CX
Timing Issues

Input Timing

The cause of problems with input timing can be explained with a rough outline of the process of receiving a mouse click. Assume that a CX program is running in a window. The user clicks inside of the window. At this point, the operating system detects that the click has occured and notes that the location of the click is within the window. The operating system then attempts to tell the program that a mouse click has occured. In order to be notified about input events like mouse click, the program has a message queue for incoming messages from the operating system. The OS puts the mouse event into the message queue. In order for the program to find out about the message it needs to check to message queue. This is what happens when CX::CX_InputManager::pollEvents() is called: The message queue is checked and all messages in the queue are processed, given timestamps, and routed to the next queue (e.g. the message queue in CX::CX_Mouse that is accessed with CX::CX_Mouse::availableEvents() and CX::CX_Mouse::getNextEvent()). The timestamps are not given by the operating system*, so if CX::CX_InputManager::pollEvents() is not called regularly, input events will be received and everything will appear to be working correctly, but the timestamps will indicate that the event occurred later than it actually did.

Of course, the actual process extends all the way back to the input device itself. The user presses the button and the microcontroller in the input device senses that a button has been pressed. It places this button press event into its outgoing message queue. At the next polling interval (typically 1 ms), the USB host controller on the computer polls the device for messages, discovers that a message is waiting and copies the message to the computer. At some point, the operating system checks to see if the USB host controller has received messages from any devices. It discovers the message and moves the message into the message queue of the program. At each step in which the message moves from one message queue to the next, the data contained in the message likely changes a little. At the start in the mouse, the message might just be "button 1 pressed". At the next step in the USB host controller, the message might be "input device 1 (type is mouse) button 1 pressed". Once the operating system gets the message it might be "mouse button 1 pressed while cursor at location (367, 200) relative to clicked window". Eventually, the message gets into the message queue that users of CX work with, in CX::CX_Mouse, for example.

This process sounds very long and complicated, suggesting that it might take a long time to complete, throwing off timing data. That is true: Input timing data collected by CX is not veridical, there are invariably delays, including non-systematic delays. However, there are many steps in the process that no experimental software can get around, so the problems with timing data are not unique to CX. It might be possible to write a custom driver for the mouse or keyboard that allows the software to bypass the operating system's message queue, but it is very difficult to avoid the USB hardware delays, which can be on the order of milliseconds for many kinds of standard input devices. If you do not use some kind of low-latency button box, my expectation is that you simply allow any error in response latencies to be dealt with statistically. Typically, any systematic error in response times will be subtracted out when conditions are compared with one another (just don't systematically use different computers for different participant groups). Any random error will simply slightly inflate the estimated response latency variance, but not to any meaningful extent given the base magnitude of human variability.

If you would like to learn more about the internals of how input is handled in CX, you can see how GLFW (the windowing system used by openFrameworks) and openFrameworks manage input by examining the source code in the respective repositories (also check out the CX source code, of course). There is no documentation on input timing that I am aware of for either GLFW or openFrameworks. For most applications, computers are fast enough that it is not an area that is emphasized by most software developers.

*Technically, on Windows the messages that are given to a program do have a timestamp. However, the documentation doesn't actually say what the timestamp represents. My searching turns up the suggestion that it is a timestamp in milliseconds from system boot, but that the timestamp is set using the GetTickCount function, which typically has worse than 10 ms precision. This makes the timestamp attached to the message of very little value. See this page for documentation of what information comes with a Windows message: http://msdn.microsoft.com/en-us/library/windows/desktop/ms644958%28v=vs.85%29.aspx. The only page on which I actually found a definition of what the time member stores is this page http://msdn.microsoft.com/en-us/library/aa929818.aspx, which gives information pertaining to Windows Mobile 6.5, which is an obsolete smartphone operating system.

Stimulus Timing

When it comes to stimulus presentation, CX, like several other pieces of psychology experiment software, uses OpenGL for visual stimuli. With OpenGL, rendering commands are put into buffers and eventually flushed to the video card for actual rendering and presentation. One function that sends the commands is called glFlush(). The OpenGL documentation has this to say about what happens to rendering commands after glFlush() is called: "Though this execution may not be completed in any particular time period, it does complete in finite time." (https://www.opengl.org/sdk/docs/man2/xhtml/glFlush.xml) That is quite a strong timing guarantee. I mean, it's a lot stronger than allowing for the possibility that the commands might never complete. But seriously, OpenGL makes no timing guarantees. You get what you get and have to empirically verify that it's all going fast enough. Thus, I cannot make any stong claims about stimulus timing in CX because I depend on something that makes no claims about timing, other than that things happen in "finite time". No one who uses OpenGL to render stimuli can make any strong claims about stimulus timing without substantial emperical validation. (As a side note, you don't need to worry about calling glFlush(); it's taken care of for you by CX.)

Although the kinds of error introduced into response time data can often be dealt with statistically, errors in stimulus presentation can be more serious. For example, if a visual stimulus is systematically presented for an extra frame throughout an experiment, then the method of the experiment has been altered without the experimenter learning about the alteration. Even if the extra frame does not always happen, on average participants are seeing more of that stimulus than they should be. An error on the magnitude of an extra frame is very hard to detect by eye in most cases, so it is important that there is some way to detect errors in stimulus presentation. The best way is with external hardware that provides timestamps giving actual stimulus onset. Barring that, there are some software mechanisms that can be used to attempt to detect errors.

The primary method of presenting time-locked visual stimuli is the CX_SlidePresenter which has built in error-detection features that pick up on certain kinds of errors. Information about presentation errors can be found by using CX::CX_SlidePresenter::checkForPresentationErrors() once a presentation of stimuli is complete. Although it is nice to be made aware of errors when they occur, it is better to not have the errors happen in the first place. For this reason, stimulus presentation in CX is designed around avoiding errors. For visual stimuli, the CX_SlidePresenter provides a very easy-to-use way to present visual stimuli. The backend code of the CX_SlidePresenter is designed to minimize the potential for timing errors by carefully tracking the passage of time, monitor refreshes, and timing of stimulus rendering.

On the audio front, CX provides the CX_SoundBufferPlayer, which plays CX_SoundBuffers. If several sounds are to be presented in a time-locked sequence, playing the sounds individually at their intended onset time can result in unequal startup delays for each sound, but if all of the sounds are combined together into a single audio buffer this possibility is eliminated. CX_SoundBuffers are designed to make combining multiple sound stimuli together easy, which helps to prevent timing errors that could have otherwise occurred between sounds. CX also includes CX_SoundStream, which provides a method for directly acessing and manipulating the contents of audio buffers that are received from or sent to audio hardware. More information about audio input and output can be found on the Audio Input and Output page.

Response Latency

Usually, we are not interested in response time (the absolute time at which a response occurred), but rather response latency of a response to a specific stimulus. However, the time at which the stimulus was actually presented may be misreported by audio or video hardware/software, so even if the response time data had no error whatsoever, when it is compared with the stimulus presentation time, the response latency (response time minus stimulus presentation time) would be wrong due to errors in measured stimulus presentation time. If the timestamps for either stimulus presentation time or response time are off, response latency will be off. No matter how low-latency of a button box you have, if stimulus presentation times are wrong, your button box will not solve the problem of measuring accurate response latencies. Based on this, it is my firm belief that the only way to accurately measure response latency is with a button box that measures actual stimulus onset time with a light or sound sensor and also measures the time of a button press or other response, with the response latency calculated based on these accurate event time measurements. If you do not have a setup like this, then just try to relax and don't worry too much about timing. The vast majority of psychological effects are not that time sensitive and standard computers provide sufficient timing precision for them. However, if what you are studying is dependent on timing, then get proper hardware and software and verify that it all works within the desired error tolerance.

Millisecond Precision

Given the timing issues outlined in this section, it seems natural to end with a brief discussion of the mythical millisecond precision of psychology stimulus software. Psychologists claim to be very concerned with obtaining millisecond precision from their equipment, yet they consistently use off-the-shelf hardware and software that are incapable of providing the desired level of precision. Then they install expensive software for psychology experiments that make claims about the submillisecond precision provided by the software. These claims are usually at best lies of omission. For example, a claim like "CX is accurate to the millisecond" is completely true in a very literal sense. However, the missing footnote on my claim is that it is trivial to write software that is accurate to the millisecond. I can write a little loop that gets a timestamp every time through the loop and show that it takes less than a microsecond to run through the loop on most computers. In that sense, CX is accurate to the microsecond. Awesome! But wait, I said nothing in my claim about controls over stimulus presentation and response collection timing. The reason I didn't is that because CX is software running in a normal operating system, it has very little control over timing. As I have outlined in this section, modern computers are almost literally Rube Goldberg machines, with multiple layers of queues that all data must pass through to get from one point to another. At each queue, there is software that is in charge of managing that queue and moving the data from that queue onward. Each step involves a delay, with both systematic and non-systematic components of the delay. As software running in an operating system, CX is far removed from each of these stages, with no ability to directly affect them. As such, it would be irresponsible to make strong guarantees about timing of stimulus presentation and response collection. Empirically verify that your hardware and software configuration meets your timing needs. I hope that CX is helpful to that end.