User Reference:BCI2000Certification: Difference between revisions

From BCI2000 Wiki
Jump to navigation Jump to search
Line 278: Line 278:
== Troubleshooting ==
== Troubleshooting ==
Things to watch out for:
Things to watch out for:
*<tt>BCI2000_Certification</tt> assumes BCI2000 path is <tt>C:\BCI2000\</tt>.  Please set your BCI2000 path to this.
*Parameter files for gUSBamp with channels greater than 16 have pre-defined amplifier IDs.  Please set these to your amplifier IDs.
*Parameter files for gUSBamp with channels greater than 16 have pre-defined amplifier IDs.  Please set these to your amplifier IDs.
*BCI2000Certification runs BCI2000 applications from the working directory: <tt>C:\BCI2000\tools\BCI2000Certification</tt>. The images folder there doesn't contain all required images for <tt>CursorTask</tt> or a sounds folder for "StimulusPresentation".  If necessary, paste the images and sounds folder from the <tt>BCI2000/prog</tt> directory into the <tt>BCI2000/tools/BCI2000Certification</tt> directory - overwriting what is currently in that directory.
*BCI2000Certification runs BCI2000 applications from the working directory: <tt>C:\BCI2000\tools\BCI2000Certification</tt>. The images folder there doesn't contain all required images for <tt>CursorTask</tt> or a sounds folder for "StimulusPresentation".  If necessary, paste the images and sounds folder from the <tt>BCI2000/prog</tt> directory into the <tt>BCI2000/tools/BCI2000Certification</tt> directory - overwriting what is currently in that directory.
*Removing tasks from BCI2000 Certification using the "-" button causes an access violation.  This does not affect program execution, and can be safely ignored.
*If you change the global Signal Source in the GUI, it isn't saved into .ini file.  This can be easily changed using a text editor and manually editing the .ini
*Older versions of <tt>BCI2000_Certification</tt> required the Borland runtime.  This has since been fixed.
*Older versions of <tt>BCI2000_Certification</tt> required the Borland runtime.  This has since been fixed.

Revision as of 02:08, 13 February 2010

Introduction

BCI2000 v2.0 and higher includes a certification procedure that can determine whether a computer system is capable of running all or some BCI2000 applications. Because BCI2000 does not have a standardized hardware configuration, i.e., it can run on potentially any PC, this testing procedure is capable of running on any PC with BCI2000, with the results reporting which standard BCI2000 applications meet the minimum timing requirements. Different configurations are included in the certification procedure to test a range of likely setups for a given application, in which timing characteristics such as the sampling rate, inter-stimulus intervals, and stimulus duration, and other parameters, such as the number and size of the stimuli, are changed for each configuration.

This document serves as the instruction manual for using the certification procedure, and describes how the procedure works to determine the timing characteristics of the BCI2000 system. The primary use for the certification procedure is to determine if the PC configuration used with BCI2000 is capable of running the core BCI2000 system, which is comprised of:

  • Support for the the g.tec series of amplifiers, including the g.USBamp, g.MOBIlab, and the g.MOBIlab+ (bluetooth) amplifiers.
  • The AR, P3, FFT, and Matlab signal processing modules.
  • The CursorTask, P3Speller, and StimulusPresentation application modules.

Additionally, due to the extensibility of BCI2000, researchers who develop custom algorithms and applications, or use amplifiers not made by g.tec, can use the certification procedure to determine whether their custom modules are capable of running on a particular PC configuration.

The certification procedure works by comparing the differences in time of when BCI2000 expects an event to occur to when that event actually occurred. These events include a change in the display, an audio output through the system speakers, or an EEG sample being stored in the amplifier buffer. The times at which such events occur are recorded with the amplifier and stored in a data file. Changes in the display are found using an optical sensor in combination with the g.TRIGbox. The g.TRIGbox is a signal conditioner that generates trigger pulses for various sensors or input signals. When the sensor exceeds a threshold, the g.TRIGbox outputs a digital signal which is recorded on the amplifier. A similar procedure is used for audio detection: the sound output from the PC is input to the g.TRIGbox, and when the volume exceeds threshold, a high digital signal is output and recorded by the amplifier. This information can be used because BCI2000 stores a time-stamp for every event during an experiment. The difference between a time-stamp value and the time that the recorded event changes values are compared to find the latency for that particular event.

These latencies can be affected by many external influences, including hardware deficiencies (such as low RAM, slow CPU speed, or a video card of low quality), and other software running in the background while the experiment is being run.

This document serves as the BCI2000Certification manual. A discussion and derivation of the specific system latencies are found HERE.

Compiling the BCI2000Certification Programs

The BCI2000Certification procedure actually consists of two separate programs, including a graphical user interface that simplifies the task of selecting and starting tests, and a program that performs the data analysis and returns results. These programs are compiled automatically with the other BCI2000 programs. Compilation is only supported in BCI2000 v3.0 with the Visual Studio or MinGW compilers. Instructions can be found at Programming_Howto:Building_BCI2000

Using the BCI2000Certification Procedure

Starting the Tester User Interface

  1. The compiled programs can be found in BCI2000/tools/BCI2000Certification.
  2. Navigate to this folder, and double-click the BCI2000CertificationTester.exe file.
  3. The certification tool opens, as below.

This program shows a list of all of the procedures to be tested for the loaded configuration. Detailed instructions for using the interface follow.

The BCI2000Certification user interface.
The BCI2000Certification user interface.

Loading Configuration Files

The BCI2000 certification procedure uses different combinations of BCI2000 parameter fragments to control the tests. While it is possible to edit these by hand, in most circumstances it is simpler to use the graphical user interface provided.

  1. In the BCI2000Certification tool, select File->Open *.ini
  2. Navigate to the folder BCI2000/tools/BCI2000Certification.
  3. Load the file named BCI2000Certification_16ch_100ms.ini if using a 16-channel system, or BCI2000Certification_32ch_100ms.ini if using 32 channels. (This tutorial assumes a 16-channel system).
  4. If using more than 16 channels, then the amplifier parameter files must be modified.
    1. Navigate to BCI2000/tools/BCI2000Certification/parms/, and open the file Certification_gUSBamp_24.prm in Notepad.
    2. Find the parameter titled DeviceIDMaster, and replace the existing device ID (e.g., UA-2007.04.12 with the ID of your master device.
    3. Next, find the parameter titled DeviceIDs, and replace the IDs with the IDs of your amplifiers. The first in the list should be identical to the DeviceIDMaster value. For example, you might enter: DeviceIDs= 2 UA-2007.04.12 UA-2006.10.13 (the \#2 corresponds to the number of devices being used).
    4. Repeat these steps for the file Certification_gUSBamp_32.prm.
    5. Note that these steps are not necessary if only one g.USBamp is being used.
  5. All 100 tests in this configuration are loaded into the GUI.

User Interface Components

This section describes each user interface component, referring to the figure below.

The BCI2000Certification user interface components.

1. Configuration: Task List

The task list shows all tasks that have been loaded from the configuration file, or created within the GUI. To run a specific test, it should have a check mark next to it. In order to select or deselect all tasks, press the Select All check box at the bottom of the list.

It is possible to create new tasks, remove tasks, and copy tasks within the interface as well. To create a new task, press the + button. A new task is created a the bottom of the list; the task details must be filled in (described next). It is often easier to copy a similar task than to create a new one from scratch. To do so, select the desired task, and press Copy. The new task should be renamed appropriately. To remove a task, select a task and press the - button.

2. Configuration: Task Details

The right side of the GUI displays the details of the currently selected task. Each option is described here.

Task Name
The name of the task. This must not contain any spaces, and must be unique.
Signal Source
The amplifier that should be used, if different from the amplifier set in the global options. Generally, this should not need to be used, since all tasks should use the same amplifier.
Signal Processing
The signal processing module that should be started and used for the given task. This should usually be either ARSignalProcessing, P3SignalProcessing, or DummySignalProcessing, depending on the specific application.
Application
The application that is run for the task. This will typically be CursorTask, P3Speller, or StimulusPresentation
Parameters
A list of the parameter fragments that should be used for this task. Typically, at least two parameter files are required. The first should be the amplifier-specific fragment that configures the amplifier for the correct number of channels. The second should be the application-specific fragment that configures how the application appears on the screen. Parameter files can be added and removed with the + and - buttons, respectively. All files should be placed in the BCI2000/tools/BCI2000Certification/parms folder.
Sample Rate
The sampling rate that should be used for this task. Note that it must be supported by the amplifier used.
Sample Block Size
The sample block size (in samples). In the provided parameter files, the sample block size is always equal to 100ms, regardless of the sampling rate.
Amp Channel
This is the analog channel that will contain the amplifier digital output (i.e., the digital output recorded back in on an analog channel). This signifies the onset of acquisition of a block. This value should be between 1 and the number of channels.
Digital Amp Channel
This is the digital channel that contains the amplifier digital output (i.e., the digital output recorded back in on a different digital input channel).
Video Channel
The recorded channel containing video event markers.
(Video) State
The BCI2000 state that contains the event markers for the video event, e.g., TargetCode or StimulusCode.
(Video) State Values
A list of state change values that should be detected by the analysis. For example, if values 4 11 are used, then the analysis tool will examine when the StimulusCode state changes to either 4 or 11, and then examine when the Video Channel changes.
Audio Channel
The recorded channel containing audio event markers.
(Audio) State
The BCI2000 state that contains the event markers for the audio event, e.g., TargetCode or StimulusCode.
(Audio) State Values
A list of state change values that should be detected by the analysis. For example, if values 4 11 are used, then the analysis tool will examine when the StimulusCode state changes to either 4 or 11, and then examine when the Video Channel changes.

3. Control: Global Settings

The BCI2000Certification user interface.

As the name suggests, the options under Global Settings are applied to all tasks, and are described here.

Window Left, Top
The position of the left and and top sides of the application window. If using multiple monitors, this position should take this into account. For example, if the application window is to the right of the main monitor, and the main monitor has a resolution of 1280x1024 pixels, then Window Left should be set to 1280, and Window Top to 0.
Window Width, Height
The width and height of the application window.
Global Signal Source
The executable that should be used by all applications. This value is overwritten by the Signal Source value set for individual tasks.
Data Output Directory
The directory where all of the data should be saved during the task. This will appear relative to BCI2000/tools/BCI2000Certification/, or at a given absolute path. NOTE that if data already exists in this directory, additional data will be saved to it! To avoid combining new data with earlier results, the existing directory with the same name should be renamed or deleted.
BCI2000 Directory
This is the directory containing the BCI2000 program files, e.g., C:\BCI2000\prog

4. Control: Controls

This panel handles starting and stopping the procedure, and running the analysis.

Auto Set Config
Automatically press Set Config upon starting BCI2000. This is required for complete automation of the procedure.
Auto Start
Automatically start the test after Set Config is pressed. This is required for complete automation of the procedure.
Auto Quit
Automatically quit BCI2000 after each task. This is required for complete automation of the procedure.
Start/Cancel
Starts the selected tasks. While running, this button changes to Cancel, which will stop the procedure after the current task finishes.
Analysis
Start the analysis tool. This uses the directory entered in the Data Output Directory setting. See the documentation for the analysis tool for more information on the analysis procedure.
Progress
The progress bar and text provide information about the current running task, and how many tasks remain.

Setting Up and Running the Certification Procedure

  1. Load the appropriate configuration file, based on the number of amplifiers that you will be using and as described above.
  2. Plug-in and turn on the amplifier(s).
  3. Configure the multi-monitor system, if applicable.
  4. Enter the appropriate screen dimensions under Window Width, etc.
  5. Prepare to place the optical detector on the monitor by using a ring of double sided tape. The exact placement on the monitor will depend on the tasks used, and will be determined shortly. Plug the detector into the Optical Input A on the g.TRIGbox.
  6. Using a 1/8th in cable, connect the headphone output on the PC to the High Level input B of the g.TRIGbox.
  7. Next, the connectors should be configured. The procedure for creating connector cables is described in detail later in this document. To summarize the procedure for the g.USBamp, and as shown in the figure below:
    1. Using the digital I/O breakout connector for the USBamp, tie pin 3 (digital out 1) to pin 4 (digital in 0).
    2. Connect the digital out 1 female connector to a male connector, and plug the signal wire of the male connector into channel 1 of the amp.
    3. Connect output A of the g.TRIGbox to channel 2 of the amplifier.
    4. Connect output B of the g.TRIGbox to channel 3 of the amplifier.
    5. Using several EEG jumper wires, jumper the 3 ground cables together and input them to the Ground and Ref inputs of the amplifier.
  8. Next, the location to place the optical detector will be determined. The first task will be run, showing the location on the screen to place the connector.
  9. Press the Start button. BCI2000 will start shortly, and the application window will appear. In this test, the P300 Speller application is shown first.
  10. Place the detector over the "X" on the screen.
  11. Adjust the threshold on the g.TRIGbox for connector A so that it blinks on and off with the appearance of the "X". It may be necessary to turn off the lights in the room so that ambient light does not interfere with the optical detector.
  12. When finished placing the detector and setting the threshold, press Cancel in the BCI2000Certification GUI, and then Quit BCI2000. Navigate to the location of the Data folder (e.g., BCI2000/tools/BCI2000Certification/) and delete the data folder.
  13. Make sure that the computer volume is turned all the way up, and not muted.To set the audio threshold, set audio volume on the computer to maximum. On Windows, to make this process easier in the future, go to the Start menu, click the Control Panel, and go to Sounds. In the Sound window, click the box that puts the volume icon in the task bar. A small speaker should now appear in the bottom-right of the screen in the task bar. Click this icon, and slide the volume up to the max; when you click on this slider, Windows emits a short beep to give an idea of the volume level, which can be used to set the threshold. To do so, make sure that the stereo audio cable is plugged into the g.TRIGbox, and click the slider to emit a sound, checking that the green LED is flashing when you click the slider. If it is not, turn the threshold knob for channel B counter-clockwise, and click the slider again. If it is still not working after several tries, unplug the audio cable from the computer, and check that you can hear the beep through the speakers. If you cannot, make sure that the sound is not muted, and that the sound drivers are properly installed. If you can hear it, plug the audio cable back into the computer audio output (NOT the microphone input), and make sure the g.TRIGbox has power; the light next to the power switch should flash momentarily if the battery levels are OK. You can also use a wall power source (see the g.TRIGbox manual for more information). The threshold is set correctly when the channel B LED is off at rest, and is on when a sound is played.
Connectors schematic.
  1. You are now ready to run all of the tests. To do so, press the Start button on the GUI.
  2. While testing, some tasks may not complete correctly, either because they are too computationally intensive for the selected computer system, or some other mistake or artifact occurred during the test. In this case, BCI2000 can be Quit manually, and the remaining tasks will be run. At the end of all tasks, you can then de-select all tasks, and re-check just the ones that need to be completed. It is necessary to delete data files in the data folder from unusable runs prior to running them again, so that the bad data is not included in the analysis.
  3. When all tasks are complete, press the Analyze button to analyze the collected data. This procedure is detailed later in this document.

Analyzing Data

The BCI2000 analysis tool.

Simply pressing the Analyze button in the Tester will start the Analysis program. It is also possible to run the program in the BCI2000Certification directory by starting BCI2000CertificationAnalysis.exe.

  1. If using the GUI to analyze data, be sure to save the current *.ini file in the BCI2000Certification program prior to starting the analysis. The analysis program uses the information in this file to determine how to analyze and interpret the data files.
  2. If the tasks have been altered in the BCI2000Certification program, go to File->Save *.ini, and create a new ini file. We recommend not overwriting the existing configuration files, and instead creating new ones if necessary.
  3. The analysis program will open in a new console window. The appropriate folders and files must be entered for this session.
    1. Press the "+" button to select the folder containing your data files. When you select the folder, all data files contained within its subfolders are added automatically.
    2. Select the *.ini file used for this session.
    3. The configuration file should be left as-is, i.e., as BCI2000Certification.cfg.
    4. Finally, set the directory for the output file. Typically, this should be the same folder that contains all of the data files.
  4. When the parameters have been correctly configured, press "Run" to start the analysis, which can take several minutes.
  5. When it is finished, the results are saved to a text file, which can be found in the BCI2000 Certification folder. The file name will have the format results_(date)_(time).txt. Additionally, this contents of this file will be opened automatically at the end of the process in a new window.
BCI2000 analysis results.

Interpreting Analysis Results

The results text file contains detailed results for all tasks. A sample output for a single task is shown here:

File	Task	Pass/Fail	Avg(ms)	Med(ms)	Std(ms)	Min(ms)	Max(ms)
[0] CursorTaskLow_1200_16ch_100msS001R01.dat	Duration=23.00s
	Amp     : pass		1.80	1.67	0.91	0.83	10.00
	DigAmp  : pass		0.16	0.00	0.98	0.00	9.17
	VidSys  : pass		39.77	38.33	10.72	20.00	62.50
	Jitter  : pass		0.00	0.00	1.38	-9.00	9.00
	ProcLat : pass		13.77	14.00	0.52	13.00	15.00
	VidOut  : pass		23.80	22.37	10.72	4.04	46.54
	Overall	PASSED

On the far left, the task number is shown in square brackets (e.g., "[0]"). In the next column, the specific latencies measured for this task are listed (e.g., "Amp", "DigAmp", etc). In the next column, the latency is given a "pass" or "fail." Finally, the next columns list the mean, median, standard deviation, minimum, and maximum latencies in ms. The last row ("Overall") shows whether the entire task passed or failed.

At the bottom of this file, more results are shown, e.g.:

-------------------------------
Video Output Latency: 17.58 +/- 11.83 ms
Audio Output Latency: 62.12 +/- 10.24 ms

-------------------------------
79/100 tests passed.

Warning: the following test configurations did not pass:
 32 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99

The overall video and audio output latencies means and standard deviations for the system are shown, determined from all of the tasks combined. Next, a summary of the tests that passed are shown ("79/100 tests passed"), and warnings are provided for the tests that did not pass.

A Note on "Pass/Fail"

The values that determine whether or not a task passes or fails are determined somewhat arbitrarily. For example, we determined that an audio output latency of more than 50 ms is unacceptable for psychophysical research, e.g., BCIs. In the example shown, the audio output latency was ~62 ms for all tasks with audio, and thus these tasks registered as a "fail." However, it is ultimately up to the researcher to determine what is acceptable system timing; this method provides a method of quantifying the system latency values to simplify the process of making these decisions.

Assembling Required Cables

Several cables must be assembled in order to use the system. The final system schematic can be seen above, in the Connectors Schematic.

Amplifier Input

Parts List:

  • 3 Male Mono Audio Connectors
  • 6 12in+ Safety Connector Wires (to USB amplifier input), different colors
  • Soldering Iron and Solder

This section describes how to assemble cables that connect the output of the g.TRIGbox to the amplifier input channels.

  1. Strip the free ends of the 6 safety connector wires (~1/4 in)
  2. Take one wire, and solder the wire to the "signal" pad of one of the mono audio connectors.
  3. Solder another wire to the "ground" pad of the same connector.
  4. If necessary, pass the wires through end of the connector that screws in to cover the connection (see below, in the g.USBamp Amplifier Plug).
g.USBamp Amplifier Plug

Each connector now has a signal and ground wire that can be connector to the g.USBamp inputs, or any amplifier that accepts the standard EEG connector.

Digital I/O Cable

Parts List:

  • 3 Colored Electrical Wires
  • 7-Pin Digital Connector from g.tec
  • Soldering Iron and Solder
  • Silicone or electrical tape for insulation
  • 1 Female Mono Audio Plug

In order to access the digital I/O pins on the g.USBamp, a connector needs to be assembled. The 7-pin digital I/O connector should be ordered through g.TEC, the manufacturer of the g.USBamp.

At least three wires are needed for the circuit, which connect to the digital output 1 pin, the digital input 0 pin, and the ground pin.

g.USBamp Digital I/O Pins
g.USBamp Digital I/O Pins
  1. Strip the ends of the 3 wires (~1/4 in)
  2. Take one wire, and solder the wire to the "signal" pad of the female mono audio connectors.
  3. Solder another wire to the "ground" pad of the same connector.
  4. If necessary, pass the wires through end of the connector that screws in to cover the connection (see below, in the g.USBamp Amplifier Plug).
  5. Solder the ground wire to the center ground pin on the digital I/O connector.
  6. Locate the top of the connector by locating the small plastic notch, and locate pins 3 and 4 relative to the notch.
  7. Solder the signal wire from the mono connector to pin 3 (digital out 1)
  8. Finally, solder the remaining wire (it has not been connected to anything yet) between pin 3 and pin 4. This connects the digital output directly to the digital input.
  9. It may be helpful to insulate the wires from the casing using either a piece of black electrical tape, or an insulating gel, like silicone. If a gel is used, be sure to let it set for several hours or overnight. It is also probably a good idea to check the electrical connection with a multimeter prior to insulating.
  10. Next, attach the rubber sleeve over the end of the part with the female threads, as shown in the panel.
  11. Slide the smaller metal sleeve over the wires, then slide the larger rubber sleeve over the wires.
g.USBamp Digital Wire Plug
  1. For the next step, attach the two metallic pieces that surround the cables, shown laying to the sides in the figure below.
  2. One of the metal pieces has a small groove in it, which should be placed over the notch on the connector; the other metal piece fits on the opposite side of the connector.
  3. Push the small metal sleeve up to the side piece with the groove, and rotate the sleeve until the notch fits in the groove.
  4. Place the end piece over the connector, with the red dot on top and aligned with the connector notch.
  5. Finally, slide the rubber sleeve over the connector, and screw the sleeve into the end piece.
g.USBamp Digital I/O Cable

The plug is assembled next, similar to the. Solder the signal wire from Pin 3 (Digital Out 1) on the digital I/O connector to the signal connector on the female mono plug, and solder the ground wire to the outer ground connector on the plug. The below figure shows this; in this figure, the red wire is the signal wire, and the brown wire is the ground.

Troubleshooting

Things to watch out for:

  • Parameter files for gUSBamp with channels greater than 16 have pre-defined amplifier IDs. Please set these to your amplifier IDs.
  • BCI2000Certification runs BCI2000 applications from the working directory: C:\BCI2000\tools\BCI2000Certification. The images folder there doesn't contain all required images for CursorTask or a sounds folder for "StimulusPresentation". If necessary, paste the images and sounds folder from the BCI2000/prog directory into the BCI2000/tools/BCI2000Certification directory - overwriting what is currently in that directory.
  • Older versions of BCI2000_Certification required the Borland runtime. This has since been fixed.