View source for User Reference:P3SpellerTask

==Synopsis==
The ''P3SpellerTask'' filter implements Donchin's matrix speller paradigm (Sellers, Donchin, Schalk, 2004).
Such a matrix speller presents a matrix of options; rows and columns of that matrix are flashed to elicit an evoked response  (ERP) whenever the user's attended matrix element is flashed.
From ERP classification results, the speller then derives which of the matrix elements was attended to, and executes the action corresponding to that matrix element. Typically, this action will consist in adding a character to a text window. 

Beyond the basic features, the ''P3SpellerTask'' provides the ability to handle multiple speller menus, saving and recovering the text buffer, and an option to exchange information with an external program.

Typically, the ''P3SpellerTask'' is used in conjunction with the [[User Reference:P3TemporalFilter|P3TemporalFilter]] signal processing filter.

==Functional Description==
===Introduction===

The ''P3SpellerTask'' displays matrix of characters, where the 6 by 6 matrix is the most common size (see Figure 1). 
The user's task is to focus attention on characters in a word that is prescribed by the investigator (i.e., one character at a time), and displayed on the user's screen.
Alternatively, the user may freely choose letters he attends to, using the speller as a true communication device.

All rows and columns of this matrix are successively and randomly intensified.
For a full set of intensifications of rows or columns, two contain the desired character (i.e., one particular row and one particular column).
The responses evoked by these infrequent stimuli are different from those evoked by the stimuli that did not contain the desired character, in a manner similar to the classical P300 paradigm (Donchin, 1988).

The ''P3SpellerTask'' will catch such responses and then determine the character that user had focused on, and will print the character to the screen (below the word to spell) for the user to verify.

The entire set of stimuli is usually flashed several times before the ''P3SpellerTask'' calculates the desired character.
To ensure maximum efficiency of character prediction, it is important that the stimuli are infrequent even between sequences. Therefore, the ''P3SpellerTask'' avoids starting a sequence with the row or column that was intensified last in the previous sequence.

[[Image:P3SpellerScreen.PNG|center|frame|Figure1: A 6 by 6 speller matrix. Here, the user's task is to spell the word "SEND" (one character at a time).  For each character, all rows and columns in the matrix are intensified a number of times (here, the third row is shown intensified).]]

===Visual Representation===

The visual representation is divided into three parts (Figure 2):
*''Text to Spell'' displays the text that the user needs to spell (only used in copy spelling mode).
*''Text Result'' holds the letters spelled up to the current moment.
*''Speller Display'' is the area that contains the speller matrix.  

[[Image:P3SpellerScreenElements.PNG|center|frame|Figure 2: Elements of the user's screen. ''Text To Spell'' indicates the pre-defined text. The speller will analyze evoked responses, and will append the selected text to ''Text Result''.]]

In addition to the text characters, it is possible to display icons (bitmaps) in the P3Speller matrix. This can be achieved by entering the appropriate file name for the icon in column 4 of the desired cell in the ''TargetDefinition'' parameter. The icon will be flashed (highlighted) using a number of methods (see the description of the ''IconHighlightMode'' parameter below).Alternatively, an additional version of the icon may be displayed during stimulus presentation by entering the file name in an appropriately-labeled column. Although the sixth column has the correct label by default, any column except the first five may be used provided the column label is '''Intensified Icon'''. If an intensified icon is specified, it will be displayed as the highlighted version of the icon instead of using the ''IconHighlightMode'' parameter. 

As new characters are added to the right, the ''Text Result'' area gets filled; once there is no space left, the text is scrolled to the left in order to accommodate for additional characters.

It is also possible to play a sound file or to "speak" text (using a Text-to-Speech engine) when a cell is selected. To play a sound file, the sound file name should be entered in column 5 of the desired cell in the target definition matrix. To enable a text-to-speech function, the text to be "spoken" should be entered in column 5 of the desired cell in the target definition matrix within single quotes (e.g. 'text'), as shown below.

[[Image:P3SpellerCommandExample.PNG|center|frame|TargetDefinitionMatrix]] 

The Text-to-Speech engine uses the system's "default voice" setting (Control Panel->Speech->TextToSpeech->Voice Selection).  Male or female voices may be selected.

===Nested Matrices Howto===
To allow for more flexibility in real-world applications, multiple speller matrices can be specified, and may be traversed in a "nested" manner.
For example, one of the cells in the first matrix may be tied to another matrix which is displayed when this cell is selected by the user. 
This capability may be used to design a menu based user interface.

To use the nested menu capability, the following steps need to be executed:

====Set up the Target Definition matrix====

[[Image:NestedMatricesWindow.PNG|center|frame|Figure 3: Target Definition matrix for 4 nested menus]]

To maintain backward compatibility and for ease of configuration, the application will not support nested functionality by simply converting one cell of the 'old' target definition matrix to a sub-matrix. To define nested matrices, the ''TargetDefinition'' parameter must be defined with 1 column and each cell as a sub-matrix. If it has more than one column, the application will treat it as a single matrix and will ensure that each of its cells is defined as a single value.

====Convert each cell to a sub-matrix====
In the GUI, this is done by right-clicking on each cell as shown in Figure 4, and choosing "Convert to sub-matrix" from the context menu.

[[Image:NestedMatricesConvert.PNG|center|frame|Figure 4: Converting cells to sub-matrices]]

====Configure individual sub matrices====
Each cell in a nested matrix is a sub-matrix (as shown in Figure 3) and needs to be configured individually.  
In the GUI, clicking on a cell that is a sub-matrix will bring up that matrix in another window, as shown in Figure 5.

[[Image:NestedMatricesSubmatrix.PNG|center|frame|Figure 5: Configuring individual sub-matrices]]

Each sub-matrix should be configured to have a minimum of 3 columns and can have up to 5 columns if it needs to display icons or play sounds.

====Enable transition from one menu to another====
Special control codes (speller commands) allow transition from one matrix to another.
To go to a different matrix when an item is selected, enter the <GOTO#> command into its "Enter" column, replacing "#" with the (one-based) index of the corresponding matrix (menu).
The <BK> command will go back to the previous menu/matrix (see Figure 5).

====Number of Rows and Columns of each nested menu (matrix)====
The number of rows and columns in each sub-matrix needs to be entered into the ''NumMatrixRows'' and ''NumMatrixColumns'' parameters.  There, individual entries are separated by space characters. 

[[Image:NestedMenusRowsColumns.PNG|center|frame|Figure 6: Rows and Columns of each sub-matrix]]

Figure 6 indicates that the first three sub-matrices are 2x2, while the sub-matrix at index 4 is a 6x6 matrix.

In case of a single (non-nested) matrix, there will be only one entry in the NumMatrixRows and NumMatrixColumns parameters.

====Select first menu to be displayed====

The index of the menu to be displayed first (i.e. when the application is started) is entered in the ''FirstActiveMenu'' parameter. In case of non-nested matrix configuration, this parameter should be left at its default value of 1.

The overall appearance of a nested matrix configuration is shown in Figure 7 below.

[[Image:P3SpellerMultipleMenus.PNG|center|frame|Figure 7:  Nested matrix Configuration]]

===Testing Matrix Menus===
In order to test matrix menu configurations, the ''P3SpellerTask'' may be put into a [[#TestMode|test mode]].
In this mode, mouse clicks into matrix elements are registered during sequence presentation, and force selection of the respective matrix element.

''Please note that the [[#DisplayResults|DisplayResults]] parameter needs to be turned on in order to be able to select sub matrices.''

===Pause and Sleep===
There are two speller commands that allow the user to suspend operation of the ''P3SpellerTask'' either momentarily or for an extended period of time, <PAUSE> and <SLEEP>.
To associate speller commands with certain matrix elements, specify them in the "Enter" column of the desired matrix element in the target definition matrix.

The <PAUSE> speller command pauses the ''P3SpellerTask'': While the system is paused, the matrix will continue to flash but target selections shall be ignored until the user resumes system operation, which is achieved by selecting <PAUSE> again. 
The "goal text" line of the status bar indicates that the system is paused.
Data recording continues while the system is paused.

For a matrix having N entries, one of which is <PAUSE>, the chance to erroneously resume system operation while paused is 1/N per selection. On average, this will happen after N selections in paused state. A second speller command, <SLEEP>, is provided as a safer option. Once in sleep state, the system will resume only after receiving two consecutive <SLEEP> selections, which requires N^2 selections on average to occur by chance.
When in sleep mode, the "goal text" line of the status bar will instruct the user to select the <SLEEP> command twice to restart.
Data recording is also suspended while the system is sleeping.

===Text Window===
The capability to display user selected text in a text window is available.
This feature can be activated by selecting the ''TextWindowEnabled'' parameter. 
The text window can be enabled only in online (free spelling) mode. 
Position and size of the window, as well as display font, are configurable.

When the text window is enabled, any text that the user selects will appear in the text window, in addition to the text result area of the ''P3SpellerTask'' display. The text window will scroll automatically.

Two speller commands are available to perform Save and Retrieve operations on the text window. When the <SAVE> command is selected, the text in the text window will get written to a file and be erased from the window. The file name will be auto generated with the date and time stamp. The directory to which the file gets sent is configured in the ''TextWindowFilePath'' parameter.  The <RETR> (retrieve) function reads the latest file that was saved by the user and recalls the text into the text window. 

A separate "clear" command has not been provided in the text window functions to prevent unintentional deletion of its contents. The only way to clear the contents of the text window is using <SAVE>.

==Parameters==
{{ApplicationWindowParams}}
{{StimulusTaskParams}}

====NumberOfSequences====
The number of intensification sequences performed prior to each classification (selection of matrix elements).
For an NxM speller matrix, a single intensification sequence comprises N+M intensifications, one for each row, and one for each column.
Usually, this parameter is set to the same value as the P3TemporalFilter's [[User Reference:P3TemporalFilter#EpochsToAverage|EpochsToAverage]] parameter.

====TargetDefinitions====
A matrix defining elements of the speller matrix. 
Each row corresponds to a single matrix element; matrix elements are enumerated row-wise, beginning with the top left matrix element.

For each matrix element, the value in the first column, labeled '''Display''', contains a text string to be displayed in the matrix element, i.e. the matrix element's caption.

The second column, labeled '''Enter''', specifies the speller action to be performed upon the item's selection; in most cases, this action consists in entering a text string, and is specified by that string.
E.g., for the top left matrix element to display the caption "A", and also enter the letter "A" on selection, both the ''Display'' and ''Enter'' columns will contain the letter "A".

The third column is labeled '''Size'''. It allows to specify an individual size for the matrix element, relative to the size of the other matrix elements.

The fourth and fifth columns, labeled '''Icon File''' and '''Sound''', respectively, contain the path to an icon file to be displayed, and a sound to be played, when the matrix element is selected. For icon files, common image formats are accepted. Sounds are given as a path to a sound file, or as a text enclosed in single quotes. On Windows, when a text is given, it will be spoken using the system's Text-To-Speech engine. 
{{FilePath}}

The sixth column is labeled '''Intensified Icon''' by default, and may contain the path to an additional icon file to be displayed as the highlighted version of an icon. Except for the first five columns, any column may be used for this purpose as long as it is labeled appropriately. Neither the text string nor the icon will be visible underneath the intensified icon during presentation.

=====Speller Commands=====
Speller commands are specified in the ''TargetDefinitions'' matrix' second column.
Speller commands may be sequences of characters, which are added to the speller text when the respective item is selected. Additionally, speller commands may be ''speller control commands'' enclosed in a pair of <> characters.
Any combination of characters and commands is allowed, and will be executed in sequence.

Available '''speller control commands''' are
*<BS> (backspace) -- delete the last character from the current text.
*<DW> (delete word) -- delete the last word from the current text.
*<UNDO> -- undo the effect of the previous speller action.
*<END> -- end spelling, put BCI2000 in suspended mode.
*<SLEEP> -- suspend spelling; resume when <SLEEP> is selected two additional times.
*<PAUSE> -- suspend spelling; resume when <PAUSE> is selected again.
*<GOTO x> -- go to speller menu #x (see [[#Multiple Menus|Multiple Menus]]).
*<BACK>, <BK> -- return to the previously active speller menu.
*<SAVE> -- write the text window's content into a file located at [[#TextWindowFilePath|TextWindowFilePath]].
*<RETR> -- load the text window's content from the most recently saved file.

=====Multiple Menus=====
The ''P3SpellerTask'' allows you to specify a number of speller menus, and to switch between them using the <GOTO> and <BACK> speller commands.
For multiple speller menus, the ''TargetDefinitions'' matrix needs to be configured as a list of matrices, rather than a single matrix.
Then, each submatrix should have the form described above.
Each of the submatrices may have an individual number of rows and columns, and its own set of matrix elements.

Also, switching to Multiple Menus implies additional entries in the following parameters:
*NumMatrixColumns, NumMatrixRows
*AudioStimuliRowsFiles, AudioStimuliRowsFiles (additional columns)
*TargetWidth, TargetHeight, TargetTextHeight, BackgroundColor
*TextColor, TextColorIntensified, IconHighlightMode, IconHighlightFactor

====NumMatrixColumns, NumMatrixRows====
The number of columns/rows in the speller matrix.
For [[#Multiple Menus|Multiple Menus]], a list of numbers representing each menu's number of columns/rows.

====AudioStimuliOn====
Switches playback of audio stimuli on or off.

====AudioStimuliRowsFiles, AudioStimuliColsFiles====
Each of these parameters is a single-column matrix specifying audio files associated with speller rows or columns, respectively.
Whenever a row/column is highlighted, the associated audio file is played back.
For audio files, Windows WAV format is expected.
{{FilePath}}
Rather than the path to an audio file, text may be given, enclosed in single quotes. In this case, the text is rendered using the system's Text-To-Speech engine.

====TargetWidth, TargetHeight====
A single matrix element's width/height in percent of screen width/height.

====TargetTextHeight====
Matrix elements' text height in percent of screen height.

====BackgroundColor====
Matrix elements' background color, given in RGB encoding.

====TextColor, TextColorIntensified====
Text color in standard and highlighted (intensified) mode, given in RGB encoding.

====IconHighlightMode====
An enumerated value specifiying how icons are highlighted on stimulus presentation:
*0 Show/Hide: icons are only visible during stimulus presentation,
*1 Intensify: highlight by increasing icon brightness,
*2 Grayscale: display a grayscale version during stimulus presentation,
*3 Invert: invert color/brightness values during stimulus presentation,
*4 Dim: decrease brightness during presentation.

This value won’t be used if an intensified icon is specified.

====IconHighlightFactor====
If ''IconHighlightMode'' is 1 or 4, this parameter defines the brightness scaling factor.
Dimming is equivalent to intensifying with a scaling factor less than 1.

====FirstActiveMenu====
For multiple menus, the index of the menu that should be active at the start of a run.

====StatusBarSize, StatusBarTextHeight====
The size and text height of the status bar in percent of screen height.
The status bar is located on top of the screen, and displays a line of text currently spelled.
In [[#InterpretMode|copy mode]], it also displays the text that the user is supposed to spell.

====TextToSpell====
In [[#InterpretMode|copy mode]], a string of characters defining the text to be spelled by the user.
This text is displayed in the status bar, above the actually spelled text.
From the difference between ''TextToSpell'', and the actually spelled text, the speller automatically derives which matrix element the user will need to select next.
This information is then used to set the [[#StimulusType|StimulusType]] state.

====TextResult====
At the beginning of a run, this parameter's content is copied into the lower part of the status bar.
At the end of a run, the status bar's content is copied back into this parameter.

====TestMode====
If this is switched on, clicking on a matrix element with the mouse will select it once the current sequence of intensifications is finished.
This is useful to test speller matrix configurations.

====DestinationAddress====
A network address to receive speller output, given in IP:port format, e.g. <tt>localhost:3582</tt>.
To this address, a UDP socket is opened, and written into, such that an external application may connect and receive information about selected matrix elements.
For each selection, the speller will write whatever is contained in the selected matrix element's ''Enter'' field, preceded with <tt>P3Speller_Output</tt> and a space character, and followed with a <tt>\r\n</tt> sequence (i.e., an MSDOS style line ending). E.g., the output will be <tt>P3Speller_Output A\r\n</tt> when a matrix element is selected that enters the letter "A", and <tt>P3Speller_Output <BS>\r\n</tt> for a matrix element corresponding to the backspace command.

====TextWindowEnabled====
If this flag is switched on, a separate window is displayed.
Once the status bar is filled, text flows into the window, and back in case of text deletion.

====TextWindowLeft, TextWindowTop, TextWindowWidth, TextWindowHeight====
Position and dimension of the text window in pixels.

====TextWindowFontName, TextWindowFontSize====
Text window font name and size.

====TextWindowFilePath====
A (relative or absolute) path to a directory.
Upon the <SAVE> and <RETR> speller commands, the text window's contents are saved to/retrieved from a file located in that directory.
Repeated <SAVE> commands do not result in overwriting existing files.
Rather, existing files are preserved, and the most recent file's name is written into a pointer file.
{{FilePath}}

==States==
{{ApplicationBaseStates}}
{{StimulusTaskStates}}

===SelectedTarget, SelectedRow, SelectedColumn===
Upon classification, these states are set to the selected target's ID, and its associated row and column, respectively.
A target's ID matches its row number in the ''TargetDefinitions'' matrix.
===SpellerMenu===
This state contains the one-based index of the current speller menu. It may be set to a different value in order to switch between menus.

==Timeline==
{{StimulusTaskTimeline}}

==See also==
[[User Reference:P3TemporalFilter]], [[User Reference:StimulusPresentationTask]], [[Programming Reference:StimulusTask Class]]

[[Category:Filters]][[Category:User Application]][[Category:Specification]]