User Reference:Command Line Processing

2011-01-20T06:24:45Z

Cdasalla: /* Processing data with parameters different from the ones contained in the file */

==Introduction==

Command line interfaces are a powerful tool for dynamically combining pieces of software which are typically small programs optimized for performing a very limited functionality. With a command line interface, data processing is usually performed ''sequentially'' on a stream of data. This allows the processing of indefinitely large amounts of data while consuming but a finite, and often very small, amount of memory.

Moreover, combining command line programs to perform complex tasks can be done interactively by entering complex commands directly and also in a "scripting" manner, by entering sequences of commands into text files which allow for iteration and branching. That way, automation of complex tasks can be achieved by rather simple means.

With the BCI2000 command line interface, BCI2000 data files can be

*''converted'' into various formats, ready for further processing with available software tools, or for visual inspection in human readable format;
*''processed'' off-line with literally the same code as used in the on-line system.

Together, these two concepts provide versatile access to recorded data in all stages of processing.
The BCI2000 command line interface may prove useful for

*automated ''analysis'' of large amounts of recorded data,
*''development'' of new filter classes,
*''verification'' of external off-line analysis methods.

This document explains the command line tools available for working with BCI2000 files, and how to build BCI2000 filters as single executables that may be combined to form a chain of filters.

==Command line interfaces==

Basically, the interface of a command line tool consists of

*an input stream <tt>stdin</tt>,
*an output stream <tt>stdout</tt>,
*an error stream <tt>stderr</tt>,
*optional arguments (switches) controlling details of its behavior.
The most important feature of a command line interface is its ability to ''redirect'' streams.

*<tt>stdin</tt> can be redirected to read from a file using the <tt><</tt> operator;
*<tt>stdout</tt> can be redirected to write to a file using the <tt>></tt> operator;
*one program's <tt>stdout</tt> can be plugged into another program's <tt>stdin</tt> using the <tt>|</tt> operator ("pipe").

Because redirection is a feature of the program's execution environment ("shell"), and not a feature of the program itself, programs can be very simple, avoiding all explicit handling of input or output files.

The following example command line will extract the "sampling rate" parameter from the BCI2000 data file <tt>mydata.dat</tt>, and display the result on the text console:
bci_dat2stream < mydata.dat | bci_stream2prm | grep SamplingRate

==Building BCI2000 command line tools==

Usually, command line tools are built automatically from the main makefile.
You will need to manually build the command line tools in one of the following situations:
*You are using the Borland C++ Builder 6 IDE rather than <tt>make</tt>.
*You are using Borland C++ Builder 2006/2007, and thus [[Programming Howto:Building BCI2000#Compiling_BCI2000_with_Borland_C.2B.2B_Builder_2006.2F2007|cannot use make to build all of BCI2000]].
*You are going create a command line executable from your own <tt>GenericFilter</tt> descendant class.

To do a manual build, open a shell (cmd) window. Change to the <tt>BCI2000/src/core/Tools/cmdline</tt> directory, and execute <tt>make install</tt> from there; this will also move the resulting executables to the <tt>BCI2000/Tools/cmdline</tt> directory. To build an executable from a file <tt>MyFilter.cpp</tt> containing a BCI2000 filter <tt>MyFilter</tt>, execute
make MyFilter.install
This will build an executable <tt>MyFilter.exe</tt> and add it to the <tt>BCI2000/Tools/cmdline</tt> directory.

Building command line tools does not require the full Borland C++ Builder IDE; you may use the [http://www.borland.com/downloads/download_cbuilder.html freely available command line version] of the Borland C++ compiler instead. In this case, you will need to comment out the line reading <code>#include <vcl.h></code> from the file <tt>BCI2000/src/shared/PCHIncludes.h</tt>

In order to use the command line tools, you will need to add the full path
to the <tt>BCI2000/Tools/cmdline</tt> directory to your system's PATH environment variable. (Please consult your operating system documentation for details on environment variables.)

==Stream format==

The common format used to exchange data between BCI2000 command line tools is called a "BCI2000 binary stream," or just "stream." Such a "stream" transfers parameters, states, and data in exactly the same binary format as is used for socket communication between the four main modules of the BCI2000 real-time system.

The stream format is different from the BCI2000 <tt>dat</tt> and <tt>prm</tt> file formats, and is not human readable. A number of tools are available to convert data either to or from stream format.
As the stream format is only used between tools, and generally not of interest to the user, a typical conversion will use ''two'' of the tools provided, one to translate the original file into a "stream," and one to translate the stream into the desired format.

E.g., to display a <tt>dat</tt> file's contents in a human readable format, one would use a tool called <tt>bci_dat2stream</tt> to convert it into a stream, and then use another tool called <tt>bci_stream2asc</tt> to convert the stream into text. If you want the output to appear in the text window as a sequence of pages, you pipe it into the "more" program:
bci_dat2stream < myfile.dat | bci_stream2asc | more

===Conversion into stream format===
====<tt>bci_dat2stream</tt>====
=====Input format=====
<tt>bci_dat2stream</tt> converts a BCI2000 data (<tt>dat</tt>) file into a stream.
=====Options=====
The <tt>--transmit</tt> option may be used to select states, parameters, and data for transmission, as in
bci_dat2stream --transmit-sp < myfile.dat
In the above example, only states and parameters but no data will be contained in the resulting stream. "Data" comprises signal and state vector data.
Omitting a <tt>--transmit</tt> option corresponds to<tt>--transmit-spd</tt>. This transmits all information contained in the <tt>dat</tt> file.

When the <tt>--raw</tt> option is given, data are transmitted in raw, i.e. uncalibrated, form. By default, data are transmitted in calibrated form, using the calibration information from the <tt>SourceChOffset</tt> and <tt>SourceChGain</tt> parameters.

====<tt>bci_prm2stream</tt>====

<tt>bci_prm2stream</tt> converts a BCI2000 parameter (<tt>prm</tt>) file into a stream.

===Conversion from stream format===

====<tt>bci_stream2prm</tt>====

<tt>bci_stream2prm</tt> converts a stream into a BCI2000 parameter (<tt>prm</tt>) file. As a parameter file is just a sequence of parameter lines, this is also a text-only, human readable format.

====<tt>bci_stream2asc</tt>====

<tt>bci_stream2asc</tt> converts a stream into a human readable format. Each object contained in the stream will appear as its C++ type name, followed by an opening brace, its content, and a closing brace. The content will appear as defined by the stream inserter <tt>operator>></tt> for the object's type.

In the output of BCI2000 state vector information, each state will appear on its own line, thus values of certain states may be easily extracted using the <tt>grep</tt> program.

====<tt>bci_stream2table</tt>====

<tt>bci_stream2table</tt> converts a stream into a tab-separated table containing state and signal values in ASCII format. Each state, and each signal entry, has its own column. The first line of output begins with a <tt>#</tt> comment character, and contains a tab-separated list of column headers. This format is best suited for data import into applications that use tables.

====<tt>bci_stream2mat</tt>====

<tt>bci_stream2mat</tt> converts a stream into a Matlab binary file. The output <tt>.mat</tt> file contains two Matlab variables called <tt>Index</tt> and <tt>Data</tt>. Of these, the <tt>Data</tt> variable is a matrix with each column representing a BCI2000 data block (comprising state information and signal data). <tt>Index</tt> is a Matlab structure that contains indices into the <tt>Data</tt> rows, allowing access to BCI2000 states by name, as in:
myMatlabVariable = squeeze( Data( Index.TargetCode, : ) );

As each BCI2000 data block contains a signal which is a two-dimensional matrix (channels by elements), the signal index is itself a matrix. To copy the first channel's data into a Matlab variable, write
myChannel1 = squeeze( Data( Index.Signal( 1, : ), : ) );

For convenience, there is a Matlab function provided that simplifies reading <tt>bci_stream2mat</tt> output files into Matlab variables:
[ mySignal, myTargetCode ] = load_bcimat( 'eegdata.mat', 2, 'TargetCode' );

This function takes the file name as its first argument. In a second argument, specify the number of dimensions your output signal will have -- typically, this will be 2 for EEG-like data (samples by channels), and 3 for spectral data (blocks by bins by channels). Remaining arguments are treated as state names; the associated state data will be written into the variables specified as remaining output arguments. State variables will always be one-dimensional, with their number of entries matching the first dimension of the signal variable.

==Applying BCI2000 filters to streams==

===Compiling an existing filter as a command line tool===

A filter defined in a file <tt>MyFilter.cpp</tt> may be compiled and linked into its own executable by executing
make MyFilter.exe

from the Windows command prompt when in <tt>BCI2000/Tools/cmdline</tt>. For this to work, it is necessary that the directory containing the filter's <tt>cpp</tt> file be contained in the makefile's <tt>SIGPROC</tt> variable; you may have to adapt it to fit your needs.

If the filter's code depends on code not contained within its <tt>cpp</tt> file, you will get linker complaints about unresolved externals. To solve the problem, add the missing <tt>cpp</tt> files to the makefile's <tt>BCIOBJ</tt> variable, and execute
make clean && make MyFilter.exe

To check whether the executable works, enter
MyFilter --help

You should get a message stating that the program applies the "MyFilter" filter to its input.

===Off-line only filters===

For off-line analysis, data must often be partitioned into "segments" before performing statistics. As there is no notion of "segments" in the on-line data and file format, we suggest using the "Running" state to indicate segments in the following way:

When performing a segmenting task, a filter sets the "Running" state to zero outside segments. A statistics filter will then perform buffering from its <tt>Process()</tt> function, and act on the buffered data from its <tt>StartRun()</tt> and <tt>StopRun()</tt> functions.

Note that setting the "Running" state to zero inside the on-line system will suspend the on-line system, so this kind of segmenting and statistics filtering cannot be used on-line.

==Examples==

The following examples work from the Windows NT command prompt. Nevertheless, we would like to point the reader to the free ''cygwin'' collection of GNU tools ported to the Win32 API. ''Cygwin'' provides the power of the <tt>bash</tt> shell, and of programs like the <tt>sed</tt> stream editor.
Although the BCI2000 command line tools cannot be compiled with <tt>gcc</tt>, they work fine when called from ''cygwin'' .

===Extracting parameters===

To extract parameters from a data file, convert it into a stream using <tt>bci_dat2stream --transmit-p</tt>, and convert the stream into a parameter file using <tt>bci_stream2prm</tt> as in
bci_dat2stream < mydata.dat | bci_stream2prm > myprms.prm

===Processing data with parameters different from the ones contained in the file===

To combine a data file with parameters other than those contained in it, use <tt>bci_dat2stream</tt>'s <tt>--transmit</tt> option to suppress parameters, and combine its output with <tt>bci_prm2stream</tt>'s into a single stream. To affect processing, parameters must precede the data in the stream. Combining the output is effected by the <code>( ... && ... )</code> construct.
(bci_prm2stream < myparameters.prm && bci_dat2stream --transmit-sd < mydata.dat) | MyFilter | bci_stream2table > mytable.txt

===Processing data with BCI2000 filters and importing the results into Matlab===

To process data with the filters used in the mu-training on-line system, saving the AR spectrum as Matlab file, execute
bci_dat2stream < mydata.dat | TransmissionFilter | SpatialFilter | ARFilter | bci_stream2mat > myspectra.mat

Load the data into Matlab using
[ signal, TargetCode ] = load_bcimat( 'myspectra.mat', 3, 'TargetCode' );

This requires that the file <tt>load_bcimat.m</tt> is accessible from the Matlab search path.

===Exporting BCI2000 data into a table suitable for import into other applications (MS Excel, SPSS)===

To process data with the filters used in the mu-training on-line system, saving the AR spectrum as a table in ASCII format, execute

bci_dat2stream < mydata.dat | TransmissionFilter | SpatialFilter | ARFilter | bci_stream2table > mytable.txt

===Testing a modified filter on existing data===

To verify that changes to a filter's code don't change its behavior with respect to existing data, apply both versions to a stream, convert the output stream into human readable format, and have a file comparison program display any differences. For the following example, we will compare a previous version of the ARFilter, renamed <tt>prev_ARFilter</tt>, to the current one.

#Create a stream suitable for input to the ARFilter:<br /><code>bci_dat2stream < mydata.dat | TransmissionFilter | SpatialFilter > test.bcistream</code>
#Apply both filter versions to the stream, and save the results in human readable format:<br /><code>ARFilter < test.bcistream | bci_stream2asc > ARresult.txt && prev_ARFilter < test.bcistream | bci_stream2asc > prev_ARresult.txt</code>
#Compare the results (using the Windows NT analog to the <tt>diff</tt> program):<br /><code>comp /a /l prev_ARresult.txt ARresult.txt | more</code>

==Further information==

*A comprehensive and up-to-date description of a number of command line shells, and their scripting, is provided at <tt>http://www.ss64.com/</tt>.

*The ''cygwin'' Win32 port of GNU tools can be downloaded at <tt>http://www.cygwin.com/</tt>.

*The free command line version of the Borland C++ compiler is available at <tt>http://www.borland.com/downloads/download_cbuilder.html</tt>.

[[Category:External Interfaces]][[Category:Data Analysis Tools]][[Category:Development]]

BCI2000 Wiki - User contributions [en]

User Reference:Command Line Processing