User Reference:Command Line Processing

From BCI2000 Wiki
Revision as of 21:12, 12 March 2012 by Jhill (talk | contribs) (Compiling an existing filter as a command line tool)

Jump to: navigation, search

The purpose of the BCI2000 command-line tools is to recreate exactly, offline, the processing that BCI2000 modules perform in an online session. This is possible because online BCI2000 modules consist of a chain of filters, each filter being implemented in a self-contained source file. We can therefore take the individual filter implementations and build them singly, as separate executables called "filter tools" that can be called from the system command-line. Recreating a preprocessing chain offline is then a question of passing a stream of data through a chain of these filter tools, connected by an operating-system "pipe". Supporting the beginning and end of the filter-tool chain we have additional command-line tools called format converters—for example, a tool that reads a data file to start the chain off, or tools that save the results in ASCII or Matlab formats to finish the chain.

This page tells you how to use the command-line tools from a system shell, such as the Windows Command Prompt (you might know it as the "DOS window") or from a bash, tcsh or similar shell on a Unix-like system (for example, in the Mac OSX Terminal application).

If you are using Matlab, you may wish to examine bci2000chain instead: this uses the command-line tools, but hides the details and presents itself to you as a single Matlab function.

Introduction

Command line interfaces are a powerful tool for dynamically combining pieces of software which are typically small programs optimized for performing a very limited functionality. With a command line interface, data processing is usually performed sequentially on a stream of data. This allows the processing of indefinitely large amounts of data while consuming but a finite, and often very small, amount of memory.

Moreover, combining command line programs to perform complex tasks can be done interactively by entering complex commands directly and also in a "scripting" manner, by entering sequences of commands into text files which allow for iteration and branching. That way, automation of complex tasks can be achieved by rather simple means.

With the BCI2000 command line interface, BCI2000 data files can be

  • converted into various formats, ready for further processing with available software tools, or for visual inspection in human readable format;
  • processed off-line with literally the same code as used in the on-line system.

Together, these two concepts provide versatile access to recorded data in all stages of processing. The BCI2000 command line interface may prove useful for

  • automated analysis of large amounts of recorded data,
  • development of new filter classes,
  • verification of external off-line analysis methods.

This document explains the command line tools available for working with BCI2000 files, and how to build BCI2000 filters as single executables that may be combined to form a chain of filters.

Command line interfaces

Basically, the interface of a command line tool consists of

  • an input stream stdin,
  • an output stream stdout,
  • an error stream stderr,
  • optional arguments (switches) controlling details of its behavior.

The most important feature of a command line interface is its ability to redirect streams.

  • stdin can be redirected to read from a file using the < operator;
  • stdout can be redirected to write to a file using the > operator;
  • one program's stdout can be plugged into another program's stdin using the | operator ("pipe").

Because redirection is a feature of the program's execution environment ("shell"), and not a feature of the program itself, programs can be very simple, avoiding all explicit handling of input or output files.

The following example command line will extract the "sampling rate" parameter from the BCI2000 data file mydata.dat, and display the result on the text console:

bci_dat2stream < mydata.dat | bci_stream2prm | grep SamplingRate

Building BCI2000 command line tools

Command line tools are built automatically by the BCI2000 build system if building of BCI2000 tools is enabled.

In order to use the command line tools, you will need to add the full path to the BCI2000/Tools/cmdline directory to your system's PATH environment variable. (Please consult your operating system documentation for details on environment variables.)

Stream format

The common format used to exchange data between BCI2000 command line tools is called a "BCI2000 binary stream," or just "stream." Such a "stream" transfers parameters, states, and data in exactly the same binary format as is used for socket communication between the four main modules of the BCI2000 real-time system.

The stream format is different from the BCI2000 dat and prm file formats, and is not human readable. A number of tools are available to convert data either to or from stream format. As the stream format is only used between tools, and generally not of interest to the user, a typical conversion will use two of the tools provided, one to translate the original file into a "stream," and one to translate the stream into the desired format.

E.g., to display a dat file's contents in a human readable format, one would use a tool called bci_dat2stream to convert it into a stream, and then use another tool called bci_stream2asc to convert the stream into text. If you want the output to appear in the text window as a sequence of pages, you pipe it into the "more" program:

bci_dat2stream < myfile.dat | bci_stream2asc | more

Conversion into stream format

bci_dat2stream

Input format

bci_dat2stream converts a BCI2000 data (dat) file into a stream.

Options

The --transmit option may be used to select states, parameters, and data for transmission, as in

bci_dat2stream --transmit-sp < myfile.dat

In the above example, only states and parameters but no data will be contained in the resulting stream. "Data" comprises signal and state vector data. Omitting a --transmit option corresponds to--transmit-spd. This transmits all information contained in the dat file.

When the --raw option is given, data are transmitted in raw, i.e. uncalibrated, form. By default, data are transmitted in calibrated form, using the calibration information from the SourceChOffset and SourceChGain parameters.

The -p or --parameters option allows some or all of the parameter values stored in the data file to be overridden by values from a named parameter file—for example:

bci_dat2stream -pMyParameters.prm < myfile.dat

bci_prm2stream

bci_prm2stream converts a BCI2000 parameter (prm) file into a stream.

Conversion from stream format

bci_stream2prm

bci_stream2prm converts a stream into a BCI2000 parameter (prm) file. As a parameter file is just a sequence of parameter lines, this is also a text-only, human readable format.

bci_stream2asc

bci_stream2asc converts a stream into a human readable format. Each object contained in the stream will appear as its C++ type name, followed by an opening brace, its content, and a closing brace. The content will appear as defined by the stream inserter operator>> for the object's type.

In the output of BCI2000 state vector information, each state will appear on its own line, thus values of certain states may be easily extracted using the grep program.

bci_stream2table

bci_stream2table converts a stream into a tab-separated table containing state and signal values in ASCII format. Each state, and each signal entry, has its own column. The first line of output begins with a # comment character, and contains a tab-separated list of column headers. This format is best suited for data import into applications that use tables.

bci_stream2mat

bci_stream2mat converts a stream into a Matlab binary file. The output .mat file contains two Matlab variables called Index and Data. Of these, the Data variable is a matrix with each column representing a BCI2000 data block (comprising state information and signal data). Index is a Matlab structure that contains indices into the Data rows, allowing access to BCI2000 states by name, as in:

myMatlabVariable = squeeze( Data( Index.TargetCode, : ) );

As each BCI2000 data block contains a signal which is a two-dimensional matrix (channels by elements), the signal index is itself a matrix. To copy the first channel's data into a Matlab variable, write

myChannel1 = squeeze( Data( Index.Signal( 1, : ), : ) );

For convenience, there is a Matlab function provided that simplifies reading bci_stream2mat output files into Matlab variables:

[ mySignal, myTargetCode ] = load_bcimat( 'eegdata.mat', 2, 'TargetCode' );

This function takes the file name as its first argument. In a second argument, specify the number of dimensions your output signal will have -- typically, this will be 2 for EEG-like data (samples by channels), and 3 for spectral data (blocks by bins by channels). Remaining arguments are treated as state names; the associated state data will be written into the variables specified as remaining output arguments. State variables will always be one-dimensional, with their number of entries matching the first dimension of the signal variable.

Note that if you are using Matlab, you may prefer to use bci2000chain from the suite of BCI2000 matlab tools, instead of calling the command-line tools plus load_bcimat by hand.

Applying BCI2000 filters to streams

Compiling an existing filter as a command line tool

Adding a new command-line filter tool to your BCI2000 solution file or Makefile requires use of CMake as described in the build system documentation. The simplest example is when you have a filter implementation in files MyFilter.cpp and MyFilter.h, in a directory MyDir on their own. Let us assume that this is a subdirectory of src/custom since this is an area of the BCI2000 distribution reserved for your own code (if src/custom does not already exist, create it—the BCI2000 CMake system will find it automatically). To add the command-line filter, first ensure that CMake reaches your subdirectory. This is done by appending the appropriate line to src/custom/CMakeLists.txt :

ADD_SUBDIRECTORY(  MyDir  )

Then you need to create a CMakeLists.txt file inside MyDir, containing the following line:

BCI2000_ADD_CMDLINE_FILTER( MyFilter )

Finally you need to go back to the top-level build and run CMake using one of the scripts there (Make......bat on Windows, or Make Unix Makefiles.sh on OSX and other unix-like systems). You need to ensure that your build is configured to include the BCI2000 Tools: if you have previously run CMake and need to change this setting, you can either delete CMakeCache.txt from the build directory before running the script, or run cmake -i (decline the offer to see advanced options).

Assuming this is successful, you can re-open your solution file and see MyFilter listed among the targets—or, for Makefile-based builds, you can simply type

make MyFilter

The executable MyFilter (or MyFilter.exe) is built in the tools/cmdline directory. Assuming this directory is on the system path, you can verify that the executable has built successfully by typing

MyFilter --help

You should get a message stating that the program applies the "MyFilter" filter to its input.

One potential complication is that the BCI2000_ADD_CMDLINE_FILTER macro cannot come in the same CMakeLists.txt file as the declaration of a BCI2000 module. Let us suppose you have followed the programmers' quickstart guide to create a custom signal-processing module in src/custom/MyModule and add a custom filter MyFilter to it. A safe procedure (which can be automated used the NewBCI2000FilterTool utility) is as follows:

  1. Create a subdirectory src/custom/MyModule/cmdline
  2. Insert the line ADD_SUBDIRECTORY( cmdline ) at the beginning of src/custom/MyModule/CMakeLists.txt (it must come before the module declaration).
  3. Create the file src/custom/MyModule/cmdline/CMakeLists.txt containing the following line:
BCI2000_ADD_CMDLINE_FILTER( MyFilter FROM .. )

Further instructive examples, of what to do when your filter needs to be linked to additional C++ source or external libraries, can be found in src/core/Tools/cmdline/CMakeLists.txt :

SET( SIGPROC_DIR ${BCI2000_ROOT_DIR}/src/shared/modules/signalprocessing )

BCI2000_ADD_CMDLINE_FILTER( ARFilter              FROM ${SIGPROC_DIR}    # because ARFilter.cpp and ARFilter.h reside there
                            INCLUDING "MATH"                             # because the "MATH" external library is required
                            EXTRA_SOURCES ${SIGPROC_DIR}/ARThread.cpp    # because the filter relies on one or more additional source files
                            EXTRA_HEADERS ${SIGPROC_DIR}/ARThread.h )    # ...and their corresponding headers

BCI2000_ADD_CMDLINE_FILTER( LinearClassifier      FROM ${SIGPROC_DIR} )  # a simpler example

Offline-only filters

For off-line analysis, data must often be partitioned into "segments" before performing statistics. As there is no notion of "segments" in the on-line data and file format, we suggest using the "Running" state to indicate segments in the following way:

When performing a segmenting task, a filter sets the "Running" state to zero outside segments. A statistics filter will then perform buffering from its Process() function, and act on the buffered data from its StartRun() and StopRun() functions.

Note that setting the "Running" state to zero inside the on-line system will suspend the on-line system, so this kind of segmenting and statistics filtering cannot be used on-line.

Examples

The following examples work from the Windows NT command prompt. Nevertheless, we would like to point the reader to the free cygwin collection of GNU tools ported to the Win32 API. Cygwin provides the power of the bash shell, and of programs like the sed stream editor. Although the BCI2000 command line tools cannot be compiled from within cygwin, they work fine when called from cygwin .

Extracting parameters

To extract parameters from a data file, convert it into a stream using bci_dat2stream --transmit-p, and convert the stream into a parameter file using bci_stream2prm as in

bci_dat2stream < mydata.dat | bci_stream2prm > myprms.prm

Processing data with parameters different from the ones contained in the file

To combine a data file with parameters other than those contained in it, use bci_dat2stream's -p or --parameters option:

bci_dat2stream -pMyParameters.prm < MyData.dat | MyFilter | bci_stream2table > MyTable.txt

Processing data with BCI2000 filters and importing the results into Matlab

First, consider using the higher-level Matlab interface bci2000chain instead. But if you decide you want to do it by hand, and you want to process data with the filters used in the mu-training on-line system, saving the AR spectrum as Matlab file, execute

bci_dat2stream < mydata.dat | TransmissionFilter | SpatialFilter | ARFilter | bci_stream2mat > myspectra.mat

Load the data into Matlab using

[ signal, TargetCode ] = load_bcimat( 'myspectra.mat', 3, 'TargetCode' );

This requires that the file load_bcimat.m is accessible from the Matlab search path.

Exporting BCI2000 data into a table suitable for import into other applications (MS Excel, SPSS)

To process data with the filters used in the mu-training on-line system, saving the AR spectrum as a table in ASCII format, execute

bci_dat2stream < mydata.dat | TransmissionFilter | SpatialFilter | ARFilter | bci_stream2table > mytable.txt

Testing a modified filter on existing data

To verify that changes to a filter's code don't change its behavior with respect to existing data, apply both versions to a stream, convert the output stream into human readable format, and have a file comparison program display any differences. For the following example, we will compare a previous version of the ARFilter, renamed prev_ARFilter, to the current one.

  1. Create a stream suitable for input to the ARFilter:
    bci_dat2stream < mydata.dat | TransmissionFilter | SpatialFilter > test.bcistream
  2. Apply both filter versions to the stream, and save the results in human readable format:
    ARFilter < test.bcistream | bci_stream2asc > ARresult.txt && prev_ARFilter < test.bcistream | bci_stream2asc > prev_ARresult.txt
  3. Compare the results (using the Windows NT analog to the diff program):
    comp /a /l prev_ARresult.txt ARresult.txt | more

Further information

  • A comprehensive and up-to-date description of a number of command line shells, and their scripting, is provided at http://www.ss64.com/.