BCI dat File format

Forum for software developers to discuss BCI2000 software development
Locked
jack123
Posts: 3
Joined: 07 Mar 2013, 08:48

BCI dat File format

Post by jack123 » 11 Mar 2013, 16:40

Hi there,

I am trying to construct a BCI2000 dat file from raw data taken originally from an EPOC emotive.

The timestamps and associated 14 sensors readings for a period covering a number of tests (over a few days)are currently in an excel spreadsheet.

I've looked at the wiki regarding the File format, but cannot see exactly how to parse the available data into the binary file format that can then be loaded directly into BCI2000.

Does anybody know of a reference/tutorial for the exact binary file format, or have a similar xml/csv/xls to bci2000 format conversion tool they could point me to?

Many Thanks
Jack

boulay
Posts: 382
Joined: 25 Dec 2011, 21:14

Re: BCI dat File format

Post by boulay » 12 Mar 2013, 02:35

If you have MATLAB, look in your BCI2000\tools\mex directory for save_bcidat. To learn how to format the signal, states, parameters arguments you should try load_bcidat on a tutorial file. Better yet, record some dummy data using BCI2000 configured as you like then open that file with load_bcidat to seed your parameters and states variable.

If you don't have MATLAB, you should probably write your own code in C++. Start by looking here:
BCI2000\src\shared\fileio

I think part of the reason why non-matlab tools to do things like this don't exist is: "Why would you want to?"
BCI2000's dat format is not the most common format for brain-signal analysis tools. Some other formats have features that make them arguably superior. E.g., I'm a big fan of GDF's ability to seek to a point in the data without reading the entire data file into memory. GDF is also supported by all third-party tools that support BCI2000 files + more. The only legitimate reason I can think of for writing a BCI2000 dat file outside of a BCI2000 recording session is because you want to test your data analysis pipeline for analyzing many BCI2000 files to be collected later, and the most likely environment for doing that is MATLAB.

If there are other reasons for writing BCI2000 files outside of a BCI2000 recording session then please tell me. Someone else asked how to write BCI2000 dat files fairly recently and I couldn't think of a reason then either.

boulay
Posts: 382
Joined: 25 Dec 2011, 21:14

Re: BCI dat File format

Post by boulay » 12 Mar 2013, 02:43

Please see Juergen's answer in another thread here:
http://www.bci2000.org/phpbb/viewtopic.php?f=15&t=2071

jack123
Posts: 3
Joined: 07 Mar 2013, 08:48

Re: BCI dat File format

Post by jack123 » 12 Mar 2013, 04:34

Hi there,

Thank you for your replies.

The purpose of creating the dat file directly is to allow me to load the raw values (currently stored in cvs type format) into BCI2000 in order to investigate using the BCI2000 P300 classifier over them. I did have a look previously at the Technical_Reference:BCI2000_File_Format wiki page, but didn't fully understand how the various sections (source/storage/system) are encoded and how they relate to the binary raw signal data. I'll have a look at the bci2000 src file as recommended (the solution will likely be c++ rather than matlab) and see if that helps.

Thanks again,

Jack

boulay
Posts: 382
Joined: 25 Dec 2011, 21:14

Re: BCI dat File format

Post by boulay » 12 Mar 2013, 06:06

I guess you were planning on using the FilePlayback + P3SignalProcessing + P3Speller modules? The classification result will be meaningless unless you get the parameters and states exactly correct in both time and value.

If all you want to do is see if your data are classifiable then you can use some other matlab/python classification algorithms. If you're into Python, I can point you to a pretty good classification pipeline available in the BCPy2000 source but it is not supported and there is no documentation beyond the code comments.

If you want to see how the P3TemporalFilter fares specifically, then I think your easiest path would be as follows:
-Collect some data using BCI2000 {SignalGenerator, P3SignalProcessing, P3Speller}
Be sure to config it to match the conditions under which your Emotiv data were collected.
-Load the dummy data into Matlab with load_bcidat
-Modify the dummy data replacing the data with your Emotiv data and modify the states' values to match what happened when you collected the Emotiv data
-Write the file with save_bcidat
-Analyze the newly saved .dat file with BCI2000\tools\matlab\bci2000chain.m

The reason why I advise against trying to create a new .dat file from scratch in C++ is because you will spend a lot of time trying to get the states and parameters exactly right, and eventually you'll get to a point where the file is being output in the correct format but there won't be an easy way for you to see if the states' values are correct without loading the data in another program. This cycle can be done much faster in Matlab.

And I think it's easier to use bci2000chain than to use FilePlayback because it's faster and you can get the output at any stage of the processing pipeline to see where things went wrong.

One more thing: Depending on how much data you intend to convert and test and whether the conditions it was collected under are reproducible or not, it may just be easier to collect new data using the Emotiv source module in BCI2000.

-Chad

jack123
Posts: 3
Joined: 07 Mar 2013, 08:48

Re: BCI dat File format

Post by jack123 » 12 Mar 2013, 06:44

You're basically right - I would like in the first instance to see if the data is classifiable, and was pointed to the BCI2000 as a starting point.

If you could point me to any python classifiers that would be a great help, and sounds like it might be an easier starting point.

Thanks for all your input

Jack

boulay
Posts: 382
Joined: 25 Dec 2011, 21:14

Re: BCI dat File format

Post by boulay » 12 Mar 2013, 07:20

Install the python packages numpy, scipy, and BCPy2000 (found in BCI2000/src/contrib/BCPY2000 and InstallFramework.bat)

Code: Select all

from BCPy2000.BCI2000Tools import DataFiles, Classification

x = 3d numpy array size exemplars * channels * samples
y = classes (target or non-target) size examplars,
channel_names = list of strings of channel_names
fs = sampling rate
f_dumped = DataFiles.dump(temp_file, x=temp_x, y=y, channels=channel_names, fs=fs)
f_dumped.close()
f_dumped = open(temp_file, 'rb')
u, c = Classification.ClassifyERPs(f_dumped)
accuracy = 1 - c.loss.train

There's a lot more you can do. Check out BCPy2000.SigTools.LearningTools.predictor for some examples.

boulay
Posts: 382
Joined: 25 Dec 2011, 21:14

Re: BCI dat File format

Post by boulay » 12 Mar 2013, 07:23

This software was written by Jeremy Hill and colleagues. They had some papers published in late 2012 on auditory ERP classification. The following offers a really nice explanation of what's going on:
http://www.ncbi.nlm.nih.gov/pubmed/23250668

mellinger
Posts: 1163
Joined: 12 Feb 2003, 11:06

Re: BCI dat File format

Post by mellinger » 14 Mar 2013, 11:48

Hi Chad,
BCI2000's dat format is not the most common format for brain-signal analysis tools. Some other formats have features that make them arguably superior. E.g., I'm a big fan of GDF's ability to seek to a point in the data without reading the entire data file into memory.
there is nothing in the BCI2000 file format that would keep a program from doing random access on it. Also, load_bcidat supports reading any range of subsamples from a BCI2000 data file.

So I'm wondering whether there is some erroneous information lying somewhere on the wiki? I would appreciate to know where it is, so I can fix it.

Regards,
Juergen

boulay
Posts: 382
Joined: 25 Dec 2011, 21:14

Re: BCI dat File format

Post by boulay » 14 Mar 2013, 22:19

No. It was my error.

Random access was one reason why I chose GDF (for a specific task) over other formats. I can't remember the reason why I chose it over BCI2000's dat format right now, so I guess I assumed it was the same reason. Maybe it was simply the availability of tools. Sorry about my mistake.

Locked

Who is online

Users browsing this forum: No registered users and 2 guests