Matlab functions for accessing BCI2000 .dat format

[tadel]

Dear BCI2000 developers,

I'm managing the developments of the Brainstorm software for MEG/EEG analysis (http://neuroimage.usc.edu/brainstorm, ~7000 users worldwide).
One of our users asked last week on our forum how he could import BCI2000 .dat files in Brainstorm. We are offering support for a wide range of MEG and EEG systems, but not for BCI2000 yet.
For all those formats, I integrated or developed specific functions for reading efficiently small blocks of recordings from the files, to be able to dynamically browse through continuous long recordings. The I/O functions are integrated tighter in the software core than in EEGLAB.

I was wondering if you also had all-matlab or -java based versions available for those functions, that do not rely on any compiled library, or if you were planning to develop some.
I would need just two functions: one that reads the header and the sensors descriptions, and one that reads an arbitrary segment of time in a long continuous file.

I've seen that you're offering a plug-in for EEGLAB, that is based on pre-compiled mex functions in your distribution.
But I found out in the past that offering long-term support for pre-compiled mex files was a nightmare on many platforms, and the total size of your mex folder is over 10Mb. It is too big for me to integrate in the Brainstorm distribution, and I don't like too much this idea of having to download and install external plug-ins, I would prefer to keep Brainstorm as a 100% plug and play software.
For file reading, there are ways to write pure Matlab code (or Java code in Matlab scripts) that runs almost as fast as compiled code. If you avoid loops, the vectorized calls to fread/fseek are not so far behind the system calls performance.

Would you be interested in working on some code that we could share?
For now, is there an easy way for your users to export the BCI2000 recordings to some other format that Brainstorm could read (like EDF)?

Let me know what you think.
Cheers,
Francois

[mellinger]

Hi,

thank you for your question.

Reading BCI2000 data files without using the BCI2000 framework library is not encouraged. This has nothing to with the speed of the resulting implementation, rather it is a problem of keeping the external code up-to-date.

As long as interfacing code is maintained within BCI2000 itself, BCI2000 developers are free to extend the BCI2000 file format in any way they feel appropriate. Obviously, this will break any interfacing code maintained outside of BCI2000.

A major problem in writing and maintaining your own code for reading BCI2000 data files will be proper support for BCI2000 "parameter lines", which are rather complicated to parse correctly, due to the many back-compatible updates that format has seen (see the BCI2000 wiki for a description of a "parameter line"). Don't be deceived by the simple appearance of most parameter lines in a BCI2000 data file -- there are a lot of special cases on which a separate parser will probably break down sooner or later. Also, maintaining a Matlab or Java-based additional parser within BCI2000 will incur a major duplication of effort for the same reason.

But I found out in the past that offering long-term support for pre-compiled mex files was a nightmare on many platforms, and the total size of your mex folder is over 10Mb.

We stick to the most important platforms, and only update mex files when a BCI2000 binary release comes out, so it is not so difficult to keep those mex files updated. You are welcome to distribute the load_bcidat mex file with your software, and update it whenever a BCI2000 binary release comes out. Also, changes to the BCI2000 parameter format will not break the load_bcidat mex file, so there is no need to update it on a regular basis (except when you want to take advantage of the newer features, of course).

Would you be interested in working on some code that we could share?

In general, yes, but not if this implies duplication of code within BCI2000 itself. One interesting option would be the development of a file format definition meta-language, and a code generation backend that creates interfacing code in high-level languages such as C++, Matlab, or Python. I have often wished that such a language existed, but I'm not aware of any. As I am experienced in writing parsers using the bison parser generator, implementing a parser for such a language would not be a problem for me. Also, I don't think it would be too difficult to design a backend that makes use of the fact that all languages forward most of the functions available in the C stdlib practically without much change.

So the more interesting part would be to define a software interface suited to access all the information available in common file formats, and a language to describe all relevant features. Would that be an interesting project for you?

For now, is there an easy way for your users to export the BCI2000 recordings to some other format that Brainstorm could read (like EDF)?

BCI2000 allows for recording data directly into EDF. In addition, it is possible to use load_bcidat in order to load data given in dat format, and then use save_bcidat on the same data, writing it out in EDF or GDF format (specify a .edf or .gdf extension for the output file).

Regards,
Juergen

[tadel]

Hi Juergen,

Thank you for your detailed answer. I completely understand your development scheme, it doesn't make much sense to maintain the double amount of code, just to save a few Mb per install.
I'm not immediately interested in investing time in the development of a new I/O library, as I already have a very stable and flexible structure in Brainstorm, that supports most of the file formats that our users are requesting. I was asking in case something was readily available.

For now, I'm going to ask this user to convert their files back to EDF before importing them into Brainstorm.
If at some point the demand is increasing, I will integrate your Matlab and mex functions as external code, asking the user to point at the local install of BCI2000.

Please let me know if you are contacted by users who have some interest into importing data from BCI2000 into Brainstorm.
Thanks,
Francois