Writing/reading .dat files

[Binderup]

Hello.

My name is Asbjoern Binderup and I am doing my master thesis project under Alvaro Cabrera at Aalborg University.

I am making a LabView program that picks up data from the Acquire program made by Neuroscan and tries to store the data following the format of BCI2000 so I can use the BCI2000 tools made by Alvaro for offline analysis of the data.

The problem is that the load_bcidat.dll which is used to load the .dat files refuses to read my "home-made" .dat files even though they follow the format described in the project outline for BCI2000.

The error is "Could not open "Testy_1_1.dat" as a BCI2000 data file." which I have tried to narrow down reading the source code description for the load_bcidat.ccp provided by this site.

Following I will list my .dat header. I have only included the system parameters that are described as minimum by the project outline. The states and their bit lengths are something I have decided, and I do not hope that should provide any problem for the file to be read.

By testing it seems so far the error has something to do with minimum file length or something.

Any help would be appriciated.

-Asbjoern

[Binderup]

HeaderLen= 2025 SourceCh= 10 StatevectorLen= 3
[ State Vector Definition ]
SourceTime 16 0 0 0
Running 1 0 2 0
Recording 1 0 2 1
TargetCode 1 0 2 2
ResultCode 1 0 2 3
RestPeriod 1 0 2 4
[ Parameter Definition ]
Source int SoftwareCh= 10 33 1 129 //the number of digitized and stored channels including marker channel
Source int SampleBlockSize= 100 20 1 128 //the number of samples transmitted at a time
Source int TransmitChList= 10 1 2 3 4 5 6 7 8 9 10 1 1 128 //list of transmitted channels (# of channels MUST equal TransmitCh)
Source int SamplingRate= 100 20 1 128 //the sample rate
Storage string SubjectName= Testy Name a z //subject alias
Storage string SubjectSession= 1 001 0 999 //session number (max. 3 characters)
Storage string SubjectRun= 1 00 0 99 //digit run number (max. 3 characters)
Storage string FileInitials= C:\Documents and Settings\Asbjørn\My Documents\Sem9_Project a z 0 //Initials of file name
Filtering int NumControlSignals= 0 1 1 128 //the number of transmitted control signals
Filtering int AlignChannels= 0 0 0 1 //align channels in time (0=no, 1=yes)
Filtering floatlist SourceChOffset= 10 0 0 0 0 0 0 0 0 0 0 0 -500 500 //offset for channels in A/D units
Filtering floatlist SourceChGain= 10 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.003 -500 500 //gain for each channel (A/D units -> muV)
Filtering floatlist SourceChTimeOffset= 1 -1 0 0 1 //time offset for all source channels (not used if -1)
System string EEGsourceIP= 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0//this module's listening IP
System string EEGsourcePort= 0 4200 1024 32768 //this module's listening port
System string SignalProcessingIP= 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0//this module's listening IP
System string SignalProcessingPort= 0 4200 1024 32768 //this module's listening port
System string ApplicationIP= 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0//this module's listening IP
System string ApplicationPort= 0 4200 1024 32768 //this module's listening port
System int StateVectorLength= 3 16 1 30 //length of the state vector in bytes

(binary data excluded)

[Binderup]

Update: By substituting the header of file originally recorded with BCI2000 with my header I get no errors. Guess it must be reading the binary data that is the problem.

I do think I store it correctly: 2 bytes for every channel, 2 bytes for the time state and 1 byte for the rest. I have a Matlab script made by Febo Cincotti which is quite capable to read both originally BCI2000 .dat files and my home made files in the correct way, that goes for both the header and binary data.

Sadly this script do not interface well with Alvaros work so it would be better to use the load_bcidat.dll.

[gschalk]

Asbjoern,

I have a few comments. First, it does not seem to be clear why your data files should not work. In particular if you get the header written right (the hard part). Do make sure that the HeaderLen is correct. The way you write the binary seems fine. 2 bytes per sample + 3 bytes StateVector, as defined in the header.

So, you could reverse engineer the error by debugging load_bcidat to see why it gives this error.

The biggest question seems to be why you don't use BCI2000 to acquire and store the data in the first place. Even if you need access to other programs, you could either use a BCI2000 built-in capacity (like the AppConnector) or your own little code to get data out of the system.

The Gerv
[/i]

[mellinger]

Asbjørn,

your FileInitials parameter contains space characters in its value.
According to the parameter format as defined in the project outline, these characters should be encoded as "%20" (remove the quotes).
In a parameter line, white space separates individual values, thus your FileInitials parameter line is interpreted to contain 7 values in total before the comment separator. This makes it syntactically wrong, as single-valued parameters may have a maximum of 4 values there (actual value, default value, lower bound, upper bound).

Generally, I suggest everyone to avoid writing code that reads or writes BCI2000 parameter syntax.
- Parameters have become pretty complex objects, so it is hard to get it right in the first place.
- Although we try to keep changes to the syntax at a minimum, such changes will inevitably occur from time to time. When your code does writing only, this should not be a problem, as changes will be backward compatible. Code that does reading will, however, inevitably break in the future, requiring continued maintenance effort on your side.

So, if writing your own parameter code is discouraged, what are the alternatives?
1) If you are using Matlab: Use the convert_bciprm mex file available in the src/tools/mex directory (will be built into tools/mex when you use Borland's make, and into src/tools/mex if you use buildmex.m to build it from inside Matlab).
2) If you are writing a program in C++:
Add the file src/shared/UParameter.cpp to your project, it is pretty independent of the rest of BCI2000.
Then, you may read a PARAM object from an STL std::istream, and write it out to an STL std::ostream.
If you use std::stringstream for the streams, this allows you to easily manipulate parameter lines.
3) If you are using any other language/coding environment:
I'm prepared to wrap the parameter code into a DLL with a C-style interface.
Such a DLL may then be used from any coding environment that allows for accessing external library functions, i.e., from practically any environment that I know of.
E.g., for LabView, there is a "Node" called "Call Library Function" that does the trick.

Regards,
Juergen

[Binderup]

Thanks for the feedback.

Though you say my parameters in my header does seem to follow the syntax it can indeed be read by load_bcidat.dll correctly as long as I put some binary data after it that has been recorded by BCI2000 and not my own system. EDIT: It does read that line completely wrong though, thanks.

To the question why I do not just use BCI2000 to do all my recordings in the first place is a very good one, some times I ask my self that, considering the troubles of making a run-time system.

Primarely me and my group mate just find it easier to work in the LabView enviroment than C++, not that we do not know how to program in C++. Also we want to take advantage of the inbuild signal processing tools LabView offers that we would perhaps othervise be without in C++ or had to program our self. Also, LabView interfaces nicely with C++, and in particular MATLAB through ActiveX so "borrowing" functionality is very easy. That can not be said the other way around.

Lastly, we have discovered that Neuroscan made software development kit for their amplifiers last year and that one is made in LabView, so if we make some good stuff now it can maybe be used in that enviroment too.

But still, we could just have ignored all that and gone with BCI2000 from the start. It do seems like an awesome program, but I guess we are stubborn young engineers that want to prove our self.

EDIT: Seems like I have some sort of format problem with the data. Besides remembering to save the data in little endian format I seem to be missing something.

EDIT2: Does little endian mean that I have to save the byte with "minor" values before the byte with "major" values for all channels and the time state?

[gschalk]

Hi,

I have two more comments regarding the interaction of BCI2000 and other software. First, it is already possible to fully integrate and execute Matlab code within BCI2000 so that, for example, certain filter calculations can be done in Matlab. If you have questions about that, please post.

Also, I would be very interested in seeing similar support for other programs such as LabView. This way, one could execute LabView code from within BCI2000 just like it is now possible with Matlab. If you were interested in writing something like this, I could make sure that it gets appropriately included in the BCI2000 distribution. I would also prominently highlight this work on the BCI2000 web site.

The Gerv

[mellinger]

Though you say my parameters in my header does seem to follow the syntax it can indeed be read by load_bcidat.dll correctly as long as I put some binary data after it that has been recorded by BCI2000 and not my own system.

In a BCI2000 data file, lines should be terminated the Windows way, i.e. with CRLFs rather than single newline characters. In this format, your header has a length of 2054 bytes rather than the 2025 of your HeaderLen field.

Lastly, we have discovered that Neuroscan made software development kit for their amplifiers last year and that one is made in LabView, so if we make some good stuff now it can maybe be used in that enviroment too.

But still, we could just have ignored all that and gone with BCI2000 from the start. It do seems like an awesome program, but I guess we are stubborn young engineers that want to prove our self.

All these are good reasons! I was just suggesting my help to achieve the same thing more efficiently, using some piece of code wrapped up in a DLL, so you don't have to write it yourself.

EDIT: Seems like I have some sort of format problem with the data. Besides remembering to save the data in little endian format I seem to be missing something.

EDIT2: Does little endian mean that I have to save the byte with "minor" values before the byte with "major" values for all channels and the time state?

For signal data, the values' layout in the file should match their memory layout in a Little Endian machine.
For data contained in the state vector, the layout is actually Big Endian, i.e., in your example, you would have to swap the SourceTime state's two bytes when writing them to disk.

Regards,
Jürgen

[Binderup]

If you will make some sort of save_bcidat.dll with functionality for saving header and binary data that I can use in the LabView enviroment it would be much appriciated.

Right now I store the data in parts to a file when the program is running instead of putting it all in memory and save by program termination. The header gets saved during program initialization and the the data gets saved every program cycle. This means that a file handle goes through the program connecting these blocks together.

I do not now much about .dll's but could there be a problem passing this file handle from one library node to another?

-Asbjoern

[mellinger]

Right now I store the data in parts to a file when the program is running instead of putting it all in memory and save by program termination. The header gets saved during program initialization and the the data gets saved every program cycle. This means that a file handle goes through the program connecting these blocks together.

I do not now much about .dll's but could there be a problem passing this file handle from one library node to another?

I'm not sure how LabView handles it when you access the same DLL from more than one node.
I cannot give you support on LabView, so you should make sure that you understand well what happens inside the "black boxes" when you access DLL functions.
For the suggested BCI2000 DLL to work, it needs to have a persistent and consistent internal state over all LabView nodes that call its functions.
This means, if node A calls function X, and function X allocates memory in the DLL's memory space, then a function Y from the same DLL, called from node B, should operate on the same data and the same memory addresses as function X did from node A.
When LabView chooses to remove DLLs from memory between calls, or at arbitrary "garbage collection" times, then the internally allocated memory will be overwritten, and the DLL's internal state is lost. Likewise, if LabView maintains separate copies of the DLL's memory space for each node, its internal state will not be consistent across nodes.

Regards,
Juergen