P300's .dat file

Forum for discussion on different brain signals
Locked
sandra2014
Posts: 17
Joined: 09 Sep 2014, 09:41

P300's .dat file

Post by sandra2014 » 15 Sep 2014, 08:55

Hi,everyone!
I have some questions about the .dat file of P300Speller.
1.After the BCI2000 run,when the .dat file began to write?
2.Was the .dat file have pass through the DataIOFilter? If is not , where and when did the SourceChOffset and SourceChGain used ?
3.If I want to analog the SpatialFilter P3TemporalFilter and LinearClassifier with the .dat file as the Input Signal ,will I get the same result with BCI2000 ?
I really need your help !

Thanks!
Sandra

boulay
Posts: 382
Joined: 25 Dec 2011, 21:14

Re: P300's .dat file

Post by boulay » 15 Sep 2014, 11:17

Hi Sandra,

First I'd like to make something very clear. The .dat file and the flow of data through the BCI2000 filter pipe are mostly independent. You can run BCI2000 perfectly fine without recording any data to a .dat file. Your data are collected form the amplifier and placed into a buffer. From this buffer, the data are simultaneously written to a dat file and passed through the BCI2000 filter pipe. Please tell me if this is confusing to you. I spent over 1 month helping someone else with their problem and, in the end, most of that time could have been saved if they understood this fundamental concept.

1.
I'm not certain, but I think the dat file is created as soon as you press "Start". Then, with each new block of data acquired from your amplifier, new data are appended to the dat file.
In terms of the processing chain, the .dat file is written to by DataIOFilter. See here and click on the link for DataIOFilter.

2. A dat file does not pass through anything. Data are written to the dat file by a FileWriter, controlled by DataIOFilter.

3. I don't understand your question. Are you asking about using a previously recorded dat file as your input into BCI2000 to "replay" your experiment? Or are you asking about analyzing a previously recorded dat file offline? Or something else?

boulay
Posts: 382
Joined: 25 Dec 2011, 21:14

Re: P300's .dat file

Post by boulay » 15 Sep 2014, 14:20

Hi Sandra,

From reading your other posts, I surmise that your goal is to reverse-engineer BCI2000's online processing in the P300Speller and reproduce it in Matlab using only the dat file as input.

First, in case you didn't know already, many of BCI2000's filters have command-line versions that can be chained together to reproduce specific segments of the BCI2000 online pipeline during offline analysis, and these can be called from within Matlab. Search the wiki for bci2000chain to get started.

Otherwise, I recommend you look at the source code.
In BCI2000\src\core\SignalProcessing\P3\PipeDefinition.cpp you'll see

Code: Select all

Filter( SpatialFilter, 2.A );
Filter( P3TemporalFilter, 2.B );
Filter( LinearClassifier, 2.C );
This describes the signal processing filters used by P3SignalProcessing. All of these filters' source code files can be found in BCI2000\src\shared\modules\signalprocessing

The source code can be a little hard to navigate but is ultimately what you'll need to do if you want to reproduce exactly what happens online. If you just want a quick summary of each step then you can find them in the wiki: SpatialFilter, P3TemporalFilter, LinearClassifier.

Note that the P3SpellerTask is specially designed to work with the output from P3SignalProcessing. I'm sorry its source code is a little hard for me to follow, and I don't have the time to go through it.

-Chad

sandra2014
Posts: 17
Joined: 09 Sep 2014, 09:41

Re: P300's .dat file

Post by sandra2014 » 16 Sep 2014, 05:15

Hi Chad!
Thanks for your reply!
I had little confuse about how the dat file written . Now I understand what your reply about "the data are simultaneously written to a dat file and passed through the BCI2000 filter pipe" .
1.I have read the DataIOFilter's source code ,the SourceChOffset and SourceChGain were used in the code. And I want to confirm that the data ,which was written into a .dat file, is the raw data have been applied the SourceChOffset and SourceChGain ,that is, the data in .dat file was the raw data that dealt with the SourceChOffset and SourceChGain .
2.I have read the source code of : SpatialFilter, P3TemporalFilter, LinearClassifier, and I didn't find any other data process part different with my matlab code . I mean , I use the same method as these three Filter in matlab code , my result was different with the .applog file which was produced after the BCI2000 run . So I guess , there must be some other data process part in BCI2000 online/offline analysis , but I don't find what and where they are .
Can you help resolve this problem ? I have been stuck in this problem so long .

Thanks ,sincerely !
Sandra

boulay
Posts: 382
Joined: 25 Dec 2011, 21:14

Re: P300's .dat file

Post by boulay » 16 Sep 2014, 09:33

Are you loading the data into Matlab for offline analysis using load_bcidat? If so, from the wiki:
By default, signal data will be in raw A/D units, and will be represented by the smallest Matlab data type that accommodates them. To obtain signal data calibrated into physical units (microvolts), specify '-calibrated' as an option anywhere in the argument list.
Try with and without that flag.Could that be the difference?

To test everything out, I suggest doing the following:

Use a function generator as input to your amplifier. (Try a slow ramp as input)
Set your gains to 1 and your offsets to 0.
Set your spatial filter to diagonal.
Set your classifier and other parameters to be as simple as possible. (Sorry, I'm on a Mac now and it's difficult for me to find the exact parameters.)
For example, set your classifier to only use one small window from one channel.
Set your application to flash the rows and columns in order with a relatively large inter-stimulus-interval. Only do one sequence.

After a sequence of flashing is done, the classifier output should follow the same pattern as your amplifier input from the function generator.

This should be fairly easy to reproduce in Matlab.
Note that there will be slight differences between BCI2000 and Matlab related to their different rounding errors and how they represent floats, but these should be small. Try doing a scatterplot between the applog values and your calculated values.

sandra2014
Posts: 17
Joined: 09 Sep 2014, 09:41

Re: P300's .dat file

Post by sandra2014 » 17 Sep 2014, 09:47

Thanks for your help ,Chad !
In fact , I did the work what your suggestion a few days ago, I set the gains to 1 and offsets to 0, AlignChannels = 0;TransmitChList = 10 (the channel number) , the classifier is :1 1 1 1 (that is, just use one element of the TransmitChList channel as feature). The result is ,there are still some error between the .applog file and my matlab Classification result .
I also re-check the workflow of BCI2000 , and I re-write my matlab code today , this time , I set the workflow in my matlab code as BCI2000's .
In detail ,
1.I wrote a DataIOFilter that use the SourceChGain and SourceChOffset to adjust the signal which was load from .dat file,and save the dealt signal as the input for next filters;
2.Because AlignChannels = 0 ,so I didn't need to write the AlignmentFilter ;
3.The next filter is TransmissionFilter ,as we know ,this filter transmit the data which is set in TransmitChList . So I realize this function in my matlab code too. Data that have passed through this filter will be reduce the dimension in channel;
4.The important part is SpatialFilter , my P300Speller experiment set the SpatialFilterType as CAR ,I have read some articles that said the CAR calculates the mean of all channels , I think what is the "all channels "refers to ? Is the all channels such as 64? Or the all channel that set in TransmitChList ? In my classification experiment in matlab , the answer is the "all channels "refers to the channel that set in TransmitChList.
5.The next two filter is P3TemporalFilter and LinearClassifier ,I realized them followed the BCI2000 wiki page .
Then ,my matlab Classification result is equal to the corresponding .applog file's result .This .dat file was acquired by means of NeuroScan.
I hope this detail describe would help someone who meet the same problem as me !
While there still is some problem :
1. When I apply my re-write matlab code to the .dat file which was acquired by P3Speller_SignalGenerator.bat ,I can not get the same result as the result in the corresponding .applog file.
2.
By default, signal data will be in raw A/D units, and will be represented by the smallest Matlab data type that accommodates them. To obtain signal data calibrated into physical units (microvolts), specify '-calibrated' as an option anywhere in the argument list.
How can I know what the value of '-calibrated' should be set ?



Thanks for your timely reply ,
Sandra.

boulay
Posts: 382
Joined: 25 Dec 2011, 21:14

Re: P300's .dat file

Post by boulay » 17 Sep 2014, 10:35

Are you using the function load_bcidat? You have two options

Code: Select all

[signal, states, parameters] = load_bcidat(filename);
[signal, states, parameters] = load_bcidat(filename, '-calibrated');
The first one will load the data in whatever type it was saved to disk. For example, if the source module saves the data in int16 (to save space), it will load the raw int16 values.
The second one will load the data in uV. For this to work, the datafile must know its conversion factor to translate int16 values to uV. In the example of a National Instruments card with a range of +/- 5 V and 16-bits, this corresponds to X_uV = X_int16 * (10,000,000 uV) / 2^16, or about 152.6. This is not good enough for EEG so an amplifier is needed in front of the NI DAQ, and the gain of that amplifier would be represented in the SourceChGain parameter (1/amplification, I think).

It's possible that the NeuroScan source module saves the data in uV so the -calibrated option will do nothing, whereas the SignalGenerator saves the data in int16 so the -calibrated flag makes a large difference.

By the way, I'm not sure, but I think using '-calibrated' when loading your dat files gets rid of the need to use your DataIOFilter. Please check.

And yes, CAR uses only what's available to it. It only sees the data that the previous filter (TransmissionFilter) gives it. For your testing, however, I'd suggest not using CAR spatial filter and instead using all 1's on the diagonal of the full matrix (i.e., no spatial filtering).

sandra2014
Posts: 17
Joined: 09 Sep 2014, 09:41

Re: P300's .dat file

Post by sandra2014 » 17 Sep 2014, 23:09

Yes , I use the function load_bcidat .
1.

Code: Select all

[signal, states, parameters] = load_bcidat(filename, '-calibrated');
I tested this code with my matlab code ,and I can get same result when apply to my data acquired from NeuroScan .In fact the data which is loaded by '-calibrated' code is equal to the output acquired form raw data pass through DataIOFilter . So you are right ,if use the '-calibrated' code I can get rid of the need to use DataIOFilter .
2.
boulay wrote: It's possible that the NeuroScan source module saves the data in uV so the -calibrated option will do nothing, whereas the SignalGenerator saves the data in int16 so the -calibrated flag makes a large difference.
I also tested this code with the .dat file which acquired from P3Speller_SignalGenerator , but there is still some error .Beside that the result is same as the data pass through DataIOFilter . Where the problem is ?

Thanks ,I have learnt many within these reply !
Sandra
Attachments
Error.png

boulay
Posts: 382
Joined: 25 Dec 2011, 21:14

Re: P300's .dat file

Post by boulay » 18 Sep 2014, 11:29

Maybe you've found a bug and signalgenerator does not output the data to the dat file exactly the same as it sends it through the signal processing pipeline. This is a bit beyond my time/ability to investigate right now. I hope Juergen will take a look when he returns (in October, I think).

I think the fact that it works with your NeuroScan data indicates that your offline processing in Matlab is correct. I hope you are no longer stuck on this problem.

sandra2014
Posts: 17
Joined: 09 Sep 2014, 09:41

Re: P300's .dat file

Post by sandra2014 » 23 Sep 2014, 08:54

Thanks, sincerely !

Sandra.

Locked

Who is online

Users browsing this forum: No registered users and 19 guests