Modifying .dat file

Forum for software developers to discuss BCI2000 software development
gigim
Posts: 6
Joined: 26 May 2014, 10:54

Modifying .dat file

Post by gigim » 29 Jul 2014, 06:52

Hi,
I've modified the spatial filter, but I don't see the correct results after the execution of P3Speller. Pratically I've assigned zero to the GenericSignal& Output elements (Output(i,j)=0.0), but the .dat file is not modified.
Please, help me!
Thanks
Ps: Sorry for my english!

boulay
Posts: 382
Joined: 25 Dec 2011, 21:14

Re: Modifying .dat file

Post by boulay » 29 Jul 2014, 09:07

The dat file is stored by the Signal Source module, before the data get passed on to the Signal Processing module (which the spatial filter is part of), so it is completely expected that the dat file will not reflect any changes you make to the spatial filter.

gigim
Posts: 6
Joined: 26 May 2014, 10:54

Re: Modifying .dat file

Post by gigim » 30 Jul 2014, 14:12

Thank you so much.
And if i want to modify the .dat file with an on-line filter, what do I have to do?
When I use this files in the classifier, I can't influence the files for the training with a filter (ex. artifact removal filter). Is it possible?
Thank you again

boulay
Posts: 382
Joined: 25 Dec 2011, 21:14

Re: Modifying .dat file

Post by boulay » 30 Jul 2014, 14:30

It is possible, for example with Brain Vision Recorder, to have one system record the raw data before it is passed to BCI2000. In these cases, it is OK to heavily modify the data within the signal source module before it is stored to .dat file because you know that you have access to the raw data through the other system. However, if you do not have such a system, then it is generally a bad idea to store heavily processed data; you should store the (almost) raw data instead. A few things that you can do to the raw data before storing are basic filtering and converting data types. I would advise against doing anything else to the data before it is stored.

Though the data are stored in their (almost) raw format, it is possible to process them online to convert them into a useful feedback signal. Spatial filters, temporal filters, spectral estimates, etc, can all be applied to the data online, even though these processing steps will not be saved to disk. If you really want, you can create a new state variable for each of your processing steps and copy the processed data into the state variable.

For off-line analysis, you would load the raw data file (e.g., into Matlab) then either try to reproduce your online processing steps (BCI2000 has some commandline functions to help with this; see bci2000chain) or to apply an entirely new processing pipeline to it.

I hope this makes sense.

-Chad

gigim
Posts: 6
Joined: 26 May 2014, 10:54

Re: Modifying .dat file

Post by gigim » 07 Aug 2014, 21:11

Thank you.
I want to convert the dat files into useful feedback signal after the filtering, but I'm not practical. How can save the processed data into state variable? There are example?

boulay
Posts: 382
Joined: 25 Dec 2011, 21:14

Re: Modifying .dat file

Post by boulay » 07 Aug 2014, 21:34

I'm sorry I don't fully understand.

Do you already have pre-recorded dat files and you want to do online signal processing to them and convert them into a useful feedback? If so then you should use BCI2000's FilePlayback module.

Or do you mean you want you have some dat files and you want to do offline signal processing to them to extract a control feature, then you'll run that control feature through some other software? If so, then this analysis should be done outside of BCI2000. BCI2000 has tools for loading the data into Matlab and Python, so those are probably your best places to start.

Or do you mean you have brain signals being acquired from a living creature and you want to convert those signals, in real-time, to useful feedback?

gigim
Posts: 6
Joined: 26 May 2014, 10:54

Re: Modifying .dat file

Post by gigim » 08 Aug 2014, 03:03

I'm sorry, but my English is so bad.
I want to convert the signals, in real-time, to useful feedback.
Thanks again!

boulay
Posts: 382
Joined: 25 Dec 2011, 21:14

Re: Modifying .dat file

Post by boulay » 08 Aug 2014, 09:55

OK. Then this really has nothing to do with dat files or storing data to disk. The online processing and feedback happen separately from data storage.

Have you gone through the BCI2000 User Tutorials?

gigim
Posts: 6
Joined: 26 May 2014, 10:54

Re: Modifying .dat file

Post by gigim » 08 Aug 2014, 14:29

Ok, thanks. But if I want to use the feedback signal for the classification, I must save the filtered signals to disk.
I'm sorry, but I'm very confused

boulay
Posts: 382
Joined: 25 Dec 2011, 21:14

Re: Modifying .dat file

Post by boulay » 08 Aug 2014, 14:57

The output of the SignalProcessing module is your "Control Signal" or "Feedback Signal". How you use this on-line (in real-time) is up to your Application module. E.g., If your application module is the 1-D cursor control task, then the trial classification depends on whether the integral of your control signal over X seconds is positive or negative. This is all done in volatile memory. There is no need to store the control signal to disk.

The only reason you would need to store ANYTHING to disk is if you want to do offline analysis. The only reason I can think of to store your control signal (to a state) is if you want to tinker with different feedback styles offline (e.g. compare bars to arrows to cursors) and you don't want to recreate the signal processing that happened online. If you want to recreate the feedback exactly then you should save your feedback object's values to state variables (e.g., CursorPosX, CursorPosY).

Maybe if you explained what you want to do in more detail then I might be able to better answer your question.

gigim
Posts: 6
Joined: 26 May 2014, 10:54

Re: Modifying .dat file

Post by gigim » 09 Aug 2014, 07:10

I'm sorry for my infinite questions. I want to use a simple on-line filter for artifact removal in the P3 Speller. The idea was: during the training step I create the .dat files and after I use this files in P300Classifier for the classification. The signals must be filtered on-line to obtain better classification, but if my filter doesn't modify the resulting files, it's all to no avail.
That's all.
Thanks for your answers and excuse me again

boulay
Posts: 382
Joined: 25 Dec 2011, 21:14

Re: Modifying .dat file

Post by boulay » 09 Aug 2014, 23:32

gigim wrote:I'm sorry for my infinite questions. I want to use a simple on-line filter for artifact removal in the P3 Speller. The idea was: during the training step I create the .dat files and after I use this files in P300Classifier for the classification. The signals must be filtered on-line to obtain better classification, but if my filter doesn't modify the resulting files, it's all to no avail.
There's nothing simple about online artifact removal.

If I understand you correctly, you want to do online artifact removal in the training step applied before the data are stored so you have better data available for the P300Classifier. However, you should continue to store the raw data before the artifact removal step and instead do artifact removal offline before classification.

If you are using Matlab, you can use tools to load the dat file into matlab, modify the signal, then save it back as a (different) dat file.

Or, if you want to use your own classifier instead of the P300Classifier then you can implement a command-line version of your artifact removal filter or you can implement it directly in Matlab. There's nothing special or secret about the P300Classifier; it decimates the data then runs stepwisefit.

It is really important to store your data before the artifact removal step because your artifact removal might misbehave and then it would be impossible to recover the real data.

loop
Posts: 10
Joined: 27 May 2014, 13:50

Re: Modifying .dat file

Post by loop » 13 Aug 2014, 09:37

boulay wrote:The output of the SignalProcessing module is your "Control Signal" or "Feedback Signal". How you use this on-line (in real-time) is up to your Application module. E.g., If your application module is the 1-D cursor control task, then the trial classification depends on whether the integral of your control signal over X seconds is positive or negative. This is all done in volatile memory. There is no need to store the control signal to disk.

The only reason you would need to store ANYTHING to disk is if you want to do offline analysis. The only reason I can think of to store your control signal (to a state) is if you want to tinker with different feedback styles offline (e.g. compare bars to arrows to cursors) and you don't want to recreate the signal processing that happened online. If you want to recreate the feedback exactly then you should save your feedback object's values to state variables (e.g., CursorPosX, CursorPosY).

Maybe if you explained what you want to do in more detail then I might be able to better answer your question.
Hi guys, sorry for my intrusion but I have the same problem.
So, if I understand fine, for create a "realtime filter" I must work on Control Signal or Feedback Signal (what's the diference between this two?), before I process it with my filter that's will be passed to the classifier? This is the only way to process signals in realtime? Do you have any document to suggest me?
Sorry but I'm a newbie
Thank you

boulay
Posts: 382
Joined: 25 Dec 2011, 21:14

Re: Modifying .dat file

Post by boulay » 13 Aug 2014, 11:29

loop wrote: Hi guys, sorry for my intrusion but I have the same problem.
So, if I understand fine, for create a "realtime filter" I must work on Control Signal or Feedback Signal (what's the diference between this two?), before I process it with my filter that's will be passed to the classifier? This is the only way to process signals in realtime? Do you have any document to suggest me?
Sorry but I'm a newbie
Thank you
Sorry if I confused you. "Control Signal" and "Feedback Signal" are the same thing. I should have said "a.k.a." instead of "or".

The control signal is the output of the SignalProcessing module (i.e., after all your filtering) that goes to the application.

The signal goes from Source -> SignalProcessing (many filters in a chain) -> Application. The signals are stored to disk in the Source module in parallel but independent to the transmission of the signals to the signalprocessing module. Then, the SignalProcessing module takes care of the online/realtime filter. Finally, the application converts the output of the SignalProcessing module (i.e. the "Control Signal") into some sort of feedback.

Browse around the wiki at doc.bci2000.org. If you can't find what you need then please ask a specific question on the bulletin board.

loop
Posts: 10
Joined: 27 May 2014, 13:50

Re: Modifying .dat file

Post by loop » 13 Aug 2014, 11:44

Thank you very much for your answer, I think that I've proceded in the right direction (I must insert my filter in SignalProcessing), now I read some documentation if I still have some problems yet I knock on your door.
Thank's Again

Locked

Who is online

Users browsing this forum: No registered users and 1 guest