Modifying .dat file

[loop]

Sorry for my insistence but something escapes me. I read more documentation and I based my filter as this http://www.bci2000.org/wiki/index.php/P ... ing_Filter .
Is a bad way?

Go back...

The signal goes from Source -> SignalProcessing (many filters in a chain) -> Application. The signals are stored to disk in the Source module in parallel but independent to the transmission of the signals to the signalprocessing module. Then, the SignalProcessing module takes care of the online/realtime filter. Finally, the application converts the output of the SignalProcessing module (i.e. the "Control Signal") into some sort of feedback.

Now I think, if the signal are stored in paralell but indipendent to the transmission, what arrive at classifier? The classifier use the dat file, if I don't modify dat file I not will see any difference before the online filter was applied. is so or I misunderstood?

Thank you so much

[boulay]

The dat file gets filled with a copy of the parameters in the header and a continuous copy of the raw (unprocessed) signal and a continuous copy of the state values.

The copying of the data to the dat file happens in the SignalSource module and is independent of the transmission of the signals to the SignalProcessing module. Indeed, you can set your source module to not create a dat file at all but the online processing and feedback still works.

If you want to copy information about something that happens AFTER the source module, then you should use States. For example, in your artifact rejection module, you can create a state that indicates when a segment of data was rejected. Then, when you examine your data offline, you can look for this state value then you'll know easily whether or not a segment of data was rejected online.

If, for some reason, you want to do your artifact rejection before the data gets copied to the dat file then you would have to implement your artifact rejection in the source module. I've stated before why I think this is a terrible idea, and there are many better alternatives to achieve the same result. The one exception to this rule is if the software for your EEG amplifier allows you to record the data independent of BCI2000. For example, BrainVision Recorder is required to get data from Brain Products amplifiers; it acts as a server that BCI2000 can connect to and pull data. You can configure BrainVision Recorder to save your data to disk, and then BCI2000 can process the data before it saves them to disk because you know the original data are saved by Recorder.

[loop]

The dat file gets filled with a copy of the parameters in the header and a continuous copy of the raw (unprocessed) signal and a continuous copy of the state values.

The copying of the data to the dat file happens in the SignalSource module and is independent of the transmission of the signals to the SignalProcessing module. Indeed, you can set your source module to not create a dat file at all but the online processing and feedback still works.

If you want to copy information about something that happens AFTER the source module, then you should use States. For example, in your artifact rejection module, you can create a state that indicates when a segment of data was rejected. Then, when you examine your data offline, you can look for this state value then you'll know easily whether or not a segment of data was rejected online.

If, for some reason, you want to do your artifact rejection before the data gets copied to the dat file then you would have to implement your artifact rejection in the source module. I've stated before why I think this is a terrible idea, and there are many better alternatives to achieve the same result. The one exception to this rule is if the software for your EEG amplifier allows you to record the data independent of BCI2000. For example, BrainVision Recorder is required to get data from Brain Products amplifiers; it acts as a server that BCI2000 can connect to and pull data. You can configure BrainVision Recorder to save your data to disk, and then BCI2000 can process the data before it saves them to disk because you know the original data are saved by Recorder.

First of all thank you very much for your help.
Unfortunately I continue to confuse, I try to explain my problem and see if you can tell me the way. So, I work with P300 Speller, as you now signals are noise affected, I would remove blink eyes noise in real time and I write a filter for that but if I understand the classifier work on dat file but my filter will not write on dat file, hence classification will do always on signal noised. I understand right? What do you advise me to solve my problem?
I hope I was a little clearer.
Thank you again

[loop]

The dat file gets filled with a copy of the parameters in the header and a continuous copy of the raw (unprocessed) signal and a continuous copy of the state values.

The copying of the data to the dat file happens in the SignalSource module and is independent of the transmission of the signals to the SignalProcessing module. Indeed, you can set your source module to not create a dat file at all but the online processing and feedback still works.

If you want to copy information about something that happens AFTER the source module, then you should use States. For example, in your artifact rejection module, you can create a state that indicates when a segment of data was rejected. Then, when you examine your data offline, you can look for this state value then you'll know easily whether or not a segment of data was rejected online.

If, for some reason, you want to do your artifact rejection before the data gets copied to the dat file then you would have to implement your artifact rejection in the source module. I've stated before why I think this is a terrible idea, and there are many better alternatives to achieve the same result. The one exception to this rule is if the software for your EEG amplifier allows you to record the data independent of BCI2000. For example, BrainVision Recorder is required to get data from Brain Products amplifiers; it acts as a server that BCI2000 can connect to and pull data. You can configure BrainVision Recorder to save your data to disk, and then BCI2000 can process the data before it saves them to disk because you know the original data are saved by Recorder.

First of all thank you very much for your help.
Unfortunately I continue to confuse, I try to explain my problem and see if you can tell me the way. So, I work with P300 Speller, as you now signals are noise affected, I would remove blink eyes noise in real time and I write a filter for that but if I understand the classifier work on dat file but my filter will not write on dat file, hence classification will do always on signal noised. I understand right? What do you advise me to solve my problem?
I hope I was a little clearer.
Thank you again

[boulay]

Yes that was clearer. Actually I answered that question previously but I wasn't clear the first time.

First, I should say that you are not the target audience for the P300Classifier. The P300Classifier is designed for people that do not stray from the standard protocol. If you write your own custom artifact rejection filter and you want to use that filter in your offline analysis then it is expected that your offline analysis will involve a lot of customization and therefore you cannot use the P300Classifier.

The P300Classifier expects raw (unprocessed) data. I didn't write the P300Classifier, but a brief look at the code suggests to me that the first thing the P300Classifier does is run the data through a common average reference. Thus, if you really want to use the P300Classifier, and you really want to use your artifact rejection filter on the data before you use the P300Classifier, then you have to take the raw data, process it offline up until your artifact rejection step, then undo the other processing steps, then save the data to a dat file. This is complicated.

First, build a
command-line version of your filter and any other filters you want to use on the data before it is passed to P300Classifier.

Then you will have to use these filters offline to take your data from the raw .dat format into an intermediate processed format.

Then you will have to undo the actions of the filters other than your artifact rejection filter.

Then you will have to write the processed data back to a .dat file.

Now, once that is all done, you can load the data in the P300Classifier.

Before you go and do all of that, please answer back here to give me a general idea of how your artifact rejection filter works. If it uses ICA, then the above is not a good method for you.


Also, I recommend not doing the above anyway. If you want to use your filter in the offline analysis then that's fine. Use the techniques I mentioned above. However, instead of undoing the actions of the other filters, writing a dat file, then using P300Classifier, you would be better off leaving the data in the processed state then using custom signal analysis techniques to build your classifier.

[loop]

Thank you boulay, now it's more clear.
For answer at your first part: I use wavelet trasform for my filter and as I said I need to use online filter.

So, you write:

However, instead of undoing the actions of the other filters, writing a dat file, then using P300Classifier, you would be better off leaving the data in the processed state then using custom signal analysis techniques to build your classifier.

can you advise any documentation for replace P300Classifier with mine classifier? It's too complicated?
Thank you for your patience

[boulay]

I don't think it's too complicated for you. I will write several posts. Please choose the method you like best.

Do you have Matlab? If so, then you should do the bci2000chain in Matlab (see BCI2000\tools\matlab\ExampleBCI2000Chain.m) to get your data in the processed state. I don't know what your filter chain look like, but you definitely want to stop before you get to the classifier.

If your filter chain has not already done so, you should then downsample your data.

Then, isolate your responses to each stimulus. Note that the responses overlap so you will have to duplicate data.
e.g.

Code: Select all

R1 --------------
R2        --------------
R3                --------------

This becomes your X matrix.

Then, using the state variables, identify the class of each response, either target (attended) or non-target (unattended).
This becomes your y-variable.

Finally, put your X and y into stepwisefit.

stepwisefit will return the coefficients of the features that maximize the between class discrimination. You can then use these feature to create a parameter file.

Admittedly, this process is much more complicated than using P300Classifier, but it gives you full control. You can do cross-validation, you can change the machine-learning techniques, you can change how much you downsample, you can balance your instances of each class, etc.

[boulay]

Another way:
Forget about the command-line implementation of your filter.

Load the data with EEGLAB toolbox for Matlab.
Remove the artifacts with a Matlab implementation of your filter.
Segment the data into "epochs" around your stimuli.
Process the data with BCILAB.
Output the classifier into a prm file.

Another way:
Edit the source code for the P300Classifier and add your own filter to it.
I'm sorry I didn't think about this earlier. This is probably the easiest thing to do, though I don't know how. I've never touched the P300Classifier source code.

Another way:
Process the data in Python using BCPy2000.
Filter it using a Python implementation of your artifact rejection filter.
Machine-learning either with the tools in BCPy2000 or with scikit-learn.
Export the classifier parameters to a prm file.

By the way, whichever method you choose, this paper is worth a read. They used BCPy2000 for this paper but the ideas translate to other environments.

-Chad

[loop]

I've matlab but I suspect that when I interact with it maybe loss realtime.
At this time, I try to modify P300Classifier as you say I will be in touch with you about this.

Process the data in Python using BCPy2000.
Filter it using a Python implementation of your artifact rejection filter.
Machine-learning either with the tools in BCPy2000 or with scikit-learn.
Export the classifier parameters to a prm file.

I don't know, is more performance than matlab? I could work in realtime with it?
Big Thank you

[boulay]

P300Classifier is offline-only. I was merely telling you how to make a substitute for that.

I think I'm confused again.

My understanding is that you want to use your artifact-rejection technique (including wavelets) as part of the online signal processing.

Then, after collecting some training data, you want to do an offline analysis (not in real time) using the P300Classifier. This will give you a set of weights in a prm file. **

Then, you will load these weights into BCI2000 and do another session that gives feedback. This has online processing but no longer has anything to do with P300Classifier.

Are you maybe confusing BCI2000\prog\P3SignalProcessing and BCI2000\tools\P300Classifier?

** As I understood it, our whole discussion was about how to do the offline analysis in a way that included the artifact rejection. There is no need for this to be in real-time as it will be done offline, after you've collected some data.

[loop]

I'm sorry if I confused you, maybe there is a misunderstanding.
Now, my only interest was create an online filter. Can you say me what I do in a simple step?

[boulay]

OK. For future reference, the dat file and P300Classifier have nothing to do with online processing. Please forget about them when talking about online processing.

As for building your own signal processing filter, the quickstart guide has some instructions to help you get started, but I can't really help you with the details of the implementation. That's up to you.

[loop]

Thank you so much boulay for your help and your patience, at this point I think that there isn't great possibility of implement online filters in bci2000, all it's based on offline filter.

Thank you again, I will see what I do
Bye