Synchronising two amplifiers over 2 com ports

[ageronimo]

We are pulling data from 2 custom amplifiers over two different usb connections. As expected, we have a synchronisation issue. We think it is a hardware miscmatch with one board consistently recording data faster than the other (clock disrepancy issue?). Since the software pulls the data at the same rate from the buffers in Process, this causes the board to add data more quickly to the com buffer, and eventually an overload of that buffer.
You can see in this image below we are recording a 100 Hz signal from channel 1 of each amplifier. The "faster" amplifier is the orange signal, which fits fewer 100Hz cycles into 1000 points of data. The second figure shows how the latency between cycle peaks changes over time. Over one second of data, the faster amp accumulates 4 additional samples in the buffer.
We are thinking this cant be resolved in software without hardware synchronization. Do you agree?

[mellinger]

I agree that hardware synchronization would be the optimal solution to the problem.

Do you need to process the output of both amplifiers online, e.g. for feedback or stimulation? If not, then there _is_ a software solution. It's called SecondaryBCI2000Controller and must be enabled under "Extensions" in CMake before it can be used.
Basically, it allows you to run two (or more) instances of BCI2000 in parallel, with each instance writing its own data file.
You may then synchronize the two amplifier's signals in software by doing appropriate resampling in your analysis environment, e.g. Matlab, matching the SourceTime timestamp from each file to the other.

Please let me know if you need more information about that solution.

[ageronimo]

Thanks Jurgen, I havent used the secondary controller before. Ill give it a try and reach back out if i have more questions.

Where does this become impossible in an online implementation?

Two BCI2000 instances sending data to a Matlab/Python processing filter, which combines the channels using the resampling method you describe, filters and classifies, and sends an output signal to one instance for application updating. Resampling process too slow? Buffer of cached data from the "fast" amplifier becomes too large?

[mellinger]

Actually, a software solution might be possible if both data streams come with a time stamp from the amplifier side, and amplifier realtime clocks are synchronized (e.g., using PTP). Alternatively, if the timing of the data blocks from the amplifiers is regular, time stamps on the BCI2000 side taken when blocks come in might suffice for synchronizing.

The buffer would need to be about 2 blocks of data, which might be too much delay for an online system. If that is not an issue, resampling could probably be done online. You would use the first amplifier (doesn't matter which one) as a primary clock, and write its data into the buffer when it arrives. When data arrives from the second amplifier, a secondary time stamp is taken and used to line up the second amplifier's samples at the right positions in the buffer, relative to the first amplifier's samples. Resampling would be minimal and would not require any anti-alias filtering.

[ageronimo]

An update -- we did get the secondary BCI controller to work. If we acquire from the "faster" amplifier (~1002Hz sampling rate) with the secondary BCI2000 instance, then the run from the secondary instance ends before the primary instance, causing an error at the end of the run when it sends the StopRun() command to the second instance. If we put the faster amplifier in the primary spot, we don't get this issue.

Unfortunately, we have no hardware time stamp. We will have to synchronize by BCI2000 SourceTime. Why do you say the buffer would only need to be 2 blocks? If the sampling rates of 4Hz out of sync, wouldnt we need to keep a buffer of the faster data that is 4 times the expected number of seconds in a run?

[mellinger]

then the run from the secondary instance ends before the primary instance, causing an error at the end of the run

Typically, you don't have runs in the secondary instance. It should just be dummy signal processing and dummy application. This way, the secondary instance is under full control of the first instance.

If for some reason you need to have runs in the secondary instance it gets complicated, yes.

Why do you say the buffer would only need to be 2 blocks?

Because the block from a secondary amplifier is about the same size as the block from the primary amplifier. Thus, a secondary block will arrive at any time from the beginning of a primary block, to the end of a primary block. If it arrives at the end of a primary block, the secondary samples must be aligned to the primary block. If it arrives at the beginning of a primary block, the secondary samples must be aligned to one block before the current primary block.