Video Frame Rate real time control

[bcilab]

Hi,
I would like to implement an application module to control the speed of a video in closed loop. I was thinking about considering the Cursortask module and using the EEG control signal to control the frame rate in real time. Is this possible? How can I control the speed of a video? That is, which variable class will allow me to do that?
Thanks in advance.

[mellinger]

You may use the VideoPlayer class which allows to adjust the frame rate using its SetNominalFrameRate() member function.

Code: Select all

Ratio frameRate(30, Time::Interval::OneSecond());
pVideoPlayer->SetNominalFrameRate(frameRate);

The property is called NominalFrameRate because the VideoPlayer object will actually modify the desired rate to better match a simple ratio between the actual frame rate and the screen refresh rate to avoid visual artifacts.

If you need to adjust the frame rate while the video is running, you will need to take into account that the frame rate is used to determine the current position from the beginning of the file, not just the difference between adjacent frames. I.e., if you reduce the rate too much, the video will move backwards for a frame.

[bcilab]

Thank you very much for your kind and quick reply. I'm trying to implement the application module with the new variable mpVideoPlayer of class videoPlayer a suggested, but something is not working.

In the .h file I have added the line class VideoPlayer*mpVideoPlayer but in the .cpp file I'm not able to define the mpVideoPlayer as a new VideoPlayer as the error said that no default constructor exists for the VideoPlayer class.

So I tried to bypass this step and directly load the file in OnInitialize as: mpVideoPlayer->SetFile(video_string); and no errors were reported. But when I run the BCI batch file with this modification, the system log refers to the following message: ApplicationError: Win32 unhandled exception of type ACCESS_VIOLATION (0xc0000005). Lost connection to application.

So may I ask what is the best way to just load the video first and play it in my module?
I apologize for my questions, I'm a beginner with the BCI2000 coding.
Thanks again

[mellinger]

The error "no default constructor exists" means that there is no constructor without arguments declared for the class you are trying to instantiate.

As you can see in the VideoPlayer.h header file, the constructor expects a GUI::GraphDisplay reference which is necessary to link it to the window which it is supposed to be rendered into.

You may get the window reference by calling ApplicationBase::Window(). So writing

Code: Select all

   ... = new VideoPlayer(ApplicationBase::Window());

should work.

If you don't call "new" then no instance of class VideoPlayer is created, and no valid address is assigned to mpVideoPlayer. Consequently, trying to access the non-existing object, e.g. by calling its member functions, will result in a segfault.

Also, in C++, each occurrence of "new" should be balanced by a call to "delete". If you decide to create the VideoPlayer in OnInitialize(), you should delete the previous one first by calling

Code: Select all

 delete mpVideoPlayer;

immediately before calling "new".

Otherwise, VideoPlayer instances will pile up on your screen, possibly with the top one always showing the initial video, and the others hidden behind.

Moreover, to avoid that your initial "delete" crashes because of mpVideoPlayer pointing to an invalid memory address, you will need to make sure that mpVideoPlayer is initialized to nullptr in the constructor (or at its point of declaration).

Finally, to delete the last instance of VideoPlayer, you will need to add a

Code: Select all

 delete mpVideoPlayer;

to the task filter's destructor.

[bcilab]

Hi, thanks again for your help.
I have followed your instructions and suggestions but still have some problems:

• 1)

I was able to use the constructor for my variable mpVideoPlayer correctly and no errors occur, but when I run my batch file two windows appear on the screen. I probably need to advice you that I'm using the FeedbackDemo application as a template, so all the instructions and feedback
are displayed on mpWindow (initialized as a QWidget) as declared in the Feedback Demo task. How can I have only one window for
everything?

• 2)

I was able to open the file but not to play it, i.e. no errors are prompted but the video is not visible on the window. Here's the line of code I'm using:

Code: Select all

mpVideoPlayer->Show();
mpVideoPlayer->Open(source_video);
if (mpVideoPlayer->IsOpen()){
   Applog << "open" << std::endl;
}
mpVideoPlayer->Play();
if (mpVideoPlayer->IsPlaying()){
   Applog << "play" << std::endl;
}

In the applog window the open string is reported.
Do I'm missing something?


Thanks again in advance

[mellinger]

Hi,

I was able to use the constructor for my variable mpVideoPlayer correctly and no errors occur, but when I run my batch file two windows appear on the screen. I probably need to advice you that I'm using the FeedbackDemo application as a template, so all the instructions and feedback
are displayed on mpWindow (initialized as a QWidget) as declared in the Feedback Demo task. How can I have only one window for
everything?

As you noticed, the FeedbackDemo application uses a plain QWidget, which does not provide BCI2000 GraphDisplay functionality. When you call ApplicationBase::Window(), no such window is found, and a new one is created.

So, basically, you cannot use VideoPlayer on a plain QWidget. I suggest that you switch from FeedbackDemo to CursorTask as a starting point for your application module.

I was able to open the file but not to play it, i.e. no errors are prompted but the video is not visible on the window. Here's the line of code I'm using:

Your code is correct, but I think there is an issue with the additional window being created. I think it should work when you switch to CursorTask.

[mellinger]

As a side note:
Instead of using VideoPlayer::SetNominalFrameRate() you might override VideoPlayer::OnAdvance() in a derived class (e.g., MyVideoPlayer), and call the inherited function with a rescaled argument to modify the speed of playback. This might be easier than computing the proper nominal frame rate corresponding to a desired speed.

Let me know if you need further information.

[bcilab]

Hi, thank you for your support.
I'm trying to follow your suggestions.
Hope I will not need to disturb you again.
Thank you again

[bcilab]

Hi,
I am writing to request assistance once more.
I have successfully converted my code into a cursor task template as you suggested, which has enabled me to accurately reproduce the video. However, I am still encountering issues related to the control of the frame rate.
The second solution you proposed appears to offer a more straightforward approach to controlling the frame rate. However, I am uncertain about the specifics of its implementation. Could you kindly provide me with further details?
In particular how to override and in which way I can input the new frame rate relating to the OnAdvance.
Thank you again in advance

[mellinger]

Hi,

to override function in question, declare a class that inherits from VideoPlayer in a header file:

Code: Select all

#ifndef MY_MODIFIED_VIDEO_PLAYER_H
#define MY_MODIFIED_VIDEO_PLAYER_H

#include "VideoPlayer.h"

class MyModifiedVideoPlayer : public VideoPlayer
{
  public:
    MyModifiedVideoPlayer(GUI::GraphDisplay&, int);
    
    void SetSpeedup(double);
    double Speedup() const;
    
    const char* OnAdvance(int) override;

  private:
    std::atomic<double> mSpeedup;
};

#endif // MY_MODIFIED_VIDEO_PLAYER_H

In the implementation (MyModifedVideoPlayer.cpp), forward the arguments to the VideoPlayer constructor:

Code: Select all

#include "MyModifiedVideoPlayer.h"

MyModifiiedVideoPlayer::MyModifiedVideoPlayer(GUI::GraphDisplay& display, int zpos)
: VideoPlayer(display, zpos), mSpeedup(1.0)
{
}

void MyModifiedVideoPlayer::SetSpeedup(double d)
{
  mSpeedup = d;
}

double MyModifiedVideoPlayer::Speedup() const
{
  return mSpeedup;
}

// In OnAdvance(), rescale the argument and call the base class function:
const char* MyModifiedVideoPlayer::OnAdvance(int count)
{
  int rescaledCount = std::round(count * mSpeedup);
  return VideoPlayer::OnAdvance(rescaledCount);
}

Please let me know if you have further questions, or experience problems with the code.

[bcilab]

Hi,
thank you a lot for your help again.
I followed your instruction but I have the following issues:

• 1. I have generated a new header file for the MyModifiedVideoPlayer class but in the line with OnAdvance it is reported to be the error: retur type is not identical to nor covariant with return type const char

• 2. Is not that clear to me in which file I have to insert the code lines related to the construct and the new OnAdvance that you suggested to me. can you please provide me still more details?

I'm sorry for all my questions.
Thank you

[mellinger]

Sorry, I wrote the code from memory and left out some information.

I corrected and completed the code now.

Let me know if you still have problems.

[bcilab]

Hi,
I am most grateful to you for your helpful suggestions and the codes you provided. At present, all components appear to be functioning correctly.
I would be grateful if you could provide me with one further suggestion regarding the speed parameter.
I would like to connect the mSpeed with the ControlSingnal(0,0) in order to control the frame rate with the eeg activity.
In order to implement a variation of the mSpeed that is visible, can you suggest me a value for the gain?

Code: Select all

mSpeed = Gain * ControlSignal(0,0);
mpVideoPlayer->SetSpeedup(mSpeed):

In general, which nominal values should be taken into consideration with regard to the speed?

Thank you again for the support

[mellinger]

By definition, the Control Signal should be zero mean and unit variance, so you cannot directly apply a gain to it unless you want the possibility for the video to go not only forward but backwards as well.

I don't know what your goal is with this experiment. If it's only about variation of the forward speed of the video, and to try a range of modest speedups/slowdowns, I would suggest to map the speed to the control signal as follows:
71% at -1
100% at 0
141% at +1.

This may be achieved by choosing speed = 2^((ControlSignal)/2):

Code: Select all

mSpeed = ::pow(2.0, ControlSignal(0, 0)/2.0);
mpVideoPlayer->SetSpeedup(mSpeed);

[bcilab]

Hi,
Thanks for your assistance and support.
I am pleased to inform you that the desired functionality has now been achieved.