BCI2000 SMR + Unity3D integration

Forum for discussion on different user applications
Post Reply
karlmor
Posts: 5
Joined: 12 Mar 2016, 09:54

BCI2000 SMR + Unity3D integration

Post by karlmor » 12 Mar 2016, 11:22

Hi all,

I am fairly new to BCI2000, but want to use it for my research. At the moment I have a game in Unity which I would like to control with motor imagery using BioSemi Active 2. Moreover, I would like Unity3D to send event triggers (start of game, showing of certain scenes, etc.) for BCI2000 to save in my files.

I have taken the introductory tour and the Mu rhythm tutorial, but I am not sure what to change in order to launch my game instead of Cursor Task, for example. What are the steps to establish a connection between BCI2000 and an external application?

There is a lot of information available on the wiki, but I am having a hard time filtering what's important. Hope here I can get a little clarification on the matter.

Best,
Karl

pbrunner
Posts: 344
Joined: 17 Sep 2010, 12:43

Re: BCI2000 SMR + Unity3D integration

Post by pbrunner » 14 Mar 2016, 10:14

Karlmor,

the easiest solution is to use the BCI2000 App Connector. This allows you to send event trigger information from an external application to BCI2000 over a UDP connection.

http://www.bci2000.org/wiki/index.php/T ... _Connector

Let me know if this resolves your issue.

Regards, Peter

karlmor
Posts: 5
Joined: 12 Mar 2016, 09:54

Re: BCI2000 SMR + Unity3D integration

Post by karlmor » 15 Mar 2016, 12:37

Hi Peter,

Thanks for you reply. I was able to use Appconnector in Unity and it seems to be working fine for now.

I have another request though: can you elaborate on the changes that I have to make in the core modules to have the game controlled with SMR? I assume I should use DummyApplication and perhaps use the simulate key presses filter, but I'm lost in what concerns the Signal Processing module and parameters.

Best,
Karl

pbrunner
Posts: 344
Joined: 17 Sep 2010, 12:43

Re: BCI2000 SMR + Unity3D integration

Post by pbrunner » 23 Mar 2016, 09:21

Karl,

for the training part you need the BioSemi source, the ARSignalProcessing and the CursorTask as your three modules. The CursorTask will be used to provide feedback using a basket task. Once you have trained the human subject, you have three options on how to proceed:

(1) If you need continuous control, you can replace the CursorTask with the DummyApplication and just have the subject perceive the feedback through your Unity game. Your game is then controlled through the Signal variable.

(2) If you need a discrete decision signal (e.g., left/right) you can keep the CursorTask and place the screen next to your gaming screen and only have the game react on the TargetCode variable.

(3) If you need a more tight integration, you can program your own Application from a template [1] that provides more tight integration, e.g., only presents the feedback bar during certain stages of the game. This the requires a bidirectional communication with your Unity game.

Regards, Peter

[1] http://www.bci2000.org/wiki/index.php/P ... a_template

karlmor
Posts: 5
Joined: 12 Mar 2016, 09:54

Re: BCI2000 SMR + Unity3D integration

Post by karlmor » 05 Apr 2016, 07:42

Hi Peter,

Thanks a lot for the help, it was very useful to get started.

I followed tip 1) and connected Unity using the AppConector and seems to be working fine. However, I wish to have more control over BCI2000 from inside the game (ex. when to stop/start recording etc.) so maybe it would be better to use the BCI2000Automation library.

I was following this [1] thread hopping to get the same result, however Unity can't get access to the library. I've included the BCI2000Automation.dll in the Assets folder, and get the error: "The type or namespace name `BCI2000Automation' could not be found. Are you missing a using directive or an assembly reference?".

According to here [2] it could be because the library is written in C++ and not C#, but my version is Unity 5 which should deal well with unmanaged code.
I tried to follow the [2] tutorial, adding the BCI2000Automation.dll as a solution but I'm pretty sure it didn't work due to my non-existing skills working with this type of things :) here's the error [3] after setting Properties to .NET Framework 3.5

Any tip on how to proceed would be greatly appreciated!

[1] - http://www.bci2000.org/phpbb/viewtopic.php?t=1282
[2] - http://ericeastwood.com/blog/17/unity-a ... -unmanaged
[3] - http://i.imgur.com/BztGgny.jpg?1

pbrunner
Posts: 344
Joined: 17 Sep 2010, 12:43

Re: BCI2000 SMR + Unity3D integration

Post by pbrunner » 07 Apr 2016, 11:20

Karlmor,

I think the first step is to ensure that you have properly installed the BCI2000Automation.dll. For this please follow the instructions in [1]. Then follow one of the two examples. If the example works, the issue is with your C# code. Please report back and we then can look into that issue.

Regards, Peter

[1] http://www.bci2000.org/wiki/index.php/C ... stallation

karlmor
Posts: 5
Joined: 12 Mar 2016, 09:54

Re: BCI2000 SMR + Unity3D integration

Post by karlmor » 07 Apr 2016, 14:51

Hi Peter,

I have tried running the Matlab example you referenced and it worked in a 32-bit installation of Matlab, after unsuccessfully trying in the 64-bit version which reported this error:

Code: Select all

Error using feval
Server Creation Failed: Class not registered

Error in actxserver (line 86)
        h=feval(['COM.' convertedProgID], 'server', machinename, interface);

Error in testAutomation (line 2)
bci = actxserver( 'BCI2000Automation.BCI2000Remote' );
So now I believe the problem is either in my C# code or with importing the .dll in Unity.

Best,
Karl

pbrunner
Posts: 344
Joined: 17 Sep 2010, 12:43

Re: BCI2000 SMR + Unity3D integration

Post by pbrunner » 07 Apr 2016, 17:02

Karl,

this brings us closer to the issue. The bit-ness of your application has to match that of BCI2000. Are you using the latest v3.6 beta, which comes in 32- and 64-bit flavors, or the previous release that only comes in 32-bit? If your Application is in 64-bit try to use the v3.6 beta x64 binaries:

http://www.bci2000.org/phpbb/viewtopic.php?f=1&t=2955

Let me know if this resolves the issue.

Regards, Peter

karlmor
Posts: 5
Joined: 12 Mar 2016, 09:54

Re: BCI2000 SMR + Unity3D integration

Post by karlmor » 08 Apr 2016, 09:14

Peter,

I was using the previous 32-bit Contrib version, and now tested the latest 64-bit version which works under 64-bit Matlab but not Unity. I suspect the problem might be the .Net Framework of the Automation.dll due to the following:

> Tried running the code in [1] with Visual Studio 2015, opening my Unity project .sln and adding the BCI2000Automation64.dll to the project References. This worked well without any errors, and all functions under the BCI2000Remote class were recognized.

> Tried replicating the above, opening my Unity project in Unity and manually adding the reference in MonoDevelop. The following error appeared:

Code: Select all

File 'C:\BCI2000\BCI2000 v3.6.beta.R5304\BCI2000.x64\prog\BCI2000Automation64.dll' is not a valid .Net Assembly
And this one on the Unity console:

Code: Select all

Assets/bci2000_controller.cs(4,7): error CS0246: The type or namespace name `BCI2000AutomationLib' could not be found. Are you missing a using directive or an assembly reference?
> Tried importing the .dll directly calling '[DllImport("BCI2000AutomationLib64.dll")]' instead of 'using BCI2000AutomationLib'. This reports the same 'missing assembly reference' above error. This might be because the syntax is incorrect, and I should declare my functions/classes after the DlllImport according to here[2]. However I don't know how to correctly call them.

Hope this is enough to clarify the issue. Let me know which steps to take next.

Best,
Karl

[1] - http://www.bci2000.org/phpbb/viewtopic.php?t=1282#p4582
[2] - http://stackoverflow.com/questions/3150 ... sharp-code

pbrunner
Posts: 344
Joined: 17 Sep 2010, 12:43

Re: BCI2000 SMR + Unity3D integration

Post by pbrunner » 08 Apr 2016, 12:32

Karl,

unfortunately, I am not an expert in .NET, so I also had to google how to do that. You can try the following example:

http://stackoverflow.com/questions/1592 ... in-unity3d

If this does not resolve the issue, you might have to post the specific question into the Unity forums. I am sure people developing in Unity must have been in similar situations before.

Let me know if this helps.

Regards, Peter

Post Reply

Who is online

Users browsing this forum: No registered users and 23 guests