Tytbutler at 02:31, 16 August 2024

2024-08-16T02:31:14Z

← Older revision		Revision as of 02:31, 16 August 2024
Line 3:		Line 3:
	</pre>		</pre>

	This tutorial goes over how to use BCI2000 as a logging service for an existing Unity project. For a more comprehensive tutorial on integrating Unity with BCI2000, see [User Tutorial:UnityBCI2000]. This tutorial will use a sample Unity project, the same as the full tutorial.		This tutorial goes over how to use BCI2000 as a logging service for an existing Unity project. For a more comprehensive tutorial on integrating Unity with BCI2000, see [[User Tutorial:UnityBCI2000]]. This tutorial will use a sample Unity project, the same as the full tutorial.

	===Download Unity===		===Download Unity===

Tytbutler: Created page with "

 Red text blocks contain detailed instructions for the higher-level instructions preceding them.

This tutorial goes over how to use BCI2000 as a logging service for an existing Unity project. For a more comprehensive tutorial on integrating Unity with BCI2000, see [User Tutorial:UnityBCI2000]. This tutorial will use a sample Unity project, the same as the full tutorial. ===Download Unity=== Download Unity Hub from [https://unity3d.com Uni..."

2024-08-16T02:30:12Z

Created page with "<pre style="color:red"> Red text blocks contain detailed instructions for the higher-level instructions preceding them. </pre> This tutorial goes over how to use BCI2000 as a logging service for an existing Unity project. For a more comprehensive tutorial on integrating Unity with BCI2000, see [User Tutorial:UnityBCI2000]. This tutorial will use a sample Unity project, the same as the full tutorial. ===Download Unity=== Download Unity Hub from [https://unity3d.com Uni..."

New page

<pre style="color:red">
Red text blocks contain detailed instructions for the higher-level instructions preceding them.
</pre>

This tutorial goes over how to use BCI2000 as a logging service for an existing Unity project. For a more comprehensive tutorial on integrating Unity with BCI2000, see [User Tutorial:UnityBCI2000]. This tutorial will use a sample Unity project, the same as the full tutorial.

===Download Unity===

Download Unity Hub from [https://unity3d.com Unity].

===Setting Up BCI2000===

Follow the instructions starting with [https://www.bci2000.org/mediawiki/index.php/Programming_Howto:Install_Prerequisites] to download BCI2000.

===Setting Up UnityBCI2000===
Download UnityBCI2000 from its GitHub page, [https://github.com/neurotechcenter/UnityBCI2000].
<pre style="color:red">
Click on the "Releases" section on the right side of the page. This tutorial is intended to be used with UnityBCI2000 version 2.0.0, so scroll to that release and click on the files BCI2000RemoteNETStandard.dll and UnityBCI2000.cs to download them.
</pre>

===Download Tutorial Project===
Download the CursorTask3 repository from [https://github.com/Personator01/CursorTask3 GitHub].
<pre style="color:red">
Either:
-Clone the git repository
-Download the respository directly from GitHub
-> Click the green "Code" button
-> Select "Download ZIP"
-> Extract the zip file
</pre>
The download directory contains two versions of the CursorTask3 project. One, CursorTask3-BCI2K, contains an example implementation of the task with BCI2000 integration. The other, CursorTask3-NoBCI2K, contains the same task without BCI2000 integration. This tutorial is a step by step guide on turning that second project into the first.

===Opening the project===
Open the project in the Unity Editor, and open the CursorTask scene.
<pre style="color:red">
Open Unity Hub.
Click the "Add" button in the top right corner.
Navigate to the CursorTask3 directory downloaded previously.
Select the CursorTask3-NoBCI2K directory to add it to Unity Hub.
Click the CursorTask3-NoBCI2K project in the Unity Hub.
If prompted to select a version of Unity to use or install, select Unity 2022.3. The specific release number of Unity 2022.3 should not matter, so select the one labeled "LTS" for consistency.
The lower panel (the asset browser) should already show the Assets/Scenes directory, which contains a single scene called SampleScene.
Double-click SampleScene. This should open a scene with a box containing various lights and objects.
</pre>
If you wish to view the completed implementation, repeat these steps, but instead open the CursorTask3-BCI2K-Barebones directory. Note that in order to use the completed implementation, you will still need to set the Operator Path value of the UnityBCI2000 component of the BCI2000 game object.

===Adding UnityBCI2000 to the project===
Add BCI2000RemoteNETStandard.dll and UnityBCI2000.cs to your project's assets.
<pre style="color:red">
In the Unity Editor's menu bar, click Assets > Open Containing Folder to open your project directory.
Place the BCI2000RemoteNETStandard.dll and UnityBCI2000.cs files in the Assets folder.
</pre>

Add an empty GameObject called 'BCI2000'. This will hold the scripts for controlling BCI2000.
<pre style="color:red">
In the Unity Editor's menu bar, click GameObject > Create Empty to create an empty GameObject, and name it 'BCI2000'. (The object's specific name does not actually matter, it is only needed in order to reference it within other scripts)
You will notice that it now appears in the Scene Hierarchy panel, on the left side of the screen.
</pre>

Add the UnityBCI2000 component to the object.
<pre style="color:red">
Click on the object in the Hierarchy panel. This will open it in the Inspector panel (the panel on the right side of the screen)
In the inspector, below the new object's Transform component, click the "Add Component" button, which will open up a small window for adding components.
Within this window, select Scripts > Unity BCI2000 to add the UnityBCI2000 component.
</pre>

===Configuring UnityBCI2000===
The inspector panel now contains the configuration options for UnityBCI2000. Set the options as follows:

<code>Start Local Operator</code>: Checked

This will start the operator on your computer when the Unity scene initializes. If we were instead connecting to an already-running instance of BCI2000, or an instance on another computer, this box would be deselected.

<code>Operator Path</code>: The path to the Operator executable. This will look something like this on Windows (C://path/to/bci2000/prog/Operator.exe)

<code>Operator Address</code>: 127.0.0.1

This is the address of the machine on which BCI2000 is running. Since we are running BCI2000 on the same computer as Unity, we leave it as 127.0.0.1, the loopback address.

<code>Operator Port</code>: 3999

This is the port on which BCI2000 is listening for commands. By default, it is 3999.

<code>Start Modules</code>: Checked

This tells BCI2000 to start the requested Signal Source, Signal Processing, and Application modules when the Unity scene initializes. Similarly to Start Local Operator, we would deselect this box if connecting to an already-running instance.

<code>Start With Scene</code>: Checked

This tells BCI2000 to start a data collection run when the scene starts. Since we will be using BCI2000 itself to set experiment parameters, we will instead wait for BCI2000 to start from Unity, and thus will leave the box unchecked.

<code>Stop With Scene</code>: Checked

This tells BCI2000 to stop collecting data when the scene stops. Since we will be controlling BCI2000 directly, rather than entirely through Unity, we will leave this unchecked.

<code>Shutdown With Scene</code>: Unchecked

This tells BCI2000 to shut down alongside the Unity scene. Whether or not this value is set ultimately doesn't matter much, especially if Start Local Operator is checked. The data will be saved whether or not BCI2000 shuts down.

<code>Module 1</code>: SignalGenerator

The signal source module to start. We will use SignalGenerator, which generates a signal without any connected hardware.

<code>Module 2</code>: DummySignalProcessing

The signal processing module to start. We will use DummySignalProcessing, as there is no processing to do.

<code>Module 3</code>: DummyApplication

The Application module to start. Since we are using Unity, we will use DummyApplication.

===Setting References===
In order for the game's scripts to communicate with BCI2000, they need to hold a reference to the UnityBCI2000 component.
There are three scripts which will need to communicate with BCI2000. They are <code>GameControl.cs, BallControl.cs, and MCursorControl.cs</code>.
These scripts are each located within the Assets directory of the Unity project.
<pre style="color:red">
As before, select Assets > Open Containing Folder to open the project directory, then open the Assets folder.
For each script, open it in a text editor.
Add a data member of type UnityBCI2000 to the class, like so:
UnityBCI2000 bci;
Place the member definition above the Awake() method, for readability.
Within the Awake() method, set this reference to the UnityBCI2000 component.
bci = GameObject.Find("BCI2000").GetComponent<UnityBCI2000>();
</pre>
Each of the three scripts should contain a section like this:
<pre>
...
UnityBCI2000 bci;
void Awake() {
bci = GameObject.Find("BCI2000").GetComponent<UnityBCI2000>();
...
</pre>

===Adding Events===
[https://www.bci2000.org/mediawiki/index.php/Programming_Reference:Events Events] are the primary way that non-signal experiment data is recorded in BCI2000. They are timestamped integer values which are encoded alongside the signal data in BCI2000 output files. Due to BCI2000's design, events must be added during a very specific part of its startup sequence, which is immediately after the BCI2000 operator starts, and before any of the modules start. As such, we cannot simply call <code>AddEvent()</code> whenever we want. Furthermore, the order in which Unity objects initialize is undefined, so it cannot even be guaranteed that calling <code>AddEvent</code> at a specific time will be consistent across multiple projects. As such, UnityBCI2000 provides a couple of methods for sending commands at well-defined points within the startup sequence. These two methods, <code>OnIdle</code> and <code>OnIdle</code> and <code>OnConnected</code> allow BCI2000 commands to be sent while the operator is in the state Idle (immediately before starting its modules) and when the operator is in the state Connected (after starting and connecting to the modules. Below is an example of using those methods to add and show an event in BCI2000.

<pre>
class Script : MonoBehaviour {
UnityBCI2000 bci;
void Awake() {
bci = GameObject.Find("BCI2000Object").GetComponent<UnityBCI2000>();
bci.OnIdle(remote => {
remote.AddEvent("AnEvent", 32);
});
}
void Start() {
bci.OnConnected(remote => {
remote.Visualize("AnEvent");
}
}
}
</pre>

As seen above, <code>OnIdle</code> and <code>OnConnected</code> take a delegate (C#'s term for a callback/closure/functor/etc.) with a single parameter of type <code>BCI2000Remote</code> (in this case called "remote"). The lambda expression given to the call to <code>OnIdle</code> uses the method <code>BCI2000Remote.AddEvent()</code> to add an event called "AnEvent" with a bit width of 32 bits. The call to <code>OnConnected</code> tells BCI2000 to show the event's value in a graphical window. A description of the BCI2000Remote class can be found [https://www.bci2000.org/mediawiki/index.php/Contributions:BCI2000RemoteNET here], and API documentation can be found [https://bci2000.org/BCI2000RemoteNET/netstandard2/classBCI2000RemoteNET_1_1BCI2000Remote.html here].

We will now add the events relevant to the Cursor Task. Open the <code>GameControl.cs</code> script, and modify its <code>Awake()</code> to add the events <code>PreFeedback</code>, <code>Feedback</code>, <code>PostFeedback</code>, <code>TargetHit</code>, and <code>Timeout</code> with bit width 1. Additionally, add the events <code>TrialNumber</code>, <code>TargetPositionX</code>, and <code>TargetPositionY</code> with width 16. These will encode the task state and number of trials.
We also need to set a parameter value so that the signal will be set to the mouse position. Using <code>BCI2000Remote.SetParameter()</code>, we will set the <code>ModulateAmplitude</code> parameter to 1.

Open the <code>BallControl.cs</code> script, and within the <code>Awake()</code> function, add events <code>CursorPositionX</code> and <code>CursorPositionY</code>, with width 16. Show these events in a visualization window with <code>Visualize</code>

Your two scripts should now look like this:
<pre>
>>> GameControl.cs
...
void Awake() {
...
bci.OnIdle(remote => {
remote.AddEvent("PreFeedback", 1);
remote.AddEvent("Feedback", 1);
remote.AddEvent("PostFeedback", 1);
remote.AddEvent("TargetHit", 1);
remote.AddEvent("Timeout", 1);
remote.AddEvent("TrialNumber", 16);
remote.AddEvent("TargetPositionX", 16);
remote.AddEvent("TargetPositionY", 16);
remote.SetParameter("ModulateAmplitude", "1");
});
}

>>> BallControl.cs
...
void Awake() {
bci.OnIdle(remote => {
remote.AddEvent("CursorPositionX", 16);
remote.AddEvent("CursorPositionY", 16);
});
bci.OnConnected(remote => {
remote.Visualize("CursorPositionX");
remote.Visualize("CursorPositionY");
}
</pre>

===Reading the control signal===
We will be using the control signal to control the cursor. Open the <code>MCursorControl.cs</code> script.

The commented out section of the <code>GetPos()</code> method contains the code required to turn the control signal waveform coming from the Signal Source and Signal Processing modules into screen coordinates.
Uncomment the commented part and delete the line <code>Ray r = camera.ScreenPointToRay(Input.MousePosition);</code>.
Change the <code>double signalX = 0;</code> and <code>double signalY = 0;</code> to.

Your <code>GetPos()</code> method should look like this:
<pre>
Vector3 GetPos() {
double signalX = bci.Control.GetSignal(1, bci.CurrentSampleOffset());
double signalY = bci.Control.GetSignal(2, bci.CurrentSampleOffset());
signalsX[signalIndex] = signalX;
signalsY[signalIndex] = signalY;

signalIndex = signalIndex + 1 >= rollingMaximumAmount ? 0 : signalIndex + 1;
double max_x = signalsX.Max();
double max_y = signalsY.Max();
Ray r = camera.ScreenPointToRay(new Vector3((float) max_x * Screen.width, (float) max_y * Screen.height, 0));
float p;
if (!plane.Raycast(r, out p)) {
throw new Exception("error casting ray to plane, invalid mouse position?");
}
return r.GetPoint(p);
}
</pre>

Note the use of the <code>UnityBCI2000.CurrentSampleOffset()</code> method. This is a special method which gets the offset into the current block such that the sample is exactly one block length later than when it was collected by the hardware. This is to normalize the latency between when the hardware collects the signal and the software receives the signal, due to how BCI2000 processes data in blocks.

===Sending events back to BCI2000===

The primary way to communicate game state back to BCI2000 is via the use of Events, which are integer values encoded alongside the signal data.

We will send the events that we added in a previous section.

First, we will send back the current position of the cursor. Within the <code>BallControl.cs</code> script, within the <code>Update()</code> method, immediately after <code>Move()</code> is called, set the <code>CursorPositionX</code> and <code>CursorPositionY</code> events.

<pre>
>>> BallControl.cs
void Update() {
...
if (isTrialRunning) {
...
Move();
bci.Control.SetEvent("CursorPositionX", (uint) ((transform.position.x + 7) * 1000));
bci.Control.SetEvent("CursorPositionY", (uint) ((transform.position.y + 4.5) * 1000));
}
...
</pre>

Notice that we transform the value of the cursor's position. This is because events in BCI2000 are represented as unsigned integers, so, for example, the cursor's range of movement in the x axis, -7 to 7, would not be directly representable within a BCI2000 event, so we add 7 so it is positive, and multiply by 1000 so that we have a more precise measure of the cursor's position. The range of [-7,7] becomes [0,14000].

We will also set the events corresponding to the game state within <code>GameControl.cs</code>. First we will set <code>PreFeedback</code>, <code>Feedback</code>, and <code>PostFeedback</code> to represent when the game is in these states. To do this, at the beginning and end of <code>PreTrial()</code>, <code>Trial()</code>, and <code>PostTrial()</code>, we will set the corresponding event values to 1 and 0, respectively.

<pre>
>>> GameControl.cs
IEnumerator PreTrial() {
bci.Control.SetEvent("PreFeedback", 1);
...
bci.Control.SetEvent("PreFeedback", 0);
}
IEnumerator Trial() {
bci.Control.SetEvent("Feedback", 1);
...
bci.Control.SetEvent("Feedback", 0);
}
IEnumerator PostTrial() {
bci.Control.SetEvent("PostFeedback", 1);
...
bci.Control.SetEvent("PostFeedback", 0);
}
</pre>

We will also set the events which happen at the end of each trial. The <code>TargetHit</code> event is activated when the subject hits the target, and the <code>Timeout</code> event is activated when the subject runs out of time.

<pre>
>>> GameControl.cs
IEnumerator Trial() {
...
bci.Control.SetEvent("Feedback", 0);
if (lastTrialSucceeded) {
bci.Control.PulseEvent("TargetHit", 1);
} else {
bci.Control.PulseEvent("Timeout", 1);
}
}

Notice that we use <code>BCI2000Remote.PulseEvent()</code> rather than <code>BCI2000Remote.SetEvent()</code>. This results in the event being set to the value <code>1</code> for exactly one sample duration, then returning to zero.

We will also record the position of the target. Similarly to the cursor, we will transform the target's coordinates to be a positive integer.

<pre>
>>> GameControl.cs
IEnumerator PreTrial() {
...
target.SetActive(true);
bci.Control.SetEvent("TargetPositionX", (uint) ((target.transform.position.x + 7) * 1000));
bci.Control.SetEvent("TargetPositionY", (uint) ((target.transform.position.y + 4.5) * 1000));
...
}
</pre>

The last event we need to set is the trial number.

<pre>
IEnumerator ControlLoop() {
...
while (IsContinue() && trials < n_trials) {
bci.Control.SetEvent("TrialNumber", trials + 1);
...
}
}
</pre>

===Unity Player Settings===
Additionally, due to how Unity detects changes in BCI2000 state, it must be allowed to run in the background.

<pre style="color:red">
In the Unity menu bar:
Edit > Project Settings > Player > Resolution and Presentation
Check the "Run In Background" box.
</pre>

===Usage===

Now, when the Unity application runs, BCI2000 will open and start collecting data.

User Tutorial:UnityBCI2000 Barebones - Revision history

Tytbutler at 02:31, 16 August 2024