Contributions:NetworkHID
Synopsis
Functionality for sending commands from BCI2000 to an external USB device which emulates a USB Human Interface Device (HID), that is, a keyboard and mouse. This consists of three components: a snippet of Python code which runs on a CircuitPython-capable microcontroller with Wi-Fi and USB functionality, a C++ class within BCI2000 which provides an interface for sending commands to the microcontroller from within a BCI2000 module, and an BCI2000 extension which mirrors mouse and keyboard from the BCI2000 computer to the microcontroller.
Location
Microcontroller code: http://www.bci2000.org/svn/trunk/src/extlib/NetworkHID/code.py
BCI2000 internal interface http://www.bci2000.org/svn/trunk/src/extlib/NetworkHID/
BCI2000 Extension http://www.bci2000.org/svn/trunk/src/extlib/NetworkHID/
Authors
Ty Butler (butler@neurotechcenter.org)
Functional Description
Certain assistive devices allow input only via USB keyboard and mouse. Enabling BCI2000 to send keyboard and mouse inputs allows for the control of these devices by biosignal data.
A NetworkHID setup consists of one computer running BCI2000, and a microcontroller connected to a target device. The microcontroller acts as a wireless access point, that is, the BCI2000 computer can connect to it over Wi-Fi, and can then send it commands. These commands can direct the microcontroller to press and release keys or mouse buttons, as well as move the mouse of the target device. In this way, BCI2000 can control the target device, either mirroring the keyboard and mouse inputs of the BCI2000 via the extension, or sending inputs in response to BCI2000's internal state, for example, moving the mouse in accordance with the cursor in a Cursor Task, or entering the chosen letters in a Speller Task.
NetworkHID Protocol Details
When a computer connects to the microcontroller's Wi-Fi network, the microcontroller itself will also be present on the network with its own IP address. This address is device-dependent, but on a Pi Pico 2 W, it is 192.168.4.1. It will expose a port (by default 1024 as set in code.py) to receive commands.
A NetworkHID command consists of 16 bytes, in the following format:
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Magic | Type | Arg1 | Arg2 | ||||||||||||
Each of Magic, Type, Arg1, and Arg2 is a 32-bit integer in big-endian form. Magic and Type are unsigned, while Arg1 and Arg2 are signed.
Magic
The constant value 0x0bc12000.
Type
The type of command being sent. This can be one of four values:
- Key Press
- Key Release
- Move Mouse
- Mouse Button
Arg1 and Arg2
These depend on the Type, as detailed below
NetworkHID Commands
These commands use the Adafruit HID library to emulate an HID.
Key Press
Press a key. Takes Arg1 to be an ASCII character value, and presses the corresponding key, or combination of keys, necessary to input the character. The keys will remain held until they are released by a Key Release command. Currently, only ASCII characters are supported, and sending non-ASCII values will cause the device to crash, at which point it must be unplugged and plugged back in, and the client computer must reconnect to its Wi-Fi network.
Key Release
Release the keys corresponding to the ASCII key value given by Arg1.
Move Mouse
Move the mouse by (Arg1,Arg2) units. The movement is relative to the current position of the mouse, and the units are those defined by the Adafruit HID library.
Due to limitations of the library, these values are clamped to the range (-100,100). In practice, if you are implementing precise control of the mouse using the NetworkHID, you will need to test until you find suitable movement values. Additionally, due to the relative nature of mouse movement, it may be difficult to maintain a consistent mouse position. The workaround for this is, when you need the mouse in a precise location, to move it to the top left corner of the screen to establish a known position by repeatedly sending Mouse Move commands with parameter (-100,100), and then moving the mouse to the desired location.
Mouse Button
Press or release mouse buttons. Arg1 and Arg2 correspond to buttons Mouse 1 and Mouse 2. If the argument is 1, the button is pressed, and if the argument is 0, the button is released.
Building a NetworkHID device
The NetworkHID CircuitPython code has been tested with a Raspberry Pi Pico 2 W microcontroller, although theoretically it should work on any device which supports CircuitPython and is capable of acting as a wireless access point.
Acquire a microcontroller and install CircuitPython on it.
After installing CircuitPython, the microcontroller should appear as a storage device when plugged into your computer. Copy the code.py file into the device, replacing the existing one. (This file can be found within the BCI2000 source code at src/extlib/NetworkHID/code.py. Alternatively, it is also located externally at https://codeberg.org/personator01/bci2000-nethid.
Once you plug the microcontroller into the target device, it will begin to act as a wireless access point. It will host a Wi-Fi network, with default name "pico" and default password "pass12345" (these values can be changed by editing the code.py file). On the BCI2000 computer, connect to this Wi-Fi network.
The device is now ready to receive commands from BCI2000.
The NetworkHID Extension
The NetworkHID extension is a source module extension that takes keyboard and mouse inputs made on the BCI2000 computer and mirrors them on the target device, via the NetworkHID microcontroller. It does this by reading the built-in KeyDown, KeyUp, MousePosX, MousePosY, and MouseKeys state variables, detecting when they have changed, and translating them into commands sent over the network to the microcontroller. Due to the synchronous nature of BCI2000, this happens once per sample block processing loop.
It is compiled by enabling the EXTENSIONS_NETWORKHIDEXTENSION CMake variable during the BCI2000 build process (If you have not build BCI2000 from source before, follow the instructions in Programming Howto:Building and Customizing BCI2000).
Once compiled, the NetworkHID Extension can be enabled by adding the --EnableNetworkHIDLogging flag to the source module in your BCI2000 startup script.
Parameters
EnableKeyboardLogging
Send keystrokes from the BCI2000 computer to the microcontroller. Default: 1 (true)
EnableMouseMovementLogging
Send mouse movements from the BCI2000 computer to the microcontroller. Default: 1 (true)
EnableMouseButtonLogging
Send mouse keypresses from the BCI2000 computer to the microcontroller. Default: 1 (true)
MouseMovementLoggingScale
This decimal parameter scales the logged mouse movement. Default: 1.0
NetworkHIDAddress
The network address of the microcontroller's input, in ipv4:port form. On a Pi Pico 2 W with default deployed code.py, this should be 192.168.4.1:1024.
Default: 192.168.4.1:1024
Implementing NetworkHID communication within a custom module
For behavior beyond just mirroring the keyboard and mouse inputs of the host machine, the NetworkHID can be used from a custom BCI2000 module. The NetworkHID class provides an interface within BCI2000 for sending commands to the microcontroller. Its interface is located within src/extlib/NetworkHID/NetworkHID.h. Additionally, the git repository for NetworkHID contains examples of it being used within a custom Cursor Task, and a P3 Speller task.