Contributions:AudioExtension: Difference between revisions
m Setting up format |
|||
| (14 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
==Synopsis== | ==Synopsis== | ||
An environment extension which manages multichannel, low latency audio I/O. | |||
==Location== | ==Location== | ||
http://{{SERVERNAME}}/svn/trunk/src/contrib/Extensions/ | http://{{SERVERNAME}}/svn/trunk/src/contrib/Extensions/AudioExtension | ||
==Versioning== | ==Versioning== | ||
===Authors=== | ===Authors=== | ||
Griffin Milsap (griffin.milsap@gmail.com) | Griffin Milsap (griffin.milsap@gmail.com) | ||
===Version History=== | ===Version History=== | ||
06/ | * 2012/06/11: Initial public release; | ||
===Source Code Revisions=== | ===Source Code Revisions=== | ||
*Initial development: | *Initial development: 4095 | ||
*Tested under: | *Tested under: 4095 | ||
*Known to compile under: | *Known to compile under: 4095 | ||
*Broken since: -- | *Broken since: -- | ||
===Todo=== | ===Todo=== | ||
* | * Fix Known Issues | ||
* | * Add per-sample resolution to envelopes | ||
===Known Issues=== | |||
* Leaving the module in halted state exhibits some sort of bug regarding state logging. When running state is resumed, envelope states may fail to update for 15-30 seconds. The bug seems to be unrelated to how long system was halted -- Not sure if this is an issue with the extension itself, or an issue with the [[Programming_Reference:Events|bcievent]] interface. This bug will never happen on the first run after BCI2000 is started up. If you do see the behavior, either wait for it to go away or restart BCI2000 and perform a new recording. | |||
* Bandpass filtering in filterbanks doesn't appear to function correctly. | |||
* AudioExtension processes audio in a separate thread and the internal audio callback is called from yet-another context (Sometimes a system interrupt). In order to prevent deadlock, the Audio callback must not lock or wait for external threads. As such, it will simply copy the last good audio buffer to the output stream if the audio thread has not posted new data to use yet. This can result in slowed "timestretching" effects on audio input files if the audio thread cannot keep up with the audio callbacks. To prevent this behavior, ensure your audio block size is large enough (at least 1024 frames). If you are using a lower latency audio API (such as ASIO) you are probably okay to use audio block sizes around 512 frames. Either way, be mindful that audio playback may not necessarily operate in real-time and you will receive NO WARNING WHATSOEVER when it fails to. | |||
==Functional Description== | ==Functional Description== | ||
Experiments which require audio input or real-time audio synthesis based on system state are now possible with the AudioExtension. This extension is capable of recording multiple channels of audio input, synthesizing tones or noise, and reading encoded audio files. These channels are input to a mixing matrix which mixes those inputs to multiple channels of audio output. Both input and output are run through a simple filterbank, then they have their envelope extracted and logged into states via the bcievent interface. Audio input and output channels can be recorded into audio files losslessly and can be resynchronized offline. The mixing matrix is a matrix of expressions which can be used to dynamically change audio mixing based on the system state. | |||
==Integration into BCI2000== | ==Integration into BCI2000== | ||
Compile the extension into your source module by enabling contributed extensions in your CMake configuration. You can do this by going into your root build folder and deleting <code>CMakeCache.txt</code> and re-running the project batch file, or by running <code>cmake -i</code> and enabling ''' | Compile the extension into your source module by enabling contributed extensions in your CMake configuration. You can do this by going into your root build folder and deleting <code>CMakeCache.txt</code> and re-running the project batch file, or by running <code>cmake -i</code> and enabling '''BUILD_AUDIOEXTENSION'''. Once the extension is built into the source module, enable it by starting the source module with the <code>--EnableAudioExtension=1</code> command line argument (NB, as explained below, the numeric value here matters, and denotes the audio API to be used: =1 means DirectSound). | ||
==Block Diagram== | |||
[[Image:AudioExtensionBlockDiagram.png]] | |||
==Parameters== | ==Parameters== | ||
The | The AudioExtension is configured in the Source tab within the AudioExtension section. The configurable parameters are: | ||
===EnableAudioExtension=== | |||
* | Enables/Disables the AudioExtension. This parameter performs double-duty as an audio host API selector. The following values of this parameter are valid. NOTE: Not all audio APIs are available on all platforms. | ||
* | **[0] - Disabled | ||
* | **[1] - DirectSound | ||
* | **[2] - MME | ||
* | **[3] - ASIO | ||
* | **[4] - SoundManager | ||
* | **[5] - CoreAudio | ||
* | **[6] - Disabled | ||
**[7] - OSS | |||
**[8] - ALSA | |||
**[9] - AL | |||
**[10] - BeOs | |||
**[11] - WDMKS | |||
**[12] - JACK | |||
**[13] - WASAPI | |||
**[14] - AudioScienceHPI | |||
===AudioMixer=== | |||
This matrix of expressions mixes input (rows) to output(columns). It must be dimensioned with exactly <code>n</code> columns where <code>n</code> is the number of outputs. Row labels define the input source. Change row labels by double clicking on the row. The following inputs are valid row labels. | |||
The | *<code>X</code> - This is automatically interpreted as INPUT[X] | ||
*<code>INPUT[X]</code> - This input will come from channel X on the sound card input. | |||
*<code>FILE[X]</code> - This input will come from channel X in the AudioInputFile. | |||
*<code>TONE[X]</code> - This input will be a synthesized sine wave with the frequency of X Hz. | |||
*<code>NOISE[X]</code> - This input will be generated white noise at X Hz. NOTE: NOISE[] is white noise at the audio sampling rate (which defaults to 44100) | |||
===AudioInputDevice=== | |||
The index for the device to use as the audio input device on the current Host API. See the operator log after "Set Config" for valid device indices on the selected host API. A value of -1 for this parameter selects the default input device on this host API. | |||
===AudioOutputDevice=== | |||
The index for the device to use as the audio input device on the current Host API. See the operator log after "Set Config" for valid device indices on the selected host API. A value of -1 for this parameter selects the default output device on this host API. | |||
===AudioInputFile=== | |||
Audio file to use as audio input to AudioMixer. The selected file can have any non-zero number of channels and be encoded in almost any format (except MP3), but MUST be encoded at 44100 Hz. | |||
===AudioRecordInput=== | |||
Enables/Disables recording of audio data to a file in the DataDirectory. | |||
===AudioRecordOutput=== | |||
Enables/Disables recording of audio data to a file in the DataDirectory. | |||
===AudioRecordingFormat=== | |||
Changes the file format and encoding options of the recorded output files. This parameter has the following three options: | |||
*Raw - Records to 16 bit Microsoft formatted WAV files with no compression. These files open directly in MATLAB if that's interesting to you. | |||
*Lossless - Records to FLAC formatted files. These files are slightly smaller than RAW files, but have no quality loss. | |||
*Lossy - Records to Ogg Vorbis files. These files are similar to MP3 but do not have the associated licensing issues. They are compressed using a lossy algorithm, so the resulting files are very small but sound slightly worse than lossless encoding. This format is good for long recordings where perfect quality is not necessary. | |||
===AudioInputFilterbank, AudioOutputFilterbank=== | |||
A filterbank which filters audio input and output before rectification/smoothing for envelope extraction. These butterworth filters will not be applied to the audible signal. The format of the filter bank is as follows: | |||
*Type - The characteristic of the filter. The following values are valid. | |||
**Lowpass - Creates a low pass filter | |||
**Highpass - Creates a high pass filter | |||
**Bandpass - Creates a band pass filter [[Contributions:AudioExtension#Known_Issues|*See Known Issues*]] | |||
**Bandstop - Creates a band stop, or notch filter | |||
*Order - The order of the filter model. Higher order filters are more accurate but more expensive computationally. | |||
*Cutoff1 - The cutoff frequency for Lowpass and Highpass filters, and the cut-on frequency for Bandpass and Bandstop filters. | |||
*Cutoff2 - The cut-off frequency for Bandpass and Bandstop filters. | |||
The matrix can have as many rows as necessary to filter the signal. Filters can be applied in any order and their transfer functions are multiplied before filtering occurs. | |||
===AudioEnvelopeSmoothing=== | |||
The cutoff frequency for the low pass filter which is applied to the filtered and full-wave rectified audio data. This should be set to the highest frequency you want to see in the resulting audio envelope. | |||
==State Variables== | ==State Variables== | ||
The AudioExtension outputs the following state variables: | |||
===Audio[In/Out]Envelope[0-3]=== | |||
These are the envelope values of each channel (up to channel 4) of the audio inputs and outputs (in the AudioMixer matrix). These 16 bit unsigned values correspond to the resulting envelope after the audio envelope extraction. For architectural reasons, it is not possible to publish states after system startup, so you are limited to four channels of input and output. The AudioExtension can be easily modified to change the number of channels by editing the <code>#define NUM_INPUT_ENVELOPES 4</code> and <code>#define NUM_OUTPUT_ENVELOPES</code> lines in AudioExtension.cpp, and recompiling your source module. | |||
=== | ===AudioFrame=== | ||
This 32 bit unsigned number corresponds to the current frame of audio data in the recorded output files. This can be used to resynchronize the lossless audio to the resulting .dat file offline. Audio is sampled internally at 44100 Hz, so this number will roll over once every 27 hours or so. | |||
==See also== | ==See also== | ||
Revision as of 15:53, 29 June 2012
Synopsis
An environment extension which manages multichannel, low latency audio I/O.
Location
http://www.bci2000.org/svn/trunk/src/contrib/Extensions/AudioExtension
Versioning
Authors
Griffin Milsap (griffin.milsap@gmail.com)
Version History
- 2012/06/11: Initial public release;
Source Code Revisions
- Initial development: 4095
- Tested under: 4095
- Known to compile under: 4095
- Broken since: --
Todo
- Fix Known Issues
- Add per-sample resolution to envelopes
Known Issues
- Leaving the module in halted state exhibits some sort of bug regarding state logging. When running state is resumed, envelope states may fail to update for 15-30 seconds. The bug seems to be unrelated to how long system was halted -- Not sure if this is an issue with the extension itself, or an issue with the bcievent interface. This bug will never happen on the first run after BCI2000 is started up. If you do see the behavior, either wait for it to go away or restart BCI2000 and perform a new recording.
- Bandpass filtering in filterbanks doesn't appear to function correctly.
- AudioExtension processes audio in a separate thread and the internal audio callback is called from yet-another context (Sometimes a system interrupt). In order to prevent deadlock, the Audio callback must not lock or wait for external threads. As such, it will simply copy the last good audio buffer to the output stream if the audio thread has not posted new data to use yet. This can result in slowed "timestretching" effects on audio input files if the audio thread cannot keep up with the audio callbacks. To prevent this behavior, ensure your audio block size is large enough (at least 1024 frames). If you are using a lower latency audio API (such as ASIO) you are probably okay to use audio block sizes around 512 frames. Either way, be mindful that audio playback may not necessarily operate in real-time and you will receive NO WARNING WHATSOEVER when it fails to.
Functional Description
Experiments which require audio input or real-time audio synthesis based on system state are now possible with the AudioExtension. This extension is capable of recording multiple channels of audio input, synthesizing tones or noise, and reading encoded audio files. These channels are input to a mixing matrix which mixes those inputs to multiple channels of audio output. Both input and output are run through a simple filterbank, then they have their envelope extracted and logged into states via the bcievent interface. Audio input and output channels can be recorded into audio files losslessly and can be resynchronized offline. The mixing matrix is a matrix of expressions which can be used to dynamically change audio mixing based on the system state.
Integration into BCI2000
Compile the extension into your source module by enabling contributed extensions in your CMake configuration. You can do this by going into your root build folder and deleting CMakeCache.txt and re-running the project batch file, or by running cmake -i and enabling BUILD_AUDIOEXTENSION. Once the extension is built into the source module, enable it by starting the source module with the --EnableAudioExtension=1 command line argument (NB, as explained below, the numeric value here matters, and denotes the audio API to be used: =1 means DirectSound).
Block Diagram
Parameters
The AudioExtension is configured in the Source tab within the AudioExtension section. The configurable parameters are:
EnableAudioExtension
Enables/Disables the AudioExtension. This parameter performs double-duty as an audio host API selector. The following values of this parameter are valid. NOTE: Not all audio APIs are available on all platforms.
- [0] - Disabled
- [1] - DirectSound
- [2] - MME
- [3] - ASIO
- [4] - SoundManager
- [5] - CoreAudio
- [6] - Disabled
- [7] - OSS
- [8] - ALSA
- [9] - AL
- [10] - BeOs
- [11] - WDMKS
- [12] - JACK
- [13] - WASAPI
- [14] - AudioScienceHPI
AudioMixer
This matrix of expressions mixes input (rows) to output(columns). It must be dimensioned with exactly n columns where n is the number of outputs. Row labels define the input source. Change row labels by double clicking on the row. The following inputs are valid row labels.
X- This is automatically interpreted as INPUT[X]INPUT[X]- This input will come from channel X on the sound card input.FILE[X]- This input will come from channel X in the AudioInputFile.TONE[X]- This input will be a synthesized sine wave with the frequency of X Hz.NOISE[X]- This input will be generated white noise at X Hz. NOTE: NOISE[] is white noise at the audio sampling rate (which defaults to 44100)
AudioInputDevice
The index for the device to use as the audio input device on the current Host API. See the operator log after "Set Config" for valid device indices on the selected host API. A value of -1 for this parameter selects the default input device on this host API.
AudioOutputDevice
The index for the device to use as the audio input device on the current Host API. See the operator log after "Set Config" for valid device indices on the selected host API. A value of -1 for this parameter selects the default output device on this host API.
AudioInputFile
Audio file to use as audio input to AudioMixer. The selected file can have any non-zero number of channels and be encoded in almost any format (except MP3), but MUST be encoded at 44100 Hz.
AudioRecordInput
Enables/Disables recording of audio data to a file in the DataDirectory.
AudioRecordOutput
Enables/Disables recording of audio data to a file in the DataDirectory.
AudioRecordingFormat
Changes the file format and encoding options of the recorded output files. This parameter has the following three options:
- Raw - Records to 16 bit Microsoft formatted WAV files with no compression. These files open directly in MATLAB if that's interesting to you.
- Lossless - Records to FLAC formatted files. These files are slightly smaller than RAW files, but have no quality loss.
- Lossy - Records to Ogg Vorbis files. These files are similar to MP3 but do not have the associated licensing issues. They are compressed using a lossy algorithm, so the resulting files are very small but sound slightly worse than lossless encoding. This format is good for long recordings where perfect quality is not necessary.
AudioInputFilterbank, AudioOutputFilterbank
A filterbank which filters audio input and output before rectification/smoothing for envelope extraction. These butterworth filters will not be applied to the audible signal. The format of the filter bank is as follows:
- Type - The characteristic of the filter. The following values are valid.
- Lowpass - Creates a low pass filter
- Highpass - Creates a high pass filter
- Bandpass - Creates a band pass filter *See Known Issues*
- Bandstop - Creates a band stop, or notch filter
- Order - The order of the filter model. Higher order filters are more accurate but more expensive computationally.
- Cutoff1 - The cutoff frequency for Lowpass and Highpass filters, and the cut-on frequency for Bandpass and Bandstop filters.
- Cutoff2 - The cut-off frequency for Bandpass and Bandstop filters.
The matrix can have as many rows as necessary to filter the signal. Filters can be applied in any order and their transfer functions are multiplied before filtering occurs.
AudioEnvelopeSmoothing
The cutoff frequency for the low pass filter which is applied to the filtered and full-wave rectified audio data. This should be set to the highest frequency you want to see in the resulting audio envelope.
State Variables
The AudioExtension outputs the following state variables:
Audio[In/Out]Envelope[0-3]
These are the envelope values of each channel (up to channel 4) of the audio inputs and outputs (in the AudioMixer matrix). These 16 bit unsigned values correspond to the resulting envelope after the audio envelope extraction. For architectural reasons, it is not possible to publish states after system startup, so you are limited to four channels of input and output. The AudioExtension can be easily modified to change the number of channels by editing the #define NUM_INPUT_ENVELOPES 4 and #define NUM_OUTPUT_ENVELOPES lines in AudioExtension.cpp, and recompiling your source module.
AudioFrame
This 32 bit unsigned number corresponds to the current frame of audio data in the recorded output files. This can be used to resynchronize the lossless audio to the resulting .dat file offline. Audio is sampled internally at 44100 Hz, so this number will roll over once every 27 hours or so.