Spatial Audio Encoder in Web Audio API
npm install songbird-audio
Songbird is a real-time spatial audio encoding JavaScript library for WebAudio
applications. It allows web developers to dynamically spatially-encode
streaming audio content into scalable
ambisonics signal, which is rendered
internally using
Omnitone to output stereo sound,
for realistic and quality-scalable 3D audio.
Hear Songbird in action (currently the examples only work on laptops/desktops):
- vs. PannerNode
- Room Models
- Flock of Birds
- Treasure Hunt
The implementation of Songbird is based on the
Google spatial media specification.
It expects mono input to its Source instances and outputs
ambisonic (multichannel) ACN channel layout with SN3D normalization. Detailed
documentation may be found
here.
Table of Contents
- How it works
- Installation
- Usage
- "Hello World" Example
- Positioning Sources and the Listener
- Room Properties
- Creation Arguments
- Differences to PannerNode
- Cost
- Quality
- Room Acoustics
- Porting PannerNode projects to Songbird
- Building
- Testing
- Testing Songbird Locally
- Related Resources
- Acknowledgments
- Support
- License
Songbird is a JavaScript API that supports real-time spatial audio encoding for
the Web using Higher-Order Ambisonics (HOA). This is accomplished by attached
audio input to a Source which has associated spatial object parameters.Source objects are attached to a Songbird instance, which models the
listener as well as the room environment the listener and sources are in.
Binaurally-rendered ambisonic output is generated using
Omnitone, and raw ambisonic output
is exposed as well.
alt="Songbird Diagram">
Songbird is designed to be used for web front-end projects. So
NPM is recommended if you want to install the library
to your web project. You can also clone this repository and use the library
file as usual.
``bash`
npm install songbird-audio
The first step is to include the library file in an HTML document.
`html
`
Spatial encoding is done by creating a Songbird scene using an associatedAudioContext and then creating any number of associated Source objectsSongbird.createSource()
using . Any number of AudioNodes can be connectedSource
directly to objects. The Songbird scene models a physicalSource
listener while adding room reflections and reverberation. The
instances model acoustic sound sources. The library is designed to be easily
integrated into an existing WebAudio audio graph.
Let's see how we can create a scene and generate some audio. Let's begin by
constructing an AudioContext and Songbird scene and connecting it to the
audio output. You can view a live demo of this example
here.
`js
var audioContext = new AudioContext();
// Create a (1st-order Ambisonic) Songbird scene.
var songbird = new Songbird(audioContext);
// Send songbird's binaural output to stereo out.
songbird.output.connect(audioContext.destination);
`
Next, let's add a room. By default, the room size is 0m x 0m x 0m (i.e. there
is no room and we are in free space). To define a room, we simply need to
provide the dimensions in meters (the room's center is the origin). We can also
define the materials of each of the 6 surfaces (4 walls + ceiling + floor). A
range of materials are predefined in Songbird, each with different reflection
properties. To create hidden/missing walls, select the 'transparent' material.
`js`
// Set room acoustics properties.
var dimensions = {
width : 3.1,
height : 2.5,
depth : 3.4
};
var materials = {
left : 'brick-bare',
right : 'curtain-heavy',
front : 'marble',
back : 'glass-thin',
down : 'grass',
up : 'transparent'
};
songbird.setRoomProperties(dimensions, materials);
Next, we create an audio element, load some audio and feed the audio element
into the audio graph as an AudioNode. We then create a Source and connect theSource
AudioNode to it. The default position for a is the origin.
`js
// Create an audio element. Feed into audio graph.
var audioElement = document.createElement('audio');
audioElement.src = 'resources/SpeechSample.wav';
// Create an AudioNode from the audio element.
var audioElementSource = audioContext.createMediaElementSource(audioElement);
// Create a Source, connect desired audio input to it.
var source = songbird.createSource();
audioElementSource.connect(source.input);
`
Finally, we can position the source relative to the listener and then playback
the audio with the familiar .play() method. This will binaurally render the
scene we have just created.
`js
// The source position is relative to the origin (center of the room).
source.setPosition(-0.707, -0.707, 0);
// Playback the audio.
audioElement.play();
`
Source objects can be placed with cartesian coordinates relative to the origin
(center of the room). Songbird uses a right-handed coordinate system, similar
to OpenGL and Three.js.
`js`
// Or Source's and Listener's positions.
source.setPosition(x, y, z);
songbird.setListenerPosition(x, y, z);
The Source and Listener orientations can be set using forward and up vectors:
`js`
// Set Source and Listener orientation.
source.setOrientation(forward_x, forward_y, forward_z, up_x, up_y, up_z);
songbird.setListenerOrientation(forward_x, forward_y, forward_z, up_x, up_y, up_z);
Alternatively, the Source's and Listener position and orientation can be set
using Three.js Matrix4 objects:
`js`
source.setFromMatrix(matrix4);
songbird.setListenerFromMatrix(matrix4);
Room properties can be set to control the characteristics of spatial
reflections and reverberation. We currently support the following named
materials:
- transparent
- acoustic-ceiling-tiles
- brick-bare
- brick-painted
- concrete-block-coarse
- concrete-block-painted
- curtain-heavy
- fiber-glass-insulation
- glass-thin
- glass-thick
- grass
- linoleum-on-concrete
- marble
- metal
- parquet-on-concrete
- plaster-smooth
- plywood-panel
- polished-concrete-or-tile
- sheetrock
- water-or-ice-surface
- wood-ceiling
- wood-panel
- uniform
When constructing a Songbird scene, optional scene-related arguments may be
provided to override default values.
`js`
var songbirdOptions = {
ambisonicOrder: 1,
listenerPosition: [1, 0, 0],
listenerForward: [1, 0, 0],
listenerUp: [0, 1, 0],
dimensions: { width: 3, height: 4, depth: 5 },
materials: { left: 'uniform', right: 'uniform', down: 'uniform',
up: 'uniform', front: 'uniform', back: 'uniform' },
speedOfSound: 340
};
var songbird = new Songbird(audioContext, songbirdOptions)
Likewise, when creating a new Source, source-related optional arguments may
be provided:
`js`
var sourceOptions = {
position: [0, 10, 10],
forward: [0, 0, -1],
up: [0, 1, 0],
minDistance: 0.1,
maxDistance: 1000,
rolloff: 'logarithmic',
gain: 0.1,
alpha: 0,
sharpness: 1,
sourceWidth: 0
}
var source = songbird.createSource(sourceOptions);
See the
documentation
for more details on all optional arguments.
There are several advantages to using Songbird over WebAudio's
PannerNode.
- Cost
- Quality
- Room Acoustics
PannerNode requires two convolutions per encoded source. But because we employ
ambisonics, there is a fixed cost associated with rendering from Songbird, with
nominal costs per source. Developers can adjust the desired ambisonic order
(from 1 to 3) to control the majority of computational costs.
In addition to controlling computational costs, adjusting the ambisonic order
controls the quality of spatialization (Higher order typically yields better
direct source localization). Additionally, we offer direct ambisonic output
(which bypasses the rendering), allowing developers total control over how to
render their content.
Songbird comes with room modelling effects, which includes both an early and
late reflection model based on the room properties. These effects are likewise
ambisonically-encoded and fully spatialized.
For projects already employing PannerNode, it is fairly simple to switch to
Songbird. Below is a basic PannerNode example:
`js
// Create a "PannerNode."
var panner = audioContext.createPanner();
// Initialize properties
panner.panningModel = 'HRTF';
panner.distanceModel = 'inverse';
panner.refDistance = 0.1;
panner.maxDistance = 1000;
// Connect input to "PannerNode".
audioElementSource.connect(panner);
// Connect "PannerNode" to audio output.
panner.connect(audioContext.destination);
// Set "PannerNode" and Listener positions.
panner.setPosition(x, y, z);
audioContext.listener.setPosition(x, y, z);
`
And below is the same example converted to Songbird:
`js
// Create a Songbird "Source" with properties.
var source = songbird.createSource({
rolloff: 'logarithmic',
minDistance: 0.1,
maxDistance: 1000
});
// Connect input to "Source."
audioElementSource.connect(source.input);
// Connect Songbird’s output to audio output.
songbird.output.connect(audioContext.destination);
// Set "Source" and Listener positions.
source.setPosition(x, y, z);
songbird.setListenerPosition(x, y, z);
`
Songbird uses WebPack to build the minified
library and to manage dependencies.
`bash`
npm install # install dependencies.
npm run build # build a non-minified library.
npm run watch # recompile whenever any source file changes.
npm run build-all # build a minified library.
npm run build-doc # generate documentation.
npm run eslint # lint code for ES6 compatibility.
Songbird uses Travis and
Karma test runner for
continuous integration. To run the test suite locally, clone the repository,
install dependencies and launch the test runner:
`bash`
npm test
Note that unit tests require the promisified version of OfflineAudioContext`,
so they might not run on non-spec-compliant browsers. Songbird's Travis CI is
using the latest stable version of Chrome.
For the local testing with Karma test runner, Chrome/Chromium-based browser is
required. Windows and Linux platforms are supported.
* Omnitone
* Google Spatial Media
* Web Audio API
* WebVR
Special thanks to Alper Gungormusler, Hongchan Choi, Marcin Gorzel, and Julius
Kammerl for their help on this project.
If you have found an error in this library, please file an issue at:
https://github.com/Google/songbird/issues.
Patches are encouraged, and may be submitted by forking this project and
submitting a pull request through GitHub. See CONTRIBUTING for more detail.
Copyright © 2017 Google Inc. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use
this file except in compliance with the License. You may obtain a copy of the
License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed
under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
CONDITIONS OF ANY KIND, either express or implied. See the License for the
specific language governing permissions and limitations under the License.