web-media-effects

Web Media Effects (WFX) is a suite of media effects developed for web SDKs and WebRTC media applications.

Introduction

There are three effects included in this library:

- Virtual background (e.g., blur, image replacement, video replacement)
- Noise reduction (e.g., background noise removal)
- Gain (for testing)

Common Methods

The effects are built on top of a plugin interface that makes building and extending effects more straight-forward.

Each effect has four primary methods to control the plugin:

- load(input) accepts a track or stream and returns a new track or stream with the effect applied
- enable() enables the plugin after it's loaded
- disable() disables the plugin after it's loaded
- dispose() tears down the effect
- preloadAssets() fetches all assets (e.g., WASM files, ONNX models) to optimize the load sequence

Upon enabling or disabling the effect, an event is fired.

``ts
effect.on('track-updated', (track: MediaStreamTrack) => {
// do something with the new track.
});
`

Additionally, there are a few convenience methods:

- getOutputStream() returns the new outgoing (i.e., "effected") stream
- getOutputTrack() returns the active output track
- setEnabled(boolean) sets the effect state by passing in a boolean (convenient for state managers)

Preloading Assets

In an effort to optimize startup time for applying media effects, there is a preloading mechanism. This mechanism fetches critical assets, such as ONNX models for image segmentation, WASM modules for audio processing, and web workers for background processing, in advance of media availability. This ensures smoother integration of effects once the media stream is ready to improve the overall user experience. Assets can be preloaded using either a provided factory function or directly using preloadAssets() API.

$3

The library includes factory functions for scenarios that require asynchronous operations. Utilizing the async/await pattern, these functions provide a simple method for creating effects with their assets already preloaded. The factory function's second parameter is a boolean that indicates whether the assets should be preloaded.

`typescript
const noiseReductionEffect = await createNoiseReductionEffect({
authToken: 'your-auth-token',
// ...other options
},
true
);

const virtualBackgroundEffect = await createVirtualBackgroundEffect({
authToken: 'your-auth-token',
mode: 'BLUR',
// ...other options
},
true
);
`

By incorporating asset preloading, the preload API aims to minimize delays and performance hitches when activating effects to keep the UI fluid and responsive.

$3

For more fine-grained control over the preloading process, you can also directly call the preloadAssets() method on each effect instance. This approach allows you to manually manage when and how assets are preloaded, providing flexibility to fit various application architectures and workflows:

`typescript
const virtualBackgroundEffect = new VirtualBackgroundEffect(options);
await virtualBackgroundEffect.preloadAssets();

const noiseReductionEffect = new NoiseReductionEffect(options);
await noiseReductionEffect.preloadAssets();
`

This direct method is useful in scenarios where you might want to preload assets independently of the effect instantiation or in response to specific application states or events. It gives you the ability to strategically preload assets at the most appropriate moment.

Virtual Background Effect

The virtual background effect is a wrapper around ladon-ts that provides a virtual background for video calling. The virtual background may be an image, an mp4 video, or the user's background with blur applied. The blur option allows for varied levels of strength and quality where higher levels require more compute resources.

The virtual-background-effect takes an optional VirtualBackgroundEffectOptions config object in its constructor. The effect's options can be changed at runtime via an updateOptions() method. When disabled, the effect simply passes through the original video images so that the outgoing stream does not need to be changed.

The effect uses a background thread worker to prevent slowdowns on the main UI thread.

> NOTE: For backwards compatibility, the default mode is set to BLUR.

Options

There are a few different options that can be supplied to the constructor or updateOptions() method that affect the behavior

| Name | Description | Values | Required |
| --------------------------- | ----------------------------------------------------------------------------------------------------- | ------------------------------------------------- | ---------------------------------- |
| authToken | Used to authenticate the request for the backend models | An encoded string token | Yes |
| frameRate | Determines how many frames per second are sent to the model | 0-60 | Defaults to 30 |
| quality | Determines the accuracy of the model (higher requires more CPU) | LOW MEDIUM HIGH ULTRA | Defaults to LOW |
| mirror | Whether the output image should be flipped horizontally | true false | Defaults to false |
| mode | Determines what kind of background to render behind the user | BLUR IMAGE VIDEO | Defaults to BLUR |
| blurStrength | How strongly the background should be blurred | WEAK MODERATE STRONG STRONGER STRONGEST | Required in BLUR mode |
| bgImageUrl | Path to the background image to replace the original background | Fully qualified URL | Required in IMAGE mode |
| bgVideoUrl | Path to the background video to replace the original background | Fully qualified URL (mp4 only) | Required in VIDEO mode |
| env | Which environment the effect is running in. | EffectEnv.Production EffectEnv.Integration | Defaults to EffectEnv.Production |
| avoidSimd | Avoid using the SIMD processor, if SIMD is supported (for testing) | true, false | Defaults to false |
| preventBackgroundThrottling | If set to true, prevents the browser from throttling the effect frame rate when the page is hidden. | true, false | Defaults to false |

$3

The virtual background plugin applies a background effect to the original media stream by performing image segmentation on the incoming video frames. The plugin is capable of applying four different kinds of effects called modes: background blur, background image replacement, background video replacement, and passthrough.

The mode configuration option determines what background effect to apply. There are four accepted values for the mode: BLUR, IMAGE, VIDEO, and PASSTHROUGH. Each mode has at least one required option that needs to be set in the options object, which is outlined below in the Options section.

- BLUR: Applies a blur effect to the background, with configurable blur strength levels.
- IMAGE: Replaces the background with a static image specified by the bgImageUrl option.
- VIDEO: Replaces the background with a video specified by the bgVideoUrl option.
- PASSTHROUGH: Runs the Ladon AI model without rendering a virtual background, allowing plugins to process the inference results (e.g., motion detection, faces and landmarks).

> NOTE: For Typescript users, the mode can be selected by using the exported VirtualBackgroundMode enum, for convenience.

Usage

Supply a video stream to the effect and when loaded, it will return a new stream with the effect applied.

`ts
// Create a new video stream by a getting user's video media.
const originalVideoStream = await navigator.mediaDevices.getUserMedia({ video: { width, height } });

// Create the effect.
const effect = new VirtualBackgroundEffect({
authToken: 'YOUR_AUTH_TOKEN',
mode: BLUR,
blurStrength: STRONG,
quality: LOW,
});

// Load the effect with the input stream.
const newStream = await effect.load(originalVideoStream);

// Attach the new stream to a video element to see the effect in action.
myVideoElement.srcObject = myStream;
`

Plugins

The virtual background effect supports the use of plugins to extend and customize its functionality. Plugins can be registered, initialized, and disposed of through the plugin manager. The two primary base classes to extend when creating plugins are BaseBeforeInferencePlugin and BaseAfterInferencePlugin.

$3

Model inference refers to the process where the virtual background effect performs calculations (like segmentation or motion analysis) on each video frame to apply the selected effect (e.g., blur, image replacement, or video replacement). Plugins can hook into this process at two key points:

- Before Inference: Plugins that need to analyze or modify the video frame before the virtual background effect processes it can use the Before Inference stage. This is useful for plugins that might want to control whether the model should perform inference at all, based on conditions like motion in the frame. For more details, see Adaptive Frame Skipper.

- After Inference: Plugins that need to work with the results of the model inference can use the After Inference stage. These plugins get access to the results of the frame processing (e.g., segmented image or detected motion) and can use that information to make decisions or apply further effects. For more details, see Be Right Back and Rate Estimator.

$3

Plugins should extend one of the following base classes depending on whether they operate before or after inference:

`typescript
abstract class BaseBeforeInferencePlugin {
initialize(effect: VirtualBackgroundEffect): void;
dispose(): void;
onBeforeInference(timestamp: number, lastResult: InferenceResult): Promise;
updateOptions(newOptions: Partial): void;
}

abstract class BaseAfterInferencePlugin {
initialize(effect: VirtualBackgroundEffect): void;
dispose(): void;
onAfterInference(timestamp: number, result?: InferenceResult): Promise;
updateOptions(newOptions: Partial): void;
}
`

These base classes automatically handle registering and unregistering the appropriate callbacks (addBeforeInferenceCallback, addAfterInferenceCallback, removeBeforeInferenceCallback, removeAfterInferenceCallback) when the plugin is initialized or disposed. Plugin developers only need to implement the onBeforeInference or onAfterInference methods based on their plugin’s needs.

$3

The VirtualBackgroundEffect also supports the following plugin methods:

- registerPlugin(name: string, plugin: PluginType): Register a plugin with the virtual background effect.
- initializePlugins(): Initialize all registered plugins.
- getPlugin\(name: string): Retrieve a plugin by name.

> Note: addBeforeInferenceCallback, addAfterInferenceCallback, removeBeforeInferenceCallback, and removeAfterInferenceCallback are handled automatically by the base plugin classes (BaseBeforeInferencePlugin and BaseAfterInferencePlugin), so plugin developers typically don’t need to call them directly.
>


$3

Plugins should be registered after creating the effect instance using the registerPlugin method. Here's an example of how to create and register plugins like the BeRightBackPlugin, FrameSkipperPlugin, and RateEstimatorPlugin. The example also demonstrates how to overwrite default plugin options using coreOptions.

For more details on these advanced features, see the documentation:

- Adaptive Frame Skipping
- Be Right Back Detection
- Rate Estimation
- Debug Overlay

`typescript
// Create the BeRightBack plugin
const beRightBackPlugin = new BeRightBackPlugin({
"mode": "conservative",
"debug": true,
"coreOptions": {
"motionIouThreshold": 0.9,
"onHysteresisMaxMs": 3000,
"offHysteresisMaxMs": 2000
}
});

// Create the FrameSkipper plugin
const frameSkipperPlugin = new FrameSkipperPlugin({
"mode": "aggressive",
"debug": true,
"coreOptions": {
"baseMinSkipTime": 50,
"baseMaxSkipTime": 1000,
"historySize": 100,
"skipTimeIncrement": 50,
"forcedInferenceInterval": 2000,
"highMotionThreshold": 0.9,
"smoothingFactor": 0.5
}
});

// Create the RateEstimator plugin
const rateEstimatorPlugin = new RateEstimationPlugin({
"targetRate": 30,
"debug": true
});

// Register the plugins with the effect instance
effect.registerPlugin("beRightBack", beRightBackPlugin);
effect.registerPlugin("frameSkipper", frameSkipperPlugin);
effect.registerPlugin("rateEstimator", rateEstimatorPlugin);

// Initialize the plugins
effect.initializePlugins();

// Enable the effect
await effect.enable(); // or await effect.setEnabled(true);
`

$3

To retrieve a registered plugin, use the getPlugin method and provide the plugin's name.

`typescript
const brbPlugin = effect.getPlugin("beRightBack");
`

By using coreOptions, you can customize the behavior of plugins according to your application's requirements.

$3

The Rate Estimator plugin monitors the processing rate (such as frame rates) of media effects and emits events when the rate changes. This is helpful to dynamically adjust the system's behavior based on performance conditions.

#### Configuration Options

| Option | Description | Default Value |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------ | ------------- |
| targetRate | The desired target rate (frames per second) the estimator should maintain. | Required |
| debug | Whether to show debug information. | false |
| coreOptions | Optional overrides for specific rate estimator behavior, such as hysteresis margin, low rate threshold, and more (see core options). | - |

#### Core Options

| Option | Description | Default Value |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------ |
| hysteresisMargin | Margin of tolerance around the low threshold to prevent rapid toggling between states, expressed as a percentage of the lowThreshold. | 0.05 (5%) |
| lowDuration | Duration in seconds that the rate must be below the lowThreshold before considering the rate sustainedly low. | 5 seconds |
| lowThreshold | Threshold below which the rate is considered low, expressed as a percentage of the target rate. | 80% of target rate |
| minSamples | Minimum number of samples to accumulate before making a rate estimation. | 30 |
| maxSamples | Maximum number of samples to consider for rate estimation to prevent using stale data. | 120 |

#### Methods

- on(event: RateEstimatorEvent, callback: (rate: number) => void): Attach a handler to rate change events.
- getRate(): Returns the current rate estimation.
- getStatus(): Returns the current status of the rate estimator.


#### Events

The Rate Estimator emits events to indicate changes in the processing rate. You can use string values or, if using TypeScript, enums provided by RateEstimatorEvent.

| Event | Description |
| -------------------------------------------------- | ------------------------------------------------------------------------------------ |
| rate-ok or RateEstimatorEvent.RateOk | Fired when the estimated rate returns to normal, above the lowThreshold. |
| rate-low or RateEstimatorEvent.RateLow | Fired when the estimated rate falls below the lowThreshold. |
| rate-lagging or RateEstimatorEvent.RateLagging | Fired when the low rate is sustained beyond the duration specified by lowDuration. |

#### Usage

`typescript
import { RateEstimatorPlugin, RateEstimatorEvent } from '@webex/web-media-effects';

const rateEstimatorPlugin = new RateEstimatorPlugin({
targetRate: 30, // Target fps
debug: true,
coreOptions: {
hysteresisMargin: 0.05,
lowThreshold: 24, // Consider rate low if below 24 fps
lowDuration: 5,
minSamples: 30,
maxSamples: 120,
}
});

rateEstimatorPlugin.on(RateEstimatorEvent.RateLow, (rate) => {
console.log(Rate is low: ${rate});
});
rateEstimatorPlugin.on(RateEstimatorEvent.RateOk, (rate) => {
console.log(Rate is ok: ${rate});
});
rateEstimatorPlugin.on(RateEstimatorEvent.RateLagging, (rate) => {
console.log(Rate is lagging: ${rate});
});
`

$3

The Frame Skipper plugin is designed to optimize the performance of media effects by selectively skipping frames based on motion detection and other criteria. This helps reduce the computational load while maintaining acceptable quality.

#### Configuration Options

| Option | Description | Default Value |
| ------------- | -------------------------------------------------------------------------------------------------------- | -------------- |
| mode | The frame skipping mode to use. | conservative |
| debug | Whether to enable debug logging for the frame skipper. | false |
| coreOptions | Optional overrides for specific frame skipping behavior, such as base skip time, motion thresholds, etc. | - |

#### Core Options

| Option | Description | Default Value |
| ------------------------- | -------------------------------------------------------------------------- | ------------- |
| baseMinSkipTime | The minimum time to wait before performing inference, in milliseconds. | 50 ms |
| baseMaxSkipTime | The maximum time to wait before performing inference, in milliseconds. | 1000 ms |
| historySize | The number of recent motion data points to consider for calculating stats. | 100 |
| skipTimeIncrement | The amount by which skip time is adjusted based on motion variance. | 50 ms |
| forcedInferenceInterval | The maximum time before inference is forced, in milliseconds. | 2000 ms |
| highMotionThreshold | The motion threshold for determining high motion. | 0.9 |
| smoothingFactor | The smoothing factor for motion value calculation. | 0.5 |

#### Methods

- updateOptions(newOptions: Partial): Update the options for the frame skipper.
- getLatestDebugInfo(): Returns the latest debug information about frame skipping.

#### Usage

`typescript
import { FrameSkipperPlugin } from '@webex/web-media-effects';

const frameSkipperPlugin = new FrameSkipperPlugin({
mode: 'aggressive', // default
debug: true,
coreOptions: {
baseMinSkipTime: 50,
baseMaxSkipTime: 1000,
historySize: 100,
skipTimeIncrement: 50,
forcedInferenceInterval: 2000,
highMotionThreshold: 0.9,
smoothingFactor: 0.5
}
});
`

$3

The Be Right Back (BRB) plugin is designed to detect when the user is away from the camera and apply appropriate actions, such as displaying a placeholder image or message.

#### Configuration Options

| Option | Description | Default Value |
| ------------- | --------------------------------------------------------------------------------------------- | -------------- |
| mode | The mode for controlling the be right back behavior. | conservative |
| debug | Whether to enable debug logging for the be right back plugin. | false |
| coreOptions | Optional overrides for specific BRB behavior, such as motion thresholds and hysteresis times. | - |

#### Core Options

| Option | Description | Default Value |
| -------------------- | ------------------------------------------------------------------- | ------------- |
| motionIouThreshold | The amount of motion required to trigger the be right back state. | 0.9 |
| onHysteresisMaxMs | The amount of time required to trigger the be right back state on. | 3000 ms |
| offHysteresisMaxMs | The amount of time required to trigger the be right back state off. | 2000 ms |

#### Methods

- on(event: BeRightBackEvent, callback: () => void): Attach handlers for BRB events like on, off, and state-change.
- updateOptions(newOptions: Partial): Update the options for BRB.

#### Events

The Be Right Back plugin emits events to indicate changes in the user's presence in front of the camera. You can use string values or, if using TypeScript, enums provided by BeRightBackEvent.

| Event | Description |
| ------------------------------------------------ | ---------------------------------------------------------------------------- |
| on or BeRightBackEvent.On | Fired when the user is detected to have left the camera. |
| off or BeRightBackEvent.Off | Fired when the user returns to the camera. |
| state-change or BeRightBackEvent.StateChange | Fired when the user's state changes, whether they leave or return to camera. |

#### Usage

`typescript
import { BeRightBackPlugin, BeRightBackEvent } from '@webex/web-media-effects';

const beRightBackPlugin = new BeRightBackPlugin({
mode: 'conservative',
debug: true,
coreOptions: {
motionIouThreshold: 0.9,
onHysteresisMaxMs: 3000,
offHysteresisMaxMs: 2000
}
});

beRightBackPlugin.on(BeRightBackEvent.On, () => {
console.log('User has left the camera');
});
beRightBackPlugin.on(BeRightBackEvent.Off, () => {
console.log('User has returned to the camera');
});
beRightBackPlugin.on(BeRightBackEvent.StateChange, (newState, oldState) => {
console.log('User has changed state: ', newState, oldState);
});
`

$3

The Debug Overlay Plugin provides real-time performance monitoring and debugging information for virtual background effects through a draggable, resizable interface. It wraps the core debug overlay functionality and integrates seamlessly with the plugin manager system.

#### Configuration Options

| Option | Description | Default Value |
| ------------- | -------------------------------------------------------------------------------------------------------- | ------------- |
| debug | Whether to enable debug logging for the plugin. | false |
| coreOptions | Optional overrides for the core debug overlay behavior, such as visibility, update interval, and limits. | - |

#### Core Options

| Option | Description | Default Value |
| ---------------- | ------------------------------------------------------------------------- | ------------- |
| showOnInit | Whether to display the overlay immediately when initialized. | false |
| updateInterval | How frequently the overlay polls for updated statistics, in milliseconds. | 250 |
| maxErrors | Maximum number of recent error messages to display. | 10 |
| debug | Whether to enable debug logging for the core overlay. | false |

#### Methods

- show(): Show the debug overlay.
- hide(): Hide the debug overlay.
- toggle(): Toggle overlay visibility.
- getDebugOverlay(): Get the underlying debug overlay instance for direct control.

#### Usage

`typescript
import { DebugOverlayPlugin } from '@webex/web-media-effects';

// Create the debug overlay plugin
const debugOverlayPlugin = new DebugOverlayPlugin({
debug: true,
coreOptions: {
showOnInit: false,
updateInterval: 250,
maxErrors: 15
}
});

// Register with the effect
effect.registerPlugin('debugOverlay', debugOverlayPlugin);

// Initialize plugins
effect.initializePlugins();

// Control programmatically
const overlay = effect.getPlugin('debugOverlay');
overlay.show();
overlay.hide();
overlay.toggle();

// Or use keyboard shortcut: Ctrl+Shift+F12
`

For complete details on all features and sections, see the Debug Overlay documentation.

Noise Reduction Effect

The noise reduction effect removes background noise from an audio stream to provide clear audio for calling.

The noise-reduction-effect takes a NoiseReductionEffectOptions config object in its constructor. A developer can optionally pass a workletProcessorUrl parameter (or legacyProcessorUrl) in the config to use a different of test version of the audio processor. An audioContext parameter can be passed into the config as well in order to supply an existing AudioContext; otherwise, a new one will be created.

The effect loads the background thread AudioWorkletProcessor into the main thread AudioWorklet in order to keep the audio computations from impacting UI performance.

Options

There are a few different options that can be supplied to the constructor or updateOptions() method that affect the behavior

| Name | Description | Values | Required |
| ------------------- | ----------------------------------------------------------------------------------------- | ---------------------------------------------- | --------------------- |
| authToken | Used to authenticate the request for the backend processors | An encoded string token | Yes |
| audioContext | An optional AudioContext for custom behavior | AudioContext | No |
| mode | Determines whether to run in WORKLET mode or LEGACY mode for older browsers | WORKLET LEGACY | Defaults to WORKLET |
| legacyProcessorUrl | A url to fetch the legacy processor that attaches to the deprecated ScriptProcessorNode | A fully qualified URL | No |
| workletProcessorUrl | A url to fetch the AudioWorkletProcessor to attach to the AudioWorkletNode | A fully qualified URL | No |
| env | Which environment the effect is running in. | EffectEnv.Production EffectEnv.Integration | No |

Supported Bitrates

The noise reduction effect supports the following audio bitrates:
- 16 kHz
- 32 kHz
- 44.1 kHz
- 48 kHz

If an unsupported bitrate is detected, the noise reduction effect will throw the following error: Error: noise reduction: worklet processor error, "Error: Sample rate of X is not supported.

Usage

Supply an audio track or stream to the effect, the effect will handle updating the stream on enable/disable. In the case of a track being passed, listen to the 'track-updated' event to receive the updated track on enable/disable.

`ts
// Create a new audio stream by getting a user's audio media.
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });

// Create the effect.
const effect = new NoiseReductionEffect({
authToken: 'YOUR_AUTH_TOKEN',
workletProcessorUrl: 'https://my-worklet-processor-url', // For 'WORKLET' mode
legacyProcessorUrl: 'https://my-legacy-processor-url', // For 'LEGACY' mode
mode: 'WORKLET', // or 'LEGACY'
});

// Load the effect with the input stream.
await effect.load(stream);
`

Example

The example app included in this repo is designed to help test functionality and troubleshoot issues. You can run the example app by following the instructions in the README in the example folder. You can also view a live example at https://effects.webex.com.

Development

1. Run yarn to install dependencies.
2. Run yarn prepare to prepare dependencies.
3. Run yarn watch to build and watch for updates.
4. Run yarn test` to build, run tests, lint, and run test coverage.

$3

Install the recommended extensions when first opening the workspace (there should be a prompt). These plugins will help maintain high code quality and consistency across the project.

> NOTE: VS Code is setup to apply formatting and linting rules on save (Prettier runs first, then ESLint). The rules applied are defined in settings.json.

Publishing for Downstream Dependencies

This project gets copied to a Docker container downstream in Quay. There is a GitHub Action as part of this repository that copies assets to the repository.