[TOC]

Agora-RTE-Extension

Introduction

Agora RTE Extension provides the ability for extension developer to interact with Agora RTC SDK NG's VideoTrack and
AudioTrack object, making video and audio processing possible.

By receiving MediaStreamTrack or AudioNode as input, running custom processing procedure such as WASM module
or AudioWorkletNode, and finally generating processed MediaStreamTrack or AudioNode, it will construct a media
processing pipeline to allow custom media processing provided by developers.

How Extension and Processor Interacts With Agora RTC SDK NG

A Processor basically connects to other Processors with pipe method:

``typescript
processorA.pipe(processorB);
`

The pipe method returns the Processor passed as parameter itself, making a function chaining style:

`typescript
//processor actually is processorB
const processor = processorA.pipe(processorB);

//function chaining
processorA.pipe(processorB).pipe(processorC);
`

On AgoraRTC SDK NG v4.10.0 and afterwards, the ILocalVideoTrack and ILocalAudioTrack objects also have pipe method
on it:

`typescript
const localVideoTrack = AgoraRTC.createCameraVideoTrack();

localVideoTrack.pipe(videoProcessor);
`

To make the processed media rendering locally and transmitting through WebRTC, processorDestination property
on ILocalVideoTrack and ILocalAudioTrack has to be the final processor through the pipeline:

`typescript
localVideoTrack.pipe(videoProcessor).pipe(localVideoTrack.processorDestination);
`

----
An Extension receives injected utility functionality such as logger and reporter
during AgoraRTC.registerExtensions function call:

`typescript
AgoraRTC.registerExtensions([videoExtension, audioExtension]);
`

Extension also provides createProcessor method for constructing Processor instance:

`typescript
const videoProcessor = videoExtension.createProcessor();
`

----
Wrap it up:

`typescript
const videoExtension = new VideoExtension();
AgoraRTC.registerExtensions([videoExtension]);

const localVideoTrack = await AgoraRTC.createCameraVideoTrack();
const videoProcessor = videoExtension.createProcessor();

localVideoTrack.pipe(videoProcessor).pipe(localVideoTrack.processorDestination);
`

Extension and Processor APIs for extension developers

$3

#### Extension._createProcessor

Abstract class Extension has one abstract method _createProcessor needs to be implemented:

`typescript
abstract class Extension {
abstract _createProcessor(): T;
}
`

When implemented, it should return a VideoProcessor or AudioProcessor instance.

AgoraRTC developer calling extension.createProcessor() will return the processor returned by _createProcessor.

#### Extension.setLogLevel

Abstract class Extension has one static method setLogLevel :

`typescript
abstract class Extension {
public static setLogLevel(level: number): void
}
`

AgoraRTC developer calling Extension.setLogLevel(level) will set the output log level of the extension.

#### Extension.checkCompatibility

Abstract class Extension has one optional abstract public method checkCompatibility could be implemented:

`typescript
abstract class Extension {
public abstract checkCompatibility?(): boolean;
}
`

When implemented, it should return a boolean value indicating whether extension could be run inside current browser environment.

$3

#### VideoProcessor.name

Abstract property name on VideoProcessor has to be implemented in order to name processor:

`typescript
abstract name: string;
`

#### VideoProcessor.onPiped

Abstract optional method onPiped could be implemented in order to be notified when processor connected to a pipeline
with ILocalVideoTrack as it's source:

`typescript
abstract onPiped?(context: IProcessorContext): void;
`

It will only be called when an ILocalVideoTrack object from AgoraRTC was connected to the pipeline, or when the
processor was connected to a pipeline with ILocalVideoTrack as its source.
> Pipeline without an ILocalVideoTrack as it's source, onPiped method will not be called for processors belonging to this pipeline until an ILocalVideoTrack connected to it.

`typescript
videoTrack.pipe(processor);//will be called

processorA.pipe(processorB);//will NOT be called
videoTrack.pipe(processorA);//will be called for both processorA and processorB
`

#### VideoProcessor.onUnpiped

Abstract optional method onUnpiped could be implemented in order to be notified when processor disconnected to a
pipeline:

`typescript
abstract onUnPiped?(): void;
`

#### VideoProcessor.onTrack

Abstract optional method onTrack could be implemented in order to be notified when the previous processor
or ILocalVideoTrack feeds output MediaStreamTrack to the current processor:

`typescript
abstract onTrack?(track: MediaStreamTrack, context: IProcessorContext): void;
`

#### VideoProcessor.onEnableChange
Abstract optional method onEnableChange could be implemented in order to be notified when processor's _enabled
property has changed:
`typescript
abstract onEnableChange?(enabled: boolean): void | Promise;
`
AgoraRTC developer calling processor.enable() and processor.disable() may change _enabled property and consequently calling onEnableChange, but enabling an already enabled processor or disabling an already disabled processor will not.
#### VideoProcessor._enabled
property _enabled describes enabled status of the current processor.

`typescript
protected _enabled :boolean = true;
`

It defaults to true , but could be change inside processor constructor:

`typescript
class CustomProcessor extends VideoProcessor {
public constructor(){
this._enabled = false;
}
}
`

Other than that, it should not be modified directly.

#### VideoProcessor.enabled

Getter enabled describes enabled status of the current processor.

`typescript
public get enabled(): boolean;
`

#### VideoProcessor.inputTrack

Optional property inputTrack will be setted when the previous processor or ILocalVideoTrack feeds output track on the current processor:

`typescript
protected inputTrack?:MediaStreamTrack;
`

#### VideoProcessor.outputTrack

Optional property outputTrack will be setted when the current processor calling output() to generate output MediaStreamTrack:

`typescript
protected outputTrack?:MediaStreamTrack;
`

#### VideoProcessor.ID

Readonly property ID is a random ID for the current processor instance:

`typescript
public readonly ID:string;
`

#### VideoProcessor.kind

Getter kind describes current processor's kind, which is either audio or video:

`typescript
public get Kind():'video' | 'audio';
`

#### VideoProcessor.context

Optional property context is the current processor's IProcessorContext :

`typescript
protected context?: IProcessorContext;
`

#### VideoProcessor.output
method output should be called when processor was about to generate processed MediaStreamTrack:
`typescript
output(track: MediaStreamTrack, context: IProcessorContext): void;
`




$3

AudioProcessor shares almost all the property/methods with VideoProcessor, with 1 exception that AudioProcessor's processorContext is IAudioProcessorContext; and with several additions:

#### AudioProcessor.onNode

Abstract optional method onNode could be implemented in order to be notified when the previous processor
or ILocalAudioTrack feeds output AudioNode to the current audio processor:

`typescript
abstract onNode?(node: AudioNode, context: IAudioProcessorContext): void;
`

#### AudioProcessor.output

method output should be called when audio processor was about to generate processed MediaStreamTrack or AudioNode:

`typescript
output(track: MediaStreamTrack | AudioNode, context: IProcessorContext): void;
`

#### AudioProcessor.inputNode

Optional property inputNode will be setted when the previous processor or ILocalAudioTrack feeds output audio node on the current processor:

`typescript
protected inputNode?:AudioNode;
`

####AudioProcessor.outputNode

Optional property outputNode will be setted when the current processor calling output() to generate output AudioNode:

`typescript
protected outputNode?:AudioNode;
`

$3

ProcessorContext provides the ability to interact with the process pipeline's source which is ILocalVideoTrack or ILocalAudioTrack, and possiblly affecting media capture.

ProcessorContext will be assgined to the processor once the processor was connected with a pipeline has ILocalVideoTrack or ILocalAudioTrack as it's source.

#### ProcessorContext.requestApplyConstraints

Method requestApplyConstraints provides the ability to change the MediaTrackConstraints used for getting pipeline source's MediaStreamTrack :

`typescript
public requestApplyConstraints(constraints: MediaTrackConstraints, processor: IVideoProcessor): Promise;
`

Constraints supplied in requestApplyConstraints will be merged with the original constraints used for creating ICameraVideoTrack. If several processors inside the same pipline all request to apply additional constraints, the pipe order will be considered to make the final constraints.

#### ProcessorContext.requestRevertConstraints

MethodrequestRevertConstraints provides the ability to revert previous constraints request using requestApplyConstraints:

`typescript
public requestRevertConstraints(processor: IVideoProcessor):void;
`


$3


AudioProceesorContext inherits all the methods provided by ProcessorContext, with one addition getAudioContext.
#### getAudioContext
Method getAudioContext provides the ability to get AudioContext object of the current pipeline:
`typescript
public getAudioContext(): AudioContext;
`

$3

Ticker is a utitly class that helps with periodic tasks.

Ticker provides simple interface for choosing periodic task implementation, add/remove task and start/stop task.

#### new Ticker

Ticker constructor requires ticker type and tick interval as parameter:

`typescript
class Ticker{
public constructor(type:"Timer" | "RAF" | "Oscillator", interval: number):Ticker;
}
`

Ticker has three implementation to choose from:

- Timer: uses setTimeout as the internal timer
- RAF: uses requestAnimationFrame as the internal timer. Most users should choose this type of Ticker as it provides best rendering performance
- Osciilator: uses WebAudio's OscillatorNode as the internal timer. Can still keep running even the browser tab is not focused.

interval sets the time between the next callback. It is a best effort timing not an exactly timing.

#### Ticker.add

Ticker.add adds a task to the ticker:

`typescript
public add(fn: Function): void;
`

#### Ticker.remove

Ticker.remove removes the task added to the ticker previously:

`typescript
public remove():void;
`

#### Ticker.start

Ticker.start starts the already add task with settled ticker type and interval:

`typescript
public start():void;
`

####Ticker.stop

Ticker.stop stops the previously add task:

`typescript
public stop():void;
`

$3

Logger is a global utility singleton that helps the logging. It provides four log levels to log to the console.

When the extension was registered with AgoraRTC.registerExtension, and the AgoraRTC developer choose to upload log, extension logs loged with Logger will also been uploaded.

#### Logger.info, Logger.debug, Logger.warning, Logger.error

Theses methods log with different level:

`typescript
public info(...args:any[]):void;
public debug(...args:any[]):void;
public warning(...args:any[]):void;
public error(...args:any[]):void;
`

#### Logger.setLogLevel

Logger.setLogLevel set the output log level of the extension.

`typescript
public setLogLevel(level: number): void;
`

$3

Reporter is a global utility singleton that helps with event reporting to Agora analysis platform:

#### Reporter.reportApiInvoke

Repoter.reportApiInvoke can report public API calling event to Agora analysis platform:

`typescript
interface ReportApiInvokeParams {
name: string;
options: any;
reportResult?: boolean;
timeout?: number;
}
interface AgoraApiExecutor {
onSuccess: (result: T) => void;
onError: (err: Error) => void;
}

public reportApiInvoke(params: ReportApiInvokeParams): AgoraApiExecutor;
`

It accepts ReportAPIInvokeParams as parameter:

- ReportAPIInvokeParams.name: the name of the public API
- options: the arguments, or any other options related to this API invoke
- reportResult: whether to report API invoke result
- timeout: specifies how long it is Reporter thinks the API calling is timeout.

It reports two callback methods, onSuccess and onEror, which can be called when the API calling success or failed accordingly.

Extending Extension


Extending an Extension is fairly straightforward as we only need to implement _createProcessor abstract method:

`typescript
import {Extension} from 'agora-rte-extension'

class YourExtension extends Extension {
protected _createProcessor(): YourProcessor {
return new YourProcessor();
}
}
`

Extending Processor

There are several abstract methods could be implemented and they will be called at the different timing of the processing pipeline.

$3

onTrack and onNode method will be called when the previous processor/LocalTrack generated output. They are the main entry point for us to process media:

`typescript
class CustomVideoProcessor extends VideoProcesor {
protected onTrack(track: MediaStreamTrack, context: IProcessorContext){}
}

class CustomAudioProcessor extends AudioProcessor {
protected onNode(node: AudioNode, context: IAudioProcessorContext){}
}
`

$3

Typically, doing video processing requests extracting each video frame as ImageData or ArrayBuffer.

As for now InsertableStream have not been globally supported by browser vendors yet, we use canvas API here to extract video frame data:

`typescript
class CustomVideoProcessor extends VideoProcessor {
private canvas: HTMLCanvasElement;
private ctx: CanvasRenderingContext2D;
private videoElement:HTMLVideoElement;

constructor(){
super();

//initialize canvas element
this.canvas = document.createElement('canvas');
this.canvas.width = 640; // canvas's width and height will be your output video streams video dimension
this.canvas.height = 480;
this.ctx = this.canvas.getContext('2d')!;

//initialize video element
this.videoElement = document.createElement('video');
this.videoElement.muted = true;
}

onTrack(track:MediaStreamTrack, context: IProcessorContext){
//loding MediaStreamTrack into HTMLVideoElement
this.videoElement.srcObject = new MediaStream([track]);
this.videoElement.play();

//extract ImageData
this.ctx.drawImage(this.videoElement, 0, 0);
const imageData = this.ctx.getImageData(0, 0, this.canvas.width, this.canvas.height);
}
}
`

As we can see here video frame data was only been eatracted once inside the onTrack method, but we need to run it inside a constant loop to ouput constant frame rate. Luckily, we can leverage requestAnimationFrameto do this for us:

`typescript
class CustomVideoProcessor extends VideoProcessor {
onTrack(track:MediaStreamTrack, context: IProcessorContext){
this.videoElement.srcObject = new MediaStream([track]);
this.videoElement.play();

this.loop();
}

loop(){
this.ctx.drawImage(this.videoElement, 0, 0);
const imageData = this.ctx.getImageData(0, 0, this.canvas.width, this.canvas.height);

this.process(imageData);

requestAnimationFrame(()=>this.loop());
}

process(){
//your custom video processing logic
}
}
`


$3

When we've done video processing, Processor's output method should be used to generate video putput. output methods requires MediaStreamTrack and IProcessorContext as it's parameter, so we will need to assemble video buffer into a MediaStreamTrack.

Usually canvas's captureStream helps us with it:

`typescript
class CustomVideoProcessor extends VideoProcessor {
doneProcessing(){
// making an MediaStream from canvas and get MediaStreamTrack
const msStream = this.canvas.captureStream(30);
const outputTrack = msStream.getVideoTracks()[0];

//output processed track
if(this.context){
this.output(outputTrack, this.context);
}
}
}
`

$3


Audio processing differs with video processing as that audio processing typically requires WebAudio's capability to do custom audio processing.

We can implement onNode method to receive notification when the previous audio processor/ILocalAudioTrack generated output AudioNode:

`typescript
class CustomAudioProcessor extends AudioProcessor {
onNode(node: AudioNode, context: IAudioProcessorContext) {}
}
`
We can call IAudioProcessorContext.getAudioContext to get AudioContext to create our own audioNode:
`typescript
class CustomAudioProcessor extends AudioProcessor {
onNode(node: AudioNode, context: IAudioProcessorContext) {
//accuire AudioContext
const audioContext = context.getAudioContext();

//create custom gaiNode
const gainNode = audioContext.createGain();
}
}
`
Also don't forget to connect the input audio node to our custom audio node:
`typescript
class CustomAudioProcessor extends AudioProcessor {
onNode(node: AudioNode, context: IAudioProcessorContext) {
const audioContext = context.getAudioContext();

const gainNode = audioContext.createGain();

//connect
node.connect(gainNode);
}
}
`


$3


When we've done audio processing, Processor's output method should be used to generate audio output. output methods requires MediaStreamTrack/AudioNode and IAudioProcessorContext as its parameter:
`typescript

class CustomAudioProcessor extends AudioProcessor {
onNode(node: AudioNode, context: IAudioProcessorContext) {
const audioContext = context.getAudioContext();

const gainNode = audioContext.createGain();

node.connect(gainNode);

//output
this.output(gainNode, context);
}
}
``

Testing

WIP

Best Practices

$3

$3

$3