@spatialwalk/avatarkit-rtc

Unified RTC adapter for avatarkit - supports LiveKit, Agora and other RTC providers.

📚 Documentation: https://docs.spatialreal.ai/web-sdk/introduction

> For server-side RTC SDK integration, please refer to the online documentation.

Installation

⚠️ IMPORTANT: This package does NOT work standalone. You MUST install the dependencies below.

$3

| Package | Purpose | Required |
|---------|---------|----------|
| @spatialwalk/avatarkit | Web SDK for avatar rendering | ✅ Yes |
| @spatialwalk/avatarkit-rtc | This package - RTC adapter | ✅ Yes |
| livekit-client | LiveKit RTC SDK | ⚡ Choose one |
| agora-rtc-sdk-ng | Agora RTC SDK | ⚡ Choose one |

$3

``bash


With LiveKit

pnpm add @spatialwalk/avatarkit @spatialwalk/avatarkit-rtc livekit-client

With Agora

pnpm add @spatialwalk/avatarkit @spatialwalk/avatarkit-rtc agora-rtc-sdk-ng
`

$3

Step 1: Install the Web SDK (required dependency)

`bash
pnpm add @spatialwalk/avatarkit
`

Step 2: Install the RTC adapter

`bash
pnpm add @spatialwalk/avatarkit-rtc
`

Step 3: Install your chosen RTC provider SDK

`bash


For LiveKit

pnpm add livekit-client

For Agora

pnpm add agora-rtc-sdk-ng
`

$3

| Error | Cause | Solution |
|-------|-------|----------|
| No matching version found for @spatialwalk/avatarkit-rtc@^0.x.x | Wrong version number | Use pnpm add @spatialwalk/avatarkit-rtc (no version) |
| No matching version found for @spatialwalk/avatarkit@^0.x.x | Wrong version number | Use pnpm add @spatialwalk/avatarkit (no version) |
| Cannot find module '@spatialwalk/avatarkit' | Missing dependency | Install it: pnpm add @spatialwalk/avatarkit |
| Avatar placeholder - SDK integration required | SDK not initialized | Call AvatarSDK.initialize() before creating AvatarView |

Get Credentials

Before using the SDK, you need to obtain authentication credentials:

$3

- Visit the Developer Platform to create your App
- Get your App ID from the platform dashboard
- The App ID is required for SDK initialization

$3

- Session Token is required for avatar loading and SDK authentication
- Session Token should be obtained from your backend server
- Your backend server should request the Session Token from the AvatarKit Server
- Session Token has a maximum validity of 1 hour

$3

`
Your Client → Your Backend Server → AvatarKit Server → returns Session Token (max 1 hour)
`

> Note: App ID and Session Token are paired and must be used together. Each App ID corresponds to a specific list of avatars that can be driven.

Quick Start

When initializing @spatialwalk/avatarkit, use DrivingServiceMode.host since network communication is managed by this SDK.

`typescript
import { AvatarPlayer, LiveKitProvider, AgoraProvider } from '@spatialwalk/avatarkit-rtc';
import { AvatarSDK, AvatarView, AvatarManager, DrivingServiceMode, Environment } from '@spatialwalk/avatarkit';

// 0. Initialize SDK (use host mode)
await AvatarSDK.initialize(appId, {
environment: Environment.cn, // or Environment.intl
drivingServiceMode: DrivingServiceMode.host,
});

// 1. Set session token (required for avatar loading)
AvatarSDK.setSessionToken(sessionToken);

// 2. Create Avatar and AvatarView
const avatar = await AvatarManager.shared.load(characterId);
const avatarView = new AvatarView(avatar, container);

// 3. Create Provider (choose LiveKit or Agora)
const provider = new LiveKitProvider(); // or new AgoraProvider()

// 4. Create Player
const player = new AvatarPlayer(provider, avatarView, {
logLevel: 'warning', // optional
});

// 5. Connect to RTC server
await player.connect({
url: 'wss://your-livekit-server.com',
token: 'your-token',
roomName: 'room-name',
});

// 6. Start microphone
await player.startPublishing();

// 7. Stop microphone
await player.stopPublishing();

// 8. Disconnect
await player.disconnect();
`

API Reference

$3

Main entry class that manages RTC connection and avatar rendering.

#### Constructor

`typescript
new AvatarPlayer(provider: RTCProvider, avatarView: AvatarView, options?: AvatarPlayerOptions)
`

Parameters:

| Parameter | Type | Description |
|-----------|------|-------------|
| provider | LiveKitProvider \| AgoraProvider | RTC provider instance |
| avatarView | AvatarView | AvatarView instance from avatarkit |
| options | AvatarPlayerOptions | Optional configuration |

#### AvatarPlayerOptions

`typescript
interface AvatarPlayerOptions {
/* Start speaking transition frames, default 5 (~200ms at 25fps) /
transitionStartFrameCount?: number;

/* End speaking transition frames, default 40 (~1600ms at 25fps) /
transitionEndFrameCount?: number;

/* Log level: 'info' | 'warning' | 'error' | 'none', default 'warning' /
logLevel?: LogLevel;
}
`

#### Properties

| Property | Type | Description |
|----------|------|-------------|
| isConnected | boolean | Whether connected to RTC server |

#### Methods

##### connect(config)

Connect to RTC server.

`typescript
await player.connect(config: RTCConnectionConfig): Promise
`

LiveKit Config:

`typescript
interface LiveKitConnectionConfig {
url: string; // LiveKit server URL (wss://...)
token: string; // Auth token
roomName: string; // Room name
}
`

Agora Config:

`typescript
interface AgoraConnectionConfig {
appId: string; // Agora App ID
channel: string; // Channel name
token?: string; // Auth token (required in production)
uid?: number; // User ID (optional, 0 = auto assign)
}
`

##### disconnect()

Disconnect RTC connection and clean up all resources.

`typescript
await player.disconnect(): Promise
`

##### reconnect()

Reconnect to RTC server using the last connection configuration. Useful for recovering from connection issues or stream stalls.

`typescript
await player.reconnect(): Promise
`

Note: Throws an error if no previous connection exists. Call connect() first.

##### startPublishing()

Start publishing microphone audio. Automatically requests microphone permission.

`typescript
await player.startPublishing(): Promise
`

##### stopPublishing()

Stop publishing microphone audio and release the device.

`typescript
await player.stopPublishing(): Promise
`

##### publishAudio(track)

Publish custom audio track (advanced usage). For non-microphone audio sources like audio file playback.

`typescript
await player.publishAudio(track: MediaStreamTrack): Promise
`

Supported audio sources:

| Source | How to obtain |
|--------|---------------|
| 🎤 Microphone | Use startPublishing() for convenience |
| 🎵

Example:

`typescript
// Play audio from browser `

##### unpublishAudio()

Stop publishing custom audio track.

`typescript
await player.unpublishAudio(): Promise
`

Note: unpublishAudio() does not stop the track - caller is responsible for managing track lifecycle:

`typescript
await player.unpublishAudio();
stream.getTracks().forEach(t => t.stop()); // Clean up externally
`

##### getConnectionState()

Get current connection state.

`typescript
player.getConnectionState(): ConnectionState
`

ConnectionState enum:

| Value | Description |
|-------|-------------|
| 'disconnected' | Not connected |
| 'connecting' | Connecting |
| 'connected' | Connected |
| 'reconnecting' | Reconnecting |
| 'failed' | Connection failed |

##### getNativeClient()

Get underlying RTC client object for platform-specific features.

`typescript
player.getNativeClient(): unknown
`

Return value:

| Provider | Return type |
|----------|-------------|
| LiveKit | Room instance (exported as LiveKitRoom) |
| Agora | IAgoraRTCClient instance (exported as AgoraClient) |

Type safety notes:

- Direct Provider usage: Returns concrete type, no assertion needed
- Via AvatarPlayer: Returns unknown, requires manual type assertion

Example:

`typescript
// Method 1: Use Provider directly (recommended, full type hints)
import { LiveKitProvider, LiveKitRoom } from '@spatialwalk/avatarkit-rtc';

const provider = new LiveKitProvider();
const room = provider.getNativeClient(); // Type: LiveKitRoom | null
console.log('Remote participants:', room?.remoteParticipants.size);

// Method 2: Via AvatarPlayer (requires assertion)
import { AvatarPlayer, LiveKitRoom } from '@spatialwalk/avatarkit-rtc';

const room = player.getNativeClient() as LiveKitRoom | null;
console.log('Remote participants:', room?.remoteParticipants.size);

// Agora example
import { AgoraProvider, AgoraClient } from '@spatialwalk/avatarkit-rtc';

const provider = new AgoraProvider();
const client = provider.getNativeClient(); // Type: AgoraClient | null
console.log('Connection state:', client?.connectionState);
`

##### on(event, handler)

Listen to events.

`typescript
player.on(event: string, handler: Function): void
`

Supported events:

| Event | Callback params | Description |
|-------|-----------------|-------------|
| 'connected' | () | Connection successful |
| 'disconnected' | () | Disconnected |
| 'error' | (error: Error) | Error occurred |
| 'connection-state-changed' | (state: ConnectionState) | Connection state changed |
| 'stalled' | () | Data stream stalled (no frames for 3s). Avatar auto-transitions to idle. |

Example: Handling stream stalls

`typescript
player.on('stalled', async () => {
console.log('Stream stalled, attempting reconnection...');
try {
await player.reconnect();
} catch (error) {
console.error('Reconnection failed:', error);
}
});
`

##### off(event, handler)

Remove event listener.

`typescript
player.off(event: string, handler: Function): void
`

---

$3

LiveKit RTC provider implementation.

`typescript
import { LiveKitProvider } from '@spatialwalk/avatarkit-rtc';

const provider = new LiveKitProvider();
`

---

$3

Agora RTC provider implementation.

`typescript
import { AgoraProvider } from '@spatialwalk/avatarkit-rtc';

const provider = new AgoraProvider({
debugLogging: true, // optional: enable debug logs
});
`

AgoraProviderOptions:

`typescript
interface AgoraProviderOptions {
/* Enable verbose debug logging, default false /
debugLogging?: boolean;
}
`

---

$3

Type guard functions for config type checking.

`typescript
import { isLiveKitConfig, isAgoraConfig } from '@spatialwalk/avatarkit-rtc';

if (isLiveKitConfig(config)) {
// config is LiveKitConnectionConfig
}

if (isAgoraConfig(config)) {
// config is AgoraConnectionConfig
}
`

---

$3

SDK exports type aliases for underlying RTC clients, useful when using getNativeClient():

`typescript
import type { LiveKitRoom, AgoraClient } from '@spatialwalk/avatarkit-rtc';

// LiveKitRoom = Room type from livekit-client
// AgoraClient = IAgoraRTCClient type from agora-rtc-sdk-ng
`

---

Log Levels

SDK log level is configured via AvatarPlayerOptions.logLevel:

| Level | Output |
|-------|--------|
| 'info' | All logs (connection state, frame processing, debug info) |
| 'warning' | Warnings + errors (default) |
| 'error' | Errors only |
| 'none' | Disable all logs |

`typescript
const player = new AvatarPlayer(provider, avatarView, {
logLevel: 'info', // Enable for debugging
});
`

---

Complete Example

$3

`typescript
import { AvatarPlayer, LiveKitProvider } from '@spatialwalk/avatarkit-rtc';
import { AvatarSDK, AvatarView, AvatarManager, DrivingServiceMode, Environment } from '@spatialwalk/avatarkit';

async function initLiveKit() {
// Initialize SDK (use host mode)
await AvatarSDK.initialize('your-app-id', {
environment: Environment.cn,
drivingServiceMode: DrivingServiceMode.host,
});

// Set session token (required)
AvatarSDK.setSessionToken('your-session-token');

// Create Avatar
const avatar = await AvatarManager.shared.load('character-id');
const container = document.getElementById('avatar-container')!;
const avatarView = new AvatarView(avatar, container);

// Create Player
const provider = new LiveKitProvider();
const player = new AvatarPlayer(provider, avatarView, {
logLevel: 'info',
transitionStartFrameCount: 5,
transitionEndFrameCount: 40,
});

// Listen to events
player.on('connected', () => console.log('Connected!'));
player.on('disconnected', () => console.log('Disconnected!'));
player.on('error', (err) => console.error('Error:', err));

// Connect
await player.connect({
url: 'wss://your-livekit-server.com',
token: 'your-token',
roomName: 'my-room',
});

// Start microphone
await player.startPublishing();

// Stop microphone
// await player.stopPublishing();

// Disconnect
// await player.disconnect();
}
`

$3

`typescript
import { AvatarPlayer, AgoraProvider } from '@spatialwalk/avatarkit-rtc';
import { AvatarSDK, AvatarView, AvatarManager, DrivingServiceMode, Environment } from '@spatialwalk/avatarkit';

async function initAgora() {
// Initialize SDK (use host mode)
await AvatarSDK.initialize('your-app-id', {
environment: Environment.cn,
drivingServiceMode: DrivingServiceMode.host,
});

// Set session token (required)
AvatarSDK.setSessionToken('your-session-token');

// Create Avatar
const avatar = await AvatarManager.shared.load('character-id');
const container = document.getElementById('avatar-container')!;
const avatarView = new AvatarView(avatar, container);

// Create Player
const provider = new AgoraProvider({ debugLogging: true });
const player = new AvatarPlayer(provider, avatarView);

// Connect
await player.connect({
appId: 'your-agora-app-id',
channel: 'my-channel',
token: 'your-token', // Required in production
uid: 0, // 0 = auto assign
});

// Start microphone
await player.startPublishing();
}
`

---

Browser Compatibility

| Feature | Chrome | Firefox | Safari | Edge |
|---------|--------|---------|--------|------|
| LiveKit (VP8 + RTCRtpScriptTransform) | ✅ 94+ | ✅ 117+ | ✅ 15.4+ | ✅ 94+ |
| Agora (H.264 + SEI) | ✅ 74+ | ✅ 78+ | ✅ 14.1+ | ✅ 79+ |

Note: LiveKit requires browser support for RTCRtpScriptTransform` API (Safari 15.4+ supported).

---

License

MIT