@convai/web-sdk

JavaScript/TypeScript SDK for Convai AI voice assistants. Build voice-powered AI interactions for web applications with real-time audio/video streaming. Supports both React and Vanilla JavaScript/TypeScript.

Installation

``bash
npm install @convai/web-sdk
`

Basic Setup

$3

`tsx
import { useConvaiClient, ConvaiWidget } from "@convai/web-sdk";

function App() {
const convaiClient = useConvaiClient({
apiKey: "your-api-key",
characterId: "your-character-id",
});

return ;
}
`

$3

`typescript
import { ConvaiClient, createConvaiWidget } from "@convai/web-sdk/vanilla";

// Create client with configuration
const client = new ConvaiClient({
apiKey: "your-api-key",
characterId: "your-character-id",
});

// Create widget - auto-connects on first user click
const widget = createConvaiWidget(document.body, {
convaiClient: client,
});

// Cleanup when done
widget.destroy();
`

Exports

$3

Components:

- ConvaiWidget - Main chat widget component

Hooks:

- useConvaiClient(config?) - Main client hook
- useCharacterInfo(characterId, apiKey) - Fetch character metadata
- useLocalCameraTrack() - Get local camera track

Core Client:

- ConvaiClient - Core client class

Types:

- ConvaiConfig - Configuration interface
- ConvaiClientState - Client state interface
- ChatMessage - Message interface
- IConvaiClient - Client interface
- AudioControls - Audio control interface
- VideoControls - Video control interface
- ScreenShareControls - Screen share control interface

Components:

- AudioRenderer - Audio playback component
- AudioContext - Audio context provider

$3

Functions:

- createConvaiWidget(container, options) - Create widget instance
- destroyConvaiWidget(widget) - Destroy widget instance

Classes:

- ConvaiClient - Core client class
- AudioRenderer - Audio playback handler

Types:

- VanillaWidget - Widget instance interface
- VanillaWidgetOptions - Widget options interface
- IConvaiClient - Client interface
- ConvaiConfig - Configuration interface
- ConvaiClientState - Client state interface
- ChatMessage - Message interface

$3

Classes:

- ConvaiClient - Main client class
- AudioManager - Audio management
- VideoManager - Video management
- ScreenShareManager - Screen share management
- MessageHandler - Message handling
- EventEmitter - Event emitter base class

Types:

- All types from React/Vanilla exports
- ConvaiClientType - Type alias for ConvaiClient

Props and Configuration

$3

`tsx
interface ConvaiWidgetProps {
/* Convai client instance (required) /
convaiClient: IConvaiClient & {
activity?: string;
isAudioMuted: boolean;
isVideoEnabled: boolean;
isScreenShareActive: boolean;
};
/* Show video toggle button in settings (default: true) /
showVideo?: boolean;
/* Show screen share toggle button in settings (default: true) /
showScreenShare?: boolean;
}
`

$3

`typescript
interface VanillaWidgetOptions {
/* Convai client instance (required) /
convaiClient: IConvaiClient & {
activity?: string;
chatMessages: ChatMessage[];
};
/* Show video toggle button in settings (default: true) /
showVideo?: boolean;
/* Show screen share toggle button in settings (default: true) /
showScreenShare?: boolean;
}
`

$3

`typescript
interface ConvaiConfig {
/* Your Convai API key from convai.com dashboard (required) /
apiKey: string;
/* The Character ID to connect to (required) /
characterId: string;
/**
* End user identifier for speaker management (optional).
* When provided: enables long-term memory and analytics
* When not provided: anonymous mode, no persistent memory
*/
endUserId?: string;
/* Custom Convai API URL (optional, defaults to production endpoint) /
url?: string;
/**
* Enable video capability (default: false).
* If true, connection_type will be "video" (supports audio, video, and screenshare).
* If false, connection_type will be "audio" (audio only).
*/
enableVideo?: boolean;
/**
* Start with video camera on when connecting (default: false).
* Only works if enableVideo is true.
*/
startWithVideoOn?: boolean;
/**
* Start with microphone on when connecting (default: false).
* If false, microphone stays off until user enables it.
*/
startWithAudioOn?: boolean;
/* Enable text-to-speech audio generation (default: true) /
ttsEnabled?: boolean;
}
`

Features

$3

To enable video capabilities, set enableVideo: true in your configuration. This enables audio, video, and screen sharing.

React:

`tsx
import { useConvaiClient, ConvaiWidget } from "@convai/web-sdk";

function App() {
const convaiClient = useConvaiClient({
apiKey: "your-api-key",
characterId: "your-character-id",
enableVideo: true,
startWithVideoOn: false, // Camera off by default
});

return (
convaiClient={convaiClient}
showVideo={true}
showScreenShare={true}
/>
);
}
`

Vanilla:

`typescript
import { ConvaiClient, createConvaiWidget } from "@convai/web-sdk/vanilla";

const client = new ConvaiClient({
apiKey: "your-api-key",
characterId: "your-character-id",
enableVideo: true,
startWithVideoOn: false,
});

const widget = createConvaiWidget(document.body, {
convaiClient: client,
showVideo: true,
showScreenShare: true,
});
`

Manual Video Controls:

`typescript
// Enable video camera
await convaiClient.videoControls.enableVideo();

// Disable video camera
await convaiClient.videoControls.disableVideo();

// Toggle video
await convaiClient.videoControls.toggleVideo();

// Check video state
const isVideoEnabled = convaiClient.videoControls.isVideoEnabled;

// Set video quality
await convaiClient.videoControls.setVideoQuality("high"); // 'low' | 'medium' | 'high'

// Get available video devices
const devices = await convaiClient.videoControls.getVideoDevices();

// Set specific video device
await convaiClient.videoControls.setVideoDevice(deviceId);
`

Screen Sharing:

`typescript
// Enable screen share
await convaiClient.screenShareControls.enableScreenShare();

// Enable screen share with audio
await convaiClient.screenShareControls.enableScreenShareWithAudio();

// Disable screen share
await convaiClient.screenShareControls.disableScreenShare();

// Toggle screen share
await convaiClient.screenShareControls.toggleScreenShare();

// Check screen share state
const isActive = convaiClient.screenShareControls.isScreenShareActive;
`

Video State Monitoring:

`typescript
// React
const { isVideoEnabled } = convaiClient;

// Core API (event-based)
convaiClient.videoControls.on("videoStateChange", (state) => {
console.log("Video enabled:", state.isVideoEnabled);
console.log("Video hidden:", state.isVideoHidden);
});
`

$3

Enable lipsync to receive blendshape data for animating 3D character faces in sync with speech:

`typescript
const client = new ConvaiClient({
apiKey: "your-api-key",
characterId: "your-character-id",
enableLipsync: true,
blendshapeConfig: {
format: "arkit", // or "mha" for MetaHuman
},
});

await client.connect();

// In your 3D render loop (60 FPS)
let conversationStartTime = 0;

client.on("speakingChange", (speaking) => {
if (speaking) conversationStartTime = Date.now();
});

function render() {
const elapsedSeconds = (Date.now() - conversationStartTime) / 1000;
const result = client.blendshapeQueue.getFrameAtTime(elapsedSeconds);

if (result) {
// Apply blendshape values to your 3D character
myCharacter.morphTargets["jawOpen"] = result.frame[0];
myCharacter.morphTargets["mouthSmile"] = result.frame[1];
// ... apply remaining blendshapes
}

requestAnimationFrame(render);
}
`

Blendshape Formats:

- arkit - 61 blendshapes (iOS ARKit standard)
- mha - 251 blendshapes (MetaHuman)

$3

Interrupt the character's current response to allow the user to speak immediately.

React:

`tsx
function ChatInterface() {
const convaiClient = useConvaiClient({
/ config /
});

const handleInterrupt = () => {
// Interrupt the bot's current response
convaiClient.sendInterruptMessage();
};

return Interrupt;
}
`

Vanilla:

`typescript
const interruptButton = document.getElementById("interrupt-btn");

interruptButton.addEventListener("click", () => {
client.sendInterruptMessage();
});
`

Voice Mode Interruption Pattern:

When implementing voice mode, interrupt the bot when the user starts speaking:

`typescript
// When user enters voice mode
const enterVoiceMode = async () => {
// Interrupt any ongoing bot response
convaiClient.sendInterruptMessage();

// Unmute microphone
await convaiClient.audioControls.unmuteAudio();
};

// When user exits voice mode
const exitVoiceMode = async () => {
// Interrupt any ongoing bot response
convaiClient.sendInterruptMessage();

// Mute microphone
await convaiClient.audioControls.muteAudio();
};
`

$3

Control the user's microphone input.

React:

`tsx
function AudioControls() {
const convaiClient = useConvaiClient({
/ config /
});

const handleMute = async () => {
await convaiClient.audioControls.muteAudio();
};

const handleUnmute = async () => {
await convaiClient.audioControls.unmuteAudio();
};

const handleToggle = async () => {
await convaiClient.audioControls.toggleAudio();
};

return (


`typescript
// Mute microphone
await client.audioControls.muteAudio();

// Unmute microphone
await client.audioControls.unmuteAudio();

// Toggle mute state
await client.audioControls.toggleAudio();

// Check mute state
const isMuted = client.audioControls.isAudioMuted;

// Enable audio (request permissions if needed)
await client.audioControls.enableAudio();

// Disable audio
await client.audioControls.disableAudio();
`

Audio Device Management:

`typescript
// Get available audio devices
const devices = await convaiClient.audioControls.getAudioDevices();

// Set specific audio device
await convaiClient.audioControls.setAudioDevice(deviceId);

// Monitor audio level
convaiClient.audioControls.startAudioLevelMonitoring();

convaiClient.audioControls.on("audioLevelChange", (level) => {
console.log("Audio level:", level);
// level is a number between 0 and 1
});

convaiClient.audioControls.stopAudioLevelMonitoring();
`

Audio State Monitoring:

`typescript
// React
const { isAudioMuted } = convaiClient;

// Core API (event-based)
convaiClient.audioControls.on("audioStateChange", (state) => {
console.log("Audio enabled:", state.isAudioEnabled);
console.log("Audio muted:", state.isAudioMuted);
console.log("Audio level:", state.audioLevel);
});
`

$3

Control whether the character's responses are spoken aloud (text-to-speech).

React:

`tsx
function TTSControls() {
const convaiClient = useConvaiClient({
/ config /
});

const handleToggleTTS = (enabled: boolean) => {
convaiClient.toggleTts(enabled);
};

return (


`typescript
// Enable text-to-speech (character will speak responses)
client.toggleTts(true);

// Disable text-to-speech (character will only send text, no audio)
client.toggleTts(false);
`

Initial TTS Configuration:

`typescript
// Set TTS state during connection
const client = new ConvaiClient({
apiKey: "your-api-key",
characterId: "your-character-id",
ttsEnabled: true, // Enable TTS by default
});

// Or disable initially
const client = new ConvaiClient({
apiKey: "your-api-key",
characterId: "your-character-id",
ttsEnabled: false, // Disable TTS
});
`

$3

Voice mode allows users to speak instead of typing. The widget automatically handles voice mode, but you can implement it manually.

React - Manual Voice Mode:

`tsx
import { useConvaiClient } from "@convai/web-sdk";
import { useState, useEffect } from "react";

function CustomChatInterface() {
const convaiClient = useConvaiClient({
/ config /
});
const [isVoiceMode, setIsVoiceMode] = useState(false);

const enterVoiceMode = async () => {
// Interrupt any ongoing bot response
convaiClient.sendInterruptMessage();

// Unmute microphone
await convaiClient.audioControls.unmuteAudio();

setIsVoiceMode(true);
};

const exitVoiceMode = async () => {
// Interrupt any ongoing bot response
convaiClient.sendInterruptMessage();

// Mute microphone
await convaiClient.audioControls.muteAudio();

setIsVoiceMode(false);
};

// Monitor user transcription for voice input
useEffect(() => {
const transcription = convaiClient.userTranscription;
if (transcription && isVoiceMode) {
// Display real-time transcription
console.log("User is saying:", transcription);
}
}, [convaiClient.userTranscription, isVoiceMode]);

return (


`typescript
let isVoiceMode = false;

const enterVoiceMode = async () => {
// Interrupt any ongoing bot response
client.sendInterruptMessage();

// Unmute microphone
await client.audioControls.unmuteAudio();

isVoiceMode = true;
updateUI();
};

const exitVoiceMode = async () => {
// Interrupt any ongoing bot response
client.sendInterruptMessage();

// Mute microphone
await client.audioControls.muteAudio();

isVoiceMode = false;
updateUI();
};

// Monitor user transcription
client.on("userTranscriptionChange", (transcription) => {
if (isVoiceMode && transcription) {
// Display real-time transcription
document.getElementById("transcription").textContent = transcription;
}
});

function updateUI() {
const voiceButton = document.getElementById("voice-btn");
const transcriptionDiv = document.getElementById("transcription");

if (isVoiceMode) {
voiceButton.textContent = "Stop Voice Mode";
transcriptionDiv.style.display = "block";
} else {
voiceButton.textContent = "Start Voice Mode";
transcriptionDiv.style.display = "none";
}
}
`

Voice Mode with State Monitoring:

`typescript
// Monitor agent state to handle voice mode transitions
convaiClient.on("stateChange", (state) => {
if (isVoiceMode) {
switch (state.agentState) {
case "listening":
// User can speak
console.log("Bot is listening");
break;
case "thinking":
// Bot is processing
console.log("Bot is thinking");
break;
case "speaking":
// Bot is responding
console.log("Bot is speaking");
// Optionally interrupt if user wants to speak
break;
}
}
});
`

$3

Connect:

`typescript
// React - config passed to hook
const convaiClient = useConvaiClient({
apiKey: "your-api-key",
characterId: "your-character-id",
});

// Or connect manually
await convaiClient.connect({
apiKey: "your-api-key",
characterId: "your-character-id",
});

// Vanilla
const client = new ConvaiClient();
await client.connect({
apiKey: "your-api-key",
characterId: "your-character-id",
});
`

Disconnect:

`typescript
await convaiClient.disconnect();
`

Reconnect:

`typescript
await convaiClient.reconnect();
`

Reset Session:

`typescript
// Clear conversation history and start new session
convaiClient.resetSession();
`

Connection State:

`typescript
// React
const { state } = convaiClient;
console.log("Connected:", state.isConnected);
console.log("Connecting:", state.isConnecting);
console.log("Agent state:", state.agentState); // 'disconnected' | 'connected' | 'listening' | 'thinking' | 'speaking'

// Core API (event-based)
convaiClient.on("stateChange", (state) => {
console.log("State changed:", state);
});

convaiClient.on("connect", () => {
console.log("Connected");
});

convaiClient.on("disconnect", () => {
console.log("Disconnected");
});
`

$3

Send Text Message:

`typescript
convaiClient.sendUserTextMessage("Hello, how are you?");
`

Send Trigger Message:

`typescript
// Trigger specific character action
convaiClient.sendTriggerMessage("greet", "User entered the room");

// Trigger without message
convaiClient.sendTriggerMessage("wave");
`

Update Context:

`typescript
// Update template keys (e.g., user name, location)
convaiClient.updateTemplateKeys({
user_name: "John",
location: "New York",
});

// Update dynamic information
convaiClient.updateDynamicInfo({
text: "User is currently browsing the products page",
});
`

Message History:

`typescript
// React
const { chatMessages } = convaiClient;

// Core API (event-based)
convaiClient.on("message", (message: ChatMessage) => {
console.log("New message:", message.content);
console.log("Message type:", message.type);
});

convaiClient.on("messagesChange", (messages: ChatMessage[]) => {
console.log("All messages:", messages);
});
`

Message Types:

`typescript
type ChatMessageType =
| "user" // User's sent message
| "convai" // Character's response
| "user-transcription" // Real-time speech-to-text from user
| "bot-llm-text" // Character's LLM-generated text
| "emotion" // Character's emotional state
| "behavior-tree" // Behavior tree response
| "action" // Action execution
| "bot-emotion" // Bot emotional response
| "user-llm-text" // User text processed by LLM
| "interrupt-bot"; // Interrupt message
`

$3

Agent State:

`typescript
// React
const { state } = convaiClient;

// Check specific states
if (state.isListening) {
console.log("Bot is listening");
}

if (state.isThinking) {
console.log("Bot is thinking");
}

if (state.isSpeaking) {
console.log("Bot is speaking");
}

// Combined state
console.log(state.agentState); // 'disconnected' | 'connected' | 'listening' | 'thinking' | 'speaking'
`

User Transcription:

`typescript
// React
const { userTranscription } = convaiClient;

// Core API (event-based)
convaiClient.on("userTranscriptionChange", (transcription: string) => {
console.log("User is saying:", transcription);
});
`

Bot Ready State:

`typescript
// React
const { isBotReady } = convaiClient;

// Core API (event-based)
convaiClient.on("botReady", () => {
console.log("Bot is ready to receive messages");
});
`

Getting Convai Credentials

1. Visit convai.com and create an account
2. Navigate to your dashboard
3. Create a new character or use an existing one
4. Copy your API Key from the dashboard
5. Copy your Character ID from the character details

Import Paths

`typescript
// Default: React version (backward compatible)
import { useConvaiClient, ConvaiWidget } from "@convai/web-sdk";

// Explicit React import
import { useConvaiClient, ConvaiWidget } from "@convai/web-sdk/react";

// Vanilla JS/TS
import { ConvaiClient, createConvaiWidget } from "@convai/web-sdk/vanilla";

// Core only (no UI, framework agnostic)
import { ConvaiClient } from "@convai/web-sdk/core";
`

TypeScript Support

All exports are fully typed:

`typescript
import type {
ConvaiClient,
ConvaiConfig,
ConvaiClientState,
ChatMessage,
AudioControls,
VideoControls,
ScreenShareControls,
IConvaiClient,
} from "@convai/web-sdk";
``

Support

- Convai Forum
- API Reference
- Convai Website