@khaveeai/react

🎭 React components and hooks for intelligent VRM AI avatars with advanced animation and lip synchronization.


Features

- 🤖 Smart VRM Avatars - 3D character rendering with Three.js/R3F
- 🎤 Real-time Voice Chat - OpenAI Realtime API integration with WebRTC
- 👄 Automatic Lip Sync - MFCC-based phoneme detection works automatically with OpenAI Realtime
- 💬 Talking Animations - Automatic gesture animations during AI speech
- 🎬 Animation System - FBX animation loading with automatic Mixamo remapping
- 😊 Facial Expressions - Smooth VRM expression control with transitions
- 👁️ Natural Blinking - Randomized blinking animations for lifelike characters
- 🔊 Audio Lip Sync - File-based lip sync analysis with MFCC and DTW algorithms
- 🛠️ Function Calling - OpenAI tool integration for RAG and custom functions
- ⚡ High Performance - Optimized rendering with automatic cleanup

Installation

``bash


Core SDK

`tsx
import { Canvas } from '@react-three/fiber';
import { KhaveeProvider, VRMAvatar } from '@khaveeai/react';

function App() {
return (





src="/models/character.vrm"
position={[0, -1, 0]}
/>


);
}
`

$3

`tsx
import { VRMAvatar, useVRMAnimations } from '@khaveeai/react';

function AnimatedAvatar() {
const { animate } = useVRMAnimations();

// Define animations (just provide URLs!)
const animations = {
idle: '/animations/idle.fbx', // Auto-plays
dance: '/animations/dance.fbx',
wave: '/animations/wave.fbx'
};

return (
<>
src="/models/character.vrm"
animations={animations} // SDK handles everything!
/>

animate('dance')}>💃 Dance
animate('wave')}>👋 Wave

);
}
`

$3

`tsx
import { VRMAvatar, useVRMExpressions } from '@khaveeai/react';

function ExpressiveAvatar() {
const { setExpression } = useVRMExpressions();

return (
<>


setExpression('happy', 1)}>😊 Happy
setExpression('sad', 1)}>😢 Sad

);
}
`

$3

`tsx
import { useAudioLipSync } from '@khaveeai/react';

function LipSyncDemo() {
const { analyzeLipSync, stopLipSync, isAnalyzing, currentPhoneme } = useAudioLipSync();

return (


####

Root provider that manages VRM state and optional provider configuration.

`tsx
interface KhaveeConfig {
llm?: LLMProvider; // Optional: Chat AI provider
tts?: TTSProvider; // Optional: Text-to-speech provider
realtime?: RealtimeProvider; // Optional: Real-time voice chat provider
tools?: RealtimeTool[]; // Optional: Custom functions
}

{/ config is optional /}
{children}

`

####

3D VRM character component with automatic animation, lip sync, and talking animations.

`tsx
interface VRMAvatarProps {
src: string; // Path to .vrm file
position?: [number, number, number];
rotation?: [number, number, number];
scale?: [number, number, number];
animations?: AnimationConfig; // FBX animation URLs
enableBlinking?: boolean; // Enable natural blinking (default: true)
enableTalkingAnimations?: boolean; // Enable gestures during AI speech (default: true)
}
`

Animation Config:
`tsx
interface AnimationConfig {
[name: string]: string; // Animation name -> FBX file URL
}

// Example
const animations = {
idle: '/animations/breathing.fbx', // Auto-plays on load
walk: '/animations/walking.fbx',
dance: '/animations/dancing.fbx',
talking: '/animations/talking.fbx', // Played during AI speech
gesture1: '/animations/gesture.fbx' // Also played during speech
};
// Note: Animations with 'talk', 'gesture', or 'speak' in the name
// are automatically played randomly when chatStatus === 'speaking'
`

$3

#### useRealtime()

Real-time voice chat with OpenAI Realtime API.

`tsx
const {
// Connection
isConnected: boolean,
connect: () => Promise,
disconnect: () => Promise,

// Chat state
chatStatus: 'stopped' | 'ready' | 'listening' | 'speaking' | 'thinking',
conversation: Conversation[],
currentVolume: number,
isThinking: boolean,

// Lip sync (automatic with VRMAvatar)
currentPhoneme: PhonemeData | null,
startAutoLipSync: () => Promise,
stopAutoLipSync: () => void,

// Actions
sendMessage: (text: string) => Promise,
interrupt: () => void,
registerFunction: (tool: RealtimeTool) => void
} = useRealtime();
`

#### useAudioLipSync()

Analyze audio files for phoneme detection and lip sync.

`tsx
const {
analyzeLipSync: (audioUrl: string, options?: {
sensitivity?: number; // 0.1 to 1.0
smoothing?: number; // 0.1 to 1.0
intensityMultiplier?: number; // 1.0 to 5.0
minIntensity?: number; // 0.0 to 1.0
onPhonemeChange?: (phoneme: PhonemeData) => void;
}) => Promise,
stopLipSync: () => void,
isAnalyzing: boolean,
mouthState: MouthState, // Current mouth state
currentPhoneme: PhonemeData | null,
audioElement: HTMLAudioElement | null
} = useAudioLipSync();
`

#### useVRMExpressions()

Control VRM facial expressions with smooth transitions.

`tsx
const {
expressions: Record,
setExpression: (name: string, value: number) => void,
resetExpressions: () => void,
setMultipleExpressions: (expressions: Record) => void
} = useVRMExpressions();
`

#### useVRMAnimations()

Control VRM body animations with smooth transitions.

`tsx
const {
currentAnimation: string | null,
animate: (name: string) => void, // Play animation by name
stopAnimation: () => void, // Stop current animation
availableAnimations: string[] // List of loaded animations
} = useVRMAnimations();
`

#### useVRM()

Access the raw VRM model instance.

`tsx
const vrm: VRM | null = useVRM();
`

#### useKhavee()

Access the complete SDK context (advanced usage).

`tsx
const {
config, // Optional provider config
vrm, // VRM instance
setVrm, // Set VRM instance
expressions, // Current expressions
setExpression, // Set single expression
resetExpressions, // Reset all expressions
setMultipleExpressions, // Set multiple expressions
currentAnimation, // Current animation name
animate, // Play animation
stopAnimation, // Stop animation
availableAnimations, // Available animations
realtimeProvider // Realtime provider instance
} = useKhavee();
`

Advanced Usage

$3

When using VRMAvatar with OpenAIRealtimeProvider, lip sync happens automatically! The avatar's mouth movements are synchronized with the AI's speech using MFCC-based phoneme detection:

`tsx
import { KhaveeProvider, VRMAvatar } from '@khaveeai/react';
import { OpenAIRealtimeProvider } from '@khaveeai/providers-openai-realtime';

const realtime = new OpenAIRealtimeProvider({
apiKey: process.env.NEXT_PUBLIC_OPENAI_API_KEY!,
voice: 'coral',
});

function App() {
return (


{/ Lip sync happens automatically when AI speaks! /}
src="/models/avatar.vrm"
enableBlinking={true} // Natural blinking
enableTalkingAnimations={true} // Gestures during speech
/>


);
}
`

The system automatically:
- ✅ Analyzes AI voice audio in real-time
- ✅ Detects phonemes (aa, ee, ih, ou, oh)
- ✅ Applies mouth shapes to VRM expressions
- ✅ Plays talking/gesture animations randomly
- ✅ Resets mouth to neutral when AI stops speaking

$3

For pre-recorded audio files, use the useAudioLipSync hook with advanced MFCC (Mel-Frequency Cepstral Coefficients) analysis and Dynamic Time Warping:

`tsx
import { useAudioLipSync } from '@khaveeai/react';

function AdvancedLipSync() {
const { analyzeLipSync, currentPhoneme, mouthState } = useAudioLipSync();

const startAnalysis = () => {
analyzeLipSync('/audio/speech.wav', {
sensitivity: 0.8, // Higher = more sensitive
intensityMultiplier: 3.0, // Boost mouth movement
minIntensity: 0.3, // Minimum threshold
onPhonemeChange: (phoneme) => {
console.log('Detected:', phoneme.phoneme, phoneme.intensity);
}
});
};

return (