@marcosremar/cabecao

Modern React 3D avatar component with real-time chat, WebSocket support, and lip-sync capabilities.

Features

- 🎭 Real-time 3D Avatar: Powered by React Three Fiber and Three.js
- 🎤 Voice Chat: WebSocket-based real-time audio processing
- 👄 Lip-sync: Automatic mouth movement synchronization
- 😊 Facial Expressions: Dynamic facial expressions based on content
- 🎨 Customizable: Easy configuration for gaze direction, animations, and more
- 📦 Streaming: Efficient chunk-based audio streaming
- 🔊 VAD: Voice Activity Detection for hands-free interaction

Installation

``bash
npm install @marcosremar/cabecao
`

Peer Dependencies

Make sure to install the required peer dependencies:

`bash
npm install @react-three/drei @react-three/fiber @ricky0123/vad-react leva react react-dom socket.io-client three
`

Basic Usage

`jsx
import { Cabecao } from '@marcosremar/cabecao';

function App() {
return (
wsUrl="ws://localhost:4002"
apiUrl="http://localhost:4001"
/>
);
}
`

Configuration

$3

| Prop | Type | Default | Description |
|------|------|---------|-------------|
| wsUrl | string | "http://localhost:4002" | WebSocket server URL |
| apiUrl | string | "http://localhost:4001" | REST API server URL |
| r2Url | string | undefined | Cloudflare R2 URL for models |
| modelPath | string | undefined | Custom model path |
| showControls | boolean | false | Show Leva controls |
| autoStartVAD | boolean | false | Auto-start voice detection |
| showStartButton | boolean | true | Show start button |
| vadEnabled | boolean | true | Enable voice activity detection |
| gazeConfig | object | See below | Eye gaze configuration |
| style | object | {} | Custom CSS styles |
| className | string | undefined | CSS class name |

$3

`jsx
const gazeConfig = {
enabled: true,
talking0: {
rightIntensity: 0.15, // 0-1, how much to look right
downIntensity: 0.08 // 0-1, how much to look down
}
};


`

Advanced Usage

$3

`jsx
modelPath="/path/to/custom/model.glb"
r2Url="https://your-r2-bucket.com"
wsUrl="ws://your-websocket-server"
/>
`

$3

`jsx
style={{
width: '100%',
height: '500px',
borderRadius: '12px'
}}
className="my-avatar"
showControls={true}
/>
`

WebSocket API

The component expects a WebSocket server that handles:

$3


- chat: Audio data with sample rate
`js
{
audio: Float32Array,
sampleRate: 16000
}
`

$3

- audio-chunk: Audio response with visemes
`js
{
text: "Hello there!",
audio: "data:audio/webm;codecs=opus;base64,UklGRiQF...",
visemes: [
{ v: "X", start: 0, end: 100 },
{ v: "H", start: 100, end: 200 }
],
animation: "Talking_0",
facialExpression: "smile"
}
`

- chat-error: Error handling
`js
{
error: "Error message"
}
`

Animations

Supported animations:
- Idle - Default idle animation
- Talking_0 - Primary talking animation
- Talking_1 - Secondary talking animation
- Talking_2 - Tertiary talking animation

Facial Expressions

Supported expressions:
- default - Neutral expression
- smile - Happy/positive expression
- sad - Sad/negative expression
- surprised - Surprised expression
- angry - Angry expression

Visemes

The component supports standard visemes A-H and X for lip-sync:
- A - Bilabial sounds (P, B, M)
- B - Velar sounds (K, G)
- C - Vowel I
- D - Vowel A
- E - Vowel O
- F - Vowel U
- G - Fricative sounds (F, V)
- H - Dental sounds (TH, T, D)
- X - Silence

Example Backend Integration

`js
// Express + Socket.IO server example
const express = require('express');
const { Server } = require('socket.io');
const http = require('http');

const app = express();
const server = http.createServer(app);
const io = new Server(server);

io.on('connection', (socket) => {
socket.on('chat', async (data) => {
const { audio, sampleRate } = data;

// Process audio, generate response
const response = await processAudio(audio, sampleRate);

// Send chunk with visemes
socket.emit('audio-chunk', {
text: response.text,
audio: response.audioBase64,
visemes: response.visemes,
animation: response.animation,
facialExpression: response.facialExpression
});
});
});

server.listen(4002);
`

Development

`bash


Clone the repository

git clone https://github.com/marcosremar/cabecao-npm.git
cd cabecao-npm

Install dependencies

npm install --legacy-peer-deps

Build the package

npm run build

Test locally

npm link
`

Contributing

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add some amazing feature')
4. Push to the branch (git push origin feature/amazing-feature`)
5. Open a Pull Request

License

MIT © Marcos Remar

Changelog

$3

- ✨ Added WebSocket support for real-time communication
- 🎯 Configurable gaze direction for better eye contact
- 🎭 Dynamic facial expressions based on content
- 📦 Streaming audio chunks for improved performance
- 🔧 Enhanced configuration options

$3

- 🎉 Initial release
- 🎭 Basic 3D avatar with lip-sync
- 🎤 Voice chat capabilities
- 📱 Responsive design