Vision plugin for ElizaOS - provides camera integration and visual awareness
npm install @elizaos/plugin-vision-rootA powerful visual perception plugin for ElizaOS that provides agents with
real-time camera integration and scene analysis capabilities. This plugin
enables agents to "see" their environment, describe scenes, detect people and
objects, and make decisions based on visual input.
This plugin is implemented in multiple languages with complete feature parity:
| Language | Directory | Status |
| ---------- | --------- | ------------------- |
| TypeScript | src/ | ā
Production Ready |
| Python | python/ | ā
Production Ready |
| Rust | rust/ | š In Development |
The TypeScript implementation is the primary and most feature-complete version.
- ā
Camera detection and connection (platform-specific)
- ā
Real-time frame capture and processing
- ā
Scene description using Vision Language Models (VLM)
- ā
Motion-based object detection
- ā
Basic person detection with pose estimation
- ā
Configurable pixel change threshold
- ā
Image capture action with base64 attachments
- ā
Non-dynamic vision provider (always active)
- ā
Integration with autonomy plugin (kill switch)
- ā
Enhanced object detection with COCO-like classification
- ā
Advanced pose detection with keypoint estimation
- ā
Improved person detection and tracking
- ā
Object classification (person, monitor, chair, keyboard, furniture, etc.)
- ā
Configurable computer vision models
- ā
Fallback to motion detection when CV is disabled
- ā
Real-time object tracking with IDs
- ā
Face detection and recognition
- ā
Screen capture and OCR integration
- ā
Entity tracking with persistent IDs
- ā
Multi-display support
- ā
Circuit breaker pattern for error resilience
- ā
Florence2 model integration for advanced scene understanding
- ā
Worker-based processing for high-FPS operations
- š WebAssembly (WASM) integration for browser compatibility
- š Gesture recognition
- š Emotion detection
- š Advanced scene understanding and spatial relationships
``bash`
npm install @elizaos/plugin-visionor
cd plugins/plugin-vision
bun install
bun run build
`bash`
cd plugins/plugin-vision/python
pip install -e .
`bash`
cd plugins/plugin-vision/rust
cargo build --release
The plugin requires platform-specific camera tools:
- macOS: brew install imagesnapsudo apt-get install fswebcam
- Linux:
- Windows: Install ffmpeg and add to PATH
`envCamera selection (partial name match, case-insensitive)
CAMERA_NAME=obsbot
$3
`json
{
"name": "VisionAgent",
"plugins": ["@elizaos/plugin-vision"],
"settings": {
"CAMERA_NAME": "obsbot",
"PIXEL_CHANGE_THRESHOLD": "30",
"ENABLE_OBJECT_DETECTION": "true",
"ENABLE_POSE_DETECTION": "true"
}
}
`Actions
$3
Analyzes the current visual scene and provides a detailed description.
Similes:
ANALYZE_SCENE, WHAT_DO_YOU_SEE, VISION_CHECK, LOOK_AROUNDExample:
`
User: "What do you see?"
Agent: "Looking through the camera, I see a home office setup with a person sitting at a desk. There are 2 monitors, a keyboard, and various desk accessories. I detected 5 objects total: 1 person, 2 monitors, 1 keyboard, and 1 chair."
`$3
Captures the current frame and returns it as a base64 image attachment.
Similes:
TAKE_PHOTO, SCREENSHOT, CAPTURE_FRAME, TAKE_PICTUREExample:
`
User: "Take a photo"
Agent: "I've captured an image from the camera." [Image attached]
`$3
Changes the vision mode (OFF, CAMERA, SCREEN, or BOTH).
Similes:
CHANGE_VISION_MODE, SET_VISION, TOGGLE_VISION$3
Assigns a name to a detected entity for tracking.
Similes:
LABEL_ENTITY, NAME_OBJECT, IDENTIFY_ENTITY$3
Identifies a person using face recognition (requires face recognition to be enabled).
Similes:
RECOGNIZE_PERSON, IDENTIFY_FACE$3
Starts tracking an entity with a persistent ID.
Similes:
START_TRACKING, FOLLOW_ENTITY$3
Stops the autonomous agent loop (useful for debugging with autonomy plugin).
Similes:
STOP_AUTONOMOUS, HALT_AUTONOMOUS, KILL_AUTO_LOOPVision Provider
The vision provider is non-dynamic (always active) and provides:
- Current scene description
- Camera connection status
- Detected objects count and types
- Detected people count with poses
- Scene change percentage
- Time since last update
$3
`typescript
{
visionAvailable: boolean,
sceneDescription: string,
cameraStatus: string,
cameraId?: string,
peopleCount?: number,
objectCount?: number,
sceneAge?: number,
lastChange?: number
}
`Detection Modes
$3
- Lightweight and fast
- Detects movement between frames
- Groups motion blocks into objects
- Basic size-based classification
$3
Enable with
ENABLE_OBJECT_DETECTION=true and/or ENABLE_POSE_DETECTION=true- Object Detection: Enhanced object recognition with COCO-like classes
- Pose Detection: 17-keypoint pose estimation
- Better Classification: Distinguishes between person, monitor, chair,
keyboard, etc.
- Higher Accuracy: Edge detection and color variance analysis
Integration with Autonomy
- Continuous environmental monitoring
- Autonomous responses to visual changes
- Visual memory persistence
- Scene-based decision making
Example autonomous behavior:
`typescript
// Agent autonomously monitors environment
"I notice someone just entered the room.";
"The lighting has changed significantly.";
"A new object has appeared on the desk.";
`Performance Considerations
- Frame processing runs every 100ms by default
- VLM is only called when pixel change exceeds threshold
- Motion detection uses 64x64 pixel blocks with 50% overlap
- Advanced CV models add ~50-100ms processing time per frame
- Memory usage increases with resolution (1280x720 recommended)
Security & Privacy
- Camera access requires system permissions
- No images are stored permanently by default
- All processing happens locally
- Base64 images in messages are ephemeral
- Consider privacy implications in your implementation
Architecture
`
plugin-vision/
āāā README.md # This file
āāā package.json # TypeScript package config
āāā src/ # TypeScript implementation (primary)
ā āāā index.ts # Plugin entry point
ā āāā service.ts # Vision service
ā āāā provider.ts # Vision provider
ā āāā action.ts # All actions
ā āāā entity-tracker.ts # Entity tracking
ā āāā screen-capture.ts # Screen capture
ā āāā ocr-service.ts # OCR service
ā āāā face-recognition.ts # Face recognition
ā āāā florence2-model.ts # Florence2 model integration
ā āāā vision-worker-manager.ts # Worker management
ā āāā tests/ # E2E tests
āāā python/ # Python implementation
ā āāā pyproject.toml
ā āāā elizaos_vision/
ā āāā __init__.py # Plugin entry point
ā āāā service.py # Vision service
ā āāā provider.py # Vision provider
ā āāā actions.py # All actions
ā āāā ...
āāā rust/ # Rust implementation (in development)
āāā Cargo.toml
āāā src/
āāā ...
`Development
$3
`bash
TypeScript - Run E2E tests
cd plugins/plugin-vision
npx vitestTypeScript - Run local E2E tests
bun run test:e2e:localPython - Run tests
cd plugins/plugin-vision/python
pytestRust - Run tests
cd plugins/plugin-vision/rust
cargo test
``- Service initialization
- Camera detection and connection
- Scene description generation
- Object and person detection
- Image capture
- Provider integration
- Autonomy integration
1. Ensure camera tools are installed (imagesnap/fswebcam/ffmpeg)
2. Check camera permissions in system settings
3. Try without CAMERA_NAME to use default camera
4. Verify camera is not in use by another application
1. Ensure good lighting conditions
2. Adjust PIXEL_CHANGE_THRESHOLD (lower = more sensitive)
3. Enable advanced CV with ENABLE_OBJECT_DETECTION=true
4. Check camera resolution (higher is better for detection)
1. Increase frame processing interval in code
2. Disable advanced CV features if not needed
3. Reduce camera resolution
4. Increase pixel change threshold
- TensorFlow.js WASM backend
- Browser-compatible vision processing
- Real-time object tracking
- Face detection and recognition
- Gesture recognition
- Emotion detection
- Scene understanding
- Spatial relationship mapping
- Multi-camera support
Contributions are welcome! Please see the main ElizaOS repository for
contribution guidelines.
MIT
For issues and feature requests, please use the GitHub issue tracker.