React Native module for image classification and description using Vision framework (iOS) and ML Kit (Android)
npm install @dariyd/react-native-image-description

React Native module for image classification and description using native ML capabilities:
- iOS: Vision framework's VNClassifyImageRequest for image classification
- Optional: merge results with Google ML Kit Image Labeling (400+ labels)
- Android: ML Kit Image Labeling + GenAI Image Description API
✨ Image Classification
- iOS and Android support
- iOS: Apple Vision + optional Google ML Kit merge (recommended ON for better results)
- Android: ML Kit Image Labeling base model (400+ labels)
- Configurable thresholds and result limits
- On-device, fast inference
✨ Image Description (Android Only)
- Natural language descriptions of images
- On-device GenAI model
- Model download management with progress tracking
✨ Modern Architecture
- Full React Native new architecture support (iOS TurboModule)
- TypeScript with complete type definitions
- Promise-based async API
- Backward compatible with old architecture
``bash`
npm install @dariyd/react-native-image-descriptionor
yarn add @dariyd/react-native-image-description
`bash`
cd ios && pod install
Requirements:
- iOS 15.0 or higher
- Xcode 14 or higher
The module will be automatically linked. No additional setup required.
Requirements:
- Android SDK 26 or higher
- Google Play Services
`typescript
import { classifyImage } from '@dariyd/react-native-image-description';
// Classify an image
const result = await classifyImage('file:///path/to/image.jpg', {
minimumConfidence: 0.5,
maxResults: 10
});
if (result.success) {
result.labels.forEach(label => {
console.log(${label.identifier}: ${(label.confidence * 100).toFixed(1)}%);
});
}
// Example output:
// dog: 95.2%
// animal: 92.8%
// pet: 89.3%
// mammal: 87.1%
`
By default on iOS, the module merges labels from Apple Vision and Google ML Kit
Image Labeling to improve coverage. You can toggle ML Kit with iosUseMlKit:
`typescript
// Vision-only (disable ML Kit on iOS)
const resVisionOnly = await classifyImage(uri, {
minimumConfidence: 0.5,
iosUseMlKit: false,
});
// Vision + ML Kit (default)
const resMerged = await classifyImage(uri, {
minimumConfidence: 0.5,
iosUseMlKit: true,
});
`
Runtime logs clearly indicate the source:
- Vision labels are logged as Vision label: …ML Kit label: …
- ML Kit labels are logged as Vision
- Final summary shows vs Vision and ML Kit.
| iOS (Vision + ML Kit) | Android (ML Kit) |
| --- | --- |
|
|
|
`typescript
import {
describeImage,
checkDescriptionModelStatus,
downloadDescriptionModel
} from '@dariyd/react-native-image-description';
// Check if model is available
const status = await checkDescriptionModelStatus();
console.log('Model status:', status); // 'available', 'downloadable', etc.
// Download model if needed
if (status === 'downloadable') {
const success = await downloadDescriptionModel((progress) => {
console.log(Download progress: ${(progress * 100).toFixed(0)}%);
});
if (success) {
console.log('Model downloaded successfully!');
}
}
// Generate description
const result = await describeImage('file:///path/to/image.jpg');
if (result.success) {
console.log('Description:', result.description);
// Example: "A golden retriever playing with a ball in a park"
}
`
`typescript
import { isAvailable } from '@dariyd/react-native-image-description';
const available = await isAvailable();
if (available) {
console.log('Image description module is ready!');
}
`
Classify an image and return labels with confidence scores.
Parameters:
- imageUri (string): Local file path or file:// URI to the imageoptions
- (object, optional):minimumPrecision
- (number): 0.0-1.0, default 0.1 (iOS only)recallThreshold
- (number): 0.0-1.0, default 0.8 (iOS only)minimumConfidence
- (number): 0.0-1.0, filter results by confidenceconfidenceThreshold
- (number): 0.0-1.0 (Android only, default 0.5)maxResults
- (number): Limit number of resultsiosUseMlKit
- (boolean): iOS only. Merge Google ML Kit labels with Visiontrue
results. Defaults to .
Returns: Promise`typescript`
{
success: boolean;
labels: Array<{
identifier: string;
confidence: number; // 0.0-1.0
index?: number;
}>;
error?: string;
}
Generate a natural language description of an image.
Parameters:
- imageUri (string): Local file path or file:// URI to the imageoptions
- (object, optional): Reserved for future use
Returns: Promise`typescript`
{
success: boolean;
description: string;
error?: string;
modelStatus?: 'available' | 'downloading' | 'not_available';
}
Note: On iOS, this always returns an error indicating the feature is not available.
Check the status of the GenAI description model.
Returns: Promise
Possible values:
- 'available' - Model is downloaded and ready'downloadable'
- - Model needs to be downloaded'downloading'
- - Model is currently downloading'not_available'
- - Model cannot be used'not_supported'
- - Platform doesn't support this feature (iOS)
Download the GenAI description model.
Parameters:
- onProgress (function, optional): Callback for download progress (progress: number) => voidprogress
- : 0.0-1.0
Returns: Promise - true if download succeeds
Check if the module is available on the current platform.
Returns: Promise - true if available
Classification:
- ✅ High-accuracy classification using Vision framework
- ✅ Taxonomy labels (e.g., "dog", "animal", "pet", "mammal")
- ✅ Configurable precision/recall thresholds
- ✅ Confidence scores for all labels
- ✅ iOS 15+ required
Description:
- ❌ Natural language description not available
- 💡 Use classifyImage() for classification labels instead
- 💡 For descriptions, consider cloud solutions (OpenAI Vision API, etc.)
Classification:
- ✅ Fast on-device labeling with ML Kit
- ✅ 400+ entity categories
- ✅ Configurable confidence threshold
- ✅ No model download required
Description (GenAI Image Description):
- ✅ On-device GenAI descriptions via ML Kit GenAI Image Description API
- ⚠️ Only available on a limited set of supported devices (AICore/Gemini Nano required). See device support list: https://developers.google.com/ml-kit/genai#feature-device
- ⚠️ May initially report downloadable/downloading and require time on Wi‑Fi to become available
- ⚠️ API is in Beta and subject to change
- ⚠️ This feature has not been fully validated by this library across the device matrix; treat as experimental
The Vision framework provides sophisticated filtering options:
High-Recall Filter (default):
`typescript`
classifyImage(imageUri, {
minimumPrecision: 0.1,
recallThreshold: 0.8
});
- Returns more labels (broader range)
- May include some false positives
- Good for discovery and exploration
High-Precision Filter:
`typescript`
classifyImage(imageUri, {
minimumPrecision: 0.9,
recallThreshold: 0.01
});
- Returns fewer labels (more conservative)
- Higher accuracy, fewer false positives
- Good for critical applications
`typescript`
classifyImage(imageUri, {
confidenceThreshold: 0.7 // Only labels with 70%+ confidence
});
The example/ directory contains a full React Native app demonstrating:
- Image picker integration
- Classification with confidence visualization
- Description generation (Android)
- Model download management
- Platform-specific UI
To run the example:
`bashInstall dependencies
yarn bootstrap
Performance Tips
1. Image Size: Resize large images before processing for faster results
2. Model Download: On Android, download the description model during app setup
3. Caching: Cache classification results for frequently used images
4. Batch Processing: Process multiple images sequentially, not concurrently
Troubleshooting
$3
"iOS 15.0 or later is required"
- Update your iOS deployment target to 15.0 in Xcode
- Update
platform :ios in your PodfileClassification returns empty results
- Check that the image file exists and is readable
- Verify the URI format (use
file:// prefix)
- Try lowering minimumConfidence threshold$3
"Model needs to be downloaded"
- Call
downloadDescriptionModel() before using describeImage()
- Ensure device has internet connection for initial download
- Check available storage space (~50MB required)GenAI Image Description returns
not_available
- Feature requires a supported device with AICore/Gemini Nano. Verify the device is listed here: https://developers.google.com/ml-kit/genai#feature-device
- Ensure Google Play services/AICore are up to date; give the device time to finish initialization on Wi‑Fi, then reboot
- Not supported on emulators or devices with unlocked bootloaderOut of memory errors
- Reduce image resolution before processing
- Process images one at a time
- Ensure proper cleanup (module handles this automatically)
ML Kit not available
- Verify Google Play Services is installed and up to date
- Check minimum SDK version is 26+
- Ensure app has internet permission in AndroidManifest.xml
React Native New Architecture
This module fully supports the React Native new architecture:
$3
✅ Native TurboModule implementation
- Automatically detected when RCT_NEW_ARCH_ENABLED=1
- Seamless fallback to Bridge mode on old architecture
- Full CodeGen integration$3
✅ Bridge mode with full compatibility
- Works seamlessly with newArchEnabled=true (new arch apps)
- Works with newArchEnabled=false (old arch apps)
- Uses React Native's interop layer for maximum compatibilityComparison with Other Solutions
| Feature | This Library | react-native-mlkit | react-native-text-detector |
|--------------------|---------------------|-------------------|---------------------------|
| iOS Support | ✅ Vision API | ❌ | ✅ |
| Android Support | ✅ ML Kit v2 | ✅ ML Kit | ✅ |
| Image Classification | ✅ | ✅ | ❌ |
| Image Description | ✅ Android | ❌ | ❌ |
| New Architecture | ✅ iOS TurboModule | ❌ | ❌ |
| TypeScript | ✅ | ⚠️ Partial | ❌ |
| Active Maintenance | ✅ | ⚠️ | ❌ |
Requirements
- React Native >= 0.77.3
- iOS 15.0+
- Android SDK 26+
- Xcode 14+ (for iOS development)
- Android Studio (for Android development)
Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
1. Fork the repository
2. Create your feature branch (
git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature`)Apache-2.0 © dariyd
- iOS implementation uses Apple's Vision framework
- Android implementation uses Google ML Kit
- Inspired by @dariyd/react-native-text-recognition
- @dariyd/react-native-text-recognition - Text recognition using Vision and ML Kit
- react-native-vision-camera - Camera library for React Native
- react-native-image-picker - Image picker for React Native
If you find this project useful, please consider:
- ⭐ Starring the repository
- 🐛 Reporting issues
- 📖 Improving documentation
- 🔧 Contributing code
See CHANGELOG.md for a detailed list of changes.
- [ ] Custom Core ML model support (iOS)
- [ ] TensorFlow Lite model support (Android)
- [ ] Image similarity search
- [ ] Batch processing API
- [ ] Cloud-based description for iOS
- [ ] Video frame classification