PyTorch core library for React Native


The PyTorch core library for React Native is part of the PlayTorch project. Please follow the instructions provided on the PlayTorch website as outlined below!

🎉 Getting started with PlayTorch

Follow the Getting Started guide. Use the PlayTorch app to get started with no development environment required, or set up a local development environment to build a standalone app by following the Manual Environment Setup guide. We also have a few tutorials for you to keep going after getting started:

* Image Classification
* Question Answering
* Prepare Custom Model

📖 Documentation

The full documentation for PlayTorch can be found on our website.

Install

``
npm install react-native-playtorch
`

Modify android/app/build.gradle:
`
android {
....
packagingOptions {
// doNotStrip "**/libc++_shared.so"
pickFirst '*/.so'
}
...
}
`
Modify android/gradle.properties:
`
org.gradle.jvmargs=-Xmx4g
`
Modify metro.config.js:
`
const defaultAssetExts = require('metro-config/src/defaults/defaults')
.assetExts;

module.exports = {
resolver: {
assetExts: [...defaultAssetExts, 'ptl'],
},
};
`

Patch to fix __emutls_get_address crash on Android


If RN0.71+ and run crash on Android java.lang.UnsatisfiedLinkError: dlopen failed: cannot locate symbol "__emutls_get_address" referenced by "/data/app/~~Bu6UWdRieDpDrpvvyvNNVQ==/com.foo.bar-w8nusksLnLfSCCsWG3cEkg==/lib/arm64/libfolly_runtime.so", you need (e.g. on Linux)
`
cd tools/android-sdk/ndk

mv ./21.4.7075529/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/aarch64-linux-android/libc++_shared.so ./21.4.7075529/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/aarch64-linux-android/origin_libc++_shared.so
cp ./23.1.7779620/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/aarch64-linux-android/libc++_shared.so ./21.4.7075529/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/aarch64-linux-android/

mv ./21.4.7075529/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/arm-linux-androideabi/libc++_shared.so ./21.4.7075529/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/arm-linux-androideabi/origin_libc++_shared.so
cp ./23.1.7779620/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/arm-linux-androideabi/libc++_shared.so ./21.4.7075529/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/arm-linux-androideabi/

mv ./21.4.7075529/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/x86_64-linux-android/libc++_shared.so ./21.4.7075529/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/x86_64-linux-android/origin_libc++_shared.so
cp ./23.1.7779620/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/x86_64-linux-android/libc++_shared.so ./21.4.7075529/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/x86_64-linux-android/

mv ./21.4.7075529/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/i686-linux-android/libc++_shared.so ./21.4.7075529/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/i686-linux-android/origin_libc++_shared.so
cp ./23.1.7779620/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/i686-linux-android/libc++_shared.so ./21.4.7075529/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/i686-linux-android/
`
The crash reason is, even the libc++_shared.so in prefab
`
readelf -s ~/.gradle/caches/transforms-3/9915f55612e7d9d2d8676faa1872c696/transformed/jetified-react-android-0.71.6-debug/jni/arm64-v8a/libc++_shared.so | grep __emutls_get_address
374: 00000000000ec5bc 448 FUNC WEAK DEFAULT 16 __emutls_get_address
`
is WEAK not LOCAL, but with pickFirst '*/.so', the libc++_shared.so in .apk will be picked from node_modules/react-native-playtorch/android/build/intermediates/library_jni/debug/jni/arm64-v8a/libc++_shared.so, and
`
readelf -s node_modules/react-native-playtorch/android/build/intermediates/library_jni/debug/jni/arm64-v8a/libc++_shared.so | grep __emutls_get_address
3885: 00000000000b60a0 344 FUNC LOCAL DEFAULT 11 __emutls_get_address
`
is LOCAL not WEAK.

If enable doNotStrip "**/libc++_shared.so" then extract the libc++_shared.so from .apk, use readelf you will also find it's LOCAL not WEAK.

The libfolly_runtime.so will call __emutls_get_address, if it's LOCAL, then run into crash.

The __emutls_get_address in libc++_shared.so of NDK21.4.7075529 is LOCAL, and it's WEAK for NDK23.1.7779620.

For now, react-native-playtorch only can be compiled in NDK21.4.7075529.

So comes the patch above.

Example Usage

`javascript
// Import dependencies
import * as React from 'react';
import { StyleSheet, Text, View } from 'react-native';
import {
Camera,
Image,
media,
MobileModel,
Module,
Tensor,
torch,
torchvision,
} from 'react-native-playtorch';
import { useSafeAreaInsets } from 'react-native-safe-area-context';

// Alias for torchvision transforms
const T = torchvision.transforms;

// URL to the image classification model that is used in this example
const MODEL_URL =
'https://github.com/facebookresearch/playtorch/releases/download/v0.1.0/mobilenet_v3_small.ptl';

// URL to the ImageNetClasses JSON file, which is used below to map the
// processed model result to a class label
const IMAGENET_CLASSES_URL =
'https://github.com/facebookresearch/playtorch/releases/download/v0.1.0/ImageNetClasses.json';

// Variable to hold a reference to the loaded ML model
let model: Module | null = null;

// Variable to hold a reference to the ImageNet classes
let imageNetClasses: string[] | null = null;

// App function to render a camera and a text
export default function App() {
// Safe area insets to compensate for notches and bottom bars
const insets = useSafeAreaInsets();
// Create a React state to store the top class returned from the
// classifyImage function
const [topClass, setTopClass] = React.useState(
"Press capture button to classify what's in the camera view!",
);

// Function to handle images whenever the user presses the capture button
async function handleImage(image: Image) {
// Get image width and height
const width = image.getWidth();
const height = image.getHeight();

// Convert image to blob, which is a byte representation of the image
// in the format height (H), width (W), and channels (C), or HWC for short
const blob = media.toBlob(image);

// Get a tensor from image the blob and also define in what format
// the image blob is.
let tensor = torch.fromBlob(blob, [height, width, 3]);

// Rearrange the tensor shape to be [CHW]
tensor = tensor.permute([2, 0, 1]);

// Divide the tensor values by 255 to get values between [0, 1]
tensor = tensor.div(255);

// Crop the image in the center to be a squared image
const centerCrop = T.centerCrop(Math.min(width, height));
tensor = centerCrop(tensor);

// Resize the image tensor to 3 x 224 x 224
const resize = T.resize(224);
tensor = resize(tensor);

// Normalize the tensor image with mean and standard deviation
const normalize = T.normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]);
tensor = normalize(tensor);

// Unsqueeze adds 1 leading dimension to the tensor
tensor = tensor.unsqueeze(0);

// If the model has not been loaded already, it will be downloaded from
// the URL and then loaded into memory.
if (model === null) {
const filePath = await MobileModel.download(MODEL_URL);
model = await torch.jit._loadForMobile(filePath);
}

// Run the ML inference with the pre-processed image tensor
const output = await model.forward(tensor);

// Get the index of the value with the highest probability
const maxIdx = output.argmax().item();

if (imageNetClasses === null) {
const response = await fetch(IMAGENET_CLASSES_URL);
imageNetClasses = (await response.json()) as string[];
}

// Resolve the most likely class label and return it
const result = imageNetClasses[maxIdx];

// Set result as top class label state
setTopClass(result);

// Release the image from memory
image.release();
}

return (

{/ Render camara and make it parent filling /}
style={[StyleSheet.absoluteFill, { bottom: insets.bottom }]}
// Add handle image callback on the camera component
onCapture={handleImage}
/>
{/ Label container with custom render style and a text /}

{/ Change the text to render the top class label /}
{topClass}


);
}

// Custom render style for label container
const styles = StyleSheet.create({
labelContainer: {
padding: 20,
margin: 20,
marginTop: 40,
borderRadius: 10,
backgroundColor: 'white',
},
});
``

👏 How to Contribute

The main purpose of this repository is to continue evolving PlayTorch. We want to make contributing to this project as easy and transparent as possible, and we are grateful to the community for contributing bug fixes and improvements. Read below to learn how you can take part in improving PlayTorch.

$3

Meta has adopted a Code of Conduct that we expect project participants to adhere to.
Please read the [full text][code] so that you can understand what actions will and will not be tolerated.

[code]: https://opensource.fb.com/code-of-conduct/

$3

Read our [Contributing Guide][contribute] to learn about our development process, how to propose bugfixes and improvements, and how to build and test your changes to PlayTorch.

[contribute]: CONTRIBUTING.md

License

PlayTorch is MIT licensed, as found in the [LICENSE][license] file.

[license]: LICENSE.md