TFJS + own custom model, TensorCamera silent failure, onReady never fires
Has anyone here had success using TFJS + React Native on NEW projects?
I am trying to use TFJS because react-native-fast-tflite
package is not able to load VertexAI models 🤷🏻♂️. So I tried following example TFJS + ReactNative repo, however, it is severely outdated. I have installed these versions:
"@tensorflow/tfjs": "^4.22.0",
"@tensorflow/tfjs-backend-cpu": "^4.22.0",
"@tensorflow/tfjs-react-native": "^1.0.0",
"expo-camera": "~15.0.16",
"expo-gl": "~14.0.2",
"expo": "^51.0.0",
"react-native": "0.74.5",
So far, I am 1) having to exclude using React Native Vision Camera as the TensorCamera is specifically looking for expo-camera...ok no biggie. 2) I am running into an issue after I open the camera:
The typeerror has no stacktrace...and it's just a warning. Here is my code:
const [model, setModel] = useState<LayersModel | null>(null);
const [isModelReady, setIsModelReady] = useState(false);
const loadModel: LoadModelType = async (setModel, setIsModelReady) => {
try {
await ready();
const modelJson = require('../../assets/tfjs/model.json');
const modelWeights1 = require('../../assets/tfjs/1of3.bin');
const modelWeights2 = require('../../assets/tfjs/2of3.bin');
const modelWeights3 = require('../../assets/tfjs/3of3.bin');
const bundle = bundleResourceIO(modelJson, [
modelWeights1,
modelWeights2,
modelWeights3,
]);
const modelConfig = await loadGraphModel(bundle);
setModel(modelConfig);
setIsModelReady(true);
} catch (e) {
console.error((e as Error).message)
}
};
useEffect(
function initTFJS() {
if (hasPermission) {
(async () => {
// const isGPUSupported = await TFGPUManager.enableGPU();
// setGpuSupported(isGPUSupported);
console.log('load model');
await loadModel(setModel, setIsModelReady);
})();
}
},
[hasPermission]
return isModelReady && <TensorCamera
autorender
cameraTextureHeight={textureDims.height}
cameraTextureWidth={textureDims.width}
onReady={handleCameraStream} // never fires onReady
resizeDepth={3}
resizeHeight={TENSOR_HEIGHT}
resizeWidth={TENSOR_WIDTH}
style={{ flex: 1 }}
useCustomShadersToResize={false}
/>
my model
appears to be loaded correctly, and isModelReady
is true
, so the camera is rendered. However, the onReady={handleCameraStream}
callback never seems to fire, so while the camera appears active, something fails.
Any ideas? Is TFJS+reactnative a good idea with modern RN versions?