Support React Native

[hans00]

Make it support React Native.

And I made a example for it.
https://github.com/hans00/react-native-transformers-example

Guide to get it start

TODO:

• Check models are works fine
• Research more efficiently image processing
• Merge v3
• Check everything are works fine on v3

[xenova]

Woah nice! There are a lot of people who are interested in doing stuff like this! Looking forward to reviewing this when you're ready! cc @pcuenca

[jhen0409]

I'm investigating a performance issue with Whisper tiny.en, and it looks like the performance is not as expected in the example.

I quickly add log for onnxruntime-react-native/lib/backend.ts, this is the result:

# Encode result
 LOG  ONNX session run finished 767 ms

# Decode result
 LOG  ONNX session run finished 4518 ms
# ... 4 runs ...
 LOG  ONNX session run finished 4856 ms
 LOG  Time: 40016
 LOG  Result: {"text": " (buzzing)"}

Click to expand

const t0 = performance.now()
// this.#inferenceSession === NativeModules.Onnxruntime
const results: Binding.ReturnType = await this.#inferenceSession.run(this.#key, input, outputNames, options);
const output = this.decodeReturnType(results);
console.log('ONNX session run finished', performance.now() - t0)

Also see logs of native part (added log to OnnxruntimeModule.java), it is significantly less than the time taken by JS part:

# Encode result
09:47:59.203 ONNXRUNTIME_RN run() is finished, time: 273 ms

# Decode result
09:48:00.280 ONNXRUNTIME_RN run() is finished, time: 339 ms
# ... 4 runs ...
09:48:23.807 ONNXRUNTIME_RN run() is finished, time: 541 ms

I think this problem may come from blocking caused by native module passing too large data. I think request/help onnxruntime for migrate to JSI module may solve this problem.

EDIT: It seems the logger have some bugs leads me to think that the problem is from the native bridge. I add timeout await to the decodeReturnType call and found the issue is from this function.

[axe-me]

@TowhidKashem same as the fugood one, merge branch.
Or you can just install it : npm i @axe-dev/transformers

[hans00]

It seems it might have come from the native-universal-fs lib you're using, I tried replacing those instances with expo-file-system directly and using patch-package to update @fugood/transformers but the errors remain.
Any idea what could be the fix?

I had same issue in my recent trials...

• Not working directly with Expo mybigday/native-universal-fs#1

This should fixed.
But currently uses webpack bundle to solve package alias, so that could not install from source.

[an-upfeat]

It seems it might have come from the native-universal-fs lib you're using, I tried replacing those instances with expo-file-system directly and using patch-package to update @fugood/transformers but the errors remain.
Any idea what could be the fix?

I had same issue in my recent trials...

• Not working directly with Expo mybigday/native-universal-fs#1

This should fixed. But currently uses webpack bundle to solve package alias, so that could not install from source.

the 3.6.1-beta.1 release works for my app, thanks!

[rogueturnip]

Just curious, is this work integrated in the latest release (I see the PR is still open) or is this PR left hanging?

[bernardro]

@jhen0409 @kungfooman could one of you kindly merge this, sorry for the ping

[llop00]

@TowhidKashem I ran into same issue. I found it was because the metro reads the exports field of the package then loaded the browser version build. You can use my fork for now: https://www.npmjs.com/package/@axe-dev/transformers (I couldn't guarantee that I won't add my changes in there) or you can figure out some ways to load the build that bundled the universal fs package.

Does any of you could help on how they got to make it work on Expo? Im a newbie on mobile development and it seems to be impossible but saw you guys achieved it hahaha

[hans00]

@TowhidKashem I ran into same issue. I found it was because the metro reads the exports field of the package then loaded the browser version build. You can use my fork for now: https://www.npmjs.com/package/@axe-dev/transformers (I couldn't guarantee that I won't add my changes in there) or you can figure out some ways to load the build that bundled the universal fs package.

Does any of you could help on how they got to make it work on Expo? Im a newbie on mobile development and it seems to be impossible but saw you guys achieved it hahaha

As far as I know, this tutorial may also works with Expo.
https://github.com/hans00/react-native-transformers-example/blob/main/DEVELOPMENT.md

And you need run local dev.