TFJS app (using webcam) to Android app --> webcam doesn't work?

[Wout Dispa]

So, I have a TFJS web app which uses my webcam and does pose estimation. I want to create a mobile Android app, and the easiest way I imagined was to just my current web app into a WebView jacket if you will (which I know will work for Android, and not for the Apple ecosystem but that's not my concern for now), load the browser window in the app and done. 

However, the webcam doesn't work when using WebView (I think it might be a security issue?) and possible solutions on StackOverflow etc. seem very vague in explaining what's going on and what needs to happen...

Also, I've been recommended by a few folks to check out PWAs in case WebView can't solve it... thoughts? Solutions? Many thanks!

[Na Li]

Hi Yannick and Matt,

What's your thoughts on the questions above? Does tfjs-react-native work with a native module (not an entire app)? Yannick, do you have any insights on WebView restrictions?

Na

--
You received this message because you are subscribed to the Google Groups "TensorFlow.js Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tfjs+uns...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/d61d8e2b-6fef-41df-a364-42c28d7a9ef5n%40tensorflow.org.

[Jason Mayes]

Aside: PWAs on Android are very well supported I believe, though again Apple is yet to catch up with this one - but they act and behave like native apps. The only downside of PWA is that you dont install  them from the "app store" - you just go to a website and it asks to install the app on phone - so slightly different approach to discovery - but if that is not an issue (personally i think it is more frictionless!) then that may work for you. 

If you are using a PWA approach I recommend using some of the PWA frameworks to make your life easier getting all the scaffolding setup (web workers and configs etc) to save yourself some time.  I think the one someone at Google mentioned to me was Workbox - https://developers.google.com/web/tools/workbox

I will let the other folk chime in on native ports though as that is not my area of expertise (my understanding was same as yours that you can just wrap in a webview - though that may be CPU only - which is why we have the react native specific build that pushes perf further for JS side of things eg see this post: https://blog.tensorflow.org/2020/02/tensorflowjs-for-react-native-is-here.html

Jason

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/CALZ9_DAcV%2BUhNOht6_AwwmyLvrjRmZdiqNL4z6gT87ZVvU%3DGug%40mail.gmail.com.

--

•  Jason Mayes   •  Senior Developer Advocate, TensorFlow.js   •  Google   •   Follow me or get inspired with #MadeWithTFJS demos

This email may be confidential and privileged. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.

[Yannick Assogba]

I'm afraid I don't know too much about android webviews. I would suggest seeing if there are any permissions you can enable that might give webcam access to the webview.

tfjs-react-native skips the webview completely so may or may not be a good fit depending on what else is going on in your app.

[Wout Dispa]

Thank you Na for kicking this off.

@Jason the fact that it won't be in an app store = 30% future revenue saved :) ... not sure how much of a need I'll have in the future to reconsider that decision, but for now in MVP stage it's a non-issue. Thank you for the Workbox advice, definitely checking that out!

@Yannick giving it permissions doesn't seem to work... 

<!-- needed -->
<uses-permission android:name="android.permission.INTERNET"/>
<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.permission.RECORD_AUDIO"/>

<!-- not sure if needed -->
<user-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />
<uses-permission android:name="android.permission.SYSTEM_ALERT_WINDOW"/>
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />

These I've already got in my manifest and it does nothing. I read somewhere that since build 23 or whatnot of Android I might also have to give permission in runtime, but... ¯\_(ツ)_/¯

As I'm not familiar with React Native either, I will make a few steps towards a PWA and see how things work out. Wish there was an easier solution though. These are the best "solutions" I found so in regards to WebView and everyone seems to suggest a different workaround, without too much of an explanation as to why

 

https://stackoverflow.com/questions/40659198/how-to-access-the-camera-from-within-a-webview

https://stackoverflow.com/questions/63695705/enable-input-file-camera-on-android-webview

https://stackoverflow.com/questions/47872078/how-to-load-an-url-inside-a-webview-using-android-kotlin