PPAPI outside the sandbox?

[ric...@batemansr.us]

Hi,

I've gotten conflicting reports on this, hoping someone here can tell me.  Is there still a way to use PPAPI outside of the NaCl sandbox?  The reason I ask is that I work on the open source framework FireBreath and we're investigating ways to deal with Chrome's decision to drop NPAPI.  Some of the use cases translate well into NaCl, but unfortunately many of us use plugins currently to provide an interface to proprietary hardware and/or closed-source DLLs, etc that aren't accessible from inside the NaCl sandbox.

My understanding was that PPAPI originally could be used without NaCl and that this is most likely what Flash uses in Chrome; does that capability still exist as an option for those of us for whom none of Chrome's other technologies will solve our issue?

Thanks!

Richard Bateman

FireBreath Development

GradeCam (http://www.gradecam.com)

[Andrey Khalyavin]

Yes, this capability exists but it is developers only because there is no way to install a trusted PPAPI plugin. Trusted PPAPI plugins must be configured with chrome.exe command line flags. You can also look into Native Messaging API: https://plus.google.com/102860501900098846931/posts/KtawbYAQUGX. May be it fits your requirements.

-- Andrey Khalyavin

[Richard Bateman]

We have certainly looked at the native messaging API; however, my understanding is that the native messaging API does not have any performant way to transfer large chunks of binary data (such as a video stream).

There are a great many different things that FireBreath users need this sort of functionality for, but the use that I professional have is with GradeCam. GradeCam is a technology used by a large and rapidly growing number of teachers in the US (and it’s starting to become more common elsewhere as well) that utilizes a webcam and a proprietary image processing system to read bubble answer sheets on the teacher’s computer. It’s a really cool system, and it helps a lot of people.  Since Chrome plans to drop NPAPI support, we’re trying to find the best alternative for providing this functionality.

NaCl is obviously the first choice people think of when considering alternatives to NPAPI, but from looking at the documentation I’m not seeing any way that we can get video from the webcam other than using the HTML5 APIs, but in that case I don’t see a good way to get the data into the NaCl plugin for processing, so that doesn’t seem like it will work.  In addition, those APIs don’t support some of the more common USB document cameras that we need to support (such as ELMO).  Native Messaging would be an option, but I don’t see a way to transfer the data fast enough to display the image in the browser; and can you use native messaging from inside NaCl? The docs I’ve found are unclear about any of that.

Any insight you or anyone can provide would be appreciated; feeling a little bit abandoned by my favorite browser these days, as there don’t seem to be any alternatives to NPAPI that cover most of our use cases :-(

Richard Bateman
FireBreath Development
GradeCam (http://www.gradecam.com)

--
You received this message because you are subscribed to the Google Groups "Native-Client-Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to native-client-di...@googlegroups.com.
To post to this group, send email to native-cli...@googlegroups.com.
Visit this group at http://groups.google.com/group/native-client-discuss.
For more options, visit https://groups.google.com/groups/opt_out.

[Andrey Khalyavin]

On Wednesday, October 16, 2013 11:20:32 PM UTC+4, Richard Bateman wrote:

We have certainly looked at the native messaging API; however, my understanding is that the native messaging API does not have any performant way to transfer large chunks of binary data (such as a video stream).

I would make an experiment to be sure. 

There are a great many different things that FireBreath users need this sort of functionality for, but the use that I professional have is with GradeCam. GradeCam is a technology used by a large and rapidly growing number of teachers in the US (and it’s starting to become more common elsewhere as well) that utilizes a webcam and a proprietary image processing system to read bubble answer sheets on the teacher’s computer. It’s a really cool system, and it helps a lot of people.  Since Chrome plans to drop NPAPI support, we’re trying to find the best alternative for providing this functionality.

NaCl is obviously the first choice people think of when considering alternatives to NPAPI, but from looking at the documentation I’m not seeing any way that we can get video from the webcam other than using the HTML5 APIs, but in that case I don’t see a good way to get the data into the NaCl plugin for processing, so that doesn’t seem like it will work.  

You can send ArrayBuffer from javascript to NaCl. Messages up to several megabytes work fine. There is also the video capture interface in the dev folder. There are no near term plans to move it out of dev though. NPAPI deprecation is a surprise for us which we are not ready for.

In addition, those APIs don’t support some of the more common USB document cameras that we need to support (such as ELMO).  Native Messaging would be an option, but I don’t see a way to transfer the data fast enough to display the image in the browser; and can you use native messaging from inside NaCl? The docs I’ve found are unclear about any of that.

It is not available inside NaCl. This is, unfortunately, a slow process.

Any insight you or anyone can provide would be appreciated; feeling a little bit abandoned by my favorite browser these days, as there don’t seem to be any alternatives to NPAPI that cover most of our use cases :-(

The worst case, you can do all heavy processing outside of the browser and send only scores through native messaging API to javascript application (which will not need NaCl).

-- Andrey Khalyavin 

[Andrey Khalyavin]

I modified native messaging example to measure its throughput. 1024 messages 100Kb each were transferred from browser and back (100Mb in each direction total) in 13.4 seconds. So it is about 7 Mb per second on my machine.

-- Andrey Khalyavin

[Richard Bateman]

That’s promising; can an ArrayBuffer be sent over native messaging, or some other binary format?

Thank you so much for your help,

Richard

On Oct 17, 2013, at 4:29 , Andrey Khalyavin <haly...@google.com> wrote:

I modified native messaging example to measure its throughput. 1024 messages 100Kb each were transferred from browser and back (100Mb in each direction total) in 13.4 seconds. So it is about 7 Mb per second on my machine.

-- Andrey Khalyavin

[Andrey Khalyavin]

On Thursday, October 17, 2013 9:35:06 PM UTC+4, Richard Bateman wrote:

That’s promising; can an ArrayBuffer be sent over native messaging, or some other binary format?

 

Native Messaging sends and receives JSON which is decoded and encoded into javascript dictionary, arrays, string and numbers (no binary formats available). I don't think that ArrayBuffer can pass through.

ArrayBuffers are possible to send between Javascript and NaCl. They may be useful if you get video from HTML 5 API or a file.

So if you need to send data from native app to NaCl, you have to either reencode data to ArrayBuffer or use something else. Array of strings or single very long string for example.

-- Andrey Khalyavin

[David Michael]

On Wed, Oct 16, 2013 at 1:20 PM, Richard Bateman <ric...@batemansr.us> wrote:

We have certainly looked at the native messaging API; however, my understanding is that the native messaging API does not have any performant way to transfer large chunks of binary data (such as a video stream).

There are a great many different things that FireBreath users need this sort of functionality for, but the use that I professional have is with GradeCam. GradeCam is a technology used by a large and rapidly growing number of teachers in the US (and it’s starting to become more common elsewhere as well) that utilizes a webcam and a proprietary image processing system to read bubble answer sheets on the teacher’s computer. It’s a really cool system, and it helps a lot of people.  Since Chrome plans to drop NPAPI support, we’re trying to find the best alternative for providing this functionality.

NaCl is obviously the first choice people think of when considering alternatives to NPAPI, but from looking at the documentation I’m not seeing any way that we can get video from the webcam other than using the HTML5 APIs, but in that case I don’t see a good way to get the data into the NaCl plugin for processing, so that doesn’t seem like it will work.  In addition, those APIs don’t support some of the more common USB document cameras that we need to support (such as ELMO).  Native Messaging would be an option, but I don’t see a way to transfer the data fast enough to display the image in the browser; and can you use native messaging from inside NaCl? The docs I’ve found are unclear about any of that.

WebRTC might help here:

http://www.html5rocks.com/en/tutorials/webrtc/basics/

We're also planning to add support in Pepper for WebRTC. This will allow doing video and audio processing for input and/or output. We have a Private interface that handles video only right now that you could play with. Here's the example:

https://code.google.com/p/chromium/codesearch#chromium/src/ppapi/examples/video_effects/video_effects.cc

We want to make this interface public/stable eventually, at which point it would be usable by NaCl. It's going to change a bit before the final version, and I can't promise any target release right now. But it is being worked on right now.

[Richard Bateman]

I will be watching progress on this, then. 

Thanks!  We have considered webrtc; it looks like the closest thing to a usable option for our needs, though it doesn't help most of the firebreath users who need drawing.

Richard

[Eric Woods]

Hi David. Is there any news to report on WebRTC for NaCL? Fingers crossed they made some progress on this in the last 5 months.

Cheers,

Eric

To unsubscribe from this group and stop receiving emails from it, send an email to native-client-discuss+unsub...@googlegroups.com.

To post to this group, send email to native-cli...@googlegroups.com.
Visit this group at http://groups.google.com/group/native-client-discuss.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "Native-Client-Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to native-client-discuss+unsub...@googlegroups.com.

[Eric Woods]

Hi Richard. I'm a bit out of my depth here, but if NaCL can draw 3D graphics (allbeit OpenGL ES), then I would have thought it could draw video too (without the concern of a bottleneck passing it back to the browser to display)?

I know I have seen WebRTC examples  taking webcam video, adding filters, and displaying the result. But I don't know what path it took - I think it stayed in JavaScript, so may have never gone to Native Code.

So maybe the problem with drawing is only if you are using WebRTC without using NaCL.

Cheers, Eric 

To unsubscribe from this group and stop receiving emails from it, send an email to native-client-discuss+unsub...@googlegroups.com.

To post to this group, send email to native-cli...@googlegroups.com.
Visit this group at http://groups.google.com/group/native-client-discuss.
For more options, visit https://groups.google.com/groups/opt_out.

-- 
You received this message because you are subscribed to the Google Groups "Native-Client-Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to native-client-discuss+unsub...@googlegroups.com.

To post to this group, send email to native-cli...@googlegroups.com.
Visit this group at http://groups.google.com/group/native-client-discuss.
For more options, visit https://groups.google.com/groups/opt_out.

-- 
You received this message because you are subscribed to the Google Groups "Native-Client-Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to native-client-discuss+unsub...@googlegroups.com.

[Richard Bateman]

The issue is that NaCL can draw the video but it can’t get it from all the various hardware sources that people use, access parts of the system that people need, etc.

It might be possible to combine the two technologies; that would require an external application on the system or maybe a native messaging-launched program to provide the webrtc stream I’m assuming, but it’s still an option.

Of course, the main problem we have with this is that FireBreath users use fire breath in order to find a solution that will work on all browsers — since Firefox seems completely uninterested in NaCL most FireBreath users want to find a solution that doesn’t require using it.  Unfortunately it seems unlikely that they will find one, so we’ll probably all have to create a hodgepodge of different solutions trying to reimplement the same things in a different way for each browser.

You can imagine why we’re all a bit frustrated with it =]

Richard

To unsubscribe from this group and stop receiving emails from it, send an email to native-client-di...@googlegroups.com.

To post to this group, send email to native-cli...@googlegroups.com.
Visit this group at http://groups.google.com/group/native-client-discuss.

For more options, visit https://groups.google.com/d/optout.

[David Michael]

Hi there. Sorry I didn't see this thread pop up again sooner. We're currently working on two new APIs that will allow you to do video and audio input & output in a way that's compatible with getUserMedia:
https://code.google.com/p/chromium/codesearch#chromium/src/ppapi/cpp/media_stream_video_track.h&l=27

https://code.google.com/p/chromium/codesearch#chromium/src/ppapi/cpp/media_stream_audio_track.h&l=27

It's still early days, so these are subject to change. But they should be working today for input in Dev Channel and Canary. They'll stay that way until we feel they're stable enough to ship.

See also this thread on stackoverflow, including links to examples:

http://stackoverflow.com/questions/22144813/is-it-possible-to-record-video-stream-in-chrome-plugin-using-nacl-ppapi

For anybody interested in these APIs, we would appreciate it if you could start experimenting and filing bugs at new.crbug.com. (You can CC dmichael on the bug to make sure I see it).

Best,

Dave

[Faldegast]

On Thursday, October 17, 2013 8:53:22 AM UTC+2, Andrey Khalyavin wrote:

On Wednesday, October 16, 2013 11:20:32 PM UTC+4, Richard Bateman wrote:

We have certainly looked at the native messaging API; however, my understanding is that the native messaging API does not have any performant way to transfer large chunks of binary data (such as a video stream).

I would make an experiment to be sure. 

Its been told below that it uses json which would give quite inferior capacity. 

There are a great many different things that FireBreath users need this sort of functionality for, but the use that I professional have is with GradeCam. GradeCam is a technology used by a large and rapidly growing number of teachers in the US (and it’s starting to become more common elsewhere as well) that utilizes a webcam and a proprietary image processing system to read bubble answer sheets on the teacher’s computer. It’s a really cool system, and it helps a lot of people.  Since Chrome plans to drop NPAPI support, we’re trying to find the best alternative for providing this functionality.

NaCl is obviously the first choice people think of when considering alternatives to NPAPI, but from looking at the documentation I’m not seeing any way that we can get video from the webcam other than using the HTML5 APIs, but in that case I don’t see a good way to get the data into the NaCl plugin for processing, so that doesn’t seem like it will work.  

You can send ArrayBuffer from javascript to NaCl. Messages up to several megabytes work fine. There is also the video capture interface in the dev folder. There are no near term plans to move it out of dev though. NPAPI deprecation is a surprise for us which we are not ready for.

While I am working on a different plugin my problems are similar. For one webcam stream that may work fine, but what happens when you have multiple streams that need to pass on many such messages each. A binary protocol would be much faster but still not good. The fastest way to do platform independent IPC today are shared memory, which have a lot of overhead in its setup which only makes it suitable for bulk data transfar. One alternative would be to have a shared memory area where the nacl sandbox enforce access restriction, however I have no idea of how much overhead that would be. I know that some embedded processors have build in secure IPC hardware accessible trough the MCAPI standard, but on x86 the MCAPI implementation just uses shared memory. However comparing raw shared memory and MCAPI, I would suggest MCAPI because at least in some deployments it support something better than shared memory...

That's a problem because Chrome are dropping NPAPI, which leaves plugin developers hanging. To drop NPAPI without having PPAPI would really suck.

Being constrained to a sandbox like the current PPAPI will not work for all use cases, and perhaps never will. With java Applets we at least had installable extensions and something similar could fill the gap in PPAPI. The sandbox are useful even if I want to be able to access native libraries, so being able to install an extension to the API and use pnacl for most of the code would be a really useful solution. That would mean that I only have to think about security when writing the API extension and can relax when I write the generic and sandboxed code that runs on top of it. It would also mean that less code have to be tested with multiple platforms as my plugin needs to work in a mixed environment.

In addition, those APIs don’t support some of the more common USB document cameras that we need to support (such as ELMO).  Native Messaging would be an option, but I don’t see a way to transfer the data fast enough to display the image in the browser; and can you use native messaging from inside NaCl? The docs I’ve found are unclear about any of that.

It is not available inside NaCl. This is, unfortunately, a slow process.

Well, at least the process are faster than the development of Java Applets which have been stalled since the 1990:s, and still lacks any support for hardware accelerated multimedia and graphics.

Any insight you or anyone can provide would be appreciated; feeling a little bit abandoned by my favorite browser these days, as there don’t seem to be any alternatives to NPAPI that cover most of our use cases :-(

The worst case, you can do all heavy processing outside of the browser and send only scores through native messaging API to javascript application (which will not need NaCl).

That's a really bad worst case scenario.

-- Andrey Khalyavin 

[Faldegast]

Yeah, that would be to slow for me.