USB and Decoding from PNaCl, Re: Android Remoting Over USB for ChromeOS

115 views
Skip to first unread message

David Michael

unread,
Apr 21, 2014, 3:29:53 PM4/21/14
to Ryan Sleevi, David Reese, Chromium-dev
Hi,
+native-client-dev, which is a good forum for NaCl and PNaCl questions.
I'm going to BCC chromium-dev for this response, since I'm focusing on the NaCl/PNaCl and Pepper parts.

Unfortunately, there might not be any better way to accomplish what you're doing right now, but we are working to improve Pepper to help out.


On Mon, Apr 21, 2014 at 12:42 PM, Ryan Sleevi <rsl...@chromium.org> wrote:
Hi David,

I've cross-posted this to chromium-dev, and moved chromium-os-dev@ to BCC. The stacks that you're interested in (PNaCL, USB I/O, VP8 decoding/display) are all things that generally fall under the "Chromium" side of the "Chromium OS", so these teams can help.



On Mon, Apr 21, 2014 at 11:21 AM, David Reese <da...@bluestacks.com> wrote:
Hello,
BlueStacks currently has an Android stick that can be attached to a Mac or PC via USB.  The user then interacts with the Android UI via a companion program which runs on the Mac/PC.  We'd like to get the same device working on ChromeOS, as we see a lot of potential selling it as an accessory for the Chromebook.

To provide a meaningful user experience, we need to send user input events from the Chromebook to the stick and receive video updates and audio samples in the other direction.  User input isn't a problem, but both video and audio have proven to be more of a challenge.

In our current model, we handle USB transfers in JavaScript using the chrome.usb API's and perform wire encoding/decoding/routing of the payload in PNaCL.  To process video frames, we call back into JavaScript to feed the VP8 encoded payload to the MediaSource() API, which decodes and displays the frame to the user.  This means that for a single video frame, we do the following:
  1. JavaScript:  Complete a USB transfer to receive a message from the stick.
We definitely need a USB Pepper API to fix this. Unfortunately, we're not working concretely on this right now, so I couldn't give you an ETA. (+sammc & yzshen)
  1. JavaScript -> PNaCL transition
How are you packaging your data? Your best chance of good performance will be to pack any big binary data in to an ArrayBuffer. You can use it as a field in a simple object, which will be received in HandleMessage in PNaCl as a "Dictionary" Var.
 
  1. PNaCL:  Unwrap the message payload and find the appropriate handler.
  2. PNaCL -> JavaScript transition
  3. JavaScript:  Feed the payload to the MediaSource() API for decoding and display.
We have an API in development for decoding video from within PNaCl, so that may help you out when it's available:
 
Thus, we have two unwanted transitions between JavaScript and PNaCL.  We can avoid this by either moving our message codec/routing to JavaScript or moving USB I/O and VP8 decode/display to PNaCL.  For a number of reasons, we would much prefer the latter option.

One of the many issues that we have with the MediaSource() API is occasional stream hangs.  After the stick has attached and we start remoting video frames, the display reliably hangs.  It may take five seconds or ten minutes, but it always hangs.  Trying to pause/play the video element doesn't help at all.  An earlier batch of our sticks had an issue where the VP8 encoder would occasionally produce junk when encoding a frame with VP8.  I'm wondering if this is a possible cause of these hangs.  We don't see this problem when handing off the video stream to libvpx instead.
This doesn't sound like it's Pepper or PNaCl related, so maybe somebody else can answer. It might be worthwhile to file a bug for this:
 

Audio is a slightly different story.  As with video, we handle the USB transfers in JavaScript and then transition to PNaCL to decode/route the payload.  However, the handler for audio messages is in PNaCL, so we do the following:
  1. JavaScript:  Complete a USB transfer to receive a message from the stickl.
  2. JavaScript -> PNaCL transition
  3. PNaCL:  Unwrap the message payload to find the appropriate handler.
  4. PNaCL:  Feed the payload to the pp::Audio API for playback.
The issue with this approach is that we cannot seem to handle a high enough USB transfer rate as required by our audio remoting protocol.  We get a burst of completed USB transfers, which are then handled by pp::Audio in PNaCL.  It seems that performing USB transfers at this rate in JavaScript is consuming enough CPU to starve the PNaCL process, which is performing playback.  This results very poor, jerky audio playback.
Until there's a way to access USB from NaCl or PNaCl, your best bet might be to use Web Audio for playback, if it can do what you need:
 

Is there currently any push to move USB I/O and VP8 decoding/display to PNaCL?  These changes could get us much closer to having a viable product for Chromebook.

Thank you.

--
David Reese

--
--
Chromium OS Developers mailing list: chromiu...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-os-dev?hl=en


--
--
Chromium Developers mailing list: chromi...@chromium.org

View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

Ami Fischman

unread,
Apr 21, 2014, 4:03:38 PM4/21/14
to David Michael, Ryan Sleevi, David Reese, Chromium-dev, native-c...@google.com
  1. JavaScript:  Feed the payload to the MediaSource() API for decoding and display.
We have an API in development for decoding video from within PNaCl, so that may help you out when it's available:

FWIW, only ARM ChromeOS devices have VP8 decode in HW today, and the PNaCl API in the CL above only offers HW codecs (i.e. it is not planned to offer SW fallback via that API AFAIK).
 
One of the many issues that we have with the MediaSource() API is occasional stream hangs.  After the stick has attached and we start remoting video frames, the display reliably hangs.  It may take five seconds or ten minutes, but it always hangs.  Trying to pause/play the video element doesn't help at all.  An earlier batch of our sticks had an issue where the VP8 encoder would occasionally produce junk when encoding a frame with VP8.  I'm wondering if this is a possible cause of these hangs.  We don't see this problem when handing off the video stream to libvpx instead.
This doesn't sound like it's Pepper or PNaCl related, so maybe somebody else can answer. It might be worthwhile to file a bug for this: new.crbug.com

Yes please (and please mention the bug # in this thread once filed).
The best bug report would include a dump of the packets you're feeding MediaSource.
 
Audio is a slightly different story.  As with video, we handle the USB transfers in JavaScript and then transition to PNaCL to decode/route the payload.  However, the handler for audio messages is in PNaCL, so we do the following:
  1. JavaScript:  Complete a USB transfer to receive a message from the stickl.
  2. JavaScript -> PNaCL transition
  3. PNaCL:  Unwrap the message payload to find the appropriate handler.
  4. PNaCL:  Feed the payload to the pp::Audio API for playback.
The issue with this approach is that we cannot seem to handle a high enough USB transfer rate as required by our audio remoting protocol.  We get a burst of completed USB transfers, which are then handled by pp::Audio in PNaCL.  It seems that performing USB transfers at this rate in JavaScript is consuming enough CPU to starve the PNaCL process, which is performing playback.  This results very poor, jerky audio playback.
Until there's a way to access USB from NaCl or PNaCl, your best bet might be to use Web Audio for playback, if it can do what you need:

What he said.  Also, if the audio & video streams are related, you could mux them together (e.g. to webm) on the stick and then play _both_ back through MSE, which would get you A/V sync "for free".

Cheers,
-a

David Michael

unread,
Apr 21, 2014, 4:08:28 PM4/21/14
to Ami Fischman, Ryan Sleevi, David Reese, native-c...@googlegroups.com
(+native-client-dev for real this time, sorry about that)

David Reese

unread,
Apr 23, 2014, 1:30:49 AM4/23/14
to chromi...@chromium.org, Ryan Sleevi, David Reese
David,
Thanks for your feedback.  I have some followup statements inline.


On Monday, April 21, 2014 12:29:53 PM UTC-7, David Michael wrote:
Hi,
+native-client-dev, which is a good forum for NaCl and PNaCl questions.
I'm going to BCC chromium-dev for this response, since I'm focusing on the NaCl/PNaCl and Pepper parts.

Unfortunately, there might not be any better way to accomplish what you're doing right now, but we are working to improve Pepper to help out.


On Mon, Apr 21, 2014 at 12:42 PM, Ryan Sleevi <rsl...@chromium.org> wrote:
Hi David,

I've cross-posted this to chromium-dev, and moved chromium-os-dev@ to BCC. The stacks that you're interested in (PNaCL, USB I/O, VP8 decoding/display) are all things that generally fall under the "Chromium" side of the "Chromium OS", so these teams can help.



On Mon, Apr 21, 2014 at 11:21 AM, David Reese <da...@bluestacks.com> wrote:
Hello,
BlueStacks currently has an Android stick that can be attached to a Mac or PC via USB.  The user then interacts with the Android UI via a companion program which runs on the Mac/PC.  We'd like to get the same device working on ChromeOS, as we see a lot of potential selling it as an accessory for the Chromebook.

To provide a meaningful user experience, we need to send user input events from the Chromebook to the stick and receive video updates and audio samples in the other direction.  User input isn't a problem, but both video and audio have proven to be more of a challenge.

In our current model, we handle USB transfers in JavaScript using the chrome.usb API's and perform wire encoding/decoding/routing of the payload in PNaCL.  To process video frames, we call back into JavaScript to feed the VP8 encoded payload to the MediaSource() API, which decodes and displays the frame to the user.  This means that for a single video frame, we do the following:
  1. JavaScript:  Complete a USB transfer to receive a message from the stick.
We definitely need a USB Pepper API to fix this. Unfortunately, we're not working concretely on this right now, so I couldn't give you an ETA. (+sammc & yzshen)

If it would be of any use, I'd be willing to provide feedback on any USB Pepper API proposals.  This is something that we seriously need.
  1. JavaScript -> PNaCL transition
How are you packaging your data? Your best chance of good performance will be to pack any big binary data in to an ArrayBuffer. You can use it as a field in a simple object, which will be received in HandleMessage in PNaCl as a "Dictionary" Var. 
 
Yes, we're already using an ArrayBuffer passed via a dictionary.
  1. PNaCL:  Unwrap the message payload and find the appropriate handler.
  2. PNaCL -> JavaScript transition
  3. JavaScript:  Feed the payload to the MediaSource() API for decoding and display.
We have an API in development for decoding video from within PNaCl, so that may help you out when it's available:

This is very exciting.  How soon will I be able to get these changes in the canary build?

Thus, we have two unwanted transitions between JavaScript and PNaCL.  We can avoid this by either moving our message codec/routing to JavaScript or moving USB I/O and VP8 decode/display to PNaCL.  For a number of reasons, we would much prefer the latter option.

One of the many issues that we have with the MediaSource() API is occasional stream hangs.  After the stick has attached and we start remoting video frames, the display reliably hangs.  It may take five seconds or ten minutes, but it always hangs.  Trying to pause/play the video element doesn't help at all.  An earlier batch of our sticks had an issue where the VP8 encoder would occasionally produce junk when encoding a frame with VP8.  I'm wondering if this is a possible cause of these hangs.  We don't see this problem when handing off the video stream to libvpx instead.
This doesn't sound like it's Pepper or PNaCl related, so maybe somebody else can answer. It might be worthwhile to file a bug for this:

Getting MediaSource() working isn't a high priority for me, but if I get some cycles, I'll file a bug with sample payloads.
 
Audio is a slightly different story.  As with video, we handle the USB transfers in JavaScript and then transition to PNaCL to decode/route the payload.  However, the handler for audio messages is in PNaCL, so we do the following:
  1. JavaScript:  Complete a USB transfer to receive a message from the stickl.
  2. JavaScript -> PNaCL transition
  3. PNaCL:  Unwrap the message payload to find the appropriate handler.
  4. PNaCL:  Feed the payload to the pp::Audio API for playback.
The issue with this approach is that we cannot seem to handle a high enough USB transfer rate as required by our audio remoting protocol.  We get a burst of completed USB transfers, which are then handled by pp::Audio in PNaCL.  It seems that performing USB transfers at this rate in JavaScript is consuming enough CPU to starve the PNaCL process, which is performing playback.  This results very poor, jerky audio playback.
Until there's a way to access USB from NaCl or PNaCl, your best bet might be to use Web Audio for playback, if it can do what you need:

Thanks for the pointer, I'll take a look and see if it fits our needs.

David Reese

unread,
Apr 23, 2014, 1:38:14 AM4/23/14
to chromi...@chromium.org, David Michael, Ryan Sleevi, David Reese, native-c...@google.com
On Monday, April 21, 2014 1:03:38 PM UTC-7, Ami Fischman wrote:
  1. JavaScript:  Feed the payload to the MediaSource() API for decoding and display.
We have an API in development for decoding video from within PNaCl, so that may help you out when it's available:

FWIW, only ARM ChromeOS devices have VP8 decode in HW today, and the PNaCl API in the CL above only offers HW codecs (i.e. it is not planned to offer SW fallback via that API AFAIK).

That's okay for now.  We're currently looking to put together a proof point for the Samsung Chromebook, which is ARM.

One of the many issues that we have with the MediaSource() API is occasional stream hangs.  After the stick has attached and we start remoting video frames, the display reliably hangs.  It may take five seconds or ten minutes, but it always hangs.  Trying to pause/play the video element doesn't help at all.  An earlier batch of our sticks had an issue where the VP8 encoder would occasionally produce junk when encoding a frame with VP8.  I'm wondering if this is a possible cause of these hangs.  We don't see this problem when handing off the video stream to libvpx instead.
This doesn't sound like it's Pepper or PNaCl related, so maybe somebody else can answer. It might be worthwhile to file a bug for this: new.crbug.com

Yes please (and please mention the bug # in this thread once filed).
The best bug report would include a dump of the packets you're feeding MediaSource.

As I responded to David, I'll go ahead and file a bug if I can get some free cycles.

Audio is a slightly different story.  As with video, we handle the USB transfers in JavaScript and then transition to PNaCL to decode/route the payload.  However, the handler for audio messages is in PNaCL, so we do the following:
  1. JavaScript:  Complete a USB transfer to receive a message from the stickl.
  2. JavaScript -> PNaCL transition
  3. PNaCL:  Unwrap the message payload to find the appropriate handler.
  4. PNaCL:  Feed the payload to the pp::Audio API for playback.
The issue with this approach is that we cannot seem to handle a high enough USB transfer rate as required by our audio remoting protocol.  We get a burst of completed USB transfers, which are then handled by pp::Audio in PNaCL.  It seems that performing USB transfers at this rate in JavaScript is consuming enough CPU to starve the PNaCL process, which is performing playback.  This results very poor, jerky audio playback.
Until there's a way to access USB from NaCl or PNaCl, your best bet might be to use Web Audio for playback, if it can do what you need:

What he said.  Also, if the audio & video streams are related, you could mux them together (e.g. to webm) on the stick and then play _both_ back through MSE, which would get you A/V sync "for free".

That's a very good point.  Not sure if we want to spend time on this internally, as we really want all of this stuff moved to PNaCL.

Thank you.
Reply all
Reply to author
Forward
0 new messages