Intent to Prototype: Client-side video editing (MediaBlob)

736 views
Skip to first unread message

Yash Khandelwal

unread,
May 26, 2020, 5:41:12 PM5/26/20
to blink-dev, ste...@microsoft.com, harn...@microsoft.com

Intent to Prototype: Client-side video editing (MediaBlob) 

 

Contact emails 

Explainer 

 

Design doc/Spec 

Summary 

MediaBlob is a high-level API that enables video editing on the browser providing operations such as trim, concatenation, split etc. 
 

Motivation 
This API solves these common overheads/workarounds that web developers currently use for editing media: 

  1. Use external JavaScript/Web Assembly libraries usually in several MBs (affects bandwidth) 

  1. Server roundtrip (affects both time and bandwidth) 

Apart from solving the above overheads, this API also provides these advantages: 
1. Web developerdo not need to know the details of low-level media concepts (like muxingdemuxing, decoding, encoding) to use this API. 
2. Hardware optimizations when available. 
3. Inherits from Blob so easy to use with MediaRecorder and File API. 
4. Web developers do not have to worry about codecs licensing. 
 

Risks 

Interoperability and Compatibility 

 
Edge: In development 
FirefoxInterest shown for this kind of implementation 
Safari: No public signals 
Web/Framework developers:  Positive 

 

Ergonomics 

MediaBlob inherits the Blob API so it will be used often with Blob. 
 

Activation 

This API intends to replace the use of polyfills and libraries meant for video editing. 
 

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)? 

Yes. 

 

Link to entry on the feature dashboard 

No dashboard entry planned until requesting approval to ship. 

 

Requesting approval to ship? 

Not at this time, will prioritize testing with interested partner sites first. 

 
 

Dale Curtis

unread,
May 26, 2020, 6:04:56 PM5/26/20
to Yash Khandelwal, blink-dev, ste...@microsoft.com, Harneet Sidhana
I'm surprised to see this. Last we (the media team) spoke with Microsoft (sorry I don't recall if you were in the meeting) the conclusion appeared to be that this effort wasn't necessary with the advent of WebCodecs. Can you elaborate on why we'd want to build this into the browser versus delegating to JavaScript? Unlike encoding and decoding, muxing/demuxing aren't computationally expensive -- so it feels like a small JS library using WebCodecs can handle all your use cases.

The only piece I'd expect to have some size would be bundling whatever demuxer you need; though most are probably just going to want mp4 (which JS solutions or tiny WASM/C++ solutions exist). The muxing can be handled by MediaRecorder as you note in your explainer. So trim, join, etc, are all just demux -> filter_by_pts -> WebCodecs::encode -> MediaRecorder.

- dale

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/3ba12a05-4800-4199-8cba-3b9300ee582b%40chromium.org.

Joe Medley

unread,
May 28, 2020, 11:15:37 AM5/28/20
to Yash Khandelwal, blink-dev, Steve Becker, harn...@microsoft.com
Do you have a tracking bug I can star?
Joe Medley | Technical Writer, Chrome DevRel | jme...@google.com | 816-678-7195
If an API's not documented it doesn't exist.


On Tue, May 26, 2020 at 2:41 PM 'Yash Khandelwal' via blink-dev <blin...@chromium.org> wrote:
--

Yash Khandelwal

unread,
May 28, 2020, 6:04:46 PM5/28/20
to blink-dev, yakh...@microsoft.com, ste...@microsoft.com, harn...@microsoft.com

@Dale, Thanks for your response.

Yes, we are aware that one of the many use cases of WebCodecs is media editing and also agree that muxing and demuxing are not computation heavy operations.
You mentioned  “So trim, join, etc, are all just demux -> filter_by_pts -> WebCodecs::encode -> MediaRecorder”. Can you provide more details on the WebCodecs::Encode part ?

We are trying to understand on how to use WebCodecs to achieve the same functionality as MediaBlob to ensure that it meets developer needs. 


On Tuesday, 26 May 2020 15:04:56 UTC-7, Dale Curtis wrote:
I'm surprised to see this. Last we (the media team) spoke with Microsoft (sorry I don't recall if you were in the meeting) the conclusion appeared to be that this effort wasn't necessary with the advent of WebCodecs. Can you elaborate on why we'd want to build this into the browser versus delegating to JavaScript? Unlike encoding and decoding, muxing/demuxing aren't computationally expensive -- so it feels like a small JS library using WebCodecs can handle all your use cases.

The only piece I'd expect to have some size would be bundling whatever demuxer you need; though most are probably just going to want mp4 (which JS solutions or tiny WASM/C++ solutions exist). The muxing can be handled by MediaRecorder as you note in your explainer. So trim, join, etc, are all just demux -> filter_by_pts -> WebCodecs::encode -> MediaRecorder.

- dale

On Tue, May 26, 2020 at 2:41 PM 'Yash Khandelwal' via blink-dev <blin...@chromium.org> wrote:

Intent to Prototype: Client-side video editing (MediaBlob) 

 

Contact emails 

To unsubscribe from this group and stop receiving emails from it, send an email to blin...@chromium.org.

Dale Curtis

unread,
May 28, 2020, 6:30:36 PM5/28/20
to Yash Khandelwal, blink-dev, ste...@microsoft.com, Harneet Sidhana
On Thu, May 28, 2020 at 3:04 PM 'Yash Khandelwal' via blink-dev <blin...@chromium.org> wrote:

You mentioned  “So trim, join, etc, are all just demux -> filter_by_pts -> WebCodecs::encode -> MediaRecorder”. Can you provide more details on the WebCodecs::Encode part ?

One example in the explainer covers this somewhat: https://github.com/WICG/web-codecs/blob/master/explainer.md#example-of-transcoding-or-offline-encodedecode -- you'd modify that to pre-filter the data going into the decoder. I.e., for trimming a stream, you'd either not send the packets to the decoder or when the trim doesn't begin with a keyframe you'd send the GOP to the decoder and then only send some subset of it to the encoder. For joining, you'd send both streams through the decoder to the encoder. 

Note: We currently don't actually have integration with MediaRecorder for muxing, see https://github.com/WICG/web-codecs/issues/61. It's unclear if folks actually want to use that versus just using mux.js or similar. Please add feedback to that issue if you have users which would prefer to mux through MediaRecorder. It would need some changes to handle transcoding; currently MediaRecorder doesn't have mp4 support and generates its own frame timestamps based on real time.

- dale

Yash Khandelwal

unread,
May 28, 2020, 8:13:18 PM5/28/20
to blink-dev, yakh...@microsoft.com, ste...@microsoft.com, harn...@microsoft.com
Thanks for this.
Follow-up question:
For joining, you'd send both streams through the decoder to the encoder
For trim/join, why do we want the data to go through the WebCodecs decode and encode workflow ? Or is it only needed when we want the output in a different encoding than the input ?

I will also create an issue on the WebCodecs explainer to request adding an example demonstrating media editing scenarios.

Dale Curtis

unread,
May 28, 2020, 8:31:35 PM5/28/20
to Yash Khandelwal, blink-dev, ste...@microsoft.com, Harneet Sidhana
On Thu, May 28, 2020 at 5:13 PM 'Yash Khandelwal' via blink-dev <blin...@chromium.org> wrote:
For trim/join, why do we want the data to go through the WebCodecs decode and encode workflow ? Or is it only needed when we want the output in a different encoding than the input ?

Sorry, I was assuming you needed re-encoding. If you don't need encoding, then simply slicing or appending w/ consideration of keyframe boundaries is sufficient. You'd only need mux.js and WebCodecs would not be involved at all.
 
I will also create an issue on the WebCodecs explainer to request adding an example demonstrating media editing scenarios.

Great; thanks!

- dale

Yash Khandelwal

unread,
Jun 3, 2020, 7:53:27 PM6/3/20
to blink-dev, yakh...@microsoft.com, ste...@microsoft.com, harn...@microsoft.com


On Thursday, 28 May 2020 08:15:37 UTC-7, Joe Medley wrote:
Do you have a tracking bug I can star?
Joe Medley | Technical Writer, Chrome DevRel | jme...@google.com | 816-678-7195
If an API's not documented it doesn't exist.


On Tue, May 26, 2020 at 2:41 PM 'Yash Khandelwal' via blink-dev <blin...@chromium.org> wrote:

Intent to Prototype: Client-side video editing (MediaBlob) 

 

Contact emails 

To unsubscribe from this group and stop receiving emails from it, send an email to blin...@chromium.org.

Ilya Grigorik

unread,
Jun 4, 2020, 6:17:35 PM6/4/20
to Yash Khandelwal, blink-dev, ste...@microsoft.com, harn...@microsoft.com
Hey folks.

On Thu, May 28, 2020 at 5:13 PM 'Yash Khandelwal' via blink-dev <blin...@chromium.org> wrote:
Thanks for this.
Follow-up question:
For joining, you'd send both streams through the decoder to the encoder
For trim/join, why do we want the data to go through the WebCodecs decode and encode workflow ? Or is it only needed when we want the output in a different encoding than the input ?

I will also create an issue on the WebCodecs explainer to request adding an example demonstrating media editing scenarios.

+1 to the above. I would love to see a hands-on example of how WebCodecs and/or mux.js addresses the same use cases, before we make a call in either direction.

The developer ergonomics of client-side media (image and video) editing capabilities that we offer today (to the extent that we do) today are, well.. not very good. While I'm excited about WASM and WebCodecs, I don't think we should rule out providing convenience APIs that address common use cases via simple and intuitive APIs.

To that end, it would be great to gather some feedback from developers that would rely on it.. 
Yash: I believe you've worked with some teams at MSFT to validate this API? It would be great to get their input and feedback on the alternate paths that we're discussing here.




On Wed, Jun 3, 2020 at 4:53 PM 'Yash Khandelwal' via blink-dev <blin...@chromium.org> wrote:
Here is the tracking bug
https://bugs.chromium.org/p/chromium/issues/detail?id=1090959

On Thursday, 28 May 2020 08:15:37 UTC-7, Joe Medley wrote:
Do you have a tracking bug I can star?
Joe Medley | Technical Writer, Chrome DevRel | jme...@google.com | 816-678-7195
If an API's not documented it doesn't exist.


On Tue, May 26, 2020 at 2:41 PM 'Yash Khandelwal' via blink-dev <blin...@chromium.org> wrote:

Intent to Prototype: Client-side video editing (MediaBlob) 

 

Contact emails 

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/d9b1401f-b385-450e-9a74-ec8e5451f3c0%40chromium.org.

Yash Khandelwal

unread,
Jun 8, 2020, 3:59:35 PM6/8/20
to blink-dev, yakh...@microsoft.com, ste...@microsoft.com, harn...@microsoft.com

@Dale, Have we considered extending WebCodecs to also provide demuxing and muxing ? I believe this will greatly help building media editing scenarios on top of WebCodecs.

I looked into external JavaScript libraries for demuxing and muxing. Here are my observations:
1. For mp4, videojs/mux.js does not seem to provide a direct API to get the encoded packets.
muxjs.mp4.tools.inspect only provides the structure of the mp4 file.
2. I also looked into these JavaScript webm demuxers: https://github.com/Yahweasel/mkvdemuxjs and https://github.com/jscodec/jswebm

These do provide direct access to encoded packets but I found these issues:

a. For some webm videos, I got "Color not supported" error.
b. Number of packets provided by ffprobe did not match the number of packets outputted by jscodec/jswebm.

c. Yahweasel/mkvdemuxjs: Timestamps were not accurate. They were resetting back to zero for some videos.

 

I believe other issues will arise when doing muxing using external JavaScript libraries. Hence, I believe these libraries do not provide a complete solution for web developers.

 

@Ilya: Yes, we have validated this approach with Flipgrid.  Their earlier approach was editing media files on the server side and switching to MediaBlob greatly decreased their turnaround time. Their feedback around WebCodecs and MediaBlob is to put efforts in a single API for media editing needs that includes shaping WebCodecs to make it easier editing videos on the client.

Also, we presented MediaBlob API to Media and Entertainment IG last week and the feedback was positive around the need of a direct and easy to use API for media editing. However, there was some feedback regarding using a playlist model instead of the current proposed model for performing these operations. Switching to this playlist model will make it more in sync with native video editing suites providing a lot more capabilities than the ones currently proposed.
MediaBlob was also discussed briefly during the Microsoft Build event last month and it resonated quite well with the developers.


On Thursday, 4 June 2020 15:17:35 UTC-7, Ilya Grigorik wrote:
Hey folks.

On Thu, May 28, 2020 at 5:13 PM 'Yash Khandelwal' via blink-dev <blin...@chromium.org> wrote:
Thanks for this.
Follow-up question:
For joining, you'd send both streams through the decoder to the encoder
For trim/join, why do we want the data to go through the WebCodecs decode and encode workflow ? Or is it only needed when we want the output in a different encoding than the input ?
I will also create an issue on the WebCodecs explainer to request adding an example demonstrating media editing scenarios.

+1 to the above. I would love to see a hands-on example of how WebCodecs and/or mux.js addresses the same use cases, before we make a call in either direction.

The developer ergonomics of client-side media (image and video) editing capabilities that we offer today (to the extent that we do) today are, well.. not very good. While I'm excited about WASM and WebCodecs, I don't think we should rule out providing convenience APIs that address common use cases via simple and intuitive APIs.

To that end, it would be great to gather some feedback from developers that would rely on it.. 
Yash: I believe you've worked with some teams at MSFT to validate this API? It would be great to get their input and feedback on the alternate paths that we're discussing here.




On Wed, Jun 3, 2020 at 4:53 PM 'Yash Khandelwal' via blink-dev <blin...@chromium.org> wrote:
Here is the tracking bug
https://bugs.chromium.org/p/chromium/issues/detail?id=1090959

On Thursday, 28 May 2020 08:15:37 UTC-7, Joe Medley wrote:
Do you have a tracking bug I can star?
Joe Medley | Technical Writer, Chrome DevRel | jme...@google.com | 816-678-7195
If an API's not documented it doesn't exist.


On Tue, May 26, 2020 at 2:41 PM 'Yash Khandelwal' via blink-dev <blin...@chromium.org> wrote:

Intent to Prototype: Client-side video editing (MediaBlob) 

 

Contact emails 

Ilya Grigorik

unread,
Jun 8, 2020, 5:02:37 PM6/8/20
to Yash Khandelwal, blink-dev, ste...@microsoft.com, harn...@microsoft.com
On Mon, Jun 8, 2020 at 12:59 PM 'Yash Khandelwal' via blink-dev <blin...@chromium.org> wrote: 

@Ilya: Yes, we have validated this approach with Flipgrid.  Their earlier approach was editing media files on the server side and switching to MediaBlob greatly decreased their turnaround time. Their feedback around WebCodecs and MediaBlob is to put efforts in a single API for media editing needs that includes shaping WebCodecs to make it easier editing videos on the client.

Also, we presented MediaBlob API to Media and Entertainment IG last week and the feedback was positive around the need of a direct and easy to use API for media editing. However, there was some feedback regarding using a playlist model instead of the current proposed model for performing these operations. Switching to this playlist model will make it more in sync with native video editing suites providing a lot more capabilities than the ones currently proposed.
MediaBlob was also discussed briefly during the Microsoft Build event last month and it resonated quite well with the developers.


That's great to hear. Thank you! Excited to see where this route will take us.


On Thursday, 4 June 2020 15:17:35 UTC-7, Ilya Grigorik wrote:
Hey folks.

On Thu, May 28, 2020 at 5:13 PM 'Yash Khandelwal' via blink-dev <blin...@chromium.org> wrote:
Thanks for this.
Follow-up question:
For joining, you'd send both streams through the decoder to the encoder
For trim/join, why do we want the data to go through the WebCodecs decode and encode workflow ? Or is it only needed when we want the output in a different encoding than the input ?
I will also create an issue on the WebCodecs explainer to request adding an example demonstrating media editing scenarios.

+1 to the above. I would love to see a hands-on example of how WebCodecs and/or mux.js addresses the same use cases, before we make a call in either direction.

The developer ergonomics of client-side media (image and video) editing capabilities that we offer today (to the extent that we do) today are, well.. not very good. While I'm excited about WASM and WebCodecs, I don't think we should rule out providing convenience APIs that address common use cases via simple and intuitive APIs.

To that end, it would be great to gather some feedback from developers that would rely on it.. 
Yash: I believe you've worked with some teams at MSFT to validate this API? It would be great to get their input and feedback on the alternate paths that we're discussing here.




On Wed, Jun 3, 2020 at 4:53 PM 'Yash Khandelwal' via blink-dev <blin...@chromium.org> wrote:
Here is the tracking bug
https://bugs.chromium.org/p/chromium/issues/detail?id=1090959

On Thursday, 28 May 2020 08:15:37 UTC-7, Joe Medley wrote:
Do you have a tracking bug I can star?
Joe Medley | Technical Writer, Chrome DevRel | jme...@google.com | 816-678-7195
If an API's not documented it doesn't exist.


On Tue, May 26, 2020 at 2:41 PM 'Yash Khandelwal' via blink-dev <blin...@chromium.org> wrote:

Intent to Prototype: Client-side video editing (MediaBlob) 

 

Contact emails 

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/8894b66e-a833-4210-af88-ea1331052631o%40chromium.org.

Dale Curtis

unread,
Jun 8, 2020, 5:26:51 PM6/8/20
to Yash Khandelwal, blink-dev, ste...@microsoft.com, Harneet Sidhana
On Mon, Jun 8, 2020 at 12:59 PM 'Yash Khandelwal' via blink-dev <blin...@chromium.org> wrote:

I looked into external JavaScript libraries for demuxing and muxing. Here are my observations:
1. For mp4, videojs/mux.js does not seem to provide a direct API to get the encoded packets.
muxjs.mp4.tools.inspect only provides the structure of the mp4 file.
2. I also looked into these JavaScript webm demuxers: https://github.com/Yahweasel/mkvdemuxjs and https://github.com/jscodec/jswebm

These do provide direct access to encoded packets but I found these issues:

a. For some webm videos, I got "Color not supported" error.
b. Number of packets provided by ffprobe did not match the number of packets outputted by jscodec/jswebm.

c. Yahweasel/mkvdemuxjs: Timestamps were not accurate. They were resetting back to zero for some videos.

 

I believe other issues will arise when doing muxing using external JavaScript libraries. Hence, I believe these libraries do not provide a complete solution for web developers.


Thanks for investigating. Did you look at mp4box.js? It sounds like you've just encountered bugs and/or feature limitations in these libraries, not problems which could not be fixed with some upstream work. Presumably a MediaBlob.js implementation could provide its own mp4, webm, etc muxers and demuxers that don't suffer from such issues.
 

 

@Ilya: Yes, we have validated this approach with Flipgrid.  Their earlier approach was editing media files on the server side and switching to MediaBlob greatly decreased their turnaround time. Their feedback around WebCodecs and MediaBlob is to put efforts in a single API for media editing needs that includes shaping WebCodecs to make it easier editing videos on the client.


By single API did they mean a browser API or just a library which used WebCodecs? I.e., would they be equally happy with a ~20-30kb zipped library like MediaBlob.js that was just as performant and used WebCodecs for transcoding?
 

Also, we presented MediaBlob API to Media and Entertainment IG last week and the feedback was positive around the need of a direct and easy to use API for media editing. However, there was some feedback regarding using a playlist model instead of the current proposed model for performing these operations. Switching to this playlist model will make it more in sync with native video editing suites providing a lot more capabilities than the ones currently proposed.
MediaBlob was also discussed briefly during the Microsoft Build event last month and it resonated quite well with the developers.


Glad to hear the developer reception is positive.

Do most developers just want to trim and join segments though? Or are they looking to do more complicated things like effects, cropping, etc. I.e., if we consider a "stories" like feature that is common these days: users record or choose a clip, trim it, apply some effects/stickers/text/etc, and finally submit. The best of the story UIs offer a live preview as this is happening. The current API doesn't seem to provide a preview mechanism, so one would have to repeatedly finalize MediaBlob operations as the user chooses different effects. In a preview case you'd really only want trim + decode + effects w/o finalizing. Which seems more suited to a demuxer + WebCodecs => MediaRecorder / custom muxer approach.

- dale

Yash Khandelwal

unread,
Jun 8, 2020, 9:30:41 PM6/8/20
to blink-dev, yakh...@microsoft.com, ste...@microsoft.com, harn...@microsoft.com
Thanks for investigating. Did you look at mp4box.js? It sounds like you've just encountered bugs and/or feature limitations in these libraries, not problems which could not be fixed with some upstream work. Presumably a MediaBlob.js implementation could provide its own mp4, webm, etc muxers and demuxers that don't suffer from such issues.

Thanks for pointing me to mp4box.js. Using extract.js, I was able to fetch the encoded packets from the mp4 container and did some performance comparison on my machine.
For this sample mp4 file, this library took 1181 ms to extract all 3182 video samples and 2214 ms to extract all 5968 audio samples. Whereas the browser implementation (using FFmpeg av_read_frame) took less than 100 ms to extract all 9150 audio and video samples.
The browser implementation is significantly better than these JavaScript libraries and I do not think we may be able to match it even if we work upon optimizing these libraries.

By single API did they mean a browser API or just a library which used WebCodecs? I.e., would they be equally happy with a ~20-30kb zipped library like MediaBlob.js that was just as performant and used WebCodecs for transcoding?
I meant a single Browser API to handle all media editing scenarios. Based on the above performance data, it does not seems like that using JavaScript libraries will provide the same performance as the browser would.
 
Do most developers just want to trim and join segments though? Or are they looking to do more complicated things like effects, cropping, etc. I.e., if we consider a "stories" like feature that is common these days: users record or choose a clip, trim it, apply some effects/stickers/text/etc, and finally submit. The best of the story UIs offer a live preview as this is happening. The current API doesn't seem to provide a preview mechanism, so one would have to repeatedly finalize MediaBlob operations as the user chooses different effects. In a preview case you'd really only want trim + decode + effects w/o finalizing. Which seems more suited to a demuxer + WebCodecs => MediaRecorder / custom muxer approach.
 
Yes, the current proposed implementation using finalize will not handle the preview case efficiently.

Dale Curtis

unread,
Jun 9, 2020, 5:33:52 PM6/9/20
to Yash Khandelwal, blink-dev, ste...@microsoft.com, Harneet Sidhana
On Mon, Jun 8, 2020 at 6:30 PM 'Yash Khandelwal' via blink-dev <blin...@chromium.org> wrote:
Thanks for investigating. Did you look at mp4box.js? It sounds like you've just encountered bugs and/or feature limitations in these libraries, not problems which could not be fixed with some upstream work. Presumably a MediaBlob.js implementation could provide its own mp4, webm, etc muxers and demuxers that don't suffer from such issues.

Thanks for pointing me to mp4box.js. Using extract.js, I was able to fetch the encoded packets from the mp4 container and did some performance comparison on my machine.
For this sample mp4 file, this library took 1181 ms to extract all 3182 video samples and 2214 ms to extract all 5968 audio samples. Whereas the browser implementation (using FFmpeg av_read_frame) took less than 100 ms to extract all 9150 audio and video samples.
The browser implementation is significantly better than these JavaScript libraries and I do not think we may be able to match it even if we work upon optimizing these libraries.

Can you share the sample code you used to test this? If I go to https://gpac.github.io/mp4box.js/test/filereader.html and load your sample video, it loads instantly.


By single API did they mean a browser API or just a library which used WebCodecs? I.e., would they be equally happy with a ~20-30kb zipped library like MediaBlob.js that was just as performant and used WebCodecs for transcoding?
I meant a single Browser API to handle all media editing scenarios. Based on the above performance data, it does not seems like that using JavaScript libraries will provide the same performance as the browser would. 
 
Do most developers just want to trim and join segments though? Or are they looking to do more complicated things like effects, cropping, etc. I.e., if we consider a "stories" like feature that is common these days: users record or choose a clip, trim it, apply some effects/stickers/text/etc, and finally submit. The best of the story UIs offer a live preview as this is happening. The current API doesn't seem to provide a preview mechanism, so one would have to repeatedly finalize MediaBlob operations as the user chooses different effects. In a preview case you'd really only want trim + decode + effects w/o finalizing. Which seems more suited to a demuxer + WebCodecs => MediaRecorder / custom muxer approach.
 
Yes, the current proposed implementation using finalize will not handle the preview case efficiently.
 
The explainer hints at an ecosystem assessment; do you have more insight into what use cases you expect to be most common?

- dale

Dale Curtis

unread,
Jun 9, 2020, 8:31:21 PM6/9/20
to Yash Khandelwal, blink-dev, ste...@microsoft.com, Harneet Sidhana
On Tue, Jun 9, 2020 at 2:33 PM Dale Curtis <dalec...@chromium.org> wrote:
On Mon, Jun 8, 2020 at 6:30 PM 'Yash Khandelwal' via blink-dev <blin...@chromium.org> wrote:
Thanks for investigating. Did you look at mp4box.js? It sounds like you've just encountered bugs and/or feature limitations in these libraries, not problems which could not be fixed with some upstream work. Presumably a MediaBlob.js implementation could provide its own mp4, webm, etc muxers and demuxers that don't suffer from such issues.

Thanks for pointing me to mp4box.js. Using extract.js, I was able to fetch the encoded packets from the mp4 container and did some performance comparison on my machine.
For this sample mp4 file, this library took 1181 ms to extract all 3182 video samples and 2214 ms to extract all 5968 audio samples. Whereas the browser implementation (using FFmpeg av_read_frame) took less than 100 ms to extract all 9150 audio and video samples.
The browser implementation is significantly better than these JavaScript libraries and I do not think we may be able to match it even if we work upon optimizing these libraries.

Can you share the sample code you used to test this? If I go to https://gpac.github.io/mp4box.js/test/filereader.html and load your sample video, it loads instantly.

For the sake of my own curiosity, I wrote a test page using MP4Box.js here: http://storage.googleapis.com/dalecurtis/test-load.html loading both tracks from the sample you provided takes 130ms. I'm assuming you tested the node version, since you mention extract.js, so I'm not sure why that one is so much slower.

Yash Khandelwal

unread,
Jun 11, 2020, 1:45:21 PM6/11/20
to blink-dev, yakh...@microsoft.com, ste...@microsoft.com, harn...@microsoft.com
Thanks a lot for looking into this. I can confirm the numbers that you are seeing.
I was using the node terminal which might have affected the performance. I am convinced now that demuxing (and probably muxing) can be efficiently handled by JS :)
Reply all
Reply to author
Forward
0 new messages