Intent to Prototype: Client-side video editing (MediaBlob)
Yash Khandelwal
Intent to Prototype: Client-side video editing (MediaBlob)
Contact emails
Explainer
Design doc/Spec
Summary
MediaBlob is a high-level API that enables video editing on the browser providing operations such as trim, concatenation, split etc.
Motivation
This API solves these common overheads/workarounds that web developers currently use for editing media:
Use external JavaScript/Web Assembly libraries usually in several MBs (affects bandwidth)
Server roundtrip (affects both time and bandwidth)
Apart from solving the above overheads, this API also provides these advantages:
1. Web developers do not need to know the details of low-level media concepts (like muxing, demuxing, decoding, encoding) to use this API.
2. Hardware optimizations when available.
3. Inherits from Blob so easy to use with MediaRecorder and File API.
4. Web developers do not have to worry about codecs licensing.
Risks
Interoperability and Compatibility
Edge: In development
Firefox: Interest shown for this kind of implementation
Safari: No public signals
Web/Framework developers: Positive
Ergonomics
MediaBlob inherits the Blob API so it will be used often with Blob.
Activation
This API intends to replace the use of polyfills and libraries meant for video editing.
Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?
Yes.
Link to entry on the feature dashboard
No dashboard entry planned until requesting approval to ship.
Requesting approval to ship?
Not at this time, will prioritize testing with interested partner sites first.
Dale Curtis
--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/3ba12a05-4800-4199-8cba-3b9300ee582b%40chromium.org.
Joe Medley
--
Yash Khandelwal
@Dale, Thanks for your response.
Yes, we are aware that one of the many use cases of WebCodecs is media editing and also agree that muxing and demuxing are not computation heavy operations.
You mentioned “So trim, join, etc, are all just demux -> filter_by_pts -> WebCodecs::encode -> MediaRecorder”. Can you provide more details on the WebCodecs::Encode part ?
We are trying to understand on how to use WebCodecs to achieve the same functionality as MediaBlob to ensure that it meets developer needs.
On Tuesday, 26 May 2020 15:04:56 UTC-7, Dale Curtis wrote:
I'm surprised to see this. Last we (the media team) spoke with Microsoft (sorry I don't recall if you were in the meeting) the conclusion appeared to be that this effort wasn't necessary with the advent of WebCodecs. Can you elaborate on why we'd want to build this into the browser versus delegating to JavaScript? Unlike encoding and decoding, muxing/demuxing aren't computationally expensive -- so it feels like a small JS library using WebCodecs can handle all your use cases.The only piece I'd expect to have some size would be bundling whatever demuxer you need; though most are probably just going to want mp4 (which JS solutions or tiny WASM/C++ solutions exist). The muxing can be handled by MediaRecorder as you note in your explainer. So trim, join, etc, are all just demux -> filter_by_pts -> WebCodecs::encode -> MediaRecorder.- dale
On Tue, May 26, 2020 at 2:41 PM 'Yash Khandelwal' via blink-dev <blin...@chromium.org> wrote:
Intent to Prototype: Client-side video editing (MediaBlob)
Contact emails
To unsubscribe from this group and stop receiving emails from it, send an email to blin...@chromium.org.
Dale Curtis
You mentioned “So trim, join, etc, are all just demux -> filter_by_pts -> WebCodecs::encode -> MediaRecorder”. Can you provide more details on the WebCodecs::Encode part ?
Yash Khandelwal
Follow-up question:
For joining, you'd send both streams through the decoder to the encoder
I will also create an issue on the WebCodecs explainer to request adding an example demonstrating media editing scenarios.
Dale Curtis
For trim/join, why do we want the data to go through the WebCodecs decode and encode workflow ? Or is it only needed when we want the output in a different encoding than the input ?
I will also create an issue on the WebCodecs explainer to request adding an example demonstrating media editing scenarios.
Yash Khandelwal
https://bugs.chromium.org/p/chromium/issues/detail?id=1090959
On Thursday, 28 May 2020 08:15:37 UTC-7, Joe Medley wrote:
Do you have a tracking bug I can star?If an API's not documented it doesn't exist.
On Tue, May 26, 2020 at 2:41 PM 'Yash Khandelwal' via blink-dev <blin...@chromium.org> wrote:
Intent to Prototype: Client-side video editing (MediaBlob)
Contact emails
To unsubscribe from this group and stop receiving emails from it, send an email to blin...@chromium.org.
Ilya Grigorik
Thanks for this.
Follow-up question:For joining, you'd send both streams through the decoder to the encoder
For trim/join, why do we want the data to go through the WebCodecs decode and encode workflow ? Or is it only needed when we want the output in a different encoding than the input ?
I will also create an issue on the WebCodecs explainer to request adding an example demonstrating media editing scenarios.
Here is the tracking bug
https://bugs.chromium.org/p/chromium/issues/detail?id=1090959
On Thursday, 28 May 2020 08:15:37 UTC-7, Joe Medley wrote:
Do you have a tracking bug I can star?If an API's not documented it doesn't exist.
On Tue, May 26, 2020 at 2:41 PM 'Yash Khandelwal' via blink-dev <blin...@chromium.org> wrote:
Intent to Prototype: Client-side video editing (MediaBlob)
Contact emails
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/d9b1401f-b385-450e-9a74-ec8e5451f3c0%40chromium.org.
Yash Khandelwal
@Dale,
Have we considered extending WebCodecs to also provide demuxing and muxing ? I
believe this will greatly help building media editing scenarios on top of
WebCodecs.
I looked into external JavaScript libraries for demuxing and muxing. Here are
my observations:
1. For mp4, videojs/mux.js
does not seem to provide a direct API to get the encoded packets.
muxjs.mp4.tools.inspect only provides
the structure of the mp4 file.
2. I also looked into these JavaScript webm demuxers: https://github.com/Yahweasel/mkvdemuxjs
and https://github.com/jscodec/jswebm
These do provide direct access to encoded packets but I found these issues:
a.
For some webm videos, I got "Color not supported" error.
b. Number of packets provided by ffprobe
did not match the number of packets outputted by jscodec/jswebm.
c. Yahweasel/mkvdemuxjs: Timestamps were not accurate. They were resetting back to zero for some videos.
I believe other issues will arise when doing muxing using external JavaScript libraries. Hence, I believe these libraries do not provide a complete solution for web developers.
@Ilya:
Yes, we have validated this approach with Flipgrid. Their earlier approach was editing media
files on the server side and switching to MediaBlob greatly decreased their
turnaround time. Their feedback around WebCodecs and MediaBlob is to put
efforts in a single API for media editing needs that includes shaping WebCodecs
to make it easier editing videos on the client.
Also, we presented MediaBlob API to Media and Entertainment IG
last week and the feedback was positive around the need of a direct and easy to
use API for media editing. However, there was some feedback regarding using a playlist model
instead of the current proposed model for performing these operations.
Switching to this playlist model will make it more in sync with native video
editing suites providing a lot more capabilities than the ones currently
proposed.
MediaBlob was also discussed briefly during the Microsoft Build event last
month and it resonated quite well with the developers.
On Thursday, 4 June 2020 15:17:35 UTC-7, Ilya Grigorik wrote:
Hey folks.On Thu, May 28, 2020 at 5:13 PM 'Yash Khandelwal' via blink-dev <blin...@chromium.org> wrote:Thanks for this.
Follow-up question:For joining, you'd send both streams through the decoder to the encoderFor trim/join, why do we want the data to go through the WebCodecs decode and encode workflow ? Or is it only needed when we want the output in a different encoding than the input ?
I will also create an issue on the WebCodecs explainer to request adding an example demonstrating media editing scenarios.+1 to the above. I would love to see a hands-on example of how WebCodecs and/or mux.js addresses the same use cases, before we make a call in either direction.The developer ergonomics of client-side media (image and video) editing capabilities that we offer today (to the extent that we do) today are, well.. not very good. While I'm excited about WASM and WebCodecs, I don't think we should rule out providing convenience APIs that address common use cases via simple and intuitive APIs.To that end, it would be great to gather some feedback from developers that would rely on it..Yash: I believe you've worked with some teams at MSFT to validate this API? It would be great to get their input and feedback on the alternate paths that we're discussing here.
On Wed, Jun 3, 2020 at 4:53 PM 'Yash Khandelwal' via blink-dev <blin...@chromium.org> wrote:
Here is the tracking bug
https://bugs.chromium.org/p/chromium/issues/detail?id=1090959
On Thursday, 28 May 2020 08:15:37 UTC-7, Joe Medley wrote:
Do you have a tracking bug I can star?If an API's not documented it doesn't exist.
On Tue, May 26, 2020 at 2:41 PM 'Yash Khandelwal' via blink-dev <blin...@chromium.org> wrote:
Intent to Prototype: Client-side video editing (MediaBlob)
Contact emails
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/d9b1401f-b385-450e-9a74-ec8e5451f3c0%40chromium.org.
Ilya Grigorik
@Ilya: Yes, we have validated this approach with Flipgrid. Their earlier approach was editing media files on the server side and switching to MediaBlob greatly decreased their turnaround time. Their feedback around WebCodecs and MediaBlob is to put efforts in a single API for media editing needs that includes shaping WebCodecs to make it easier editing videos on the client.
Also, we presented MediaBlob API to Media and Entertainment IG last week and the feedback was positive around the need of a direct and easy to use API for media editing. However, there was some feedback regarding using a playlist model instead of the current proposed model for performing these operations. Switching to this playlist model will make it more in sync with native video editing suites providing a lot more capabilities than the ones currently proposed.
MediaBlob was also discussed briefly during the Microsoft Build event last month and it resonated quite well with the developers.
On Thursday, 4 June 2020 15:17:35 UTC-7, Ilya Grigorik wrote:
Hey folks.On Thu, May 28, 2020 at 5:13 PM 'Yash Khandelwal' via blink-dev <blin...@chromium.org> wrote:Thanks for this.
Follow-up question:For joining, you'd send both streams through the decoder to the encoderFor trim/join, why do we want the data to go through the WebCodecs decode and encode workflow ? Or is it only needed when we want the output in a different encoding than the input ?
I will also create an issue on the WebCodecs explainer to request adding an example demonstrating media editing scenarios.+1 to the above. I would love to see a hands-on example of how WebCodecs and/or mux.js addresses the same use cases, before we make a call in either direction.The developer ergonomics of client-side media (image and video) editing capabilities that we offer today (to the extent that we do) today are, well.. not very good. While I'm excited about WASM and WebCodecs, I don't think we should rule out providing convenience APIs that address common use cases via simple and intuitive APIs.To that end, it would be great to gather some feedback from developers that would rely on it..Yash: I believe you've worked with some teams at MSFT to validate this API? It would be great to get their input and feedback on the alternate paths that we're discussing here.
On Wed, Jun 3, 2020 at 4:53 PM 'Yash Khandelwal' via blink-dev <blin...@chromium.org> wrote:
Here is the tracking bug
https://bugs.chromium.org/p/chromium/issues/detail?id=1090959
On Thursday, 28 May 2020 08:15:37 UTC-7, Joe Medley wrote:
Do you have a tracking bug I can star?If an API's not documented it doesn't exist.
On Tue, May 26, 2020 at 2:41 PM 'Yash Khandelwal' via blink-dev <blin...@chromium.org> wrote:
Intent to Prototype: Client-side video editing (MediaBlob)
Contact emails
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/8894b66e-a833-4210-af88-ea1331052631o%40chromium.org.
Dale Curtis
I looked into external JavaScript libraries for demuxing and muxing. Here are my observations:
1. For mp4, videojs/mux.js does not seem to provide a direct API to get the encoded packets.
muxjs.mp4.tools.inspect only provides the structure of the mp4 file.
2. I also looked into these JavaScript webm demuxers: https://github.com/Yahweasel/mkvdemuxjs and https://github.com/jscodec/jswebmThese do provide direct access to encoded packets but I found these issues:
a. For some webm videos, I got "Color not supported" error.
b. Number of packets provided by ffprobe did not match the number of packets outputted by jscodec/jswebm.c. Yahweasel/mkvdemuxjs: Timestamps were not accurate. They were resetting back to zero for some videos.
I believe other issues will arise when doing muxing using external JavaScript libraries. Hence, I believe these libraries do not provide a complete solution for web developers.
@Ilya: Yes, we have validated this approach with Flipgrid. Their earlier approach was editing media files on the server side and switching to MediaBlob greatly decreased their turnaround time. Their feedback around WebCodecs and MediaBlob is to put efforts in a single API for media editing needs that includes shaping WebCodecs to make it easier editing videos on the client.
Also, we presented MediaBlob API to Media and Entertainment IG last week and the feedback was positive around the need of a direct and easy to use API for media editing. However, there was some feedback regarding using a playlist model instead of the current proposed model for performing these operations. Switching to this playlist model will make it more in sync with native video editing suites providing a lot more capabilities than the ones currently proposed.
MediaBlob was also discussed briefly during the Microsoft Build event last month and it resonated quite well with the developers.
Yash Khandelwal
Thanks for investigating. Did you look at mp4box.js? It sounds like you've just encountered bugs and/or feature limitations in these libraries, not problems which could not be fixed with some upstream work. Presumably a MediaBlob.js implementation could provide its own mp4, webm, etc muxers and demuxers that don't suffer from such issues.
Thanks for pointing
me to mp4box.js. Using extract.js,
I was able to fetch the encoded packets from the mp4 container and did some
performance comparison on my machine.
For this sample
mp4 file, this library took 1181 ms to extract all 3182 video samples and
2214 ms to extract all 5968 audio samples. Whereas the browser implementation
(using FFmpeg av_read_frame) took less than 100 ms to extract all 9150 audio
and video samples.
The browser implementation is significantly better than these JavaScript
libraries and I do not think we may be able to match it even if we work upon
optimizing these libraries.
By single API did they mean a browser API or just a library which used WebCodecs? I.e., would they be equally happy with a ~20-30kb zipped library like MediaBlob.js that was just as performant and used WebCodecs for transcoding?
Do most developers just want to trim and join segments though? Or are they looking to do more complicated things like effects, cropping, etc. I.e., if we consider a "stories" like feature that is common these days: users record or choose a clip, trim it, apply some effects/stickers/text/etc, and finally submit. The best of the story UIs offer a live preview as this is happening. The current API doesn't seem to provide a preview mechanism, so one would have to repeatedly finalize MediaBlob operations as the user chooses different effects. In a preview case you'd really only want trim + decode + effects w/o finalizing. Which seems more suited to a demuxer + WebCodecs => MediaRecorder / custom muxer approach.
Dale Curtis
Thanks for investigating. Did you look at mp4box.js? It sounds like you've just encountered bugs and/or feature limitations in these libraries, not problems which could not be fixed with some upstream work. Presumably a MediaBlob.js implementation could provide its own mp4, webm, etc muxers and demuxers that don't suffer from such issues.Thanks for pointing me to mp4box.js. Using extract.js, I was able to fetch the encoded packets from the mp4 container and did some performance comparison on my machine.
For this sample mp4 file, this library took 1181 ms to extract all 3182 video samples and 2214 ms to extract all 5968 audio samples. Whereas the browser implementation (using FFmpeg av_read_frame) took less than 100 ms to extract all 9150 audio and video samples.
The browser implementation is significantly better than these JavaScript libraries and I do not think we may be able to match it even if we work upon optimizing these libraries.
By single API did they mean a browser API or just a library which used WebCodecs? I.e., would they be equally happy with a ~20-30kb zipped library like MediaBlob.js that was just as performant and used WebCodecs for transcoding?I meant a single Browser API to handle all media editing scenarios. Based on the above performance data, it does not seems like that using JavaScript libraries will provide the same performance as the browser would.
Do most developers just want to trim and join segments though? Or are they looking to do more complicated things like effects, cropping, etc. I.e., if we consider a "stories" like feature that is common these days: users record or choose a clip, trim it, apply some effects/stickers/text/etc, and finally submit. The best of the story UIs offer a live preview as this is happening. The current API doesn't seem to provide a preview mechanism, so one would have to repeatedly finalize MediaBlob operations as the user chooses different effects. In a preview case you'd really only want trim + decode + effects w/o finalizing. Which seems more suited to a demuxer + WebCodecs => MediaRecorder / custom muxer approach.Yes, the current proposed implementation using finalize will not handle the preview case efficiently.
Dale Curtis
On Mon, Jun 8, 2020 at 6:30 PM 'Yash Khandelwal' via blink-dev <blin...@chromium.org> wrote:Thanks for investigating. Did you look at mp4box.js? It sounds like you've just encountered bugs and/or feature limitations in these libraries, not problems which could not be fixed with some upstream work. Presumably a MediaBlob.js implementation could provide its own mp4, webm, etc muxers and demuxers that don't suffer from such issues.Thanks for pointing me to mp4box.js. Using extract.js, I was able to fetch the encoded packets from the mp4 container and did some performance comparison on my machine.
For this sample mp4 file, this library took 1181 ms to extract all 3182 video samples and 2214 ms to extract all 5968 audio samples. Whereas the browser implementation (using FFmpeg av_read_frame) took less than 100 ms to extract all 9150 audio and video samples.
The browser implementation is significantly better than these JavaScript libraries and I do not think we may be able to match it even if we work upon optimizing these libraries.Can you share the sample code you used to test this? If I go to https://gpac.github.io/mp4box.js/test/filereader.html and load your sample video, it loads instantly.
Yash Khandelwal
I was using the node terminal which might have affected the performance. I am convinced now that demuxing (and probably muxing) can be efficiently handled by JS :)