Capture CSS Animations into video with frame precision

[Andrey Utkin]

Hi web engine developers,

We want to render arbitrary CSS animations into video file, on our servers.
This is to provide CSS animations effects to viewers which don't support CSS
animation (e.g. IE 9).

To best of my understanding, just running a browser in Xvfb and recording the
screen is not a perfect solution, because moments at which screenshots are
taken won't perfectly align with the moments at which browser updates rendered
image, which will result in skewed animation speed.

So what seems to be the right thing to do is to build an application which
gives HTML and CSS code as input to web browser engine, and also controls time
flow in browser engine by triggering clock events. This way, animation
rendering engine renders just frames we want to grab, exactly when our app is
ready to consume it, and rendering engine waits for our app for an order to
render next frame.

Currently I am considering WebKit, Blink and Gecko engines, the question is
which of them allows to achieve above with least effort.

Any practical hints are highly appreciated.

If you would consider helping with this on paid basis, please let me know.

[Eric Seckler]

Hi Andrey,

We're working on a deterministic rendering control for Headless Chrome (see https://crbug.com/646774), which will allow you to take screenshots at given points in time.

We've got a first version of it in TOT, but at this point it doesn't yet allow fully-deterministic rendering in all circumstances. That said, we're working on combining this with virtual time to render animations deterministically at emulated points in time. Feel free to watch the bug for updates.

Cheers,

Eric

--
You received this message because you are subscribed to the Google Groups "headless-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev...@chromium.org.
To post to this group, send email to headle...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/20171023102441.GA29334%40UK-EOV-NEX0BOC-S.pb.com.

[Andrey Utkin]

Thanks Eric, that's awesome.

Running headless Chrome and executing DevTools protocol command HeadlessExperimental.beginFrame() fetching the screenshots seems like the way to go.

Now I'm stuck at finding a tool to send such protocol command. What are you Eric using to test these protocol commands? Or are you utilizing just headless/lib/headless_web_contents_browsertest.cc ?

Unfortunately chrome-remote-interface misses HeadlessExperimental domain. I have patched its lib/protocol.json and chrome-remote-interface.js with what I found here https://github.com/ChromeDevTools/devtools-protocol/blob/master/json/browser_protocol.json#L4031 , but HeadlessExperimental still doesn't show up in CRI shell, and commands result in errors:

>>> HeadlessExperimental.enable();
ReferenceError: HeadlessExperimental is not defined
at repl:1:1
at ContextifyScript.Script.runInContext (vm.js:53:29)
at REPLServer.defaultEval (repl.js:241:29)
at bound (domain.js:301:14)
at REPLServer.runBound [as eval] (domain.js:314:12)
at REPLServer.onLine (repl.js:433:10)
at emitOne (events.js:115:13)
at REPLServer.emit (events.js:210:7)
at REPLServer.Interface._onLine (readline.js:278:10)
at REPLServer.Interface._line (readline.js:625:8)

I am running it against headless session of Chrome 63.0.3236.0.

I am currently setting up "chromewhip" as it seems to support these protocol commands, will see how it works.
________________________________________
From: Eric Seckler <esec...@google.com>
Sent: 23 October 2017 11:52:21
To: Andrey Utkin; blin...@chromium.org; graphi...@chromium.org; headle...@chromium.org
Cc: Andrey Utkin
Subject: Re: Capture CSS Animations into video with frame precision

Hi Andrey,

We're working on a deterministic rendering control for Headless Chrome<https://developers.google.com/web/updates/2017/04/headless-chrome> (see https://crbug.com/646774), which will allow you to take screenshots at given points in time.

We've got a first version of it in TOT, but at this point it doesn't yet allow fully-deterministic rendering in all circumstances. That said, we're working on combining this with virtual time<https://docs.google.com/document/d/1y9KDT_ZEzT7pBeY6uzVt1dgKlwc1OB_vY4NZO1zBQmo/edit#heading=h.tn3gd1y9ifml> to render animations deterministically at emulated points in time. Feel free to watch the bug for updates.

Cheers,
Eric

On Mon, Oct 23, 2017 at 11:24 AM Andrey Utkin <andrey...@fastmail.com<mailto:andrey...@fastmail.com>> wrote:
Hi web engine developers,

We want to render arbitrary CSS animations into video file, on our servers.
This is to provide CSS animations effects to viewers which don't support CSS
animation (e.g. IE 9).

To best of my understanding, just running a browser in Xvfb and recording the
screen is not a perfect solution, because moments at which screenshots are
taken won't perfectly align with the moments at which browser updates rendered
image, which will result in skewed animation speed.

So what seems to be the right thing to do is to build an application which
gives HTML and CSS code as input to web browser engine, and also controls time
flow in browser engine by triggering clock events. This way, animation
rendering engine renders just frames we want to grab, exactly when our app is
ready to consume it, and rendering engine waits for our app for an order to
render next frame.

Currently I am considering WebKit, Blink and Gecko engines, the question is
which of them allows to achieve above with least effort.

Any practical hints are highly appreciated.

If you would consider helping with this on paid basis, please let me know.

--
You received this message because you are subscribed to the Google Groups "headless-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev...@chromium.org<mailto:headless-dev%2Bunsu...@chromium.org>.
To post to this group, send email to headle...@chromium.org<mailto:headle...@chromium.org>.

[Eric Seckler]

I believe chrome-remote-interface is what a lot of folks are using, I'm not sure how to update its domains though. Headless also exposes a C++ API for sending DevTools commands (which is used in the browsertests). If nothing else works, the protocol is based on fairly simple JSON messages that you could build/parse yourself, too.

Make sure you're running against a current ToT build of chromium - these commands are very new :)

[Sami Kyostila]

In theory chrome-remote-interface should query the protocol from the target browser automatically, but I've had problems with that before. Try instead using this script for updating the protocol description:

https://github.com/cyrus-and/chrome-remote-interface/blob/master/scripts/update-protocol.sh

- Sami

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev...@chromium.org.
To post to this group, send email to headle...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/CAHZJZiGApJbCZujb1P-GB7MgmO%3DaaeD8agT4N%2BdFkhZGTdn4zQ%40mail.gmail.com.

[Andrey Utkin]

Thanks Eric, Sami,

I have been able to use HeadlessExperimental DevTools API after I switched to latest Chromium.

With repeated calls to HeadlessExperimental.beginFrame() indeed I can get screenshots. But I don't get any useful results when I pass 'frameTime' parameter - i get "mainFrameContentUpdated: true" in returned object and picture stays the same on all screenshots.

I guess this is expected due to rendering at "emulated time" being not supported yet according to Eric. But if that's just my wrong usage of API, please let me know.

So now I can record screenshots at real time, but not faster. And also possibly with no perfect precision of timestamps: "then()" of beginFrame() call from interval timer deviates from perfect by up to 100 ms, don't know how aligned is the actual time at which frame rendering is happening.

Happy to track development of your feature, hope in future what I am trying to achieve will be completely possible.