Screencast frame doesn't trigger every frame as expected

[Cédric Nirousset]

Hello,

I'm not sure if it's a bug or if I can find a workaround, so I'm starting here.

I'm using Chrome Canary 67 in MacOs, starting with this command :

chrome --headless --no-sandbox --hide-scrollbars --remote-debugging-address=127.0.0.1 --remote-debugging-port=9222 --disable-setuid-sandbox

I'm trying to build a system where a node script will trigger function to update the HTML, and then take a screenshot of it.

I implemented 2 different script to compare screenshot and screencast.

With screenshot, there is no problem, but it's quite slow.

With screencast, there is some missing frame, that never comes to Page.screencastFrame.

I attached the code sample to be simpler to test.

Here is how I send a function to the HTML to update it:

    return client.Runtime.evaluate({
        expression: 'showFrame(' + currentFrame + ');',
        awaitPromise: true
    }).then(function(showRes) {
        if (!showRes.result) {
            throw new Error('Failed showFrame ' + currentFrame);
        }
        return saveScreenshot(client, width, height, currentFrame);
    });

For the screenshot, here is the code :

    return Page.captureScreenshot({
        format: 'png',
        clip: {
            x: 0,
            y: 0,
            width: width,
            height: height,
            scale: 1
        }
    }).then(function(screenshot) {
        return new Buffer(screenshot.data, 'base64');
    }).then((buffer) => {
        let outputFile = filePath + formatFrameNumber(currentFrame) + '.png';
        return new Promise((resolve, reject) => {
            fs.writeFile(outputFile, buffer, 'base64', async (err) => {
                if (err) {
                    console.error(err);
                    reject();
                } else {
                    console.log('Screenshot saved to "' + outputFile + '"');
                    resolve();
                }
            });
        });
    }).catch((err) => {
       console.error('ERORO WRITE BUFFER '+err);
    });

For screencast, it's requested with this code just after the load:

            Page.screencastFrame(({ sessionId, data, metadata }) => {
                console.log('screencastFrame');
                lastScreencastData = data;
                lastScreencastMetadata = metadata;
                Page.screencastFrameAck({ sessionId });
            });

            await Page.startScreencast({
                format: 'png'
            });

The last screencastFrame is saved in a global variable, and then used when saving is requested:

    return new Promise((resolve, reject) => {
        if (!lastScreencastData || !lastScreencastMetadata) {
            throw new Error('screencast frame '+currentFrame+' is empty');
            reject();
        }

        resolve(new Buffer(lastScreencastData, 'base64'));
    }).then((buffer) => {
        lastScreencastData = false;
        let outputFile = filePath + formatFrameNumber(currentFrame) + '.png';
        return new Promise((resolve, reject) => {
            fs.writeFile(outputFile, buffer, 'base64', async (err) => {
                if (err) {
                    console.error(err);
                    reject();
                } else {
                    console.log('Screenshot saved to "' + outputFile + '"');
                    resolve();
                }
            });
        });
    }).catch((err) => {
       console.error('ERORO WRITE BUFFER '+err);
    });

In some cases, there is error with "screencast frame XXX is empty".

In some other cases, the resulting PNG is the same as the previous one.

There is 2 screencastFrame which looks exactly the same.

In the HTML, here is the code called by the node script:

async function showFrame(frame) {
    frameDiv.innerHTML+= ' '+frame;
    console.log('show frame '+frame);
    return new Promise(function(resolve, reject) {
        requestAnimationFrame(function() {
            requestAnimationFrame(function() {
                console.log('show frame END '+frame);
                resolve({
                    result: true
                });
            });
        });
    });
};

As you can see, I already add a double requestAnimationFrame, but it's still not sufficient for screencasting...

The difference of timing between the 2 scripts :

• 8 seconds for screenshot
  
• 1 second for screencast
  

For screen casting, if I add a setTimeout in the HTML of 100ms, there is no frame missing or problem.

But I'd like to avoid this problem in order to be as fast as possible...

Could anybody already experience same problem and found any solutions?

Thanks in advance.

[Cédric Nirousset]

Here is the code for the 2 different node script

[Eric Seckler]

+headless-dev, bcc chromium-discuss.

Screencasts are best-effort, i.e. you'll get the frame that chromium managed to produce in the browser process. If your machine is slowish, this may well be a stale frame.

Page.captureScreenshot avoids this problem, but as you've discovered, is more expensive.

Alternatively, there's a more or less experimental rendering mode that ensures that each produced frame is up-to-date. It's exposed in the HeadlessExperimental domain right now, see this test for an example [1] and bit.ly/headless-rendering for some background reading.

[1] https://cs.chromium.org/chromium/src/headless/lib/headless_web_contents_browsertest.cc?sq=package:chromium&l=1103

--
--
Chromium Discussion mailing list: chromium...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-discuss

[Cédric Nirousset]

Thanks Eric for your quick answer.

After reading the linked documents and testing, I ended with this gotchas:

1. As mentioned in the doc, enableBeginFrameControl on Target.createTarget() is not working on MacOs.
Moreover, this should be activated while creating the target in order to make beginFrame enabled and working.

2. In this mode, requestAnimationFrame doesn't seem to be usable. I fixed it by executing this dirty code before starting doing anything related to my stuff in the HTML page:

await Runtime.evaluate({
  expression: 'window.requestAnimationFrame = function(clb) {clb();};',
  awaitPromise: true
});

Am I missing something or is it intended?

3. If you want to use the screenshot param of the beginFrame, it doesn't seem to work on the first one. Thus, before starting my frame loop, I start with an initial beginFrame to let the next one working with screenshotData.
As far as I can see, the linked test seems to indicate that is intended and/or the expected behavior.

4. For my first try, I didn't want to use the screenshot param for beginFrame and still use the screencast.
But when receiving the result of beginFrame, I have to wait and multiple callback in order to be sure the frame was painted.
Moreover, the screencastAck was always called AFTER the last beginframe result with hasDamage = false.
Here is the code I used in this try (working)

const takeScreenshots = async (client, width, height, currentFrame, totalFrameCount) => {

    return client.Runtime.evaluate({
        expression: 'showFrame(' + currentFrame + ');',
        awaitPromise: true
    }).then(function(showRes) {

        return waitNoDamageAndScreencast(client);
    }).then(function() {
        return saveLastScreencast(client, width, height, currentFrame);
    }).then(function() {
        currentFrame++;
        if (currentFrame < totalFrameCount) {
            return takeScreenshots(client, width, height, currentFrame, totalFrameCount);
        }
        return true;
    });
};

const waitScreencast = async () => {

  return new Promise((resolve, reject) => {

    if (lastScreencastData) {
      resolve(lastScreencastData);
    } else {
      console.log('wait for screencast');
      return delay(10).then(waitScreencast).then(resolve)
    }
  });
};


const waitNoDamageAndScreencast = async (client) => {
  return client.HeadlessExperimental.beginFrame()
   .then(function({hasDamage}) {
     if (!hasDamage) {
       return true;
     }
     console.log('damaged');
     return waitNoDamageAndScreencast(client);
   }).then(waitScreencast);
};

This worked, but as you can see, it's not very efficient because there is a loop of test and a delay of 10ms...

As far as I can see and understand, there is no way to listen for a frame to be completely painted?

5. Regarding the timing, on my debian machine, I now have these result :

• Screenshot version: 4.117s (no image missing)
  
• initial screencast version: 0.512s (but missing frames, the script might be executed too fast regarding Chrome rendering)
  
• Screencast with waiting loop: 2.578s (no image missing)
  
• beginFrame using screenshot param: 2.392s (no image missing)
  

So it seems the usage of beginFrame with screenshot param is better than simple captureScreenshot, by a factor of almost 2.
I guess I will use this version for now.

6. I read that the "hasDamage" might be removed in the future.
Why is that? If you remove it, do you intend to add an event to listen when a frame have been completly painted?