how to detect finishing all ajax calls?

[headless-man]

any "smart" way to detect finishing all ajax calls? Page.loadEventFired doesn't work,

e.g. google-chrome-unstable --headless --disable-gpu --no-sandbox --hide-scrollbars --screenshot http://html5test.com, showing only spinning, need to way until the ajax will be finish and then take screenshot.

[Anton Bacaj]

Try using the events on Network

https://chromedevtools.github.io/devtools-protocol/tot/Network/#event-responseReceived

You can filter by type including XHR(Ajax).

You need to implement this yourself because remember how load event works. Load events works for resources that are defined on the HTML page, so any XHR that comes after those resources is not considered part of that event.

On May 20, 2017 9:15 AM, "headless-man" <bakht...@gmail.com> wrote:

any "smart" way to detect finishing all ajax calls? Page.loadEventFired doesn't work,

e.g. google-chrome-unstable --headless --disable-gpu --no-sandbox --hide-scrollbars --screenshot http://html5test.com, showing only spinning, need to way until the ajax will be finish and then take screenshot.

--
You received this message because you are subscribed to the Google Groups "headless-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev+unsubscribe@chromium.org.
To post to this group, send email to headle...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/1d0acefc-027f-408f-838d-a2734584fab1%40chromium.org.

[headless-man]

How to recognize is it last response received or left?

i did something like this, but didn't help

    Network.requestWillBeSent(params => {
        requestCount++;
    });

    Network.loadingFailed(params => {
        requestCount--;
    });

    Network.loadingFinished(params => {
        requestCount--;

      if (requestCount === 0) {
        console.info('are all requests completed?');
      }
    });

On Sunday, May 21, 2017 at 2:21:41 AM UTC+5, Anton Bacaj wrote:

Try using the events on Network

https://chromedevtools.github.io/devtools-protocol/tot/Network/#event-responseReceived

You can filter by type including XHR(Ajax).

You need to implement this yourself because remember how load event works. Load events works for resources that are defined on the HTML page, so any XHR that comes after those resources is not considered part of that event.

On May 20, 2017 9:15 AM, "headless-man" <bakht...@gmail.com> wrote:

any "smart" way to detect finishing all ajax calls? Page.loadEventFired doesn't work,

e.g. google-chrome-unstable --headless --disable-gpu --no-sandbox --hide-scrollbars --screenshot http://html5test.com, showing only spinning, need to way until the ajax will be finish and then take screenshot.

--
You received this message because you are subscribed to the Google Groups "headless-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev...@chromium.org.

[Anton Bacaj]

I am trying this now, seems there might be a bug in the way the responseRecevied / loadFinished events are being triggered.

When I run `network.requestWillBeSent` and add a variable it amounts to 29, when I count how many loaded (separate variable) it amounts to 26.

I also have a event listener for `network.loadingFailed` but there are no failures. 

Problem is that there are 29 requests sent and only 26 are loadingFinished. Is there a reason why those aren't == to each other?

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev+unsubscribe@chromium.org.

To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/1e2a76d4-cc15-4407-883e-deb4cdedd762%40chromium.org.

[headless-man]

yeah, i've such experiences, but can chrome team confirm that it's bug or not?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/1e2a76d4-cc15-4407-883e-deb4cdedd762%40chromium.org.

[Anton Bacaj]

Yea I don't know if we are just using the API wrong or there is a bug. 

Either way, need some official response.

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev+unsubscribe@chromium.org.

To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/a37fadb4-7565-4769-8a20-d546b9c6eb6f%40chromium.org.

[Raffaele Sena]

I have just tried, and you may just need to listen to more events.

- if the request causes a redirect you will get TWO or more "requestWillBeSent" (one for the original request plus one per redirect). They all have the same "requestId" and there will be only one associated "responseReceived" event.

- if the request is cached you will get a "requestServedFromCache" event instead of "responseReceived" (and no "loadingFinished" event)

So, my suggestion is that instead of counting events by type, you add and remove events from a map (indexed by requestId), and at the end you check if the map is empty.

-- Raffaele

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/CAK%3DPu0AOM-iKX24FQNyo4ZhdLf3qDYpf9kQppZQE7B8twNroxQ%40mail.gmail.com.

[Anton Bacaj]

Thanks for pointing those out Raffaele! I tried it and it you are correct, the map will be empty because the requests correlate with the id.

The problem in this specific case with html5test.com, even if we wait for all ajax to complete - the page is still not ready because the page is executing javascript code and rendering new html.

It is not enough to just wait for all ajax to complete, because the page is rendering - we do not know how long it will take for that javascript / html to execute so the screenshot still returns with a loading spinner.

[headless-man]

Btw, what's the difference between responseReceived & loadingFinsihed?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/a37fadb4-7565-4769-8a20-d546b9c6eb6f%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "headless-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev...@chromium.org.
To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/CAK%3DPu0AOM-iKX24FQNyo4ZhdLf3qDYpf9kQppZQE7B8twNroxQ%40mail.gmail.com.

[Raffaele Sena]

My guess is that you get responseReceived after the client gets the response headers and loadingFinished after the full response is read (so if you are downloading a large file you should see a reasonably large interval between the two events)

Sent from my iPad

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/0c312001-396a-44b8-be49-0eadd3f60f14%40chromium.org.

[Alex Clarke]

Looking at the code, responseReceived is issued when the ResourceMsg_ReceivedResponse IPC is sent to the renderer.  That happens when the headers are available for a resource request.

The loadingFinsihed event is issued when the ResourceMsg_RequestComplete IPC is sent to the renderer. That happens when the the full resource has been downloaded by the browser process.

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev+unsubscribe@chromium.org.

To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/0c312001-396a-44b8-be49-0eadd3f60f14%40chromium.org.

[headless-man]

Thanks

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/0c312001-396a-44b8-be49-0eadd3f60f14%40chromium.org.

[headless-man]

maybe --deterministic-fetch & --virtual-time-budget options help?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/a37fadb4-7565-4769-8a20-d546b9c6eb6f%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "headless-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev...@chromium.org.
To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/CAK%3DPu0AOM-iKX24FQNyo4ZhdLf3qDYpf9kQppZQE7B8twNroxQ%40mail.gmail.com.

[Anton Bacaj]

Have you been able to figure this out yet? I still cannot get the page screenshot without a hard timer.

Needs a smarter way to wait for page execution.

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev+unsubscribe@chromium.org.

To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/609440c8-1ebd-4779-a275-d0ffd6397d32%40chromium.org.

[headless-man]

no, but with --virtual-time-budget option did a bit, it's also some kind of timer

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/609440c8-1ebd-4779-a275-d0ffd6397d32%40chromium.org.

[Jeff Tierney]

one option could be to inject some code into the page to be executed that checks for the existence of a certain DOM element that you know will only exist once the page has fully finished loading and has rendered like so:

function checkLoadedAndScreenshot() {

  myClient.Runtime.evaluate({

    // need to cast result to a bool here because you can't return

    // HTML elements in these responses

    expression: '!!document.querySelector("#someElementThatOnlyExistsAfterLastRender")',

    returnByValue: true,

  }).then((clientResult) => {

    if (!clientResult.result.value) {

      // selector returned null... page not rendered yet... try again shortly

      setTimeout(checkLoadedAndScreenshot, 100); 

    } else {

      // we have the element that we were looking for...

      // go ahead and take the screenshot

      takeScreenshot();

    }

  });

}

you'd need to be careful to limit either the number of times it tries to check, or set some overall timeout on that though, because if some loading error occurs and the page never ends up loading completely and rendering, you'd get stuck in this infinite loop of checking for something that will never exist.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/CAK%3DPu0D0%3DY2w1Hhp%3DeHOLbB7RxPMJoV%2BcmnqLbpwukbctUQdLA%40mail.gmail.com.

[Anton Bacaj]

Exactly my thoughts this is just like writing automated tests.

I have an idea of how it can work, a threshold for the maximum wait and polling the Dom. This way we can avoid any network inspection.

[Simon Luetzelschwab]

I'm using a slightly different approach taking advantage of the (experimental) screencast feature.

The basic idea is to take a screenshot after the page hasn't changed for some fix timeout.

Here's a fully working example for html5test using the latest build with a 1sec timeout -

const fs = require('fs');

const CDP = require('chrome-remote-interface');

const URL = 'http://html5test.com';

const TIMEOUT = 1 * 1000; // 1sec

const FILENAME = 'html5test.png';

CDP(async (client) => {

  const {Network, Page} = client;

  let timeoutId; 

  Page.screencastFrame(async (params) => {

    if (timeoutId) {

      clearTimeout(timeoutId);

      console.log('still loading');

    }

    timeoutId = setTimeout(() => {

      fs.writeFileSync(FILENAME, params.data, 'base64'); 

      console.log('written screenshot and exiting');

      client.close();

    }, TIMEOUT);

    //console.log(params, params.data, params.metadata);

    await Page.screencastFrameAck({sessionId: params.sessionId});

  });

  await Promise.all([

    Network.enable(),

    await Page.enable()]);

  await Page.navigate({url: URL});

  await Page.startScreencast({format: 'png'});

}).on('error', (err) => {

  console.log(err);

});

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/CAK%3DPu0DA%3D4hS3CO%3DupMbvPkN0FdDShxCDh0oGcB30SV-p%3DDnng%40mail.gmail.com.

--

Alpeware LLC - 548 Market St #35286, San Francisco, CA 94104 - +1 415 200 3094

[Raffaele Sena]

For the specific case, it seems that the last call is always for "http://html5test.com/api/submit", so you could wait until that requests complete + a few more seconds and then take the screenshot.

Here is an example:

frameStartedLoading 27648.1

navigating to 27648.1

waiting...

requestWillBeSent 27648.66 Document http://html5test.com/

responseReceived 27648.66

frameNavigated

loadingFinished 27648.66

   0

requestWillBeSent 27648.67 Other http://html5test.com/css/main.css

requestWillBeSent 27648.68 Other http://html5test.com/scripts/base.js

requestServedFromCache 27648.68

   1

responseReceived 27648.68

loadingFinished 27648.68

   1

requestWillBeSent 27648.69 Other http://html5test.com/scripts/8/engine.js

requestServedFromCache 27648.69

   1

responseReceived 27648.69

loadingFinished 27648.69

   1

requestWillBeSent 27648.70 Other http://html5test.com/scripts/8/data.js

requestServedFromCache 27648.70

   1

responseReceived 27648.70

loadingFinished 27648.70

   1

responseReceived 27648.67

loadingFinished 27648.67

   0

requestWillBeSent 27648.75 Other http://api.whichbrowser.net/rel/detect.js?ua=Mozilla%2F5.0%20(Macintosh%3B%20Intel%20Mac%20OS%20X%2010_12_6)%20AppleWebKit%2F537.36%20(KHTML%2C%20like%20Gecko)%20HeadlessChrome%2F60.0.3107.0%20Safari%2F537.36&e=48&f=127&r=qr5pwk&w=1680&h=1050

requestWillBeSent 27648.76 Other http://html5test.com/fonts/html5test.woff

requestServedFromCache 27648.76

   1

responseReceived 27648.76

loadingFinished 27648.76

   1

requestWillBeSent 27648.78 Other http://html5test.com/fonts/leaguegothic-regular-webfont.woff

requestServedFromCache 27648.78

   1

responseReceived 27648.78

loadingFinished 27648.78

   1

requestWillBeSent 27648.80 Other http://html5test.com/images/ht...@2x.png

requestServedFromCache 27648.80

   1

responseReceived 27648.80

loadingFinished 27648.80

   1

requestWillBeSent 27648.81 Other http://html5test.com/images/sponsors/clou...@2x.png

requestServedFromCache 27648.81

   1

responseReceived 27648.81

loadingFinished 27648.81

   1

domContentEventFired

responseReceived 27648.75

loadingFinished 27648.75

   0

loadEventFired

frameStoppedLoading 27648.1

requestWillBeSent 27648.82 Other http://html5test.com/

requestWillBeSent 27648.83 XHR http://html5test.com/assets/detect.html?7i0jn

requestWillBeSent 27648.84 XHR http://html5test.com/assets/detect.html?v7f90

requestWillBeSent 27648.85 XHR http://html5test.com/assets/detect.html?804ga

requestWillBeSent 27648.86 XHR http://html5test.com/assets/detect.html?uarow

frameStartedLoading 27648.5

requestWillBeSent 27648.87 Document http://html5test.com/assets/csp.html

requestWillBeSent 27648.88 Other data:text/javascript;charset=utf-8,window

requestServedFromCache 27648.88

   6

responseReceived 27648.88

loadingFinished 27648.88

   6

requestWillBeSent 27648.90 Other http://s3.buysellads.com/ac/bsa.js

requestWillBeSent 27648.91 Other http://www.google-analytics.com/ga.js

responseReceived 27648.82

loadingFinished 27648.82

   7

responseReceived 27648.87

frameNavigated

loadingFinished 27648.87

   6

responseReceived 27648.90

loadingFinished 27648.90

   5

requestWillBeSent 27648.91 Other https://www.google-analytics.com/ga.js

  redirectReponse http://www.google-analytics.com/ga.js

requestWillBeSent 27648.92 Other http://s3.buysellads.com/r/s_6c18c40d896c427f049e74e4c708ab51.js?v=1495479600000

frameStoppedLoading 27648.5

responseReceived 27648.91

loadingFinished 27648.91

   5

responseReceived 27648.92

loadingFinished 27648.92

   4

requestWillBeSent 27648.93 Other http://s3.buysellads.com/ac/pro.js

responseReceived 27648.93

loadingFinished 27648.93

   4

requestWillBeSent 27648.94 Other http://srv.buysellads.com/ads/get/ids/CWSICKV;CWSICKE/?r=1495479600000

responseReceived 27648.83

loadingFinished 27648.83

   4

responseReceived 27648.94

loadingFinished 27648.94

   3

responseReceived 27648.85

loadingFinished 27648.85

   2

requestWillBeSent 27648.95 Other https://s3.buysellads.com/1236118/204861-1389626817.jpg

responseReceived 27648.86

loadingFinished 27648.86

   2

responseReceived 27648.95

loadingFinished 27648.95

   1

responseReceived 27648.84

loadingFinished 27648.84

   0

requestWillBeSent 27648.96 XHR http://html5test.com/api/submit

responseReceived 27648.96

loadingFinished 27648.96

   0

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/CAHKjGjrFpTm_%2B48b-tbd7udRdG%3D4zZjv57GDC3wz_51SM%3DGgOg%40mail.gmail.com.

[Anton Bacaj]

While I agree that waiting is a good idea Raffaele, that approach is not very deterministic. We do not know how long to wait, how do you decide? Some random time?

Would be better to poll the DOM and wait for a maximum threshold i.e 10seconds, polling every 1second similar to what Selenium web driver does.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/CANKfucanOy7PF4ah7gaXnWhdsmeU4%3D_EqzbAJ%3D5Pr_%3DmJ%3D3Tmg%40mail.gmail.com.

[Alex Clarke]

Virtual time might help here. The idea is that while there are any network transactions outstanding DOM timers do not fire.  Depending on your content 2-10 virtual seconds should be sufficient for the page to fully load.

I'm not super familiar with the node bindings but the commands should be of the form:

Emulation.enable();
Emulation.setVirtualTimePolicy({policy: pauseIfNetworkFetchesPending, budget, 5000})

You can wait till you receive an Emulation.virtualTimeBudgetExpired event.

Here is an example:

requestWillBeSent 27648.80 Other http://html5test.com/images/htm...@2x.png

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/CAK%3DPu0DoAnXUZLPN7Khhwa2k8%2BhEnP9E3rwWnAwwa-EYTqMV5A%40mail.gmail.com.

[headless-man]

maybe directly passing --virtual-time-budget instead of programmically?

Here is an example:

requestWillBeSent 27648.80 Other http://html5test.com/images/ht...@2x.png

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/609440c8-1ebd-4779-a275-d0ffd6397d32%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "headless-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev...@chromium.org.
To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/CAK%3DPu0D0%3DY2w1Hhp%3DeHOLbB7RxPMJoV%2BcmnqLbpwukbctUQdLA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "headless-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev...@chromium.org.
To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/CAK%3DPu0DA%3D4hS3CO%3DupMbvPkN0FdDShxCDh0oGcB30SV-p%3DDnng%40mail.gmail.com.

--

Alpeware LLC - 548 Market St #35286, San Francisco, CA 94104 - +1 415 200 3094

--
You received this message because you are subscribed to the Google Groups "headless-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev...@chromium.org.
To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/CAHKjGjrFpTm_%2B48b-tbd7udRdG%3D4zZjv57GDC3wz_51SM%3DGgOg%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "headless-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev...@chromium.org.
To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/CANKfucanOy7PF4ah7gaXnWhdsmeU4%3D_EqzbAJ%3D5Pr_%3DmJ%3D3Tmg%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "headless-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev...@chromium.org.
To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/CAK%3DPu0DoAnXUZLPN7Khhwa2k8%2BhEnP9E3rwWnAwwa-EYTqMV5A%40mail.gmail.com.

[Raffaele Sena]

I am very confused on how setVirtualTimePolicy works. I am testing on Chrome canary on MacOS (HeadlessChrome/60.0.3108.0) and I am getting very confusing results:

- if policy is NOT pauseIfNetworkFetchesPending but budget is set, the "Emulation.virtualTimeBudgetExpired" fires right away (I would have expected NOT to fire)

- if policy IS pauseIfNetworkFetchesPending, the event fires at random times. Some times it seems to fire right away (or as soon as one network request completes) independent of the "budget" value, some times it seems to wait until the timeout (budget value) expires (i.e. if I set to 20000, for 20 seconds, it will go through all the network requests and then wait until the 20 seconds (or close to that) pass, after which the event fires.

How is this really supposed to work ?

thanks!

-- Raffaele

[Alex Clarke]

On 23 May 2017 at 23:59, Raffaele Sena <raf...@gmail.com> wrote:

I am very confused on how setVirtualTimePolicy works. I am testing on Chrome canary on MacOS (HeadlessChrome/60.0.3108.0) and I am getting very confusing results:

- if policy is NOT pauseIfNetworkFetchesPending but budget is set, the "Emulation.virtualTimeBudgetExpired" fires right away (I would have expected NOT to fire)

Virtual time is described a bit more here and timer fast forwarding basically means all the budget will get gobbled up immediately.

 

- if policy IS pauseIfNetworkFetchesPending, the event fires at random times. Some times it seems to fire right away (or as soon as one network request completes) independent of the "budget" value, some times it seems to wait until the timeout (budget value) expires (i.e. if I set to 20000, for 20 seconds, it will go through all the network requests and then wait until the 20 seconds (or close to that) pass, after which the event fires.

Cutting and pasting from one of the comments related to pauseIfNetworkFetchesPending

    // In this policy virtual time is allowed to advance unless there are
    // pending network fetches associated any child WebFrameScheduler, or a
    // document is being parsed on a background thread. Initially virtual time
    // is not allowed to advance until we have seen at least one load. The aim
    // being to try and make loading (more) deterministic.

I'd suggest opening a new tab with about:blank, then set pauseIfNetworkFetchesPending with a suitable budget (say 5000 ms) and then issue a Page.navigate to open the desired web content.  Depending on what the page is doing you may need more or less virtual time budget for the page to consistently load to the same place.

As a side note determining when a page is loaded is actually very tricky, some pages load new content when you scroll down (perhaps infinitely), others continuously stream in data e.g. to display a stock ticker.  

[headless-man]

tested with the snippet and got empty blank image.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/609440c8-1ebd-4779-a275-d0ffd6397d32%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "headless-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev...@chromium.org.
To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/CAK%3DPu0D0%3DY2w1Hhp%3DeHOLbB7RxPMJoV%2BcmnqLbpwukbctUQdLA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "headless-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev...@chromium.org.
To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/CAK%3DPu0DA%3D4hS3CO%3DupMbvPkN0FdDShxCDh0oGcB30SV-p%3DDnng%40mail.gmail.com.