Can this be used to build a crawler?

[vvowns]

Hi, I am looking for ways to use chrome headless for crawling a website, in an efficient way.

From my first experiences with chrome-remote-interface, it appears that spinning up 10 tabs, then navigating inside them in parallel (to a local website) leads to not so good performance.

What I have been doing:

- create ten tabs

- disallow jpg, css, js ... (setBlockedUrls)

- take a start url and load it in first tab

- wait for domContentEventFired

- get all links from that page

- push all links as new tasks in a queue handled by async

- then the process is the same and will try to use all tabs

When I do this, I reach a maximum number of tasks being done every second. Mostly because

loading the page and waiting for domContentLoaded can take from 80ms to 2s as soon as I try to use

multiple tabs in parallel. Same for getting all the links, it can take up to 50ms.

Any comment, feedback or remarks on what I am doing is welcomed. Thanks.

[Isaac Dawson]

I'm not sure this is the correct list for this, but I had similar challenges and spoke about them at a conference last year: https://youtu.be/aqeBM9Q3aY8 

--
You received this message because you are subscribed to the Google Groups "headless-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev...@chromium.org.
To post to this group, send email to headle...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/fc32b28f-a77a-4ebf-8d1d-e780d2a25602%40chromium.org.

[Sami Kyostila]

This is interesting. Could you using the --trace-startup and, say, --trace-startup-duration=30 command line flags to record some performance traces and attach them to a bug? I suspect since all tabs are trying to render animations in parallel, they'll end up using a lot of CPU time in total.

- Sami

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/CANdP%3DScMGnkFKs25UcUxNxUzoB48-F7MCW1uEK2g5eJf28_utA%40mail.gmail.com.

[Anton Bacaj]

You said you disable CSS, and JS... Isn't that the purpose of even using Chrome Headless? 

To me, it sounds like you can crawl a webpage quiet easily without a browser, just use a http library etc

On Apr 26, 2017 1:08 PM, "Sami Kyostila" <skyo...@chromium.org> wrote:

This is interesting. Could you using the --trace-startup and, say, --trace-startup-duration=30 command line flags to record some performance traces and attach them to a bug? I suspect since all tabs are trying to render animations in parallel, they'll end up using a lot of CPU time in total.

- Sami

ke 26. huhtik. 2017 klo 15.09 Isaac Dawson <isaac....@gmail.com> kirjoitti:

I'm not sure this is the correct list for this, but I had similar challenges and spoke about them at a conference last year: https://youtu.be/aqeBM9Q3aY8 

On Wed, Apr 26, 2017 at 10:56 PM vvowns <wou...@gmail.com> wrote:

Hi, I am looking for ways to use chrome headless for crawling a website, in an efficient way.

From my first experiences with chrome-remote-interface, it appears that spinning up 10 tabs, then navigating inside them in parallel (to a local website) leads to not so good performance.

What I have been doing:

- create ten tabs

- disallow jpg, css, js ... (setBlockedUrls)

- take a start url and load it in first tab

- wait for domContentEventFired

- get all links from that page

- push all links as new tasks in a queue handled by async

- then the process is the same and will try to use all tabs

When I do this, I reach a maximum number of tasks being done every second. Mostly because

loading the page and waiting for domContentLoaded can take from 80ms to 2s as soon as I try to use

multiple tabs in parallel. Same for getting all the links, it can take up to 50ms.

Any comment, feedback or remarks on what I am doing is welcomed. Thanks.

--
You received this message because you are subscribed to the Google Groups "headless-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev+unsubscribe@chromium.org.

To post to this group, send email to headle...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/fc32b28f-a77a-4ebf-8d1d-e780d2a25602%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "headless-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev+unsubscribe@chromium.org.

To post to this group, send email to headle...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/CANdP%3DScMGnkFKs25UcUxNxUzoB48-F7MCW1uEK2g5eJf28_utA%40mail.gmail.com.

--

You received this message because you are subscribed to the Google Groups "headless-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev+unsubscribe@chromium.org.

To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/CAPuLczsU_5Vh2ZPz4Crgr%2B1Zx4oxCehEAC9MX6Wn7t9mO75VyQ%40mail.gmail.com.

[Alex Clarke]

Chrome is currently trying to render at 60fps even in headless mode, that can burn a lot of CPU, although a trace would be useful to confirm theres not some other problem.  We are working on adding controls for rendering which we hope will help reduce CPU usage.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/CAK%3DPu0A449Z6bbK2uadXaANwiNJgyJtUpSov6VPMhjrr%3D0qx%2BQ%40mail.gmail.com.

[Alex Clarke]

On 26 April 2017 at 21:29, Vincent Voyer <wou...@gmail.com> wrote:

On 26 April 2017 at 22:16, Alex Clarke <alexc...@google.com> wrote:

Chrome is currently trying to render at 60fps even in headless mode, that can burn a lot of CPU, although a trace would be useful to confirm theres not some other problem.  We are working on adding controls for rendering which we hope will help reduce CPU usage.

I will do that trace, did not even know it was feasible, nice! Ultimately I do not need any rendering, I just want to use this DOM features and being able to query the DOM.

A note of caution, depending on which pages you're crawling you'll have to do at least some rendering.  Quite a few pages won't function without animation triggers or a few requestAnimationFrames firing.

 

 

On 26 April 2017 at 20:48, Anton Bacaj <aba...@gmail.com> wrote:

You said you disable CSS, and JS... Isn't that the purpose of even using Chrome Headless? 

I disabled the loading of CSS, JS, images. Those are still loaded by chrome headless while for my current case I want to use chrome headless in two cases:

- no external dependencies, just the html of the page

- all external dependencies, with CSS, JS ... loaded

What I will be using from chrome headless: the DOM loading and DOM/Network methods.

 

To me, it sounds like you can crawl a webpage quiet easily without a browser, just use a http library etc

Yes you can do that but to understand the DOM you will have to use some libraries that will try to mimic a browser without reaching it. And then when you will want to have JS, CSS, images loaded, you will have to use another system (like a real browser, chrome headless).

I want a single solution, if feasible.

 

On Apr 26, 2017 1:08 PM, "Sami Kyostila" <skyo...@chromium.org> wrote:

This is interesting. Could you using the --trace-startup and, say, --trace-startup-duration=30 command line flags to record some performance traces and attach them to a bug? I suspect since all tabs are trying to render animations in parallel, they'll end up using a lot of CPU time in total.

- Sami

I'll do that as soon as tomorrow, thanks for suggestion.

 

ke 26. huhtik. 2017 klo 15.09 Isaac Dawson <isaac....@gmail.com> kirjoitti:

I'm not sure this is the correct list for this, but I had similar challenges and spoke about them at a conference last year: https://youtu.be/aqeBM9Q3aY8 

Thanks for this, was very interesting and yes some of the challenges you faced I will face them too.

--

Vincent Voyer
06 13 92 69 96

--
You received this message because you are subscribed to the Google Groups "headless-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev+unsubscribe@chromium.org.
To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/CAPdVrO-fpxCtiAn2GDEiAGGM3RNNKK3d756ROoKCrDo_oM6LGg%40mail.gmail.com.

[vvowns]

Hi Sami,

I did a trace, it's available here: https://dl.dropboxusercontent.com/u/3508235/chrometrace.log.zip

I did not open a bug yet because I am not sure there's a bug, I don't want to create useless work for maintainers.

Still I would love your feedback on this trace as I am currently unable to understand it.

Especially if you know that soon I will be able to get an even better performance.

After fiddling a bit more I was able to get a better performance (something like 70 pages parsed by second). Nice.

Thanks a lot.

[Sami Kyostila]

Thanks for the trace Vincent! Looks like you're running 16 renderers in parallel and each one is getting a little preempted. How may cores does your machine have? Also, there is a GPU process which is rendering at about 50 fps, which is causing some extra load on the browser IO thread (which also handles network connections). I would suggest trying to disable GPU emulation (use the --disable-gpu command line flag if this is the headless shell). Also, soon it will be possible to control rendering frame rate more explicitly which should help further reduce the CPU load. However each navigation seems to result in about two renders which isn't too bad -- maybe it would be better to tell each tab that it is backgrounded to avoid any rendering at all (see Browser.setWindowBounds).

- Sami

pe 28. huhtik. 2017 klo 11.06 'Vincent Voyer' via headless-dev <headle...@chromium.org> kirjoitti:

Hi Sami,

I did a trace, it's available here: https://dl.dropboxusercontent.com/u/3508235/chrometrace.log.zip

I did not open a bug yet because I am not sure there's a bug, I don't want to create useless work for maintainers.

Still I would love your feedback on this trace as I am currently unable to understand it.

Especially if you know that soon I will be able to get an even better performance.

After fiddling a bit more I was able to get a better performance (something like 70 pages parsed by second). Nice.

Thanks a lot.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/26b04d4f-6b64-4681-bfee-c23b8a22dd50%40chromium.org.

[daniel....@zwoop.biz]

Hi,

Are you willing to share the code you're using for crawling? 

Which programming language are you using with Chromium and the chromedriver?

Best regards