headless_shell dump-dom command never print the dom

[Philippe Namias]

Hi,

When running the headless shell to dump the dom , for example :
headless_shell --no-sandbox --dump-dom http://weibo.com/login.php

This will never print the dom, due to some  javascript file (like ua.js) constantly polling and leaving the network request always open.
Is there a way to force the dump of dom in those situation?

Thanks for any help/hint.

Philippe

[Alex Clarke]

I'm going to have a look into this. You might be able to use the Network domain's EmulateNetworkConditions to pretend you're offline which should cancel all  network requests.  If that doesn't work I'll build something to support cancellation of pending (and future) network fetches.

--
You received this message because you are subscribed to the Google Groups "headless-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev+unsubscribe@chromium.org.
To post to this group, send email to headle...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/28a25abc-3390-4b07-81d7-f8908235c8bf%40chromium.org.

[Philippe Namias]

It also affect the --screenshot headless_shell command as this will
also never complete.

For example with another URL (this one will freeze in part because of
Google Analytic javascript) : https://roxxlyn.com/shop/

If you want a set of another URL that are affected let me know.

Thanks for looking into it,

Philippe

[Sebastian Cato]

Hello,

--dump-dom and --screenshot both work for me with the headless_shell
target built from master at 66cf403e747dc5dd959f9ee94852b269c0f3a312
(Sat Nov 19 01:11:28 2016 -0800) on both https://roxxlyn.com/shop/ and
http://weibo.com/login.php

Is it possible for you to replicate this issue with a small, isolated
.html-file as a testcase? Preferably one possible to host locally
without relying on Internet access or third parties.

//Sebastian

> --
> You received this message because you are subscribed to the Google Groups "headless-dev" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev...@chromium.org.

> To post to this group, send email to headle...@chromium.org.

> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/1479750538.2317.3.camel%40gmail.com.

[Philippe Namias]

Hi Sebastian,


I haven't been able to reproduce the error on a small html file (even
by letting multiple connection open the dom is printed, so the culprit
seems elsewhere).
I rebuilt from master today till commit
d42d7efd5e208eb564d0f4c060fc24121d7e6919

I still have the bug when running :

headless_shell --no-sandbox --dump-dom https://roxxlyn.com/shop/

it never print the dump.

By using strace on the main process i get the following result in loop
(something is stuck on a file descriptor but i have no clue why):

read(13, "\0", 1)                       = 1
epoll_wait(10, [], 32, 0)               = 0
epoll_wait(10, [{EPOLLIN, {u32=13, u64=13}}], 32, -1) = 1

I will do a full rebuild tomorow and keep you and Alex updated.

Thanks for the help,

Philippe

[Philippe Namias]

Hi,

So I did a full rebuild (today) of headless_shell on a brand new server
that is only used for this.
So this URL : https://roxxlyn.com/shop/ doesnt work definitively (dump-
dom and screenshot) for me, even using chrome-remote-debugger and
connecting to it, the page doesnt load at all. The only way to make it
work is to use http instead of https.

But the new build fixed all my other URL problem (weibo for example).
I will continue digging as soon as I can replicate this one, I will
post back to the list.

Thanks for the help

Philippe

On Mon, 2016-11-21 at 20:37 +0100, Sebastian Cato wrote:

[Alex Clarke]

To work around this sort of thing I'm going to add an optional timeout to headless_shell. It'll be functionally equivalent to hitting the browser's stop button after N milliseconds.  I'll try and implement this in terms of a new devtools Stop command.

> > send an email to headless-dev+unsubscribe@chromium.org.

> > To post to this group, send email to headle...@chromium.org.
> > To view this discussion on the web visit https://groups.google.com/
> > a/chromium.org/d/msgid/headless-
> > dev/1479750538.2317.3.camel%40gmail.com.

--
You received this message because you are subscribed to the Google Groups "headless-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev+unsubscribe@chromium.org.

To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/1479763981.15151.5.camel%40gmail.com.

[Philippe Namias]

Thanks Alex, that will be great, looking forward to test it.

Philippe

> > send an email to headless-dev...@chromium.org.

> > To post to this group, send email to headle...@chromium.org.
> > To view this discussion on the web visit https://groups.google.com/
> > a/chromium.org/d/msgid/headless-
> > dev/1479750538.2317.3.camel%40gmail.com.

--
You received this message because you are subscribed to the Google Groups "headless-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev...@chromium.org.

To post to this group, send email to headle...@chromium.org.

[Alex Clarke]

I'm not sure if this will land today (the chromium CQ is having some trouble) but the patch is here:  https://codereview.chromium.org/2525903003/

> > send an email to headless-dev+unsubscribe@chromium.org.

> > To post to this group, send email to headle...@chromium.org.
> > To view this discussion on the web visit https://groups.google.com/
> > a/chromium.org/d/msgid/headless-
> > dev/1479750538.2317.3.camel%40gmail.com.

--
You received this message because you are subscribed to the Google Groups "headless-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev+unsubscribe@chromium.org.

To post to this group, send email to headle...@chromium.org.

[Alex Clarke]

This patch has landed.  Please let us know if you're still having any issues.

[Philippe Namias]

Hi Alex,

I have been using this feature via the chrome remote debugger and also
as an headless shell switch for some of my customer (in production for
the past 5 days, around 10000 URL passed to it) and it work great.

Thank you very much,

Philippe

> > > > > > On Mon, Nov 21, 2016 at 6:48 PM, Philippe Namias <pnamias@g

> > > > > > > > On 17 November 2016 at 16:44, Philippe Namias <pnamias@

> > > > > > > send an email to headless-dev...@chromium.org.
> > > > > > > To post to this group, send email to headless-dev@chromiu
> > > > > m.org.

> > > > > > > To view this discussion on the web visit https://groups.g
> > > > > oogle.com/
> > > > > > > a/chromium.org/d/msgid/headless-
> > > > > > > dev/1479750538.2317.3.camel%40gmail.com.
> > > > >
> > > > > --
> > > > > You received this message because you are subscribed to the
> > > > > Google Groups "headless-dev" group.
> > > > > To unsubscribe from this group and stop receiving emails from

> > > > > it, send an email to headless-dev...@chromium.org.

> > > > > To post to this group, send email to headle...@chromium.or
> > > > > g.
> > > > >
> > > > > To view this discussion on the web visit https://groups.googl
> > > > > e.com/a/chromium.org/d/msgid/headless-

> > > > > dev/1479763981.15151.5.camel%40gmail.com.
> > > > >
> > > >
> > > >
> >
> >
>
>

[Simon Pick]

We have our very own beta-tester :)

Simon

> > > > > > > send an email to headless-dev+unsubscribe@chromium.org.

> > > > > > > To post to this group, send email to headless-dev@chromiu
> > > > > m.org.
> > > > > > > To view this discussion on the web visit https://groups.g
> > > > > oogle.com/
> > > > > > > a/chromium.org/d/msgid/headless-
> > > > > > > dev/1479750538.2317.3.camel%40gmail.com.
> > > > >
> > > > > --
> > > > > You received this message because you are subscribed to the
> > > > > Google Groups "headless-dev" group.
> > > > > To unsubscribe from this group and stop receiving emails from

> > > > > it, send an email to headless-dev+unsubscribe@chromium.org.

> > > > > To post to this group, send email to headle...@chromium.or
> > > > > g.
> > > > >
> > > > > To view this discussion on the web visit https://groups.googl
> > > > > e.com/a/chromium.org/d/msgid/headless-

> > > > > dev/1479763981.15151.5.camel%40gmail.com.
> > > > >
> > > >
> > > >
> >
> >
>
>

--
You received this message because you are subscribed to the Google Groups "headless-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev+unsubscribe@chromium.org.
To post to this group, send email to headle...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/1480765009.14735.1.camel%40gmail.com.

[Philippe Namias]

What can I say , I like living on the edge:)
But on more serious note, for sure if you want to test some feature in
headless on a medium scale just ask me.

Let me also thanks the chromium headless dev team for such a great work
and for being so responsive.

Philippe

> > > > > On Wed, 23 Nov 2016 at 19:16, Alex Clarke <alexclarke@google.

> > com>
> > > > > wrote:
> > > > > > To work around this sort of thing I'm going to add an
> > optional
> > > > > > timeout to headless_shell. It'll be functionally equivalent
> > to
> > > > > > hitting the browser's stop button after N milliseconds. 
> > I'll
> > > > > > try and implement this in terms of a new devtools Stop
> > command.
> > > > > >

> > > > > > On 21 November 2016 at 21:33, Philippe Namias <pnamias@gmai
> > l.co

> > > > > > > > > send an email to headless-dev...@chromium.or
> > g.

> > > > > > > > > To post to this group, send email to headless-dev@chr
> > omiu
> > > > > > > m.org.
> > > > > > > > > To view this discussion on the web visit https://grou
> > ps.g
> > > > > > > oogle.com/
> > > > > > > > > a/chromium.org/d/msgid/headless-
> > > > > > > > > dev/1479750538.2317.3.camel%40gmail.com.
> > > > > > >
> > > > > > > --
> > > > > > > You received this message because you are subscribed to
> > the
> > > > > > > Google Groups "headless-dev" group.
> > > > > > > To unsubscribe from this group and stop receiving emails
> > from

> > > > > > > it, send an email to headless-dev...@chromium.or
> > g.

> > > > > > > To post to this group, send email to headless-dev@chromiu
> > m.or

> > > > > > > g.
> > > > > > >
> > > > > > > To view this discussion on the web visit https://groups.g
> > oogl
> > > > > > > e.com/a/chromium.org/d/msgid/headless-
> > > > > > > dev/1479763981.15151.5.camel%40gmail.com.
> > > > > > >
> > > > > >
> > > > > >
> > > >
> > > >
> > >
> > >
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "headless-dev" group.
> > To unsubscribe from this group and stop receiving emails from it,

> > send an email to headless-dev...@chromium.org.

> > To post to this group, send email to headle...@chromium.org.
> > To view this discussion on the web visit https://groups.google.com/
> > a/chromium.org/d/msgid/headless-
> > dev/1480765009.14735.1.camel%40gmail.com.
> >
>

> -- 
> You received this message because you are subscribed to the Google
> Groups "headless-dev" group.
> To unsubscribe from this group and stop receiving emails from it,

> send an email to headless-dev...@chromium.org.

> To post to this group, send email to headle...@chromium.org.
> To view this discussion on the web visit https://groups.google.com/a/
> chromium.org/d/msgid/headless-

> dev/CAHOyddZ1%2BfzUy1yCPP_JszgSAWxTeFsMbhWuUjvqs1u2XMc-
> %3Dg%40mail.gmail.com.