[Dillo-dev] Google broke recently with 3.0.4 causing repush loop

[Dennis New]

This past week or so Google (main search site, image.google) broke for
me -- Dillo (3.0.4) goes into an endless "repush" loop:

Nav_open_url: new url='http://images.google.com/images?q=bra'
Dns_server [0]: images.google.com is 96.127.250.44 96.127.250.42
96.127.250.38 96.127.250.34 96.127.250.30 96.127.250.29 96.127.250.27
96.127.250.23 96.127.250.19 96.127.250.15 96.127.250.59 96.127.250.57
96.127.250.53 96.127.250.49 96.127.250.45 2607:f8b0:4006:80f::200e

Connecting to 96.127.250.44
Nav_open_url: new url='http://images.google.com/search?q=bra&tbm=isch'
Connecting to 96.127.250.44

>>>> a_Nav_repush <<<<
Nav_open_url: new url='http://images.google.com/search?q=bra&tbm=isch'
a_Nav_expect_done: repush!
Nav_open_url: new
url='http://images.google.com/search?q=bra&ie=UTF-8&tbm=isch&gbv=1&sei=zkNNVd6sIoiSyATahoHACA'
Connecting to 96.127.250.44

>>>> a_Nav_repush <<<<
Nav_open_url: new
url='http://images.google.com/search?q=bra&ie=UTF-8&tbm=isch&gbv=1&sei=zkNNVd6sIoiSyATahoHACA'
a_Nav_expect_done: repush! Nav_open_url: new
url='http://images.google.com/search?q=bra&ie=UTF-8&tbm=isch&gbv=1&sei=zkNNVYTpKIiWyQTprYC4Ag'
Connecting to 96.127.250.44
>>>> a_Nav_repush <<<<

... and so on, extermely quickly, which then results in Google
temporarily blocking me since Dillo is behaving like a DDOS attacker :s.

Any idea what's causing these repush'es? I'm pretty sure nothing
changed on my end. Other browsers work for me (eg. Links, Uzbl).
Changing the User Agent had no effect.

_______________________________________________
Dillo-dev mailing list
Dill...@dillo.org
http://lists.dillo.org/cgi-bin/mailman/listinfo/dillo-dev

[James C]

Here's a guess without justification: it's trying to get confirmation
that you accepted a cookie, and failing.

The basis for my guess is that login to gmail is possible, but
requires more than five redirects, so I run a hacked up dillo 3.1,
with a dialog about whether to keep redirecting.

[eocene]

It looks like google is misbehaving.
http://www.google.com/search?ie=UTF-8&oe=UTF-8&q=whatever
contains
<noscript><meta content="0;url=/search?q=whatever&amp;ie=UTF-8&amp;oe=UTF-8&amp;gbv=1&amp;sei=sV5NVfbJMtfsoASOo4GQBw" http-equiv="refresh"><style>table,div,span,p{display:none}</style><div style="display:block">Please click <a href="/search?q=whatever&amp;ie=UTF-8&amp;oe=UTF-8&amp;gbv=1&amp;sei=sV5NVfbJMtfsoASOo4GQBw">here</a> if you are not redirected within a few seconds.</div></noscript>

so we follow the refresh, and the new page has

<noscript><meta content="0;url=/search?q=whatever&amp;ie=UTF-8&amp;gbv=1&amp;sei=PV9NVej-Noj9oQSd-ICQDA" http-equiv="refresh"><style>table,div,span,p{display:none}</style><div style="display:block">Please click <a href="/search?q=whatever&amp;ie=UTF-8&amp;gbv=1&amp;sei=PV9NVej-Noj9oQSd-ICQDA">here</a> if you are not redirected within a few seconds.</div></noscript>

...etc....

If I use a firefox user agent string, though, I am redirected from http to
an https page which works and doesn't loop because...it puts the meta refresh
down in the body instead of the head, so Dillo disallows it.

[eocene]

James wrote:
> Here's a guess without justification: it's trying to get confirmation
> that you accepted a cookie, and failing.

I did make a quick attempt with accept_session in cookiesrc, and it
didn't appear to make a difference...

> The basis for my guess is that login to gmail is possible, but
> requires more than five redirects, so I run a hacked up dillo 3.1,
> with a dialog about whether to keep redirecting.

How many redirects does it need? The only site that I tend to run into
redirect problems with is nytimes.com (Not logging into any account;
just looking at articles that people post links to...), and the last
time I checked it was something like 12 redirects before it will give
up and provide the page. And I wonder whether raising the redirect
limit might be good, overall.

Does gmail still need all of those redirects if you let it have a
cookie?

[James C]

All of this is with the cookiesrc below, which lets google have
cookies on two domains.

If I:
- log out of google
-- it stalls on a meta refresh tag
- select home, which for me is https://mail.google.com/?ui=html
- provide a username and password
-- I am eventually shown my email messages
then the code below redirects:
- once during the logout
- twice before asking for a password
- eight (8) times before showing me my mail

Adding back the hack to share cookies probably makes no difference to
these figures, but I'm not sure, because I think that I remember
inconsistently getting seven and nine redirects, so I think that
google is A/B testing me.

This code cannot be put into production, because it produces messages
about redirecting 0 times reasonably often, for reasons that I haven't
tracked down yet.

--------------------------------
#host action
DEFAULT DENY
.google.com ACCEPT_SESSION
.google.co.nz ACCEPT_SESSION
--------------------------------

render/dillo-jfw01$ hg diff
diff -r 07c0c28f8960 src/cache.c
--- a/src/cache.c Fri Mar 06 13:12:11 2015 +1300
+++ b/src/cache.c Sat May 09 22:03:31 2015 +1200
@@ -1007,11 +1007,24 @@
return 0;
}

- /* if there's a redirect loop, stop now */
- if (bw->redirect_level >= 5)
- entry->Flags |= CA_RedirectLoop;
+// if (0 == bw->redirect_level % 5) {
+ if (0 == bw->redirect_level % 1) {
+ char *redirMessage;
+ int choice;
+
+ asprintf(&redirMessage, "Dillo: redirected %d times",
bw->redirect_level);
+
+ choice = a_Dialog_choice(redirMessage, URL_STR(entry->Url),
+ "Stop", "Continue", NULL);
+
+ if (1 == choice)
+ entry->Flags |= CA_RedirectLoop;
+
+ free(redirMessage);
+ }

if (entry->Flags & CA_RedirectLoop) {
+ MSG("Redirect loop for: %s\n", URL_STR_(entry->Url));
a_UIcmd_set_msg(bw, "ERROR: redirect loop for: %s",
URL_STR_(entry->Url));
bw->redirect_level = 0;
return 0;
diff -r 07c0c28f8960 src/cookies.c
--- a/src/cookies.c Fri Mar 06 13:12:11 2015 +1300
+++ b/src/cookies.c Sat May 09 22:03:31 2015 +1200
@@ -196,6 +196,15 @@

if (requester == NULL) {
/* request made by user */
+#if 0
+ } else if (!strcmp("accounts.google.co.nz", URL_HOST(requester)) &&
+ !strcmp("mail.google.com", URL_HOST(query_url))) {
+ MSG("Cookies: sent for google NZ special case 1.\n");
+ } else if (!strcmp("accounts.google.com", URL_HOST(requester)) &&
+ !strcmp("accounts.google.co.nz", URL_HOST(query_url))) {
+ MSG("Cookies: sent for google NZ special case 2.\n");
+
+#endif
} else if (!a_Url_same_organization(query_url, requester)) {
MSG("Cookies: not sent for request by '%s' for '%s'\n",
URL_HOST(requester), URL_HOST(query_url));

[Andreas Kemnade]

Hi,

On Sat, 9 May 2015 01:28:40 +0000
eocene <eoc...@gmx.com> wrote:


> If I use a firefox user agent string, though, I am redirected from http to
> an https page which works and doesn't loop because...it puts the meta refresh
> down in the body instead of the head, so Dillo disallows it.
>

trying with lynx gives no refresh at all!

http request resulting from a search with lynx:


GET /search?ie=ISO-8859-1&hl=de&source=hp&q=dillo&btnG=Google-Suche&gbv=1 HTTP/1.0
Host: www.google.de
Accept: text/html, text/plain, text/css, text/sgml, */*;q=0.01
Accept-Encoding: gzip, compress, bzip2
Accept-Language: en
User-Agent: Lynx/2.8.8dev.12 libwww-FM/2.14 SSL-MM/1.4.1 GNUTLS/2.12.18
Referer: http://www.google.de/


It gives javascript in the output but no meta refresh.
Lynx works perfectly with google.

Greetings
Andreas

[eocene]

James wrote:
> This code cannot be put into production, because it produces messages
> about redirecting 0 times reasonably often, for reasons that I haven't
> tracked down yet.

Having a dialog when we're looping does sound like a good idea to me...

[Dennis New]

On Sat, 9 May 2015 15:59:56 +0000, eocene wrote:
> James wrote:
> > This code cannot be put into production, because it produces
> > messages about redirecting 0 times reasonably often, for reasons
> > that I haven't tracked down yet.
>
> Having a dialog when we're looping does sound like a good idea to
> me...

Why automatically follow the meta redirect at all? I notice that when
the time period is non-zero, it doesn't automatically follow, but
instead outputs something like:

"The author wanted you to go here after 1 second"

That seems to make more sense to me -- it avoids loops, and is more
informative for users?

[eocene]

Dennis wrote:
> On Sat, 9 May 2015 15:59:56 +0000, eocene wrote:
> > James wrote:
> > > This code cannot be put into production, because it produces
> > > messages about redirecting 0 times reasonably often, for reasons
> > > that I haven't tracked down yet.
> >
> > Having a dialog when we're looping does sound like a good idea to
> > me...
>
> Why automatically follow the meta redirect at all? I notice that when
> the time period is non-zero, it doesn't automatically follow, but
> instead outputs something like:
>
> "The author wanted you to go here after 1 second"
>
> That seems to make more sense to me -- it avoids loops, and is more
> informative for users?

Jorge put it in. I don't remember why now.
http://lists.dillo.org/pipermail/dillo-dev/2009-April/006294.html

In any case, if we don't automatically redirect based on the meta tags,
there are still the redirections from Location headers in HTTP.

[James C]

Some pages which show lots of 'redirected 0 times', show less when I
block *.gravatar.com in domainrc.