Crawl updated to 500k URLs

[Patrick Meenan]

FYI, the crawl for 12/1 was expanded to crawl 500k URLs (up from 300k).  That mmeans results will probably be a couple of days later than normal.

Thanks,

-Pat

[Charlie Clark]

Am .12.2014, 17:01 Uhr, schrieb Patrick Meenan <patm...@gmail.com>:

> FYI, the crawl for 12/1 was expanded to crawl 500k URLs (up from 300k).
> That mmeans results will probably be a couple of days later than normal.

Thanks very much for the info, Patrick.

Charlie
--
Charlie Clark
Managing Director
Clark Consulting & Research
German Office
Kronenstr. 27a
Düsseldorf
D- 40217
Tel: +49-211-600-3657
Mobile: +49-178-782-6226

[Charlie Clark]

Am .12.2014, 17:01 Uhr, schrieb Patrick Meenan <patm...@gmail.com>:

> FYI, the crawl for 12/1 was expanded to crawl 500k URLs (up from 300k).
> That mmeans results will probably be a couple of days later than normal.

Hiya Pat,

from the New Relic dashboard it looks like the number crunching has
finished and the mobile stats are available but not the desktop ones. Are
they still being processed? Or has there been a hold-up?

[Patrick Meenan]

After the main crawl is done there are a few retry cycles for straggler URLs that failed the first time around and then the aggregation.  Looking at the test backlog it looks like it may still be doing small batches of the cleanup sweeps: http://httparchive.webpagetest.org/getLocations.php

At some point it would probably be worthwhile to resubmit the failures as they are detected instead of at the end (or some other logic to minimize the retries).

--
You received this message because you are subscribed to the Google Groups "HTTP Archive" group.
To unsubscribe from this group and stop receiving emails from it, send an email to httparchive+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Charlie Clark]

Am .12.2014, 14:06 Uhr, schrieb Patrick Meenan <patm...@gmail.com>:

> After the main crawl is done there are a few retry cycles for straggler
> URLs that failed the first time around and then the aggregation. Looking
> at the test backlog it looks like it may still be doing small batches of
> the cleanup sweeps: http://httparchive.webpagetest.org/getLocations.php

Looks empty now. Does it then run the stat generation?

> At some point it would probably be worthwhile to resubmit the failures as
> they are detected instead of at the end (or some other logic to minimize
> the retries).

How many is it? Likely to be exponential with the new intake of websites.

[Patrick Meenan]

Sorry, I don't know the specifics, hopefully Steve will chime in.  Looking at the newrelic graphs it looks like there were a couple of bumps after the main crawl finished (and when I peeked a minute ago there were 20k tests).  I know Steve recently increased the retry count but I'm not sure what it is set to or how the logic works.

[Charlie Clark]

Am .12.2014, 18:28 Uhr, schrieb Patrick Meenan <patm...@gmail.com>:

> Sorry, I don't know the specifics, hopefully Steve will chime in.
> Looking
> at the newrelic graphs it looks like there were a couple of bumps after
> the
> main crawl finished (and when I peeked a minute ago there were 20k
> tests).
> I know Steve recently increased the retry count but I'm not sure what it
> is
> set to or how the logic works.

Now done to about 12k which suggests another couple of hours. It was
totally idle a couple of hours ago.

[Charlie Clark]

Am .12.2014, 19:13 Uhr, schrieb Charlie Clark
<charli...@clark-consulting.eu>:

> Now done to about 12k which suggests another couple of hours. It was
> totally idle a couple of hours ago.

Everything seems to have finished now. Guessing based on the New Relic
dashboard (unhelpfully not localised in region and timezone) suggests
everything finished around 00:00 GMT on 9th December.