10/1 Crawl will be a bit delayed

[Patrick Meenan]

We had one of the 3 original VM servers go belly-up towards the end of the last crawl and just got it mostly back up (if hobbling a bit) on 10/2 so the crawl got off to a bit of a slow start.  It will still complete before the 10/15 crawl kicks off but it may be a day or two later than normal (or at least relative to when things are going smoothly).

Thanks to SOASTA jumping in as a sponsor we were able to order some new SSD's which should fix most of the stability issues we have had recently.

Thanks,

-Pat

[Charlie Clark]

Hi Pat.

Am .10.2015, 04:51 Uhr, schrieb Patrick Meenan <patm...@gmail.com>:

> We had one of the 3 original VM servers go belly-up towards the end of
> the last crawl and just got it mostly back up (if hobbling a bit) on
> 10/2 so
> the crawl got off to a bit of a slow start. It will still complete
> before the 10/15 crawl kicks off but it may be a day or two later than
> normal (or at least relative to when things are going smoothly).

Thanks for the update but it seems to be going very slow all the same:
still over 300000 crawls to do. Or am I looking at the wrong queue?

> Thanks to SOASTA jumping in as a sponsor we were able to order some new
> SSD's which should fix most of the stability issues we have had recently.

+1 on that. Those disks must indeed take quite a pounding.

Charlie
--
Charlie Clark
Managing Director
Clark Consulting & Research
German Office
Kronenstr. 27a
Düsseldorf
D- 40217
Tel: +49-211-600-3657
Mobile: +49-178-782-6226

[Patrick Meenan]

I'll have a closer look.  It looks to still be running ~50% of the speed of the chrome crawl while it should be running a little faster so something else is also going on.  Thanks for pointing it out.

--
You received this message because you are subscribed to the Google Groups "HTTP Archive" group.
To unsubscribe from this group and stop receiving emails from it, send an email to httparchive...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Charlie Clark]

Am .10.2015, 19:08 Uhr, schrieb Patrick Meenan <patm...@gmail.com>:

> I'll have a closer look. It looks to still be running ~50% of the speed
> of the chrome crawl while it should be running a little faster so
> something
> else is also going on. Thanks for pointing it out.

At the moment it looks like a number of the machines are out again. Even
so, only around 25,000 tests are being performed a day, so it looks like
the run for 2015-10-01 might not be complete before the 15th.

[Patrick Meenan]

Yep. I'm in the process of rebuilding all of the VM's right now. The existing ones were spinning the CPU with background services which is why it is going so slow. I'm just finishing setting up a clean new windows install (updates take forever) and I'll be cloning it out across the servers this morning. 

Fingers crossed that will nail the issue and the testing rate will go back up to its normal rate and still finish on time but it's going to be tight. 

On Wed, Oct 7, 2015 at 12:36 AM -0700, "Charlie Clark" <charli...@clark-consulting.eu> wrote:

-- 
You received this message because you are subscribed to the Google Groups "HTTP Archive" group.
To unsubscribe from this group and stop receiving emails from it, send an email to httparchive...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Charlie Clark]

Am .10.2015, 12:47 Uhr, schrieb Patrick Meenan <patm...@gmail.com>:

> Yep. I'm in the process of rebuilding all of the VM's right now. The
> existing ones were spinning the CPU with background services which is
> why it is going so slow. I'm just finishing setting up a clean new
> windows install (updates take forever) and I'll be cloning it out across
> the servers this morning.

Ouch! I recently ran a Windows 7 VM install and the 150+ updates took over
a day! I suppose it isn't possible to get something like one of the
relevant VMs from modern.ie and use that instead? At least for a while MS
was keeping the images up to date.

> Fingers crossed that will nail the issue and the testing rate will go
> back up to its normal rate and still finish on time but it's going to be
> tight.

Recent runs have been taking about 10 days. Still not half-way through but
if you can get things back up to speed then it might just work.

[Patrick Meenan]

Sadly the modern.ie VM's have a 90-day activation and aren't really licensed for what the HA uses so it's cleaner to just use the licenses we have with full VM's.  The last rounds of updates are installing now and then it's really easy to clone it across the servers (does take a few hours though).

I have a few machines in Dulles that I just decommissioned from the public WPT that I might be able to bring online as well to help give it a kick and get through the remaining tests.

[Patrick Meenan]

Looks like we are back in business.

The bandwidth through the server is back up where it normally is when things are running well: http://httparchive.org/newrelic-dash/ and that doesn't include the browsing traffic from the extra VM's I have running in Dulles to help catch up.

Looks like it is running close to 4,000 tests per hour which should put the initial batch completing in ~3 days.  There are a couple of resubmits to re-try error pages but looks like we should be in good shape to complete the crawl before the next one kicks off.

[Charlie Clark]

Am .10.2015, 00:34 Uhr, schrieb Patrick Meenan <patm...@gmail.com>:

> Looks like we are back in business.

> The bandwidth through the server is back up where it normally is when
> things are running well: http://httparchive.org/newrelic-dash/ and that
> doesn't include the browsing traffic from the extra VM's I have running
> in Dulles to help catch up.

> Looks like it is running close to 4,000 tests per hour which should put
> the initial batch completing in ~3 days. There are a couple of
> resubmits to
> re-try error pages but looks like we should be in good shape to complete
> the crawl before the next one kicks off.

Indeed, thanks very much for getting things working again. One tiny
niggle: outside the US "10/1" really is *extremely* confusing, can we
please always either use ISO or abbreviate the month (Oct 1 / 1 Oct)?

Regarding the switch of the main browser. My corporate clients are all
finally switching from IE 8 to IE 11 and Microsoft has announced EOL for
IE 8 at the end of the year. So, switching the basic browser on 2016-01-01
might make sense.

[Patrick Meenan]

FWIW - http://chrome.httparchive.org/ has been running for a few months so we have Chrome data in parallel (not fully validated yet though).

I think the main concern with switching IE (either to Chrome or to 11) is going to be the potentially radical change in stats and the lack of ability to trend them (i.e. the Chrome data is showing 1.7MB page averages vs 2.2MB for IE9).  We have to figure out a good way to transition in general.  With modern evergreen browsers (Chrome, Firefox, Edge) it becomes a bit easier as you can just track the stable version.

That said, huge +1 from me since IE 9 won't be getting patches anymore and isn't all that common anymore.

[Steve Souders]

400K (80%) done with 6 days left to go. Looking good.

-Steve

[Charlie Clark]

Am .10.2015, 18:56 Uhr, schrieb Steve Souders <st...@souders.org>:

> 400K (80%) done with 6 days left to go. Looking good.

I think we should treat this run a brown bag and remove it from the
records. From the analyses I've done it looks like more than 50% of the
crawls were impacted by the problems with timings often 3 times long-term
averages.