UA sniffing on webpagetest.org?

[Charlie Clark]

I think is one for Pat - have you put a block on certain UAs for some
resources?

I'm currently getting 403 when I try and get the HAR file via Python eg.
http://httparchive.webpagetest.org/export.php?test=140815_A_69JF&run=2&cached=0&pretty=1

But this works fine in a browser or curl which suggests there's a block.

* Connected to httparchive.webpagetest.org (149.20.63.13) port 80 (#0)
> GET /export.php?test=140801_2_1PSK HTTP/1.1
> User-Agent: curl/7.37.1
> Host: httparchive.webpagetest.org
> Accept: */*
>
< HTTP/1.1 200 OK
* Server nginx is not blacklisted
< Server: nginx
< Date: Thu, 21 Aug 2014 10:44:02 GMT
< Content-Type: application/json
< Transfer-Encoding: chunked
< Connection: keep-alive
< Vary: Accept-Encoding
< Set-Cookie: o=47bdd03e1d9a42021f103e206bab60af4c0c615e; expires=Fri,
21-Aug-2015 10:44:02 GMT; Max-Age=31536000; path=/
< Set-Cookie: tid=140801_2_1PSK
< Content-disposition: attachment; filename=www.bayer.com.140801_2_1PSK.har

Charlie
--
Charlie Clark
Managing Director
Clark Consulting & Research
German Office
Kronenstr. 27a
Düsseldorf
D- 40217
Tel: +49-211-600-3657
Mobile: +49-178-782-6226

[Patrick Meenan]

I had to put the block in place because someone had a script that was pulling HARs for every test in every crawl we had ever run which was causing some pretty significant server problems.  I just dropped the block but if the crawling of the results picks back up again I may have to put it back (if I remember correctly, blocking at the IP level was a game of whack-a-mole because they switched IP blocks every few hours).

--
You received this message because you are subscribed to the Google Groups "HTTP Archive" group.
To unsubscribe from this group and stop receiving emails from it, send an email to httparchive+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Charlie Clark]

Am .08.2014, 14:13 Uhr, schrieb Patrick Meenan <patm...@gmail.com>:

> I had to put the block in place because someone had a script that was
> pulling HARs for every test in every crawl we had ever run which was
> causing some pretty significant server problems. I just dropped the
> block
> but if the crawling of the results picks back up again I may have to put
> it
> back (if I remember correctly, blocking at the IP level was a game of
> whack-a-mole because they switched IP blocks every few hours).

That's entirely understandable: inconsiderate crawlers can be a real pain.
I seem to remember someone posting on the list about wanting to do that.

I think there is still an IP block for my server: 213.131.254.165. I'd be
surprised if that was generating a lot of traffic. It may have done for a
while because of bug related to how the thumbnails were stored.

[Patrick Meenan]

ok, give it a shot now.

[Charlie Clark]

Am .08.2014, 14:35 Uhr, schrieb Patrick Meenan <patm...@gmail.com>:

> ok, give it a shot now.

Fantastic! Thanks very much.