Is BCP blocking LWP::Simple UAs?

[Jeffrey Darlington]

I have been a long time user of Browser Capabilities Project files for a number of years. For the past year or so, I have successfully automated my checks and downloads with a simple Perl script that uses LWP::Simple. I follow all of the rules: My script runs once per week, first checks the version file and compares it to a local cache of the browscap file I already have installed, and if the version is newer, then and only then do I download the full file.

In recent weeks, this script has started to fail, not on the full file download, but on the version check. The BCP main page states that version check requests are not rate limited. I am able to request the version check URL manually via Lynx from the same server, as well as manually grab the actual INI file with wget, so I'm guessing my IP has not been banned. After adding a bit more error checking to my script, I can see that my requests to fetch the version file are getting an HTTP 403 (Forbidden) response.

The only conclusion I can reach now is that BCP is now blocking all requests that use the LWP::Simple Perl module.

Can I request that this block be lifted? I really don't have time to rewrite this script, and it seems rather arbitrary to block this module when you're already rate limiting and banning individual IPs, especially when my scripts and IP have always been well-behaved. This is a widely used Perl module, not just by "script kiddies" and ne'er-do-wells, but by legitimate users for its simplicity of use.

Alternatively, and assuming I'll have to rewrite my script to use the heavier-weight LWP::UserAgent module, can you suggest a specific user agent my script should be using to avoid this arbitrary ban in the future?

Jeffrey T. Darlington
General Protection Fault
http://www.gpf-comics.com/

[James Titcumb]

Hi Jeffrey,

The 403 you see - is this branded with Cloudflare? I don't think our website issues 403 for rate limiting, though I'd need to check. I think it's possible that Cloudflare is returning this...

https://support.cloudflare.com/hc/en-us/articles/200169226-Why-am-I-getting-a-403-error-

If you could confirm where the 403 is coming from that would help me investigate, thanks! I doubt it'll be necessary to rewrite anything, I'm sure we can figure out the source of the problem :)

Thanks

James

--
You received this message because you are subscribed to the Google Groups "browscap" group.
To unsubscribe from this group and stop receiving emails from it, send an email to browscap+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/27e382f7-53ad-4891-8f6a-79fb5945fb2e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[James Titcumb]

For the record, we don't have any blanket band for UAs, only individual IPs.

[Jeffrey Darlington]

> To unsubscribe from this group and stop receiving emails from it, send an email to browscap+u...@googlegroups.com.

>
> To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/27e382f7-53ad-4891-8f6a-79fb5945fb2e%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

Yeah, it's Cloudflare. When I do a DNS look-up for "browscap.org" from that server, then do a reverse look-up on the returned IP, it's coming up with a Cloudflare domain. So that's probably where the problem lies.

Thanks for investigating.

[Jay Dansand]

Sorry to zombify this thread, but I just stumbled onto this same issue and wanted the maintainers to know this is going on (and may be quietly stomping on scripted updates for others as well).

I am following all the rules, fetching once a week and checking the version number first. Only for some amount of time, when using Perl::LWP, I now receive the (Cloudflare-branded, but they blame browscap.org) error message:

The owner of this website (browscap.org) has banned your access based
on your browser's signature (434ac75e55612531-ua24).

The IP isn't banned - if I change the request to use libCURL it works just fine. This happens on multiple servers.

[James Titcumb]

Hi Jay,

Indeed we use Cloudflare in front of the site; could you let me know your IP and I'll try and look up in the CF firewall events for you.

I haven't had to unban anyone by IP now for quite some time (which is good!) in our own system, but indeed that error comes from Cloudflare.

Anyway - let me know the IP and I'll do some digging for you.

Thanks

James

--
You received this message because you are subscribed to the Google Groups "browscap" group.
To unsubscribe from this group and stop receiving emails from it, send an email to browscap+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/4a4da8ef-7ff6-4d7f-a5a7-25b7533721a0%40googlegroups.com.

[Jeffrey Darlington]

On Tuesday, July 3, 2018 at 12:56:40 PM UTC-4, James Titcumb wrote:
> Indeed we use Cloudflare in front of the site; could you let me know your IP and I'll try and look up in the CF firewall events for you.
>
> I haven't had to unban anyone by IP now for quite some time (which is good!) in our own system, but indeed that error comes from Cloudflare.

For what it's worth, I was the original reporter for this issue. I ended up switching to having my script call my system's copy of wget and haven't had an issue since. Why Cloudflare distrusts Perl libraries but allows wget (which can just as easily be scripted) is beyond me.

Jeffrey T. Darlington
General Protection Fault

https://www.gpf-comics.com/

[Jay Dansand]

Thanks for looking into it!

I'm pretty sure the issue isn't an IP ban because switching the user agent works, but one of the affected servers is 143.44.200.14. Here's hoping the events log will shed more light on why Cloudflare is blocking LWP.

On Tue, Jul 3, 2018 at 11:56 AM, browscap on behalf of James Titcumb <brow...@googlegroups.com> wrote:

Hi Jay,

Indeed we use Cloudflare in front of the site; could you let me know your IP and I'll try and look up in the CF firewall events for you.

I haven't had to unban anyone by IP now for quite some time (which is good!) in our own system, but indeed that error comes from Cloudflare.

Anyway - let me know the IP and I'll do some digging for you.

Thanks

James

On Tue, 3 Jul 2018 at 17:52, Jay Dansand via browscap <browscap+APn2wQdQrYdduZJz3Eu0sx9-Ew2iK0H6-lNd7_B1BdtK32xmourCmIr@googlegroups.com> wrote:

Sorry to zombify this thread, but I just stumbled onto this same issue and wanted the maintainers to know this is going on (and may be quietly stomping on scripted updates for others as well).

I am following all the rules, fetching once a week and checking the version number first. Only for some amount of time, when using Perl::LWP, I now receive the (Cloudflare-branded, but they blame browscap.org) error message:

  The owner of this website (browscap.org) has banned your access based
  on your browser's signature (434ac75e55612531-ua24).

The IP isn't banned - if I change the request to use libCURL it works just fine. This happens on multiple servers.

--
You received this message because you are subscribed to the Google Groups "browscap" group.

To unsubscribe from this group and stop receiving emails from it, send an email to browscap+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/4a4da8ef-7ff6-4d7f-a5a7-25b7533721a0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "browscap" group.

To unsubscribe from this group and stop receiving emails from it, send an email to browscap+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/CAF5o2a82N7QdE8foekjV76_D2ciyyAbRGPCLGCw74qVq%3D2w0Mg%40mail.gmail.com.

[James Titcumb]

Folks,

Oddly, there's nothing showing in the firewall events for that IP :/

A quick google suggests might help if you use set up LWP to use cookies? https://stackoverflow.com/questions/29057331/waiting-for-cloudflare-ddos-protection-lwp-perl

If not, can you change the User Agent at all? By setting something identifiable (i.e. not whatever the default of the LWP library is) that might help to distinguish the requests? Looks like it's the "Browser Integrity Check" kicking in (see https://support.cloudflare.com/hc/en-us/articles/200170086-What-does-the-Browser-Integrity-Check-do- ) so changing user agent (e.g. to one that Chrome/Edge/Firefox might use) may indeed help...

Thanks

James

On Tue, 3 Jul 2018 at 18:12, browscap on behalf of Jay Dansand <brow...@googlegroups.com> wrote:

Thanks for looking into it!

I'm pretty sure the issue isn't an IP ban because switching the user agent works, but one of the affected servers is 143.44.200.14. Here's hoping the events log will shed more light on why Cloudflare is blocking LWP.

On Tue, Jul 3, 2018 at 11:56 AM, browscap on behalf of James Titcumb <brow...@googlegroups.com> wrote:

Hi Jay,

Indeed we use Cloudflare in front of the site; could you let me know your IP and I'll try and look up in the CF firewall events for you.

I haven't had to unban anyone by IP now for quite some time (which is good!) in our own system, but indeed that error comes from Cloudflare.

Anyway - let me know the IP and I'll do some digging for you.

Thanks

James

On Tue, 3 Jul 2018 at 17:52, Jay Dansand via browscap <browscap+APn2wQdQrYdduZJz3Eu0sx9...@googlegroups.com> wrote:

Sorry to zombify this thread, but I just stumbled onto this same issue and wanted the maintainers to know this is going on (and may be quietly stomping on scripted updates for others as well).

I am following all the rules, fetching once a week and checking the version number first. Only for some amount of time, when using Perl::LWP, I now receive the (Cloudflare-branded, but they blame browscap.org) error message:

  The owner of this website (browscap.org) has banned your access based
  on your browser's signature (434ac75e55612531-ua24).

The IP isn't banned - if I change the request to use libCURL it works just fine. This happens on multiple servers.

--
You received this message because you are subscribed to the Google Groups "browscap" group.

To unsubscribe from this group and stop receiving emails from it, send an email to browscap+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/4a4da8ef-7ff6-4d7f-a5a7-25b7533721a0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "browscap" group.

To unsubscribe from this group and stop receiving emails from it, send an email to browscap+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/CAF5o2a82N7QdE8foekjV76_D2ciyyAbRGPCLGCw74qVq%3D2w0Mg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "browscap" group.

To unsubscribe from this group and stop receiving emails from it, send an email to browscap+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/CA%2BqvCsi%2B67qo0gC2ViyR0%3DuNx145ZO23ZWhiRZbravncHWfPPA%40mail.gmail.com.

[Jay Dansand]

Thanks for checking into it! I ran some tests, and can confirm it's literally just blocking the LWP user agent string - there is no redirect cookie check (despite what the stackoverflow answer says). The obvious question is what security does this provide? If anyone was using LWP to script an attack, they can change the User-Agent header. This only penalizes legitimate users.

Fails:

$ GET -UsSed http://browscap.org/version-number
GET http://browscap.org/version-number
User-Agent: lwp-request/5.827 libwww-perl/5.833

GET http://browscap.org/version-number --> 403 Forbidden

Works:

$ GET -UuSsed http://browscap.org/version-number -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:59.0) Gecko/20100101 Firefox/59.0'
GET http://browscap.org/version-number
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:59.0) Gecko/20100101 Firefox/59.0

GET http://browscap.org/version-number --> 200 OK

But here's another oddball thing:

Works: GET -UusSed http://browscap.org/version-number -H 'User-Agent: curl'

Fails: GET -UusSed http://browscap.org/version-number -H 'User-Agent: lwp'

I even changed User-Agent to the made-up string "lpw" (reordered the letters from LWP) and it actually works, so there's no whitelist of allowable strings, just a blacklist on which "lwp" resides.

On Tue, Jul 3, 2018 at 3:01 PM, browscap on behalf of James Titcumb <brow...@googlegroups.com> wrote:

Folks,

Oddly, there's nothing showing in the firewall events for that IP :/

A quick google suggests might help if you use set up LWP to use cookies? https://stackoverflow.com/questions/29057331/waiting-for-cloudflare-ddos-protection-lwp-perl

If not, can you change the User Agent at all? By setting something identifiable (i.e. not whatever the default of the LWP library is) that might help to distinguish the requests? Looks like it's the "Browser Integrity Check" kicking in (see https://support.cloudflare.com/hc/en-us/articles/200170086-What-does-the-Browser-Integrity-Check-do- ) so changing user agent (e.g. to one that Chrome/Edge/Firefox might use) may indeed help...

Thanks

James

On Tue, 3 Jul 2018 at 18:12, browscap on behalf of Jay Dansand <brow...@googlegroups.com> wrote:

Thanks for looking into it!

I'm pretty sure the issue isn't an IP ban because switching the user agent works, but one of the affected servers is 143.44.200.14. Here's hoping the events log will shed more light on why Cloudflare is blocking LWP.

On Tue, Jul 3, 2018 at 11:56 AM, browscap on behalf of James Titcumb <brow...@googlegroups.com> wrote:

Hi Jay,

Indeed we use Cloudflare in front of the site; could you let me know your IP and I'll try and look up in the CF firewall events for you.

I haven't had to unban anyone by IP now for quite some time (which is good!) in our own system, but indeed that error comes from Cloudflare.

Anyway - let me know the IP and I'll do some digging for you.

Thanks

James

On Tue, 3 Jul 2018 at 17:52, Jay Dansand via browscap <browscap+APn2wQdQrYdduZJz3Eu0sx9-Ew2iK0H6-lNd7_B1BdtK32xmourCmIr@googlegroups.com> wrote:

Sorry to zombify this thread, but I just stumbled onto this same issue and wanted the maintainers to know this is going on (and may be quietly stomping on scripted updates for others as well).

I am following all the rules, fetching once a week and checking the version number first. Only for some amount of time, when using Perl::LWP, I now receive the (Cloudflare-branded, but they blame browscap.org) error message:

  The owner of this website (browscap.org) has banned your access based
  on your browser's signature (434ac75e55612531-ua24).

The IP isn't banned - if I change the request to use libCURL it works just fine. This happens on multiple servers.

--
You received this message because you are subscribed to the Google Groups "browscap" group.

To unsubscribe from this group and stop receiving emails from it, send an email to browscap+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/4a4da8ef-7ff6-4d7f-a5a7-25b7533721a0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "browscap" group.

To unsubscribe from this group and stop receiving emails from it, send an email to browscap+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/CAF5o2a82N7QdE8foekjV76_D2ciyyAbRGPCLGCw74qVq%3D2w0Mg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "browscap" group.

To unsubscribe from this group and stop receiving emails from it, send an email to browscap+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/CA%2BqvCsi%2B67qo0gC2ViyR0%3DuNx145ZO23ZWhiRZbravncHWfPPA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "browscap" group.

To unsubscribe from this group and stop receiving emails from it, send an email to browscap+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/CAF5o2a9LGUBu8ifi-gFMuA34VSpkMBpx9cLs%3DW7WYNdQWysHtQ%40mail.gmail.com.

[James Titcumb]

Ahh, glad that worked at least. I guess it stops some abuse, but yeah. Not sure why CF blames us... the setting is enabled, but it's not like I have a way of configuring the option... it's all or nothing heh.

Thanks
James

To unsubscribe from this group and stop receiving emails from it, send an email to browscap+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/4a4da8ef-7ff6-4d7f-a5a7-25b7533721a0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "browscap" group.

To unsubscribe from this group and stop receiving emails from it, send an email to browscap+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/CAF5o2a82N7QdE8foekjV76_D2ciyyAbRGPCLGCw74qVq%3D2w0Mg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "browscap" group.

To unsubscribe from this group and stop receiving emails from it, send an email to browscap+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/CA%2BqvCsi%2B67qo0gC2ViyR0%3DuNx145ZO23ZWhiRZbravncHWfPPA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "browscap" group.

To unsubscribe from this group and stop receiving emails from it, send an email to browscap+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/CAF5o2a9LGUBu8ifi-gFMuA34VSpkMBpx9cLs%3DW7WYNdQWysHtQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "browscap" group.

To unsubscribe from this group and stop receiving emails from it, send an email to browscap+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/CA%2BqvCsisGaKRwDWadWj0zZJq4QZ-9a9nEQ2tH%2Brufsgv2ABKgg%40mail.gmail.com.