In recent weeks, this script has started to fail, not on the full file download, but on the version check. The BCP main page states that version check requests are not rate limited. I am able to request the version check URL manually via Lynx from the same server, as well as manually grab the actual INI file with wget, so I'm guessing my IP has not been banned. After adding a bit more error checking to my script, I can see that my requests to fetch the version file are getting an HTTP 403 (Forbidden) response.
The only conclusion I can reach now is that BCP is now blocking all requests that use the LWP::Simple Perl module.
Can I request that this block be lifted? I really don't have time to rewrite this script, and it seems rather arbitrary to block this module when you're already rate limiting and banning individual IPs, especially when my scripts and IP have always been well-behaved. This is a widely used Perl module, not just by "script kiddies" and ne'er-do-wells, but by legitimate users for its simplicity of use.
Alternatively, and assuming I'll have to rewrite my script to use the heavier-weight LWP::UserAgent module, can you suggest a specific user agent my script should be using to avoid this arbitrary ban in the future?
Jeffrey T. Darlington
General Protection Fault
http://www.gpf-comics.com/
--
You received this message because you are subscribed to the Google Groups "browscap" group.
To unsubscribe from this group and stop receiving emails from it, send an email to browscap+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/27e382f7-53ad-4891-8f6a-79fb5945fb2e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
I am following all the rules, fetching once a week and checking the version number first. Only for some amount of time, when using Perl::LWP, I now receive the (Cloudflare-branded, but they blame browscap.org) error message:
The owner of this website (browscap.org) has banned your access based
on your browser's signature (434ac75e55612531-ua24).
The IP isn't banned - if I change the request to use libCURL it works just fine. This happens on multiple servers.
--
You received this message because you are subscribed to the Google Groups "browscap" group.
To unsubscribe from this group and stop receiving emails from it, send an email to browscap+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/4a4da8ef-7ff6-4d7f-a5a7-25b7533721a0%40googlegroups.com.
For what it's worth, I was the original reporter for this issue. I ended up switching to having my script call my system's copy of wget and haven't had an issue since. Why Cloudflare distrusts Perl libraries but allows wget (which can just as easily be scripted) is beyond me.
Jeffrey T. Darlington
General Protection Fault
Hi Jay,Indeed we use Cloudflare in front of the site; could you let me know your IP and I'll try and look up in the CF firewall events for you.I haven't had to unban anyone by IP now for quite some time (which is good!) in our own system, but indeed that error comes from Cloudflare.Anyway - let me know the IP and I'll do some digging for you.ThanksJames
On Tue, 3 Jul 2018 at 17:52, Jay Dansand via browscap <browscap+APn2wQdQrYdduZJz3Eu0sx9-Ew2iK0H6-lNd7_B1BdtK32xmourCmIr@googlegroups.com> wrote:
Sorry to zombify this thread, but I just stumbled onto this same issue and wanted the maintainers to know this is going on (and may be quietly stomping on scripted updates for others as well).
I am following all the rules, fetching once a week and checking the version number first. Only for some amount of time, when using Perl::LWP, I now receive the (Cloudflare-branded, but they blame browscap.org) error message:
The owner of this website (browscap.org) has banned your access based
on your browser's signature (434ac75e55612531-ua24).
The IP isn't banned - if I change the request to use libCURL it works just fine. This happens on multiple servers.
--
You received this message because you are subscribed to the Google Groups "browscap" group.
To unsubscribe from this group and stop receiving emails from it, send an email to browscap+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/4a4da8ef-7ff6-4d7f-a5a7-25b7533721a0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "browscap" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/CAF5o2a82N7QdE8foekjV76_D2ciyyAbRGPCLGCw74qVq%3D2w0Mg%40mail.gmail.com.To unsubscribe from this group and stop receiving emails from it, send an email to browscap+unsubscribe@googlegroups.com.
Thanks for looking into it!I'm pretty sure the issue isn't an IP ban because switching the user agent works, but one of the affected servers is 143.44.200.14. Here's hoping the events log will shed more light on why Cloudflare is blocking LWP.
On Tue, Jul 3, 2018 at 11:56 AM, browscap on behalf of James Titcumb <brow...@googlegroups.com> wrote:
Hi Jay,Indeed we use Cloudflare in front of the site; could you let me know your IP and I'll try and look up in the CF firewall events for you.I haven't had to unban anyone by IP now for quite some time (which is good!) in our own system, but indeed that error comes from Cloudflare.Anyway - let me know the IP and I'll do some digging for you.ThanksJames
On Tue, 3 Jul 2018 at 17:52, Jay Dansand via browscap <browscap+APn2wQdQrYdduZJz3Eu0sx9...@googlegroups.com> wrote:
Sorry to zombify this thread, but I just stumbled onto this same issue and wanted the maintainers to know this is going on (and may be quietly stomping on scripted updates for others as well).
I am following all the rules, fetching once a week and checking the version number first. Only for some amount of time, when using Perl::LWP, I now receive the (Cloudflare-branded, but they blame browscap.org) error message:
The owner of this website (browscap.org) has banned your access based
on your browser's signature (434ac75e55612531-ua24).
The IP isn't banned - if I change the request to use libCURL it works just fine. This happens on multiple servers.
--
You received this message because you are subscribed to the Google Groups "browscap" group.
To unsubscribe from this group and stop receiving emails from it, send an email to browscap+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/4a4da8ef-7ff6-4d7f-a5a7-25b7533721a0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "browscap" group.
To unsubscribe from this group and stop receiving emails from it, send an email to browscap+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/CAF5o2a82N7QdE8foekjV76_D2ciyyAbRGPCLGCw74qVq%3D2w0Mg%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "browscap" group.
To unsubscribe from this group and stop receiving emails from it, send an email to browscap+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/CA%2BqvCsi%2B67qo0gC2ViyR0%3DuNx145ZO23ZWhiRZbravncHWfPPA%40mail.gmail.com.
Folks,Oddly, there's nothing showing in the firewall events for that IP :/A quick google suggests might help if you use set up LWP to use cookies? https://stackoverflow.com/questions/29057331/waiting-for-cloudflare-ddos-protection-lwp-perlIf not, can you change the User Agent at all? By setting something identifiable (i.e. not whatever the default of the LWP library is) that might help to distinguish the requests? Looks like it's the "Browser Integrity Check" kicking in (see https://support.cloudflare.com/hc/en-us/articles/200170086-What-does-the-Browser-Integrity-Check-do- ) so changing user agent (e.g. to one that Chrome/Edge/Firefox might use) may indeed help...ThanksJames
On Tue, 3 Jul 2018 at 18:12, browscap on behalf of Jay Dansand <brow...@googlegroups.com> wrote:
Thanks for looking into it!I'm pretty sure the issue isn't an IP ban because switching the user agent works, but one of the affected servers is 143.44.200.14. Here's hoping the events log will shed more light on why Cloudflare is blocking LWP.
On Tue, Jul 3, 2018 at 11:56 AM, browscap on behalf of James Titcumb <brow...@googlegroups.com> wrote:
Hi Jay,Indeed we use Cloudflare in front of the site; could you let me know your IP and I'll try and look up in the CF firewall events for you.I haven't had to unban anyone by IP now for quite some time (which is good!) in our own system, but indeed that error comes from Cloudflare.Anyway - let me know the IP and I'll do some digging for you.ThanksJames
On Tue, 3 Jul 2018 at 17:52, Jay Dansand via browscap <browscap+APn2wQdQrYdduZJz3Eu0sx9-Ew2iK0H6-lNd7_B1BdtK32xmourCmIr@googlegroups.com> wrote:
Sorry to zombify this thread, but I just stumbled onto this same issue and wanted the maintainers to know this is going on (and may be quietly stomping on scripted updates for others as well).
I am following all the rules, fetching once a week and checking the version number first. Only for some amount of time, when using Perl::LWP, I now receive the (Cloudflare-branded, but they blame browscap.org) error message:
The owner of this website (browscap.org) has banned your access based
on your browser's signature (434ac75e55612531-ua24).
The IP isn't banned - if I change the request to use libCURL it works just fine. This happens on multiple servers.
--
You received this message because you are subscribed to the Google Groups "browscap" group.
To unsubscribe from this group and stop receiving emails from it, send an email to browscap+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/4a4da8ef-7ff6-4d7f-a5a7-25b7533721a0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "browscap" group.
To unsubscribe from this group and stop receiving emails from it, send an email to browscap+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/CAF5o2a82N7QdE8foekjV76_D2ciyyAbRGPCLGCw74qVq%3D2w0Mg%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "browscap" group.
To unsubscribe from this group and stop receiving emails from it, send an email to browscap+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/CA%2BqvCsi%2B67qo0gC2ViyR0%3DuNx145ZO23ZWhiRZbravncHWfPPA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "browscap" group.
To unsubscribe from this group and stop receiving emails from it, send an email to browscap+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/CAF5o2a9LGUBu8ifi-gFMuA34VSpkMBpx9cLs%3DW7WYNdQWysHtQ%40mail.gmail.com.
To unsubscribe from this group and stop receiving emails from it, send an email to browscap+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/4a4da8ef-7ff6-4d7f-a5a7-25b7533721a0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "browscap" group.
To unsubscribe from this group and stop receiving emails from it, send an email to browscap+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/CAF5o2a82N7QdE8foekjV76_D2ciyyAbRGPCLGCw74qVq%3D2w0Mg%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "browscap" group.
To unsubscribe from this group and stop receiving emails from it, send an email to browscap+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/CA%2BqvCsi%2B67qo0gC2ViyR0%3DuNx145ZO23ZWhiRZbravncHWfPPA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "browscap" group.
To unsubscribe from this group and stop receiving emails from it, send an email to browscap+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/CAF5o2a9LGUBu8ifi-gFMuA34VSpkMBpx9cLs%3DW7WYNdQWysHtQ%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "browscap" group.
To unsubscribe from this group and stop receiving emails from it, send an email to browscap+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/browscap/CA%2BqvCsisGaKRwDWadWj0zZJq4QZ-9a9nEQ2tH%2Brufsgv2ABKgg%40mail.gmail.com.