Open Rail Data Wiki slowness

[Peter Hicks]

All,

Yesterday, as an experiment, I turned off 'under attack' mode on Cloudflare which protects the Open Rail Data Wiki.  Unfortunately, numerous bots started to scrape the content again, resulting in response times going from "fine" to "nope":

I've switched 'under attack' mode back on, because this is the new normal.  Thanks. AI(!)

Peter

[Mike Wooldridge]

Hi Peter.

Just be aware (I look after RailUK Forums technically and have been battling with similar bot-related issues in recent weeks) - Under Attack mode is undesirable if you want legitimate bots (such as Google's crawlers) to be able to access the Wiki too, and Cloudflare does give you a reasonable level of control, even on the free tier.

I'm happy to share the much more granular set of rules I've settled on as a compromise with you if that'd be helpful.

Cheers,

Mike

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/openraildata-talk/oVyNDa-uC2hwBw5lsbchm0YYZ5WyzvPgAp9dINjcvBEOX9NHohDLQ1KJI4B6KNfe-ogh8alPD6kn06V4hDgMaDTqCb7JAjXSHuHjjb2h7Ik%3D%40poggs.co.uk.

[Matthew Burdett]

Hi Peter

I've always found these bots to be outside of the UK. 

One of my own site rules presents all non-UK traffic with a "Managed" JS challenge. It therefore only shows that under attack mode page to non UK visitors. I exclude USA for Google bot.

It can be found under Security -> Security Rules

--

[Peter Hicks]

Hi Mike

On Friday, 18 September 2026 at 15:41, Mike Wooldridge <m...@mikewooldridge.co.uk> wrote:

Just be aware (I look after RailUK Forums technically and have been battling with similar bot-related issues in recent weeks) - Under Attack mode is undesirable if you want legitimate bots (such as Google's crawlers) to be able to access the Wiki too, and Cloudflare does give you a reasonable level of control, even on the free tier.

I'm happy to share the much more granular set of rules I've settled on as a compromise with you if that'd be helpful.

That would be immensely helpful, thank you!  Whilst I am not too concerned about pages not appearing in search engines, it's clearly useful for everyone if they do.

That, and Matthew's idea of whitelisting UK traffic seems like a sensible approach.

Peter