RaceDB and robots.txt

[stuart lynne]

I just enabled access (temporarily) to my RaceDB system to outside world, so that someone could clean up the teams for 2026.

Just happened to leave a terminal window open tailing apache logs, and see openai crawler busily crawling the hub results.

I'm not quite sure how big of a problem this is, although if it got into the non Hub area and logged in, it would have access to more licenseholder data (DoB etc.)

It did try and get a robots.txt file. But I don't know if it is possible to add that easily in RaceDB/django.

Seemed to be easy to add a 404 to any requests to /hub in the apache2 server I was using to redirect and proxy.

104.210.140.138 - - [12/Mar/2026:12:40:15 +0000] "GET /robots.txt HTTP/1.1" 404 435 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatib
le; OAI-SearchBot/1.0; +https://openai.com/searchbot"





acedb.wimsey.co:80 74.7.241.57 - - [12/Mar/2026:23:29:01 +0000] "GET /RaceDB/Hub/ResultAnalysis/381/0/1281/ HTTP/1.1" 404 439 "https://racedb.wimsey.co" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible
; GPTBot/1.3; +https://openai.com/gptbot)"
racedb.wimsey.co:443 74.7.241.57 - - [12/Mar/2026:23:29:01 +0000] "GET /RaceDB/Hub/ResultAnalysis/29/1/230/QRCode/ HTTP/1.1" 200 3756 "https://racedb.wimsey.co/RaceDB/Hub/ResultAnalysis/29/1/230/" "Mozilla/5.0 AppleW
ebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.3; +https://openai.com/gptbot)"
racedb.wimsey.co:443 74.7.241.57 - - [12/Mar/2026:23:29:02 +0000] "GET /RaceDB/Hub/ResultAnalysis/410/0/13072/ HTTP/1.1" 200 22584 "https://racedb.wimsey.co/RaceDB/Hub/LicenseHolderResults/1958" "Mozilla/5.0 AppleWeb
Kit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.3; +https://openai.com/gptbot)"
racedb.wimsey.co:443 74.7.241.57 - - [12/Mar/2026:23:29:02 +0000] "GET /RaceDB/Hub/ResultAnalysis/212/0/9695/ HTTP/1.1" 200 22954 "https://racedb.wimsey.co/RaceDB/Hub/LicenseHolderResults/1718" "Mozilla/5.0 AppleWebK
it/537.36 (KHTML, like Gecko; compatible; GPTBot/1.3; +https://openai.com/gptbot)"
racedb.wimsey.co:443 74.7.241.57 - - [12/Mar/2026:23:29:03 +0000] "GET /RaceDB/Hub/ResultAnalysis/177/0/8740/ HTTP/1.1" 200 22425 "https://racedb.wimsey.co/RaceDB/Hub/LicenseHolderResults/6449" "Mozilla/5.0 AppleWebK
it/537.36 (KHTML, like Gecko; compatible; GPTBot/1.3; +https://openai.com/gptbot)"
racedb.wimsey.co:443 74.7.241.57 - - [12/Mar/2026:23:29:03 +0000] "GET /RaceDB/Hub/ResultAnalysis/232/0/6543/ HTTP/1.1" 200 29767 "https://racedb.wimsey.co/RaceDB/Hub/LicenseHolderResults/4030" "Mozilla/5.0 AppleWebK
it/537.36 (KHTML, like Gecko; compatible; GPTBot/1.3; +https://openai.com/gptbot)"
racedb.wimsey.co:443 74.7.241.57 - - [12/Mar/2026:23:29:04 +0000] "GET /RaceDB/Hub/ResultAnalysis/230/0/10889/ HTTP/1.1" 200 23233 "https://racedb.wimsey.co/RaceDB/Hub/LicenseHolderResults/6610" "Mozilla/5.0 AppleWeb
Kit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.3; +https://openai.com/gptbot)"
racedb.wimsey.co:443 74.7.241.57 - - [12/Mar/2026:23:29:04 +0000] "GET /RaceDB/Hub/ResultAnalysis/431/0/12781/ HTTP/1.1" 200 27232 "https://racedb.wimsey.co/RaceDB/Hub/LicenseHolderResults/7622" "Mozilla/5.0 AppleWeb
Kit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.3; +https://openai.com/gptbot)"
racedb.wimsey.co:443 74.7.241.57 - - [12/Mar/2026:23:29:05 +0000] "GET /RaceDB/Hub/ResultAnalysis/232/0/6601/ HTTP/1.1" 200 23221 "https://racedb.wimsey.co/RaceDB/Hub/LicenseHolderResults/6610" "Mozilla/5.0 AppleWebK
it/537.36 (KHTML, like Gecko; compatible; GPTBot/1.3; +https://openai.com/gptbot)"

--

__________O___________
_______-\<,____________
_____(_)/_(_)___________
_________________________
Stuart_Lynne____<stuart...@gmail.com>____604-518-1749(m)__604-461-7532(h)