Rate limit for HSP API and academic use allowance
Selom Caleb Arcmann Ackummey
I'm collecting historical performance data for five routes from Liverpool Lime Street across a twelve-month period, querying serviceMetrics per day and time band, then serviceDetails per service. After successfully retrieving around one route-month of data, I began receiving sustained 429 responses, and these have persisted well beyond short retry intervals.
I have three questions:
1. What is the documented rate limit for the HSP API (requests per minute, hour or day), and over what window does the quota reset? I would like to pace my collection properly rather than repeatedly hitting the limit.
2. Is there any provision for a higher allowance for academic or research use? If not, I'm happy to work within the standard limit, but knowing the exact figure would let me schedule collection accordingly.
3. The serviceDetails response returns a numeric late_canc_reason (e.g. 886). Where can I find the lookup that maps these numeric codes to their meanings or to the standard delay attribution codes?
My work is non-commercial, and I'm caching all responses locally so that no request is ever repeated.
Thank you for any guidance you can offer.
Peter Hicks
I'm collecting historical performance data for five routes from Liverpool Lime Street across a twelve-month period, querying serviceMetrics per day and time band, then serviceDetails per service. After successfully retrieving around one route-month of data, I began receiving sustained 429 responses, and these have persisted well beyond short retry intervals.
I suggested, quite some time ago, that the underlying HSP database was published so that people like you didn't have to scrape data to get a set of data. That suggestion didn't go anywhere.
1. What is the documented rate limit for the HSP API (requests per minute, hour or day), and over what window does the quota reset? I would like to pace my collection properly rather than repeatedly hitting the limit.
As far as I'm aware, there is no published guidance on how many requests can be made over what period of time, and whether it's per IP address or per calling user. You might be able to email CACI Support to ask, and if you get some answers, please feel free to report them here!
2. Is there any provision for a higher allowance for academic or research use? If not, I'm happy to work within the standard limit, but knowing the exact figure would let me schedule collection accordingly.
I'm not aware there's any solution for bulk usage - the original platform I worked on was designed to handle something like 100 requests per second, but the platform which went live some years later was a wholesale rebuild and I don't think was load-tested.
3. The serviceDetails response returns a numeric late_canc_reason (e.g. 886). Where can I find the lookup that maps these numeric codes to their meanings or to the standard delay attribution codes?
The late/cancellation reason numeric data is in the Darwin reference data, which you can download from several places including the Open Rail Data Wiki and probably the Darwin product on the RDM. They don't map on to two-character reason codes as used in Network Rail, as Darwin is designed to be customer-friendly. For this reason, you might want to re-think you use of the HSP and instead make an FOI request for the same data from Network Rail.
Peter