A closer look at the Lift and Escalator API v2
Peter Hicks
1. Apache HttpClient seems to have a bug where if you make an HTTP POST without any content (which is permissible, to my surprise!), it won't send a zero Content-Length header, and the remote server will send back a 411 Length Required. Not a problem with the API as such, but I was also trying to send the x-lne-api-key in the POST body which didn't work either. Likely my fault.2. The JSON Web Token (JWT) returned from a call to POST /api/token appears to have an invalid signature. It also includes a payload referencing "Demo User", and the user_id field of the JSON changes every time I request a new token. I'm not sure what's going on here.3. The 'Get Assets by Station Code' REST query fails with a validation error, even from the web interface, so something's wrong4. The 'Get Sensors' REST query works, but has no data indicating where the sensor is located, because the 'Get Assets by Station Code' query fails. Also, getting a list of sensors and no locational metadata seems a bit silly unless you already know where everything is5. The 'Get Asset Info by ID' REST query returns an error, as it seems to want to use the 'publicFacing' field in the 'assets' objects which no longer exists6. Introspection is disabled on the v2 GraphQL interface, which makes it tricky to explore given the documentation is lacking (and having documents on Google Drive isn't really that great, because you can't see when and what was changed in them)7. The v2 documentation at https://docs.google.com/document/d/e/2PACX-1vTBUhyCXJoCvKUdIjE5GSyV3CqPgXg5d4K7DAghGilat4j8DTlWwMaTIQwTee-zvifFuW4NVB_0Hd5S/pub suggests a query under 'CRS codes' with the fields 'inCommision' and 'publicFacing', which aren't valid. Remove these and it works.8. The documentation suggests that more than 15 requests per minute will result in an HTTP 200 response with the error in the body. An HTTP 429 'Too Many Requests' response would be more suitable, because a 200 suggests the API call worked... except it didn't9. I can make GraphQL queries to /gateway/v2 without authentication, but I need authentication on /graphql/v2 - this seems like a mis-feature10. The documentation suggests I need to supply the x-lne-api-key on both the token request endpoint (fine) and on the GraphQL query endpoint, and in the latter, I need to specify the bearer token as well. In reality, I can get away with just specifying the API key, no need for the token. Why do I need to request and send a token if it's not even used?
Watch this space for updates.
Alireza Parandeh
Hi Peter,
Thank you for thoroughly testing the API, contributing to the API documentation and for your feedback on the V2 release! I appreciate the speedy feedback as we only released the V2 last Friday.
I've passed your feedback to the API team to rectify any of the bugs you've identified.
Please see my responses to the above queries below:
L&E V2 API Endpoints and Portal URLs
The L&E V2 Portal is located at: https://portal.nr-lift-and-escalator.net
In regard to the API endpoints, they should be listed beginning with the following URL: https://nr-lift-and-escalator.azure-api.net/
We never had initially planned to release the https://nr-lift-and-escalator.net/ endpoint to the public, however, we have made it available under the request of the owner of the API; Network Rail. We've had time to incorporate user feedback from our public preview V1 for the full V2 release, and continue to welcome more feedback!
Sending zero Content-Length header
Sounds like you’re attempting to send the x-lne-api-key header without a value?
We're not familiar with the Apache HttpClient setup - Do you have some time this week to have a chat about this for me to note down your experience and input?
/gateway/v2 vs /graphql/v2 endpoints
We wish to make the API available to all users without the requirement of signing up, by authenticating as an anonymous user. There are many reasons, such as testing the API, and why a user may not wish to or not be able to sign up. Please consider that the API is designed to be available for accessibility users who may have additional requirements.
Users who do not wish to sign up will not be able to:
- Access the new features (such as the stations table introduced in V2)
- Access the interactive portal for API testing
- Be notified of any changes to the API
- Have full access to the help provided by the website and help desk
All users will still benefit from any data fixes brought in by the V2 update.
Validation Errors caused by missing required headers - For instance, The 'Get Assets by Station Code' REST query fails with a validation error
The validation error is what you receive when not the authenticated correctly. You require both your JWT token in Authorization header and your subscription key in x-lne-api-key header. This includes the stations endpoint that requires access to the stations table. We are investigating the limitations on sending detailed error responses using the current Azure platform setup.
This should be encompassed by the V2 API Authentication Docs, which are on the documentation.
Any unauthenticated users trying to access the new V2 content will receive the validation failed error.
The 'Get Sensors' REST query endpoint
By providing the correct authentication headers you should be able to query the station assets and then Get Sensor Info By ID endpoint to receive the status of asset sensors. This endpoint is mostly used to create a table of sensors in a UI to investigate them one by one using Get Sensor Info By ID endpoint.
The 'Get Asset Info by ID' REST query endpoint
This is now reported to the team and fixed by the time of this response. The publicfacing field is no longer required.
Requiring both Authentication JWT Token + Subscription Key in GraphQL Endpoint
This is for security reasons. You can only generate a subscription key via the portal in a browser and generate your JWT token by passing your subscription token on the server to the /auth endpoint.
We feel it is safer to do the user authentication (name/id + password) on the API portal, rather than transmitting personal user information across different cloud instances. The GraphQL endpoint asks for a pair of keys to ensure the client and their server are both authenticated.
inCommission and publicFacing fields
In V2 API only public-facing and in-commission assets are returned by default. Therefore, these fields are no longer exposed in the V2 public API. We have made adjustments to the inCommission and publicFacing fields on the documentation, since these are not fields available to all users.
GraphQL Schema Introspection
On V2 the schema introspection is disabled for security reasons but can be viewed through the GraphQL console in the portal: https://portal.nr-lift-and-escalator.net/api-details#api=graphql-api-test
V2 API Documentation
I understand the documentation can be improved. Would you please be able to provide specifics on how to extend it?
Demo User shown in JWT + API Rate Limit Reached Error Response
Thank you for this report - These have been identified as bugs by the engineering team. These should not affect anyone using the API. I've passed these to the developers to see if this can be addressed.
The user_id is mocked as we validate only using a key from the server and client to validate your user permissions on the API backend - hence the required x-lne-api-key. We feel it is safer to do the user authentication (name/id + password) on the API portal, rather than transmitting personal user information across different cloud instances. We can look into the issues with values of the JWT token headers such as user id, however, it is not required for validation.
Further issues & Feedback
I encourage all to reach us at networkra...@hackpartners.com for reporting further issues. I will make sure these are addressed to improve the quality of the L&E V2 API and documentation.
Ali
------------------------------------------
Hack Partners Limited is a company registered in England and Wales. Registered Address: Hack Partners, 10 Market Place, Devizes, Wiltshire, England, SN10 1HT. Registered Number: 9274301.
This e-mail, and any files transmitted with it, are confidential and intended solely for the use of the individual or entity to whom it is addressed. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. If you have received this e-mail in error, please notify the sender by return and then destroy it.
The views and opinions expressed in this email are the author’s own and may not reflect the views and opinions of Hack Partners Limited.
This e-mail message has been scanned by antivirus software, however Hack Partners Limited cannot guarantee that it is virus free.
Evelyn Snow
I've taken some time today to investigate V2 of the lifts and escalators API
for myself.
To go over Peter's observations, skipping anything I haven't observed or
isn't applicable:
3. Currently works, but requires bearer auth in addition to x-lne-api-key,
unlike any other non-graphql endpoint that works
4. Still contains no information on location
5. Broken but for another reason: Field 'uprn' not found in type: 'assets'
6. Haven't tried GraphQL introspection; of course, the documentation is
still on google docs
7. No longer in the documentation, but the query builder on Azure still
helpfully has inCommision and publicFacing
8. Not changed in the documentation (presumably still the case?)
9. This one was seemingly fixed
10. As it stands, bearer authentication is mandatory for:
- The stations table in GraphQL
- One of the REST endpoints (maybe one other too, but that's broken for
other reasons so I'm not sure), mostly just because they're very thin
wrappers around GraphQL
- Going three layers down in GraphQL (There's only three tables anyway)
x-lne-api-key authentication is mandatory for:
- All of the REST endpoints
Normal authentication for normal logic tables for normal people!
My own observations:
11. Despite updatedTime appearing in the documentation for the status table,
it doesn't exist any more, and you can't select it. This means there's
seemingly now no way to tell how stale the information is.
12. The documentation states that V1-equivalent information in V2 graphQL
doesn't require authentication, this is not the case. If you're not
authenticated, you can't access the stations table, which contains no
new fields except CRS (which is also in the assets table in any case!)
13. The sensors REST endpoint has two nominally optional parameters, num and
offset, the documentation doesn't explain these, but they appear to be
for paging. This would be fine, if the default for num wasn't 50. There's
fortunately no upper limit, if you set this to 20000 you'll get all of
them (but be warned! beware! it'll take up 170K on your hard disk!)
14. The REST API is very much an afterthought, it's acting as a very thin
wrapper around GraphQL, and none of the endpoints do what I'd want in any
case, i.e. giving all of the information at once with no filtering.
One simply doesn't work, and since Peter also reported this same as being
broken a year ago, I wonder – did it ever work?
15. For that matter, why is this GraphQL? This is the equivalent of a few
thousand rows of a flat table, no more than a few megabytes, and it's
being treated like linked data – why? An overhaul could have been an ideal
time to revisit a decision to smash an acorn with an elephant.
16. There's three separate documents for documentation, all hosted on google
docs. They're incomplete, contain factual errors, and contain a degree of
overlapping information.
17. The security stance doesn't appear to make sense. I can request almost
everything from GraphQL without authentication (albeit not stations), and
yet document 1 justifies token rotation as offering "enhanced security".
I've also noticed that I'm regularly confronted with cloudflare "checking
if the site connection is secure".
18. I notice that the portal has been updated to feature prominent "CrossTech"
and "Powered by CrossTech" branding. For those unaware, CrossTech is the
new trading name of Hack Partners Ltd, I note that the footer also informs
me that they are, apparently, "Leaders in AI technology for transport
efficiency and safety".
All this is to say that it's quite disappointing to see that we've had a year
of v2 of the lifts and escalators API, without some of the significant issues
being resolved in that time.
Evelyn
Peter Hicks
--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/openraildata-talk/ZPudZ5JKjwbT/QcC%40evelyn.moe.
Evelyn Snow
It's entirely possible the updated time is still in there somewhere, I've been
going off the official documentation, and the query builder on the portal, but
neither of these is an accurate description of the actual underlying schema.
Since we don't have introspection, ostensibly for security reasons, I think the
only real approach here is to ask Hackpartners themselves. I haven't yet
approached them with my concerns, but lack of last updated time is one of the
more serious regresions/problems in my opinion, since it can't be worked around.
I'd also tend to agree that some sort of small cache or proxy would be the
approach to take here. We can try to paper over the cracks with documentation
littered with caveats and warnings, but that doesn't make the feed less arcane
to use, it just means we're the ones stuck writing the spellbook!
Evelyn
Evelyn Snow
Seems that in spite of what Hack Partners have said about this, it's possible
to gain some information through introspection. In particular, I can confirm
that there is no field for the last updated time.
https://wiki.openraildata.com/images/6/6c/Nr_lift_escalator_api_introspected_schema_2023-09-09.txt
Evelyn