Clarification of "X-Experience-API-Consistent-Through" header

[James Snowden]

Currently implementing the xAPI so if there's a better place for such questions perhaps you can point me to it. There are probably going to be more of them ;)

According to the spec, the LRS must include the "X-Experience-API-Consistent-Through" header on all responses to statement requests, 'with a value of the timestamp for which all Statements that have or will have a "stored" property before that time are known with reasonable certainty to be available for retrieval'.

I can't see the meaning of this header in the context of a GET request. If the statement isn't yet available for retrieval, how/why are we responding to requests to retrieve it? I can see the potential use-case in a response for a PUT/POST request, but it's listed as a requirement for GET.

Can anybody enlighten me here?

Thanks

James

[Andrew Downes]

Hi James,
An LRS might take some time to process statements it receives before they are available for retrieval. That header lets the client receiving statements know that the list of retrieved statements may be missing statements stored after the timestamp provided.

Does that help?

Andrew

[Andrew Downes]

The phrase "with reasonable certainty" does worry me a bit as it means that statements stored before the timestamp could be missed out. This could mean some statements get lost in systems that pull all statements from an LRS at regular intervals using the since parameter.

I may raise an issue for this on github later. LRS vendors - how much pain would it cause you if we remove that phrase?

Andrew

[James Snowden]

Hi Andrew,

Thanks for your prompt reply. That clears up my question, I had it a bit backwards in my head.

James

[Ben Clark]

Hi Andrew,

Russell convinced me that phrase should be in there, so let me try to explain. The point of including the header is to explicitly allow scenarios where an LRS accepts statements after doing initial validation, and writes them to an incoming statement queue of some sort, and then responds with a success status to the request to store the statement, while asynchronously persisting that statement in the LRSs main statement store which will drive queries. (Note, this includes writing to any sort of distributed DB even if there is no 'queue' concept at the application level). 

Consider the case where a machine involved in the process of that asynchronous save crashes. The LRS will presumably find that it has some statements in its queue that have not be successfully persisted, but will only detect it when those crashed machines come back up or some sort of recovery is run on the storage they were writing to. At that point, since we already acknowledged the statement it makes sense to continue to persist it to the main LRS tables.

So another way to put this could be "except in extraordinary circumstances", but without that sort of caveat LRSs will either have to ensure the statement is written everywhere it should finally wind up before returning success (which effectively negates the benefits of using distributed DBs), or it would sometimes fail to meet this requirement. It's quite a corner case though.

Sorry I didn't send this reply sooner, just noticed I wrote a draft and then didn't send.

Regards,

-Ben

Andrew

To unsubscribe from this group and stop receiving emails from it, send an email to xapi-spec+...@adlnet.gov.