Simulation/Gaming developer with questions

[Scott Casey]

The xAPI stuff is starting to show up in training system RFPs that I have been seeing.  The xAPI statements appear to be largely text based messages.  This isn't really a scaleable solution in a simulation context where enumerations are much more efficient as packet data.  Especially with the amount of messages that will be sent, not just from a single instance, but from many instances running in a single classroom, and many classrooms within a school house.  In addition, the data being stored will rapidly fill up disk space with all of the strings.  Is there work being done to optimize the statements?  Also, I am having problems finding a schema for the xAPI LRS so any link would be appreciated.

[Christian Glahn]

Hi Scott

Regarding your performance remarks, you want to use minimal statements. In fact you almost always want to use minimal statements unless for long term archiving. Statement issuers should use only those elements that are required for describing an experience for automatic processing.

At the beginning the XAPI semantics can appear bloated, but most elements are optional so applications can focus on what they need. This dramatically reduces the statement size. If you add data compression for the client server communication, this can handle large amounts of statements efficiently. We run XAPI apps with large cohorts in similar scenarios as you describe for several years and never really had a problem.

Also keep in mind that XAPI is designed for interoperability between different educational technologies. This means that other XAPI processors may hook into the activity stream that are completely independent from the initial system. This is the true scope of the spec: one can integrate different systems into complex learning experiences and link the data of these systems. There are similar specs from other domains, but they lack the verbosity that is required for education and training.

However, if minimal statements are really not enough for your performance requirements, you may have a look at the learninghub project :
https://github.com/janschneiderou/LearningHub

This LRS has been designed to handle high throughputs of sensor data for real-time analytics and feedback in multi-modal learning environments. I recall that Jan implemented upstream XAPI support for aggregate data. From the upstream perspective This will loose raw data that is only required in the concrete situation, but keeps the relevant information about the learning process. Normally, you won’t have to document that low level data outside the actual training situation, so for most cases this should not be a real problem.

Regarding the schema, you will find the spec repo helpful I recall a reference to a JSON Schema there.
https://github.com/adlnet/xAPI-Spec

Hope that helps your design decisions.

Christian

—
Innovative Technology-enhanced Learning:
EC-TEL and mLearn 2019 Conference 16-18 September 2019 at TU Delft.
Submission deadline for research papers and demos: 29 April 2019
http://www.ec-tel.eu/index.php?id=950
http://iamlearn.org/mlearn/call-for-papers/

[swarwi...@gmail.com]

Is there a product opportunity for a compressed record store? It would seem that there would be a high level of compression possible given the level of redundancy in the messages and the very limited number of times the information is accessed at the message level

[Scott Casey]

Thank you for your response.  I have a lot of experience developing interoperable simulations using DIS and HLA protocols.  I did not see a database schema for the LRS in the links that you have provided, although it may be buried within the other links on the various pages?  I had already read through a lot of the JSON based stuff for statements, but did not find anything detailed in order to implement an LRS. This is why I started this dialog.  I might be missing where an item in a statement is flagged as optional.  Again it is probably because I am use to standards (even draft) as defined by IEEE.   I am thinking xAPI folks might want to familiarize themselves with those standard protocols as a way forward when working with the training systems type use cases.  Reason being that there is a lot of verbose data in the form of strings that could be reduced to an enumeration represented as an integer.  The standard should then dictate how to decode those enumerations into meaningful information for the human reader.

As an example from https://github.com/adlnet/xAPI-Spec/blob/master/xAPI-Data.md#Appendix2A

{

"id":"fd41c918-b88b-4b20-a0a5-a4c32391aaa0",

"timestamp": "2015-11-18T12:17:00+00:00",

"actor":{

"objectType": "Agent",

"name":"Project Tin Can API",

"mbox":"mailto:us...@example.com"

},

"verb":{

"id":"http://example.com/xapi/verbs#sent-a-statement",

"display":{ 

"en-US":"sent" 

}

},

"object":{

"id":"http://example.com/xapi/activity/simplestatement",

"definition":{

"name":{ 

"en-US":"simple statement" 

},

"description":{ 

"en-US":"A simple Experience API statement. Note that the LRS 

does not need to have any prior information about the Actor (learner), the 

verb, or the Activity/object." 

}

}

}

}

This is what will be sent over the network for the above (in hex): 7ba9226964223a2266643431633931382d623838622d346232302d613061352d613463333233393161616130222ca92274696d657374616d70223a2022323031352d31312d31385431323a31373a30302b30303a3030222ca9226163746f72223a7ba99226f626a65637454797065223a20224167656e74222ca99226e616d65223a2250726f6a6563742054696e2043616e20415049222ca99226d626f78223a226d61696c746f3a75736572406578616d706c652e636f6d22a97d2ca92276657262223a7ba99226964223a22687474703a2f2f6578616d706c652e636f6d2f786170692f76657262732373656e742d612d73746174656d656e74222ca9922646973706c6179223a7b20a99922656e2d5553223a2273656e742220a997da97d2ca9226f626a656374223a7ba99226964223a22687474703a2f2f6578616d706c652e636f6d2f786170692f61637469766974792f73696d706c6573746174656d656e74222ca9922646566696e6974696f6e223a7ba999226e616d65223a7b20a999922656e2d5553223a2273696d706c652073746174656d656e742220a9997d2ca999226465736372697074696f6e223a7b20a999922656e2d5553223a22412073696d706c6520457870657269656e6365204150492073746174656d656e742e204e6f7465207468617420746865204c525320a9999646f6573206e6f74206e65656420746f206861766520616e79207072696f7220696e666f726d6174696f6e2061626f757420746865204163746f7220286c6561726e6572292c2074686520a9999766572622c206f72207468652041637469766974792f6f626a6563742e2220a9997da997da97da7d

This is 662 bytes.  You will have to forgive my ignorance, but I am not sure why you would send the entire question with possible responses, and then the response itself in a statement.  If I were implementing this, the database (LRS) would already have a tables the set up with the verbose information already.  Time Stamp instead of a string would just be an integer (4 bytes) per ISO 6081.  The Actor would be reduced to one integer ID (4 bytes) - ActorInstanceId, the email address string, string name, and "objectType" would just columns in the ActorIsntance table in the database.  Like wise verb would be split into two integer IDs (8 bytes), VerbTypeId and VerbId. And then finally Object would just be one integer ID (4 bytes), the description and name strings would already be defined in the database, you don't need to keep sending them.  The message would then be (line breaks for clarity):

TimeStamp

ActorInstanceId

VerbTypeId

VerbId

ObjectId

And could look something like this (again, in hex): 546865717569636b62726f776e666f786a756d70

This reduces the network traffic for this one statement from 662 bytes to 20 bytes (ignoring any padding - not needed in this example, but that can be mitigated through the standard).  Roughly 33 time smaller in size and there is no processor cycles used on compression/decompression or string compares - O(n) and contains the same exact information as the verbose version. For interoperability, a listener could either be listening to network traffic and already have queried the database for the human readable information that the IDs represent, or they can query the database (LRS) either periodically using UPDATE if the LRS is SQL based (still not sure if it is, need to see a schema for the LRS standard).  This will also make the database way more efficient, even if it is represented by a flat file.  I would really like to see the proposed database design for the LRS.

Again, this is coming from someone that is pretty much new to xAPI, but have done a lot of simulation training stuff with interoperability for live, virtual, and constructive simulations.

[Scott Casey]

Please ignore " You will have to forgive my ignorance, but I am not sure why you would send the entire question with possible responses, and then the response itself in a statement. "  I was looking at a different example, but that will likely be a separate conversation.  

And please don't take my questions as an attack on xAPI, I am just trying to figure out how it is to be implemented with training systems.  Thanks again. 

[Rob Chadwick]

Hey Scott,

This is Rob from Veracity. I worked at ADL for several years, and have a bit of experience with the DIS/HLA world as well has a long history with xAPI. Let me weigh in with what may be a controversial opinion. 

It sounds to me like you're looking for a wire protocol type solution for simulators - but a RESTFul web API is nothing like that.   

You might have the wrong technology for your use case. xAPI is explicitly a REST(like) web API, with all the overhead that entails. Not only are you restricted to the rather verbose JSON encoding, but you also must use UTF8. I think that on the wire, even decimal numbers like 10.53252353 are represented as a string of characters. That's not just xAPI - that's REST and HTTP. Moreover, absent optimizations like HTTP2 or Keep-Alive, you're even going to be setting up and tearing down connections with each POST.  

As far as efficiency: xAPI is intended to represent learner events: things like playing a video or answering a question. The most verbose use I could possibly support would be tracking clicks on user interface elements, like buttons. Even that should keep the cadence to no more than 1 statement per second per user. To be clear - one statement per second per user is an incredible rate, and tracking clicks is of debatable value. This is just a maximum case example. Most of the content on LRS.io sends between 1 and 10 statements every few minutes, and I think that is generally the typical rate.  So, in this case, a classroom of 24 students would generate roughly 24KB/s, and 24 statements per second. This should be well withing the capabilities of even a small LRS. The whole point of a web API to set on top of a massive infrastructure that already exists - HTTP. It's not even close to the most efficient way to transmit data. 

You should be using xAPI for tracking the final score, or the completion of some objective. Perhaps join/leave or play/pause events. If you're sending player position or mouse movements or head tracking, xAPI is not the right technology. 

Anyway, that's just my 2 cents. I'm interested to see what the rest of the community has to say. 

-Rob C

[Scott Casey]

Thanks Rob.  Ok, this is making more sense, I have been assuming that for simulators that the LRS would be like an after action review and assessment database that other applications (not the simulation producing the data) would be consuming.  I am only investigating xAPI as it is starting to creep into training systems requirement's in DoD request for proposals that are not web based applications.  So, being pro-active I started investigating.

[Christian Glahn]

Hi Scott 

Re Schema: I was referring to a JSON schema for validating the XAPI statements. There is no such thing as a database schema for XAPI because this is implementation specific from the viewpoint of the spec. 

Re data volume: I think your key misunderstanding is that you compare the JSON notation and binary data formats, which requires a separate parser, is not "human" readable, and not extensible. This is entirely unrelated to REST (sorry Rob).

I would not go as far as Rob and suggest what to use XAPI for, because what should be in the activity stream and document store really depends on the educational setting. Therefore, I recommend that you analyse the ecosystem of your learners and teachers and use XAPI for linking different tools and services to have better tracking for more complex assessment of learning and performance. You will find that XAPI can offer interesting integration patterns beyond web-based systems (for tested ideas see https://www.slideshare.net/phish108/new-patterns-for-adl-architectures-and-learning-designs, slides 32-39).

We use XAPI for multi-device and multi modal tracking on a finer granularity than Rob's case. On a device we track almost all learning relevant interactions from device sensors. This triggers up to 10 statements per second per device and user. In order to cope with this, we cascade several LRSes with different databases (and database schemas) for partial system intelligence. For us the internal storage of the LRS is determined by performance and device requirements. E.g., a dumb IoT system may use a FIFO storage or a key-value store, a smart PWA would use IndexDB whereas one backend would use a document DB and another backend would use a SQL DB. XAPI is entirely agnostic to that.

When it comes to payload volume, you find that many examples that include "definition" and "description" attributes in the statements. IMHO, these attributes only make sense for long term archiving and are not suited in production environments. The idea behind these descriptive attributes is to allow humans to evaluate the meaning of a statement when the statement issuers and documentation are no longer available. In a production environment you can strip these attributes and if extra information is needed, you can use the ID's as linked data sources in order to provide the same information.  

Also you want to abandon mbox in favour of openid URLs for data privacy reasons (see https://www.slideshare.net/phish108/bridging-xapi-into-higher-education-learning-analytics-ownership-and-privacy). 

Your example then becomes: 

{

"id":"fd41c918-b88b-4b20-a0a5-a4c32391aaa0",

"timestamp": "2015-11-18T12:17:00+00:00",

"actor":{

"objectType": "Agent",

"openid":"https://example.com/xapi/user/xaAdB5lU"

},

"verb":{

"id":"http://example.com/xapi/verbs#sent-a-statement"

},

"object":{

"id":"http://example.com/xapi/activity/simplestatement"

}

}

In compressed JSON this is 298 octets (because you have long URIs). You could bring it down to roughly 250 octets with some URL compression. Our statements contain a fair bit more information and are typically around 400 octets, which is compact enough for most interactions we have in our systems. In peak situations this leads to ~ 4KB/s per device and user. If you take that as a baseline and take a small higher education institution with 2000 students where all students have 2 devices and perform at peak level on both devices we talk about 16M/s for the payload. This is not really a challenge for today's infrastructure.

Finally you refer to the wrong specs when it comes to binary data compression in relation to XAPI, which leads to wrong assumptions about the ID value space: rfc4122 and rfc5905 ground on 128bit (16 octets) not 32bit (4 octets) values, although a 32bit format exists for timestamps if it is ensured that temporal granularity will never be an issue (and never can be closer than one might expect). Your 20 octet statement would therefore be really 80 octets. Arguably, this is still smaller than 250 octets, but now just by factor 3 and not 30. Moreover, with binary compression you also throw the semantics, all linked data concepts, and extensibility out the window. This adds complexity to a learning environment, which reduces extensibility.

I hope this helps to get your head around XAPI and performance. 

Christian.