Demo Statement Review

[andrew...@watershedlrs.com]

Hi,

I'm spending some time today looking at the statements generated by the video profile demo. As the goal of profiles is to make data easier for reporting tools to handle, I think it's important that profiles are tested against existing tools including Watershed. I do a lot of testing of xAPI products so I'm treating this in the same way, first doing a visual review of the statements (using this checklist) and then testing the data against Watershed reports. While the testing is specifically in Watershed, I would assume that other reporting tools that go to a similar level of detail would also encounter similar issues and have similar feedback (It would be great to see if other LRS vendors that have reporting can do similar testing to confirm that).

As I've not been closely involved in the development of the video profile, so I don’t know how much of this feedback relates to the profile itself or to the demo. I hope that an outsider's perspective is useful too.

Here's feedback from my review of the statements (I'll post additional feedback from testing in reports later):

Interoperability with existing video implementations and organization data

There are a number of applications (e.g. DigitalChalk, Kaltura, some home-brewed implementations) that already track video with xAPI and organizations using these applications (I'm aware of multiple Kaltura implementations amongst Watershed's clients). Many of these implementations use the activity streams verbs and activity types following the video profile hosted on Registry. There are some advantages to using the w3id.org types in terms of metadata, but also costs in terms of lack of compatibility with existing video data. The more differences there are, the harder it will be to get existing implementers to adopt the new profile so I encourage you to take care that any differences add sufficient value to be worth that lack of backwards interoperability.

In addition to the differences in verb and activity type IRIs, there's also differences in the extensions used to record start and end point. In this case, I can't see any benefit to the change, and there is also a greater cost. It's relatively easy for a reporting tool to account for multiple verbs and activity types that mean the same thing, but much harder to reconcile completely different methods of recording pause and play positions.

On that last point, it’s worth noting that Kaltura does not implement the starting and ending point extensions from the Registry video profile. This is because when implementing xAPI in Kaltura MediaSpace, the developers used hooks for events that they already tracked: milestones for watching within the video at 25%, 50%, 75% and 100%, rather than play and pause positions. This approach actually ends up being pretty nice to work with from a reporting perspective since you can see how far through videos users are getting.

It looks like this use case could be supported by the video profile using the progress extension. Kaltura currently uses the extension https://w3id.org/xapi/cmi5/result/extensions/progress for this and I encourage you to re-use that extension, rather than creating a new one specifically for video progress.

Another extension that already exists is http://id.tincanapi.com/extension/browser-info. Since this extension is used across statements regardless of content type and use case, it’s even more important to re-use the extension here.

You should also consider removing the word “video” from some of the other generic extensions such as screen size, volume and session id with the expectation that these will be re-used in other profiles. Don’t fall into the trap of thinking the uri conformity is a good thing that provides any practical benefit. It doesn’t, but it does create practical problems.

Session id vs context.registration

Where possible it’s a good idea to use existing statement properties rather than coining extensions. Is there any reason not to use context.registration here?

Assuming the reason is something like “context.registration will be set by the application/course the video is embedded in”, consider recommending that registration is also set with the same value of session id if no other value of registration is available. This would help reporting features that specifically look at the context.registration property.

Time extension

The time extension is used to denote the position in the video where the event happened. It’s using a number data format to represent the number of seconds since the start of the video.

xAPI itself uses an ISO 8601 duration format to represent this sort of data. Because of that, Watershed has some specific ways of handling iso 8601 durations as durations (they get displayed in reports as durations rather than numbers) that can also be applied to extensions. Since all reporting tools have to handle xAPI durations, I would assume that they’d all need to develop functionality to handle iso 8601 durations at some point.

All that is to say, if you switch this extension to use an iso 8601 duration string it could make it easier for reporting tools to parse as a duration.

Played segments extension

If you only follow  one piece of my advice here, I hope this is the piece you’ll listen to. The played segments extension isn’t going to be usable by Watershed because of the way the data is formatted: combining multiple pieces of data in the same string.

This extension is re-using the data format used in response patterns for matching type questions. There’s two reasons why this is an odd and unhelpful thing to do:

1. This data format was only included in the xAPI spec for the purposes of backwards compatibility with SCORM. It’s generally agreed by the xAPI working group that this isn’t a good format and if there is a future major release of xAPI, I would assume that this data format would be dropped. See this spec issue.

2. The segments of a video that have been watched is a totally different set of data to the sources and targets matched in a matching question. It doesn’t make sense to re-use this data structure with a completely different meaning.

The data transmitted about segments is also problematic because the data is repeated in each paused statement. For a reporting tool to get a complete list of segments watched, they need to look at the last statement for each session only. This is more complex than just looking at all the paused statements to compile a list.

The video profile on registry already has a good way of recording the segments watched by using a separate statement with the verb ‘watched’ and the starting and end point extensions (which support either floats or ISO 8601 durations - you should restrict this to just ISO 8601 durations). I can’t see any reason why you shouldn’t continue this approach?

Nice to have: Result.duration

It might be nice to also record the amount of time spent watching the video

‘Reached the end’ verb

When I get to the end of the video the demo sends a statement that I paused the video. This is misleading because I didn’t actually pause the video and could throw off reports counting the number of pauses. Since the completed statement (ADL completed verb) is sent when I’ve watched every section of the video, there would need to be a different verb to represent getting to the end of the video.

The Registry profile uses the activity streams completed verb to represent this, which you could use, but it might be confusing to have two verbs with the same display in one profile even if the ids are different. Probably better to coin a new one.

I hope this is helpful feedback. Let me know if you have any questions, or if it would be helpful for me to attend a call one time to talk this through. I’m planning to test the data with watershed reports later, which may lead to some additional feedback.

Andrew

[andrew...@watershedlrs.com]

Having tested the data in configured reports, I didn't come across any new issues, but it did underline some of the issues outlined above. I wasn't able to find a way to report on the segments and the pause count was inaccurate as it showed extra pauses at the end of the video. To be fair, I could have filtered out pauses at the end of the video since I know the position of the end of the video, but in the real world with lots of videos of different lengths, the complexity of configuration required to filter out pauses at the end of every video would make this unfeasible. 

Below is a screenshot of the dashboard I configured. If anybody wants access to be able to drill down into and explore the reports, please let me know. The data set is VERY limited - 10 statements covering 1 person watching 1 video on 1 day, which is why several of the reports look very empty (the line chart is more of a dot chart because there's only one item on the y axis). I'd love to get my hands on a larger data set covering multiple people, videos and days if anybody has one.

Again, I hope this is helpful feedback for development of the profile. Please let me know if you have any questions!

Andrew

[J. Pablo Caballero]

Andrew, thank you for taking the time to share this.

I think that this CoP should consider all these points. It seems to me that aligning the profile with general xAPI practice (e.g. using ISO 8601 for time and durations) and general industry  (video) practice (more reuse of IRIs, etc.) is a good idea.

[Jason Haag]

Thanks for the detailed feedback and suggestions Andrew!

[Pankaj Agrawal]

Hi Andrew,

Thanks for testing and feedback.

I believe most of the points you mentioned have been discussed during our meetings.  


1. Using existing profiles

We looked at different profiles and implementations including tincan registry and GrassBlade xAPI Companion which uses an earlier implementation from ADL YouTube demo. We at GrassBlade use 25%,50%,75%, like Kaltura, however, I haven't seen their statements so I am not sure if that matches. However, when we started Video Profile we discussed all implementations and it was decided that a fresh implementation is required to achieve required use cases.

2. Using existing IRIs

I think Jason can talk about that better.

For progress we had considered both result.score.scaled as well as cmi5 progress. If I am not wrong I had personally talked to xAPI Spec community and cmi5 community and was discouraged to use these options. 

3. Session id vs context.registration

The two are used for different purposes in Video Profile. Session ID identifies one session which starts at a launch and finishes when the user closes the browser or if the session ends in any other way.

context.registration can be same in multiple sessions to determine continuity. If there are multiple launches for same registration, it means continuation of the registration and the played-segments/progress data will be used from last session (last paused statement for that registration) 

4. Played Segments

Format: Yes the format was chosen based on the format used in the spec.

 

I am not a fan of the formating as well. So, I am open to discussions. However, most LRSes already understand it, it should not be extra pain processing it. So, I am feeling (lazyish) on thoughts of making changes at this stage. 

Usage: We had many discussions on this, and probably some arguments too. However, after the demo is out, I guess, everyone has agreed to the utility. Few main usage:

(context: last paused statement is last paused statement for the specific registration) 

a. Easy to pull last paused statement for the specific registration and use played segments information to resume the video from where the user left last. 

b. Easy to pull last paused statement for the specific registration and use played segments information to build on new played segments, and eventually the calculation of progress and completion. 

c. Simplify sending completion based on actual completion of all parts of video. 

5. ISO Time

We had discussed ISO time format, and I personally hate it, and others did not like it either. Also calling "point in time" in the video, as duration sounds odd (not sure). Using some cryptic duration syntax just to change "10" to "10 seconds" in a reporting tool doesn't make sense. 

In the xAPI Spec there was only one good argument in favour of ISO time/duration format that I know of, and that doesn't apply to videos: 

Argument in favor of ISO time format: There can be duration in terms of weeks, months or years (for example, duration of a course) which can not be represented in seconds. 

 

6. result.duration
We already have it required with the completion statement. 

7.  ‘Reached the end’ verb

We had a lot of discussion on this.There was an earlier proposed and planned verb "ended". I was actually inclined towards keeping it, however, it was dropped by the community with these arguments: 

a. Current uses of 'reached the end' verb as completion verb is not useful  when a more complete and real 'completed' verb is available. 

b. 'reached the end' is already available as paused verb with time equalling the length of the video. So, two statements with same information is not required. 

8. "paused" for "Reached the end" 

Earlier most of the video players (both physical and virtual) used to have separate pause and stop buttons. And reaching the end was considered a stop event. 

However, almost all current video players use "paused" for the stop event. And hence we decided to continue using paused verb for two reasons:

1. Keep consistency with current video player events. 

2. Simplify utility of played segments by pulling last paused verb to facilitate resume, progress and completion behaviour. 

I hope it makes sense. 

Regards,
Pankaj Agrawal

--
You received this message because you are subscribed to the Google Groups "xAPI Video CoP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xapi-video-cop+unsubscribe@googlegroups.com.
To post to this group, send email to xapi-vi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/xapi-video-cop/c548dd5c-7142-4a89-8725-8f91c025d0dc%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[andrew...@watershedlrs.com]

Thanks for the responses Pankaj. Let me response back to few of these..

2. I'd be interested to hear the reasoning behind not using the cmi5 progress extension. I can't think of any benefit and the big disadvantage is you then have to look at two different extensions to report on progress across cmi5 launched things and videos. In the event you launch a video via cmi5 I guess you'd need to use both. 

3. Thanks for the explanation - that makes a lot of sense. 

4. Glad that you're looking into change this. With regards to the usage examples, all of these would be better off using the State Resource rather than the Statements resource. 

5. It's not a cryptic syntax; it's the ISO standard way of formatting data about durations. Surely the interoperbility of the data is more important than whether you like how the raw data looks? 

7a. I understand that having a completion verb that records when you watched all of the video is useful. But it's also useful to record that the video stopped when it reached the end rather than the user actually pausing it. It's untrue to record that the video was paused. 

7b. Its not feasible to expect a reporting tool to calculate reaching the end of the video based on the pause position because all; videos are not the same length. Consider a report on how many times people paused a video in a library of thousands of videos with thousands of different lengths. Even if the video length were available in the statement data (it's not), the logic to report on "count of number of statements where the verb is paused and the value of the position extension is not equal to the value of the video length extension" is so much more complex than "count of the number of statements where the verb is paused". That second logic would be the one to use if you don't use the paused verb at the end of the video. 

8.1. All the xAPI video platforms I've worked with don't use paused for when the video reaches the end and stops. 

8.2. Again, the State Resource (or the application's internal storage) is what should be used for bookmarking, not the Statement Resource.