WebIM: An Open Web Interactive Media Format
Adam Sobieski
WebM Community,
Hello. I wonder whether there is any interest in an open standard for state-of-the-art interactive videos. By interactive videos [1], I mean videos like the popular “Black Mirror: Bandersnatch”, “Puss in Book: Trapped in an Epic Tale”, and “Minecraft Story Mode”.
One can envision interactive videos providing viewers with menu options and utilizing JavaScript scripting environments provided by interactive video player applications. Depending on users’ interactions with menus, random number generators, JavaScript program logic, users’ settings and configurations, and other variables or data stored, persisted, and available through JavaScript scripting environments, interactive videos could branch to portions of video content.
I am interested in advancing interactive video technologies and in open standards. I am hoping to propose and to discuss a “WebIM” interactive video format. If there is some interest, perhaps a group of us could put together a fuller proposal?
Best regards,
Adam Sobieski
[1] https://en.wikipedia.org/wiki/Interactive_film
MahanStreamer Management
fond memories of playing those types of games when I was a young kid.
Also look into the Sega CD, Philips CDI, and 3DO video game consoles
from the 1990s.
And then eko.com / interlude (before they changed their name), but that was a
fad for like 2 weeks back when I was
like 13 or something like that. I think it's only because pewdiepie
said something about it that I went on the site.
Also look into WebTV's efforts to bring interactive video:
http://sepwww.stanford.edu/sep/jon/family/jos/webtv/developer3/itv/embedtv/
That being said, I'm not sure if I would necessarily base the
technology on one singular codec like webm. I could envision an
interactive video framework using webm as one of the many
codecs/formats supported by it. Let me know where you decide to take
your project.
Relevant links:
https://www.youtube.com/watch?v=nTY_uZKSFhc
https://www.youtube.com/watch?v=-ylUb_NuFBM
https://www.youtube.com/watch?v=9ZIbnZuqUuo (funny commercial)
https://www.youtube.com/watch?v=HCQRcinZYH8 (another funny commercial)
(Side note: who couldn't forget everyone's favorite FMVs - "Night Trap" and
"Plumbers Don't Wear Ties?)
> You received this message because you are subscribed to the Google Groups "WebM Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to webm-discuss...@webmproject.org.
> To view this discussion on the web visit https://groups.google.com/a/webmproject.org/d/msgid/webm-discuss/CH2PR12MB4184F4D705817F11E4C45017C5219%40CH2PR12MB4184.namprd12.prod.outlook.com.
Adam Sobieski
Thank you.
In addition to entertainment scenarios, I think about educational
applications of interactive video. More broadly, I think about educational applications of choose-your-own-adventure stories, interactive films, and serious games.
With respect to interactive storytelling and branching narratives, there exists an “authorial bottleneck”, or an exponential amount of narrative content required to create truly branching storylines. In (Stefnisson & Thue, 2018), the authors indicate that “authoring in the context of interactive storytelling is inherently difficult and there is a need for authoring tools that both enable and assist authors in the creation of new content.” New software tools can facilitate authoring interactive stories, can facilitate visualizing their structure, can facilitate writing screenplays for interactive films, can facilitate organizing the production of interactive films, can facilitate editing in post-production, and can facilitate outputting resultant content to files. Authoring interactive stories and producing interactive films is difficult and with new software tools we could expect more content.
Flash was deprecated in 2017 and reached end-of-life last year in 2020. There is a niche for new interactive video formats. I see your point about a framework or ecosystem with multiple codecs for interactive video; basing one on WebM and Matroska seems reasonable.
In these regards, there are some exciting Web scenarios to consider. Web browsers could play interactive videos, could provide the videos with secure JavaScript scripting environments (environments provided by the same engines which power WWW JavaScript scenarios), could sandbox them, could provide users with intuitive and familiar user permissions systems, and could securely handle origin/domain-based access to users’ configurations, settings, and other data persisted across interactive video viewing sessions. Perhaps there could be an <ivideo> HTML element someday.
It's an interesting set of technical topics.
Best regards,
Adam
Stefnisson, Ingibergur, and
David Thue. "Mimisbrunnur: AI-assisted authoring for interactive
storytelling." In Proceedings of the AAAI Conference on Artificial Intelligence
and Interactive Digital Entertainment, vol. 14, no. 1. 2018.
MahanStreamer Management
framework based on HTML5 and JS, no need to somehow create something
totally different. Unless you want to somehow abstract it so the
browser is one of the places it can run out of many, in that case it
sounds like a good idea.
The last thing we need is more web standards, browsers like Pale Moon
can't keep up and even Chromium doesn't get to them all. There should
be less standards, not more.
Adam Sobieski
Yes, I want to abstract it so that Web browsers would be but one of many places that new interactive video format(s) could be run. To run interactive videos, any video player software applications would, as envisioned, need to provide a JavaScript scripting environment (perhaps utilizing an engine like V8).
Advantages of popular new format(s) would also include enabling the processing of collections of interactive videos, e.g. with artificial intelligence tools, for various research projects [1][2]. To process large collections of interactive videos, the interactive videos would best be stored using standard format(s).
In addition to entertainment and educational uses of interactive stories, perhaps AI systems could, someday, be trained using choose-your-own-adventure stories, interactive videos, and serious games as they are trained, today, using classic video games (see also: [3][4][5]).
Best regards,
Adam
[1] Partlan, Nathan, Elin Carstensdottir, Sam Snodgrass, Erica Kleinman, Gillian Smith, Casper Harteveld, and Magy Seif El-Nasr. "Exploratory automated analysis of structural features of interactive narrative." In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, vol. 14, no. 1. 2018.
[2] Carstensdottir, Elin. Automated Structural Analysis of Interactive Narratives. Northeastern University, 2020.
[3] Ammanabrolu, Prithviraj, and Mark O. Riedl. "Playing text-adventure games with graph-based deep reinforcement learning." arXiv preprint arXiv:1812.01628 (2018).
[4] Yin, Xusen, and Jonathan May. "Comprehensible context-driven text game playing." In 2019 IEEE Conference on Games (CoG), pp. 1-8. IEEE, 2019.
[5] Hausknecht, Matthew, Ricky Loynd, Greg Yang, Adith Swaminathan, and Jason D. Williams. "Nail: A general interactive fiction agent." arXiv preprint arXiv:1902.04259 (2019).
Adam Sobieski
I would like to clarify on the Flash/SWF comparison and to indicate a cool scenario for interactive video.
Firstly, when considering interactive video and films, of primary interest is Flash/SWF’s ability to include scripts in videos and to go to – or to branch to – frames, segments, or chapters of videos, e.g. based on menu selections. Not under consideration for a new video format are Flash/SWF features for: shapes, gradients, bitmaps, shape morphing, fonts and text, sounds, sprites, 3D graphics, and so forth.
The cool scenario: with standard format(s) for interactive videos, some chatbots and dialogue systems could be stored as interactive videos. This could be useful for instructional materials, how-to videos, and automated call center systems. In addition to static interactive video files, new standards could support streaming scenarios such that video content could be dynamically generated on servers.
Interestingly, interactive video formats could include support for utilization of pronunciation lexicons and speech recognition grammars and/or be interoperable with remote speech recognition services to enable users to select menu options with spoken natural language. In Web browsers, this functionality could be achieved via the Web Speech API (https://wicg.github.io/speech-api/).
I will circle back in a few months with a more detailed letter. Thank you for the interesting discussion.
Best regards,
Adam
Jasim Peringattuthodi
Adam Sobieski
Steve Lhomme
Just saw this thread and thought I would bring some light on existing
building blocks in Matroska, the cousin of WebM.
In Matroska there are "ordered chapters" [1] which is basically telling
the player what parts of the file its playing and in what order.
There are also "chapter codecs" [2] that can attach commands to ordered
chapters when they start and/or end. You could plug a javascript based
codec that can seek to different parts of the edition of switch to
another edition based on user interactions.
The missing part is the visual interaction. That is the buttons on the
screen to make the choice with possible animations, highlights, etc.
There are "button tracks" that can embed overlay information like on
DVDs but it's not well defined and hardly ever used. And these buttons
tracks need to be able to talk to the "chapter codec". Or rather provide
some standardized information to the player so it can do some action,
regardless of the button track type, and regardless of the chapter codec
type.
So right now we're missing 2 parts:
- the interface between players and button tracks
- the interface between players and chapter codecs
Both have been supported in VLC for ~15 years as a proof of concept of
how you can transfer a DVD into an MKV file. VideoLAN has been looking
for GSoC students to help make this happen [3]. But so far no student
that would be capable of doing it has shown up.
As for the file vs web approach, I feel like WebM was originally a
format that you can put on a website and web browsers would be able to
deal with it properly. The same should be expected from interactive WebM
files. Seeking lag needs to be taken in account though to have a smooth
experience. For example while a choice is being made you might need to
preload the start of each option to continue playing when the choice is
final (or allow a gap between the deadline and the moment you actually
switch).
[1]
https://www.ietf.org/archive/id/draft-ietf-cellar-matroska-07.html#name-editionflagordered
[2]
https://www.ietf.org/archive/id/draft-ietf-cellar-matroska-07.html#name-menu-features
[3] https://wiki.videolan.org/SoC_2021/#Interactive_movie_support
>
> On Wednesday, June 2, 2021 at 7:17:27 PM UTC-4 mahans...@gmail.com wrote:
>
> This already exists/existed: It was called Flash/Shockwave. I have
> fond memories of playing those types of games when I was a young kid.
>
> Also look into the Sega CD, Philips CDI, and 3DO video game consoles
> from the 1990s.
>
> fad for like 2 weeks back when I was
> like 13 or something like that. I think it's only because pewdiepie
> said something about it that I went on the site.
>
> Also look into WebTV's efforts to bring interactive video:
>
> http://sepwww.stanford.edu/sep/jon/family/jos/webtv/developer3/itv/embedtv/
> <http://sepwww.stanford.edu/sep/jon/family/jos/webtv/developer3/itv/embedtv/>
>
>
> That being said, I'm not sure if I would necessarily base the
> technology on one singular codec like webm. I could envision an
> interactive video framework using webm as one of the many
> codecs/formats supported by it. Let me know where you decide to take
> your project.
>
> Relevant links:
> https://www.youtube.com/watch?v=nTY_uZKSFhc
> https://www.youtube.com/watch?v=-ylUb_NuFBM
> <https://www.youtube.com/watch?v=-ylUb_NuFBM>
> https://www.youtube.com/watch?v=9ZIbnZuqUuo
>
> --
> You received this message because you are subscribed to the Google
> Groups "WebM Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to webm-discuss...@webmproject.org
> <https://groups.google.com/a/webmproject.org/d/msgid/webm-discuss/133a2b13-52ea-477b-b762-a12df7f22c2dn%40webmproject.org?utm_medium=email&utm_source=footer>.