Intent to Implement: Web MIDI API

[Chris Wilson]

This mail declares our intent to implement the Web MIDI API.  This is *not* declaring an intent to ship at this time.

Primary emails

eng: Takashi Toyoshima (toyo...@chromium.org), pm de facto: Chris Wilson (cwi...@chromium.org)

Spec

Web MIDI spec is here.  More information on MIDI and Web MIDI API implementation (technical details in Blink, security, etc) is here.

Summary

MIDI is a widely-used industry standard protocol used to enable communication between music keyboards, control surfaces such as mixers, DJ controllers, computers and other musical instruments. This API adds support for MIDI devices to the web platform, enabling web applications to list available MIDI input and output devices on the client system, as well as send and receive MIDI messages.  Playing a key on a music keyboard triggers an event handler, just like receiving keyboard or mouse events.  “MIDI” is often confused with the playback of Standard MIDI Files (.SMFs), which are similar to .MOD files; this feature is unrelated.

This API is a thin layer on top of industry standard MIDI APIs such as CoreMIDI on OSX and iOS, Windows MIDI, and ALSA MIDI.  This API confines itself to exposing MIDI devices and message data, analogous to these existing MIDI platform APIs.

In short:

• Industry standard protocol for past three decades

• Foundational protocol for nearly all consumer and pro music scenarios

• Very limited technical scope, thin layer on existing industry standard APIs

Motivation

MIDI is an industry standard protocol used to enable communication between musical controllers, computers, and other musical instruments.  It was originally designed in 1983 and was quickly adopted across a wide variety of devices - keyboard controllers, drum machines, synthesizers, effect racks, computers, hardware sequencers, and from then on, an expanding variety of controllers (DJ controllers, basic knob/button/slider controllers, guitar controllers, wind controllers, drum triggers, etc.).  Additionally, other non-musical control devices (particularly, professional lighting controllers) have adopted the MIDI standard as a communication protocol.

Today, MIDI is a critically important standard protocol to the music industry.  It is the foundational communication protocol used in professional, prosumer and consumer studios and workflows, allowing a large variety of common control surfaces to be used to control audio and music software such as mixing control surfaces with faders and knobs, DJ control surfaces and more, in addition to interconnecting musical instruments like keyboards, guitars, drums, effects units and other equipment .  It is supported and used by nearly every music production application, from professional applications down to Apple’s GarageBand included with every Mac, and is the foundation of interoperability across music devices in studios, from consumer to pro. MIDI is supported on Apple’s iOS (more than 400 MIDI applications in the App Store), along with every other major operating system (OSX, Windows, Linux), and MIDI controllers are available locally nationwide at consumer-focused stores like Best Buy and the Apple Stores, as well as music-focused stores like Guitar Center.

The difference between tapping out melodies on a computer keyboard and playing a weighted 88-key piano controller represents the gap between web toy and a serious music production tool to a large audience of musicians, sound designers and audio engineers, from amateurs to pros.

Compatibility Risk

The specification is a stable working draft (first published working draft was approximately six months ago), and the working group considers it ready for implementation experience.  It has been implemented as a plugin-based polyfill that works across browsers on OSX and Windows, but there are no other current native implementations, and no other browser has yet declared their intent to implement.  Mozilla has made positive comments about the specification (see links in chromestatus.com), but is not focused on in-depth review of the MIDI API right now; they are very focused on Web Audio as a higher priority (which is natural). Web Audio implementations in general turn up the need for MIDI support, so we expect this to be more interesting as they work through their Web Audio implementation.

This API has a very small footprint- four object types - and directly maps onto existing APIs that have been successful in the industry for decades, and requires few changes in the Blink code (see linked Web MIDI doc referenced above for technical details).  Although small changes to the specification may happen in the future (we expect implementation to turn up some changes, for example), the overall API design is unlikely to change dramatically, and there is just not that much to change.  Because this API is based on well-established industry standards, we fully expect this API to be compatibly and interoperably implemented.

Without this API, MIDI devices can only be accessed on the web platform via plugins such as the Jazz plugin, or general-purpose platform plugins that support MIDI such as Adobe Director or Java, but cannot access MIDI devices without a plugin.

This API does not directly interoperate with existing APIs, so should not cause issues outside its own area, both in terms of the JavaScript API’s interaction with other features, and in the underlying Blink/Chromium implementation (for example, the code does not share or interfere with the Web Audio implementation).  

OWP launch tracking bug

Yes, there is a row on the feature dashboard.  (Search for “MIDI”.)

We are not requesting simultaneous permission to ship.

This feature will be implemented behind a runtime flag in order to get early developer experience.  We will send an Intent to Ship email when we believe it is ready to enable by default.

-Chris, Takashi, Chris, et al.

[Emil A Eklund]

Chris,

Thanks for the detailed motivation and risk sections. I'm a bit
concerned about this feature though as at least one other browser
vendor is opposed to this. Also, the objections that where bought up
in the webkit thread about this feature earlier this year still
stands, in short:

- Are any other browser engines planning to implement this?
- Is it a meaningful feature to at least a large subset of users?
- Is it a meaningful feature without dedicated and uncommon hardware?
- Why is it implemented at this level and not at a lower level to
allow for a js implementation of this (and other similar) features.

Before we continue the discussion it would be useful to get your take
on those issues.

Thanks

--
Emil

[Chris Wilson]

Hey Emil-

those are (and were) all great questions.  We specifically tried to address them with this intent; there are more detailed answers to them are in the email or the linked Web MIDI doc, but as a short version,

• [other browsers] No other browser has declared their intent to implement at this point (I'm obviously discounting the NPAPI-/ActiveX-based prollyfill that I maintain), but MIDI is fairly tightly (though not completely) bound in scenarios to Web Audio support - that is, once you have a powerful audio subsystem, you really want MIDI controllers to control it.  I would classify Mozilla (the only other engine that has Web Audio support under way) as "unopposed, but not directly reviewing" - which is natural, as they're highly focused on Web Audio right now.  This Intent to Implement is to gather the developer experience we need; the decision to ship or not will be made afterward as a separate Intent to Ship, and I expect we'd have better information on other browsers as well as developer experience to make that decision.
• [meaning for large subset of users] Yes.  The MIDI market is huge (see doc), and MIDI devices are sold in Best Buy, the Apple store, and other consumer places, as well as music-focused stores like Guitar Center.  MIDI is the foundational communications protocol for the music industry.
• [meaningful without dedicated/uncommon hardware] First, it is a myth that the hardware is uncommon.  That $10 USB-based "learn to play piano" keyboard bought for your child is likely a MIDI device.  Every DJ, keyboard player, guitar player with a modelling amp - they're all using MIDI.  Apple actually included CoreMIDI support in their iOS 4.2 release, which sparked an enormous focus in the music industry on iPad MIDI apps (>400 in the App Store today, and a number of companies sell dedicated iPad stands for music use) and hardware (dedicated keyboard controllers with iPad slots, e.g.).  Second, there are a few scenarios (built-in MIDI software synth) that are interesting without hardware; largely, though, MIDI as an input controller technology is the most compelling.
• [Why not a lower-level API] See the doc for this one, as it's a long answer, but in short - providing low-level access makes the security implications far, far more concerning, as well as making the hardware hard to use; MIDI is far more like keyboards and mice as a scenario - a tightly defined, narrow messaging API for a specific "class" of device, that may be connected by one of several hardware protocols (USB, FireWire, BlueTooth, et al).

-Chris

[Dan Beam]

On Wed, Apr 17, 2013 at 11:38 AM, Chris Wilson <cwi...@google.com> wrote:

Hey Emil-

those are (and were) all great questions.  We specifically tried to address them with this intent; there are more detailed answers to them are in the email or the linked Web MIDI doc, but as a short version,

• [other browsers] No other browser has declared their intent to implement at this point (I'm obviously discounting the NPAPI-/ActiveX-based prollyfill that I maintain), but MIDI is fairly tightly (though not completely) bound in scenarios to Web Audio support - that is, once you have a powerful audio subsystem, you really want MIDI controllers to control it.  I would classify Mozilla (the only other engine that has Web Audio support under way) as "unopposed, but not directly reviewing" - which is natural, as they're highly focused on Web Audio right now.  This Intent to Implement is to gather the developer experience we need; the decision to ship or not will be made afterward as a separate Intent to Ship, and I expect we'd have better information on other browsers as well as developer experience to make that decision.
• [meaning for large subset of users] Yes.  The MIDI market is huge (see doc), and MIDI devices are sold in Best Buy, the Apple store, and other consumer places, as well as music-focused stores like Guitar Center.  MIDI is the foundational communications protocol for the music industry.

+1, cannot agree more; MIDI is pervasive.  An overwhelming majority of performance interfaces support MIDI.  If we want to push the web forward with regards to music performance or creation, this would be a great next step.

--
Dan Beam
db...@chromium.org

[PhistucK]

Unless I am mistaken, the Web MIDI API is totally unrelated to playing .mid files. Correct?

(I took a peek at the document, but I just want to make sure I understand correctly)

So, while apparently unrelated, if anything regarding MIDI is implemented, I think the playback of .mid files is a much much higher priority.

While MIDI devices are common, I obviously do not have numbers here, obviously, but I personally know almost no one with a MIDI device and those who own one, are music artists.

While music production can be a very nice use case, I think there are many other areas to improve first.

☆PhistucK

[Alex Russell]

hey Chris,

Excited to see this happening! I've got some feedback on API aspects of the proposed spec. What's the right forum for those?

[Ehsan Akhgari]

The public-audio mailing list is the right place for sending feedback.

Thanks!

--
Ehsan
<http://ehsanakhgari.org/>

[Alex Russell]

Thanks. Re-sending notes there.

[Chris Rogers]

On Fri, Apr 19, 2013 at 3:38 AM, PhistucK <phis...@gmail.com> wrote:

Unless I am mistaken, the Web MIDI API is totally unrelated to playing .mid files. Correct?

(I took a peek at the document, but I just want to make sure I understand correctly)

So, while apparently unrelated, if anything regarding MIDI is implemented, I think the playback of .mid files is a much much higher priority.

It's already possible to parse .mid (SMF standard MIDI files) in JS and playback using the Web Audio API, so we're covered there.

[PhistucK]

Do you mean it is possible to use the local MIDI sound bank (not sure how it is really called) using the Web Audio API?

Unless I am mistaken, this is not possible and it means every web application must share its own sound bank and make the browser download it. There is a huge difference here. Parsing a standard MIDI file and playing it back like a native MIDI player that uses the sound bank of the sound card are totally different features and I was referring to the latter (in addition to the former, obviously).

☆PhistucK

[Dimitri Glazkov]

Hi folks!

Yesterday, we had an API owners review with the MIDI implementers. Web MIDI is LGTMed to implement behind a runtime flag, but we’ll need to carefully consider whether to ship by default when the time comes. The primary concerns are:

1) whether something so niche belongs in a browser

2) fingerprinting/privacy

3) what a malicious web author can do to your MIDI devices

4) whether this should just be a Chrome Apps API (and/or whether the more security sensitive bits should only be exposed to Chrome Apps).

Stronger positive sentiment from other browser vendors about what they are willing to ship will also have an impact on the decision to ship.

:DG<

[Rafael Weinstein]

Not that anyone asked my opinion, but I'm super excited to see this
move forward.

To my mind, you just need to look at the number of YouTube videos
devoted to music instruction to make the case for this API. (Try
searching for: "[song] [instrument] lesson".)

(To wit: I think framing this as "niche" is doing a disservice to the use case).

I very much hope that we can persuade other venders to be excited
about web-based music creation, production, instruction and art.

[Chris Wilson]

Yes, this is pretty unrelated to playing Standard MIDI Files.  That can actually be accomplished already, by writing a software synthesizer in Web Audio and reading the files/doing the sequence playback in code (not particularly hard, has been done; I've written the .MID reader myself; with Web MIDI support, you could leverage the system General MIDI synths on Mac and Windows), or forking the playback to a system player (built in in Windows Media Player, Android media player, and some versions of QuickTime, lots of players available in iOS).  The scenario here in actual usage, though, ends up being largely "play cheesy background music" in my experience, which I don't think is a high priority.

Indeed, MIDI devices are far, far more common in the music community; however, that spans the consumer and pro markets, not just the high-end production case.  Every digital DJ, every keyboard player, most modelling guitar amp users, most "learn to play piano" keyboards... it's a very broad market.

-Chris

On Fri, Apr 19, 2013 at 3:38 AM, PhistucK <phis...@gmail.com> wrote:

[Chris Palmer]

On Fri, Apr 19, 2013 at 1:56 PM, Dimitri Glazkov <dgla...@chromium.org> wrote:

> 1) whether something so niche belongs in a browser

We should apply this razor often, indeed.

> 2) fingerprinting/privacy

The passive fingerprinting problem is extremely hard to solve, and
other already-approved and already-shipped APIs exacerbate the problem
more than Web MIDI would.

> 3) what a malicious web author can do to your MIDI devices

Limiting that is my bailiwick, and I've committed to doing it.

> 4) whether this should just be a Chrome Apps API (and/or whether the more
> security sensitive bits should only be exposed to Chrome Apps).

Yes, possibly. But the Open Web is the best place to be, and it's
worth trying to get there.

[PhistucK]

Why should every developer bother building a MIDI sound bank by themselves (and the user has to download them for every application that needs them) when practically every operating system or sound card already has it? Why should we not just expose it?

☆PhistucK

[Chris Wilson]

Indeed, this software synthesizer typically *IS* exposed - as a MIDI device.  The use case for a General MIDI device, however, is a separate one from the basic support for MIDI, and I'm personally not yet convinced that today, it is a compelling enough case.  The music software cases are generally covered by Web MIDI; games for the most part have moved to more in-depth audio production, because the control offered by sending your music to an arbitrary GM synth is pretty poor (some sound great, many sound like junk; you'd be far better off streaming straight digital audio).

To define terms for others on the thread:

MIDI: communication protocol of note-on/note-on/controller/program change etc messages

Standard MIDI File (SMF, ".MID") - a file format that contains "tracks" of the above messages with timestamps.

General MIDI (GM) - a well known system of "patch 00 is an acoustic piano; drums are on channel 10, and arranged like this."

Downloadable Sounds - an extension that allows the composer to install new sound samples/patches in a General MIDI synthesizer

[PhistucK]

So using a JavaScript based .MID parser and the Web MIDI API, a developer can play back .MID files just like native players do?

☆PhistucK

[Chris Wilson]

In the case where there's a system-wide general MIDI synth exposed (e.g. on OSX and Windows), yes.

[PhistucK]

Well, that settles it for me.

Thank you for the clarification!

☆PhistucK

[Chris Wilson]

No problem!  The challenging part about explaining Web MIDI is that "MIDI" has been associated with .SMF files too strongly.

[yw.w...@gmail.com]

Hi,

I'm new in Web audio API and Web MIDI API and see a big future. What I'm understood is that the MIDI support for Linux isn’t yet implemented, for simple use the Jazz Plugin e.g.   Is there any chance to implement the Web MIDI API for Linux without extreme development?
Thanks,
Yosef

[Takashi Toyoshima]

Hi Yosef,

Thank you for being interested in Web MIDI API.

Actually, we already started to implement it for Linux and Chrome OS. In the trunk build, you can see the flag in chrome://flags, and output devices work behind the flag. Input devices is coming soon.

--

Takashi Toyoshima
Software Engineer, Google

[killers...@gmail.com]

Takashi 

  Hello just found the Web MIDI API info on the web a short time ago. Any chance on an update to input devices? Very interested in the progress of this project. A classroom full of kids with chromebooks and one midi keyboard are itching to give this a try. 

Hi Yosef,

Thank you for being interested in Web MIDI API.

Actually, we already started to implement it for Linux and Chrome OS. In the trunk build, you can see the flag in chrome://flags, and output devices work behind the flag. Input devices is coming soon.

[Chris Wilson]

You should be able to use your chromebooks and USB MIDI keyboard, input included, today - provided you enable the @enable-web-midi flag in chrome://flags.