sheet music generation transformer model (with demo)

[Joseph Weidinger]

Hey there!

Like many folks here, I've been following the amazing progress of Onsets and Frames / MT3  (among other Magenta releases) over the years. One thing I wish existed was the generation of equivalent sheet music representations. I've been lately experimenting with this: coming up with a novel musicxml tokenization scheme (375 vocab size) and training a stock gpt2-like implementation on tons of musicxml I found on the web. NLL goes from 6.0 to 0.16 in 8 hours on a 3090. Here's a 30s demo: https://www.youtube.com/watch?v=5C0jD6tEpqc. As you can see the results are mostly coherent.

I next want to add a cross-attention component so the model can attend to some note seq. The ultimate goal is to have a model that can take arbitrary midi input and output coherent sheet music without a tempo map or quantized performance. My hunch is that this is tractable for single-staff or piano. But I'm a hacker-type, not an ML expert, so it may just be naive optimism.

Anyways, I'm curious to know if anyone else in the Magenta community/team has experimented with this. If so, what were your results? I'd also be curious to know what you used as training data. It'd be nice to maybe one day get a good quality dataset for musicxml going.

Also if people are interested I can probably find a way to share the code -- I'd just have to clean it up a bit and get it cleared by Google.

- Joseph Weidinger

[Joseph Weidinger]

FWIW: I went through the process of getting approval to release the project and it is now available here: https://github.com/jsphweid/xamil

[Jamie]

This is really cool! Are the weights available?

[Joseph Weidinger]

Hey thanks! Unfortunately the weights are not available. The primary reason is that there isn't to my knowledge I nice open collection of MusicXML files to train on. The ones I trained on for this proof of concept were ones I found online through various means.