sheet music generation transformer model (with demo)

125 views
Skip to first unread message

Joseph Weidinger

unread,
Mar 13, 2023, 10:16:34 PM3/13/23
to Magenta Discuss
Hey there!

Like many folks here, I've been following the amazing progress of Onsets and Frames / MT3  (among other Magenta releases) over the years. One thing I wish existed was the generation of equivalent sheet music representations. I've been lately experimenting with this: coming up with a novel musicxml tokenization scheme (375 vocab size) and training a stock gpt2-like implementation on tons of musicxml I found on the web. NLL goes from 6.0 to 0.16 in 8 hours on a 3090. Here's a 30s demo: https://www.youtube.com/watch?v=5C0jD6tEpqc. As you can see the results are mostly coherent.

I next want to add a cross-attention component so the model can attend to some note seq. The ultimate goal is to have a model that can take arbitrary midi input and output coherent sheet music without a tempo map or quantized performance. My hunch is that this is tractable for single-staff or piano. But I'm a hacker-type, not an ML expert, so it may just be naive optimism.

Anyways, I'm curious to know if anyone else in the Magenta community/team has experimented with this. If so, what were your results? I'd also be curious to know what you used as training data. It'd be nice to maybe one day get a good quality dataset for musicxml going.

Also if people are interested I can probably find a way to share the code -- I'd just have to clean it up a bit and get it cleared by Google.

- Joseph Weidinger

Joseph Weidinger

unread,
Apr 18, 2023, 4:49:48 PM4/18/23
to Magenta Discuss, Joseph Weidinger
FWIW: I went through the process of getting approval to release the project and it is now available here: https://github.com/jsphweid/xamil

Jamie

unread,
May 7, 2023, 4:30:52 PM5/7/23
to Magenta Discuss, jweid...@google.com
This is really cool! Are the weights available?

Joseph Weidinger

unread,
May 8, 2023, 1:21:40 AM5/8/23
to Magenta Discuss, jhl...@gmail.com, jweid...@google.com
Hey thanks! Unfortunately the weights are not available. The primary reason is that there isn't to my knowledge I nice open collection of MusicXML files to train on. The ones I trained on for this proof of concept were ones I found online through various means.
Reply all
Reply to author
Forward
0 new messages