Onset and Frames vs MT3 - Piano transcription accuracy

250 views
Skip to first unread message

Joris Astier

unread,
Dec 13, 2022, 12:39:19 PM12/13/22
to Magenta Discuss
Hello everyone!

I'm writing to you because we are currently working on a tool to transcribe audio piano songs into sheet music that our users can download in PDF, MIDI and XML formats.

I would like to know your opinion about the accuracy of the Onset and Frames and MT3 models on piano transcriptions.

Onset and Frames specializes in piano song transcription while MT3 works on multi-instrument.

At first glance, Onset and Frames seems to be more efficient. However, I have seen in a few research articles that the accuracy of MT3 is superior to Onset and Frames on piano inputs. I did some tests with some songs and it seems that MT3 is indeed quite accurate on piano, but I can't say that it is better than Onset and Frames.

Do you have any information about this?

Thanks in advance for your help!

Have a nice day!

Curtis "Fjord" Hawthorne

unread,
Dec 13, 2022, 3:33:02 PM12/13/22
to Joris Astier, Magenta Discuss
Hi Joris,

You can see the comparisons on the MAESTRO (all piano) dataset in our ISMIR and ICLR papers about the architecture:

Sequence-to-Sequence Piano Transcription with Transformers (ISMIR): https://arxiv.org/abs/2107.09142
MT3: Multi-Task Multitrack Music Transcription (ICLR): https://arxiv.org/abs/2111.03017

The colab supports either of these models (keep in mind the MT3 one doesn't support velocity, but the ISMIR one does): https://colab.research.google.com/github/magenta/mt3/blob/main/mt3/colab/music_transcription_with_transformers.ipynb

Depending on the metric used, they get better scores than the Onsets and Frames model.

However, I've heard anecdotally that Onsets and Frames might generalize better outside of the MAESTRO dataset, so depending on your exact use case, it's probably still worth doing some comparisons. I suspect Onsets and Frames is also faster to run because it's a smaller model, but I haven't actually done the benchmark and it may depend on the hardware you're using.

Sorry there's not a clear answer to your question, but hopefully those details help some.

I'd love to hear how the project goes!

-Fjord

--
Magenta project: magenta.tensorflow.org
To post to this group, send email to magenta...@tensorflow.org
To unsubscribe from this group, send email to magenta-discu...@tensorflow.org
---
To unsubscribe from this group and stop receiving emails from it, send an email to magenta-discu...@tensorflow.org.
Reply all
Reply to author
Forward
0 new messages