Analyzing music with Magenta

[Keith Doelling]

Hi all!

Thanks in advance for any time you give to this post! I'm interested in using really any of the Magenta music generation RNNs preferably Polyphony RNN and Music Transformer to analyze a given piece of music. How probable is each note given the preceding notes?

I figure theoretically I could use the last layer of the output after each step to calculate entropy and surprisal over the probability distribution. Practically however, it's not so clear to me how to access this information from the generators. I would appreciate any suggestions you may have on how I could accomplish this or if this is any obvious question, a quick pointer to the documentation so I can sort it out for myself.

Thanks again for the guidance!

Keith

[Ian Simon]

It's a little buried, but here's one place to get started (using Performace RNN as an example, but the other RNN models are similar): https://github.com/magenta/magenta/blob/master/magenta/models/performance_rnn/performance_model.py#L83

If you trace that method which computes the log likelihood of an existing sequence, you can see where the softmax values are accessed here: https://github.com/magenta/note-seq/blob/master/note_seq/encoder_decoder.py#L309

You can modify that code to do other computations using the probability distribution at each step.

Hope that helps somewhat!

-Ian

--
Magenta project: magenta.tensorflow.org
To post to this group, send email to magenta...@tensorflow.org
To unsubscribe from this group, send email to magenta-discu...@tensorflow.org
---
To unsubscribe from this group and stop receiving emails from it, send an email to magenta-discu...@tensorflow.org.

[Jeremy Sawruk]

This sounds great! Keith, what kind of analysis are you planning on doing? I am wondering if this could be adapted to functional harmonic analysis.

[Keith Doelling]

Thanks Ian, I will play with it today and report back if I'm missing something I can't figure out.

Jeremy, I do research in cognitive neuroscience and I'm interested in comparing the output of this analysis with behavioral and neural data to study neural mechanisms of prediction. By extracting entropy and surprisal from the output layer, I would interpret these in the cognitive sense as prediction uncertainty and prediction error. Both are very relevant to how neuroscientists think about perception these days. I'm also interested in the question of learning. I.e. how much (as humans) do we update our internal models of music from hearing a surprising note. I could imagine that for some people a given note could be too surprising given the context, and we therefore don't learn how it relates to the previous notes, while others are also surprising but allow us to re-interpret the previous music in a new light or allow us to expand our understanding to appreciate this as being more probably in the future (e.g. maybe something like a piccadilly third). It would be interesting to see if we could model this with the RNN. I would like to break down the training algorithm to go note by note to see how much each note would change the weights. I'm not sure what exactly would be the best metric on this front yet. But I'm open to suggestions!

I imagine if each piece you run through this analysis has some functional harmonic analysis completed, you might be able to use this kind of analysis to see what kind of harmonic relationships the RNN has learned over the course of the training. And perhaps what it is most ready to learn next. Is this the sort of idea you were thinking of?

Keith

[Keith Doelling]

Hi again, all!

I'm running up on some errors using the performance_log_likelihood function that I haven't been able to sort out. Some appeared to be python 2 vs python 3 issues that I tried to fix on my own but those just led to further errors so I figured I should put it down before I hurt myself. Here is a copy of the code I am attempting to run as a test. Any advice would be appreciated!

https://colab.research.google.com/drive/1W0d89HHhabTlMKdHww-m_qNy1Kc7M2Bo?usp=sharing

Keith

[Ian Simon]

You're right, there are some Python 3 compatibility issues.  However, once I fix them, it seems to work: https://colab.research.google.com/drive/1whh6ZYRoj7Yr9Qtj1kdoDVViCfz8ofAM?usp=sharing

-Ian