Detect audio notes and assign color ML/AI

[Athena Maxwell]

Hello! I am completely new to this forum and program but have been researching possible ways to create AI-generated artwork based on music. 

I have synesthesia and make art by transcribing music note-for-note into a colorized version of the score in Adobe Illustrator. Obviously, this is very tedious.

Is there any way to create something which would pre-assign stand-alone notes, and/or notes within a chord a specific color, so that when audio is filtered in, it "regurgitates" the piece of music with the colors assigned? And maybe is able to learn/intuit which notes will be which color based on the musical key or inversion of the chord, etc.? 

Any insight or ideas would be greatly appreciated, thank you!

Athena

[Jamie Hlusko]

That sounds cool! Could you share some examples of the art you've made? What rules for colour assignment do you use?

I don't know if Magenta is the best tool for your project because you need a score. Perhaps modifying something like this automatic fingering generator would be a better fit?

https://github.com/marcomusy/pianoplayer

[Athena Maxwell]

Thank you so much for your response! 

Below is a link to some of my work. Basically when I hear a note, it has a color. Put that note into a chord, and the colors change a little. They key and tempo of the piece can change colors too. That's where the ML might come in. So say, an "A flat" is always a bright red color, unless it's part of a diminished D chord- then it becomes maroon.

I don't necessarily need a score; if it were possible to develop something that could even just turn audio input into midi and color assign each note that way, that could be cool too. I'm just curious about the whole thing and want to experiment with it. I've seen so many cool demos and applications, but so far nothing that specifically assigns color to a given note. 

https://opensea.io/assets/dechnicolors?search[sortAscending]=false&search[sortBy]=BIRTH_DATE&search[resultModel]=ASSETS

Thank you again for your insight!

Athena

[Krish]

Super cool!

For audio->MIDI transcription, you could try Magenta's Onsets and Frames https://magenta.tensorflow.org/onsets-frames, and then color assign notes as you mentioned. If you want to learn the color assignment, you could perhaps use your existing transcriptions and add more to come up with a dataset to train a network that does just that. You could also use Onsets and Frames to generate a bunch of MIDI transcriptions (and scrape other existing MIDI transcriptions online) and then use you color assignment algorithm to generate a large dataset for the purposes of learning a new model that does audio -> color transcription automatically (and maybe learns some interesting interpolations/behavior).

If you're open to collaboration on this, I'd definitely be interested in learning more and helping!

--
Magenta project: magenta.tensorflow.org
To post to this group, send email to magenta...@tensorflow.org
To unsubscribe from this group, send email to magenta-discu...@tensorflow.org
---
To unsubscribe from this group and stop receiving emails from it, send an email to magenta-discu...@tensorflow.org.