Human-Robot Collaboration for Musical Tasks

[JunShern Chan]

Hi everyone! 

First of all, let me thank you all for the amazing work you've done on Magenta! Very cool and very inspiring. :) 

My name is Jun Shern, I'm a Masters student at Imperial College London working on my Masters project titled "Human-Robot Collaboration for Musical Tasks". The project will (hopefully) eventually involve having a robotic musician to physically play an instrument in a duet setting, but I have decide to focus my work right now on composing musical accompaniments based on a user's (MIDI keyboard) input. 

Here are some goals I defined for my project:

• The generation must work in real-time
• Both members of the duet would play the piano, but in contemporary pop music styles (think anything from Adele to Disney songs) instead of classical
• I would like the model to have flexibility in its accompaniment role - eg. when the human plays a leading melody, the robot plays accompanying chords or harmonies. When the human plays background, the robot takes a more leading role. If necessary, I am happy to have some external inputs to the model such as a "forwardness" knob which controls the leading/background style of the generation but am not sure how this could be integrated into a model.
  

I have been reading about and exploring the various models you have available, and I have tried out the online AI-Duet which is close to what I hope to achieve. However, my experience with AI Duet seems to give poorer composition results than the sound bites given on the blog posts for Performance-RNN or RL Tuner, so I was hoping to pick your brains with a few questions to help me get started:

1. Regarding AI Duet, I presume that live accompaniment falls short of the offline compositions because the Melody-RNN model is built for composition rather than accompaniment, is that right? If so, how could we handle accompaniment versus composition differently? 

2. How do the current existing models compare in terms of composition quality, and which seems most adaptable to my objective? In particular, I thought that Performance RNN, RL Tuner and Music VAE are all fantastic and very cool architectures, but they are very different and I'm not sure which I should focus more on.

I know that these are long questions, but I would be hugely grateful for some pointers and ideas! Alternatively, I would love to jump on a hangouts call for brainstorming if anyone is willing to do so. 

Thanks so much in advance!

[Adam Roberts]

Hi JunShern,

None of our current models are trained for accompaniment. The closest thing to what you want would be to use one of the NoteRNN models (MelodyRNN etc) like with AI Duet. However, AI Duet does not work as well as these models typically do because the simple app is having to infer the tempo from what you're playing and it's often failing.

You may want to have a look at "AI Jam", which does the timing more explicitly and will work better: https://github.com/tensorflow/magenta-demos/tree/master/ai-jam-js

You may also want to have a look at Mason Bretan's work, which sounds similar to what you're trying to accomplish: http://www.masonbretan.com/

--
Magenta project: magenta.tensorflow.org
To post to this group, send email to magenta...@tensorflow.org
To unsubscribe from this group, send email to magenta-discuss+unsubscribe@tensorflow.org
---
You received this message because you are subscribed to the Google Groups "Magenta Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to magenta-discuss+unsubscribe@tensorflow.org.