multitrack MusicVAE

[Ian Simon]

Magenta has a new blog post up about a version of MusicVAE that works on single measures of multitrack MIDI, with optional chord conditioning: https://magenta.tensorflow.org/multitrack 

Here are a couple of interactive CodePens (also embedded in the blog post) that let you interact with the model in your browser, using magenta.js:

https://codepen.io/iansimon/pen/GGRYJZ 

https://codepen.io/iansimon/pen/Bxgbgz 

Feel free to play with everything, and let us know if you make something interesting!

-Ian

[Sean Farrell]

Yay amazing!  Can't wait to try it out.

Sean

--
Magenta project: magenta.tensorflow.org
To post to this group, send email to magenta...@tensorflow.org
To unsubscribe from this group, send email to magenta-discuss+unsubscribe@tensorflow.org
---
You received this message because you are subscribed to the Google Groups "Magenta Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to magenta-discuss+unsubscribe@tensorflow.org.

[Kyle Kastner]

It's awesome stuff! Congrats to all - the samples really blew my mind.

On Tue, Jun 5, 2018 at 2:17 PM, Sean Farrell <se...@a52.com> wrote:

Yay amazing!  Can't wait to try it out.

Sean

[Jebin Matthew]

That was totally cool, Actually, I was trying to build multitrack based on an implementation of improv_rnn and drums_rnn, and it kinda worked. But then, guess that's not needed now, since this is based on performance_rnn plus, it has chords!!

It's just that, I was wondering if tensorflow implementation of multitrack is available at github, wanna try it so bad! (Not good with 'JS' and the 'browsers' yet!)

Thanks!

[Ian Simon]

Hi Jebin, the code to run (or train) the multitrack MusicVAE is on GitHub, though there's no notebook demo just yet.  In the meantime, you can do the following:

1) Download a checkpoint (these are pretty large, almost 1 GB):

(chord-conditioned)

https://storage.googleapis.com/magentadata/js/checkpoints/music_vae/multitrack_med_chords/model.ckpt-100000.data-00000-of-00001

https://storage.googleapis.com/magentadata/js/checkpoints/music_vae/multitrack_med_chords/model.ckpt-100000.index

https://storage.googleapis.com/magentadata/js/checkpoints/music_vae/multitrack_med_chords/model.ckpt-100000.meta

(non-chord-conditioned)

https://storage.googleapis.com/magentadata/js/checkpoints/music_vae/multitrack_med/model.ckpt-100000.data-00000-of-00001

https://storage.googleapis.com/magentadata/js/checkpoints/music_vae/multitrack_med/model.ckpt-100000.index

https://storage.googleapis.com/magentadata/js/checkpoints/music_vae/multitrack_med/model.ckpt-100000.meta

1.5) Make sure you have the latest version of magenta installed ("pip install --upgrade magenta").

2) Open a Python REPL or notebook and run the following to load the model:

import magenta.music as mm

from magenta.models import music_vae

BATCH_SIZE = 4

TOTAL_LENGTH = 512

NUM_CHORDS = 49

TEMPERATURE = 0.2

config = music_vae.configs.CONFIG_MAP['hier-multiperf_vel_1bar_med_chords']  # (or 'hier-multiperf_vel_1bar_med' for no chord conditioning)
model = music_vae.TrainedModel(config, BATCH_SIZE, '/path/to/model.ckpt-100000')

3) Sample from the model.

# Construct a chord conditioning vector (skip if not using chord conditioning).
index = 1  # 1-12 = major, 13-24 = minor, 25-36 = augmented, 37-48 = dimished
c_input = np.zeros([TOTAL_LENGTH, NUM_CHORDS])
c_input[0, 0] = 1.0
c_input[1:, index] = 1.0

# Generate samples.
seqs = model.sample(1, TOTAL_LENGTH, temperature=TEMPERATURE, c_input=c_input)

# Write to MIDI.
mm.sequence_proto_to_midi_file(seqs[0], 'sample.mid')

That should be enough to get you started.  Let me know if things aren't working.

-Ian

--

Magenta project: magenta.tensorflow.org
To post to this group, send email to magenta...@tensorflow.org

To unsubscribe from this group, send email to magenta-discu...@tensorflow.org

---
You received this message because you are subscribed to the Google Groups "Magenta Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to magenta-discu...@tensorflow.org.

[Jebin Matthew]

Many thanks for your detailed reply Simon, much appreciated. I'll set this up tonight, (or at least within this weekend), and post the results ASAP!

In the meantime, if possible, please let us know if the model and the corresponding .mag files are out.

Thanks a lot!

[Jebin Matthew]

I've finally set that up, Ian. Thanks a lot for the support! 

I've generated a melody successfully and it sounds awesome, but then the 'TOTAL LENGTH' seems to have no effect, as the resulting midi file is of one bar only. (Makes sense since the config name itself had '1_bar' mentioned in it.)

Anyways, I tried to fiddle with the config file but in vain. I'd be happy to know about extending the length of generated melody and why TOTAL_LENGTH had no effect.

And, what if I wanted to customize the 'style' it generates, (like jazzy, heavy metal, shuffle, world, even odd time-signatures like 5/4 or 7/4), will it be possible? 

Can I have control on what instruments it generates music with too? (like only these 3: saxophone, brass and strings) 

[I saw max_num_instruments field in Config file, though, but how are instruments chosen?]

Many Thanks,

Jebin Matthew.

[Ian Simon]

Hi Jebin, right now the constraints you've pointed out are baked into the model: it only handles single measures of 4/4 music.  The longer interpolations and chord progressions from the blog post are formed by concatenating multiple measures generated by the model.

We should have a colab notebook out in the next few days with example code for producing these multi-bar interpolations.

-Ian

[Gopi]

Thanks a ton, Ian.

[Ian Simon]

Hi Jebin, just following up on this.  A colab notebook for multitrack MusicVAE is now available here: https://colab.research.google.com/notebooks/magenta/music_vae/multitrack.ipynb

Let me know if you have any questions.

-Ian

[Jebin Matthew]

Thank you ever so much, Ian. I'll definitely get back to you about the results!

Gracias!

[Jebin Matthew]

Thanks a ton, Ian.

I've copied bits of code from the notebook, and used it along the checkpoint that you gave earlier, and the results are sweet!
Now the length is not an issue, and chords too sound pretty. Next thing would be to maybe control the style and choice of instruments... which I too shall try trying after probably digging more into MusicVAE's implementation!


Anyways, Thanks again!

Jebin Matthew.

[andrey...@gmail.com]

Hello Ian,

Could you please help?

I am trying to train MusicVAE  on MIDIs of one band expecting to get something similar on output using random sampling from the prior distribution.  

I found 20-30 midis of band in internet and did the following:

tensorflow (1.8.0)
magenta (0.3.8)

 

!convert_dir_to_note_sequences \
  --input_dir=RancidMidi \
  --hparams=sampling_rate=1000.0\
  --output_file=tmp/notesequences_RancidMidi.tfrecord \
  --recursive

coverted

!music_vae_train \
--config=hier-multiperf_vel_1bar_med \
--run_dir=/tmp/music_vae/ \
--hparams=batch_size=32,learning_rate=0.0005 \
--mode=train \
--examples_path=tmp/notesequences_RancidMidi.tfrecord

Got the following exception:

INFO:tensorflow:Reading examples from: tmp/notesequences_RancidMidi.tfrecord
INFO:tensorflow:Building MusicVAE model with HierarchicalLstmEncoder, HierarchicalLstmDecoder, and hparams:
{'learning_rate': 0.0005, 'decay_rate': 0.9999, 'use_cudnn': False, 'free_bits': 0.0, 'sampling_rate': 0.0, 'conditional': True, 'batch_size': 32, 'clip_mode': 'global_norm', 'residual_decoder': False, 'dec_rnn_size': [512, 512, 512], 'beta_rate': 0.0, 'grad_norm_clip_to_zero': 10000, 'dropout_keep_prob': 1.0, 'min_learning_rate': 1e-05, 'max_seq_len': 512, 'max_beta': 1.0, 'grad_clip': 1.0, 'enc_rnn_size': [1024], 'sampling_schedule': 'constant', 'residual_encoder': False, 'z_size': 512}
INFO:tensorflow:
Hierarchical Encoder:
input length: 512
level lengths: [64, 8] INFO:tensorflow:Level 0 splits: 8
INFO:tensorflow:
Encoder Cells (bidirectional):
units: [1024] INFO:tensorflow:Level 1 splits: 1
INFO:tensorflow:
Encoder Cells (bidirectional):
units: [1024] INFO:tensorflow:
Hierarchical Decoder:
input length: 512
level output lengths: [8, 64] INFO:tensorflow:
Decoder Cells:
units: [512, 512, 512] Traceback (most recent call last):
File "/opt/conda/bin/music_vae_train", line 11, in <module>
sys.exit(console_entry_point())
File "/opt/conda/lib/python3.5/site-packages/magenta/models/music_vae/music_vae_train.py", line 325, in console_entry_point
tf.app.run(main)
File "/opt/conda/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 126, in run
_sys.exit(main(argv))
File "/opt/conda/lib/python3.5/site-packages/magenta/models/music_vae/music_vae_train.py", line 321, in main
run(configs.CONFIG_MAP)
File "/opt/conda/lib/python3.5/site-packages/magenta/models/music_vae/music_vae_train.py", line 303, in run
task=FLAGS.task)
File "/opt/conda/lib/python3.5/site-packages/magenta/models/music_vae/music_vae_train.py", line 164, in train
optimizer = model.train(**_get_input_tensors(dataset, config))
File "/opt/conda/lib/python3.5/site-packages/magenta/models/music_vae/base_model.py", line 296, in train
input_sequence, output_sequence, sequence_length, control_sequence)
File "/opt/conda/lib/python3.5/site-packages/magenta/models/music_vae/base_model.py", line 260, in _compute_model_loss
x_input, x_target, x_length, z, control_sequence)[0:2]
File "/opt/conda/lib/python3.5/site-packages/magenta/models/music_vae/lstm_models.py", line 1121, in reconstruction_loss
hier_input = self._reshape_to_hierarchy(x_input)
File "/opt/conda/lib/python3.5/site-packages/magenta/models/music_vae/lstm_models.py", line 1081, in _reshape_to_hierarchy
perm.insert(num_levels, perm.pop(0))
AttributeError: 'range' object has no attribute 'insert'

Don't you know what  I am doing wrong? Could you please help how can I train it with my own midis?

Many thanks!

суббота, 16 июня 2018 г., 21:15:48 UTC+3 пользователь Ian Simon написал:

[Ian Simon]

Ah, I guess it's broken in python3.  I'll look in to it further, but in the meantime can you try python2?

-Ian

[Ian Simon]

Could also try changing line 1080 in lstm_models.py to:

perm = list(range(len(hier_shape)))

Though I haven't tested the multitrack model with py3 so there may be other issues as well.

-Ian