Wave2Midi2Wave Code/Checkpoints

[jps...@g.rit.edu]

Hey there,

Very appreciative of the work the Magenta team is doing and am having fun programmatically getting to know you guys through reading the papers and re-implementing stuff!

Quick question: Anyone know of plans to release code/checkpoints for Wave2Midi2Wave, particularly the modified Conditional WaveNet and/or Music Transformer models?  Curious simply because I'm attempting to re-implement the Conditional WaveNet to make some Frankenstein music, but realized that training would be excruciating for me on a 1070.

Thanks!

Jason St. George

Rochester Institute of Technology

[Curtis "Fjord" Hawthorne]

Hi Jason,

We've released some parts of the paper:

The Onsets and Frames model trained on the MAESTRO dataset (Wave2Midi) had code and a checkpoint here: https://github.com/tensorflow/magenta/tree/master/magenta/models/onsets_frames_transcription

A version of the Music Transformer model (Midi) has code here (we hope to release a pre-trained model at some point too): https://github.com/tensorflow/magenta/tree/master/magenta/models/score2perf

Unfortunately, the WaveNet model (Midi2Wave) isn't in a releasable state right now, but we may release a version of it some point in the future.

Hope that helps!

-Fjord

--
Magenta project: magenta.tensorflow.org
To post to this group, send email to magenta...@tensorflow.org
To unsubscribe from this group, send email to magenta-discu...@tensorflow.org
---
You received this message because you are subscribed to the Google Groups "Magenta Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to magenta-discu...@tensorflow.org.

[Jason St George]

Thanks Fjord and Jong!

This is very helpful.  I’ve been enjoying playing with Onsets & Frames model a bit, didn’t realize there was Transformer code out!  I’ll poke around the nvidia reference implementation and the score2perf.

Excited for a Midi2Wave model/ckpt to be released!  

Jason

P.S. Fjord, I’m a meager graduate student with time on his hands, so if there’s any way I could assist, please let me know and I’d be glad to help.

On Wednesday, February 6, 2019, Curtis "Fjord" Hawthorne <fj...@google.com> wrote:

Hi Jason,

We've released some parts of the paper:

The Onsets and Frames model trained on the MAESTRO dataset (Wave2Midi) had code and a checkpoint here: https://github.com/tensorflow/magenta/tree/master/magenta/models/onsets_frames_transcription

A version of the Music Transformer model (Midi) has code here (we hope to release a pre-trained model at some point too): https://github.com/tensorflow/magenta/tree/master/magenta/models/score2perf

Unfortunately, the WaveNet model (Midi2Wave) isn't in a releasable state right now, but we may release a version of it some point in the future.

Hope that helps!

-Fjord

On Wed, Feb 6, 2019 at 8:47 AM <jps...@g.rit.edu> wrote:

Hey there,

Very appreciative of the work the Magenta team is doing and am having fun programmatically getting to know you guys through reading the papers and re-implementing stuff!

Quick question: Anyone know of plans to release code/checkpoints for Wave2Midi2Wave, particularly the modified Conditional WaveNet and/or Music Transformer models?  Curious simply because I'm attempting to re-implement the Conditional WaveNet to make some Frankenstein music, but realized that training would be excruciating for me on a 1070.

Thanks!

Jason St. George

Rochester Institute of Technology

--
Magenta project: magenta.tensorflow.org
To post to this group, send email to magenta...@tensorflow.org

To unsubscribe from this group, send email to magenta-discuss+unsubscribe@tensorflow.org

---
You received this message because you are subscribed to the Google Groups "Magenta Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to magenta-discuss+unsubscribe@tensorflow.org.

--

Jason St George

"Talking about music is like dancing about architecture"

[robin meier]

Hello,

I’ve successfully been using PerformanceRNN for quite some time now and would like to try out the Music Transformer model. I have been reading through the readme of score2perf and the music transformer paper, but there’s something very basic I don’t understand, please excuse the noob question: What’s happening during the Data generation / preprocessing step? I suppose it’s creating the "sparse, MIDI-like, event-based representation from (Oore et al., 2018)”?

But what is the model actually learning from? The Piano-e-competition dataset? Maestro dataset (what is this??)

Or from my own data? then how should that data be formatted (standard midi?) and where does it go?

Thank you for your help,

r

[Jason St George]

Robin, I am in a similar boat.  Have been learning about it for a week or so but still have some questions about the tfrecord formatting at the end of the process.

Here’s what I’ve learned and hope I can be helpful: (please correct me if I’m wrong, guys)

The Onsets model produces MIDI representations of the spectrogram images created from the wav files in an online fashion.  The transformer model takes in the performance encoding proposed by Sageev Oore in this paper (https://arxiv.org/abs/1808.03715), start at pg 13.

That encoding is a one-hot vector of dimension 413, but we can’t get there just from midi.  

There is an intermediate step, converting the midi files to what are called NoteSequence Protos, which is a format developed by google (protobuf, https://github.com/tensorflow/magenta/tree/master/magenta/protobuf) that make the data easier to work with.  The midi is read in as string and then converted to a note sequence.

From here the note sequence protos are converted into the performance one-hot vector encoding (Oore) and then serialized to tfrecord files.  These will be consumed by the transformer model.

This is where I’m still fuzzy.  The tfrecord files that are given at the gs:// (cloud flow) bucket locations are already in note sequence format, it seems the conversation to performance encoding is what’s happening in the Apache beam datagen script you mentioned.  To train on your own midis it looks like we need to structure our tfrecord files the same way as Magenta team has in their tfrecords at the gs:// locations. 

So it goes something like this:

MIDI -> String -> Note Sequence -> Performance encoding (one-hot) -> Serialization -> TFRecord

Hope this helps,

Jason

--
Magenta project: magenta.tensorflow.org
To post to this group, send email to magenta...@tensorflow.org
To unsubscribe from this group, send email to magenta-discuss+unsubscribe@tensorflow.org
---
You received this message because you are subscribed to the Google Groups "Magenta Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to magenta-discuss+unsubscribe@tensorflow.org.

[robin meier]

Thank you Jason! This is very helpful. Actually I’m interested in the Piano-e-Performance dataset. Is that available somewhere whilst we wait for a pre-trained transformer model? thank you, Robin

[Jason St George]

You’re welcome.

Yes, the MAESTRO dataset contained several years of the Piano-e-Competition disklvier (midi) and raw (wav) recordings available at: https://magenta.tensorflow.org/datasets/maestro

It’s ~103gb uncompressed

[robin meier]

Hello,

two questions:

A) I’d like to prime a transformer model with melody fragments and obtain midifiles as a result. ideally i’d constrain the model to a list of pitch-class-histograms as you can do in performanceRNN. Is this currently possible?

B) i’ve installed the latest magenta, including score2perf using python3. i’ve setup my google cloud platform and ran the wordcount demo successfully with python2.7, i’m now trying to run following command:

PROBLEM=score2perf_maestro_language_uncropped_aug

BUCKET=bucket_name

PROJECT=project_name

PIPELINE_OPTIONS=\

"--runner=DataflowRunner,"\

"--project=${PROJECT},"\

"--temp_location=gs://${BUCKET}/tmp,"\

"--setup_file=/path/to/setup.py"

t2t_datagen \

  --data_dir=gs://${BUCKET}/datagen \

  --problem=${PROBLEM} \

  --pipeline_options="${PIPELINE_OPTIONS}" \

  --alsologtostderr

however, I used python 3 to install magenta (as it won’t compile with python 2.7)

but as you say, apache beam uses python 2. so i now get the error

  File "/Users/meier/Dropbox/transformer-gymno/lib/python2.7/site-packages/pkg_resources/__init__.py", line 786, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'matplotlib>=1.5.3' distribution was not found and is required by magenta

when running t2t_datagen

what am I doing wrong?

should I run in a different environment?

thank you for your help.

also - please let me know if this should be an issue on github rather than posted here?

r

[Ian Simon]

Hi Robin, in PIPELINE_OPTIONS are you setting --setup_file to the actual path of the setup.py file in Magenta's root directory?

-Ian

Hello,

To unsubscribe from this group, send email to magenta-discu...@tensorflow.org

---
You received this message because you are subscribed to the Google Groups "Magenta Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to magenta-discu...@tensorflow.org.

--

Jason St George

"Talking about music is like dancing about architecture"

--

Jason St George

"Talking about music is like dancing about architecture"

--
Magenta project: magenta.tensorflow.org
To post to this group, send email to magenta...@tensorflow.org

To unsubscribe from this group, send email to magenta-discu...@tensorflow.org

---
You received this message because you are subscribed to the Google Groups "Magenta Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to magenta-discu...@tensorflow.org.

[jps...@g.rit.edu]

Robin,

Have you had any luck priming a transformer model with a sequence?

Jason

Hello,

To unsubscribe from this group, send email to magenta-discu...@tensorflow.org

---
You received this message because you are subscribed to the Google Groups "Magenta Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to magenta-discu...@tensorflow.org.

--

Jason St George

"Talking about music is like dancing about architecture"

[robin meier]

Hello Magenters,

is there a pre-trained checkpoint available to experiment with music transformer?

https://magenta.tensorflow.org/music-transformer

i’ve been looking through the magenta github but couldn’t find anything.

unfortunately i do not have access to a gpu and from past experience i’m under the impression that training my own model would take months :-/

thank you in advance for your help,

robin