Evaluation Metrics for Symbolic Music Generation

[Sonnery Hugo]

Hi Magenta !

[Note : Sorry for the cross-posting!]

I am a Masters student currently working on a research project at INRIA/TAU (Paris) dedicated to symbolic music generation. It's great to be part of this exciting group !!

I am currently looking into the analogy between NLP and symbolic music processing. I am aware of how ad-hoc metrics assessing one specific property of an ideal composition have already been proposed (cf. LSTM RL Tuner, 2016) ; however, do you know if there have been any proposed comprehensive, general-purpose scores for assessing musical quality and / or musical similarity wrt. another reference score or a reference dataset of scores (like BLEU and its variants in NLP for example) ? Being still unfamiliar with the MIR literature, it seems like some papers tried to answer that question partially in the past, but it remains unclear to me whether one metric has become prevalent now ...

Thank you very much in advance for your time reading my short message, and I look forward to reading you !

Best,

Hugo S.

[Bob Sturm]

Hi Hugo!

A variety of such approaches have been proposed:

@article{Yang2018a,
author = {L.-C. Yang and A. Lerch},
journal = {Neural Computing and Applications},
title = {On the evaluation of generative models in music},
year = {2018}}

@inproceedings{Ens2018,
address = {Salamanca, Spain},
author = {J. Ens and P. Pasquier},
booktitle = {Proc. Int. Conf. Computational Creativity},
month = {June},
title = {A Cross-Domain Analytic Evaluation Methodology for Style Imitation},
year = {2018}}

@inproceedings{Colombo2017a,
author = {F. Colombo and A. Seeholzer and W. Gerstner},
booktitle = {Proc. EvoMUSART},
title = {Deep Artificial Composer: A Creative Neural Network Model for Automated Melody Generation},
year = {2017}}

@article{Sturm2016c,
author = {B. L. Sturm and O. Ben-Tal},
journal = {J. Creative Music Systems},
month = {Sep.},
number = {1},
title = {Taking the Models back to Music Practice: Evaluating Generative Transcription Models built using Deep Learning},
volume = {2},
year = {2017}}

Best of luck with the thesis!

--
Magenta project: magenta.tensorflow.org
To post to this group, send email to magenta...@tensorflow.org
To unsubscribe from this group, send email to magenta-discu...@tensorflow.org
---
To unsubscribe from this group and stop receiving emails from it, send an email to magenta-discu...@tensorflow.org.

--

Bob L. T. Sturm, Associate Professor

Speech, Music and Hearing

School of Electronic Engineering and Computer Science
Royal Institute of Technology KTH, Sweden
https://highnoongmt.wordpress.com

[Tom Collins]

Hi Hugo,

Here are a few mostly more recent suggestions:

@inproceedings{yin2021good,
  title={“A Good Algorithm Does Not Steal--It Imitates”: The Originality Report as a Means of Measuring When a Music Generation Algorithm Copies Too Much},
  author={Yin, Zongyu and Reuben, Federico and Stepney, Susan and Collins, Tom},
  booktitle={Artificial Intelligence in Music, Sound, Art and Design. EvoMUSART 2021},
  pages={360--375},
  year={2021},
  organization={Springer}
}

@inproceedings{janssen2019algorithmic,
  title={Algorithmic Ability to Predict the Musical Future: Datasets and Evaluation.},
  author={Janssen, Berit and Collins, Tom and Ren, Iris Yuping},
  booktitle={ISMIR},
  pages={208--215},
  year={2019}
}

I still favour a rigorously conducted listening study with listeners of appropriate expertise:

@article{collins2017computer,
  title={Computer-generated stylistic compositions with long-term repetitive and phrasal structure},
  author={Collins, Tom and Laney, Robin},
  journal={Journal of Creative Music Systems},
  volume={1},
  number={2},
  year={2017}
}

@incollection{jordanous2019evaluating,
  title={Evaluating evaluation: Assessing progress and practices in computational creativity research},
  author={Jordanous, Anna},
  booktitle={Computational Creativity},
  pages={211--236},
  year={2019},
  publisher={Springer}
}

Kind regards,

Tom

Tom Collins, PhD

"Circus" by human-AI band, The elephants and the, is released Tue 1st Jun

https://ai-circus.glitch.me

https://tomcollinsresearch.net
https://mstrcyork.org
Lecturer (Assistant Professor) in Music Technology
Department of Music
University of York

[Sonnery Hugo]

Dear Professor Collins, Dear Professor Sturm,

Thank you very much for the great references ! 

I will make sure to keep you posted on my advancement !

Sincerely yours,

Hugo S.