App Interoperability, XML etc.

[Aaron Andrew Hunt]

Hello everyone. It was great to meet you all and I want to continue come conversations started at MELM and start some new ones.

First, several of us had started discussing XML, and how we can work with it as a means to achieve interoperability, or at least interchange between our softwares. So I want to open up that topic here.

I think all of our software to be as compatible with each other as possible, so how do we make that happen?

For example, some of my customers who use my Custom Scale Editor (CSE) on Windows have asked for features in my software to allow them to work with Robert Walker's Fractal Tune Smithy (FTS) in conjunction with CSE. Over the years, I have added features to CSE so that data can be exchanged by cut and paste from FTS, just by working with customers. I never even talked to Robert before this event. Now I have some contact with Robert, I want to know more about how I can make CSE more friendly to FTS.

The point is, if we have maximum compatibility, including file exchange, this will benefit everyone. We already have Scala files as a default, so that's fine, but these files don't work for anything beyond simple scale data.

For Mus2, Notist, Nota, FTS, and all my software, let's look into ways to share output and make interchange easier. XML and XSL seem to be the way to go for file formats.

Just a comment as well -- with all these things, I am wary that there is some initial interest and energy behind the idea when it begins, and some more ideas are exchanged, and then nothing happens after that. At worst, this message will sit here on the server with maybe a few replies, and nothing to come of it. We're all busy, but it would be great if we could be energetic and involved and not let the typical thing happen ; )

Looking forward to how this can develop.

Cheers,
Aaron
=====

[Utku Uzmen]

Hi,

For starters I'd like to brainstorm on what kind of information would be exchanged. From the Mus2 point of view, the most obvious is importing tuning data into Mus2, and in the present state, this can be done with Scala files and Mus2's own tuning files. For export, aside from the obvious MusicXML format, at the moment I can only think of MIDI files.

Speaking of file formats, the Mus2 formats are simply XML files compressed with zlib. If there is interest I can now put up a basic decompression utility that we've used internally for testing, so that those interested can look at the plain-text XML trees.

Utku Uzmen

--
MANAGERS ONLY: To post to MELM" group, send email to mton...@googlegroups.com
To unsubscribe from this group, send email to mtonalist+...@googlegroups.com

GOOGLE TRANSLATE IT: http://translate.google.com.tr

[Aaron Andrew Hunt]

Hi Utku.

Yes, please post the zlib utility, if it's not too much trouble. I don't know how files work on Google Groups...

Anyway, I have zlib functions in my IDE which I can investigate, but others might not have them; if you have a utility, it will be easier for everyone to just use that.

Cheers,
Aaron

[Utku Uzmen]

I've uploaded the project source for the utility to http://db.tt/kieEHzzR. You will need the Qt SDK from http://qt.nokia.com to compile and run it.

If you want to use generic zlib functions to decompress the files, you will need to skip the first 4 bytes in the file and pass the remaining data to zlib. (Qt uses an extra 4 bytes at start to denote the length of the data)

Utku Uzmen

[Aaron Andrew Hunt]

Thanks a lot for the utility.

Decompressing a .mus2 file, I see that your XML style is heavy on attribute lists. I understand why, as it is more compact than atomic tags. Just for comparison, for example, you have this line:

<level at="1" value="275.242" cents="87.8048780488"/>

Which in tag atoms would be:

<level>
<at>1</at>
<value>275.242</value>
<cents>87.8048780488</cents>
</level>

This is a question of style and preference, but it also has a pretty huge impact on the way XSLT has to be written to work on the documents, and how a schema or DTD is written for error checking. It also has an impact on how the XML tree can be modified in future versions of your file format.

The arguments against using attribute lists are many, so I guess you are already familiar with them and made your choice deliberately. Please pardon me if it appears I am challenging your choice; I am no expert on XML, I only wonder if you would consider moving towards atomic tag structures for the sake of interoperability and also just for the future of Mus2? Of course these are your own formats, so it's totally your preference.

I feel this issue should be discussed because right now I see that our XML styles are at radical opposite ends, and if we have common style guidelines it helps us all work together more easily. Can others interested in this thread please comment briefly on their preferred XML style? I don't want to get into debates over it; I just want to see where we all are and how we feel about having common styles for the sake of collaboration.

Cheers,
Aaron
=====

[Robert Walker]

Hi everyone,

sorry been off-line for a while - mainly because of a lightning strike 1.30 am on New years day. I'm in a rural area with many houses affected by the same storm, most of them on their own lines, it took the repairman a week to get around to fixing my broadband and telephone.

As far as tuning is concerned, FTS does the .tun, and the .scl formats.

It has only partial support for SCALA keyboard maps at present, can't remember the details now as it's rare for anyone to use the feature to import them, some kind of minor incompatibility issue with the internal format FTS uses (the Scala format is the better one but I didn't know about it when I originally wrote FTS).

Also there's an option to add several extra sections to the .tun file format, which other programs will normally ignore,to help with sharing .tun files between different copies of FTS. This gives the pitch of the 1/1, the scale description and scale, and the SCALA keyboard map, also the intervals as intervals from the 1/1 instead of absolute frequencies.

They aren't part of the official .TUN format, and if you choose to add them to the .TUN file FTS will warn you about that, but the sections should be ignored by any .TUN file parser written to be forward compatible when new sections are added to the format.

The scala keyboard map or some way to specify how the notes are to be played from a midi keyboard is the only other main thing I can think of that you would want to add that FTS uses currently.

I don't have any xml format files, but doubtless could add the capability to read them if we devise a new format. 

No particular views a present about details of xml format as I've done no research into it and don't know what the merits or disadvantages are of different ways of doing it.

Robert

[Robert Walker]

Hi everyone, 

oh there's one other thing I include in the FTS unofficial extension of the .tun format, which is the mode, similar idea to the SCALA modes file, basic idea is just that if you have a large scale, then the piece is probably in some smaller subset of it so rather than give the subset as a scale of its own, it is clearer to give both the scale (tuning) and the mode (a subset from the larger scale)

In FTS I call them "arpeggios" because with the fractal tunes especially they don't need to be strictly ascending, you can get arpeggios like 0 1 2 -1 -3 5 or whatever, so a bit confusing for the user to call them modes and also many people think of a mode as a rotation of the diatonic scale rather than a selection of notes from a larger scale, while the word "arpeggio" for my users more clearly explains what it does.

I think that is about it as far as currently exported tuning related file formats in FTS, may be useful in deciding how to do the xml.

Robert

[Aaron Andrew Hunt]

Hi Robert,

As I think I mentioned at MELM, for all these innovative but non-standard things you're putting in the files, the new (several years old now) TUN standard likely has provisions. See version 2.0 here <http://www.mark-henning.de/download_e.php#Tuning>

Actually, we all should look at the TUN 2.0 as another exchange format. There is free C++ source there and also Its developer Mark Henning modeled many of the features of TUN 2.0 on my sketches of TuningXML. The .MSF file format was made specifically for allowing multi-channel tuning tables for Tonal Plexus keyboards. It is implemented in his Windows softsynth Anamark, but I don't know of other software supporting it.

Aaron

[Robert Walker]

Hi Aaron,

Yes I had a look at that, after you mentioned it, but it's hard to understand the pdf, not enough simple examples (no actual scala scales included).

Do you have any examples we can look at, to see how it works for a simple scale + pitch of 1/1 plus keyboard mapping? Is there any support for modes?

The relevant sections seem to be

[Scale Begin]

; The data of the first scale dataset

[Scale End]

I didn't understand this as he gave no actual examples, just a comment "; The data of the first scale dataset".

Can you just include the contents of a scala .scl file there (complete with description, numbers of notes etc) or what can you put there, if just the scale values, do you do them one to a line or can they all be on one line, and what do you do about the pitch of the 1/1 for the scale?

 [Mapping]

seems straightforward but would like to see an example to make sure I understand.

[Functional Tuning]
At first sight seems far too complex for most microtonal purposes, features that aren't needed for tuning exchange, and won't be easy for users to read the .tun file and edit it by hand, basically a script and if you e.g. leave out an * or get the syntax wrong won't import successfully.

If I understand it right, you could do a scala scale plus the 1/1 as, say

InitEqual(261.6255653)

note 1 = "*9/8"

note 2 = "*5/4"

note 3 = "*3/2"

note 4 = "*5/3"

note 5 = "*2/1"

but I'm not sure from reading it how you make the scale periodic if defined in ratios like that.

If anyone wants the scales to be exported in .tun format the format is surely flexible enough to do it and it is probably just a matter of reading it  more carefully and asking for help if needed or looking at examples.

 

I know he has free source code, and I could add that to my program as an importer of the files. I can look into that if anyone ever asks for it. I think the version 2 of the .tun format isn't widely used yet as I've never been asked to support it, ditto never been asked to add support for MSF.

To make it possible to export data from FTS without use of the new optional sections, I would need a simple easy user editable way of embedding the scale, the pitch of the 1/1 and the mode (arpeggio) and keyboard mapping.

It's not a big deal, the optional FTS sections are probably rarely used, hidden in the "more" versrion of a rarely used part of the program - but there might be some users who use them to make it possible to read old .tun files and see in FTS what scale, arpeggio and keyboard mapping they had in FTS when they generated the file.

Any examples of MSF to look at?

The original .tun format is good, a complete mapping of midi note numbers to exact frequencies is a good optional section to have, and I can understand why it became so popular.

Robert

[Utku Uzmen]

Attribute lists are more convenient to code for with a streaming XML processor as opposed to loading the entire tree into memory, and streaming is generally more efficient than loading the entire tree. IMO it also corresponds better to an object-oriented structure, with tags as objects and attributes as properties. And in Mus2 the XML tree is just a reflection of the internal objects.

So I have no desire to change the file formats and end up with even more compatibility functions, but there's no reason a new format has to be based on Mus2 formats or conventions.

Utku Uzmen

[Aaron Andrew Hunt]

Yes, I understand and I agree completely that the structure of objects with properties is very logically translated into tags with attributes. This is why I drafted TuningXML in exactly that way.

But I ran into problems with DTD and schema, and trying to make changes was very difficult. Not for debate, but just to state the major problems with XML attributes (taken from <http://www.w3schools.com/xml/xml_attributes.asp> but the same is found many other places online)

(1) attributes cannot contain multiple values (elements can)
(2) attributes cannot contain tree structures (elements can)
(3) attributes are not easily expandable (for future changes)

That particular page ends with this admonition: "Attributes are difficult to read and maintain. Use elements for data. Use attributes for information that is not relevant to the data."

So, after some time I came to reject attributes for these reasons, and now I only use atomic tags, and only use attributes for nonessential metadata. Since this is the recommended style for XML by coding "authorities", most people are also working this way now.

I understand using the XMLReader versus XMLDocument paradigm for loading are two different ways of working, but (for me at least) I find that reading in a stream is more error prone than reading the whole document, and the difference in time to process isn't that significant.

But I am not telling anyone what to do, of course, only giving some information for consideration. After I changed my mind away from attributes, I never went back and rewrote TuningXML because it was too much (tedious) work. I assume with some kind of automatic processing Qt can save XML style either way, but I don't know much about Qt ... yet : )

Cheers,
Aaron
=====

[Utku Uzmen]

I have to admit I never considered DTD, schemas, etc.

(1) attributes cannot contain multiple values (elements can)
(2) attributes cannot contain tree structures (elements can)

These are true and in the Mus2 formats only single, simple values are contained in attributes. (3) is open to debate but there's some truth there as well. As for streaming vs. loading the entire tree, the difference in speed and memory is indeed irrelevant for a modern PC, but it's noticeable for a mobile app or processing on a web server. (As a side note, Qt has classes for both methods, but their developers stated that they see streaming as the future and may discontinue the other classes)

In any case, I do think a file format should be designed for usefulness before coding convenience and I can see the case for elements.

Utku Uzmen

[Utku Uzmen]

On Jan 13, 2012, at 2:41 AM, Robert Walker wrote:

The scala keyboard map or some way to specify how the notes are to be played from a midi keyboard is the only other main thing I can think of that you would want to add that FTS uses currently.

This is also something we will need for Mus2 once the MIDI recording functionality is finished. There will be a facility for creating mappings inside Mus2, but I suspect there is other software out there that is more convenient for creating mappings.

Utku Uzmen

[Aaron Andrew Hunt]

On Jan 14, 2012, at 12:01 PM, Utku Uzmen wrote:
> There will be a facility for creating mappings inside Mus2, but I suspect there is other software out there that is more convenient for creating mappings.

This is what CSE is for. Tunings can be created in a huge number of ways, including importing any number of Scala files and mapping to a keyboard. CSE files can be exported in a variety of formats, including .scl + .kbm. There is a .Hz format that is simply a list of frequencies in a text file. No headers, nothing else, just Hz values, in chunks of 128, each chunk being a MIDI channel, for multi-channel support (TBX1 supports 4 channels, Tonal Plexus keyboard support all 16 MIDI channels).

Generally, the problem with all apps that support Scala files is the mapping. VSTs, AUs, standalone synths, you name it, it is the keyboard mapping that is the problem.

Apart from being the companion to TBX1 hardware, this whole problem of mapping scales is why CSE exists. It's supposed to make that process as easy as possible and provide export options for as many uses as possible. I guess I did a really bad job demonstrating it at MELM because I obviously didn't get that across; sorry.

I personally do not like having 2 files .scl + .kbm. I find this combination of 2 files an extremely unattractive option, for two reasons. First, because it forces every program that supports it to have intelligent parsing, which is too discouraging for developers who don't know anything about tuning and see it as a bother. Second, simply because it requires these two different files that have no internal connection so that it is impossible to error check. It may offer some flexibility for the user, but I really don't see any advantage, when an app like CSE exists to create 1 file that is exactly what is wanted, clearly showing "this MIDI key = this pitch" relationships for every note.

CSE provides formats that have 128 values for each MIDI channel you want to tune, up to 16 channels in a file in two formats .Hz and .csv. Valid file sizes are in increments of 128 values and those values correspond directly to MIDI notes. I can't think of anything simpler than that. The parser doesn't need to do any calculation, or hardly any after it gets the values. CSV is direct pitch bend data. Hz is frequency data.

If we want this stuff to be supported, we need standards that are as simple as possible to implement, and we need documentation and free code to do it. We're not there yet. I wanted to do it with CSE, it's not there yet. So let's do it already!?

Cheers,
Aaron

[Aaron Andrew Hunt]

I haven't looked at them for a long time, but I thought the docs for TUN 2.0 were well done. CSE can export .msf files if you'd like to see some, try a scale in CSE and export .msf. About the "Functional Tuning", this was a result of one-ups-manship going on with its developer trying to obviate the need for an app like CSE, which I thought was a little nuts. Equations in a tuning file doesn't make sense unless the file is for an editing program, IMO.

[Robert Walker]

Aaron, yes it's nicely laid out and the content is clearly stated, so in that sense the document was fine.

But not really what I'd call a complete specification as it was incomplete, there wasn't enough information there to write your own code to input or output format 2 tun files - if that was the intention. 

MY FIRST IMPRESSIONS OF THE .TUN VERSION 2 FILE SPEC

If it was just meant as a user guide for programmers using the provided source code to output and input .tun files, perhaps it was okay, assuming there is more documentation in the code itself.

The documentation was fine about the tags you use but didn't say that much about what goes into the sections, for the ones I was interested in - in the case of the scala scales, said nothing at all about what goes into the section except for that comment " ; The data of the first scale dataset". 

It should give guidelines about how the data is formatted. 

It is probably very easy to fix, in this case anyway. He just needs to add a line or two to the documentation, with a link to the scala scale specification. If there are any differences, then the documenation should say how it is different (e.g. that it is exactly the same except, you can, or must, omit the description or omit the number of notes field, or that scale values are permitted to be separated by any white space and don't need to be one to a line).

It always helps to have a few examples, (which he doesn't have in case of the scales), and a good spec should have a few examples for any significant type of tag or data type, to help the user to confirm their understanding of the document. See e.g. the html spec which is full of examples:

http://www.w3.org/TR/1999/REC-html401-19991224/struct/lists.html

But examples aren't enough, you need documentation as well, e.g. for the scl format, an example wouldn't be enough to clarify whether the scale values have to be one to a line or are permitted to be all on the same line separated only by white space. 

The scala scl format page I think is a fine model of how things like this should be done:

http://www.huygens-fokker.org/scala/scl_format.html

- and, as he does, you should always include examples of valid data, and an example of a complete valid file so the user can check their understanding of the specification. 

When I read the .tun version 2.0 spec, then I was left not knowing how a program should format or parse the file. Also, that was because the information wasn't there, neither explicitly nor as references to follow up to get the information - and not because there was something I failed to understand.

I don't know if that is anything to do with why the version 2.0 hasn't been widely adopted, it might be, if I was inclined to add it to my program and read that specification I'd go away scratching my head.If very keen on it would contact the author, but otherwise I'd probably not take it any further. I might well reason that because the spec doesn't describe everything in full detail, it just doesn't seem fully developed, more like after the event documentation of what his code does than a file spec proper.

Which, I think is perhaps what it is, perhaps he assumes that most users will just use the provided file spec and not bother to try to write their own code to read or write .tun files.

IS IT A GOOD MODEL TO FOLLOW?

So anyway - it's not far off from what we've been talking about, with a clearer specification, it could be quite good, lets you combine the .tun, .scl and .kbm files together into a single file. 

VALUE OF HUMAN READABILITY OF THE FILE

The original .tun format is successful because it is easy for a program to read and to output, and also easy for a human to read and edit by hand - I think that's an important point, especially since it is used for tuning synths, and for that the note number = value is perhaps better than just a list of 128 values without the note numbers.

I'm not sure if the simplicity of just the values without numbers is worth the loss in readability for the user, since it is pretty easy to parse the original .tun file, and really easy to output it. It works well with scales, but for keyboard mappings, the user wants to know "what pitch is played by this midi key?" so it's good to be able to just look at the file and see the answer.

That's also an advantage of simple text formats like scl, kbm and the original .tun over xml which isn't easy for a human to read or edit, especially if not a programmer

The scl format ditto is very easy to read and edit by hand, and you can copy / paste the contents of a .scl file into an email even.

The kbm format is less often used, but it works well too.

Since I have code to output scl files, kbm files and the original .tun files, then it is probably quite easy to combine them together to output a version 2 .tun file without need to use his source code, if I have a clear spec for it.

REASONS FOR INCLUDING THE SCL AND KBM SECTIONS

There is a value in having separate .scl and .kbm formats,as options..

It's then obvious what the scale is that you are playing, which can be hard to see with just the .tun file.Basically I see that as the value of the .kbm format, it lets you separate the scale out so you can see what it is.

In a combined file, if for general use I think you should require a simple section such as the original .tun which any program can read without need to examine the other sections such as the .scl if they don't need them - for retuning VSTi for instance, where the programmer just needs to know how to tune each midi note.

APPLICATION TO THE XML IDEAS

So anyway the .tun format might be an idea perhaps, if we work with the developer and get those sections clarified and extra sections added to it. But that would mean dropping xml. It depends on whether you want easy editingn and readability of a simple text format, or want something you mainly edit using software.

It is probably also a good idea to think about what the new format is for, is it a general format for all microtonal software (including non octave, large scales with arbitrarily many notes to an octave, non octave scales etc), or is it specifically for Turkish maqams, or perhaps more generally for music based on a twelve tone system with retuning of the notes, and accidentals?

I'm not clear yet why you feel the need for a new file format, so it could be useful to explain that perhaps?

FOR FTS

For my own use, then I don't need to support the version 2.0 .tun format at present, but will certainly look into it if anyone does need it at a future date.

Though I have the option in FTS to set up separate scales for each part, until you mentioned the MSF I didn't know of any other program that could use that data if exported. The FTS parts correspond to the midi input channels when midi relaying notes from another program (e.g. retuning the midi output of notation software).

So anyway if it is needed by anyone it's probably easy to add the MSF file format to FTS for import and export, I can add that to the wish list, do you think that is useful to have? Have any of your users asked yet for a way to exchange multiple scales between FTS and your software?

I imagine it's probably low priority, as not many people use multiple scales like that, though enough so that there is some interest in them - you get example compositions of this type posted to the Xenharmonic Alliance from time to time, 

If so I'll just add it to the wish list and can look into it later when I'm working on the next release of the main FTS program.

Robert

[Aaron Andrew Hunt]

Hi Robert.

First, thanks for such a long message, but please don't feel you need to respond to everything in so much detail. I worry a little that you have other more important things to do ; )

I wasn't suggesting that TUN 2.0 be adopted by anyone actually. I wouldn't have done anything with TuningXML otherwise. I wouldn't suggest worrying about it for FTS, Robert, seriously. I only suggested looking at it in order perhaps to make your extra sections fit in with an existing specification.

If we want to discuss the specifics of TUN 2.0, they were really modeled on TuningXML. I guess I should point to my initial sketches of that. They appear rather ridiculous to me now, especially because I didn't understand XML at all back then. I was just exploring the idea and trying out different things. I was doing a lot of things simply wrong back then. You'll see, if you know XML at all. I asked for help, but nobody helped me then. Here it is anyway. <http://www.h-pi.com/wordpress/?p=24>

As far as human readability, XML is verbose and more human readable than any of these other files. XML allows us to clearly state what the data is in tags with specific names for everything. This is one of the main reasons to use it, to me at least.

Consider that a format like Scala really isn't readable on the same way. You have to know what a scala file is representing in order to make sense out of it. The fact that a number with a point and a number with a slash don't represent the same thing is absolutely not intuitive, for example. With XML, you may not know what a tag is if you don't know that word, but consulting a dictionary will answer the question.

The XML language gives structured data with human readable elements, so that the whole structure of the data in the file makes logical sense without any other reference whatsoever. This is very useful!

Why have a new format? The reasons are many; here are a few:

(1) syntax standardization - XML is a standard for any kind of data, we should have ghettoized formats for our stuff
(2) human readability - tags explain everything in an WML file, comments make up for any ambiguity
(3) interchange - XML parsers automatically error check and get the data already structured
(4) versatility - the file can represent things simple or complex, from a tuning to a notation file
(5) expandability - just add more tags
(6) adaptability - the data can be translated using XSL into any form imaginable

OK if I go on I'll start talking at cross purposes, and contradicting myself.

... and I also have a lot of other work I need to be doing ; )

Cheers,
Aaron
=====

[Aaron Andrew Hunt]

correction -- "we should *NOT* have ghettoized formats for our stuff"

[Robert Walker]

Hi Aaron,

Thanks for your reply. I type fast so it took less time than you might think, though did spend a bit of time thinking about the issues.

I understand what you are saying. Will probably be mainly a spectator for this discussion from now on.

I can understand the values of xml. It depends what you mean by readability, xml is more readable in some ways, plain text in other ways. I think an xml version of the scl file format would never have become as popular amongst microtonalists generally as the plain text version. The verboseness of xml can make it harder to read and edit the text file, and means more characters to type if you need to create it by hand, and more possibilities to get things wrong e.g. to misspell a tag or forget to close it or leave out a < or > which your average non programmer is very likely to do.

Also for the .tun format perhaps. 

So, XML files won't so easy to edit by hand for a non programmer though easy enough for someone who is already familiar with raw html.

In practise I think an xml file will only be edited by hand by users who are web page coders or programmers. But it probably doesn't need to be depending on how it is meant to be used.

Anyway that's about all I want to say, don't want to get in a long discussion of all this especially as FTS doesn't use xml at all so I'm happy to just go along with whatever you come up with when the time comes to add it to FTS. But will keep an eye on what you all say and come in again if there is anything else I feel I might be able to contribute.

Thanks,

Robert

In the case of the scl file format, it depends what you mean by readability, 

[Ozan Yarman]

I am observing with quite some satisfaction that new avenues for collaboration has been made possible due to our recent gathering. :)

I also once suggested to monz that he could perhaps allow his Tonescape code be ported into Mus2.

Oz.

✩ ✩ ✩
www.ozanyarman.com