I'm at it again. I recently stumbled onto my old Midi code (I've been
doing a lot of other stuff) and looked at my Midi Device Schema again.
I still think its a good idea but thought it might be cool to see what
you think of it.
First let me explain what I've been trying to do. My idea was to build
the ultimate universal patch editor: a software program that has full
access to all the settings of a (hardware/virtual) device. In order to
gain access to all settings of a device, Midi System Exclusive message
will have to be sent and received. So I started investigating all
different kind of SysEx messages from different manufacturers. I
analysed examples from Roland and Yamaha but also from small firms
(that had a really crappy SysEx impl. ;-).
I didn't want to hard-code device-specific SysEx support into my app
as is the common way to do things (as far as I could see). I wanted my
application to support ALL devices. So that is how I came up with the
idea of a device schema.
A Midi Device Schema is a description of the SysEx messages a specific
Midi device supports. It also describes the full contents of these
SysEx messages at a logical level. So, theoretically when you are able
to describe the SysEx messages of a device, the application could
access all its settings. As a user, to get your device supported by my
app., you just needed to provide the app. with the correct schema of
your device and it would work. I think that this is still a great
(commercial) argument to justify the effort put into it (already).
Each field in a device schema represents a logical value inside the
SysEx message and is defined with a name and a data type. The data
type says something about the physical representation of that field.
Some logical fields are physical bits, some are nibbles or a single
byte, others are multiple bytes (text for instance).
Each data type is related to a converter in code. This converter knows
how to perform the conversion from logical to physical and visa versa.
When a (part of a) device schema is read, a converter for each field
is put in sequence. Reading a SysEx message is nothing more than
putting the binary stream of data through a chain of converters
yielding logical values for that message. Writing a SysEx message
requires the application to supply the logical values for each field
and the chain of converters to create a physical SysEx message.
This is the basic mechanism that allows you to read and write SysEx
messages for any device, provided you have the correct device schema.
On Tue, 22 Sep 2009, obiwanjacobi wrote: > What do you think? > Is it worth pursuing or is it a stupid idea?
You seem to be talking only about manufacturer sysex - there are many more types of sysex messages, mainly in the real time and non-real time categories. I would focus on them... ;-)
> You seem to be talking only about manufacturer sysex - there are many more types
> of sysex messages, mainly in the real time and non-real time categories. I
> would focus on them... ;-)
> Especially MIDI Show Control messages....
Yes, that was my initial goal: to be able to talk to all Midi devices
using device/manufacturer-specific SysEx messages.
But the Device Schema allows you to describe *any* sysex message: also
the official MMA sysex messages or Show control messages (those are
just sysex, right?).
The only concern for real-time messages is the processing speed of the
"engine" that interprets the Device Schema and puts out the sysex
message or interprets it. But that is not something that has my focus
right now: just getting all the features in the framework is. Because
besides being able to describe a sysex message and read and write its
data, there are some other 'features' that manufacturers like Roland
and Yamaha use: address maps (ask if you want to know more about
that ;-).
yeah , i am finding the bytes of my casio cz101 a tad difficult to decipher.
there were many attempts at such a controller during the atari era .. most
serve good working purpose .
but when there are 1000's of synths with its own sysex it imply that the
program would be very large in size and be ridiculos cause most of us only
have 1-10 different models of synthesizer... the rest would be dead weight
----- Original Message -----
From: "obiwanjacobi" <obiwanjac...@hotmail.com>
To: "mididev" <mididev@googlegroups.com>
Sent: Tuesday, September 22, 2009 11:26 PM
Subject: [mididev] Re: The ultimate uber SysEx code library compatible with
*ALL* Midi devices
> You seem to be talking only about manufacturer sysex - there are many more
types
> of sysex messages, mainly in the real time and non-real time categories. I
> would focus on them... ;-)
> Especially MIDI Show Control messages....
Yes, that was my initial goal: to be able to talk to all Midi devices
using device/manufacturer-specific SysEx messages.
But the Device Schema allows you to describe *any* sysex message: also
the official MMA sysex messages or Show control messages (those are
just sysex, right?).
The only concern for real-time messages is the processing speed of the
"engine" that interprets the Device Schema and puts out the sysex
message or interprets it. But that is not something that has my focus
right now: just getting all the features in the framework is. Because
besides being able to describe a sysex message and read and write its
data, there are some other 'features' that manufacturers like Roland
and Yamaha use: address maps (ask if you want to know more about
that ;-).
i might say its a great ideal , but stupid because it has no sysex
handshaking protocol.
but that might be easilly coded in too .
i know i been coding for years and to be honest us here at this group are
the only active midi musicians/programmers... that means not enough user
base to gain feedback or input.
charles
----- Original Message -----
From: "obiwanjacobi" <obiwanjac...@hotmail.com>
To: "mididev" <mididev@googlegroups.com>
Sent: Tuesday, September 22, 2009 11:13 PM
Subject: [mididev] The ultimate uber SysEx code library compatible with
*ALL* Midi devices
> I'm at it again. I recently stumbled onto my old Midi code (I've been
> doing a lot of other stuff) and looked at my Midi Device Schema again.
> I still think its a good idea but thought it might be cool to see what
> you think of it.
> First let me explain what I've been trying to do. My idea was to build
> the ultimate universal patch editor: a software program that has full
> access to all the settings of a (hardware/virtual) device. In order to
> gain access to all settings of a device, Midi System Exclusive message
> will have to be sent and received. So I started investigating all
> different kind of SysEx messages from different manufacturers. I
> analysed examples from Roland and Yamaha but also from small firms
> (that had a really crappy SysEx impl. ;-).
> I didn't want to hard-code device-specific SysEx support into my app
> as is the common way to do things (as far as I could see). I wanted my
> application to support ALL devices. So that is how I came up with the
> idea of a device schema.
> A Midi Device Schema is a description of the SysEx messages a specific
> Midi device supports. It also describes the full contents of these
> SysEx messages at a logical level. So, theoretically when you are able
> to describe the SysEx messages of a device, the application could
> access all its settings. As a user, to get your device supported by my
> app., you just needed to provide the app. with the correct schema of
> your device and it would work. I think that this is still a great
> (commercial) argument to justify the effort put into it (already).
> Each field in a device schema represents a logical value inside the
> SysEx message and is defined with a name and a data type. The data
> type says something about the physical representation of that field.
> Some logical fields are physical bits, some are nibbles or a single
> byte, others are multiple bytes (text for instance).
> Each data type is related to a converter in code. This converter knows
> how to perform the conversion from logical to physical and visa versa.
> When a (part of a) device schema is read, a converter for each field
> is put in sequence. Reading a SysEx message is nothing more than
> putting the binary stream of data through a chain of converters
> yielding logical values for that message. Writing a SysEx message
> requires the application to supply the logical values for each field
> and the chain of converters to create a physical SysEx message.
> This is the basic mechanism that allows you to read and write SysEx
> messages for any device, provided you have the correct device schema.
> i might say its a great ideal , but stupid because it has no sysex
> handshaking protocol.
Well, actually that is one of the other features I was talking about.
I assume you mean the message exchange patterns that some
manufacturers (like Roland) use to intelligently request and sent data
between devices; along with messages like 'ack' and 'abort'...?
I haven't build anything yet to support message exchange patterns -
I'm currently stuck at address maps - but I did give it some thought
and I think I have a pretty good idea how to implement it in a pretty
generic way using device schema's and a separate flow schema that
defines the message that can be exchanged. At runtime a state machine
would load in the flow schema and keep track of what messages can be
sent or received.
But still you need to decipher the content of those handshake messages
and that brings you back to the schema (core).
Over at http://www.kvraudio.com I dropped the same question on their
forum and it's awefully quiet... :-(
Would you buy a Midi software program that advertises it can handle
*ALL* Midi devices?
Initially you might think it is BS but if you would know how its done,
would you buy it?
Of course with the usual disclaimers and perhaps I could offer a
service that will get you your schema for free (and available to the
public) by supplying the original sysex documentation.
I've declared a unique data type for each checksum type (algorithm)
and it is automatically calculated or validated through a converter in
code. It's based on intercepting the binary data stream and keeping
count on what values are read or written.
> Of course with the usual disclaimers and perhaps I could offer a
> service that will get you your schema for free (and available to
the
> public) by supplying the original sysex documentation.
An alternate strategy would be to make the program available for free,
and sell the schemas and service to create them. Essentially you'd
have a business of providing custom editors. This works well iff your
schema approach is actually better/easier than writing a custom
editor.
My opinion is that given the difficulty of making a schema approach
work in the general case, and the difficulty of actually writing a
schema complete enough for a given device, the approach is best used
as an internal tool to make your own development of custom editors
more efficient.
> An alternate strategy would be to make the program available for free,
> and sell the schemas and service to create them. Essentially you'd
> have a business of providing custom editors. This works well iff your
> schema approach is actually better/easier than writing a custom
> editor.
I think the schema approach is faster and easier than writing a custom
editor.
Thanx for the suggestion.
> My opinion is that given the difficulty of making a schema approach
> work in the general case, and the difficulty of actually writing a
> schema complete enough for a given device, the approach is best used
> as an internal tool to make your own development of custom editors
> more efficient.
I don't consider writing a fully functional schema as difficult. You
just follow the docs and create a field for each piece of information
and pick the correct data type. After that you have to group it
correctly to get the checksum working, but thats about the only
complexity.
I envision a (DSL) graphic designer application that allows 'me' ;-)
to create a valid device schema quickly.
Because I analysed only a few different sysex implementations, there
is a chance that my library cannot do a certain 'feature' yet that I
might encounter in another Midi Device. But I have confidence in the
architecture that I will be able to patch those holes as we go.
handshaking and checksumis common for most roland stuff.
but i still don't believe this will be a great benifit for couple reasons,
1: too much data to sift through
2:program its self will be too big byte wise
3:too many synth /specs to adheare to
4: if this was greatly needed , by now ,they would have released something
like that before
reconfiguarable data files , self modify code, saving user setups for all
synths , sounds messy ,
i like these sysex savers to be unit specific
sooner have 10-300kb files in programs on my computer than 1-1 gb one
this thing would have to reserve space for memory and know how big a dump
file is too ,no?
i should suggest this "uni-man" by martain terraskean" -name could be little
wrong!
its a universal saver for most atari era stuff...
i wrote one for my atari which saves
dx,
tx,
yamaha pss
and
roland dr660 ,plus couple more
transmits too !
----- Original Message -----
From: "Charlie Richmond" <charl...@RichmondSoundDesign.com>
To: <mididev@googlegroups.com>
Sent: Wednesday, September 23, 2009 6:08 AM
Subject: [mididev] Re: The ultimate uber SysEx code library compatible with
*ALL* Midi devices
> On Wed, 23 Sep 2009, charlie from PARRY wrote:
> > i might say its a great ideal , but stupid because it has no sysex
> > handshaking protocol.
The schema's are parsed once and cached as an object model. Its all in
memory, its all blazingly fast.
Sysex data itself would be no different than any other solution. My
code handles it as best as it can: no extra copying, just one buffer
that is passed through the framework (as a stream).
> 2:program its self will be too big byte wise
I do not target people that run on 640k ;-) Memory is cheap and so is
storage (and it will only get cheaper). BUY MORE RAM! :-P
> 3:too many synth /specs to adheare to
Well, thats the whole idea. But it should be pretty painless to adhere
to all these different specs.
> 4: if this was greatly needed , by now ,they would have released something
> like that before
Depends on who you ask I guess. I personally as a user, I think the
quality of Midi software is very poor and the only real universal
patch editor I've seen (Midi Quest) looks like it was made in the
early 90's. It does support a whole lot of devices, but my newest
drumkit isn't among them... Bummer. Now what? I think a user would
welcome an application that would allow them to add their device
without hassle. Don't you?
> reconfiguarable data files , self modify code, saving user setups for all
> synths , sounds messy ,
Reconfig-data-file: do you mean the schemas?
There is no self-modifying code.
Saving setups is a matter of structuring data on HD.
What do you find messy about it? Isn't this just what application do
(in general)? Surely your apps has config files and needs to store
stuff on HD?
> i like these sysex savers to be unit specific
Why? What is the advantage of having a separate program for each Midi
Device? I can only see disadvantages. Biggest one: no integration. Not
to speak of how each application handles the midi ports (locked!
sorry! ;-)
> sooner have 10-300kb files in programs on my computer than 1-1 gb one
> this thing would have to reserve space for memory and know how big a dump
> file is too ,no?
Why do you think this would be 1GB?
Big sysex dumps can be processed streaming. As new midi buffers come
in from the WIN32 API the engine advances to process the next few
fields. Or you can open a stream on a file, in which case the engine
just goes ahead full speed. For processing it is not necessary to know
in advance what the total length is going to be. It is necessary to
determine the schema to use for interpretation of the sysex data. To
automatically determine the correct schema I still have to do some
work, its a difficult 'feature' ;-)
The universal patch editor that I want to make is also a little
different than what is (or was?) common. All patch editors I've seen
are device-centric: meaning they all put the device at the center of
how you work with that data.
My idea is to be song-centric (or task centric). Why do you want to
edit you Midi device? Well, because you're writing the best song ever
and need a new sound to do it with (or something like that ;-). So the
main screen of my patch editor contains a set of controls (slider,
knobs etc) that are all linked to (potentially) different devices. All
these settings are used in your song and you can tweak them and adjust
them as needed. Of course there is a point where you want to manage
your device and at that point it would resemble the common patch
editor. But it wouldn't be the starting point.
Thanx for your comments Charlie, I really appreciate it.
i know ever time i get a new midi device i make sure i write a sysex send
and reciver for the unit
, mind you i limit myself to the atari as a controller
and most of my units are from the 80's-90's
with no dump largrer than 32767 bytes ,
what field we have entered is ????what about samplers ?????
when you get them loaded with samples and the units program is hand
configured by your self for the channels and such ,
does this fall into your programs criteria?
i mean there are some weird midi things out there too.
i just assumed people used the midi sysex within there sequencers for this
type of interactive approach ,,,
most people even lie about midi in an attempt to "pry" our precious midi
files from us for use
in their
vst
soft synth ,
wav sequencers ,,,,,,
who knows where this topic will utimitally go or stop ? sorry ?
if you can do this sysex thing do it !
nothing stopping you ,right?
----- Original Message -----
From: "obiwanjacobi" <obiwanjac...@hotmail.com>
To: "mididev" <mididev@googlegroups.com>
Sent: Wednesday, September 23, 2009 11:02 PM
Subject: [mididev] Re: The ultimate uber SysEx code library compatible with
*ALL* Midi devices
> 1: too much data to sift through
The schema's are parsed once and cached as an object model. Its all in
memory, its all blazingly fast.
Sysex data itself would be no different than any other solution. My
code handles it as best as it can: no extra copying, just one buffer
that is passed through the framework (as a stream).
> 2:program its self will be too big byte wise
I do not target people that run on 640k ;-) Memory is cheap and so is
storage (and it will only get cheaper). BUY MORE RAM! :-P
> 3:too many synth /specs to adheare to
Well, thats the whole idea. But it should be pretty painless to adhere
to all these different specs.
> 4: if this was greatly needed , by now ,they would have released something
> like that before
Depends on who you ask I guess. I personally as a user, I think the
quality of Midi software is very poor and the only real universal
patch editor I've seen (Midi Quest) looks like it was made in the
early 90's. It does support a whole lot of devices, but my newest
drumkit isn't among them... Bummer. Now what? I think a user would
welcome an application that would allow them to add their device
without hassle. Don't you?
> reconfiguarable data files , self modify code, saving user setups for all
> synths , sounds messy ,
Reconfig-data-file: do you mean the schemas?
There is no self-modifying code.
Saving setups is a matter of structuring data on HD.
What do you find messy about it? Isn't this just what application do
(in general)? Surely your apps has config files and needs to store
stuff on HD?
> i like these sysex savers to be unit specific
Why? What is the advantage of having a separate program for each Midi
Device? I can only see disadvantages. Biggest one: no integration. Not
to speak of how each application handles the midi ports (locked!
sorry! ;-)
> sooner have 10-300kb files in programs on my computer than 1-1 gb one
> this thing would have to reserve space for memory and know how big a dump
> file is too ,no?
Why do you think this would be 1GB?
Big sysex dumps can be processed streaming. As new midi buffers come
in from the WIN32 API the engine advances to process the next few
fields. Or you can open a stream on a file, in which case the engine
just goes ahead full speed. For processing it is not necessary to know
in advance what the total length is going to be. It is necessary to
determine the schema to use for interpretation of the sysex data. To
automatically determine the correct schema I still have to do some
work, its a difficult 'feature' ;-)
The universal patch editor that I want to make is also a little
different than what is (or was?) common. All patch editors I've seen
are device-centric: meaning they all put the device at the center of
how you work with that data.
My idea is to be song-centric (or task centric). Why do you want to
edit you Midi device? Well, because you're writing the best song ever
and need a new sound to do it with (or something like that ;-). So the
main screen of my patch editor contains a set of controls (slider,
knobs etc) that are all linked to (potentially) different devices. All
these settings are used in your song and you can tweak them and adjust
them as needed. Of course there is a point where you want to manage
your device and at that point it would resemble the common patch
editor. But it wouldn't be the starting point.
Thanx for your comments Charlie, I really appreciate it.
> what field we have entered is ????what about samplers ?????
> when you get them loaded with samples and the units program is hand
> configured by your self for the channels and such ,
> does this fall into your programs criteria?
> i mean there are some weird midi things out there too.
Hmmm samplers. Haven't thought about those. But wouldn't the only
thing 'weird' about them being a sample being up/down loaded in sysex?
So you have a 'blob' field in the device schema (need a new data type
for variable length multi-byte value, but that should be no problem)
for those sysex messages. I could even write a converter that can
interpret the blob (RIFF?) and expose the value as a structured wav
file ;-)
The customization that a user performs on his equipment should be
retrievable with sysex messages, right? So by reading 'all' settings
from a device at runtime, you always have a correct representation of
the midi device inside your program.
> i just assumed people used the midi sysex within there sequencers for this
> type of interactive approach ,,,
I also want my patch editor to be able to run as a plugin (as well as
stand alone), if that is possible. To that purpose I started exploring
the VST standard and am working on an open source project called
VST.NET that enables you to write VST plugins in managed code (.NET/C#
language of my choice).
> most people even lie about midi in an attempt to "pry" our precious midi
> files from us for use
> in their
> vst
> soft synth ,
> wav sequencers ,,,,,,
?? I dont get this? what do you mean?
> who knows where this topic will utimitally go or stop ? sorry ?
> if you can do this sysex thing do it !
> nothing stopping you ,right?
Its just that I wanted a second opinion before I sink in a lot more
time into this project. Although I have the basics working, I still
need to address some hefty issues. The address map is the first (the
way Roland and Yamaha structure their sysex messages), after that I
need to address the message exchange patterns and I probably need a
mechanism to automatically determine the device/message schema of an
incoming message by then...
Thanx so much for you comments, Charlie. Although I understand that
you're not a big fan of the idea, you have been a good sparring
partner showing me the 'other side'. Thanx!
|
