Script to create incremental MIDI SysEx cues

[Simon VB]

Hi

I'm using QLab to control an Allen & Heath dLive mixing desk through MIDI commands.

There's an extensive MIDI Protocol available, which I've figured out how to interpret. So I can create MIDI SysEx commands in QLab to control certain functions in the dLive.

So far, so good, but ...

In order to save me many hours of tedious work, it would be great if someone could help me script something to create incremental versions of MIDI cues. 

For example, if I want to recall scene 1, I now have a MIDI cue named 'Scene 1', sending the MIDI command:

BB, 00, 00, CB, 00

Obviously, I'd like such a cue for every available scene (this goes to 500).
What I would like to do is copy this cue, and have a script cue increment both the name and the HEX value, so I get a MIDI cue named 'Scene 2', sending the MIDI command

BB, 00, 00, CB, 01

How would this work?

- Is it possible to script QLab to interpret part of a cue's name as a number and increment it? (I bet starting my numbering with 001 instead of 1 will make this easier when going to 500.)

- Is it possible to script QLab to take some byte of a MIDI command and increment it?

- Can QLab scripting 'count' in HEX, so 09 increments to 0A instead of 10, and 0F increments to 10?

Ideally, this script would be easy to change, so I can use it for incrementing all different sorts of MIDI cues. It's not always the last HEX byte that needs incrementing, you see.

I'm looking forward to the input of the smart people in this group!

[micpool]

This will get you scenes 1 to 256

tell application id "com.figure53.QLab.5" to tell front workspace
    set hexChars to "0123456789ABCDEF"
    set theMIDIconstant to "BB 00 00 CB "
    repeat with x from 0 to 255
        set hexString to (character (x div 16 + 1) of hexChars) & (character (x mod 16 + 1) of hexChars)
        set hexString to theMIDIconstant & hexString
        make type "midi"
        set theCue to last item of (selected as list)
        set the message type of theCue to sysex
        set the sysex message of theCue to hexString
        set the q name of theCue to hexString
    end repeat
end tell

Obviously that's as many scenes as can be addressed by one byte of hex.

Mic

[Rich Walsh]

BB isn’t a valid bit of hex inside a SysEx string: it’s more than 127 so is a system byte not a data byte. BB 00 00 is a bank change (on channel 11), followed by CB 01 – a program change…

This isn’t SysEx. We’ve looked at the awful A&H MIDI documentation a few times, eg: https://groups.google.com/g/qlab/c/ZJMjvOb2898/m/gFWRCJFKCQAJ.

My template does bank change: https://wiki.allthatyouhear.com/doku.php#go_ahead_make_midi. Not updated for QLab 5 yet…

Rich

[micpool]

Yes of course, I just assumed Simon had the message working  with the Allen & Heath and merely added a means of incrementing the last byte of hex.

But as a data byte is only  7 bit then obviously bank switching would be needed to achieve anything beyond Scene 128.

Does this do what's required?

--Generate Incrementing Bank + Hex for PC

tell application id "com.figure53.QLab.5" to tell front workspace
    set hexChars to "0123456789ABCDEF"

    set theBankHex to {"00", "01", "02", "03"}
    repeat with bank from 1 to 4
        repeat with program from 0 to 127
            set progNUm to ((bank - 1) * 128) + (program + 1)
            if progNUm > 500 then return
            set hexString to (character (program div 16 + 1) of hexChars) & (character (program mod 16 + 1) of hexChars)
            make type "group"
            set theGroup to last item of (selected as list)
            set the mode of theGroup to timeline
            set the q name of theGroup to "Program " & progNUm as text
            set the q number of theGroup to "PC" & progNUm
           
            make type "midi"
            set theBankCue to last item of (selected as list)
            set the message type of theBankCue to sysex
            set the sysex message of theBankCue to "BB 00  " & item bank of theBankHex
            set the q name of theBankCue to sysex message of theBankCue
            set the q number of theBankCue to ""
            set theCueID to uniqueID of theBankCue
            move cue id theCueID of parent of theBankCue to end of theGroup
            make type "midi"
            set theMidiCue to last item of (selected as list)
            set the message type of theMidiCue to sysex
            set the sysex message of theMidiCue to "CB " & hexString
            set the q name of theMidiCue to sysex message of theMidiCue
            set the q number of theMidiCue to ""
            set the pre wait of theMidiCue to 0.1
            set theCueID to uniqueID of theMidiCue
            move cue id theCueID of parent of theMidiCue to end of theGroup
        end repeat
    end repeat
end tell

Mic

[Simon VB]

Hi guys

Thank you for your quick response, and sorry, mine is considerably slower!

Rich, I'm aware the commands discussed aren't actually MIDI SysEx.

However, if you want to (or need to) use hexadecimal MIDI commands to control a piece of hardware, these can only be entered in QLab by setting the MIDI cue's 'Message type' to ... drumroll ... 'MIDI SysEx message'! Hence the confusion. I'm sorry if I went over that detail rather lightly in my original question.

Perhaps it's worth renaming this option to 'Hexadecimal MIDI string', or something of the sort?

(I'm surprised you're so appalled by the A&H MIDI documentation. I'm not saying you're wrong - it's the only brand I've ever tried this kind of stuff with, but I've never NOT found what I needed in the documentation.

However, I can imagine that some other brands use a, well, let's say, more organised scheme or something.)

I tried out your template script for creating MIDI cues, it's impressive and it works very well! However, it seems to have no option for creating the hexadecimal strings I needed.

micpool, you were correct to assume that I already had the command working and just needed a way to increment one of the hex bytes.

I liked your first script better (not creating groups of two MIDI cues, but creating just one MIDI cue for each command), but I played around with it to create neater cue names.

This is what I ended up using:

tell application id "com.figure53.QLab.4" to tell front workspace
    set hexChars to "0123456789ABCDEF"
    set theMIDIconstant to "B9 00 00 C9 "
    repeat with x from 0 to 127

        set hexString to (character (x div 16 + 1) of hexChars) & (character (x mod 16 + 1) of hexChars)
        set hexString to theMIDIconstant & hexString
        make type "midi"
        set theCue to last item of (selected as list)
        set the message type of theCue to sysex
        set the sysex message of theCue to hexString

        set nametypeofcue to "Recall scene "
        set numberofCue to x + 1
        set the q name of theCue to nametypeofcue & numberofCue
        set numbertypeofcue to "RS"
        set the q number of theCue to numbertypeofcue & numberofCue
    end repeat
end tell

I have created several instances of this script, the second version changes the theMIDIconstant to B9 00 01 C9 and the numberofCue to x + 129, and so on. I didn't necessarily need to create all 500 scenes with one script, I just needed a script that's simple enough for me to work with ;)

And now I can use this to create all kinds of commands for the dLive, eg. controlling the mutes for input channels. This uses a completely different MIDI string. The script looks like this:

tell application id "com.figure53.QLab.4" to tell front workspace
    set hexChars to "0123456789ABCDEF"
    set theMIDIconstant1 to "99 "
    set theMIDIconstant2 to " 7F "
    set theMIDIconstant3 to " 00"
    repeat with x from 0 to 127

        set hexString to (character (x div 16 + 1) of hexChars) & (character (x mod 16 + 1) of hexChars)

        set hexString to theMIDIconstant1 & hexString & theMIDIconstant2 & hexString & theMIDIconstant3

        make type "midi"
        set theCue to last item of (selected as list)
        set the message type of theCue to sysex
        set the sysex message of theCue to hexString

        set nametypeofcue to "Input channel MUTE for Input "
        set numberofCue to x + 1
        set the q name of theCue to nametypeofcue & numberofCue
        set numbertypeofcue to "IM"
        set the q number of theCue to numbertypeofcue & numberofCue
    end repeat
end tell

Thank you so much for handing me this useful tool!

[Rich Walsh]

I’m sure you didn’t mean to patronise me, but you don’t really know what you are talking about. There are no "hexadecimal MIDI commands”. MIDI is ultimately just a sequence of flashes of an LED inside an optically-decoupled connector. Messages can be one or more bytes, so any multiple of 8 flashes from 8 up – although technically they are 10-bit chunks on an asynchronous serial connection, so it’s more like Morse code than flashes. You might want to visit https://learn.sparkfun.com/tutorials/midi-tutorial/all, and everyone who works with MIDI should have read http://midi.teragonaudio.com/tech/midispec.htm. MIDI may get wrapped up in a TCP header and sent via Ethernet, but it’s still ultimately about current loops, 5-pin connectors and serial bytes.

The message bytes can be expressed (FOR HUMANS) as binary (11000000), decimal (192) or – much more usefully – hexadecimal ($C0). The two-digit hexadecimal is just a way of writing a number between 0 and 255, but takes a very meaningful form in the context of MIDI because numbers above 127 are used to identify a byte as a “status byte” rather than a “data byte”. Hence, any hex starting with a digit above 7 (ie: 8, 9, A, B, C, D, E, F) is a status byte, which is used to indicate the type of message – as opposed to a data byte, which is used to convey the message.

There are therefore 8 types of message: ALL OF WHICH can be expressed (for humans) as binary, decimal or hex. QLab uses decimal for the human interface with most of these messages, but it could equally have used hex to represent the numbers involved. Equally, other manufacturers can choose how they present these underlying numbers to the user; A&H have opted for hex in their documentation – so you have to translate (I’m assuming that the current scene isn’t displayed on the console in hex…?). I haven’t checked, but I remember Yamaha sticking with the same base (decimal) for the all the numbers you see or interact with, until you get into actual SysEx programming (like data dumps).

What’s more, because only the first nibble of a status byte is required to indicate what type of message it is, the second nibble can be used to represent MIDI channel: so $C0 is a Program Change on channel 0 (generally expressed to humans as channel 1), while $CB is a program change on channel 11 (generally expressed to humans as channel 12).

You can read the MIDI specs to discover that "BB 00 00 CB 00” is not and can not be contained within a System Exclusive message. You can type those bytes into a QLab SysEx cue, but what you will send out is:

• $F0… Listen, here comes a SysEx message – look out for the $F7 to say that we’re done
• $BB… Er, hang on, where’s the $F7? Abort: something’s gone wrong, it’s a status byte – we’re actually getting a Control Change message on channel 11 (or 12, if you’re a human), so standby for two data bytes
• $00… First data byte: Control Change 0 = Bank Select MSB
• $00… Second data byte: value 0 – so select bank 0
• $CB… Good, another status byte: Program Change on channel 11 (12); standby for a data byte
• $00… First (only) data byte: value 0 – so select program 0
• $F7… End of SysEx message, but we’ve already aborted so we’ll ignore that!

So although you think you’ve sent "hexadecimal MIDI commands” you haven’t. You’ve sent precisely:

1. Spurious start-of-SysEx status byte
2. One Control Change message
3. One Program Change message
4. Spurious end-of-SysEx status byte 

It would be completely and utterly wrong for Figure 53 to relabel a cue that generates SysEx as anything else. QLab even says that it will be adding an $F0 and an $F7 in the Inspector. They could make a new raw Message Type that does not include the SysEx status bytes, but it would only be useful if you didn’t really understand what was going on and wanted to type MIDI messages in as hex rather than decode them into what they are really doing and add the appropriate cue types.

Hence, why after 12 years my general template script for making MIDI "seems to have no option for creating the hexadecimal strings [you] needed”: you don’t need them; what you need are combined Bank Select and Program Change messages which are included.

It’s up to you if you want to bodge the messages by sending aborted SysEx, but if you want to build something that is guaranteed to work – rather than seems to work OK but one day might not – then I think you should use the correct MIDI commands required by the equipment.

There are two reasons I dislike the A&H documentation:

1. They don’t prefix their hex to indicate that it is hex: conventionally it is prefixed with 0x or $ (or suffixed with H)
2. It is presented in such a way that people on this list keep mistaking all the messages as System Exclusive – which they absolutely can not be if they take MIDI channel…

I’ve attached part of the very first time I encountered MIDI in 1990; I learnt from this document – as opposed to being able to decode the A&H documents because I’ve already fully understood what MIDI is.

Maybe don’t drumroll me next time?

Rich

[Simon VB]

Hi Rich

I DEFINITELY didn't mean to patronise you. I'm truly sorry if it came across that way.
Maybe it's my English? Not a native speaker. The drumroll was a joke, but not a good one, clearly - but definitely not meant as a jab.

I stand corrected. On many fronts, I guess. Thank you for the thorough explanation, I learned a lot.
If I'm correct, I should have written "MIDI messages written as two-digit hexadecimal text strings" whenever I wrote "hexadecimal MIDI commands" (or something like that).

The thing is, the Allen & Heath documentation 'asks' me to use these strings, and that's what I've been doing - using the SysEx version of MIDI cues to send these strings - and it's worked for me this way for several years. Actually, I guess it shows the sturdiness of the MIDI protocol that these SysEx headers have never caused confusion for the console...
So I've never felt the need to look any further and decode the content of the messages in the A&H documentation. All I was looking for was indeed a way to increment part of such a string.

But I would like to use this opportunity to learn!

If I wanted to, as you say, build something that is guaranteed to work, you suggest I should translate the strings in A&H’s documentation to ‘regular’ (or what should I call them) MIDI commands and use QLab’s MIDI Voice Messages. Right?
So to get a scene recall, I should use a Control Change message and a Program Change message in two separate MIDI cues, and fire them consecutively. Should they get a timing offset to make them consecutive and not get mixed up? And how would you go about this when firing many more MIDI commands ‘together’ (or better, after one another)?

I must admit that having to split up all of the commands in the documentation to their respective building blocks, and then building groups to reorder them into the right combinations, seems like a tedious chore… which I can avoid by using those hexadecimal strings - however unorthodox they may be. But if I ever run into an issue with those, I’ll think of your advice and try the sturdier option!

[micpool]

Here's the offending section from the A&H MIDI Protocol

h

It's clearly a very bad description of MIDI control of a mixer,  hence Rich's comment on the quality of the A&H documentation.  I think ever since MIDI wasn't just something transmitted between opto isolated  5 Pin DIN plugs, there has been a bit of a pick and mix attitude to adherence to the full MIDI spec.

However most of the time the MIDI receiver will sort it out into something readable and actionable.

So, QLab has interpreted it's own output as Select Bank 3 (MSB). The second  CC on controller 32 (Hex 20 )is the Bank Change LSB which is nearly always 00. Program Change 5

What may have been missed throughout this thread is what a MIDI bank change message is, particularly as QLab doesn't list bank change in the options for MIDI voice messages.

A Bank change (Well really the significant part of a Bank Change) is simply  Control change  control number  0.

Using 2 messages in a timeline group in QLab , Bank Change control number 0 followed by MIDI program Change shows QLab is actually chasing the bank number on the Program change. which can be seen in the attached video,.

Mic

[Rich Walsh]

Mic, I was actually using the "dLive MIDI TCP Protocol 1.90” document from https://www.allen-heath.com/dlive-home/dlive-documents/:

Even less clear than the SQ? I don’t think there’s any way in which these messages do not adhere to the MIDI specification; they are just so badly presented that you might think they are something special when they are actually just completely standard messages, such as Note On, Control Change, Pitch Bend and NRPNs. Basically, anything that takes MIDI channel must be one of the vanilla MIDI Voice Message types.

At the risk of flogging a bored horse to death, the Scene Recall commands are NOT some kind of special "MIDI messages written as two-digit hexadecimal text strings” – they are a Bank Change message written in hex and a Program Change message written in hex. There is no need for a Cue type that can handle "MIDI messages written as two-digit hexadecimal text strings” because there is already a Cue type that can handle Bank Changes, Program Changes, etc. The documentation isn’t “asking” you to use these strings: it’s saying that the desk responds to MIDI Bank Change and Program Change messages in a totally standard and expected way; it’s just saying it unclearly via the medium of hex… When it does expect you to send SysEx then it says "SysEx Header” and includes the trailing $F7 you need to close the message.

In the screenshot you can see it says “Bank and Program Change message” so it’s not a huge conceptual leap to see the two clearly-separated clusters of hex as two messages, rather than a set of illegal data bytes to include in a SysEx message. If you do use the messages described (QLab calls them all “MIDI Voice Messages”) then incrementing the numerical values becomes trivial.

I personally don’t see that it’s simpler to convert MIDI channel 12 to $B and scene 42 to $2A then type “BB 00 00 CB 2A” into a SysEx Cue than to type those decimal numbers directly into two MIDI Cues of the correct commands, particularly as it LITERALLY tells you in pink what the command types are! Much easier to read the resulting Cues too.

QLab’s logic should play nominally-simultaneous MIDI Cues in the order they appear in the cue list, but it’s not a bad idea to put 10ms between messages: something we were doing with MIDI sequencers like Vision 30 years ago. I also wouldn’t actually be surprised if the Bank Change is only required when, well, changing banks – so once you’re in the right bank a Program Change alone should recall the appropriate scene. That would be normal behaviour – it’s how Lexicons worked.

I’m done with this now.

Rich

[Jaap Schledorn]

If it is of any assistance:
For using an A&H dLive or SQ mixer in conjunction with the "dLive MIDI TCP protocol" or MIDI over USB connection (for SQ) I threw together these two scripts (see attachment). 

The "dLive scene changes" generates the actual MIDI PC and CC messages needed. I suggest using it in a separate cue list named "dLive"  You can specify number of scene changes, MIDI channel, Qlab MIDI patch, interval and the cue label.

The "dLive OSC scene recalls" makes an OSC cue which plays the correct scene change. I use this one on a hotkey to quickly generate an OSC cue in my Main Cue List.

I'm pretty sure there is cleaner way to write these scripts as I'm not an experienced AppleScript user. Furthermore I have used it in my own workflow so the script might throw you some errors.

Op vrijdag 3 februari 2023 om 01:33:41 UTC+1 schreef Rich Walsh:

[Simon VB]

Hi everyone

It's been a while - we just finished the production for which I needed these commands.

Like I'd said before, I've learned a lot from this. Apparently, the use of the hex SysEx cues to recall the dLive's scenes was not as stable or reliable as the last time I used them. Maybe because this was a busier, more hectic show, I'm not sure, but sometimes the commands just didn't have any effect. So the 'translation' or re-interpretation from SysEx to MIDI Voice Messages on the receiving end doesn't seem that sturdy after all.

In any case, I had to change my strategy and use the separate Bank and Program Change messages combined in a group, as you all suggested.

I used Jaap's script for this. Jaap, thank you very much - your script worked flawlessly for me! It's very nice to have a naming option.

If you've made any more scripts for batch creating MIDI commands to control the dLive, you're obviously very welcome to share them :D

Thanks again to everyone, for enlightening me ;)