Eos-Nomad with Qlab Instability

538 views
Skip to first unread message

Jkar Fernández

unread,
Jan 25, 2017, 2:36:59 PM1/25/17
to QLab

Mike Post

unread,
Jan 25, 2017, 4:25:57 PM1/25/17
to ql...@googlegroups.com
This is exactly what I ran into on our last show and in my research and asking around the conclusion is there seem’s to be a basic unreliability in the Mac receiving OSC messages.  Oddly enough, using QLab to trigger EOS seems fine, it’s just the other way around that doesn’t work.  I suspect something low level in the Mac OS that’s randomly dropping the OSC message.

For us, I’m simply not allowing our Ion to trigger QLab via OSC.  I will set up a MIDI link if necessary, or require that QLab triggers the Ion.
--
--
Change your preferences or unsubscribe here:
http://groups.google.com/group/qlab
 
Follow Figure 53 on Twitter: http://twitter.com/Figure53
---
You received this message because you are subscribed to the Google Groups "QLab" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qlab+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/qlab/ff00cc17-bffc-4305-94a4-a0fcae2948d8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Chris Ashworth

unread,
Jan 26, 2017, 10:07:42 AM1/26/17
to ql...@googlegroups.com
Hm, this is… possibly true in the sense that OSC sent over UDP are not running over a transport that guarantees delivery, but I would caution against the idea that there is “basic unreliability” in the Mac receiving OSC messages. To my knowledge there is no evidence that there is a bug in the Mac OS network stack or similar condition that would introduce unreliability in the way described below. 

I am having trouble following the original discussion on the ETC forum, but it appears there are plenty of other places to check before ascribing it to “Macs can’t accept UDP messages reliably”…

Some questions for Jkar:

 - Is this a closed network dedicated to the show? 
 - How many machines are on the network?
 - Does the network have a dedicated hard-wired router or are you using wifi?
 - Do you have a way of confirming when the messages leave the EOS?

Best,
C

Mike Post

unread,
Jan 26, 2017, 11:32:44 AM1/26/17
to ql...@googlegroups.com
Hi Chris,

Thanks for pursuing this.  It’s not going to be an easy one to figure out.  I stand corrected in my comment about the Mac receiving the messages.  At this time it seems to receive them, but I’m not sure what happens after that.

I was hoping to find a moment to diagnose it better but the demands of the next two shows pre-empted tinkering.  My guest designers for one of those shows happened to be a team that have some experience with this and said trying to trigger QLab via OSC is not reliable but trusted QLab OSC triggering other devices and MIDI in both directions.

Most of what I have on this is anecdotal at this point, but maybe I can get back to some real troubleshooting soon.  For the moment, I have to do what I can to ensure the reliability of the system so I just don’t allow that mechanism. I’ll repost the original thread I started here to bring the discussion back into focus.  I don’t know what Jkar has set up, but for me I can answer your questions:

 - Is this a closed network dedicated to the show? 
 - How many machines are on the network?

Yes - the only devices on the net were the Ion and the Mac. 

 - Does the network have a dedicated hard-wired router or are you using wifi?

Connection was Cat-6 through an Apple router.  We do keep Wifi on the Ion with a dedicated access point, but that was not used for OSC.  Wifi on the Mac is turned off for shows.

 - Do you have a way of confirming when the messages leave the EOS?

See Richard’s description of what he found.  ETC provides a utility that can log the Mac side and Richard’s example shows the messages arriving as expected.  In my setup, I didn’t have that utility installed and had about the same frequency of problems, so I ruled out the utility as the point of failure.

I’m using the word ‘unreliable’ for this because there was no discernible pattern to the failures.  It wasn’t a very busy show and while there were some tight visual cues, there weren't any complex sequences.  Cues that would fire just fine in one show would fail in the next.  We couldn’t get a grasp on anything that helped with troubleshooting. 

Take care!

Original thread:

Thanks Richard,

Actually, that gives me exactly what I need in the short term.  I don’t have there wherewithal to really diagnose this issue, but it does give me a means to test the overall reliability of the OSC link triggering QLab so I can decide what I’ll allow in our space.  I still feel confident from personal experience QLab triggering other devices via MIDI is quite reliable - been doing that for years.  But if I know there’s something intermittent in an OSC message successfully triggering QLab and can prove it, I can keep that scheme from messing up my shows.  Knowing about OSCRouter gives me that tool and I think I own just enough computer to mock it up without compromising my current show.  Thanks for that.

I don’t have much experience with QLab using OSC to trigger other devices but I know people who do and have never heard of any intermittent problems so I tend to think that’s a reliable scheme.  QLab seems to win out for reliability in most cases ;-)

The scenario you describe doesn’t point a finger at anything specific within the Mac for me.  We know it's received by the Mac, but once there I feel like the problem could be either in QLab or a failure in something low level between QLab and whatever the Mac does to pass OSC messages along.  I leave it to wiser heads than mine to figure out that puzzle...
On Dec 10, 2016, at 5:53 AM, richardm <rmo...@nationaltheatre.org.uk> wrote:

Hi Mike

I can't offer any great insight but i saw the exact same thing happen last night..This was an ETC Ti triggering QLab via OSC using numbered cues. I'd say of a couple of hundred cues fired, 2 did not reach Qlab.

I didnt have chance to really dig into the system set up but the OSC messages were passing through ETCs OSCRouter software on the QLab machine.  The OSCRouter log showed the cue arriving from the lighting console and being forwarded to Qlab. The console wasn't the problem i dont think (and the cues fired correctly after the show) so the packet seemed to get lost on a local socket .

This was an old version of QLab too - 3.1.2 i believe

Not much to go on i know but hope that's some use to you..I'm off the show now or i'd do some more digging..
 
richard 


On Saturday, 10 December 2016 02:50:04 UTC, Mike P wrote:
Thanks Chris but I think that tests the wrong case. It's not QLab sending OSC. It's cues in QLab that are set to trigger off an OSC message from the Ion. I suspect the Ion is the culprit  but not sure what to look for. Is there an OSC sniffer for a Mac like I have for MIDI?

Sent from my iPhone

On Dec 9, 2016, at 10:57 AM, Chris Ashworth <ch...@figure53.com> wrote:

Hi Mike,

As a debugging option, it might help to turn on some logging to look for confirmation that QLab believed the cues were triggered, and then compare that to what cues didn’t actually happen.

I don’t think you’re using v4 yet, but if you are, you can do that in the new status window by going to the logs tab and enabling logs for cue triggers.

Alternately, you can set log level 2 in the application preferences and it will log when cues start in the console logs.

If you see a cue show up in those logs that doesn’t fire, that would suggest to me that the message is getting lost somewhere in route, somehow….

-C

On December 8, 2016 at 1:43:45 PM, Mike Post (mdpost...@gmail.com) wrote:

Hi again all, 

I’m having an intermittent problem with OSC commands from an ETC Ion console triggering specific cues by number on QLab. It seems to work just fine for the most part but once in a while, a GO from the Ion will not trigger the cue on QLab. There’s much speculation about what may be causing this, maybe a specific cue, maybe a wiring problem… None of this is proving conclusive in my mind, so I wanted to ask here if anyone has run into this and has any idea what might cause a failed OSC GO. 

I’ve looked at the console log but couldn’t see anything that jumped out to me as a problem at the time of the execution. Not that I’m an expert, but I would at least expect to see some sort of network error…? Yes? 

The Ion is at the latest rev of the software and QLab is running at the latest version of 3 before V4 came out. Any comments would be appreciated. 
------------------------------------------------------------------


-- 
-- 
Change your preferences or unsubscribe here: 
http://groups.google.com/group/qlab
 
Follow Figure 53 on Twitter: http://twitter.com/Figure53
--- 
You received this message because you are subscribed to the Google Groups "QLab" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qlab+uns...@googlegroups.com.

Jkar Fernández

unread,
Jan 26, 2017, 4:43:22 PM1/26/17
to QLab
Hi Chris:

The network consists of a PC (windows 8.1) exclusively with Eos software, connected with cat-5 UTP cable (crossover) directly to the ethernet port of a mac (s.o Grand Captain).



I shoot the cue 2 of EOS to tell Qlab (/ cue / 1 / stop), for the audio ..

.. but qlab continues to play the audio .... the stop order is lost ...


Thanks for your time.

Alexander (Mailing List) Taylor

unread,
Jan 26, 2017, 4:49:38 PM1/26/17
to ql...@googlegroups.com
I don't have anything to add about the instability, but I am wondering why you're using a crossover cable.  All the Macs for quite a while (and most PCs I run into) have auto-sensing ports, so don't need crossover cables.  I've found having them around will just cause unexpected problems the vast majority of the time.

Also I can't read those screenshots at all, or I'd compare it to my setup.

Alexander

-- 
-- 
Change your preferences or unsubscribe here: 
http://groups.google.com/group/qlab
 
Follow Figure 53 on Twitter: http://twitter.com/Figure53
--- 
You received this message because you are subscribed to the Google Groups "QLab" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qlab+uns...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

The Right-To-Know Law provides that most e-mail communications to or from School District employees regarding the business of the School District are government records available to the public upon request. Therefore, this e-mail communication may be subject to public disclosure.

Jkar Fernández

unread,
Jan 26, 2017, 4:51:50 PM1/26/17
to QLab

Jkar Fernández

unread,
Jan 26, 2017, 4:59:37 PM1/26/17
to QLab




El miércoles, 25 de enero de 2017, 20:36:59 (UTC+1), Jkar Fernández escribió:

Jkar Fernández

unread,
Jan 26, 2017, 5:01:28 PM1/26/17
to QLab



El miércoles, 25 de enero de 2017, 20:36:59 (UTC+1), Jkar Fernández escribió:

Sam Kusnetz

unread,
Jan 26, 2017, 5:06:12 PM1/26/17
to ql...@googlegroups.com
Hello all

Thanks to those last few screenshots, I think I see what’s going on.

It appears you’re sending the message “/cue/1/stop” as a string, not as an OSC message. I’m gathering this from the log message in Nomad saying “send string:"

QLab accepts UDP strings on a separate port from formal OSC messages.

If you set the outgoing port to 53535 (and not 53000), does that work?

Sam
—
Sam Kusnetz | Figure 53
--

--
Change your preferences or unsubscribe here:
http://groups.google.com/group/qlab
 
Follow Figure 53 on Twitter: http://twitter.com/Figure53
---
You received this message because you are subscribed to the Google Groups "QLab" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qlab+uns...@googlegroups.com.

Jkar Fernández

unread,
Jan 30, 2017, 8:26:18 AM1/30/17
to QLab
Hi.
Thank you Alexander. Today I learned something new. You're right. I assumed that two computers were communicating with UTP wire crossover. And as you say, it is not necessary at the moment. Computers use (Auto-MDIX) or auto-sensing Ethernet port. I have tried it and better than crossover.

Hi.
Thank you for your interest Sam. I changed the port you say (53535) and yes, there it gets ... but in 53000 it also receives ...

News; UTP cable cat-5, non-crossover. Port 53000 or 53535. Trigger cues 70 times in a row and the seventy comes in working correctly ... but at the moment stops working and returns to behave in an unstable.

I have tried with different computers to discard network card failures, with the same settings ... the problem persists.

Chris Ashworth

unread,
Jan 30, 2017, 9:40:40 AM1/30/17
to Jkar Fernández, ql...@googlegroups.com
Hi Jkar,

What Sam was saying is that from the screenshot it does not look like you are sending OSC messages.

Port 53000 will only respond to OSC messages, so sending other things there will do nothing.

Best,
Chris

Jkar Fernández

unread,
Jan 30, 2017, 10:09:53 AM1/30/17
to QLab, jka...@gmail.com
Hi.
EOS diagnostics option, activating Outgoing OSC (on) .... shows the following output:
[OSC Packet] / cue / 1 / stop
Send String: / cue / 1 / stop
Is not this an OSC message?

Same configuration, always, sending same message, and sometimes works if and not others. If that were wrong, it would never work. Do not?

Chris Ashworth

unread,
Jan 30, 2017, 10:18:39 AM1/30/17
to Jkar Fernández, ql...@googlegroups.com
Hi Jkar,

I do not know the EOS software so it is hard for me to help with the screenshots, but the “OSC Packet” appears to (maybe) be an OSC message.

The one that says “send string” is probably not an OSC message — OSC messages are not strings. (Unfortunately! I wish they were.)

-C

John Evans

unread,
Jan 30, 2017, 6:56:09 PM1/30/17
to QLab, jka...@gmail.com
Hi,

I used the instructions on this F53 wiki page many times to control Eos from QLab or vice versa without using the OSC message features that were introduced in (as I recall) Eos 2.3. I used to do this successfully quite a lot before Eos fully supported OSC, so it's definitely possible to send arbitrary OSC messages from Eos as text, but as I recall there was some messing around involved - I think Sam has the correct solution above when he mentions using port 53535. Any strings written into the external links column aren't being sent as pure OSC. Eos sends OSC messages along the lines of "eos/out/event/cue/1/2/fire" - see this page for details: https://community.etcconnect.com/control_consoles/f/eos-family-show-control-support-midi-smpte-osc-rtc-etc/17390/a-guide-to-triggering-qlab-cues-from-eos-over-osc

Thanks,
John

Jkar Fernández

unread,
Jan 31, 2017, 2:02:17 AM1/31/17
to QLab, jka...@gmail.com
Hi.
I think you have found the solution to the problem ... as soon as I have time I put it into practice.
Thank you very much.

Jkar Fernández

unread,
Jan 31, 2017, 11:18:35 AM1/31/17
to QLab, jka...@gmail.com

El martes, 31 de enero de 2017, 0:56:09 (UTC+1), John Evans escribió:

Rich Walsh

unread,
Jan 31, 2017, 11:40:38 AM1/31/17
to ql...@googlegroups.com
I’m not an expert on these settings – and I haven’t been able to follow this thread completely – but from this snippet of one of your screenshots I’m not 100% convinced you have followed the instructions on the wiki: shouldn’t String TX Port be 53535?

Also, if the messages are sometimes working and sometimes not working have you tested the network for packet loss? Try pinging the QLab machine from the EOS machine – it looks like a Windows machine? You should also stick Wireshark on your QLab machine and check that all the messages from EOS are arriving at the NIC: then you’ll know if it’s the network or QLab that’s not working…

Rich

--
Contact support anytime: sup...@figure53.com

Follow Figure 53 on Twitter: http://twitter.com/Figure53
---
You received this message because you are subscribed to the Google Groups "QLab" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qlab+uns...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Jkar Fernández

unread,
Jan 31, 2017, 1:54:59 PM1/31/17
to QLab

 Hi.

Thank you Rich.

... uuhmm! The problem may be there. .. sometimes, I connect and it goes well ... I turn off the system, turn it on again and the problem arises again.


Richard comments in the ETC forum ....“ It is always best to have a switch in the middle.

Many network cards want to see the other end running before they can properly link up. If they don't start up in the right order, you may end up with no link, an intermittent connection, or a very slow (half-duplex 10 Mbps) connection.

In some cases, both ends want the other to be turned on first - which is of course, impossible to do.

If you have a switch in the middle, it can be turned on before either machine (or application) is started, avoiding this problem. “


Next week I will try with swicht the network, I suspect that maybe the solution is there. I will comment.

Jkar Fernández

unread,
Feb 1, 2017, 10:50:59 AM2/1/17
to QLab

Hi.

First of all I want to thank you for all your help.
Today I tried with a low performance swicht (10 / 100Mbps) and goes perfect. Richard, you were absolutely right. Thank you. The problem is solved. These were not erroneous configurations, but the direct connection between computers gave the failure. I followed your advice, and perfect.

So the title of the post (Eos-Nomad with Qlab Instability) does not seem very accurate now, it is confusing ... since both systems are perfectly compatible.

I want to thank you all for your help. For my part, the matter is settled.


Richard (ETC London)
It is always best to have a switch in the middle.

Many network cards want to see the other end running before they can properly link up. If they don't start up in the right order, you may end up with no link, an intermittent connection, or a very slow (half-duplex 10 Mbps) connection.

In some cases, both ends want the other to be turned on first - which is of course, impossible to do.

If you have a switch in the middle, it can be turned on before either machine (or application) is started, avoiding this problem.

Chris Ashworth

unread,
Feb 1, 2017, 11:57:49 AM2/1/17
to Jkar Fernández, ql...@googlegroups.com
Hooray, I’m glad it’s working! Thanks for the update!



On February 1, 2017 at 10:51:01 AM, Jkar Fernández (jka...@gmail.com) wrote:

Hi.

First of all I want to thank you for all your help.


Today I tried with a low performance swicht (10 / 100Mbps) and goes perfect. Richard, you were absolutely right. Thank you. The problem is solved. These were not erroneous configurations, but the direct connection between computers gave the failure. I followed your advice, and perfect.

Reply all
Reply to author
Forward
0 new messages