CobraPin Erratic Switch/Coil Firing

[kaydeeH]

Hey, it's me again. Now that we have everything up and running, and all the switches working perfectly, it was finally time to test out the coils. Some of them do exactly what we would expect. Others get pretty nuts.

This is a retheme from a Paragon, so there is a cluster of 3 pops at the top and one at the bottom. We've added a multiball trough and an autoplunger. 

So here's what's going wrong so far:

• Slingshots won't autofire (despite a correct switch hit detected), but they will fire as expected in the coil test.
• Pops are crazy.
  • Of the top three pops: The right one won't *self* autofire at all. The center one causes only the right one to fire. The left causes the right and center to fire. I get that the firing each other issue could be switch gaps too small but I don't think it's that because if the right one won't even autofire itself despite a proper intentional switch hit detection, the others shouldn't be able to vibrate it to do so either, though admittedly I do see the switches flicker on the screen.
  • The lower playfield pop never auto fires... unless you drain a ball into the trough, then it fires 2-4 times. Its switch is detected fine when you hit the skirt.
  • So then I go into coil tests to see what happens, and it's a little different. The right pop test causes the right and left to fire. The left pop test causes only the right to fire. The center pop test causes itself plus the right to fire. The lower pop test causes itself to fire just fine.
  • No other coil tests cause the pops to fire.
• When you drain a ball, the intent is that it drains into the standard outhole, then kicks over into the trough which is the true drain (the outhole is basically just a saucer that hops the balls over into the trough). That said...
  • It never detects a drain. This may be some config issues on our part, not sure yet. But...
  • When the outhole kicks, the trough switches seem to cause the lower pop to fire (as mentioned before). I do see the pop switch flicker when the outhole kicks, but again, I don't think this is tight switch gaps in the pop vibrating to close because...
  • As mentioned above, when I use the diag menu to test the outhole coil, it does NOT cause the lower pop to autofire.

Finally, when I disable the coils by disconnecting J13, every single switch works perfectly... no switch causes other switches to flicker, so it's not crosstalk or shorts between the switches. We are using a switch matrix, but it works perfectly with coils disabled, so I don't think there's any issues with high/low settings or wiring.

So that's where we are. It's not intermittent, it's not variable. It 100% of the time happens just as I've described. We're pretty certain we have all our autofires set up properly, but feel free to check them out in our source, link below. Any thoughts on what we should be looking for? If it helps, our CobraPin is a 1.02 revision and we haven't updated it in any way.

https://github.com/kaydeeH/OutpostKodelia-DoH

Thanks,
Kaydee & Olivia / Outpost Kodelia

[kaydeeH]

Oh, and as you'll see in the config, all coils we expect to see autofire are 1-X-X addresses so they'll link up with switches in the matrix. :)

[cobra18t]

Did you completely rewire the switch matrix? Your switch numbering does not match the schematic I have for Paragon at all. The schematic I have shows the slings and pops on the same strobe line, but your switches span 4 columns and 3 rows.

You may have already handled this, but I have to bring it up. Paragon used an Active High switch matrix and you got the Standard config--which has an active low matrix. This is not that big of a deal but you need to hook the Strobes on Paragon up to SWITCH_MATRIX_IN of CobraPin and the Inputs on Paragon up to SWITCH_MATRIX_OUT of CobraPin. This transposes the switch matrix numbering and biases the matrix diodes in the correct direction. The diodes in an active low matrix must always "point" to the SWITCH_MATRIX_OUT lines on CobraPin.

https://pinballmakers.com/wiki/index.php?title=CobraPin#Matrix_Numbering

This is what I read from your switch numbering:

And this is what I have for Paragon:

[cobra18t]

Also, there is nothing connected to the center GND pin on J4 and J5 of the CobraPin correct? The switch matrix does not need a GND connection. A picture of those two connectors and wires going to them might help me understand.

[kaydeeH]

So a couple clarifications. Our goal wasn't to reproduce a Paragon with a new skin on it. We're are changing a lot of functionality and adding a bunch of switches, coils, and gameplay complexity. Also, we initially planned to build this on a P3Roc platform, so we had already stripped out the matrix wiring. When we switched gears and decided to go with the CobraPin, it allowed us to reconstruct the matrix as we liked based on everything we were adding/removing for optimal wire routing. So yeah, it doesn't look anything like the Paragon matrix and it's been fully rebuilt. As to your questions on wiring, we used new color coded wiring, so all the striped wires in the pic below go into J5, and the diodes all have their striped side pointing to the striped wires, just as you described for the active low matrix. All the wires in J4 are solid colors and are, obviously, attached to the opposite end of the switch from the striped side of the diode. You can also see that there's nothing connected to the ground pin. Again, I want to stress that when we test the switch matrix switches, they all work exactly as we would expect. We can fire any switch, or groups of switches, in any combination, and the correct switches indicate activation in MPF. Likewise, all direct switches are working perfectly.

The problem seems to exist primarily on the coils side of things, as described earlier. Even then, most of the coils work exactly as expected... it's just a problem with the autofire configured coils misbehaving. And for what it's worth, we have checked that the coils are wired correctly with diodes pointing towards the HVABC header (J13). There, too, we used color coded wiring to minimize accidents, so all HVABC lines are the striped yellow wires, and we have repeatedly checked that the diodes point to them. We also ensured that all coils on banks A/B/C use the matching HVA/B/C wire. 

Anyways, here's that pic. I'm also including images of our matrix, direct switch, and coil maps.

[kaydeeH]

Fresh bit of info... first off, we realized we had mis-tagged the pin numbers for the left and right pop coils, so that explains a little bit of the issue. That's fixed. With that done, we disabled all autofires in MPF and found that they were still autofiring. So we restarted EVERYTHING, full power cycle. That successfully stopped all the autofiring. At that point, we did a coil test and all the madness went away, every coil fired itself and only itself. Then we did a little game play. Sure, nothing autofired, but things like trough switches making the lower pop coil fire also went away. So we added autofires back in the MPF config and did another full power cycle. Trough switches once again seemed to be making the lower pop coil autofire. Just to ensure it wasn't vibration activation, we slid some cardboard into the pop leaf switch and confirmed it had nullified the switch. Even then, the trough switches are making the lower pop fire. Just to see what happened next, I cardboarded up the left/right upper pop switches, but left the center open. I then activated the center switch and the upper left coil fired. Confirmed in MPF that it shows only s_thumper_center activating and the other switches show no activity, not even a flicker. Also confirmed through the coil test that c_center_thumper fires the center coil and only the center coil. Finally, checked my autofire config, and that switch/coil are paired up properly. So long story short, it still seems to me that the autofire config just isn't initializing properly when the game starts.

[Thomas Fulenwider]

Thanks for all the info. Can you put "debug: true" in your opp section, enable autofire, and post a log?

--
You received this message because you are subscribed to a topic in the Google Groups "MPF Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mpf-users/Qyyi30m0MEQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mpf-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mpf-users/58b4cf90-270e-43bd-9601-12961984d225n%40googlegroups.com.

[kaydeeH]

Sure thing. Here it is. Just so you know the physical activities the log should be reflecting: I started the game, it ejected the ball into the shooter lane, I picked up the ball, hit the center pop a few times with my finger (which fired the upper left pop each time), then manually drained the ball (which caused the lower pop to fire a few times) and quit the game. Thanks for taking a look. 

[Kaydee Helm]

Interesting note... with the game shut down, but the system still running, autofire stays active. Not sure if that's intended. If I manually eject and drain a ball, activity on the trough switches is still causing the lower pop to fire, even with cardboard blocking its switch.

You received this message because you are subscribed to the Google Groups "MPF Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mpf-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mpf-users/CABMT2j2E5USU0Yx9tzqVYkdwj6V5YB0AiHUu%2BCU77W-%3DK4Evpg%40mail.gmail.com.

[kaydeeH]

I'm suddenly wondering if I wired the trough switches up correctly. I modeled it off what I found online, but I'm not certain it's right, so maybe I got something wrong there. Here's why I wonder... trough switch 3 and the lower pop are on the same row, (yellow wire in the pics), though physically separated by other switches. If I manually articulate trough switch 3, the pop fires every time. I started the game back up to see what was registering, and I don't see the pop switch (or any other switch in the same row/column) activating, so I don't know, it seems right, but what I see in the log I sent you seems to differ with that. Anyways, I'm providing a pic of both the trough switches and the pop switch. Again, the striped wires are all columns and go to J5, solid wires are rows and go to J4. Let me know if anything looks wrong.

[kaydeeH]

Just ran another test where the only switch I activated (now with the cardboard removed) was the lower bumper switch, which caused nothing to happen. You can clearly see the switch hits in the log, and I don't see any other switch hits registering, but no autofire. Log for that is attached.

[kaydeeH]

We just found that two of our standing targets are actually making the slingshots autofire. s_rebound_500pt makes the right sling fire, and s_center_target makes the left sling fire. In both cases, they're in the same row with the corresponding sling's switch. I would think I was having classic switch matrix diode issues, EXCEPT, when I look at the logs in MPF, I don't ever see anything but the switch I actually hit. There's no indication in the logs or on screen that additional switches are being triggered by diode / wiring fails... and yet the wrong switches are triggering the coils to fire. I'm stumped.

[cobra18t]

I will review the commands that MPF is sending to OPP a little later to see if OPP is getting configured properly. Thanks for the logs.

[cobra18t]

Jan, is this a malformed command to OPP? 

0x20 0x17 0x51 0x02 0xbc 0xff is the set of values that need to be sent but it looks like 0x51 is getting converted to ASCII ("Q") so the result is b' \x17Q\x02\xbc\xff'

This is happening on the third byte of all the hardware rules which defines the input for the autofire device. Maybe this is why Kaydee has surprising but repeatable results with autofire devices.

Here is a relevant section of the log. The MPF version is v0.55.0:

2021-09-29 13:09:51,542 : DEBUG : OPP : Setting HW Rule. Driver: 1-0-2

2021-09-29 13:09:51,542 : DEBUG : OPPSolenoid 32 on 1 : Writing individual config:  0x20 0x14 0x02 0x02 0x14 0x20 0x30 0xff on 1

2021-09-29 13:09:51,542 : DEBUG : OPP : /dev/ttyACM1 sending: b' \x14\x02\x02\x14 0\xff' ( 0x20 0x14 0x02 0x02 0x14 0x20 0x30 0xff)

2021-09-29 13:09:51,542 : DEBUG : OPP : Mapping input 81 and coil 2 on 1

2021-09-29 13:09:51,542 : DEBUG : OPP : /dev/ttyACM1 sending: b' \x17Q\x02\xbc\xff' ( 0x20 0x17 0x51 0x02 0xbc 0xff)

I confirmed in one of my old logs on MPF v0.55.0-dev.55 that this ASCII conversion does not take place. One thing to note is that I am using direct switches and Kaydee is using a matrix switch so I am not sure if the issue lies in using the matrix switch or the MPF version:

2021-05-26 22:11:04,056 : DEBUG : OPP : Setting HW Rule. Driver: 0-0-8

2021-05-26 22:11:04,056 : DEBUG : OPPSolenoid 32 on 0 : Writing individual config:  0x20 0x14 0x08 0x24 0x32 0x00 0xa2 0xff on 0

2021-05-26 22:11:04,056 : DEBUG : OPP : /dev/ttyACM1 sending: b' \x14\x08$2\x00\xa2\xff' ( 0x20 0x14 0x08 0x24 0x32 0x00 0xa2 0xff)

2021-05-26 22:11:04,056 : DEBUG : OPP : Mapping input 27 and coil 8 on 0

2021-05-26 22:11:04,057 : DEBUG : OPP : /dev/ttyACM1 sending: b' \x17\x1b\x08S\xff' ( 0x20 0x17 0x1b 0x08 0x53 0xff)

[kaydeeH]

Fingers crossed you're on to something here. Thank you for the support! :D

[cobra18t]

The ASCII thing isn't it. That appears to be a binary-to-string conversion side effect that only affects the log. It converts anything in the valid character range to ASCII. It happens all over the commands and works in all other cases.

The MPF hardware rule commands look correct. This might be an OPP issue centered around using matrix switches for autofire devices. We have been recommending that people use direct switches for autofire devices for other reasons, so maybe that is why we haven't seen this before.

I was able to replicate the issue on my hardware. I created a pop bumper autofire device on switch 1-0-81, but it was only triggered by 1-0-41 just like you reported with your pop and trough. I will work with Hugh to investigate.

[cobra18t]

Ooh, maybe I spoke too soon. I don't think matrix inputs for OPP were ever implemented in MPF!

From the OPP interface document:

Each solenoid is configured using three bytes. The first byte is the type of solenoid which is a bit field.

(0x01 - USE_SWITCH, 0x02 - AUTO_CLR, 0x04 - ON_OFF_SOL, 0x08 - DLY_KICK_SOL, 0x10 –

USE_MATRIX_INP, and 0x20 - CAN_CANCEL). If the solenoid is configured to use the switch

input, when the switch input is closed, the solenoid fires. If the solenoid is configured to auto clear, the

host can kick the solenoid using the kick which is automatically cleared, and the solenoid can be

kicked another time without sending a command to clear the kick. If the solenoid is configured as an

ON/OFF solenoid, it will be driven at 100% when the input is active (used for dual wound flippers). If

the solenoid is configured as a delay kick solenoid, the third byte determines the delay after the switch

is active before kicking the solenoid. If the solenoid is configured to use a matrix input to kick the

solenoid, the the third byte is used to indicate the 8x8 switch matrix input on this board that activates

the solenoid. If the solenoid is configured as can cancel, the initial kick can be canceled to provide a

shorter pulse to enable things like flipper tap passing.

A search revealed that USE_MATRIX_INP is defined, but never used in MPF. Furthermore, the Set Solenoid Input command (0x17) that is used in your log is only stated to accept a value from 0-31 but we are sending it 81:

7.23 Set Solenoid Input, 0x17

CardAddr + 0x17 + InputIndex + SolIndex (1 byte) + CRC8

Change a solenoid to be triggered or not triggered by an input. The input index contains the index of

the input [0-31]. The solenoid index contains the index of the solenoid [0 – 15]. If the msb of the

SolIndex is set (0x80), the solenoid will stop using the input switch.

[kaydeeH]

While it's far from ideal, in our case we can remove the key switches from the matrix and rewire them up to J3, but if someone was trying to do a direct port of an older game into MPF, those autofire coil switches are always on the matrix, so they would be in a real bind. Before we go through the hassle of rewiring, though, I'll probably hold off a bit to see if Jan chimes in on it. If it truly wasn't implemented in MPF, I'm curious to know what the timeframe would be for getting it implemented. In other aspects of my life, I would normally offer to jump in and help code it myself, but I'm a bit too amateur - especially with Python - to really have a shot at doing that successfully when it comes to something as involved as this. :(

[Thomas Fulenwider]

I am still working with Hugh to see if there is something on the OPP side. If you can wait a few days, I would. This is likely a bug somewhere rather than full-blown missing features.

To view this discussion on the web visit https://groups.google.com/d/msgid/mpf-users/69c29033-f696-4660-bb17-95fda09487a8n%40googlegroups.com.

[kaydeeH]

Thank you so much! We'll just temporarily make them event driven instead of autofire so we can proceed with some basic testing. :)

[cobra18t]

Hugh found a great consistency between all the misconfigured autofire switches. They are mirrored across the matrix columns (see below). 

Not sure why this is so yet, but this may indicate a work around... In your autofire config, use the mirrored switch. Does that work?

For example:

  thumper_bottom:

    coil: c_bottom_thumper

    switch: s_trough_3  

[kaydeeH]

Well, it's official... our switch matrix comes from the evil mirror universe. Inverting the switch maps as suggested... worked. We had to add a fake switch in position 1-0-72 for the upper right pop and, even though there isn't a physical switch in that position, the upper right pop is firing 'as expected'. So for the time being we have working (evil mirror universe variant) autofire coils. Happy to try out patches or other diagnostic stuff, as needed. Thank you both! If nothing else, I'm just glad we weren't totally losing it over here, despite what my now steadfast belief in the mirror universe may say about me. ;)

[cobra18t]

I blame Dolly Parton for the evil mirror universe.

The original switch matrix in OPP was based on a Dolly Parton machine and the column numbering was reversed from what is now standard.

I submitted a pull request to MPF with an edit to opp.py to unravel the mirror universe. I verified it on my CobraPin board: https://github.com/missionpinball/mpf/pull/1600

[kaydeeH]

"I blame Dolly Parton for the evil mirror universe." Now THAT is a sentence I didn't think I'd ever hear, but it was my first laugh of the day, so I'm all for it! Glad to hear that you pinned it down and fixed it. Nice work! I'll watch the PR to see when it goes through and update accordingly. Meantime, we've got our workaround. Thanks again for the prompt support!

[jabdoa]

Thanks for the PR. Will review it later.

Jan

[cobra18t]

I confirmed this switch matrix autofire issue is resolved in dev.16. Upgrade to the latest dev version and you should be good.

[kaydeeH]

Awesome, thank you! Is there a PIP command to pull dev?

[cobra18t]

https://docs.missionpinball.org/en/dev/install/linux/index.html#keeping-mpf-up-to-date 

The --pre is the key to getting dev...

pip3 install mpf[all] mpf-mc --pre --upgrade 

[kaydeeH]

Thank you!