Game not loading past init_phase_1

[Ben Moeller]

Hey all,

My game boots fine using the virtual hardware (-X flag), but not when connected to actual Fast hardware. When booting with the real hardware using `mpf -b -t -v -V`, the log shows the init_phase_1 event, it shows loading all modes (for debugging, I've removed all modes from my config.yaml, so it just shows the default game and attract modes), and then hangs and does nothing until I ctrl-C to stop the game. It then throws a warning about attract not loading, then posts the Shutdown event and closes as expected.

I'm attaching a pair of log files, one is with virtual hardware (-X flag) that boots correctly, and one is with the physical hardware where I'm having the problem.

Has anyone seen this issue before?

[Anthony van Winkle]

Thank you for the logs! It's interesting, it successfully connects to the FAST hardware but doesn't trigger init_phase_2, thus neither any of the subsequent init phases. The init phases are asynchronous queue events, so each one automatically triggers when the previous one's queue is empty. Clearly something is happening in the init_phase_1 queue that's never completing, so the init cycle gets stuck.

I suspect it's something with initializing the hardware network connection or the ball devices, but tough to say. You could add debug: true to your fast: net: config section, which will output (when used with -v -V) all of the actual serial commands sent to and from the platform, and additional debugging logs. It may also help if you share your hardware and main config files. 

-Anthony

[Ben Moeller]

Hi Anthony,

Thanks for looking into this, here's a log with debug: true in my fast: net: section. Also attaching the config/config.yaml file that contains all the Fast and switch setup, the modes: list is entirely commented out so nothing else should be loading.

Ben

[Anthony van Winkle]

Okay! We're getting somewhere, maybe. I see that your FAST Neuron is on the latest firmware, 02.26, which is good. But also bad because if it was out of date, updating it is an easy way to solve problems :)

The FAST platform has an init_phase_1 handler to do a "soft-reset" of all the connections. There is a debug log when this happens that I don't see in your logs, but it's on the platform level rather than the neuron itself. So you should add debug: true to the root fast: config section as well as nested in the net: section, that would help.

The soft reset sends a command to each port of the FAST board and awaits the response, so if one of them is not responding that would cause the init_phase_1 to never complete, which is aligned with what you're seeing. And that's the only init_phase_1 event I see that's platform-based, so all the others would trigger on virtual mode.

So I think that's where the issue is. What the issue is still don't know. The logs don't show any commands going out to the FAST board to trigger the reset, which means it may be an MPF bug rather than an firmware issue (if MPF was sending a command and not receiving anything, that would cause the same hangup but we would see it in the log).

If the additional debug: true doesn't provide anything helpful, we can add more thorough debug logging to the flows to identify where it gets hung up (and fix it!). Sorry you're stuck in the process, getting the board up and flipping for the first time is a huge milestone and you're so close!

-Anthony

[Ben Moeller]

I'll test that the next time I can get to my game. The other interesting thing here is it was working without issue, the only thing I've changed is working on it virtually. I've been keeping everything in a GitHub repo and committing to it fairly often, but even rolling back to a last known working commit still had the issue. I'm leaning towards it being a hardware issue with that info, but again I'll add in the extra debug: true the next time I can test with the game.

Thanks,

Ben

--
You received this message because you are subscribed to a topic in the Google Groups "MPF Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mpf-users/5ZCA85vbGjM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mpf-users+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/mpf-users/75d2c70f-1fd2-4654-8ce3-16f5ba358712n%40googlegroups.com.

[Ben Moeller]

Not seeing much of a difference in the logs with the updated config.yaml. See attached log and config. 

Thanks,

Ben

[Ben Moeller]

Hi all,

I've fixed it! I tried debugging with another approach, by starting a new machine folder from scratch and following the MPF tutorial until something broke. Turns out, the thing that made it break was my lights section in the config.yaml. I'm surprised that it didn't resolve itself when I rolled back to the old code version, since I don't think I had lights in that one. But when I commented out everything in the lights: section of the config, the game booted properly.

After some more debugging, the root issue was that I had a placeholder start address in there since I don't currently have the expansion board hooked up. To parse the channel string for the FAST platform, MPF takes the channel number and splits it on the hyphens to split out board, port, and channel. Since my string didn't have any dashes, it was throwing an error, but for some reason that error never surfaced into my logs or terminal. Now that I've told MPF that the "lights" are (supposed to be) plugged into one of the EXP ports on the Neuron main board, it's much happier. So I can now boot and play my game without any issue, while still having the lights defined.

Anthony, thanks for your help trying to debug this! Even just knowing where the debug: true configs could go was helpful in working through the issue.

Ben

[Anthony van Winkle]

That's fantastic news Ben, and thanks for the detailed breakdown. With your assessment and config files, I was able to reproduce the issue. I've opened PR https://github.com/missionpinball/mpf/pull/2053 to handle exceptions during startup and exit with a clear message, instead of hanging forever. 

To view this discussion visit https://groups.google.com/d/msgid/mpf-users/ddf63317-0869-4417-8bde-65627b57e3fan%40googlegroups.com.