Charmhigh chmt36VA conversion random Gcodedriver timeout on M114

[Sam Vanheule]

Hi everyone, not sure if I have to post this here or in the Desktop pick and place Group.

I'm far into converting and configuring my charmhigh machine to openPnP. I did already overwon many challenges and have my machine almost running flawlessly on a basic board.

But for some reason my serial connection gives me problems at random intervals. It's always the same error that pops up. It seems to freeze after a M114 command. It appears to be random, sometimes when picking a part, sometimes when issuing a move,...

Anyone has any idea?

I'm using windows with latest stable OpenPnP, machine.xml attached.

Sam

[mark maker]

Please send a log of a session when this happened. OpenPnP keeps the last 100 logs around.

https://github.com/openpnp/openpnp/wiki/FAQ#where-are-configuration-and-log-files-located

--
You received this message because you are subscribed to the Google Groups "OpenPnP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openpnp+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/8661aeae-841f-4211-9558-ed9f3ae94b0an%40googlegroups.com.

[Wayne Black]

I'm having the same issue. It was minor annoyance every so often when jogging the head for calibration the machine wouldn't respond and M114 timeout messagebox would appear. After several seconds the machine typically responds again with all settings and coordinates intact. I just now suffered a major machine crash. I was performing the Primary Fiducial Calibration. When jogging to the 2nd fiducial I got the M114 messagebox and I waited for the machine to respond as typical. When I continued jogging (10mm along X) to the next fiducial the machine took off to the Charmhigh 'Too Far' position and exceeded the xy soft limits crashing into the frames. I have been weary not to jog to quickly as to not provoke the M114 exception, but it doesn't appear to be related. I had the M114 message appear after sitting idle for several minutes with no user input or job running when I attempted to jog the head. I tested this by running the machine at 25% speed and repeated/rapid jogging at 10mm before the movements had completed and there is no problem. The problem is intermittent, but relatively frequent.

I wish I had more than just anecdotal info. Unfortunately, I dont think I had logging set correctly at time of the crash. But I have attached what I have. I also attempted to 'Submit Diagnostics', but this also threw an exception, please see attached. Lastly I went to the Openpnp wiki to see about enabling logging, but the wiki references logging tabs that my version of OpenPNP doesn't have.

The machine is a CHMT 36 SN#1303, stock controller board w the FW;

ì Smoothie Running @168MHz
Build version: chmt-abda6b99, Build date: Feb 15 2021 08:56:08, MCU: STM32F4, System Clock: 168MHz
7 axis
NETWORK is disabled
NOTE: One extruder configured and enabled
NOTE: 1 extruders enabled out of 1
Watchdog enabled for 10.000000 seconds

[Wayne Black]

Sorry I was mistaken about not having the Log Tab, I do. I thought the log tab was in the main title bar. Im using OpenpnpTest version on Win11. I set the log to debug and jogged the machine for a couple minutes. Everything respoding just fine and then the machine/app hangs for a few seconds, then the M114 Exception. Attached is the Log...

2022-01-06 19:18:36.223 MessageBoxes DEBUG: Error: java.lang.Exception: GcodeDriver timeout waiting for response to M114
2022-01-06 19:18:56.278 AbstractHeadMountable DEBUG: N1.moveTo((319.373000, 258.170000, -0.150000, 0.000000 mm), 0.5)

Wayne

[tonyl...@gmail.com]

Best if you can run with logging level at trace.  That provides the most detail in the log file.

[Wayne Black]

Ok here ya go. It's pretty repeatable on my machine

[Wayne Black]

For giggles I did other logs to try and see any patterns for the exception in regards to time or cabling...

First I used my new 232-usb cable to a usb hub shared w a wired mouse. I logged several M114 exceptions with jogging.

Second I removed the hub and put the new 232-usb cable directly to the pc port. I logged and M114  exception on homing.

Lastly I used the original 232 usb cable that came with the machine that never suffered com problems with the original SW and connected directly to the pc port. Here again several M114 exceptions in just as many minutes.

Im running comms at 115200, Should I half this to the 57K in the firmware and rebuild/retest?

Logs attached

[Niclas Hedhman]

Hi,
I am not at all into OpenPnp drivers, but this sounds like there are
buffers that are not flushed, either on sender or receiver side.

Could it be that the Chmt side is looking for double linefeed or
something to start "processing"?

Niclas

>>> Anyone has any idea?
>>>
>>> I'm using windows with latest stable OpenPnP, machine.xml
>>> attached.
>>>
>>> Sam
>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "OpenPnP" group.
>>> To unsubscribe from this group and stop receiving emails from it,
>>> send an email to openpnp+u...@googlegroups.com.
>>> To view this discussion on the web visit
>>>
>>
> https://groups.google.com/d/msgid/openpnp/8661aeae-841f-4211-9558-ed9f3ae94b0an%40googlegroups.com

>>> [1].

>
> --
> You received this message because you are subscribed to the Google
> Groups "OpenPnP" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to openpnp+u...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/openpnp/3c80b371-6d6a-4d68-b055-752302311b46n%40googlegroups.com
> [2].
>
>
> Links:
> ------
> [1]
> https://groups.google.com/d/msgid/openpnp/8661aeae-841f-4211-9558-ed9f3ae94b0an%40googlegroups.com?utm_medium=email&utm_source=footer
> [2]
> https://groups.google.com/d/msgid/openpnp/3c80b371-6d6a-4d68-b055-752302311b46n%40googlegroups.com?utm_medium=email&utm_source=footer

[mark maker]

Hi,

As you see here:

2022-01-06 21:01:16.172 GcodeDriver$ReaderThread TRACE: [serial://COM4] << oM1k14
2022-01-06 21:01:36.170 AbstractMachine TRACE: Exception caught, executing pending motion: java.lang.Exception: GcodeDriver timeout waiting for response to M114
    at org.openpnp.machine.reference.driver.GcodeDriver.getReportedLocation(GcodeDriver.java:630)
    at org.openpnp.machine.reference.driver.GcodeAsyncDriver.waitForCompletion(GcodeAsyncDriver.java:338)
    at org.openpnp.machine.reference.driver.AbstractMotionPlanner.waitForDriverCompletion(AbstractMotionPlanner.java:786)
    at org.openpnp.machine.reference.driver.AbstractMotionPlanner.waitForCompletion(AbstractMotionPlanner.java:709)

the response OpenPnP gets is garbled. "oM1k14". It almost looks as if sender and receiver buffers get "interlaced".

This can have a multitude of reasons, but most strongly points to the firmware/hardware, and less likely the electrical side, I doubt such a thing has anything to do with USB packet transport, hubs, or OpenPnP.

Questions/Ideas:

1. What serial flow control do you have configured in OpenPnp on the driver?
   
2. Does it still happen, when you switch on Confirmation Flow Control, and switch off Location Confirmation on the driver?
   https://github.com/openpnp/openpnp/wiki/GcodeAsyncDriver#gcodeasyncdriver-specific-settings
   
3. Can you run with a USB isolator on the USB/Serial?
   https://www.adafruit.com/product/2107
   
4. Or with a notebook that is not earthed: two-pin mains jack, or better yet (for a quick test) running on battery?
5. How sure are we, that the serial implementation on the Smoothie port is robust and truly full duplex?
6. If it is not full duplex, you need to convert the driver back to the GcodeDriver. The GcodeAsyncDriver can only support Full Duplex. If that is the case, report back, I'll tell you how.
   This would be a big setback though (you're losing features and speed), and it really should be solved on the firmware/hardware side. I still wonder, why you guys can''t use the USB directly/natively on that STM32 board. It would be so much faster and robust.
   

_Mark

[Jan]

Hi Wayne!
The RS232 connection to the mainboard is two wire without any
flowcontrol. I'd suggest to a) check that the serial port settings have
"Flow Controll" "Off" and b) "Confirmation Flow Control?" is enabled (on
Advanced Settings, see attached picture).

Jan

> /2022-01-06 19:18:36.223 MessageBoxes DEBUG: Error:

> java.lang.Exception: GcodeDriver timeout waiting for
> response to M114
> 2022-01-06 19:18:56.278 AbstractHeadMountable DEBUG:
> N1.moveTo((319.373000, 258.170000, -0.150000, 0.000000 mm),

> 0.5)/

>> Screenshot 2022-01-06 194212.png
>>
>> Anyone has any idea?
>>
>> I'm using windows with latest stable OpenPnP,
>> machine.xml attached.
>>
>> Sam
>> --
>> You received this message because you are
>> subscribed to the Google Groups "OpenPnP" group.
>> To unsubscribe from this group and stop receiving
>> emails from it, send an email to
>> openpnp+u...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/openpnp/8661aeae-841f-4211-9558-ed9f3ae94b0an%40googlegroups.com

>> <https://groups.google.com/d/msgid/openpnp/8661aeae-841f-4211-9558-ed9f3ae94b0an%40googlegroups.com?utm_medium=email&utm_source=footer>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "OpenPnP" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to openpnp+u...@googlegroups.com

> <mailto:openpnp+u...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/openpnp/f30c0f93-58ef-4266-b521-43535909f904n%40googlegroups.com
> <https://groups.google.com/d/msgid/openpnp/f30c0f93-58ef-4266-b521-43535909f904n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Jan]

Hi Mark!
The USB-Device port of the STM32 is available on the CAN connector. One
would have to solder some bridging wires to cross the can transceiver.
Adding a USB device driver with serial communication class should not be
to difficult as CubeMX provides sample code. It's on my list - to
benefit from your motion blending - but first I've to get my machine
back and running.
Btw: do you know a simple way to get the smoothie code into an IDE? I'm
currently using the supplied build setup but this does not provide any
debugging facilities...

Jan

On 07.01.2022 09:27, mark maker wrote:
> Hi,
>
> As you see here:
>
> 2022-01-06 21:01:16.172 GcodeDriver$ReaderThread TRACE: [serial://COM4]
> << oM1k14
> 2022-01-06 21:01:36.170 AbstractMachine TRACE: Exception caught,
> executing pending motion: java.lang.Exception: GcodeDriver timeout
> waiting for response to M114
>     at
> org.openpnp.machine.reference.driver.GcodeDriver.getReportedLocation(GcodeDriver.java:630)
>     at
> org.openpnp.machine.reference.driver.GcodeAsyncDriver.waitForCompletion(GcodeAsyncDriver.java:338)
>     at
> org.openpnp.machine.reference.driver.AbstractMotionPlanner.waitForDriverCompletion(AbstractMotionPlanner.java:786)
>     at
> org.openpnp.machine.reference.driver.AbstractMotionPlanner.waitForCompletion(AbstractMotionPlanner.java:709)
>
> the response OpenPnP gets is garbled. "oM1k14". It almost looks as if
> sender and receiver buffers get "interlaced".
>
> This can have a multitude of reasons, but most strongly points to the
> firmware/hardware, and less likely the electrical side, I doubt such a
> thing has anything to do with USB packet transport, hubs, or OpenPnP.
>
> Questions/Ideas:
>

> 1. What serial flow control do you have configured in OpenPnp on the
> driver?
> 2. Does it still happen, when you switch _on _*Confirmation Flow
> Control*, and switch _off _*Location Confirmation* on the driver?
> https://github.com/openpnp/openpnp/wiki/GcodeAsyncDriver#gcodeasyncdriver-specific-settings
> 3. Can you run with a USB isolator on the USB/Serial?
> https://www.adafruit.com/product/2107
> 4. Or with a notebook that is not earthed: two-pin mains jack, or

> better yet (for a quick test) running on battery?

> 5. How sure are we, that the serial implementation on the Smoothie port

> is robust and truly full duplex?

> 6. If it is not full duplex, you need to convert the driver back to the
> GcodeDriver. The Gcode*Async*Driver can only support Full Duplex. If

> <mailto:openpnp+u...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/openpnp/c537913c-8ce6-f524-58b4-87f9348bf0e5%40makr.zone
> <https://groups.google.com/d/msgid/openpnp/c537913c-8ce6-f524-58b4-87f9348bf0e5%40makr.zone?utm_medium=email&utm_source=footer>.

[Sam Vanheule]

Ok, I think it's fixed :). The  "Confirmation Flow Control" is enabled (on Advanced Settings) seems to solve it. I already run a job multiple times with and without this setting. With it enabled I don't get the M114 error anymore, If I disable it it causes the random error to reappear.

Charmhigh machine does not support normal types of flow control.

Picture attached for reference (Thanks Jan)

Solution:

Enable "Confirmation Flow Control" under Advanced Settings in GcodeDriver

Disable Xon/Xoff flow control

Disable Rts/Cts

Many thanks for the help!

Sam

[Wayne Black]

Hi Sam,

Your solution above makes matters worse for me. Essentially a non-starter. I have everything set as you above and get the following upon homing... 

Ill have to fully review the explanations offered by Mark latter in the day and Ill update what I can find.

Thanks again to all for helping

[tonyl...@gmail.com]

Something seems to go wrong right at the start as G21 is the very first thing sent to the controller:

2022-01-07 08:57:41.285 ReferenceMachine DEBUG: setEnabled(true) 2022-01-07 08:57:42.307 GcodeAsyncDriver DEBUG: serial://COM3 commandQueue.offer(G21 ; Set millimeters mode, 10000)... 2022-01-07 08:57:42.308 GcodeAsyncDriver DEBUG: serial://COM3 commandQueue.offer(G90 ; Set absolute positioning mode, 10000)... 2022-01-07 08:57:42.309 GcodeAsyncDriver$WriterThread TRACE: [serial://COM3] >> G21 2022-01-07 08:57:42.309 GcodeAsyncDriver DEBUG: serial://COM3 commandQueue.offer(M82 ; Set absolute mode for extruder, 10000)... 2022-01-07 08:57:42.309 GcodeAsyncDriver DEBUG: serial://COM3 commandQueue.offer(M809 ; Turn off VAC, 10000)... 2022-01-07 08:57:42.309 GcodeAsyncDriver DEBUG: serial://COM3 commandQueue.offer(M801 ; Vacuum nozzle 1 off, 10000)... 2022-01-07 08:57:42.309 GcodeAsyncDriver DEBUG: serial://COM3 commandQueue.offer(M805 ; Vacuum nozzle 2 off, 10000)... 2022-01-07 08:57:42.309 GcodeAsyncDriver DEBUG: serial://COM3 commandQueue.offer(M813 ; Turn off BLOW, 10000)... 2022-01-07 08:57:42.309 GcodeAsyncDriver DEBUG: serial://COM3 commandQueue.offer(M810 ; Turn on UPLED, 10000)... 2022-01-07 08:57:42.310 GcodeAsyncDriver DEBUG: serial://COM3 commandQueue.offer(G4P100 ; Wait 750 milliseconds, 10000)... 2022-01-07 08:57:42.310 GcodeAsyncDriver DEBUG: serial://COM3 commandQueue.offer(M811 ; Turn off UPLED, 10000)... 2022-01-07 08:57:42.310 GcodeAsyncDriver DEBUG: serial://COM3 commandQueue.offer(M92 X31.94 Y31.94 ; Set steps per mm, 10000)... 2022-01-07 08:57:42.325 GcodeDriver$ReaderThread TRACE: [serial://COM3] << ok - ignored
I'm not sure why the controller responded with "ok - ignored".  Perhaps that is what the controller reports if it gets something it doesn't understand?

and then later, things get garbled:

2022-01-07 08:57:53.723 GcodeAsyncDriver$WriterThread TRACE: [serial://COM3] >> M114 2022-01-07 08:57:53.745 GcodeDriver$ReaderThread TRACE: [serial://COM3] << ok C: X:0.0000 Y:0.0000 Z:0.0000 A:0.0000 B:0.0000 C:0.0000 E:0.0000

[Wayne Black]

Ok progress. This is all preliminary, but I wanted to give all who helped a little feedback before too long. 

This am before work I attempted the settings as per Jan and Sam's settings, but this led to a G21 exception/openpnp lockup with homing and then later a 'dragpin endstop' exception upon user input. Later today I empirically went through the driver settings a came upon a combination that has the machine, jogging, homing, actuators, idling and power resetting the machine for several minutes without false exceptions upon user input. I've attached a pic of the settings and the log of the exception free operation. 

To add, Ive noticed that stepping through this process I have to hard restart both openpnp and the machine for definitive changes to take effect. Also hard resets of the machine require 10+ sec to fully discharge the system. If not sometimes the  openpnp locks and the machine goes crazy. The first crazy behavior was during fiducial calibration the machine crashed the head in XY past sw limits. Second 'crazy' time was altering driver cfg/advanced settings the dragpin started oscillating up/down uncontrollably. 

After all this I did have another XY machine crash, but I think this was due to not initially homing after a hard reset. It seems Openpnp or smoothie wants a homing performed within the watchdog timer of connecting or it throws an exception.

Lastly, in openpnp there appears to be latency in the Log files being updated from the gui/console. Is there a way to force log update to the file within the gui?

 Thanks Jan, Sam and especially Mark. Ill try and get further logs/performance update as I work through getting the setup.

[tonyl...@gmail.com]

I still see several places in working_log.txt where you are having problems:

2022-01-07 14:10:04.137 GcodeAsyncDriver$WriterThread TRACE: [serial://COM3] >> G21 2022-01-07 14:10:04.139 GcodeDriver$ReaderThread TRACE: [serial://COM3] << ok - ignored 2022-01-07 14:10:06.419 AbstractHeadMountable DEBUG: N1.moveTo((299.373000, 59.170000, -0.150000, 0.000000 mm), 0.5) 2022-01-07 14:10:06.420 GcodeAsyncDriver DEBUG: serial://COM3 commandQueue.offer(M119 ; DRAGPIN endstop status, 10000)... 2022-01-07 14:10:14.139 GcodeAsyncDriver$WriterThread TRACE: [serial://COM3] >> M82 2022-01-07 14:10:14.141 GcodeDriver$ReaderThread TRACE: [serial://COM3] << ok 2022-01-07 14:10:14.141 GcodeDriver TRACE: [serial://COM3] confirmed M82 2022-01-07 14:10:14.141 AbstractMachine TRACE: Exception caught, executing pending motion: java.lang.Exception: serial://COM3 error response from controller: serial://COM3 timeout waiting for response to G21 at org.openpnp.machine.reference.driver.GcodeDriver.bailOnError(GcodeDriver.java:1103) at org.openpnp.machine.reference.driver.GcodeAsyncDriver.bailOnError(GcodeAsyncDriver.java:293) at org.openpnp.machine.reference.driver.GcodeDriver.receiveResponses(GcodeDriver.java:1111) at org.openpnp.machine.reference.driver.GcodeDriver.receiveResponses(GcodeDriver.java:1132) at org.openpnp.machine.reference.driver.GcodeDriver.actuatorRead(GcodeDriver.java:955) at org.openpnp.machine.reference.driver.GcodeDriver.actuatorRead(GcodeDriver.java:987) at org.openpnp.machine.reference.ReferenceActuator.read(ReferenceActuator.java:328) at org.openpnp.machine.reference.ActuatorInterlockMonitor.interlockActuation(ActuatorInterlockMonitor.java:370) at org.openpnp.machine.reference.driver.AbstractMotionPlanner.moveTo(AbstractMotionPlanner.java:152) at org.openpnp.machine.reference.driver.ReferenceAdvancedMotionPlanner.moveTo(ReferenceAdvancedMotionPlanner.java:317) at org.openpnp.machine.reference.ReferenceHead.moveTo(ReferenceHead.java:141) at org.openpnp.spi.base.AbstractHeadMountable.moveTo(AbstractHeadMountable.java:242) at org.openpnp.spi.base.AbstractHeadMountable.moveTo(AbstractHeadMountable.java:269) at org.openpnp.gui.JogControlsPanel.jogTool(JogControlsPanel.java:236) at org.openpnp.gui.JogControlsPanel.lambda$jog$0(JogControlsPanel.java:169) at org.openpnp.util.UiUtils.lambda$submitUiMachineTask$0(UiUtils.java:38) at org.openpnp.spi.base.AbstractMachine$1.call(AbstractMachine.java:571) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) 2022-01-07 14:10:14.141 MessageBoxes DEBUG: Error: java.lang.Exception: serial://COM3 error response from controller: serial://COM3 timeout waiting for response to G21
.

.

.

2022-01-07 14:14:30.489 GcodeAsyncDriver$WriterThread TRACE: [serial://COM3] >> G21 2022-01-07 14:14:30.506 GcodeDriver$ReaderThread TRACE: [serial://COM3] << ok - ignored 2022-01-07 14:14:40.490 GcodeAsyncDriver$WriterThread TRACE: [serial://COM3] >> M82 2022-01-07 14:14:40.491 GcodeDriver$ReaderThread TRACE: [serial://COM3] << ok 2022-01-07 14:14:40.491 GcodeDriver TRACE: [serial://COM3] confirmed M82 2022-01-07 14:14:40.492 GcodeAsyncDriver$WriterThread TRACE: [serial://COM3] >> M809 2022-01-07 14:14:40.506 GcodeDriver$ReaderThread TRACE: [serial://COM3] << ok 2022-01-07 14:14:40.507 GcodeDriver TRACE: [serial://COM3] confirmed M809 2022-01-07 14:14:40.507 GcodeAsyncDriver$WriterThread TRACE: [serial://COM3] >> M801 2022-01-07 14:14:40.523 GcodeDriver$ReaderThread TRACE: [serial://COM3] << ok 2022-01-07 14:14:40.524 GcodeDriver TRACE: [serial://COM3] confirmed M801 2022-01-07 14:14:40.524 GcodeAsyncDriver$WriterThread TRACE: [serial://COM3] >> M805 2022-01-07 14:14:40.538 GcodeDriver$ReaderThread TRACE: [serial://COM3] << ok 2022-01-07 14:14:40.539 GcodeDriver TRACE: [serial://COM3] confirmed M805 2022-01-07 14:14:40.539 GcodeAsyncDriver$WriterThread TRACE: [serial://COM3] >> M813 2022-01-07 14:14:40.554 GcodeDriver$ReaderThread TRACE: [serial://COM3] << ok 2022-01-07 14:14:40.555 GcodeDriver TRACE: [serial://COM3] confirmed M813 2022-01-07 14:14:40.555 GcodeAsyncDriver$WriterThread TRACE: [serial://COM3] >> M810 2022-01-07 14:14:40.571 GcodeDriver$ReaderThread TRACE: [serial://COM3] << ok 2022-01-07 14:14:40.571 GcodeDriver TRACE: [serial://COM3] confirmed M810 2022-01-07 14:14:40.572 GcodeAsyncDriver$WriterThread TRACE: [serial://COM3] >> G4P100 2022-01-07 14:14:40.682 GcodeDriver$ReaderThread TRACE: [serial://COM3] << ok 2022-01-07 14:14:40.683 GcodeDriver TRACE: [serial://COM3] confirmed G4P100 2022-01-07 14:14:40.683 GcodeAsyncDriver$WriterThread TRACE: [serial://COM3] >> M811 2022-01-07 14:14:40.699 GcodeDriver$ReaderThread TRACE: [serial://COM3] << ok 2022-01-07 14:14:40.699 GcodeDriver TRACE: [serial://COM3] confirmed M811 2022-01-07 14:14:40.700 GcodeAsyncDriver$WriterThread TRACE: [serial://COM3] >> M92X31.94Y31.94 2022-01-07 14:14:40.722 GcodeDriver$ReaderThread TRACE: [serial://COM3] << X:31.940001 Y:31.940001 Z:4.444400 A:4.444400 B:4.444400 C:69.264198 E:69.264198 2022-01-07 14:14:40.722 GcodeDriver TRACE: Position report: X:31.940001 Y:31.940001 Z:4.444400 A:4.444400 B:4.444400 C:69.264198 E:69.264198 2022-01-07 14:14:40.723 GcodeDriver$ReaderThread TRACE: [serial://COM3] << ok 2022-01-07 14:14:45.811 ReferenceMachine DEBUG: homing machine 2022-01-07 14:14:45.811 AbstractMachine TRACE: Exception caught, executing pending motion: java.lang.Exception: serial://COM3 error response from controller: serial://COM3 timeout waiting for response to G21

.

.

.

2022-01-07 14:15:57.452 GcodeAsyncDriver$WriterThread TRACE: [serial://COM3] >> M105 2022-01-07 14:15:57.461 GcodeDriver$ReaderThread TRACE: [serial://COM3] << ok V1:123.3 /0.0 @0 V2:123.0 /0.0 @0 2022-01-07 14:15:57.461 GcodeDriver TRACE: actuatorRead response: ok V1:123.3 /0.0 @0 V2:123.0 /0.0 @0 2022-01-07 14:15:57.461 ReferenceActuator DEBUG: N1VAC.read(): 123.3 2022-01-07 14:15:57.461 GcodeAsyncDriver DEBUG: serial://COM3 commandQueue.offer(M400 ; Wait for moves to complete before returning, 20000)... 2022-01-07 14:16:07.463 GcodeAsyncDriver TRACE: GcodeDriver confirmation complete. 2022-01-07 14:16:07.464 AbstractMachine TRACE: Exception caught, executing pending motion: java.lang.Exception: serial://COM3 error response from controller: serial://COM3 timeout waiting for response to M105 .

.

.

2022-01-07 14:10:04.137 GcodeAsyncDriver$WriterThread TRACE: [serial://COM3] >> G21 2022-01-07 14:10:04.139 GcodeDriver$ReaderThread TRACE: [serial://COM3] << ok - ignored 2022-01-07 14:10:06.419 AbstractHeadMountable DEBUG: N1.moveTo((299.373000, 59.170000, -0.150000, 0.000000 mm), 0.5) 2022-01-07 14:10:06.420 GcodeAsyncDriver DEBUG: serial://COM3 commandQueue.offer(M119 ; DRAGPIN endstop status, 10000)... 2022-01-07 14:10:14.139 GcodeAsyncDriver$WriterThread TRACE: [serial://COM3] >> M82 2022-01-07 14:10:14.141 GcodeDriver$ReaderThread TRACE: [serial://COM3] << ok 2022-01-07 14:10:14.141 GcodeDriver TRACE: [serial://COM3] confirmed M82 2022-01-07 14:10:14.141 AbstractMachine TRACE: Exception caught, executing pending motion: java.lang.Exception: serial://COM3 error response from controller: serial://COM3 timeout waiting for response to G21

[Wayne Black]

Thanks for the insights Tony. Im at a loss reading/understanding the trace logs. I appreciate your all's expertise in spotting anomalies. 

With the machine now jogging reasonably well I went back to working through the Issues and Solutions, but didn't get far before the machine starts stumbling again. I noticed that though I go thru the homing process, Openpnp never sets home. Something I should have recognized in the gui with the home icon staying yellow. I'm now stepping back to reread the Openpnp wiki before proceeding on. First goal is to get the machine/Openpnp to set home. 

[mark maker]

Hi Wayne,

I think you need a larger connect wait time on the driver (you seem to have 1000ms).

2022-01-07 08:57:41.285 ReferenceMachine DEBUG: setEnabled(true)
2022-01-07 08:57:42.307 GcodeAsyncDriver DEBUG: serial://COM3 commandQueue.offer(G21 ; Set millimeters mode, 10000)...

Note, I don't know why such a high wait time is needed (especially for Smoothie, it seems). While I have changed a lot in the GcodeDriver/GcodeAsyncDriver itself, I have never really touched or looked at the communications stuff.

_Mark

To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/fb3ea0c5-3177-407a-ad8a-5c591a7a3369n%40googlegroups.com.

[Wayne Black]

Hi Mark,

I will increase the connection wait time and report back. Can I ask you what if any driver board would you recommend to replace the original Charmhigh driver board?

To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/efc59267-6378-a7a6-b2c7-772b3a0f5a83%40makr.zone.

--

Wayne Black

Owner

Black Box Embedded, LLC

black...@blackboxembedded.com

1.831.682.4964

[mark maker]

> Can I ask you what if any driver board would you recommend to replace the original Charmhigh driver board?

I still think you could make it work well with the original board.

But if you really want to exchange it, AFAIK the original board has 7 axes, right?

In this case I recommend Duet3D boards:

https://github.com/openpnp/openpnp/wiki/Motion-Controller-Firmwares#duet3d

_Mark

To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/CABUTZN_%2BUvN7aen-36vX-m-LGvWxcGFhEUJAawwSauF47U4otQ%40mail.gmail.com.

[bert shivaan]

And they are in stock here if you are in the USA

Use this up to 5 motors:

https://www.dllpdf.com/electronics-and-controls/duet-3d/duet-3-mini-5

Add 1 of these to get 2 more axis:

https://www.dllpdf.com/electronics-and-controls/duet-3d/duet-3-expansion-mini-2

To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/74015aa6-e19a-dd8d-1e73-1683423eafa0%40makr.zone.

[Wayne Black]

The 36V so far as I know has 6 individual axis drivers. Typical 5  motion axis + 1 feeder uptake motor. For the 48V Im not sure because dual of dual feeder strip uptake motors, perhaps the dual motors are driven in parallel with a single driver?

I went ahead and purchased the Duet 6HC without a lot research as they seem to be running low of stock. The goal for the machine is proto/beta pcb builder that is QUIETER and won't oscillate other machines in my little lab like the oem 36 does. Its amazing how much more tolerable it is to be in the same room of the 36V with just changing out the fan and insulating the vacuum pump mount a bit better.

Any rate I don't mean to derail Sam's original post further. In regard to the M114 timeout my issue seems to have resolved with the above mentioned settings. Regardless the veterans of the message boards are still noticing anomalies within my trace logs. Im currently researching the move to Linux as recommended by Chris and further reviewing Openpnp wiki.

[Jarosław Karwik]

Well, no wonder stocks are getting low - no one is able to restock e.g. processors and Ethernet chips ( my order for these from 12.2021 is supposed to arrive 01.2023)

Today, you cannot find any STM chips available in stock in any major distributors ( Digikey, Farnell, Mouser etc. )

So this year will be challenging..... you need to be fast with your purchase

[vespaman]

fredag 7 januari 2022 kl. 09:27:29 UTC+1 skrev ma...@makr.zone:

2022-01-06 21:01:16.172 GcodeDriver$ReaderThread TRACE: [serial://COM4] << oM1k14

2022-01-06 21:01:36.170 AbstractMachine TRACE: Exception caught, executing pending motion: java.lang.Exception: GcodeDriver timeout waiting for response to M114
    at org.openpnp.machine.reference.driver.GcodeDriver.getReportedLocation(GcodeDriver.java:630)
    at org.openpnp.machine.reference.driver.GcodeAsyncDriver.waitForCompletion(GcodeAsyncDriver.java:338)
    at org.openpnp.machine.reference.driver.AbstractMotionPlanner.waitForDriverCompletion(AbstractMotionPlanner.java:786)
    at org.openpnp.machine.reference.driver.AbstractMotionPlanner.waitForCompletion(AbstractMotionPlanner.java:709)

the response OpenPnP gets is garbled. "oM1k14". It almost looks as if sender and receiver buffers get "interlaced".

This can have a multitude of reasons, but most strongly points to the firmware/hardware, and less likely the electrical side, I doubt such a thing has anything to do with USB packet transport, hubs, or OpenPnP.

Hi,

While looking for a M400 timeout issue, I found this thread (there are many others relating to this same issue!).

The reason for this is a bug in the interrupt rx code of Charmhigh-Smoothie, that probably has been around since day one. It actually mixes the tx with the rx, just as you suggest @Mark.

I fixed this one earlier in the year, but since I have moved on to a DMA solution, where I have rewritten the interrupt code, I don't really have a proper code base for interrupt driven rx on the chmt.

But if I remember correctly, I tried to do a clean check-in of this issue, so if anyone wants to test...

My guess is that with the fix, lots of strange issues can be resolved, maybe even with the confirmation flow control enabled.

 -  Micael

 

[Jan]

Hi Micael!
Just for reference: I'm running my machine with the firmware I've
modded to trigger I&S correctly
(https://github.com/janm012012/Smoothieware-CHMT) with confirmation flow
control enabled without communication issues and I don't remember that I
ever had any. With confirmation flow control enabled the communication
is basically half duplex and likely does not trigger the issue you found.
It's cool, that you fund such a fundamental error. Hopefully I can
participate from your work by merging your changes back to run them on
unmodified hardware.

Jan

> --
> You received this message because you are subscribed to the Google
> Groups "OpenPnP" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to openpnp+u...@googlegroups.com

> <mailto:openpnp+u...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/openpnp/6f37cf11-e6c2-4e5e-b176-c7e659ab2432n%40googlegroups.com <https://groups.google.com/d/msgid/openpnp/6f37cf11-e6c2-4e5e-b176-c7e659ab2432n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[vespaman]

Hi Jan,

Yes, I thought as much! (That the confirmation flow control hides the issue).

I have read so many threads (way back) with people having had issues with the serial, and I can say that all the strange things that people have reported in the past, is due to the broken rx interrupt code, so I just wanted to make everyone aware, perhaps newcomers especially. 

For everyone that has setup the chmt with confirmation flow control, or given up on the stock controller, and are happily operating their machines, I fully understand if they don't want to open it up and re-flash etc etc.

 - Micael

[mark maker]

Hi Micael,

Just so I understand correctly: with the TX/RX issue sorted we could move to XON/XOFF flow control, right? because RTS/CTS are still missing physically, right?

_Mark

To unsubscribe from this group and stop receiving emails from it, send an email to openpnp+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/4839c3ae-5e47-420d-be1a-a26e0694b292n%40googlegroups.com.

[vespaman]

Hi Mark,

Just so I understand correctly: with the TX/RX issue sorted we could move to XON/XOFF flow control, right? because RTS/CTS are still missing physically, right?

I thought about how to answer this for  a rather long time.

Yes it would be possible, but...

There's no code in the original Smoothie port, and I did not add it to my dma branch.

So either code must be added to original Smoothie port (with the rx IRQ bug fix applied of course), or onto the dma solution.

Benefit of doing this on the original Smoothie port, is that it is rather simple if we can count on the fact that the USB-Serial bridge does the XON/OFF itself. I know e.g. the XR21B1420 does this, but I wouldn't bet my savings on the rest of the pack.

Negative is that it will not accept any faster baudrate. Even 115200 is probably on the edge, afaict. For Chmt36 users this is anyway what they can hope for on stock hw. But for Chmt48 users, I doubt it will work up to their max (460kbaud). The solution is very fragile, since the UART can only hold one character. Serial interrupt is the lowest prio, so it is not deterministic.

Benefit of doing this on the dma branch, is that we have plenty of buffers, so the transfer is solid in it self.

Negative is that the CPU is not involved. Only between each message, or as a safety check, every 128bytes. So detecting own buffers getting filled will be easy, albeit a little late. But now we need to send the XOFF. Don't think this can be don in interrupt code, since the tx DMA might be pushing bytes out, so we need to set a flag for the application layer that it (gcode dispatcher probably) needs to send an XOFF ASAP. Problem here is that ASAP might not cut it (it looks to me, that gcode dispatcher can be away for quite a while e.g. after M400).

Here at least it will be possible in theory to go for higher speeds, but this means that we also have to consider the XON/XOFF from the usb-serial bridge, since it may not have sufficient buffers, and then we need to maintain the tx start/stop in code. Once again in application layer.

I fear that it would risk become a monster, and worse; very hard to do QA since we are stretching how XON/XOFF normally.

Disclaimer here, is that I don't really know how Smoothie does everything, so for someone else, this might be more straightforward. :-)

Jan once mentioned that porting reprat had come to his mind. And I have also considered this.. If you have a rtos things are easier.

But then again, on the upside, hardware handshaking is just two short wires away (on the CHMT36)!  ;-)

 - Micael

[mark maker]

Reading that I think it is sensible to say RTS/CTS is the only way.

> we have plenty of buffers, so the transfer is solid in it self.

I'm not sure what you mean by that.

With confirmation flow control enabled, the buffer needs to be large enough for the longest possible single G-code command. You can then forget about serial flow control. If that's what you meant, we're all on the same page. 😉

But you cannot blindly rely on your buffer being large enough and just forget about any flow control. With motion interpolation OpenPnP could send huge amounts of motion data before it needs to pause, i.e. wait for motion completion.

_Mark

--
You received this message because you are subscribed to the Google Groups "OpenPnP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openpnp+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/2e9a56be-b48d-4b03-a482-c33125c20ddan%40googlegroups.com.

[vespaman]

> we have plenty of buffers, so the transfer is solid in it self.

I'm not sure what you mean by that.

With confirmation flow control enabled, the buffer needs to be large enough for the longest possible single G-code command. You can then forget about serial flow control. If that's what you meant, we're all on the same page. 😉

But you cannot blindly rely on your buffer being large enough and just forget about any flow control. With motion interpolation OpenPnP could send huge amounts of motion data before it needs to pause, i.e. wait for motion completion.

We say the same thing, but talk about different buffers :-) I think.

I am talking about the low level byte reception buffer. Which with DMA can be set to anything really. Without DMA, it is one byte(!). Every single byte that is received has to be promptly moved out of the way from the single byte buffer by the CPU before the next byte arrives. Old code -> Interrupt on each byte.

Therefore, with the DMA solution, the buffers can be made as big as we like, so a little slack with the XON/XOFF is not a problem in it self, (as opposed to the old code that had only one byte buffer, and hence HAD to deal with XON/XOFF directly). But the buffer slack is a double edged sword, it is way more complex to manage the XON/XOFF now.

  - Micael

[mark maker]

> We say the same thing, but talk about different buffers :-) I think.

No, I don't think so.

From my (more conceptual) stand-point, I do not care whether it is a software buffer that is filled byte-by-byte using IRQ, or a DMA buffer that is filled by hardware autonomously. Both the software buffer and the DMA buffer are in RAM, both can be small or large, both can still fill up, even when large, both need flow control!

I'm quite sure there was a software buffer before or are these duds?

https://github.com/vespaman/Smoothieware-CHMT/blob/f9f60250f4b760967281daba0f920ea5b4f69243/src/libs/USBDevice/USBSerial/USBSerial.h#L47-L48

Using DMA is offloading the MCU, surely, and perhaps making it easier to fulfill hard real time guarantees for the stepper signals, everything else is more or less the same, AFAIK. Agree?

_Mark

--
You received this message because you are subscribed to the Google Groups "OpenPnP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openpnp+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/51e50828-f732-40c9-b2ab-b0c4de965c49n%40googlegroups.com.

[vespaman]

So this is for rx only, just to be clear;

>>From my (more conceptual) stand-point, I do not care whether it is a software buffer that is filled byte-by-byte using IRQ, or a DMA buffer that is filled by hardware autonomously. Both the >>software buffer and the DMA buffer are in RAM, both can be small or large, both can still fill up, even when large, both need flow control!

Well, sure, but as fw developer you need to consider the pace. The IRQ/one buffer solution has very hard time constraints that are impossible to guarantee when going for higher bit rates. This is not the case with DMA.

That software buffer was around before yes, and still is in the dma version (at least so far - I'd prefer to remove it, and maybe I will in future, but it has its benefits still, since there's no means to communicate a complete message to the gcode dispatcher - it has to be dealt with from the main "sequencer" ).

But that buffer is populated by the CPU only. So it will not help for increasing the speed, since the CPU cannot keep up with emptying the hardware one byte buffer. I.e. that buffer is only to collect data until a LF has arrived.

This is why keeping the interrupt version (which I did this summer) never did improve things much, it simply took too long time to empty the one hw buffer (yes, with RTS/CTS of course, otherwise it would be overrun on every single byte even at modest rates. XON/OFF will not help here, since this is sent on the serial as well, so the actual XOFF could be lost).  Same goes with confirmation flow control - the rx data in the controller is simply destroyed.

So on a bit/byte/hw level the interrupt solution is very fragile. This is not the case with the DMA solution.

With the DMA solution, we have RAM buffers that are manage by the DMA engine, that populates the DMA buffers and emits an interrupt to the CPU when there's a complete message (well, it does not know that there is a message, but it knows that no data has arrived in one byte time.

So, The DMA will collect everything up until idle line (this is the normal trigger) or every time it (the dma) arrives to buffersize/2. If the cpu does not have time to deal with this right now, no worry, since the DMA will continue to receive what ever is remaining, until the CPU finds time. And of course, if the DMA buffers gets too cramped, and the CPU cannot stuff any more into the secondary (software) buffer, the RTS line will be pulled. Still without any sort of time pressure.

And:- Of course any solution needs flow control. Have I said anything else? I am the designated "RTS/CTS guy" for Gods sake. ;-)

 -  Micael

[mark maker]

Thanks for the clarifications, I think we're more or less on the same page 😉

If you want to do some stress tests/benchmarks using OpenPnP, use Simulated3rdOrder as your driver Motion Control Type, then setup very fine interpolation. You'll quickly get quite large bursts of G-code (you need some CPU oomph too, otherwise OpenPnP/Java will become the bottleneck):

https://github.com/openpnp/openpnp/wiki/GcodeAsyncDriver#motion-control-type

https://github.com/openpnp/openpnp/wiki/GcodeAsyncDriver#interpolation

https://youtu.be/cH0SF2D6FhM

_Mark

To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/cf5e6d34-576f-4bef-85f4-e6a582a0a388n%40googlegroups.com.

[vespaman]

Hi Mark,

Actually I did that last night, and yes, it ended with a controller watchdog reset and an Java error message that I had not seen before. Definitely a proper stress test! :-)
Given the hardware at hand, do you think you could give me some starting values that are relevant and might work? The settings under gcode async driver are complex to say the least.

I set the Maximum Number of Steps to 128, which is what is in the controller.

Maximum Number of Jerk Steps I guess can around 5 or so(?).

Minimum Time Step was default at a very small value, 0.0000001 or something. That did not work. Looking at some of your example pics, 0.01 or thereabouts seems reasonable?

Minimum Axis Resolution Ticks - here it is getting tricky for me. I think I grasp what it does, just can't find a starting point.

Maximum Junction Deviation - also here, I struggle.

I have set the backlash comp to none.

I am guessing the CPU Juice is needed for calculating the step start/stop step frequency & motion for each block? I see that the code uses 64bit integer math. Is that still superior even if we have a floating point unit on the stm32f407, do you think? (I suspect it is, but then again, I know little about the calculations).

 - Micael

[mark maker]

Hi Micael,

1. First I want to make sure you run with RTS/CTS or some other serial flow control. Like I said before, it won't work without, not unless you have a megabyte DMA buffer or something.
2. Also make sure you have G-code compression enabled in the driver. The stress test should be realistic.
   
3. Then set the axes' resolutions properly, I believe that is well documented in Wiki and in tooltips and should answer your question about "ticks", otherwise ask specifically what is unclear (I just expanded the Wiki a bit):
   https://github.com/openpnp/openpnp/wiki/Machine-Axes#controller-settings
4. For the other interpolation settings, please ask specifically. Offhand, I cannot explain better than what I wrote in the Wiki (as linked before):
   https://github.com/openpnp/openpnp/wiki/GcodeAsyncDriver#interpolation

_Mark

To view this discussion on the web visit https://groups.google.com/d/msgid/openpnp/645a8add-aaf5-4fff-b1a0-afceea5712d8n%40googlegroups.com.