Duet 3 6XD + two 3HC - Driver "Does not exist"

[Zdenko Stanec]

Hi,

I found myself in an issue with this error:

2023-01-28 21:31:26.235 ReferenceMachine DEBUG: setEnabled(true)
2023-01-28 21:31:26.334 GcodeDriver$ReaderThread TRACE: [serial://COM21] << HTTP is enabled on port 80
2023-01-28 21:31:26.335 GcodeDriver$ReaderThread TRACE: [serial://COM21] << FTP is disabled
2023-01-28 21:31:26.336 GcodeDriver$ReaderThread TRACE: [serial://COM21] << TELNET is disabled
2023-01-28 21:31:26.337 GcodeDriver$ReaderThread TRACE: [serial://COM21] << Error: Driver 10.0 does not exist
2023-01-28 21:31:26.337 GcodeDriver$ReaderThread TRACE: [serial://COM21] << Error: Driver 10.1 does not exist
2023-01-28 21:31:26.337 GcodeDriver$ReaderThread TRACE: [serial://COM21] << Error: Driver 10.2 does not exist
2023-01-28 21:31:26.338 GcodeDriver$ReaderThread TRACE: [serial://COM21] << Error: Driver 11.0 does not exist
2023-01-28 21:31:26.338 GcodeDriver$ReaderThread TRACE: [serial://COM21] << Error: Driver 11.1 does not exist
2023-01-28 21:31:26.339 GcodeDriver$ReaderThread TRACE: [serial://COM21] << Error: Driver 11.2 does not exist
2023-01-28 21:31:26.339 GcodeDriver$ReaderThread TRACE: [serial://COM21] << Done!
2023-01-28 21:31:27.334 GcodeAsyncDriver DEBUG: serial://COM21 commandQueue.offer(G21 ; Set millimeters mode, 5000)...
2023-01-28 21:31:27.334 GcodeAsyncDriver DEBUG: serial://COM21 commandQueue.offer(G90 ; Set absolute positioning mode, 5000)...
2023-01-28 21:31:27.334 GcodeAsyncDriver$WriterThread TRACE: [serial://COM21] >> G21; Set millimeters mode
2023-01-28 21:31:27.335 GcodeAsyncDriver$WriterThread TRACE: [serial://COM21] >> G90; Set absolute positioning mode
2023-01-28 21:31:27.335 GcodeDriver$ReaderThread TRACE: [serial://COM21] << RepRapFirmware for Duet 3 MB6XD is up and runninok
2023-01-28 21:31:27.336 GcodeDriver$ReaderThread TRACE: [serial://COM21] << ok
2023-01-28 21:31:27.390 AbstractMachine TRACE: Machine entering idle state.

Using the latest stable FW for 6XD and 3HC.

IOs are working on both 3HC PCBAs, but it seems it is not connecting to Drivers.

Br,

Zdenko,

[Zdenko Stanec]

Hi,

I was adjusting the config.g file to switch some drivers to a different axis, 

In OpenPNP I pressed "Detect Firmware" as some axis were not visible, but now it is ok.

FIRMWARE_NAME: RepRapFirmware for Duet 3 MB6XD FIRMWARE_VERSION: 3.4.5 ELECTRONICS: Duet 3 MB6XD v1.0 or later FIRMWARE_DATE: 2022-11-30 19:41:59

X:0.000 Y:0.000 Z:0.000 U:0.000 V:0.000 W:0.000 A:0.000 B:0.000 E:0.000 Count 0 0 0 0 0 0 0 0 Machine 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 Bed comp 0.000

Driver assignments: X0.0 Y0.1 Z10.1 U11.1 (c)V10.0 (c)W11.0 (c)A10.2 (c)B11.2, 8 axes visible

Br,

[dc42]

It looks to me that the Duet 3 main board has not detected a board at CAN address 10 or 11. That can happen if the 6XD board resets unexpectedly, but of course that should not happen in normal use. Have you set the address switches on the 3HCs to CAN addresses 10 and 11?

You can check what boards the 6XD knows about from the RRF object model, using any of these methods:

1. Connect the Ethernet port of the 6XD to your router or directly to a PC or laptop, load Duet Web Control, start the Object Model Browser plugin, and look at the boards key.

2. Send this command via USB or serial: M409 "boards"

[Zdenko Stanec]

Hey dc42,

To answer your question, yes 3HCs are on addresses 10 and 11, and the last one in line is terminated.

I managed to get it working yesterday, and currently, I have the same issue:

______________________________

m409
{"key":"","flags":"","result":{"boards":[{},{},{}],"directories":{},"fans":[],"global":{},"heat":{},"inputs":[{},{},{},{},{},{},{},{},{},{},{},{}],"job":{},"limits":{},"move":{},"network":{},"sensors":{},"seqs":{},"spindles":[{},{},{},{}],"state":{},"tools":[],"volumes":[{},{}]}}
ok
m584
Driver assignments: X0.0 Y0.1   (c) (c) (c) (c), 8 axes visible
ok

_____________________________

It seems the 3HCs Drivers are not seen again,

But I have to mention that even with this issue with Drivers, Outputs 0_1_2 where I have solenoid valves are still working fine and I can actuate them, same as Inputs for pressure sensors, when I send the Comand I see Green light on CAN that it is transmitting data and valves/pressure sensor inputs are working.

Working IOs on 3HCs even if Drivers are not seen:

_____________________________

2023-01-29 19:25:32.288 GcodeDriver$ReaderThread TRACE: [serial://COM21] << Error: Driver 10.1 does not exist
2023-01-29 19:25:32.289 GcodeDriver$ReaderThread TRACE: [serial://COM21] << Error: Driver 11.1 does not exist
2023-01-29 19:25:32.290 GcodeDriver$ReaderThread TRACE: [serial://COM21] << Error: Driver 10.0 does not exist
2023-01-29 19:25:32.291 GcodeDriver$ReaderThread TRACE: [serial://COM21] << Error: Driver 11.0 does not exist
2023-01-29 19:25:32.292 GcodeDriver$ReaderThread TRACE: [serial://COM21] << Error: Driver 10.2 does not exist
2023-01-29 19:25:32.293 GcodeDriver$ReaderThread TRACE: [serial://COM21] << Error: Driver 11.2 does not exist
2023-01-29 19:25:32.294 GcodeDriver$ReaderThread TRACE: [serial://COM21] << Done!
2023-01-29 19:25:32.295 GcodeDriver$ReaderThread TRACE: [serial://COM21] << RepRapFirmware for Duet 3 MB6XD is up and running.
2023-01-29 19:25:32.296 GcodeDriver$ReaderThread TRACE: [serial://COM21] << Ethernet running, IP address = 192.168.1.146

2023-01-29 19:35:32.589 GcodeAsyncDriver$WriterThread TRACE: [serial://COM21] >> M42P4S1
2023-01-29 19:35:32.591 GcodeDriver$ReaderThread TRACE: [serial://COM21] << ok
2023-01-29 19:35:32.643 AbstractMachine TRACE: Machine entering idle state.
2023-01-29 19:35:36.024 GcodeAsyncDriver DEBUG: serial://COM21 commandQueue.offer(M409 K"sensors.gpIn[0].value", 5000)...
2023-01-29 19:35:36.025 GcodeAsyncDriver$WriterThread TRACE: [serial://COM21] >> M409K"sensors.gpIn[0].value"
2023-01-29 19:35:36.026 GcodeDriver$ReaderThread TRACE: [serial://COM21] << {"key":"sensors.gpIn[0].value","flags":"","result":1}
2023-01-29 19:35:36.026 GcodeDriver$ReaderThread TRACE: [serial://COM21] << ok
2023-01-29 19:35:36.026 GcodeDriver TRACE: actuatorRead response: {"key":"sensors.gpIn[0].value","flags":"","result":1}
2023-01-29 19:35:36.027 ReferenceActuator DEBUG: AVAC1.read(): 1
2023-01-29 19:35:36.029 AbstractMachine TRACE: Machine entering idle state.

______________________________

The only issue is these Drives that keep "dropping" for some reason and are not visible sometimes.

Another strange issue I have with drivers is if I get them to work with resetting the 3HC boards and 6XD board, sometimes it seems that the motor current is too low and they hardly move, so I reset the 3HCs again and they are rotating fine.

Maybe I have an issue somewhere in config.g ?

Br,

[dc42]

@zdenko I see that you already have an account at https://forum.duet3d.com/ so I suggest you post there for faster response, because our support engineers monitor that forum; whereas I am only on this one a few times a week. Meanwhile I have a few suggestions:

- Run commands M122 B10 and M122 B11. In the responses check that the firmware version is correct (the latest stable expansion board firmware is version 3.4.4) and also check the reported bootloader version. If the bootloader version is earlier than 2.4 or reported as "not available", upgrade it to 2.4. Older bootloader versions are sometimes slow to start up, and slow startup of the 3HC boards could be causing this issue.

- If the bootloader versions are already up to date, change the G4 S2 command in config.g to G4 S4.

- If the problem occurs again, try running M98 P"config.g". That will either fix the problem or report errors trying to initialise the drivers.

What's happening is that when the commands in config.g to initialise the drivers are run, the main board hasn't received announcement messages from the 3HC boards yet, so it considers that the driver numbers are out of range.

[Zdenko Stanec]

Hey dc42,

I will check/try all of what you have mentioned,

Btw, is there any difference in resetting 6XD Board with the hardware reset button and for example over DWC when I change/save the config.g file, should be the same, but just want to check?

I will report on Duet forum if the issue continues.

Thank you for support.

Zdenko.

[dc42]

@zdenko if you use the reset button then I think the issue you report will always occur, because the 3HC boards won't know that the main board has reset. If you do any sort of software reset then the main board notifies the 3HC boards to reset too.

[Zdenko Stanec]

Hey dc42,

What about in the case if I use HW rest buttons on both 3HC boards one after another and then on the last step I use HW rest on 6XD, in that case, 6XD should initialize 3HC boards? Or I should just never use HW reset and just go with software reset over DWC.

Sorry for these questions, but I just want to limit possible handling mistakes.

Br,

[dc42]

You would need to release the reset button on the 3HCs just after releasing it on the 6XD, so that the 3HCs announce themselves to the 6XD after it has started up but before it tries to configure the drivers. TBH we don't really expect users to need to use the Reset pin, except on the 6XD when programming it using Bossa.

Here's another possibility. Replace the G4 S2 delay command in config.g with a loop that waits for the other boards to have announced themselves. Something like this:

while #boards < 3 && iterations < 300

  G4 P100

This will cause it to wait for up to 30 seconds for the other boards to announce themselves. So you could then hardware reset the 6XD, then you have 30 seconds to hardware reset both 3HCs.

[Zdenko Stanec]

Well, this makes all things much clearer, thank you for explaining.

I will report later today.

Zdenko,

[Zdenko Stanec]

Hi dc42,

I tested all possible solutions and re-created a problem again, and the route cause is me. 🎉

I was the issue as I did not know how the process of initialization of CAN extension hardware is working together with 6XD, if in some case with power applied on 3HCs I use hardware reset on 6XD which should never be a case, I lose connection only to 3HC Drivers, so proper reset from DWC is needed for all things to work/initialize again (for example after changing config.g file)

Just for a note, when I received the 6XD and later 3HCs, my first step was flashing the new FW and bootloader, so all should be the latest stable rev. as we see below.

__________________________

M122 B10
Diagnostics for board 10:
Duet EXP3HC rev 1.02 or later firmware version 3.4.4 (2022-10-14 11:45:56)
Bootloader ID: SAME5x bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 158932, free system stack 187 words
Tasks: Move(notifyWait<null>,0.0%,125) HEAT(notifyWait<null>,0.0%,95) CanAsync(notifyWait<null>,0.0%,69) CanRecv(notifyWait<null>,0.0%,80) CanClock(notifyWait<null>,0.0%,71) TMC(notifyWait<null>,7.5%,65) MAIN(running<null>,91.1%,450) IDLE(ready<null>,0.0%,40) AIN(delaying<null>,1.3%,263), total 100.0%
Last reset 00:16:06 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 8.9 steps/mm,standstill, SG min 0, mspos 8, reads 43338, writes 19 timeouts 0, steps req 7112 done 7112
Driver 1: pos 0, 20.0 steps/mm,standstill, SG min 0, mspos 8, reads 43342, writes 16 timeouts 0, steps req 0 done 0
Driver 2: pos 0, 8.9 steps/mm,standstill, SG min 0, mspos 8, reads 43340, writes 19 timeouts 0, steps req 7824 done 7824
Moves scheduled 18, completed 18, in progress 0, hiccups 0, step errors 0, maxPrep 27, maxOverdue 3, maxInc 2, mcErrs 0, gcmErrs 0
Peak sync jitter -6/6, peak Rx sync delay 177, resyncs 0/0, no step interrupt scheduled
VIN voltage: min 24.4, current 24.4, max 24.4
V12 voltage: min 12.3, current 12.3, max 12.3
MCU temperature: min 19.8C, current 31.6C, max 31.9C
Last sensors broadcast 0x00000000 found 0 226 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 7770, send timeouts 0, received 4876, lost 0, free buffers 37, min 37, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 407, adv 4628/37191
ok

M122 B11
Diagnostics for board 11:
Duet EXP3HC rev 1.02 or later firmware version 3.4.4 (2022-10-14 11:45:56)
Bootloader ID: SAME5x bootloader version 2.4 (2021-12-10)
All averaging filters OK
Never used RAM 158804, free system stack 192 words
Tasks: Move(notifyWait<null>,0.0%,125) HEAT(notifyWait<null>,0.0%,108) CanAsync(notifyWait<null>,0.0%,69) CanRecv(notifyWait<null>,0.0%,80) CanClock(notifyWait<null>,0.0%,71) TMC(notifyWait<null>,7.5%,65) MAIN(running<null>,91.1%,441) IDLE(ready<null>,0.0%,40) AIN(delaying<null>,1.3%,263), total 100.0%
Last reset 00:16:35 ago, cause: power up
Last software reset data not available
Driver 0: pos 0, 8.9 steps/mm,standstill, SG min 0, mspos 8, reads 22087, writes 19 timeouts 0, steps req 6400 done 6400
Driver 1: pos 0, 20.0 steps/mm,standstill, SG min 0, mspos 8, reads 22090, writes 16 timeouts 0, steps req 0 done 0
Driver 2: pos -10844, 8.9 steps/mm,standstill, SG min 0, mspos 584, reads 22088, writes 19 timeouts 0, steps req 17244 done 17244
Moves scheduled 28, completed 28, in progress 0, hiccups 0, step errors 0, maxPrep 27, maxOverdue 3, maxInc 3, mcErrs 0, gcmErrs 0
Peak sync jitter -6/5, peak Rx sync delay 178, resyncs 0/0, no step interrupt scheduled
VIN voltage: min 24.4, current 24.4, max 24.4
V12 voltage: min 12.2, current 12.2, max 12.2
MCU temperature: min 18.8C, current 30.6C, max 30.6C
Last sensors broadcast 0x00000000 found 0 161 ticks ago, 0 ordering errs, loop time 0
CAN messages queued 8003, send timeouts 0, received 5032, lost 0, free buffers 37, min 37, error reg 0
dup 0, oos 0/0/0/0, bm 0, wbm 0, rxMotionDelay 412, adv 9992/37192
ok
__________________________

Thank you,

Zdenko,

[dc42]

@zdenko I'm glad you found a solution. BTW after editing config.g it's usually sufficient just to re-run config.g instead of rebooting. If not, then rebooting via DWC or sending M999 should be sufficient.