Connecting HAL components across multiple CPUs

[Joshua Dickerson]

Hello everyone,

I am working on and off again on writing my own machine controller, largely to understand LinuxCNC and related projects like MachineKit.  For reference, here's the controller I wrote running a few years ago:

https://www.youtube.com/watch?v=qasLhuJFZNU


I'm starting back up on the project.  Right now it works very similar to LinuxCNC, a shared memory space, all HAL components running on a single controller.  What I want to do is make a machine controller that could have conceivably worked on 80s MCUs, but also work on single desktop machine.  The idea would be to have a distributed HAL that would function across multiple MCUs- but of course works in the degenerate case of a single machine/PC with shared memory space, as with LinuxCNC.  For instance, there would be a main MCU which reads files, runs the GUI/HMI and handles the trajectory planning, homing, probing, tapping, etc.  There would also be a dedicated MCU for each joint controller which is updating encoders and sensors, generating step pulses, closing servo loops, etc.  

My naive approach would be to have a registry of output signals which includes which MCU each actually resides on.  So if a particular HAL component task is reading a signal that's already on the MCU it's running on, it's immediately returned- otherwise it pulls the data over a network layer (which could shared memory bus, RS485, ethernet, etc.)  Is there something out there that does this?  Maybe some kind of mutant hybrid between HAL/NML/SCADA?

Of course this complicates things over simply using pointers to a single shared memory space.  I also get the sense the linuxCNC group balks at this kind of idea, they want to keep it all on one machine in the interest of latency anyway.  Just searching various machine controller project communities for where this kind of idea might be interesting.  I would really like something like a Raspberry Pi tied into a MCU which does all the hard-RT/IO stuff, but not quite in the way Klipper does it.  Klipper basically sends commands to be followed at some specific time stamp.  That's actually pretty cool, but I want something that allows for more feedback and flexibility between the various systems.  

Anyhow, thanks in advance for your input.

[Charles Steinkuehler]

On 9/17/2018 10:07 PM, Joshua Dickerson wrote:
>
> My naive approach would be to have a registry of output signals which
> includes which MCU each actually resides on. So if a particular HAL
> component task is reading a signal that's already on the MCU it's running
> on, it's immediately returned- otherwise it pulls the data over a network
> layer (which could shared memory bus, RS485, ethernet, etc.) Is there
> something out there that does this? Maybe some kind of mutant hybrid
> between HAL/NML/SCADA?
>
> Of course this complicates things over simply using pointers to a single
> shared memory space. I also get the sense the linuxCNC group balks at this
> kind of idea, they want to keep it all on one machine in the interest of
> latency anyway. Just searching various machine controller project
> communities for where this kind of idea might be interesting. I would
> really like something like a Raspberry Pi tied into a MCU which does all
> the hard-RT/IO stuff, but not quite in the way Klipper does it. Klipper
> basically sends commands to be followed at some specific time stamp.
> That's actually pretty cool, but I want something that allows for more
> feedback and flexibility between the various systems.

Look into the haltalk stuff using Zero-MQ messaging. It was
specifically written to enable connecting various HAL "islands" which
may or may not have a shared memory space. The messaging protocol is
transport agnostic, so you can setup HAL nodes that communicate via
shared memory, Ethernet, serial, or whatever.

The existing haltalk may not do everything you want, but it's probably
the best place to start:

https://machinekoder.com/machinetalk-explained-part-4-hal-remote/

--
Charles Steinkuehler
cha...@steinkuehler.net

[mngr]

I am not that expert on machinekit, but I will try to give my opinion, maybe someone will support or correct it

The problem is communicating with lots of machine pieces, in Machinekit exists Machinetalk but it makes more Machinekit instances talking to each other (and Machinekit runs on Linux).

It is based on ZMQ, that requires some Unix system calls, so moving to MCU world may be quite hard, always if it is possible to port ZMQ to firmware.

Talking about low level communication protocol CAN may be a good option, since it is already used in automotive and supports multiple endpoints talking (one at time) with a master. Or even multi master if needed. But this would require some work above that comunicate in a proper way with machinekit, at least a HAL module/ direver that manages the CAN communication and shows all the connected MCU to the other HAL module

Another system that try to solve this is EtherCAT, LinuxCNC can surely talk with ethercat modules

I tried it a couple of years ago, and i found that the industrial world use EtherCAT on windows and has the xml config file written in a different way from the LinuxCNC world.

Manufacturer will give the config in the standard way, and you will have to translate it for yourself.

For example, I made a xml file to work with that arduino shield , but I died while trying to interface with a Weidmuller EtherCAT module (again, it was a couple of years ago, and I am a very beginner).

talking about the Raspberry, I have seen everyone avoided it's Ethernet controller is on the USB-bus, so it's inherently non-real-time. Other boards available are the asus Tinkerboard, or the banana Pi, but I have seen few people using that alternatives.

mngr

[Chris Albertson]

Again the first step is to define the kind of things the machine is going to do.  Generally there is a divide between machines that run a fixed script and those that react to what they see in the environment and generate trajectories in real-time.

Then decide what you are going to use to build the controller with.   Today I would use standardized MCU development boards.  These cost between $2 and $16 each.  Connect them with serial links (CAN BUS, Ethercat, SPI, I2C or whatever...)    Then use a generic PC/Mac/iPhone browser interface for the user interface and display.  You might even allow multiple screens.  I can imagine a big monitor on the wall and a hand held controller running in an Android tablet.

But you have to chose al this up front.

I hate to say it but the "run everything on one big computer" design that MK and LCNC use is not something I'd use in a new design.  

Also, to avoid massive wheel re-invention run an RTOS on the micro controllers.  It will handle the details of running timers, I/O devices bus protocols and task scheduling and allows you to swap out hardware with no code change.

The last question to answer is "Why?" what will your controller do that others don't?

--
website: http://www.machinekit.io blog: http://blog.machinekit.io github: https://github.com/machinekit
---
You received this message because you are subscribed to the Google Groups "Machinekit" group.
To unsubscribe from this group and stop receiving emails from it, send an email to machinekit+...@googlegroups.com.
Visit this group at https://groups.google.com/group/machinekit.
For more options, visit https://groups.google.com/d/optout.

--

Chris Albertson
Redondo Beach, California

[Joshua Dickerson]

Thanks for all the suggestions, Chris!  I would have to say you truly have the mindset of a developer, all important questions to address before any major undertaking.

I have ported my machine controller over to use FreeRTOS, totally agree about not reinventing the wheel there.  Right now the sole purpose of my machine controller is to educate myself on machine controllers and industrial control systems.  Some of the different architectures out there make more sense to me, but only after my own attempts at writing my own. I also chose the particular route I'm on because it's more challenging (and possibly more flexible.)  Also, sometimes solving problems like this lead to breakthroughs for other related problems, right?

What would be cool is to have a very light weight, real-time equivalent of SCADA.  Something with a centralized "DB" so the data is always consistent, some provision for logging and monitoring to alert when things goes wrong.  In the immediate future, I am leaning towards a shared memory node that all the other systems go through. Might be a good starting point before figuring out something more distributed. 

[Joshua Dickerson]

Wow thanks, Charles!  Never heard of Zero-MQ, totally diving into the material now! :)  Also reading into haltalk so I can better understand how it works.

Hopefully I can learn enough to contribute meaningfully to the project!  Thanks again!

[Joshua Dickerson]

Hi there!  Thanks so much for your suggestions, currently looking into and will get back to everyone on what I learn.