ANN: POSIXClock - writing hard real-time Julia code

[Islam Badreldin]

Hi,

I'm excited to announce the first release of POSIXClock, a package that provides Julia bindings to clock_*() functions from POSIX real-time extensions (librt on Linux).
https://github.com/ibadr/POSIXClock.jl

But first, a full disclosure: I'm very new to Julia programming as well as to real-time programming. That being said, I think this first release is working as expected, and I tested it on a laptop powered by Intel Celeron processor running Julia v0.4.5 as well as on a BeagleBone Black board powered by AM335x armv7-based processor running Julia v0.5.0-dev nightly build.

The focus of the first release was on showing a proof-of-concept demonstration that it's possible to execute hard real-time Julia code, with emphasis on wrapping the clock_gettime() and the clock_nanosleep() functions using CLOCK_MONOTONIC and absolute-time sleeps. Special care was devoted to completely avoiding memory allocations in the real-time section of the code by using in-place operations and pre-allocating all the needed variables, as well as by disabling the garbage collector. The latency histogram (see README.md) demonstrates such hard real-time functionality, with worst-case latency of 40 us on a simple Intel Celeron processor. A similar histogram was obtained for the BeagleBone Black, albeit with worst-case latency of up to 140 us.

This package should appeal to roboticists interested in Julia (I have successfully tested this package with blinking a GPIO on the BeagleBone Black using the mraa library), as well as to scientists conducting closed-loop experiments with soft or hard real-time requirements. (Hard real-time performance requires a recent Linux kernel with the PREEMPT_RT patch.)

In addition to this announcement, I have a couple of questions pertaining to best practices for writing real-time Julia code and avoiding memory allocations, as well as to sharing arrays between two Julia instances (one is real-time and the other is regular). Is it best to post these questions in this thread, or to create a new thread for these questions?

Cheers,
Islam

[Páll Haraldsson]

On Monday, June 6, 2016 at 8:07:22 AM UTC, Islam Badreldin wrote:

Hi,

I'm excited to announce the first release of POSIXClock, a package that provides Julia bindings to clock_*() functions from POSIX real-time extensions (librt on Linux).
https://github.com/ibadr/POSIXClock.jl

Accurate clock might be of interest to not only real-time computing? [Unclear to me if librt/clock_*() works without PREEMPT_RT patch.] And of interest to e.g. the real-time satellite telescope in Julia.. see link below:
 

Special care was devoted to completely avoiding memory allocations in the real-time section of the code by using in-place operations and pre-allocating all the needed variables, as well as by disabling the garbage collector.

I'm not convinced disabling the GC is needed [or even helpful when there are allocations, see my rambling at link below and comments from others and pointers to Julia issues.]

 

This package should appeal to roboticists interested in Julia (I have successfully tested this package with blinking a GPIO on the BeagleBone Black using the mraa library), as well as to scientists conducting closed-loop experiments with soft or hard real-time requirements. (Hard real-time performance requires a recent Linux kernel with the PREEMPT_RT patch.)

Note in the link below, a link to Linus Torvalds on real-time Linux. Caches are a problem. Be careful, even with this patch..
 

In addition to this announcement, I have a couple of questions pertaining to best practices for writing real-time Julia code and avoiding memory allocations, as well as to sharing arrays between two Julia instances (one is real-time and the other is regular). Is it best to post these questions in this thread, or to create a new thread for these questions?

Feel free to look at this recent link on [soft, hard?] real-time and add questions there (since it's for a specific project, you may want to start a new thread):

https://groups.google.com/forum/#!topic/julia-users/qgnNbbuwMIY

[If you add to that thread, you could point there to this thread.]

--
Palli.

[Islam Badreldin]

Hi Páll,

Thanks for your reply. Please see below.

On Mon, Jun 6, 2016 at 5:30 AM, Páll Haraldsson <pall.ha...@gmail.com> wrote:

On Monday, June 6, 2016 at 8:07:22 AM UTC, Islam Badreldin wrote:

Hi,

I'm excited to announce the first release of POSIXClock, a package that provides Julia bindings to clock_*() functions from POSIX real-time extensions (librt on Linux).
https://github.com/ibadr/POSIXClock.jl

Accurate clock might be of interest to not only real-time computing? [Unclear to me if librt/clock_*() works without PREEMPT_RT patch.] And of interest to e.g. the real-time satellite telescope in Julia.. see link below:

AFAIK clock_*() exists and 'works' (in the sense of executing the code with no errors, but no real-time guarantees!) on any modern Linux system. However, whether it's able to meet the real-time demands is dependent on the Linux kernel configuration (CONFIG_PREEMPT_RT=y and friends; see https://rt.wiki.kernel.org/index.php/Cyclictest) _and_ the real-time priority of the user-space process that is making the calls (SCHED_FIFO is recommend with a priority above 49; same link).

 

 

Special care was devoted to completely avoiding memory allocations in the real-time section of the code by using in-place operations and pre-allocating all the needed variables, as well as by disabling the garbage collector.

I'm not convinced disabling the GC is needed [or even helpful when there are allocations, see my rambling at link below and comments from others and pointers to Julia issues.]

I agree that disabling the GC may not be strictly needed, but I thought to add it any way as a safety measure. I also add the ccall to mlockall() to provide a fast fail mechanism in case any unexpected memory allocations are encountered, in some sense that is similar to your proposed @nogc macro on the real-time astronomy thread.

 

 

This package should appeal to roboticists interested in Julia (I have successfully tested this package with blinking a GPIO on the BeagleBone Black using the mraa library), as well as to scientists conducting closed-loop experiments with soft or hard real-time requirements. (Hard real-time performance requires a recent Linux kernel with the PREEMPT_RT patch.)

Note in the link below, a link to Linus Torvalds on real-time Linux. Caches are a problem. Be careful, even with this patch..

I completely agree with you here! I'd say that in its current state, the real-time performance I'm expecting is more of 95% hard real-time. (see e.g. https://www.osadl.org/fileadmin/dam/rtlws/12/Brown.pdf)

That being said, I think the PREEMPT_RT patch has matured enough by now that the most of it made its way up to the vanilla Linux kernel. It got to the point where Xenomai 3 is providing the Mercury core to enable Xenomai-based applications to run on the vanilla Linux kernel (optionally compiled with PREEMPT_RT feature; see Single kernel configuration under https://xenomai.org//embedded-hardware/). I'm also planning to benchmark the stock low-latency kernel (CONFIG_PREEMPT_RT=y; a.k.a. soft real-time ) provided by Ubuntu and compare it to the CONFIG_PREEMPT_RT_FULL=y (a.k.a. hard real-time) kernel that I compiled.

 

 

In addition to this announcement, I have a couple of questions pertaining to best practices for writing real-time Julia code and avoiding memory allocations, as well as to sharing arrays between two Julia instances (one is real-time and the other is regular). Is it best to post these questions in this thread, or to create a new thread for these questions?

Feel free to look at this recent link on [soft, hard?] real-time and add questions there (since it's for a specific project, you may want to start a new thread):

https://groups.google.com/forum/#!topic/julia-users/qgnNbbuwMIY

[If you add to that thread, you could point there to this thread.]

I think I will add to that thread where appropriate, and otherwise will start a new thread for the specific questions that I have.

 

--
Palli.

Thanks,

Islam

[Uwe Fechner]

I am very happy about your work. I had the hope, that the built-in method sleep could be improved to reach this level of performance: https://github.com/JuliaLang/julia/issues/12770

But perhaps it is better to have this in a separate package.

Uwe

[Islam Badreldin]

Hi Uwe,

On Monday, June 6, 2016 at 12:55:28 PM UTC-4, Uwe Fechner wrote:

I am very happy about your work. I had the hope, that the built-in method sleep could be improved to reach this level of performance: https://github.com/JuliaLang/julia/issues/12770

But perhaps it is better to have this in a separate package.

Thank you! Yes, I'm aware of the issue you linked, and in fact I got convinced by the arguments of the Julia developers in there that this special use case may as well deserve its own package. You also mentioned tickless kernels in there, which I'm also interested in. I think one of the most promising configurations is to have a dedicated CPU core to the Julia real-time process using the isolcpus=1 kernel command line option. This is the configuration by which I got 200 us sleep with worst case latency of 40 us on a humble Intel Celeron dual core CPU as shown in the results of the 'rt_histogram.jl' example in the package. I also used the

CONFIG_NO_HZ_FULL=y  Kconfig option such that the CPU core dedicated to Julia is an adaptive-ticks CPU. I'm planning on documenting more of these details in the near future.

 

Uwe

Best,
Islam