ach - Xenomai - Modeswitches

[Jan Carstensen]

Hi,

i am trying to use ach under ubuntu 14.4 with xenomai 2.6.4. I am able to compile ach using:

 ./configure CFLAGS="`xeno-config --skin=posix --cflags`" LDFLAGS="`xeno-config --skin=posix --ldflags`" CC="`xeno-config --skin=posix --cc`"

but when i do so ach, triggers a bunch of mode switches. These switches occur when using the library and also, when running achbench. I am not cross compiling.

What am i doing wrong?

Regards,
Jan

[Neil T. Dantam]

Hi Jan,

I have not examined mode switching under Xenomai. For achbench,
though, perhaps this is due to I/O when recording the latencies.
Would you be able to investigate this further to identify the source
of the mode switching?

Please note that kernel-space channels run in the Linux side rather
than in Xenomai's Adeos kernel, so they will not benefit from
Xenomai's reduced latency. Kernel-space channels do, however, work on
PREEMPT_RT.

What performance are you obtaining? Under Xenomai, you should have
latencies below 10 microseconds for user-space channels.

Cheers,
-ntd

> --
> You received this message because you are subscribed to the Google
> Groups "ach-help" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to ach-help+u...@googlegroups.com
> <mailto:ach-help+u...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[Jan Carstensen]

HI,

thanks for the quick response. Currently it seems to be a problem with xenomai. I was able to get a simple setup running without switches.

I am running all the communication on the Adeos Kernel, and have latencies around 16 us. I think this is due to the system is not very well tuned. (Still using a Nvidia driver). I will get our testsystem up and running soon and can post some benchmarks.

I am half way through switching the build system to catkin, after this is complete i will take a look into achbench.

Thanks,
Jan

[Neil T. Dantam]

On 06/22/2015 03:04 AM, Jan Carstensen wrote:
> I am running all the communication on the Adeos Kernel, and have
> latencies around 16 us.

What CPU is this? For an ARM or low-end Intel, that may be a reasonable
number. Higher-end Intel should be a little better.

Cheers,
-ntd

[Jan Carstensen]

I am using an i7-4790k.

I belive that xenomai is not tuned as it should be, i am trying to fix some issues. For instance the nvidia driver resulting in high system latencies.

I am setting up a test system with optimized settings, after i can get a basic example working in real-time over network i will start to optimize latency.

Regards,
Jan

[Jan Carstensen]

do you have some sources to how you got your xenomai setup up and not mode-switching?

i switched ach to catkin for a better ROS integration. And managed to get one sample up an running using rt_task_create(..) but when i try to switch the current task (rt_task_shadow(..)) which i can not trace.

What i have done so far:
1) used autotools again, to rule out porting errors
2) tried to figure out where the mode-switches occur.
-> ach_create, ach_open and ach_put all seem to produce some
3) tried achtest
-> A lot of Mode-switches

At his point i am a little out of ideas, since the Mode-switches cant be found using a sig_debug handler i cannot tell where exactly where they are coming from. Since i am pretty new to xenomai as well i am not sure where my mistake is.

Do you have any suggestions?

Regads,
Jan

[Neil T. Dantam]

> I am using an i7-4790k.

FWIW, I have that same CPU and in initial testing on PREEMPT_RT, I'm
getting 15-20us latency. This is slightly worse that what I saw
previously on a Xeon 1270v2.

> And managed to get one sample up an running using rt_task_create(..)
> but when i try to switch the current task (rt_task_shadow(..)) which
> i can not trace.

Is it acceptable to mix the Xenomai native and POSIX skins? When I
tested Ach on Xenomai, I used only the POSIX skin. (To preserve
portability, I think it is preferable to use POSIX whenever possible.)

> do you have some sources to how you got your xenomai setup up and not
> mode-switching?

I have not investigated mode switching under Xenomai. What I have
seen is that Ach on Xenomai had lower latency than PREEMPT_RT (and
that Xenomai had some stability issues).

> 2) tried to figure out where the mode-switches occur.
> -> ach_create, ach_open

ach_create() and ach_open() open/close files and handle memory
mapping. I can see that these may cause mode switching.

> and ach_put all seem to produce some/

The only system calls ach_put() performs deal with pthread mutexes and
condition variables. Assuming Xenomai has implemented these
appropriately, then this should not mode switch. HOWEVER, if there is
contention on the mutex, ach_put() may yield until the other mutex
holder finishes. Priority-inheriting mutexes are used (when
available) though, so there should be no priority inversion. Maybe if
a low-priority task gets scheduled out while holding the mutex, this
could cause a mode switch, so all processes accessing the channel may
need to have real-time priority.

> 3) tried achtest
> -> A lot of Mode-switches

achtest does I/O, so mode-switches are expected.

> Do you have any suggestions?

1. Does unaligned memory access on AMD64 cause a Xenomai to mode
switch? Ach does not (currently) make an effort to align memory
accesses (but this wouldn't be too hard to fix).

2. One question here is if and how Xenomai mutex and condition
variable operations can cause a mode switch. Those are the only
type of system calls that (userspace) ach_put() and ach_get()
perform. Does a contention on a mutex between two primary-mode
processes register as a mode-switch?

3. What are your latency requirements? If they can be met with
PREEMPT_RT, I think that would be a far easier option.

4. The Right Way (TM) to implement Ach on Xenomai is probably as an
RTDM kernel module similar to the current Linux kernel module.
This would make channels select()-able and hopefully bypass any
potential quirks in Xenomai pthreads synchronization.

Hope that helps...

Cheers,
-ntd

[Jan Carstensen]

Thank you for the input. This helped allot. I was able to get my test setup working. I used the POSIX skin and it worked. I was able to get a message latency of roughly 5 us when running without load and up to 18 us under full load.

I am using catkin as a build tool since i am using catkin as an real-time alternative to ROS publish subscribe. It works quite well but needs further testing.

Sorry for taking so long to reply.