Basic thread questions
Zerex71
I am writing a m/t application at work that basically deals with
several software-hardware interfaces. As one example, when the box
(Linux RHEL 5.1 in a ruggedized form factor) reads data from an RS-422
(serial) port from another box, it then converts the data into UDP
datagrams to be sent on to yet a third device. In my app, I have a
listener thread which pulls from the RS-422 card and interprets the
raw data coming in.
Now what I have discovered is in a test box which is driving my
RS-422, if I don't throttle the messaging, the test box (a PC with an
MFC app constantly throwing out 422 data) overwhelms my app and,
despite the fact that every one of my worker threads ends its while
loop with a sleep of equal duration (on the theory that sleep both
delays its return and yields to the next thread in line), another
thread which processes the incoming data and converts it to UDP
messages still gets hung up. For example, if the test box has a sleep
of 1ms, it slams my box with data so much that my 422-to-UDP thread
gets backlogged to the tune of several thousand messages.
When I adjust the test app's sleep to a 500ms delay, things are
considerably more calmed down and the 422-to-UDP thread is able to
keep up with the incoming data.
So my questions are,
1. Is there a way to mitigate this problem if on a real 422 box I'm
not able to adjust the data rate coming out?
2. Does sleep() actually wait the duration (I believe it does) AND
yield to other threads?
3. Is there a way to adjust thread priorities if need be if, in the
real system, I find that no matter what I do, having equal thread
priorities give me problems?
I should clarify by saying that our particular RS-422 interface
operates at a rate of 38.4 kHz but I have no idea the real rate that
test messages are being produced by the test box to send, i.e. I
assume it's on the order of 2 Hz with a 500ms delay and 1 kHz with a
1ms delay.
Thanks,
Mike
Eric Sosman
> Hi group,
>
> I am writing a m/t application at work that basically deals with
> several software-hardware interfaces. As one example, when the box
> (Linux RHEL 5.1 in a ruggedized form factor) reads data from an RS-422
> (serial) port from another box, it then converts the data into UDP
> datagrams to be sent on to yet a third device. In my app, I have a
> listener thread which pulls from the RS-422 card and interprets the
> raw data coming in.
>
> Now what I have discovered is in a test box which is driving my
> RS-422, if I don't throttle the messaging, the test box (a PC with an
> MFC app constantly throwing out 422 data) overwhelms my app and,
> despite the fact that every one of my worker threads ends its while
> loop with a sleep of equal duration (on the theory that sleep both
> delays its return and yields to the next thread in line), another
> thread which processes the incoming data and converts it to UDP
> messages still gets hung up. For example, if the test box has a sleep
> of 1ms, it slams my box with data so much that my 422-to-UDP thread
> gets backlogged to the tune of several thousand messages.
These "worker threads" you mention: Are they part of the
message-generating program or the message-transforming program?
That is, which box do they run on? (This may turn out to be
unimportant, but I'm trying to get a mental picture of your setup
and the focus is pretty blurry ...)
> When I adjust the test app's sleep to a 500ms delay, things are
> considerably more calmed down and the 422-to-UDP thread is able to
> keep up with the incoming data.
>
> So my questions are,
> 1. Is there a way to mitigate this problem if on a real 422 box I'm
> not able to adjust the data rate coming out?
Insufficient information.
> 2. Does sleep() actually wait the duration (I believe it does) AND
> yield to other threads?
The sleep() function I know of takes an interval specified in
seconds, so I don't understand how you're getting these shorter
durations. Also, the sleep() function I know of may wake up either
earlier or later than requested, for a number of reasons described
on the man page.
It is conceivable that a sleep() of very short duration would
be a no-op, or that a sleep() of an interval too short for the
system clock's resolution might be implemented as a busy loop (so
the thread wouldn't really sleep, just waste some time).
If a thread is not running, other threads certainly have the
opportunity to run. Whether they actually do or not is something
for them to discuss with the scheduler.
> 3. Is there a way to adjust thread priorities if need be if, in the
> real system, I find that no matter what I do, having equal thread
> priorities give me problems?
Again, where are these threads? If they're all doing pretty
much the same thing, juggling priorities is unlikely to make much
difference. General rule of thumb, applicable to most parallel
programs: Juggling priorities may alter performance, but seldom
fixes a bug.
> I should clarify by saying that our particular RS-422 interface
> operates at a rate of 38.4 kHz but I have no idea the real rate that
> test messages are being produced by the test box to send, i.e. I
> assume it's on the order of 2 Hz with a 500ms delay and 1 kHz with a
> 1ms delay.
You'd better find a way to measure it. If you don't, you're
wearing a blindfold.
David Schwartz
> Now what I have discovered is in a test box which is driving my
> RS-422, if I don't throttle the messaging, the test box (a PC with an
> MFC app constantly throwing out 422 data) overwhelms my app and,
> despite the fact that every one of my worker threads ends its while
> loop with a sleep of equal duration (on the theory that sleep both
> delays its return and yields to the next thread in line), another
> thread which processes the incoming data and converts it to UDP
> messages still gets hung up. For example, if the test box has a sleep
> of 1ms, it slams my box with data so much that my 422-to-UDP thread
> gets backlogged to the tune of several thousand messages.
I'm not sure exactly what your issue is, but the sleep's are the wrong
solution to it. When the 422-to-UDP thread gets backlogged, is the CPU
maxed out?
If the CPU is maxed out, the problem is either that you just don't
have enough CPU to keep up, or that you aren't doing the most
important work. You need to rig things so that if there are too many
backlogged messages, the threads that do less important work are not
ready-to-run. A barrier of some kind that can be raised when work
backs up and lowered when you catch up may be the right solution.
If the CPU is not maxed out, then the problem probably has nothing to
do with what other threads are doing and is simply because the 422-to-
UDP thread can't make forward progress for some reason. You need to
figure out why the 422-to-UDP thread is not ready to run. Is it
blocked on something?
Tweaking the sleep is like hitting an engine harder or software with a
hammer. The 'sleep' is the wrong tool for the job, and you haven't
figured out what's wrong.
DS