Re: Changing Kernel thread priorities
Johannes Bauer
However, if you are running the system on an embedded platform, where the _WHOLE_ system (including priorities) is preconfigured and never touched, starting a irq thread with the right prio from start is a more straightforward method than having to invoke a script that changes it using userspace chrt tool.
Regards JB
----- Ursprüngliche Nachricht -----
Von: "Peter Zijlstra" <pet...@infradead.org>
Erhalten: 07.06.2011 00:36
An: hannes...@aon.at
"Monica Puig-Pey" wrote:
>
> I need to change the priority from inside the driver, when creating the
> kernel thread.
No you don't. How does you driver know about what priority is correct
wrt all the other running RT tasks on the system?
Determining the right priority in a fixed priority scheduling system is
a system wide problem, nobody but the administrator can possibly even
begin to solve it.
There's a reason all RT irq threads are started at 50, its plain
impossible to do better.
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Monica Puig-Pey
> Absolutly correct!
>
> However, if you are running the system on an embedded platform, where the _WHOLE_ system (including priorities) is preconfigured and never touched, starting a irq thread with the right prio from start is a more straightforward method than having to invoke a script that changes it using userspace chrt tool.
>
> Regards JB
> ----- Ursprüngliche Nachricht -----
> Von: "Peter Zijlstra"<pet...@infradead.org>
> Erhalten: 07.06.2011 00:36
> An: hannes...@aon.at
>
> "Monica Puig-Pey" wrote:
>>
>> I need to change the priority from inside the driver, when creating the
>> kernel thread.
>
> No you don't. How does you driver know about what priority is correct
> wrt all the other running RT tasks on the system?
>
> Determining the right priority in a fixed priority scheduling system is
> a system wide problem, nobody but the administrator can possibly even
> begin to solve it.
>
> There's a reason all RT irq threads are started at 50, its plain
> impossible to do better.
>
That's it!
If I work with embedded system where I know all my tasks running and
there is not a user how could I do it?
What I tried is create the kernel thread in my init_module using:
#include <linux/kthread.h>
struct task_struct *kthread_create(int (*threadfn)(void *data),
void *data,
const char namefmt[], ...)
and then running it with:
#include <linux/sched.h>
extern int wake_up_process(struct task_struct *tsk);
These functions stars a Kthread which has a NON RT priority. I can see
this using the ps command from user space.
Because it's not a real time thread is why I want, better need, to
change its priority, to have only real time threads running in my
driver. I want to use the Kthread as a bottom half for the interrupts.
How could I create real time kernel threads then? is kthread_create
wrong for real time enviroment?
--
__________________________________________________________________________________
Mónica Puig-Pey González E-mail: puig...@unican.es
Grupo de Computadores y Tiempo Real, Departamento de Electrónica y
Computadores.
Facultad de Ciencias - Universidad de Cantabria
Av. de los Castros s/n. 39005 - Santander, España
__________________________________________________________________________________
Thomas Gleixner
Please stop top-posting and use proper line breaks at 78
> > Peter Zijlstra wrote:
> > > "Monica Puig-Pey" wrote:
> > > I need to change the priority from inside the driver, when creating the
> > > kernel thread.
> >
> > No you don't. How does you driver know about what priority is correct
> > wrt all the other running RT tasks on the system?
> >
> > Determining the right priority in a fixed priority scheduling system is
> > a system wide problem, nobody but the administrator can possibly even
> > begin to solve it.
> >
> > There's a reason all RT irq threads are started at 50, its plain
> > impossible to do better.
> Absolutly correct!
>
> However, if you are running the system on an embedded platform,
> where the _WHOLE_ system (including priorities) is preconfigured and
> never touched, starting a irq thread with the right prio from start
> is a more straightforward method than having to invoke a script that
> changes it using userspace chrt tool.
Feel free to do that for your embedded system and carry the patch for
yourself if you think it's worth to avoid the extra init script.
But we do _not_ add stuff like this to the mainline simply because
there is no way to find a prio setting which is appropriate for all
users of a particular driver.
Aside of that the extra init script is definitely less annoying to
maintain than the crap you need to hack into random drivers.
Thanks,
tglx
Thomas Gleixner
> El 07/06/11 10:27, Johannes Bauer escribió:
> > impossible to do better.
> >
>
> That's it!
> If I work with embedded system where I know all my tasks running and there is
> not a user how could I do it?
init scripts run from user space and you can adjust the priority there.
> What I tried is create the kernel thread in my init_module using:
>
> #include <linux/kthread.h>
>
> struct task_struct *kthread_create(int (*threadfn)(void *data),
> void *data,
> const char namefmt[], ...)
> and then running it with:
>
> #include <linux/sched.h>
>
> extern int wake_up_process(struct task_struct *tsk);
>
> These functions stars a Kthread which has a NON RT priority. I can see this
> using the ps command from user space.
> Because it's not a real time thread is why I want, better need, to change its
> priority, to have only real time threads running in my driver. I want to use
> the Kthread as a bottom half for the interrupts.
Use threaded interrupt handlers. That's what they are made for.
Thanks,
tglx
Armin Steinhoff
Hi,
when I read all these confusing statements here ( in german it looks
like an "Eiertanz") ... I can only say:
- do the basic stuff in a minimal kernel driver
- use UIO (or VFIO for PCI devices)
and you get clean control about your real-time priorities.
I think changing the priorities of "interrupt threads" inside the kernel
could lead to strange race conditions in the kernel.
That seems to be the reason for that "Eiertanz" here :)
--Armin
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
Peter Zijlstra
> Hi,
>
> when I read all these confusing statements here ( in german it looks
> like an "Eiertanz") ... I can only say:
>
> - do the basic stuff in a minimal kernel driver
> - use UIO (or VFIO for PCI devices)
I see no requirement for any of those horrid things to be used. You can
write a full on proper kernel driver, it just cannot set kernel thread
priorities to a sane value (let them all default to 50 or so).
Then have a user space script or whatever set the kthread priorities.
> and you get clean control about your real-time priorities.
>
> I think changing the priorities of "interrupt threads" inside the kernel
> could lead to strange race conditions in the kernel.
No, changing the priority in the kernel is a perfectly sound operation,
it just doesn't make any sense to do so since its impossible to
determine a proper priority.
Therefore setting a priority is a pure user policy and should not be
done by the driver itself -- it simply cannot do it right so why bother
doing it.
Remy Bohmer
2011/6/7 Peter Zijlstra <pet...@infradead.org>:
> On Tue, 2011-06-07 at 11:40 +0200, Armin Steinhoff wrote:
>> Hi,
>>
>> when I read all these confusing statements here ( in german it looks
>> like an "Eiertanz") ... I can only say:
>>
>> - do the basic stuff in a minimal kernel driver
>> - use UIO (or VFIO for PCI devices)
>
> I see no requirement for any of those horrid things to be used. You can
> write a full on proper kernel driver, it just cannot set kernel thread
> priorities to a sane value (let them all default to 50 or so).
>
> Then have a user space script or whatever set the kthread priorities.
>> and you get clean control about your real-time priorities.
>> I think changing the priorities of "interrupt threads" inside the kernel
>> could lead to strange race conditions in the kernel.
Well, I 100% agree that it must be under full userspace control to be
able to set the priorities. But, the kernel default assumption of
starting everything at 50 is wrong as well.
Imagine the following situation:
* Realtime application is running and has threads active in the range
of prios 20 - 90.
* Now bring up a network device, it immediately starts spamming the
system at prio 50 _before_ you have the chance to set it below 20 by
means of chrt.
* RT behaviour is gone!
So, in that case and in many other hotplug cases, you ruin the RT
behaviour of the system just by the
default-50-is-probably-right-assumption of the kernel.
For systems where you have everything under control as a user/system
designer, hotplug can also be under control as well.
To solve this we have the patchset in use as attached to this mail. It
is a newer version of the old set mentioned earlier in this mail
thread.
The opinion of Thomas about this subject is quite clear so I will not
post it as a cleaned up patchset, although I believe we reworked all
his previous major remarks about this set.
For everyone else who can do something useful with it: go ahead. It
should apply to 2.6.33.9-rt31
It creates entries in /proc/irq for you to setup the priorities after
booting. (/proc/irq/hirq-prio and /proc/irq/sirq-prio), but also per
driver in /proc/irq/<irq-id>/<name>/irqprio
In /proc/irq/[hs]irq-prio you can, for example, enter the following text:
* at91_udc:22,ohci_hcd:usb1:22,atmel_spi:22,33
This results in:
* starting the at91_udc at prio 22
* starting ohci:usb1 also at prio 22
* atmel_spi at 22
* overall default moved from 50 to 33
Kind regards,
Remy
Peter Zijlstra
Good point I guess, Thomas should we default to 1 for everything?
Lucas De Marchi
> > Well, I 100% agree that it must be under full userspace control to be
> > able to set the priorities. But, the kernel default assumption of
> > starting everything at 50 is wrong as well.
> > Imagine the following situation:
> > * Realtime application is running and has threads active in the range
> > of prios 20 - 90.
> > * Now bring up a network device, it immediately starts spamming the
> > system at prio 50 _before_ you have the chance to set it below 20 by
> > means of chrt.
> > * RT behaviour is gone!
Why is the application using priorities in the range 20-90 if it's
well known that we default RT kernel threads to 50?
IMO it's a mistake in the application.
Lucas De Marchi
Thomas Gleixner
> On Tue, 2011-06-07 at 13:02 +0200, Remy Bohmer wrote:
> > Well, I 100% agree that it must be under full userspace control to be
> > able to set the priorities. But, the kernel default assumption of
> > starting everything at 50 is wrong as well.
> > Imagine the following situation:
> > * Realtime application is running and has threads active in the range
> > of prios 20 - 90.
> > * Now bring up a network device, it immediately starts spamming the
> > system at prio 50 _before_ you have the chance to set it below 20 by
> > means of chrt.
> > * RT behaviour is gone!
>
> Good point I guess, Thomas should we default to 1 for everything?
No objections.
Tim Sander
> Use threaded interrupt handlers. That's what they are made for.
I just started using threaded interrupt handlers in 2.6.39 for an
arm system (pcm043 - arm1136jfs). The results where not so good as
i expected. I set the rt prio of the interrupt to 99 (the scheduling
type made difference since it was the only interrupt at this level).
The user io driver for this where set to 98. I hadn't instrumented
the irqthread but the latencies where to high for our buffer which
can hold data for about 400µs. After configuring the IRQ as
non-threaded, but still using threaded irqs the latencies where fast
enough for the mentioned buffer size.
Nevertheless are there still scheduling latencies for the usermode
handler thread which then runs at rt prio 99 for about 1ms. I know
this is not an preempt-rt kernel but i hoped to get better values out
of this configuration.
So if anybody has an idea how to get better latencies out of a 2.6.39
kernel, please let me know.
I triead SLAB and SLOB allocator and they didn't made any difference for me.
Best regards
Tim
PS: i patched the kernel to get threadedirqs with:
diff --git a/arch/arm/mach-mx3/Kconfig b/arch/arm/mach-mx3/Kconfig
index 4e05237..28ae32c 100644
--- a/arch/arm/mach-mx3/Kconfig
+++ b/arch/arm/mach-mx3/Kconfig
@@ -162,6 +162,7 @@ config MACH_PCM043
select IMX_HAVE_PLATFORM_SDHCI_ESDHC_IMX
select IMX_HAVE_PLATFORM_SPI_IMX
select MXC_ULPI if USB_ULPI
+ select IRQ_FORCED_THREADING
help
Include support for Phytec pcm043 platform. This includes
specific configurations for the board and its peripherals.
Thomas Gleixner
> Nevertheless are there still scheduling latencies for the usermode
> handler thread which then runs at rt prio 99 for about 1ms. I know
> this is not an preempt-rt kernel but i hoped to get better values out
> of this configuration.
>
> So if anybody has an idea how to get better latencies out of a 2.6.39
> kernel, please let me know.
The mainline forced irq thread handling is no guarantee for lower
latencies. We need to disable preemption via local_bh_disable() for
the forced threaded interrupts to satisfy the handler vs. softirq
assumptions. So you might get long lasting preempt disabled regions
due to long running interrupt handlers. Aside of that you still can
get long periods due to code which runs with preemption or interrupts
disabled. The forced threaded option has no way to change that.
It would be interesting to find the root cause for those >1ms
latencies. Tracing is your friend.
Thanks,
tglx
Nicholas Mc Guire
> Hi All,
>
> 2011/6/7 Peter Zijlstra <pet...@infradead.org>:
> > On Tue, 2011-06-07 at 11:40 +0200, Armin Steinhoff wrote:
> >> Hi,
> >>
> >> when I read all these confusing statements here ( in german it looks
> >> like an "Eiertanz") ?... I can only say:
> >>
> >> - do the basic stuff in a minimal kernel driver
> >> - use UIO (or VFIO for PCI devices)
> >
> > I see no requirement for any of those horrid things to be used. You can
> > write a full on proper kernel driver, it just cannot set kernel thread
> > priorities to a sane value (let them all default to 50 or so).
> >
> > Then have a user space script or whatever set the kthread priorities.
> >> and you get clean control about your real-time priorities.
> >> I think changing the priorities of "interrupt threads" inside the kernel
> >> could lead to strange race conditions in the kernel.
>
> Well, I 100% agree that it must be under full userspace control to be
> able to set the priorities. But, the kernel default assumption of
> starting everything at 50 is wrong as well.
> Imagine the following situation:
> * Realtime application is running and has threads active in the range
> of prios 20 - 90.
> * Now bring up a network device, it immediately starts spamming the
> system at prio 50 _before_ you have the chance to set it below 20 by
> means of chrt.
> * RT behaviour is gone!
>
> So, in that case and in many other hotplug cases, you ruin the RT
> behaviour of the system just by the
> default-50-is-probably-right-assumption of the kernel.
> For systems where you have everything under control as a user/system
> designer, hotplug can also be under control as well.
>
I dont't quite see that - the 50 default is well dokumented so you can
plan it into the rt design at system level. It simply means that you
would need to put your hard-rt tasks in the range of 50<prio<99.
The actual value is quite irrelevant aslong as it is well defined, and
leaves a range free above suitably large for a rt task set (if you need
more than 5 distinct priorities for a rt task-set it generally means you
are using implicid locking any way)
I dont see the utility of adding a further interface that provides the same
level of configuration that the current user space tools do, and having one
interface for interrupt threads and one for tasks does not really simplify
things - from a rt-task-set perspective the differentiation betwen rt-tasks
and rt-related interrupt threads does not really make sens so keeping the
management in one interface seems more resonable to me.
hofrat
Nicholas Mc Guire
> On Tue, 7 Jun 2011, Peter Zijlstra wrote:
> > On Tue, 2011-06-07 at 13:02 +0200, Remy Bohmer wrote:
> > > Well, I 100% agree that it must be under full userspace control to be
> > > able to set the priorities. But, the kernel default assumption of
> > > starting everything at 50 is wrong as well.
> > > Imagine the following situation:
> > > * Realtime application is running and has threads active in the range
> > > of prios 20 - 90.
> > > * Now bring up a network device, it immediately starts spamming the
> > > system at prio 50 _before_ you have the chance to set it below 20 by
> > > means of chrt.
> > > * RT behaviour is gone!
> >
> > Good point I guess, Thomas should we default to 1 for everything?
>
> No objections.
I think that splitting the range makes more sense - you are now potentially
reverting the argument brought up before.
"My drivers behave properly until I load any RT-task"
hofrat
Remy Bohmer
>> So, in that case and in many other hotplug cases, you ruin the RT
>> behaviour of the system just by the
>> default-50-is-probably-right-assumption of the kernel.
>> For systems where you have everything under control as a user/system
>> designer, hotplug can also be under control as well.
>>
>
> I dont't quite see that - the 50 default is well dokumented so you can
> plan it into the rt design at system level. It simply means that you
> would need to put your hard-rt tasks in the range of 50<prio<99.
This is another assumption as well. You assume here that any task
bound to certain latencies must be above _all_ interrupt handlers, and
tasks not bound to certain latencies must be set below _all_ interrupt
handlers.
In fact, in real life, the prioritisation will be much more fine
grained during runtime. One can have a RT task that depends on the
networking stack, but not on mass-storage. In that case you want to
move them out of the range of 50.
The whole idea about threaded interrupts is, is that you can give them
priorities such that they suit your specific application. It is
therefore common practice that priorities does not stay at 49/50.
In real life you may want, for EXAMPLE, this setup:
* prio 70: high priority motor control loop
* prio 60: network device irq
* prio 59: network softirqs
* prio 55: some realtime task depending on networkingstack
* prio 54: mass storage irq
* prio 53: block device softirq
* prio 52: some realtime task depending on mass-storage
* prio 50: all remaining irq threads
* prio 49: all remaining softirqs
Assume here you do a ifconfig down and ifconfig up, in the current
kernel behaviour you will see that the irq thread switches from prio
60 to 50.
The irq-thread will become of a lower priority compared to its related
softirqs due to this reason, which can result in a complete die of
this network interface... even before it ever came back up again...
As mentioned before by Thomas, the configuration is a policy issue and
must be set from user-context. I understand what he means by that and
I agree, but there still has to be a mechanism to make the kernel
remember the configuration set by the user to prevent all kinds of
race conditions. You cannot demand from the user to run after
executing a shell command like ifconfig or modprobe to run some sort
of init-script that repairs what the kernel assumed wrong. The wrong
assumptions the kernel does results in: deadlocks, priority inversion
issues between irq-threads and softirqs and realtime behaviour impact.
Even UDEV cannot_fix_this_problem_ since it runs _after_ the kernel
has set the wrong priorities of the irq threads and the problem it
imposes already may have occurred.
Setting the priorities right once is already complicated enough, it
makes it far more complex if all kinds of race conditions and
limitations need to be taken into account due to this mentioned
auto-reset-to-50(-or-1)-assumptions of the kernel...
True, there must be a safe default for booting, and 49/50 is good
enough for that.
So, you might disagree the way our patches to solve this problem are
implemented, I can buy that, it is just at the level of 'works-for-me'
. Since this discussion frequently appears back on the mailinglist
makes clear that this _is_ an issue that is relevant. Instead of
ignoring or denying this issue, we should figure out what is the
_best_way_ to solve it.
I hope to see great and constructive suggestions soon on this list, I
am very willing to implement it :-)
Kind regards,
Remy
Thomas Gleixner
> In real life you may want, for EXAMPLE, this setup:
> * prio 70: high priority motor control loop
> * prio 60: network device irq
> * prio 59: network softirqs
> * prio 55: some realtime task depending on networkingstack
> * prio 54: mass storage irq
> * prio 53: block device softirq
> * prio 52: some realtime task depending on mass-storage
> * prio 50: all remaining irq threads
> * prio 49: all remaining softirqs
>
> Assume here you do a ifconfig down and ifconfig up, in the current
> kernel behaviour you will see that the irq thread switches from prio
> 60 to 50.
> The irq-thread will become of a lower priority compared to its related
> softirqs due to this reason, which can result in a complete die of
> this network interface... even before it ever came back up again...
Not really. If that's the case it needs to be investigated and
fixed.
> As mentioned before by Thomas, the configuration is a policy issue and
> must be set from user-context. I understand what he means by that and
> I agree, but there still has to be a mechanism to make the kernel
> remember the configuration set by the user to prevent all kinds of
> race conditions. You cannot demand from the user to run after
Which race conditions?
> executing a shell command like ifconfig or modprobe to run some sort
> of init-script that repairs what the kernel assumed wrong. The wrong
> assumptions the kernel does results in: deadlocks, priority inversion
> issues between irq-threads and softirqs and realtime behaviour impact.
If you do an ifdown/up then your prio 55 task is totally irrelevant
until the interface is back to full operation again, which includes
setting the priority right.
There is another gotcha with your approach. It only ever works when
the interrupt descriptors are static and not dynamically
allocated/freed. If they are fully dynamic then you have no
possibility to store the prio information after a full teardown of a
device.
So moving the base priority down to 1 or 2 is probably the most
sensible solution to avoid that a newly brought up interrupt thread
interferes with anything in the rt domain and it's not rocket science
to adjust the priority in a ifup.post or with an udev rule.
Thanks,
tglx
Nicholas Mc Guire
> Hi,
>
> >> So, in that case and in many other hotplug cases, you ruin the RT
> >> behaviour of the system just by the
> >> default-50-is-probably-right-assumption of the kernel.
> >> For systems where you have everything under control as a user/system
> >> designer, hotplug can also be under control as well.
> >>
> >
> > I dont't quite see that - the 50 default is well dokumented so you can
> > plan it into the rt design at system level. It simply means that you
> > would need to put your hard-rt tasks in the range of 50<prio<99.
>
> This is another assumption as well. You assume here that any task
> bound to certain latencies must be above _all_ interrupt handlers, and
> tasks not bound to certain latencies must be set below _all_ interrupt
> handlers.
> In fact, in real life, the prioritisation will be much more fine
> grained during runtime. One can have a RT task that depends on the
> networking stack, but not on mass-storage. In that case you want to
> move them out of the range of 50.
No implications are made on the "correct" priority - but as there is no
correct priority setting a well defined default and leaving it to the
user is the most resonable solution. The main issue with not changing it
from the current value is simply that it may change behavior of existing
systems that update and that is, if it provides no obvious benifit, not
a good thing to happen.
but you equally can not assume that a device comming back on-line always
should have the same priority. i.e. DAQ-card is opened by process A prioX
and then closed to be reopened by procsee B prio Y - the priority of the
DAQ-card IRQ thread is not related and persistence here is dependant on
the particular circumstances.
> Even UDEV cannot_fix_this_problem_ since it runs _after_ the kernel
> has set the wrong priorities of the irq threads and the problem it
> imposes already may have occurred.
>
> Setting the priorities right once is already complicated enough, it
> makes it far more complex if all kinds of race conditions and
> limitations need to be taken into account due to this mentioned
> auto-reset-to-50(-or-1)-assumptions of the kernel...
> True, there must be a safe default for booting, and 49/50 is good
> enough for that.
>
> So, you might disagree the way our patches to solve this problem are
> implemented, I can buy that, it is just at the level of 'works-for-me'
> . Since this discussion frequently appears back on the mailinglist
> makes clear that this _is_ an issue that is relevant. Instead of
> ignoring or denying this issue, we should figure out what is the
> _best_way_ to solve it.
> I hope to see great and constructive suggestions soon on this list, I
> am very willing to implement it :-)
>
so maybe a comlet set of requirements is needed first rather than ad-hoc
solutions for special cases - Im quite sure the non-persistant case is only
one of a few issues one can find here.
hofrat
Remy Bohmer
2011/6/8 Thomas Gleixner <tg...@linutronix.de>:
> On Wed, 8 Jun 2011, Remy Bohmer wrote:
>> In real life you may want, for EXAMPLE, this setup:
>> * prio 70: high priority motor control loop
>> * prio 60: network device irq
>> * prio 59: network softirqs
>> * prio 55: some realtime task depending on networkingstack
>> * prio 54: mass storage irq
>> * prio 53: block device softirq
>> * prio 52: some realtime task depending on mass-storage
>> * prio 50: all remaining irq threads
>> * prio 49: all remaining softirqs
>>
>> Assume here you do a ifconfig down and ifconfig up, in the current
>> kernel behaviour you will see that the irq thread switches from prio
>> 60 to 50.
>> The irq-thread will become of a lower priority compared to its related
>> softirqs due to this reason, which can result in a complete die of
>> this network interface... even before it ever came back up again...
>
> Not really. If that's the case it needs to be investigated and
> fixed.
I, of course, agree with that, but these cases are usually extremely
hard to find, and occur typically only in the once-a-month-condition
that you cannot reproduce...
Do you remember why the priority of the softirqs was moved down from
50 to 49 ? IIRC this was because of the very same reason and IIRC
still valid
We do not have control over all kernel code, and new drivers are
continuously being developed that make wrong implicit assumptions
about the order of irq->sirq->everything else. Of course this is
wrong, and there is no excuse, but it is a fact of life...
In practice the softirq prio can be set to a higher value than 50 (or
1), and a hirq thread that is started at 50 (or 2) will result in
situations that are not expected.
>> As mentioned before by Thomas, the configuration is a policy issue and
>> must be set from user-context. I understand what he means by that and
>> I agree, but there still has to be a mechanism to make the kernel
>> remember the configuration set by the user to prevent all kinds of
>> race conditions. You cannot demand from the user to run after
>
> Which race conditions?
Race conditions that occur when a softirq preempts a related hardirq
what the driver did not expect or was designed for.
>> executing a shell command like ifconfig or modprobe to run some sort
>> of init-script that repairs what the kernel assumed wrong. The wrong
>> assumptions the kernel does results in: deadlocks, priority inversion
>> issues between irq-threads and softirqs and realtime behaviour impact.
>
> If you do an ifdown/up then your prio 55 task is totally irrelevant
> until the interface is back to full operation again, which includes
> setting the priority right.
I already expected that remark after I pressed the send button of that
mail... This was just meant as an example, in which you can probably
shoot more holes in. It is not about the example, it is about the
essence of what I am trying to explain here.
> There is another gotcha with your approach. It only ever works when
> the interrupt descriptors are static and not dynamically
> allocated/freed. If they are fully dynamic then you have no
> possibility to store the prio information after a full teardown of a
> device.
It depends how it is being implemented.
A mechanism to specify the policy does not mean everything has to be
already in place the policy is about at the moment you specify the
policy.
In other words, a policy may describe situations that are going to
happen in the future, not necessarily situations that are actual now.
For example, something like this:
* A user specifies a table with policy information about what each
interrupt handler in the system should do when they are being created.
* When the interrupt handler is being installed, it is looked up in
the table at what priority and scheduling policy it needs to run. If
not specified, go for a default.
* Additionally: When the table is being updated, the already running
threads can being adjusted to the new policy.
> So moving the base priority down to 1 or 2 is probably the most
> sensible solution to avoid that a newly brought up interrupt thread
> interferes with anything in the rt domain and it's not rocket science
> to adjust the priority in a ifup.post or with an udev rule.
At prio 1 or 2, _every_ RT-thread in the system is to be assumed to be
more low-latency bound compared to _any_ interrupt handler. And you
assume here that no user RT-thread in the system shall use any
functionality of any driver that has an interrupt handler (otherwise
you get the priority inversions issue)
As mentioned in this thread before by someone else, you will get this
old issue back: 'My drivers start to behave weird when I create a
RT-thread...'
The prio inversion issue between hirq/sirq will even become more
worse, since there will be a smaller chance that softirqs will stay at
prio 1 and thus there is less guarantee that they will stay below the
hirq-prio all the time.
Furthermore, I prefer the principle: _Nothing_ goes above interrupt
(thread) priority unless there is a very special reason for it and it
has been investigated that it is safe to do so. And a user-thread that
requires functionality of a certain driver shall be set below the
priority of the hirq-thread of that driver. The prio of the softirq
must _always_ be between that user-thread and hirq-thread if there is
a relation between the driver and softirq.
In that light I think prio 1/2 is more worse compared to 49/50. I
think the current _default_ is okay, it makes the system at least
boot.
Kind regards,
Remy
Thomas Gleixner
No, it's not. The root cause was a problem with the network softirq
and a network driver, the softirq ->49 was a temporary workaround
until we had enough information to find the real root cause. I wish
I'd never committed that change at all.
> We do not have control over all kernel code, and new drivers are
> continuously being developed that make wrong implicit assumptions
> about the order of irq->sirq->everything else. Of course this is
> wrong, and there is no excuse, but it is a fact of life...
> In practice the softirq prio can be set to a higher value than 50 (or
> 1), and a hirq thread that is started at 50 (or 2) will result in
> situations that are not expected.
>
> >> As mentioned before by Thomas, the configuration is a policy issue and
> >> must be set from user-context. I understand what he means by that and
> >> I agree, but there still has to be a mechanism to make the kernel
> >> remember the configuration set by the user to prevent all kinds of
> >> race conditions. You cannot demand from the user to run after
> >
> > Which race conditions?
>
> Race conditions that occur when a softirq preempts a related hardirq
> what the driver did not expect or was designed for.
And making it the other way round hides the problem, which is even
worse. We want stuff to explode right away. You can run into the same
problem when the softirq holds a lock and the high prio irq thread
boosts it.
> > So moving the base priority down to 1 or 2 is probably the most
> > sensible solution to avoid that a newly brought up interrupt thread
> > interferes with anything in the rt domain and it's not rocket science
> > to adjust the priority in a ifup.post or with an udev rule.
>
> At prio 1 or 2, _every_ RT-thread in the system is to be assumed to be
> more low-latency bound compared to _any_ interrupt handler. And you
> assume here that no user RT-thread in the system shall use any
> functionality of any driver that has an interrupt handler (otherwise
> you get the priority inversions issue)
Sigh. People who use RT threads should better know what they do and
configure their damned system correct. We cannot provide a solution
which takes every incarnatation of lusers into account.
> As mentioned in this thread before by someone else, you will get this
> old issue back: 'My drivers start to behave weird when I create a
> RT-thread...'
And I do not care at all. The answer is: Do not use an RT-thread when
you are not knowing what you are doing.
> The prio inversion issue between hirq/sirq will even become more
> worse, since there will be a smaller chance that softirqs will stay at
> prio 1 and thus there is less guarantee that they will stay below the
> hirq-prio all the time.
There is no such thing and if it's there, then it needs to be found
and fixed.
> Furthermore, I prefer the principle: _Nothing_ goes above interrupt
> (thread) priority unless there is a very special reason for it and it
> has been investigated that it is safe to do so. And a user-thread that
> requires functionality of a certain driver shall be set below the
> priority of the hirq-thread of that driver. The prio of the softirq
> must _always_ be between that user-thread and hirq-thread if there is
> a relation between the driver and softirq.
>
> In that light I think prio 1/2 is more worse compared to 49/50. I
> think the current _default_ is okay, it makes the system at least
> boot.
It boots with 50 or whatever you set it to as well.
Thanks,
tglx
Remy Bohmer
> No, it's not. The root cause was a problem with the network softirq
> and a network driver, the softirq ->49 was a temporary workaround
> until we had enough information to find the real root cause. I wish
> I'd never committed that change at all.
Clear. Did not know it was already solved. I thought it was still an
issue. This changes things :-)
>> Race conditions that occur when a softirq preempts a related hardirq
>> what the driver did not expect or was designed for.
>
> And making it the other way round hides the problem, which is even
> worse. We want stuff to explode right away.
100% Agreed
> You can run into the same
> problem when the softirq holds a lock and the high prio irq thread
> boosts it.
OK.
Thanks for the explanation. I see no reason any more why setting the
prios default to 1 would be a bad thing.
The rest of the configuration in that case can then indeed done be
done by udev and other userland friends.
Kind regards,
Remy