Question about multiple modules talking to one adapter
Timur Tabi
We have a PCI adapter that supports three different APIs / interfaces. The adapter is an
OpenIB device, a regular 10GB ethernet Netdev device, and there's also a proprietary
interface called CCIL that we invented. The CCIL interface is really just a bunch of
custom IOCtls.
The problem we have is that our adapter appears as just one PCI device, so it has one
memory buffer and one IRQ. All three interfaces need access to adapter memory and need an
ISR.
We're going to have a separate module for each interface. For simplicity sake, I'll call
these openib.ko, netdev.ko, and ccil.ko. openib.ko will be in the drivers/inifiniband
directory. netdev.ko will be in the drivers/network directory. I don't know yet where
we'll put ccil.ko.
The way I see it, there are four different ways to implement these drivers, and I would
like to know which method the Linux community would prefer:
1) Each driver registers its own ISR and has its own mapping the adapter memory. This is
the simplest approach, but the problem is that every time an ISR is called, it does a read
across the PCI bus to determine whether the interrupt is for it. Believe it or not, this
is actually pretty expensive in terms of performance. If anyone has an idea on how to
cleanly solve that particular problem, then this is the approach we'll take. Otherwise,
we'd rather do implement one of the other two methods.
2) Create a single driver which does nothing but register an ISR and map the kernel
memory. Let's call this the CRM driver. The other three drivers can then use XXXXXX to
provide callbacks for the ISR and obtain the address of the mapping. The ISR will then
query the adapter and call the appropriate callback.
3) A variation on #2: Instead of having a separate driver with the CRM code, one of the
three modules will have that code. The other two modules will then have a dependency on
that first module. The problem is that I don't know which of the three modules should
have the CRM. Plus, this creates an artificial dependency.
4) Another variation on #2: Each module has a copy of the CRM code, but only the CRM in
the first module that's loaded is used. As each module loads, it tries to find one of the
other two modules. If it does find one of the other modules, it uses that other module's
CRM instead of its own. Otherwise, it exports its own CRM entry points.
Thanks.
--
Timur Tabi
Staff Software Engineer
timur...@ammasso.com
One thing a Southern boy will never say is,
"I don't think duct tape will fix it."
-- Ed Smylie, NASA engineer for Apollo 13
--
Kernelnewbies: Help each other learn about the Linux kernel.
Archive: http://mail.nl.linux.org/kernelnewbies/
FAQ: http://kernelnewbies.org/faq/
Jim Cromie
> Hi,
>
> We have a PCI adapter that supports three different APIs / interfaces.
> We're going to have a separate module for each interface. For
> simplicity sake, I'll call these openib.ko, netdev.ko, and ccil.ko.
> openib.ko will be in the drivers/inifiniband directory. netdev.ko will
> be in the drivers/network directory. I don't know yet where we'll put
> ccil.ko.
>
Excellent question - I look forward to learning the answer.
allow me to rephrase:
You have A,B,C, all of which want to own X
you suggest 3 alternatives:
(I answer in parens)
1. A,B,C are all independent.
this is inefficient cuz each has to look in detail at the card state to
determine
whether it should act upon the interrupt.
(IIUC, this is not interrupt-sharing per-se, but something else or is it?)
2. create P (parent), on which A,B,C are dependent
P has ISR code in it, A,B,C use that.
IOW, P export_symbols a register_isr_callback of some flavor,
A,B,C call it from their module_init()s (this is *the* dependency)
(this seems best. code is minimal, loaded footprint is probly also
close to minimal,
if only P+A are loaded, having P linked in statically is probly smaller.
One difficulty here is compiled-in vs loadable-module.
It seems simple to have them all the same - ie all modules, or all built-in.
I dont know how to do that with Kconfig depends statements.
having A be compiled, but B,C be loadable, makes for interesting
handling of P)
3. put ISR code M (for methods) in one of them , say A, so that B, C
use it.
thus B, C are dependent on A, since they use those exported symbols
(this seems best, *iff* you need other parts of A, but youve said you didnt.
it may come down to q of how big rest of A is, presumably the smallest
of the 3
(least kernel mem wasted on unused functionality) - this seems like a
poor criterion
for chosing which of ABC to put M code into.)
4, A,B,C each have copies of M
(I dont like this, I doubt real kernel folk will either. Its either
cut-paste (worse)
or a common .o file thats linked into each .ko (better - no cut-paste,
but..))
<related Q>
Is module loading/initialization safely serialized ?
if A, B are independent, and are simultaneously modprobed by 2 different
user-procs, is there locking to insure that one finishes before the
other is allowed to start ?
Assuming 2: given dependencies, P will load 1st, presumably 2nd
modprobe will
never try loading P, either cuz its locked out, or cuz it sees P inprogress.
Timur Tabi
> allow me to rephrase:
> You have A,B,C, all of which want to own X
> you suggest 3 alternatives:
>
> (I answer in parens)
>
> 1. A,B,C are all independent.
> this is inefficient cuz each has to look in detail at the card state to
> determine
> whether it should act upon the interrupt.
>
> (IIUC, this is not interrupt-sharing per-se, but something else or is it?)
No, it would be bona fide interrupt sharing. Each driver would register an ISR with the
same IRQ, and then whenever that IRQ is posted, the kernel would call each ISR in turn
until one handled it.
> 2. create P (parent), on which A,B,C are dependent
> P has ISR code in it, A,B,C use that.
> IOW, P export_symbols a register_isr_callback of some flavor,
> A,B,C call it from their module_init()s (this is *the* dependency)
>
> (this seems best. code is minimal, loaded footprint is probly also
> close to minimal,
Yes, this seems to be the consensus.
> if only P+A are loaded, having P linked in statically is probly smaller.
> One difficulty here is compiled-in vs loadable-module.
> It seems simple to have them all the same - ie all modules, or all
> built-in.
> I dont know how to do that with Kconfig depends statements.
> having A be compiled, but B,C be loadable, makes for interesting
> handling of P)
That's a good point. If Kconfig doesn't allow it, then we'll either have to tell the user
what configurations work, or force A, B, C to be loadable only, and P can be either static
or loadable.
> <related Q>
> Is module loading/initialization safely serialized ?
It should be, but I don't know enough yet about how the kernel knows which modules to load
when.
> if A, B are independent, and are simultaneously modprobed by 2 different
> user-procs, is there locking to insure that one finishes before the
> other is allowed to start ?
The only thing I was planning on having are semaphores to serialize two modules that
attempt to register their ISR callbacks simultaneously.
> Assuming 2: given dependencies, P will load 1st, presumably 2nd
> modprobe will
> never try loading P, either cuz its locked out, or cuz it sees P
> inprogress.
Yes, that's a requirement.
--
Timur Tabi
Staff Software Engineer
timur...@ammasso.com
One thing a Southern boy will never say is,
"I don't think duct tape will fix it."
-- Ed Smylie, NASA engineer for Apollo 13
--