I am facing some issues regarding kernel module compliancy with a dedicated
kernel. I am working on an embedded platform with a 2.6.22.6 kernel that has
been built by someone else, and for several reasons not worth explaining, I
can't change the kernel on this platform. Problem is, I needed to use usblp
on this platform, but the usblp.ko driver was missing. So I downloaded a
source tree for 2.6.22.6, I altered the config to include USB printer
support, and I ran a build which produced (among many things) a shiny new
usblp.ko module. So far, so good. However, when I candidly tried to insmod'
this module on my platform, this resulted in a bunch of error messages:
===================================================
root@platform:~# insmod
/lib/modules/2.6.22.6/kernel/drivers/usb/class/usblp.ko
[ 4239.380000] usblp: disagrees about version of symbol usb_alloc_urb
[ 4239.390000] usblp: Unknown symbol usb_alloc_urb
[ 4239.410000] usblp: disagrees about version of symbol usb_free_urb
[ 4239.420000] usblp: Unknown symbol usb_free_urb
<snip>
[ 4239.620000] usblp: disagrees about version of symbol usb_buffer_alloc
[ 4239.630000] usblp: Unknown symbol usb_buffer_alloc
[ 4239.640000] usblp: disagrees about version of symbol device_remove_file
[ 4239.640000] usblp: Unknown symbol device_remove_file
insmod: cannot insert
'/lib/modules/2.6.22.6/kernel/drivers/usb/class/usblp.ko': unknown symbol in
module, or unknown parameter
root@platform:~#
===================================================
Needless to say that the upsetting symbols exist in the kernel:
===================================================
root@platform:~# cat /proc/kallsyms | grep usb_alloc
c01c9d40 T usb_alloc_dev
c01cffa0 T usb_alloc_urb
c0357280 r __ksymtab_usb_alloc_urb
c035be68 r __kcrctab_usb_alloc_urb
c0365f63 r __kstrtab_usb_alloc_urb
root@platform:~#
===================================================
Well, I have since managed to get the config used by the developer who built
the installed kernel, I have rebuilt usblp.ko with this config, and it loads
just fine now. Nevertheless, I don't feel comfortable with this
"disagreement" between kernel and module. I have searched the Net (including
this newsgroup), finding only scarce info on this subject, and I have
examined the sources, which vaguely pointed me to an issue associated with
kernel symbol table mismatch. I'm sure there is a solid and valid reason for
this, but still, it seems weird to me that someone has to rebuild / update
all kernel modules just because of a slight change in the kernel config.
Please, could someone point me at a document where this match/mismatch stuff
is explained ?
Many thanks in advance !
--
JPI (e-mail address invalid)
[...]
> ===================================================
> root@platform:~# insmod
> /lib/modules/2.6.22.6/kernel/drivers/usb/class/usblp.ko
> [ 4239.380000] usblp: disagrees about version of symbol usb_alloc_urb
> [ 4239.390000] usblp: Unknown symbol usb_alloc_urb
> [ 4239.410000] usblp: disagrees about version of symbol usb_free_urb
> [ 4239.420000] usblp: Unknown symbol usb_free_urb
>
> <snip>
>
> [ 4239.620000] usblp: disagrees about version of symbol usb_buffer_alloc
> [ 4239.630000] usblp: Unknown symbol usb_buffer_alloc
> [ 4239.640000] usblp: disagrees about version of symbol device_remove_file
> [ 4239.640000] usblp: Unknown symbol device_remove_file
> insmod: cannot insert
> '/lib/modules/2.6.22.6/kernel/drivers/usb/class/usblp.ko': unknown symbol in
> module, or unknown parameter
> root@platform:~#
> ===================================================
[...]
> Well, I have since managed to get the config used by the developer who built
> the installed kernel,
This is just a guess since I have never used this feature myself, but
could it be that one configuration had module versioning enabled and
the other didn't?
Hi Rainer, thanks for your messsage.
Nope, module versioning ("CONFIG_MODVERSIONS=y") was enabled in both builds.
Well, to be exhaustive, here is the diff between the config that doesn't
work and the new one that works:
=========================================
diff -r1.1 defconfig
4c4
< # Tue Feb 22 09:00:25 2011
---
> # Sat May 14 11:15:35 2011
41c41
< CONFIG_LOCALVERSION_AUTO=y
---
> # CONFIG_LOCALVERSION_AUTO is not set
165c165
< CONFIG_MX27_FEC_POLLING=y
---
> # CONFIG_MX27_FEC_POLLING is not set
171c171
< CONFIG_I2C_MXC_SELECT1=y
---
> # CONFIG_I2C_MXC_SELECT1 is not set
173,175c173,175
< # CONFIG_UART_MXC_HWFLOW1 is not set
< # CONFIG_UART_MXC_HWFLOW2 is not set
< # CONFIG_UART_MXC_HWFLOW3 is not set
---
> CONFIG_UART_MXC_HWFLOW1=y
> CONFIG_UART_MXC_HWFLOW2=y
> CONFIG_UART_MXC_HWFLOW3=y
246,248c246
< CONFIG_DPM=y
< # CONFIG_DPM_STATS is not set
< CONFIG_DPM_PROCFS=y
---
> # CONFIG_DPM is not set
281,285c279
< CONFIG_PM=y
< # CONFIG_PM_LEGACY is not set
< # CONFIG_PM_DEBUG is not set
< # CONFIG_PM_SYSFS_DEPRECATED is not set
< # CONFIG_APM_EMULATION is not set
---
> # CONFIG_PM is not set
334a329
> # CONFIG_IP_VS is not set
339c334,432
< # CONFIG_NETFILTER is not set
---
> CONFIG_NETFILTER=y
> # CONFIG_NETFILTER_DEBUG is not set
>
> #
> # Core Netfilter Configuration
> #
> CONFIG_NETFILTER_NETLINK=m
> # CONFIG_NETFILTER_NETLINK_QUEUE is not set
> # CONFIG_NETFILTER_NETLINK_LOG is not set
> CONFIG_NF_CONNTRACK_ENABLED=m
> CONFIG_NF_CONNTRACK=m
> CONFIG_NF_CT_ACCT=y
> CONFIG_NF_CONNTRACK_MARK=y
> # CONFIG_NF_CONNTRACK_EVENTS is not set
> # CONFIG_NF_CT_PROTO_SCTP is not set
> # CONFIG_NF_CONNTRACK_AMANDA is not set
> # CONFIG_NF_CONNTRACK_FTP is not set
> # CONFIG_NF_CONNTRACK_H323 is not set
> # CONFIG_NF_CONNTRACK_IRC is not set
> # CONFIG_NF_CONNTRACK_NETBIOS_NS is not set
> # CONFIG_NF_CONNTRACK_PPTP is not set
> # CONFIG_NF_CONNTRACK_SANE is not set
> # CONFIG_NF_CONNTRACK_SIP is not set
> # CONFIG_NF_CONNTRACK_TFTP is not set
> # CONFIG_NF_CT_NETLINK is not set
> CONFIG_NETFILTER_XTABLES=m
> CONFIG_NETFILTER_XT_TARGET_CLASSIFY=m
> # CONFIG_NETFILTER_XT_TARGET_CONNMARK is not set
> # CONFIG_NETFILTER_XT_TARGET_DSCP is not set
> CONFIG_NETFILTER_XT_TARGET_MARK=m
> CONFIG_NETFILTER_XT_TARGET_NFQUEUE=m
> CONFIG_NETFILTER_XT_TARGET_NFLOG=m
> CONFIG_NETFILTER_XT_TARGET_TCPMSS=m
> CONFIG_NETFILTER_XT_MATCH_COMMENT=m
> # CONFIG_NETFILTER_XT_MATCH_CONNBYTES is not set
> # CONFIG_NETFILTER_XT_MATCH_CONNMARK is not set
> # CONFIG_NETFILTER_XT_MATCH_CONNTRACK is not set
> CONFIG_NETFILTER_XT_MATCH_DCCP=m
> CONFIG_NETFILTER_XT_MATCH_DSCP=m
> CONFIG_NETFILTER_XT_MATCH_ESP=m
> # CONFIG_NETFILTER_XT_MATCH_HELPER is not set
> CONFIG_NETFILTER_XT_MATCH_LENGTH=m
> CONFIG_NETFILTER_XT_MATCH_LIMIT=m
> CONFIG_NETFILTER_XT_MATCH_MAC=m
> CONFIG_NETFILTER_XT_MATCH_MARK=m
> # CONFIG_NETFILTER_XT_MATCH_POLICY is not set
> # CONFIG_NETFILTER_XT_MATCH_MULTIPORT is not set
> # CONFIG_NETFILTER_XT_MATCH_PKTTYPE is not set
> # CONFIG_NETFILTER_XT_MATCH_QUOTA is not set
> # CONFIG_NETFILTER_XT_MATCH_REALM is not set
> # CONFIG_NETFILTER_XT_MATCH_SCTP is not set
> # CONFIG_NETFILTER_XT_MATCH_STATE is not set
> # CONFIG_NETFILTER_XT_MATCH_STATISTIC is not set
> # CONFIG_NETFILTER_XT_MATCH_STRING is not set
> # CONFIG_NETFILTER_XT_MATCH_TCPMSS is not set
> # CONFIG_NETFILTER_XT_MATCH_HASHLIMIT is not set
>
> #
> # IP: Netfilter Configuration
> #
> CONFIG_NF_CONNTRACK_IPV4=m
> CONFIG_NF_CONNTRACK_PROC_COMPAT=y
> # CONFIG_IP_NF_QUEUE is not set
> CONFIG_IP_NF_IPTABLES=m
> CONFIG_IP_NF_MATCH_IPRANGE=m
> CONFIG_IP_NF_MATCH_TOS=m
> CONFIG_IP_NF_MATCH_RECENT=m
> CONFIG_IP_NF_MATCH_ECN=m
> CONFIG_IP_NF_MATCH_AH=m
> CONFIG_IP_NF_MATCH_TTL=m
> CONFIG_IP_NF_MATCH_OWNER=m
> CONFIG_IP_NF_MATCH_ADDRTYPE=m
> CONFIG_IP_NF_FILTER=m
> CONFIG_IP_NF_TARGET_REJECT=m
> CONFIG_IP_NF_TARGET_LOG=m
> CONFIG_IP_NF_TARGET_ULOG=m
> CONFIG_NF_NAT=m
> CONFIG_NF_NAT_NEEDED=y
> CONFIG_IP_NF_TARGET_MASQUERADE=m
> CONFIG_IP_NF_TARGET_REDIRECT=m
> CONFIG_IP_NF_TARGET_NETMAP=m
> CONFIG_IP_NF_TARGET_SAME=m
> # CONFIG_NF_NAT_SNMP_BASIC is not set
> # CONFIG_NF_NAT_FTP is not set
> # CONFIG_NF_NAT_IRC is not set
> # CONFIG_NF_NAT_TFTP is not set
> # CONFIG_NF_NAT_AMANDA is not set
> # CONFIG_NF_NAT_PPTP is not set
> # CONFIG_NF_NAT_H323 is not set
> # CONFIG_NF_NAT_SIP is not set
> CONFIG_IP_NF_MANGLE=m
> CONFIG_IP_NF_TARGET_TOS=m
> CONFIG_IP_NF_TARGET_ECN=m
> CONFIG_IP_NF_TARGET_TTL=m
> # CONFIG_IP_NF_TARGET_CLUSTERIP is not set
> # CONFIG_IP_NF_RAW is not set
> CONFIG_IP_NF_ARPTABLES=m
> CONFIG_IP_NF_ARPFILTER=m
> CONFIG_IP_NF_ARP_MANGLE=m
734a828
> CONFIG_I2CARMSLAVE=m
742c836
< CONFIG_I2C_ALGOBIT=y
---
> # CONFIG_I2C_ALGOBIT is not set
751c845,846
< CONFIG_I2C_MXC_GPIO=y
---
> CONFIG_I2C_SLAVE_MXC=m
> # CONFIG_I2C_MXC_GPIO is not set
876c971
< CONFIG_MXC_CAMERA_OV7675=m
---
> CONFIG_MXC_CAMERA_OV7740=m
955a1051
> CONFIG_SND_AC97_CODEC=y
963a1060,1061
> CONFIG_SND_MX27_PCM=y
> CONFIG_SND_MX27_AC97=y
976,977c1074,1075
< CONFIG_SND_MXC_SOC_SSI=y
< CONFIG_SND_SOC_IMX27_CX20703=y
---
> CONFIG_SND_MXC_SOC_SSI=m
> CONFIG_SND_SOC_IMX27_CX20703=m
979c1077
< CONFIG_SND_SOC_CX20703=y
---
> CONFIG_SND_SOC_CX20703=m
984a1083
> CONFIG_AC97_BUS=y
1001d1099
< CONFIG_USB_SUSPEND=y
1132a1231
> # CONFIG_MMC_MXC_GPIO_DETECT is not set
1237a1337
> CONFIG_MXC_VPU_BUFFER_STATIC=y
1284,1286c1384
< CONFIG_NTFS_FS=m
< # CONFIG_NTFS_DEBUG is not set
< # CONFIG_NTFS_RW is not set
---
> # CONFIG_NTFS_FS is not set
1346c1444
< CONFIG_NFSD=m
---
> CONFIG_NFSD=y
1354c1452
< CONFIG_EXPORTFS=m
---
> CONFIG_EXPORTFS=y
1360,1367c1458,1459
< CONFIG_SMB_FS=m
< # CONFIG_SMB_NLS_DEFAULT is not set
< CONFIG_CIFS=m
< # CONFIG_CIFS_STATS is not set
< # CONFIG_CIFS_WEAK_PW_HASH is not set
< # CONFIG_CIFS_XATTR is not set
< # CONFIG_CIFS_DEBUG2 is not set
< # CONFIG_CIFS_EXPERIMENTAL is not set
---
> # CONFIG_SMB_FS is not set
> # CONFIG_CIFS is not set
1485c1577
< CONFIG_CRYPTO_ALGAPI=m
---
> CONFIG_CRYPTO_ALGAPI=y
1492c1584
< CONFIG_CRYPTO_MD5=m
---
> CONFIG_CRYPTO_MD5=y
=========================================
Basically, some modules have vanished (e.g. CONFIG_NTFS_FS), other have been
integrated in kernel instead of being generated as kernel modules (e.g.
CONFIG_CRYPTO_MD5) , and some drivers have been added to the kernel (e.g.
CONFIG_SND_AC97_CODEC). AFAICS, this should only affect the kernel map,
nothing else ?
Does it ring a bell ?
'later,
--
JPI
Since you were trying to load usblp, this looks like the answer. That power
management code has lots of tentacles.
--
Alan Curry
> [...]
>>1001d1099
>>< CONFIG_USB_SUSPEND=y
>
> Since you were trying to load usblp, this looks like the answer. That
> power
> management code has lots of tentacles.
Hello Alan,
Hhmmmm [scratching head]... I would agree if it had prevented my usblp
module from *running*, but from *loading* ? With a dozen "disagrees about
version of symbol" error messages targeting even the most common USB API
functions like usb_alloc_urb() ? I don't really buy it, except if it's a
side effect, e.g. perhaps this option resulted in kernel USB support code
being shifted up or down in memory by a couple bytes, which could be the
cause for the "version mismatch" ?
Well, I can't figure Linux not being able to link dynamically at module load
time, even if the target functions have moved a bit in memory. Windows knows
how to do this at kernel level (been there, done that in a previous life),
and Linux wouldn't ? I can't believe it. However, this seems to be close to
the heart of the issue, and this is where I'd be happy to find some
clarification.
Anyway, thanks for the idea. Moves brain in the right direction. :-)
Bye,
--
JPI
Yes, that's probably what happened. I haven't tracked down exactly how the
connection goes between them, but for example the size of struct usb_hcd is
affected by CONFIG_USB_SUSPEND, since it contains this field:
#ifdef CONFIG_USB_SUSPEND
struct work_struct wakeup_work; /* for remote wakeup */
#endif
And that's going to affect a lot of things.
--
Alan Curry
(Setting aside the fact that you got the original config and rebuilt the
whole thing:)
I think in this situation you may have to treat the installed kernel as a
given, and consider the usblp module you're about to build a module of
your own creation, ie. a third party, external module. How to build such a
module is documented here:
http://tldp.org/LDP/lkmpg/2.6/html/x181.html
Example 2-2. Makefile for a basic kernel module
----v----
obj-m += hello-1.o
all:
make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules
clean:
make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean
----^----
(Supposing you're running the same kernel that you want to compile the
module for.) Most probably, you should just enter the directory containing
the usblp module in the extracted vanilla 2.6.22.6 tree, hack up a similar
makefile and run "make all".
To test this theory: on a Debian Lenny box ("uname -r" returns
"2.6.26-2-amd64"), I extracted 2.6.22.6, then:
$ cd linux-2.6.22.6/drivers/usb/class
$ cat >>Makefile <<EOT
all:
make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules
EOT
$ make all
make -C /lib/modules/2.6.26-2-amd64/build M=/.../linux-2.6.22.6/drivers/usb/class modules
make[1]: Entering directory `/usr/src/linux-headers-2.6.26-2-amd64'
CC [M] /.../linux-2.6.22.6/drivers/usb/class/cdc-acm.o
CC [M] /.../linux-2.6.22.6/drivers/usb/class/usblp.o
Building modules, stage 2.
MODPOST 2 modules
CC /.../linux-2.6.22.6/drivers/usb/class/cdc-acm.mod.o
LD [M] /.../linux-2.6.22.6/drivers/usb/class/cdc-acm.ko
CC /.../linux-2.6.22.6/drivers/usb/class/usblp.mod.o
LD [M] /.../linux-2.6.22.6/drivers/usb/class/usblp.ko
make[1]: Leaving directory `/usr/src/linux-headers-2.6.26-2-amd64'
$ su -c 'insmod ./usblp.ko' root
Password:
$ lsmod | grep usblp
usblp 16256 0
Note that the kernel version I run has minor version 26, while the
extracted/built module comes from minor version 22. Still this is no
problem as long as the driver is source-level compatible with your kernel
development package (eg. "linux-headers-2.6.26-2-amd64").
Of course altering the Makefile can be avoided like this:
$ cd "/lib/modules/$(uname -r)/build"
$ make M=/.../linux-2.6.22.6/drivers/usb/class modules
CC [M] /.../linux-2.6.22.6/drivers/usb/class/cdc-acm.o
CC [M] /.../linux-2.6.22.6/drivers/usb/class/usblp.o
Building modules, stage 2.
MODPOST 2 modules
CC /.../linux-2.6.22.6/drivers/usb/class/cdc-acm.mod.o
LD [M] /.../linux-2.6.22.6/drivers/usb/class/cdc-acm.ko
CC /.../linux-2.6.22.6/drivers/usb/class/usblp.mod.o
LD [M] /.../linux-2.6.22.6/drivers/usb/class/usblp.ko
If you look at the contents of the "/lib/modules/$(uname -r)/build"
directory, among other things you find ".config" and "Module.symvers". I
think those might play some role in getting your driver built correctly.
(The last access time of "Module.symvers" is refreshed during the above
build.)
Whoever built/installed your kernel (manually?): if he used "make
modules_install", the /lib/modules/$(uname -r)/build directory should be
there.
One thing which I obviously forgot above: since your installation did not
come with usblp.ko originally, chances are that CONFIG_USB_PRINTER was not
set at all. In this case I think you must change the Makefile anyhow, from
obj-$(CONFIG_USB_PRINTER) += usblp.o
to
obj-m += usblp.o
I'm afraid if CONFIG_USB_PRINTER depends on further stuff you don't have
enabled in your current kernel, and/or CONFIG_USB_PRINTER affects other
modules as well, then it may be easiest to do a full rebuild.
(A "real" third party, external kernel module would (a) simply state the
required CONFIG_XXXX flags as dependencies, (b) not affect anything else.)
Tangentially, since the upstream kernel does not (can not) guarantee even
source-level compatibility to external modules over time, vendors have two
choices:
- get their driver in the kernel,
- develop their driver against a Linux distribution (yes, now as in
"Linux, the kernel") that guarantees binary compatibility between minor
updates, for a well-defined set of kernel interfaces. This way users don't
even have to recompile the 3rd party module when a new kernel build is
pushed to them (nor must the vendor).
HTH,
lacos
It would be nice though to know exactly how the build system and
module loading translates "affects a lot of things" to "disagrees
about version of symbol".
I'm sure the information is out there somewhere. I used to know how it
worked in the Linux 2.2 timeframe. I can't remember /how/, but back
then it was straightforward and googleable. The kernel programming
books surely cover it.
/Jorgen
--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .
Yes ! With this one, I think you're right on the point, since as far as I
understood, the computed "hash" used for the "mismatch check" includes
values derived from arguments of API functions, down to the deepest byte of
the deepest member of the deepest structure.
So, I think the whole story is not so much a matter of addresses as a matter
of API functions arguments, which makes sense: if my module expects usb_hcd
to include a wakeup_work member, and it actually doesn't (or the opposite),
Bad Things (tm) will happen sooner or later. So better prevent the module
from loading...
If I can find some time, I will go back to the original version of the
config file I have used for building the module that didn't load, changing
only this CONFIG_USB_SUSPEND parameter to have identical options at this
level, and I will check if the resultant usblp.ko can load despite the other
differences in the configuration.
Thanks a lot Alan.
--
JPI
Thanks Laszlo, your message, in addition to the right-on-the-point final
response of Alan Curry, did help me a lot.
I'll try to find some time to keep you both informed on the results of my
tests.
'later,
--
JPI
Hi again,
And the winner is... Alan. :-)
In addition of CONFIG_USB_SUSPEND, I had to alter CONFIG_PM (and
subsequently CONFIG_DPM and CONFIG_DPM_PROCFS) in my original config file,
because CONFIG_PM affects usb_device structure (defined in
include/linux/usb.h). I have made these options identical to the ones that
had been used for building the installed kernel, and these minor changes are
sufficient to allow me to build a usblp.ko module that is "API compliant"
and can be loaded/used with the installed kernel.
As a conclusion, when being faced with "disagrees about version of symbol
xxx" error messages, look for configuration options that may affect the
presence/absence of API functions used by the module *and the structure of
their arguments*. The actual address of the API functions doesn't matter and
doesn't prevent the module from being loaded.
Thanks again to everyone that helped on this topic.
--
JPI