Udev-based serial number dependent symlinks for PCIe UIO devices

[Wojciech Zabolotny]

I'm working at a system, where a server manages multiple PCIe data acquisition cards. Each card has its individual serial number available in the extended PCIe configuration space. The cards are controlled as UIO devices.
Unfortunately, it appears that the card are sometimes enumerated in the random order. Proper management of the system requires that there are symlinks created with names permanently associated with the individual cards.
Of course, I wanted to use udev for that purpose. Unfortunately, udev does not provide the Device Serial Number capability between the device attributes.
The DSN may be read with "lspci -vvv":

[...]
Capabilities: [100 v1] Device Serial Number 12-34-56-78-90-ab-cd-ef
[...]

My solution consists of two parts.
1. The udev rule (file: /etc/udev/rules.d/30-daq.rules)
===========
SUBSYSTEM=="uio" PROGRAM="/opt/bin/uio_namer" SYMLINK="%c"
===========
2. The program that is run for each found uio device and:
a. Checks if it is our board (end returns non-zero status if it is not)
b. Outputs the name of the desired symlink to the standard output
and returns 0 status when it is our device.
I have implemented it fully in bash. Of course it can be implemented
in C using the libpci, or in Python.
The program receives the information from udev in the environment
variables. The most important is DEVPATH.

(file: /opt/bin/uio_namer)
=======================
#!/bin/sh
DEVID=abba:3342
SLOTNAME=`basename \`readlink /sys/${DEVPATH}/device\``
DLSPCI=`lspci -vvv -s ${SLOTNAME}`
#Check if it is our device
if [ -z "`echo ${DLSPCI} | grep Device | grep ${DEVID}`" ] ; then
# It is not our device
exit 1
fi
DSNCAP=`lspci -vvv -s ${SLOTNAME} | grep "Device Serial Number" `
echo daq/${DSNCAP##*" "}
========================

For recognized boards, the device files are created in the /dev/daq directory as the board's serial numbers. For example: /dev/daq/12-34-56-78-90-ab-cd-ef

I hope that the above solution may be useful for others as well.
With best regards,
Wojtek

[Don Y]

On 5/18/2021 2:17 PM, Wojciech Zabolotny wrote:
> I'm working at a system, where a server manages multiple PCIe data
> acquisition cards. Each card has its individual serial number available in
> the extended PCIe configuration space. The cards are controlled as UIO
> devices. Unfortunately, it appears that the card are sometimes enumerated in
> the random order. Proper management of the system requires that there are
> symlinks created with names permanently associated with the individual
> cards.

[I don't run Linux, so...]

Can't you adjust the kernel configuration so that it forces
certain probes to yield certain "devices"?

E.g., instead of letting NetBSD probe disk drives in whatever order
it encounters them -- and assigning device names in that order -- I
build my kernels so certain device names are set aside for specific
devices (which may or may not be present at any given boot).

So, the first "free" probed device might be /dev/xx4 instead of
/dev/xx0 -- because I want to ensure /dev/xx0 is available for
a *specific* device instead of letting it go to the "first one
found".

[Theo]

Don Y <blocked...@foo.invalid> wrote:
> Can't you adjust the kernel configuration so that it forces
> certain probes to yield certain "devices"?
>
> E.g., instead of letting NetBSD probe disk drives in whatever order
> it encounters them -- and assigning device names in that order -- I
> build my kernels so certain device names are set aside for specific
> devices (which may or may not be present at any given boot).

That's what udev does. udev has a set of rules that indicate how devices
should be turned into device nodes, be it by physical location (PCI slot),
vendor/product ID, serial number, etc. It also sets ownership and
permissions of those device nodes (so only certain users can access a
device node). It handles creating and removing device nodes as devices are
hot plugged. It's all just some textual config files, no kernel compilation
needed.

The problem in this case is that PCI devices don't as standard support a
serial number in the way that USB or SATA does, and so this manufacturer has
done their own thing in the way the serial number is embedded. That is why
some helper code is needed to expose the serial number to udev to use in its
device naming.

(although I would have expected a rule that sets up device nodes named by
PCI slot to be equally predictable)

Theo

[Don Y]

On 5/19/2021 1:30 AM, Theo wrote:
> Don Y <blocked...@foo.invalid> wrote:
>> Can't you adjust the kernel configuration so that it forces
>> certain probes to yield certain "devices"?
>>
>> E.g., instead of letting NetBSD probe disk drives in whatever order
>> it encounters them -- and assigning device names in that order -- I
>> build my kernels so certain device names are set aside for specific
>> devices (which may or may not be present at any given boot).
>
> That's what udev does. udev has a set of rules that indicate how devices
> should be turned into device nodes, be it by physical location (PCI slot),
> vendor/product ID, serial number, etc. It also sets ownership and
> permissions of those device nodes (so only certain users can access a
> device node).

OK. In NetBSD, that's part of the kernel configuration.
Things like permissions and device types (char vs block)
are handled in reasonably static files:

---------
device-major ccd char 300 block 300 ccd
device-major vnd char 301 block 301 vnd
device-major md char 302 block 302 md
device-major ld char 303 block 303 ld
device-major raid char 304 block 304 raid
device-major cgd char 305 block 305 cgd

# scsibus and its children
device-major scsibus char 310 scsibus
device-major sd char 311 block 311 sd
device-major st char 312 block 312 st
device-major cd char 313 block 313 cd
device-major ch char 314 ch
device-major ss char 315 ss
device-major uk char 316 uk
device-major ses char 317 ses
device-major se char 318 se

device-major fd char 320 block 320 fdc
...
device-major com char 260 com
device-major lpt char 261 lpt
...
-------

while the binding of device identifiers is part of the "config"
file:

------
# SCSI devices
sd* at scsibus? target ? lun ? # SCSI disk drives
st* at scsibus? target ? lun ? # SCSI tape drives
cd* at scsibus? target ? lun ? # SCSI CD-ROM drives
ch* at scsibus? target ? lun ? # SCSI autochangers
ses* at scsibus? target ? lun ? # SCSI Enclosure Services devices
ss* at scsibus? target ? lun ? # SCSI scanners
uk* at scsibus? target ? lun ? # SCSI unknown
----

E.g., in the above, I could ensure sd3 was always a particular
device on a particular scsibus at a particular SCSI ID by
creating an entry with the various wildcards (? *) replaced
by specific values. The probe will preallocate those so
they aren't assigned to some other device that happened
to be Nth in order of detection.

> It handles creating and removing device nodes as devices are
> hot plugged. It's all just some textual config files, no kernel compilation
> needed.
>
> The problem in this case is that PCI devices don't as standard support a
> serial number in the way that USB or SATA does, and so this manufacturer has
> done their own thing in the way the serial number is embedded. That is why
> some helper code is needed to expose the serial number to udev to use in its
> device naming.

Ah. I'd have thought an alternative would just be to ensure devices
of that specific *type* (manufacturer) were probed in an order that
could be enforced by the equivalent of the scheme I mentioned above.

> (although I would have expected a rule that sets up device nodes named by
> PCI slot to be equally predictable)

I guess one unanswered question is whether Wojciech is bolting his solution
onto an existing system *or* designing a system from scratch (in which case,
you would assume he could exercise some control over how the hardware is
"built"/pieced together)

[Wojciech Zabolotny]

The end user wants to be able to run it on standard Debian GNU/Linux OS.

BR,
Wojtek