Can't generate traffic with trex in docker(using latest master)

[Robert Blomberg]

When I have set-up trex with docker, and wanted to bind-interfaces, I got the following error, though everything works:

root@5184ecaa2f3f:/scratch/trex-core/scripts# ./dpdk_nic_bind.py -s
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12


Network devices using DPDK-compatible driver
============================================
<none>


Network devices using kernel driver
===================================
0000:02:00.0 'Ethernet 10G 2P X520 Adapter' if= drv=ixgbe unused=
0000:02:00.1 'Ethernet 10G 2P X520 Adapter' if= drv=ixgbe unused=
0000:05:00.0 'I210 Gigabit Network Connection' if= drv=igb unused=
0000:06:00.0 'I210 Gigabit Network Connection' if= drv=igb unused=


Other network devices
=====================
0000:01:00.0 'Ethernet 10G 2P X520 Adapter' unused=
0000:01:00.1 'Ethernet 10G 2P X520 Adapter' unused=
root@5184ecaa2f3f:/scratch/trex-core/scripts#

Also, when I try to generate some traffic, I get this error:

root@a72f3bb891eb:/scratch/trex-core/scripts# ./dpdk_nic_bind.py -s
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12


Network devices using DPDK-compatible driver
============================================
<none>


Network devices using kernel driver
===================================
0000:01:00.0 'Ethernet 10G 2P X520 Adapter' if= drv=ixgbe unused=
0000:01:00.1 'Ethernet 10G 2P X520 Adapter' if= drv=ixgbe unused=
0000:05:00.0 'I210 Gigabit Network Connection' if= drv=igb unused=
0000:06:00.0 'I210 Gigabit Network Connection' if= drv=igb unused=


Other network devices
=====================
0000:02:00.0 'Ethernet 10G 2P X520 Adapter' unused=
0000:02:00.1 'Ethernet 10G 2P X520 Adapter' unused=
root@a72f3bb891eb:/scratch/trex-core/scripts# ./dpdk_nic_bind.py -u 01:00.0 01:00.1
Warning - no supported modules(DPDK driver) are loaded
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
root@a72f3bb891eb:/scratch/trex-core/scripts# ./dpdk_nic_bind.py -b ixgbe 01:00.0 01:00.1
Warning - no supported modules(DPDK driver) are loaded
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
root@a72f3bb891eb:/scratch/trex-core/scripts# ./dpdk_setup_ports.py -c 01:00.0 01:00.1 -o 1.cfg
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
File 1.cfg already exist, overwrite? (y/N)y
Saved to 1.cfg.


root@a72f3bb891eb:/scratch/trex-core/scripts# ./t-rex-64 --cfg 1.cfg -f avl/sfr_delay_10_no_bundeling.yaml
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
lspci: Unable to load libkmod resources: error -12
sh: 1: modprobe: not found
Trying to bind to vfio-pci ...
Trying to compile and bind to igb_uio ...
sh: 1: modprobe: not found
Failed inserting uio module, please check if it is installed
ERROR encountered while configuring trex system
root@a72f3bb891eb:/scratch/trex-core/scripts#

[Itay Marom]

Hi Robert,

Did you use our provided .Dockerfile or created your own ?

Itay

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/c8f06d6e-738b-4b94-a972-5d50e0dd01a3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Itay Marom

[Itay Marom]

but the reason for your error is probably docker privileged mode was not set.

docker sees you as root but if privileged was not set to true then lspci will not succeed in some of the actions.

Itay

--

Itay Marom

[Itay Marom]

Ok...

I see the problem...

when calling strace on the command I get this:

open("/lib/modules/3.13.0-32-generic/modules.softdep", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/proc/cmdline", O_RDONLY|O_CLOEXEC) = 4
read(4, "BOOT_IMAGE=/boot/vmlinuz-3.13.0-"..., 4095) = 130
read(4, "", 3965)                       = 0
close(4)                                = 0
open("/lib/modules/3.13.0-32-generic/modules.dep.bin", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
write(2, "lspci: Unable to load libkmod re"..., 51lspci: Unable to load libkmod resources: error -12

so we need to mount the lib/modules library inside docker:

try to add this:

-v /lib/modules:/lib/modules

now everything is fine:

docker run -it --privileged --cap-add=ALL -v /mnt/huge:/mnt/huge -v /sys/bus/pci/devices:/sys/bus/pci/devices -v /sys/devices/system/node:/sys/devices/system/node -v  /lib/modules:/lib/modules -v /dev:/dev trex-dev-ubuntu:16.10

root@4a5991db52f5:/scratch/trex-core/scripts# ./dpdk_nic_bind.py -s

Network devices using DPDK-compatible driver
============================================

0000:0b:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio unused=ixgbe
0000:0b:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio unused=ixgbe
0000:13:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio unused=ixgbe
0000:13:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio unused=ixgbe

Network devices using kernel driver
===================================
0000:03:00.0 'VMXNET3 Ethernet Controller' if= drv=vmxnet3 unused=igb_uio

Other network devices
=====================
<none>

--

Itay Marom

[Robert Blomberg]

Hi Itay!

I followed the README which was inclued in the docker-directory.

Also, I am running the commands as root.

So to create docker-image, I used following command:

"docker run -it --privileged --cap-add=ALL -v /mnt/huge:/mnt/huge -v /sys/bus/pci/devices:/sys/bus/pci/devices -v /sys/devices/system/node:/sys/devices/system/node -v /dev:/dev trex-dev-ubuntu:16.10"

I am currently running debian-strech, which maybe is not compatible running with ubuntu-16.10 in a docker-instance.

root@sqa-pktgen10:~/trex-core/docker/ubuntu/16.10# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 9.0 (stretch)
Release: 9.0
Codename: stretch
root@sqa-pktgen10:~/trex-core/docker/ubuntu/16.10#

To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.

To post to this group, send email to trex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/c8f06d6e-738b-4b94-a972-5d50e0dd01a3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Itay Marom

--

Itay Marom

[Itay Marom]

Great..

So I'll fix the README.txt to include mounting the modules library as well.

please let us know how it goes with Docker and TRex together.

Thanks,

Itay

To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.

To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/29c70852-5327-4b1e-b71b-e2cfd822b176%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Itay Marom

[Robert Blomberg]

Hi Itay!

Now I don't see the error printed out all the time.

Though, now I got this error:

root@41da40f51fca:/scratch/trex-core/scripts# ./t-rex-64 --cfg 1.cfg -f avl/sfr_delay_10.yaml
sh: 1: modprobe: not found

Trying to bind to vfio-pci ...
Trying to compile and bind to igb_uio ...
sh: 1: modprobe: not found
Failed inserting uio module, please check if it is installed
ERROR encountered while

configuring trex system
root@41da40f51fca:/scratch/trex-core/scripts#

Am I missing any dependencies?

Using ixgbe-driver.

Thanks again.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/29c70852-5327-4b1e-b71b-e2cfd822b176%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Itay Marom

[Itay Marom]

apt-get install kmod 

I'll fix the missing libraries in the readme...

To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+unsubscribe@googlegroups.com.

To post to this group, send email to trex...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/8cf0d842-58b3-4024-9e1c-3eb1e555632e%40googlegroups.com.

[Robert Blomberg]

Hi again!

Seems that I got into this also:

root@b3cb264a2415:/scratch/trex-core/scripts# ./t-rex-64 --cfg 1.cfg -f avl/sfr_delay_10.yaml

Trying to bind to vfio-pci ...
Trying to compile and bind to igb_uio ...

ERROR: We don't have precompiled igb_uio.ko module for your kernel version
Will try compiling automatically - make sure you have file-system read/write permission


 ERROR:  Automatic compilation failed: ([Errno 2] No such file or directory)
Make sure you have file-system read/write permission
You can try compiling yourself, using the following commands:
  $cd ko/src
  $make
  $make install
  $cd -
Then, try to run TRex again.
Note: you might need additional Linux packages for that:
  * yum based (Fedora, CentOS, RedHat):
        sudo yum install kernel-devel-`uname -r`
        sudo yum group install "Development tools"
  * apt based (Ubuntu):
        sudo apt install linux-headers-`uname -r`
        sudo apt install build-essential

ERROR encountered while configuring trex system

Trying to run bellow command but just get this:

make -C /lib/modules/4.9.0-2-amd64/build M=/scratch/trex-core/scripts/ko/src modules
make[1]: *** /lib/modules/4.9.0-2-amd64/build: No such file or directory.  Stop.
Makefile:20: recipe for target 'default' failed
make: *** [default] Error 2
root@b3cb264a2415:/scratch/trex-core/scripts/ko/src# apt install linux-headers-`uname -r`
Reading package lists... Done
Building dependency tree      
Reading state information... Done
E: Unable to locate package linux-headers-4.9.0-2-amd64
E: Couldn't find any package by glob 'linux-headers-4.9.0-2-amd64'
E: Couldn't find any package by regex 'linux-headers-4.9.0-2-amd64'
root@b3cb264a2415:/scratch/trex-core/scripts/ko/src#

[hanoh haim]

See this,

https://builders.intel.com/docs/container-and-kvm-virtualization-for-nfv.PDF

You will need to load igb_uio in the host.

Compile the igb_uio in the host and load it once by trying to run TRex(again from the host) once it was loaded the container will work.

Another alternative is to use vfio that is part of the kernel in specific distribution (CentOS/Red Hat) - this is how TRex is used inside docker by nfvbench.

 

Thanks,

Hanoh 

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/95ebdbed-d610-4518-be2b-27b4952f69a5%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Hanoh

Sent from my iPhone

[Robert Blomberg]

Hi Hanoh!

It works now Hanoh.

Just did:

modprobe uio_pci_generic

And used that instead of ixgbe.

Thanks for the help.