problem with free hugepages

[llzz22.55ssll@gmail.com]

hi,
I have installed and runned seastar with DPDK, but encountered with this problem:
after dynamically allocating 2048 persistent huge pages that was successfully created based on contents of meminfo after doing allocation which were:
HugePages_Total:  2048
HugePages_Free:   2048
HugePages_Surp:      0
Hugepagesize:       2048 kB

I ran the udp-server program in seastar/tests folder that has been already compiled, with given port and --dpdk-pmd option,
but seastar gives me this error:
EAL: Detected 4 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB

I have tried with smaller and bigger number of huge pages but still the same error
after this error when I read the contents of meminfo file it says there is no free huge pages:

HugePages_Total:    2048
HugePages_Free:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

it seems some application (maybe DPDK itself?) consumes these free hugepages right after i run the executable server but seastar cannot get access to them and give me this error = No free hugepages reported in hugepages-1048576kB

can anyone please tell me what to do in order to be able to use dpdk for receiving packets in server side (like 1 million packets per second)?

I really appreciate if you help me on this issue.

thanks,

[Takuya ASADA]

On Thu, Oct 26, 2017 at 2:48 AM, <llzz22...@gmail.com> wrote:

hi,
I have installed and runned seastar with DPDK, but encountered with this problem:
after dynamically allocating 2048 persistent huge pages that was successfully created based on contents of meminfo after doing allocation which were:
HugePages_Total:  2048
HugePages_Free:   2048
HugePages_Surp:      0
Hugepagesize:       2048 kB

I ran the udp-server program in seastar/tests folder that has been already compiled, with given port and --dpdk-pmd option,
but seastar gives me this error:
EAL: Detected 4 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB

Could you provide us a full parameters you have passed to udp_server, and also the command you have executed to allocate hugepages, attaching NIC to UIO driver to use NIC from DPDK?

There are two size of hugepages 1GB and 2MB, the warning message "EAL: No free hugepages reported in hugepages-1048576kB" means that DPDK unable to find free 1GB hugepage, doesn't mean failed to allocate 2MB and giving up program execution.

Here's example to run udp_server with DPDK mode, on Intel 10G NIC(ens4f0) and allocating 256 hugepages for each NUMA node (it's saying port link down because I haven't connected cable on the NIC but at least it able to launch in DPDK mode):

$ sudo scripts/dpdk_nic_bind.py --status

Network devices using DPDK-compatible driver

============================================

<none>

Network devices using kernel driver

===================================

0000:00:19.0 'Ethernet Connection (2) I218-V' if=eno1 drv=e1000e unused=uio_pci_generic *Active*

0000:01:00.0 'Ethernet Controller 10-Gigabit X540-AT2' if=ens4f0 drv=ixgbe unused=uio_pci_generic *Active*

0000:01:00.1 'Ethernet Controller 10-Gigabit X540-AT2' if=ens4f1 drv=ixgbe unused=uio_pci_generic 

Other network devices

=====================

<none>

$ sudo ifconfig ens4f0 down

$ sudo scripts/dpdk_nic_bind.py --unbind 0000:01:00.0

$ sudo modprobe uio_pci_generic

$ sudo scripts/dpdk_nic_bind.py --force --bind=uio_pci_generic 0000:01:00.0

$ sudo scripts/dpdk_nic_bind.py --status

Network devices using DPDK-compatible driver

============================================

0000:01:00.0 'Ethernet Controller 10-Gigabit X540-AT2' drv=uio_pci_generic unused=ixgbe

Network devices using kernel driver

===================================

0000:00:19.0 'Ethernet Connection (2) I218-V' if=eno1 drv=e1000e unused=uio_pci_generic *Active*

0000:01:00.1 'Ethernet Controller 10-Gigabit X540-AT2' if=ens4f1 drv=ixgbe unused=uio_pci_generic 

Other network devices

=====================

<none>

$ sudo su -

# for n in /sys/devices/system/node/node?; do

> echo 256 > $n/hugepages/hugepages-2048kB/nr_hugepages

> done

# exit

$ sudo ./build/release/tests/udp_server --dpdk-pmd --default-log-level debug --host-ipv4-addr 172.16.0.1 --netmask-ipv4-addr 255.255.255.0 --network-stack native

EAL: Detected 20 lcore(s)

EAL: No free hugepages reported in hugepages-1048576kB

EAL: Probing VFIO support...

EAL: PCI device 0000:00:19.0 on NUMA socket 0

EAL:   probe driver: 8086:15a1 net_e1000_em

EAL: PCI device 0000:01:00.0 on NUMA socket 0

EAL:   probe driver: 8086:1528 net_ixgbe

EAL: PCI device 0000:01:00.1 on NUMA socket 0

EAL:   probe driver: 8086:1528 net_ixgbe

ports number: 1

Port 0: max_rx_queues 128 max_tx_queues 64

Port 0: using 20 queues

Port 0: RSS table size is 128

LRO is on

RX checksum offload supported

TX ip checksum offload supported

TSO is supported

TX TCP&UDP checksum offload supported

Port 0 init ... done: 

Creating Tx mbuf pool 'dpdk_pktmbuf_pool0_tx' [1024 mbufs] ...

Creating Rx mbuf pool 'dpdk_pktmbuf_pool0_rx' [1024 mbufs] ...

Creating Tx mbuf pool 'dpdk_pktmbuf_pool1_tx' [1024 mbufs] ...

Creating Tx mbuf pool 'dpdk_pktmbuf_pool11_tx' [1024 mbufs] ...

Creating Tx mbuf pool 'dpdk_pktmbuf_pool10_tx' [1024 mbufs] ...

Creating Tx mbuf pool 'dpdk_pktmbuf_pool6_tx' [1024 mbufs] ...

Creating Tx mbuf pool 'dpdk_pktmbuf_pool16_tx' [1024 mbufs] ...

Creating Tx mbuf pool 'dpdk_pktmbuf_pool2_tx' [1024 mbufs] ...

Creating Tx mbuf pool 'dpdk_pktmbuf_pool18_tx' [1024 mbufs] ...

Creating Tx mbuf pool 'dpdk_pktmbuf_pool14_tx' [1024 mbufs] ...

Creating Tx mbuf pool 'dpdk_pktmbuf_pool4_tx' [1024 mbufs] ...

Creating Tx mbuf pool 'dpdk_pktmbuf_pool12_tx' [1024 mbufs] ...

Creating Tx mbuf pool 'dpdk_pktmbuf_pool8_tx' [1024 mbufs] ...

Creating Tx mbuf pool 'dpdk_pktmbuf_pool19_tx' [1024 mbufs] ...

Creating Tx mbuf pool 'dpdk_pktmbuf_pool13_tx' [1024 mbufs] ...

Creating Tx mbuf pool 'dpdk_pktmbuf_pool17_tx' [1024 mbufs] ...

Creating Tx mbuf pool 'dpdk_pktmbuf_pool5_tx' [1024 mbufs] ...

Creating Tx mbuf pool 'dpdk_pktmbuf_pool15_tx' [1024 mbufs] ...

Creating Tx mbuf pool 'dpdk_pktmbuf_pool9_tx' [1024 mbufs] ...

Creating Tx mbuf pool 'dpdk_pktmbuf_pool3_tx' [1024 mbufs] ...

Creating Rx mbuf pool 'dpdk_pktmbuf_pool1_rx' [1024 mbufs] ...

Creating Rx mbuf pool 'dpdk_pktmbuf_pool10_rx' [1024 mbufs] ...

Creating Rx mbuf pool 'dpdk_pktmbuf_pool18_rx' [1024 mbufs] ...

Creating Rx mbuf pool 'dpdk_pktmbuf_pool8_rx' [1024 mbufs] ...

Creating Rx mbuf pool 'dpdk_pktmbuf_pool16_rx' [1024 mbufs] ...

Creating Rx mbuf pool 'dpdk_pktmbuf_pool6_rx' [1024 mbufs] ...

Creating Rx mbuf pool 'dpdk_pktmbuf_pool11_rx' [1024 mbufs] ...

Creating Rx mbuf pool 'dpdk_pktmbuf_pool3_rx' [1024 mbufs] ...

Creating Rx mbuf pool 'dpdk_pktmbuf_pool13_rx' [1024 mbufs] ...

Creating Tx mbuf pool 'dpdk_pktmbuf_pool7_tx' [1024 mbufs] ...

Creating Rx mbuf pool 'dpdk_pktmbuf_pool2_rx' [1024 mbufs] ...

Creating Rx mbuf pool 'dpdk_pktmbuf_pool17_rx' [1024 mbufs] ...

Creating Rx mbuf pool 'dpdk_pktmbuf_pool12_rx' [1024 mbufs] ...

Creating Rx mbuf pool 'dpdk_pktmbuf_pool14_rx' [1024 mbufs] ...

Creating Rx mbuf pool 'dpdk_pktmbuf_pool9_rx' [1024 mbufs] ...

Creating Rx mbuf pool 'dpdk_pktmbuf_pool4_rx' [1024 mbufs] ...

Creating Rx mbuf pool 'dpdk_pktmbuf_pool19_rx' [1024 mbufs] ...

Creating Rx mbuf pool 'dpdk_pktmbuf_pool5_rx' [1024 mbufs] ...

Creating Rx mbuf pool 'dpdk_pktmbuf_pool15_rx' [1024 mbufs] ...

Creating Rx mbuf pool 'dpdk_pktmbuf_pool7_rx' [1024 mbufs] ...

Port 0: Enabling HW FC

Checking link status 

Created DPDK device

..........................................................................................done

Port  Link Down

 

I have tried with smaller and bigger number of huge pages but still the same error
after this error when I read the contents of meminfo file it says there is no free huge pages:

HugePages_Total:    2048
HugePages_Free:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

it seems some application (maybe DPDK itself?) consumes these free hugepages right after i run the executable server but seastar cannot get access to them and give me this error = No free hugepages reported in hugepages-1048576kB

can anyone please tell me what to do in order to be able to use dpdk for receiving packets in server side (like 1 million packets per second)?

I really appreciate if you help me on this issue.

thanks,

--
You received this message because you are subscribed to the Google Groups "seastar-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to seastar-dev+unsubscribe@googlegroups.com.
To post to this group, send email to seast...@googlegroups.com.
Visit this group at https://groups.google.com/group/seastar-dev.
To view this discussion on the web visit https://groups.google.com/d/msgid/seastar-dev/48cbe985-4e55-4ccc-ba96-1a84cfdc82c2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Takuya ASADA]

It isn’t documented enough, sorry about that.

Will add how to run in DPDK mode page on our Wiki.

To unsubscribe from this group and stop receiving emails from it, send an email to seastar-dev...@googlegroups.com.

[llzz22.55ssll@gmail.com]

Thank you so much sir for your fast and informative response.
I did use 1 GB huge pages by changing the GRUB_CMDLINE_LINUX_DEFAULT value in grub file and it worked fine!
I also had to re-run the DPDK-setup.py again in order to bind network device with igb_uio module to DPDK.
---
I have a new question though that might seem a little bit stupid !! but i'm a newbie in this field some how ... any ways from this link :
https://github.com/scylladb/seastar/wiki/Networking
I couldn't get my answer and I hope you can help me on this:
==> If I want to send packets to the server with DPDK parameter enabled and with network stack as native (like the case you mentioned in your example) what would be the port number of the server to be passed to client code? (cause as much as I know and see! in network stack = native mode, no port is passed to the udp_server)
I prefer not to just use the udp_client.cc code because I want the client part to be as flexible as possible because packets based on the needs of the project I am involved in it, might come from different clients with different circumstances...

Thank you so much again.

[Takuya ASADA]

On Sat, Oct 28, 2017 at 4:49 AM <llzz22...@gmail.com> wrote:

Thank you so much sir for your fast and informative response.
I did use 1 GB huge pages by changing the GRUB_CMDLINE_LINUX_DEFAULT value in grub file and it worked fine!
I also had to re-run the DPDK-setup.py again in order to bind network device with igb_uio module to DPDK.
---
I have a new question though that might seem a little bit stupid !! but i'm a newbie in this field some how ... any ways from this link :
https://github.com/scylladb/seastar/wiki/Networking
I couldn't get my answer and I hope you can help me on this:
==> If I want to send packets to the server with DPDK parameter enabled and with network stack as native (like the case you mentioned in your example) what would be the port number of the server to be passed to client code? (cause as much as I know and see! in network stack = native mode, no port is passed to the udp_server)

The option “server” is pair of IP:port, so just passing “—server xx.xx.xx.xx:yy” to the client should work.

To view this discussion on the web visit https://groups.google.com/d/msgid/seastar-dev/70097fa6-bf9e-4261-acd9-7f1a1b89c8f7%40googlegroups.com.

[llzz22.55ssll@gmail.com]

Thank you so much sir,
I managed to bind dpdk with the network device that I had connected cable to it before (took some time to find out how to unbind it from linux kernel and bind it to DPDK ...) and I ran udp-server code with dpdk enabled and then I realized it will listen to port 10000 by defualt !!! My lack of attention :|

these are the parameters that I passed to server:
./udp_Server --dpdk-pmd --default-log-level debug --host-ipv4-addr 192.168.1.53 --gw-ipv4-addr 192.168.1.1 --netmask-ipv4-addr 255.255.255.0 --network-stack native -c 4

and these are the results:

EAL: Detected 4 lcore(s)
EAL: Probing VFIO support...
EAL: PCI device 0000:05:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10fb net_ixgbe
EAL: PCI device 0000:05:00.1 on NUMA socket -1
EAL:   probe driver: 8086:10fb net_ixgbe
EAL: PCI device 0000:0a:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10a7 net_e1000_igb
EAL: PCI device 0000:0a:00.1 on NUMA socket -1
EAL:   probe driver: 8086:10a7 net_e1000_igb
EAL: PCI device 0000:0b:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10a7 net_e1000_igb
EAL: PCI device 0000:0b:00.1 on NUMA socket -1
EAL:   probe driver: 8086:10a7 net_e1000_igb
EAL: PCI device 0000:0c:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10a7 net_e1000_igb
EAL: PCI device 0000:0c:00.1 on NUMA socket -1
EAL:   probe driver: 8086:10a7 net_e1000_igb
EAL: PCI device 0000:0d:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10a7 net_e1000_igb
EAL: PCI device 0000:0d:00.1 on NUMA socket -1
EAL:   probe driver: 8086:10a7 net_e1000_igb
EAL: PCI device 0000:12:00.0 on NUMA socket -1
EAL:   probe driver: 8086:10d3 net_e1000_em
ports number: 1
Port 0: max_rx_queues 4 max_tx_queues 4
Port 0: using 4 queues

Port 0: RSS table size is 128

LRO is off

RX checksum offload supported
TX ip checksum offload supported
TSO is supported
TX TCP&UDP checksum offload supported
Port 0 init ... done:
Creating Tx mbuf pool 'dpdk_pktmbuf_pool0_tx' [1024 mbufs] ...
Creating Rx mbuf pool 'dpdk_pktmbuf_pool0_rx' [1024 mbufs] ...

Creating Tx mbuf pool 'dpdk_pktmbuf_pool2_tx' [1024 mbufs] ...

Creating Tx mbuf pool 'dpdk_pktmbuf_pool3_tx' [1024 mbufs] ...

Creating Tx mbuf pool 'dpdk_pktmbuf_pool1_tx' [1024 mbufs] ...

Creating Rx mbuf pool 'dpdk_pktmbuf_pool2_rx' [1024 mbufs] ...

Creating Rx mbuf pool 'dpdk_pktmbuf_pool1_rx' [1024 mbufs] ...

Creating Rx mbuf pool 'dpdk_pktmbuf_pool3_rx' [1024 mbufs] ...

Port 0: Enabling HW FC

Checking link status
Created DPDK device

..........................done
Port 0 Link Up - speed 1000 Mbps - full-duplex
Seastar UDP server listening on port 10000 ...

Out: 63423 pps
Out: 0 pps
Out: 0 pps
Out: 0 pps
Out: 64479 pps
Out: 0 pps
Out: 0 pps
Out: 0 pps
Out: 62911 pps
Out: 0 pps
Out: 0 pps
Out: 0 pps
Out: 64287 pps
.
.
.
.