hugepages

[bre...@versa-networks.com]

Hello I have a system with both 1G and 2MB hugepages.  When I run trex it will consume all my 1G hugepages and will not release them.  I am running 2.41.

I set the limit_memory however they are all consumed once I run trex.

[root@trex ~]# cat /etc/trex_cfg.yaml

- port_limit      : 2

  version         : 2

  limit_memory    : 2048

#List of interfaces. Change to suit your setup. Use ./dpdk_setup_ports.py -s to see available options

  interfaces    : ["11:00.2","11:00.3"]

  port_info       :  # Port IPs. Change to suit your needs. In case of loopback, you can leave as is.

          - ip         : 4.4.0.1

            default_gw : 4.4.0.8

            #vlan       : 78

          - ip         : 12.12.0.1

            default_gw : 12.12.0.254

            #vlan       : 10

HugePages_Total:     128

HugePages_Free:        0

HugePages_Rsvd:        0

HugePages_Surp:        0

Hugepagesize:    1048576 kB

[root@osp1 usertools]# mount | grep huge

cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)

hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)

none on /dev/hugepages2M type hugetlbfs (rw,relatime,pagesize=2MB)

[root@osp1 usertools]#

[root@osp1 usertools]# cat /proc/cmdline

BOOT_IMAGE=/vmlinuz-3.10.0-862.3.2.el7.x86_64 root=/dev/mapper/rhel_osp1-root ro crashkernel=auto rd.lvm.lv=rhel_osp1/root rd.lvm.lv=rhel_osp1/swap rhgb quiet default_hugepagesz=1G hugepagesz=1G hugepages=128 hugepagesz=2M hugepages=100 iommu=pt intel_iommu=on isolcpus=4-11,12-23

[root@osp1 usertools]#

[Matt Callaghan]

noticed a small note in https://trex-tgn.cisco.com/trex/doc/trex_manual.html#_low_end_machines, that states a reboot is required to free hugepage allocation

PS: where in the documentation does it even say that hugepages require tuning as you have done?

[Brendan Kelly]

Hello Matt.  It’s not the release of the hugepages that is a big concern.  It’s the fact it takes all 128GB of the 1GB pages and then I cannot launch VMs as I am out.

 

Brendan Kelly

303-513-0846

Sales Engineer

--
You received this message because you are subscribed to the Google Groups "TRex Traffic Generator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to trex-tgn+u...@googlegroups.com.
To post to this group, send email to trex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/43aba772-4add-496c-9309-3c48f7b66341%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Matt Callaghan]

The only place in documentation (triple checked) we even mention hugepages are these two spots:

$ grep -rins hugepages *
trex_book.asciidoc:615:If you have already started without low_end argument, reboot the Linux to free hugepages.
trex_book.asciidoc:662:* Does not require hugepages.

Yet clearly the dpdk script is doing magic for us

https://github.com/cisco-system-traffic-generator/trex-core/blob/master/scripts/dpdk_setup_ports.py#L650

[Matt Callaghan]

Checkout the code in the script ... there are numerous logic assumptions that I see related to NUMA node0 as well as some hard coded allocation rules ...

You may want to debug the script according to your hardware configuration: https://github.com/cisco-system-traffic-generator/trex-core/blob/master/scripts/dpdk_setup_ports.py#L650

 (you may uncover a bug for your scenario)

[Brendan Kelly]

I will look at it.  Thanks.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/cfb51e07-5602-4a05-abd7-94b35159830f%40googlegroups.com.

[Brendan Kelly]

Well I looked at that script and it seems to just setup the hugepages if needed.  It seems like it is in the astf code that grabs all my 1G hugepages.

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/F7AC6484-1134-4E22-9354-F55C19B4FF98%40versa-networks.com.

[Brendan Kelly]

Ok found and old thread on this topic.

 

You have to set both prefix: and limit_memory in the cfg file otherwise it doesn’t work.

 

prefix : trex1

limit_memory : 2048

To view this discussion on the web visit https://groups.google.com/d/msgid/trex-tgn/68B57BC6-8B2A-4013-9F03-C3A9AC000644%40versa-networks.com.

[Matt Callaghan]

submit a bug and/or fix the parsing so that it errors?

[Matt Callaghan]

Brendan submitted: https://github.com/cisco-system-traffic-generator/trex-core/issues/129