TRex lcore limit

[Haklat haklat]

Hi,
we use TRex in K8s cloud environment.
lately with Servers having more CPUs we have had issued to start due to this.

Starting  TRex v3.04 please wait  ...
EAL: lcore 156 >= RTE_MAX_LCORE (128)
EAL: To use high physical core ids, please use --lcores to map them to lcore ids below RTE_MAX_LCORE, e.g. --lcores 0@52,1@156,2@53
EAL: invalid core list syntax

or:
Starting  TRex v3.04 please wait  ... 
ERROR, physical thread id is 157 higher than max 120

I have fixt (seems working) it by compiling with two define changes:

src/pal/linux_dpdk/dpdk_2303_x86_64/rte_build_config.h

#define RTE_MAX_LCORE 128  ---->>>>  #define RTE_MAX_LCORE 256

 

src/trex_defs.h:#define MAX_THREADS_SUPPORTED   (120) ---->>>> #define MAX_THREADS_SUPPORTED   (256)

Is there any downsides with increasing those compile variables, or could it be some defaults that could be changed in coming releases?

BR//Håkan

[hanoh haim]

Hi Haklat, 

I think there is a small memory and overhead, but I would create a PR to increase it

Thanks

Hanoh

Thban

[Todd Blevins]

i have a setup with 512 cores:

NUMA node0 CPU(s):   0-127,256-383
NUMA node1 CPU(s):   128-255,384-511

made the changes above using 512, but getting this error starting trex:

Total Memory :  
MBUF_64                                   : 65520  
MBUF_128                                  : 49140  
MBUF_256                                  : 24570  
MBUF_512                                  : 24570  
MBUF_1024                                 : 16380  
MBUF_2048                                 : 49140  
MBUF_4096                                 : 4096  
MBUF_DP_FLOWS                             : 10000000  
MBUF_GLOBAL_FLOWS                         : 5120  
get_each_core_dp_flows                    : 2500000  
Total memory                              :       1.98 Gbytes   
core_list : 378,379,124,125,126,127  
sockets : 0   
active sockets : 1  
ports_sockets : 1  
0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
phy   |   virt    
124      4    
125      3    
126      2    
127      1    
380      4    
381      3    
382      2    
383      1    
DPDK args  
xx  -l  378,379,124,125,126,127  -n  4  --log-level  8  --main-lcore  122  -a  0000:21:00.0  -a  0000:21:00.1  --legacy-mem  --file-prefix  trex1  --socket-mem  2048,0   
EAL: Detected CPU lcores: 512
EAL: Detected NUMA nodes: 2
EAL: Main lcore is not enabled for DPDK

where does --main-lcore 122 come from?  here's the config file:

[root@localhost v1.0]# more /tmp/cfg_file_1.yaml    
- interfaces: ["21:00.0", "21:00.1"]
 limit_memory: 2048
 platform:
   dual_if:
   - socket: 0
     threads:
     - 383
     - 382
     - 381
     - 380
   latency_thread_id: 379
   master_thread_id: 378
 prefix: trex1
 memory:
  dp_flows: 10000000
  mbuf_1025: 16380
  mbuf_128:  16380
  mbuf_2048: 16380
  mbuf_256:  16380
  mbuf_512:  16380
  mbuf_64:   32760
  traffic_mbuf_1025: 16380
  traffic_mbuf_64:   32760
 stack: linux_based
 version: 2
 zmq_pub_port: 8101
 zmq_rpc_port: 8102