ntb_tool cannot dma_alloc_coherent with CMA (over switchtec)

48 views
Skip to first unread message

Glimcher, Boris

unread,
May 31, 2018, 12:33:16 PM5/31/18
to linu...@googlegroups.com

Hi,

 

I’m trying to enlarge buffers I use in ntb_tool by using cma:

$ dmesg | grep cma

[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.14.17 root=UUID=71fb5fd6-0242-444e-8e17-d31d65051f61 ro crashkernel=auto rhgb quiet cma=32G hugepages=100000 intel_pstate=disable

[    0.000000] cma: Reserved 32768 MiB at 0x000000f87fc00000

[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-4.14.17 root=UUID=71fb5fd6-0242-444e-8e17-d31d65051f61 ro crashkernel=auto rhgb quiet cma=32G hugepages=100000 intel_pstate=disable

[    0.000000] Memory: 6057140K/1073389456K available (10252K kernel code, 2231K rwdata, 3476K rodata, 2252K init, 3040K bss, 17104596K reserved, 33554432K cma-reserved)

 

grep -i cma /proc/meminfo

CmaTotal:       33554432 kB

CmaFree:        33501788 kB

 

After loading drivers I allocate memory (uses dma_alloc_coherent):

echo 20971520 > /sys/kernel/debug/ntb_tool/0000\:3b\:00.1/peer_trans1

 

I get this stack:

 

[  313.224544] switchtec 0000:3b:00.1: swiotlb buffer is full (sz: 20971520 bytes)

[  313.224546] swiotlb: coherent allocation failed for device 0000:3b:00.1 size=20971520

[  313.224550] Call Trace:

[  313.224560]  dump_stack+0x5c/0x83

[  313.224567]  swiotlb_alloc_coherent+0x14c/0x160

[  313.224572]  tool_peer_mw_trans_write+0x1b2/0x320 [ntb_tool]

[  313.224578]  ? perf_trace_mm_compaction_migratepages+0x5a/0x120

[  313.224584]  full_proxy_write+0x5a/0xa0

[  313.224589]  __vfs_write+0x33/0x170

[  313.224594]  ? handle_mm_fault+0xc4/0x1d0

[  313.224599]  ? __audit_syscall_entry+0xaf/0x100

[  313.224601]  vfs_write+0xad/0x1a0

[  313.224603]  SyS_write+0x52/0xc0

[  313.224608]  do_syscall_64+0x61/0x1a0

[  313.224614]  entry_SYSCALL64_slow_path+0x25/0x25

[  313.224616] RIP: 0033:0x7f79ac50e840

[  313.224617] RSP: 002b:00007ffc8e89afe8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001

[  313.224619] RAX: ffffffffffffffda RBX: 0000000000000009 RCX: 00007f79ac50e840

[  313.224619] RDX: 0000000000000009 RSI: 00007f79ace2d000 RDI: 0000000000000001

[  313.224620] RBP: 00007f79ace2d000 R08: 000000000000000a R09: 00007f79ace27740

[  313.224621] R10: 00007ffc8e89ad90 R11: 0000000000000246 R12: 00007f79ac7e2400

[  313.224622] R13: 0000000000000009 R14: 0000000000000001 R15: 0000000000000000

 

Any help please?

 

--

Best regards, Boris

 

Logan Gunthorpe

unread,
May 31, 2018, 2:48:38 PM5/31/18
to Glimcher, Boris, linu...@googlegroups.com

On 31/05/18 10:32 AM, Glimcher, Boris wrote:
> I’m trying to enlarge buffers I use in ntb_tool by using cma:

Hmm, not sure what the problem is. Is this x86?

It would also be useful to see a complete dmesg trace.

Logan

Dave Jiang

unread,
May 31, 2018, 3:31:11 PM5/31/18
to Glimcher, Boris, linu...@googlegroups.com


On 5/31/2018 9:32 AM, Glimcher, Boris wrote:

Hi,

 

I’m trying to enlarge buffers I use in ntb_tool by using cma:

$ dmesg | grep cma

[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.14.17 root=UUID=71fb5fd6-0242-444e-8e17-d31d65051f61 ro crashkernel=auto rhgb quiet cma=32G hugepages=100000 intel_pstate=disable

[    0.000000] cma: Reserved 32768 MiB at 0x000000f87fc00000

[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-4.14.17 root=UUID=71fb5fd6-0242-444e-8e17-d31d65051f61 ro crashkernel=auto rhgb quiet cma=32G hugepages=100000 intel_pstate=disable

[    0.000000] Memory: 6057140K/1073389456K available (10252K kernel code, 2231K rwdata, 3476K rodata, 2252K init, 3040K bss, 17104596K reserved, 33554432K cma-reserved)

 

grep -i cma /proc/meminfo

CmaTotal:       33554432 kB

CmaFree:        33501788 kB

 

After loading drivers I allocate memory (uses dma_alloc_coherent):

echo 20971520 > /sys/kernel/debug/ntb_tool/0000\:3b\:00.1/peer_trans1

 

I get this stack:

 

[  313.224544] switchtec 0000:3b:00.1: swiotlb buffer is full (sz: 20971520 bytes)

[  313.224546] swiotlb: coherent allocation failed for device 0000:3b:00.1 size=20971520

So it's quite difficult to allocate CMA memory on x86 without iommu enabled as I discovered. Anything larger than 4M and you are pretty much doing it by luck. When you turn the iommu on it does not go through swiotlb and that seems to work fairly well.


[  313.224550] Call Trace:

[  313.224560]  dump_stack+0x5c/0x83

[  313.224567]  swiotlb_alloc_coherent+0x14c/0x160

[  313.224572]  tool_peer_mw_trans_write+0x1b2/0x320 [ntb_tool]

[  313.224578]  ? perf_trace_mm_compaction_migratepages+0x5a/0x120

[  313.224584]  full_proxy_write+0x5a/0xa0

[  313.224589]  __vfs_write+0x33/0x170

[  313.224594]  ? handle_mm_fault+0xc4/0x1d0

[  313.224599]  ? __audit_syscall_entry+0xaf/0x100

[  313.224601]  vfs_write+0xad/0x1a0

[  313.224603]  SyS_write+0x52/0xc0

[  313.224608]  do_syscall_64+0x61/0x1a0

[  313.224614]  entry_SYSCALL64_slow_path+0x25/0x25

[  313.224616] RIP: 0033:0x7f79ac50e840

[  313.224617] RSP: 002b:00007ffc8e89afe8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001

[  313.224619] RAX: ffffffffffffffda RBX: 0000000000000009 RCX: 00007f79ac50e840

[  313.224619] RDX: 0000000000000009 RSI: 00007f79ace2d000 RDI: 0000000000000001

[  313.224620] RBP: 00007f79ace2d000 R08: 000000000000000a R09: 00007f79ace27740

[  313.224621] R10: 00007ffc8e89ad90 R11: 0000000000000246 R12: 00007f79ac7e2400

[  313.224622] R13: 0000000000000009 R14: 0000000000000001 R15: 0000000000000000

 

Any help please?

 

--

Best regards, Boris

 

--
You received this message because you are subscribed to the Google Groups "linux-ntb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to linux-ntb+...@googlegroups.com.
To post to this group, send email to linu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/linux-ntb/8A1F3ACA2F40C04FBF91985D6F491D7C0176649B%40MX302CL01.corp.emc.com.
For more options, visit https://groups.google.com/d/optout.

Glimcher, Boris

unread,
May 31, 2018, 3:42:59 PM5/31/18
to Logan Gunthorpe, linu...@googlegroups.com
Logan,

Yes, it is x86 machine.

When using smaller buffer size everything works fine:

$ echo 8192 > /sys/kernel/debug/ntb_tool/0000\:3b\:00.1/peer_trans1

$ cat /sys/kernel/debug/ntb_tool/0000\:3b\:00.1/peer_trans1
Peer MW 1 Information:
Physical Address 0x00001af400000000[p]
Window Size 17179869184
Alignment 4096
Size Alignment 4096
Size Max 17179869184
Ready Y
Allocated Size 8192

Dmesg:
[ 211.417152] switchtec: loading out-of-tree module taints kernel.
[ 211.417222] switchtec: module verification failed: signature and/or required key missing - tainting kernel
[ 211.418519] switchtec 0000:3b:00.1: enabling device (0140 -> 0142)
[ 211.421294] switchtec switchtec0: Management device registered.
[ 211.421556] switchtec 0000:86:00.1: enabling device (0140 -> 0142)
[ 211.424389] switchtec switchtec1: Management device registered.
[ 211.424439] switchtec: loaded.
[ 211.428558] ntb_hw_switchtec:switchtec_ntb_init_sndev:913: switchtec switchtec0: Partition ID 0 of 4
[ 211.428713] ntb_hw_switchtec:switchtec_ntb_init_mw:1227: switchtec switchtec0: MWs: 2 direct, 32 lut
[ 211.428828] ntb_hw_switchtec:switchtec_ntb_init_mw:1237: switchtec switchtec0: Peer MWs: 2 direct, 32 lut
[ 211.480112] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec0: Requester ID 00:00.0 -> BB:01.0
[ 211.480134] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec0: Requester ID 3A:00.0 -> BB:01.1
[ 211.532022] switchtec switchtec0: Using crosslink configuration
[ 211.532099] ntb_hw_switchtec:crosslink_enum_partition:1128: switchtec switchtec0: Crosslink BAR0 addr: 0
[ 211.532135] ntb_hw_switchtec:crosslink_enum_partition:1128: switchtec switchtec0: Crosslink BAR2 addr: 1000000000
[ 211.532172] ntb_hw_switchtec:crosslink_enum_partition:1128: switchtec switchtec0: Crosslink BAR4 addr: 2000000000
[ 211.792034] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec0: Requester ID 00:01.0 -> BB:02.0
[ 211.792056] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec0: Requester ID 00:01.1 -> BB:02.1
[ 211.844037] ntb_hw_switchtec:switchtec_ntb_init_db:1274: switchtec switchtec0: dbs: shift 0/0, mask 0fffffffffffffff
[ 211.948006] ntb_hw_switchtec:switchtec_ntb_init_shared_mw:1370: switchtec switchtec0: Shared MW Ready
[ 211.948029] ntb_hw_switchtec:switchtec_ntb_init_db_msg_irq:1442: switchtec switchtec0: irqs - event: 2, db: 0, msgs: 1
[ 211.948080] switchtec switchtec0: NTB device registered
[ 211.948131] ntb_hw_switchtec:switchtec_ntb_init_sndev:913: switchtec switchtec1: Partition ID 1 of 4
[ 211.948253] ntb_hw_switchtec:switchtec_ntb_init_mw:1227: switchtec switchtec1: MWs: 2 direct, 32 lut
[ 211.948373] ntb_hw_switchtec:switchtec_ntb_init_mw:1237: switchtec switchtec1: Peer MWs: 2 direct, 32 lut
[ 212.000049] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec1: Requester ID 00:00.0 -> BB:01.4
[ 212.000072] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec1: Requester ID 85:00.0 -> BB:01.5
[ 212.051993] switchtec switchtec1: Using crosslink configuration
[ 212.052069] ntb_hw_switchtec:crosslink_enum_partition:1128: switchtec switchtec1: Crosslink BAR0 addr: 0
[ 212.052106] ntb_hw_switchtec:crosslink_enum_partition:1128: switchtec switchtec1: Crosslink BAR2 addr: 1000000000
[ 212.052143] ntb_hw_switchtec:crosslink_enum_partition:1128: switchtec switchtec1: Crosslink BAR4 addr: 2000000000
[ 212.312006] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec1: Requester ID 00:01.4 -> BB:02.4
[ 212.312028] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec1: Requester ID 00:01.5 -> BB:02.5
[ 212.363990] ntb_hw_switchtec:switchtec_ntb_init_db:1274: switchtec switchtec1: dbs: shift 0/0, mask 0fffffffffffffff
[ 212.468009] ntb_hw_switchtec:switchtec_ntb_init_shared_mw:1370: switchtec switchtec1: Shared MW Ready
[ 212.468034] ntb_hw_switchtec:switchtec_ntb_init_db_msg_irq:1442: switchtec switchtec1: irqs - event: 2, db: 0, msgs: 1
[ 212.468115] switchtec switchtec1: NTB device registered
[ 212.474696] ntb_hw_switchtec:switchtec_ntb_link_enable:603: switchtec switchtec0: enabling link
[ 212.474700] ntb_tool:tool_link_event:170: ntb_tool 0000:3b:00.1: link is down speed 0 width 0
[ 212.474771] ntb_hw_switchtec:switchtec_ntb_link_enable:603: switchtec switchtec1: enabling link
[ 212.474774] ntb_tool:tool_link_event:170: ntb_tool 0000:86:00.1: link is down speed 0 width 0
[ 212.493665] ntb_hw_switchtec:switchtec_ntb_mw_set_trans:319: switchtec switchtec0: MW 1: part 0 addr 0x0000000034c02000 size 0x0000000000002000

$ echo 20971520 > /sys/kernel/debug/ntb_tool/0000\:3b\:00.1/peer_trans1

[ 113.848900] switchtec: loading out-of-tree module taints kernel.
[ 113.848947] switchtec: module verification failed: signature and/or required key missing - tainting kernel
[ 113.850239] switchtec 0000:3b:00.1: enabling device (0140 -> 0142)
[ 113.852935] switchtec switchtec0: Management device registered.
[ 113.853166] switchtec 0000:86:00.1: enabling device (0140 -> 0142)
[ 113.856024] switchtec switchtec1: Management device registered.
[ 113.856089] switchtec: loaded.
[ 113.860198] ntb_hw_switchtec:switchtec_ntb_init_sndev:913: switchtec switchtec0: Partition ID 0 of 4
[ 113.860348] ntb_hw_switchtec:switchtec_ntb_init_mw:1227: switchtec switchtec0: MWs: 2 direct, 32 lut
[ 113.860464] ntb_hw_switchtec:switchtec_ntb_init_mw:1237: switchtec switchtec0: Peer MWs: 2 direct, 32 lut
[ 113.911797] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec0: Requester ID 00:00.0 -> BB:01.0
[ 113.911819] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec0: Requester ID 3A:00.0 -> BB:01.1
[ 113.963711] switchtec switchtec0: Using crosslink configuration
[ 113.963787] ntb_hw_switchtec:crosslink_enum_partition:1128: switchtec switchtec0: Crosslink BAR0 addr: 0
[ 113.963824] ntb_hw_switchtec:crosslink_enum_partition:1128: switchtec switchtec0: Crosslink BAR2 addr: 1000000000
[ 113.963860] ntb_hw_switchtec:crosslink_enum_partition:1128: switchtec switchtec0: Crosslink BAR4 addr: 2000000000
[ 114.223723] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec0: Requester ID 00:01.0 -> BB:02.0
[ 114.223745] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec0: Requester ID 00:01.1 -> BB:02.1
[ 114.275731] ntb_hw_switchtec:switchtec_ntb_init_db:1274: switchtec switchtec0: dbs: shift 0/0, mask 0fffffffffffffff
[ 114.379693] ntb_hw_switchtec:switchtec_ntb_init_shared_mw:1370: switchtec switchtec0: Shared MW Ready
[ 114.379715] ntb_hw_switchtec:switchtec_ntb_init_db_msg_irq:1442: switchtec switchtec0: irqs - event: 2, db: 0, msgs: 1
[ 114.379768] switchtec switchtec0: NTB device registered
[ 114.379819] ntb_hw_switchtec:switchtec_ntb_init_sndev:913: switchtec switchtec1: Partition ID 1 of 4
[ 114.379940] ntb_hw_switchtec:switchtec_ntb_init_mw:1227: switchtec switchtec1: MWs: 2 direct, 32 lut
[ 114.380060] ntb_hw_switchtec:switchtec_ntb_init_mw:1237: switchtec switchtec1: Peer MWs: 2 direct, 32 lut
[ 114.431741] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec1: Requester ID 00:00.0 -> BB:01.4
[ 114.431764] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec1: Requester ID 85:00.0 -> BB:01.5
[ 114.483686] switchtec switchtec1: Using crosslink configuration
[ 114.483754] ntb_hw_switchtec:crosslink_enum_partition:1128: switchtec switchtec1: Crosslink BAR0 addr: 0
[ 114.483792] ntb_hw_switchtec:crosslink_enum_partition:1128: switchtec switchtec1: Crosslink BAR2 addr: 1000000000
[ 114.483829] ntb_hw_switchtec:crosslink_enum_partition:1128: switchtec switchtec1: Crosslink BAR4 addr: 2000000000
[ 114.743705] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec1: Requester ID 00:01.4 -> BB:02.4
[ 114.743727] ntb_hw_switchtec:config_req_id_table:999: switchtec switchtec1: Requester ID 00:01.5 -> BB:02.5
[ 114.795686] ntb_hw_switchtec:switchtec_ntb_init_db:1274: switchtec switchtec1: dbs: shift 0/0, mask 0fffffffffffffff
[ 114.899664] ntb_hw_switchtec:switchtec_ntb_init_shared_mw:1370: switchtec switchtec1: Shared MW Ready
[ 114.899688] ntb_hw_switchtec:switchtec_ntb_init_db_msg_irq:1442: switchtec switchtec1: irqs - event: 2, db: 0, msgs: 1
[ 114.899731] switchtec switchtec1: NTB device registered
[ 114.906116] ntb_hw_switchtec:switchtec_ntb_link_enable:603: switchtec switchtec0: enabling link
[ 114.906120] ntb_tool:tool_link_event:170: ntb_tool 0000:3b:00.1: link is down speed 0 width 0
[ 114.906185] ntb_hw_switchtec:switchtec_ntb_link_enable:603: switchtec switchtec1: enabling link
[ 114.906188] ntb_tool:tool_link_event:170: ntb_tool 0000:86:00.1: link is down speed 0 width 0
[ 114.925778] switchtec 0000:3b:00.1: swiotlb buffer is full (sz: 20971520 bytes)
[ 114.925779] swiotlb: coherent allocation failed for device 0000:3b:00.1 size=20971520
[ 114.925781] CPU: 30 PID: 1862 Comm: ntb_driver_setu Tainted: G OE 4.14.17 #2
[ 114.925783] Call Trace:
[ 114.925792] dump_stack+0x5c/0x83
[ 114.925798] swiotlb_alloc_coherent+0x14c/0x160
[ 114.925804] tool_peer_mw_trans_write+0x1b2/0x320 [ntb_tool]
[ 114.925808] ? perf_trace_mm_compaction_migratepages+0x5a/0x120
[ 114.925814] full_proxy_write+0x5a/0xa0
[ 114.925820] __vfs_write+0x33/0x170
[ 114.925824] ? handle_mm_fault+0xc4/0x1d0
[ 114.925830] ? __audit_syscall_entry+0xaf/0x100
[ 114.925831] vfs_write+0xad/0x1a0
[ 114.925833] SyS_write+0x52/0xc0
[ 114.925838] do_syscall_64+0x61/0x1a0
[ 114.925844] entry_SYSCALL64_slow_path+0x25/0x25
[ 114.925846] RIP: 0033:0x7effb38e6840
[ 114.925847] RSP: 002b:00007ffefbae83f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[ 114.925849] RAX: ffffffffffffffda RBX: 0000000000000009 RCX: 00007effb38e6840
[ 114.925849] RDX: 0000000000000009 RSI: 00007effb4205000 RDI: 0000000000000001
[ 114.925850] RBP: 00007effb4205000 R08: 000000000000000a R09: 00007effb41ff740
[ 114.925851] R10: 00007ffefbae81a0 R11: 0000000000000246 R12: 00007effb3bba400
[ 114.925852] R13: 0000000000000009 R14: 0000000000000001 R15: 0000000000000000

--
Best regards, Boris
Let me google that for you
--
You received this message because you are subscribed to the Google Groups "linux-ntb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to linux-ntb+...@googlegroups.com.
To post to this group, send email to linu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/linux-ntb/52648710-cf4a-9072-c03b-9a0f3fe7c19d%40deltatee.com.

Glimcher, Boris

unread,
May 31, 2018, 3:44:20 PM5/31/18
to Dave Jiang, linu...@googlegroups.com

Dave,

 

How can I enable IOMMU?

Are you referring to “NTB: PCI Quirk to Enable Switchtec NT Functionality with IOMMU On” patch?

 

--

Best regards, Boris

Let me google that for you

 

From: linu...@googlegroups.com [mailto:linu...@googlegroups.com] On Behalf Of Dave Jiang
Sent: Thursday, May 31, 2018 22:31
To: Glimcher, Boris; linu...@googlegroups.com
Subject: Re: ntb_tool cannot dma_alloc_coherent with CMA (over switchtec)

 

 

On 5/31/2018 9:32 AM, Glimcher, Boris wrote:

Dave Jiang

unread,
May 31, 2018, 4:15:34 PM5/31/18
to Glimcher, Boris, linu...@googlegroups.com


On 05/31/2018 12:44 PM, Glimcher, Boris wrote:
> Dave,
>
>  
>
> How can I enable IOMMU?

Turn VTd on in your BIOS. Set intel_iommu=on as a kernel boot parameter.
If you are using Intel CPU.

>
> Are you referring to “NTB: PCI Quirk to Enable Switchtec NT
> Functionality with IOMMU On” patch?
>
>  
>
> --
>
> Best regards, Boris
>
> Let me google that for you <http://lmgtfy.com/>
>
>  
>
> *From:*linu...@googlegroups.com [mailto:linu...@googlegroups.com]
> *On Behalf Of *Dave Jiang
> *Sent:* Thursday, May 31, 2018 22:31
> *To:* Glimcher, Boris; linu...@googlegroups.com
> *Subject:* Re: ntb_tool cannot dma_alloc_coherent with CMA (over switchtec)
> <mailto:linux-ntb+...@googlegroups.com>.
> To post to this group, send email to linu...@googlegroups.com
> <mailto:linu...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/linux-ntb/8A1F3ACA2F40C04FBF91985D6F491D7C0176649B%40MX302CL01.corp.emc.com
> <https://groups.google.com/d/msgid/linux-ntb/8A1F3ACA2F40C04FBF91985D6F491D7C0176649B%40MX302CL01.corp.emc.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google
> Groups "linux-ntb" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to linux-ntb+...@googlegroups.com
> <mailto:linux-ntb+...@googlegroups.com>.
> To post to this group, send email to linu...@googlegroups.com
> <mailto:linu...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/linux-ntb/1f327d5f-5bd9-12ee-2569-8f66eb0034c2%40intel.com
> <https://groups.google.com/d/msgid/linux-ntb/1f327d5f-5bd9-12ee-2569-8f66eb0034c2%40intel.com?utm_medium=email&utm_source=footer>.
Reply all
Reply to author
Forward
0 new messages