Re: Problem with ntb_transport + ntb_netdev

[Jon Mason]

Adding the NTB mailing list

On Mon, Jan 9, 2017 at 11:27 AM, perplexus <perp...@ya.ru> wrote:
> Hello.
>
> I'm working with NTB (kernel version 4.10) on Xeon D-1527 CPU and have a problem with couple ntb_transport + ntb_netdev.
>
> On the local side I run:
> sudo /etc/init.d/irqbalance stop
> sudo modprobe ntb_transport dyndbg=+p
> sudo modprobe ntb_netdev dyndbg=+p
> sudo ifconfig eth0 192.168.1.1 up
>
> On the remote side I run:
> sudo /etc/init.d/irqbalance stop
> sudo modprobe ntb_transport dyndbg=+p
> sudo modprobe ntb_netdev dyndbg=+p
> sudo ifconfig eth0 192.168.1.2 up
>
> Along with these steps I run ping 192.168.1.2 and get about 2-3 successfull transmitted/received packets then get 'Destination host unreacheble'.
>

Do pings in the other direction work?

[Dave Jiang]

How are the NTBs connected? Where did the get the hardware? Also can you
dump the stats in debugfs? And which distro are you running? Sometimes
Network Manager will yank the IP for no reason if you don't configure it
correctly and thus your interface can't route.

[shle...@gmail.com]

> Do pings in the other direction work? 

Similarly

>How are the NTBs connected?

B2B

>And which distro are you running?

Ubuntu 16.10 with kernel 4.10.0-rc1 from git

>  Sometimes 

Network Manager will yank the IP for no reason if you don't configure it 

correctly and thus your interface can't route.

I stop network-manager service before load the modules and set IP manually.

> Also can you dump the stats in debugfs?

ntb_hw_intel_info_1 - one side

ntb_hw_intel_info_2 - second side

ntb_transport_info_1 - one side

ntb_transport_info_2 - second side

среда, 11 января 2017 г., 1:24:17 UTC+3 пользователь dave.jiang написал:

[shle...@gmail.com]

dmesg

[Dave Jiang]

On 01/11/2017 03:08 AM, shle...@gmail.com wrote:
>> Do pings in the other direction work?
> Similarly
>>How are the NTBs connected?
> B2B
>>And which distro are you running?
> Ubuntu 16.10 with kernel 4.10.0-rc1 from git
>> Sometimes
> Network Manager will yank the IP for no reason if you don't configure it
> correctly and thus your interface can't route.
> I stop network-manager service before load the modules and set IP manually.
>
>> Also can you dump the stats in debugfs?

> ntb_hw_intel_info_1 <http://pastebin.com/eyMZzSik> - one side
> ntb_hw_intel_info_2 <http://pastebin.com/vF94zLBk> - second side
> ntb_transport_info_1 <http://pastebin.com/Ktq1yAqa> - one side
> ntb_transport_info_2 <http://pastebin.com/w8r08ezB> - second side

Your secondary BAR2 size is not the same as your primary. They both need
to be 22. Also all the upstream memory miss is concerning. You should
have none. You may have signal integrity issues. But fix the bar size
first and see if that helps.

> --
> You received this message because you are subscribed to the Google
> Groups "linux-ntb" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to linux-ntb+...@googlegroups.com
> <mailto:linux-ntb+...@googlegroups.com>.
> To post to this group, send email to linu...@googlegroups.com
> <mailto:linu...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/linux-ntb/c18c3dc2-d242-4352-946e-931ff13e1d28%40googlegroups.com
> <https://groups.google.com/d/msgid/linux-ntb/c18c3dc2-d242-4352-946e-931ff13e1d28%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Dave Jiang]

On 01/11/2017 09:42 AM, Dave Jiang wrote:
>
>
> On 01/11/2017 03:08 AM, shle...@gmail.com wrote:
>>> Do pings in the other direction work?
>> Similarly
>>> How are the NTBs connected?
>> B2B
>>> And which distro are you running?
>> Ubuntu 16.10 with kernel 4.10.0-rc1 from git
>>> Sometimes
>> Network Manager will yank the IP for no reason if you don't configure it
>> correctly and thus your interface can't route.
>> I stop network-manager service before load the modules and set IP manually.
>>
>>> Also can you dump the stats in debugfs?
>> ntb_hw_intel_info_1 <http://pastebin.com/eyMZzSik> - one side
>> ntb_hw_intel_info_2 <http://pastebin.com/vF94zLBk> - second side
>> ntb_transport_info_1 <http://pastebin.com/Ktq1yAqa> - one side
>> ntb_transport_info_2 <http://pastebin.com/w8r08ezB> - second side
>
> Your secondary BAR2 size is not the same as your primary. They both need
> to be 22. Also all the upstream memory miss is concerning. You should
> have none. You may have signal integrity issues. But fix the bar size
> first and see if that helps.

Also, if you are using Broadwell Xeon D, you should use the kernel from
here and not upstream.
https://github.com/davejiang/linux
branch: davejiang/ntb

[shle...@gmail.com]

Both bars size are equal to 22.

[shle...@gmail.com]

I built this kernel.

Unfortunately link doesn't train.

ntb_hw_intel_1 - 1'st side

ntb_hw_intel_2 - 2'nd side

среда, 11 января 2017 г., 19:45:05 UTC+3 пользователь dave.jiang написал:

[Dave Jiang]

Need to see your dmesg with dyndbg for the ntb_hw_intel.ko but your
SBAR23SZ is still wrong. You are not configuring the secondary BAR2/3
size correctly in BIOS. Can you provide all your NTB BIOS config options?

On 01/12/2017 08:29 AM, shle...@gmail.com wrote:
> I built this kernel.
> Unfortunately link doesn't train.
>

> ntb_hw_intel_1 <http://pastebin.com/ZvCK9yu5> - 1'st side
> ntb_hw_intel_2 <http://pastebin.com/p8BJ83pv> - 2'nd side

> https://github.com/davejiang/linux <https://github.com/davejiang/linux>

> >> an email to linux-ntb+...@googlegroups.com <javascript:>
> >> <mailto:linux-ntb+...@googlegroups.com <javascript:>>.

> >> To post to this group, send email to linu...@googlegroups.com

> <javascript:>
> >> <mailto:linu...@googlegroups.com <javascript:>>.

> >> To view this discussion on the web visit
> >>
> https://groups.google.com/d/msgid/linux-ntb/c18c3dc2-d242-4352-946e-931ff13e1d28%40googlegroups.com
> <https://groups.google.com/d/msgid/linux-ntb/c18c3dc2-d242-4352-946e-931ff13e1d28%40googlegroups.com>
>
> >>

> <https://groups.google.com/d/msgid/linux-ntb/c18c3dc2-d242-4352-946e-931ff13e1d28%40googlegroups.com?utm_medium=email&utm_source=footer

> <https://groups.google.com/d/msgid/linux-ntb/c18c3dc2-d242-4352-946e-931ff13e1d28%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> >> For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

> >
>
> --
> You received this message because you are subscribed to the Google
> Groups "linux-ntb" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to linux-ntb+...@googlegroups.com
> <mailto:linux-ntb+...@googlegroups.com>.
> To post to this group, send email to linu...@googlegroups.com
> <mailto:linu...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/linux-ntb/1d63eddd-e9b2-4023-8e0c-6bc658633b9a%40googlegroups.com
> <https://groups.google.com/d/msgid/linux-ntb/1d63eddd-e9b2-4023-8e0c-6bc658633b9a%40googlegroups.com?utm_medium=email&utm_source=footer>.

[shle...@gmail.com]

> your SBAR23SZ is still wrong.

Oh, thank you. It works!