Unable to load kernel modules (/lib/modules does not exist)?

[Marc Sune]

Hi folks,

First, I would like to thank the community for all the efforts around ONL.

We have some EdgeCore AS5712 (Atom) in which we have compiled and loaded ONL without problems. However, we are unable to load some kernel modules of the Broadcom SDK on them. The kernel does have the kernel modules support (/proc/modules exist), but /lib/modules folder does not even exist, which lead the insmod/modprobe to fail:

root@onl-x86:/# insmod ./<module_name>

libkmod: ERROR ../libkmod/libkmod.c:554 kmod_search_moddep: could not open moddep file '/lib/modules/3.2.65-1+deb7u2-OpenNetworkLinux/modules.dep.bin'

I think I am missing something trivial here; is there any configuration build step that we missed?

Thank you and regards

marc

[Rob Sherwood]

Hi Marc,

Good to hear from you.

A couple of questions: are you sure you're running ONL on the 5712?  We have a pull request for 5712 support in the queue that we're testing, but the master branch of ONL does not yet have 5712 support.  If it really is a 5712 (`cat /etc/machine.conf`), where did you get the code from?

Independent of that, check out the /usr/sbin/brcm-modules-init.sh script for how the modules are supposed to be loaded (https://github.com/opennetworklinux/ONL/blob/master/builds/swi/all/shared/usr/sbin/brcm-modules-init.sh).  There are some devices that need to be made and there are typically module parameters that need to be set.

If you can confirm that it's a 5712 and where you got the code from, I can probably better direct you.

Thanks,

- Rob

.

--
You received this message because you are subscribed to the Google Groups "opennetworklinux" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opennetworklin...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Marc Sune]

On Wednesday, August 26, 2015 at 12:47:11 AM UTC+2, Rob Sherwood wrote:

Hi Marc,

Good to hear from you.

HI Rob,

Nice to hear again from you too. I hope you are doing fine ;)

 

A couple of questions: are you sure you're running ONL on the 5712?  We have a pull request for 5712 support in the queue that we're testing, but the master branch of ONL does not yet have 5712 support.  If it really is a 5712 (`cat /etc/machine.conf`), where did you get the code from?

I was not building the ONL image myself, but I am quite sure it was compiled in master branch. I will double check the exact model and the machine.conf file too.

Looking at the github repository, there _seems_ to be some PRs with support for this switch already merged:

https://github.com/opennetworklinux/ONL/pull/119

https://github.com/opennetworklinux/ONL/pull/113

https://github.com/opennetworklinux/ONL/pull/130

Kernel matches PR #119 (3.2.65)

 

Independent of that, check out the /usr/sbin/brcm-modules-init.sh script for how the modules are supposed to be loaded (https://github.com/opennetworklinux/ONL/blob/master/builds/swi/all/shared/usr/sbin/brcm-modules-init.sh).  There are some devices that need to be made and there are typically module parameters that need to be set.

We were creating them manually, but thanks for pointing to the script. The path /lib/modules/`uname -r`/ simply does not exist in our ONL installation, not even /etc/modules. Maybe it boils down to this switch not being yet fully supported (though the system boots)?

 

If you can confirm that it's a 5712 and where you got the code from, I can probably better direct you.

I will.

thanks

marc

[Marc Sune]

Rob, all,

Here are the details:

switch: AS5712-54X

ONL compiled in master branch, HEAD:

admin@builder:/opt/onl$ git show HEAD

commit 770b95cb0e4cba7281a1957edac153b2dbdf0e69

Merge: 5dde334 044361b

Author: Jeffrey Townsend <>

Date:   Sun Aug 2 11:21:11 2015 -0700

    Merge pull request #130 from brandonchuang/master

    

    [as5712_54x] Add initial script for device drivers(psu/cpld/sfp/thermal)

Built following the guide:

• apt-get install lxc docker.io
  
• make docker
  
• make onl-x86
  

Installed:

• Get installer from previous build: /opt/onl/builds/installer/amd64/all/onl-770b95c-amd64-all.2015.08.19.09.38.installer
  
• From ONIE; onie-nos-install onl-770b95c-amd64-all.2015.08.19.09.38.installer
  

Linux onl-x86 3.2.65-1+deb7u2-OpenNetworkLinux #1 SMP Wed Aug 19 09:18:39 UTC 2015 x86_64 GNU/Linux

There is no /etc/machine.conf in the ONL installation.

Thank you and regards
marc

[Rob Sherwood]

Ha -- ok.  I think I understand what happened.  You've grabbed code from master while some but not all of the as5712 patches have landed.  I believe more patches have landed today but not everything that's needed to run Indigo + OFDPA.  I'll send mail once everything has landed properly so that you can use this platform with Indigo and OF-DPA.

Sound good?

- Rob

.

[Marc Sune]

On Wednesday, August 26, 2015 at 11:42:36 PM UTC+2, Rob Sherwood wrote:

Ha -- ok.  I think I understand what happened.  You've grabbed code from master while some but not all of the as5712 patches have landed.  I believe more patches have landed today but not everything that's needed to run Indigo + OFDPA.

We are not using OFDPA / Indigo, but BRCM SDK. Should the patches already merged upstream be sufficient to boot the switch and have /etc/machine.conf load kernel modules/

 

  I'll send mail once everything has landed properly so that you can use this platform with Indigo and OF-DPA.

Sound good?

We will wait for the latest patches.

Thanks!

marc

[Rob Sherwood]

So, this mail confused me.  Are you trying to run your own build of the Broadcom SDK (meaning you're under NDA with Broadcom and have gotten the source from them)?  If so, you shouldn't need *our* copy of the kernel modules to load; you've build your own.

If you're looking for ONL to provide the SDK, unfortunately it does not provide raw access to the SDK.   The Broadcom SDK is both closed source *and* closed API, meaning we can't publish it nor can we publish code that references the SDK calls.  What we can publish is OF-DPA: it is closed source but *open* API.  OF-DPA is an "openflow-ish" view of the SDK, but can be used for non-OpenFlow as well as OpenFlow programs.

Please let me know which end you're interested in and I can try to help you in the right direction.

- Rob

.

[Marc Sune]

Hi Rob

On Sat, Aug 29, 2015 at 1:50 AM, Rob Sherwood <rob.sh...@bigswitch.com> wrote:

So, this mail confused me.  Are you trying to run your own build of the Broadcom SDK (meaning you're under NDA with Broadcom and have gotten the source from them)?  If so, you shouldn't need *our* copy of the kernel modules to load; you've build your own.

Yes. I am trying to run the kernel modules built with the closed-source OpenNSL + BRCM SDK.

Thanks

marc

[Rob Sherwood]

So, if you're trying to run OpenNSL, you're going to need the kernel modules that were compiled against the kernel that ONL uses -- which you basically need the source code for.  Also, there are some platform specific drivers that you will need as well.  Fwiw, downloading the binaries off of Broadcom's github site does not provide you any of these things.

If you have all of these things, you actually don't need anything else from us/ONL.  

- Rob

.

[Marc Sune]

I do,

If I may come back to the original question; is it normal that  /lib/modules does not exist in a standard ONL compilation?

Maybe it is and I am missing something trivial

Thanks

Marc

[Rob Sherwood]

No -- on any build that has orc support or OF-DPA (really, anything that installs kernel modules), the /lib/modules directory exists and is created as part of that support.  The AS5712 currently has support for neither of those applications, so no /lib/modules directory exists -- but that's simply a matter of the efficiency of the installer rather than some deeper design decision.

Hope that helps,

- Rob

.

[Liang-Wei Kang]

Hi Rob,

 

 We have OpenNSL and OFDPA debian packages ready for AS4600, AS5710, AS5712 and AS6712 swiches.

 Now, we are using dpkg -i "package_name" to install our OpenNSL and OFDPA packages on the above switches.

 

 Users have to execute dpkg -i XXX after poweron, or include it in /mnt/flash/rc.boot for auto installation.

 

 My questions are

1. is there a better way for auto installation of a debian package which is not in ONL swi image?

2. any easy way to add the packages into ONL swi image so as to be installed by default?

3. Could we put our deb packages  in apt.opennetliniux.org/debian/ and add our package names to ONL/builds/swi/$ARCH/all/rootfs/repo.$ARCH file?

 

Could you give me some suggestions? Thanks in advance.

 

/Lewis

Rob Sherwood於 2015年8月31日星期一 UTC+8下午3時38分51秒寫道：

[Junho Suh]

Hi Liang-Wei,

Couple of month ago in mailing list, you said Accton is ready to OFDPA debian package for AS5712 and AS6712 switches.

So here is my question that is it possible the OFDPA debian package working on Facebook Wedge?

Actually, recently we got Wedge and now I am trying to test Wedge + ONL with SDN controller (i.e., ONOS).

But in basic ONL package, there is no OFDPA debian packages for x86 based platform (i.e., Wedge).

I think the architecture of AS5712, AS6712 switches are same as Wedge, so the package might be working.

Thanks~

Junho Suh

[Steven Noble]

Hi Junho,

If you want to use the wedge, we only support opennsl.

We have a new build on the site under http://opennetlinux.org/binaries/accton-wedge

http://opennetlinux.org/binaries/accton-wedge/ONL-2.0.0-Deb8-Wedge-2015-12-19.2121-547e306-AMD64-INSTALLER is the base image.

This image will install directly to the SSD in the wedge so you can install software and it will persist.

Once installed you can do apt-get update; apt-get install fboss and it will install the necessary drivers for fboss along with the fboss binaries. You don't need to use fboss, but it's all we support at this time.

You can also install docker on the system if you want to: apt-get install docker-engine

As for OFDPA we have not looked at how much work it would be to support it.

--

[Junho Suh]

Thx Steven.

I installed the new build as you mentioned and installed fboss with apt, but there is still a problem when the opennsl modules are insmod.

When I insmod the modules which was installed with apt (linux-kernel-bde.ko, linux-user-bee.ko, ...), the result turns out as below:

linux_kernel_bde: version magic '3.18.25-OpenNetworkLinux SMP mod_unload ' should be '3.18.25-deb8-OpenNetworkLinux SMP mod_unload '

insmod: ERROR: could not insert module linux-kernel-bde.ko: Invalid module format

I think the module version does seem to not match with kernel version.

So, is there any build for "3.18.25-deb8-OpenNetworkLinux SMP” I can successfully insmod?

Thanks in advance.

[Steven Noble]

Hi Junho,

Things have been moving quickly and I forgot to push an updated version..

I just pushed it up and you can download/install it from here:

http://opennetlinux.org/binaries/accton-wedge/ONL-2.0.0-Deb8-Wedge-2016-01-26.2138-2fe0c93_AMD64_INSTALLER

If you install that build, it should have the correct kernel, etc.  It's the one I am currently running for testing.

Thanks!

[Junho Suh]

Thanks Steven.

At this time, the link doesn’t seem to work :)

Would you check one more time?

Cheers~

[Steven Noble]

Sorry there is a issue with _ on our server,

http://opennetlinux.org/binaries/accton-wedge/ONL-2.0.0-Deb8-Wedge-2016-01-26.2138-2fe0c93-AMD64-INSTALLER

The link works now.

Junho Suh

February 2, 2016 at 3:26 PM

Thanks Steven.

At this time, the link doesn’t seem to work :)

Would you check one more time?

Cheers~

Steven Noble

February 2, 2016 at 8:03 AM

Hi Junho,

Things have been moving quickly and I forgot to push an updated version..

I just pushed it up and you can download/install it from here:

http://opennetlinux.org/binaries/accton-wedge/ONL-2.0.0-Deb8-Wedge-2016-01-26.2138-2fe0c93_AMD64_INSTALLER

If you install that build, it should have the correct kernel, etc.  It's the one I am currently running for testing.

Thanks!

Junho Suh

February 2, 2016 at 5:49 AM

Thx Steven.

I installed the new build as you mentioned and installed fboss with apt, but there is still a problem when the opennsl modules are insmod.

When I insmod the modules which was installed with apt (linux-kernel-bde.ko, linux-user-bee.ko, ...), the result turns out as below:

linux_kernel_bde: version magic '3.18.25-OpenNetworkLinux SMP mod_unload ' should be '3.18.25-deb8-OpenNetworkLinux SMP mod_unload '

insmod: ERROR: could not insert module linux-kernel-bde.ko: Invalid module format

I think the module version does seem to not match with kernel version.

So, is there any build for "3.18.25-deb8-OpenNetworkLinux SMP” I can successfully insmod?

Thanks in advance.

Steven Noble

January 3, 2016 at 10:05 PM

Hi Junho,

If you want to use the wedge, we only support opennsl.

We have a new build on the site under http://opennetlinux.org/binaries/accton-wedge

http://opennetlinux.org/binaries/accton-wedge/ONL-2.0.0-Deb8-Wedge-2015-12-19.2121-547e306-AMD64-INSTALLER is the base image.

This image will install directly to the SSD in the wedge so you can install software and it will persist.

Once installed you can do apt-get update; apt-get install fboss and it will install the necessary drivers for fboss along with the fboss binaries. You don't need to use fboss, but it's all we support at this time.

You can also install docker on the system if you want to: apt-get install docker-engine

As for OFDPA we have not looked at how much work it would be to support it.

Junho Suh

January 3, 2016 at 9:50 PM

[Junho Suh]

Sorry for disturbing Steven.

There is still a problem after installing the image.

After boot up, the error messages are shown as below:

error: file `/kernel-3.18-x86_64-all' not found.

error: you need to load the kernel first.

Junho Suh

[Steven Noble]

Hi Junho,

Not a bother.. I am confused and will look in a bit. Can you mount the /dev/sdb3 on /mnt/onlboot and see if the kernel is there?

[Steven Noble]

Hi Junho,

I just realized that you would not have access to the loader.. This is a comedy of errors on my side sorry. I will get you a confirmed working build tonight.

[Junho Suh]

Thanks for your kind help. I am just writing an email about this and cancel :). I will wait until that moment.

Cheers~

[Steven Noble]

Hi Junho,

Image is up, same name, has the 3.18 kernel.

There is an issue with python in the build that requires you to do the following:

dhclient ma1 (to get proper dhcp for the interface)
apt-get update
apt-get remove python2.7 (answer yes to all the removal stuff)
apt-get install fboss

once that is done, run the service fboss_wedge_agent start (you may have to due it twice due to a library initialization issue) and then you should be good.

root@localhost:~# fboss_route.py list_routes
Route 10.11.0.110/31 --> 10.11.0.111
Route 10.11.8.110/31 --> 10.11.8.111
Route 10.11.16.110/31 --> 10.11.16.111

Junho Suh

February 2, 2016 at 4:43 PM

Thanks for your kind help. I am just writing an email about this and cancel :). I will wait until that moment.

Cheers~

Junho Suh

February 2, 2016 at 3:55 PM

Sorry for disturbing Steven.

There is still a problem after installing the image.

After boot up, the error messages are shown as below:

error: file `/kernel-3.18-x86_64-all' not found.

error: you need to load the kernel first.

Junho Suh

[Junho Suh]

Now all modules of OpenNSL are added successfully. yeah~

But there is still a problem when I connect fboss agent through thrift with fboss_route.py. The error msg is as below:

Traceback (most recent call last):

  File "/usr/local/bin/fboss_route.py", line 241, in <module>

    args.func(args)

  File "/usr/local/bin/fboss_route.py", line 82, in list_routes

    with get_client(args) as client:

  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__

    return self.gen.next()

  File "/usr/local/bin/fboss_route.py", line 144, in get_client

    transport.open()

  File "/usr/local/lib/python2.7/dist-packages/thrift/transport/TSocket.py", line 277, in open

    raise TTransportException(TTransportException.NOT_OPEN, message)

thrift.transport.TTransport.TTransportException: Could not connect to localhost:5909: error(111, 'Connection refused')

I can fix this error :)

Anyway, so far, our side doesn't need a fboss agent yet. Now is fine for me if OpenNSL modules are correctly working.

For our use case, I am now porting indigo agent on ONL on wedge with OpenNSL. I will share this try soon.

Thanks~

Junho Suh

[Steven Noble]

Hi Junho,

I find that I need to restart the fboss_wedge_agent if I get that error, service fboss_wedge_agent status will probably show not running.

[Steven Noble]

Btw, super excited about the indigo port! That would be awesome. If you run into issues, you can go to the 3.2 kernel to debug by updating the grub.cfg on the onl boot partition. I have kernel drivers for opennsl for both kernels.

mkdir /mnt/onlboot
mount /dev/sdb3 onlboot

You will find the grub.cfg under the grub directory.

[Junho Suh]

Yes. Now all things gonna be alright.

Cheers~

[Junho Suh]

Yes this work would be awesome, but one big challenge is how we implement OpenFlow features with OpenNSL, instead of OF-DPA.

Now the current version of port supports the partial OF functionalities such as L2, L3 tables.

Junho Suh

[Steven Noble]

Btw we are running an older (October 6.4.6.6) libopennsl due to issues with fboss. If you want to run the latest (Dec 22nd 6.4.6.8) the kernel modules should be compatible.