[Lustre-discuss] Installing Lustre client on 2.6.18-194 kernel

[Arya Mazaheri]

Hi,
I have installed lustre client packages on a client node. But it doesn't mount the lustre file system from lustre server. It gets the following famous error:

$ mount -t lustre 192.168.0.1:/lustre /mnt/lustre
mount.lustre: mount 172.16.113.232:/lustre at /mnt/lustre failed: No such device
Are the lustre modules loaded?
Check /etc/modprobe.conf and /proc/filesystems
Note 'alias lustre llite' should be removed from modprobe.conf


As I was searching through the mailing list, I have noticed that lustre.ko should be present in this directory:
/lib/modules/2.6.18-194.17.4.el5/kernel/fs/lustre/lustre.ko

My current kernel is 2.6.18-194.17.4.el5. but lustre.ko is in 2.6.18-164.11.1.el5 instead. So I guessed that this may be the source of problem.

Any ideas?

Thanks

[Brian J. Murrell]

On 11-02-19 09:34 AM, Arya Mazaheri wrote:
>
> As I was searching through the mailing list, I have noticed that lustre.ko
> should be present in this directory:
> /lib/modules/2.6.18-194.17.4.el5/kernel/fs/lustre/lustre.ko
>
> My current kernel is 2.6.18-194.17.4.el5. but lustre.ko is in
> 2.6.18-164.11.1.el5 instead.

You need the client modules package that matches your kernel. If one is
not available you will have to build it from the source.

b.

[Arya Mazaheri]

Well, I want to install client module on rocks 5.5 x86_64.
there are some packages in source section of lustre download area. I am confused which one to choose? what are the differences between them?
lustre-client-source-2.0.0.1-2.6.18_164.11.1.0.1.el5_lustre.2.0.0.1.i686.rpm
lustre-client-source-2.0.0.1-2.6.16_60_0.42.8_lustre.2.0.0.1_smp.x86_64.rpm
lustre-client-source-2.0.0.1-2.6.27_23_0.1_lustre.2.0.0.1_default.x86_64.rpm
lustre-client-source-2.0.0.1-2.6.18_164.11.1.0.1.el5_lustre.2.0.0.1.x86_64.rpm
lustre-client-source-2.0.0.1-2.6.16_60_0.42.8_lustre.2.0.0.1_bigsmp.i686.rpm
lustre-client-source-2.0.0.1-2.6.27_23_0.1_lustre.2.0.0.1_default.i686.rpm
lustre-client-source-2.0.0.1-2.6.18_164.11.1.el5_lustre.2.0.0.1.i686.rpm
lustre-client-source-2.0.0.1-2.6.18_164.11.1.el5_lustre.2.0.0.1.x86_64.rpm

and one other thing! where is the source for lustre-client-modules?

_______________________________________________
Lustre-discuss mailing list
Lustre-...@lists.lustre.org
http://lists.lustre.org/mailman/listinfo/lustre-discuss

[Albert Everett]

Our kernel is also 2.6.18_194.17.4.el5.

We installed OFED 1.5.2 from source, following this guide:

https://wiki.rocksclusters.org/wiki/index.php/Install_OFED_1.5.x_on_a_Rocks_5.3_cluster

... which left us, among other things, a folder /usr/src/ofa_kernel.

Lustre on the server side is handled by our vendor, so all we needed
to worry about is the client.

To build a lustre client, we then installed lustre-1.8.5.tar.gz from
source, not from rpms. Our first compile produced the error you show
below.
# ./configure --with-linux=/lib/modules/`uname -r`/build
# make rpms

To get the lustre installation to use our new OFED, we tried this and
it worked.

# ./configure --with-o2ib=/usr/src/ofa_kernel --with-linux=/lib/
modules/`uname -r`/build
# make rpms

RPMs showed up in /usr/src/redhat/RPMS/x86_64, and we are using
lustre-1.8.5*.rpm and lustre-modules-*.rpm on our client machines.

Albert

[Arya Mazaheri]

Hi Albert,
It seems that you have made a new kernel in order to run lustre on clients. Am I right?
I don't want to change kernel on clients at all...

On Sat, Feb 19, 2011 at 8:57 PM, Albert Everett <aeev...@ualr.edu> wrote:

Our kernel is also 2.6.18_194.17.4.el5.

We installed OFED 1.5.2 from source, following this guide:

 https://wiki.rocksclusters.org/wiki/index.php/Install_OFED_1.5.x_on_a_Rocks_5.3_cluster

... which left us, among other things, a folder /usr/src/ofa_kernel.

Lustre on the server side is handled by our vendor, so all we needed to worry about is the client.

To build a lustre client, we then installed lustre-1.8.5.tar.gz from source, not from rpms. Our first compile produced the error you show below.
# ./configure --with-linux=/lib/modules/`uname -r`/build
# make rpms

To get the lustre installation to use our new OFED, we tried this and it worked.

# ./configure --with-o2ib=/usr/src/ofa_kernel --with-linux=/lib/modules/`uname -r`/build
# make rpms

[Albert Everett]

For lustre client, we did not need to alter our kernel at all. We just
made and installed lustre-1.8.5 and lustre-modules-1.8.5 rpms. /etc/
modprobe.conf needs a tweak.

For lustre server, I believe you will need to deal with a patched
kernel. We have not been down this road yet since our vendor includes
lustre server software with their hardware.

Albert

[Arya Mazaheri]

I have done what you said. I will test my client to the server tomorrow. but would you tell me the tweaks you have done on /etc/modprobe.conf ?

On Sun, Feb 20, 2011 at 1:56 AM, Albert Everett <aeev...@ualr.edu> wrote:

For lustre client, we did not need to alter our kernel at all. We just made and installed lustre-1.8.5 and lustre-modules-1.8.5 rpms. /etc/modprobe.conf needs a tweak.

For lustre server, I believe you will need to deal with a patched kernel. We have not been down this road yet since our vendor includes lustre server software with their hardware.

Albert

On Feb 19, 2011, at 12:18 PM, Arya Mazaheri wrote:

Hi Albert,
It seems that you have made a new kernel in order to run lustre on clients. Am I right?
I don't want to change kernel on clients at all...

On Sat, Feb 19, 2011 at 8:57 PM, Albert Everett <aeev...@ualr.edu> wrote:
Our kernel is also 2.6.18_194.17.4.el5.

We installed OFED 1.5.2 from source, following this guide:

 https://wiki.rocksclusters.org/wiki/index.php/Install_OFED_1.5.x_on_a_Rocks_5.3_cluster

... which left us, among other things, a folder /usr/src/ofa_kernel.

Lustre on the server side is handled by our vendor, so all we needed to worry about is the client.

To build a lustre client, we then installed lustre-1.8.5.tar.gz from source, not from rpms. Our first compile produced the error you show below.
# ./configure --with-linux=/lib/modules/`uname -r`/build
# make rpms

To get the lustre installation to use our new OFED, we tried this and it worked.

# ./configure --with-o2ib=/usr/src/ofa_kernel --with-linux=/lib/modules/`uname -r`/build
# make rpms

[Albert Everett]

Here's what's in our /etc/modprobe.conf related to IB and lustre:

options ib_mthca msi_x=1
options lnet networks=o2ib0(ib0)
options ko2iblnd ipif_name=ib0

We have Mellanox Infinihost (III?) DDR cards and IPs defined for them.

$ /sbin/ifconfig ib0
ib0 Link encap:InfiniBand HWaddr 80:00:04:04:FE:
80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
inet addr:192.168.2.1 Bcast:192.168.2.255 Mask:
255.255.255.0
inet6 addr: fe80::202:c902:29:b341/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1
RX packets:1719 errors:0 dropped:0 overruns:0 frame:0
TX packets:30 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:256
RX bytes:96564 (94.3 KiB) TX bytes:2420 (2.3 KiB)

Albert

[Arya Mazaheri]

Thanks Albert, I really appreciate you...
Now everything is working...

On Mon, Feb 21, 2011 at 7:44 PM, Albert Everett <aeev...@ualr.edu> wrote:

Here's what's in our /etc/modprobe.conf related to IB and lustre:

options ib_mthca msi_x=1
options lnet networks=o2ib0(ib0)
options ko2iblnd ipif_name=ib0

We have Mellanox Infinihost (III?) DDR cards and IPs defined for them.

$ /sbin/ifconfig ib0
ib0       Link encap:InfiniBand  HWaddr 80:00:04:04:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
         inet addr:192.168.2.1  Bcast:192.168.2.255  Mask:255.255.255.0
         inet6 addr: fe80::202:c902:29:b341/64 Scope:Link
         UP BROADCAST RUNNING MULTICAST  MTU:65520  Metric:1
         RX packets:1719 errors:0 dropped:0 overruns:0 frame:0
         TX packets:30 errors:0 dropped:0 overruns:0 carrier:0
         collisions:0 txqueuelen:256
         RX bytes:96564 (94.3 KiB)  TX bytes:2420 (2.3 KiB)

Albert

On Feb 20, 2011, at 2:20 PM, Arya Mazaheri wrote:

I have done what you said. I will test my client to the server tomorrow. but would you tell me the tweaks you have done on /etc/modprobe.conf ?

On Sun, Feb 20, 2011 at 1:56 AM, Albert Everett <aeev...@ualr.edu> wrote:

For lustre client, we did not need to alter our kernel at all. We just made and installed lustre-1.8.5 and lustre-modules-1.8.5 rpms. /etc/modprobe.conf needs a tweak.

For lustre server, I believe you will need to deal with a patched kernel. We have not been down this road yet since our vendor includes lustre server software with their hardware.

Albert


On Feb 19, 2011, at 12:18 PM, Arya Mazaheri wrote:

Hi Albert,
It seems that you have made a new kernel in order to run lustre on clients. Am I right?
I don't want to change kernel on clients at all...

On Sat, Feb 19, 2011 at 8:57 PM, Albert Everett <aeev...@ualr.edu> wrote:
Our kernel is also 2.6.18_194.17.4.el5.

We installed OFED 1.5.2 from source, following this guide:

 https://wiki.rocksclusters.org/wiki/index.php/Install_OFED_1.5.x_on_a_Rocks_5.3_cluster

... which left us, among other things, a folder /usr/src/ofa_kernel.

Lustre on the server side is handled by our vendor, so all we needed to worry about is the client.

To build a lustre client, we then installed lustre-1.8.5.tar.gz from source, not from rpms. Our first compile produced the error you show below.
# ./configure --with-linux=/lib/modules/`uname -r`/build
# make rpms

To get the lustre installation to use our new OFED, we tried this and it worked.

# ./configure --with-o2ib=/usr/src/ofa_kernel --with-linux=/lib/modules/`uname -r`/build
# make rpms

[Albert Everett]

Excellent - thanks for letting us know.

Thanks again to Tim Carlson for his OFED wiki, and to Terascala for
help with building up the lustre client.

Albert