Re: [zfs-discuss] ARM support?

100 views
Skip to first unread message

Durval Menezes

unread,
Apr 1, 2014, 1:25:12 PM4/1/14
to zfs-...@googlegroups.com, Gordan Bobic, ray vantassle
Hi Gordan, Ray, 

(Moving this to the zfs-fuse list which is more on-topic) 

On Tue, Apr 1, 2014 at 11:17 AM, Durval Menezes <durval....@gmail.com> wrote:
[...]
I plan on using it here in my RPi (as soon as I get the leisure to modify my OpenELEC image to load it on boot), will keep you posted as to the developments..

On Tue, Apr 1, 2014 at 11:18 AM, Durval Menezes <durval....@gmail.com> wrote:
[...]
Thanks! Will give it a try in case the binaries I extracted from Gordan's RPM cause me any trouble. 

As promised (threatened? :-): 

A) With Gordan's (redsleeve) binaries: 

cd /tmp/zfs-fuse-0.7.0.20121023-6.el6.armv5tel/usr/local/bin
./zfs-fuse 
./zfs-fuse: error while loading shared libraries: libaio.so.1: cannot open shared object file: No such file or directory

ldd ./zfs-fuse
/lib/libarmmem.so (0xb6f01000)
librt.so.1 => /lib/librt.so.1 (0xb6ef2000)
libpthread.so.0 => /lib/libpthread.so.0 (0xb6ed3000)
libfuse.so.2 => /usr/lib/libfuse.so.2 (0xb6eae000)
libdl.so.2 => /lib/libdl.so.2 (0xb6ea3000)
libz.so.1 => /usr/lib/libz.so.1 (0xb6e91000)
libaio.so.1 => not found
libcrypto.so.10 => not found
libbz2.so.1 => not found
liblzo2.so.2 => not found
liblzma.so.0 => not found
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0xb6e6b000)
libc.so.6 => /lib/libc.so.6 (0xb6d46000)
ld-linux.so.3 => /lib/ld-linux.so.3 (0xb6d1e000)
/lib/ld-linux.so.3 (0xb6f0b000)
 
The situation with Ray's binaries are much improved,  just libaio.so missing:

cd /tmp/zfs-fuse-0.7.0-arm-bin.tar.gz/usr/local/sbin
ldd ./zfs-fuse
        ./zfs-fuse: /usr/lib/libcrypto.so.1.0.0: no version information available (required by ./zfs-fuse)
/lib/libarmmem.so (0xb6edf000)
librt.so.1 => /lib/librt.so.1 (0xb6ed0000)
libpthread.so.0 => /lib/libpthread.so.0 (0xb6eb1000)
libfuse.so.2 => /usr/lib/libfuse.so.2 (0xb6e8c000)
libdl.so.2 => /lib/libdl.so.2 (0xb6e81000)
libz.so.1 => /usr/lib/libz.so.1 (0xb6e6f000)
libaio.so.1 => not found
libcrypto.so.1.0.0 => /usr/lib/libcrypto.so.1.0.0 (0xb6d1e000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0xb6cf8000)
libc.so.6 => /lib/libc.so.6 (0xb6bd3000)
/lib/ld-linux.so.3 (0xb6ee9000)

Oh my... shared library hell... :-/ and to think that the initial rationale for using shared libraries was to 1. simplify things (ha!) and 2. make executables smaller (double ha!).

Any chance of getting a statically-linked version without having to do it myself? :-) 

Cheers, 
-- 
   Durval.



 
Cheers, 
-- 
   Durval.


 


Gordan

To unsubscribe from this group and stop receiving emails from it, send an email to zfs-discuss...@zfsonlinux.org.


Gordan Bobic

unread,
Apr 1, 2014, 1:33:17 PM4/1/14
to Durval Menezes, zfs-...@googlegroups.com, ray vantassle
I don't understand why exactly you can't build the binaries yourself or add the missing DLLs to your distro. On RH/Fedora you would say something like: 'yum provides "*/libaio.so.1" and it would tell you what package provides it.

I'm pretty sure you will have to modify the build scripts to build static binaries.

Durval Menezes

unread,
Apr 1, 2014, 3:40:10 PM4/1/14
to Gordan Bobic, zfs-...@googlegroups.com, ray vantassle
Hi Gordan, 

On Tue, Apr 1, 2014 at 2:33 PM, Gordan Bobic <gordan...@gmail.com> wrote:
I don't understand why exactly you can't build the binaries yourself or add the missing DLLs to your distro.

Oh my. I must be doing a very sorry job of explaining my situation... :-( please bear with me while I do it again (hopefully better this time):

1) I don't have an ARM machine capable of compiling anything: my only ARM machine is a Raspberry Pi where I only have OpenELEC installed, and OpenELEC runs partly from a R/O partition in an SD card, and part  from a RAMDISK (for temporary files, etc).  I can't install the toolchain in this environment, although I can install the zfs-fuse plus dynamic libraries to the RAMdisk and (setting LD_LIBRARY_PATH and PATH appropriately) run them from there, which should be more than appropriate for my testing purposes. I could install a full (or "fuller") Linux distro on it, perhaps RaspBIAN, so as to be able to compile ARM binaries there... but then I will get lynched by the family which wants to use it to watch movies and the like... :-)

2) My main machines  (where I have disk space etc to install larger things) are all x86_64 machines; I understand I would have to install an ARM cross-toolchain (arm-gcc plus ARM toolchain plus ARM libraries etc etc) in order to compile things. According to this, the process doesn't  seem too involved but is rather long and labor intensive, so it's something I would like to avoid doing for the time being (ie, just to test zfs-fuse). 

 
On RH/Fedora you would say something like: 'yum provides "*/libaio.so.1" and it would tell you what package provides it.

No question. But again I don't have any ARM machine where I can run this (OpenELEC doesn't have yum nor any other package manager available... )
 

I'm pretty sure you will have to modify the build scripts to build static binaries.

Ditto.

Cheers, 
-- 
   Durval.

Gordan Bobic

unread,
Apr 1, 2014, 6:13:34 PM4/1/14
to Durval Menezes, zfs-...@googlegroups.com, ray vantassle
On 04/01/2014 08:40 PM, Durval Menezes wrote:
> Hi Gordan,
>
> On Tue, Apr 1, 2014 at 2:33 PM, Gordan Bobic <gordan...@gmail.com
> <mailto:gordan...@gmail.com>> wrote:
>
> I don't understand why exactly you can't build the binaries yourself
> or add the missing DLLs to your distro.
>
>
> Oh my. I must be doing a very sorry job of explaining my situation...
> :-( please bear with me while I do it again (hopefully better this time):
>
> 1) I don't have an ARM machine capable of compiling anything: my only
> ARM machine is a Raspberry Pi where I only have OpenELEC installed, and
> OpenELEC runs partly from a R/O partition in an SD card, and part from
> a RAMDISK (for temporary files, etc). I can't install the toolchain in
> this environment, although I can install the zfs-fuse plus dynamic
> libraries to the RAMdisk and (setting LD_LIBRARY_PATH and PATH
> appropriately) run them from there, which should be more than
> appropriate for my testing purposes. I could install a full (or
> "fuller") Linux distro on it, perhaps RaspBIAN, so as to be able to
> compile ARM binaries there... but then I will get lynched by the family
> which wants to use it to watch movies and the like... :-)
>
> 2) My main machines (where I have disk space etc to install larger
> things) are all x86_64 machines; I understand I would have to install an
> ARM cross-toolchain (arm-gcc plus ARM toolchain plus ARM libraries etc
> etc) in order to compile things. According to this
> <http://www.kitware.com/blog/home/post/426>, the process doesn't seem
> too involved but is rather long and labor intensive, so it's something I
> would like to avoid doing for the time being (ie, just to test zfs-fuse).
>
> On RH/Fedora you would say something like: 'yum provides
> "*/libaio.so.1" and it would tell you what package provides it.
>
>
> No question. But again I don't have any ARM machine where I can run this
> (OpenELEC doesn't have yum nor any other package manager available... )

You can run a different distro in a chroot. Grab the RSEL6 tarball,
extract into a subdirectory called, e.g. /mnt/rsel, then do:

mount --bind /dev /mnt/rsel/dev
mount --bind /sys /mnt/rsel/sys
mount --bind /proc /mnt/rsel/proc
chroot /mnt/rsel
yum update redsleeve-release
yum groupinstall buildsys-build

and now you can buid whatever you need (possibly after installing other
dependencies). Or you an just use yum to find out what packages provide
the dlls you are missing, install them, then just grab the dlls
themselves from the chroot and put them into /usr/local/lib on your main
rootfs (make sure you add /usr/local/lib to your /etc/ld.so.conf or
ld.so.conf.d and run ldconfig.

Or you can try to build a more statically linked set of zfs-fuse binaries.

Gordan

Durval Menezes

unread,
Apr 1, 2014, 6:33:45 PM4/1/14
to Gordan Bobic, zfs-...@googlegroups.com, ray vantassle
Hi Gordan, 

Very interesting. Perhaps XBMC can even keep running and the family happily watching movies etc while I compile stuff inside the chroot ;-) The problem is, as I said in #1 above, that the writable area on OpenELEC is a RAMDisk which has only 128MB RAM... I checked and the RSEL6 tarball is 120MB+ in size, and that's with BZ2-compression... not a chance it will fit there. OTOH, perhaps I can create a separate NFS export in my main server, mount it on the RPi and do the above there... humrmrmrmr.... it's certainly going to be slow with the Pi accessing the NFS server over wifi, so perhaps I can connect a local USB harddrive on the Pi and... humrmrmr... 

Heck, seems I've just been dragged into building a compile environment on my RPi, which was exactly what I was trying to avoid :-)

Seriously now, thanks for pointing the different-distro-in-a-chroot possibility. I run chroot stuff day-to-day but had never thought about running an entire (and different) distro inside one... and now that you point it, can't see (as long as the kernel has all the necessary options & modules the distro needs) a reason for it not to work. 

Will try that and see what happens.

Cheers, 
-- 
   Durval.
 


Gordan

Gordan Bobic

unread,
Apr 1, 2014, 6:40:51 PM4/1/14
to Durval Menezes, zfs-...@googlegroups.com, ray vantassle
On 04/01/2014 11:33 PM, Durval Menezes wrote:
> Hi Gordan,
>
> On Tue, Apr 1, 2014 at 7:13 PM, Gordan Bobic <gordan...@gmail.com
> <mailto:gordan...@gmail.com>> wrote:
>
> On 04/01/2014 08:40 PM, Durval Menezes wrote:
>
> Hi Gordan,
>
> On Tue, Apr 1, 2014 at 2:33 PM, Gordan Bobic
> <gordan...@gmail.com <mailto:gordan...@gmail.com>
> <mailto:gordan...@gmail.com
> <http://www.kitware.com/blog/__home/post/426
USB stick is probably the easiest option if you have one handy. 2GB
might be enough. 4GB definitely will be.

> Heck, seems I've just been dragged into building a compile environment
> on my RPi, which was exactly what I was trying to avoid :-)

Sorry. :)

> Seriously now, thanks for pointing the different-distro-in-a-chroot
> possibility. I run chroot stuff day-to-day but had never thought about
> running an entire (and different) distro inside one... and now that you
> point it, can't see (as long as the kernel has all the necessary options
> & modules the distro needs) a reason for it not to work.

If the kernel is booting a distro already, it should be sufficient for
basic RSEL capable of compiling stuff.

Gordan

ray vantassle

unread,
Apr 2, 2014, 2:25:12 PM4/2/14
to Durval Menezes, zfs-...@googlegroups.com
I was able to rebuild the binaries with a static AIO library.  Same link: https://www.dropbox.com/s/7csfpd1ar1fi0ku/zfs-fuse-0.7.0-arm-bin.tar.gz

Man, is it ever a pain to build with just 1 library static and the rest shared!

I figured out how to build a deb package now, too.   I think I'll poke around on my systems and see if I can find the ashift option code and then incorporate that -- since I can now create a deb.

ray vantassle

unread,
Apr 8, 2014, 1:05:32 PM4/8/14
to Durval Menezes, zfs-...@googlegroups.com
Durval, did you ever try the binaries I uploaded, and did they work?

Anyway, I managed to incorporate the code to support the ashift option in zpool create.  I uploaded the new ARM binaries.  same link https://www.dropbox.com/s/7csfpd1ar1fi0ku/zfs-fuse-0.7.0-arm-bin.tar.gz

Ray

Durval Menezes

unread,
Apr 8, 2014, 1:34:08 PM4/8/14
to ray vantassle, zfs-...@googlegroups.com
Hi Ray, 

Thanks for the followup. I tried in my OpenELEC installation, extracting the only missing shared library (libaio.so.1) from a RedSleeve RPM into a directory and pointing LD_LIBRARY_PATH there, to no avail: I get the following error trying to run zfs-fuse: 

./zfs-fuse: /storage/ZFS_TESTING_20140402/usr/local/lib/libcrypto.so.1.0.0: no version information available (required by ./zfs-fuse)

ldd zfs-fuse, of course, shows the same thing: it finds the library but complains about the missing version information. I tried both with OpenELEC's built-in libcrypto.so.1.0.0 and also with the same file extracted from the RedSleeve OpenSSL RPM. My next step would be to bring up a full development environment in my RPi (according to Gordan's excellent hint) so as to be able to compile zfs-fuse from source, but I really don't have the leisure for it right now. 

If you could  send me a pointer to your libcrypto.so.1.0.0, or put it somewhere I can download it, it would rather expedite matters... 

Thanks, 
-- 
   Durval.



ray vantassle

unread,
Apr 8, 2014, 2:35:26 PM4/8/14
to Durval Menezes, zfs-...@googlegroups.com
Here it is. https://www.dropbox.com/s/h1uj8hkvq9upkps/arm-cryptolib.tar.gz

FWIW, some ldd output:
PogoPlug>ldd /sbin/zfs-fuse  /sbin/zpool                                                                                     
/sbin/zfs-fuse:
        librt.so.1 => /lib/arm-linux-gnueabi/librt.so.1 (0xb6f4b000)
        libpthread.so.0 => /lib/arm-linux-gnueabi/libpthread.so.0 (0xb6f2a000)
        libfuse.so.2 => /lib/arm-linux-gnueabi/libfuse.so.2 (0xb6ef7000)
        libdl.so.2 => /lib/arm-linux-gnueabi/libdl.so.2 (0xb6eec000)
        libz.so.1 => /lib/arm-linux-gnueabi/libz.so.1 (0xb6ece000)
        libcrypto.so.1.0.0 => /usr/lib/arm-linux-gnueabi/libcrypto.so.1.0.0 (0xb6d63000)
        libgcc_s.so.1 => /lib/arm-linux-gnueabi/libgcc_s.so.1 (0xb6d39000)
        libc.so.6 => /lib/arm-linux-gnueabi/libc.so.6 (0xb6c00000)
        /lib/ld-linux.so.3 (0xb6f68000)
/sbin/zpool:
        libpthread.so.0 => /lib/arm-linux-gnueabi/libpthread.so.0 (0xb6f3a000)
        libm.so.6 => /lib/arm-linux-gnueabi/libm.so.6 (0xb6e90000)
        libdl.so.2 => /lib/arm-linux-gnueabi/libdl.so.2 (0xb6e85000)
        libcrypto.so.1.0.0 => /usr/lib/arm-linux-gnueabi/libcrypto.so.1.0.0 (0xb6d1a000)
        libgcc_s.so.1 => /lib/arm-linux-gnueabi/libgcc_s.so.1 (0xb6cf0000)
        libc.so.6 => /lib/arm-linux-gnueabi/libc.so.6 (0xb6bb8000)
        /lib/ld-linux.so.3 (0xb6f68000)
        libz.so.1 => /lib/arm-linux-gnueabi/libz.so.1 (0xb6b9a000)

I'd hate to see you have to install a full development system just to build this thing.  Hopefully this is your last library problem. If you need, I should be able to easily make a build with static libcrypt, now that I know how to do it.
 I can do a complete build from scratch in 2 minutes on my desktop PC, but my pogoplug V2 takes 22 minutes.  I think the PI has an ever slower CPU.  

Emmanuel Anne

unread,
Apr 8, 2014, 3:12:17 PM4/8/14
to zfs-...@googlegroups.com
For info I merged your 2 patches, which seemed to be for another version of zfs-fuse apparently. Thanks for the work, with even the docs updated, nice thing, I guess this should have been merged a long time ago, but it would be nice that someone else does it.
The 2nd part of the arm support patch uses libaio.a instead of the shared libaio for zfs-fuse for everyone, I guess it's not really an issue, I don't know any other program using libaio so the shared library is not really useful here, but maybe it should not be in the patch for everyone.

Anyway I just compiled it to check that it compiles, but I didn't even run it... !
But the patch should be ok, for those interested.


--
--
To post to this group, send email to zfs-...@googlegroups.com
To visit our Web site, click on http://zfs-fuse.net/
---
You received this message because you are subscribed to the Google Groups "zfs-fuse" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zfs-fuse+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

ray vantassle

unread,
Apr 8, 2014, 5:14:13 PM4/8/14
to zfs-...@googlegroups.com
Yes, my patches were for the current Debian release, which is based on zfs_fuse-0.7.0.  There's a lot of complex #if's in the atomic code, so I hope my patches there went into the right places.

The static libaio was purely for Durval.

I guess zfs-fuse has largely been abandoned in favor of ZOL, but ZOL has massive memory requirements and simply won't run on little machines like Pogoplug or PI.  I really really wanted zfs for my Pogoplug, so I had to fix the atomic stuff.  
Reply all
Reply to author
Forward
0 new messages