Install S3QL on Amazon Linux AMI Current 2015 Edition

264 views
Skip to first unread message

Jamie Nelson

unread,
Sep 29, 2015, 11:33:22 PM9/29/15
to s3ql
Hi guys,
I have just put together this setup script to deploy s3ql on Amazon Linux March 2015 edition.
It is very likely to work on newer editions too.

Let me know if you have any improvements to this

#!/bin/bash
# Install s3ql
#install pre-reqs
sudo yum install -y gcc-c++ make psmisc libattr-devel fuse-libs fuse-devel fuse bzip2 unzip xz-libs xz-devel xz python34 python34-devel python34-pip python34-setuptools sqlite sqlite-devel
sudo alternatives --set python /usr/bin/python3.4
sudo pip install --upgrade pip
sudo hash -r

sudo pip install --upgrade pycrypto
sudo pip install --upgrade defusedxml
sudo pip install --upgrade dugong
sudo pip install --upgrade requests
sudo pip install --upgrade llfuse
sudo pip install --upgrade wheel
easy_install -H *.python.org distribute
easy_install -H *.python.org argparse
easy_install -H *.python.org pycryptopp
easy_install -H *.python.org pyliblzma
easy_install -H *.python.org Sphinx


tar zxfv sqlite-autoconf-3081101.tar.gz
cd sqlite-amalgamation-3081101
./configure
make
sudo make install
sudo ldconfig
cd ..

rm -rf apsw-master
unzip master.zip
rm -f master.zip
cd apsw-master
python setup.py install
cd ..

unzip master.zip
rm -f master.zip
cd main-master
cd ..

python3 setup.py build_ext --inplace
sudo python3 setup.py install
cd ..



Nikolaus Rath

unread,
Sep 30, 2015, 2:20:14 PM9/30/15
to s3...@googlegroups.com
On Sep 29 2015, Jamie Nelson <ja...@analogueltd.com> wrote:
> easy_install -H *.python.org argparse
> easy_install -H *.python.org pycryptopp
> easy_install -H *.python.org pyliblzma
> easy_install -H *.python.org Sphinx

You shouldln't need these four. 1+3 are included in Python 3, 2 is not
used by S3QL, 4 is only required when re-building the documentation.


> wget <http://sqlite.org/2015/sqlite-amalgamation-3081101.zip>
> <http://sqlite.org/2015/sqlite-amalgamation-3081101.zip>http:
> //sqlite.org/2015/sqlite-autoconf-3081101.tar.gz
> tar zxfv sqlite-autoconf-3081101.tar.gz
> cd sqlite-amalgamation-3081101
> ./configure
> make
> sudo make install
> sudo ldconfig
> cd ..

You already installed SQLite above (with yum install sqlite
sqlite-devel), either the Yum install or this one is not needed.

> wget <https://github.com/s3ql/main/archive/master.zip>https:
> //github.com/s3ql/main/archive/master.zip
> unzip master.zip
> rm -f master.zip
> cd main-master
> cd ..

That's not necessary (note that you're not doing anything here).


Best,
-Nikolaus

--
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

»Time flies like an arrow, fruit flies like a Banana.«

Jamie Nelson

unread,
Sep 30, 2015, 10:36:18 PM9/30/15
to s3ql
Thanks for that.
When i did this install, i didn't get an error until ran s3ql.


Traceback (most recent call last):
  File "/usr/local/bin/mount.s3ql", line 9, in <module>
    load_entry_point('s3ql==2.15', 'console_scripts', 'mount.s3ql')()
  File "/usr/lib/python3.4/dist-packages/pkg_resources/__init__.py", line 521, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3.4/dist-packages/pkg_resources/__init__.py", line 2632, in load_entry_point
    return ep.load()
  File "/usr/lib/python3.4/dist-packages/pkg_resources/__init__.py", line 2312, in load
    return self.resolve()
  File "/usr/lib/python3.4/dist-packages/pkg_resources/__init__.py", line 2318, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/local/lib64/python3.4/site-packages/s3ql-2.15-py3.4-linux-x86_64.egg/s3ql/mount.py", line 10, in <module>
  File "/usr/local/lib64/python3.4/site-packages/s3ql-2.15-py3.4-linux-x86_64.egg/s3ql/fs.py", line 10, in <module>
  File "/usr/local/lib64/python3.4/site-packages/s3ql-2.15-py3.4-linux-x86_64.egg/s3ql/deltadump.py", line 7, in <module>
  File "/usr/local/lib64/python3.4/site-packages/s3ql-2.15-py3.4-linux-x86_64.egg/s3ql/deltadump.py", line 6, in __bootstrap__
  File "src/s3ql/deltadump.pyx", line 80, in init s3ql.deltadump (src/s3ql/deltadump.c:8447)
    import apsw
ImportError: /usr/local/lib64/python3.4/site-packages/apsw.cpython-34m.so: undefined symbol: sqlite3_status64


I saw you respond to someone else saying that that can mean a mismatch in sqlite versions in the compiled components.

Are you able to point me in the right direction?

Nikolaus Rath

unread,
Sep 30, 2015, 11:09:19 PM9/30/15
to s3...@googlegroups.com
On Sep 30 2015, Jamie Nelson <ja...@analogueltd.com> wrote:
> Thanks for that.
> When i did this install, i didn't get an error until ran s3ql.
>
> File "src/s3ql/deltadump.pyx", line 80, in init s3ql.deltadump
> (src/s3ql/deltadump.c:8447)
> import apsw
> ImportError: /usr/local/lib64/python3.4/site-packages/apsw.cpython-34m.so:
> undefined symbol: sqlite3_status64
>
>
> I saw you respond to someone else saying that that can mean a mismatch in
> sqlite versions in the compiled components.
>
> Are you able to point me in the right direction?

Probably because you installed SQLite two times. See my previous email.

Jamie Nelson

unread,
Sep 30, 2015, 11:46:53 PM9/30/15
to s3ql


On Thursday, October 1, 2015 at 4:09:19 PM UTC+13, Nikolaus Rath wrote:
On Sep 30 2015, Jamie Nelson <ja...@analogueltd.com> wrote:
> Thanks for that.
> When i did this install, i didn't get an error until ran s3ql.
>
>   File "src/s3ql/deltadump.pyx", line 80, in init s3ql.deltadump
> (src/s3ql/deltadump.c:8447)
>     import apsw
> ImportError: /usr/local/lib64/python3.4/site-packages/apsw.cpython-34m.so:
> undefined symbol: sqlite3_status64
>
>
> I saw you respond to someone else saying that that can mean a mismatch in
> sqlite versions in the compiled components.
>
> Are you able to point me in the right direction?

Probably because you installed SQLite two times. See my previous email.


I have removed the sqlite libraries from the /usr/lib64 and replaced them with the libraries from /usr/local/lib
No errors now

Jamie Nelson

unread,
Sep 30, 2015, 11:48:48 PM9/30/15
to s3ql


On Wednesday, September 30, 2015 at 4:33:22 PM UTC+13, Jamie Nelson wrote:
Hi guys,
I have just put together this setup script to deploy s3ql on Amazon Linux March 2015 edition.
It is very likely to work on newer editions too.

Let me know if you have any improvements to this

Here is an improved version, based on suggestions from Nik, and fixing the sqlite library install path.

#!/bin/bash
# Install s3ql
#install pre-reqs
sudo yum install -y gcc-c++ make psmisc libattr-devel fuse-libs fuse-devel fuse bzip2 unzip xz-libs xz-devel xz python34 python34-devel python34-pip python34-setuptools
sudo alternatives --set python /usr/bin/python3.4
sudo pip install --upgrade pip
sudo hash -r

sudo pip install --upgrade pycrypto
sudo pip install --upgrade defusedxml
sudo pip install --upgrade dugong
sudo pip install --upgrade requests
sudo pip install --upgrade llfuse
sudo pip install --upgrade wheel
easy_install -H *.python.org distribute


tar zxfv sqlite-autoconf-3081101.tar.gz
cd sqlite-autoconf-3081101
./configure
make
sudo make install
sudo \cp -f /usr/local/lib/libsqlite* /usr/lib64/
sudo hash -r
sudo ldconfig
cd ..

rm -rf apsw-master
unzip master.zip
rm -f master.zip
cd apsw-master
python setup.py build install
cd ..

Nikolaus Rath

unread,
Oct 1, 2015, 4:29:59 PM10/1/15
to s3...@googlegroups.com
On Sep 30 2015, Jamie Nelson <ja...@analogueltd.com> wrote:
> sudo alternatives --set python /usr/bin/python3.4

This won't interfere with S3QL operation, but is very like to get you
into trouble. The "python" command should always refer to Python 2.x,
while "python3" should give you a Python 3.x version.

> easy_install -H *.python.org distribute

That should not be necessary.

> wget http://sqlite.org/2015/sqlite-autoconf-3081101.tar.gz
> tar zxfv sqlite-autoconf-3081101.tar.gz
> cd sqlite-autoconf-3081101
> ./configure
> make
> sudo make install
> sudo \cp -f /usr/local/lib/libsqlite* /usr/lib64/

That line is pretty terrible. You should never put stuff into /usr/. Did
you encounter any problems when you *not* install SQlite by hand, or if
you *not* install SQLite by yum and install to /usr/local by hand?

> sudo hash -r

This doesn't do anything.

Jamie Nelson

unread,
Oct 1, 2015, 6:57:10 PM10/1/15
to s3ql


On Friday, October 2, 2015 at 9:29:59 AM UTC+13, Nikolaus Rath wrote:
On Sep 30 2015, Jamie Nelson <ja...@analogueltd.com> wrote:
> sudo alternatives --set python /usr/bin/python3.4

This won't interfere with S3QL operation, but is very like to get you
into trouble. The "python" command should always refer to Python 2.x,
while "python3" should give you a Python 3.x version.

Thanks for that tip.
 

> easy_install -H *.python.org distribute

That should not be necessary.

Cool, so it can be removed from the script.
 

> wget http://sqlite.org/2015/sqlite-autoconf-3081101.tar.gz
> tar zxfv sqlite-autoconf-3081101.tar.gz
> cd sqlite-autoconf-3081101
> ./configure
> make
> sudo make install
> sudo \cp -f /usr/local/lib/libsqlite* /usr/lib64/

That line is pretty terrible. You should never put stuff into /usr/. Did
you encounter any problems when you *not* install SQlite by hand, or if
you *not* install SQLite by yum and install to /usr/local by hand?

It looked like the Amazon Linux distribution comes with sqlite pre installed, even if i hadn't installed it using yum it was already there.
And the applications all compiled against the version in /usr/lib64 instead of the new one installed.
And the version that comes with Amazon Linux is too old for apsw to run, i had a min version error.

 

> sudo hash -r
This was added after pip was upgraded because pip wasn't working after the update until this was run. I added it after replacing the sqlite libs just in case it caused the same error, but if you think that in this case it wouldn't do anything, it can be removed too.



This doesn't do anything.


Best,
-Nikolaus

Thanks for helping make this better. 

Spice Pearl

unread,
Nov 9, 2016, 9:53:31 AM11/9/16
to s3ql
Hi,

I tried to install S3QL and having issue with the below error.


2016-11-09 14:26:53.407 27952:MainThread root.excepthook: Uncaught top-level exception:
Traceback (most recent call last):
  File "/usr/local/bin/mount.s3ql", line 9, in <module>
    load_entry_point('s3ql==2.15', 'console_scripts', 'mount.s3ql')()
  File "/usr/local/lib64/python3.4/site-packages/s3ql-2.15-py3.4-linux-x86_64.egg/s3ql/mount.py", line 214, in main
    llfuse.main(options.single)
  File "src/fuse_api.pxi", line 304, in llfuse.main (src/llfuse.c:35376)
ValueError: No workers is not a good idea


any solution??

Thanks..

Nikolaus Rath

unread,
Nov 9, 2016, 12:45:36 PM11/9/16
to s3...@googlegroups.com
On Nov 09 2016, Spice Pearl <spice...@gmail.com> wrote:
> Hi,
>
> I tried to install S3QL and having issue with the below error.
>
>
> 2016-11-09 14:26:53.407 27952:MainThread root.excepthook: Uncaught
> top-level exception:
> Traceback (most recent call last):
> File "/usr/local/bin/mount.s3ql", line 9, in <module>
> load_entry_point('s3ql==2.15', 'console_scripts', 'mount.s3ql')()
> File
> "/usr/local/lib64/python3.4/site-packages/s3ql-2.15-py3.4-linux-x86_64.egg/s3ql/mount.py",
> line 214, in main
> llfuse.main(options.single)
> File "src/fuse_api.pxi", line 304, in llfuse.main (src/llfuse.c:35376)
> ValueError: No workers is not a good idea
>
> any solution??

Upgrade to the newest version. Your python-llfuse version is newer than
your S3Ql release, thus the error.

Best,
-Nikolaus
Reply all
Reply to author
Forward
0 new messages