iRODS 4.0.3 downloa from source

[Navya D.A.Y]

Hello,

 

If we would like to download a specific version of iRODs say 4.0.3 from source on github https://github.com/irods/irods. Can you advice on how to do this? As using the git resource we see a version 4.1.0 build during execution.

 

Regards,

Navya.

[Adil Hasan]

Hello Navya,
I think there may be 2 ways:

git clone https://github.com/irods/irods.git <some dir>

then:
cd <some dir>
git checkout tags/4.0.3

(the advantage of this is you can switch between different tags). Another
option (maybe what you want) is to click on the 'releases' on the menu
and there you can download just the 4.0.3 release:

https://github.com/irods/irods/archive/4.0.3.tar.gz

hth
adil

> --
> --
> "iRODS: the Integrated Rule-Oriented Data-management System; A community driven, open source, data grid software solution" https://www.irods.org
>
> iROD-Chat: http://groups.google.com/group/iROD-Chat
>
> ---
> You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Terrell Russell]

The build.sh script is currently designed to build the master branch - if you want to build an older version from source, you can do this:

$ git checkout 4.0.3

$ ./packaging/build.sh icat postgres    # this will fail, as the downloaded externals will be incorrect

$ cd external

$ make clean

$ make generate

$ cd ..

$ ./packaging/build.sh icat postgres

This should result in newly built packages in the /build directory.

Terrell

[Wayne Schroeder]

Hi Terrell,

I'm trying this too (for a project separate from my DICE job) on a Debian host, and I found that the 'make generate' in external is failing (after running quite a while) because mysql is missing (cant find mysql library).  Do I need to install MySQL to run 'make generate' like that?  Or is there a way to indicate we're using Postgres and not MySQL?  I'm using Adil's first method to get the source tree, which I assume is equivalent to the one you suggested, the 'git checkout 4.0.3', but if I got the tar file like Adil mentioned as an alternative would that be set up to build more easily?  Since 4.0.3 is the current release, it seems like there should be an easy way to build it.

Thanks,

 - Wayne -
Integrated Data Management Solutions, LLC
i-data-mgmt.com

> To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
--
"iRODS: the Integrated Rule-Oriented Data-management System; A community driven, open source, data grid software solution"   https://www.irods.org

 iROD-Chat:  http://groups.google.com/group/iROD-Chat

---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.

To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+unsubscribe@googlegroups.com.

[Terrell Russell]

Hi Wayne,

The downloadable 4.0.3 tar.gz file from GitHub is simply a snapshot of the repository at the 4.0.3 tag.  The code is identical regardless of how you retrieve it.

The dependency you're hitting (mysql.h) is coming from lib_mysqludf_preg.

That library is included in external because the mysql database plugin requires its use.

You have two options:

1) satisfy the header file requirement with a package (sudo apt-get install libmysqlclient-dev)

or

2) remove lib_mysqludf_preg from the external/Makefile, line 82:

  build : cmake libarchive jansson boost epm hdf5 netcdf lib_mysqludf_preg libs3 check

You definitely do not need to install MySQL, just its development libraries.

Additionally, please note you can use the "-j" option for "make generate" to take advantage of multiple cores.

The build system is designed for use in continuous integration.  All supported releases are available as packages on the website.  Building the code is easy with the proper prerequisites in place.

Terrell

To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.

[Terrell Russell]

We've just reconfigured the RENCI ftp site for pre-built external packages.

You should now be able to build from a source checkout of 4.0.3 without any extra work (make generate).

The version string is now a part of the pre-built tar.gz files and will allow us to support building older snapshots directly.

Terrell

[Wayne Schroeder]

Thanks guys, that sounds good.  I'll give it a try soon, maybe tomorrow.

The 'make generate' did seem to work for me (with your help), but now the setup_irods_database.sh fails when connecting to the DB (Postgres) server (this is a familiar conversation for me, but I use to be on the other side of it... :-) ).  I'm using localhost as the db host name.  I believe Adil said that we need to check a setting in  /etc/postgresql/9.1/main/pg_hba.conf to make sure an authentication method uses md5.  There are a few that use md5, so it might be fine as it is (i.e. the default from Postgre), but there are a couple lines that might be it, do you know which?  And do I need to restart the postgres server for those to take effect?

Thanks,
 - Wayne -

[Terrell Russell]

I think by default, 'localhost' is a special string for postgres and it will connect over a local socket, rather than through TCP.   This may be the distinction that makes all the difference when setting up the postgres authentication.

To test the configuration, generally:

- First make sure 'psql' can connect as expected.

- Then connect via 'isql', as it will go through ODBC.

- Then go through iRODS (through ODBC).

Terrell

To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.

[Wayne Schroeder]

Yes, good point, 'localhost' may be special for postgres.  I updated the the postgres conf file use 'md5' instead of 'peer' authentication for the two cases where it was using 'peer' and restarted the postgres servers to make sure it re-read the conf file.  Now I can connect via 'psql' without the -h localhost, so apparently it was doing different connects with -h localhost and with no host.  When I use the -h with the IP address or -h with the hostname, it still fails but that may be something with the networking setup on this VM (altho ping to either seems fine).

Thanks for the 'isql' suggestion, that might be good to try sometime.  In this case tho,
./plugins/database/packaging/setup_irods_database.sh  does not set up my .odbc.ini file for me (since it fails), so I'd have to set up .odbc.ini manually and I'm not sure what all should be in there in this situation.

But with that 'peer'/'md5' change, and your changes for the externals, I thought it would be worth trying again.  So I got the 4.0.3 source tree again,  updated packaging/server_config.py with Adil's fix again, and tried './packaging/build.sh --run-in-place icat postgres' again, but it failed again with:
*** No rule to make target `/home/wayne/irods3/external/boost_1_55_0z/stage/lib/libboost_system.a', needed by `/home/wayne/irods3/iRODS/lib/core/obj/base64.o'
again (still).  Is there something else I need to do to access your revised version?  I guess I can try the 'make clean' and 'make generate' in the external directory again (perhaps tomorrow).

Thanks,
 - Wayne -
( with my i-data-mgmt.com hat on )

[Terrell Russell]

Try 'make clean' in the external/ directory - you should have nothing in there but a Makefile to start.

Then a regular ./packaging/build.sh again and it should get the correct precompiled external tar.gz file from RENCI.  No more need for 'make generate' unless you really want to - we fixed that for you.

Terrell

To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.

[Wayne Schroeder]

It's still not working for me.  I using new source directory, and there's only Makefile in external.  Are there other setup files, perhaps in /tmp/user, that I need to remove too?  If not, I may be doing something wrong, but somehow your fix doesn't seem to be working for me.

This morning, I did a:
git clone https://github.com/irods/irods.git irods4
cd irods4
git checkout tags/4.0.3
cd external
and then:
wayne@li889-20:~/irods4/external$ ls
Makefile
wayne@li889-20:~/irods4/external$ cd ..

Then:
./packaging/build.sh --run-in-place icat postgres

And it displayed a few lines about building and then the display paused while downloading the large file:

Getting irods_external_Debian.tar.gz -> irods_external.tar.gz...

And then it failed with:

Unpacking irods_external.tar.gz...
Libraries
------------------------------------------------------------------------
Compile flags:
make[2]: *** No rule to make target `/home/wayne/irods4/external/boost_1_55_0z/stage/lib/libboost_system.a', needed by `/home/wayne/irods4/iRODS/lib/core/obj/base64.o'.  Stop.
make[2]: *** Waiting for unfinished jobs....

This is the same host I've been using to try to install 4.0.3 from source, so I have previously done these steps:

1) Install postgres
sudo apt-get install postgresql

2) Create the irods user in Postgres
sudo su - postgres
psql
create user irods with password '[redacted]';
create database ICAT;
grant all privileges on database ICAT to irods;
\q

3) Check the Postgres authentication settings:
  edit: /etc/postgresql/9.1/main/pg_hba.conf
  and make sure that the METHOD is md5 for authentication.
  I changed both 'peer' auth settings to 'md5' and restarted the postgres server.

4) Install the set of software reported as needed the first time I ran ./packaging/build.sh --run-in-place icat postgres:

sudo apt-get install g++ make python-dev help2man unixodbc libfuse-dev libcurl4-gnutls-dev libbz2-dev zlib1g-dev libpam0g-dev libssl-dev libxml2-dev libkrb5-dev

5) Install boost:
sudo apt-get install libboost-all-dev

 - Wayne -

[Terrell Russell]

Debian has not been an officially supported distribution.  We will have it in CI soon, but not yet.

I've manually rebuilt the .tar.gz for Debian and placed it on the ftp site.

I've just re-run with success:

  git clean -xffd

  git checkout 4.0.3

  ./packaging/build.sh --run-in-place icat postgres

  ./plugins/database/packaging/setup_irods_database.sh

Terrell

To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.

[Terrell Russell]

One additional note, if you are building 4.0.3, you should be targeting the new 1.4 database plugins.

  git checkout 4.0.3-with-v1.4-database-plugins

Terrell

[Wayne Schroeder]

Thanks Terrell.  With some additional help from Adil (thanks Adil), we now have this building and running OK and I thought I'd post the steps we used to do this.  Please let me know if I missed anything.

Thanks,

 - Wayne -

These are based on Adil's notes, irods4-install.txt, and an updated
one (2), a recent irod-chat post of Adil's (Feb 9) describing how to
get the 4.0.3 source code from git, the manual irods-manual-4.0.3.pdf
from RENCI's irods.org web site, and some additional advice from Adil
and Terrell.

Since I had to backup and retry a few times, I'm not sure I have
everything correct here, but I believe these are the steps that were
needed to install irods 4.0.3, from source, on the VM under my
user id.

Note that we are installing from source so that we can run more than
one iRODS instance on the host.  This also makes it easy to update the
source if needed and gives us assurance that this source matches the
binaries.  iRODS is middleware in this application and we are building
from source for a number of components.  But a binary install should
be somewhat easier.

There was in issue for which we were advised to try build the
components in the external tree, and then there were some workarounds
for dealing with issues in that, but it should no longer be necessary
to build this.  But those steps (done after 7 or so) would be:

$ cd external
$ make clean
$ make generate

1) Install postgresql

sudo apt-get install postgresql

2) Create the irods user in Postgres
sudo su - postgres
psql

create user irods with password '[password selected]';

create database ICAT;
grant all privileges on database ICAT to irods;
\q

3) Check and update (if needed) Postgres authentication settings.

edit: /etc/postgresql/9.1/main/pg_hba.conf
and change the METHOD on one type (one line) from peer to md5.
The revised file has this change:
wayne@li889-20:/etc/postgresql/9.1/main$ sudo diff pg_hba.conf pg_hba.conf.orig
90c90
< local   all             all                                     md5
---
> local   all             all                                     peer

Then restart the postgres server for these setting to take effect:
  sudo /etc/init.d/postgresql restart

Verify that you can login as user postgres without a password and as
user 'irods' with its password (set above):

$ sudo su - postgres
postgres@li889-20:~$ psql
psql (9.1.15)
Type "help" for help.

postgres=# \q
postgres@li889-20:~$ exit
logout
wayne@li889-20:~$ psql -U irods -d icat -W
Password for user irods:
(login/connect succeeded)

4) Get the iRODS 4.0.3 source tree.

First, get git:
  sudo apt-get install git

One of these (the first two are equivalent as far as what's created in
the tree, I believe). 

  git checkout 4.0.3-with-v1.4-database-plugins

This was recommended by Terrell so it will get the new 1.4 database
plugins so should be the one used.  In general though, one could
use one of these alternatives:

    git clone https://github.com/irods/irods.git irods6
    cd irods6
    git checkout tags/4.0.3
or

    git clean -xffd
    git checkout 4.0.3

5) Update the script packaging/server_config.py with a bug fix that Adil
made: the username was omitted from the psql command.  Once updated it
will look like this:
wayne@li889-20:~/irods6/packaging$ diff server_config.py server_config.py.orig
88d87
<         db_user = self.values['DBUsername']
92d90
<             " -U " + db_user + \
99d96
<             " -U " + db_user + \
wayne@li889-20:~/irods6/packaging$

Without this, a finishing script in step 11 will fail.  I believe this
fix is integrated into 4.1 now, but we need to back port it to 4.0.3.

6) Install Boost
  sudo apt-get install libboost-all-dev

7) Install unixodbc-dev and odbc-postgresql:

  sudo apt-get install unixodbc-dev
  sudo apt-get install odbc-postgresql

I'm not sure unixodbc-dev is needed but perhaps so.

8) Do a test build of iRODS 4.0.3 with the --run-in-place option, to
get a list of packages we need to install.  This does not find the
Boost and odbc requirement (steps 6 and 7 above) but does find others.

./packaging/build.sh --run-in-place icat postgres

This produced:
ERROR :: build.sh requires some software to be installed
      :: try: sudo apt-get install g++ make python-dev help2man unixodbc libfuse-dev libcurl4-gnutls-dev libbz2-dev zlib1g-dev libpam0g-dev libssl-dev libxml2-dev libkrb5-dev

9) Install the needed packages:

sudo apt-get install g++ make python-dev help2man unixodbc libfuse-dev libcurl4-gnutls-dev libbz2-dev zlib1g-dev libpam0g-dev libssl-dev libxml2-dev libkrb5-dev

10) Build iRODS 4.0.3 with the --run-in-place option.

./packaging/build.sh --run-in-place icat postgres

11) Run the setup script to complete the setup.

This is similar to the 3.x irodssetup.pl script in that it prompts for
values, suggesting some defaults, and then sets up what's needed. 

./plugins/database/packaging/setup_irods_database.sh

I used 'localhost' as the host name for the postgres server. 

Be sure to set a vault path in a directory that the current user has
write access to.