Error during iRODS install / setup - PostgreSQL connectivity

[Aleksandr Blekh]

Hello everyone,

Some time ago I have successfully installed iRODS 4.1.8 on a host. Today I decided to upgrade it to the latest version. I tried to upgrade via packages, but it wasn't very successful (perhaps, due to using 'apt-get install' and official APT repo, while the previous version has been installed via .deb packages; or I did something wrong). Long story short, I decided to remove the previous installation for good and install the latest version of iRODS from the scratch on the same host. Also, this time, I decided to go with the latest version of PostgreSQL as well. I updated APT repos info on the host and successfully installed the database server (even though slightly hesitated, as I thought that the DB plugin might not be compatible). I also did the firewall setup on the host (Ubuntu 14.04), since it's been activated previously. Then I proceeded with iRODS install per official documentation. However, during execution of the setup script ("python /var/lib/irods/scripts/setup_irods.py"), I experienced the following error message (after quite a significant delay after "Updating /etc/irods/server_config.json..."):

=====
[output omitted...]
-------------------------------------------
Database Type: postgres
ODBC Driver:   PostgreSQL ANSI
Database Host: localhost
Database Port: 5432
Database Name: ICAT
Database User: irods
-------------------------------------------

Please confirm [yes]:
Database password:
Updating /etc/irods/server_config.json...
Error encountered running setup_irods:
Traceback (most recent call last):
  File "/var/lib/irods/scripts/setup_irods.py", line 436, in main
    setup_server(irods_config, json_configuration_file=options.json_configuration_file)
  File "/var/lib/irods/scripts/setup_irods.py", line 115, in setup_server
    database_interface.setup_database_config(irods_config)
  File "/var/lib/irods/scripts/irods/database_interface.py", line 160, in setup_database_config
    if database_already_in_use_by_irods(irods_config):
  File "/var/lib/irods/scripts/irods/database_interface.py", line 70, in database_already_in_use_by_irods
    with contextlib.closing(database_connect.get_database_connection(irods_config)) as connection:
  File "/var/lib/irods/scripts/irods/database_connect.py", line 206, in get_database_connection
    sys.exc_info()[2])
  File "/var/lib/irods/scripts/irods/database_connect.py", line 189, in get_database_connection
    return pypyodbc.connect(connection_string.encode('ascii'), ansi=True)
  File "/var/lib/irods/scripts/irods/pypyodbc.py", line 2452, in __init__
    self.connect(connectString, autocommit, ansi, timeout, unicode_results, readonly)
  File "/var/lib/irods/scripts/irods/pypyodbc.py", line 2501, in connect
    check_success(self, ret)
  File "/var/lib/irods/scripts/irods/pypyodbc.py", line 988, in check_success
    ctrl_err(SQL_HANDLE_DBC, ODBC_obj.dbc_h, ret, ODBC_obj.ansi)
  File "/var/lib/irods/scripts/irods/pypyodbc.py", line 966, in ctrl_err
    raise DatabaseError(state,err_text)
IrodsError: pypyodbc encountered an error connecting to the database:
('08001', '[08001] [unixODBC]could not connect to server: Connection timed out\n\tIs the server running on host "localhost" (::1) and accepting\n\tTCP/IP connections on port 5432?\ncould not connect to server: Connection refused\n\tIs the server running on host "localhost" (127.0.0.1) and accepting\n\tTCP/IP connections on port 5432?\n')
Exiting...
=====

Just in case, my firewall setting are:

===== IPv4 =====
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [4538:480396]
-A INPUT -m state --state INVALID -j DROP
-A INPUT -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m state --state NEW -j DROP
-A INPUT -f -j DROP
-A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -j DROP
-A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,SYN,RST,PSH,ACK,URG -j DROP
-A INPUT -p icmp -m limit --limit 5/sec -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 1248 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 1247 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 20000:20199 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 4443 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 5432 -j ACCEPT
-A INPUT -j LOG
-A INPUT -j DROP
COMMIT
=====

I would appreciate any advice / help in fixing this issue. Thank you very much in advance.

Best regards,
Aleksandr

[Terrell Russell]

Can you confirm the local postgresql is running and listening on 5432?

On my local development machine, I see this:

$ netstat -ln | grep 5432
tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN    
unix  2      [ ACC ]     STREAM     LISTENING     12401    /var/run/postgresql/.s.PGSQL.5432

Terrell

--
--
"iRODS: the Integrated Rule-Oriented Data-management System; A community driven, open source, data grid software solution" https://www.irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat

---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Aleksandr Blekh]

Terrell,

The command you suggested produces no output on my host. Any advice on how to troubleshoot from here?

Thank you,
Aleksandr

Terrell

To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.

[Aleksandr Blekh]

I just tried to start/restart PostgreSQL server manually (just in case) and "service postgresql (re)start" results in the following messages:

 * Starting PostgreSQL 9.6 database server                                                                                     [ OK ]
 * Restarting PostgreSQL 9.6 database server                                                                                     [ OK ]

-Aleksandr

On Sunday, February 12, 2017 at 8:20:57 AM UTC-5, Terrell Russell wrote:

Terrell

To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.

[Jason Coposky]

please ensure that you have the database created with:

sudo su - postgres -c 'createdb ICAT'

as well creating the irods database user:

sudo su - postgres -c 'createuser -s irods'

before reinstalling irods.

Thanks,

—
Jason Coposky
Executive Director, iRODS Consortium
RENCI at the University of North Carolina at Chapel Hill
(919)445-9675
jas...@renci.org

irods.org — Take Control of Your Data

---

From: irod...@googlegroups.com <irod...@googlegroups.com> on behalf of Aleksandr Blekh <aleksan...@gmail.com>
Sent: Sunday, February 12, 2017 6:05 PM
To: iRODS-Chat
Subject: Re: [iROD-Chat:16086] Error during iRODS install / setup - PostgreSQL connectivity

 

[Aleksandr Blekh]

I, indeed, has created the DB and iRODS DB user prior to (re)installing iRODS. This is what I have now:

root@hostname:~$ su - postgres
postgres@hostname:~$ psql
psql (9.6.2)
Type "help" for help.

postgres=# \l
                                  List of databases
   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
 ICAT      | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =Tc/postgres         +
           |          |          |             |             | postgres=CTc/postgres+
           |          |          |             |             | irods=CTc/postgres
 postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres
 template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres
(4 rows)

postgres=# \du
                                   List of roles
 Role name |                         Attributes                         | Member of
-----------+------------------------------------------------------------+-----------
 irods     |                                                            | {}
 postgres  | Superuser, Create role, Create DB, Replication, Bypass RLS | {}

Thank you,
Aleksandr

[Aleksandr Blekh]

UPDATE
=======

After some more tinkering and Internet searching, I have finally figured out what was the problem, which is weird and IMHO the correct way (until the packaging issue is not fixed) - specifying the version of PostgreSQL about to be installed implicitly - should be mentioned in the iRODS installation instructions. Relevant info include this answer (http://askubuntu.com/a/334730) as well as mentioned in the comments there advice on complete removal of a PostgreSQL server installation (http://johnmee.com/how-to-reinstall-postgresql-on-ubuntu). I have followed both recommendations and, after installing PostgreSQL 9.6 was happy to see that the server is not only running, but also listening on the default post:

root@matin-irods:~# netstat -ln | grep 5432

tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN

tcp6       0      0 ::1:5432                :::*                    LISTEN
unix  2      [ ACC ]     STREAM     LISTENING     1505327  /var/run/postgresql/.s.PGSQL.5432

However, unfortunately, continuing following the prompts of the iRODS setup script, I have experienced the following error (sorry about the long output, but I want you to see most of the relevant context). Any help with this issue will be much appreciated.

=======
root@matin-irods:~# python /var/lib/irods/scripts/setup_irods.py
Updating /var/lib/irods/VERSION.json...
The iRODS service account name needs to be defined.
iRODS user [irods]:
iRODS group [irods]:

+--------------------------------+
| Setting up the service account |
+--------------------------------+

Existing Group Detected: irods
Existing Account Detected: irods
Setting owner of /var/lib/irods to irods:irods
Setting owner of /etc/irods to irods:irods
iRODS server's role:
1. provider
2. consumer
Please select a number or choose 0 to enter a new value [1]:
Updating /etc/irods/server_config.json...

+-----------------------------------------+
| Configuring the database communications |
+-----------------------------------------+

You are configuring an iRODS database plugin. The iRODS server cannot be started until its database has been properly configured.

ODBC driver for postgres:
1. PostgreSQL ANSI
2. PostgreSQL Unicode
Please select a number or choose 0 to enter a new value [1]:
Database server's hostname or IP address [localhost]:
Database server's port [5432]:
Database name [ICAT]:
Database username [irods]:

-------------------------------------------
Database Type: postgres
ODBC Driver:   PostgreSQL ANSI
Database Host: localhost
Database Port: 5432
Database Name: ICAT
Database User: irods
-------------------------------------------

Please confirm [yes]:
Database password:
Updating /etc/irods/server_config.json...

Listing database tables...
Salt for passwords stored in the database:
Updating /etc/irods/server_config.json...

+--------------------------------+
| Configuring the server options |
+--------------------------------+

iRODS server's zone name [matinZone]:
iRODS server's port [1247]:
iRODS port range (begin) [20000]:
iRODS port range (end) [20199]:
Control Plane port [1248]:
Schema Validation Base URI (or off) [https://schemas.irods.org/configuration]:
iRODS server's administrator username [rods]:

-------------------------------------------
Zone name:                  matinZone
iRODS server port:          1247
iRODS port range (begin):   20000
iRODS port range (end):     20199
Control plane port:         1248
Schema validation base URI: https://schemas.irods.org/configuration
iRODS server administrator: rods
-------------------------------------------

Please confirm [yes]:
iRODS server's zone key:
iRODS server's negotiation key (32 characters):
Control Plane key (32 characters):
Updating /etc/irods/server_config.json...

+-----------------------------------+
| Setting up the client environment |
+-----------------------------------+


iRODS server's administrator password:

Updating /var/lib/irods/.irods/irods_environment.json...

+--------------------------+
| Setting up default vault |
+--------------------------+

iRODS Vault directory [/var/lib/irods/Vault]:

+-------------------------+
| Setting up the database |
+-------------------------+

Listing database tables...
Creating database tables...

+-------------------+
| Starting iRODS... |
+-------------------+

Validating [/var/lib/irods/.irods/irods_environment.json]... Success
Validating [/var/lib/irods/VERSION.json]... Success

Error encountered running setup_irods:
Traceback (most recent call last):
  File "/var/lib/irods/scripts/setup_irods.py", line 436, in main
    setup_server(irods_config, json_configuration_file=options.json_configuration_file)

  File "/var/lib/irods/scripts/setup_irods.py", line 126, in setup_server
    IrodsController(irods_config).start()
  File "/var/lib/irods/scripts/irods/controller.py", line 43, in start
    self.config.validate_configuration()
  File "/var/lib/irods/scripts/irods/configuration.py", line 256, in validate_configuration
    name=config_file['path'])
  File "/var/lib/irods/scripts/irods/json_validation.py", line 77, in validate_dict
    sys.exc_info()[2])
  File "/var/lib/irods/scripts/irods/json_validation.py", line 58, in validate_dict
    jsonschema.validate(config_dict, schema)
  File "/usr/lib/python2.7/dist-packages/jsonschema/validators.py", line 432, in validate
    cls(schema, *args, **kwargs).validate(instance)
  File "/usr/lib/python2.7/dist-packages/jsonschema/validators.py", line 117, in validate
    raise error
IrodsError: ERROR: Validation Failed for [/etc/irods/server_config.json]:
        against [https://schemas.irods.org/configuration/v3/server_config.json]
        ValidationError: u'authentication' is a required property

Failed validating u'required' in schema[u'properties'][u'plugin_configuration']:
    {u'properties': {u'authentication': {u'properties': {},
                                         u'type': u'object'},
                     u'database': {u'additionalProperties': {u'$ref': u'database_config.json'},
                                   u'maxProperties': 1,
                                   u'minProperties': 1,
                                   u'type': u'object'},
                     u'network': {u'properties': {}, u'type': u'object'},
                     u'resource': {u'properties': {}, u'type': u'object'},
                     u'rule_engines': {u'items': {u'$ref': u'rule_engine.json'},
                                       u'minItems': 1,
                                       u'type': u'array'}},
     u'required': [u'authentication',
                   u'network',
                   u'resource',
                   u'rule_engines'],
     u'type': u'object'}

On instance[u'plugin_configuration']:
    {'database': {'postgres': {'db_host': 'localhost',
                               'db_name': 'ICAT',
                               'db_odbc_driver': 'PostgreSQL ANSI',
                               'db_password': '**********',
                               'db_port': 5432,
                               'db_username': 'irods'}}}
Exiting...
=======

Thank you,
Aleksandr

[Ben Keller]

I believe the root cause of this issue (the failing schema validation) is that the 4.1 install was removed, which unfortunately deletes the /var/lib/irods/VERSION.json file but does not delete the /etc/irods/server_config.json file. Then when 4.2 is installed, the upgrade path is not taken, but also a new copy of the 4.2 server_config.json file is not created because of the already existing copy.

The easiest way to fix this would be to:

1. uninstall irods packages
   
2. rm -rf /etc/irods /var/lib/irods
3. drop and recreate the ICAT database:
   
1. sudo su - postgres -c "psql --command='drop database \"ICAT\";'"
   
2. sudo su - postgres -c "psql --command='create database \"ICAT\";'"
   
4. reinstall irods packages
   
5. run setup_irods.py

--

--
"iRODS: the Integrated Rule-Oriented Data-management System; A community driven, open source, data grid software solution" https://www.irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat

---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.

To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+unsubscribe@googlegroups.com.

[Aleksandr Blekh]

Thank you very much, Ben. I followed your advice and surely enough it fixed the problem. However, there is the next issue. Here's the output that I'm getting now:

=======

+-------------------+
| Starting iRODS... |
+-------------------+

Validating [/var/lib/irods/.irods/irods_environment.json]... Success
Validating [/var/lib/irods/VERSION.json]... Success

Validating [/etc/irods/server_config.json]... Success
Validating [/etc/irods/host_access_control_config.json]... Success
Validating [/etc/irods/hosts_config.json]... Success
Ensuring catalog schema is up-to-date...
Updating to schema version 2...
Updating to schema version 3...
Updating to schema version 4...
Updating to schema version 5...
Catalog schema is up-to-date.
Starting iRODS server...
Failure

Error encountered running setup_irods:
Traceback (most recent call last):
  File "/var/lib/irods/scripts/setup_irods.py", line 436, in main
    setup_server(irods_config, json_configuration_file=options.json_configuration_file)
  File "/var/lib/irods/scripts/setup_irods.py", line 126, in setup_server
    IrodsController(irods_config).start()

  File "/var/lib/irods/scripts/irods/controller.py", line 121, in start
    six.reraise(IrodsError, e, sys.exc_info()[2])
  File "/var/lib/irods/scripts/irods/controller.py", line 115, in start
    raise IrodsError('iRODS server failed to start.')
IrodsError: iRODS server failed to start.
Exiting...
=======

Best regards,
Aleksandr

To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.

[Ben Keller]

Can you send me your server log file, found at /var/lib/irods/log/rodsLog*

You can zip it up and email it to me directly if you wish.

To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+unsubscribe@googlegroups.com.