iRODS 4.3 on Debian11

[Alexander Haller]

Hello all,

I tried compiling iRODS 4.3 on Debian11 and have some troubles you might
be able to help me with.

I did the following steps (and installing additional packages when needed):
A)I build https://github.com/irods/externals
B)I build https://github.com/irods/irods (master)
C)I build https://github.com/irods/irods_client_icommands

I installed the following debs:

irods-externals-avro1.11.0-1_1.0~bullseye_amd64.deb
irods-externals-boost1.78.0-0_1.0~bullseye_amd64.deb
irods-externals-catch22.13.8-0_1.0~bullseye_amd64.deb
irods-externals-clang13.0.0-0_1.0~bullseye_amd64.deb
irods-externals-clang-runtime13.0.0-0_1.0~bullseye_amd64.deb
irods-externals-cmake3.21.4-0_1.0~bullseye_amd64.deb
irods-externals-cppzmq4.8.1-1_1.0~bullseye_amd64.deb
irods-externals-fmt8.1.1-0_1.0~bullseye_amd64.deb
irods-externals-json3.10.4-0_1.0~bullseye_amd64.deb
irods-externals-libarchive3.5.2-0_1.0~bullseye_amd64.deb
irods-externals-nanodbc2.13.0-1_1.0~bullseye_amd64.deb
irods-externals-qpid-proton0.36.0-0_1.0~bullseye_amd64.deb
irods-externals-spdlog1.9.2-1_1.0~bullseye_amd64.deb
irods-externals-zeromq4-14.1.8-0_1.0~bullseye_amd64.deb

irods-database-plugin-postgres_4.3.0-1~bullseye_amd64.deb
irods-dev_4.3.0-1~bullseye_amd64.deb
irods-runtime_4.3.0-1~bullseye_amd64.deb
irods-server_4.3.0-1~bullseye_amd64.deb

irods-icommands_4.3.0-1~bullseye_amd64.deb


Now there are a couple of issues:

1)python3 /var/lib/irods/scripts/setup_irods.py
iRODS user [irods]:
iRODS group [irods]:
iRODS server's role:
1. provider
2. consumer
Please select a number or choose 0 to enter a new value [1]:
Error encountered running setup_irods:
Traceback (most recent call last):
File "/var/lib/irods/scripts/setup_irods.py", line 445, in main
setup_server(irods_config,
File "/var/lib/irods/scripts/setup_irods.py", line 118, in setup_server
database_interface.setup_database_config(irods_config)
File "/var/lib/irods/scripts/irods/database_interface.py", line 97,
in setup_database_config
db_type = get_database_type()
File "/var/lib/irods/scripts/irods/database_interface.py", line 90,
in get_database_type
raise IrodsError('Database type must be one of postgres,
cockroachdb, mysql, or oracle.')
irods.exceptions.IrodsError: Database type must be one of postgres,
cockroachdb, mysql, or oracle.

I do not understand why. The file
"/usr/lib/irods/plugins/database/libpostgres.so" is avaiable.

Just to see if this is the only problem I then patched def
get_database_type(): to always "return 'postgres'"

After that the setup went trough.

2)After trying to start /usr/sbin/irodsServer it complains about 2 things:

2a)
WARNING: Validation Failed for [/etc/irods/server_config.json]:
against
[file:///var/lib/irods/configuration_schemas/v4/server_config.json]
RefResolutionError: unknown url type: 'host_access_control.json'
Warning encountered in json_validation for
/etc/irods/server_config.json, skipping validation...
WARNING: Validation Failed for
[/var/lib/irods/.irods/irods_environment.json]:
against
[file:///var/lib/irods/configuration_schemas/v4/service_account_environment.json]
RefResolutionError: unknown url type: 'client_environment.json'
Warning encountered in json_validation for
/var/lib/irods/.irods/irods_environment.json, skipping validation...
Warning encountered in validation:
Traceback (most recent call last):
File "/var/lib/irods/scripts/irods/controller.py", line 94, in start
self.config.validate_configuration()
File "/var/lib/irods/scripts/irods/configuration.py", line 280, in
validate_configuration
raise IrodsWarning('%s\n%s' % (
irods.exceptions.IrodsWarning: Skipped validation for the following files:
/etc/irods/server_config.json
/var/lib/irods/.irods/irods_environment.json


2b)
Error encountered running setup_irods:
Traceback (most recent call last):
File "/var/lib/irods/scripts/setup_irods.py", line 445, in main
setup_server(irods_config,
File "/var/lib/irods/scripts/setup_irods.py", line 129, in setup_server
IrodsController(irods_config).start(test_mode=test_mode)
File "/var/lib/irods/scripts/irods/controller.py", line 197, in start
six.reraise(IrodsError, e, sys.exc_info()[2])
File "/var/lib/irods/scripts/irods/six.py", line 672, in reraise
raise value
File "/var/lib/irods/scripts/irods/controller.py", line 168, in start
lib.execute_command(cmd,
File "/var/lib/irods/scripts/irods/execute.py", line 117, in
execute_command
check_command_return(args, out, err, returncode, input=input, **kwargs)
File "/var/lib/irods/scripts/irods/execute.py", line 110, in
check_command_return
raise IrodsError('\n'.join([
irods.exceptions.IrodsError: Call to open process with
['/usr/sbin/irodsServer'] returned an error:
Options passed to Popen:
cwd: /usr/sbin
env: HIDDEN
Return code: 127
Error output:
/usr/sbin/irodsServer: error while loading shared libraries:
libavrocpp.so.1.11.0: cannot open shared object file: No such file or
directory

I can workaround the 2b) error with
export PATH=/opt/irods-externals/avro1.11.0-1/lib:$PATH

However now irods still will not start. In the log I can see the
following error:


3)
in the logs I see now:
irodsServer[1064554]:
{"log_category":"legacy","log_facility":"local0","log_level":"error","log_message":"environment_properties::capture:
missing environment file. should be at [/root/.irods/irods_environment.json]

/var/lib/irods/.irods/irods_environment.json exists, I assume it has the
wrong infos.


At this point I want to ask:
I) any ideas what went wrong on my try?
II) has someone successfuly build irods 4.3 on Debian 11 an would be so
nice to share his steps
III) When 4.3 is officially released is it planned to provide packages
not only for Ubuntu/Centos but also for Debian?


Thank you for any pointers!
best,
Alexander

[Alan King]

Hi,

I have compiled and run packages for the main branch on Debian 11 using the iRODS Development Environment and iRODS Testing Environments:

 https://github.com/irods/irods_development_environment

 https://github.com/irods/irods_testing_environment

These are Docker-based environments which are based on the official debian:11 image tag (although, it may have updates which I haven't pulled down yet). The only issue that I encountered is that the rsyslogd service cannot be started inside the container, so that's still being worked out. Here's the output I get:

iRODS user [irods]: 
iRODS group [irods]: Traceback (most recent call last):
  File "/var/lib/irods/scripts/setup_irods.py", line 459, in <module>

    sys.exit(main())
  File "/var/lib/irods/scripts/setup_irods.py", line 445, in main
    setup_server(irods_config,
  File "/var/lib/irods/scripts/setup_irods.py", line 93, in setup_server
    setup_rsyslog_and_logrotate(register_tty=False)
  File "/var/lib/irods/scripts/irods/logging_infrastructure.py", line 39, in setup_rsyslog_and_logrotate
    subprocess.call(['systemctl', 'restart', 'rsyslog'])
  File "/usr/lib/python3.9/subprocess.py", line 349, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.9/subprocess.py", line 1823, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'systemctl'

After I installed systemd, setup completed and then I was able to start the server using irodsctl as per usual.

I haven't seen the other issues you've described here except for 2a, which is known and captured in an issue here: https://github.com/irods/irods/issues/6160

The third problem you mentioned about the environment file is likely due to the fact that the server was started as root and not the irods service account (Linux user), which should have been created as part of the setup script. Have you tried something like su - irods -c './irodsct start' ?

So let's take the questions now...

1) The most interesting piece of your story to me is the initial failure in the setup script to find the DB plugin .so file. Do you have any idea why that wouldn't be reachable by the setup script?

2) For my steps, the Dockerfiles in the development and testing environments outline the environment. Beyond that, the steps are pretty much just running "apt install -fy <paths-to-deb-files>" for the externals and then the iRODS packages. Were there any issues during the installation itself?

3) We are planning to officially support the following platforms for 4.3.0:

 - ubuntu:18.04

 - ubuntu:20.04

 - debian:11

 - centos:7

 - almalinux:8

In theory, the packages would also work on any RHEL 8-like platform, but these are the platforms for which package repositories will be maintained.

Hopefully that helps us get started on figuring out what's going on

--
--
The Integrated Rule-Oriented Data System (iRODS) - https://irods.org

 iROD-Chat:  http://groups.google.com/group/iROD-Chat
---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/1c8203b6-6d4f-33ed-a6a5-5a7c5d1bd648%40gmail.com.

--

Alan King

Senior Software Developer | iRODS Consortium

[Alan King]

Ack, typos. :)

su - irods -c './irodsctl start'

[Alexander Haller]

Thank you Alan,
with running as the irods user (completly forgot to do this...) I get further but I get this error:

irods@dev-storage-001:~/$ ./irodsctl start

WARNING: Validation Failed for [/etc/irods/server_config.json]:
against [file:///var/lib/irods/configuration_schemas/v4/server_config.json]
RefResolutionError: unknown url type: 'host_access_control.json'
Warning encountered in json_validation for /etc/irods/server_config.json, skipping validation...

WARNING: Validation Failed for [/var/lib/irods/VERSION.json]:
against [file:///var/lib/irods/configuration_schemas/v4/VERSION.json]
RefResolutionError: unknown url type: 'VERSION.json'
Warning encountered in json_validation for /var/lib/irods/VERSION.json, skipping validation...

WARNING: Validation Failed for [/var/lib/irods/.irods/irods_environment.json]:
against [file:///var/lib/irods/configuration_schemas/v4/service_account_environment.json]
RefResolutionError: unknown url type: 'client_environment.json'
Warning encountered in json_validation for /var/lib/irods/.irods/irods_environment.json, skipping validation...
Warning encountered in validation:
Traceback (most recent call last):
File "/var/lib/irods/scripts/irods/controller.py", line 94, in start
self.config.validate_configuration()
File "/var/lib/irods/scripts/irods/configuration.py", line 280, in validate_configuration
raise IrodsWarning('%s\n%s' % (
irods.exceptions.IrodsWarning: Skipped validation for the following files:
/etc/irods/server_config.json

/var/lib/irods/VERSION.json
/var/lib/irods/.irods/irods_environment.json
Error encountered running irods_control start:

Traceback (most recent call last):

File "/var/lib/irods/./scripts/irods_control.py", line 124, in main
operations_dict[operation]()
File "/var/lib/irods/./scripts/irods_control.py", line 70, in <lambda>
operations_dict['start'] = lambda: irods_controller.start(write_to_stdout=options.write_to_stdout, test_mode=options.test_mode)

File "/var/lib/irods/scripts/irods/controller.py", line 197, in start
six.reraise(IrodsError, e, sys.exc_info()[2])
File "/var/lib/irods/scripts/irods/six.py", line 672, in reraise
raise value

File "/var/lib/irods/scripts/irods/controller.py", line 189, in start
raise IrodsError('iRODS server failed to start.')
irods.exceptions.IrodsError: iRODS server failed to start.

> 1) The most interesting piece of your story to me is the initial failure in the setup script to find the DB plugin .so file. Do you have any idea why that wouldn't be reachable by the setup script?

I have maybe an idea now, could a restrictive default umask (I set it to 0077 on most of my servers) play a role there? Does /usr/lib/irods/plugins/database/libpostgres.so need to be accessible by someone other than root while installing? I ran the setup script as root.

ls -lah /usr/lib/irods/plugins/database/
total 2,5M
drwx------  2 root root 4,0K 2022-03-09 11:32 .
drwx------ 10 root root 4,0K 2022-03-09 11:32 ..
-rw-r--r--  1 root root 2,5M 2022-03-08 22:32 libpostgres.so

But packages should set the correct permissions? Will test this later.

> 2) For my steps, the Dockerfiles in the development and testing environments outline the environment. Beyond that, the steps are pretty much just running "apt install -fy <paths-to-deb-files>" for the externals and then the iRODS packages. Were there any issues during the installation itself?

I think the packages installed fine. I had to install some additional packages initially to get them build:

apt install libpam-dev libkrb5-dev super odbc-postgresql python3-psutil python3-pyodbc ninja-build

Great to hear that you plan to expand the offered packages!

Thank you again,

Alexander

[Alexander Haller]

On 09/03/2022 17.38, Alexander Haller wrote:
> I have maybe an idea now, could a restrictive default umask (I set it to
> 0077 on most of my servers) play a role there? Does
> /usr/lib/irods/plugins/database/libpostgres.so need to be accessible by
> someone other than root while installing? I ran the setup script as root.
> ls -lah/usr/lib/irods/plugins/database/
> total 2,5M
> drwx------ 2 root root 4,0K 2022-03-09 11:32 .
> drwx------ 10 root root 4,0K 2022-03-09 11:32 ..
> -rw-r--r-- 1 root root 2,5M 2022-03-08 22:32 libpostgres.so
>
> But packages should set the correct permissions? Will test this later.

Infos so far from the first install:
In the deb:
irods-database-plugin-postgres_4.3.0-1~bullseye_amd64.deb:
/usr/lib/irods/plugins/database/libpostgres.so
all folders in this tree are packaged 700 root:root
libpostgres.so is 644 root:root

There is a call in postinst to
python3 /var/lib/irods/scripts/chown_directories_for_postinstall.py
for dirpath, _, _ in os.walk(paths.irods_directory()):
os.chown(dirpath, service_account_uid, service_account_gid)

from paths.py:
def irods_directory():
return os.path.join(root_directory(), 'var', 'lib', 'irods')

So it seems that all files under /usr/lib/irods/ are ignored in the
permissions "fix".

Also it seems that there is only a chown never a chmod. (not sure if
this will ever be a problem as long as the irods user has access.
For example /var/lib/irods/packaging/sql/ is 700 irods:irods on my
system atm.

I checked on another system and /usr/lib/irods/plugins/database/
is 755 root:root.

Is this permissions chain correct from your perspective and if yes
should/could this be an improvement to set owner:group and perms more
explicitly in the packages?

best,
Alexander

[Alan King]

Sorry about the delayed response here...

Actually, everything you've shown so far looks fine/normal to me, so I'm a bit at a loss. Any errors which are occurring during setup may show up in /var/lib/irods/log/setup_log.txt, but otherwise this is very strange.

I am not sure whether the explicit user/group permission setting would be an improvement as I'm not a package expert, but someone else from my team might have more opinions on that.

All I could offer at this point is the suggestion to start over, making sure the steps are performed as expected by the setup script. That is, after installing, the setup script should be run as root (i.e. sudo python /var/lib/irods/scripts/setup_irods.py), and any invocations of /var/lib/irods/irodsctl - in other words, starting the server - should be done by the iRODS service account.

--
--
The Integrated Rule-Oriented Data System (iRODS) - https://irods.org

 iROD-Chat:  http://groups.google.com/group/iROD-Chat
---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/99b09516-ecac-8d7c-7aaf-6fddae974cb3%40gmail.com.

[Terrell Russell]

Hi Alexander,

I think you might be onto something - maybe we are missing an additional chown for /usr/lib/irods.

Would that make your setup/configuration work?

We don't want to make the packaging do the work, but rather the irods controller (on startup/upgrade).

Thanks,

Terrell

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/CADnp3x7KbEAAbqfq41wA2TZE%3Dnr%3DjYDV9njKNia1%3DESMbChWrg%40mail.gmail.com.