iRODS Testing Environment on M1 Mac

[David Reynolds]

Hello all,

Is it possible to run the testing environment in Docker on an M1 Mac?  I've set "platform: linux/amd64" in the compose file, and I'm trying to stand up a single zone with the following command:

$ python stand_it_up.py --project-directory ./projects/ubuntu-20.04/ubuntu-20.04-postgres-10.12

The postgres container and provider container are created, but I get the following error:

Creating ubuntu-2004-postgres-1012_catalog_1 ... done
Creating ubuntu-2004-postgres-1012_irods-catalog-provider_1 ... done
2023-03-12 20:27:00,826 - exception raised while setting up iRODS Zone [<irods_testing_environment.irods_setup.zone_info object at 0x10c904a90>]
2023-03-12 20:27:00,826 - failed to set up iRODS server [ubuntu-2004-postgres-1012_irods-catalog-provider_1]
Traceback (most recent call last):
  File "/Users/david/irods_testing_environment/stand_it_up.py", line 66, in <module>
    services.create_topology(ctx,
  File "/Users/david/irods_testing_environment/irods_testing_environment/services.py", line 77, in create_topology
    return create_topologies(ctx,
  File "/Users/david/irods_testing_environment/irods_testing_environment/services.py", line 53, in create_topologies
    irods_setup.setup_irods_zones(ctx, zone_info_list, odbc_driver=odbc_driver)
  File "/Users/david/irods_testing_environment/irods_testing_environment/irods_setup.py", line 637, in setup_irods_zones
    raise RuntimeError('failed to set up one or more iRODS Zones, ec=[{}]'.format(rc))
RuntimeError: failed to set up one or more iRODS Zones, ec=[1]

And the tail of the postgres container log:

2023-03-12 20:25:57 PostgreSQL init process complete; ready for start up.
2023-03-12 20:25:57
2023-03-12 20:25:57 2023-03-13 00:25:57.877 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2023-03-12 20:25:57 2023-03-13 00:25:57.877 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2023-03-12 20:25:57 2023-03-13 00:25:57.878 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2023-03-12 20:25:57 2023-03-13 00:25:57.907 UTC [56] LOG:  database system was shut down at 2023-03-13 00:25:57 UTC
2023-03-12 20:25:57 2023-03-13 00:25:57.914 UTC [1] LOG:  database system is ready to accept connections
2023-03-12 20:26:54 2023-03-13 00:26:54.030 UTC [63] LOG:  incomplete startup packet
2023-03-12 20:26:54 2023-03-13 00:26:54.189 UTC [78] FATAL:  database "ICAT" does not exist
2023-03-12 20:26:54 2023-03-13 00:26:54.629 UTC [107] FATAL:  role "irods" does not exist

When I log into the postgres container to check, both the "ICAT" database and "irods" role exist, so I'm not sure what's going on.  Does anyone have any ideas or suggestions?

Thank you,

David

[Alan King]

Hi,

Those error messages are likely coming from a check that occurs before attempting to create the database for the catalog, so those can likely be ignored.

Can you share what messages are in the setup log for iRODS? This file can be found in /var/lib/irods/log/setup_log.txt.

You can also try increasing the verbosity level and you may see some additional information in the script output that you can share and/or will help with investigation.

Alan

--
--
The Integrated Rule-Oriented Data System (iRODS) - https://irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat
---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/407f1114-a63a-4453-8ef9-c5db43f37055n%40googlegroups.com.

--

Alan King

Senior Software Developer | iRODS Consortium

[David Reynolds]

Thank you for the suggestions, Alan.  With more debugging here's what I've discovered.  It actually succeeds in setting up and running the iRODS server, but the failure is happening towards the end of setup_irods.py when it tries to stop the server process after the installation and setup is complete.  It looks like the issue is a side effect of things running through the emulator when a non-native platform is specified. 

2023-03-16 12:57:24,833 - Traceback (most recent call last):
  File "/var/lib/irods/scripts/setup_irods.py", line 529, in <module>
    sys.exit(main())
  File "/var/lib/irods/scripts/setup_irods.py", line 515, in main
    setup_server(irods_config,
  File "/var/lib/irods/scripts/setup_irods.py", line 162, in setup_server
    IrodsController(irods_config).stop()
  File "/var/lib/irods/scripts/irods/controller.py", line 221, in stop
    server_proc = self.get_server_proc()
  File "/var/lib/irods/scripts/irods/controller.py", line 77, in get_server_proc
    if server_proc.exe() and os.path.samefile(self.config.server_executable, server_proc.exe()):
  File "/usr/lib/python3.8/genericpath.py", line 101, in samefile
    s2 = os.stat(f2)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/qemu-x86_64'

In controller.py, it gets the server_pid and then attempts to check that the server process executable associated with that pid matches the executable path stored in the config.  That can be seen at "controller.py, line 77, in get_server_proc" in the traceback where it is checking server_proc.exe().  At that point, server_proc has been set to psutil.Process(server_pid).  The problem is that when the emulator is involved, psutil.Process().exe() doesn't return "/usr/sbin/irodsServer" as would be expected.  We can see in the ps output that when run on Linux natively or in Docker when the native architecture is amd64, things are run as would be expected:

david@irods-ubuntu:~$ ps ax | grep irods
    805 ?        Ssl    0:01 /usr/sbin/irodsServer
    806 ?        S      0:01 /usr/sbin/irodsServer
    827 ?        Sl     0:00 irodsDelayServer

And therefore, psutil.Process().exe() returns the path to the iRODS server as expected:

>>> psutil.Process(805).exe()
'/usr/sbin/irodsServer'

However, when run in Docker with a non-native platform (in this case - specified platform: amd64, native architecture: arm64), the emulator gets involved:

irods@0fe1b03f67b4:/$ ps ax | grep irods
 4036 ?        Ssl    0:01 /usr/bin/qemu-x86_64 /usr/sbin/irodsServer /usr/sbin/irodsServer
 4038 ?        Sl     0:00 /usr/bin/qemu-x86_64 /usr/sbin/irodsServer /usr/sbin/irodsServer
 4058 ?        Sl     0:00 /usr/bin/qemu-x86_64 irodsDelayServer irodsDelayServer

And psutil.Process().exe() returns the emulator instead of the iRODS server path:

>>> psutil.Process(4036).exe()
'/usr/bin/qemu-x86_64'

That gets passed to os.path.samefile() and eventually when it tries to stat that file we get the FileNotFoundError, causing setup_irods.py to fail which leads to the "failed to set up iRODS server" and "failed to set up one or more iRODS Zones" error messages.

I suppose a potential workaround could be to recognize when the path returned is actually the emulator, and instead of exe() use psutil.Process().cmdline() to get to the other arguments.  I'm going to try that for now, but I'm all ears if anyone has any other thoughts.

Thanks again,

David

[Alan King]

Thanks for the detailed findings! Yes, it looks like this may be an issue with the controller script and how it identifies the iRODS server processes.

This issue seems similar to if not related to what you're seeing: https://github.com/irods/irods/issues/6144

Please let us know how it goes with implementing your suggested changes.

To view this discussion on the web visit https://groups.google.com/d/msgid/irod-chat/d3881746-9651-4a1f-8774-e8afc1e31aadn%40googlegroups.com.