Cannot get server to work after upgrading to v5

[Marc Hoeppner]

Hi,

so I did a dumb thing - which is running a distro update on my iRODS Server (AlmaLinux, 9.4->9.7). And that included the irods update from 4.3.3 to 5.0.2. So now I have v5 on my system - and cannot get it to work. 

What is the issue:

After the upgrade, my clients could no longer connect - claiming the server was not setup for TLS (which was running before). 

```

Level 0: Error occurred while authenticating user [mhoeppner] [SYS_NOT_ALLOWED: Client communications with this server are not secure and this authentication plugin is configured to require TLS communication. Authentication is not allowed unless this server is configured to require TLS in order to prevent leaking sensitive user information.

] [ec=-169000]
```

So, how do I require the server to use TLS? In /etc/irods/server_config.json, I have:

```

"tls_server": {
        "certificate_chain_file": "/etc/irods/ssl/chain.pem",
        "certificate_key_file": "/etc/irods/ssl/irods.key",
        "dh_params_file": "/etc/irods/ssl/dhparams.pem"
    },
    "zone_auth_scheme": "native",

```

Which, as far as I can tell, should be ok?

Any help appreciated, really. 

Cheers,

Marc

[Marc Hoeppner]

Checking the SSL status of the server gives:

```

imiscsvrinfo
RCAT_ENABLED
relVersion=rods5.0.2
apiVersion=d
rodsZone=lsh
SSL/TLS Info:
    enabled: false
```

So it would seem the server is not enabling SSL. Since I have not changed a thing about the certs...no idea why not. The cert is self-signed, for 10 years (the infrastructure runs in a private network). 

I am assuming that some helpful information should be in the logs... but I cannot find the logs ;) (AlmaLinux, RHEL). 

There is no:

/var/log/irods

/var/log/syslog

/var/lib/irods/IRODS

[Antoine Migeon]

Hello,

To get my irods v5 works, I have valid SSL certificates and this on server side : 

```

     "client_server_policy": "CS_NEG_REQUIRE",
    "tls_server": {
        "certificate_chain_file": "/etc/irods/certs/fullchain.pem",
        "certificate_key_file": "/etc/irods/certs/privkey.pem",
        "dh_params_file": "/etc/irods/certs/dhparams.pem"
    },
    "tls_client": {
        "ca_certificate_file": "/etc/irods/certs/cert.pem",
        "ca_certificate_path": "/etc/ssl/certs",
        "verify_server": "hostname"
    },
```

And this on client :
```
    "irods_ssl_verify_server": "hostname",
    "irods_client_server_policy": "CS_NEG_REQUIRE",
```

If irods server is started, you can check SSL info with `imiscsvrinfo`

Regards,

Antoine

[Antoine Migeon]

Check logs in /var/log/messages or `journalctl -u irods`

[Kory Draughn]

Hi Marc,

iRODS 5 does not auto-reload changes to the configuration. That requires a SIGHUP. Another thing is that the server no longer relies on the service account's irods_environment.json file.

I recommend reviewing the release notes for 5.0.0 and later if you haven't already.

There are a few things that are important to be aware of for iRODS 5. Links follow.

• https://docs.irods.org/5.0.2/getting_started/upgrading/
• https://docs.irods.org/5.0.2/system_overview/process_model/
• https://docs.irods.org/5.0.2/system_overview/configuration/
• https://docs.irods.org/5.0.2/system_overview/server_log/
• https://docs.irods.org/5.0.2/getting_started/running/
• https://docs.irods.org/5.0.2/system_overview/tls/

The primary question that comes to mind is - did you run the upgarde_irods.py script after the packages were upgraded? If not, take a look at the page I linked to which talks about upgrading.

Let us know if you continue to run into issues.

Kory Draughn

Chief Technologist

iRODS Consortium

--
--
The Integrated Rule-Oriented Data System (iRODS) - https://irods.org
 
iROD-Chat: http://groups.google.com/group/iROD-Chat
---
You received this message because you are subscribed to the Google Groups "iRODS-Chat" group.
To unsubscribe from this group and stop receiving emails from it, send an email to irod-chat+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/irod-chat/9a5d97f1-7f82-4bd9-a9dd-ca5e430bb628n%40googlegroups.com.

[Marc Hoeppner]

Hi,

thanks for the link about upgrading and all. Can't see any immediate solutions to my problem though. I did run the upgrade script; I also generated a fresh self-signed cert - just in case. Not sure how the PEPs would play into this (I haven't touch those at all, just running iRODS as a kind of metadata-enabled backup space). 

What I still don't get is why the server reports as not using SSL/TSL. There is no error I can see in the logs. I started the server in test mode,  just in case - and all I get is this:

```

cat /var/lib/irods/log/test_mode_output.log
{"log_category":"server","log_level":"info","log_message":"main: Initializing shared memory for main server process.","server_host":"localhost","server_pid":2867,"server_timestamp":"2026-01-22T06:42:58.557Z","server_type":"server","server_zone":"lsh"}
{"log_category":"server","log_level":"info","log_message":"main: Initializing access time queue for main server process.","server_host":"localhost","server_pid":2867,"server_timestamp":"2026-01-22T06:42:58.566Z","server_type":"server","server_zone":"lsh"}
{"log_category":"server","log_level":"info","log_message":"main: Launching Agent Factory.","server_host":"localhost","server_pid":2867,"server_timestamp":"2026-01-22T06:42:58.570Z","server_type":"server","server_zone":"lsh"}
{"log_category":"server","log_level":"info","log_message":"launch_agent_factory: Agent Factory PID = [2871].","server_host":"localhost","server_pid":2867,"server_timestamp":"2026-01-22T06:42:58.570Z","server_type":"server","server_zone":"lsh"}
{"log_category":"agent_factory","log_level":"info","log_message":"main: Initializing loggers for agent factory.","server_host":"localhost","server_pid":2871,"server_timestamp":"2026-01-22T06:42:58.584Z","server_type":"agent_factory","server_zone":"lsh"}
{"log_category":"agent_factory","log_level":"info","log_message":"main: Initializing signal handlers for agent factory.","server_host":"localhost","server_pid":2871,"server_timestamp":"2026-01-22T06:42:58.584Z","server_type":"agent_factory","server_zone":"lsh"}
{"log_category":"agent_factory","log_level":"info","log_message":"main: Initializing client allowlist for agent factory.","server_host":"localhost","server_pid":2871,"server_timestamp":"2026-01-22T06:42:58.584Z","server_type":"agent_factory","server_zone":"lsh"}
{"log_category":"agent_factory","log_level":"info","log_message":"main: Initializing shared memory for agent factory.","server_host":"localhost","server_pid":2871,"server_timestamp":"2026-01-22T06:42:58.584Z","server_type":"agent_factory","server_zone":"lsh"}
{"log_category":"agent_factory","log_level":"info","log_message":"main: Initializing access time queue for agent factory.","server_host":"localhost","server_pid":2871,"server_timestamp":"2026-01-22T06:42:58.585Z","server_type":"agent_factory","server_zone":"lsh"}
{"log_category":"agent_factory","log_level":"info","log_message":"main: Initializing zone information for agent factory.","server_host":"localhost","server_pid":2871,"server_timestamp":"2026-01-22T06:42:58.585Z","server_type":"agent_factory","server_zone":"lsh"}
{"log_category":"agent_factory","log_level":"info","log_message":"initServerMain: Server Release version rods5.0.2 - API Version d is up","server_host":"localhost","server_pid":2871,"server_timestamp":"2026-01-22T06:42:58.601Z","server_type":"agent_factory","server_zone":"lsh"}

``` 

And then, when trying to query the server from the client (or the server itself with a user account):

```

imiscsvrinfo
RCAT_ENABLED
relVersion=rods5.0.2
apiVersion=d
rodsZone=lsh
SSL/TLS Info:
    enabled: false
```

No errors about invalid SSL certs or whatever, so .. not sure how to debug this. 

Just in case, the relevant server config section:

```

"client_server_policy": "CS_NEG_DONT_CARE",

"tls_client": {
        "ca_certificate_file": "/etc/irods/ssl/irods.cert",
        "verify_server": "cert"
    },

    "tls_server": {
        "certificate_chain_file": "/etc/irods/ssl/chain.pem",
        "certificate_key_file": "/etc/irods/ssl/irods.key",
        "dh_params_file": "/etc/irods/ssl/dhparams.pem"
    },

```

and the client config:

```

{
    "irods_authentication_scheme": "pam_password",
    "irods_client_server_negotiation": "request_server_negotiation",
    "irods_client_server_policy": "CS_NEG_DONT_CARE",
    "irods_encryption_algorithm": "AES-256-CBC",
    "irods_encryption_key_size": 32,
    "irods_encryption_num_hash_rounds": 16,
    "irods_encryption_salt_size": 8,
    "irods_host": "lab-2-ngs-007.local",
    "irods_port": 1247,
    "irods_ssl_ca_certificate_file": "/etc/irods/irods.crt",
    "irods_ssl_verify_server": "cert",
    "irods_user_name": "SOMEUSER",
    "irods_zone_name": "lsh"
}

``` 

Where the irods.crt file is the same on the host and the client (verified with md5sum). 

[Marc Hoeppner]

oh and chain.pem and irods.crt are also the same file; some legacy thing I kept from an earlier iRODS instruction on the use of self-signed certs. 

[John Constable]

Hi Marc,

Before digging into it, if you have not uploaded any files into your iRODS system since the upgrade it might be worth trying going back to the version you had prior to the unintentional upgrade, and scheduling the upgrade.  You could then try an upgrade on a test system with less pressure.

I also recommend pinning the package versions which you can do in most distributions, which helps with this sort of accidental upgrade - without that people are (rightly) nervous to run any package updates, which leads to systems falling behind on patches and becoming more vulnerable.

I have some time this afternoon (UK) between 2-5 if you wanted someone to talk it through in person / screen share? I find a lot of my issues are solved by just walking someone else through the problem!

A few things to check, building on what others have said;

1. when you run imiscsrvinfo can you do so as a normal user and the irods user on the server? Check that both users have the irods_environment.json properties set to require SSL https://docs.irods.org/5.0.2/system_overview/tls/#client-tls-setup also, the SSL setup notes don't refer to the config changes to need in the users irods_environment files which are at https://docs.irods.org/5.0.2/plugins/pluggable_network/
2. for the end user (but not the server user!) try running `iinit --with-ssl` (as long as you know the password) and compare with the values that are in the ~/.irods/irods_environment.json file. If you find the values are different, keep a copy of the old files and adjust them on the live ones.
3. Has your SSL cert expired? Unlikely timing, but you had some bad luck, perhaps more also happened? ;-)

If you had SSL working before the upgrade, then with any luck its just a matter of finding the right combination of configuration.

In my (limited) experience of configuring iRODS this way, most of the issues have been from missing sections out of the users or server config files. With 5.X now only using the local users environment file and the server using the server_config you have less places to check, which is something!

Hope that helps,

John

--

Want to stay abreast of developments in iRODS but can’t read every bug report?

Sign up to https://theresource.metadata.school/  for a monthly update on the iRODS community.

Limited consultancy opportunities available  - reply and let me know your interest or take a look at https://metadataschool.carrd.co/#home

To view this discussion visit https://groups.google.com/d/msgid/irod-chat/6a04b132-2b0c-4e78-bfc3-69a21ef6adefn%40googlegroups.com.

[Kory Draughn]

Marc,

Please post your server_config.json and core.re.

Don't forget to mask sensitive information before posting.

Thanks,

Kory Draughn

Chief Technologist

iRODS Consortium

To view this discussion visit https://groups.google.com/d/msgid/irod-chat/mhLYrKAGVIYpsKkoODrn0oQ7ee837kbbuO9uky1bMBZPi-ss7KV6NEDaJjjFhbO4Ysnt8BfFj9AAXSiTazUr80y1BEooQuvhAif1c9NcnKg%3D%40metadata.school.

[Marc Hoeppner]

Hi,

I am leaning towards rolling back to 4.3. - the icat database is dumped daily, so shouldn't be a big issue. But for the sake of getting to the bottom of this:

server_config.json: https://pastebin.com/RwY1TPsT

core.re: https://pastebin.com/VPVpZwJE

Kind rergards,

Marc

[Kory Draughn]

Your core.re file shows that your iRODS server is configured to refuse secure connections. See acPreConnect in your core.re.

See https://docs.irods.org/5.0.2/plugins/pluggable_network/.

Kory Draughn

Chief Technologist

iRODS Consortium

To view this discussion visit https://groups.google.com/d/msgid/irod-chat/e5bbf2c2-5a59-47f7-8576-bc8dc72eccdbn%40googlegroups.com.

[Marc Hoeppner]

You, Sir, are a superstar. Changing this to CS_NEG_DONT_CARE fixes my issue. 

(that's an oddly hidden setting tho). 

Thanks for the help!

To view this discussion visit https://groups.google.com/d/msgid/irod-chat/CAA-7h7%3DaWnEy%2BQRm_e%2BbFEpMK8-oPYLGRm2kOfPFCVZZO7P2pQ%40mail.gmail.com.

[Marc Hoeppner]

Maybe as a closing remark - the documentation is rather dense, What I would have found helpful is mentioning of the need to fundamentally enable/disable ssl in the core.re file here:

https://docs.irods.org/5.0.2/system_overview/tls/

Because that bits reads like it explains how to make SSL work and just skips (unless I really missed that...) over a very fundamental prerequisite. 

[Kory Draughn]

Happy to hear it's working now.

And yes on the documentation. We'll look into improving that moving forward.

Thanks,

Kory Draughn

Chief Technologist

iRODS Consortium

To view this discussion visit https://groups.google.com/d/msgid/irod-chat/CAJVSKcDABt_cpcSN0fF0WsQgTqTBjDvOeMQ07zftVHfN8T%3Dwfw%40mail.gmail.com.