LDAP connections issues after upgrading to Erlang 26.1.2

[michael sleeuwagen]

Hi,

We are running RabbitMQ 3.12.6 with Erlang 25.2.3 which works fine. After we upgrade Erlang to version 26.1.2, the ldap connections doesn't work anymore. We only can logon to the RabbitMQ site with a local account but not with the AD account anymore(ldap_connect_error). We did not change anything to our ldap server or to the RabbitMQ settings. When I uninstall the newest version of Erlang and install the old one, everything works fine again. We want to update to the latest version, so if anybody has any ideas, shoot!

Thanks!

[Luke Bakken]

Hi Michael,

Are you using LDAP over TLS?

[michael sleeuwagen]

Hi Luke,

We use ssl to connect to ldap.

Kind Regards,

Michael

Op donderdag 12 oktober 2023 om 18:51:56 UTC+2 schreef Luke Bakken:

[Gert van den Berg]

I have some strange issues with AMQPS and Erlang 26.something... TLS 1.2 mostly doesn't work if TLS 1.3 is enabled (it strangely sometimes work). (testssl doesnot find TLS 1.2, but openssl s_client (and testssl --ssl-native) manages to connect using it...) (I posted about it on the Slack, came here to start a thread)

You can try and disable TLS 1.3 and see if it works then... (or do the testssl and openssl checks to see if it might be the same issue) (The problem goes away on Erlang 25 for me)

Gert

[Luke Bakken]

Somehow my response was deleted...

We need your full RabbitMQ configuration. There are TLS settings that are more strictly enforced by Erlang 26.

If you can provide log files at the "network_unsafe" level that might help.

Which LDAP server are you using?

[michael sleeuwagen]

Erlang version 25 works for me also. It is the version 26 that gives me the troubles.

Op woensdag 18 oktober 2023 om 09:18:19 UTC+2 schreef Gert van den Berg:

[michael sleeuwagen]

Hi Luke,

We are running RabbitMQ on a Windows 2019 server and connect to Microsoft LDAP.

Extract from the log file with the network_unsafe log level on. I replaced my account with <<account>> and the ldap server with <<ldapserver>>

2023-10-20 09:10:30.351000+02:00 [info] <0.930.0>     LDAP connect error: {error,"connect failed"}
2023-10-20 09:10:30.351000+02:00 [info] <0.139717.0> LDAP DECISION: login for <<account>>: {error,
2023-10-20 09:10:30.351000+02:00 [info] <0.139717.0>                                                               ldap_connect_error}
2023-10-20 09:10:30.351000+02:00 [warning] <0.139717.0> HTTP access denied: user '<<account>>' - invalid credentials
2023-10-20 09:10:33.257000+02:00 [info] <0.140226.0> LDAP CHECK: login for <<account>>
2023-10-20 09:10:33.257000+02:00 [info] <0.140226.0>         LDAP filling template "${username}" with
2023-10-20 09:10:33.257000+02:00 [info] <0.140226.0>             [{username,<<"<<account>>">>}]
2023-10-20 09:10:33.257000+02:00 [info] <0.140226.0>         LDAP template result: "<<account>>"
2023-10-20 09:10:33.257000+02:00 [info] <0.140226.0>     LDAP connecting to servers: ["<<ldapserver>>"]
2023-10-20 09:10:33.257000+02:00 [info] <0.140227.0>     LDAP network traffic: Connect: "<<ldapserver>>" failed {error,
2023-10-20 09:10:33.257000+02:00 [info] <0.140227.0>                                                                     {options,
2023-10-20 09:10:33.257000+02:00 [info] <0.140227.0>                                                                      incompatible,
2023-10-20 09:10:33.257000+02:00 [info] <0.140227.0>                                                                      [{verify,
2023-10-20 09:10:33.257000+02:00 [info] <0.140227.0>                                                                        verify_peer},
2023-10-20 09:10:33.257000+02:00 [info] <0.140227.0>                                                                       {cacerts,
2023-10-20 09:10:33.257000+02:00 [info] <0.140227.0>                                                                        undefined}]}}.

The above logging is mentioning invalid credentials but my credentials are correct. I think RabbitMQ just can't connect to ldap to verify my credentials.

This is an extract from our config file with again some sensitive data replaced with placeholders:

# try LDAP first
auth_backends.1 = ldap
# fall back to the internal database
auth_backends.2 = internal
auth_ldap.servers.1 = <<ldapserver>>
auth_ldap.use_ssl = true
auth_ldap.port = 636
#listeners.ssl.default = 5671
auth_ldap.idle_timeout = 10000
auth_ldap.timeout = 15000
auth_ldap.log = network_unsafe
auth_ldap.dn_lookup_attribute = userPrincipalName
auth_ldap.dn_lookup_base = <<ldap path>>

Op woensdag 18 oktober 2023 om 16:53:12 UTC+2 schreef Luke Bakken:

[Luke Bakken]

Hello,

The error is right in the output - incompatible options:

failed {error,
2023-10-20 09:10:33.257000+02:00 [info] <0.140227.0>                                                                     {options,
2023-10-20 09:10:33.257000+02:00 [info] <0.140227.0>                                                                      incompatible,
2023-10-20 09:10:33.257000+02:00 [info] <0.140227.0>                                                                      [{verify,
2023-10-20 09:10:33.257000+02:00 [info] <0.140227.0>                                                                        verify_peer},
2023-10-20 09:10:33.257000+02:00 [info] <0.140227.0>                                                                       {cacerts,
2023-10-20 09:10:33.257000+02:00 [info] <0.140227.0>                                                                        undefined}]}}.

I asked for your full RabbitMQ configuration, but you didn't provide all of it, so I can't check the ssl options you're using.

Thanks,

Luke

[michael sleeuwagen]

Hi Luke,

This is the complete contents of my rabbitmq.conf file:

# try LDAP first
auth_backends.1 = ldap
# fall back to the internal database
auth_backends.2 = internal
auth_ldap.servers.1 = <<ldapserver>>
auth_ldap.use_ssl = true
auth_ldap.port = 636
#listeners.ssl.default = 5671

ssl_options.cacertfile = <<cacertfile>>
ssl_options.certfile   = <<cerfile>>
ssl_options.keyfile    = <<keyfile>>
ssl_options.verify     = verify_peer
ssl_options.fail_if_no_peer_cert = true

auth_ldap.idle_timeout = 10000
auth_ldap.timeout = 15000
auth_ldap.log = network_unsafe
auth_ldap.dn_lookup_attribute = userPrincipalName

auth_ldap.dn_lookup_base = <<lookup_base>>
# logsettings
log.file.rotation.date = $D0
log.file.rotation.count = 365

This is the contents of the advanced.config

[{rabbitmq_auth_backend_ldap,[
     {tag_queries,           [{management,    {in_group, "<<ldap path>>"}},
              {monitoring,    {in_group, "<<ldap path>>"}},
      {administrator, {in_group, "<<ldap path>>"}}]},

      {resource_access_query,      
     {'or', [
                {for, [{permission, configure, {in_group, "<<ldap path>>"}},
                       {permission, write, {in_group, "<<ldap path>>"}},                    
                       {permission, read,{in_group, "<<ldap path>>"}}]},
      {for, [{permission, configure, {in_group, "<<ldap path>>"}},
                       {permission, write, {in_group, "<<ldap path>>"}},                    
                       {permission, read,{in_group, "<<ldap path>>"}}]}]}}

                   
]}
].

Do you need something else?

Kind Regards,

Michael

Op vrijdag 20 oktober 2023 om 15:49:05 UTC+2 schreef Luke Bakken:

[Luke Bakken]

Hi Michael,

The simplest thing for you to try is to add this setting to rabbitmq.conf

auth_ldap.ssl_options.verify = verify_none

What that does is turn off TLS server verification. If that fixes your issue, and you have access to the CA certificate (and intermediates) that signed your LDAP server cert, you can add the following settings:

auth_ldap.ssl_options.verify = verify_peer

auth_ldap.ssl_options.cacertfile = /path/to/ca_certificates.pem

ca_certificates.pem should be the concatenation of the full CA and intermediate cert chain.

Let me know how it goes.

Thanks,

Luke