LDAP Failure after upgrading RabbitMQ from 3.8.7 to 3.8.9.

[ipock]

This was previously brought up on the Google Groups email group by me. Reposting here it per request.

I have had an odd LDAP error that occurred right after a RabbitMQ upgrade. I've worked around it, but thought it was worthy for other to know about, in the off chance that it was related to the upgrade.

Approximately a week and a half ago, I performed an in-place upgrade of RabbitMQ from version 3.8.7 to RabbitMQ 3.8.9, each running Erlang 23.0. The upgrade was performed on four environments: our development, QA, staging, and production environments, all running Windows Server 2016. The first two environments are single-server environments and the last two are both clusters of three servers. Each server has TLS 1.2 configured using certificates that do not expire for approximately 18 months. AuthN and AuthZ with the exchanges and queues by RabbitMQ clients use a username / password combination over a TLS-enabled connection (Moving them to certificate authentication is on my to-do list). Signing into the web management portal uses a secured connection to our LDAP servers (Active Directory).

All environments were working fine prior to the upgrade. There have been no reports of bad connections, missed messages, or any issues at all. The web management portal could be signed into using the Active Directory credentials and the proper rights were given based upon group membership. Again, no problems at all. After the upgrade, I'm reasonably certain I signed into the web portal of all four environments to double-check the upgrade / node status. So I had no indication anything was wrong.

After about a week after the upgrade (which is to say, earlier this past week), I attempted to login to our QA environment. It would consistently give a failed login message when trying to go into the web management portal. After turning on network-level logging, it was clear that the error was a failed handshake with the LDAP server. All client communication with the exchanges and nodes remained just fine. After experimenting with some configurations, I found that by turning off the peer verify setting (setting it to verify_none), the LDAP sign in started working again. This was NOT happen in our dev, staging, or production environments that also had the same identical upgrade. Weird, right?

Yesterday, the same faulty behavior show up in the development, staging, and production environments: any attempt to sign-in with AD credentials into the Web Management portal would fail, due to a bad TLS handshake with the LDAP servers. Again, setting the verify_peer setting to verify_none corrected the issue. I re-checked the local RabbitMQ server certificates and they were all valid. It also should be noted that I was assured by others in the IT department that nothing was done regarding our LDAP configuration or internal certificates. So, it seems like the only change made was the RabbitMQ upgrade.

I didn't post this to the google group or Github until now, as I suspect it was something related to our internal environment. I still am leaning that way. However, someone posted in the Google group an issue with LDAP and thought that perhaps something else is going on.

Here are the steps I've performed during the upgrade, from the perspective of the QA server:

0. Approximately a 10-14 days ago - Accessed Web Management pre-upgrade to confirm RabbitMQ was at 3.8.7 / Erlang 23.0. Successful secured LDAP communication via a valid TLS certificate which does not expire until 3rd Quarter 2022. No problems at all.
1. Stopped RabbitMQ node via rabbitmqctl in the rabbitmq_server-3.8.7 folder (rabbitmqctl stop)
2. Upgraded RabbitMQ to 3.8.9, via the EXE on Github. No errors noted in the dialog box.
3. Started RabbitMQ node via rabbitmqctl in the rabbitmq_server-3.8.9 folder (rabbitmqctl start_app) Appears working. rabbitmqctl status report looks fine. Clients to the Exchanges / Queues reconnect without any obvious issue.
4. Went into RabbitMQ Web Management portal - successful sign in via secured LDAP communication via the same valid TLS certificate as in step 0.
5. Approximately a week after the upgrade performed in the previous steps, I attempted to go into the QA environment RabbitMQ Web management portal and couldn't sign in.
   -> Other environments that followed steps 0 through 4 worked just fine when going into their Web Management.
   --> As of yesterday, the other three environments started failing to accept my AD credentials until I turned off peer validation.

Current status is that LDAP logins are working without peer validation of the certificate. Less than ideal, but functional for the time being.

Sorry for being so long-winded but perhaps this might be of value to someone.

[lukebakken]

Can you give some details about the chain of trust for the certificates you're using? The reason I ask is that your symptoms are very similar to those described here, where there are several certs in the server certificate's chain of trust: https://groups.google.com/g/rabbitmq-users/c/lquxjze8UIc/m/9E5mkg88AgAJ

The Erlang VM has no knowledge of Windows certificate stores, so you must ensure that the full trust chain is included in the file pointed to by the cacertfile setting.

My guess is that the certificate(s) used to sign your LDAP server certificate are not all present in the cacertfile file used in your LDAP TLS settings. I can't explain why this environment used to work, however, unless Erlang was also upgraded.

At any rate, that rabbitmq-users discussion explains how I tested the TLS handshake using openssl s_client. Here are the full commands I used at first to test the certs:

# run a TLS server
openssl s_server -accept 8443 -cert server_certificate.pem -key private_key.pem -CAfile full_ca_chain.pem

# run a TLS client
openssl s_client -connect localhost:5671 -cert client_certificate.pem -key client_private_key.pem -CAfile full_ca_chain.pem -verify 8 -verify_hostname $SERVER_HOSTNAME

If you could run the equivalent openssl s_client command against your LDAP server while using the same certs that RabbitMQ's LDAP plugin is configured to use, we may get some insight.

[ipock]

Much thanks for your reply. I have seen this issue reported and one difference is that the version I upgraded to is 3.8.9, where as that specific issue was an upgrade to a higher version. Please note that Erlang was NOT upgraded at this time, only RabbitMQ. I have NOT tried reverting my configuration back to requiring peer verification to see if it is still broken (which I'll probably do later today).

I understand that the Erlang VM doesn't know about the Windows Certs. I currently have all of the certs available in an accessible folder on a drive, "E:\RabbitMQ\Certs". There are three files: key.pem, cert.pem, and cacert.pem. All three of which are exactly what you'd expect: key.pem contains the private key for the server cers, cert.pem contains the RabbitMQ server certs, and the cacerts.pem contains the certificate authority certificate export.

To be clear, the advanced.config file the non-working LDAP configuration was like this:
{use_ssl, true},
{ssl_options, [{cacertfile,"e:/RabbitMQ/certs/cacert.pem"},
{certfile,"e:/RabbitMQ/certs/cert.pem"},
{keyfile,"e:/RabbitMQ/certs/key.pem"},
{verify, verify_peer},
{fail_if_no_peer_cert, true}]},

I did run openssl on the server using the requested settings with one exception: I connected to the LDAP server, since that is what was where the problem was seen.

Command:
openssl s_client -connect ldap.spaceshipyamato.net:636 -key key.pem -CAfile cacert.pem -cert cert.pem -verify 8 -verify_hostname ldap.spaceshipyamato.net

Result:
openssl_ldap_sclient_connection_test_b.txt

Command:
openssl s_client -connect localhost:5671 -cert cert.pem -CAfile cacert.pem -key key.pem

Result:
openssl_sclient_localhost_test_b.txt

Nothing jumps out at me in the output as being a problem.

Any thoughts??

[ipock]

Ok, please ignore this thread. The issue is valid, and I have tracked down the issue to an improperly created certificate on one of the round-robin LDAP servers (which is why it worked fine for so long). It would seem "ldap.spaceshipyamato.net" has a number of servers behind it with at least one of them having a cert without a subject.

Thanks for listening.

[ipock]

Check to be sure that ALL of the LDAP servers have similar certificates. In this case, one server had a subject omitted.

[lukebakken]

@ipock well done figuring that out!