Unable to authenticate to mongodb during provision

[Jonathan Blok]

Hi all, I'm having an issue while provisioning OpenConext to a machine. For some reason authentication with the mongodb is failing. I have the following entries in my secrets file:

mongo_passwords:

   manage: <manage_password>

mongo_admin_password: <admin_password>

I've tried connecting to the mongodb manually, but I am not able to authenticate either. I'm very confused as to why this is happening. Any suggestions on how to proceed would be greatly appreciated. 

This is the stacktrace:

<172.19.53.26> (0, '', 'OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 30868\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')

The full traceback is:

Traceback (most recent call last):

  File "/tmp/ansible_r1HH0_/ansible_module_mongodb_user.py", line 398, in main

    client.admin.authenticate(login_user, login_password, source=login_database)

  File "/usr/lib64/python2.7/site-packages/pymongo/database.py", line 1167, in authenticate

    connect=True)

  File "/usr/lib64/python2.7/site-packages/pymongo/mongo_client.py", line 588, in _cache_credentials

    sock_info.authenticate(credentials)

  File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 620, in authenticate

    auth.authenticate(credentials, self)

  File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 486, in authenticate

    auth_func(credentials, sock_info)

  File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 466, in _authenticate_default

    return _authenticate_scram_sha1(credentials, sock_info)

  File "/usr/lib64/python2.7/site-packages/pymongo/auth.py", line 237, in _authenticate_scram_sha1

    res = sock_info.command(source, cmd)

  File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 517, in command

    collation=collation)

  File "/usr/lib64/python2.7/site-packages/pymongo/network.py", line 125, in command

    parse_write_concern_error=parse_write_concern_error)

  File "/usr/lib64/python2.7/site-packages/pymongo/helpers.py", line 145, in _check_command_response

    raise OperationFailure(msg % errmsg, code, response)

OperationFailure: Authentication failed.

fatal: [172.19.53.26]: FAILED! => {

    "changed": false,

    "invocation": {

        "module_args": {

            "database": "manage",

            "login_database": "admin",

            "login_host": "localhost",

            "login_password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",

            "login_port": "27017",

            "login_user": "admin",

            "name": "managerw",

            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",

            "replica_set": null,

            "roles": [

                "readWrite"

            ],

            "ssl": false,

            "ssl_cert_reqs": "CERT_REQUIRED",

            "state": "present",

            "update_password": "always"

        }

    },

    "msg": "unable to connect to database: Authentication failed."

}

to retry, use: --limit @/home/jblok/OpenConext-deploy-2/provision-beng-clariah-acc.retry

PLAY RECAP **************************************************************************************************************************

172.19.53.26               : ok=303  changed=1    unreachable=0    failed=1

[Bart Geesink]

Hi Jonathan,

On 9/11/18 4:13 PM, Jonathan Blok wrote:
> Hi all, I'm having an issue while provisioning OpenConext to a machine.
> For some reason authentication with the mongodb is failing. I have the
> following entries in my secrets file:
>
> mongo_passwords:
>    manage: <manage_password>
>
> mongo_admin_password: <admin_password>
>
> I've tried connecting to the mongodb manually, but I am not able to
> authenticate either. I'm very confused as to why this is happening. Any
> suggestions on how to proceed would be greatly appreciated. 
>

Can you connect to the database on the target server using the following
command?

mongo -u admin -p --authenticationDatabase admin

And is this a clean install or did it work before?

Regards,
Bart

> --
> OpenConext - Open For Collaboration
> ---
> You received this message because you are subscribed to the Google
> Groups "OpenConext Community" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to openconext+...@googlegroups.com
> <mailto:openconext+...@googlegroups.com>.
> To post to this group, send email to openc...@googlegroups.com
> <mailto:openc...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/openconext.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/openconext/8dbef4c7-6ae4-4661-b8c4-642a0fe2a81a%40googlegroups.com
> <https://groups.google.com/d/msgid/openconext/8dbef4c7-6ae4-4661-b8c4-642a0fe2a81a%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Jonathan Blok]

Hi Bart,

Thank you for your reply. When using the passwords from my secrets file I am also not able to authenticate using the command you mentioned. Again, the error is "Error: Authentication failed.". This is a clean install as I'm deploying this to a new environment. 

Kind regards,

Jonathan

Op dinsdag 11 september 2018 16:45:26 UTC+2 schreef Bart Geesink:

> <mailto:openconext+unsub...@googlegroups.com>.

[Jonathan Blok]

Does anyone else have a suggestion on how to fix this? I am completely stuck. 

Op dinsdag 11 september 2018 16:13:42 UTC+2 schreef Jonathan Blok:

[Thijs Kinkhorst]

Hoi Jonathan,

Op 11-09-18 om 16:13 schreef Jonathan Blok:

> Hi all, I'm having an issue while provisioning OpenConext to a machine.
> For some reason authentication with the mongodb is failing. I have the
> following entries in my secrets file:

The error you get is about the admin user not being authenticated to Mongo.

The admin user is created with Ansible on the condition that the Mongo
installation is completely new. This is checked in the first stanza of
roles/mongo/tasks/main.yml. This means that if the install somehow
failed before the admin user is created but a directory /var/lib/mongo
remains, then creating the admin user will be skipped on any subsequent
attempts.

I suggest you remove mongo from the system and ensure that
/var/lib/mongo does not exist, then re-run the ansible role.


Cheers,
Thijs

[Jonathan Blok]

Thank you for the reply. After manually uninstalling mongo and removing the /var/lib/mongo directory, it was necessary to recreate the /var/lib/mongo folder with the correct permissions. I ran the deployment again the "mongo" tag. The deployment failed again on the same task. 

TASK [mongo : Create mongo database users] 

In the mongo log file:

2018-09-22T10:42:52.692+0200 I ACCESS   [conn2] SCRAM-SHA-1 authentication failed for admin on admin from client 127.0.0.1:32800 ; UserNotFound: Could not find user admin@admin

Do you have any further suggestions?

Op dinsdag 18 september 2018 14:46:30 UTC+2 schreef Thijs Kinkhorst:

[Thijs Kinkhorst]

Hi Jonathan,

Op 22-09-18 om 10:47 schreef Jonathan Blok:

> Thank you for the reply. After manually uninstalling mongo and removing
> the /var/lib/mongo directory, it was necessary to recreate the
> /var/lib/mongo folder with the correct permissions. I ran the deployment
> again the "mongo" tag. The deployment failed again on the same task. 

The current ansible task will not create an admin user if the
/var/lib/mongo directory is present before the task is run (it then
assumes that Mongo is already installed and will not mess with the admin
user).

Why is it necessary to create the directory /var/lib/mongo manually?
Doesn't the rpm installation take care of that?


Cheers,
Thijs

[Jonathan Blok]

Hi Thijs,

If I don't create the directory, the mongod service fails with the following error:

2018-09-22T10:12:35.490+0200 I STORAGE  [initandlisten] exception in initAndListen: 29 Data directory /var/lib/mongo not found., terminating

so I created the directory again and adjusted the permissions manually. 

Regards,

Jonathan

Op zaterdag 22 september 2018 10:57:49 UTC+2 schreef Thijs Kinkhorst:

[Thijs Kinkhorst]

Op 22-09-18 om 14:01 schreef Jonathan Blok:

>
> If I don't create the directory, the mongod service fails with the
> following error:
>
> 2018-09-22T10:12:35.490+0200 I STORAGE  [initandlisten] exception in
> initAndListen: 29 Data directory /var/lib/mongo not found., terminating
>
> so I created the directory again and adjusted the permissions manually. 
>

So I'm not sure why your mongo package behaves differently in that
regard than with us. But that's a bit hard to debug remotely.

As a workaround, you could try the following.
In this file
https://github.com/OpenConext/OpenConext-deploy/blob/master/roles/mongo/tasks/main.yml
comment out lines 57 and 67 (" - mongo_present.stat.exists == False").

Then you skip the check that does not work for you, and those tasks will
be run.


Cheers,
Thijs