Authenticating marathon to a mesos-master (framework authentication)

[Ram Rajamony]

I spent much more time than necessary in trying to get marathon to authenticate itself to a meson-master. The problem turned out to be that the file in which I specified the secret (through --mesos_authentication_secret_file) was newline terminated, when in fact, it should have been just the raw string. This became clear after going through the marathon code, but others may make the same mistake especially considering how mesos-master drops the newline when it picks the principal and secret from a text file specified via --credentials. 

Its easy to make a note of this in the documentation (command-line-flags.md) - will you accept a pull request to this effect? [I couldn't immediately find guidelines for contribution]

Regards,

-ram.

[Dario Rexin]

Hi Ram,

pull requests are always welcome. 

Cheers,

Dario

--
You received this message because you are subscribed to the Google Groups "marathon-framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framew...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Pradeep Sekar]

Hi Ram

I m unable to setup mesos marathon framework authentication. I m getting "Master refused authentication" error when i start marathon with authentication enabled. Below is what i did.

I did install cyrus-sasl-devel cyrus-sasl-md5 packages for CRAM MD5 to work.

Mesos Master:

$cat /tmp/mesos-secret

marathon secret

$cat /etc/mesos-master/credentials

/tmp/mesos-secret

systemctl restart mesos-master (it started ok)

Marathon:

$cat /tmp/marathon-secret

secret

Starting marathon in cmdline:

java -Djava.library.path=/usr/local/lib -Djava.util.logging.SimpleFormatter.format=%2$s%5$s%6$s%n -Xmx512m -cp /usr/local/bin/marathon mesosphere.marathon.Main --zk zk://X.X.X.X:2181/marathon --master zk://X.X.X.X:2181/mesos  --mesos_authentication_principal marathon --mesos_authentication_secret_file /tmp/marathon-secret

I0303 19:48:28.260689  2606 authenticatee.hpp:250] Attempting to authenticate with mechanism 'CRAM-MD5'

I0303 19:48:28.278282  2606 authenticatee.hpp:270] Received SASL authentication step

E0303 19:48:28.299891  2606 sched.cpp:353] Master mas...@X.X.X.X:5050 refused authentication

I0303 19:48:28.299921  2606 sched.cpp:736] Got error 'Master refused authentication'

I0303 19:48:28.299932  2606 sched.cpp:1320] Asked to abort the driver

[2015-03-03 19:48:28,300] WARN Error: Master refused authentication (mesosphere.marathon.MarathonScheduler:255)

Not sure what im missing here ? Any ideas?

OTH, the logging i think can be improved in mesos at least in case of authentication failures. I m pure java/scala guy, but i ll try to send a pull request if there is something i can make it better here.

[Ram Rajamony]

The key is that the secret as given to Marathon (in /tmp/marathon-secret) should not be newline terminated. Thus, if the secret in /tmp/mesos-secret is "secret" (excluding quotes), you would have to provide this to marathon via echo -n "secret" > /tmp/marathon-secret. Note the use of -n to prevent a newline from being added. 

[Sivaram Kannan]

Hi Ram,

I am having an autthentication issue even if I don't have new line in the secret file. Am passing the file with the below content

user1 password1

as --credentials in mesos-master.

In the marathon am passing a file marathon_password which has the content password1 without the newline as you specified above. But I am getting the below error

I0304 06:12:54.169895    44 sched.cpp:284] Authenticating with master mas...@14.1.1.1:5050
I0304 06:12:54.170111    44 authenticatee.hpp:133] Creating new client SASL connection
W0304 06:12:59.175511    47 sched.cpp:379] Authentication timed out
I0304 06:12:59.175879    47 sched.cpp:339] Failed to authenticate with master mas...@14.1.1.1:5050: Authentication discarded
I0304 06:12:59.176357    47 sched.cpp:284] Authenticating with master mas...@14.1.1.1:5050
I0304 06:12:59.177878    47 authenticatee.hpp:133] Creating new client SASL connection
W0304 06:13:04.183523    51 sched.cpp:379] Authentication timed out
I0304 06:13:04.184674    51 sched.cpp:339] Failed to authenticate with master mas...@14.1.1.1:5050: Authentication discarded
I0304 06:13:04.184855    51 sched.cpp:284] Authenticating with master mas...@14.1.1.1:5050

Following is the error in mesos-master

I0303 17:17:29.586299    12 authenticator.hpp:166] Creating new server SASL connection
W0303 17:17:30.600091    16 master.cpp:3954] Failed to authenticate scheduler-7bd6b0d8-f233...@172.17.0.7:9090: Failed to communicate with authenticatee
I0303 17:17:34.592434     9 master.cpp:3901] Authenticating scheduler-7bd6b0d8-f233...@172.17.0.7:9090
I0303 17:17:34.592509     9 master.cpp:3912] Using default CRAM-MD5 authenticator

Any ideas what could be wrong ??

Thanks,
./Siva.

[Connor Doyle]

Just to make sure: does the framework successfully register without authentication enabled?  From the logs it looks a bit like the master can't talk to the scheduler.
--

Connor

[Sivaram Kannan]

Hi Connor,

Earlier I had problem in passing a file to --acl parameter since this is a container. I was passing the --acl=<the whole json content> and authorization was working. Now I mounted the acl.json to the container and since it did not throw any error I assumed it would work - but I find authorization itself is failing which I am guessing due to mounting the file. Let me debug that and come to this later.

Thanks for point it out.

./Siva.

On Wednesday, March 4, 2015 at 12:06:32 PM UTC+5:30, Connor Doyle wrote:

Just to make sure: does the framework successfully register without authentication enabled?  From the logs it looks a bit like the master can't talk to the scheduler.
--

Connor

On Mar 3, 2015, at 22:29, Sivaram Kannan <siva...@gmail.com> wrote:

Hi Ram,

I am having an autthentication issue even if I don't have new line in the secret file. Am passing the file with the below content

user1 password1

as --credentials in mesos-master.

In the marathon am passing a file marathon_password which has the content password1 without the newline as you specified above. But I am getting the below error

I0304 06:12:54.169895    44 sched.cpp:284] Authenticating with master mas...@14.1.1.1:5050
I0304 06:12:54.170111    44 authenticatee.hpp:133] Creating new client SASL connection
W0304 06:12:59.175511    47 sched.cpp:379] Authentication timed out
I0304 06:12:59.175879    47 sched.cpp:339] Failed to authenticate with master mas...@14.1.1.1:5050: Authentication discarded
I0304 06:12:59.176357    47 sched.cpp:284] Authenticating with master mas...@14.1.1.1:5050
I0304 06:12:59.177878    47 authenticatee.hpp:133] Creating new client SASL connection
W0304 06:13:04.183523    51 sched.cpp:379] Authentication timed out
I0304 06:13:04.184674    51 sched.cpp:339] Failed to authenticate with master mas...@14.1.1.1:5050: Authentication discarded
I0304 06:13:04.184855    51 sched.cpp:284] Authenticating with master mas...@14.1.1.1:5050

Following is the error in mesos-master

I0303 17:17:29.586299    12 authenticator.hpp:166] Creating new server SASL connection

W0303 17:17:30.600091    16 master.cpp:3954] Failed to authenticate scheduler-7bd6b0d8-f233-4504-bec2-3cf...@172.17.0.7:9090: Failed to communicate with authenticatee
I0303 17:17:34.592434     9 master.cpp:3901] Authenticating scheduler-7bd6b0d8-f233-4504-bec2-3cf...@172.17.0.7:9090

To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framework+unsub...@googlegroups.com.

[Sivaram Kannan]

Hi Connor,

I got it working. Earlier I forgot to pass --roles parameter for mesos-master. The above error does not reflect it since I made a stupid mistake. Not I got it working after passing --roles to the mesos-master.

Thanks,

./Siva.

--

ever tried. ever failed. no matter.
try again. fail again. fail better.
        -- Samuel Beckett

[Shingo Omura]

Hi folks,

I'm also stucked in framework authentication. I did 'echo -n' all credentials for mesos-master and secret file for marathon.  However, marathon can't be authenticated.  As Conor guided, I confirmed marathon can be registered to mesos-master without authentication.

I gist-ed my environments and logs here: https://gist.github.com/everpeace/eecdf1fcd9ee7cd6cd36

I would be glad if you pointed out what could be wrong with this setting.

Best regards,
--

Shingo Omura

ever...@gmail.com

Sivaram Kannan:

W0303 17:17:30.600091 16 master.cpp:3954] Failed to authenticate scheduler-7bd6b0d8-f233...@172.17.0.7:9090: Failed to communicate with authenticatee
I0303 17:17:34.592434 9 master.cpp:3901] Authenticating scheduler-7bd6b0d8-f233...@172.17.0.7:9090

To unsubscribe from this group and stop receiving emails from it, send an email to marathon-framew...@googlegroups.com.

[Dario Rexin]

Hi Shingo,

sorry for the delay. Seems like you found a bug. Marathon does not trim the \n at the end of the line, so the secret will never match. I just created a patch for that: https://github.com/mesosphere/marathon/pull/1290.

Cheers,

Dario

[OMURA, Shingo]

Thanks for the fix, Dario.

I could fix my PR for authentication in myriad by referring this, too.

--
Shingo Omura
ever...@gmail.com
http://everpeace.github.io/