Marathon UI breaks for HTTPS only

[pawa...@gmail.com]

Hi,

I am running a Marathon (0.8.3 RC3) cluster on three machines which only server traffic on HTTPS 8443. I explicitly disabled http using the --disable_http flag.

 java -Djava.library.path=/usr/lib/mesos/lib:/usr/lib/java64/jdk1.7.0/jre/lib/amd64/server \

     -jar marathon-0.8.2-RC3.jar \

    --ssl_keystore_path $MARATHON_HOME/conf/ssl/marathon.jks\

    --ssl_keystore_password pass \

    --mesos_authentication_principal admin \

    --mesos_authentication_secret_file "$MARATHON_HOME/conf/credentials" \

    --disable_http \

    --https_port 8443 \

    --webui_url https://192.168.1.1:8443 \

    --master zk://192.168.1.1:2181,12.168.1.2:2181,192.168.1.3:2181/mesos \

    --zk zk://192.168.1.1:2181,12.168.1.2:2181,192.168.1.3:2181/marathon \

When I access the UI using https://marathon-host:8443, it is broken. I see the following in the logs of that machine.

[2015-05-20 20:02:12,654] INFO Proxying request to leader at marathon3.abc.com:8080 (mesosphere.marathon.api.LeaderProxyFilter:62)

[2015-05-20 20:02:12,655] WARN Exception while proxying (mesosphere.marathon.api.LeaderProxyFilter:128)

java.net.ConnectException: Connection refused

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)

So when the UI requests /apps from its local machine which is NOT a leader, it automatically proxies it to the leader machines on marathon-leader-machine:8080 and NOT on the https port marathon-leader-machine:8443. As my setup is https only and there is no one listening on 8080 the connection gets refused.

I went into zookeeper data store to check the entry for each marathon instance which is what Marathon seems to be using in LeaderProxyFilter.scala. And I found this:

root@zookeeper2:/a-zookeeper/bin# ./zkCli.sh -server localhost:2181

Connecting to localhost:2181

Welcome to ZooKeeper!

JLine support is enabled

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

[zk: localhost:2181(CONNECTED) 0] ls /marathon/leader 

[member_0000000010, member_0000000009, member_0000000008]

[zk: localhost:2181(CONNECTED) 1] get /marathon/leader/member_0000000010

marathon2.abc.com:8080

cZxid = 0x200000052

ctime = Wed May 20 19:09:58 UTC 2015

mZxid = 0x200000052

mtime = Wed May 20 19:09:58 UTC 2015

pZxid = 0x200000052

cversion = 0

dataVersion = 0

aclVersion = 0

ephemeralOwner = 0x34d728a15f0000a

dataLength = 39

numChildren = 0

[zk: localhost:2181(CONNECTED) 2] 

--

The bottom line is, Marathon doesnt seem to support HTTPS only mode. Can you confirm this ? If so, this seems to be a serious flaw.

 

I can only run apps on https in our setup and was hoping there is a way we can workaround this. Please help.

--

Thanks,

pawan