SRV Discovery customization? Deploying multiple clusters within DCOS (Mesos).

[Ryan Bezdicek]

We are attempting to deploy many different etcd clusters within DCOS. In order to do this we are deploying the clusters into DCOS folders. We are currently using the etcd discovery services to bootstrap the cluster, however, we've found it to be unreliable in DCOS. We were hoping to use the SRV discovery, however there is a mismatch in how DCOS generates records and what etcd looks for. 

DCOS generates the record for an etcd cluster with an ID of /test/etcd_server as: 

    _etcd_server.test._tcp.marathon.mesos

etcd is expecting: 

    _etcd_server._tcp.test.marathon.mesos

Is there anyway to customize the search terms used for SRV discovery? Does anyone have a better solution for cluster discovery within DCOS?

When we use the etcd discovery service, roughly 1/20 container deployments will have the container fail to at some point during bootstrapping (no error messages, many hours debugging). When the container is relaunched, there is no bootstrapped info on disk (mounted persisted volume) but the containers 'spot' in the discovery key was taken.

Thanks!