Endpoint setup error second time on DNS registration

[Marshall McDonnell]

Hello,

I was going through the Globus Connect Server v5 installation on a server on a cloud instance.

I had spun up the instance and started the installation up until setting up the "data gateway".

So I had already done setup for the "endpoint" and the "data transfer node".

But, then I spun down the instance in an ephemeral manner (kept none of the files on the server after installation, such as the deployment-key.json)

I spun back up the instance with the same DNS name and client credentials and trying to re-install starting at the endpoint setup.

Yet, when I go to do "globus-connect-server endpoint setup "<name>" --client-id <id> --organization <org> --owner <email>", I get to the DNS registration and get the following error:

    GraphQLError: (GraphQLError(...), 'Object WILDCARD_DOMAIN already exists.')

I assume this is related to the fact I already did a DNS registration with this same set of hostname and client credentials.

Is there anyway to "reset" the credentials / certificates?

And also curious, is this DNS registration error on the  Globus service side of things or the Let's Encrypt service side?

Thanks and let me know if I can provide any further information!

If it helps, would like to containerize an installation of GCS v5 endpoint.

And would like to do ephemeral, automated installations (going to start looking over this project https://github.com/globus/gcsv5_installer)

Any help or advice towards that goal would be much appreciated!

Best,

Marshall

[Jason Alt]

Marshall-

You would need the client id, secret and deployment key to recover the endpoint (technically it is 'adding a new node to the endpoint' but we'll get to that). The client id and secret allow the GCS services on your endpoint to communicate with other Globus services as needed (like Globus Transfer) and the deployment key allows a node to pull down the latest, encrypted configuration for your endpoint (in your case, the configuration with the domain, endpoint id, gateway configuration).

`globus-connect-server endpoint setup` effectively creates all of the resources in the other Globus services; there really are no changes to the local node. So while you lost your node, the 'endpoint' still exists as defined in Globus Transfer (and other Globus services).

If you had the deployment-key, the correct action here would be `globus-connect-server node setup ...` to add this new node to the existing endpoint. `node setup` enables all of the GCS services on the node (ie gridftp) that everyone has come to know and expect. If you had the deployment key, you'd be up and running.

Unfortunately, there is no recovery from losing the deployment key. This is by design. The deployment key is what encrypts all of the endpoint's confidential configuration so that it is only ever visible to the node. If we (Globus) were able to recover the deployment key, that would suggest that we also have implicit access to the configuration. 

The error you received "GraphQLError: (GraphQLError(...), 'Object WILDCARD_DOMAIN already exists.'" is immaterial at this point but for reference, that error occurred because trying to store a new encrypted configuration where one already exists. If you had access to the deployment key, the command would have succeeded but updating the existing configuration. We have an internal issue to improve the error message you received, in order to improve it and make the next steps more obvious.

What you need to do now is `globus-connect-server endpoint delete ...`. This will delete the endpoint definition throughout the Globus platform. It will ask you if you still have access to the deployment key or if you want to continue. Choose the option to continue with the deletion of the endpoint. Once that is complete, you are free to delete the client registration at https://auth.test.globuscs.info/developers.

At this point, you'll want to repeat the installation and configuration steps for a new endpoint, starting with the step 'globus-connect-server endpoint create ...'. You are free to reuse the same displayname and other configuration you used for the prior endpoint. In the end, you'll end up with a different endpoint id, but since the endpoint was non functional, this shouldn't be an issue.

Finally, containers are something we are looking into. And I don't mean that passively; we are literally planning that now. The process has just begun though and since we want to release something product-worthy and generally applicable, it may be several weeks before release. Feel free to contact me directly if you are interested, I'd be interested in any additional use cases.

Also, the code at https://github.com/globus/gcsv5_installer is mine. It was written for a project and may now be a little out of date (iirc it is based on GCSv5.2) but it should serve as a good reference if you use Ansible. We also plan support for configuration management tools like Ansible, but that priority is more 'on the roadmap' than the immediate work we are doing with containers. Again, feel free to contact me directly if you are interested, although support on that repo is fairly limited beyond guidance at this time.

Jason

[Marshall McDonnell]

Hi Jason,

Thanks so much for the detailed response!

Before I read it I did end up using "globus-connect-server endpoint cleanup"

I didn't see the "globus-connect-server endpoint delete" command via the CLI but I might have an older version (Installing on CentOS 7 via yum; package 5.4.15, cli 1.0.14)

Long story short: Got it and back to installing it again!

Details:

> "Unfortunately, there is no recovery from losing the deployment key. This is by design. ... If we (Globus) were able to recover the deployment key, that would suggest that we also have implicit access to the configuration."

Gotcha and that makes a lot of sense!

I do remember seeing in the documentation how to spin up for multiple data nodes and that you would need the deployment-key.json

> We have an internal issue to improve the error message you received, in order to improve it and make the next steps more obvious.

Good to know!

The documentation does help for sure (online + CLI) so great job there!

> Finally, containers are something we are looking into. And I don't mean that passively; we are literally planning that now. The process has just begun though and since we want to release something product-worthy and generally applicable, it may be several weeks before release. Feel free to contact me directly if you are interested, I'd be interested in any additional use cases.

Oh wow! Great news!

I had literally spent the last few months working on Terraform + Docker to setup quick GCS v4 endpoints so we could do some application testing.

I know we really should mock these calls but really helps to figure out stuff with the actual endpoints!

Anyway, so glad to hear and if the development is open-source, would love to track progress and even help if I even can!

> Also, the code at https://github.com/globus/gcsv5_installer is mine.

Cool! Thanks for putting that out there and sharing it open source! Great work!

> We also plan support for configuration management tools like Ansible, but that priority is more 'on the roadmap' than the immediate work we are doing with containers.

Yeah, I do see having images to spin up containers from would actually negate my need for having to install via configuration management.

Overall, thank you so much for reaching out and definitely would like to discuss the containerization of GCS v5 in the future!

Best,

Marshall

[Jason Alt]

On Wed, Feb 10, 2021 at 1:52 PM Marshall McDonnell <cmrg...@gmail.com> wrote:

Hi Jason,

Thanks so much for the detailed response!

Before I read it I did end up using "globus-connect-server endpoint cleanup"

I didn't see the "globus-connect-server endpoint delete" command via the CLI but I might have an older version (Installing on CentOS 7 via yum; package 5.4.15, cli 1.0.14)

Oops, thanks for the correction. `endpoint cleanup` will delete the endpoint. I had my wires crossed.

[Joshua Brown]

Is there an update on this? I'm running into a similar issue, for one I don't know how to check what endpoints exist on my machine? Most of the GCS (5.4) commands only work once you have actually logged in with your endpoint UUID and I don't see anything in the API documentation on being able to list endpoints associated with a domain name from the CLI, though I could be missing something.

[Jason Alt]

It's not clear what you are trying to do. There can only be one endpoint per node and only one endpoint per domain. If the node has been setup, you can get the domain and endpoint ID (which is actually he client id) from /var/lib/globus-connect-server/info.json:

$ cat /var/lib/globus-connect-server/info.json

{

    "client_id": "ba11def8-7d2b-4ab3-8a12-1680446d4d3e",

    "DATA_TYPE": "info#1.0.0",

    "domain_name": "1008a.8540.sandbox2.zones.dnsteam.globuscs.info",

    "manager_version": "5.4.36",

    "endpoint_id": "ba11def8-7d2b-4ab3-8a12-1680446d4d3e",

    "api_version": "1.9.0"

}

Which is not terribly exciting if all you want to do is 'globus-connect-server login' from the local node because you can use the 'localhost' option:

$ globus-connect-server login localhost

Please authenticate with Globus here:

------------------------------------

https://auth.sandbox.globuscs.info/v2/oauth2/authorize?client_id=202b928b-4c6d-4b27-9c03-b00584181476&redirect_uri=https%3A%2F%2Fauth.sandbox.globuscs.info%2Fv2%2Fweb%2Fauth-code&scope=openid+profile+email+urn%3Aglobus%3Aauth%3Ascope%3Aauth.globus.org%3Aview_identity_set+urn%3Aglobus%3Aauth%3Ascope%3Aba11def8-7d2b-4ab3-8a12-1680446d4d3e%3Amanage_collections&state=_default&response_type=code&access_type=offline&prompt=login

------------------------------------

But if my chance you have the endpoint ID and you are looking for the domain, you can 'globus-connect-server login <endpoint_id>' and then 'globus-connect-server endpoint show'; you'll see the domain in the gcs_manager_url field.

Jason

[Joshua Brown]

Thanks @jason that is actually really helpful.