Running DTNs in the public cloud

[Tobin Magle]

Hi all,

 

We’re in the process of redesigning our Globus infrastructure. We’re considering running some DTNs in the public cloud (AWS/GCP/Azure) for storage services that make sense (cloud object storage, SharePoint etc.).

 

We’d like some advice on what instance types to run as DTNs and any strategies that folks have come up with for using autoscaling groups so we can price it out and see if it’s financially feasible.

 

We have discussed this with Globus staff and ESNet and are looking for actual experiences running DTNs/Globus this way. Happy to set a time to talk over Zoom if that’s easier for people.

 

Thanks,

 

Tobin

 

 

Tobin Magle, PhD (they/she)

Lead Data Management Specialist

Research Computing Services

Northwestern University IT

tobin...@northwestern.edu

847-467-0263

Request a data consultation

 

[Scott Friedman]

Hi Tobin

I would be happy to connect with you (or anyone else) about this on AWS.  I would be happy to enable some experimentation with you to explore.

Feel free to contact me by email.

-Scott

Scott Friedman, Ph.D.

AWS Higher Education Research

To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u...@globus.org.

[Kaufman, Ian]

I would be interested in such a Zoom ...

Ian

Sent via the Samsung Galaxy S22 5G, an AT&T 5G smartphone
Get Outlook for Android

---

From: dis...@globus.org <dis...@globus.org> on behalf of Tobin Magle <tobin...@northwestern.edu>
Sent: Wednesday, July 24, 2024 10:23:04 AM
To: dis...@globus.org <dis...@globus.org>
Subject: [Globus Discuss] Running DTNs in the public cloud

 

To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u...@globus.org.

[Lev Gorenstein]

Aye, me too!

To summarize for everyone what we discussed with Tobin prior to the post, we do not have specific instance recommendations because the necessary specs might vary greatly depending on how busy the endpoint is.  So while one could infer a few general ideas ("Globus does checksums" ==> "needs CPU cores and RAM", "data is transferred" ==> "needs bandwidth to external network and to storage", "concurrency and maintainability" ==> "better to have two smaller nodes than one beefy one", etc), a lot of this ends up being a trial and error improvement method. Deploy, watch performance and DTN load, add resources if necessary.  And the elasticity of VM/cloud approach makes this even easier.

And a very important aspect of Globus is that no matter what instance type you pick, your endpoint will still work (at least initially). What I mean is that even if the initial chosen DTN configuration is insufficient for the [future] steady state active usage, any failures and overloads will not be sudden, but rather will ramp up and develop gradually (and even when they happen, the fire-and-forget nature of Globus will make them much less painful). So your server-side monitoring will spot the trends in CPU/RAM/network usage, and will alert you with enough time to mediate... and mediation would likely be of the easiest kind (just throw in an extra server or bigger instance).

Lev

--

Lev Gorenstein
Solutions Architect
Globus // University of Chicago

e: l...@globus.org

[Scott Friedman]

Lev, I'd like to suggest that you and I arrange something offline. Then, anyone on this list that wants to participate can do so. I can facilitate something via Globus or if any of you want to try things in your own environment we can do that as well.

BTW - my Amazon email is: sco...@amazon.com

-Scott