Storing disaster recovery secrets

[Jason Healy]

Question about how small shops handle this. We only have a single location, so I don't have multiple offices where I can get some georedundancy.

What do you do with critical information needed for disaster recovery (such as encryption secrets)? I mean, worst-case scenario there's a meteor strike here and it takes out all our servers as well as me and my staff. Where should I keep the passwords that would allow for restoration from backup?

Do you put everything in a safe deposit box somewhere remote?

Have our legal counsel keep it in their safe?

Use some kind of an escrow service?


I figure the information has to be physically secured, but not encrypted (as that would mean someone would have to know the key). I'm just not sure where the balance of security and actual feasibility lies in this case.

Thanks,

Jason

[Matt Finnigan]

Any of those would work, I've done similar in the past. You could also use a hosted service like lastpass, or a Keepass DB in GDrive or Dropbox, that contains only the minimum passwords to bootstrap a DR, if that can be done without leaking actual live/production secrets.

--
This list provided by the League of Professional System Administrators
http://lopsa.org/
---
You received this message because you are subscribed to the Google Groups "LOPSA Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u...@lopsa.org.
To view this discussion on the web visit https://groups.google.com/a/lopsa.org/d/msgid/discuss/CF583AE7-A223-4727-8302-AA4399AA195E%40logn.net.

[Steve Potter]

If you're a single location and a small group, what's the probability
that the company will still be a going concern if a meteor strike
happens?  Does the company have any kind of offsite storage for anything
else?  If so, I'd just leverage that. Otherwise a cloud-based credential
solution should be good enough or even a password protected file stored
in the cloud.  No matter what, someone who survives has to have a
minimal amount of knowledge to even start back up.

-spp

[Dan Ritter]

Jason Healy wrote:
> Question about how small shops handle this. We only have a single location, so I don't have multiple offices where I can get some georedundancy.
>
> What do you do with critical information needed for disaster recovery (such as encryption secrets)? I mean, worst-case scenario there's a meteor strike here and it takes out all our servers as well as me and my staff. Where should I keep the passwords that would allow for restoration from backup?

In the case of a meteor strike that has killed you, your staff,
and all your servers... do you care anymore?

> Do you put everything in a safe deposit box somewhere remote?

That can work.

> Have our legal counsel keep it in their safe?

That can work.

> Use some kind of an escrow service?

That is unlikely to work. Escrow services are geared to taking
things in, and then being very reluctant to hand them out.

For example, my employer sends a copy of all the instructions to
build a copy of our main service, minus encryption secrets and
authentication secrets, to a major escrow service. It usually
takes them a week to acknowledge receipt. It's not a backup for
us; it's a promise to our customers that if we go bankrupt, they
can try to build the service again.

>
> I figure the information has to be physically secured, but not encrypted (as that would mean someone would have to know the key). I'm just not sure where the balance of security and actual feasibility lies in this case.

How much is it worth to an attacker?

You need to buy the security level that will deter an attacker
from spending that much on acquiring your secret.

-dsr-

[Edward Harvey]

Create a shared folder in https://sync.com (which you should be using for everything, instead of dropbox or whatever else you're actually using)

Use it to share all your team documentation, including network diagrams, license keys, and everything else.

[Kourosh Ghassemieh]

Some ideas:  A safe deposit box at a local bank. A fire-resistant safe in the owner's house. A 3rd party DR/BCP service provider (Iron mountain, etc).  A related firm in another area (maybe a joint DR/BCP scenario where you help them if they help you).

My suggestion would be a bank's safe-deposit box for hard copies of access passwords and keys and a sheet with basic instructions and the locations of the DR/BCP documentation that can help a 3rd party consultant or service provider get the firm back up and running.  This info will need to be updated any time changes are made to the key and/or passwords.

Some other, longer notes below.

Really, this is a thought exercise and should be a part of your DR/BCP planning and HAS to include the high-level execs and/or the owner.  There are statistics I've heard that the majority of small businesses close if faced with a disaster because they have no recovery plans.

You can start by having table-top exercises with the execs and tell them that there's just been a fire/flood/earthquake/some other disaster and the entire building has been red-tagged and you can't get anything out.  What happens?  Can the business survive?  What do you need to get up and running?  How much would that cost?  Cost does come into play and there are plenty of options at all price levels but the execs and/or owner needs to be aware of potential scenarios and the repercussions.  It doesn't even have to be an area-wide natural disaster.  What happens if there's a fire that destroys your buildings?   What if IT staff can't or won't come in?  

Some DR/BCP scenarios that generally cover various disasters can help you think about what you and the company really need to do.  It could be as simple as a copy of various materials are kept at a 3rd party location or as complex as having a contract with a 3rd party DR services provider.  At that same time, you would have to think about who needs to access the information in a DR/BCP scenario and what they need to do with it.

Different companies have different needs for DR/BCP.  A consulting firm's DR/BCP procedures will be different than those of a manufacturing firm, which will be different than those of a financial services firm, etc...

There may also be regulations covering some of the data and your firm such as PCI, HIPAA, SEC regulations, FDIC regulations, etc.

Just as a note, we have a small branch office that we use as our DR site with copies of all kinds of information but that location is going away next summer so I'm thinking about some of this as well.  Someone has offered us the use of a room in their office but I may also get a safe-deposit box in a bank near that office to keep more sensitive materials.

Sorry for the long-winded reply, and you may have done a lot of this already but there may be others who haven't and I hope this info can help them improve their own DR/BCP plans.

Kourosh Ghassemieh

kou...@ghassemieh.com

On Mon, Nov 25, 2019 at 12:36 PM Jason Healy <jhe...@logn.net> wrote:

[Jason Healy]

On Nov 25, 2019, at 3:58 PM, Dan Ritter <d...@randomstring.org> wrote:
>
> In the case of a meteor strike that has killed you, your staff,
> and all your servers... do you care anymore?

Well, *I* don't... ;->

Maybe a plane crash instead of a meteor strike? Anyhow, we will be reviewing this along with the overall DR plan so this is all good. I'll see if the admin team has suggestions for other secure storage that we're already using.

Thank you all for the suggestions. We do have a DR plan, but up until now we haven't involved cloud storage and basically took the "if the site burned down we're screwed anyway" approach. However, now that we're going to send some high-value stuff off site I wanted to think about how to access it if I'm not here.

For the last option, "escrow" may have been the wrong choice of words... I just meant someone like Iron Mountain who would surrender access to a known controlling entity (like acting executive officers). The issue with encryption and cloud storage is no amount of paperwork will recover a lost encryption key; there has to be a human in the loop who can authenticate those requests under exceptional circumstances.

Thanks again for the pointers!

Jason

[Dan Ritter]

Jason Healy wrote:
> On Nov 25, 2019, at 3:58 PM, Dan Ritter <d...@randomstring.org> wrote:
> >
> > In the case of a meteor strike that has killed you, your staff,
> > and all your servers... do you care anymore?
>
> Well, *I* don't... ;->
>
> Maybe a plane crash instead of a meteor strike? Anyhow, we will be reviewing this along with the overall DR plan so this is all good. I'll see if the admin team has suggestions for other secure storage that we're already using.

$WORK's security and DR plan includes "a meteor destroys the Boston
datacenter and all employees in the area" and concludes: at that point, we
don't care. On the other hand, we have another datacenter in the DC area
because there are other scenarios like "fire in the Boston datacenter"
and "snowstorm takes out datacenter for longer than their diesel supplies
can last".

> Thank you all for the suggestions. We do have a DR plan, but up until now we haven't involved cloud storage and basically took the "if the site burned down we're screwed anyway" approach. However, now that we're going to send some high-value stuff off site I wanted to think about how to access it if I'm not here.
>
> For the last option, "escrow" may have been the wrong choice of words... I just meant someone like Iron Mountain who would surrender access to a known controlling entity (like acting executive officers). The issue with encryption and cloud storage is no amount of paperwork will recover a lost encryption key; there has to be a human in the loop who can authenticate those requests under exceptional circumstances.

No, that's exactly what escrow means. Iron Mountain will happily
take your secrets, and it will be days to weeks before they
return the secrets to a named party -- because they need to
ascertain that they aren't being party to fraud.

-dsr-

[Jerald Sheets]

If you’re using something like Puppet, you can encrypt your on-disk secrets using enamel and shove everything up to Amazon Glacier that is a digital asset.

You can also create an encrypted volume (LUKS, Sparse images, etc.) and push that up there too.

What I do as an added plus for the company for an “I’m now deceased” scenario is I push everything into a spreadsheet and password protect it. It goes in a sparse image and that gets put on our file server in a privileged location. Then, the spreadsheet password is printed out, put in an envelope and there are three copies. One with the CEO. One with the CFO. One in the company safe.

I’ve had some companies put those in a safe deposit box too.


Jerald Sheets
que...@gmail.com

> --
> This list provided by the League of Professional System Administrators
> http://lopsa.org/
> ---
> You received this message because you are subscribed to the Google Groups "LOPSA Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u...@lopsa.org.

> To view this discussion on the web visit https://groups.google.com/a/lopsa.org/d/msgid/discuss/20191126123341.z6gkddonuo2swtst%40randomstring.org.

[Edward Ned Harvey (lopser)]

I want to repeat this because I didn't get any replies:

Use sync.com. Everything is encrypted client-side so there is zero exposure, not even their employees can decrypt your stuff. You can share your folder with whomever should have access in a DR scenario. You can and should use this, not only for DR data, but also all your other important IT stuff such as license keys, ssh keys, and so on, which you want always with you and always available (even if there's an outage that prevents you from reaching the wiki or network filesystem in your datacenter) and you want always up-to-date.

On 11/25/19, 4:06 PM, "'Edward Harvey' via LOPSA Discussion" <dis...@lopsa.org> wrote:

Create a shared folder in https://sync.com (which you should be using for everything, instead of dropbox or whatever else you're actually using)

Use it to share all your team documentation, including network diagrams, license keys, and everything else.

--
This list provided by the League of Professional System Administrators
http://lopsa.org/
---
You received this message because you are subscribed to the Google Groups "LOPSA Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss+u...@lopsa.org.

To view this discussion on the web visit https://groups.google.com/a/lopsa.org/d/msgid/discuss/A0A24C07-FE02-4427-9620-29DE2C5460F6%40nedharvey.com.

[Jason Healy]

On Nov 26, 2019, at 11:19 AM, 'Edward Ned Harvey (lopser)' via LOPSA Discussion <dis...@lopsa.org> wrote:
>
> I want to repeat this because I didn't get any replies:
>
> Use sync.com.

Duly noted, and that makes sense for the availability of critical information. The issue of controlling access to the critical information remains, which goes back to my original question as it can't be a purely technical solution.

Thanks,

Jason

[Edward Ned Harvey (lopser)]

> From: Jason Healy <jhe...@logn.net>

>
> Duly noted, and that makes sense for the availability of critical information.
> The issue of controlling access to the critical information remains,
> which goes back to my original question as it can't be a purely technical solution.

As long as you got the suggestion and for whatever reason don't like it, I'm happy to move on, but based on your response, I'm not sure if we've had a miscommunication of expectations and capabilities. For clarification, I am imagining you create a shared folder with your CEO, CFO, COO, Director of IT, and that's probably good enough, but you could also add on the company lawyer, someone from the board of directors, or whoever else management deems appropriate. In the folder you have things like critical passwords and so on that a new IT team and management team would require in order to continue after a disaster. Most companies consider this level of protection to be adequate because if something wipes out all those people simultaneously, they just acknowledge that the risk is low enough and the damage would be high enough, that there's no way the company could survive such a terrible loss (of all its people). If you're trying to protect against the situation where every single one of your officers and managers and the company lawyer are all lost at the same time, I'll agree this is not sufficient. Very few companies could survive such a terrible loss, even if all their data were recovered, so very few companies try to protect against that situation.

[Jason Healy]

On Nov 26, 2019, at 1:25 PM, 'Edward Ned Harvey (lopser)' via LOPSA Discussion <dis...@lopsa.org> wrote:
>
> As long as you got the suggestion and for whatever reason don't like it, I'm happy to move on

Yes, suggestion received and it's a good one. I've just mentally separated the means (have a shared cloud drive with critical documents in it) from the process (who should I invite to have access? how do I distribute that keying information and keep it up to date?) so that's what I'm still working through. Per everyone's suggestions, I'm 90% of the way there, so just some details to iron out.

Thanks,

Jason

[Edward Ned Harvey (lopser)]

On 11/26/19, 3:04 PM, "Jason Healy" <jhe...@logn.net> wrote:

> Yes, suggestion received and it's a good one. I've just mentally separated the means
> (have a shared cloud drive with critical documents in it) from the process (who should
> I invite to have access? how do I distribute that keying information and keep it up to
> date?) so that's what I'm still working through. Per everyone's suggestions, I'm 90%
> of the way there, so just some details to iron out.

This response still leaves me feeling like you didn't get it - there is no keying information to distribute. You just sign in (make up your own password) and create a team share, and add the email addresses of the managers you want included. If they already use sync, they just click "Accept" to accept the shared folder. Otherwise they do as you did; sign up for their own account and make up their own password, and then Accept. Voila. You all have a shared directory and whatever you put in there, they all have access to it. The only part for you to decide is what humans to include. There's nothing to keep up to date, except you have to make a good practice of storing sensitive information in the shared folder, so whenever you change the root password, change it in the folder, and so on. Don't keep a separate spreadsheet or something outside the directory, which creates redundant information and risk of being out of sync. Just keep your stuff in the folder.

Each person may optionally install the sync client app. Although people can certainly use the web interface, the web interface is only the most convenient for rare, occasional usage. At least you, probably should install the app, for convenience of uploading / updating files, because you'll likely use it all the time. People who don't use it all the time probably won't install the app; just know that they could if they wanted to.

[Dan Ritter]

'Edward Ned Harvey (lopser)' via LOPSA Discussion wrote:
> On 11/26/19, 3:04 PM, "Jason Healy" <jhe...@logn.net> wrote:
>
> > Yes, suggestion received and it's a good one. I've just mentally separated the means
> > (have a shared cloud drive with critical documents in it) from the process (who should
> > I invite to have access? how do I distribute that keying information and keep it up to
> > date?) so that's what I'm still working through. Per everyone's suggestions, I'm 90%
> > of the way there, so just some details to iron out.
>
> This response still leaves me feeling like you didn't get it - there is no keying information to distribute. You just sign in (make up your own password) and create a team share, and add the email addresses of the managers you want included. If they already use sync, they just click "Accept" to accept the shared folder. Otherwise they do as you did; sign up for their own account and make up their own password, and then Accept. Voila. You all have a shared directory and whatever you put in there, they all have access to it. The only part for you to decide is what humans to include. There's nothing to keep up to date, except you have to make a good practice of storing sensitive information in the shared folder, so whenever you change the root password, change it in the folder, and so on. Don't keep a separate spreadsheet or something outside the directory, which creates redundant information and risk of being out of sync. Just keep your stuff in the folder.
>

Clearly there is key information happening, or else there is no
encryption. Perhaps you meant to say "sync.com keeps those
details completely hidden from you, and you have to trust
whatever they say and also that they will have a transition plan
for the glorious day on which their incredible journey is over"

https://ourincrediblejourney.tumblr.com/

-dsr-

[Edward Ned Harvey (lopser)]

From: Dan Ritter <d...@randomstring.org>

> Clearly there is key information happening, or else there is no encryption. Perhaps you meant to say "sync.com keeps those details completely hidden from you, and you have to trust whatever they say and also that they will have a transition plan for the glorious day on which their incredible journey is over"

I didn't say there's no encryption keys. I said there's no keying information to distribute, in response to Jason saying he still needed to figure out how to distribute keying information (which made it seem like he didn't get the idea, because there is no such thing).

To your second point about having to trust them, this is an unavoidable fact that applies to any product, even open source products. You can argue in favor of open source, or in favor of closed source. Both have vulnerabilities, pros and cons. It's not clear that one is actually better than the other. I know that during my days of working crypto, I reviewed many open source products, and every single one of them was full of glaring flaws and problems. You report them, and for the most part, they don't get fixed anyway. The best measure that I know of to determine whether or not I have faith in a crypto product is to read their architecture documentation, and see if they're using standard algorithms in standard ways.

During my days of working crypto for Concept Blossom, we were the real deal. We seriously put in real effort to making everything perfectly secure. We actually worked with the sync.com engineers for a period, as well as protonmail, and I was convinced that each of them is also the real deal. Spideroak is garbage; they are a farce. In the case of sync.com, they publish a whitepaper under https://www.sync.com/your-privacy/, they are using pbkdf2 to generate keys from password (which is good). From that key they generate RSA 2048 (which is good). This all happens client-side so they never have your password or private key (this is good). They don't expose a password verifier to unauthenticated clients (this is good). Long story short, I'm satisfied by their architecture.

To your third point, about sync.com coming to an end, if that happens, all your people still have the latest copies of information on their systems, so no information is lost; only the ability to keep syncing new changes comes to an end. At that point, you can search for a new sync solution. Also they're a profitable company, so it's an unlikely risk. (They were break-even several years ago, when we worked with them).