In the midst of a disaster recovery

[Philip Elder]

I’m trying to diarize as I go along: https://x.com/MPECSInc/status/1961645439092142245

 

VMs were on a Storage Spaces Direct cluster.

Veeam backing up to local SOBR and Cloud Tier.

Backups are all encrypted.

Restore is being run to one of our stand-by servers.

Internet will be routed to their environment.

 

They’ll be productive today though most won’t be available until Tuesday.

 

Lots of tips in the thread for all y’all and more to come.

 

Happy long weekend! 😊

 

Philip Elder MCTS

Senior Technical Architect

Microsoft High Availability MVP

MPECS Inc.

E-mail: Phili...@mpecsinc.ca

Phone: +1 (780) 458-2028

Web: www.mpecsinc.com

Blog: blog.mpecsinc.com

Twitter: Twitter.com/MPECSInc

Teams: Phili...@MPECSInc.Cloud

 

Please note: Although we may sometimes respond to email, text and phone calls instantly at all hours of the day, our regular business hours are 8:00 AM - 5:00 PM, Monday thru Friday.

 

[Severino Juan Miguel]

Sh... good luck with your DR.

 

We do DR tests yearly (one time we even deenergized the HQ for 6 hours) and we do a full recovery test every 12-15 months too.

 

In the full recovery test, we simulate a total loss, may be because of fire like your case or in case of a ransoware attack and the servers got seized for investigation:

- Only air-gapped backups

- Only the documentation in the air-gapped backups

- Encryption keys (we keep them separated).

- Brand new laptops, "new" server (server with no data/configuration).

 

And then you discover that the AD tombstone lifetime is screwing with your DC replication, the EDR solution is panicking because the signature cache is expired and bluescreening servers at boot, how to renew certificates and rejoin ADFS application proxies, rebuild Citrix images, etc.

 

That's recovering the infrastructure. That's the really easy part.

 

A different beast is to recover the applications: Ensure you don't issue dupplicate serial numbers, invoice numbers, personal numbers, patient IDs (hospitals), don't transfer money twice (pay the payroll/invoices twice, etc.) plus missing references: the document Nr referenced in system A does in fact exist in the recovered system B and it's the correct one instead of a new one, etc.

 

So, yeah. It is difficult and in most companies, never done. Tha't why many companies take months to recover after a ransomware attack, even if they payed.

Best luck with your recovery.

 

Von: ntsys...@googlegroups.com <ntsys...@googlegroups.com> Im Auftrag von Philip Elder
Gesendet: Samstag, 30. August 2025 17:41
An: ntsys...@googlegroups.com
Betreff: [ntsysadmin] In the midst of a disaster recovery

 

-!-!-!- EXTERNAL EMAIL -!-!-!- This email originated from outside of SKAN. Do not click links or open attachments unless you recognize the sender and know the content is safe

--
You received this message because you are subscribed to the Google Groups "ntsysadmin" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ntsysadmin+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/ntsysadmin/cf4d9daf8d8e42bb88c13fb6b104eda8%40MPECSInc.Ca.

The content of this message is confidential and shall be used solely for the intended purpose and by the intended recipient. If you received this email by mistake, please inform us immediately and delete this message without disclosing its content to any other person. Any viewing, copying or distribution of, or reliance on this message by unintended recipients is strictly prohibited. The integrity and security of this email cannot be guaranteed over the internet. The sender shall not be held liable for any damage caused by this message.

[Philip Elder]

Thank you!

 

I’m very impressed with the company’s awareness to be prepared! Well done! 😊

 

All of our clients are on our backup and recovery services where we do full test recoveries of their environment to verify.

 

We’re past all of the recovery steps now, their environment is fully alive on our DRaaS systems, and we’ve re-established a backup regimen for the recovered servers.

 

Now it’s the little things to work through on the app side as you mention.

 

They’ll be live for tomorrow AM as requested.

 

Leadership is already working.

 

Mail was re-established about 40 hours in with Proofpoint shuffling their cached e-mails over after that. They have their Emergency Inbox on Proofpoint for those that needed it.

 

All in all, this was a yuge win. :0)

 

Philip Elder MCTS

Senior Technical Architect

Microsoft High Availability MVP

MPECS Inc.

E-mail: Phili...@mpecsinc.ca

Phone: +1 (780) 458-2028

Web: www.mpecsinc.com

Blog: blog.mpecsinc.com

Twitter: Twitter.com/MPECSInc

Teams: Phili...@MPECSInc.Cloud

 

Please note: Although we may sometimes respond to email, text and phone calls instantly at all hours of the day, our regular business hours are 8:00 AM - 5:00 PM, Monday thru Friday.

 

To view this discussion visit https://groups.google.com/d/msgid/ntsysadmin/DBBP189MB1340CD2AD6D7B5255F4768658307A%40DBBP189MB1340.EURP189.PROD.OUTLOOK.COM.

[Henry Awad]

Well done! I haven't been involved in DR/BC drills in a few years. I remember working on my first DR system for an investment company in 2000-2001 where management was trying to do the bare minimum before September 11 events and suddenly I had to build a fully functional DR/BC systems. My company was the second Exchange 2000 server implementation for the replication software that we used at the time and I ended up writing the documentation for the vendor because theirs was useless. We also ran quarterly drills for a few systems and one yearly full test where we would shutdown the entire main site. We always ran into issues during the yearly test because ultimately there were a few applications that were not installed or updated at the DR site. So we would update our documentation and make the necessary adjustments until we had very few minor issues. 

I am working on using Veeam replication for our DR/BC systems and hopefully that should make life a lot easier. Unfortunately, we have very limited funds and staff to work on this critical project. But I'm hoping to have it completed by this time next year along with updated hardened Veeam repositories onsite (we already have cloud based ones). I'll try to document the process and maybe share it with others once it's done.

Henry Awad

Principal Engineer

Technology Services

The Catholic University of America

(D) 202-319-5404 | (M) 571-233-4479

To view this discussion visit https://groups.google.com/d/msgid/ntsysadmin/bb9622dfb88c4418b2933efd7efa4aff%40MPECSInc.Ca.

[Severino Juan Miguel]

Congratulations!

 

Great preparation and great team. Without people and management support, it won’t be possible.

 

Veeam is amazing too.

 

Best regards and don’t forget to get some sleep 😉

To view this discussion visit https://groups.google.com/d/msgid/ntsysadmin/bb9622dfb88c4418b2933efd7efa4aff%40MPECSInc.Ca.