AWS US-EAST-1 service disruption

143 views
Skip to first unread message

Micheal Espinola

unread,
Oct 20, 2025, 11:00:02 AMOct 20
to ntsys...@googlegroups.com
If you haven't noticed yet or have been otherwise unaware, there has been an ongoing AWS disruption in US-EAST-1 for the past 6+ hours. It is disrupting Amazon DynamoD services, which impacts upwards of 110 different AWS services, including the CloudFront CDN.

---
Espi

Michael Leone

unread,
Oct 20, 2025, 11:18:00 AMOct 20
to ntsysadmin
I happened to get up at 4:30AM this morning, and noticed my Alexa wasn't responding to voice commands to turn on the lights (I have them on a smart plug). The specific app for the smart plug worked fine, but voice control by Alexa did not.  I went back to bed.

By the time I finally woke up as normal at 6AM, it was all back working. 


--
You received this message because you are subscribed to the Google Groups "ntsysadmin" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ntsysadmin+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/ntsysadmin/CAAfzEuySFm_pSCqLvr5E%3DvNZPGFy_J%3DvraC7E0uh%3DOpOqQ3dhg%40mail.gmail.com.

Micheal Espinola

unread,
Oct 20, 2025, 11:38:46 AMOct 20
to ntsys...@googlegroups.com
I noticed some chatter about it last night before I went to bed, and was surprised to see it still an issue when I woke up this morning. I've so far noticed issues on certain websites that currently still aren't accepting data submissions.

I was glad to see there was otherwise no clamoring about it here, and hopefully most people have not been effected. I figured I'd give a late heads-up for anyone out of the loop but hearing people complain about random issues with online services, etc.

To view this discussion visit https://groups.google.com/d/msgid/ntsysadmin/CA%2BZrOWyKKnhHwGMZJ-PtpbC-N3Srkv4eJm7%2B6S7TKVw%2BuQSw6g%40mail.gmail.com.


--

---
Espi

Erik Goldoff

unread,
Oct 20, 2025, 11:48:22 AMOct 20
to ntsys...@googlegroups.com
I think there might have been a lot of people affected... just one example:

Erik

To view this discussion visit https://groups.google.com/d/msgid/ntsysadmin/CAAfzEuyuO90TbenG_cp0AxmrSqZbkz8PXJLhtE7SMEuX4ysoCA%40mail.gmail.com.

Kurt Buff

unread,
Oct 20, 2025, 12:39:48 PMOct 20
to ntsys...@googlegroups.com
It's impacting our web site very seriously - images not loading, pages not responding, and we're not hosted by them, though our images are coming from an S3 bucket.

Kurt

--

Rick McClure

unread,
Oct 20, 2025, 12:43:14 PMOct 20
to ntsys...@googlegroups.com

This has also impacted my Oncologist site and even though it was working fine 2-3 hrs ago, now it’s not working at all.

 

I thought there was a benefit using the cloud?

 

Rick.

Gabriel Clifton

unread,
Oct 20, 2025, 12:45:39 PMOct 20
to ntsysadmin
We are seeing several sites impacted. Did the AWS team take notes from the Crowdstrike team on how to take everyone down?
It has gone down, up, degraded, down, waiting for a full up.

Henry Awad

unread,
Oct 20, 2025, 1:38:40 PMOct 20
to ntsys...@googlegroups.com
I saw on the news this morning the widespread impact that this outage has caused. T-Mobile, AT&T, are among the many companies impacted as well as several Amazon services like Ring and Alexa. What was interesting is that they haven't disclosed the reason for the outage but I'm sure they'll eventually disclose the root cause once things settle down. But it's really concerning how many of these large infrastructure companies can have such a major impact every time they have an outage. 

I wonder if this outage was caused by an engineering error or a major breach.


To view this discussion visit https://groups.google.com/d/msgid/ntsysadmin/febb68f1-e567-4182-90ee-81d855f79492n%40googlegroups.com.

Mike Leone

unread,
Oct 20, 2025, 2:02:32 PMOct 20
to ntsys...@googlegroups.com
Here's ZDNet is saying:

Amazon's cloud database technology. Engineers later identified a Domain Name System (DNS) resolution problem affecting the DynamoDB API endpoint, which cascaded across dependent systems.


To view this discussion visit https://groups.google.com/d/msgid/ntsysadmin/CAGaCHK58BYt8%2BTowcYkmqxE_AOc-sfyVZ-_Y1OHaZ0La2F00xg%40mail.gmail.com.


--

Mike. Leone, <mailto:tur...@mike-leone.com>

PGP Fingerprint: 0AA8 DC47 CB63 AE3F C739 6BF9 9AB4 1EF6 5AA5 BCDF
Photo Gallery: <http://www.flickr.com/photos/mikeleonephotos>

Lee Wilbur

unread,
Oct 20, 2025, 2:04:00 PMOct 20
to ntsys...@googlegroups.com

Enough of these and maybe people will realize that the cloud isn’t the end-all-be-all of computing existence.

 

 

Lee Wilbur

A+, MCSA, MCTS, MS-MVP 2006-2018

Multiverse Enterprises Inc

Providing Technical Support for IT Professionals and SMBs

In the Long Island and New York City areas

516.901.2558   www.multiverseit.com

 

Please note: Although we may sometimes respond to email, text, and phone calls instantly at all hours of the day, our regular business hours are 9:00 AM - 6:00 PM, Monday thru Friday.

 

 

Twitter: MultiverseIT / ThatWindowsGeek

Facebook: https://www.facebook.com/pages/Multiverse-Enterprises/102713849814574

LinkedIn: http://www.linkedin.com/in/leewilbur 

Wright, John M

unread,
Oct 20, 2025, 2:05:37 PMOct 20
to ntsys...@googlegroups.com

Yes, we’re affected by it.  Some emails are going through, others not, and some come through but with a weirdly long lag between when they show up in message trace and when they hit Outlook.  Your email to this list showed in my inbox (per message trace) 10-15 minutes before it appeared in my Outlook client, despite hitting F9 or restarting the app entirely.

 

--

John Wright

IT Support Specialist

1800 Old Bluegrass Avenue, Louisville, KY 40215

502.708.9953

Please submit IT requests to Hazelwoo...@bluegrass.org

24 Hour Helpline 1.800.928.8000

  

CONFIDENTIALITY NOTICE: This message contains confidential information and is intended only for the individual(s) addressed in the message. If you are not the named addressee, you should not disseminate, distribute, or copy this e-mail. If you are not the intended recipient, you are notified that disclosing, distributing, or copying this e-mail is strictly prohibited.

--

Philip Elder

unread,
Oct 20, 2025, 2:14:09 PMOct 20
to ntsys...@googlegroups.com

The Cloud First/Cloud Native push started in 2008 with BPOS.

 

The horrible service in the public cloud was “justified” because “we’re just setting this up so there’s be a few issues to begin with”.

 

Heh … it’s 2025 and those issues have only gotten worse.

 

Me thinks the AI Revolution isn’t what the marketing folks want us to believe it means. :P

 

Philip Elder MCTS

Senior Technical Architect

Microsoft High Availability MVP

MPECS Inc.

E-mail: Phili...@mpecsinc.ca

Phone: +1 (780) 458-2028

Web: www.mpecsinc.com

Blog: blog.mpecsinc.com

Twitter: Twitter.com/MPECSInc

Teams: Phili...@MPECSInc.Cloud

 

Please note: Although we may sometimes respond to email, text and phone calls instantly at all hours of the day, our regular business hours are 8:00 AM - 5:00 PM, Monday thru Friday.

Michael B. Smith

unread,
Oct 20, 2025, 2:18:36 PMOct 20
to ntsys...@googlegroups.com

Before then.

 

We called it “ASP” (Application Service Providers) back in the day. 😊

 

I offered hosted Exchange with an admin console based on Exchange 2003. We also offered hosted web sites (IIS and Linux) and hosted VPS’ (virtual private servers). And of course, we would co-lo servers all day long. Great profit margin on that.

Philip Elder

unread,
Oct 20, 2025, 2:24:01 PMOct 20
to ntsys...@googlegroups.com

We started into hosting with SPLA, IIRC, using Virtual Server 2005 R2 Enterprise Edition. It may have been an earlier version.

 

That’s way back and the grey matter has had a continual snowstorm. 😉

 

And yes, hosting was, and still is very profitable when done right.

 

Philip Elder MCTS

Senior Technical Architect

Microsoft High Availability MVP

MPECS Inc.

E-mail: Phili...@mpecsinc.ca

Phone: +1 (780) 458-2028

Web: www.mpecsinc.com

Blog: blog.mpecsinc.com

Twitter: Twitter.com/MPECSInc

Teams: Phili...@MPECSInc.Cloud

 

Please note: Although we may sometimes respond to email, text and phone calls instantly at all hours of the day, our regular business hours are 8:00 AM - 5:00 PM, Monday thru Friday.

 

Kurt Buff

unread,
Oct 20, 2025, 2:29:15 PMOct 20
to ntsys...@googlegroups.com
It's killing our email, inbound and outbound - Check Point HEC is apparently highly depending on AWS.

I'll bet other mail shielding services are as well.

Kurt

CR Hiestand

unread,
Oct 20, 2025, 2:33:06 PMOct 20
to ntsys...@googlegroups.com
I'm surprised to hear this. We also use Check Point and haven't seen any impacts from the AWS outage. 

To view this discussion visit https://groups.google.com/d/msgid/ntsysadmin/CADy1Ce5HRzbbBFZe-%3DZJBPbATYHSm9%2BD%3Dp-ZLd1SPe%3De2BUUUQ%40mail.gmail.com.

Kurt Buff

unread,
Oct 20, 2025, 2:37:42 PMOct 20
to ntsys...@googlegroups.com
Very interesting.

We contacted CP support, and they're saying they have lots of customers affected.

Aren't you the lucky one!

Kurt

To view this discussion visit https://groups.google.com/d/msgid/ntsysadmin/CA%2BDi7PtW5nFBJRP3cCWV_%2BKJds9aW49rYcp2Yoe3-qe%3D3H0OmQ%40mail.gmail.com.

Michael B. Smith

unread,
Oct 20, 2025, 2:54:02 PMOct 20
to ntsys...@googlegroups.com

Not having any Proofpoint issues.

 

I’ve also got lots of clients on Mimecast and no one has complained to me (but that doesn’t mean it’s up).

 


Sent: Monday, October 20, 2025 2:29 PM
To: ntsys...@googlegroups.com

Wright, John M

unread,
Oct 20, 2025, 2:58:46 PMOct 20
to ntsys...@googlegroups.com

Evidently it’s not resolved yet.  My message below took nearly an hour from the time I sent it to appear in this list.

 

--

John Wright

IT Support Specialist

1800 Old Bluegrass Avenue, Louisville, KY 40215

502.708.9953

Please submit IT requests to Hazelwoo...@bluegrass.org

24 Hour Helpline 1.800.928.8000

  

CONFIDENTIALITY NOTICE: This message contains confidential information and is intended only for the individual(s) addressed in the message. If you are not the named addressee, you should not disseminate, distribute, or copy this e-mail. If you are not the intended recipient, you are notified that disclosing, distributing, or copying this e-mail is strictly prohibited.

 

From: ntsys...@googlegroups.com <ntsys...@googlegroups.com> On Behalf Of Wright, John M


Sent: Monday, October 20, 2025 2:05 PM
To: ntsys...@googlegroups.com

Rick McClure

unread,
Oct 20, 2025, 3:24:38 PMOct 20
to ntsys...@googlegroups.com

I heard on CNN that it was resolved but the sites I need to chat with are not up….yet.  Maybe later today as buffers empty.

 

Rick

Philip Elder

unread,
Oct 20, 2025, 3:31:25 PMOct 20
to ntsys...@googlegroups.com

We’re not having any issues nor clients complaining here either. Proofpoint

 

Philip Elder MCTS

Senior Technical Architect

Microsoft High Availability MVP

MPECS Inc.

E-mail: Phili...@mpecsinc.ca

Phone: +1 (780) 458-2028

Web: www.mpecsinc.com

Blog: blog.mpecsinc.com

Twitter: Twitter.com/MPECSInc

Teams: Phili...@MPECSInc.Cloud

 

Please note: Although we may sometimes respond to email, text and phone calls instantly at all hours of the day, our regular business hours are 8:00 AM - 5:00 PM, Monday thru Friday.

 

Erik Goldoff

unread,
Oct 20, 2025, 5:14:09 PMOct 20
to ntsys...@googlegroups.com
image.png

Rick McClure

unread,
Oct 20, 2025, 5:30:51 PMOct 20
to ntsys...@googlegroups.com

And that was in 2021….redundancy…?  Who needs stinking redundancy….?....it’s the internet….it’ll never go down.  Right after we update it to 256meg of RAM.  And what year was that?

Wright, John M

unread,
Oct 22, 2025, 8:07:38 AMOct 22
to ntsys...@googlegroups.com

They’re saying it was triggered by DNS issues:  AWS outage cause: It’s always DNS, but sometimes it’s...

 

--

John Wright

IT Support Specialist

1800 Old Bluegrass Avenue, Louisville, KY 40215

502.708.9953

Please submit IT requests to Hazelwoo...@bluegrass.org

24 Hour Helpline 1.800.928.8000

  

CONFIDENTIALITY NOTICE: This message contains confidential information and is intended only for the individual(s) addressed in the message. If you are not the named addressee, you should not disseminate, distribute, or copy this e-mail. If you are not the intended recipient, you are notified that disclosing, distributing, or copying this e-mail is strictly prohibited.

 

From: ntsys...@googlegroups.com <ntsys...@googlegroups.com> On Behalf Of Henry Awad
Sent: Monday, October 20, 2025 1:38 PM
To: ntsys...@googlegroups.com
Subject: Re: [ntsysadmin] AWS US-EAST-1 service disruption

 

EXTERNAL EMAIL - This email was sent by a person from outside your organization. Exercise caution when clicking links, opening attachments or taking further action, before validating its authenticity.

Secured by Check Point

Michael B. Smith

unread,
Oct 22, 2025, 8:44:35 AMOct 22
to ntsys...@googlegroups.com

“Someone else’s” computer can crash just like your on-prem computer.

 

But since thousands/millions/billions(?) depend upon it, it’s a lot worse.

Philip Elder

unread,
Oct 22, 2025, 1:23:00 PMOct 22
to ntsys...@googlegroups.com

I remember our experiment in using DHCP failover along with the multiple DCs with DNS that had static IPs assigned.

 

It turned out that the DHCP service didn’t behave they way I expected it do when the service locked up on one of the partners. The other would just sit there doe eyed and we’d lose our clusters.

 

After troubleshooting things we ditched DHCP and went back to static IPs for everything and nary an issue since.

 

Philip Elder MCTS

Senior Technical Architect

Microsoft High Availability MVP

MPECS Inc.

E-mail: Phili...@mpecsinc.ca

Phone: +1 (780) 458-2028

Web: www.mpecsinc.com

Blog: blog.mpecsinc.com

Twitter: Twitter.com/MPECSInc

Teams: Phili...@MPECSInc.Cloud

 

Please note: Although we may sometimes respond to email, text and phone calls instantly at all hours of the day, our regular business hours are 8:00 AM - 5:00 PM, Monday thru Friday.

 

Dave Lum

unread,
Oct 22, 2025, 2:45:46 PMOct 22
to ntsys...@googlegroups.com

“Sorry we deleted the HOSTS file everybody”

 

From: ntsys...@googlegroups.com <ntsys...@googlegroups.com> On Behalf Of Wright, John M


Sent: Wednesday, October 22, 2025 5:08 AM
To: ntsys...@googlegroups.com

Subject: RE: [ntsysadmin] AWS US-EAST-1 service disruption

 

CAUTION: This email originated from outside of OCHIN’s network

Do not click links or open attachments unless you recognize the sender and know the content is safe. If you suspect this email is phishing or a scam, use the report button in the Outlook toolbar to report it to Desktop Support.

 

Attention: Information contained in this message and or attachments is intended only for the recipient(s) named above and may contain confidential and or privileged material that is protected under State or Federal law. If you are not the intended recipient, any disclosure, copying, distribution or action taken on it is prohibited. If you believe you have received this email in error, please contact the sender with a copy to compl...@ochin.org, delete this email and destroy all copies.

Philip Elder

unread,
Oct 22, 2025, 3:17:45 PMOct 22
to ntsys...@googlegroups.com

Holy smokes that brought out some PTSD! :D

 

Philip Elder MCTS

Senior Technical Architect

Microsoft High Availability MVP

MPECS Inc.

E-mail: Phili...@mpecsinc.ca

Phone: +1 (780) 458-2028

Web: www.mpecsinc.com

Blog: blog.mpecsinc.com

Twitter: Twitter.com/MPECSInc

Teams: Phili...@MPECSInc.Cloud

 

Please note: Although we may sometimes respond to email, text and phone calls instantly at all hours of the day, our regular business hours are 8:00 AM - 5:00 PM, Monday thru Friday.

 

Henry Awad

unread,
Oct 22, 2025, 3:25:47 PMOct 22
to ntsys...@googlegroups.com
Lol, we all have battle scars dealing with DNS. I was working on a case with Microsoft support to fix DNS issues between forest and domain and as I was providing an update to one of the directors the Microsoft support engineer proceeded to wipe the domain DNS zone and forced propagation taking the entire company down. Luckily, I had exported the DNS zone before I made the call. Still, I had to remotely login to one DC in each site and copy back the DNS zone since the DCs couldn't replicate without DNS. 



To view this discussion visit https://groups.google.com/d/msgid/ntsysadmin/a451eb1384a24f70aa000df73d3778de%40MPECSInc.Ca.

Michael B. Smith

unread,
Oct 22, 2025, 3:34:06 PMOct 22
to ntsys...@googlegroups.com

I’ve been around so long that I remember when the “all Internet hosts file” was distributed daily by UUCP.

 

“rm /etc/hosts” could cause many many screams.

Philip Elder

unread,
Oct 22, 2025, 3:41:18 PMOct 22
to ntsys...@googlegroups.com

Woah, really?

The HOSTS file was DNS?

 

Yeah, I could see that being a single point of failure but don’t tell Amazon that!

 

Philip Elder MCTS

Senior Technical Architect

Microsoft High Availability MVP

MPECS Inc.

E-mail: Phili...@mpecsinc.ca

Phone: +1 (780) 458-2028

Web: www.mpecsinc.com

Blog: blog.mpecsinc.com

Twitter: Twitter.com/MPECSInc

Teams: Phili...@MPECSInc.Cloud

 

Please note: Although we may sometimes respond to email, text and phone calls instantly at all hours of the day, our regular business hours are 8:00 AM - 5:00 PM, Monday thru Friday.

 

Michael B. Smith

unread,
Oct 22, 2025, 4:13:50 PMOct 22
to ntsys...@googlegroups.com

Yeah, DNS wasn’t released until 1983 and it took a while for it to be widespread. And back then the Internet was a whole bunch of networks that weren’t all interconnected very well or very often.

Rick McClure

unread,
Oct 22, 2025, 4:16:57 PMOct 22
to ntsys...@googlegroups.com

I thought the hosts file was needed (as a last resort) for the machines to operate and that the file was always “in use” and couldn’t easily be removed?

 

But what do I know….

 

Rick.

Michael B. Smith

unread,
Oct 22, 2025, 4:24:49 PMOct 22
to ntsys...@googlegroups.com

In my experience, that was true later.

 

At least in the early 80’s the hosts file was read by a statically linked “resolv.lib” for each program that needed IP resolution.

 

When shared libraries came around and the hosts file was cached – what you say was absolutely true. And resolv.lib gained DNS capabilities and the hosts file was accessed only as a last resort.

Kurt Buff

unread,
Oct 22, 2025, 4:49:55 PMOct 22
to ntsys...@googlegroups.com
The hosts file is the FIRST resort for resolving names, and depending on WINS/NBNS configuration on the machine might consult the LMHOSTS file, but administrators on the machine can easily delete either one. If the name isn't found in the hosts file, then it goes to DNS.

image.png

Kurt

Henry Awad

unread,
Oct 22, 2025, 5:04:43 PMOct 22
to ntsys...@googlegroups.com
And the hosts/lmhosts file created its own problems when people messed around with it for testing and forgot to remove the additional entries. That was a lot of fun to troubleshoot several months later when they forgot about it.



Dave Lum

unread,
Oct 23, 2025, 11:29:25 AMOct 23
to ntsys...@googlegroups.com

Yikes! I can imagine!   Modern technology allows the same interruption but in new, more novel ways! I took a network routing class waaay back when and it mainly taught me to be amazed the Internet even works.

A dizzying number of ways to kill propagation.

 

Dave

 

From: ntsys...@googlegroups.com <ntsys...@googlegroups.com> On Behalf Of Michael B. Smith
Sent: Wednesday, October 22, 2025 12:34 PM
To: ntsys...@googlegroups.com
Subject: RE: [ntsysadmin] AWS US-EAST-1 service disruption

 

CAUTION: This email originated from outside of OCHIN’s network

Mike Leone

unread,
Oct 23, 2025, 11:33:24 AMOct 23
to ntsys...@googlegroups.com
I wasn't aware that the /\windows32\system\drivers\etc\hosts was checked *FIRST*, and *THEN* DNS if the entry is not found. I always thought it was the other way around. I know you can change the order of resolution in Linux ...




--

Mike. Leone, <mailto:tur...@mike-leone.com>

PGP Fingerprint: 0AA8 DC47 CB63 AE3F C739 6BF9 9AB4 1EF6 5AA5 BCDF
Photo Gallery: <http://www.flickr.com/photos/mikeleonephotos>

Erik Goldoff

unread,
Oct 23, 2025, 11:39:06 AMOct 23
to ntsys...@googlegroups.com
a once popular technique to control a system, much easier than direct DNS cache poisoning, also a good technique for deception baiting and redirect away from certain sites for specific computers (like kiosks)

Erik

To view this discussion visit https://groups.google.com/d/msgid/ntsysadmin/CAHBr%2B%2BiLe_Fo8sxe8BhVP0%2BFFSNHABakmroDmjh75Ydte0knKg%40mail.gmail.com.

Kurt Buff

unread,
Oct 23, 2025, 11:50:41 AMOct 23
to ntsys...@googlegroups.com
Yep - it's the reason why malware often changes the host file. Well, perhaps not as often now, but in earlier versions of Windows it was a very popular technique to poison the hosts file.

Kurt

To view this discussion visit https://groups.google.com/d/msgid/ntsysadmin/CAHBr%2B%2BiLe_Fo8sxe8BhVP0%2BFFSNHABakmroDmjh75Ydte0knKg%40mail.gmail.com.

Erik Goldoff

unread,
Oct 23, 2025, 12:08:36 PMOct 23
to ntsys...@googlegroups.com
going back at least 20 years or more, I often used FIMs to monitor for any attempts to change the HOSTS file, just need to be aware that many VPN clients legitimately also dynamically change the hosts file when connecting, and breaking the tunnel.

Erik

To view this discussion visit https://groups.google.com/d/msgid/ntsysadmin/CADy1Ce63qEspu7BPXO%2BTTrhqia%2B40Ct3EVOziPkqHcT3BZn15w%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages