Nuke Teardown

[Victorino Eagle]

Ihave an AWS account where multiple EC2 instances, load balancers, target groups, security groups etc are setup by multiple owners. We use terraform to set this up but sometimes due to corruption, the state becomes inconsistent. Current mechanism to recover is to manually destroy all resources in that account owned by a particular owner.Is there an easy way to nuke all resources in an AWS account belonging to a particular owner?

But I see the original issue is that the terraform state is corrupted. You can checkout the terraform import feature which lets you generate state file from aws resources. In that way you can connect your config to resources again.

Of course, you could script it, fairly trivially, by wrapping calls to aws-cli to custom code to iterate through the resources and generate additional requests to destroy them... but if you do, lock that code away, since such capability is inherently dangerous.

Run the following command to execute the nuke script. The scripts will ask you to enter the account alias name and then hit enter. It will list down the resources from the specified account, what can and cannot be deleted. It will take a while based on resources available in the target account.

Deleting a large number of stacks manually is a time-consuming task. If the stack we are trying to delete has some dependencies, then we need to delete the dependent stacks first. We can't imagine doing this for an environment/platform consisting large set of stacks.

In my company, we have been following a process of creating ephemeral dev/testing environments. A single environment has more than 500 stacks and the number keeps growing daily. The requirement was to clean up the non-production environment and experimental stuff from the AWS account periodically. Also, we needed to filter the core infrastructural components during the clean-up so that the base AWS setup would be intact and we could just spin a new environment on top of it.

First, we tried AWS-NUKE library which was meant for the whole AWS account, not just CloudFormation, and would cover all regions. It served our purpose for some time. Sometime later, we introduced custom cloudformation resources that were linked across different AWS accounts and had to be deleted gracefully. So, the brute force strategy used by this library did not work.

First, we prepared a dependency tree of all stacks which gave us more context on how to approach the teardown. We found that many stacks had tight coupling and during deletion, we needed to respect such dependency for determining the deletion order. We went for a hybrid solution where we created a custom script to teardown our non-production environment stacks first and then used AWS-NUKE for deleting residual and experimental stuff in the account.

Of course, despite your best efforts, occasionally cleanup will fail, perhaps due to the CI server going down, or a bugin your code, or a temporary network outage. To handle those cases, we run a tool calledcloud-nuke in our test AWS account on a nightly basis to clean up anyleftover resources.

how about the mind boggling insanity in normalising and cutsifying talking about weapons of mass destruction

maybe next time work in something with nukes

and 3d printed kittens

ya!,nothing smooths over the end of civilisation

like some lucky kitty charms

Additionally munitions might have to be disposed of quickly, including foreign munitions. We have teams around the country ready to go at a moments notice. Disposal is even in their name (EOD). So whilst old munitions might not have thought about disposal at design, modern ones do.

Bob, the elderly armed guard at Humboldt Bay Nuclear Power Plant's reactor building, sealed the oval air-lock hatch with a hard twist of both shaky hands on the big wheel. Bob was outside. I was inside with a utility public relations flack. The top of the sunken reactor itself was on the floor, with a cap on it about 20 feet wide. Next to the reactor, the wildly toxic spent fuel pool shimmered.

The public relations guy kept talking about how benign the whole set up was. He declared that much of the radioactivity had cooled down. You could practically swim in the spent fuel pool, he said, and if all that water suddenly drained, there would be no chain reaction. No fire. No meltdown. It was all perfectly safe, he said. I began to get nervous. I started slowly back up the metal stairs to where the submarine portal beckoned. I mentioned to the public relations guy that I really, really do have claustrophobia. He ignored me. He kept talking. I inched my way up the stairs. I pleaded, "I'm claustrophobic. Can't we get out of here?" (I am, in fact, claustrophobic, but what I am more is radiation-phobic.) The PR guy kept talking. He had me locked in and wasn't about to let me out until he was damn well ready. I started whimpering. He kept talking. After another 10 minutes, the PR guy called Bob to unscrew the portal and let us out. With clammy hands, I signed out and stood in a phone booth-sized box that was supposed to read out my radioactive exposure. I was trembling.

The PR guy kept talking as we headed to the main office. There, I signed some government paperwork and handed over my radiation-detection badge. Six months later I got a letter from the federal government reporting I had not been unduly exposed.

That was two decades ago but the memory is vivid. It was like being locked inside a submarine. Exactly like being locked inside a submarine, actually, because the power plant was built like an old sub, vertically inserted at the edge of the bay at King Salmon.

I was in that claustrophobic steel sausage to report on the progress of decommissioning the nuke for another publication. The sideways sub, along with the cooling tower that used to dominate the waterfront at the south end of Eureka, and that leaking spent fuel pool, are now demolished, and the old nuclear plant finally made new news. PG&E officially declared an end to its decades-long plant clean up Oct. 22, when it filed for a license transfer with the Nuclear Regulatory Commission.

PG&E deemed the area ready to pass federal muster, yet the radioactive detritus accumulated over its 13 years of operation in the middle of the last century are still here, stuffed in concrete casks. The site will pose an ongoing threat to Humboldt Bay. It will likely never be clean.

Some other decommissioned nukes, like Rancho Seco in Sacramento and Trojan in Oregon, advertise their post clean-up use as "family fun" for visitors. Rancho Seco, for instance, offers camping. But Humboldt's will remain closed to the public for the foreseeable future. "The desired state" for the former reactor site is for "post-industrial residential farming use," PG&E spokesperson Carina Corral said in an email to the Journal, but that would be after "meeting an extensive set of standards."

The Humboldt Bay Nuclear Power Plant was shut down in 1976 because, suddenly, 13 years after it came online, seismologists found it was built practically on top of the Little Salmon earthquake fault. Its design was not nearly strong enough to be retrofitted against potential shaking and if an earthquake broke open any critical part of the plant, the results could have been catastrophic.

Why be concerned about nuclear waste? Because it's one of the most toxic substances on earth. On a basic physiological level, when radioactivity is absorbed by a body, the radioactivity becomes part of living tissue. It can be breathed in, swallowed or assimilated just by exposing some skin. Scientists report that once inside the body, different types of radioactive elements that appear as a result of nuclear fission behave like calcium in bones, potassium in muscles. There, they emit alpha, beta and gamma particles. That, in turn, ionizes molecules and starts breaking up chemical bonds, damaging a body's cells. It wreaks havoc, particularly, on DNA molecules and studies show it leads to increase risk of cancers, such as leukemia.

Like a genie bottle, all those horrid, toxic, contaminated pieces are now sealed up tight in storage at the edge of Humboldt Bay. The idea was those casks wouldn't be opened until, with the wave of a newer magic wand, the federal government transports them to a permanent waste disposal site where the radioactive genie will be stored in a bigger, more powerfully built bottle until a long, long time from now, when the contamination will be degraded into nothing but bottle rust. That's the theory.

The casks now holding the irradiated parts of Humboldt's old nuke are "heavy, cylindrical, multilayered steel vessel[s]," according to their manufacturer, Holtec, a New Jersey company that designs and manufactures both parts for nuclear reactors and the equipment used to decommission them. These vessels serve as both "a missile barrier" for terrorist targets and a "radiation shield" to keep the toxics inside, according to the company.

The casks are made up of three shells. An inner shell for containment, a series of thick steel intermediate shells for gamma shielding and an outer enclosure shell that houses the neutron shielding material, according to Holtec. The casks, in turn, are tucked inside a vault made of reinforced concrete between 3-feet and 7-feet thick. Holtec contends they are not only safe, but scenic. According to the company, "Humboldt Bay became the first plant to feature subterranean storage, which is so unobtrusive and produces such a negligible [radiation] dose that the path to the beach runs in close proximity to the [storage site]!"

The casks are meant to "temporarily" hold the waste for 100 years or so. If the scientists who predict a 240,000-year toxicity are correct, that would leave 239,900 years, and for that, there's no plan. Yet, some local environmentalists are laying the groundwork to plan for site protection at least into the next century. They foresee a drowning bluff and an unsuspecting future generation.

3a8082e126