Cerner outage hit the DOD and Coast Guard also

48 views
Skip to first unread message

Nancy Anthracite

unread,
Apr 11, 2022, 3:37:10 PM4/11/22
to hard...@googlegroups.com

David Whitten

unread,
Apr 11, 2022, 4:03:54 PM4/11/22
to Hard Hats Mailing List
Thank you Nancy. 

VA looking into ‘root cause’ of EHR outage that also hit DoD, Coast Guard

Jory Heckman@jheckmanWFED
April 8, 2022 8:04 am
5 min read
      

The Veterans Affairs Department said the second go-live of its Electronic Health Record at a VA medical center in Walla Walla, Washington went smoothly, building on the lessons learned from a rocky first launch in Spokane.

The VA, however, said it’s investigating a broader outage Wednesday that brought down all federal Cerner systems across the Defense Department, VA and the Coast Guard.

Terry Adirim, the executive director of the EHR Modernization Integration Office, told reporters in a call Thursday that the cause was a bug in an Oracle server, but said an investigation is still ongoing.

“Speaking as a health care provider, you don’t want disruption to your work, and I would not want them to expect that this is going to happen routinely,” Adirim said regarding VA staff.

Deputy VA Secretary Donald Remy said the agency is working closely with Cerner to diagnose the problem.

“We are in close contact with Cerner, and we are exercising all of our authority to make sure that we can position ourselves for success and assure that when these types of things happen, we get to the root cause. We conduct a root cause analysis, we identify how the problem arose, and we make sure that we don’t see the same thing happen in the future,” Remy said.

The VA, meanwhile, is dealing with a second technical glitch, this one localized to the Mann-Grandstaff VA Medical Center in Spokane, the site of the first EHR go-live.

Adirim said the glitch in Spokane happened last month, and was caused by a defect introduced into an interface between systems that provides demographic information to Cerner.

“That defect was fixed in short order. However, [it] went through some testing to be 100% certain that it was safe to turn the system back on,” Adirim said.

Adirim said that during the outage, staff at Mann-Grandstaff went through “downtime procedures” that are standard whenever the facility experiences electrical outages.

“Anytime there’s an electrical outage or anything like that, you go to these procedures. You still provide care, and during that time, that facility provided clinical operations, most of the clinical operations. They had to use pen and paper to document patient encounters,” Adirim said.

These outages are just the latest in a series of complications the VA has encountered since the start of its EHR rollout.

The VA launched a strategic review in March 2021, which effectively paused the rollout of the multi-billion dollar project. The review uncovered widespread patient safety issues, technical problems, and training shortfalls, as well as other budgetary and organizational challenges at the department’s first go-live site in Spokane, Washington.

The VA inspector general’s office, in a series of reports released last month, substantiated many of the EHR concerns that agency employees reported in Spokane. The new EHR went live at that facility in October 2020.

Remy said the VA is building on lessons learned from the first go-live in Spokane. The agency made over 2,000 content changes in training, expanded the training course catalog and virtual training options.

Remy said that at this point, there aren’t any unaddressed help tickets flagged as potential patient safety issues in Walla Walla.

The VA also plans for its new EHR to launch in Columbus, Ohio on April 30.

“We’ve got to get it right, that’s our goal. Our goal is to make sure that as we deploy this Electronic Health Record, we’re deployed in a fashion that’s workable for the clinicians there, and that is going to provide the benefit for veterans that we know these types of systems can provide,” Remy said.

Adirim said the VA is also learning from DoD, which has deployed the same EHR to over half their health system, with over 90,000 users successfully using this system to deliver care to service members, retirees and their families.

“EHR implementations are never simple, and never without challenges. The technology is complicated enough, but we also need to account for the enormous change to how we are delivering health care, which relies on people to adopt the new ways. And people are more complex than the technology. We know this, and this is why we prepare for problems and have plans to address those that come up,” Adirim said.

Scott Kelter, the director of the Walla Walla VA Medical Center, said Cerner has doubled the number of personnel onsite to train VA employees, compared to the first go-live in Spokane.

Cerner usually provides one adoption coach per 20 personnel, for about 10-14 days. At Walla Walla, Cerner has one trainer for every 10 personnel for the first two weeks. After that, Cerner will provide one trainer per 20 personnel.

“People are very engaged in how do we experiment and learn the new system, especially while we have the Cerner team supporting us. I’ve really been encouraging everybody to try to break it while they’re there. Experiment with everything you can, test every capability you can,” Kelter said.

Kelter said productivity is improving as staff becomes more familiar with the new EHR.

“In any workforce, you’re going to have people that are more easily able to learn new technologies, some that take a little bit longer, and some that have been using the legacy system for decades, and might take a little bit longer to unlearn what they expect to see, versus those that are newer employees that are not so entrenched in the VistA and [Computerized Patient Record System],” Kelter said.

To prepare for the go-live, Kelter said veteran employees came into the facility for real appointments with their providers.

“Our employees obviously knew that this was the first time that their providers were using the system, and they had the opportunity to really explore the system,” Kelter said.

The VA is also looking at taking best practices from Walla Walla and applying them back in Spokane.

Adirim said Cerner has been in Spokane continuously, and that the Veterans Health Administration sent a patient safety team last summer to work with them.

“Clearly there were things that we should have done better in implementing there, and we’re not giving up on them. As we can move forward with our new approaches and new ways of doing things, we’re going to continue to go back to Spokane and help them, support them and determine what their needs are,” she said.

Jory Heckman

Jory Heckman is a reporter at Federal News Network covering U.S. Postal Service, IRS, big data and technology issues.

 

Follow @jheckmanWFED



--
--
http://groups.google.com/group/Hardhats
To unsubscribe, send email to Hardhats+u...@googlegroups.com

---
You received this message because you are subscribed to the Google Groups "Hardhats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hardhats+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hardhats/4339067.ez7UBJjR4b%40owl.

Tom Munnecke

unread,
Apr 11, 2022, 4:09:40 PM4/11/22
to hardhats
Am I right in assuming that Cerner is using a centralized data center for all of these locations?


Jim St. Clair

unread,
Apr 11, 2022, 4:13:40 PM4/11/22
to hardhats
Tom, that's a great question. I remember years ago that there was discussion about Cerner hosting all the instances from their data center so I think the answer is yes...

Best Regards,

Jim St.Clair

Executive Director

Linux Foundation Public Health

(228) 273-4893

www.lfph.io

Connect with me: https://calendly.com/jstclair-4



From: hard...@googlegroups.com <hard...@googlegroups.com> on behalf of Tom Munnecke <munn...@gmail.com>
Sent: Monday, April 11, 2022 3:09 PM
To: hardhats <hard...@googlegroups.com>
Subject: Re: [Hardhats] Cerner outage hit the DOD and Coast Guard also
 

mikegi...@aol.com

unread,
Apr 11, 2022, 4:25:21 PM4/11/22
to hard...@googlegroups.com

Every time I read about this miserable project, I think of this song: https://youtu.be/24VOo7-ctKU

--

Tom Munnecke

unread,
Apr 11, 2022, 4:51:33 PM4/11/22
to hard...@googlegroups.com
I guess the proper assessment of the “root causes” is 1)  a single point of failure in a central database and 2) the choice of Cerner. 

Nancy Anthracite

unread,
Apr 11, 2022, 5:46:44 PM4/11/22
to hard...@googlegroups.com, mikeginsburg via Hardhats

:-) Appreciate it!

 

--

Nancy Anthracite

rrichards

unread,
May 10, 2022, 7:31:13 PM5/10/22
to Hardhats
Cerner is designed around its single database in a single data center for all its clients in USA, all via Citrix.

Cerner runs the worlds largest Citrix farm with 500,000 Citrix clients going to its data center in Kansas.
It is a single point of failure for all of these clients.

Darren Coolidge

unread,
May 10, 2022, 9:45:10 PM5/10/22
to Hardhats
Reply all
Reply to author
Forward
0 new messages