MSFT patch breaks SolarWinds Orion?

[Kurt Buff]

All,

We applied KB5007152 to our Orion box, which talks with SQLServer on
another box. No patches were applied to the SQLServer machine.

It broke communications with the SQLServer, though I did see that
there were connections to port 1433 on the SQLServer.

One of our sysadmins opened a case, and according to our internal
ticketing system the following was done to resolve the problem:

reindex SQL database - SQL server
shutdown all services in Orion service manager - IIS server
reset admin pw - PW was lost - IIS server
reconfigure user setting in the orion interface to allow user access
to the pages again - web interface
confirm settings in IIS - no changes needed - IIS server
open Orion database manager - IIS server
run several custom queries to update SQL tables that were changed due
to MS patching - IIS server
update system resources for the SQL server - IIS server
reboot SQL server - SQL server
all services are now operational - IIS, SQL, web interface

Has anyone else run into this? I'm looking to see of our Orion box was
pre-broken, of if the patch really caused the problem - I reviewed the
KB article for the patch, and noted that MSFT observed no problems
with it.

Thanks,
Kurt

[Mike]

My first thought was something with Kerberos delegation as there were some known issues with that, but  KB5007152 is for .NET. Any chance other patches were installed at the same time? Also...

"run several custom queries to update SQL tables that were changed due

to MS patching - IIS server"

What? A patch changed SQL tables? Which tables and what was changed?

--
You received this message because you are subscribed to the Google Groups "ntsysadmin" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ntsysadmin+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/CADy1Ce5tTS4zW5%3D7Ap5_OhBjhHRZObQPz8mV7uDDdExjekE%3DiA%40mail.gmail.com.

[Kevin Lundy]

I would look or post in Thwack.

To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/CA%2BSdsNEY2kDZY_26NBCOJhvESGumZV9stT3RQ%3Dh7Gt%3Dvbdp09A%40mail.gmail.com.

[Kurt Buff]

No other patches were installed on either the SQLServer box or the
Orion application box.

I was not party to the ticket, and the sysadmin is a bit unhappy that
Orion was broken this morning.

No patches were applied last night to the SQLServer box, but on the
11th KB5007192 was applied to both boxes - and they're both 2016.

What I find very strange is the statement:

"run several custom queries to update SQL tables that were changed due
to MS patching - IIS server"

That makes no sense to me.A patch on the application server should not
change tables on the SQL box. AFAICT.

Kurt

> To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/CA%2BSdsNEY2kDZY_26NBCOJhvESGumZV9stT3RQ%3Dh7Gt%3Dvbdp09A%40mail.gmail.com.

[Solodow, Damien]

I didn’t see anything on there about it.

Kurt – do you have/can you send me the Solarwinds ticket/support number? I’m one of their Thwack MVPs so I have a few extra strings I can tug to investigate this.

 

Like several of the other responders, I’m rather concerned about the assertion that the patch changed the SQL tables.

​

Damien Solodow IS Senior Systems Engineer Gaylor Electric, Inc. 5750 Castle Creek Pkwy N Drive, Suite 400 Indianapolis ,  IN .  46250 O: 317.815.3103  |  M: 317.506.8521 317.759.0077 emergency IS support

To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/CAB0geZO%2BZj4YHhdMaHEN5KM2d5pb9PPYsyPgVcEOtEOU2zGHXw%40mail.gmail.com.

[Kurt Buff]

Damien,

Ticket number sent privately.

Thanks,

Kurt

To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/CH0PR04MB81294FE8D88B5059DA7D904EBC9B9%40CH0PR04MB8129.namprd04.prod.outlook.com.

[Michael B. Smith]

I'm not going to say "impossible", but it strikes me as very unlikely.

Loss of the admin password (expiration?) seems most concerning of those items listed...

[Solodow, Damien]

Most likely thing there is that their Orion install authenticates people by either Windows Auth or SAML, and the local built-in account wasn’t used anymore, and so no-one had the password and/or it expired.

 

​

Damien Solodow IS Senior Systems Engineer Gaylor Electric, Inc. 5750 Castle Creek Pkwy N Drive, Suite 400 Indianapolis ,  IN .  46250 O: 317.815.3103  |  M: 317.506.8521 317.759.0077 emergency IS support

 

To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/CADy1Ce5tTS4zW5=7Ap5_OhBjhHRZObQPz8mV7uDDdExjekE=i...@mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "ntsysadmin" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ntsysadmin+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/49267818929741ff...@smithcons.com.

[Kurt Buff]

I agree - I'll bet that the admin password was expired but in use in
an active session, so was still alive, but broke when the application
box rebooted.

Kurt

> To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/49267818929741ff886ac0a6f0785aa5%40smithcons.com.

[Kurt Buff]

That's what I get out of it. Here are the closing notes from the support rep - I think it supports the hypothesis: The earlier notes were from the sysadmin.

"we worked in a webex

get website error

has error regarding IPAM

doesn't know the admin password

reset the admin password

now web console comes up

changed IPAM for your account admin

now your account works

you closed the case from the customer portal"

Kurt

To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/CH0PR04MB81298A04B626F71351C42EECBC9B9%40CH0PR04MB8129.namprd04.prod.outlook.com.

[Kurt Buff]

Per my response on the patchmangement list, this is the final note
from the SW support rep - the earlier note was from the sysadmin, and
I don't think his analysis was accurate:

"we worked in a webex
get website error
has error regarding IPAM
doesn't know the admin password
reset the admin password
now web console comes up
changed IPAM for your account admin
now your account works
you closed the case from the customer portal"

This seems to support password expiration as the root cause.

Kurt

On Thu, Nov 18, 2021 at 12:10 PM Michael B. Smith <mic...@smithcons.com> wrote:
>

> To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/49267818929741ff886ac0a6f0785aa5%40smithcons.com.

[Philip Elder]

I read that as being IIS had some tweaks due to the patches that caused the IIS/App to fall out of alignment with the SQL setup.

Philip Elder MCTS
Microsoft High Availability MVP
E-mail: Phili...@mpecsinc.ca
Phone: +1 (780) 458-2028
Web: www.mpecsinc.com
Blog: blog.mpecsinc.com
Twitter: Twitter.com/MPECSInc
Skype: MPECSInc.
 
Please note: Although we may sometimes respond to email, text and phone calls instantly at all hours of the day, our regular business hours are 8:00 AM - 5:00 PM, Monday thru Friday.

-----Original Message-----
From: ntsys...@googlegroups.com <ntsys...@googlegroups.com> On Behalf Of Kurt Buff

To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/CADy1Ce4-19ccDN842n_UgkS430deC_0WF1_D9NCSV2FSxPMpFg%40mail.gmail.com.

[Miller, Jon]

Not the latest MSFT patches, but the latest Orion patches caused problems for us.  Maybe an Orion patch that got applied on restart?

 

From: ntsys...@googlegroups.com <ntsys...@googlegroups.com> On Behalf Of Kurt Buff

Sent: Thursday, November 18, 2021 1:43 PM
To: ntsys...@googlegroups.com; patchma...@googlegroups.com

--
You received this message because you are subscribed to the Google Groups "ntsysadmin" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ntsysadmin+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/CADy1Ce5tTS4zW5=7Ap5_OhBjhHRZObQPz8mV7uDDdExjekE=i...@mail.gmail.com.

This message and any attachments may contain legally privileged or confidential information, and are intended only for the individual or entity identified above as the addressee. If you are not the addressee, or if this message has been addressed to you in error, you are not authorized to read, copy, or distribute this message and any attachments, and we ask that you please delete this message and attachments (including all copies) and notify the sender. Delivery of this message and any attachments to any person other than the intended recipient(s) is not intended in any way to waive confidentiality or a privilege. All personal messages express views only of the individual sender, and may not be copied or distributed without this statement.

[Henry Awad]

We have a similar setup (Windows Server 2016) running the latest version of Orion and we just applied the November patches yesterday morning without any issues. So very unlikely that the November patches broke anything. 

Henry Awad

Senior Systems Engineer

Technology Services

The Catholic University of America

(D) 202-319-5404 | (M) 571-233-4479

To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/9aada464feb14898a2567c651c49991d%40DT-EX151.boselaw.com.

[Kurt Buff]

Thanks for that confirmation. That's what I would have expected.

Kurt

> To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/CAGaCHK7-JPP%3D_4R3Ut7UY736GHAcB%3DVgXhsFgJaH9kk0Ri5E0w%40mail.gmail.com.

[Kurt Buff]

I suppose that's a possibility.

We recently (late October) upgraded our licensing to their new
Observability licensing - the latest install dates listed for their
products on this machine are on 2021-10-27, and the machine has
rebooted as least twice between then and last night, with the latest
reboot before last night being the 11th.

Kurt.

> To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/9aada464feb14898a2567c651c49991d%40DT-EX151.boselaw.com.

[Dave Lum]

One of the recent Orion patches put an expiration on all Orion-specific accounts, so if you were using local Orion accts and not AD-integrated, this might trip up some functions using the local accounts.

 

I have patched Orion to current Windows patches without issue.

 

Dave

 

From: ntsys...@googlegroups.com <ntsys...@googlegroups.com> On Behalf Of Miller, Jon

Sent: Thursday, November 18, 2021 11:23 AM
To: ntsys...@googlegroups.com; patchma...@googlegroups.com

To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/9aada464feb14898a2567c651c49991d%40DT-EX151.boselaw.com.

Attention: Information contained in this message and or attachments is intended only for the recipient(s) named above and may contain confidential and or privileged material that is protected under State or Federal law. If you are not the intended recipient, any disclosure, copying, distribution or action taken on it is prohibited. If you believe you have received this email in error, please contact the sender with a copy to compl...@ochin.org, delete this email and destroy all copies.

[Kurt Buff]

I never did find out exactly what happened.

One of life's little mysteries, I suppose..

Kurt

To view this discussion on the web visit https://groups.google.com/d/msgid/ntsysadmin/CO1PR17MB5308848F22739CFE4DD3C250DD9F9%40CO1PR17MB5308.namprd17.prod.outlook.com.