Im sure you've read all the articles in the Success Center and even contemplated that one method you read about doing this with "Zero Downtime". Well instead, I'm just going to tell you what I did and hope it at least helps a couple people who might be currently pressured into upgrading. *cough*
Start by powering down that NTA server of yours. You wont need it. Orion version 2020.2.1 now creates a new NTA database on the same SQL server that the Orion database gets placed on. This will happen during the configuration wizard for the 2020.2.1 install.
At this point, you should now have (2) Windows 2019 Servers. The one with SQL 2019 is setup and ready to go and another that will be your new SolarWinds Orion server, which is just sitting there waiting in anticipation.
Log in as the Local Administrator on the new Orion server and run the installer. DO NOT PROCEED WITH THE CONFIGURATION WIZARD. We will do this when it's time to cut over. Just cancel when you get to this step.
Once the backup is done, go ahead and copy it to your new SQL 2019 SQL Server and restore it using the same link in the last step. You will no longer need that old SQL Server of yours. (For SolarWinds at least...)
Once the database has been restored, you will need to select the newly restored database on your new SQL 2019 server an execute the following query, replacing 'Server 1' with the NetBIOS name of your OLD Orion server and 'Server 2' with the NetBIOS name of your NEW Orion server:
Just to recap, we've deactivated our licenses, backed up our database on the old server, restored it on the new server, updated the databased to reflect the NetBIOS name of our new server, and powered down the old Orion server.
Now on your new Orion server, go ahead and start the configuration wizard I told you to skip earlier. Follow the steps on the configuration wizard and select the option to use an existing database. Point it to your new SQL 2019 Server and use the SA account to create the initial connection. The configuration wizard will then ask if you want to create a new account for SolarWinds to use instead. Please do this and do not actually use your SQL SA account.
The configuration wizard will now prompt for the same thing, but now for the new NTA database. Choose to create a new database for it on the new 2019 SQL server. Use that new account you created in the last step for accessing the Orion database.
While I can't guarantee the same process goes as smoothly for everyone else, especially if you have multiple polling engines (please see ahbrook comment below for help with APE's) this process only took me (1.5) hours to complete after I had already pre-staged everything along side my old environment.
Should for whatever reason you need to fall back, all you will have to do is power on the old Orion server and reactivate your licenses. (Assuming you haven't changed the hostname and IP of your new Orion server to match. This is why I created a CNAME and tested my access to the portal first.)
This is good in case you're not on an affected version and someone still thinks it's smart to upgrade to 2020.2.1 HF2 right now! I'm not sure why anyone would think this is a good idea right away but... that's just me.
My intention with this post was specifically help anyone that might be under extreme pressure to get to the latest version. I'm sure we can all agree this was not on our Holiday wish list, especially mine, but felt I should at least offer what I can for anyone who has not had to do this previously.
In this scenario, are you leaving the agents installed on your managed nodes? I've not seen much guidance on whether or not they would be compromised as well.. We're estimating that we will have to completely uninstall and reinstall our agents, and I do not know how our database would like that -- if we decide we are keeping our DB and it isn't infected.
When I performed this upgrade / migration, I completely left our SolarWinds agents as is on the servers they were installed on. Once I created the CNAME record of the old Orion server that points to my new one, the agents checked in and updated their version of the agent software automatically.
The only thing you would still need to do after is modify the agents settings from control panel on each server that has it installed, so the hostname or IP now points to the new server name. That's assuming you're not eventually changing the name on your new Orion server to match the old one. Then you can skip having to do this part.
On Friday, we got the call from our security office that we were safe to proceed with a migration scenario. I used this thread as part of my work to develop steps, and learned a few things in the process that may be helpful for others.
Main poller running 2020.2.0 on physical hardware
Additional polling engine running as a VM
Production database running on a physical server
Test main poller running as a VM.
Test database running as a VM.
Wipe and rebuild the physical machine, giving it a new name (app02) but the same IP as the previous server
Build a new APE, giving it a new name (ape02) but the same IP as the previous server
Build a new test main poller, giving it a new name (app02-t) but the same IP as the previous server.
Changed the passwords that Orion accesses the databases with, but left the database servers more or less alone.
- The Test environment would not see the Test database. This turned out to be because firewall rules that allowed inbound communications were lost on the DB. We don't know why at the moment; I suspect they were pulled out due to an abundance of caution.
- Eventually we moved app02 to a unique IP address, and this allowed the configuration manager to run and the website to work, just extremely slowly. It sometimes hung trying to access the orion agent to download it.
- To finalize cleanup on prod, I needed to remove the now-useless extra poller (the extra app02 that was created by my mistake), repair the license using the steps outlined above, and then re-apply the licenses we deactivated before starting. We also reset the IP back to its original, and ran the configuration wizard again.
After taking these steps, production looked good - the instance was running, I could see all of our nodes in the maintenance mode we put them in, and security was working as expected. The agents were not checking in, however, but we are waiting on networking to put in the CNAME to see if that fixes it.
As far as test goes, the process was simpler because we had tested in prod... as you do. (Given that prod was physical and was not having the database access issues, I focused on it first). For test, I was also able to go in to edit the database and rename the references. In its case, the license seemed healthy, but it did not work and allow us to manage nodes until we deactivated and re-applied all our licenses. I also needed to clean up the extra polling engine that it thought existed.
If you have a multi server or HA setup, it is extremely important that you go in and check the database references in between installing the new software and running the configuration manager. There can only be one main poller, and likely the system still thinks it exists if you are using the same database servers. It doesn't take much to check this, so better to be safe than sorry!
We're still not 100% up, but we are looking better for the holiday break. I am super grateful to this thread and other places that have been sharing tips and tricks to getting things secured and back to full operational status, and that's why I wanted to post this - to help others who may follow in our footsteps.
Monitor, analyze, diagnose, and optimize database performance and data ops that drive your business-critical applications. Unify on-premises and cloud database visibility, control, and management with streamlined monitoring, mapping, data lineage, data integration, and tuning across multiple vendors.
Modernize your service desk with intelligent and automated ticketing, asset, configuration, and service-level agreement (SLA) management; a knowledge base; and a self-service portal with secure remote assistance. SolarWinds offers an easy-to-use IT service management (ITSM) platform designed to meet your service management needs to maximize productivity while adhering to ITIL best practices.
Ensure user experience with unified performance monitoring, tracing, and metrics across applications, clouds, and SaaS. Robust solutions offering rich visualization, synthetic and real user monitoring (RUM), and extensive log management, alerting, and analytics to expedite troubleshooting and reporting.
Reduce attack surface, manage access, and improve compliance with IT security solutions designed for accelerated time-to-value ranging from security event management, access rights management, identity monitoring, server configuration monitoring and patching, and secure gateway and file transfer.
Orion Platform version 2020.2.5 adds to these enhancements with additional security fixes and protections. We recommend you upgrade to the latest available release (2020.2.5) as soon as is practical. For more information, review the Release Notes here, and KB article here.
If you have recently upgraded to 2020.2.1 HF2 or 2019.4 HF 6, you are also protected from SUNBURST and SUPERNOVA. However, as part of our response to the SUNBURST vulnerability, the code-signing certificate used by SolarWinds to sign the affected software versions was revoked March 8, 2021, and you may experience performance issues if you do not apply the more recent updates. Please visit our SolarWinds New Digital Code-Signing Certificate page at
solarwinds.com/trust-center/new-digital-certificate for more information.
Disconnecting any product from the internet can reduce your attack surface. The Orion Platform is fully functional without an internet connection. Based on our investigations to date, SUNBURST requires an internet connection to be activated and disconnecting your Orion server from the internet should immediately protect your environment from SUNBURST. The vulnerability in the product that allows for the deployment of SUPERNOVA, however, may still be exploited by a malicious actor who accesses your network from inside and not over the internet. Disconnecting from the internet will limit the ability of an external actor exploiting this vulnerability over an internet connection.
3a8082e126