Manually Migrating Secret Keys from H2 to another DB backend.

[John Apple II]

Hi, Rundeck-discuss list,

Since I've just figured out how to do this over the past 48 hours (hours of intense suffering on my part), I thought I'd leave this here in case it helps anyone. I hunted all over the web for the same details and found nothing helpful. Next time, someone remind me that "simple tests to see if I can" are never really that simple and take 10 times more time than you allocate for it.

It's a totally manual process, but you can move the secrets over between H2 and - say - MySQL without manually finding/recreating each one in your vault-or-secrets-spreadsheet again. Make sure you do this in a secure system/laptop, since you're fiddling with secrets that I'm sure nefarious actors would love to snarfle off your system one day.

NOTE BEFORE WE BEGIN: YMMV, you could break everything and destroy the planet, test-test-test, no warranty is provided or implied, your corporate InfoSec team might have SEVERE reservations  about doing this at work, etc., etc., etc. (AND I AM COMPLETELY CERTAIN THAT THIS IS IN NO WAY SUPPORTED BY RUNDECK!)

But if you really, really, really, REALLY need to, then this worked for me. In the spirit of OSS I leave this here in the hope this could help some poor, intrepid soul who finds themselves in my same position one day.

The secret - as I figured out the hard way - was the STORAGE tables in both H2 and the new DB, AND keeping the exact same encryption key between the two backends during this process.  Why - you ask - did it take me so long to figure this out?  Because I'm a bit dense at times.

You will need to:

1. have the H2 binaries installed so that you can access the DB console of H2 (make sure it's the same version as your source Rundeck-instance-in-question uses)
2. have the new database setup with your target Rundeck set (potentially setup a mirror copy of Rundeck with execution-mode and schedule-mode disabled) where you restore each project archive and have it setup the DB schema in the new backend.
   (NOTE: I used DBeaver to open both the H2 and new DB Backend to make the process easier on myself. A similar tool might help you, but you should be able to do this on the CLI if you can handle character escaping.)
3. have the new copy of Rundeck running the same version as your source server but attached to the new backend - with the SAME encryption key configured.
4. (optional) securely copy the entire H2 DB across to a secure vm so as to protect the source host DB during this process.  Working on the live DB is potentially dangerous.
5. The same version of Java that your Rundeck is using on the H2 hos
6. You probably want to make sure no one modifies the source Rundeck server during this process, else you'll be missing jobs and keys from the time the H2 database was made.

Steps:

1. Create blank keys mirroring the type, name, and folder structure in the new, target Rundeck instance that is attached to the new DB backend. 
   • Put something for each value you know that will not match any stored secret already existing (like "password1", then "password2", etc) which is - I hope - not a secret anyone stores for use!).  Trust me - this helps and prevents strange errors with SHA duplication message later).
   • DO NOT copy the rows between the tables - that won't work, and the target copy of Rundeck will not understand them (ask me how I know sometime!), but they WILL cause errors on creating the new secrets. Create each key individually with a separate secret (maybe the `rd` tool can make this easier for you)
2. Connect to the H2 database and the new Backend database in your preferred method.
   • Be sure your H2 binary server matches exactly with the version used in your Rundeck instance or it likely will not open the tables.
     
   • My connection string was something like this:
     
     jdbc:h2:tcp://10.0.0.2:9092/rundeckdb
     
     
   • The java command to start the H2 server I used was 
     
     java -cp "$dir/h2-{version-string}.jar:$H2DRIVERS:$CLASSPATH" org.h2.tools.Server "$@" -baseDir /home/user/rundeckdb/data/
     
     
3. Open the Storage Table in both databases. The relevant fields in both are:
   
   • DIR
   • NAME
   • DATA
4. For each secret, match the row with DIR and NAME between the H2 and new Database backend STORAGE table.
5. Copy and paste the DATA field from the H2 table for that match into the new Database table.  BE SURE THE VALUE MATCHES EXACTLY between the two tables!
6. When you're done, make sure to commit the changes into the new Database table.
7. Whalla! After several hours of careful verification, copy and paste, your secrets should now exist in the new backend.
8. Test a few jobs that use secrets and make sure they don't complain about the secret not being found.

NOTE: If you have errors in jobs after this, it's likely because your secret-type does not match (e.g. it's a PASSWORD type, and you accidentally created it as a PRIVATE KEY.  Of course I NEVER did that while I was working through this process and it DID NOT take me 2 hours to figure it out why.)

I'd highly recommend making sure you wipe any copies of the clipboard you were using to make the copies, and potentially securely wiping the VM you used to run this migration if you're dealing with any InfoSec teams, depending on the secrets you stored in Rundeck and the environment you're in.

Best of luck Rundeck-ers - and I hope this helps someone!

- John

[rac...@rundeck.com]

Thanks a lot, John!