Netbox-docker start up scripts

[Christopher Mills]

I’m trying to find a way of merging two netbox instances in to a new single instance. I’m aiming towards running NetBox via an augmented docker container based on the work of netbox-docker project. Augmented to include my own plugins and required python packages.

I’ve just discovered the collection of start up scripts shipped with the netbox-docker source and have the idea that I could use those to pull data from my existing instances using PyNetbox and inject them in to the new instance at that point. Keeping the new one in sync during migration seems like it could just be a simple case of running 'docker-compose restart netbox’.

I’ve made some progress on this and successfully pulled in and created all my manufacturers and device types, but the interfaces etc defined within the device type template don’t seem to be catered for. Am I missing something or is this missing functionality? Has anyone any idea where to start with this? There are startup scripts that seem to act upon interfaces, could these be repurposed to support the template items?

I won’t raise an issue to netbox-docker if I’m being a bit dim and just missing something, but if there is missing functionality here and others think it would add value, I’ll raise an FR (and or try and write something).

[Brian Candler]

I'm not sure what you mean by "merging two Netbox instances".  If it's just a question of replacing one Netbox docker container with your augmented Netbox container, then all you need is to do a postgres dump and restore from one to the other (and also the media directory, which really ought to be in a docker volume anyway).

"Merging" Netbox instances, in the sense of importing data from instance A into instance B which already has different pre-existing objects, is hard.  Netbox doesn't use UUIDs, it uses sequential integer IDs, so you'll have to keep track of the IDs which were used in the old database versus the IDs allocated on insert in the new database, and modify all foreign key references accordingly.  You'll also have to insert things in the right order to meet foreign key dependencies (e.g. create devices before interfaces), as the REST API doesn't give any way to import a subtree of related objects.  In some cases, even that is not enough: e.g. you can't set a primary IP address on a device (or VM) until you've first created its interfaces *and* linked an IP address to the interface.  This means you have to create the device/VM without primary_ip4/ip6, and update its primary_ip4 and primary_ip6 fields afterwards.

I really wouldn't recommend going down this road unless you have no alternative.

[Christopher Mills]

“The best thing about sources of truth is that you can have as many of them as you need/want” - said no one, ever.

For historic reasons (two teams merging) I have the need to combine two instances of NetBox in to one - this is to reduce the TCO of the two systems, to reduce the need for repeated permissions, to ensure consistency of device types and to ensure all team members know there is indeed only one source of truth. The two instances I currently look after represent many many hundreds of hours of work as we have documented devices, virtual machines, IPAM and cables between approximately 250 racks. There is complete inconsistency between the two in terms of tenants, device types, roles etc that I will need to deal with in time, but first and foremost I want to get all the objects for all the models imported. I can work on sanitising the data later and plan to do that with a combination of reports, external scripts using PyNetbox, bulk edits and sheer hard work!

I know it’s going to be hard to merge the two and I appreciate this is going to mean changes to all the object IDs - that alone is going to have a huge knock on impact to all the systems I have that feed off of my (two) sources of truth at the moment and one that I know I am going to need to deal with. Hence the desire to find the easiest way to import all the data from both in to one. The start-up scripts and initialisers supplied with the netbox-docker project seem to have the potential to be the easiest way to make this happen by re-creating new objects that fundamentally appear (to the user at least) similar to the existing ones. As far as I can see they already deal with the need to create certain objects before other objects through the use of sequential file naming to ensure, for example, devices are created before the interfaces that belong to them.

Incidentally, how much input does the NetBox core team have in to the development and support of the netbox-docker project? It seems this functionality might have been developed to facilitate unit and integration testing to guarantee a known collection of data on container start up?

My question was whether the need to create template interfaces, ports etc had been catered for or not and whether the approach would need to be similar to the creation of interfaces on live objects - i.e. it appears that had been catered for, how easy would it be to adapt that to create template interfaces instead?  I’m willing and able to put some hours in to Python coding, I have little experience however with Django and the ORM, but I’m learning. So far I’ve successfully written a script to create devices and connections from an Excel spreadsheet (not CSV) so I know it should be possible.

As an aside, I guess it would be a challenge, but would suggesting a move to UUID based keys be an FR worth raising with Jeremy et al?  I had an idea about how to create a UUID based on current data, but it seems Django already has a UUIDField type. I won’t pretend I know anything about how it would work in reality nor how it could be added to an existing NetBox instance - I’ll look to people cleverer than me for that!

Many thanks

--
You received this message because you are subscribed to the Google Groups "NetBox" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netbox-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/netbox-discuss/f9f9fe24-5fef-4ea0-9634-5155adabb28ao%40googlegroups.com.

[Brian Candler]

If this is a one-time merge, then personally I'd approach it at the SQL level, not via the REST API.  That is:

- bump all the sequences on the target system to leave a big enough gap for everything being inserted.  The target system can then carry on as normal.

- renumber all objects being imported, so that their ID sits between max(id) of the target table and the starting point of the new sequences (*)

- do a test insert into an empty database, to check all the foreign keys have been updated correctly

- insert into the live database in one transaction with deferred constraints - or if that's too big, as a bunch of smaller transactions (e.g. one per site, or at worst as small as one per device/VM)

(*) This is the tricky part. You could in principle manipulate a text SQL dump, remembering to find and update all the FK key references. However I think it's safer to work directly on a copy of the SQL database, making individual transactions to renumber each item (device ID for example) at the same time as updating all the related FK fields.  If you get this wrong, the transaction will fail (except for generic foreign keys - you need to be careful with those).  Once this is complete, then dumping and inserting into the target database is trivial.

Or: you could just create a map of old_id to new_id for each table.  Then when you import any id or foreign key field, remember to replace its value with a lookup of id to new_id.  This is mostly straightforward, although generic foreign keys are more of a pain (e.g. in Netbox 2.9 an IP address can link to either a device Interface or a VMInterface).

- you'll still likely need to do tidying up afterwards: e.g. identifying and merging duplicate IP addresses, sites, tenants etc - unless these two Netbox instances had absolutely no overlap in scope.

[Jeremy Stretch]

> Incidentally, how much input does the NetBox core team have in to the development and support of the netbox-docker project?

There's not much regular overlap between the two. Occasionally Christian (ciminine on GitHub) will raise an idea with us but he's done a tremendous job running with the Docker project mostly on his own.

> As an aside, I guess it would be a challenge, but would suggesting a move to UUID based keys be an FR worth raising with Jeremy et al?

That would result in a tremendous amount of disruption for current users, and isn't likely to introduce a proportionate amount of value. (Merging two NetBox instances isn't a very common task.) There are also concerns around ordering that make the use of UUID as a primary key somewhat unappealing.

I'd caution that, before starting any migration work in earnest, you need to take a hard look at both instances and classify each model as one of the following:

1. No overlap among instances (e.g. one instance has sites from North America, the other from Europe)

2. Exact overlap (e.g. both instances have the same sites defined, but naming and other attributes are consistent)

3. Mismatched overlap (e.g. both instances have the same sites, but use a different naming scheme)

Case 1 obviously allows for the most direct migration path. Case 2 is fairly direct, but you'll need to consolidate objects and their relations. Case 3 requires a lot more planning, but you may be able to do some cleanup on the originating system first to help it better match the destination. The more you can consolidate the two before beginning the migration work, the easier it'll be.

Hope that helps!

To view this discussion on the web visit https://groups.google.com/d/msgid/netbox-discuss/DEF0FAA0-E719-4D1E-A9F7-0CB7EDDA3CA7%40discreet-its.co.uk.

--

Jeremy Stretch
Sr. Network Automation Engineer
Network to Code, LLC