How is everyone importing?

[joey_joe_joe]

I may have missed an important part of the docs but I don't see a section on importing data from an existing infrastructure. I see the ability to create via the API but not an actual import utility? How is everyone importing and could you share examples?

[Jacob Mansfield]

Most list pages have a green import button in the top-right, which allows you to upload a CSV.

Regards,

Jacob Mansfield

www.jacobmansfield.co.uk

@KingCyberJacob

On 9 February 2018 at 16:43, joey_joe_joe <80scybo...@gmail.com> wrote:

I may have missed an important part of the docs but I don't see a section on importing data from an existing infrastructure. I see the ability to create via the API but not an actual import utility? How is everyone importing and could you share examples?

--
You received this message because you are subscribed to the Google Groups "NetBox" group.
To unsubscribe from this group and stop receiving emails from it, send an email to netbox-discuss+unsubscribe@googlegroups.com.
To post to this group, send email to netbox-discuss@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/netbox-discuss/47987f5f-de72-4d5e-bfd8-148b632debab%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Joseph Wegner]

Our old IPAM solution, IPPlan, didn't support IPv6, which was most of the reason that we migrated to Netbox in the first place. I wrote a Python script to parse out the MySQL dump and format it into CSV, but it's pretty specific to our organization. Then I used the CSV import to import VLANs, prefixes, and IPs.

I also wrote a script in Powershell/PowerCLI that synchronizes our vCenter inventory with Netbox through the API. It did most of the import flawlessly, but there are still some issues I need to resolve before I would consider posting it.

[Chris Petrolino]

If you have any details to share on your sync between PowerCLI and the Netbox API I would LOVE to take a look! 

[Niels Dutry]

I upvote this as well :) Would also like to have some sync between vCenter VMs and NetBox.

[Kevin W]

Hi Joseph, I would be veru interested in how you set the vCenter to Netbox Sync up nightly, exactly what I would like to do.

If you could share this please, Im sure we would al be extremely grateful.

Kind regards.

[Joseph Wegner]

I pulled out most of the things specific to our organization and posted the code here: https://github.com/jwegner89/netbox-utilities

You will likely have to tweak these for your own use. I welcome suggestions though. They're not perfect, and I wasn't able to test after pulling out our custom stuff, so there might be issues.

[Kevin W]

Hi Joseph, this is great, many thanks for sharing this. I have taken a quick look at the scripts and the notes. Do al the scripts run independently, or do they all run from the Sync-Netbox.ps1 script?

Kind regards.

[Joseph Wegner]

They are independent for the most part. In particular, Sync-Netbox.ps1 is self-contained. I wrote Sync-Netbox.ps1 to be able to run as a Windows scheduled task on a control station type host so that you can synchronize the inventory at some set interval, although it works fine from a regular workstation.

For our environment of roughly 900 VMs and 37,000 IPs (not all associated with those VMs ;) ), a single run will take perhaps 40 minutes at most, so it's nice to be able to run it in the background. There are certainly some ways to optimize this, but I call it good enough for my purposes.

[Kevin W]

Cool, can't wait to give this a try, I'm working away for a couple of weeks, so will report back once I've got my head around this on my return.

Great work, and thanks again for your time.

Kind regards.

[Joseph Wegner]

Sounds good, hope it helps. The big things that you will need to change with the script are

1. Your Netbox instance's URL
2. vCenter Server hostname(s) - separate with commas if you have multiple, e.g. vcenter1.example.com,vcenter2.example.com
3. API authorization token
4. VM host cluster names

I'd recommend commenting out anything that isn't a GET and run it in verbose on a subset of your VMs. Maybe restrict the auth token for read-only access to begin just to be safe.

I'll work on getting more substantial documentation written up, but I figured I would post what I have to serve as a quick reference since people are interested.

[Kevin W]

Hi Joseph, I'm sorry to ask, but I can't seem to find exactly where to enter the info in the Sync-Netbox.ps1 file, could you point me in the right direction please?

1. What line in the Sync-Netbox.ps1 file where do I enter / change my 'Netbox instance's URL' ?
   
2. What line in the Sync-Netbox.ps1 file where do I enter / change my 'vCenter Server hostname(s)' ?
   
3. For the 'API authorization token' is this entered into line 499 of the Sync-Netbox.ps1 file '$Token =' ?
   
4. What line in the Sync-Netbox.ps1 file where do I enter / change my 'VM host cluster names' ?
   
5. Is there any danger running the Sync-Netbox.ps1 file that it will write anything at all to Vsphere or ESXi?

Kind regards

[Joseph Wegner]

Hi Kevin,

 

For all line numbers mentioned below, I am referring to the master version available here: https://github.com/jwegner89/netbox-utilities/blob/master/Sync-Netbox.ps1

 

1. Change “netbox.example.com” to your Netbox instance’s hostname on line 34.
2. Change “vcenter.example.com” to your vCenter server’s hostname on line 494.
3. Yes, you will change line 499 to 
   $Token = "68b959d575e6028937c2d91825202e6ff5573b8e"
   
   or whatever your newly generated API token is.
4. By default, the script assumes that your clusters are named the same in both vCenter and Netbox. If they differ (and you require this for some reason) then you will need to write some custom code somewhere in lines 88 - 114 that will create the mapping between them.
5. No, all calls made to vCenter/ESX are read-only. However, if you want extra protection against any potential issues, you could configure a read-only user account in vCenter/ESX and use that for running that script. I would recommend that anyway if you are going to set this up as an automated process.

Hope that helps!

[Kevin W]

Hi Joseph, thanks for the below info. I have created a demo setup on my home ESXi server, so I can test this out in my own time and understand each stage.

At work, we have ESXi v6, but at home I am on ESXi v6.7. I setup a clean new Netbox install at home, added a device (my host and interface), added a cluster (same name as in ESXi), then after updating the script with the correct info I tried running the script (Sync-Netbox.ps1), but keep getting these errors (see attached error log). Im sure its something obvious, but could you point me in the right direction please?

[Joseph Wegner]

Hi Kevin,

The issue is that you need to create a custom field on the virtual machine object in your Netbox instance.

Invoke-RESTMethod : {"custom_fields":["Invalid custom field for virtual machine objects: vcenter_persistent_id"]}

All objects in vCenter have a PersistentID field that I used to match up Netbox VMs even if names change. You can add a new custom field at the URL https://netbox.example.com/admin/extras/customfield/add/ (substituting your domain). Here's a screenshot of the settings I have on mine:

I hope that helps!

[Kevin W]

Hi Joseph, perfect, thanks for that info, worked great, all my vm's (on my home test) imported.

Although I did still get one small error below, is this anything to worry about?

Name                           Port  User

----                           ----  ----

192.168.0.110                  443   VSPHERE.LOCAL\Administrator

Invoke-RESTMethod : {"name":["This field may not be null."]}

At Z:\_CUPBOARD\Netbox\netbox-utilities-master\Sync-Netbox.ps1:320 char:37

+ ... $Response = Invoke-RESTMethod -Method POST -Headers $Headers -Content ...

+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebExc

   eption

    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

[Joseph Wegner]

Glad to hear that! I still get some of those issues as well, but I haven't had a chance to investigate further.

---

From: netbox-...@googlegroups.com <netbox-...@googlegroups.com> on behalf of Kevin W <kevj...@gmail.com>
Sent: Wednesday, May 23, 2018 1:17:48 AM
To: NetBox
Subject: [netbox-discuss] Re: How is everyone importing?

 

--
You received this message because you are subscribed to a topic in the Google Groups "NetBox" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/netbox-discuss/p3BdgW1IEIc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to netbox-discus...@googlegroups.com.
To post to this group, send email to netbox-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/netbox-discuss/4f15512a-3d23-4093-81a9-d3165253421d%40googlegroups.com.

[Joseph Wegner]

I just pushed an update which I believe should fix this. I think there are still some outstanding issues, but I've been running it as a nightly scheduled task without issue now.

[Pett Computers]

Hi Joseph, I've just found your script and I had a question I hope you might be able to help with.

I've set my Netbox API token to be read only for testing and when I run the script in verbose mode, it appears it's trying to remove all my existing VM's in Netbox.  Is this expected behavior since you're using the VM UUID to track changes with VM's?

[Joseph Wegner]

Yes, in that case it would be expected. Your options would be:

1. Manually enter the UUID for any VM objects that already exist in Netbox (or script it somehow, probably not too hard)
2. Delete all of your existing VMs from Netbox and let the script recreate them for you (not great because you'll lose any history/references/notes/etc)
3. Modify the sync script to search for matching names and update the UUID accordingly (probably not too hard, but most likely more involved than option 1)

Hope that helps.

[Kevin W]

Hi Joseph, you've done some great work on this, many thanks.

I noticed you have updated your scripts. I am currently running and single VM on version 2.3.3 (need to work out a 1+1 / DB sync, etc), just wanted to ask, am I good to upgrade to latest Netbox version 2.4.4 and run your latest sync script?

Regards.

[Joseph Wegner]

Hi Kevin,

You should be fine updating, I haven't had issues with any of the stable releases so far. The version we're running is 2.4.4. I would always recommend taking a Postgres backup before the upgrade and double check the configuration file for any new options.

I know my script has some outstanding bugs that I'm working on when I have time. As always, feel free to report issues on my Gthub repo.

-Joe

[Kevin W]

All good thanks, updated (alos updated with your scripts) and all working great. One issue I am having is automating the sync script, as auto logging into Vsphere is proving  difficult, keeps prompting me for my username an password. Any tips there please?

Regards.

[Kevin W]

I sorted that out by adding in:

$Username = 'myUsername'

$Password = 'myPassword'

$pass = ConvertTo-SecureString -AsPlainText $Password -Force

$SecureString = $pass

# Users you password securly

$Credential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $Username,$SecureString

Gonna write a python app which runs this script nightly then sends an email with attached log.

[Igor Stojnov]

Hi all,

I am using Invoke-RestMethod in PowerShell to send API calls to NetBox.

The standard use of headers & body  works well:

$Hdrs = @{}
$body = @{}

$Hdrs.Add("Authorization","Token 123123123123123123123123123123")
$Hdrs.Add("Accept","application/json; indent=4")
$Hdrs.Add("Content-Type","application/x-www-form-urlencoded")

$body.add("serial",$blade.SN)
$body.add("id",$blade.id)

$body.add("comments",$blade.Comments)

Invoke-RestMethod -Method patch -Uri "https://netbox.oglan.local/api/dcim/devices/$id/" -Headers $Hdrs -Body $body

So far I've done successfull tests for GET, POST and PATCH... I've used ConvertTo-Json cmdlet to check the JSON content, all is well.

{
    "comments":  "",
    "serial":  "CZJ21806Y0",
    "id":  "4858"
}

But this works only for standard fileds.

For custom_fields, I have problems defining values using the same approach. This synthax:

$body.add("custom_fields",@{"RAM"=$blade.RAM;"cpu_count"=$blade.cpu_count})

Return the following JSON layout:

{
    "comments":  "",
    "serial":  "CZJ21806Y0",
    "custom_fields":  {
                          "RAM":  "32",
                          "cpu_count":  "2"
                      },
    "id":  "4858"
}

So, this looks to be correct to me. But if I try to patch from PowerShell, it fails wit the error.

&lt;class &#39;AttributeError&#39;&gt;
&#39;str&#39; object has no attribute &#39;items&#39;

Interestingly enough, in Postman, it works.

Regards,

Igor

[Kevin W]

Hi, I am using this sync Script nightly and is working fine, with one exception.

When syncing VM's which have no VMWare Tools installed (as they are a closed linux system, so it's not possible to install VMWare Tools), in Netbox we manually enter the interfaces and IP details, but then on next sync, all this info is removed.

Any suggestions as to how to keep the manually entered interfaces and IP's please?

Regards.

[Anthony Ruhier]

Hi,

If you're interested to import network devices (switches/routers), at Online.net we wrote a tool for that: https://github.com/Anthony25/netbox-netprod-importer

We would be very happy to receive contributions if any feature is missing.

[Frenshie :)]

Hi Joseph,

Thank you very much for sharing with us this script. It's working well for us too.

However, I m still fighting with your script to find a simply way to match the "Notes" field of each VMs in vsphere 6.5 with the "comments" field in Netbox please ?

How could I simply add this process please ?

Thank you very much again for your support..

Cheers,

J.