django-ipam - Import and Export feature details

[Anurag Sharma]

Hi all,

I am currently working on implementing the export and import feature on the django-ipam app - https://github.com/openwisp/django-ipam/pull/31

As of now the user can -

• Export a list of subnets.
• Export a list of IP Addresses under a particular subnet.
• Import a list of IP Addresses for a particular subnet.

Testing the export feature -

One can easily test the export feauture by just going on the admin interface, and clicking on the "Export Subnets" or "Export IP Address" buttons. The export button for subnet is on the change list view of subnets and the export button for IP addresses is on the change view of any specific subnet.

Testing the import feature -

In order to test it, I have attached a sample csv file, which contains a list of IP Addresses(IPv6) under fdb6:21b:a477::/64 subnet. This would require fdb6:21b:a477::/64 subnet instance to be created on the database beforehand to work.

Thank you,

Anurag Sharma

[Federico Capoano]

Anurag,

a few questions:

• didn't we say we wanted to import an entire subnet from phpipam? We want to be able to move content from an existing phpipam instance into this new module, if we don't achieve this goal this feature won't be very useful
  
• what if we want to import address descriptions as well?
  
• the export format is not re-importable in the application itself, don't you think this wrong? What's the use of developing the export feature if we already have the API which can be used to retrieve data?

To summarize:

• the import format should be compatible with the exported CSV of phpipam
• our export format could be similar to the one used by phpipam
• we need to be able to import an entire subnet and its addresses exported from phpipam
• we need to be able to import a group of addresses into an existing subnet
• we need to be able to import a CSV exported from django-ipam itself, there's no point in having an export format which cannot be reimported

This feature needs to be designed more carefully.

Federico 

[Anurag Sharma]

Here is an exported xls file from PHPipam. 

During export, we get an option to select the fields as well, I have only exported the IP address value and descriptions, in order to be compatible with django-ipam models.

[Federico Capoano]

The first things that come into my mind are the following:

- the system should convert any xls to csv automatically - but this can be implemented as the last thing, you can convert it manually to CSV for now

- the system should be able to create the subnet if it doesn't exist

- the system should be able to import the same file again, if no additions were made to the file it will simply do nothing. If new addresses were added, the system will add the new addresses to the existing subnet

- our export format should be very similar to that one and the import feature should recognize the additional fields and import those too, the additional fields should be shown after (on the right) the most relevant fields (ip addresses, subnets, names, descriptions, daes and id last)

- keep in mind that since IDs are present in the exported CSV generated by django-ipam, when re-importing into django-ipam, the system must use the same ID

Is this clear? Does it sound like a good implementation to you?

It does sound good enough to me.

Keep in mind that the best thing to do now is write failing tests for the main cases I described and then implement the code which makes those tests pass.

Avoid testing manually or you will waste a lot of time.

Federico

On Thu, Jul 12, 2018 at 1:14 PM Anurag Sharma <anssh...@gmail.com> wrote:

Here is an exported xls file from PHPipam. 

During export, we get an option to select the fields as well, I have only exported the IP address value and descriptions, in order to be compatible with django-ipam models.

--
You received this message because you are subscribed to the Google Groups "OpenWISP" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openwisp+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Marco Giuntini]

Hi All,

I did some test on phpAdmin on the import/export feature and it doesn't work properly there as well:

- The export file is not in the same format of the import one. If you want to use the import feature you need to download the template fill it in and use it.

- The import of list ip address in a new subnet seems it doesn't work.

- the system should convert any xls to csv automatically - but this can be implemented as the last thing, you can convert it manually to CSV for now

We can use only CVS file for now.

 

- the system should be able to create the subnet if it doesn't exist

True.

 

- the system should be able to import the same file again, if no additions were made to the file it will simply do nothing. If new addresses were added, the system will add the new addresses to the existing subnet

Maybe we can implement some check to show the differences before submit the changes. What's happen if I want remove a cert amount of ip address?

 

- our export format should be very similar to that one and the import feature should recognize the additional fields and import those too, the additional fields should be shown after (on the right) the most relevant fields (ip addresses, subnets, names, descriptions, daes and id last)

- keep in mind that since IDs are present in the exported CSV generated by django-ipam, when re-importing into django-ipam, the system must use the same ID

I think that the ID should't be in the export/import file. But the system should be able to check if an IP or Subnet already exist with the fields ipaddress and subnet into the CSV.

Regards,

Marco

[Federico Capoano]

On Thu, Jul 12, 2018 at 5:26 PM Marco Giuntini <marco.g...@gmail.com> wrote:

Hi All,

I did some test on phpAdmin on the import/export feature and it doesn't work properly there as well:

- The export file is not in the same format of the import one. If you want to use the import feature you need to download the template fill it in and use it.

- The import of list ip address in a new subnet seems it doesn't work.

 

We surely should do better than this.

- the system should convert any xls to csv automatically - but this can be implemented as the last thing, you can convert it manually to CSV for now

We can use only CVS file for now.

I think our system will need to handle the xls coming from phpipam, that will make migration very easy. Converting XLS to CSV should be easy.

 

 

- the system should be able to create the subnet if it doesn't exist

True.

 

- the system should be able to import the same file again, if no additions were made to the file it will simply do nothing. If new addresses were added, the system will add the new addresses to the existing subnet

Maybe we can implement some check to show the differences before submit the changes.

This would take too much time to implement and we are behind schedule.

 

What's happen if I want remove a cert amount of ip address?

What do you mean exactly?

 

 

- our export format should be very similar to that one and the import feature should recognize the additional fields and import those too, the additional fields should be shown after (on the right) the most relevant fields (ip addresses, subnets, names, descriptions, daes and id last)

- keep in mind that since IDs are present in the exported CSV generated by django-ipam, when re-importing into django-ipam, the system must use the same ID

I think that the ID should't be in the export/import file. But the system should be able to check if an IP or Subnet already exist with the fields ipaddress and subnet into the CSV.

Having the ID can be useful in many cases and doesn't hurt. Remember we are dealing with UUIDs and not classic mysql auto-increment IDs, so there's virtually no risk of collision when importing.

F.

[Marco Giuntini]

  What's happen if I want remove a cert amount of ip address?

What do you mean exactly?

I mean, what should be the expected behaviour in case of an IP address is present in the django-ipam but not in the import file?  

 

Having the ID can be useful in many cases and doesn't hurt. Remember we are dealing with UUIDs and not classic mysql auto-increment IDs, so there's virtually no risk of collision when importing.

What happen if the import file contains an existing IP with a different UUID or an existing UUID with a different IP? 

If we are going to import an PhpIpam export the UUID should be generated automatically by django-ipam.

Marco

[Federico Capoano]

On Thursday, July 12, 2018 at 9:51:35 PM UTC+2, Marco Giuntini wrote:

  What's happen if I want remove a cert amount of ip address?

What do you mean exactly?

I mean, what should be the expected behaviour in case of an IP address is present in the django-ipam but not in the import file?  

At the moment nothing. Let's just import new addresses and leave existing addresses alone.

 

 

Having the ID can be useful in many cases and doesn't hurt. Remember we are dealing with UUIDs and not classic mysql auto-increment IDs, so there's virtually no risk of collision when importing.

What happen if the import file contains an existing IP with a different UUID or an existing UUID with a different IP? 

If we are going to import an PhpIpam export the UUID should be generated automatically by django-ipam.

BTW: Anurag has given up on the UUID.
I thought it was a good idea but I have no strong feelings about this, but I'll reply to your questions for the sake of making things straight.

If the IP exists with a different UUID, no big deal, the system can ignore it or may issue a warning after the operation is completed.

Existing UUID with different IP is impossible unless artificially fabricated:

UUIDs have 122 bits of entropy so the chance of two random UUIDs colliding is about 10^-37.

If you generate 2^46 UUIDs (approximately 1 petabyte of entropy) the chance of getting a collision is 1 in 50 billion.[1]  You're 170 times more likely to win the jackpot in the US Powerball.[2]

[1]: https://en.wikipedia.org/wiki/Universally_unique_identifier#Collisions

[2]: https://www.lotteryusa.com/powerball/

 

Fed