IP addresses are not directly linked to devices (*). The relationship goes like this:
device -----< interface -----< IPAddress
Therefore you need to:
1. import the devices;
2. import the interfaces (if they are not already created as part of the device template);
3. import the IP Addresses linked to *interface*.
In order to identify an interface, you need to specify both the *device* and *interface* columns in the CSV (since interface names are only unique to a device).
For VMs, you would use virtualmachine+interface instead of device+interface.
(*) That's not entirely true, as there is also a direct link on device pointing to primary_ip4 and primary_ip6. Therefore it looks like this:
device ----------< interface -----< IPAddress
primary_ip4 -+---------------------' '
primary_ip6 -+-----------------------'
However, you can only point primary_ip4 or primary_ip6 to an IPAddress which is also linked to an interface on the same device. This means you have to import the IPAddress first *before* setting the device primary_ip4/ip6. Since CSV upload doesn't support modifying existing objects (nor uploading device+interface+IPAddress in the same transaction), you cannot do this via CSV.