Typed version of jexp's batch import?

[James Thornton]

Michael -

Is there a version of the batch importer (https://github.com/jexp/batch-import) that handles typed data? The version on GitHub looks like it only handles strings.

Thanks.

- James

[Michael Hunger]

James,

you're right, but it should be pretty simple to add.

I think of either specifying the type in the column name like age:int or in a secondary column below.

Another option that I could think of is to treat the cell contents as json and have the json parser take care of those. And then run the importer with -json

What do you think?

Michael

[Friso van Vollenhoven]

Here's my solution to this:

private static Pattern fpNumber = Pattern.compile("^[-]?\\d+[\\.]\\d+$");
private static Pattern intNumber = Pattern.compile("^[-]?\\d+$");
private static Pattern hexNumber = Pattern.compile("^0x[0-9A-Fa-f]+$");
etc...


And then match properties against the different patterns to figure out what it is. This is computationally a bit expensive, though. You could however do some sampling on the data; e.g. only run the matchers against the first N records and then assume the same type for the rest of the set.




Friso

[Michael Hunger]

Good idea,

sampling makes sense.

And if you assume wrongly your conversion will tell you :)

Michael

[Michael Hunger]

Ok, James, I added the type support.

But with explicit type information on the header row, sampling can lead to properties that should stay strings to be converted to numbers.

Michael

[James Thornton]

On Thursday, August 9, 2012 9:23:57 PM UTC-5, Michael Hunger wrote:

Ok, James, I added the type support.

But with explicit type information on the header row, sampling can lead to properties that should stay strings to be converted to numbers.

Ha! -- that's exactly what I recommended this morning to the person who asked me about it, but you may have beat him to it :)

Thanks Michael, I'll let him know!

- James

 

[Amergin]

First of all, many thanks to Michael for adding the support for primitive types in the importer.

Would it be possible to add typing support for index properties as well? It seems currently there is no such support for index property values.

Also, once numeric properties for indexes are supported, it'd be great to have support for numeric range searches from an index as explained in http://docs.neo4j.org/chunked/stable/indexing-lucene-extras.html