Support for fields of IP type?

[Christos Vontas]

Hi there,

any plans to add fields of IP type in the future?

If I wanted to start working on it, would you have any pointers for me? 

Maybe look into Lucene and follow the geo paradigm?

Thanks,

Chris

[Marty Schoch]

We have no immediate plans, but it seems like a reasonable addition.

I have not looked at how ES/Lucene implement it, but IPv4 addresses and CIDR style searches could probably be reasonably well supported by our existing numeric fields.  Unfortunately they only support 64-bits, so that wouldn't work for IPv6.  Probably my first instinct would be to introduce a new field type that can support wider numbers, say up to 128bits, and then in the Index Mappings, add support for treating string fields as "IP" type, and do the conversion there (much like we do for dates already).

marty

--
You received this message because you are subscribed to the Google Groups "bleve" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bleve+unsubscribe@googlegroups.com.
To post to this group, send email to bl...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bleve/ed2c9064-4631-47da-a4a9-8e9ee2951c96%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Christos Vontas]

Here is what ES/Lucene is doing:

- latest version is using k-d trees to store values instead of the older trie data structure, which is the paradigm that Bleve seems to be following, so their current implementation does not apply

- for older versions I can not find any signs for storing BigIntegers or IPs using PrefixCoded byte arrays

I assume that if I try to store an IP (which is already a []byte) without prefix coding I'll break compatibility (from comment in prefix_coded.go) or I won't get the benefits of Precision Steps is that correct?

If I expand the prefix encoding up to 128bits will that make sense?

Alternatively, I could reuse the existing prefix_coding methods by splitting an IPv6 in four uint32 parts (most methods use int64 internally so we will loose precision there if split in two). In that case would CompositeField be the best way to treat 4 uint32 as one field?

Maybe you have some other idea?

Thanks for the help!

Chris

[Marty Schoch]

Thanks for doing the research on the Lucene/ES side.

Rather than respond to all of this inline, I think its best to start by asking this:

What kinds of queries do you want to run on the indexed IP address data?  Understanding that will guide which path to go down...

marty

--

You received this message because you are subscribed to the Google Groups "bleve" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bleve+unsubscribe@googlegroups.com.
To post to this group, send email to bl...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bleve/9aad8a87-b7b1-4061-a4e1-92bfba6001f0%40googlegroups.com.

[chri...@redsift.io]

Mainly CIDR queries. Also exact matches wouldn’t be bad, but we can work around that with /32 or /128.

Chris

Sent from my iPhone

You received this message because you are subscribed to a topic in the Google Groups "bleve" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bleve/FBiUWJKWMZg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bleve+un...@googlegroups.com.

To post to this group, send email to bl...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bleve/CAE2t7bqaqU-ZFuuCE8bQiE8XiQeQhp-bSgJc6PXh3%3DnW5YrX7Q%40mail.gmail.com.

[Christos Vontas]

Hi Marty!

Happy New Year!

You left me hanging a bit there, still waiting for your comments.

In the meantime I have opened a PR taking a stab at this, reusing preexisting components.

Looking forward to your reply.

Regards,

Chris

To unsubscribe from this group and all its topics, send an email to bleve+unsubscribe@googlegroups.com.

To post to this group, send email to bl...@googlegroups.com.

[Marty Schoch]

Yeah sorry for the long delay, I've been busy working on the new index scheme and traveling over the holidays.

I did look at the PR.  I think the reuse of existing fields is clever, but not really consistent with the rest of bleve.  I think it would be better to create a new IP field type (in bleve/document package) which supports a 128-bit number directly in a single field.  Then support for IPv4 is handled in the same way as lucene:

"IPv4 addresses are converted to IPv4-Mapped IPv6 Addresses: indexing 1.2.3.4 is the same as indexing ::FFFF:1.2.3.4."

The 128-bit field would work just like our existing numeric field (multiple terms indexed for a single value).  Then we support range queries on the 128-bit field, and CIDR queries are just bit-prefix queries which get converted to range queries.

marty

To view this discussion on the web visit https://groups.google.com/d/msgid/bleve/fcceb5c6-b0ce-4174-a814-86ac70e2bd81%40googlegroups.com.

[Amnon BC]

Hi @marty and all.

Happy New Year!

I have resurrected the PR, following you advice to create a native IP field type.

The new version is in 
https://github.com/blevesearch/bleve/pull/1536

Please let me know if this is what you were thinking. Once I have the basic approach in a form you 
are happy with, I'll extend it to add support for ipv6.

Thanks,
Amnon

Chris

To unsubscribe from this group and stop receiving emails from it, send an email to bleve+un...@googlegroups.com.

To post to this group, send email to bl...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bleve/9aad8a87-b7b1-4061-a4e1-92bfba6001f0%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "bleve" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bleve/FBiUWJKWMZg/unsubscribe.

To unsubscribe from this group and all its topics, send an email to bleve+un...@googlegroups.com.

To post to this group, send email to bl...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bleve/CAE2t7bqaqU-ZFuuCE8bQiE8XiQeQhp-bSgJc6PXh3%3DnW5YrX7Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "bleve" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bleve+un...@googlegroups.com.

To post to this group, send email to bl...@googlegroups.com.

[Amnon BC]

I have added ipv6 support to the PR - basically by normalising all IPs and searches to ipv6.

- Amnon