Support for fields of IP type?

131 views
Skip to first unread message

Christos Vontas

unread,
Nov 3, 2017, 1:26:14 PM11/3/17
to bleve
Hi there,

any plans to add fields of IP type in the future?
If I wanted to start working on it, would you have any pointers for me? 
Maybe look into Lucene and follow the geo paradigm?

Thanks,
Chris

Marty Schoch

unread,
Nov 3, 2017, 1:38:05 PM11/3/17
to bl...@googlegroups.com
We have no immediate plans, but it seems like a reasonable addition.

I have not looked at how ES/Lucene implement it, but IPv4 addresses and CIDR style searches could probably be reasonably well supported by our existing numeric fields.  Unfortunately they only support 64-bits, so that wouldn't work for IPv6.  Probably my first instinct would be to introduce a new field type that can support wider numbers, say up to 128bits, and then in the Index Mappings, add support for treating string fields as "IP" type, and do the conversion there (much like we do for dates already).

marty

--
You received this message because you are subscribed to the Google Groups "bleve" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bleve+unsubscribe@googlegroups.com.
To post to this group, send email to bl...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bleve/ed2c9064-4631-47da-a4a9-8e9ee2951c96%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Christos Vontas

unread,
Nov 16, 2017, 1:49:50 PM11/16/17
to bleve
Here is what ES/Lucene is doing:
- latest version is using k-d trees to store values instead of the older trie data structure, which is the paradigm that Bleve seems to be following, so their current implementation does not apply
- for older versions I can not find any signs for storing BigIntegers or IPs using PrefixCoded byte arrays

I assume that if I try to store an IP (which is already a []byte) without prefix coding I'll break compatibility (from comment in prefix_coded.go) or I won't get the benefits of Precision Steps is that correct?

If I expand the prefix encoding up to 128bits will that make sense?

Alternatively, I could reuse the existing prefix_coding methods by splitting an IPv6 in four uint32 parts (most methods use int64 internally so we will loose precision there if split in two). In that case would CompositeField be the best way to treat 4 uint32 as one field?

Maybe you have some other idea?

Thanks for the help!

Chris

Marty Schoch

unread,
Nov 16, 2017, 2:16:07 PM11/16/17
to bl...@googlegroups.com
Thanks for doing the research on the Lucene/ES side.

Rather than respond to all of this inline, I think its best to start by asking this:

What kinds of queries do you want to run on the indexed IP address data?  Understanding that will guide which path to go down...

marty

--
You received this message because you are subscribed to the Google Groups "bleve" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bleve+unsubscribe@googlegroups.com.
To post to this group, send email to bl...@googlegroups.com.

chri...@redsift.io

unread,
Nov 16, 2017, 5:15:04 PM11/16/17
to bl...@googlegroups.com
Mainly CIDR queries. Also exact matches wouldn’t be bad, but we can work around that with /32 or /128.

Chris

Sent from my iPhone
You received this message because you are subscribed to a topic in the Google Groups "bleve" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bleve/FBiUWJKWMZg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bleve+un...@googlegroups.com.

To post to this group, send email to bl...@googlegroups.com.

Christos Vontas

unread,
Jan 3, 2018, 8:54:27 AM1/3/18
to bleve
Hi Marty!

Happy New Year!

You left me hanging a bit there, still waiting for your comments.
In the meantime I have opened a PR taking a stab at this, reusing preexisting components.

Looking forward to your reply.

Regards,
Chris
To unsubscribe from this group and all its topics, send an email to bleve+unsubscribe@googlegroups.com.

To post to this group, send email to bl...@googlegroups.com.

Marty Schoch

unread,
Jan 3, 2018, 4:09:01 PM1/3/18
to bl...@googlegroups.com
Yeah sorry for the long delay, I've been busy working on the new index scheme and traveling over the holidays.

I did look at the PR.  I think the reuse of existing fields is clever, but not really consistent with the rest of bleve.  I think it would be better to create a new IP field type (in bleve/document package) which supports a 128-bit number directly in a single field.  Then support for IPv4 is handled in the same way as lucene:

"IPv4 addresses are converted to IPv4-Mapped IPv6 Addresses: indexing 1.2.3.4 is the same as indexing ::FFFF:1.2.3.4."

The 128-bit field would work just like our existing numeric field (multiple terms indexed for a single value).  Then we support range queries on the 128-bit field, and CIDR queries are just bit-prefix queries which get converted to range queries.

marty

Amnon BC

unread,
Jan 13, 2021, 11:25:11 AM1/13/21
to bleve
Hi @marty and all.

Happy New Year!

I have resurrected the PR, following you advice to create a native IP field type.

The new version is in 
https://github.com/blevesearch/bleve/pull/1536

Please let me know if this is what you were thinking. Once I have the basic approach in a form you 
are happy with, I'll extend it to add support for ipv6.

Thanks,
Amnon


Chris
To unsubscribe from this group and stop receiving emails from it, send an email to bleve+un...@googlegroups.com.

To post to this group, send email to bl...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "bleve" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bleve/FBiUWJKWMZg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bleve+un...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "bleve" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bleve+un...@googlegroups.com.

To post to this group, send email to bl...@googlegroups.com.

Amnon BC

unread,
Jan 15, 2021, 1:18:17 PM1/15/21
to bleve
I have added ipv6 support to the PR - basically by normalising all IPs and searches to ipv6.

- Amnon
Reply all
Reply to author
Forward
0 new messages