How to: Regular expressions lookup table?

[Eliezer Croitoru]

I am working for quite some time with postfix and I wanted to understand how can I implement a lookup table similar to the one postfix has.

I will try to be more specific about the details:

Postfix has the next functionality: http://www.postfix.org/DATABASE_README.html

Which is based on some kind of lookup table compilation using a command like

postmap /postfix/aliases            (local aliasing)

postmap /postfix/header_checks (content filtering)

postmap /postfix/transport      (routing table)

postmap /postfix/virtual    (address rewriting)

And when I am using the file tool to understand what DB post uses I see that it's BDB ie:

$ ls|grep db|xargs -l1 file

generic_maps.db: Berkeley DB (Hash, version 9, native byte-order)

postscreen_access.cidr.db: Berkeley DB (Hash, version 9, native byte-order)

rbl_override.db: Berkeley DB (Hash, version 9, native byte-order)

saslpass.db: Berkeley DB (Hash, version 9, native byte-order)

transport.db: Berkeley DB (Btree, version 9, native byte-order)

Now I am trying to understand how exactly postfix uses these BDB files to store regex?

The answer that I was thinking about was that they are using a key->value storage and
postfix loads all the keys into ram.

I assume that the order of the rules is important.

I am asking since I am not a C expert and every time I sit to read any C code my head starts to spin in circles and twist and GoLang for me is much easier.

Thanks,

Eliezer

 

----

Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: eli...@ngtech.co.il

 

[Konstantin Khomoutov]

On Wed, 01 Jun 2016 14:16:34 +0300
Eliezer Croitoru <eli...@ngtech.co.il> wrote:

> I am working for quite some time with postfix and I wanted to
> understand how can I implement a lookup table similar to the one
> postfix has.

[...]

> Now I am trying to understand how exactly postfix uses these BDB
> files to store regex?

[...]

The question seems to be a manifestation of the classic XY Problem [1]:
you tell us lots of details, but it's hard to see what is your exact
problem.

To being with, the document you've referred to says that Postfix
supports lots of table types with the regexp-based being only one of
them [2]. So let's suppose this is the format you're interested in.
Now the question: what do you want to do?
Do you need to read DB files generated by `postmap` for thar "pcre" map
type yourself?
Or do you want to implement functionality similar to that of Postfix
"pcre" maps in your own program? If yes, then what the use case -- in
moderate level of details?

1. http://meta.stackexchange.com/questions/66377/what-is-the-xy-problem
2. http://www.postfix.org/regexp_table.5.html

[Eliezer Croitoru]

It's pretty simple.
I was looking at the way that postfix does things from couple angles.
First I was interested in understanding how the regex\pcre is being used with a BDB DB in a form of either a hash or btree.
Second is what would be the best way to work with a big list of regex\pcre in Golang as a black\white list?

To clear out the use case and the wanted functionality I will try to be more specific.
I want to run a series of tests against a bunch of regex\pcre as a tool to test a policy.
In this specific case I have a URLs categorizing DB and I am considering to add a functionality of "regex" lists(mostly exceptions).
I have used this DB based on the idea of SquidGuard which creates a hash BDB file that stores values of urls and domains as a key with the value "1", then it run's lots of lookup against the BDB file which is similar to what postfix postmap does.(assuming I understood things correctly)

So my goal is to use\have some kind of persistent storage for regex lists and to be able to test against it every time, and also be able to update it live.
I was thinking about an option to use an iterator like:
iter := db.NewIterator(util.BytesPrefix([]byte("foo-")), nil)
for iter.Next() {
// Use key/value.
...
}
iter.Release()
err = iter.Error()
# from: https://godoc.org/github.com/syndtr/goleveldb/leveldb

But I am not sure if I'm even in the right direction since maybe I will need to recompile regex over and over again.

Eliezer

----
Eliezer Croitoru
Linux System Administrator
Mobile: +972-5-28704261
Email: eli...@ngtech.co.il

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tamás Gulácsi]

AFAIK you'll have to recompile those regexps, yes. Maybe cache them in memory? Or compile them in a separate goroutine, and use them coming from a channel?