Is it possible to inject custom behaviour into parsing logic (return custom types after parsing not evaluation)?

[Patryk Małek]

I've created an issue on Github (https://github.com/google/cel-go/issues/279) but I believe this place is more appropriate for this type of questions.

I'm working on an internal Go package that would translate user (other services') expressions into Elasticsearch queries.

My goal is to declare parsing logic which would let's say allow something like:

label.id == "my-id1" && label.value == 100

or

( label.id == "id1" && label.value == 100 ) || ( label.id == "id2" && label.value == 900 )

So the parsing logic has to be aware that it looks for (in the second example):

* a label with id `id1` *and* `value` of `100`

or

* a label with id `id2` *and* `value` of `900`

What I'm looking for is to generate a query that would do the filtering etc. on the DB side (Elasticsearch) instead of getting all the relevant results (for instance from a particular time range) and then evaluating the filter on those results.

I believe that aligns with usage of cel in other projects like those mentioned on [1]

> Google uses CEL as the expression component of IAM and Firebase security policies, as well as within Istio Mixer configs.

> 

> Cloud IAM Conditions

> Firebase Rules

> Istio Mixer

Can I achieve this with cel? If so then can I do that?

[1] https://opensource.google/projects/cel

[Tristan Swadell]

Hi Patryk,

Taking a look at the Elastic Search Query DSL, it looks like the structure of the DSL is an abstract syntax tree, but that there is also a human readable query form which can be parsed to the JSON-like representation.

The CEL parse step generates a cel.Ast result whose .Expr() function will return the underlying protobuf representation of the abstract-syntax tree. You can either walk the AST and build a Elastic Search query string format or a JSON-like DSL directly.

It might be useful to look at the cel.AstToString implementation which lives in parser/unparser.go. This is an example of how the Ast is walked in order to produce a human-readable output. I could imagine the same or similar Ast walk being used to generate query strings or JSON-like query DSLs for Elastic Search.

CEL generates type information during the env.Check call. It is possible to make limited inferences post-parse purely based on the function call operator name, though this doesn't ensure type-agreement of the variables. 

It sounds like you mostly need a way to translate CEL to Elastic Search, but if you need the type information I'd recommend checking the expression as well.

-Tristan

--
You received this message because you are subscribed to the Google Groups "CEL Go Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cel-go-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cel-go-discuss/17ce4922-1086-4a68-9c34-a2ef9cd0626c%40googlegroups.com.

[Patryk Małek]

Hi Tristan,

Thanks for replying.

So what I'd like to do is to generate JSON-like representation of Elastic Search query but using an already existing package with object oriented abstractions (https://github.com/olivere/elastic) so it a bit easier.

I need to further investigate what's possible with CEL but from what I understand I'll need to add my own types  with e.g.

env, err := cel.NewEnv(
cel.Types(
types.NewObjectTypeValue("label"),
),
)

And then fulfil appropriate interfaces like for instance Int does https://github.com/google/cel-go/blob/master/common/types/int.go#L32 and define my own return values etc. assuming I'd for instance want to return a Bool Query with `must` clause when an expression with && operand is defined for lhs and rhs being another both Bool Query, correct?

Regards,

Patryk

On Tuesday, November 5, 2019 at 6:17:51 PM UTC+1, Tristan Swadell wrote:

Hi Patryk,

Taking a look at the Elastic Search Query DSL, it looks like the structure of the DSL is an abstract syntax tree, but that there is also a human readable query form which can be parsed to the JSON-like representation.

The CEL parse step generates a cel.Ast result whose .Expr() function will return the underlying protobuf representation of the abstract-syntax tree. You can either walk the AST and build a Elastic Search query string format or a JSON-like DSL directly.

It might be useful to look at the cel.AstToString implementation which lives in parser/unparser.go. This is an example of how the Ast is walked in order to produce a human-readable output. I could imagine the same or similar Ast walk being used to generate query strings or JSON-like query DSLs for Elastic Search.

CEL generates type information during the env.Check call. It is possible to make limited inferences post-parse purely based on the function call operator name, though this doesn't ensure type-agreement of the variables. 

It sounds like you mostly need a way to translate CEL to Elastic Search, but if you need the type information I'd recommend checking the expression as well.

-Tristan

On Tue, Nov 5, 2019 at 6:35 AM Patryk Małek <patry...@nauto.com> wrote:

I've created an issue on Github (https://github.com/google/cel-go/issues/279) but I believe this place is more appropriate for this type of questions.

I'm working on an internal Go package that would translate user (other services') expressions into Elasticsearch queries.

My goal is to declare parsing logic which would let's say allow something like:

label.id == "my-id1" && label.value == 100

or

( label.id == "id1" && label.value == 100 ) || ( label.id == "id2" && label.value == 900 )

So the parsing logic has to be aware that it looks for (in the second example):

* a label with id `id1` *and* `value` of `100`

or

* a label with id `id2` *and* `value` of `900`

What I'm looking for is to generate a query that would do the filtering etc. on the DB side (Elasticsearch) instead of getting all the relevant results (for instance from a particular time range) and then evaluating the filter on those results.

I believe that aligns with usage of cel in other projects like those mentioned on [1]

> Google uses CEL as the expression component of IAM and Firebase security policies, as well as within Istio Mixer configs.

> 

> Cloud IAM Conditions

> Firebase Rules

> Istio Mixer

Can I achieve this with cel? If so then can I do that?

[1] https://opensource.google/projects/cel

--
You received this message because you are subscribed to the Google Groups "CEL Go Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cel-go-...@googlegroups.com.

[Jim Larson]

Hi, Patryk

The easiest way to add new types to CEL is to import a protocol buffer definition, then add whatever functions you'd like to operate on data of the new type.  You don't need to access the internals of the evaluator, so following common/types/int.go is unnecessary.  For instance, the label type can simply be a protocol buffer message with fields "id" and "value".

As Tristan said, it looks like what you want to do is walk the AST returned by the Parse() or Check() calls to translate the CEL expression to a semantically equivalent Elastic Search query.  You probably want to restrict your translator to only cover the subset of CEL that you actually use.

We'd be happy to answer any questions you have about the details of CEL semantics.

Jim

To unsubscribe from this group and stop receiving emails from it, send an email to cel-go-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cel-go-discuss/1dd07df6-1b70-4360-ab45-f3517eb3af66%40googlegroups.com.

[Patryk Małek]

Hi Jim,

Ok I'll take a look at the logic in unparser.visit() to get more familiar with the code.

But to be clear: what you're suggesting is to both add types by importing protocol buffers definitions (mostly to allow "types" to have particular fields that I want to allow) *and* to walk the tree returned from Parse() or Check()?

I'm not sure I'm asking the correct question at this level of understanding but how do I refer to my types in the logic similar to what parser/unparser.go does? Do I hardcode it in the parsing logic is instead of adding some custom code to the types themselves when creating cel.Env?

Regards,

Patryk

[Tristan Swadell]

Hi Patryk,

My understanding of the Elastic Search API is limited, so we're trying to map the capabilities of two systems in real-time.

Maybe you could give the example of the input CEL and the output Elastic Search and we can work from there?

As general information for how to use CEL, the following would map the environment you'd want to configure for the original example:

env, _ := cel.NewEnv(

  cel.Declarations(

    decls.Ident("label.id", decls.String, nil),

    decls.Ident("label.value", decls.Int, nil),

  ),

)

parsed, iss := env.Parse(`label.id == 'my-id' && label.value == 100`)

// check for parse issues

checked, iss := env.Check(parsed)

// result type ...

// checked.ResultType() == decls.Bool

// reference / type map ..

// ce := cel.AstToCheckedExpr(checked)

// types of each expression in the AST. 

// tm := ce.GetTypeMap()

-Tristan

To view this discussion on the web visit https://groups.google.com/d/msgid/cel-go-discuss/CAEddPXwogHwzq2MDxsZFtu%3D76cVOP_%3DHv52gvo24UVUJau%3DKRg%40mail.gmail.com.

[Patryk Małek]

Hi Tristan,

Please take a look at a simplified code that I hope will depict what I'm aiming for:

package main

import (

"encoding/json"

"flag"

"log"

"github.com/olivere/elastic"

"github.com/tidwall/pretty"

)

var queryFlag = flag.Int("query", -1, "Specify which query to parse and print")

func main() {

flag.Parse()

var q elastic.Query

switch *queryFlag {

case 1:

q = query1()

case 2:

q = query2()

default:

log.Fatalf("Unknown ID")

}

src, err := q.Source()

if err != nil {

log.Fatalf("Source error %v", err)

return

}

b, err := json.Marshal(src)

if err != nil {

log.Fatalf("Source marshal error %v", err)

return

}

log.Printf("Source %s", pretty.Pretty(b))

// ast, _ = env.Check(ast)

// Here I'd hope to get an elastic.Query instead of a decls.Bool

// log.Printf("ResultType %v", ast.ResultType())

}

func query1() elastic.Query {

// ast, _ := env.Parse(`label.id == "my-id" && label.value == "my-value"`)

return elastic.NewBoolQuery().Filter(

elastic.NewTermQuery("label.id", "my-id"),

elastic.NewTermQuery("label.value", "my-value"),

)

}

func query2() elastic.Query {

// ast, _ := env.Parse(

// `(label.id == "my-id" && label.value == "my-value") || (label.id == "my-other-id")`)

return elastic.NewBoolQuery().Should(

elastic.NewBoolQuery().Filter(

elastic.NewTermQuery("label.id", "my-id"),

elastic.NewTermQuery("label.value", "my-value"),

),

elastic.NewBoolQuery().Filter(

elastic.NewTermQuery("label.id", "my-other-id"),

),

)

}

Let me know if you need more information from my side.

Regards,

Patryk

On Tuesday, November 5, 2019 at 11:46:03 PM UTC+1, Tristan Swadell wrote:

Hi Patryk,

My understanding of the Elastic Search API is limited, so we're trying to map the capabilities of two systems in real-time.

Maybe you could give the example of the input CEL and the output Elastic Search and we can work from there?

As general information for how to use CEL, the following would map the environment you'd want to configure for the original example:

env, _ := cel.NewEnv(

  cel.Declarations(

    decls.Ident("label.id", decls.String, nil),

    decls.Ident("label.value", decls.Int, nil),

  ),

)

parsed, iss := env.Parse(`label.id == 'my-id' && label.value == 100`)

// check for parse issues

checked, iss := env.Check(parsed)

// result type ...

// checked.ResultType() == decls.Bool

// reference / type map ..

// ce := cel.AstToCheckedExpr(checked)

// types of each expression in the AST. 

// tm := ce.GetTypeMap()

-Tristan

To view this discussion on the web visit https://groups.google.com/d/msgid/cel-go-discuss/1dd07df6-1b70-4360-ab45-f3517eb3af66%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "CEL Go Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cel-go-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cel-go-discuss/CAEddPXwogHwzq2MDxsZFtu%3D76cVOP_%3DHv52gvo24UVUJau%3DKRg%40mail.gmail.com.

[Jim Larson]

Patryk,

Just to be clear, you're looking for some process by which your query1() and query2() code could be replaced by something like:

func query1() elastic.Query {

return celToElastic(`label.id == "my-id" && label.value == "my-value"`)

}

Is that correct?

Jim

To unsubscribe from this group and stop receiving emails from it, send an email to cel-go-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cel-go-discuss/d14d1c13-f11b-4a24-a4d4-b43e61ef4d51%40googlegroups.com.

[Patryk Małek]

Jim,

Yes, something like that.

Regards,

Patryk

[Jim Larson]

Patryk,

Ok, good, I think we understand what you're doing.  Unfortunately, we don't have the expertise in Elastic Search to spell out the exact algorithm, but it seems like you want a specialized pattern-matcher on the CEL AST produced by parsing that recognizes comparisons on the fields of a common variable, joined together with && and ||, and can translate that to the equivalent Elastic Search.

Let's go through your example to see how it would work.

( label.id == "id1" && label.value == 100 ) || ( label.id == "id2" && label.value == 900 )

should become:

 a label with id `id1` *and* `value` of `100`
or

 a label with id `id2` *and* `value` of `900`

You might have to tell the translation which variable to look for, in this case "label".  It would then scan the expression to look for leaf expressions like

label.FIELD COMPARISON LITERAL

and then translate to search clauses like

with FIELD COMPARISON LITERAL

(maybe specializing on the particular comparison) and then translate the && and || to "and" and "or" respectively.

It would probably be too ambitious to translate the full CEL language, but you could handle this specialized subset that your patterns match against.

We'll be able to answer any questions about the meaning of various CEL expressions if the language spec or cel-go implementation don't make that clear.

Jim

[Patryk Małek]

Unfortunately, we don't have the expertise in Elastic Search to spell out the exact algorithm

That's totally understandable.

It would probably be too ambitious to translate the full CEL language, but you could handle this specialized subset that your patterns match against.

Yup that's what I'm looking for (I hope to start small and then add some more operators etc.)

Can you describe in terms of cel-go functions and subpackages what am I looking for there? What exactly do I want to do in terms of cel-go? Do I want to add new types (Tristan has already suggested that this is not what I'm looking for, though)?

Taking into account my previous email am I correct that I still want to parse the AST like it's done in unparser.visit()?

Regards,

Patryk

 

[Jim Larson]

You don't need any functionality from cel-go other than the Env.Parse() to get an Ast, then call Ast.Expr() to get the proto.  The unparser shows one way of walking the proto structure, but you'll want to process it in your own particular way, so don't feel like you need to follow the unparser exactly.

No, you don't need to add new types to the runtime.

[Patryk Małek]

Alright but if I'd like to define that my type (e.g. label type from the example) has an attribute .value that is a string and .number which is an int should I then define those with proto definitions or I'd be better of doing some custom "type checking" when walking the Ast?

[Tristan Swadell]

HI Patryk,

Protos are the easiest way to teach CEL about custom types. However, you could also implement the ref.TypeProvider interface to implement your own type system. 

Here's an example from Istio:
https://github.com/anespo/istio-demo/blob/8b90baa684da305481cf2988b489f35e1a33eb46/mixer/pkg/lang/cel/provider.go

To view this discussion on the web visit https://groups.google.com/d/msgid/cel-go-discuss/CAEddPXxXVZ1aoPWmmc9p%3DTYnMNefabHaAUfH0e0H9C7QosqBmQ%40mail.gmail.com.