Using cel-go as a rules engine for JSON objects?

[Philip Gardner]

Hi,

I'm planning on using cel-go in essentially what is meant to be a rules engine for streaming data.  Essentially what I want to do is to take messages off of a Cloud Pub/Sub topic and route those to other Pub/Sub topics based on the outcome of a set of rules. I'd like to use CEL as this would allow team members to contribute new rules without having to learn Golang, so I figured I'd just provide a standard way to declare rules in TOML or something and then load these rules and match them as necessary.

My problem is the data field of a PubsubMessage can have any kind of shape once decoded.  This poses a bit of a problem when I'm trying to create a new environment where I have to declare all the potential fields:

e, err := cel.NewEnv(

//cel.Declarations(decls.NewIdent("event", decls.Dyn, nil)),

cel.Declarations(decls.NewIdent("service", decls.String, nil)),

cel.Declarations(decls.NewIdent("proto", decls.String, nil)),

)

if err != nil {

log.Fatalf("cel.NewEnv: %v", err)

}

Ideally, I could just umarshal the byte slice  into a map[string]interface{} (or into a known struct) and then pass that to a matching cel.Program.

Hopefully that makes sense, I'm pretty excited about this project! Here's a gist of some sample code I have so far that works, but I definitely feel like I'm missing something to implement my end goal in a much simpler fashion.

https://gist.github.com/gaahrdner/13a7c265f90940af1c0da247076589ab

Thanks!

[Jim Larson]

Hi, Philip!

Yes, CEL could be a good fit for your use case.  CEL has a natural mapping from JSON objects to CEL maps, so that sounds like the best way to import the pubsub payload.  I'll have a look at your link and see if I can make more detailed comments.

Jim

--
You received this message because you are subscribed to the Google Groups "CEL Go Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cel-go-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cel-go-discuss/5a5fd870-a071-4bf9-87de-0c1900fc91a0%40googlegroups.com.

[Jim Larson]

Philip,

Your initial guess is very nearly correct.  You can get a slightly more precise declaration of "event" by making it a

decls.NewMapType(decls.String, decls.Dyn)

Based on your code, you also want to have variables "service" and "proto", so your call to program.Eval should look like:

out, _, err := program.Eval(map[string]interface{}{

"event": event,

"proto": "some string",

"service": "another string",

})

where "event" is a map[string]interface{} unpacked from JSON.  The CEL variable "event" will now be a map that you can access in your CEL expression as described in the "JSON Data Conversion" section of https://github.com/google/cel-spec/blob/master/doc/langdef.md

Let us know if you run in to any other snags.

Jim

[Michael Cheng]

Hi Philip,

The open source project I'm working on has very similar use case as yours. Mine is to take JSON pub/sub messages from one source, perform transformations, and then either cause side effects, or re-format and re-route to a destination.  I'm currently using YAML as a specification language so that the "mediation logic" can be custom code. We are still at the early stages. Below is a sample for processing github webhook messages. Our http listener stores the webhook JSON data and http header into a JSON message and publishes it. The framework interprets the YAML, performs a subscription, and calls CEL to perform variable assignment, and evaluation of conditionals.  I'll be happy to discuss more if you find it of interest.

eventTriggers:
  - eventSource: default
    input: message
    body:
      - if: ' has(message.webhook)' # webhook event
        body:
          -  build.repositoryEvent : ' message.webhook.header["X-Github-Event"][0] ' # push, pull_request, tag, etc

Michael

To view this discussion on the web visit https://groups.google.com/d/msgid/cel-go-discuss/CAEhWN%2B8VNUhPektZWALGfTHb3B1iaZea5_SEfMRYonhfy9P7ew%40mail.gmail.com.

[Philip Gardner]

Hi Jim,

Thanks very much for the clarification, The NewMapType looks like what I'm looking for when wrapping up potentially arbitrary JSON blobs. I built another quick test but can't seem to access the arbitrary field names though: https://gist.github.com/gaahrdner/4de545768444cd2c3464a5e77a0255f3

As an aside, if I know the shape of the data beforehand, would it be simpler to create a NewObjectType using https://github.com/golang/protobuf/blob/master/ptypes/struct/struct.proto?

Thanks again for all your help!

To unsubscribe from this group and stop receiving emails from it, send an email to cel-go-...@googlegroups.com.

[Philip Gardner]

Hi Michael,

Your project sounds very interesting! I'd be happy to discuss more about your project, though yours sounds much farther along than mine; feel free to email me though!

Thanks!

Philip

To unsubscribe from this group and stop receiving emails from it, send an email to cel-go-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cel-go-discuss/5a5fd870-a071-4bf9-87de-0c1900fc91a0%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "CEL Go Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to cel-go-...@googlegroups.com.

[Jim Larson]

Philip,

Try removing the single quotes from around your map lookup expressions.  The quotes tell CEL that it's a string, not a map lookup.  I.e.

e1 := "'event.service' == 'ssl'" e2 := "'event.mac'.contains('de')"

should be

e1 := "event.service == 'ssl'" e2 := "event.mac.contains('de')"

Jim

To unsubscribe from this group and stop receiving emails from it, send an email to cel-go-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cel-go-discuss/ea6fa38d-a099-414b-8025-c2e003c8537d%40googlegroups.com.

[Philip Gardner]

Thanks Jim, that did it.

On Thursday, November 7, 2019 at 2:06:48 PM UTC-6, Jim Larson wrote:

Philip,

Try removing the single quotes from around your map lookup expressions.  The quotes tell CEL that it's a string, not a map lookup.  I.e.

e1 := "'event.service' == 'ssl'" e2 := "'event.mac'.contains('de')"

should be

e1 := "event.service == 'ssl'" e2 := "event.mac.contains('de')"

Jim

To view this discussion on the web visit https://groups.google.com/d/msgid/cel-go-discuss/ea6fa38d-a099-414b-8025-c2e003c8537d%40googlegroups.com.

[Leo Rudberg]

Hello all,

Apologies for necroposting, but for anyone else trying to solve this problem in the future, here's a complete, self-contained gist[0] that shows how to convert a JSON-friendly Go struct into the proper format for running in a CEL program. Hope this helps!

[0]: https://gist.github.com/LOZORD/573749028b570e694658d1dd3273e74d

- Leo Rudberg (ljr/LOZORD)

[Tristan Swadell]

Thanks for sharing Leo! I added a brief comment to the gist about some alternative ways to go that offer better performance, but what you've posted is definitely the fastest way to get started. 

If you want to file a feature request on the cel-go repo to make this use case (Go structs as types) something cel-go supports directly, I think it's worth adding.

Cheers,

-Tristan

To unsubscribe from this group and stop receiving emails from it, send an email to cel-go-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cel-go-discuss/d535a5ae-1da5-467a-975a-3bb83647448b%40googlegroups.com.