Is there a sample using dynamic JSON

[Michael Cheng]

Hi,

I'm planning to use CEL in a project, but can't find an example for my use case. What I want to do is to enable expression parsing on an arbitrary incoming JSON message whose internal structure is unknown at compile time of my Go program.  For example, one message may be:

 {

  "eventType" : "type1",

    "owner" : "ower_id",

  "data" : {

        "attr1": "val1",

        "attr2": "val2",

         ...

   }

}

while another message maybe:

{

  "eventyType" : "type2",

  "repository": "repository_name",

  "data": {

        "attr3": "val3",

        "attr4": "val4",

        "attr5": [ "...", "..." ]

   }

}

I would like to define an incoming message as a variable, and the use it in an expression. For example, the expression may be something like:

    message.eventType == "type1"? message.owner: ""

or

    message.eventType == "type2"? message.data.attr3: ""

I would also like to support pre-defined variables that the user can provide in a file. I will parse this file to define the variables before calling the expression parser. For example, the variables may be:

    owners: [ "owner1", "owner2" ]

It's OK to restrict the variables to JSON supported types.  Once I add owners as a pre-defined variable, the expression can then be:

   message.owner in owners

[Jim Larson]

Hi, Michael!

Just this week we updated the CEL language spec with a section on JSON-to-CEL data conversion, but it doesn't go into specifics of how you do so in a particular implementation:  https://github.com/google/cel-spec/blob/master/doc/langdef.md

To import JSON data in our current implementations, you convert it to a protobuf message of type google.protobuf.Value.  CEL will automatically coerce the variable's value to the appropriate CEL type (double, string, boolean, null, list, or map).  If you want to generate JSON output, write an expression that produces one of these types and export it as a google.protobuf.Value, which can then be written as JSON.

If you need more details on the specifics, then please let us know which CEL implementation you're working with.

Jim

--
You received this message because you are subscribed to the Google Groups "CEL Go Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cel-go-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cel-go-discuss/4a3851df-30d6-4ba1-8080-bdd7cd1a0112%40googlegroups.com.

[Michael Cheng]

Hi Jim,

Thanks for the pointer. I'm using Go and cel-go package.

I am currently experimenting with handcrafting a map[string]interface{}.  If I populate it with only values whose types are JSON comformant, I'm able to define a CEL variable and use it in an expression. Is this an acceptable approach, or do I need to go the protobuf route as you suggest and use google.protobuf.Value.  An example would be appreciated.

Michael

[Tristan Swadell]

Hi Michael, 

Thanks for reaching out. Using a map[string]interface{} as input is definitely the simplest way to go. CEL also treats protobuf.Value messages like JSON, but unless you are writing expressions against proto messages with protobuf.Value fields in them, this probably isn't very useful for your case.  It's also possible to use a custom Go struct with CEL as long as the struct implements the ref.Val interface and it's type is specified as a ref.Type.  For an example of a type that you might want to copy, I would subset / extend / override the common/types/map.go implementation for a specialized Go type. Within the declaration set, you could mark the variable as a decls.Dyn type to indicate that it's JSON and its structure is dynamic.

If you would like richer type-checking, I've been exploring how to use JSON schema within the CEL type-provider for richer static analysis and would be curious to know if this is something you're interested in as well.

-Tristan

p.s. You have the exact right idea with variable definition in the 'owners' example. Nice work!

To view this discussion on the web visit https://groups.google.com/d/msgid/cel-go-discuss/CAE3KxMAbK%2BQ-bDZ4vf15-d%3DD%3Dd%3DMRetQQ0cex7jAZfJSpMoBCg%40mail.gmail.com.

[Michael Cheng]

Hi Tristan,

Thanks for confirming that map[string]interface{} is supposed to work. My project involving processing events whose schema is not known at compile time. So static type checking will not work.  Thanks for the quick response, and I will continue to experiment. 

Michael

[Tristan Swadell]

Glad we could help. If you run into any issues, please don't hesitate to let us know or file a GitHub issue.

-Tristan

[Evan Burns]

Hey Tristan, reviving an old thread here but seemed the most pertinent place to post. I came across this while investigating whether there was any support for JSON Schema with CEL and looks like you had considered pursing this at some point? It's certainly something of interest within my project and I had been considering a round about approach which I was curious to get your thoughts on. 

My use case involves providing the ability for users to define expressions that evaluate, manipulate and extract (for which i'm looking at CEL) API response data from a variety of third party APIs. Each of the APIs supported by the system are registered by providing their OpenAPI specification. This  means we have the JSON Schema definition of their response structures ahead of time. I would very much like to incorporate this in a way such that expressions can be type checked against the associated JSON Schema to minimize runtime errors from malformed expressions. I've been considering a layer in our system that would convert the JSON Schema to a protobuf definition and taking the dynamic proto approach (since we can have new APIs added to the our system dynamically and would not want to re-compile/deploy each time) as outlined at https://github.com/google/cel-go/blob/master/cel/cel_test.go#L567. Presumably this would provide the desired type check safety? Perhaps there's a better approach of having more native support for JSON Schema within CEL?

Thanks & really exciting project! Looking forward to learning more.

Cheers,

Evan

[Tristan Swadell]

Hi Evan,

I actually have implemented support for Open API Schema (similar to JSON Schema, but YAML) in the CEL Policy Templates project. I haven't upstreamed the feature into CEL since it's technically a subset of what CEL can support, but you are welcome to take what's there and combine it with the CEL Go source to more or less accomplish your aims.

Hope that helps!

-Tristan

To view this discussion on the web visit https://groups.google.com/d/msgid/cel-go-discuss/a187da14-4cc9-4b7e-86bc-1fb624670f64n%40googlegroups.com.