queries about flattening nested values and how to construct struct types

[Dan Kortschak]

Hi,

I found out about CEL from a comment in the Gophers' Slack last week
and thought it would match a use case we have very well, so I'm
noodling around what it can do and how it would fit. So far it looks
very promising.

I've come up against issues that I don't understand how to address
though (illustrated in https://go.dev/play/p/oWApK6T47Px). Ignoring for
a moment whether this is a good idea or not (the CEL expressions will
be user-provided and so adding native functions is likely not an option
for many cases, and this is just exploration), I want to be able to
calculate the Cartesian product of a small set of lists and return a
flattened list of that product. So far I have been able to get the
Cartesian product as nested lists through nested map macros, like so
(expressed as JSON):

[
[
[
[
"2022-03-19T22:40:28.528615063Z",
"random information for first",
[
"1",
"2",
"a",
"b"
],
"1",
"a"
],
[
"2022-03-19T22:40:28.52861847Z",
"random information for first",
[
"1",
"2",
"a",
"b"
],
"1",
"b"
]
],
[
[
"2022-03-19T22:40:28.528621907Z",
"random information for first",
[
"1",
"2",
"a",
"b"
],
"2",
"a"
],
<snip>


I would also like to be able to construct a list struct types of the
result rather than of the arrays that I'm using to play with the
system. Something like this:

[
{
"timestamp": "2022-03-19T22:40:28.528615063Z",
"other": "random information for first",
"numlet": [
"1",
"2",
"a",
"b"
],
"num": "1",
"let": "a"
},
{
"timestamp": "2022-03-19T22:40:28.52861847Z",
"other": "random information for first",
"numlet": [
"1",
"2",
"a",
"b"
],
"num": "1",
"let": "b"
},
{
"timestamp": "2022-03-19T22:40:28.528621907Z",
"other": "random information for first",
"numlet": [
"1",
"2",
"a",
"b"
],
"num": "2",
"let": "a"
},
{
<snip>

This second part I think is just an issue of my ignorance in how to set
up a destination type (it feels like something related to Exercise 6 in
the code lab, but it's not entirely clear to me how I would set this up
for my own types).

thanks
Dan

[Dan Kortschak]

On Sat, 2022-03-19 at 23:40 +0000, 'Dan Kortschak' via CEL Go

Discussion Forum wrote:
> Hi,
>
> I found out about CEL from a comment in the Gophers' Slack last week
> and thought it would match a use case we have very well, so I'm
> noodling around what it can do and how it would fit. So far it looks
> very promising.
>
> I've come up against issues that I don't understand how to address
> though (illustrated in https://go.dev/play/p/oWApK6T47Px). Ignoring
> for
> a moment whether this is a good idea or not (the CEL expressions will
> be user-provided and so adding native functions is likely not an
> option
> for many cases, and this is just exploration), I want to be able to
> calculate the Cartesian product of a small set of lists and return a
> flattened list of that product. So far I have been able to get the
> Cartesian product as nested lists through nested map macros, like so
> (expressed as JSON):
>

Solved: https://go.dev/play/p/VdNSlPUfiy4

1. Implement a flatten extension function to handle the de-nesting.
2. Understand how to import protobuf message types properly.

[Tristan Swadell]

Hi Dan,

Thanks for all of the contributions lately to CEL as well. It means a lot when people who use CEL want to improve it. I hope that you're finding it useful and if there are areas to improve ease of use, I'd be happy to take them as feature requests.

I have a couple of code suggestions which may simplify / optimize your approach:

1. CEL map literals are generally compatible with conversion to protobuf.Struct values. If you replace the `Struct{}` creation with a map `{}`, then you can remove the `cel.Container` reference since you won't be instantiating new protobuf types.

s.map(e, 

   has(e.other) && e.other != '', // inline filter

   e.num.map(v1, e.let.map(v2, {  // cartesian product with map literal

       "timestamp": now, 

       "other": e.other,

       "numlet": e.num+e.let, 

       "num": v1, 

       "let": v2})))

2. Instead of providing a function which returns a new timestamp every time now() is invoked, provide a lazy value

out, _, err := prg.Eval(map[string]interface{}{

    "now": func() { time.Now().In(time.UTC) },

    "s": ...
})

3. For the flatten method, prefer using the `types.NewMutableList(types.DefaultTypeAdapter)` and calling `ToImmutableList()` (new in CEL v0.10.*) on the result of flattenParts as it will avoid a large number of allocations. It's only safe to use the mutable list approach for intermediate calculations:

func flatten(arg ref.Val) ref.Val {
   obj := arg
   l, ok := obj.(traits.Lister)
   if !ok {
     return types.ValOrErr(obj, "no such overload")
   }
   dst := types.NewMutableList(types.DefaultTypeAdapter)
   flattenParts(dst, l)

   return dst.ToImmutableList()
}

4. When iterating, consider using a predicate of `for it.HasNext() == types.True`

As long as the CEL map literals you're constructing only contain string keys, I believe all CEL simple types are convertible to protobuf.Struct since it's a special type that has a native mapping in CEL.

Cheers,

-Tristan

--
You received this message because you are subscribed to the Google Groups "CEL Go Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cel-go-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cel-go-discuss/2e3382adfe201290ff79a0ca82a3f75e2ed1f358.camel%40kortschak.io.

[Dan Kortschak]

Thanks for those, they are very helpful and I think that CEL will be
very useful in what I'm attempting to do.

On Sun, 2022-03-20 at 11:15 -0700, 'Tristan Swadell' via CEL Go

Discussion Forum wrote:
> Hi Dan,
>
> Thanks for all of the contributions lately to CEL as well. It means a
> lot when people who use CEL want to improve it. I hope that you're
> finding it useful and if there are areas to improve ease of use, I'd
> be happy to take them as feature requests.
>

I'm impressed by the quality of the codebase, it makes on-boarding so
much easier when you can ask questions of the implementation and get
sensible answers from the source code.

> I have a couple of code suggestions which may simplify / optimize
> your approach:
>
> 1. CEL map literals are generally compatible with conversion to
> protobuf.Struct values. If you replace the `Struct{}` creation with a
> map `{}`, then you can remove the `cel.Container` reference since you
> won't be instantiating new protobuf types.
> > s.map(e,
> > has(e.other) && e.other != '', // inline filter
> > e.num.map(v1, e.let.map(v2, { // cartesian product with map
> > literal
> > "timestamp": now,
> > "other": e.other,
> > "numlet": e.num+e.let,
> > "num": v1,
> > "let": v2})))
>

This is cool. I think what I was doing wrong previously was not using
quoted strings and failing at that and so moving on. The detour through
google.protobuf.Struct helped me understand that.

The inline filter is a nice touch.

> 2. Instead of providing a function which returns a new timestamp
> every time now() is invoked, provide a lazy value
>
> > out, _, err := prg.Eval(map[string]interface{}{
> > "now": func() { time.Now().In(time.UTC) },
> > "s": ...
> > })
> >

I was unable to get this to work (adding a return value to the func
literal and adding a decls.NewVar("now", decls.Timestamp) to the
cel.Declarations — what am I missing?).

> 3. For the flatten method, prefer using the
> `types.NewMutableList(types.DefaultTypeAdapter)` and calling
> `ToImmutableList()` (new in CEL v0.10.*) on the result of
> flattenParts as it will avoid a large number of allocations. It's
> only safe to use the mutable list approach for intermediate
> calculations:
>
> > func flatten(arg ref.Val) ref.Val {
> > obj := arg
> > l, ok := obj.(traits.Lister)
> > if !ok {
> > return types.ValOrErr(obj, "no such overload")
> > }
> > dst := types.NewMutableList(types.DefaultTypeAdapter)
> > flattenParts(dst, l)
> > return dst.ToImmutableList()
> > }
>

I noticed that type, but wasn't sure how to use it; I can see that the
returned value from (*mutableList).Add does not need to be retained
since it mutates in-place, I think it would be helpful to add this in
the doc comment. Also, I'd suggest that NewMutableList returns a type
that indicates that it has the ToImmutableList() Lister method. I've
tested the following change and it all passes, and makes the code you
have above compile without a type assertion. If you are OK with the
change, I'm happy to send it.

diff --git a/common/types/list.go b/common/types/list.go
index f63aef4..0011a97 100644
--- a/common/types/list.go
+++ b/common/types/list.go
@@ -99,7 +99,7 @@ func NewJSONList(adapter ref.TypeAdapter, l *structpb.ListValue) traits.Lister {
//
// The mutable list only handles `Add` calls correctly as it is intended only for use within
// comprehension loops which generate an immutable result upon completion.
-func NewMutableList(adapter ref.TypeAdapter) traits.Lister {
+func NewMutableList(adapter ref.TypeAdapter) traits.MutableLister {
return &mutableList{
TypeAdapter: adapter,
baseList: nil,
diff --git a/common/types/traits/lister.go b/common/types/traits/lister.go
index f40e7ee..5cf2593 100644
--- a/common/types/traits/lister.go
+++ b/common/types/traits/lister.go
@@ -28,5 +28,6 @@ type Lister interface {

// MutableLister interface which emits an immutable result after an intermediate computation.
type MutableLister interface {
+ Lister
ToImmutableList() Lister
}


Since there is no function signature to match here, this seems
reasonable to change.

> 4. When iterating, consider using a predicate of `for it.HasNext() ==
> types.True`
>

Nice. I was wondering what the idiomatic way to express this was.

> As long as the CEL map literals you're constructing only contain
> string keys, I believe all CEL simple types are convertible to
> protobuf.Struct since it's a special type that has a native mapping
> in CEL.
>
> Cheers,
>
> -Tristan

thanks
Dan

[Tristan Swadell]

Hi Dan,

Happy to have your changes in cel-go to make the MutableLister easier to use. I was shy about exposing it in a usable way since I don't want people to misuse it. It should only ever be used when you're computing an intermediate result that the user can't see or interact with using their own expressions.

This example shows how to use lazy function bindings for things like 'now':.https://go.dev/play/p/E1NDkbtqeht

I got the lazy function signature wrong initially.

I'm glad you're enjoying the library. I think there are some definite areas for improvement in how concepts are surfaced and supported, but at the moment we're focusing on features and semantic correctness knowing that the top-level API will need to consolidate some of these concepts in the 'cel' package a bit better than it does now. Feedback is always welcome. :)

Cheers,

-Tristan

--
You received this message because you are subscribed to the Google Groups "CEL Go Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cel-go-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cel-go-discuss/9b0cdd080a96b55157f8d1cfab9c8c0957070a99.camel%40kortschak.io.

[Dan Kortschak]

On Sun, 2022-03-20 at 15:47 -0700, 'Tristan Swadell' via CEL Go

Discussion Forum wrote:
> This example shows how to use lazy function bindings for things like
> 'now':.https://go.dev/play/p/E1NDkbtqeht
> I got the lazy function signature wrong initially.

Ah, OK. Yes, this subtly different in behaviour to the implementation
that I had since this will give all the structs in the result stream
the same time stamp. Which is a valid goal for probably most uses, but
not what I was attempting to get here.

thanks
Dan

[Tristan Swadell]

Fair enough, generally the benefit of having side-effect free functions is that the behavior is consistently reproducible. That said, if this isn't a goal, then you're free to ignore the suggestion and stick with the `now()` function instead.

-Tristan

--
You received this message because you are subscribed to the Google Groups "CEL Go Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cel-go-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/cel-go-discuss/0eec3a7818419e0ef12451844367c7da8f6bbaee.camel%40kortschak.io.