Capturing executed node ID during run time

[Dan Kortschak]

I would like to be able to collect the node ID of a CEL AST during run
time. This is to be able to perform coverage analysis of a CEL program.

I can see that cel.CustomDecorator will allow me to inject
instrumentation into a program so that when Eval is called on each of
the interpreter.Interpretables the node's ID can be recorded to later
collate and present as coverage. However, when I do this, I get blocked
because the type I use to record evals is not an
interpreter.InterpretableAttribute and I don't see an obvious way to
implement this interface; all the implementations are internal and
access details of the planner that I do not have access to as an
external package.

How should I go about this?

thanks
Dan

[Dan Kortschak]

On Wed, 2024-07-17 at 22:40 +0000, 'Dan Kortschak' via CEL Go

Following this up for any suggestions. I'll have time available to work
on this coming up soon. Any suggestions would be greatly appreciated.

thanks
Dan

[Tristan Swadell]

Hi Dan,

Is part of the issue that the InterpretableAttribute intercepts some portion of the coverage graph? In general, you should be able to rely on a similar mechanism to the decorator used for doing state tracking: https://pkg.go.dev/github.com/google/cel...@v0.21.0/cel#EvalOption

-Tristan

--
You received this message because you are subscribed to the Google Groups "CEL Go Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cel-go-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cel-go-discuss/5466b760135d497cfca1caf857e828c5e140fdce.camel%40kortschak.io.

[Dan Kortschak]

On Thu, 2024-09-05 at 18:14 -0700, 'Tristan Swadell' via CEL Go

Discussion Forum wrote:
> Hi Dan,
>
> Is part of the issue that the InterpretableAttribute intercepts some
> portion of the coverage graph? In general, you should be able to rely
> on a similar mechanism to the decorator used for doing state
> tracking: 
> https://pkg.go.dev/github.com/google/cel...@v0.21.0/cel#EvalOption
>
> -Tristan

Thanks. That would work, but it's significantly heavier than I'd like
since it keeps all of the states as well, which I don't need. If I
could define the implementation of the interpreter.EvalState, then it
would be fine, but this is essentially the same issue that I had with
the previous approaches that I tried because it happens internally;
there are interfaces that should be able to be used, but the details of
the interfaces make it impossible for me to satisfy them because the
required states are not made available, only being constructed
internally.

Essentially what I'd like to do is this


type coverage struct {
node interpreter.Interpretable
all map[int64]bool
cov map[int64]bool
}

func (c coverage) ProgramOption() cel.ProgramOption {
return cel.CustomDecorator(func(i interpreter.Interpretable)
(interpreter.Interpretable, error) {
c.all[i.ID()] = true
return coverage{node: i, cov: c.cov}, nil
})
}

func (c coverage) ID() int64 {
return c.node.ID()
}

func (c coverage) Eval(a interpreter.Activation) ref.Val {
c.cov[c.node.ID()] = true
return c.node.Eval(a)
}


which I can then use like so


c := coverage{all: make(map[int64]bool), cov: make(map[int64]bool)}
prg, err := env.Program(ast, c.ProgramOption())


and examine the two maps after to get a coverage percentage and
potentially be able to render the source in a meaningful way to guide
testing. But it fails with

failed program instantiation: failed program instantiation: invalid
attribute decoration: {0xc00040cb40 map[] map[]}(main.coverage)



Dan

[Dan Kortschak]

On Fri, 2024-09-06 at 04:53 +0000, 'Dan Kortschak' via CEL Go

Discussion Forum wrote:
>
> Essentially what I'd like to do is this
>
>
> type coverage struct {
> node interpreter.Interpretable
> all  map[int64]bool
> cov  map[int64]bool
> }
>
> func (c coverage) ProgramOption() cel.ProgramOption {
> return cel.CustomDecorator(func(i interpreter.Interpretable)
> (interpreter.Interpretable, error) {
> c.all[i.ID()] = true
> return coverage{node: i, cov: c.cov}, nil
> })
> }
>
> func (c coverage) ID() int64 {
> return c.node.ID()
> }
>
> func (c coverage) Eval(a interpreter.Activation) ref.Val {
> c.cov[c.node.ID()] = true
> return c.node.Eval(a)
> }

I have come to a reasonable approach for this. Though I think there are
some possible sharp edges.

The implementation is at [1]. The concerns that I have are that it is
necessary to have knowledge of all of the possible implementers of the
various interpreter.Interpretable interface since by embedding, I lose
access to methods that are not defined in the embedded interface type.
I defensively check for exported methods to reject cases where the
value supports more than the interpreter.Interpretable in the case that
it is not one of the exported extended interfaces of this type.

I also notice that in this test[2], not all nodes are detected, even
though the OptTrackState option does show that the node was evaluated.
This can be seen by comparing the state dump to the coverage profile.

In the state dump, we see that the "has_x_y_z" node is evaluated, but
the coverage marks it as not having been evaluated.

dump:
> <input>:1:1
> | {
> | ^
> {has_x_y_z: false}
>
> <input>:2:2
> | "has_x_y_z": has(state.?x.?y.z),
> | .^
> has_x_y_z
>
> <input>:2:18
> | "has_x_y_z": has(state.?x.?y.z),
> | .................^
> false

coverage:
> 1: 1.00 (1/1)
> 2: 0.14 (1/7) [3 5 6 7 8 9]
> | "has_x_y_z": has(state.?x.?y.z),
> | ! ! ! !! !

cheers
Dan

[1]https://github.com/elastic/mito/pull/76/files#diff-6b7d41fe7a25250e318c3a936d61d03110061742202c1e725a155902e6bd027b
[2]https://github.com/elastic/mito/pull/76/files#diff-80c6571985eb1fb9088517b18ea28bc54bed1a5ee00f68d8bec20c3fd51d0019