index in for loop

[Juergen Etzlstorfer]

Hi everyone, 


I am quite new to Jsonnet and using Grafonnet to build my Grafana dashbaords.
What I am currently struggling with is that I can not get the index of the currently processed item when doing this in a "for" loop.


My example:
dashboard.new(...)
.addPanels(
[
environment(e){ gridPos: { h:5,w:6, x:0, y:0} },
for e in std.extVar('stages')
]


In the example above I would like to calculate the x and y values based on the index of the "stages" array to be able to have the panels next to each other. Basically I would like to do something like:
dashboard.new(...)
.addPanels(
[
environment(e){ gridPos: { h:5,w:6, x:index*6, y:0} },
for e in std.extVar('stages')
]


How could this be achieved?


Thanks a lot for your help!


Cheers,
Juergen.

[Brett Viren]

Hi Juergen,

You can maybe find std.mapWithIndex(func, arr) useful.

The passed function is called on each element as func(index, element)
where element is arr[index].

Eg:

$ jsonnet -e 'std.mapWithIndex(function(i,e) e*i, std.range(0,10))'
[
0,
1,
4,
9,
16,
25,
36,
49,
64,
81,
100
]


-Brett.

[juergen.e...@dynatrace.com]

Thanks Brett!

That already helped a lot. 

I have a follow-up question since I can not manage to get my code running correctly. Sorry for the noob question but I am really new to jsonnet ;) 

I have the following fragment as part of my jsonnet template file:

[

row('dev',) { gridPos: { h:5,w:18, x:0, y:0} },

]+

[

response_time(std.extVar('service')+"-"+std.extVar('project')+"-dev") { gridPos: { h:10,w:6, x:0, y:0 } },

throughput(std.extVar('service')+"-"+std.extVar('project')+"-dev") { gridPos: { h:10,w:6, x:6, y:0 } },

error_rate(std.extVar('service')+"-"+std.extVar('project')+"-dev") { gridPos: { h:10,w:6, x:12, y:0 } },

]+

[

row('staging') { gridPos: { h:5,w:24, x:0, y:0} },

]+

[

response_time(std.extVar('service')+"-"+std.extVar('project')+"-staging") { gridPos: { h:10,w:6, x:0, y:0 } },

throughput(std.extVar('service')+"-"+std.extVar('project')+"-staging") { gridPos: { h:10,w:6, x:6, y:0 } },

error_rate(std.extVar('service')+"-"+std.extVar('project')+"-staging") { gridPos: { h:10,w:6, x:12, y:0 } },

]

As you can see it is basically two blocks that start with "row" and only distinguish that I first use "dev" and in the second block I use "staging" as a variable. I want to pass this as an array via std.extVar. The passing of the variable works fine, but I struggle to refactor the given code block into a foreach or mapWithIndex structure. I would envision something like this:

std.mapWithIndex(function(i, env) [

row(env) { gridPos: { h:5,w:18, x:0, y:0} },

]+

[

response_time(std.extVar('service')+"-"+std.extVar('project')+"-"+env) { gridPos: { h:10,w:6, x:0, y:0 } },

throughput(std.extVar('service')+"-"+std.extVar('project')+"-"+env) { gridPos: { h:10,w:6, x:6, y:0 } },

error_rate(std.extVar('service')+"-"+std.extVar('project')+"-"+env) { gridPos: { h:10,w:6, x:12, y:0 } },

]

, std.extVar('stages'));

Unfortunately, this gives me the error: STATIC ERROR: keptn.jsonnet:95:26: expected a comma before next function argument. 

I am happy to share the completely file as well, I really appreciate your help here! 

Thanks and best regards,

Jürgen.

[Brett Viren]

Hi again,

Hmm, I don't spot anything obviously wrong with your example. Knowing
what code is at

keptn.jsonnet:95:26

may be more telling.

My only guess is that....

> , std.extVar('stages'));

...is not producing what you think it is.

Here is, I think, a minimal example of your pattern:

$ jsonnet -e 'std.mapWithIndex(function(i,e) ["in",i,e] + ["out",i,e], std.range(0,1))'
[
[
"in",
0,
0,
"out",
0,
0
],
[
"in",
1,
1,
"out",
1,
1
]
]

If this truly is representative of your code then I think the problem
must be with what pops out of std.extVar(). Especially when using this
function to inject code (instead of just strings) I often have trouble
getting it right on the command line side of the injection. A little
test with just your std.extVar() may be helpful.


If I may give one unsolicited suggestion and just because it is fresh in
my own work: consider using "top-level arguments" (TLA) instead of
std.extVar(). I used to rely on std.extVar() heavily to "inject" data
from the command line but found it rather awkward and its use has two
big issues which TLA overcomes:

- you *must* provide every extVar (no way to set a default so a extVar
may be optional)

- you can not provide an extVar from Jsonnet (no way to reuse
extVar-using Jsonnet from other Jsonnet which may know the proper
extVar values to use).

The only downside with TLA (that I found) is that it requires factoring
existing extVar-using code into a top-level function definition.
Arguably, the effort to do that will almost always make the code far
better but it can still be a small pain to do the work. It tends to
turn big monoliths into many small files.

Because I *just* worked this out for myself yesterday, here is a simple
but full example. The "printf" part is some Bash to form a JSON string
from a matching set of file names.


$ cat junk.jsonnet
function(msg="",foo=[]) [msg]+foo

## make a Jsonnet array from one scalar and a JSON array formed from
## matching file names
$ jsonnet --tla-str msg="hello" --tla-code foo="[ $(printf '"%s", ' *.org) ]" junk.jsonnet
[
"hello",
"README.org"
]

## show call relying on default argument values
$ jsonnet junk.jsonnet
[
""
]

## show use of original function fully inside Jsonnet
$ cat junk2.jsonnet
local junk = import "junk.jsonnet";
junk("goodbye", ["a", "b", "c"])

$ jsonnet junk2.jsonnet
[
"goodbye",
"a",
"b",
"c"
]


Cheers,
-Brett.

[juergen.e...@dynatrace.com]

Hi Brett,

thanks again for helping me out here.

I've uploaded already finished my project and uploaded the file here: https://github.com/keptn-sandbox/grafana-service/blob/master/keptn.jsonnet#L88 

The problematic code starts around line 88. 

As said, I am not able to use the constructs of row, response_time, throughput and error_rate in a loop. The part with std.extVar is working fine. If I pass it via 

jsonnet -J ./grafonnet-lib/ --ext-code stages='["dev","staging"]' --ext-str service=carts --ext-str project=sockshop keptn.jsonnet > gen/keptn.json 

it is working fine and stages is a valid array which I can go through. 

I mainly fight with having the afore mentioned constructs row, response_time, throughput and error_rate as part of the loop.

Really appreciate any help on this!

As a next step I can then take a look at your suggestion to switch to TLA instead of extVar.

Cheers,

Jürgen.

[Brett Viren]

"juergen.e...@dynatrace.com" <juergen.e...@dynatrace.com> writes:

> https://github.com/keptn-sandbox/grafana-service/blob/master/keptn.jsonnet#L88

One of these days, I gotta learn grafonnet!

> The problematic code starts around line 88.
> As said, I am not able to use the constructs of row, response_time,
> throughput and error_rate in a loop. The part with std.extVar is
> working fine. If I pass it via
> jsonnet -J ./grafonnet-lib/ --ext-code stages='["dev","staging"]' --ext-str service=carts --ext-str project=sockshop keptn.jsonnet > gen/keptn.json
> it is working fine and stages is a valid array which I can go through.

Okay. And, yes I can reproduce this command successfully with your
code.

> I mainly fight with having the afore mentioned constructs row,
> response_time, throughput and error_rate as part of the loop.

I may be losing the thread of the problem but looking at the exhaustive
array you have typed out for .addPanels(), I believe it may boil down to
building an intermediate "array of arrays" as you iterate over "stages"
and then flattening that compound array to a single flat array. In
short, using this pattern:

$ jsonnet -e 'std.flattenArrays([[1,2],[3,4]])'
[
1,
2,
3,
4
]

If I'm still on track, you can write a function to produce an array of
row, response_time, throughput and error_rate function calls each
parameterized by the "stage":

local make_one(stage, stagenum) = [
row(stage, std.extVar(....),...) {gridPos:....},
response_time(...),
...
];

and then "map" this function over the "stages" array and flatten the
result:

.addPanels(std.flattenArrays(std.mapWithIndex(make_one, std.extVar('stages'))))


BTW, I don't see how the index to the "stages" array is supposed to get
used. Maybe that original problem is no longer material? If so, you
could simplify the above to use std.map() and remove the "stagenum" arg
to make_one().

-Brett.

[juergen.e...@dynatrace.com]

Brett thank you so much for your help!

I finally got it to work :)

You are right, I got rid of the index completely since I decided to change the layout of the dashboard I want to create and therefore remove the need to calculate some parts coordinates based on the index. 

I now introduced a new construct to make a complete row including the panels:

local make_row(stage) = [

row(stage, std.extVar('service')+"-"+std.extVar('project')+"-"+stage) { gridPos: { h:5,w:24, x:0, y:0} },

response_time(std.extVar('service')+"-"+std.extVar('project')+"-"+stage) { gridPos: { h:10,w:6, x:0, y:0 } },

throughput(std.extVar('service')+"-"+std.extVar('project')+"-"+stage) { gridPos: { h:10,w:6, x:6, y:0 } },

error_rate(std.extVar('service')+"-"+std.extVar('project')+"-"+stage) { gridPos: { h:10,w:6, x:12, y:0 } },

];

And finally I create them via 

dashboard.new(..).addPanels(

std.flattenArrays(std.map(make_row, std.reverse(std.extVar('stages'))))

)

Works pretty well now! 

Thanks again and would love to reconnect to see what you are building with Grafonnet or to showcase what we are doing with it :)

Cheers!

[Brett Viren]

"'juergen.e...@dynatrace.com' via Jsonnet" <jso...@googlegroups.com>
writes:

> I finally got it to work :)

Very cool. I'm glad to hear it.

> Thanks again and would love to reconnect to see what you are building
> with Grafonnet or to showcase what we are doing with it :)

So far I use Jsonnet for configuring scientific software toolkits, data
acquisition sub-systems and as part of a code generation package. In
fact, it is rare for me to see any sort of system and NOT think "this
would be so much better with a Jsonnet interface". So, it makes total
sense to me to use Jsonnet for configuring Grafana. Next time I play
with Grafana I'll refer to your repo and maybe reach out to you for any
help! :)

Cheers,
-Brett.

[juergen.e...@dynatrace.com]

By the way, I've now released the first version of my tool.

It integrates with an open-source control plane for continuous delivery called Keptn and generates Grafana dashboards based on services that are monitored. The data from Prometheus is not only used for the dashboards but also for quality evaluation of the services in the CI/CD pipeline (but that was outside of the scope of my integration and is anyway provided by Keptn already).

In case you are interested please take a look: https://github.com/keptn-sandbox/grafana-service 

Cheers,

Jürgen.