max stack frames exceeded on array

[Duologic<jer...@grafana.com>]

Hello

While processing an array with 16K items, I ran into a `RUNTIME ERROR: max stack frames exceeded.` using foldl(). I could reproduce this with this snippet:

local appsAggregate(apps) =
  std.foldl(
    function(acc, app)
      acc {
        [app.appid]+: {
          [app.name]+: [app],
        },
      },
    apps,
    {}
  );

appsAggregate([
  {
    appid: 'id1',
    name: 'yo',
    id: i,
  }
  for i in std.range(0, 497)
])

The runtime error happens when there are 497 items or more in the array, when lowering the range to 496, all is fine. Can someone explain what is happening here?

FWIW, I've created a workaround for this by aggregating with an iterative procedure rather than std.foldl.

Cheers

[Duologic<jer...@grafana.com>]

Hi

This just crossed my path in another context: https://github.com/jsonnet-libs/docsonnet/pull/39

In this issue we're trying to render a jsonnet library for the AWS terraform provider, which has 1K+ resources, I didn't investigate deeply yet but as we're using `std.foldl` here too, I assume this is another instance of the same problem.

I just found https://github.com/google/jsonnet/issues/426 which seems to raise the same concerns and a follow up issue (https://github.com/google/jsonnet/issues/427) discusses ways that can avoid the a big stack.

I wonder if there is something else we can do upstream to improve this.

[Preston Podaril]

I filed this issue with go-jsonnet a while back

https://github.com/google/go-jsonnet/issues/590 

I found regression in the stack usage in std.foldl (we use it in a lot of places in our Jsonnet files) from 0.17 to 0.18. I tested the same with 0.19.1 when it was published but it still remains. Seems like my issue was never looked at, and we only use the go-jsonnet version now, so I don't know if this is something specific to the Go version or not. We have just remained on v0.17 since I found this, though there are new features I wish we could take advantage of in later versions. I just don't want to have to increase our stack numbers to allow version update only to have possibility of some executions still hit problems if it was iterating through enough values (haven't had the time to really test out whether we could increase the stack enough to be safe in all cases or if we would just be accommodating it for now until our needs ended up hitting that raised limit).

Wanted to bring it up here since this seems eerily similar to that issue I reported.

[Preston Podaril]

For additional clarity, this is the reason I don't just want to bump our max-stack up and start using the new versions, because it seems this would probably only get us working for a period time, until we would just need to up it more and more..

[C:\>]$ gojsonnet18.exe -e '{ "a": std.foldl(function(p, c) p{ "test"+: { "t": c }}, std.range(0, 2000), {}) }'

RUNTIME ERROR: max stack frames exceeded.

        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        ...
        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        Field "test"
        Field "a"
        During manifestation

[C:\> ]$ gojsonnet18.exe -e --max-stack 2005 '{ "a": std.foldl(function(p, c) p{ "test"+: { "t": c }}, std.range(0, 2000), {}) }'
{
   "a": {
      "test": {
         "t": 2000
      }
   }
}
[C:\> ]$ gojsonnet18.exe -e --max-stack 2005 '{ "a": std.foldl(function(p, c) p{ "test"+: { "t": c }}, std.range(0, 2002), {}) }'

RUNTIME ERROR: max stack frames exceeded.

        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        ...
        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        <cmdline>:1:41-49       +:
        Field "test"
        Field "a"
        During manifestation

Have you tried increasing the max-stack to see if you can avoid exceeding it? How about trying with different versions of jsonnet? Really curious if this is the same thing I ran into.

Thanks,

Preston

[Jeroen Op 't Eynde]

Interesting that this is a regression. The workaround I posted in the initial post has been working well but probably does not cover for all use cases.

Out of curiosity I ran your example with jrsonnet (rust port) and that worked well:

➜ jrsonnet -e '{ "a": std.foldl(function(p, c) p{ "test"+: { "t": c }}, std.range(0, 2000), {}) }'
{
   "a": {
      "test": {
         "t": 2000
      }
   }
}

--
You received this message because you are subscribed to the Google Groups "Jsonnet" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jsonnet+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jsonnet/81315c70-6417-4fe2-90c7-77d9b17c059fn%40googlegroups.com.

--

Jeroen Op 't Eynde

[JuanJo Ciarlante]

woOT `jrsonnet`, amazing.

Anyhow doing a bit of bisecting, it'd look like max stacks implementation limits indeed:

```

$ jrsonnet -e '{ "a": std.foldl(function(p, c) p{ "test"+: { "t": c }}, std.range(0, 18690), {}) }'
[...OK...]

$ jrsonnet -e '{ "a": std.foldl(function(p, c) p{ "test"+: { "t": c }}, std.range(0, 18691), {}) }'

thread 'main' has overflowed its stack
fatal runtime error: stack overflow
[1]    5462 abort      jrsonnet -e

$ jsonnet -e '{ "a": std.foldl(function(p, c) p{ "test"+: { "t": c }}, std.range(0, 496), {}) }'

[...OK...]

$ jsonnet -e '{ "a": std.foldl(function(p, c) p{ "test"+: { "t": c }}, std.range(0, 497), {}) }'

RUNTIME ERROR: max stack frames exceeded.

[...]

```

BTW `jrsonnet --max-stack <N>` didn't help, tried upping it to 1e6 and moar even.

[Dave Cunningham]

What you're doing there is building an inheritance hierarchy that is 2000 layers deep, which is why it's running out of stack.  The difference between these versions may just be whether they are counting this towards the stack limit.

To view this discussion on the web visit https://groups.google.com/d/msgid/jsonnet/3f6bdcad-ee67-49fc-b03e-5ed87b93b855n%40googlegroups.com.

[Dave Cunningham]

Here's a version of the original example that does not do that:

local appsAggregate(apps) =
{
  acc: {
    [appid]: {
      [name]: [app for app in apps if app.appid == appid && app.name == name]
      for name in std.set([app.name for app in apps if app.appid == appid])
    }
    for appid in std.set([app.appid for app in apps])

  }
};

appsAggregate([
  {
    appid: 'id1',
    name: 'yo',
    id: i,
  }
  for i in std.range(0, 497)
])