Facing issues while using builtin functions while executing golang templates

[Ritesh]

We are facing a strange issue within one of our services where randomly one of the docker containers in production out of (N,  N>50)  starts randomly failing on an API while trying to execute a golang template with an error "Err: template: <template name>: executing \"<template name>\" at len: \"len\" is not a defined function".  `len` is a builtin function within golang so am stumped why this error comes.  Some more context

• Golang Version: 1.15
• The same Pod was serving this API and executing the template correctly before this issue started coming
• Parsing the template works fine, only executing fails. We are parsing and executing in sequence.
• After this error was encountered first, all subsequent API calls to the same container failed with the same error
• The same API on all other containers works perfectly fine.  We removed the container from serving prod traffic, and this issue disappeared
• This is the second time the issue is happening, first time we restarted the container and hence weren't able to debug much
•  Binary is stripped of symbol tables and ptrace is not enabled

We have the container running (not serving prod traffic) so if there are any hints on how to debug this issue would appreciate.

[Michael Ellis]

I did a quick search for "is not a defined function".  That message appears once in https://go.dev/src/text/template/exec.go.  It's triggered when findFunction() fails while executing a template.

Hope that's of some use.

[Amnon]

Unrelated. But it may be worth updating to Go 1.19 as Go 1.15 over 2 years old, and no longer supported.

To get an answer to your original question, I would post a code snippet here which reproduces the problem. 
Also, do you know what triggers this error condition?

[Martin Schnabel]

Hi Ritesh,

I am no expert on these matters, but have you thought about defective
memory on this specific machine? I certainly looks very wrong to me and
the first thing i would try is run an extensive memory test. If your
systems memory is faulty, then all bets are off and trying to debug
memory corruptions is a fools errant.

Hope this helps

On 10/19/22 14:20, 'Ritesh' via golang-nuts wrote:
>
> We are facing a strange issue within one of our services where randomly
> one of the docker containers in production out of (N,  N>50)  starts
> randomly failing on an API while trying to execute a golang template

> with an error *"Err: template: <template name>: executing \"<template
> name>\" at len: \"len\" is not a defined function". *`len` is a builtin

> function within golang so am stumped why this error comes.  Some more
> context
>

> * *Golang Version: 1.15*
> * The same Pod was serving this API and executing the template

> correctly before this issue started coming

> * Parsing the template works fine, only executing fails. We are

> parsing and executing in sequence.

> * After this error was encountered first, all subsequent API calls to

> the same container failed with the same error

> * The same API on all other containers works perfectly fine.  We

> removed the container from serving prod traffic, and this issue
> disappeared

> * This is the second time the issue is happening, first time we

> restarted the container and hence weren't able to debug much

> *  Binary is stripped of symbol tables and ptrace is not enabled

>
> We have the container running (not serving prod traffic) so if there are
> any hints on how to debug this issue would appreciate.
>

> --
> You received this message because you are subscribed to the Google
> Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to golang-nuts...@googlegroups.com
> <mailto:golang-nuts...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/9cbda66a-f0ce-4f39-a1a7-4e3dac5ff4d4n%40googlegroups.com <https://groups.google.com/d/msgid/golang-nuts/9cbda66a-f0ce-4f39-a1a7-4e3dac5ff4d4n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Vishal Sharma]

Hi @mb0,

> the first thing i would try is run an extensive memory test.

what tests do you mean? how can I run them? can you point me to a link/video or a list of tools that can help me in doing so? or any other useful link/reference will be great.

Thanks in advance.

[Ritesh]

Hi Michael, 

Thanks for replying, we did identify this as well. Inside `findFunction` it first tries to see if the `function` is a user defined, and if yes returns that. If not then it tries to find it within `builtIn` func map. The `builtIn` funcMap is populated using `sync.Once` and we can clearly see that `len` is a built in function.  The thing that stumps me the most is that,  before the error started appearing , the same code was working fine on the same pod -> ie it was able to find `len` template function before.