Sidekiq losing environment variables

[Floyd Wallace]

Hi all --

Anyone have any experience with Sidekiq needing to be restarted periodically in order to retain environment variables?

I have an Elastic Beanstalk app (AWS) with Sidekiq v. 5.2.7 that has a background job that fails after a few months running as it loses the configuration/environment variables needed to call out to an external API. [In this case, Shopify and the Shopify api key, password, and shop name]. The app used to be on an EC2 instance but had severe memory leaks.

Is this just "something with Sidekiq" and I need to regularly (every couple of days/weeks) restart Sidekiq as it just loses its ability to read environment variables from the application.yml file? I am running Rails 5.1.7 btw.

I did notice on the EC2 instance that Sidekiq would balloon in memory used, and no easy malloc trick seemed to fix it. Only rebooting seemed to fix the issue. Stopping and restarting Sidekiq did not seem to garbage collect. 

Is the answer just "reboot your server every week, Sidekiq can't run longer than that" ??

If that's the answer that's OK, I just need some real world guidance on this one. Mostly I've run Resque for back end jobs (this one was handed to me by a vendor).

Thanks.

[Mike Perham]

Strange, I’ve never heard of the environment just mysteriously clearing itself. As for memory please read the problems and troubleshooting wiki page. It has links to various resources.

On Apr 2, 2020, at 16:44, Floyd Wallace <flwall...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "Sidekiq" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sidekiq+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sidekiq/53606bc3-28ec-434c-8cfb-c499f52f22d0%40googlegroups.com.

[Floyd Wallace]

Thanks Mike. 

I've read all your posts and comments regarding bloat/leaks, in the interest of brevity I did not note that we tried tuning malloc without any improvements. Have you seen people get success with using jemalloc instead? I have not tried that solution. I had thought that Elastic Beanstalk would solve many of my issues with memory bloat as it rotates and restarts VMs, no joy. Just a day after restarting the whole Elastic Beanstalk cluster of 4 with a code deploy,  Sidekiq is at it again, losing the api keys.

Is it better just to hard code keys into the jobs? I do have a private github repo.

Has anyone used jemalloc on Elastic Beanstalk? Success, failure, any gotchas?

To unsubscribe from this group and stop receiving emails from it, send an email to sid...@googlegroups.com.

[Floyd Wallace]

Replying to myself --

Is it just better to upgrade to Sidekiq Pro? Are there significant advantages in terms of memory bloat (I suspect that might be causing my environment variable issues) since again this morning after a reboot yesterday I get the same issues -- Sidekiq jobs lose their environment variables. I'm happy to upgrade if that will solve the issues.

Anyone seen these bloat issues go away or be reduced with Sidekiq Pro?

[Mike Perham]

Sidekiq Pro has nothing that will help with any memory issues. Sidekiq Enterprise has multi-process mode with Bundler preload and memory monitoring which can definitely help but they are not a silver bullet.  If your app is bloating today, it will bloat with Enterprise. jemalloc can help with bloat a bit but nothing will automatically solve a memory leak. Most important for leaks is to upgrade all of your native gems and Ruby version as that's usually where memory leaks are found.

https://github.com/mperham/sidekiq/wiki/Ent-Multi-Process#bundler-preload

https://github.com/mperham/sidekiq/wiki/Problems-and-Troubleshooting#my-sidekiq-process-is-crashing-what-do-i-do

To unsubscribe from this group and stop receiving emails from it, send an email to sidekiq+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sidekiq/a6448d31-4496-48ab-a47c-711bb1c95852%40googlegroups.com.

--

Mike Perham – CEO, Contributed Systems

Smart, effective open source infrastructure for your apps.

[Floyd Wallace]

Mike --

Those recommendations particularly how to set up Sidekiq on Elastic Beanstalk did the trick so far (fingers crossed). There was also a problem with the Rails Web UI and the Sidekiq instance using different redis addresses randomly when the Sidekiq instance fell into staging not production mode randomly. We fixed that in configuration. So that solved that mystery of missing environment variables. The jobs would sometimes be picked up randomly from the different redis server.

Thanks so much!