customize sidekiq connection pool configuration

723 views
Skip to first unread message

shama anjum

unread,
May 25, 2021, 6:21:14 PM5/25/21
to Sidekiq

Hi

In our application sidekiq configuration uses the default connection pool. 

Can we make it configurable so that it can be increased as the number of concurrent jobs increases.

Or is it recommended to have default connection pool.

We are currently seeing redis connection timeouts. Would it help to customize connection pool configuration? If yes, how do we do it?

just fyi - We are currently using sentinel set up for redis

Best regards,

Shama.

Mike Perham

unread,
May 25, 2021, 6:28:31 PM5/25/21
to sid...@googlegroups.com
Sidekiq will respond to RAILS_MAX_THREADS to control concurrency so use that:

RAILS_MAX_THREADS=10 bundle exec sidekiq

And in config/database.yml:

pool: <%= (ENV[“RAILS_MAX_THREADS”] || 5).to_i %>

Now the concurrency and database pool size will stay in sync.

Mike

--
You received this message because you are subscribed to the Google Groups "Sidekiq" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sidekiq+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sidekiq/5a576274-c958-442e-bcac-96be8d8d1a69n%40googlegroups.com.
--
Mike Perham, CEO — Contributed Systems

shama anjum

unread,
May 25, 2021, 6:39:31 PM5/25/21
to Sidekiq

We have different values of concurrency per job process. On each server we have:

1 x 30 concurrency

5 x 25 concurrency

3 x 10 concurrency

2 x  8 concurrency

1 x  5 concurrency

2 x  4 concurrency

 

That is a total of 214 possible concurrent jobs across 6 servers, that is 1284.

How do we configure?

can we configure different RAILS_MAX_THREADS value for each sidekiq init ?

Mike Perham

unread,
May 25, 2021, 6:48:20 PM5/25/21
to sid...@googlegroups.com
I will let you handle the devops work, good luck!

shama anjum

unread,
May 25, 2021, 7:02:15 PM5/25/21
to Sidekiq
sorry may be I mislead you with those numbers. 
My point was to ask that in our application, there is no sidekiq.yml, I have different init files for different sidekiq processes with different values of concurrency. 
So can I use RAILS_MAX_THREADS instead of -c option in my init files? If yes what should the pool value be configured to in database.yml?
Do I also need to make changes in sidekiq.rb sidekiq client configuration as mentioned here - https://github.com/mperham/sidekiq/issues/3806

shama anjum

unread,
May 25, 2021, 7:13:31 PM5/25/21
to Sidekiq
Also does it help to use redis connection pool configuration in sidekiq.rb for server and client config? Or leave it use default size based on concurrency? Please suggest.
config.redis = ConnectionPool.new(size: 25) { Redis.new }

Mike Perham

unread,
May 25, 2021, 7:15:16 PM5/25/21
to sid...@googlegroups.com
Don’t configure the redis pool sizes at all. Let Sidekiq do it.

shama anjum

unread,
May 25, 2021, 7:18:05 PM5/25/21
to Sidekiq
Thank you. Really appreciate the quick response.
So for my case do you recommend using RAILS_MAX_THREAD ?

Mike Perham

unread,
May 25, 2021, 7:19:52 PM5/25/21
to sid...@googlegroups.com
I gave you a tip. You are free to use it or not to solve your issue.

shama anjum

unread,
May 25, 2021, 7:32:11 PM5/25/21
to Sidekiq

ok I will try and see if it resolves my issue. 
Right now we have pool:60 in our database.yml
so if I go with RAILS_MAX_THREAD what should my pool config be?

Mike Perham

unread,
May 25, 2021, 7:34:24 PM5/25/21
to sid...@googlegroups.com
I’ve already explained the pool config.

shama anjum

unread,
May 25, 2021, 7:43:58 PM5/25/21
to Sidekiq
yeah you mentioned this
pool: <%= (ENV[“RAILS_MAX_THREADS”] || 5).to_i %>

but my question is each of my sidekiq process has different concurrency,so the each initializer will have different RAILS_MAX_THREAD value if I understood the usage correctly. In that case what should the pool look like?
I am sorry if this is a dumb question. I am really new to sidekiq and trying to solve an issue assigned to me.

shama anjum

unread,
Jun 16, 2021, 8:05:54 PM6/16/21
to Sidekiq
Hi I need help with usage od RAILS_MAX_THREADS

So I have my sidekiq initializer .init file to start sidekiq like below
function start_sidekiq
{
  bundle exec sidekiq -c 10 \
  -q project_releases -q service_manager -q remote_update -q aggregator \
  -e $RAILS_ENV -P $sidekiq_pidfile $@ >> $sidekiq_logfile 2>&1
}

We have a bunch of other sidekiq initializers with different concurrency values.
We are seeing redis connection timeouts. We want to experiment using RAILS_MAX_THREADS as you recommended. Is this the correct usage. Is it same as -c option for sidekiq concurrency? Is there a difference?

function start_sidekiq
{
  RAILS_MAX_THREADS=10 bundle exec sidekiq \
  -q project_releases -q service_manager -q remote_update -q aggregator \
  -e $RAILS_ENV -P $sidekiq_pidfile $@ >> $sidekiq_logfile 2>&1
}

Mike Perham

unread,
Jun 16, 2021, 11:19:18 PM6/16/21
to sid...@googlegroups.com
They are interchangeable.

Reply all
Reply to author
Forward
0 new messages