Multiple thin servers - how to set it up and what are the benefits

[Miroslav Schlossberg]

Hi all,

I've noticed that the thin server process sometimes gets stuck and during that time IPZ website usually reports error 500 (gateway error). I'm currently handling that with the watchdog script that is checking the website on port 3000 and then if stuck, trigger alaveteli restart. But I have stumbled upon this information in changelog notes for version 0.24.0.0 (https://github.com/mysociety/alaveteli/blob/2c2d49d97d880a4d648d1d0b63efba1a02420f70/doc/CHANGES.md) about setting up thin to run as multiple servers:

• To switch to running multiple thin servers with nginx:
  • stop the running processes using service alaveteli stop
  • regenerate your SysVinit daemon file using the instructions at: http://alaveteli.org/docs/installing/manual_install/#thin (but don't restart the site yet!)
  • Edit the upstream alaveteli directive in your /etc/nginx/sites-available/alaveteli_https (or /etc/nginx/sites-available/alaveteli if you are not running your site over SSL) file as per http://alaveteli.org/docs/installing/manual_install/#running-over-ssl so that nginx knows how to use the extra server processes
  • restart your site with service alaveteli start

BTW I have checked system uptime and my load is less than 1 (0.12) on 2-CPU server, and there are no significant CPU or memory usage that would lock a thin server resource.

So in order to investigate if multiple thin servers would be helpful for IPZ issues, could you tell me what are general benefits of running multiple thin servers? Why would one start running multiple thin servers?

p.s. Link in the Changelog that should lead to "#thin" heading does not work: http://alaveteli.org/docs/installing/manual_install/#thin

Can you confirm if these are only changes that need to be done in order to start running multiple thin servers:

A) NGINX site config:

If you have configured thin to use more than one server, you will need to edit the Alaveteli upstream directive in your /etc/nginx/sites-enabled/alaveteli file so that it has a server line for each instance you are running (otherwise nginx will only know how to send requests to the first server process). Each address line needs to have a unique port number - start at 3000 and add 1 for each new process. For example if you started your cluster with 4 servers, the upstream directive should look like this:

upstream alaveteli {
    server 127.0.0.1:3000;
    server 127.0.0.1:3001;
    server 127.0.0.1:3002;
    server 127.0.0.1:3003;
}

B) and this SysVinit configuration:

Using Thin#

Thin is a lighter-weight application server which can be run independently of a web server. Thin will be installed in the application bundle and used to run Alaveteli by default.

Run the following to get the server running:

cd /var/www/alaveteli
bundle exec thin \
  --environment=production \
  --user=alaveteli \
  --group=alaveteli \
  --servers=1 \ <------- I assume this needs to be set to number different than "1"
  start

By default the server listens on all interfaces. You can restrict it to the localhost interface by adding --address=127.0.0.1

The server should have told you the URL to access in your browser to see the site in action.

You can daemonize the process by starting it with the --daemonize option.

Next we’ll actually create a SysVinit daemon to run the application, so stop any thin processes you’ve started here.

Is there anything else to do for multiple thin servers?

Thanks,

Miroslav

Miroslav Schlossberg Code for Croatia k:keybase.io/schlossy | skype:darvvon | t:darvvon | w:https://codeforcroatia.org

Code for Croatia brine o zaštiti Vaše privatnosti. Molimo vas da ograničite dijeljenje svojih osobnih podataka putem elektroničke pošte.

Vodite računa o okolišu. Ne tiskajte ovu poruku ako nije neophodno.

[Miroslav Schlossberg]

Hello,

Any feedback on multiple thin servers would be appreciated.

Cheers,

Miroslav

[Mattias Axell]

I'd love to hear how it goes. Would love to pool resources for Ansible playbook sharing too.

--
You received this message because you are subscribed to the Google Groups "Alaveteli Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alaveteli-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/alaveteli-dev/CAH3C4kteDAh7hKG7XF3ZT846PiTLZbtFdK0%3Dm%2BRJ8UhEpmS2WA%40mail.gmail.com.

[Gareth Rees]

Hey Miroslav,

> Why would one start running multiple thin servers?

The TL;DR is that more servers == ability to handle more concurrent requests [1].

> Can you confirm if these are only changes that need to be done in order to start running multiple thin servers A) NGINX site config B) and this SysVinit configuration:

This was true at the time of writing the documentation, though it's been a long time since it was last tested. I can't see why not though.

We don't use thin here at mySociety, and it's been an eternity since I've used it personally.

> I assume this needs to be set to number different than "1"

Yup. I don't recall how one arrives at a good number. Maybe 1 per CPU core? But that may have been the advice for Mongrel/Unicorn back in the day. You'll have to do some investigation on that.

Sorry that we're of limited help here.

Best,

Gareth

[1] https://stackoverflow.com/questions/13549037/why-would-you-start-multiple-thin-servers

[Miroslav (Code for Croatia, ImamoPravoZnati.org)]

Hi Gareth,

I missed this one - I never got notification on email, not sure why would Google Groups unsubscribe me for the topic.

Thanks for the feedback, will investigate.

Also I think I saw somewhere on Github that you maybe plan to switch from Thin to different server. Found it - it's Puma:

https://github.com/mysociety/alaveteli/pull/7581

If this is planned soon-ish not sure if I should spend time fixing the thin.

Cheers,

Miroslav

[Graeme Porteous]

Yes, We're considering switching to Puma for the next release. Thin hasn't seen much active development other than security patches. Puma is actively being developed and has default web server included with Rails since version 5 and recommended by some cloud hosting providers.

Also we've just added Sidekiq 6 as a background job queue. In Sidekiq 7 (which we can't yet ship due to Redis 6.2 not being available for Debian Bullseye) we can embedded the job queue processor in Puma process which is main reason I opened PR 7581. See:

https://github.com/mysociety/alaveteli/pull/7637/commits/406083485a3d6cd1a9a6f51820247b4810651ef8#diff-24409379bdb75ad446bb4e2c18fd4cced1a263b99a3ff96fc3777d8fd8faeab9

Puma would be configured differently to thin. We will document on alaveteli.org and/or the changelog before the next release.

--
Graeme Porteous
gra...@mysociety.org

To view this discussion on the web visit https://groups.google.com/d/msgid/alaveteli-dev/6d6d15f5-b8ad-4161-a124-ff7e382d0462n%40googlegroups.com.