Warm restart setup for dummies

[Even Onsager]

My site runs on one webserver and we rely heavily on memcached to make it snappy, to the extent that a reboot will make the site unresponsive for hours. So imagine my joy when I saw the warm restart addition, and the fact that Ubuntu Server 20.04 LTS has a new enough version in its repo.
 
But the wiki left me scratching my head. This is what I have:
 
- The standard apt package for Ubuntu 20.04 (version 1.5.22)
- `-e /tmpfs_mount_memcached/memory_file` in memcached.conf
- `-m 920` in memcached.conf
- `tmpfs /tmpfs_mount_memcached tmpfs nodev,nosuid,size=930M 0` in /etc/fstab (generated by Ansible's mount module)
 
No type of restart (neither of the systemd service nor the server itself) seems to work. After restarting the size of the cache store is 0 and all pages take forever to load. But is it supposed to work like this with tmpfs mounts? I thought tmpfs wasn't meant to survive reboots? Am I misreading the wiki?
 
I'm obviously in way over my head here (I don't even really know what a SIGUSR1 is), so I'd really appreciate some help as to what I'm missing!
 
And thanks for memcached - it's served my site well for years now, and after upgrading to Rails 5 with better caching, we're using it more and more. :)

[dormando]

Hey,

I might have to look at how ubuntu's install works.. it might not be set
up for this.

These are the basic steps for a restart:

1) set up memcached as you did, tmpfs/etc.
2) when you want to stop gracefully, issue a `kill -SIGUSR1 $(pidof
memcached)`
(kill is the command to send signals to a process)
3) start memcached again with the same options, and it will recover its
data.

This will _not_ survive reboots. This will survive software upgrades,
which ubuntu isn't going to do anyway :)

To survive reboots you need a few more steps:

1) once memcached has stopped, copy the files created in
/tmpfs_mount_memcached/ to an actual harddrive somewhere.
2) reboot.
3) copy the datafiles back in place.
4) start memcached again.

For this to be reliable you probably don't want ubuntu to automatically
manage the process (disable auto start/restart in systemd).

You might want to engage some community/chat/something for some systems
admnistration help to get this going. Sounds like you're a bit over your
head :(

good luck,
-Dormando

> --
>
> ---
> You received this message because you are subscribed to the Google Groups "memcached" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to memcached+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/memcached/afb47849-1915-4183-9765-457e7e4bc153o%40googlegroups.com.
>
>

[Even Onsager]

That's extremely helpful, thank you so much for this! I will look into it and test on my staging server. I don't think systemd has ever killed or restarted the process apart from once before I upgraded the RAM, so I'm not too worried about the daily usage. But even systemd supports custom kill signals, so it should be possible to set this up?

Anyway, it's the reboots I'm trying to get to work. I never upgrade apt packages or reboot directly, only with Ansible after kernel upgrades or similar, so I should be able to disable the systemd services (should probably temporarily disable the puma webserver service too) and automate a copy to disk task before the reboot takes place. A good thing with Ansible is that it can automate reboots and continue with more tasks after reboot is complete, so it should be ideal for this scenario. I will post back if I can get it to work, should be interesting for more than me. :)

[dormando]

Absolutely. That's exactly the workflow it's designed for, we just haven't
updated any of the systemd scripts to be more friendly for it.

Also a caveat; there _was_ a bug fixed relatively recently with the
restart code. I don't know if ubuntu backports these. If you use large
objects (> 512k) there's a chance restart won't work sometimes. Worst case
you can probably file a bug with them to backport the patch or upgrade
memcached.

Good luck!

On Wed, 10 Jun 2020, Even Onsager wrote:

> That's extremely helpful, thank you so much for this! I will look into it and test on my staging server. I don't think systemd has ever killed or restarted the process apart from once before I upgraded the RAM, so I'm not too worried about the daily usage. But even systemd supports custom kill signals, so it should be possible to set this up?
>
> Anyway, it's the reboots I'm trying to get to work. I never upgrade apt packages or reboot directly, only with Ansible after kernel upgrades or similar, so I should be able to disable the systemd services (should probably temporarily disable the puma webserver service too) and automate a copy to disk task before the reboot takes place. A good thing with Ansible is that it can automate reboots and continue with more tasks after reboot is complete, so it should be ideal for this scenario. I will post back if I can get it to work, should be interesting for more than me. :)
>

> --
>
> ---
> You received this message because you are subscribed to the Google Groups "memcached" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to memcached+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/memcached/5ec346ab-6977-4996-b573-9d07dd0d4084o%40googlegroups.com.
>

[Even Onsager]

I got it to work! With your great help it wasn't that bad, or maybe I'm a better sysadmin than I'm giving myself credit for. ;)

There were two steps, really. After creating the fstab line, I typed sudo systemctl edit memcached.service. I entered this and saved:

[Service]

KillSignal=SIGUSR1

Then it's a matter of:

0. sudo systemctl daemon-reload

1. stopping and disabling the systemd services for the webserver

2. Stopping and disabling memcached

3. copying the memory_file and memory_file.meta to a safe location on the hard disk

4. rebooting

5. copying the files back to /tmpfs_mount_memcached

6. Starting and enabling memcached with systemctl

7. Starting and enabling the webserver

All of these steps I was able to easily automate with Ansible. Really happy about this! Thanks again!

> To unsubscribe from this group and stop receiving emails from it, send an email to memc...@googlegroups.com.