Redis replication for redundant primary mail servers

[Matt Reeve]

Hi,

I run two postfix mailservers on different sites, both are basically identically configured apart from the "myhostname" setting. They deliver via dovecot's LMTP and dovecot replication is in use to keep the mailboxes on both sites in real-time sync. Both have an external MX record pointing to them at equal priorities, and interally either server is used for imap, smtp and submission  requests.

The goal is to have an active/active mail system in the event that one site goes down, i.e. both are primary mail exchangers, no backup MX is in place. Volumes are very low as these are just home mail severs.

My question is how should redis be configured in this instance? At the moment I have followed this guide https://rspamd.com/doc/tutorials/redis_replication.html and have 3 redis instances on each site, one replicated one for bayes, one replicated for fuzzy and one that is NOT replicated for the rest of the redis cache. But this doesn't seem right, as each site needs to have access to all cached items so that it can correctly score replies and grey listed items etc. I believe, as it's random as to which of the two mail exchangers will pick up any given mail.

Should I also be replicating the main redis cache in my case?

[Alexander Moisseev]

The guide you are referencing to is an example of Redis replication for distributed separate mail servers. Replication of the main Redis instance database doesn't make sense in that scenario.
I'd consider your setup as a cluster of Rspamd servers. In your case replication of the main Redis database is a good idea. Of course, Rspamd server on the Redis slave side should write into the master Redis database.

[Matt Reeve]

Thanks for your reply. t does lead to a follow up question though - in the event that the server that is acting as the redis master is the one that has failed, the other one will obviously not be able to write any updates to redis, though it will be able to read from its own slave. Will rspamd continue to operate correctly on this server, just with reduced capabilities?

[Alexander Moisseev]

On 5/23/2018 9:46 AM, Matt Reeve wrote:
> Thanks for your reply. t does lead to a follow up question though - in the event that the server that is acting as the redis master is the one that has failed, the other one will obviously not be able to write any updates to redis, though it will be able to read from its own slave. Will rspamd continue to operate correctly on this server, just with reduced capabilities?
>

Yes, it will continue to operate with reduced functionality.

If you expect really long master downtimes, I have the following idea (just an idea, I've never tried):

You can disable read-only mode on the Redis slave and configure Rspamd on the slave side so it write to the master with high priority and to the slave with low priority. Normally it will write to the master and directly to the slave when the master is unavailable.

The data written to the slave will be discarded on the slave resynchronization as soon as connection with the master reestablished.

[Matt Reeve]

That's a really interesting idea; this is the master-slave functionality in the upstreams documentation you're talking about, something I'd not really looked at before. I'll set this up in the coming days and report back.

[Matt Reeve]

I haven't got this working correctly yet, in fact I'm having a more fundamental problem even before looking at the replication side of things. Even when I point both servers to use the same redis cache on one of my servers, they appear to continue to act completely independently. I think this is because they are not sharing the same history entries:

redis-cli keys "*"

1) "ip_score"

2) "rrz48q8ms3jw94in4zsmq3"

3) "rs_historylondon_zst"

4) "rrgx4zizhgtz37oym11iwe"

5) "rgm56swzmshrokrpmbdmobu"

6) "rs_history"

7) "rro5ygewx3sfxeggt18kbp"

8) "rgb1dy8i4r7mf9pg898y9gc"

9) "rs_historyxu4_zst"

One server's hostname is "xu4" and the other is "london". It looks like they are maintaining separate histories in lines 3 and 9. Is there a way to make both servers use the same history?

[Alexander Moisseev]

On 6/4/2018 11:46 AM, Matt Reeve wrote:
> I haven't got this working correctly yet, in fact I'm having a more fundamental problem even before looking at the replication side of things. Even when I point both servers to use the same redis cache on one of my servers, they appear to continue to act completely independently.

Do they share common greylisting, ip_score, ratelimit, e.t.c. ?

I think this is because they are not sharing the same history entries:
>

Rspamd doesn't use history for mail processing. Consider it like a log.

> redis-cli keys "*"
> 1) "ip_score"
> 2) "rrz48q8ms3jw94in4zsmq3"
> 3) "rs_historylondon_zst"
> 4) "rrgx4zizhgtz37oym11iwe"
> 5) "rgm56swzmshrokrpmbdmobu"
> 6) "rs_history"
> 7) "rro5ygewx3sfxeggt18kbp"
> 8) "rgb1dy8i4r7mf9pg898y9gc"
> 9) "rs_historyxu4_zst"
>
> One server's hostname is "xu4" and the other is "london". It looks like they are maintaining separate histories in lines 3 and 9. Is there a way to make both servers use the same history?
>

No, but if you need to view aggregated history and throughput statistics from both of your servers in WebUI, you can configure cluster: https://rspamd.com/doc/configuration/options.html#neighbours-list .

[Matt Reeve]

Yes, my mistake, they are sharing the data - I had misinterpreted what I looked at. They are certainly sharing greylisting and tracking of replies at least, I don't know how to check ip_score and ratelimit specifically.

However, the master-slave arrangement does not seem to be working for the redis server. Specifically when I shutdown redis on the server that I have configured as the master, rspamd is giving an error when trying to write the history. It's like it's completely ignoring the slave server.

Here is my redis.conf:

password = "xxxxxxx";

write_servers = "master-slave:debian,localhost";

read_servers = "localhost";

When the redis server on the host "debian" is switched off, but it still running on the localhost, rspamd logs this:

2018-06-20 16:27:19 #3430(normal) <494a5f>; lua; history_redis.lua:97: got error Connection refused when writing history row: no value