Support of Redis Sentinel

[Artyom Aleksandrov]

Hello,

Since Rspamd has redis support and greylist module I decided to migrate from SA and greylistd.
Thanks very much for all who do it possible!

My idea is to install rspamd on each incoming MTA and use Redis cluster as distributed storage.

I just want to ask if Rspamd support Redis Sentinel http://redis.io/topics/sentinel ?
It will be really cool from HA side.

Moreover as I understand correctly it can help with redis configuration because clients with sentinel support can detect which server is master and which is a slave.
So it's not required to specify read/write servers separately.

What do you think about it?

Best regards, Artyom

[petr.s...@economia.cz]

Hello I totaly agree with Artyom.

Redis sentinel is great way ho to handle master failover and it is almost  necessary to have it implemented in HA environment.

The configuration would need the list of sentinels and the name of redis group. From sentinel instance (sentinels can be balanced) you can get info on master and all slaves and in case of failure, the new master is elected and the list of slaves updated.

The implementation should not be that difficult so I wonder why the post was idle for half year already.

Dne čtvrtek 4. srpna 2016 12:04:24 UTC+2 Artyom Aleksandrov napsal(a):

[Vsevolod Stakhov]

1) Redis sentinel support is *NOT* easy
2) There is no need in supporting of Redis sentinel and other similar
systems:

Rspamd has 2 types of Redis data:

1) Dynamic stuff, such as limits, reputation, greylisting, replies etc
2) More valuable stuff, such as statistics tokens or fuzzy hashes

For (1), you have almost same amount of read and write requests,
however, Rspamd can perform sharding of all requests using consistent
hashing and can send read and write requests to different servers.
Hence, to scale this part, you can simply add new M-S pair to the
existing cluster. Moreover, you can loose this data without any
significant influence on spam scanning process.

For (2), you have almost all request being read-only requests. Hence, to
scale this data, you can just add more slaves to the existing
configuration. Master fail won't affect anything: you just stop learning
until master is restored or switched, but there is still no influence on
scanning process.

Therefore, there are no plans to support Redis sentinel.

On 02/01/2017 16:13, petr.s...@economia.cz wrote:
> Hello I totaly agree with Artyom.
> Redis sentinel is great way ho to handle master failover and it is
> almost necessary to have it implemented in HA environment.
>
> The configuration would need the list of sentinels and the name of redis
> group. From sentinel instance (sentinels can be balanced) you can get
> info on master and all slaves and in case of failure, the new master is
> elected and the list of slaves updated.
>
> The implementation should not be that difficult so I wonder why the post
> was idle for half year already.
>
> Dne čtvrtek 4. srpna 2016 12:04:24 UTC+2 Artyom Aleksandrov napsal(a):
>
> Hello,
>
> Since Rspamd has redis support and greylist module I decided to
> migrate from SA and greylistd.
> Thanks very much for all who do it possible!
>
> My idea is to install rspamd on each incoming MTA and use Redis
> cluster as distributed storage.
>
> I just want to ask if Rspamd support Redis Sentinel

> http://redis.io/topics/sentinel <http://redis.io/topics/sentinel> ?

> It will be really cool from HA side.
>
> Moreover as I understand correctly it can help with redis
> configuration because clients with sentinel support can detect which
> server is master and which is a slave.
> So it's not required to specify read/write servers separately.
>
> What do you think about it?
>
> Best regards, Artyom
>

> --
> You received this message because you are subscribed to the Google
> Groups "rspamd" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to rspamd+un...@googlegroups.com
> <mailto:rspamd+un...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/rspamd.


--
Vsevolod Stakhov

[petr.s...@economia.cz]

Hello. Thank you for the answer. We use SA for outgoing antispam. I mean
statistics on how many emails were received from single email address or
how many bounced messages it has sent. This way we currently can detect
corrupted mailboxes.

Dne 2.1.2017 v 17:21 Vsevolod Stakhov napsal(a):

> 1) Redis sentinel support is *NOT* easy
> 2) There is no need in supporting of Redis sentinel and other similar
> systems:
>
> Rspamd has 2 types of Redis data:
>
> 1) Dynamic stuff, such as limits, reputation, greylisting, replies etc
> 2) More valuable stuff, such as statistics tokens or fuzzy hashes
>
> For (1), you have almost same amount of read and write requests,
> however, Rspamd can perform sharding of all requests using consistent
> hashing and can send read and write requests to different servers.
> Hence, to scale this part, you can simply add new M-S pair to the
> existing cluster. Moreover, you can loose this data without any
> significant influence on spam scanning process.

Another M-S pair seems like a nice idea, but it is little bit unclear to me.
If I would have 4 nodes: M1-S1 and M2-S2. So far all nodes are healthy,
everything is fine.
In the moment M1 fails, all writes are performed to M2 but reads are
made from both S1 and S2. So I endup with inconsistency.
Since Redis is very good optimalized with data I did not see the point
to use Redis cluster in our infrastructure and sharding is not necessary
for us.

>
> For (2), you have almost all request being read-only requests. Hence, to
> scale this data, you can just add more slaves to the existing
> configuration. Master fail won't affect anything: you just stop learning
> until master is restored or switched, but there is still no influence on
> scanning process.

If the master fails for let's say 1 hour, the corrupted mailbox can send
big amount of unwanted stuff not reaching the limit. On the big scale it
can even mean the reputation of IPs of our servers can start to
deteriorate. That's why I thought the redis sentinel support would be
welcomed.

[Vsevolod Stakhov]

On 03/01/2017 11:10, petr.s...@economia.cz wrote:
> Hello. Thank you for the answer. We use SA for outgoing antispam. I mean
> statistics on how many emails were received from single email address or
> how many bounced messages it has sent. This way we currently can detect
> corrupted mailboxes.

In Rspamd, you can check the Clichouse module:
https://rspamd.com/doc/modules/clickhouse.html as it provides somehow
similar functionality out of the box.

> Dne 2.1.2017 v 17:21 Vsevolod Stakhov napsal(a):
>> 1) Redis sentinel support is *NOT* easy
>> 2) There is no need in supporting of Redis sentinel and other similar
>> systems:
>>
>> Rspamd has 2 types of Redis data:
>>
>> 1) Dynamic stuff, such as limits, reputation, greylisting, replies etc
>> 2) More valuable stuff, such as statistics tokens or fuzzy hashes
>>
>> For (1), you have almost same amount of read and write requests,
>> however, Rspamd can perform sharding of all requests using consistent
>> hashing and can send read and write requests to different servers.
>> Hence, to scale this part, you can simply add new M-S pair to the
>> existing cluster. Moreover, you can loose this data without any
>> significant influence on spam scanning process.
> Another M-S pair seems like a nice idea, but it is little bit unclear to me.
> If I would have 4 nodes: M1-S1 and M2-S2. So far all nodes are healthy,
> everything is fine.
> In the moment M1 fails, all writes are performed to M2 but reads are
> made from both S1 and S2. So I endup with inconsistency.
> Since Redis is very good optimalized with data I did not see the point
> to use Redis cluster in our infrastructure and sharding is not necessary
> for us.
>

Yes, you won't need sharding unless you run Rspamd on Gmail. But then
you have a different set of problems... So yes, that's mainly
theoretical possibility.

>> For (2), you have almost all request being read-only requests. Hence, to
>> scale this data, you can just add more slaves to the existing
>> configuration. Master fail won't affect anything: you just stop learning
>> until master is restored or switched, but there is still no influence on
>> scanning process.
> If the master fails for let's say 1 hour, the corrupted mailbox can send
> big amount of unwanted stuff not reaching the limit. On the big scale it
> can even mean the reputation of IPs of our servers can start to
> deteriorate. That's why I thought the redis sentinel support would be
> welcomed.

The question is why is your master down for 1 hour. That's monitoring
and administration issue and not a development one.

Redis sentinel support will require extremely huge modifications to the
all Redis support libraries (https://redis.io/topics/sentinel-clients).
It will change the overall logic of making requests and of maintaining
redis pools besides of other issues.

From my estimations, this support would require about month of my work
and I see clearly no desire in adding it to Rspamd, as I'm not convinced
that it will be useful to any Rspamd user.

--
Vsevolod Stakhov