Redis data not being cleaned up properly?

[paulc]

Hi folks,

We have a fairly large faye deployment running in production and have
been using the redis engine as a backend across a number of servers.

We noticed that our memory usage on redis was getting quite large and
on inspection I found that there were a lot (40k) of faye related keys
in the store many that seemed to be quite stale.

In addition the faye gc call didn't seem to be working at all (none of
the redis calls that should have been happening were showing up.

I cleared out this faye data, but I wanted to see if there were any
known issues as to why we'd this type of build up or why gc might have
stopped working. We've since restarted all our faye processes and I
see the gc calls hitting redis now but I want to get to the bottom of
what may have happened.

We are wondering about modifying faye redis behavior to set explicit
longer term expiries on most keys for instance (eg. expire after x
days).

Any thoughts on this would be welcome!

- Paul

[James Coglan]

On 9 March 2012 19:16, paulc <pcra...@gmail.com> wrote:

We noticed that our memory usage on redis was getting quite large and
on inspection I found that there were a lot (40k) of faye related keys
in the store many that seemed to be quite stale.

In addition the faye gc call didn't seem to be working at all (none of
the redis calls that should have been happening were showing up.

I cleared out this faye data, but I wanted to see if there were any
known issues as to why we'd this type of build up or why gc might have
stopped working.  We've since restarted all our faye processes and I
see the gc calls hitting redis now but I want to get to the bottom of
what may have happened.

I'm not sure, but I'm wondering if it could be caused by a change at some point during the 0.6.x range while I was still getting that stuff stable -- the conditions for deleting stuff might have changed in such a way that data stored using an old schema was not collected. Would be good to know if the gc() function was running but deciding to do nothing, or just not running at all (which would indicate a broken setInterval loop). If the former, we need to know why it was being called but not deleting anything.

 

We are wondering about modifying faye redis behavior to set explicit
longer term expiries on most keys for instance (eg. expire after x
days).

Potentially, if you set expire times that are much longer than the lifetime of any of your clients, this could work. But you need to consider the access pattern of each key. Faye uses these key types:

* /notifications -- pub/sub channel for inter-process messaging

* /clients -- set of all active client IDs, probably written frequently, should never go away but unused IDs should be removed

* /clients/:id/messages -- list of queued messages for a client -- may be infrequently written, should go away if client is inactive

* /clients/:id/channels -- set of subscribed channels for a client -- may be infrequently written, should go away if client is inactive

* /channels/:id -- inverse index for /clients/:id/channels, set of client IDs subscribed to a channel -- should never be removed but should not include inactive client IDs

Although I say these should 'never go away' -- Faye should be able to recover if you completely flush the database. There will be a period of message loss, but clients will re-handshake, get new IDs and re-send their subscriptions if you remove all the state from the DB.