Local read-only child processes to speed up reads

[Yichao Zhang]

Hi,

I have a scenario where I don't mind data being a little stale, but really want the read throughput to be really high. I think forking local read-only processes can handle this case. Redis can periodically kill the older children and start newer ones to keep their copy in memory only minutes or even seconds late. Forking for background save is a pretty good reference for implementation.

What do you guys think? Will it be useful to other people too? I have not implemented it, but I can if people are interested and when I have more time.

Regards,

Yichao

[Felix Gallo]

Just use a slave per core and a load balancer in front, like haproxy.  No need to get fancy.

F.

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/groups/opt_out.

[Josiah Carlson]

Forking repeatedly is not necessarily faster than keeping slaves updated. It's not necessarily slower either, but computationally speaking, forking is not an inexpensive operation for a nontrivial data set size.

 - Josiah

[Matt Palmer]

On Wed, Mar 05, 2014 at 08:49:51AM -0800, Josiah Carlson wrote:
> Forking repeatedly is not necessarily faster than keeping slaves updated.

*Keeping*, probably not. *Getting*, on the other hand... Having to dump an
RDB every time you want to spin up a new slave is something of a pest.

Yichao, I think your idea is interesting, and I'd encourage you to give it a
whirl -- the Redis codebase is *very* well laid out, and is one of the
easiest codebases to modify I've had the pleasure of working on. I doubt
you'll find widespread acceptance of your change, but anyone else who finds
your idea interesting will gain value from your work. I'd be surprised if
you got your patched merged into mainline, but the beauty of open source is
that you can modify the software to suit your needs, regardless of the
opinions of anyone else, including the "vendor".

Just remember that, while the master is recycling a slave (killing it and
respawning it), the socket you talk to won't be available, so your client
code will need to be capable of handling connections dying and retrying
their operations. Depending on how latency-sensitive your read ops are,
that *might* be enough additional overhead to make your scheme infeasible,
but I don't know what your workload is, so I can't say for sure.

- Matt

[Josiah Carlson]

If you've got updated slaves, then there's reason to create new slaves to induce further snapshotting after startup. Of course the drawback is an increase in memory use, which is what I suspect Yichao's original intent was (distributing read load without using more memory).

 - Josiah

[Yichao Zhang]

Yes, my original intent was to distribute read load without using (much) more memory.

>  Depending on how latency-sensitive your read ops are, that *might* be enough additional overhead to make your scheme infeasible,

I care more about throughput than latency, so I think it is OK for me.

I have been trying to think up some use cases other than mine, but I don't find much. I am not sure if web page caching can make use of my idea. Perhaps pages updated by other people can be served safely stale for some short time. HyperLogLog is not accurate and staleness may not be a big problem, but it's not memory hungry, so I'm not very sure. Maybe some people may use a lot of HLL sets to count different kinds of objects?

I have finally found time to dive into Redis source and implement the idea. I am not very sure how to test though. Command handling is basically untouched, and most code is to shutdown subsystems in the child process. I want to test if the subsystems (like AOF/replication) are shutdown and does nothing, but how to effectively do so?

Any code review will be very helpful to me. There is a strong feeling of some subsystem is lurking somewhere, not shutdown properly.

As to the Redis source code, I do think it's easy to modify. However, IMHO, the subsystems are not encapsulated well enough. For example, Redis uses server.masterhost to check for the master/slave status, and the checks are everywhere, even in db.c. This makes me afraid of the read-only slave going into places I  want to keep it out of. Does anybody also consider it a problem? I have some opinions I'd really like to share.

Regards,

Yichao

[Josiah Carlson]

On Wed, Apr 16, 2014 at 1:51 AM, Yichao Zhang <ech...@gmail.com> wrote:

Yes, my original intent was to distribute read load without using (much) more memory.

>  Depending on how latency-sensitive your read ops are, that *might* be enough additional overhead to make your scheme infeasible,

I care more about throughput than latency, so I think it is OK for me.

I have been trying to think up some use cases other than mine, but I don't find much. I am not sure if web page caching can make use of my idea. Perhaps pages updated by other people can be served safely stale for some short time. HyperLogLog is not accurate and staleness may not be a big problem, but it's not memory hungry, so I'm not very sure. Maybe some people may use a lot of HLL sets to count different kinds of objects?

Many high-IO read use-cases end up running into network limitations far before Redis is an issue, so slaving is usually the right answer.

I have finally found time to dive into Redis source and implement the idea. I am not very sure how to test though. Command handling is basically untouched, and most code is to shutdown subsystems in the child process. I want to test if the subsystems (like AOF/replication) are shutdown and does nothing, but how to effectively do so?

Close all unnecessary file handles in the child. Not only does that prevent accidentally reading/writing handles that are being controlled by the parent process, it also tells you pretty quickly if you forgot to update a function or two.

Any code review will be very helpful to me. There is a strong feeling of some subsystem is lurking somewhere, not shutdown properly.

As to the Redis source code, I do think it's easy to modify. However, IMHO, the subsystems are not encapsulated well enough. For example, Redis uses server.masterhost to check for the master/slave status, and the checks are everywhere, even in db.c. This makes me afraid of the read-only slave going into places I  want to keep it out of. Does anybody also consider it a problem? I have some opinions I'd really like to share.

If you have ideas for fixing the encapsulation issue, create an issue on the bug tracker with your ideas. If you are willing to work on the fix itself, providing a patch may help quite a bit, though I'd ask Salvatore first before starting on that part.

 - Josiah

 

For more options, visit https://groups.google.com/d/optout.