Simple Queue Service over Memcache - MemcacheQ 0.1.0 is release!
Steve Chu
It is a long time since last release, but anyway, we have new version
now, and new feature is ready! Lots of people here request the
multiple queue feature, now it is shipped!
MemcacheQ is a damn simple queue service over memcache protocol, just
set for append and get for consume. Any memcached clients can do queue
operations on MemcacheQ. It is very fast, and can do tens of thousands
append/comsume operations per second.
Please visit http://memcachedb.org/memcacheq for more info(also
howtos), and you can download it from:
http://code.google.com/p/memcacheq/downloads/list
Please post your bugs and issues to the mailing list, and any feedback
is welcome!
Thanks mark for his bombing on my mailbox that makes me keep working on it :P
Enjoy!
--
Steve Chu
http://stvchu.org
Dustin
On Sep 22, 6:51 am, "Steve Chu" <stv...@gmail.com> wrote:
> MemcacheQ is a damn simple queue service over memcache protocol, just
> set for append and get for consume. Any memcached clients can do queue
> operations on MemcacheQ. It is very fast, and can do tens of thousands
> append/comsume operations per second.
Does it ever drop queued items?
Can you compare it to gearmand/beanstalkd/starling?
mark
> MemcacheQ is a damn simple queue service over memcache protocol, just
> set for append and get for consume. Any memcached clients can do queue
> operations on MemcacheQ. It is very fast, and can do tens of thousands
> append/comsume operations per second.
this is the best of all the open-source queuing solution ever!!!!
simple, fast and rocks out loud!!!
> Thanks mark for his bombing on my mailbox that makes me keep working on it :P
hei steve.. thanks a lot for taking time to work on this!! this is
great!! thanks again!!!
Dustin
On Sep 22, 11:25 am, mark <markki...@gmail.com> wrote:
> this is the best of all the open-source queuing solution ever!!!!
> simple, fast and rocks out loud!!!
memcached has several properties that I think make it quite poorly
suited as a platform for building a job queue. You can certainly work
around them, but if the goal is to get it to work with existing
memcached clients, then you've recreated starling (including the part
where it forces the clients to poll for jobs, which will cause you to
try to balance between pickup latency and massive CPU utilization on
both server and client).
If it's stateless, then you also have to have semantics for jobs
that get dropped by clients. e.g. if I ask for a job and then
immediately crash, does the job get run?
Of course, I'm not trying to suggest there shouldn't be innovation
in this space, but there are a *lot* of options, and I don't
understand what the others are lacking (I've used and written several
myself).
Steve Chu
>
>
> On Sep 22, 6:51 am, "Steve Chu" <stv...@gmail.com> wrote:
>
>> MemcacheQ is a damn simple queue service over memcache protocol, just
>> set for append and get for consume. Any memcached clients can do queue
>> operations on MemcacheQ. It is very fast, and can do tens of thousands
>> append/comsume operations per second.
>
> Does it block?
>
> Does it ever drop queued items?
No, it is persistent(we use Berkeley DB TDS in our backend). But for
'-N' mode, if the server daemon crashed, it will lose the data in the
write buffer(the size determined by you with '-L' option), seem we
should use replication for this situation, but a bit complicated.
>
> Can you compare it to gearmand/beanstalkd/starling?
>
We need a queue component that we know all the details(I mean the code
base, and when something bad happened, we can solve it easily). We do
not use Perl, we do not use Ruby, so we do not use gearmand and
starling. The application developers in our company are very familiar
with memcached clients, so we take the memcache protocol as a
consideration.
Steve Chu
>
>
> On Sep 22, 11:25 am, mark <markki...@gmail.com> wrote:
>
>> this is the best of all the open-source queuing solution ever!!!!
>> simple, fast and rocks out loud!!!
>
> Which ones have you tried, and what did you not like about them?
>
> memcached has several properties that I think make it quite poorly
> suited as a platform for building a job queue.
You mean the non-blocked I/O model? What properties that make it
quite poorly? Can you give us a bit more detail?
> You can certainly work
> around them, but if the goal is to get it to work with existing
> memcached clients, then you've recreated starling (including the part
> where it forces the clients to poll for jobs, which will cause you to
> try to balance between pickup latency and massive CPU utilization on
> both server and client).
>
> If it's stateless, then you also have to have semantics for jobs
> that get dropped by clients. e.g. if I ask for a job and then
> immediately crash, does the job get run?
Yes, this is a problem we should solve. Maybe a visible timeout and
explicit delete will make this done, but also a bit more complicated
to end users.
>
> Of course, I'm not trying to suggest there shouldn't be innovation
> in this space, but there are a *lot* of options, and I don't
> understand what the others are lacking (I've used and written several
> myself).
>
--
Steve Chu
http://stvchu.org
Dustin
On Sep 22, 7:48 pm, "Steve Chu" <stv...@gmail.com> wrote:
> Yes, we can not say it does not block, but it runs fast:)
for the number of times a job is requested and none is available.
> > Does it ever drop queued items?
>
> No, it is persistent(we use Berkeley DB TDS in our backend). But for
> '-N' mode, if the server daemon crashed, it will lose the data in the
> write buffer(the size determined by you with '-L' option), seem we
> should use replication for this situation, but a bit complicated.
> > Can you compare it to gearmand/beanstalkd/starling?
>
> We need a queue component that we know all the details(I mean the code
> base, and when something bad happened, we can solve it easily). We do
> not use Perl, we do not use Ruby, so we do not use gearmand and
> starling. The application developers in our company are very familiar
> with memcached clients, so we take the memcache protocol as a
> consideration.
codebase. :) You should look at it before putting too much more
effort into where you're going here, as it already solves a lot of the
problems (priority queues, time delayed jobs, blocking and partially
blocking (including non-blocking) job reservation, automatic job
reclamation due to timeout or disconnect, per-job stats, all kinds of
stuff).
What it doesn't do yet is persistence. That probably matters less
to my applications than to other people's (since I've never had it
crash and I can afford to theoretically lose a few jobs at the
moment).
[more below]
No, I mean if the queue doesn't block on a take (at least
optionally), then it can't perform well in the general case. Clients
will have to spend a lot of time just polling the queue. That's a lot
of wasted network and CPU resources on both ends.
You want the server to just hang until a job becomes available, and
then immediately return.
> What properties that make it
> quite poorly? Can you give us a bit more detail?
can poll...but what if you need a priority queue (I've needed it every
time I've implemented a job queue so far)?
The polling thing is similarly an issue. You can block, not block,
or partially block (with timeout). You'll want to support these
things, or you're in bad shape.
The memcached protocol doesn't have any room for something like,
say, a job identifier that can be used to uniquely address the job on
the way in or out. You basically just have a queue and you're forced
to do an atomic delete and get from the queue in order to get things
to work (or maybe a get and delete which would at least allow you to
ensure the bulk of the contents were delivered before deleting it.
How do you deal with a queue with delay? That's necessary in *at
least* retry scenarios, but I've needed it directly in a few projects
as well.
> > If it's stateless, then you also have to have semantics for jobs
> > that get dropped by clients. e.g. if I ask for a job and then
> > immediately crash, does the job get run?
>
> Yes, this is a problem we should solve. Maybe a visible timeout and
> explicit delete will make this done, but also a bit more complicated
> to end users.
existing memcached clients. That's not what they do.
mark
>
>
> On Sep 22, 7:48 pm, "Steve Chu" <stv...@gmail.com> wrote:
>> > Can you compare it to gearmand/beanstalkd/starling?
>>
>> We need a queue component that we know all the details(I mean the code
>> base, and when something bad happened, we can solve it easily). We do
>> not use Perl, we do not use Ruby, so we do not use gearmand and
>> starling. The application developers in our company are very familiar
>> with memcached clients, so we take the memcache protocol as a
>> consideration.
>
> I use beanstalkd because it's a really nice and small C
> codebase. :) You should look at it before putting too much more
> effort into where you're going here, as it already solves a lot of the
> problems (priority queues, time delayed jobs, blocking and partially
> blocking (including non-blocking) job reservation, automatic job
> reclamation due to timeout or disconnect, per-job stats, all kinds of
beanstalkd is too complicated.. with tubes and stuff like that.. and
it is not persistent.. and getting started is not easy.. all i want is
a q server, that has multiple queues, and follows a very simple
protocol like memcaches.. memcacheq is perfect in such a scenario...
Dustin
On Sep 23, 6:35 pm, mark <markki...@gmail.com> wrote:
> beanstalkd is too complicated.. with tubes and stuff like that.. and
> it is not persistent.. and getting started is not easy.. all i want is
> a q server, that has multiple queues,
complicated and one of your requirements.
Agreed it's not persistent. Making it persistent will be less work
than finishing another memcached based queue server.
> and follows a very simple
> protocol like memcaches.. memcacheq is perfect in such a scenario...
pretty good for that. I think I've sufficiently stated the reasons
it's not good for a worker queue already.
For example, what's the value of a persistent queue if inspecting a
queue deletes jobs? A client get+crash (which, in my experience is
significantly more likely than a queue server going down since I've
yet to have that happen, but my workers blow up all the time) loses a
job in memcacheq. That is, if I deployed memcacheq instead of
beanstalkd in my environment, memcacheq would lose many jobs a day,
while beanstalkd currently loses none. It would also use
*considerably* more CPU on my workers and queue server.
I completely reject your assertion that beanstalkd (as an example)
is too hard to get started with. There are quite a few easy-to-use
clients and the protocol is pretty damned simple. My ocaml
implementation of the beanstalk protocol (including support for
everything in the document and custom exception types for every error
that might occur) is 192 lines. My rails-compatible job inserter in
ocaml is about 7 lines (the ruby one is a bit shorter).