General Questions about having many Queues

[Ben]

My basic understanding is that MemcacheQ uses 1 file per Queue (Q).
And that the total number of Qs you can have open at a time is limited
by the limit of number files you are allowed to open.

Is there a better way for handling situations where I want, say..
10,000 Qs, but only 10% of these Qs will need to be accessed a lot.
I.e. I don't mind if the access time is higher for Qs that are not
actively used.

I don't mind having one file per Q, but I would like to limit the
number of open file-handles to something reasonable, let's say 1000 at
a time. Can MemcacheQ or BDB take care of this for me? Or is BDB
just going to croak if it can't have an open filehandle to every
database (Q) it may want to open.

Again, I am quite new to BDB, so am not sure if these are options that
I can configure or not, but at first glance MemcacheQ looks pretty
nice, although possibly not really optimized for having a large number
of Queues.

Thanks again for the great piece of software I am looking forward to
your reply.

[Ben]

To make this problem more clear:

Is there a way that not all the queues DBs have to be open at the same
time? I.e. I can imagine writing an algorithm that
closes queue DBs that have not been used recently. So that the
maximum number of file handles at any given time remains constant.

i.e. I could define MAX_FILE_HANDLES

and if NUMBER_OF_QUEUES_OPEN > MAX_FILE_HANDLES

close the least recently used open queue.

The more I think about this, the more I realize this is probably
something that I'll need to write myself, but it seems possible that
BDB's DB manager could perform this sort of management task for you.

-Ben

[Steve Chu]

Hi, ben,

Enhancing MAX_FILE_HANDLES of a process that can open is not big
performance loss by "-c" option, and tens of thousands fds per process
is not a issue on modern operating systems.

--
Steve Chu
http://stvchu.org