Re: About performance of Posix message Queue
Jens Thoms Toerring
> I'm programming multithreading application.
> Several threads share one global variable and
> the global variable is implemented circular queue by myself.
> - my develop environment isn't supported C++, so I can't used STL
> But it suddenly occurred to me that there is message queue(MQ)
> and i find the Posix Message Queue .
Mmmm, then you throw away the advantage you get from using
threads, i.e. that they share a single address space, so that
interprocess communication methods like message queues aren't
needed...
> And I have a few Questions but I could not find the solution.
> So I ask a favor.
> Q1) Do MQ use memory? or file?
> I think if MQ use file, the performanc is much lower than
> global variable.
> if so I can't use MQ.
> Otherwise I consider using MQ intead of circluar queue made by
> myself.
As far as I know message queues typically are kept in memory
by the kernel, but I don't think that there's anything that
would forbid an implementation to store them in a file (or
somehwere else on disk). Of course, the kernel also can keep
them in a virtual file system - I don't know if you would
consider is as a "file" if the file is actually in memory.
But I would recommend to simply measure the performance and
not decide because of some possibly unfounded preconception.
> Q2) If I would use MQ, I made many ones - maybe 80~100
> Is there restriction that make many MQ?
There have to be restrictions since message queues use a
limited resource, kernel memory. There are limits on the
total number of message queues, the number of messages in
a queue, the size of the messages and the total memory that
can be used for message queues. And you must keep in mind
that these are system wide limits, i.e. they apply to all
processes running on the machine combined.
On my machine (Linux) the maximum number of queues seems to
be 256 (according to /proc/sys/fs/mqueue/queues_max), so if
three instances of your program would be running there already
might be trouble. But why do you need that many at all?
Also don't forget that message queues are persistent, i.e. if
your program forgets to delete them before it ends they will
continue to exist. That can especially be a problem if your
program gets killed by an uncatchable signal and thus has no
chance to clean up after itself.
Regards, Jens
--
\ Jens Thoms Toerring ___ j...@toerring.de
\__________________________ http://toerring.de
RickE
more familiar with the SysV, but barely.
The SysV MQ is runtime expandable so size can be changed. It is a
kernel context function, non-preemptive, but Signals are not handled,
as said.
I'm interested in the ftok(pathname, int) function. It seems the
intended use is to set up a file (perhaps in /temp) that reflects the
existence of the MQ along with the msgget function. Kind of like a /
proc filesystem.
In my adaptation (using FreePascal) I call a MQ a Message Box, and
treat it like a mailbox, where the message type is treated like a mail
Subject. The /temp/msg_box/file is used as a log file for messages.
The analogy seems to be consistent. SysV IPC MQs predate networking so
this is a simple version of email. It isn't rocket science, just a
useful tool.
Jens Thoms Toerring
> The SysV MQ is runtime expandable so size can be changed. It is a
> kernel context function, non-preemptive, but Signals are not handled,
> as said.
While (as the owner or creator of a mesage queue) you can change
it's size within certain limits I don't think that there's a
requirement for a kernel to allow changing the system-wide limits
dynamically. If I remember correctly they used to be compile time
settings for the kernel. But modern systems seem to allow dynamic
changes (e.g. on Linux via /proc/sys/kernel/msgm((ax)|(nb)|(ni)).
But you need root privileges to do that, otherwise rogue processes
could wreak havoc with the system.
> I'm interested in the ftok(pathname, int) function. It seems the
> intended use is to set up a file (perhaps in /temp) that reflects the
> existence of the MQ along with the msgget function. Kind of like a /
> proc filesystem.
If I am not completely mistaken that's not the case. Neither does
the existence of the file, whose name is used with ftok(), indicate
that a message queue exists nor the other way round, i.e. you can't
conclude from the its non-existence that the message queue does not
exist. If the file exists this does not indicate that a process has
created the corresponding message queue(s) or that it hasn't been
deleted. And if the file does not exist there still could be pro-
cesses that connected to the message queue while the file still
existed (deleting the file does not delete the message queue nor
does deleting the message queue remove the file).
As far as I understand it ftok() is just a tool that is supposed
to help the programmer to come up with unique keys suitable for
SysV IPC methods and that have to be common to unrelated processes,
but not more than that.
RickE
Excellent. Thanks Jens. Again, I agree with you. But please, lets look
further.
Looking at the man page for ftok(), the key is generated by an
existing file and a "project id." If the file is set up as a message
log file, any messages posted to the file produce a file descriptor
change notification via select. Perhaps special files like FIFOs will
work.
Further, if the project id variable is set to INBOX and OUTBOX
constants, or STD_MSG_IN and STD_MSG_OUT, something like a pipe can be
set up.
In a historical context, original Unix developers worked on the same
host, using dumb terminals. There was no reason for ethernet message
systems.
Anyway, that's what I'm playing with using FreePascal, and importing
the libc functions. Pascal, another throwback to the days when one
could read a program.
Thanks again for your thoughts...Rick
Jens Thoms Toerring
> Looking at the man page for ftok(), the key is generated by an
> existing file and a "project id." If the file is set up as a message
> log file, any messages posted to the file produce a file descriptor
> change notification via select. Perhaps special files like FIFOs will
> work.
I guess there's nothing wrong with using the file for whatever
purposes you may have. ftok() doesn't care about its contents
or about changes to the contents, all it does is check if the
file exists and that it is accessible (in the sense that your
process is able to call stat() on the file successfully). No-
thing else about the file should be of importance to ftok().
And once you have called ftok() (and don't need to call it
again to get the same key) you can also delete or rename the
file or change it's permissions or whatever, it won't have any
influence on the workings of the message queue/shared memory
segment/semaphore you created/attached to via the key you
received from ftok().
ftok() should be seen as a function that doesn't do anything
else than getting some bits from the information about the
actual file behind the path (e.g. from the inode number and
the device number), mixing them up with the 'proj_id' value
and returning the result. It's just a kind of a hash func-
tion (including the problem that collisions can't be ruled
out).
> Further, if the project id variable is set to INBOX and OUTBOX
> constants, or STD_MSG_IN and STD_MSG_OUT, something like a pipe can be
> set up.
I don't really understand what you mean here. But, of course, if
you use different constants for 'proj_id' in calls of ftok() you
get keys for different message queues (at least if the constants
you use differ in their least significant 8 bits). BTW, don't
forget to make sure that the constants (or their lowest 8 bits,
to be precise) aren't 0 since otherwise it's unspecified what
ftok() will come up with.
RickE
Thanks again.
My point was that, in a historical context, ftok did relate a file
system variable to an IPC variable. Other users become aware of this
file system flag. I believe this is a clever solution to MQ IDs.
But you are right, I have not thought through the other issues you
raise. On a brighter note, the program now compiles so I can
experiment.
Recently, I got a signals program going and learned that my linux
supports a signal set of 64 signals. Also, how the procmask function
sets an environment variable that is passed through forks and execs.
Perhaps my real point is how many keep looking for the "new"
solutions, but don't fully appreciate "old" solutions.
Best regards,
Rick.
Jens Thoms Toerring
> My point was that, in a historical context, ftok did relate a file
> system variable to an IPC variable. Other users become aware of this
> file system flag. I believe this is a clever solution to MQ IDs.
I can't comment since I don't have much knowledge about the
historical details of UNIX. I guess there are some other people
here in this group that are much more qualified.
> But you are right, I have not thought through the other issues you
> raise. On a brighter note, the program now compiles so I can
> experiment.
Congrats;-)
> Recently, I got a signals program going and learned that my linux
> supports a signal set of 64 signals. Also, how the procmask function
> sets an environment variable that is passed through forks and execs.
I don't think that sigprocmask() sets an environment variable (in
the sense that you could get at it with getenv()) but that it sets
some state of the process which then might be inherited by child
processes. But I would have to look up the details...
Moi
> On Feb 5, 7:29 am, j...@toerring.de (Jens Thoms Toerring) wrote:
>
>> I guess there's nothing wrong with using the file for whatever purposes
>> you may have. ftok() doesn't care about its contents or about changes
>> to the contents, all it does is check if the file exists and that it is
>> accessible (in the sense that your process is able to call stat() on
>> the file successfully). No- thing else about the file should be of
>> importance to ftok(). And once you have called ftok() (and don't need
>>
>> ftok() should be seen as a function that doesn't do anything else than
>> getting some bits from the information about the actual file behind the
>> path (e.g. from the inode number and the device number), mixing them up
>> with the 'proj_id' value and returning the result. It's just a kind of
>> a hash func- tion (including the problem that collisions can't be ruled
>> out).
>>
> My point was that, in a historical context, ftok did relate a file
> system variable to an IPC variable. Other users become aware of this
> file system flag. I believe this is a clever solution to MQ IDs.
>
ftok() is a way for programs to find a rendez-vous point. Compare it
to /etc/services: programs need a way to find out on which port(tcp) or
key (IPC) they should connect/attach. They could put it hardcoded into
both server and clients, but there still needs to be a way to configure
it, eg in case of "collisions" with other programs using the same key.
Portmapper serves a similar purpose in a different way and for a different
domain (RPC).
The file or filename itself is not important. it could be /tmp or
/etc/password.
ftok() is just a hashfunction: it maps the filesystem namespace (device
number+inode number, IMO) to a (smaller) key. Since server and clients use
the same (file)system and filename, they find the same key.
HTH,
AvK
RickE
All of this is very interesting. I'm reading about Group "sets." It is
hard to sort out with "process groups" and filesystem groups and
effective IDs. And portmapper is another good example with internet.
My "historical context" goes way back. And I see the Unix model
remaining important. The "big iron" mainframe model that served many
dumb terminals can now be applied to an automobile controller that
serves many microcontrollers. A little MQ capability has uses.
A long time ago I worked at a Bell Tel. switching station. The relay
switches sparked each time a number was dialed (new connection made).
And those old ttys were engineering marvels. The concepts remain
remarkably valid.