Processing workload distribution

[Ted]

The machine is a single processor/quad core server running the latest
Windows Server.

The Perl we're using is the 64 bit build of 5.8.8 from Activestate.

I created a script that uses threads to launch a series of standalone
SQL scripts (I use system to invoke mysql to run my SQL scripts). I
had thought that system was spawning child processes, but it seems
that the script that launches them waits on each child before
launching the next. It is looking like the threads launches all
without wating for the next, which is closer to what I want.

However, regardless of what I have tried so far, all the hard work is
being done by one core, with the other three mostly sitting idle.

What can I do to distribute the workload evenly over all the cores in
the machine? The whole purpose of this machine was to serve as a
compute server for my colleagues and I, and in fact only two of us
have a need to access it, and this is so we can run heavy duty DB
analysis programs on it. The code works adequately, but it seems a
shame to have invested in a quad core processor based machine only to
have all the work done on only one core. I don't care if I have to
launch threads or child processes, as long as I can distribute the
work more evenly. Is what I am after possible?

Thanks

Ted

BTW: are there any 64 bit modules on CPAN? The repository for the 64
bit build that Activestate has provided seems to be empty.

[Ben Morrow]

Quoth Ted <r.ted...@rogers.com>:

> The machine is a single processor/quad core server running the latest
> Windows Server.
>
> The Perl we're using is the 64 bit build of 5.8.8 from Activestate.
>
> I created a script that uses threads to launch a series of standalone
> SQL scripts (I use system to invoke mysql to run my SQL scripts).

In general it is easier to talk to the database directly using DBI than
to launch external mysql processes. In this case, however, you may find
it easier to make things run in parallel with an external command.

> I had thought that system was spawning child processes, but it seems
> that the script that launches them waits on each child before
> launching the next.

Yes, this is how system works. As a special case, on Windows only,
calling

my $pid = system 1, "mysql ...";

(with a literal 1 as the first argument) will spawn a new process and
*not* wait for it. You can use the returned pid as an argument to wait
or waitpid. Alternatively, for more control, you can use Win32::Process
or IPC::Run.

> It is looking like the threads launches all without wating for the
> next, which is closer to what I want.

You would have to post your code for us to see what is happening here. I
think probably each thread is launching one child and waiting for it,
but since you have several threads you have several children.

> However, regardless of what I have tried so far, all the hard work is
> being done by one core, with the other three mostly sitting idle.

Are you *certain* you end up with more than one mysql process running at
a time? You should be able to check easily with Task Manager. If you
don't, then you need to fix that (or switch to using DBI to talk to the
database and Perl threads for parallelism). If you do, yet they are all
running on the same core, then something odd is happening: you will need
to show us a (minimal) script that runs two processes at the same time
that still end up on the same core.

> BTW: are there any 64 bit modules on CPAN? The repository for the 64
> bit build that Activestate has provided seems to be empty.

Modules on CPAN are (generally-speaking) architecture-neutral, given
that CPAN only holds C and Perl source. If ActiveState don't provide
64-bit ppms, then you can install pure-Perl modules directly from CPAN,
but XS modules will require a copy of the compiler used to build your
perl, probably a 64-bit version of MSVC. You may be able to get gcc
working (the 32-bit version of AS Perl has support for building modules
with gcc), but I don't know how well the 64-bit version is supported.

Ben

[smallpond]

See perlthrtut and check your perl build with: perl -V
Look for a line like:

usethreads=define use5005threads=undef useithreads=define
usemultiplicity=define

I believe that the last part indicates whether perl tries to use
multiple
CPUs.

[xho...@gmail.com]

Ted <r.ted...@rogers.com> wrote:
> The machine is a single processor/quad core server running the latest
> Windows Server.
>
> The Perl we're using is the 64 bit build of 5.8.8 from Activestate.
>
> I created a script that uses threads to launch a series of standalone
> SQL scripts (I use system to invoke mysql to run my SQL scripts). I
> had thought that system was spawning child processes, but it seems
> that the script that launches them waits on each child before
> launching the next.

That is what Perl's "system" does. It spawns a child process, and then
waits for it.

> It is looking like the threads launches all
> without wating for the next, which is closer to what I want.
>
> However, regardless of what I have tried so far, all the hard work is
> being done by one core, with the other three mostly sitting idle.

Generally (not always, but generally) the hard work of an SQL query
is done by the server, not the client. If your MySQL database server fails
to use all the cores in doing it's job, that is not a Perl problem, that is
a MySQL problem.

If you don't think this describes your situation, then launch the four
standalone SQL scripts separately from Windows. If you launch them this
way, are all cores used? If not, then you don't have a Perl problem.

Xho

--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.

[Ted]

> CPUs.- Hide quoted text -
>
> - Show quoted text -

Yes, both on my development machine (only a dual core) and on the quad
core server, I see this: use of both threads and multiplicity are
there. Near the top of the output from perl -V, I see:

usethreads=define use5005threads=undef useithreads=define
usemultiplicity=define

And lower down I see,

Characteristics of this binary (from libperl):
Compile-time options: MULTIPLICITY PERL_IMPLICIT_CONTEXT
PERL_IMPLICIT_SYS PERL_MALLOC_WRAP
PL_OP_SLAB_ALLOC USE_ITHREADS USE_LARGE_FILES
USE_PERLIO USE_SITECUSTOMIZE

When I launch four mysql commands separately, I see four instances of
it in task manager, and the work load is distributed over all four
cores. So the problem must be in my code (See my reply to myself).

Thanks

Ted

[Ted]

On Feb 25, 12:26 pm, xhos...@gmail.com wrote:

yes, so I have a perl problem (or probably more accurately my use of
perl is the problem).

Thanks

Ted

[Ted]

Here is my code:

use threads;

my $thr0 = threads->create(\&dbspawn,'e5pf16.sql','e5pf16.log');
$thr0->join;
my $thr1 = threads->create(\&dbspawn,'gpf45.sql','gpf45.log');
$thr1->join;
my $thr2 = threads->create(\&dbspawn,'gpf46.sql','gpf46.log');
$thr2->join;
my $thr3 = threads->create(\&dbspawn,'gpf47.sql','gpf47.log');
$thr3->join;
my $thr4 = threads->create(\&dbspawn,'gpf48.sql','gpf48.log');
$thr4->join;
my $thr5 = threads->create(\&dbspawn,'gpf49.sql','gpf49.log');
$thr5->join;
my $thr6 = threads->create(\&dbspawn,'gpf50.sql','gpf50.log');
$thr6->join;

sub dbspawn {
local ($script,$log) = @_;
system("mysql -t -u rejbyers --password=jesakos yohan < $script >
$log") == 0 or die "$script failed!";
}

Is the problem the way I use "system"?

When this script is running, I only ever see one instance of mysql
running in task manager.

Is there a Win64::process, or will Win32::process work on a 64 bit
version of windows using the 64 bit build of perl?

Thanks

Ted

[Ben Morrow]

Quoth smallpond <smal...@juno.com>:

>
> See perlthrtut and check your perl build with: perl -V
> Look for a line like:
>
> usethreads=define use5005threads=undef useithreads=define
> usemultiplicity=define
>
> I believe that the last part indicates whether perl tries to use
> multiple CPUs.

Please don't make things up and post them without verifying.
Multiplicity indicates whether it is possible to have several perl
interpreters in the same process: this is a requirement for ithreads (so
any perl with useithreads=define will also have usemultiplicity=define)
but is also used without ithreads for e.g. mod_perl. It has *nothing* to
do with use of multiple CPUs: this is determined by your OS's thread
scheduling policy.

Ben

[Joost Diepenmaat]

Ted <r.ted...@rogers.com> writes:

> Here is my code:
>
> use threads;
>
> my $thr0 = threads->create(\&dbspawn,'e5pf16.sql','e5pf16.log');
> $thr0->join;

you should join() all threads /after/ creating all of them. a join()
blocks untill the thread is done. In other words, move the ->join()
calls to the end of script.

--
Joost Diepenmaat | blog: http://joost.zeekat.nl/ | work: http://zeekat.nl/

[xho...@gmail.com]

Ted <r.ted...@rogers.com> wrote:
> Here is my code:
>
> use threads;
>
> my $thr0 = threads->create(\&dbspawn,'e5pf16.sql','e5pf16.log');
> $thr0->join;

Join means you wait for $thr0 to finish. Obviously this is going
to lead to serialization. The simple answer is to move all the joins
from where they are and put them after all of the creates have been done.
The *right* answer is probably to put the thread objects into an array,
then after that loop over that array doing joins, rather than using a bunch
of different variables to hold the objects.

Xho

--
-------------------- http://NewsReader.Com/ --------------------

[Ted]

On Feb 25, 1:25 pm, Joost Diepenmaat <jo...@zeekat.nl> wrote:

Ah, OK.

While it is not likely in this case, since the scripts normally take
several hours to run, what happens if you create half a dozen threads,
and one of them finishes before you get to join it?

Also, I recall reading somewhere that some system calls block
everything in the process until the system call returns. Is that
true? If so, which are involved, and what would be the workaround?

Thanks

Ted

[Ted]

On Feb 25, 1:30 pm, xhos...@gmail.com wrote:

Yup that was it.

This script was just a test to make sure I got it all right. I was
going to worry about putting it into arrays after I got it right.

that means one array for the script name, a second for the log name,
and a third for the thread objects.

I saw Joost's post before I saw yours, and set up a trivial test,
involving selecting everything from each of three tables, and
displaying them in the command line window. Yes, that window looks
like a mess, but it is getting all the data and it looks like the work
load is being distributed evenly, and I see an instance of mysql in
Task Manager for each thread. So now just a little rewriting to make
good use of arrays, and to simplify the code.

Thanks all.

Ted

[Ben Morrow]

Quoth Ted <r.ted...@rogers.com>:

> Here is my code:
>
> use threads;
>
> my $thr0 = threads->create(\&dbspawn,'e5pf16.sql','e5pf16.log');
> $thr0->join;
> my $thr1 = threads->create(\&dbspawn,'gpf45.sql','gpf45.log');
> $thr1->join;
> my $thr2 = threads->create(\&dbspawn,'gpf46.sql','gpf46.log');
> $thr2->join;
> my $thr3 = threads->create(\&dbspawn,'gpf47.sql','gpf47.log');
> $thr3->join;
> my $thr4 = threads->create(\&dbspawn,'gpf48.sql','gpf48.log');
> $thr4->join;
> my $thr5 = threads->create(\&dbspawn,'gpf49.sql','gpf49.log');
> $thr5->join;
> my $thr6 = threads->create(\&dbspawn,'gpf50.sql','gpf50.log');
> $thr6->join;

As others have said, join waits for the thread, so you need to do it
last. Also, this is *not* an efficient way to create many threads. I
would probably write something like

$_->join for map {
threads->create(\&dbspawn, "$_.sql", "$_.log");
} qw/e5pf16 gpf45 gpf46 gpf48 gpf49 gpf50/;

or probably drop the sub altogether and use async:

$_->join for map {
async {
system "mysql -t -u rejbyers <$_.sql >$_.log"
and warn "$script failed";

# there's little point in dieing when the thread's about to
# exit anyway...
}
} qw/.../;

but you may find something a little less compressed easier to
understand:

my @jobs = qw/e5pf16 gpf45 .../;

my @threads = map {
async {
system "...";
}
} @jobs;

$_->join for @threads;

Of course, given that you only use the threads to launch processes, you
would be better off just launching processes:

my @procs = map { system 1, "..." } @jobs;

waitpid $_ for @procs;

> Is there a Win64::process, or will Win32::process work on a 64 bit
> version of windows using the 64 bit build of perl?

I don't know. That is, there isn't a Win64::Process, and I don't know
whether Win32::Process works on Win64. That depends on whether the
underlying Win32 system calls still work the same way: I suspect they
do. There are no Win64 results from CPAN Testers (surprise), which is
usually the first place to look. In any case, since there aren't any
Win64 ppms, you'd need to compile libwin32 from source. Do you have an
appropriate compiler?

Ben

[Ben Morrow]

Quoth Ted <r.ted...@rogers.com>:

> On Feb 25, 1:25 pm, Joost Diepenmaat <jo...@zeekat.nl> wrote:
> > Ted <r.ted.by...@rogers.com> writes:
> > > Here is my code:
> >
> > > use threads;
> >
> > > my $thr0 = threads->create(\&dbspawn,'e5pf16.sql','e5pf16.log');
> > > $thr0->join;
> >
> > you should join() all threads /after/ creating all of them. a join()
> > blocks untill the thread is done. In other words, move the ->join()
> > calls to the end of script.
>

> Ah, OK.
>
> While it is not likely in this case, since the scripts normally take
> several hours to run, what happens if you create half a dozen threads,
> and one of them finishes before you get to join it?

It sits there, consuming a small amount of memory in your Perl process,
until you do. This is exactly the same as a 'zombie' process under Unix.
If you don't want this to happen, and don't care about the return value
of the thread, you can ->detach it, which will cause it to clean itself
up as soon as it finishes. However, if your main thread exits, any
detached threads will be silently destroyed, so if this is a problem you
will need to join them.

> Also, I recall reading somewhere that some system calls block
> everything in the process until the system call returns. Is that
> true? If so, which are involved, and what would be the workaround?

This depends on your underlying threads implementation, and is rarely
true any more (most systems use some sort of 'real' kernel threads,
which don't behave like this).

Ben

[Joost Diepenmaat]

Ted <r.ted...@rogers.com> writes:

> While it is not likely in this case, since the scripts normally take
> several hours to run, what happens if you create half a dozen threads,
> and one of them finishes before you get to join it?

if the thread has already finished, the join() should just return
immediately (after possibly doing some cleanup). Note that you should
always either join() or detach() each thread.

> Also, I recall reading somewhere that some system calls block
> everything in the process until the system call returns.

Besides thread-specific calls, like the ones used by threads::shared's
synchronization methods, I don't know of anything like that. From perl's
point of view, you should probably regard threads as completely seperate
processes that have some nifty additional IPC mechanisms.

This also implies that unless you need those sharing mechanisms, you're
not gaining anything by using threads instead of fork() - and you're
quite likely to be losing efficiency and gaining some possible issues) -
at least on OSs that support fork natively. Though I gather that on
Windows systems, the situation is different.

[xho...@gmail.com]

Ted <r.ted...@rogers.com> wrote:

> On Feb 25, 1:25=A0pm, Joost Diepenmaat <jo...@zeekat.nl> wrote:
> > Ted <r.ted.by...@rogers.com> writes:
> > > Here is my code:
> >
> > > use threads;
> >

> > > my $thr0 =3D threads->create(\&dbspawn,'e5pf16.sql','e5pf16.log');

> > > $thr0->join;
> >
> > you should join() all threads /after/ creating all of them. a join()
> > blocks untill the thread is done. In other words, move the ->join()
> > calls to the end of script.
> >
> > --
> > Joost Diepenmaat | blog:http://joost.zeekat.nl/| work:http://zeekat.nl/
>
> Ah, OK.
>
> While it is not likely in this case, since the scripts normally take
> several hours to run, what happens if you create half a dozen threads,
> and one of them finishes before you get to join it?

Then it waits very very patiently for you to "join" it. No problem.
(Unless you accumulate a large number of them).

> Also, I recall reading somewhere that some system calls block
> everything in the process until the system call returns.

There is the Perl function named "system", then there are calls that perl
makes into the kernel, referred to as system calls. They are different
but confusingly named. Is your concern about the first or the latter?

> Is that
> true?

I suspect that it is true of certain kernel-level system calls, but such
calls should be extremely fast (otherwise, they OS designers should have
reworked them not to block.) There isn't much you can do about it in
a Perl program, or at least not without knowing exactly what the call
is.

> If so, which are involved, and what would be the workaround?

Find a better OS to run your Perl program on? :)
Anyway, don't worry about it until it happens.

Xho

--
-------------------- http://NewsReader.Com/ --------------------

[smallpond]

On Feb 25, 1:07 pm, Ben Morrow <b...@morrow.me.uk> wrote:
> Quoth smallpond <smallp...@juno.com>:

I didn't make it up, I looked at previous postings in this NG where
someone has ithreads=define and not usemultiplicity.
http://groups.google.com/group/comp.lang.perl.misc/browse_frm/thread/8f27b699aff8a37d/18a233467aaa72d8?hl=en&lnk=gst&q=usemultiplicity#18a233467aaa72d8

[Jim Gibson]

In article
<43db91c7-7e79-40bd...@i7g2000prf.googlegroups.com>, Ted
<r.ted...@rogers.com> wrote:

> On Feb 25, 1:30 pm, xhos...@gmail.com wrote:
> > Ted <r.ted.by...@rogers.com> wrote:
> > > Here is my code:
> >
> > > use threads;
> >
> > > my $thr0 = threads->create(\&dbspawn,'e5pf16.sql','e5pf16.log');
> > > $thr0->join;
> >
> > Join means you wait for $thr0 to finish.  Obviously this is going
> > to lead to serialization.  The simple answer is to move all the joins
> > from where they are and put them after all of the creates have been done.
> > The *right* answer is probably to put the thread objects into an array,
> > then after that loop over that array doing joins, rather than using a bunch
> > of different variables to hold the objects.
> >
> > Xho

.
>
> Yup that was it.
>
> This script was just a test to make sure I got it all right. I was
> going to worry about putting it into arrays after I got it right.
>
> that means one array for the script name, a second for the log name,
> and a third for the thread objects.

Better to use an "array of arrays", "array of hashes", or "hash of
hashes" to keep the bits of information together (untested):

# array of arrays:
my @aoa = (
[ "script_name1", "log_name1" ],
[ "script_name2", "log_name2" ],
...
);

for my $ref ( @aoa ) {
$ref->[2] = threads->create(\&dbspawn, $ref->[0], $ref->[1]);
}

etc.

--
Jim Gibson

Posted Via Usenet.com Premium Usenet Newsgroup Services
----------------------------------------------------------
** SPEED ** RETENTION ** COMPLETION ** ANONYMITY **
----------------------------------------------------------
http://www.usenet.com

[Ted]

On Feb 25, 1:51 pm, Ben Morrow <b...@morrow.me.uk> wrote:
>
> As others have said, join waits for the thread, so you need to do it
> last. Also, this is *not* an efficient way to create many threads. I
> would probably write something like
>
>     $_->join for map {
>         threads->create(\&dbspawn, "$_.sql", "$_.log");
>     } qw/e5pf16 gpf45 gpf46 gpf48 gpf49 gpf50/;
>

;-) I like the look of this.

> or probably drop the sub altogether and use async:
>
>     $_->join for map {
>         async {
>             system "mysql -t -u rejbyers <$_.sql >$_.log"
>                 and warn "$script failed";
>
>             # there's little point in dieing when the thread's about to
>             # exit anyway...
>         }
>     } qw/.../;
>
> but you may find something a little less compressed easier to
> understand:
>
>     my @jobs = qw/e5pf16 gpf45 .../;
>
>     my @threads = map {
>         async {
>             system "...";
>         }
>     } @jobs;
>
>     $_->join for @threads;
>
> Of course, given that you only use the threads to launch processes, you
> would be better off just launching processes:
>
>     my @procs = map { system 1, "..." } @jobs;
>
>     waitpid $_ for @procs;
>

Are there performance implications for the various forms you suggest,
apart from the obvious that the last just has the overhead of creating
the needed processes while the others create the same processes within
threads that carry their own overhead. But I doubt that this overhead
could be significant when the scripts that are launched take hours to
complete.

> > Is there a Win64::process, or will Win32::process work on a 64 bit
> > version of windows using the 64 bit build of perl?
>
> I don't know. That is, there isn't a Win64::Process, and I don't know
> whether Win32::Process works on Win64. That depends on whether the
> underlying Win32 system calls still work the same way: I suspect they
> do. There are no Win64 results from CPAN Testers (surprise), which is
> usually the first place to look. In any case, since there aren't any
> Win64 ppms, you'd need to compile libwin32 from source. Do you have an
> appropriate compiler?
>

Well, I have MS Visual Studio 2003, so I suppose I could build it here
(with a 64 bit target) and deploy it at the office. I have yet to get
cygwin to work on a 64 bit machine, so the 64 bit machine does not
have a native compiler installed. But wait a minute. On the 64 bit
machine, 'ppm list' includes libwin32 at the bottom of the list. And,
I see in the documentation Activestate provided with this build that
Win32::process is available, along with a long list of other Win32
modules. Mind you, I don't know if it works since I just found it
mere minutes ago.

Thanks

Ted

[Ted]

On Feb 25, 2:30 pm, xhos...@gmail.com wrote:

> Ted <r.ted.by...@rogers.com> wrote:
> > On Feb 25, 1:25=A0pm, Joost Diepenmaat <jo...@zeekat.nl> wrote:
> > > Ted <r.ted.by...@rogers.com> writes:
> > > > Here is my code:
>
> > > > use threads;
>
> > > > my $thr0 =3D threads->create(\&dbspawn,'e5pf16.sql','e5pf16.log');
> > > > $thr0->join;
>
> > > you should join() all threads /after/ creating all of them. a join()
> > > blocks untill the thread is done. In other words, move the ->join()
> > > calls to the end of script.
>
> > > --
> > > Joost Diepenmaat | blog:http://joost.zeekat.nl/|work:http://zeekat.nl/
>
> > Ah, OK.
>
> > While it is not likely in this case, since the scripts normally take
> > several hours to run, what happens if you create half a dozen threads,
> > and one of them finishes before you get to join it?
>
> Then it waits very very patiently for you to "join" it.  No problem.
> (Unless you accumulate a large number of them).
>

At present, we're talking a few dozen at a time. But, I wouldn't mind
having the problem of making it work for thousands, given what that
would imply about our commercial success!

> > Also, I recall reading somewhere that some system calls block
> > everything in the process until the system call returns.
>
> There is the Perl function named "system", then there are calls that perl
> makes into the kernel, referred to as system calls.  They are different
> but confusingly named.  Is your concern about the first or the latter?
>

I assumed the former was just a special case of the latter. I was
concerned about the former, though, because at present I am merely
launched child processes using it. But the latter would always
create a nagging doubt, for when I need to make more interesting
threads.

> > Is that
> > true?
>
> I suspect that it is true of certain kernel-level system calls, but such
> calls should be extremely fast (otherwise, they OS designers should have
> reworked them not to block.)  There isn't much you can do about it in
> a Perl program, or at least not without knowing exactly what the call
> is.
>
> > If so, which are involved, and what would be the workaround?
>
> Find a better OS to run your Perl program on? :)
> Anyway, don't worry about it until it happens.
>

OK. In this case, though, I don't have the option of choosing OS
since our administrator only knows Windows, and I have neither the
time nor the full suite of skills required to administer a unix box.
I develop application level code and analyze data, and even for that
there is insufficient time in the day. Now if Perl could make the
days last 48 hours ....

;-)

Thanks

Ted

[Ted]

On Feb 25, 2:04 pm, Joost Diepenmaat <jo...@zeekat.nl> wrote:

Good to know. Thanks. I didn't consider fork because i was under the
impression that it doesn't work right on Windows.

Thanks

ted

[Joost Diepenmaat]

Ted <r.ted...@rogers.com> writes:

> Good to know. Thanks. I didn't consider fork because i was under the
> impression that it doesn't work right on Windows.

With regards to issues, fork() and threads should be more or less
equivalent on (native, non-cygwin) windows perls, since on windows fork
is implemented using threads. I vaguely recall that there may still be
some issues with fork() on windows, but they should be subtle as long as
you're not relying on /exact/ POSIX semantics for fork on windows.

In any case, threads (on any platform) and fork() on windows both have
their more-or-less subtle issues, so if it works for you, just use it,
but if you run into unexplained issues, they're definitely one of places
trouble may be looming.

[Joost Diepenmaat]

Ted <r.ted...@rogers.com> writes:

>> There is the Perl function named "system", then there are calls that perl
>> makes into the kernel, referred to as system calls.  They are different
>> but confusingly named.  Is your concern about the first or the latter?
>>
> I assumed the former was just a special case of the latter. I was
> concerned about the former, though, because at present I am merely
> launched child processes using it. But the latter would always
> create a nagging doubt, for when I need to make more interesting
> threads.

System calls, on Unix, are a fairly small set of calls (a couple of
hundred, exact number depending on the OS) that can be made from the
program into the kernel. Those system calls basically define the minimum
functionality that programs may need to interact with the OS (and via
the os, with each other).

In addition, there are library functions for C language programs, a lot
of which are built on top of the system calls.

For example, fork(), wait() and exec() are always (well, probably
always) system calls on Unix, but system() is generally a library call
built on top of fork(), wait() and exec(), just like in general malloc()
is implemented as a library function built on the sbrk() syscall, which
is why perl can be built with different malloc() implementations on most
systems. Programmers not building compilers or other programs that need
to interact with the system on a very low level usually won't care which
is which.

Note that some of perl's built-in functions that interact directly with
the OS are sometimes called "syscalls" or "system calls" but may be
implemented in one of the standard C libraries or the perl binary itself
instead of directly in the OS kernel, and which is which may be
different on different OSs and perl builds.

[Ben Morrow]

Quoth smallpond <smal...@juno.com>:

> On Feb 25, 1:07 pm, Ben Morrow <b...@morrow.me.uk> wrote:
> > Quoth smallpond <smallp...@juno.com>:
> >
> > > See perlthrtut and check your perl build with: perl -V
> > > Look for a line like:
> >
> > > usethreads=define use5005threads=undef useithreads=define
> > > usemultiplicity=define
> >
> > > I believe that the last part indicates whether perl tries to use
> > > multiple CPUs.
> >
> > Please don't make things up and post them without verifying.
> > Multiplicity indicates whether it is possible to have several perl
> > interpreters in the same process: this is a requirement for ithreads (so
> > any perl with useithreads=define will also have usemultiplicity=define)
> > but is also used without ithreads for e.g. mod_perl. It has *nothing* to
> > do with use of multiple CPUs: this is determined by your OS's thread
> > scheduling policy.
>

> I didn't make it up, I looked at previous postings in this NG where
> someone has ithreads=define and not usemultiplicity.

You made up the bit about it affecting CPU affinity.

> http://groups.google.com/group/comp.lang.perl.misc/browse_frm/thread/
> 8f27b699aff8a37d/18a233467aaa72d8?hl=en&lnk=gst&q=usemultiplicity#
> 18a233467aaa72d8

[URL broken. Has the 'net collectively forgotten about msg-ids and news:
urls?]

From that posting:
| useithreads=define usemultiplicity=

I don't really know why that is; probably because the user didn't
explicitly ask for multiplicity. However, the important bits come later:

| Characteristics of this binary (from libperl):

| Compile-time options: DEBUGGING MULTIPLICITY USE_ITHREADS USE_LARGE_FILES
| PERL_IMPLICIT_CONTEXT

That list has both USE_ITHREADS and MULTIPLICITY in, so that perl was
built with both, as you can see from

| archname=i386-linux-thread-multi

right at the top.

Ben

[Ben Morrow]

Quoth Ted <r.ted...@rogers.com>:

> On Feb 25, 1:51 pm, Ben Morrow <b...@morrow.me.uk> wrote:
> >

[threads->create(sub { system "" }) vs.
async { system "" } vs.
system 1, "" on Win32
]

>
> Are there performance implications for the various forms you suggest,

No. async {...} is just a shortcut for threads->new(sub {...}).

> apart from the obvious that the last just has the overhead of creating
> the needed processes while the others create the same processes within
> threads that carry their own overhead. But I doubt that this overhead
> could be significant when the scripts that are launched take hours to
> complete.

I doubt *anything* you do in Perl will be significant compared with that
:).

> > > Is there a Win64::process, or will Win32::process work on a 64 bit
> > > version of windows using the 64 bit build of perl?
> >
> > I don't know. That is, there isn't a Win64::Process, and I don't know
> > whether Win32::Process works on Win64. That depends on whether the
> > underlying Win32 system calls still work the same way: I suspect they
> > do. There are no Win64 results from CPAN Testers (surprise), which is
> > usually the first place to look. In any case, since there aren't any
> > Win64 ppms, you'd need to compile libwin32 from source. Do you have an
> > appropriate compiler?
>
> Well, I have MS Visual Studio 2003, so I suppose I could build it here
> (with a 64 bit target) and deploy it at the office.

Hmmm... maybe, maybe not. You need to use *exactly* the same compiler;
in particular, using a different version of MSVC from the one perl was
built with causes the msvcrt.dll linking to go haywire. I've no idea
what compiler ActiveState use for Win64: for Win32 they use MSVC 6,
which is compatible with MinGW gcc, but somehow I doubt MSVC 6 has a
64-bit target.

> I have yet to get cygwin to work on a 64 bit machine, so the 64 bit
> machine does not have a native compiler installed.

Oh, no, a cygwin compiler won't do at all. :)

> But wait a minute. On the 64 bit machine, 'ppm list' includes
> libwin32 at the bottom of the list. And, I see in the documentation
> Activestate provided with this build that Win32::process is available,
> along with a long list of other Win32 modules. Mind you, I don't know
> if it works since I just found it mere minutes ago.

D'oh! Of course, libwin32 has always been a standard part of ActivePerl
(that's where it came from in the first place). If it's there I'd expect
ActiveState to have tested it.

Ben

[Ted]

On Feb 25, 5:54 pm, Ben Morrow <b...@morrow.me.uk> wrote:

> Quoth Ted <r.ted.by...@rogers.com>:> On Feb 25, 1:51 pm, Ben Morrow <b...@morrow.me.uk> wrote:
> > > > Is there a Win64::process, or will Win32::process work on a 64 bit
> > > > version of windows using the 64 bit build of perl?
>
> > > I don't know. That is, there isn't a Win64::Process, and I don't know
> > > whether Win32::Process works on Win64. That depends on whether the
> > > underlying Win32 system calls still work the same way: I suspect they
> > > do. There are no Win64 results from CPAN Testers (surprise), which is
> > > usually the first place to look. In any case, since there aren't any
> > > Win64 ppms, you'd need to compile libwin32 from source. Do you have an
> > > appropriate compiler?
>
> > Well, I have MS Visual Studio 2003, so I suppose I could build it here
> > (with a 64 bit target) and deploy it at the office.
>
> Hmmm... maybe, maybe not. You need to use *exactly* the same compiler;
> in particular, using a different version of MSVC from the one perl was
> built with causes the msvcrt.dll linking to go haywire. I've no idea
> what compiler ActiveState use for Win64: for Win32 they use MSVC 6,
> which is compatible with MinGW gcc, but somehow I doubt MSVC 6 has a
> 64-bit target.
>

While I have MSVS6, you could not pay me to install something that
antiquated on my system. Being used to a reasonably standard
compliant C++ compiler, using some that nacient and broken would be
much too painful.

> > I have yet to get cygwin to work on a 64 bit machine, so the 64 bit
> > machine does not have a native compiler installed.
>
> Oh, no, a cygwin compiler won't do at all. :)
>

Why not? Isn't it just another build of gcc, with libraries required
for programs to work on Windows?

> > But wait a minute.  On the 64 bit machine, 'ppm list' includes
> > libwin32 at the bottom of the list.  And, I see in the documentation
> > Activestate provided with this build that Win32::process is available,
> > along with a long list of other Win32 modules.  Mind you, I don't know
> > if it works since I just found it mere minutes ago.
>
> D'oh! Of course, libwin32 has always been a standard part of ActivePerl
> (that's where it came from in the first place). If it's there I'd expect
> ActiveState to have tested it.
>

Good to know.

Thanks

Ted

[Joost Diepenmaat]

Ted <r.ted...@rogers.com> writes:

>> Oh, no, a cygwin compiler won't do at all. :)
>>
> Why not? Isn't it just another build of gcc, with libraries required
> for programs to work on Windows?

Cygwin's compiler is fine, if you're building for cygwin perl. If you're
building modules for activeperl you probably do need a VC compiler that
matches the one used by active state.

Another perl version that ships with a (free) compiler suite is
strawberry perl. It's pretty new, but you might get it to work for your
stuff.

[smallpond]

The post is about this exact problem and a poster suggests
checking usemultiplicity. I inferred from that that they
were related since I see nothing in the perl documentation.

I find this in README.cygwin:
"Multiplicity is required when embedding Perl in a C program and using
more than one interpreter instance. This works with the Cygwin port."

It says nothing about ithreads. Maybe you made that up?

[Ben Morrow]

Quoth smallpond <smal...@juno.com>:

>
> The post is about this exact problem and a poster suggests
> checking usemultiplicity. I inferred from that that they
> were related since I see nothing in the perl documentation.

Then both the suggestion and your inference were incorrect. This is
understandable, as the documentation is a rather unclear in this area.

> I find this in README.cygwin:
> "Multiplicity is required when embedding Perl in a C program and using
> more than one interpreter instance. This works with the Cygwin port."
>
> It says nothing about ithreads. Maybe you made that up?

No. From perl.h:

#ifdef USE_ITHREADS
# if !defined(MULTIPLICITY)
# define MULTIPLICITY
# endif
#endif

so you can't have USE_ITHREADS without MULTIPLICITY. More obviously,
since multiplicity is what lets you have several Perl interpreters in
one process, and ithreads create a new interpreter for each thread, you
obviously can't have one without the other. And neither has anything to
do with thread scheduling, which, as I said, is entirely under the
control of your OS's thread implementation.

Ben

[Ben Morrow]

Quoth Joost Diepenmaat <jo...@zeekat.nl>:

> Ted <r.ted...@rogers.com> writes:
>
> >> Oh, no, a cygwin compiler won't do at all. :)
> >>
> > Why not? Isn't it just another build of gcc, with libraries required
> > for programs to work on Windows?
>
> Cygwin's compiler is fine, if you're building for cygwin perl. If you're
> building modules for activeperl you probably do need a VC compiler that
> matches the one used by active state.

32-bit ActivePerl is built with MSVC6. 32-bit MinGW gcc is
binary-compatible with MSVC6, and 32-bit ActivePerl ships with support
for using gcc to build extensions. Cygwin gcc, while essentially the
same compiler, has different default options (for instance, it links
with Cygwin's libc.dll by default) so it would be possible to make it
work, but difficult; and of course you'd need the MinGW import libs and
so on.

> Another perl version that ships with a (free) compiler suite is
> strawberry perl. It's pretty new, but you might get it to work for your
> stuff.

I looked at that first :). Adam isn't yet shipping a 64-bit build. I
don't even know if gcc will build Win64 binaries.

Ben

[xho...@gmail.com]

smallpond <smal...@juno.com> wrote:
> On Feb 25, 5:44 pm, Ben Morrow <b...@morrow.me.uk> wrote:
> > Quoth smallpond <smallp...@juno.com>:
>

> The post is about this exact problem and a poster suggests
> checking usemultiplicity.

I was that poster, and what I tentatively suggested was that
multiplicity looked kind of odd and perhaps should be looked into
as we had run out of more concrete ideas. Sometimes we bounce
ideas off each other here, which are intended to be less than the Word
of God.

[smallpond]

On Feb 26, 1:45 am, xhos...@gmail.com wrote:

and it looked like a good shot to me since the perl docs weren't
much help.
--S