Parallel Common-Lisp with at least 64 processors?
Robert Maas, http://tinyurl.com/uh3t
CPUs, with a version of Common Lisp that runs on that computer and
supports distributing the function-applications within a MAPCAR
call across as many CPUs as are available in order to achieve great
speed-up compared to the usual algorithm of performing each
function-application in sequence down the list? Would anybody
volunteer such a system for me to occasionally use across the net
without charge, for research purposes?
Of course if there are dependencies from one function application
to the next, this parallel-mapcar wouldn't be appropriate. But I
have an application where I need to apply a single function to a
large number of arguments in parallel. I'm running it with nearly
three hundred at the moment, whereupon it takes several minutes to
do them all in succession, which isn't too bad if done rarely, but
I envision doing the same with thousands of arguments, whereby the
time to do them in succession would be prohibitive.
Tim Bradshaw
http://tinyurl.com/uh3t) wrote:
> Does anybody you know of have access to a computer with at least 64
> CPUs, with a version of Common Lisp that runs on that computer and
> supports distributing the function-applications within a MAPCAR
> call across as many CPUs as are available in order to achieve great
> speed-up compared to the usual algorithm of performing each
> function-application in sequence down the list? Would anybody
> volunteer such a system for me to occasionally use across the net
> without charge, for research purposes?
Those systems typically don't exist any more, because the memory
coherency & latency costs are too high for distributing stuff in such
a fine-grained way. There are plenty of machines with lots of cores -
the systems I deal with mostly have up to 144, though we very seldom
use them in a single domain (and no, they can't be lent out). But
these systems typically make use of significantly coarser-grained
multiprocessing, so that you're not just killed by communication cost
all the time.
One interesting possibility though is heavily multicored/multithreaded
processors such as Sun's Niagara family. I don't know what the
latency issues are between cores for these things, but between HW
threads (which look like virtual CPUs) it is essentially zero I
think. I think the current largest of these systems is currently 16
core / 128 thread.
I doubt there are CL implementations which take advantage of these
systems however.
Patrick May
> On Apr 26, 2:21 am, lisp1.3.CalRob...@SpamGourmet.Com (Robert Maas,
> http://tinyurl.com/uh3t) wrote:
>> Does anybody you know of have access to a computer with at least 64
>> CPUs, with a version of Common Lisp that runs on that computer and
>> supports distributing the function-applications within a MAPCAR
>> call across as many CPUs as are available in order to achieve great
>> speed-up compared to the usual algorithm of performing each
>> function-application in sequence down the list? Would anybody
>> volunteer such a system for me to occasionally use across the net
>> without charge, for research purposes?
>
> Those systems typically don't exist any more, because the memory
> coherency & latency costs are too high for distributing stuff in
> such a fine-grained way.
Don't do that, then. ;-)
Space Based Architecture is one technique for avoiding the
distribution overhead. The basic idea is to co-locate the logic and
data for a particular instance of an end-to-end use case into a single
processing unit that also provides a messaging infrastructure. This
allows near linear scalability across a broad range of hardware
resources. (Full disclosure: I do this for my day job.)
[ . . . ]
> I doubt there are CL implementations which take advantage of these
> systems however.
That's too bad. I'd much rather implement this architecture in
Lisp than in Java. How good is the SMP support in commercial Lisps?
Regards,
Patrick
------------------------------------------------------------------------
S P Engineering, Inc. | Large scale, mission-critical, distributed OO
| systems design and implementation.
p...@spe.com | (C++, Java, Common Lisp, Jini, middleware, SOA)
Alex Mizrahi
TB> systems however.
1. i think what Robert does can be just launched in multiple processes, even
on different machines.
Robert is just too lazy to implement this, so he want's support in core
language.
(also he could get a speedup from just faster CPU.
probably some modern 4-core Xeon would be not much worse than a rusty
64-CPU machine or that Niagara monster)
2. i doubt there's huge need in parallelism support in core language.
most likely parallel:mapcar implemented in library will do just fine.
3. ABCL should work fine on niagaras, since it uses underlying JVM threading
4. SBCL and SCL might work too
John Thingstad
<tfb+g...@tfeb.org>:
"The Scieneer implementation of the Common Lisp language has been
developed to support enterprise and performance computing applications,
and features innovative multi-threading support for symmetrical
multi-processor systems which clearly differentiates it from the
competition."
http://www.scieneer.com/scl/index.html
It does not support vectorization if that is what you mean.
--------------
John Thingstad
Tim Bradshaw
> such a fine-grained way.
>
> Don't do that, then. ;-)
That was pretty much my suggestion.
dlwei...@gmail.com
add-on boxes to run Java. It's (at least a little bit) along the
lines
of the plug-in-board Lisp machines that Symbolics made and the bigger
ones they once contemplated. It'll be interesting to see how well
they sell.
Robert Maas, http://tinyurl.com/uh3t
> i think what Robert does can be just launched in multiple
> processes, even on different machines.
Yes, but I was wondering if any existing Lisp implementation
provided a way specify a parallel MAPCAR which would then be
automatically distributed per some configuration that had been set
up previously. For example, a cooperative set of CPUs might be in
an arrangement whereby whenever one of them wants to run a burst of
parallel computation and the others have some idle time then
FORK-MAPCAR automatically distributes across all the CPUs. Of
course for this to be practical they must all be physically located
close together so that the master algorithm can be quickly
distributed onto all the CPUs (or accessed in real time by shared
memory bus).
For launching on multiple machines connected across the InterNet,
like the way SETI@home is done, I don't imagine FORK-MAPCAR being
practical. Instead, only a small set of algorithms would be
pre-distributed to the various hosts, and each such host registered
as now available to run that particular algorithm, and then when
there's a task to be done in parallel the parallel sub-tasks would
be sent out per that registry.
> Robert is just too lazy to implement this, so he want's support
> in core language.
That's an extremely hostile/nasty thing for you to say about
somebody you've never met in person. You have no idea how much work
I do, without pay, to accomplish various major processing tasks.
I was simply inquiring whether any *one* (1) person might have
access to a single multi-CPU with such auto-distributing software
up&running and available for occasional use by outsiders when not
otherwise busy. That seemed a more reasonable request than to ask
hundreds of different people to *each* volunteer their single-CPU
servers to join me in building a brand-new protocol for
distributing software for such tasks and maitaiing a registery of
which hosts have which software and which hosts are available at
any given time and distributing tasks via this new protocol we've
invented. But if you think asking for hundreds of volunteers to
work with me on a new protocol would be a more reasonable request,
so be it.
> (also he could get a speedup from just faster CPU. probably some
> modern 4-core Xeon would be not much worse than a rusty 64-CPU
> machine or that Niagara monster)
Well if *one* person has access to a CPU that is several orders of
magnitude faster than the Intel processor used by a commercial ISP,
and would be willing to let me use it from time to time to see how
much faster it is at the kinds of tasks I have in mind, I'd be
willing to give that a try too. But that really doesn't scale well,
compared to more CPUs running in parallel. If I want to tackle
processing a batch of a hundred million records at a time, I really
don't think any single CPU or even quad-CPU that fast exists at
all.
> i doubt there's huge need in parallelism support in core
> language. most likely parallel:mapcar implemented in library will
> do just fine.
My wording must have been unclear. parallel:mapcar in a library,
using some super-efficient built-in mechanism for forking on
64-core computer, would satisfy my query just fine. But
parallel:mapcar in a library without any support for efficient
threads in the underlying CL would probably not work well.
> 3. ABCL should work fine on niagaras, since it uses underlying
> JVM threading
What's the largest number of available CPUs ("cores") that anybody
in this newsgroup has available for playing around with this sort
of thing using ABCL?
4. SBCL and SCL might work too
Same question for SBCL and SCL.
Robert Maas, http://tinyurl.com/uh3t
> "The Scieneer implementation of the Common Lisp language has been
> developed to support enterprise and performance computing
> applications, and features innovative multi-threading support for
> symmetrical multi-processor systems which clearly differentiates it
> from the competition."
> http://www.scieneer.com/scl/index.html
Hmm, it looks pretty nice. Does anyone here have direct experience
with it sufficient to rate it as to how well it actually works in
practice?
Browsing links from there:
Linkname: UFFI: Universal Foreign Function Interface for Common Lisp
URL: http://uffi.b9.com/
... Every Common Lisp implementation has a method
for interfacing to such libraries. Unfortunately, these method vary
widely amongst implementations. ^s
(typo, anybody here have authorization to fix the typo?)
... UFFI wraps this common subset of functionality with
it's own syntax ...
x (another typo)
> It does not support vectorization if that is what you mean.
<http://en.wikipedia.org/wiki/Vectorization>
Vectorization, in computer science, is the process of converting a
computer program from a scalar implementation, which does an operation
on a pair of operands at a time, to a vectorized program where a
single instruction can perform multiple operations or a pair of vector
(series of adjacent values) operands. Vector processing is a major
feature of both conventional and modern supercomputers.
OK, clarification: When I use the term, I'm not referring to the
*automatic* conversion from a conventional program to a vectorized
version. I'm merely referring to the final result, a single CPU
instruction that processes a whole array of data with the same
function with memory access and actual computing overlapped as fast
as the internal busses in the CPU can accomodate. For example, a
single instruction might compute the pairwise difference of two
arrays writing the differences to a pre-allocated third array, and
a second instruction might compute the squares of those
differences, and a third instruction might compute the sum of those
squares, thereby computing the variance between two vectors in just
three machine instructions. A fourth, non-vectorized operation,
would compute the square root of that sum of squares of
differences, thereby computing the Cartesian distance between the
two original vectors.
But this is a completely separate topic from the multi-CPU question
I posed. For my current application, I have a set of records, each
of which is to be pre-processed in exactly the same way:
- Convert to list of words, all lower case.
- Convert each word to bigrams trigrams and tetragrams, separately,
and accumulate those results separately for each list-of-words.
I imagine all of that to be runnable on parallel processes, hence
the 64k query. No vectorization happens there.
Next:
- Accumulate all those bigram trigram and tetragram statistics
separately for the entire corpus, yielding three whole-corpus
histograms.
That would be done on the main computer after getting the many
individual triples from the sub-processes.
Next:
- Divide each of the three single-record histograms for each record
by the whole-corpus histrogram for that class among the three, to
yield the three frequency-ratio histograms for each such record.
- Merge those three frequency-ratio histograms for the record into
a single ratio histogram for that record.
- Normalize that merged ratio historgram to have Cartesian length 1.
- Compute the ProxHash, which is a 64-component vector, for the record.
I imagine all of that to be runnable on parallel processes, hence
the 64k query. No vectorization happens there.
Next:
- Perform various calculations of the difference between partial or
full ProxHash vectors in the process of building a nearest-neighbor
network.
This is where vectorization would happen, but *not* on distributed
computers, merely on one (1) moderately vectorized computer, able
to compute Cartesian distance between two vectors (up to 64
elements in each vector) in four machine instructions
(diff,square,sum,sqrt). The first three operations would be
standard vectorized opcodes, whereas the sqrt would be specially
micro-coded to run extremely quickly by finite Newton's method
pipelined directly in the CPU internal bus structure. (It may be
that some commercial CPUs already include a built-in
vendor-supplied SQRT opcode, in which case of course no additional
micro-coding would be needed.)
Of course the CPU I speak of for the vectorized calculation might
be an auxiliary "vector processor", or it might be functionality
built into a high-performance main CPU.
John Thingstad
http://tinyurl.com/uh3t <lisp1.3....@SpamGourmet.Com>:
> OK, clarification: When I use the term, I'm not referring to the
> *automatic* conversion from a conventional program to a vectorized
> version. I'm merely referring to the final result, a single CPU
> instruction that processes a whole array of data with the same
> function with memory access and actual computing overlapped as fast
> as the internal busses in the CPU can accomodate. For example, a
> single instruction might compute the pairwise difference of two
> arrays writing the differences to a pre-allocated third array, and
> a second instruction might compute the squares of those
> differences, and a third instruction might compute the sum of those
> squares, thereby computing the variance between two vectors in just
> three machine instructions. A fourth, non-vectorized operation,
> would compute the square root of that sum of squares of
> differences, thereby computing the Cartesian distance between the
> two original vectors.
Have you seen this paper on the MapReduce algorithm goole uses?
http://labs.google.com/papers/mapreduce.html
It might give you soem idea of how to roll your own.
--------------
John Thingstad
Alex Mizrahi
??>> processes, even on different machines.
RM> Yes, but I was wondering if any existing Lisp implementation
RM> provided a way specify a parallel MAPCAR which would then be
RM> automatically distributed per some configuration that had been set
RM> up previously.
no, i mean different thing -- in many cases it's possible to parallelize
processing with mere shell scripts, requiring no support from
language/libraries itself.
from what you're saying, you task is like that -- it should be easy to write
some shell scripts that launch processing on multiple processes/nodes.
you only need some fancy parallelization support if you have either:
* lots of small (executed in less than, say, 0.1 sec) tasks that depend on
each other: outputs of one function are fed to input of another
* enormous amounts of input data that is not easy to share
RM> For launching on multiple machines connected across the InterNet,
RM> like the way SETI@home is done, I don't imagine FORK-MAPCAR being
RM> practical. Instead, only a small set of algorithms would be
RM> pre-distributed to the various hosts, and each such host registered
RM> as now available to run that particular algorithm, and then when
RM> there's a task to be done in parallel the parallel sub-tasks would
RM> be sent out per that registry.
between single huge SMP system and SETI@home scale there are "cluster"
solutions, and they are actually easiest to work with.
??>> Robert is just too lazy to implement this, so he want's support
??>> in core language.
RM> That's an extremely hostile/nasty thing for you to say about
RM> somebody you've never met in person.
no, it would be hostile to call you asshole (i have no problems calling
people assholes on usenets either).
"lazy" is not even always a bad thing -- maybe you just have other stuff to
do.
RM> You have no idea how much work I do, without pay, to accomplish
RM> various major processing tasks.
sorry, but often "without pay" means "what nobody needs".
RM> I was simply inquiring whether any *one* (1) person might have
RM> access to a single multi-CPU with such auto-distributing software
RM> up&running and available for occasional use by outsiders when not
RM> otherwise busy. That seemed a more reasonable request than to ask
RM> hundreds of different people to *each* volunteer their single-CPU
RM> servers to join me in building a brand-new protocol for
RM> distributing software for such tasks and maitaiing a registery of
RM> which hosts have which software and which hosts are available at
RM> any given time and distributing tasks via this new protocol we've
RM> invented. But if you think asking for hundreds of volunteers to
RM> work with me on a new protocol would be a more reasonable request,
RM> so be it.
as i've mentioned above, you could also ask for a "cluster" -- just bunch of
small servers connected via local network.
they are *much* easier to find than huge SMP machine -- for example, when i
was a student in university, we had a class with 10 machines available to
us, and sometimes we were launching distributed rendering tasks on them.
and local cluster is much easier to manage than distributed network in
internet: for example, in cluster you can launch task on a remote node as
easy as
ssh remote-node sbcl --load process10.lisp > results10.txt
also, you can get readily available "grid computing" software, that is able
to automatically find idle nodes and use them to launch processes.
but, you know, if you mention this "brand-new" protocol, this is actually
interesting thing -- if you'll actually *implement* this and release as open
source software, it would be quite an interesting thing to a Lisp community.
and it would be much easier to convince people to contribute their resources
to testing this distributed network.
RM> Well if *one* person has access to a CPU that is several orders of
RM> magnitude faster than the Intel processor used by a commercial ISP,
"Intel processor used by a commercial ISP" is hardly a benchmark. if they've
bought it some 5 years ago,
i'm pretty sure current top Intel processors are more than 10 times faster.
??>> 3. ABCL should work fine on niagaras, since it uses underlying
??>> JVM threading
RM> What's the largest number of available CPUs ("cores") that anybody
RM> in this newsgroup has available for playing around with this sort
RM> of thing using ABCL?
RM> 4. SBCL and SCL might work too
RM> Same question for SBCL and SCL.
i have 8-core machine, here, seems to be working fine with SBCL -- i was
able to launch 8 threads that do processing, and indeed all 8 cores were
utilized.
i'm pretty sure ABCL and SCL would work fine there too.
i'd be glad to give you access to this system, but i don't owned, and i
don't think i'm authorized to share access to it, sorry
Robert Maas, http://tinyurl.com/uh3t
> in many cases it's possible to parallelize processing with mere
> shell scripts, requiring no support from language/libraries itself.
> from what you're saying, you task is like that -- it should be easy
> to write some shell scripts that launch processing on multiple
> processes/nodes.
That would seem to require either re-starting Lisp multiple times
and re-loading the application-level software on each such clone
process, or somehow saving an executable Lisp that already has all
the software loaded re-starting that saved executable multiple
times. Question: If the latter option is possible, then can this be
done on an ordinary operating system in such a way that all the
clone processes share the same memory until and unless one of them
breaks out of pure-page sharing by modifying one of the pure pages
making it impure for just that one process, after which all the
other processes share that page (until each of them in turn also
modifies it) but still *all* processes share the rest of the pages
which nobody has made impure yet? I believe this was possible on
ITS, but I don't know if it works that way on Unix or Linux. If all
that is possible, then indeed that might be a feasible way to
parallelize my algorithms on a computer that had tens or hundreds
of CPU cores.
On the other hand, perhaps a computer with so many CPU cores might
use some non-standard operating system in order to better manage
all the parallel-processing that is possible. So I guess my
original query about anyone having access to such a system is more
general than I originally imagined.
> you only need some fancy parallelization support if you have either:
> * lots of small (executed in less than, say, 0.1 sec) tasks that
> depend on each other: outputs of one function are fed to input of
> another
> * enormous amounts of input data that is not easy to share
For my application, it typically takes a fraction of a second to a
few seconds per single ProxHash calculation, with all the Lisp code
interpreted (actually JIT-compiled in CMUCL). So if I had tens of
thousands of ProxHash calculations to perform (conceptually all in
parallel), and a computer with 64 or more CPU cores, would a shell
script be able to farm them off to new processes at a average rate
of 1/N second per new process where N is the number of CPU cores,
in order to keep all the CPU cores constatly busy, in order to
finish the ProxHash calculations in minimum real time?
> between single huge SMP system and SETI@home scale there are "cluster"
> solutions, and they are actually easiest to work with.
So that would be something like 16 or more quad-core computers
connected on a very fast localnet, so that ProxHash calculations
could be farmed off to localnet-remote computers rapidly enough to
keep all four of their each-cores busy?
> but often "without pay" means "what nobody needs".
More likely what I don't know how to advertise without spamming so
nobody knows what's available because I'm not willing to spam just
to find beta testers. There might be lots of people who need the
kind of software I develop, but they have no practical way to know
that I've developed anything like that because I don't have the
money to place ads on prime-time television, so they don't even
think of the kinds of solutions I invent, so they don't realize
what they're missing.
> "Intel processor used by a commercial ISP" is hardly a benchmark.
> if they've bought it some 5 years ago, i'm pretty sure current top
> Intel processors are more than 10 times faster.
I am not privy to such knowledge regarding the commercial ISP where
I have my shell account. From time to time I see system messages
about upgrading to a new CPU or replacing a CPU that has been
flaky, but I don't have total recall memory so I don't remember the
details. A few weeks ago the shell machine hung and was down for
1.5 hours, and when it came back up there was a system message
saying there'd be a new CPU soon. But now the message is gone and
there's no announcement of a new CPU so I don't know what happened.
Do you know of any good benchmark that a single user can run on his
personal account on a commercial ISP that has hundreds of users,
some time when it's not too busy? For that matter, do you know any
easy way for a single user to determine how busy the computer is at
the moment? Right now, finger shows (most columns removed, rows sorted):
Idle Login
Wed
Wed
Wed
Wed
1:30 Wed
2:02 Wed
2:21 Wed
6:42 Wed
14:41 Sun
22:42 Mon
1d Tue
3d Sun
3d Sun
From looking at that, can you estimate how busy the system is?
Otherwise, is there some other program that will tell me?
Tops-20 at Stanford had a command 'load' which gave that info,
and Tenex had control-T, at least that's how I vaguely remember it.
That was so long ago I am not at all sure, and Unix has neither,
so it's irrelevant, except to give you a general idea what info I need
before running any sort of CPU timing test. Not just because results
would be inaccurate if the system is heavily loaded, but I'm not
allowed to use so much CPU time that it interferes with other users.
Robert Maas, http://tinyurl.com/uh3t
> Have you seen this paper on the MapReduce algorithm goole [sic] uses?
> http://labs.google.com/papers/mapreduce.html
No, I never heard of it before. Taking a look now ...
Users specify a map function that processes a key/value pair to generate
a set of intermediate key/value pairs, and a reduce function that merges
all intermediate values associated with the same intermediate key.
So if I understand it correctly, if I want to reduce all the data
to a single value, then I generate the intermediate keys all the
same, but using different intermediate keys allows several
independent data-reduction tasks to run in parallel to produce
multiple keyed outputs? So the first stage runs fully parallel per
the different input pairs, but then the second stage runs at a
lesser degree of parallelism just one per intermediate key?
a typical MapReduce computation
processes many terabytes of data on thousands of machines.
It doesn't sound like Google would let me play with it for free.
> It might give you soem idea of how to roll your own.
Without access to a cluster of machines, there's no way I can
develop *any* such software myself. And if I tried to clone
thousands of active processes on this shell account just to develop
the software, I'd get in bad trouble with the admin.
But at least in principle it sounds like a nifty mathematical
abstraction that indeed should have many possible uses, just as you
and Google's WebPage claim. So thanks for sharing the info.
Madhu
Wrote on Thu, 08 May 2008 00:48:01 -0700:
| That would seem to require either re-starting Lisp multiple times and
| re-loading the application-level software on each such clone process,
| or somehow saving an executable Lisp that already has all the software
| loaded re-starting that saved executable multiple times.
This is the premise behind the
<URL:http://franz.com/support/tech_corner/forksmp.lisp>
<URL:http://franz.com/support/tech_corner/forksmp.lhtml>
Which attempts to use multiple processes over UNIX process granularity,
[search google for "cmucl forksmp" if you need an initial cmucl version
of this]
| Question: If the latter option is possible, then can this be done on
| an ordinary operating system in such a way that all the clone
| processes share the same memory until and unless one of them breaks
| out of pure-page sharing by modifying one of the pure pages making it
| impure for just that one process, after which all the other processes
| share that page [...] I believe this was possible on ITS, but I don't
| know if it works that way on Unix or Linux.
"modern" unix VMs support mmap(2) calls with MAP_PRIVATE, which does
copy on write. CMUCL, for example uses this in mapping the corefile, so
you get the behaviour for free
| If all that is possible, then indeed that might be a feasible way to
| parallelize my algorithms on a computer that had tens or hundreds of
| CPU cores.
I suspect such innovation is not encouraged in the current scenario.
Instead you are adviced to use extant thread APIs in partitioning your
tasks, and leave the memory management and maping of
threads-to-processors to the OS thread scheduler. The OS/CPU vendors
are expected to take care of caching and other shennanigans.
--
Madhu
alexey...@gmail.com
http://tinyurl.com/uh3t) wrote:
> Without access to a cluster of machines, there's no way I can
> develop *any* such software myself. And if I tried to clone
> thousands of active processes on this shell account just to develop
> the software, I'd get in bad trouble with the admin.
>
Renting instances on Amazon Elastic Cloud?
Pascal J. Bourguignon
> Do you know of any good benchmark that a single user can run on his
> personal account on a commercial ISP that has hundreds of users,
> some time when it's not too busy?
On a linux system you can have a look at /proc/cpuinfo.
cat /proc/cpuinfo
Otherwise, just run your own benchmark on your various systems. Mine
is compilation of emacs:
cd emacs-22.1 ; time (./configure && make)
> For that matter, do you know any
> easy way for a single user to determine how busy the computer is at
> the moment? Right now, finger shows (most columns removed, rows sorted):
Finger is not the right too. To know the load of a system, the right
tool is obviously:
loadavg
well, it may not exist (anymore) in which case, again obviously:
uptime
would give you the load average such as:
14:44:30 up 4:39, 2 users, load average: 0.05, 0.04, 0.13
> That was so long ago I am not at all sure, and Unix has neither,
> so it's irrelevant, except to give you a general idea what info I need
> before running any sort of CPU timing test. Not just because results
> would be inaccurate if the system is heavily loaded, but I'm not
> allowed to use so much CPU time that it interferes with other users.
But like always in unix, you don't usually test whether resources
exist before using them, because by the time you try to use them, they
may already be exhausted, given that it's a multiuser system without
resource reservation. What you just do, is to use the resources you
need. And if you want to be nice to the other users, then just that,
be nice, using the nice(1) command:
nice my-big-cpu-very-intensive-program
This will make your my-big-cpu-very-intensive-program program use all
the CPU time it can, but only when there is no other process competing
for that resource.
Another tool you may use is batch(1), which enqueues jobs, and will
start them only when the load average is low enough. That is, if it's
installed on your system. If you had a lot of little tasks to
execute, you could use batch.
--
__Pascal Bourguignon__
Tim Bradshaw
http://tinyurl.com/uh3t) wrote:
> Without access to a cluster of machines, there's no way I can
> develop *any* such software myself. And if I tried to clone
> thousands of active processes on this shell account just to develop
> the software, I'd get in bad trouble with the admin.
Of course you can. You'll just take a performance hit since you'll be
emulating a large number of cores on one. You may also not notice
bugs related to concurrency. There's a very long history of people
doing this. Indeed, time on big machines is typically so scarce and
expensive that very few people develop code on them: that would be an
absurd waste of resources.
--tim
Tim Bradshaw
http://tinyurl.com/uh3t) wrote:
> Question: If the latter option is possible, then can this be
> done on an ordinary operating system in such a way that all the
> clone processes share the same memory until and unless one of them
> breaks out of pure-page sharing by modifying one of the pure pages
> making it impure for just that one process, after which all the
> other processes share that page (until each of them in turn also
> modifies it) but still *all* processes share the rest of the pages
> which nobody has made impure yet? I believe this was possible on
> ITS, but I don't know if it works that way on Unix or Linux. If all
> that is possible, then indeed that might be a feasible way to
> parallelize my algorithms on a computer that had tens or hundreds
> of CPU cores.
That typically is the way modern Unix(-oid) systems work.
(Disclaimer: I'm only deeply familiar with Solaris.) However on any
significant modern machine things are quite a lot more complex because
access to memory is not simple. So (for instance) almost any system
will, in fact, aggressively make copies of pages it is using in the
caches of the core concerned. Even for main memory, I think most
large SMP systems are now either explicitly or implicitly NUMA,
meaning you have to be concerned about which cores pages are near, and
you may well want to move pages around within main memory because of
this. Solaris does the latter, for instance.
>
> On the other hand, perhaps a computer with so many CPU cores might
> use some non-standard operating system in order to better manage
> all the parallel-processing that is possible. So I guess my
> original query about anyone having access to such a system is more
> general than I originally imagined.
The break is almost certainly between systems with shared address
spaces (probably with hardware-supported coherency), where Unix does
very well, and scales to quite large numbers of cores (hundreds), and
systems which don't, where you probably effectively have multiple OS
instances. Any sufficiently large system is of the latter kind, but
CC-shared-address systems can get quite big. The edge case is shared-
address-space non-coherent systems: I don't know if there are many of
those (I think some of the 90s Cray boxes like the T3d and T3e were
like that, but I am not sure).
--tim
George Neuner
You can pretty well approximate a cluster on a multi-core machine.
Finding latency bugs will still be a problem though ... a debug
framework that inserts short random delays into message delivery can
help with that (don't know of a canned one offhand, but it's easy
enough to funnel messaging though a server thread that does it).
George
--
for email reply remove "/" from address
Tim Bradshaw
> You can pretty well approximate a cluster on a multi-core machine.
You can approximate one on a single-core machine: that's the wonder of
computers. Of course the performance characteristics will be a
different and you'll have to do some work to make sure bugs show up,
but it can be done, and anyone who is claiming that they can never
develop any code because they don't have access to time on very
expensive hardware is just trying to put obstacles in their own way to
avoid doing things.
Does no one remember the *Lisp simulateor? I ran that on a Sun 3.
George Neuner
<tfb+g...@tfeb.org> wrote:
>On May 13, 6:19 pm, George Neuner <gneuner2/@/comcast.net> wrote:
>
>> You can pretty well approximate a cluster on a multi-core machine.
>
>You can approximate one on a single-core machine
Of course - been there, done that.
But sans the aid of a simulator that can pretty faithfully reproduce
the behavior of a multiprocessor, it can be quite difficult to make
sure your code is clear of synchronization and latency bugs. Actually
having a several processors (or cores) to run in parallel can speed up
tripping a hidden problem.
[Not that should be any hidden problems, but IME many developers have
considerable difficulty moving between the different abstraction
levels and seeing the application design at all levels. Thus
debugging tends to be a much larger part of concurrent designs.]
>: that's the wonder of
>computers. Of course the performance characteristics will be a
>different and you'll have to do some work to make sure bugs show up,
>but it can be done, and anyone who is claiming that they can never
>develop any code because they don't have access to time on very
>expensive hardware is just trying to put obstacles in their own way to
>avoid doing things.
>
>Does no one remember the *Lisp simulateor? I ran that on a Sun 3.
Never used it. And IIRC it wasn't a multiprocessor Lisp simulation.
Tim Bradshaw
>
> Of course - been there, done that.
>
> But sans the aid of a simulator that can pretty faithfully reproduce
> the behavior of a multiprocessor, it can be quite difficult to make
> sure your code is clear of synchronization and latency bugs. Actually
> having a several processors (or cores) to run in parallel can speed up
> tripping a hidden problem.
I think we're agreeing here :-). My real point was that there's no
reason to wait for time on expensive and perhaps not-yet-existing HW
to write code which will exploit it, and indeed almost all code for
such HW is written on simulators of one kind or another.
> Never used it. And IIRC it wasn't a multiprocessor Lisp simulation.
Well, it gave you the constructs & operators which you had on the
Connection Machine (CM-2?) which was a fairly large multiprocessor,
albeit of a kind (SIMD) not seen much nowadays (though aren't a lot of
graphics cards essentially SIMD systems?)
--tim
George Neuner
<tfb+g...@tfeb.org> wrote:
>On May 14, 2:36 am, George Neuner <gneuner2/@/comcast.net> wrote:
>
>>On Tue, 13 May 2008 14:44:38 -0700 (PDT), Tim Bradshaw
>><tfb+g...@tfeb.org> wrote:
>
>>>Does no one remember the *Lisp simulator?
>
>> Never used it. And IIRC it wasn't a multiprocessor Lisp simulation.
>
>Well, it gave you the constructs & operators which you had on the
>Connection Machine (CM-2?) which was a fairly large multiprocessor,
>albeit of a kind (SIMD) not seen much nowadays
I remember the CM-2 well, though I never used *Lisp ... at the time
most of my programming was in C* with a bit in Paris (CM assembler).
>(aren't a lot of graphics cards essentially SIMD systems?)
Depends on your definitions I suppose. A single instruction does
operate on multiple data elements ...
But to my thinking, GPUs, SSE units on x86, etc. are really (short)
vector processors - all the data elements are offset from a common
address.
On the other hand, on the early CMs and other by-gone machines that I
think of as being "real" SIMD, each ALU had independent address
generation ... allowing, for example, each ALU to follow its own path
through a graph.
It's quite difficult to do the same kind of programming using vector
ops ... whether they really are technically "SIMD" or not, I don't
consider them to be in the same league.
Tim Bradshaw
>
> But to my thinking, GPUs, SSE units on x86, etc. are really (short)
> vector processors - all the data elements are offset from a common
> address.
>
yes, that's a much better view of what they are, I think.
Robert Maas, http://tinyurl.com/uh3t
> On a linux system you can have a look at /proc/cpuinfo.
That does't help me on my shell account running FreeBSD Unix.
> cat /proc/cpuinfo
% more /proc/cpuinfo
/proc/cpuinfo: No such file or directory
> To know the load of a system, the right tool is obviously:
> loadavg
% loadavg
loadavg: Command not found.
> well, it may not exist (anymore) in which case, again obviously:
> uptime
% uptime
5:01AM up 3 days, 25 mins, 9 users, load averages: 5.11, 6.07, 6.49
(Oh oh, there's that cliche about third time ... charm?)
Anyway, with load around 5 or 6 it looks too busy for me to run any
major CPU-speed benchmark.
> would give you the load average such as:
> 14:44:30 up 4:39, 2 users, load average: 0.05, 0.04, 0.13
I don't know what machine gave you *those* numbers, but that
machine is sorely underutilized?
> And if you want to be nice to the other users, then just that, be
> nice, using the nice(1) command:
> nice my-big-cpu-very-intensive-program
Since all my heavy computations are from a single CMUCL REP, I
guess I need to say "nice lisp" when starting up in the morning,
and leave it that way all day until I'm ready to (quit) and go to
bed?
> Another tool you may use is batch(1), which enqueues jobs, and
> will start them only when the load average is low enough.
That's worthless for my purpose, unless you know a way to use it to
deal with interactive Lisp session.
> If you had a lot of little tasks to execute, you could use batch.
If I have a lot of little tasks to execute, I put them into a PROGN
or LIST, and if I'm going to do the same sequence of tasks many
times over different days then I might put them into a DEFUN.
For example, here is the LIST of tasks I do when I first start up CMUCL:
(list
(setq *gc-verbose* nil)
(unless (fboundp 'funct+shortdir+fn1-mayload)
(load "/home/users/rem/LispWork/2007-2-mayload.lisp"))
(funct+shortdir+fn1-mayload 'filenamebase-std-roll-before :LISP "2007-2-roll")
(funct+shortdir+fn1-mayload 'load-file-by-method :LISP "2005-8-readers")
(funct+shortdir+fn1-mayload 'dirspec+filnam+method+globsym-may-load :LISP "2008-4-MayLoad")
(funct+shortdir+fn1-mayload 'make-empty-heap :LISP "2001-B-heap")
(funct+shortdir+fn1-mayload 'string-read-words-batch-sort :LISP "2008-3-WordHist")
(funct+shortdir+fn1-mayload 'phvec-normalize :LISP "2008-3-ProxHash")
(funct+shortdir+fn1-mayload 'device-to-ar+mr+mc :LISP "2008-3-TextGraphics")
(funct+shortdir+fn1-mayload 'trans-skills-lines :LISP "2008-3-TopPH")
(funct+shortdir+fn1-mayload 'trans-skills-3d-012-links :LISP "2008-5-TopPH")
)
I don't see how batch could possibly help me with those tasks.
Pascal J. Bourguignon
>> From: p...@informatimago.com (Pascal J. Bourguignon)
>> On a linux system you can have a look at /proc/cpuinfo.
>
> That does't help me on my shell account running FreeBSD Unix.
>
>> cat /proc/cpuinfo
>
> % more /proc/cpuinfo
> /proc/cpuinfo: No such file or directory
>
>> To know the load of a system, the right tool is obviously:
>> loadavg
>
> % loadavg
> loadavg: Command not found.
>
>> well, it may not exist (anymore) in which case, again obviously:
>> uptime
>
> % uptime
> 5:01AM up 3 days, 25 mins, 9 users, load averages: 5.11, 6.07, 6.49
> (Oh oh, there's that cliche about third time ... charm?)
> Anyway, with load around 5 or 6 it looks too busy for me to run any
> major CPU-speed benchmark.
Well, 5:00 AM, is often the time at which the last cron tasks of the
night are just started. Try it at another random time.
>> would give you the load average such as:
>> 14:44:30 up 4:39, 2 users, load average: 0.05, 0.04, 0.13
>
> I don't know what machine gave you *those* numbers, but that
> machine is sorely underutilized?
Not all the time :-) Also it's a quad-core, so the average is about
divided by four, compared to a normal processor.
>> And if you want to be nice to the other users, then just that, be
>> nice, using the nice(1) command:
>> nice my-big-cpu-very-intensive-program
>
> Since all my heavy computations are from a single CMUCL REP, I
> guess I need to say "nice lisp" when starting up in the morning,
> and leave it that way all day until I'm ready to (quit) and go to
> bed?
Not necessarily. But you can only increase the nicety level of your processes.
$ clisp -x '(loop :for i :from 0 :do (* i i))' >/dev/null 2>&1 &
[2] 13589
$ renice +4 13589
13589: old priority 0, new priority 4
$ renice +6 13589
13589: old priority 4, new priority 6
$ renice +2 13589
renice: 13589: setpriority: Permission denied
$ renice +5 13589
renice: 13589: setpriority: Permission denied
$ renice +6 13589
13589: old priority 6, new priority 6
$ renice +7 13589
13589: old priority 6, new priority 7
Only root can reduce it.
So you could start with a normal nicety, do your loading and
compilation full speed, and once you launch your computation, you can
detach your process and renice it.
>> Another tool you may use is batch(1), which enqueues jobs, and
>> will start them only when the load average is low enough.
>
> That's worthless for my purpose, unless you know a way to use it to
> deal with interactive Lisp session.
Indeed, that would be useful only if you could cut your task in
smaller batches that could be run independently and without
supervision.
>> If you had a lot of little tasks to execute, you could use batch.
>
> If I have a lot of little tasks to execute, I put them into a PROGN
> or LIST, and if I'm going to do the same sequence of tasks many
> times over different days then I might put them into a DEFUN.
> For example, here is the LIST of tasks I do when I first start up CMUCL:
> (list
> (setq *gc-verbose* nil)
> (unless (fboundp 'funct+shortdir+fn1-mayload)
> (load "/home/users/rem/LispWork/2007-2-mayload.lisp"))
> (funct+shortdir+fn1-mayload 'filenamebase-std-roll-before :LISP "2007-2-roll")
> (funct+shortdir+fn1-mayload 'load-file-by-method :LISP "2005-8-readers")
> (funct+shortdir+fn1-mayload 'dirspec+filnam+method+globsym-may-load :LISP "2008-4-MayLoad")
> (funct+shortdir+fn1-mayload 'make-empty-heap :LISP "2001-B-heap")
> (funct+shortdir+fn1-mayload 'string-read-words-batch-sort :LISP "2008-3-WordHist")
> (funct+shortdir+fn1-mayload 'phvec-normalize :LISP "2008-3-ProxHash")
> (funct+shortdir+fn1-mayload 'device-to-ar+mr+mc :LISP "2008-3-TextGraphics")
> (funct+shortdir+fn1-mayload 'trans-skills-lines :LISP "2008-3-TopPH")
> (funct+shortdir+fn1-mayload 'trans-skills-3d-012-links :LISP "2008-5-TopPH")
> )
> I don't see how batch could possibly help me with those tasks.
You could fork a lisp process for each of these tasks, suspend them by
sending them a STOP signal, and batch a kill -CONT $PID command to
resume them when the system load is low enough for the batch commands
to be run.
;; (pseudo-code, I don't have cmucl installed on this computer to check all of it).
(progn (setq *gc-verbose* nil)
(unless (fboundp 'funct+shortdir+fn1-mayload)
(load "/home/users/rem/LispWork/2007-2-mayload.lisp"))
(loop
:with pid = 0
:for task :in (list
(lambda () (funct+shortdir+fn1-mayload 'filenamebase-std-roll-before :LISP "2007-2-roll"))
(lambda () (funct+shortdir+fn1-mayload 'load-file-by-method :LISP "2005-8-readers"))
(lambda () (funct+shortdir+fn1-mayload 'dirspec+filnam+method+globsym-may-load :LISP "2008-4-MayLoad"))
(lambda () (funct+shortdir+fn1-mayload 'make-empty-heap :LISP "2001-B-heap"))
(lambda () (funct+shortdir+fn1-mayload 'string-read-words-batch-sort :LISP "2008-3-WordHist"))
(lambda () (funct+shortdir+fn1-mayload 'phvec-normalize :LISP "2008-3-ProxHash"))
(lambda () (funct+shortdir+fn1-mayload 'device-to-ar+mr+mc :LISP "2008-3-TextGraphics"))
(lambda () (funct+shortdir+fn1-mayload 'trans-skills-lines :LISP "2008-3-TopPH"))
(lambda () (funct+shortdir+fn1-mayload 'trans-skills-3d-012-links :LISP "2008-5-TopPH")))
:do (if (zerop (setf pid (unix:unix-fork)))
(progn
(unix:unix-kill 0 (unix:unix-signal-number :sigsstop))
(funcall task)
(unix:unix-exit 0))
(progn
(sleep 1)
(with-open-stream (cmd (ext:run-program "batch" :input :stream))
(format t "kill -CONT ~A~%" pid))))))
But you'll probably get better result with a nice process. It doesn't
run slower, but is only less prioritary that the other users'
processes.
In any case, for long running computing processes, you should probably
try to set up things to avoid a need for interactive use once the
computations are started. Save the results (and intermediary results)
to files, to be able to restart from new processes, etc. A good model
would be the SETI@home
Have a look at Boinc too: http://en.wikipedia.org/wiki/BOINCProject.
--
__Pascal Bourguignon__
Madhu
| % uptime
| 5:01AM up 3 days, 25 mins, 9 users, load averages: 5.11, 6.07, 6.49
[...]
|> Another tool you may use is batch(1), which enqueues jobs, and
|> will start them only when the load average is low enough.
|
| That's worthless for my purpose, unless you know a way to use it to
| deal with interactive Lisp session.
The concept is simple enough to implement in a few lines of lisp
|> If you had a lot of little tasks to execute, you could use batch.
|
| If I have a lot of little tasks to execute, I put them into a PROGN
| or LIST, and if I'm going to do the same sequence of tasks many
| times over different days then I might put them into a DEFUN.
| For example, here is the LIST of tasks I do when I first start up CMUCL:
| (list
| (setq *gc-verbose* nil)
| (unless (fboundp 'funct+shortdir+fn1-mayload)
| (load "/home/users/rem/LispWork/2007-2-mayload.lisp"))
| (funct+shortdir+fn1-mayload 'filenamebase-std-roll-before :LISP "2007-2-roll")
| (funct+shortdir+fn1-mayload 'load-file-by-method :LISP "2005-8-readers")
| (funct+shortdir+fn1-mayload 'dirspec+filnam+method+globsym-may-load :LISP "2008-4-MayLoad")
| (funct+shortdir+fn1-mayload 'make-empty-heap :LISP "2001-B-heap")
| (funct+shortdir+fn1-mayload 'string-read-words-batch-sort :LISP "2008-3-WordHist")
| (funct+shortdir+fn1-mayload 'phvec-normalize :LISP "2008-3-ProxHash")
| (funct+shortdir+fn1-mayload 'device-to-ar+mr+mc :LISP "2008-3-TextGraphics")
| (funct+shortdir+fn1-mayload 'trans-skills-lines :LISP "2008-3-TopPH")
| (funct+shortdir+fn1-mayload 'trans-skills-3d-012-links :LISP "2008-5-TopPH")
| )
(defvar *batch-job-queue* *)
| I don't see how batch could possibly help me with those tasks.
;; No need for batch:
(defun parse-uptime-load-average ()
"Call external program `uptime' and return the first load average."
(let* ((string
(with-output-to-string (stream)
(ext:run-program "uptime" nil :output stream)))
(beg (position #\: string :from-end t))
(end (position #\, string :start beg))
*read-eval*)
(read-from-string string t nil :start (1+ beg) :end end)))
(defvar +batch-threshold-loadavg+ 3.0
"When the load average drops below this, execute jobs in the job queue")
(defvar +batch-periodic+ 60
"Time interval in seconds. Our batch timer process wakes up every
+batch-periodic+ seconds to check on the load average.")
(defun run-batch-job-queue ()
(loop (cond (*batch-job-queue*
(cond ((< (parse-uptime-load-average) +batch-threshold-loadavg+)
(let ((form (pop *batch-job-queue*)))
(apply (car form) (cdr form)))) ; HANDLE-ERRORS HERE
(t (sleep +batch-periodic+))))
(t (return 'NOJOBSLEFT)))))
(run-batch-job-queue)
;; no need to lock protect the queue for this use case. start it up in
;; the background with (mp:make-process #'run-batch-job-queue)
Rob Warnock
+---------------
| > From: p...@informatimago.com (Pascal J. Bourguignon)
| > On a linux system you can have a look at /proc/cpuinfo.
|
| That does't help me on my shell account running FreeBSD Unix.
| /proc/cpuinfo: No such file or directory
Actually, if your FreeBSD provider has mounted the "linprocfs"
pseudo-device then all you need to do is look in the right place:
$ uname -rs
FreeBSD 6.2-RELEASE-p4
$ df | grep proc
linprocfs 4 4 0 100% /usr/compat/linux/proc
procfs 4 4 0 100% /proc
$ cat /proc/cpuinfo
cat: /proc/cpuinfo: No such file or directory
$ cat /usr/compat/linux/proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 1
model name : AMD Athlon(tm) 64 Processor 3500+
stepping : 0
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 b19 mmx fxsr xmm b26
cpu MHz : 2211.34
bogomips : 2211.34
$
-Rob
-----
Rob Warnock <rp...@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
Robert Maas, http://tinyurl.com/uh3t
> > 5:01AM up 3 days, 25 mins, 9 users, load averages: 5.11, 6.07, 6.49
> > major CPU-speed benchmark.
> From: p...@informatimago.com (Pascal J. Bourguignon)
> Well, 5:00 AM, is often the time at which the last cron tasks of
> the night are just started. Try it at another random time.
% uptime
12:48PM up 9:06, 18 users, load averages: 2.64, 4.13, 5.14
If it's a quad core, then it looks like one core is sitting totally
idle at the moment, and the other three are not quite fully
occupied. But if it's single core, it's pretty busy now. Is that
correct, or do I misunderstand?
What Unix command can be used to find out how many cores this
computer system has?
> >> would give you the load average such as:
> >> 14:44:30 up 4:39, 2 users, load average: 0.05, 0.04, 0.13
> >
> > I don't know what machine gave you *those* numbers, but that
> > machine is sorely underutilized?
> Not all the time :-) Also it's a quad-core, so the average is about
> divided by four, compared to a normal processor.
Do you mean the number reported by uptime is only 1/4 of the actual
load? So that wouldn't help me if my own ISP were quad core, the
2.64 reported actually means 10.56 processes competing for those
four cores? I don't suppose you know of a Web-based tutorial that
explains these sorts of things so that I wouldn't need to ask you
so many questions?
> ... you can only increase the nicety level of your processes.
> $ clisp -x '(loop :for i :from 0 :do (* i i))' >/dev/null 2>&1 &
(actually cmucl in my case; anyway, at this point I would do
various interactive stuff to debug code in preparation for a major
computation)
> [2] 13589
> $ renice +4 13589
> 13589: old priority 0, new priority 4
> $ renice +6 13589
> 13589: old priority 4, new priority 6
(at this point I would run a major computation;
done with that, now I want to work interactively again, but I can't!!)
> $ renice +2 13589
> renice: 13589: setpriority: Permission denied
> $ renice +5 13589
> renice: 13589: setpriority: Permission denied
(so basically I can set up a major computation ro run nice *only*
if it's the very last thing I want to do during my Lisp session)
> So you could start with a normal nicety, do your loading and
> compilation full speed, and once you launch your computation, you
> can detach your process and renice it.
I don't know how to detach a process, unless that process is 'screen',
whereby C-A d detaches it.
The 'detach' command doesn't detach an existing process, instead it
starts a brand new process in detached mode, which would provide me
no way to take the Lisp environment I already have built up and
then simply call one function that invokes a major compute task.
> > If I have a lot of little tasks to execute, I put them into a PROGN
> > or LIST, and if I'm going to do the same sequence of tasks many
> > times over different days then I might put them into a DEFUN. ...
> You could fork a lisp process for each of these tasks, suspend them by
> sending them a STOP signal, and batch a kill -CONT $PID command to
> resume them when the system load is low enough for the batch commands
> to be run.
That's not ANSI-CL.
I'd need to find out how to do all that in CMU Common Lisp 18b.
But for lots of little Lisp tasks that load files or otherwise
build up a single Lisp environment, such as the example I posted,
it'd be worthless to me.
> In any case, for long running computing processes, you should
> probably try to set up things to avoid a need for interactive use
> once the computations are started.
Actually whenever I get anything debugged and canned to that
degree, with all the set-up from a barebones CMUCL included in the
canned task, I usually interface it to CGI. That automatically runs
it in a separate process, and it's only a click away from a Web
page so I don't have to remember where I put text for the toplevel
function call and copy&paste from there to the REP, and I can have
all such canned tasks in a nice heirarchial menu
(single nested UL/LI, or multiple Web pages in a tree, however I
want the tasks organized).
> A good model would be the SETI@home
Setting up a system for automatically downloading tasks to milions
of personal computers around the world seems like overkill.
> Have a look at Boinc too: http://en.wikipedia.org/wiki/BOINCProject.
Wikipedia does not have an article with this exact name. Please search
for BOINCProject in Wikipedia to check for alternative titles or
spellings.
->
Search results
You searched for BOINCProject [Index]
No article title matches
* Search for "BOINCProject" in existing articles.
->
No page text matches
Robert Maas, http://tinyurl.com/uh3t
> The concept is simple enough to implement in a few lines of lisp
;; no need to lock protect the queue for this use case. start it up in
;; the background with (mp:make-process #'run-batch-job-queue)
> * (describe 'mp:make-process)
> MAKE-PROCESS is an external symbol in the MULTIPROCESSING package.
> Function documentation:
> Make a process which will run function when it starts up. ...
I assume it means call the function with no arguments, i.e. APPLY
the function to the empty list? The documentation strings visible
by DESCRIBE in CMUCL are often not quite the whole truth, leave a
lot of important info unstated.
I wonder where this sort of stuff in CMUCL is documented? Google search:
CMUCL MULTIPROCESSING package MAKE-PROCESS
lots of discussion, and some source code, but no documentation of that package
CMUCL MULTIPROCESSING package MAKE-PROCESS documentation
not there either
CMUCL MULTIPROCESSING package MAKE-PROCESS tutorial
not there either. I give up. Where is online documentation for it??
Madhu
| % uptime
| 12:48PM up 9:06, 18 users, load averages: 2.64, 4.13, 5.14
|
| If it's a quad core, then it looks like one core is sitting totally
| idle at the moment, and the other three are not quite fully
| occupied. But if it's single core, it's pretty busy now. Is that
| correct, or do I misunderstand?
`loadavg' is a measure of contention -- [see manpage definition below]
loosely it is the number of processes trying to run simultaneously
(contending for CPU) or waiting for IO (contending for Disk). Someone
from google once told me that if this number is greater than 1 "your
system is in trouble", as the system is overloaded.
| What Unix command can be used to find out how many cores this
| computer system has?
See Rob Warnock's reply to you in this thread:
<News:tCdnQJvRYdj7rDV...@speakeasy.net>
<URL:http://groups.google.com/groups?selm=tCdnQJvRYdj7rDV...@speakeasy.net>
Linux: cat /proc/cpuinfo should tell you.
Freebsd: cat /usr/compat/linux/proc/cpuinfo
|> >> would give you the load average such as:
|> >> 14:44:30 up 4:39, 2 users, load average: 0.05, 0.04, 0.13
|> >
|> > I don't know what machine gave you *those* numbers, but that
|> > machine is sorely underutilized?
|> Not all the time :-) Also it's a quad-core, so the average is about
|> divided by four, compared to a normal processor.
|
| Do you mean the number reported by uptime is only 1/4 of the actual
| load? So that wouldn't help me if my own ISP were quad core, the
| 2.64 reported actually means 10.56 processes competing for those
| four cores? I don't suppose you know of a Web-based tutorial that
| explains these sorts of things so that I wouldn't need to ask you
| so many questions?
The number should be CPU independent.
I'm on linux Here you get the loadvg by doing `cat /proc/loadavg'
=> 0.15 0.13 0.09 3/157 14446
The linux manpage explains it this way:
,---- man (5) proc
| /proc/loadavg
| The first three fields in this file are load average figures giving the
| number of jobs in the run queue (state R) or waiting for disk I/O (state
| D) averaged over 1, 5, and 15 minutes. They are the same as the load
| average numbers given by uptime(1) and other programs. The fourth field
| consists of two numbers separated by a slash (/). The first of these is
| the number of currently executing kernel scheduling entities (processes,
| threads); this will be less than or equal to the number of CPUs. The
| value after the slash is the number of kernel scheduling entities that
| currently exist on the system. The fifth field is the PID of the process
| that was most recently created on the system.
---
--
Madhu
Madhu
|> * (describe 'mp:make-process)
|> MAKE-PROCESS is an external symbol in the MULTIPROCESSING package.
|> Function documentation:
|> Make a process which will run function when it starts up. ...
|
| I assume it means call the function with no arguments, i.e. APPLY the
| function to the empty list? The documentation strings visible by
| DESCRIBE in CMUCL are often not quite the whole truth, leave a lot of
| important info unstated.
[...]
| not there either. I give up. Where is online documentation for it??
I'm afraid I found the comments in the source and the source are still
the most accurate/best documentation for CMUCL's MP implementation.
See <URL:http://www.trakt7.net/cmucl%20and%20multiprocessing>
It is based on the CLIM-SYS API which is specified in the CLIM
specification. If you can live with the fact that there may be some
differences, see the clim-sys specification (mikemac.com site linked in
the above URL), or in a slightly nicer form at:
<URL:http://bauhh.dyndns.org:8000/clim-spec/B-2.html>
--
Madhu
Madhu
| I'm on linux Here you get the loadvg by doing `cat /proc/loadavg'
| => 0.15 0.13 0.09 3/157 14446
| ,---- man (5) proc
| | /proc/loadavg
[...]
| | average numbers given by uptime(1) and other programs. The fourth field
| | consists of two numbers separated by a slash (/). The first of these is
| | the number of currently executing kernel scheduling entities (processes,
| | threads); this will be less than or equal to the number of CPUs. The
| | value after the slash is the number of kernel scheduling entities that
| | currently exist on the system. The fifth field is the PID of the process
| | that was most recently created on the system.
| ---
Re. the fourth field reported on my single CPU non SMP system, `3/157',
that documentation is obviously incorrect: The number of currently
executing kernel scheduling entities (3) is NOT less than or equal the
number of CPUs on my system (1).
--
Madhu
Madhu
| See <URL:http://www.trakt7.net/cmucl%20and%20multiprocessing>
I'm sorry I suggested this without any caveats and would like to take
this recommendation back. [I mentioned the URL because it listed some
other links, but the page has too many faults, and I am not upto fixing
them] To wit
The docstrings listed here are based on an old CMUCL 18e (Early 2003?),
current CMUCL is 19e. Although MP implementation has not changed, the
particular build mentioned seems to be buggy:
- The listed docstrings are incomplete: MAKE-PROCESS for instance does
not list the documentation on keyword arguments.
- It does not mention MULTIPROCESSING::STARTUP-IDLE-AND-TOP-LEVEL-LOOPS,
which is required for any performant system. (This has to be executed
at the REPL at startup, and will start a new top level loop)
| CMUCLs MP implementation
| is based on the CLIM-SYS API which is specified in the CLIM
| specification. If you can live with the fact that there may be some
| differences, see the clim-sys specification (mikemac.com site linked in
| the above URL), or in a slightly nicer form at:
| <URL:http://bauhh.dyndns.org:8000/clim-spec/B-2.html>
I've used and would recommend a mixture of the docstrings (which are
reasonably good) and the clim-spec as a guide to CMUCL's MP. Be warned
that the implementation (x86 only) is reasonably stable for all common
uses but is not polished and would still be billed as `experimental' in
many circles.
--
Madhu
Pascal J. Bourguignon
> * (Robert Maas, http://tinyurl.com/uh3t) <rem-2008...@yahoo.com> :
> Wrote on Thu, 05 Jun 2008 14:36:42 -0700:
>
> | % uptime
> | 12:48PM up 9:06, 18 users, load averages: 2.64, 4.13, 5.14
> |
> | If it's a quad core, then it looks like one core is sitting totally
> | idle at the moment, and the other three are not quite fully
> | occupied. But if it's single core, it's pretty busy now. Is that
> | correct, or do I misunderstand?
>
> `loadavg' is a measure of contention -- [see manpage definition below]
> loosely it is the number of processes trying to run simultaneously
> (contending for CPU) or waiting for IO (contending for Disk). Someone
> from google once told me that if this number is greater than 1 "your
> system is in trouble", as the system is overloaded.
That's not exactly what I observe on my linux system. Here, with 4
running processes, I still have a lav of 3.47:
top - 09:52:13 up 8 days, 18:49, 6 users, load average: 3.47, 2.03, 0.93
Tasks: 118 total, 5 running, 112 sleeping, 1 stopped, 0 zombie
Cpu0 : 93.4%us, 6.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.3%hi, 0.3%si, 0.0%st
Cpu1 : 96.0%us, 4.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 94.0%us, 6.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 94.0%us, 6.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 3921872k total, 2952724k used, 969148k free, 304024k buffers
Swap: 2000084k total, 160k used, 1999924k free, 942100k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
30593 pjb 20 0 20624 952 368 R 100 0.0 1:45.94 bash
30590 pjb 20 0 20624 952 368 R 99 0.0 1:45.28 bash
30592 pjb 20 0 20624 952 368 R 99 0.0 1:46.29 bash
30591 pjb 20 0 20624 952 368 R 98 0.0 1:45.60 bash
27939 pjb 20 0 319m 207m 22m S 3 5.4 150:00.75 firefox-bin
4786 root 20 0 579m 273m 5876 S 2 7.1 260:50.94 X
1 root 20 0 3696 580 488 S 0 0.0 0:06.02 init
2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:00.17 migration/0
4 root 15 -5 0 0 0 S 0 0.0 0:02.03 ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 0:00.32 watchdog/0
6 root RT -5 0 0 0 S 0 0.0 0:00.12 migration/1
7 root 15 -5 0 0 0 S 0 0.0 0:01.87 ksoftirqd/1
Moreover, the trouble, on mono-processors linux systems only really
starts with lavs above 7. Note that you can tune linux kernel
scheduling and responsiveness (the kernel preemption parameter).
> The number should be CPU independent.
No. Read it again:
> I'm on linux Here you get the loadvg by doing `cat /proc/loadavg'
> => 0.15 0.13 0.09 3/157 14446
> The linux manpage explains it this way:
>
> ,---- man (5) proc
> | /proc/loadavg
> | The first three fields in this file are load average figures giving the
> | number of jobs in the run queue (state R) or waiting for disk I/O (state
> | D) averaged over 1, 5, and 15 minutes. They are the same as the load
> | average numbers given by uptime(1) and other programs. The fourth field
> | consists of two numbers separated by a slash (/). The first of these is
> | the number of currently executing kernel scheduling entities (processes,
> | threads); this will be less than or equal to the number of CPUs. The
> | value after the slash is the number of kernel scheduling entities that
> | currently exist on the system. The fifth field is the PID of the process
> | that was most recently created on the system.
> ---
The average is done over time, not over number of CPU. If you have 4
R processes, and 4 processors, you have a situation that's 4 times
less critical than if you have 4 R processes and only 1 processor.
Another way to realize you're on a multi-processor system is time:
real 0m4.129s
user 0m10.136s
sys 0m0.890s
Mind it, the real time is less than the user+sys CPU times! In this
example, it means that the program (several processes) ran on average
on (/ (+ 10.136 0.890) 4.129) --> 2.67 processors.
--
__Pascal Bourguignon__
Tim Bradshaw
> `loadavg' is a measure of contention -- [see manpage definition below]
> loosely it is the number of processes trying to run simultaneously
> (contending for CPU) or waiting for IO (contending for Disk). Someone
> from google once told me that if this number is greater than 1 "your
> system is in trouble", as the system is overloaded.
I'm not sure how much it varies between implementations, but
traditionally, and simplifying somewhat, the load average is the
number of processes which are not waiting for anything other than I/O
in order to run (ie this does not count processes which are waiting
for you to type something, say).
Thus a load average less than or equal to the number of cores in the
system means that everything is getting CPU time that wants it
(processes may still be starving for I/O of course).
In particular, what is a "bad" load average scales as the number of
cores: for many years of course Unix systems all had a single core,
but nowadays very many systems have more than one of course. This
took me a while to get used to, but now I always check the number of
cores before panicking at a load of 25 or something...
(And "cores" is not correct: on systems such as Sun's CMT machines the
number of interest is the number of virtual processors, which is some
multiple of the number of cores (4 or 8, depending on generation I
think).)
-tim
--tim
Pascal J. Bourguignon
>> > % uptime
>> > 5:01AM up 3 days, 25 mins, 9 users, load averages: 5.11, 6.07, 6.49
>> > Anyway, with load around 5 or 6 it looks too busy for me to run any
>> > major CPU-speed benchmark.
>
>> From: p...@informatimago.com (Pascal J. Bourguignon)
>> Well, 5:00 AM, is often the time at which the last cron tasks of
>> the night are just started. Try it at another random time.
>
> % uptime
> 12:48PM up 9:06, 18 users, load averages: 2.64, 4.13, 5.14
>
> If it's a quad core, then it looks like one core is sitting totally
> idle at the moment, and the other three are not quite fully
> occupied. But if it's single core, it's pretty busy now. Is that
> correct, or do I misunderstand?
>
> What Unix command can be used to find out how many cores this
> computer system has?
That would depend on the unix system you have.
On linux, you can do: cat /proc/cpuinfo
or use top, and type 1 to get per cpu stats.
On MacOSX, IIRC, host_info will do.
>> >> would give you the load average such as:
>> >> 14:44:30 up 4:39, 2 users, load average: 0.05, 0.04, 0.13
>> >
>> > I don't know what machine gave you *those* numbers, but that
>> > machine is sorely underutilized?
>> Not all the time :-) Also it's a quad-core, so the average is about
>> divided by four, compared to a normal processor.
>
> Do you mean the number reported by uptime is only 1/4 of the actual
> load?
No, I was wrong, sorry. Those numbers are not divided by the number
or processors (at least on linux). (See my answer to Madhu).
> So that wouldn't help me if my own ISP were quad core, the
> 2.64 reported actually means 10.56 processes competing for those
> four cores?
If your system is a quad-core, and you have a lav of 2.64, it means
that 4-2.64=1.36 cores are left unused.
> I don't suppose you know of a Web-based tutorial that
> explains these sorts of things so that I wouldn't need to ask you
> so many questions?
man uptime
man loadavg
and some experiments (easier done when you 'own' the system).
reading the linux or freebsd sources.
But I guess google could find some web page explaining them too.
>> So you could start with a normal nicety, do your loading and
>> compilation full speed, and once you launch your computation, you
>> can detach your process and renice it.
>
> I don't know how to detach a process, unless that process is 'screen',
> whereby C-A d detaches it.
Yes, I would advise screen anyways. An alternative for server (lisp)
processes would be detachtty, which like screen allows for reattaching
them.
Otherwise, in general, in bash, you can detach any job with the disown
built-in command.
in bash:
lisp -x '(load "batch-run.lisp")' & disown
or if you start interactively, suspend the process (usually C-z), and then:
bg # to let it run in background and
disown # to detach it
however, for interactive processes, they usually try to read to the
tty, so detaching them might be more difficult (ideally, they should
close stdin/stdout/stderr).
> The 'detach' command doesn't detach an existing process, instead it
> starts a brand new process in detached mode, which would provide me
> no way to take the Lisp environment I already have built up and
> then simply call one function that invokes a major compute task.
Perhaps:
detach lisp -x '(load "batch-run.lisp")'
>> You could fork a lisp process for each of these tasks, suspend them by
>> sending them a STOP signal, and batch a kill -CONT $PID command to
>> resume them when the system load is low enough for the batch commands
>> to be run.
>
> That's not ANSI-CL.
Indeed not. But that's still in your lisp image. Use APROPOS,
DESCRIBE, DOCUMENTATION, and read the user manual of your
implementation.
>> Have a look at Boinc too: http://en.wikipedia.org/wiki/BOINCProject.
> Wikipedia does not have an article with this exact name. Please search
> for BOINCProject in Wikipedia to check for alternative titles or
> spellings.
> ->
> Search results
> You searched for BOINCProject [Index]
> No article title matches
>
> * Search for "BOINCProject" in existing articles.
> ->
> No page text matches
Indeed. I don't know why this 'Project' stem was added here.
Try rather: http://en.wikipedia.org/wiki/BOINC
You could have guessed it yourself, if a word made of two words glued
together doesn't give any hit, try the two words separated with
spaces, then try each word in turn alone.
I thought you where smart and a programmer. If you don't have these
rules already in your brain, you can add (program) them yourself! ;-)
--
__Pascal Bourguignon__
Rob Warnock
+---------------
| On Jun 6, 7:37 am, Madhu <enom...@meer.net> wrote:
| > `loadavg' is a measure of contention -- [see manpage definition below]
| > loosely it is the number of processes trying to run simultaneously
| > (contending for CPU) or waiting for IO (contending for Disk). Someone
| > from google once told me that if this number is greater than 1 "your
| > system is in trouble", as the system is overloaded.
|
| I'm not sure how much it varies between implementations, but
| traditionally, and simplifying somewhat, the load average is the
| number of processes which are not waiting for anything other than I/O
| in order to run (ie this does not count processes which are waiting
| for you to type something, say).
<FLAME value="on">
Unfortunately for computer science, historical practice, sanity,
and just plain common sense, Linux has decided to include in the
"load average" *all* processes that are *waiting* for *any* I/O
completions, whether from swapping disks or slow file disks or
networks or slow serial lines or paper tape readers! This results
in such nonsense, for example, as often seeing a "load average"
of 150+ on a *TOTALLY IDLE* NFS server that just happens to have
a large number of mounts on it. (*sigh*)
Whereas sane operating systems such as TOPS-10, TOPS-20, Irix, Solaris,
{Free,Net,Open}BSD, and many others include in the load average only
processes that are waiting to get a *CPU* on which to run.
</FLAME>
Rob Warnock
+---------------
| I've used and would recommend a mixture of the docstrings (which are
| reasonably good) and the clim-spec as a guide to CMUCL's MP. Be warned
| that the implementation (x86 only) is reasonably stable for all common
| uses but is not polished and would still be billed as `experimental' in
| many circles.
While technically true, I have used CMUCL's "processes"[1] in
production web applications servers which spawned a new process
for each HTTP request, and -- as far as I can tell -- in over
6 years and three sites there have been *no* crashes due to
instability in CMUCL's MP primitives.
But as usual, YMMV. In particular, I was careful to make sure
that in normal operation the CMUCL image doesn't have to handle
any Unix signals. E.g., *don't* use the (truly-experimental!!)
pre-emptive MP scheduling which relies on SIGALRM, but instead
use only the default cooperative scheduling [which uses the timeout
parameter in the "select()" system call]. *Don't* allow SIGPIPE;
set it to SIG_IGN and handle the EPIPE I/O error exception instead.
Etc., etc.
-Rob
[1] What CMUCL calls "multiprocessing" (a heritage from CLIM?)
is what most other people would call "multiprogramming"
or even simply or user-mode coroutines ("green threads").
Espen Vestre
> That's not exactly what I observe on my linux system. Here, with 4
> running processes, I still have a lav of 3.47:
Our LispWorks-delivered applications make the use of load numbers on
our servers almost useless, since (presumably) hundreds of threads
executing mp:process-wait-with-timeout create high load averages
while the cpus are really mostly idle.
The servers typically report a load average of 25-50 while the cpu is
95% idle.
--
(espen)
Tim X
> * (Robert Maas, http://tinyurl.com/uh3t) <rem-2008...@yahoo.com> :
> Wrote on Thu, 05 Jun 2008 14:36:42 -0700:
>
> | % uptime
> | 12:48PM up 9:06, 18 users, load averages: 2.64, 4.13, 5.14
> |
> | If it's a quad core, then it looks like one core is sitting totally
> | idle at the moment, and the other three are not quite fully
> | occupied. But if it's single core, it's pretty busy now. Is that
> | correct, or do I misunderstand?
>
> `loadavg' is a measure of contention -- [see manpage definition below]
> loosely it is the number of processes trying to run simultaneously
> (contending for CPU) or waiting for IO (contending for Disk). Someone
> from google once told me that if this number is greater than 1 "your
> system is in trouble", as the system is overloaded.
I've heard this before and I think its misleading. The way it was
explained to me is that a value less than 1 means that there were
spare/wasted CPU cycles i.e. cycles wehre nothing was waiting to run on
the cpu. A value of 1 means that every cpu cycle was being used. A value
above 1 indicates that there was some contention for cpu cycles. A value
of 2 would indicate that to have no contention/delay, you would need a
cpu with twice the capacity (i.e. double the cpus or possibly same
number of cpus, but operatiing at twice the speed).
However, all this is really just useful as a rough indicator. It is
rediculous to state that a system is in trouble once it gets a load
average of over 1. Other issues need to be considered, such as amount of
memory available, number of processes waiting in the queue and above
all, user expectations. I regularly see servers with averages over 5,
but its not an issue because user expectations regarding performance are
being met.
From stuff I've read, I think the load average values you see have
become even less informative with the advent of multi-core systems. I do
remember seeing some debate on various lists on how to 'fix' things so
that multi-core systems were giving accurate indicators. I've not
monitored the debate for some time and I'm not sure what current
thinking is, but at the time, there was considerable debate regarding
the best way to give reliable and meaningful indicators on a systems
load. In the end, the best measure appears to be user expectation - if
response times are adequate, then they are adequate. If they are not,
then information such as load average can be useful, but should not be
taken in isolation.
Some systems do provide more information with tools such as 'top' and
I'd recommend using that rather than just the values of load
average. Some versions of top and some operating systems will even show
individual CPU values etc.
Tim
--
tcross (at) rapttech dot com dot au
Robert Maas, http://tinyurl.com/uh3t
> > Wikipedia does not have an article with this exact name. Please search
> > for BOINCProject in Wikipedia to check for alternative titles or
> > spellings.
> > ->
> > Search results
> > You searched for BOINCProject [Index]
> > No article title matches
> > * Search for "BOINCProject" in existing articles.
> > ->
> > No page text matches
> Try rather: http://en.wikipedia.org/wiki/BOINC
> You could have guessed it yourself, if a word made of two words
> glued together doesn't give any hit, try the two words separated
> with spaces, then try each word in turn alone.
Unfortunately "boink" by itself is a code word in singles
organizations having to with social gatherings, sexual intercourse,
and penguins, including fist-punching toy penguins in lieu of
punching real people as the Three Stooges often did. I anticipated
that the variant spelling would turn up similar matches on Google
so the search for that word by itself wasn't worth doing.
> I thought you where smart and a programmer. If you don't have
> these rules already in your brain, you can add (program) them
> yourself! ;-)
I have a rule in my brain that if a particular word has an obscene
meaning, then using that word as a sole search term to find
something totally unrelated is a waste of time and a likely harm to
my mental state.
Pascal J. Bourguignon
I don't know what you're talking about.
p...@informatimago.com | My Account | Sign out
Google
Advanced Search
Preferences
Web Video Results 1 - 100 of about 14,300,000 for boinc. (0.33 seconds)
BOINC
BOINC is an open-source software platform for computing using volunteered resources.
boinc.berkeley.edu/ - 11k - Cached - Similar pages - Note this
Download
Choose
Trac
Seti@Home
Account Manager
Top 100
Web
Add-Ons Page
More results from berkeley.edu »
BOINC: compute for science
BOINC is a program that lets you donate your idle computer time to science projects like SETI@home, Climateprediction.net, Rosetta@home, World Community ...
boinc.berkeley.edu/download.php - 8k - Cached - Similar pages - Note this
Berkeley Open Infrastructure for Network Computing - Wikipedia ...
The Berkeley Open Infrastructure for Network Computing (BOINC) is a non-commercial middleware system for volunteer and grid computing. ...
en.wikipedia.org/wiki/BOINC - 45k - Cached - Similar pages - Note this
Berkeley Open Infrastructure for Network Computing - Wikipedia ...
The Berkeley Open Infrastructure for Network Computing (BOINC) is a non-commercial middleware system for volunteer computing, originally developed to ...
en.wikipedia.org/wiki/Berkeley_Open_Infrastructure_for_Network_Computing - 44k - Cached - Similar pages - Note this
More results from en.wikipedia.org »
That said, I've got the impression that google filters or orders
results depending on each user history. If you've been clicking a lot
on CS links, you should get them on top of your next results.
--
__Pascal Bourguignon__ http://www.informatimago.com/
HEALTH WARNING: Care should be taken when lifting this product,
since its mass, and thus its weight, is dependent on its velocity
relative to the user.
Tim Bradshaw
>
> I've heard this before and I think its misleading. The way it was
> explained to me is that a value less than 1 means that there were
> spare/wasted CPU cycles i.e. cycles wehre nothing was waiting to run on
> the cpu. A value of 1 means that every cpu cycle was being used. A value
> above 1 indicates that there was some contention for cpu cycles.
As others have said, this really isn't right. Firstly average is
computed over the number of cores (or virtual CPUs in some sense),
and secondly the load is not just dependent on CPU utilisation but
also on I/O. It is quite common to see machines with very high load
with almost no CPU utilisation, as they are starving for I/O.
> I do
> remember seeing some debate on various lists on how to 'fix' things so
> that multi-core systems were giving accurate indicators.
This is a non-problem, since the denominator of load is the number of
cores/virtual CPUs.
Vend
http://tinyurl.com/uh3t) wrote:
> Does anybody you know of have access to a computer with at least 64
> CPUs, with a version of Common Lisp that runs on that computer and
> supports distributing the function-applications within a MAPCAR
> call across as many CPUs as are available in order to achieve great
> speed-up compared to the usual algorithm of performing each
> function-application in sequence down the list? Would anybody
> volunteer such a system for me to occasionally use across the net
> without charge, for research purposes?
>
> Of course if there are dependencies from one function application
> to the next, this parallel-mapcar wouldn't be appropriate. But I
> have an application where I need to apply a single function to a
> large number of arguments in parallel. I'm running it with nearly
> three hundred at the moment, whereupon it takes several minutes to
> do them all in succession, which isn't too bad if done rarely, but
> I envision doing the same with thousands of arguments, whereby the
> time to do them in succession would be prohibitive.
Excuse my ignorant remark, but can't you just spawn 64 threads and let
the OS scheduler balance them?
Raymond Wiker
> On Jun 7, 1:58 am, Tim X <t...@nospam.dev.null> wrote:
>
>>
>> I've heard this before and I think its misleading. The way it was
>> explained to me is that a value less than 1 means that there were
>> spare/wasted CPU cycles i.e. cycles wehre nothing was waiting to run on
>> the cpu. A value of 1 means that every cpu cycle was being used. A value
>> above 1 indicates that there was some contention for cpu cycles.
>
> As others have said, this really isn't right. Firstly average is
> computed over the number of cores (or virtual CPUs in some sense),
> and secondly the load is not just dependent on CPU utilisation but
> also on I/O. It is quite common to see machines with very high load
> with almost no CPU utilisation, as they are starving for I/O.
I think the definition of the load average is the number of
runnable process in the run queue - i.e, processes that are not
currently waiting for I/O or sleeping. Thus, I/O bound processes
should not count, but I could be wrong about this.
John Thingstad
Not on most schedulers today.
The reason is the garbage collector.
LispWorks and ACL lock you to one processor for all processes.
Spinlocks have different properties from semaphores.
--------------
John Thingstad
Vend
If I understand correctly, the Sun Java VM has a multithreading/
multiprocessor garbage collector, hasn't it?
Would it be more difficult to implement one for Common Lisp?
Rob Warnock
+---------------
| Tim Bradshaw <tfb+g...@tfeb.org> writes:
| > As others have said, this really isn't right. Firstly average is
| > computed over the number of cores (or virtual CPUs in some sense),
| > and secondly the load is not just dependent on CPU utilisation but
| > also on I/O. It is quite common to see machines with very high load
| > with almost no CPU utilisation, as they are starving for I/O.
|
| I think the definition of the load average is the number of
| runnable process in the run queue - i.e, processes that are not
| currently waiting for I/O or sleeping. Thus, I/O bound processes
| should not count, but I could be wrong about this.
Sorry, you *are* wrong about this... for Linux.
(But only Linux. Most other O/Ss get it right.)
-Rob
Tim X
> On Jun 7, 1:58 am, Tim X <t...@nospam.dev.null> wrote:
>
>>
>> I've heard this before and I think its misleading. The way it was
>> explained to me is that a value less than 1 means that there were
>> spare/wasted CPU cycles i.e. cycles wehre nothing was waiting to run on
>> the cpu. A value of 1 means that every cpu cycle was being used. A value
>> above 1 indicates that there was some contention for cpu cycles.
>
> As others have said, this really isn't right. Firstly average is
> computed over the number of cores (or virtual CPUs in some sense),
> and secondly the load is not just dependent on CPU utilisation but
> also on I/O. It is quite common to see machines with very high load
> with almost no CPU utilisation, as they are starving for I/O.
>
I obviously wasn't clear enough. I was providing a simplistic
explination and was trying to explain that relying on the load average
values alone to judge the load on a system was insufficient. It was
mainly in response to the point put forward by another poster that
stated they had been told that a load above 1 indicated the system was
'in trouble', which I think is misleading.
>> I do
>> remember seeing some debate on various lists on how to 'fix' things so
>> that multi-core systems were giving accurate indicators.
>
> This is a non-problem, since the denominator of load is the number of
> cores/virtual CPUs.
Possibly now it is. At the time, this was not necessarily the case and
the debate I saw was on how to best represent a high level indicator of
load given the new cpu architectures. There were a number of arguments
against using a simple averaging approach. As has been seen from other
posts, the other problem is that different operating systems calculate
the load average figure differently and the value you see on one system
doesn't reflect the same calculation as you see on another. I also
suspect operating systems like Linux that allow you to select different
process scheduling and queuing algorithms probably further impact on the
accuracy of the result.
In general, I think load average is only a very rough measure and you
have to use many other indicators when considering a systems
performance, such as how IO is factored in, memory use, disk IO, network
bandwidth and user expectations.
Tim X
I think it is operating system dependent. I think Solaris does this and
many of the 'older' Unix systems did. However, not sure if this holds
for Linux and have no idea with respect to BSD. It certainly was the
definition back in the late 80s/90s IIRC.
Raymond Wiker
> Raymond Wiker <r...@RawMBP.local> wrote:
> +---------------
> | Tim Bradshaw <tfb+g...@tfeb.org> writes:
> | > As others have said, this really isn't right. Firstly average is
> | > computed over the number of cores (or virtual CPUs in some sense),
> | > and secondly the load is not just dependent on CPU utilisation but
> | > also on I/O. It is quite common to see machines with very high load
> | > with almost no CPU utilisation, as they are starving for I/O.
> |
> | I think the definition of the load average is the number of
> | runnable process in the run queue - i.e, processes that are not
> | currently waiting for I/O or sleeping. Thus, I/O bound processes
> | should not count, but I could be wrong about this.
> +---------------
>
> Sorry, you *are* wrong about this... for Linux.
> (But only Linux. Most other O/Ss get it right.)
If the other OS's get it right, and differ from Linux, and my
description differs from what Linux does, then I'm right, right? Right
:-)
Robert Maas, http://tinyurl.com/uh3t
> > organizations having to with social gatherings, sexual intercourse,
> > and penguins, including fist-punching toy penguins in lieu of
> > punching real people as the Three Stooges often did. I anticipated
> > that the variant spelling would turn up similar matches on Google
> > so the search for that word by itself wasn't worth doing.
> >> these rules already in your brain, you can add (program) them
> >> yourself! ;-)
> > I have a rule in my brain that if a particular word has an obscene
> > meaning, then using that word as a sole search term to find
> > something totally unrelated is a waste of time and a likely harm to
> > my mental state.
<http://groups.google.com/group/ba.singles/msg/a8869c3343ec6383?hl=en&dmode=source>
Search for words "penguin" and "boink".
Also take a glance at:
<http://boinkmagazine.com/>
<http://www.urbandictionary.com/define.php?term=boink>
<http://www.starma.com/penis/richardkitty/richardkitty.html>
<http://www.randyrants.com/2002/08/boink_boinkity.html>
> BOINC is an open-source software platform for computing using
> volunteered resources.
..
> BOINC is a program that lets you donate your idle computer time
> to science projects like SETI@home, Climateprediction.net,
> Rosetta@home, World Community ...
If and when I have some software that I don't making public, for
which I need massive amounts of parallel processing, that might be
an option. But my original asking if somebody had access to a
massive parallel computer was more for running a test of
proprietary software that I wouldn't want released to public access
on millions of computers around the world. There'd be an agreement
that I could run my software on such a machine and that the person
hosting it wouldn't steal my algorithms.
> That said, I've got the impression that google filters or orders
> results depending on each user history. If you've been clicking a
> lot on CS links, you should get them on top of your next results.
How would Google have the slightest idea who I am when I start up
lynx from my shell account on a commercial ISP and type google.com
and submit a search?? Does the ISP's sysadmin covertly reveal to
Google which account is making each HTTP/TCP/IP connection to
google?? I don't believe it's happening. Do you have evidence to
support your (IMO) ridiculous claim?
Robert Maas, http://tinyurl.com/uh3t
> Excuse my ignorant remark, but can't you just spawn 64 threads
> and let the OS scheduler balance them?
I suppose I could do that *once*, and immediately have my account terminated.
Once a few years ago I had a function that created a new process,
inside an UNWIND-PROTECT to make sure it got killed if the inner
code aborted. But I made a mistake in editing the function and the
process no longer got killed when an abort happened. I was trying
to debug why it was failing, unaware that I was creating a new
process and not killing it with each re-testing. Suddenly my login
session got killed by the admin and when I dialed back in I found a
nasty message from the admin complaining about my appx. 30
processes that were hogging system resources. I would never want to
do anything like that *deliberately*, as you propose.
The whole point of distributing my tasks over 64 processors is to
try to get it to run 64 times as fast as if running on a single
processor, to allow a couple orders of magnitude more data to be
processed within a reasonable time. If I spawned 64 processes on a
single CPU here on this shell account, it wouldn't run any faster,
would probably run slower, and would piss off the admin royally if
he learned I did it deliberately after already being warned not to
do anything like that after it happened accidently years ago.
By the way, I notice you're in Italy, right? I get a lot of spam
from Italy, but it's mostly from that other ISP, not yours.
-
Nobody in their right mind likes spammers, nor their automated assistants.
To open an account here, you must demonstrate you're not one of them.
Please spend a few seconds to try to read the text-picture in this box:
/--------------------------------------------------------------------\
| |\/| _ | _ ~|~' _ |_ _ _| _ _ _ | |. _ _ _ |
| | |(_||(/_. _|_ \/(/_ | |(_|(_| | (/_(_||, ||\/(/_ _\(/_><. |
\----(Rendered by means of <http://www.schnoggo.com/figlet.html>)----/
(You don't need JavaScript or images to see that ASCII-text image!!
You just need to view this in a fixed-pitch font such as Monaco.)
Then enter your best guess of the text (20-40 chars) into this TextField:
+----------------------------------------+
| |
+----------------------------------------+
Pascal J. Bourguignon
> Search for words "penguin" and "boink".
>
> Also take a glance at:
> <http://boinkmagazine.com/>
> <http://www.urbandictionary.com/define.php?term=boink>
> <http://www.starma.com/penis/richardkitty/richardkitty.html>
> <http://www.randyrants.com/2002/08/boink_boinkity.html>
Why do you keep orienting the discussion in this direction?
>> BOINC is an open-source software platform for computing using
>> volunteered resources.
> How would Google have the slightest idea who I am when I start up
> lynx from my shell account on a commercial ISP and type google.com
> and submit a search?? Does the ISP's sysadmin covertly reveal to
> Google which account is making each HTTP/TCP/IP connection to
> google?? I don't believe it's happening. Do you have evidence to
> support your (IMO) ridiculous claim?
1- Cookies.
2- The Identificatoin protocol RFC-1413. Are you sure your ISP disabled it?
--
__Pascal Bourguignon__ http://www.informatimago.com/
Wanna go outside.
Oh, no! Help! I got outside!
Let me back inside!
Tim X
http://tinyurl.com/uh3t) writes:
>
> By the way, I notice you're in Italy, right? I get a lot of spam
> from Italy, but it's mostly from that other ISP, not yours.
>
If you put your address out there, you will get a lot of spam. use an
ISp with a decent anti-spam solution and your life will be easier.
> -
> Nobody in their right mind likes spammers, nor their automated assistants.
> To open an account here, you must demonstrate you're not one of them.
> Please spend a few seconds to try to read the text-picture in this box:
>
> /--------------------------------------------------------------------\
> | |\/| _ | _ ~|~' _ |_ _ _| _ _ _ | |. _ _ _ |
> | | |(_||(/_. _|_ \/(/_ | |(_|(_| | (/_(_||, ||\/(/_ _\(/_><. |
> \----(Rendered by means of <http://www.schnoggo.com/figlet.html>)----/
> (You don't need JavaScript or images to see that ASCII-text image!!
> You just need to view this in a fixed-pitch font such as Monaco.)
>
> Then enter your best guess of the text (20-40 chars) into this TextField:
yeah, and what about blind users?
Tim Bradshaw
> I think the definition of the load average is the number of
> runnable process in the run queue - i.e, processes that are not
> currently waiting for I/O or sleeping. Thus, I/O bound processes
> should not count, but I could be wrong about this.
You are, sorry! Watch a machine with some disk-hound application and
you will often see the processors almost entirely idle with a huge
load average,
Tim Bradshaw
> Sorry, you *are* wrong about this... for Linux.
> (But only Linux. Most other O/Ss get it right.)
>
Me or him? I've never found cases where I/O bound processes don't
count towards load (but I don't manage Linux machines much nowadays).
Certainly it did on BSD, SunOS and does on Solaris. There are other
tools (on Solaris anyway) to look at CPU usage or IO usage in
isolation).
Tim Bradshaw
> Possibly now it is.
I suspect you're talking about Linux or something. I come from a
SunOS / solaris background and it's been that way since multiprocessor
support first appeared in the early 90s. Other things have changed
radically (for instance until relatively recently Solaris made it all
but impossible to know what the memory pressure was like on a system),
but load hasn't changed in this world since BSD.
Vend
Maas, http://tinyurl.com/uh3t) wrote:
> > From: Vend <ven...@virgilio.it>
> > Excuse my ignorant remark, but can't you just spawn 64 threads
> > and let the OS scheduler balance them?
>
> I suppose I could do that *once*, and immediately have my account terminated.
>
> Once a few years ago I had a function that created a new process,
> inside an UNWIND-PROTECT to make sure it got killed if the inner
> code aborted. But I made a mistake in editing the function and the
> process no longer got killed when an abort happened. I was trying
> to debug why it was failing, unaware that I was creating a new
> process and not killing it with each re-testing. Suddenly my login
> session got killed by the admin and when I dialed back in I found a
> nasty message from the admin complaining about my appx. 30
> processes that were hogging system resources. I would never want to
> do anything like that *deliberately*, as you propose.
>
> The whole point of distributing my tasks over 64 processors is to
> try to get it to run 64 times as fast as if running on a single
> processor, to allow a couple orders of magnitude more data to be
> processed within a reasonable time. If I spawned 64 processes on a
> single CPU here on this shell account, it wouldn't run any faster,
> would probably run slower, and would piss off the admin royally if
> he learned I did it deliberately after already being warned not to
> do anything like that after it happened accidently years ago.
I assumed you were working on a SMP machine with all the CPUs managed
by a single OS.
> By the way, I notice you're in Italy, right? I get a lot of spam
> from Italy, but it's mostly from that other ISP, not yours.
So?
George Neuner
wrote:
It could be a bit easier depending on whether you insist on finalizers
(a dumb idea anyway IMO). Since Lisp code generally employs scoped
handling for resources other than memory, finalizers aren't really
necessary.
George
--
for email reply remove "/" from address
Rob Warnock
+---------------
He, Raymond Wiker, was wrong about the definition for Linux (only),
and I was disagreeing with him about that O/S only. I was *agreeing*
with Raymond that the definition of load of "number of runnable process
in the run queue - i.e, processes that are not currently waiting
for I/O or sleeping" [scaled by the number of available CPU cores,
of course] is the *correct* definition of "load". [Linux therefore
uses an *incorrect* definition of "load", IMNSHO.]
I also agree with Raymond that "I/O bound processes should not count",
except of course for whatever fraction of CPU time they *do* actually
consume when they're not waiting for I/O completion events. [Again,
Linux gets this wrong, resulting in totally idle servers being shown
with load averages >100 just because that many processes are waiting
for network traffic.]
+---------------
| I've never found cases where I/O bound processes don't count
| towards load (but I don't manage Linux machines much nowadays).
| Certainly it did on BSD, SunOS and does on Solaris.
+---------------
As far as I can tell from a quick look at the FreeBSD 6.2 scheduler code,
only processes (well, threads) in CPU run queues are counted in "load".
[See the code in "/usr/src/sys/kernsched_ule.c" & thereabouts, especially
the calls to "kseq_load_add()" & "kseq_load_rem()".] If a process/thread
is taken out of the CPU run queues, its contribution to "load" is immediately
removed. AFAIK this is *always* the way BSD has worked, all the way back
to 4.1a-BSD [the earliest version that I dug into the kernel]. And from
observation of the behavior of servers with lots of processes in I/O wait,
this was the case for Irix as well.
I know [again, from direct observation] that this is most definitely
*NOT* the case on Linux.
[I cannot speak to SunOS/Solaris, due to lack of hands-on experience.]
So, yes, I guess I'm disagreeing with you, Tim.
Tim X
I think this thread sort of proves the main point I was trying to get
across, that load average is a very poor measure of anything when taken
in isolation. The fact there seems to be so many different opinions
regarding how it is calculated and what differs between different
flavors of unix just makes such measurements even less useful. To some
extent, it reminds me of all the arguments regarding MIPS calculations
and debates between the RISC and CISC camps of the late 80s/early 90s.
Remember when Linux measured processing in 'bogomips' (i.e. bogus
MIPS)?
Tim Bradshaw
>
> So, yes, I guess I'm disagreeing with you, Tim.
And that's because it turns out I was wrong, even for Solaris! Sorry!
Solaris computes the load average the way it has always been done (at
least back to BSD 4?), which is based on number of processes in the
run queue (so the way you think it should be done). Linux (it seems)
uses some function of run queue & blocked for I/O queue. So you're
right, and I was wrong.
What led me astray was that for a machine which is paging heavily,
then processes are in the run queue when in fact they are waiting for
disk (because they're waiting for memory which has been paged out).
It's *this* case where you can get huge load averages but tiny CPU
utilization. Obviously that doesn't happen as much as it did, but I
remember we used to see this a lot for Lisps in GC on the BSD machines
where I first saw Unix (30 or 30MB images on a machine with 2MB of
real memory, which did very well except during GC).
And now I think I also remember what lead me even further astray. If a
process uses mmap to map a large file into its address apce, and then
accesses the resulting array, what does that count as? If you're
naive it looks like paging and therefore this process looks like it's
runnable. But actually it is blocked for I/O: in particular it's not
starving for memory. Except, maybe it is, if the file it is paging
from is its own executable's text segment.
(This next bit is based on memory and may be wrong in places.) Solaris
uses mapped files pervasively, to the extent that normal memory
allocation is treated as mapping a file, which file happens to sit in
a special filesystem which is backed by swap. For a long time this
gave rise to all sorts of problems, one of which was that systems
which were actually I/O bound could show up as having high load
averages. Another, worse, problem was that processes doign intensive
I/O (which is most of the important ones on your average big database
box) could cause the VM system to start paging stuff that mattered
out, such as executables. This all got fixed a few years ago by
making the system more aware of whether memory was backed by normal
files, or by anonymous swap, and I think by further treating read-only
executable file mappings specially.
Anyway, I was wrong, but I now why now.
--tim
Rob Warnock
| > So, yes, I guess I'm disagreeing with you, Tim.
|
| And that's because it turns out I was wrong, even for Solaris! Sorry!
| Solaris computes the load average the way it has always been done (at
| least back to BSD 4?), which is based on number of processes in the
| run queue (so the way you think it should be done). Linux (it seems)
| uses some function of run queue & blocked for I/O queue. So you're
| right, and I was wrong.
Not a problem. The only reason I was being pretty stubborn about it
was that Linux is the *only* place I've ever seen these silly 100+
load averages on essentially idle systems!!
+---------------
| What led me astray was that for a machine which is paging heavily,
| then processes are in the run queue when in fact they are waiting for
| disk (because they're waiting for memory which has been paged out).
| It's *this* case where you can get huge load averages but tiny CPU
| utilization. Obviously that doesn't happen as much as it did, but I
| remember we used to see this a lot for Lisps in GC on the BSD machines
| where I first saw Unix (30 or 30MB images on a machine with 2MB of
| real memory, which did very well except during GC).
+---------------
Hmmm... O.k., I suppose counting processes in the "Page-In" wait queue
[as TOPS-10 called it] as part of "load" isn't all that unreasonable.
I'll have to go look at the current FreeBSD code again and see if they
still do that.
+---------------
| And now I think I also remember what lead me even further astray. If a
| process uses mmap to map a large file into its address apce, and then
| accesses the resulting array, what does that count as? If you're
| naive it looks like paging and therefore this process looks like it's
| runnable. But actually it is blocked for I/O: in particular it's not
| starving for memory. Except, maybe it is, if the file it is paging
| from is its own executable's text segment.
+---------------
Yeah, that one's a bit borderline, particularly since with the
proliferation of software that's based on some language VM
[hint, hint] the definition of "executable" is a bit broad.
+---------------
| (This next bit is based on memory and may be wrong in places.)
| Solaris uses mapped files pervasively, to the extent that normal memory
| allocation is treated as mapping a file, which file happens to sit in
| a special filesystem which is backed by swap. For a long time this
| gave rise to all sorts of problems, one of which was that systems
| which were actually I/O bound could show up as having high load averages.
+---------------
Ah, yezz! Now that you mention it, I do seem to recall something
about that with some version of Sun systems.
Anyway, whether processes waiting to be re-paged-in get counted in
"load" or not, I think we can all probably agree that current Linux
does it wrong when an idle system has a 100+ load! ;-}
Mark Wooding
> Remember when Linux measured processing in 'bogomips' (i.e. bogus
> MIPS)?
Err, it still does.
-- [mdw]
Mark Wooding
> Not a problem. The only reason I was being pretty stubborn about it
> was that Linux is the *only* place I've ever seen these silly 100+
> load averages on essentially idle systems!!
I've seen it on Irix, too.
I feel a need to set the record straight for Linux, by the way. The
load average counts
* processes which are runnable, and
* processes which are blocked /uninterruptably/.
The number of places where a process can block uninterruptably is
actually quite small. Just waiting for a network socket or terminal
won't do it. Waiting for a local block device usually will, whether it
be for filesystem access or swapping.
The only point where this gets really strange is NFS: if you mount the
fileserver `hard' then the kernel will block uninterruptably for NFS
responses, and therefore processes waiting for NFS will appear to be
`loading' the system. If you mount `soft' then this won't happen (and
you'll be able to kill processes stuck waiting for the NFS server to
resurrect itself) but you'll also run the risk of data loss on
read/write mounts.
Irix has the same definition of load, as far as I can tell: certainly I
remember seeing stupidly high load averages on perfectly responsive
systems.
Finally, if you don't like the Linux kernel's behaviour, kill the line
uninterruptible += cpu_rq(i)->nr_uninterruptible;
in nr_active() (kernel/sched.c).
-- [mdw]
Robert Maas, http://tinyurl.com/uh3t
I posted several articles under this address, but one of them got
harvested by a Nigerian spammer, who then sent me two spam to the
same address before I discovered it. As a result, I've shut down
this address so that I won't get any more spam via this address.
Any e-mail sent to this address will be accepted by the server then
just discarded without any non-delivery notice. That's why I'm
posting this warning, just in case somebody saw any of my articles
just now and wants to reply to me privately If anybody wants to
send private e-mail to me regarding anything I've posted, you'll
have to look around to find some other variant address that I
haven't yet disabled, or go to my Web site and click on "Contact
me".
As of a few days ago, Yahoo! Mail no longer provides any way to see
full headers of spam that comes in, so that's why the info isn't
included here. Previous recent Nigerian 419 spam of the same type
came from an IP number owned by Egyptian University, with a dropbox
at HotMail. Obviously neither Egypt nor MicroSoft is doing anything
to stop such spam.
Today in fact I received six spam, using four newly-harvested
SpamGourmet forwarding addresses, all of which I've now shut down:
lisp1.3....@spamgourmet.com
m6d02a4.3...@spamgourmet.com
un1.3.c...@spamgourmet.com
un1.3.c...@spamgourmet.com
usenet2.3...@spamgourmet.com
usenet2.3...@spamgourmet.com
-
Nobody in their right mind likes spammers, nor their automated assistants.
To open an account here, you must demonstrate you're not one of them.
Please spend a few seconds to try to read the text-picture in this box:
/----------------------------------------------------------------------------\
| ~|~|_ _|_| _ | _| ._ _ _|_ |o __|_ _._ _|_|_ _|_|)._ _ |
| | | |}_ _| \/\/(_)|_||(_| | |(_) | ||_\ | }_| |) | | |}_ _| | }_ |
| ._ _ _|_ |o __|_ _._ o._ (~| __|_o|| |) _._|_ _ |) _ _|_|_ _|_| |
| | |(_) | ||_\ | }_| ||| | _| _\ | |||) | }_| | |(_|| _\ | | |}_ _| |
| ._ _ _._ o|| |
| | |}_\/}_| \/\/|||o |
\----(Rendered by means of <http://www.schnoggo.com/figlet.html>)------------/
(You don't need JavaScript or images to see that ASCII-text image!!
You just need to view this in a fixed-pitch font such as Monaco.)
Then enter your best guess of the text (50-150 chars) into this TextArea:
+------------------------------------------------------------+
| |
| |
| |
| |
+------------------------------------------------------------+
George Neuner
<tfb+g...@tfeb.org> wrote:
>(This next bit is based on memory and may be wrong in places.) Solaris
>uses mapped files pervasively, to the extent that normal memory
>allocation is treated as mapping a file, which file happens to sit in
>a special filesystem which is backed by swap. For a long time this
>gave rise to all sorts of problems, one of which was that systems
>which were actually I/O bound could show up as having high load
>averages. Another, worse, problem was that processes doign intensive
>I/O (which is most of the important ones on your average big database
>box) could cause the VM system to start paging stuff that mattered
>out, such as executables.
You're right at least on the history. Solaris 2 definitely had these
problems.
>This all got fixed a few years ago by
>making the system more aware of whether memory was backed by normal
>files, or by anonymous swap, and I think by further treating read-only
>executable file mappings specially.
I'll take your word for it. I haven't touched Solaris since 2.5.1
Rob Warnock
+---------------
| Rob Warnock <rp...@rpw3.org> wrote:
| > Not a problem. The only reason I was being pretty stubborn about it
| > was that Linux is the *only* place I've ever seen these silly 100+
| > load averages on essentially idle systems!!
|
| I've seen it on Irix, too.
Really? I believe you; I've just never seen it myself
(and I worked at SGI for 13 years). Maybe I just never
ran the kind of thing that triggered it.
By the way, notice that I'm *not* talking about a system
that is busy doing a whole bunch of work with very little
consumption of CPU time, e.g., a big Lisp app that's
thrashing swap. Tim Bradshaw pointed out that "page-in wait"
is often counted as "runnable" (and thus appears in "load"),
and I can accept that as reasonable (for soem values of
"reasonable"). I'm talking about systems that are *idle*,
running *nothing* but the usual assortment of daemons,
yet with 100+ "load averages".
+---------------
| I feel a need to set the record straight for Linux, by the way.
| The load average counts
| * processes which are runnable, and
| * processes which are blocked /uninterruptably/.
...
| The only point where this gets really strange is NFS: if you mount the
| fileserver `hard' then the kernel will block uninterruptably for NFS
| responses, and therefore processes waiting for NFS will appear to be
| `loading' the system.
+---------------
Hmmm... This may be the clue, or at least related. The Linux
systems that I was complaining about were all NFS *servers*
which implemented server_NFS (or at least portions of it)
in the kernel. So at boot time they'd start up a bunch of
"nfsd" processes which would dive into the kernel and wait
to handle requests. It was probably these processes which
were viewed as "busy" even when they were completely idle.
[And, no, it was not necessary for a client to mount them
for the load average to skyrocket.]
Madhu
Wrote on Wed, 11 Jun 2008 04:14:49 -0500:
| Tim Bradshaw <tfb+g...@tfeb.org> wrote:
| +---------------
| | rp...@rpw3.org (Rob Warnock) wrote:
| | > So, yes, I guess I'm disagreeing with you, Tim.
| |
| | And that's because it turns out I was wrong, even for Solaris! Sorry!
| | Solaris computes the load average the way it has always been done (at
| | least back to BSD 4?), which is based on number of processes in the
| | run queue (so the way you think it should be done). Linux (it seems)
| | uses some function of run queue & blocked for I/O queue. So you're
| | right, and I was wrong.
| +---------------
|
| Not a problem. The only reason I was being pretty stubborn about it
| was that Linux is the *only* place I've ever seen these silly 100+
| load averages on essentially idle systems!!
By essentially idle I assume you mean no process shows 100% CPU
utilization (using `top', say)
I have observed weird numbers across a number of linux kernels across
varying implementations of the VM, some buggy. AIUI, typically weird
numbers were caused by IO on the VM system which tfb refers to [in the
next quoted section below -- linux uses this system heavily too]. In
some cases some badly behaving loadable modules [sound, fb] caused
threads which counted to the high load average without leaving a trace
on `ps' or `top' of the system. These are still indicative of a
malfunctioning system, rather than a poor load average algorithm.
[BTW when I posted the bit upthread where I quoted someone from google,
I did not mean to imply I believed what he said was correct, having
observed otherwise --- the idea was that the point that Tim X is
making, that the Original poster (Maas) could not use the load average
numbers to estimate the stuff he was talking about]
Also, BTW I noticed a Load_Average.pdf on the net, of p.62 Issue 83, Oct
2007, www.linux-magazine.com. "Understanding load average and stretch
factors" by Neil Gunther, which talks about a `strech factor' measure
but AFAICT also misses the VM I/O consideration.
| +---------------
| | What led me astray was that for a machine which is paging heavily,
| | then processes are in the run queue when in fact they are waiting for
| | disk (because they're waiting for memory which has been paged out).
| | It's *this* case where you can get huge load averages but tiny CPU
| | utilization. Obviously that doesn't happen as much as it did, but I
| | remember we used to see this a lot for Lisps in GC on the BSD machines
| | where I first saw Unix (30 or 30MB images on a machine with 2MB of
| | real memory, which did very well except during GC).
| |
Rob Warnock
| | was that Linux is the *only* place I've ever seen these silly 100+
| | load averages on essentially idle systems!!
|
| By essentially idle I assume you mean no process shows 100% CPU
| utilization (using `top', say)
No, I mean that: (1) no process was showing *any* CPU usage
[except a fraction of a percent for "top"]; (2) there was
*no* network traffic [checked with "tcpdump"]; and (3) since
the box was a NAS filer, network traffic is the *only* potential
workload there is [no "user" apps at all, except my shell].
Oh, and the "nfsd" processes were all showing *zero* CPU
consumption. And yet the load average was between 80-250
[probably higher the more NFS filesystems were exported].
-Rob
p.s. By the way, the only reason I said "essentially idle" instead
of "absolutely 100% totally idle" is that there are a few logging
tasks that run for a small fraction of a second every 5 minutes,
but given that there was nothing for them to log... ;-} ;-}
Tim X
I guess I've just not seen it amongst the absolutely huge amount of boot
messages linux now produces and at a speed that makes it impossible to
read at boot time.
(I'm old enough to remember network speeds when you could just read the
file with cat and Linux kernel boots that took most of the night to
compile !)
tim
Robert Uhl
>
> Thus a load average less than or equal to the number of cores in the
> system means that everything is getting CPU time that wants it
> (processes may still be starving for I/O of course).
>
> In particular, what is a "bad" load average scales as the number of
> cores: for many years of course Unix systems all had a single core,
> but nowadays very many systems have more than one of course. This
> took me a while to get used to, but now I always check the number of
> cores before panicking at a load of 25 or something...
Wouldn't number of cores be irrelevant to the badness of the number of
_waiting_ (as opposed to _running_) processes? I mean, if I have 14
processes waiting to run, it doesn't really matter if I have 16 cores or
1--I still have 14 processes which are not getting any work done.
Granted, with multiple cores those 14 processes will be gotten to more
quickly.
--
Robert Uhl <http://public.xdi.org/=ruhl>
Most people aren't thought about after they're gone. `I wonder where
Bob got the plutonium' is better than most get.
Robert Uhl
>
> Not a problem. The only reason I was being pretty stubborn about it
> was that Linux is the *only* place I've ever seen these silly 100+
> load averages on essentially idle systems!!
You know, one could always submit a patch...
--
Robert Uhl <http://public.xdi.org/=ruhl>
Rack mount machines look nice, no doubt about it, but they were never
meant to be worked on. If the fellow who invented them were anywhere
near me last night, he would be 1U high right now. --Tom Liston
Rob Warnock
+---------------
| > The only reason I was being pretty stubborn about it
| > was that Linux is the *only* place I've ever seen these
| > silly 100+ load averages on essentially idle systems!!
|
| You know, one could always submit a patch...
I believe the company I was working for at the time did just that,
but [IIRC] it was rejected by the Linux kernel core maintainers:
"It ain't a bug, it's a *FEE-CHURE*!" (*sigh*) :-{
One can only hope that preferences change over time, and
that this "feature" will someday be recognized as a bug.
Robert Uhl
> Robert Uhl <eadm...@NOSPAMgmail.com> wrote:
> +---------------
> | rp...@rpw3.org (Rob Warnock) writes:
> | > The only reason I was being pretty stubborn about it
> | > was that Linux is the *only* place I've ever seen these
> | > silly 100+ load averages on essentially idle systems!!
> |
> | You know, one could always submit a patch...
> +---------------
>
> I believe the company I was working for at the time did just that,
> but [IIRC] it was rejected by the Linux kernel core maintainers:
> "It ain't a bug, it's a *FEE-CHURE*!" (*sigh*) :-{
Ah, well perhaps there's a good reason to do it differently, but I sure
as heck can't see it.
Brain-dead maintainers are a problem. And it's hardly worth forking
over.
> One can only hope that preferences change over time, and
> that this "feature" will someday be recognized as a bug.
I'd actually be interested in why they consider it a feature (beyond
reflexive 'this is how it works, why change it?').
--
Robert Uhl <http://public.xdi.org/=ruhl>
Considering the number of wheels Microsoft has found reason to invent,
one never ceases to be baffled by the minuscule number whose shape
even vaguely resembles a circle. --unknown
Jon Harrop
> The whole point of distributing my tasks over 64 processors is to
> try to get it to run 64 times as fast as if running on a single
> processor, to allow a couple orders of magnitude more data to be
> processed within a reasonable time.
If you want your programs to run at a reasonable speed you should not be
using Lisp in the first place. So your best bet is probably to rewrite your
code in a modern language and benefit from their enormous performance
improvements.
--
Dr Jon D Harrop, Flying Frog Consultancy
http://www.ffconsultancy.com/products/?u
Ariel
Jon Harrop <j...@ffconsultancy.com> wrote:
> Robert Maas, http://tinyurl.com/uh3t wrote:
> > The whole point of distributing my tasks over 64 processors is to
> > try to get it to run 64 times as fast as if running on a single
> > processor, to allow a couple orders of magnitude more data to be
> > processed within a reasonable time.
>
> If you want your programs to run at a reasonable speed you should not be
> using Lisp in the first place. So your best bet is probably to rewrite your
> code in a modern language and benefit from their enormous performance
> improvements.
I thought using a modern compiler for Lisp would allow it to perform as well as any other standard modern day high level language? (Or so says Paul Graham.) Is this a false statement?
Edi Weitz
> I thought using a modern compiler for Lisp would allow it to perform
> as well as any other standard modern day high level language? (Or
> so says Paul Graham.) Is this a false statement?
The statement is correct. The person who posted the mis-information
you replied to is a well-known troll who tries everything (including
blatant lies) to sell his books and magazines. Search the archives of
this newsgroup for other postings of him.
Edi.
--
Lisp is not dead, it just smells funny.
Real email: (replace (subseq "spam...@agharta.de" 5) "edi")
Rainer Joswig
wrote:
> On Fri, 20 Jun 2008 08:18:55 -0700, Ariel <n...@mail.poo> wrote:
>
> > I thought using a modern compiler for Lisp would allow it to perform
> > as well as any other standard modern day high level language? (Or
> > so says Paul Graham.) Is this a false statement?
>
> The statement is correct. The person who posted the mis-information
> you replied to is a well-known troll who tries everything (including
> blatant lies) to sell his books and magazines. Search the archives of
> this newsgroup for other postings of him.
>
> Edi.
He (They?) reminds me of Lucius Detritus, if you know who
I mean (Germans might know him as Tullius Destruktivus). ;-)
Remember the green speech bubbles?
Ariel
I used to love reading Asterix :)
Rainer Joswig
wrote:
> On Fri, 20 Jun 2008 18:42:53 +0200
> Rainer Joswig <jos...@lisp.de> wrote:
>
> > In article <uskv8h...@agharta.de>, Edi Weitz <spam...@agharta.de>
> > wrote:
> >
> > > On Fri, 20 Jun 2008 08:18:55 -0700, Ariel <n...@mail.poo> wrote:
> > >
> > > > I thought using a modern compiler for Lisp would allow it to perform
> > > > as well as any other standard modern day high level language? (Or
> > > > so says Paul Graham.) Is this a false statement?
> > >
> > > The statement is correct. The person who posted the mis-information
> > > you replied to is a well-known troll who tries everything (including
> > > blatant lies) to sell his books and magazines. Search the archives of
> > > this newsgroup for other postings of him.
> > >
> > > Edi.
> >
> > He (They?) reminds me of Lucius Detritus, if you know who
> > I mean (Germans might know him as Tullius Destruktivus). ;-)
> > Remember the green speech bubbles?
> >
> > --
> > http://lispm.dyndns.org/
>
> I used to love reading Asterix :)
I'd say there are some parallels between Asterix and co. and
the Lisp users. ;-)
Edi Weitz
> He (They?) reminds me of Lucius Detritus, if you know who I mean
> (Germans might know him as Tullius Destruktivus). ;-) Remember the
> green speech bubbles?
Hehe. I sometimes had to think of Lucius Detritus when I read
gavino's postings. He wrote one sentence and then disappeared, but it
was usually enough to keep half of c.l.l busy for a week... :)
Jon Harrop
That certainly is a false statement, yes. Even for trivial tasks, it can be
vastly more difficult to write comparably efficient Lisp code. For example,
this toy symbolic simplifier:
http://www.lambdassociates.org/studies/study10.htm
The OCaml code is not particularly efficient and OCaml itself is not
particularly well-suited to this benchmark (MLton-compiled SML would be a
lot faster):
let rec ( +: ) f g = match f, g with
| `Int n, `Int m -> `Int (n +/ m)
| `Int (Int 0), e | e, `Int (Int 0) -> e
| f, `Add(g, h) -> f +: g +: h
| f, g -> `Add(f, g)
let rec ( *: ) f g = match f, g with
| `Int n, `Int m -> `Int (n */ m)
| `Int (Int 0), e | e, `Int (Int 0) -> `Int (Int 0)
| `Int (Int 1), e | e, `Int (Int 1) -> e
| f, `Mul(g, h) -> f *: g *: h
| f, g -> `Mul(f, g)
let rec simplify = function
| `Int _ | `Var _ as f -> f
| `Add (f, g) -> simplify f +: simplify g
| `Mul (f, g) -> simplify f *: simplify g
The obvious Lisp is 7.5x slower than the unoptimized OCaml:
(defun simplify (a)
(if (atom a)
a
(destructuring-bind (op x y) a
(let* ((f (simplify x))
(g (simplify y))
(nf (numberp f))
(ng (numberp g))
(+? (eq '+ op))
(*? (eq '* op)))
(cond
((and +? nf ng) (+ f g))
((and +? nf (zerop f)) g)
((and +? ng (zerop g)) f)
((and (listp g) (eq op (first g)))
(destructuring-bind (op2 u v) g
(simplify `(,op (,op ,f ,u) ,v))))
((and *? nf ng) (* f g))
((and *? (or (and nf (zerop f))
(and ng (zerop g)))) 0)
((and *? nf (= 1 f)) g)
((and *? ng (= 1 g)) f)
(t `(,op ,f ,g)))))))
If you openly advertise this as a programming challenge and get dozens of
expert Lisp programmers to painstakingly optimize solutions to this trivial
problem over a period of many weeks then the fastest implementation you get
is still 1.7x slower than the unoptimized OCaml but, more importantly, it
is a completely unmaintainable mess:
(defun simplify-no-redundant-checks (xexpr)
(if (atom xexpr)
xexpr
(let ((op (first xexpr))
(z (second xexpr))
(y (third xexpr)))
(let* ((f (simplify-no-redundant-checks z))
(g (simplify-no-redundant-checks y))
(nf (numberp f))
(ng (numberp g)))
(tagbody
START
(if (eq '+ op) (go OPTIMIZE-PLUS) (go TEST-MULTIPLY))
OPTIMIZE-PLUS
(when (and nf ng) (return-from simplify-no-redundant-checks (+
f g)))
TEST-PLUS-ZEROS
(when (eql f 0) (return-from simplify-no-redundant-checks g))
(when (eql g 0) (return-from simplify-no-redundant-checks f))
(go REARRANGE-EXPR)
TEST-MULTIPLY
(unless (eq '* op) (go REARRANGE-EXPR))
OPTIMIZE-MULTIPLY
(when (and nf ng) (return-from simplify-no-redundant-checks (*
f g)))
TEST-MULTIPLY-ZEROS-AND-ONES
(when (or (eql f 0) (eql g 0)) (return-from
simplify-no-redundant-checks 0))
(when (eql f 1) (return-from simplify-no-redundant-checks g))
(when (eql g 1) (return-from simplify-no-redundant-checks f))
REARRANGE-EXPR
(when (and (listp g) (eq op (first g)))
(let ((op2 (first g))
(u (second g))
(v (third g)))
(declare (ignore op2))
(return-from simplify-no-redundant-checks
(simplify-no-redundant-checks (list op (list op f u)
v)))))
MAYBE-CONS-EXPR
(if (and (eq f z) (eq g y))
(return-from simplify-no-redundant-checks xexpr)
(return-from simplify-no-redundant-checks (list op f
g))))))))
Lisp's awful performance only gets worse as your programs get more
complicated. For anything non-trivial, Lisp is inevitably incredibly slow.
Eventually, all Lisp programmers end up Greenspunning features like
optimizing pattern matchers over algebraic data types that are taken for
granted in modern functional languages. As a dynamic language, Lisp cannot
statically check even the most basic of constraints so programmers are
forced to code in the debugger and waste their lives writing unit testing
code.
Fortunately, there are a wealth of much better modern functional programming
languages out there for you to use, like SML, OCaml, Haskell, F# and Scala.
They are not only many times faster than Lisp for real work but also much
more expressive and concise with better tools and much larger communities
of friendly users. If you are interested in earning a living then I
strongly recommend taking a look at Scala and F# because they already have
vastly wealthier markets than Lisp will ever have.
Regards,
Rainer Joswig
Jon Harrop <j...@ffconsultancy.com> wrote:
> Ariel wrote:
> > On Fri, 20 Jun 2008 14:32:50 +0100
> > Jon Harrop <j...@ffconsultancy.com> wrote:
> >> Robert Maas, http://tinyurl.com/uh3t wrote:
> >> > The whole point of distributing my tasks over 64 processors is to
> >> > try to get it to run 64 times as fast as if running on a single
> >> > processor, to allow a couple orders of magnitude more data to be
> >> > processed within a reasonable time.
> >>
> >> If you want your programs to run at a reasonable speed you should not be
> >> using Lisp in the first place. So your best bet is probably to rewrite
> >> your code in a modern language and benefit from their enormous
> >> performance improvements.
> >
> > I thought using a modern compiler for Lisp would allow it to perform as
> > well as any other standard modern day high level language? (Or so says
> > Paul Graham.) Is this a false statement?
>
> That certainly is a false statement, yes. Even for trivial tasks, it can be
> vastly more difficult to write comparably efficient Lisp code. For example,
> this toy symbolic simplifier:
___________________________
/| /| | |
||__|| | Please don't |
/ O O\__ feed |
/ \ the troll |
/ \ \ |
/ _ \ \ ----------------------
/ |\____\ \ ||
/ | | | |\____/ ||
/ \|_|_|/ | __||
/ / \ |____| ||
/ | | /| | --|
| | |// |____ --|
* _ | |_|_|_| | \-/
*-- _--\ _ \ // |
/ _ \\ _ // | /
* / \_ /- | - | |
* ___ c_c_c_C/ \C_c_c_c____________
John Thingstad
<j...@ffconsultancy.com>:
Spoken like someone who has never actually programmed Lisp.
Just earlier this week I made a program to compute prime numbers.
It found all primes < 1000000 in 1.015 seconds.
That is just as fast as the same algorithm in C...
--------------
John Thingstad
John Thingstad
<j...@ffconsultancy.com>:
Running Pascal's code:
Cl-USER 6 > (time (test 10000000))
Timing the evaluation of (TEST 10000000)
User time = 1.328
System time = 0.000
Elapsed time = 1.421
Allocation = 4780 bytes
0 Page faults
That's in fact 43% faster than the OCalm program..
Of cource I use LispWorks not SBCL which seems to optimize CLOS code
better.
(Pentium 820 Dual core 2.8MHz running Windows.)
Anyhow 10 million simplifications in a little more than a second doesn't
strike me as particulary slow.
--------------
John Thingstad
Kenny
You answered Harrop? You really answered Harrop? After all this time,
you actually really seriously answered Harrop? Ariel has an excuse, for
you we need some more Shakespeare, not sure what... a, the solstice is
at hand, not exactly mid, but something from A Midsummer's Night Dream
must apply...
kt
Jon Harrop
> På Fri, 20 Jun 2008 22:19:33 +0200, skrev Jon Harrop
> <j...@ffconsultancy.com>:
>
> Spoken like someone who has never actually programmed Lisp.
Note that I didn't even write the Lisp.
Jon Harrop
Yes, of course. That task is so simple that Lisp's deficiencies are not
relevant.
Jon Harrop
> On Fri, 20 Jun 2008 08:18:55 -0700, Ariel <n...@mail.poo> wrote:
>> I thought using a modern compiler for Lisp would allow it to perform
>> as well as any other standard modern day high level language? (Or
>> so says Paul Graham.) Is this a false statement?
>
> The statement is correct. The person who posted the mis-information
> you replied to is a well-known troll who tries everything (including
> blatant lies) to sell his books and magazines. Search the archives of this
> newsgroup for other postings of him.
Note the breadth of programming language experience offered by Edi Weitz:
Jon Harrop
> I thought using a modern compiler for Lisp would allow it to perform as
> well as any other standard modern day high level language? (Or so says
> Paul Graham.) Is this a false statement?
Here is another example: the Mersenne Twister PRNG. Comparing the Common
Lisp code here:
to implementations in other languages when computing 10^8 random integers:
C: 0.61s
F#: 1.77s
OCaml: 3.4s
SBCL: 13.6s
As you can see, Lisp is very slow on this benchmark as well, at least when
compiled with SBCL.
Ariel
Jon Harrop <j...@ffconsultancy.com> wrote:
> Ariel wrote:
> > I thought using a modern compiler for Lisp would allow it to perform as
> > well as any other standard modern day high level language? (Or so says
> > Paul Graham.) Is this a false statement?
>
> Here is another example: the Mersenne Twister PRNG. Comparing the Common
> Lisp code here:
>
> http://www.cliki.net/MT19937
>
> to implementations in other languages when computing 10^8 random integers:
>
> C: 0.61s
> F#: 1.77s
> OCaml: 3.4s
> SBCL: 13.6s
>
> As you can see, Lisp is very slow on this benchmark as well, at least when
> compiled with SBCL.
You only linked to a project who's main goal was consistency across platforms and implimentations over speed. Is there a link to these benchmark comparisons?
-a
Ariel
Jon Harrop <j...@ffconsultancy.com> wrote:
> John Thingstad wrote:
> > På Fri, 20 Jun 2008 22:19:33 +0200, skrev Jon Harrop
> > <j...@ffconsultancy.com>:
> >
> > Spoken like someone who has never actually programmed Lisp.
> >
> > Just earlier this week I made a program to compute prime numbers.
> > It found all primes < 1000000 in 1.015 seconds.
> > That is just as fast as the same algorithm in C...
>
> Yes, of course. That task is so simple that Lisp's deficiencies are not
> relevant.
Saying that a task is too simple doesn't give your argument any weight, some languages process sleep cycles faster than others, so what. If instead you had said something like "it avoids these specific slower functions of Lisp" such that to actually show where slowness occurs in the language, it would become a helpful comment.
For example Perl isn't the fastest language to start with, but becomes drastically slower when processing regex. Thus if speed is your priority over ease of programming, you should avoid regex as much as possible so Perl gains back towards its maximum speed potential.
-a
Jon Harrop
> On Sat, 21 Jun 2008 10:01:20 +0100
> Jon Harrop <j...@ffconsultancy.com> wrote:
>> John Thingstad wrote:
>> > Just earlier this week I made a program to compute prime numbers.
>> > It found all primes < 1000000 in 1.015 seconds.
>> > That is just as fast as the same algorithm in C...
>>
>> Yes, of course. That task is so simple that Lisp's deficiencies are not
>> relevant.
>
> Saying that a task is too simple doesn't give your argument any weight,
> some languages process sleep cycles faster than others, so what. If
> instead you had said something like "it avoids these specific slower
> functions of Lisp" such that to actually show where slowness occurs in the
> language, it would become a helpful comment.
My original point was that Lisp makes high-level programming slow. For
example, high-level abstract constructs like pattern matching are heavily
optimized by the compilers of all modern functional languages but Lisp is
incapable of anything comparable (without drastically changing the language
and Greenspunning modern language features).
John's response that Lisp can compete with C on a low-level program is true
for trivial problems but neither relevant to my point nor interesting in
the context of real programming (unless you are trying to solve problems so
simple that it is feasible for you to use C).
Fortunately, modern functional languages are so much more effective than
Lisp that you don't have to study much more complicated programs (e.g. the
Mersenne Twister rather than a prime sieve) to appreciate just how far in
advance these modern languages are.
Jon Harrop
> On Sat, 21 Jun 2008 11:00:02 +0100
> Jon Harrop <j...@ffconsultancy.com> wrote:
>> C: 0.61s
>> F#: 1.77s
>> OCaml: 3.4s
>> SBCL: 13.6s
>
> platforms and implimentations over speed.
They also proudly claimed improved performance over a competitor:
"...faster than the JMT Mersenne Twister implementation"
Moreover, the C code is consistent between platforms, the F# code runs
under .NET or Mono but the OCaml code was 64-bit only.
> Is there a link to these benchmark comparisons? -a
I just downloaded and ran Mersenne Twisters in each language except for the
F# code which is part of our commercial F# for Numerics library.
George Neuner
Jon Harrop has a bug up his ass about brevity. His main complaint
about Lisp is not its lack of performance, but the fact that high
performance is not the default setting and his simple, naively written
code doesn't run very fast.
He has very little Lisp experience and his comments regarding Lisp
should simply be ignored.
George
--
for email reply remove "/" from address
Raymond Wiker
Terry Pratchett also has a character named "Detritus"; his
version is a troll who uses a helmet with a clockwork fan to boost his
mental capacity. Could be that Harrop could use one of those, too.
Raymond Wiker
> Edi Weitz wrote:
>> On Fri, 20 Jun 2008 08:18:55 -0700, Ariel <n...@mail.poo> wrote:
>>> I thought using a modern compiler for Lisp would allow it to perform
>>> as well as any other standard modern day high level language? (Or
>>> so says Paul Graham.) Is this a false statement?
>>
>> The statement is correct. The person who posted the mis-information
>> you replied to is a well-known troll who tries everything (including
>> blatant lies) to sell his books and magazines. Search the archives of this
>> newsgroup for other postings of him.
>
> Note the breadth of programming language experience offered by Edi Weitz:
>
> http://www.weitz.de
His web pages shows that he is a more than capable Lisp
programmer, which means that he is worth listening to in a Lisp
newsgroup. You, on the other hand, have shown yourself worth
ignoring, in Lisp newsgroups as well as a number of other places.