Cross platform way of finding number of processors on a machine?
John
Is there a way to find the number of processors on a machine (on linux/
windows/macos/cygwin) using python code (using the same code/cross
platform code)?
Nicholas Bastin
There's no single call that will give you the same info on every
platform, but you can obviously write this yourself and switch based
on os.uname()[0] in most cases.
For Darwin, you can just use the subprocess module to call 'sysctl
hw.logicalcpu'. I'm not sure if there's a more direct way in python
to use sysctl. (hw.logicalcpu_max is what the hardware maximally
supports, but someone may have started their machine with OF blocking
some of the processors and you should probably respect that decision)
For Linux you can read /proc/cpuinfo and parse that information. Be
somewhat careful with this, however, if your processors support HT,
they will show as 2, and that may or may not be what you want. You
can deterministically parse this information out if you know which
processor families are truly multi-core, and which are HT.
For Win32, the cheap and dirty way is to read the NUMBER_OF_PROCESSORS
environment variable. 99% of the time, this will be correct, so it
might be sufficient for your purposes. There is a system call
available on windows that will net you the true number of virtual
cpus, but I don't know what it is.
I don't have a cygwin install laying around, but my guess is there's a
sysctl option there too.
One note: *all* of these methods will tell you the virtual number of
CPUs in a machine, not the physical number. There's almost no reason
why you care about the distinction between these two numbers, but if
you do, you'll have to go to great lengths to probe the actual
hardware on each platform. (And pre-WinXP, Windows doesn't actually
know the difference - all processors are presumed to be physical).
Also, Darwin and Linux will easily allow you to get the speed of the
processors, but on x86 these numbers are not the maximums due to C1E
and EIST (x86 processors from all vendors are capable of changing
speeds depending on load).
--
Nick
Tim Golden
> platform, but you can obviously write this yourself and switch based
> on os.uname()[0] in most cases.
Second that point about not getting one cross-platform
answer. Under Windows, WMI is often the way to go for
these things:
http://msdn2.microsoft.com/en-us/library/aa394373.aspx
but, as Nicholas noted:
"""
Windows Server 2003, Windows XP, and Windows 2000:
This property is not available.
"""
since it's presumably not available in anything earlier either,
that leaves you with Vista or the early-adopter editions of the
next Windows Server product.
TJG
Kay Schluehr
Remarkable. I've a two years old Windows XP dual core notebook and
when I'm asking Python I get the correct answer:
>>> import os
>>> os.environ['NUMBER_OF_PROCESSORS']
2
However this feature might not be guaranteed by Microsoft for
arbitrary computers using their OS?
Kay
Tim Golden
>> """
>> Windows Server 2003, Windows XP, and Windows 2000:
>> This property is not available.
>> """
>>
>> since it's presumably not available in anything earlier either,
>> that leaves you with Vista or the early-adopter editions of the
>> next Windows Server product.
Kay Schluehr wrote:
> Remarkable. I've a two years old Windows XP dual core notebook and
> when I'm asking Python I get the correct answer:
>
>>>> import os
>>>> os.environ['NUMBER_OF_PROCESSORS']
> 2
>
> However this feature might not be guaranteed by Microsoft for
> arbitrary computers using their OS?
Ahem. I was referring (and not very clearly, now I look
back at my post) to a particular property within the
WMI class, not to the general concept of counting
processors.
TJG
Nick Craig-Wood
> On 10/4/07, John <weeken...@yahoo.com> wrote:
> >
> > Is there a way to find the number of processors on a machine (on linux/
> > windows/macos/cygwin) using python code (using the same code/cross
> > platform code)?
>
> There's no single call that will give you the same info on every
> platform, but you can obviously write this yourself and switch based
> on os.uname()[0] in most cases.
>
> For Darwin, you can just use the subprocess module to call 'sysctl
> hw.logicalcpu'. I'm not sure if there's a more direct way in python
> to use sysctl. (hw.logicalcpu_max is what the hardware maximally
> supports, but someone may have started their machine with OF blocking
> some of the processors and you should probably respect that decision)
>
> For Linux you can read /proc/cpuinfo and parse that information. Be
> somewhat careful with this, however, if your processors support HT,
> they will show as 2, and that may or may not be what you want. You
> can deterministically parse this information out if you know which
> processor families are truly multi-core, and which are HT.
On any unix/posix system (OSX and linux should both qualify) you can use
>>> import os
>>> os.sysconf('SC_NPROCESSORS_ONLN')
2
>>>
(From my Core 2 Duo laptop running linux)
--
Nick Craig-Wood <ni...@craig-wood.com> -- http://www.craig-wood.com/nick
Lawrence Oluyede
From processing <http://cheeseshop.python.org/pypi/processing/0.34> :
def cpuCount():
'''
Returns the number of CPUs in the system
'''
if sys.platform == 'win32':
try:
num = int(os.environ['NUMBER_OF_PROCESSORS'])
except (ValueError, KeyError):
pass
elif sys.platform == 'darwin':
try:
num = int(os.popen('sysctl -n hw.ncpu').read())
except ValueError:
pass
else:
try:
num = os.sysconf('SC_NPROCESSORS_ONLN')
except (ValueError, OSError, AttributeError):
pass
if num >= 1:
return num
else:
raise NotImplementedError
--
Lawrence, oluyede.org - neropercaso.it
"It is difficult to get a man to understand
something when his salary depends on not
understanding it" - Upton Sinclair
Paul Boddie
>
> From processing <http://cheeseshop.python.org/pypi/processing/0.34> :
[...]
> num = os.sysconf('SC_NPROCESSORS_ONLN')
It's interesting what new (or obscure) standard library functions (and
system functions) can be discovered through these kinds of
discussions. However, this one seems to report the number of "virtual
CPUs" in situations like mine where the CPU supports hyperthreading,
as Nicholas Bastin says. One way to get the real number of CPUs (on
Linux-based systems) is to parse /proc/cpuinfo, and to count the
number of distinct "physical id" values.
>From the experiments I've done with pprocess [1], pretending that a
hyperthreading "virtual CPU" is as good as a real CPU (or CPU core)
tends to be counter-productive: the performance benefits in moving
from the utilisation of one to two "virtual CPUs" are virtually non-
existent.
Paul
Lawrence Oluyede
> From the experiments I've done with pprocess [1], pretending that a
> hyperthreading "virtual CPU" is as good as a real CPU (or CPU core)
> tends to be counter-productive: the performance benefits in moving
> from the utilisation of one to two "virtual CPUs" are virtually non-
> existent.
We came the same conclusion at work. One of my colleagues has a machine
with HyperThreading and he noticed no real performance increase with the
processing library altough on my machine (which is a dual Core 2 Duo)
the performance boost was definitely there :-)
The good side of this is hyperthreading with parallel processes is like
using a mono-CPU machine so we don't have to take into account it like
if it was an entirely different kind of architecture.
Nick Craig-Wood
There is a bug in that code...
NotImplementedError will never be raised because num won't have been
set. It will raise "UnboundLocalError: local variable 'num'
referenced before assignment" instead
Nicholas Bastin
> John <weeken...@yahoo.com> wrote:
> > Is there a way to find the number of processors on a machine (on linux/
> > windows/macos/cygwin) using python code (using the same code/cross
> > platform code)?
>
> >From processing <http://cheeseshop.python.org/pypi/processing/0.34> :
A few reiterated notes inline, from my previous message.
> def cpuCount():
> '''
> Returns the number of CPUs in the system
> '''
> if sys.platform == 'win32':
> try:
> num = int(os.environ['NUMBER_OF_PROCESSORS'])
The user can do bad things to this environment variable, but it's
probably ok most of the time. (Hey, they change it, they pay the
consequences).
> else:
> try:
> num = os.sysconf('SC_NPROCESSORS_ONLN')
This is really bad on linux. You really want to parse /proc/cpuinfo
because HT 'cpus' are almost useless, and can be bad in situations
where you try to treat them as general purpose cpus.
--
Nick
Lawrence Oluyede
> This is really bad on linux. You really want to parse /proc/cpuinfo
> because HT 'cpus' are almost useless, and can be bad in situations
> where you try to treat them as general purpose cpus.
I'm not the author of the library. I think you should address these bug
reports elsewhere...