detecting number of (physical) CPUs from Python

Showing 1-14 of 14 messages
detecting number of (physical) CPUs from Python Guy 4/10/12 9:39 PM
Hi all,

I might need to hack up a little Python scriptlet to generate
configuration files for a particular software here at uni (the software
is Condor High Throughput Computing). The problem is, that I don't know
how to detect the number of *physical* CPU cores on a machine. I can get
the number of *virtual* cores quite simply using multiprocessing's
cpu_count() function. Under Linux I'd know how to wiggle myself through
using /proc/cpuinfo to find out how many physical cores a machine has got.

Now's the big question: Does anybody know a (good or dirty) way to
detect the number of physical CPU cores in Python on a *Windows* box
(Win 7)?

Guy

--
Guy K. Kloss
School of Computing + Mathematical Sciences
Auckland University of Technology
Private Bag 92006, Auckland 1142
phone: +64 9 921 9999 ext. 5032
eMail: Guy....@aut.ac.nz

Re: [nzpug] detecting number of (physical) CPUs from Python Danny W. Adair 4/10/12 9:50 PM
Hi Guy,

On Wed, Apr 11, 2012 at 16:39, Guy K. Kloss <guy....@aut.ac.nz> wrote:
>[...]


> Now's the big question: Does anybody know a (good or dirty) way to
> detect the number of physical CPU cores in Python on a *Windows* box
> (Win 7)?

I don't use Windows but this is how Parallel Python does it - seems
like a corresponding environment variable can be expected:

==========================
    def __detect_ncpus(self):
        """Detects the number of effective CPUs in the system"""
        #for Linux, Unix and MacOS
        if hasattr(os, "sysconf"):
            if "SC_NPROCESSORS_ONLN" in os.sysconf_names:
                #Linux and Unix
                ncpus = os.sysconf("SC_NPROCESSORS_ONLN")
                if isinstance(ncpus, int) and ncpus > 0:
                    return ncpus
            else:
                #MacOS X
                return int(os.popen2("sysctl -n hw.ncpu")[1].read())
        #for Windows
        if "NUMBER_OF_PROCESSORS" in os.environ:
            ncpus = int(os.environ["NUMBER_OF_PROCESSORS"])
            if ncpus > 0:
                return ncpus
        #return the default value
        return 1
==========================

Cheers,
Danny

Re: [nzpug] detecting number of (physical) CPUs from Python Olly Betts 4/10/12 9:58 PM
On Wed, Apr 11, 2012 at 04:39:30PM +1200, Guy K. Kloss wrote:
> I might need to hack up a little Python scriptlet to generate
> configuration files for a particular software here at uni (the software
> is Condor High Throughput Computing). The problem is, that I don't know
> how to detect the number of *physical* CPU cores on a machine. I can get
> the number of *virtual* cores quite simply using multiprocessing's
> cpu_count() function. Under Linux I'd know how to wiggle myself through
> using /proc/cpuinfo to find out how many physical cores a machine has got.

Calling sysconf(_SC_NPROCESSORS_ONLN) will work on Linux and some other
Unixy platforms too, if you care about more than just Linux:

        - _SC_NPROCESSORS_ONLN
              The number of processors currently online (available).

Not sure if sysconf() is easily accessible from Python though.

Both this and /proc/cpuinfo seem to count hyperthreaded cores as more
than one, which may or may not be what you want.

> Now's the big question: Does anybody know a (good or dirty) way to
> detect the number of physical CPU cores in Python on a *Windows* box
> (Win 7)?

Sorry, don't know about that.

Cheers,
    Olly

Re: [nzpug] detecting number of (physical) CPUs from Python Stephen Bell 4/11/12 3:57 AM
Hi
The registry registry reports hyperthreads as separate cpus, but you can get the processor identifier.
> import _winreg
>
> with _winreg.OpenKey(_winreg.HKEY_LOCAL_MACHINE, "Hardware\DESCRIPTION\System\CentralProcessor") as key:
> (N_CPUS, N_values, time) = _winreg.QueryInfoKey(key)
> for CPU in range(N_CPUS):
> with _winreg.OpenKey(key, str(CPU) ) as key2:
> (Identifier, n) = _winreg.QueryValueEx(key2,"Identifier")
> (ProcessorName, n) = _winreg.QueryValueEx(key2,"ProcessorNameString")
> print "CPU: ",CPU,Identifier,ProcessorName
>




On 11/04/2012 4:39 p.m., Guy K. Kloss wrote:
> Hi all,
>
> I might need to hack up a little Python scriptlet to generate
> configuration files for a particular software here at uni (the software
> is Condor High Throughput Computing). The problem is, that I don't know
> how to detect the number of *physical* CPU cores on a machine. I can get
> the number of *virtual* cores quite simply using multiprocessing's
> cpu_count() function. Under Linux I'd know how to wiggle myself through
> using /proc/cpuinfo to find out how many physical cores a machine has got.
>
> Now's the big question: Does anybody know a (good or dirty) way to
> detect the number of physical CPU cores in Python on a *Windows* box
> (Win 7)?
>
> Guy
>


--
Stephen Bell

56 Jeffreys Rd, Christchurch 8052
Ph:   +64 3 351 4251    Mob: 021 109 8204

Re: [nzpug] detecting number of (physical) CPUs from Python Guy 4/11/12 2:48 PM
On 11/04/12 16:50, Danny Adair wrote:
> I don't use Windows but this is how Parallel Python does it - seems
> like a corresponding environment variable can be expected:

Funny that. I've also looked at the code for Parallel Python in this
case, as well as the source of multiprocessing.cpu_count(). But they all
only report the total virtual cores (incl. hyper-threading duplicates).

Guy


--
Guy K. Kloss
School of Computing + Mathematical Sciences
Auckland University of Technology
Private Bag 92006, Auckland 1142
phone: +64 9 921 9999 ext. 5032
eMail: Guy....@aut.ac.nz

Re: [nzpug] detecting number of (physical) CPUs from Python Guy 4/11/12 2:57 PM
On 11/04/12 16:58, Olly Betts wrote:
> Calling sysconf(_SC_NPROCESSORS_ONLN) will work on Linux and some other
> Unixy platforms too, if you care about more than just Linux:
>
>         - _SC_NPROCESSORS_ONLN
>               The number of processors currently online (available).
>
> Not sure if sysconf() is easily accessible from Python though.

Yes, using os.sysconf("SC_NPROCESSORS_ONLN").

But it also reports all virtual CPU cores.

> Both this and /proc/cpuinfo seem to count hyperthreaded cores as more
> than one, which may or may not be what you want.

But one can puzzle that out from the info in there. Here's some shell
magic that can do the trick:

Number of physical individual chips (not cores):

grep 'physical id' /proc/cpuinfo | sort | uniq | wc -l

Number of physical cores that is associated with every CPU core entry in
cpuinfo:

grep 'cpu cores' /proc/cpuinfo

I got those tricks from here [1]. Unfortunately, that's a no-go for the
Win world ... :-(

Guy


[1]
https://lists.cs.wisc.edu/archive/condor-users/2011-November/msg00079.shtml

--
Guy K. Kloss
School of Computing + Mathematical Sciences
Auckland University of Technology
Private Bag 92006, Auckland 1142
phone: +64 9 921 9999 ext. 5032
eMail: Guy....@aut.ac.nz

Re: [nzpug] detecting number of (physical) CPUs from Python Guy 4/11/12 4:11 PM
Hi,

On 11/04/12 22:57, Stephen Bell wrote:
> The registry registry reports hyperthreads as separate cpus, but you can
> get the processor identifier.
>> import _winreg
>>
>> with _winreg.OpenKey(_winreg.HKEY_LOCAL_MACHINE,
> "Hardware\DESCRIPTION\System\CentralProcessor") as key:
>> (N_CPUS, N_values, time) = _winreg.QueryInfoKey(key)
>> for CPU in range(N_CPUS):
>> with _winreg.OpenKey(key, str(CPU) ) as key2:
>> (Identifier, n) = _winreg.QueryValueEx(key2,"Identifier")
>> (ProcessorName, n) = _winreg.QueryValueEx(key2,"ProcessorNameString")
>> print "CPU: ",CPU,Identifier,ProcessorName

The indentation was all messed up, but it was easy enough to sort that
out straight. That is some seriously dark magic that you're invoking
here! ;-)

Anyway, unfortunately that approach didn't help me much further, as it
also only spat out those 8 virtual cores on a HT quad core system. I've
got neither the registry foo nor the regedit.exe (disabled by admins
here) to dig more deeply into it and see if I can find out whether there
are other entries that might help. Google was also a bit fuzzy/opaque on
that with my search queries.

However, I've found one solution while ploughing through Google along
somewhat similar lines. There's something on Wintendo called "WMI" that
can help. There's also a Python wrapper available for WMI, but I need
something that works out of the box with our Python(x,y) install here.
Gladly, there's a command line tool "wmic" that can help. With the
following command line I can get the number of physical CPU cores:

H:\> wmic cpu get NumberOfCores
NumberOfCores
4

And a little bit of Python wrapped around will give me the integer value:

import subprocess
output = subprocess.check_output('wmic cpu get NumberOfCores',
                                 shell=True)
print 'NumberOfCores using wmic:', int(output.split()[1])

Maybe a bit dirty, but it works!

Guy

--
Guy K. Kloss
School of Computing + Mathematical Sciences
Auckland University of Technology
Private Bag 92006, Auckland 1142
phone: +64 9 921 9999 ext. 5032
eMail: Guy....@aut.ac.nz

Re: [nzpug] detecting number of (physical) CPUs from Python Danny W. Adair 4/11/12 5:34 PM
On Thu, Apr 12, 2012 at 11:11, Guy K. Kloss <guy....@aut.ac.nz> wrote:
>[...]
> Gladly, there's a command line tool "wmic" that can help. With the
> following command line I can get the number of physical CPU cores:

this command line tool seems to provide some more details
http://technet.microsoft.com/en-us/sysinternals/cc835722

Cheers,
Danny

Re: [nzpug] detecting number of (physical) CPUs from Python Guy 4/11/12 6:42 PM
On 12/04/12 12:34, Danny Adair wrote:
> this command line tool seems to provide some more details
> http://technet.microsoft.com/en-us/sysinternals/cc835722

Interesting! This would then probably be my solution of choice if it
wasn't for the fact that this tool first needs prior installation on the
designated machines. So not very good for a roll-out to 100s of
machines. I guess I'm gonna stick to using the rather uglier wmic ...
after all, for this particular purpose it just does the trick.

Guy

--
Guy K. Kloss
School of Computing + Mathematical Sciences
Auckland University of Technology
Private Bag 92006, Auckland 1142
phone: +64 9 921 9999 ext. 5032
eMail: Guy....@aut.ac.nz

Re: [nzpug] detecting number of (physical) CPUs from Python tiff 4/12/12 4:28 PM
Hi,
We've been down this route before, started using the wmi module, then wmi command line and have now coded all the required windows calls into a C DLL and then call that using the ctypes module.

Here is some old test python code that may help:
-----------------------------------------------------------------------------------------------

import wmi
c = wmi.WMI (find_classes=False)

for cpu in c.Win32_Processor():
    print '------ Details for %s -------' % cpu.DeviceID
    print 'Name:              %s' % cpu.Name
    print 'Description:       %s' % cpu.Description
    print 'Number of cores:   %d' % cpu.NumberOfCores
    print 'CurrentClockSpeed: %d' % cpu.CurrentClockSpeed
    print 'L2CacheSize:       %d' % cpu.L2CacheSize
    print 'LoadPercentage:    %d' % cpu.LoadPercentage
    print 'Manufacturer:      %s' % cpu.Manufacturer
    print 'MaxClockSpeed:     %d' % cpu.MaxClockSpeed
----------------------------------------------------------------------------------------------- 
Checkout the M/S page for details on Win32_Processor: 

Best Regards
Steve Hutchins
Aptelisense.com
Re: [nzpug] detecting number of (physical) CPUs from Python jbrendel 4/12/12 5:14 PM

This whole discussion makes me wonder if it's possible to determine the
number of physical CPUs in a completely platform independent, empirical
manner. Basically, a small utility that - no matter whether you run it
on Linux, BSD, Windows, Solaris, etc. - would come back with the number
of physical CPU cores.

For example, using processing, you could start a single number crunching
task, measure how long it takes to complete (let's say 10 seconds). Then
you start more tasks in parallel: 2, 4, 8 and so on. As long as there
are enough physical CPU cores for all the tasks, you should still get
results in about 10 seconds. The moment you start seeing delays, you
probably will have reached the limit and end up running more than one of
those tasks on the same CPU core.

Obviously, this doesn't work well if the machine is already busy with
other tasks. You probably want to take several measurements to get a
decent read, and so on.

But, just theoretically, I wonder if that would work.

Juergen

Re: [nzpug] detecting number of (physical) CPUs from Python donal.m...@gmail.com 4/12/12 5:35 PM
It's an interesting idea, but I think the set of conditions in which it would be effective are so limiting that it would seldom be used in practice. If it were to be used in ignorance of the caveats you mention, it would give confusing and misleading results, so to the uninformed casual user it would be worse than having no information at all.

It also overlooks the fact that different schedulers can be in operation in different OSes at different times. Even on an unburdened system, a hypothetical scheduler may choose to keep your processes adjacent to the cache on a single physical CPU for longer than you would expect.

Another point of interest is the notion of tools like taskset on Linux, which limit the subset of CPUs that your process can be scheduled to run on. It's interesting to know that your process is running on a 64-core box, but if you're spawning subprocesses (for example) it would also be interesting to know that you and your children will be limited to running on only one of those CPUs.

Ideally it would be possible to query the OS about the resources that are available to your process in a portable way, and the OS would notify the process when the available resources change (also in a portable way), perhaps with a signal.

Just braindumping

D




--
You received this message because you are subscribed to the Google Groups "New Zealand Python User Group" group.
To post to this group, send email to nz...@googlegroups.com.
To unsubscribe from this group, send email to nzpug+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/nzpug?hl=en.


Re: [nzpug] detecting number of (physical) CPUs from Python jbrendel 4/12/12 6:15 PM

Yes, it would be much nicer if there would be a decent implementation
for this in Python's os module. Certainly much nicer than just playing
guessing games with timing CPU intensive tasks.

Juergen

>         +unsub...@googlegroups.com.


>         For more options, visit this group at
>         http://groups.google.com/group/nzpug?hl=en.
>        
>        
>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "New Zealand Python User Group" group.
> To post to this group, send email to nz...@googlegroups.com.
> To unsubscribe from this group, send email to nzpug
> +unsub...@googlegroups.com.

> For more options, visit this group at
> http://groups.google.com/group/nzpug?hl=en.


Re: [nzpug] detecting number of (physical) CPUs from Python Guy 4/13/12 7:21 PM
On 13/04/12 12:14, Juergen Brendel wrote:
> This whole discussion makes me wonder if it's possible to determine the
> number of physical CPUs in a completely platform independent, empirical
> manner. Basically, a small utility that - no matter whether you run it
> on Linux, BSD, Windows, Solaris, etc. - would come back with the number
> of physical CPU cores.

Oh, yes, how I'd like that. In a completely platform independent way
would be sweet. But I'm not too sure whether that should be an empirical
way. There are two big issues:

 * The machine might be busy with (a whole bunch of) other tasks while
   doing so: If one process e. g. configures itself at start time, and
   it is started every time at boot, the machine is/can be quite busy
   starting all kinds of other services/daemons.

 * Such tasks should be *quick* to look up, so running a dummy benchmark
   with n concurrent workers to time the performance for a variety of n
   will just take a bit too long. In your example below for a quad core
   one would have to run already four sets of timing loops (with 1, 2,
   4 and 8 parallel processes, as threads won't do), and then do them
   each for a significant time (e. g. 10 secs, maybe multiply). That'll
   lead to something in the order of magnitude of a minute until such a
   call/test returns a suitable result.

> For example, using processing, you could start a single number crunching
> task, measure how long it takes to complete (let's say 10 seconds). Then
> you start more tasks in parallel: 2, 4, 8 and so on. As long as there
> are enough physical CPU cores for all the tasks, you should still get
> results in about 10 seconds. The moment you start seeing delays, you
> probably will have reached the limit and end up running more than one of
> those tasks on the same CPU core.
>
> Obviously, this doesn't work well if the machine is already busy with
> other tasks. You probably want to take several measurements to get a
> decent read, and so on.
>
> But, just theoretically, I wonder if that would work.

Guy

--
Guy K. Kloss
School of Computing + Mathematical Sciences
Auckland University of Technology
Private Bag 92006, Auckland 1142
phone: +64 9 921 9999 ext. 5032
eMail: Guy....@aut.ac.nz