Questions on 64 bit versions of Python
pyt...@bdurham.com
are 30G in size. As part of this processing I may need to create some
very large dictionary structures. I will be running my scripts on a
version of Windows 2003 Server Enterprise Edition that supports 16G of
RAM. Yes, I could use a database in place of dictionaries, but I'm
looking for maximum performance.
The following page lists two 64 bit versions of Python for Windows:
http://www.python.org/download/releases/2.5.2/
For Win64-Itanium users: python-2.5.2.ia64.msi
For Win64-AMD64 users: python-2.5.2.amd64.msi
1. It looks like the 64 bit versions of Python for Windows are CPU
vendor specific, eg. it doesn't look like there's a single, universal
executable for Windows 64 bit platforms. Is this true?
2. Are there limitations to the using the 64 bit versions of Python? I
seem to remember reading that many 3rd party modules (especially Windows
OS specific modules) may not be compatible with the 64 bit versions of
Python for Windows.
3. If I wanted to run a 64 bit version of Python under Linux, would I
need to recompile from source on 64 bit version of Linux or do 64 bit
versions of Linux automatically ship with 64 bit versions of Python?
(Any recommendations on a flavor of 64 bit of Linux for the Intel
architecture would be appreciated)
4. Is there a stable version of IronPython compiled under a 64 bit
version of .NET? Anyone have experience with such a beast?
Thank you,
Malcolm
M.-A. Lemburg
> Background: I'm going to be processing some raw transaction logs that
> are 30G in size. As part of this processing I may need to create some
> very large dictionary structures. I will be running my scripts on a
> version of Windows 2003 Server Enterprise Edition that supports 16G of
> RAM. Yes, I could use a database in place of dictionaries, but I'm
> looking for maximum performance.
>
> The following page lists two 64 bit versions of Python for Windows:
> http://www.python.org/download/releases/2.5.2/
>
> For Win64-Itanium users: python-2.5.2.ia64.msi
> For Win64-AMD64 users: python-2.5.2.amd64.msi
>
> 1. It looks like the 64 bit versions of Python for Windows are CPU
> vendor specific, eg. it doesn't look like there's a single, universal
> executable for Windows 64 bit platforms. Is this true?
Yes.
But then Intel Itanium is being phased out anyway and the AMD64
build works on both Intel and AMD processors.
> 2. Are there limitations to the using the 64 bit versions of Python? I
> seem to remember reading that many 3rd party modules (especially Windows
> OS specific modules) may not be compatible with the 64 bit versions of
> Python for Windows.
True.
The extensions usually need to be recompiled and possibly
also updated to support 64-bit architectures.
The win32 tools are available on 64-bit Windows - if that's what
you mean with "Windows OS specific modules".
> 3. If I wanted to run a 64 bit version of Python under Linux, would I
> need to recompile from source on 64 bit version of Linux or do 64 bit
> versions of Linux automatically ship with 64 bit versions of Python?
> (Any recommendations on a flavor of 64 bit of Linux for the Intel
> architecture would be appreciated)
Yes, 64-bit Linux versions ship with 64-bit builds of Python. Any
recent Linux distro should work just fine.
> 4. Is there a stable version of IronPython compiled under a 64 bit
> version of .NET? Anyone have experience with such a beast?
Can't comment on that one.
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, Jul 25 2008)
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________
:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
Fredrik Lundh
>> 4. Is there a stable version of IronPython compiled under a 64 bit
>> version of .NET? Anyone have experience with such a beast?
>
> Can't comment on that one.
Should that matter? Isn't IronPython pure CLR?
</F>
Mike Driscoll
IronPython is written in C# and runs in/with the CLR, if that's what
you mean. Well, IronPython one works with the CLR and is equivalent to
Python 2.4, whereas IronPython 2 works with the DLR and is equivalent
to Python 2.5
Mike
Dino Viehland
"Martin v. Löwis"
> 32-bit process and on a 64-bit machine it runs in a 64-bit process.
That's probably not exactly true (although I haven't checked).
When you start a .NET .exe program, the operating system needs to decide
whether to create a 32-bit or a 64-bit process (assuming the processor
supports 64-bit mode).
The Microsoft .NET commercial framework uses the PE architecture of the
executable to make that decision (or, rather, it doesn't decide at all,
but the underlying OS decides). The C# compiler (more specifically, the
assembly linker) offers a choice of setting the .NET architecture to
Itanium, AMD64, x86, or "any"; people use typically "any". This "any"
choice is implemented by setting the PE architecture to "any", and then
indicating to the .NET run-time that any other architecture would be
fine as well.
As a consequence, an architecture-any executable launches as a 32-bit
process on a 64-bit system. To have the executable launch as 64-bit
code, you must tell csc.exe to create an AMD64 binary (say), which
then means that the binary won't launch on a 32-bit system. I haven't
checked, but my guess is that IronPython uses architecture-any
executables (unless you adjust the build process).
Things are different in a DLL: an architecture-any DLL will load into
a 32-bit process as 32-bit code, and into a 64-bit process as 64-bit
code.
Regards,
Martin
"Martin v. Löwis"
> architecture would be appreciated)
My recommendation is to use Debian or Ubuntu, as that's my personal
preference. As MAL said, any recent distribution that supports AMD64
should be fine (assuming you are not interested in Itanium, SPARC64,
HPPA (2.0), PPC64, or Alpha, which are all 64-bit flavors of Linux).
Regards,
Martin
"Martin v. Löwis"
> architecture would be appreciated)
My recommendation is to use Debian or Ubuntu, as that's my personal
Rob Williscroft
comp.lang.python:
>> The end result of that is on a 32-bit machine IronPython runs in a
>> 32-bit process and on a 64-bit machine it runs in a 64-bit process.
>
>
> That's probably not exactly true (although I haven't checked).
>
> When you start a .NET .exe program, the operating system needs to
> decide whether to create a 32-bit or a 64-bit process (assuming the
> processor supports 64-bit mode).
>
> The Microsoft .NET commercial framework uses the PE architecture of the
Whats the "Commercial framework" ? I've only come accross 3, the
standard 32 bit one and 2 64 bit variants.
> executable to make that decision (or, rather, it doesn't decide at all,
> but the underlying OS decides). The C# compiler (more specifically, the
> assembly linker) offers a choice of setting the .NET architecture to
> Itanium, AMD64, x86, or "any"; people use typically "any". This "any"
> choice is implemented by setting the PE architecture to "any", and then
> indicating to the .NET run-time that any other architecture would be
> fine as well.
>
> As a consequence, an architecture-any executable launches as a 32-bit
> process on a 64-bit system.
I just tested, I built a default C# forms app using the "AnyCPU" option
and it ran as a 64 bit app (no *32 in Task Manager), this is on XP64.
I have though installed the AMD64 version of the 2.0 framework and
AFAICT neither windows update or the Visual Studio installer
will install that by default, you have to go get it your self.
> To have the executable launch as 64-bit
> code, you must tell csc.exe to create an AMD64 binary (say), which
> then means that the binary won't launch on a 32-bit system. I haven't
> checked, but my guess is that IronPython uses architecture-any
> executables (unless you adjust the build process).
I just started ipy.exe the 1.1 and 2.0B1, both ran as 64 bit processes
IronPython 1.1 (1.1) on .NET 2.0.50727.1433
Copyright (c) Microsoft Corporation. All rights reserved.
>>>
I don't know what happens if you have both 32 bit and 64 bit versions
of the framwork installed (presumably with slightly different minor
version numbers) as I uninstalled the 32 bit version before I installed the
AMD64 version.
"Martin v. Löwis"
>
> Whats the "Commercial framework" ? I've only come accross 3, the
> standard 32 bit one and 2 64 bit variants.
That's the name of the Microsoft .NET product available for Windows.
There are other implementations as well, such as the Compact Framework,
or GNU Mono.
It's correct that the commercial framework is available for three
architectures. In addition, the compact framework is also available
for ARM. Mono is available for many more platforms, such as SPARC,
PowerPC, MIPS and HPPA.
> I just tested, I built a default C# forms app using the "AnyCPU" option
> and it ran as a 64 bit app (no *32 in Task Manager), this is on XP64.
>
> I have though installed the AMD64 version of the 2.0 framework and
> AFAICT neither windows update or the Visual Studio installer
> will install that by default, you have to go get it your self.
Interesting. I only tested this in .NET 1.1. Perhaps they have changed
something since. How exactly did you launch the program? Does it change
if you use Python's os.spawn* to launch it?
Regards,
Martin
Paul Boddie
>
> But then Intel Itanium is being phased out anyway
Citation needed! ;-)
Paul
Rob Williscroft
comp.lang.python:
>> I just tested, I built a default C# forms app using the "AnyCPU"
>> option and it ran as a 64 bit app (no *32 in Task Manager), this is
>> on XP64.
>>
>> I have though installed the AMD64 version of the 2.0 framework and
>> AFAICT neither windows update or the Visual Studio installer
>> will install that by default, you have to go get it your self.
>
> Interesting. I only tested this in .NET 1.1. Perhaps they have changed
> something since. How exactly did you launch the program? Does it change
> if you use Python's os.spawn* to launch it?
If subprocess.call will do then I can report the same results, a
64 bit process, using 32 bit CPython 2.5 to call it.
Also if the programme (C#) is built with the 3.5 framework I
get the same results, however interestingly there is no 64 bit
build for the 3.5 framework (or at least I was unable to find one).
Tim Roberts
>
>For Win64-Itanium users: python-2.5.2.ia64.msi
>For Win64-AMD64 users: python-2.5.2.amd64.msi
>
>1. It looks like the 64 bit versions of Python for Windows are CPU
>vendor specific, eg. it doesn't look like there's a single, universal
>executable for Windows 64 bit platforms. Is this true?
It's true for ALL operating systems, not just Windows. The ia64 (Itanium)
and the amd64 are completely separate processors with VERY different
instruction sets.
--
Tim Roberts, ti...@probo.com
Providenza & Boekelheide, Inc.
Roel Schroeven
> pyt...@bdurham.com wrote:
>> For Win64-Itanium users: python-2.5.2.ia64.msi
>> For Win64-AMD64 users: python-2.5.2.amd64.msi
>>
>> 1. It looks like the 64 bit versions of Python for Windows are CPU
>> vendor specific, eg. it doesn't look like there's a single, universal
>> executable for Windows 64 bit platforms. Is this true?
>
> It's true for ALL operating systems, not just Windows. The ia64 (Itanium)
> and the amd64 are completely separate processors with VERY different
> instruction sets.
Maybe I'm wrong, but I have the impression that Malcolm is a bit
confused about the different 64-bit processors from Intel and AMD. So,
to clear things up a bit:
- The Itanium (ia64) is a 64-bit processor made by Intel, mostly for use
in high-end systems (servers etc.) IIRC. It's instruction set is indeed
completely different from the i386 instruction set (though I think
there's an emulation mode)
- AMD64 (or x86-64 or x64 or EMT64 or Intel64) is a 64-bit instruction
set from AMD which is an extension to the i386 instruction set, and runs
32-bit (and 16-bit) i386-code natively. But, and this is important,
despite the name the instruction set is also used by Intel (though they
call it EMT64 and made a few minor changes).
If you have or buy a 64-bit computer, it almost certainly uses the AMD64
instruction set, even if it has an Intel CPU (if you have an Itanium or
other 64-bit CPU, I suppose you would know; it would not be something
you buy from the shelf in a regular computer store). So you almost
certainly need the AMD64 version of Python.
--
The saddest aspect of life right now is that science gathers knowledge
faster than society gathers wisdom.
-- Isaac Asimov
Roel Schroeven
"Martin v. Löwis"
> set from AMD which is an extension to the i386 instruction set, and runs
> 32-bit (and 16-bit) i386-code natively. But, and this is important,
> despite the name the instruction set is also used by Intel (though they
> call it EMT64 and made a few minor changes).
Indeed, there are (unfortunately) many names for the architecture.
Originally, AMD called it x86-64, and later renamed it to AMD64. Intel
originally implemented it under the name EM64T (for Extended Memory 64
Technology), and now calls the architecture Intel 64.
Microsoft (apparently not wanting to take sides) calls it x64.
I personally believe that whoever invents a technology also
gets to name it, so I encourage use of the name AMD gives it
(which is AMD64). That's why the Python installer has that label
in its name.
Regards,
Martin
William McBrine
> Originally, AMD called it x86-64, and later renamed it to AMD64. Intel
> originally implemented it under the name EM64T (for Extended Memory 64
> Technology), and now calls the architecture Intel 64.
I hadn't heard "Intel 64" before. That's a bit nervy, isn't it? Plus it
seems to conflict with their own use of "IA-64" (Intel Architecture 64)
for the Itanium (vs. "IA-32" for traditional x86).
--
09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0 -- pass it on
"Martin v. Löwis"
> seems to conflict with their own use of "IA-64" (Intel Architecture 64)
> for the Itanium (vs. "IA-32" for traditional x86).
Indeed. Microsoft Installer has an architecture string for the MSI file;
"Intel64" there means Itanium (and AMD64 can be denoted by either
"AMD64" or "x64"). This was from a time when Intel still told people
that the future of 64-bit computing is in Itanium (and architecturally,
I still think this would have been the right choice).
Regards,
Martin
pyt...@bdurham.com
Thanks for everyone's feedback - excellent detail - all my questions
have been answered.
BTW: Roel was correct that I got confused over the AMD and Intel naming
conventions regarding the 64 bit versions of Python for Windows. (I
missed that nuance that the Intel build refered to the Itanium vs. the
standard off-the-rack 64 bit version of Intel's 586/686 CPU)
Best regards,
Malcolm
Tim Golden
Frankly, I'm very glad you did get confused: I've learnt more
about the various 64-bit architectures in this thread than
I ever thought I'd need to know :)
TJG
Fuzzyman
> Martin v. Löwis wrote innews:488B570C...@v.loewis.dein
.NET 3.5 is a set of extension assemblies and does not change the
underlying runtime - so there is probably no need for a separate 63
bit build for .NET 3.5.
Michael Foord
--
http://www.ironpythoninaction.com/