Building Python on Cray T3E
Mark Hadfield
T3E. My first attempt with 2.2.1 failed so I searched Google for "Cray
T3E group:comp.lang.python" and found several threads, including a
couple from mid-2001 on problems with Unicode. Following the advice in
this message...
http://groups.google.co.nz/groups?hl=en&selm=j4lmkqdi7d.fsf%40informatik.hu-
berlin.de
...I ran configure with "--enable-unicode=ucs4". But make stops with
the following error:
cc -DNDEBUG -O -I. -I./Include -DHAVE_CONFIG_H -c ./Modules/_sre.c
-o Modules/_sre.o
CC-147 cc: ERROR File = ./Modules/_sre.c, Line = 1791
Declaration is incompatible with "int join(char *, char *)"
(declared at line 300 of "/usr/include/unistd.h").
join(PyObject* list, PyObject* pattern)
^
Total errors detected in ./Modules/_sre.c: 1
I'm out of my depth here. Any suggestions?
--
Mark Hadfield "Ka puwaha et tai nei, Hoea tatou"
m.had...@niwa.co.nz
National Institute for Water and Atmospheric Research (NIWA)
Andrew Dalke
>...I ran configure with "--enable-unicode=ucs4". But make stops with
>the following error:
> CC-147 cc: ERROR File = ./Modules/_sre.c, Line = 1791
> Declaration is incompatible with "int join(char *, char *)"
> (declared at line 300 of "/usr/include/unistd.h").
>
> join(PyObject* list, PyObject* pattern)
> ^
>
> Total errors detected in ./Modules/_sre.c: 1
Not worked on a T3E, but what this is saying is that 'unistd.h'
declares a functions called 'join' and _sre.c defines a function
called 'join'.
_sre's join is only used in that file and it's static, so you can
rename it easily if you want. Change _sre.c (line 1791 in 2.2.1)
so the function "join" is now "srejoin" as in
static PyObject*
join(PyObject* list, PyObject* pattern)
{
/* join list elements */
--to--
static PyObject*
srejoin(PyObject* list, PyObject* pattern)
{
/* join list elements */
This function is called from one place (that I can find) so you'll
also need to change line 2244 the same way, from
/* convert list to single string (also removes list) */
item = join(list, self->pattern);
--to--
/* convert list to single string (also removes list) */
item = srejoin(list, self->pattern);
Andrew
da...@dalkescientific.com
Skip Montanaro
Mark> CC-147 cc: ERROR File = ./Modules/_sre.c, Line = 1791
Mark> Declaration is incompatible with "int join(char *, char *)"
Mark> (declared at line 300 of "/usr/include/unistd.h").
Mark> join(PyObject* list, PyObject* pattern)
Looks like _sre.c defines a static function named "join" that conflicts with
something in the Cray's standard library. Try applying this patch. It
should get you past this particular problem.
--
Skip Montanaro (sk...@pobox.com - http://www.mojam.com/)
"Excellant Written and Communications Skills [required]" - post to chi.jobs
*** /home/skip/tmp/_sre.c.~2.79~ Thu May 9 23:50:23 2002
--- /home/skip/tmp/_sre.c Thu May 9 23:50:23 2002
***************
*** 1788,1794 ****
#endif
static PyObject*
! join(PyObject* list, PyObject* pattern)
{
/* join list elements */
--- 1788,1794 ----
#endif
static PyObject*
! join_list(PyObject* list, PyObject* pattern)
{
/* join list elements */
***************
*** 2241,2247 ****
Py_DECREF(filter);
/* convert list to single string (also removes list) */
! item = join(list, self->pattern);
if (!item)
return NULL;
--- 2241,2247 ----
Py_DECREF(filter);
/* convert list to single string (also removes list) */
! item = join_list(list, self->pattern);
if (!item)
return NULL;
Michael Hudson
> [Mark Hadfield]
> > Like several others before me I am trying to build Python on a Cray
> > T3E.
[...]
> There's a join() in unistd.h on UNICOS which collides with a internal
> SRE function. Try this simple patch:
So does the resulting Python actually work after this? We got this
far with 2.2.1c1, but you never told me if the dang thing worked or
not...
Cheers,
M.
--
ROOSTA: Ever since you arrived on this planet last night you've
been going round telling people that you're Zaphod
Beeblebrox, but that they're not to tell anyone else.
-- The Hitch-Hikers Guide to the Galaxy, Episode 7
Kalle Svensson
> Kalle Svensson <ka...@lysator.liu.se> writes:
>
> > [Mark Hadfield]
> > > Like several others before me I am trying to build Python on a Cray
> > > T3E.
> [...]
> > There's a join() in unistd.h on UNICOS which collides with a internal
> > SRE function. Try this simple patch:
>
> So does the resulting Python actually work after this? We got this
> far with 2.2.1c1, but you never told me if the dang thing worked or
> not...
Nope, not yet. First I had to disable shared libraries. After that,
I've had some linking trouble. I'm getting close now, though.
Basically, I have a running Python without any useful modules.
By the way, I'm on a Cray J90se with UNICOS 10.0.0.3. I have no idea
how (dis)similar this is to a T3E. Anyone?
Peace,
Kalle
--
Kalle Svensson, http://www.juckapan.org/~kalle/
Student, root and saint in the Church of Emacs.
Holger Berger
wrote:
>
> Nope, not yet. First I had to disable shared libraries. After that,
> I've had some linking trouble. I'm getting close now, though.
> Basically, I have a running Python without any useful modules.
>
> By the way, I'm on a Cray J90se with UNICOS 10.0.0.3. I have no idea
> how (dis)similar this is to a T3E. Anyone?
>
A bit of topic...
When reading about Python on 'uncommon plattforms':
python2.2 works fine on NEC SX vector-series running SUPER-UX.
Tested on SX-5 and SX-6, with SUPER-UX 11.0 and 12.0.
Also without shared libraries, with lots of code
built into the interpreter.
regards
Holger
Mark Hadfield
news:abfjuu$djs$1...@slb4.atl.mindspring.net...
> _sre's join is only used in that file and it's static, so you can
> rename it easily if you want. Change _sre.c (line 1791 in 2.2.1)
> so the function "join" is now "srejoin" ...
Thanks for that, Andrew. (Thanks also to Skip Montanaro, who replied to my
post by email.)
The change to _sre.c worked and I now have a minimal Python installation
working (though I haven't been brave enough to try "make test"). It's
minimal at the moment because "make" tries to build extensions dynamically,
and Cray's cc compiler doesn't support dynamic linking. (At least I don't
think it does--it certainly doesn't work the way it's being called here.) So
I'm working through Modules/Setup, seeing what can and what can't be built
and linked statically.
OK, so once I have something working, how can I get my findings into the
installation notes and/or build process? Eg. can the configure script be
persuaded to turn off static linking on this platform? How do I get the _sre
patch into the source?
Michael Hudson
> OK, so once I have something working, how can I get my findings into
> the installation notes and/or build process?
Submit patches to sf, most likely.
I've been occasionally to-and-froing with Kalle Svensson about Cray
support, but as he's not sure he's actually got a working Python yet
I've been reluctant to make changes to the core.
> Eg. can the configure script be persuaded to turn off static linking
> on this platform?
I don't think so. This is arguably a limitation of the current build
procedure, but hacking the build system is such a vile job that
noone's too keen to do it. I'm certainly not.
> How do I get the _sre patch into the source?
SF again, most likely. I'd just check it in, but /F owns the sre
code, so it would probably be best to run any changes past him first.
Cheers,
M.
--
Considering that this thread is completely on-topic in the way only
c.l.py threads can be, I think I can say that you should replace
"Oblivion" with "Gravity", and increase your Radiohead quotient.
-- Ben Wolfson, comp.lang.python
Mark Hadfield
news:lkhelg7...@pc150.maths.bris.ac.uk...
> So does the resulting Python actually work after this? We got this
> far with 2.2.1c1, but you never told me if the dang thing worked or
> not...
I'm not sure who this question was directed to, but here are the
results of my attempts to build Python 2.2.1 on a Cray T3E over the
last few days. The summary is: it can be built and installed and it
runs several simple scripts, but several modules are broken to a
greater or lesser extent. It's not yet clear to me how useful it will
be or how much work is required to fix things.
The procedure is the usual "configure; make; make install" with the
following variations and notes:
- The compiler (found automatically by the configure script) is
Cray's cc. We don't have gcc on the NIWA machine and a note from
Konrad Hinsen in README warns against using it anyway.
- As has been discussed before on this group, Python's unicode
module cannot be built because the Cray compiler lacks a 16-bit
character datatype (something like that--don't trust me on the
details here). So we disable it with "configure
--enable-unicode=ucs4".
- As noted earlier in this thread, there is a C-function name clash
in the _sre module, which can be easily fixed. However, recall that
sre is one of two alternative regular-expression engines for the re
module, the other engine being pre. It turns out that sre does not
pass its tests so the path of least resistance is is to modify
Lib/re.py to use the pre engine (the change is trivial) and comment
out the _sre entry in Modules/Setup.
- The Cray compiler does not support dynamic linking, so extensions
must be explicitly listed in Modules/Setup. Here is the list of
extensions I have built & linked successfully:
posix posixmodule.c
new newmodule.c
array arraymodule.c
cmath cmathmodule.c
math mathmodule.c
struct structmodule.c
time timemodule.c
operator operator.c
_weakref _weakref.c
_codecs _codecsmodule.c
_testcapi _testcapimodule.c
_symtable symtablemodule.c
strop stropmodule.c
regex regexmodule.c regexpr.c
pcre pcremodule.c pypcre.c
_locale _localemodule.c
fcntl fcntlmodule.c
pwd pwdmodule.c
grp grpmodule.c
errno errnomodule.c
select selectmodule.c
xreadlines xreadlinesmodule.c
_socket socketmodule.c
crypt cryptmodule.c
termios termios.c
timing timingmodule.c
rotor rotormodule.c
syslog syslogmodule.c
dbm dbmmodule.c
binascii binascii.c
parser parsermodule.c
cStringIO cStringIO.c
cPickle cPickle.c
- With the above modules included, Python is built
successfully. There are quite a few warnings, which I haven't
checked out yet. I think that some of them come home to roost in the
tests.
- After building Python, make runs setup.py, which attempts (among
other things) to build extensions not already built & linked
statically. This is a futile but harmless exercise; the inevitable
failures do not cause make to abort. I have seen elsewhere (in
connection with Cygwin Python) a reference to a "--disable-shared"
option to configure that I presume would inhibit this. However this
is ignored by configure in Python 2.2.1.
- Testing is a somewhat depressing experience. On the NIWA machine,
a simple "make test" command fails to complete after 12 hours and
thrashes the swap space, with the side effect of inhibiting all
network access for 10 seconds out of every 30. (They don't call
these things supercomputers for nothing. I bet you can't do *that*
on your $1000 Linux box!) By running "make test" for a while then
completing the tests one at a time, I have established that the
following tests fail:
test test_al skipped -- No module named al
test test_audioop skipped -- No module named audioop
test test_binhex failed -- CRC error
test test_bsddb skipped -- No module named bsddb
test test_cd skipped -- No module named cd
test test_cl skipped -- No module named cl
test test_curses skipped -- No module named _curses
test test_descr skipped -- No module named xxsubtype
test test_dl skipped -- No module named dl
test test_fcntl crashed -- IO Error
test test_fpformat failed -- AssertionError: '-0' != '0'
test test_hmac failed -- No module named md5
test test_hotshot skipped -- No module named _hotshot
test test_imageop skipped -- No module named imageop
test test_imgfile skipped -- No module named imgfile
test test_linuxaudiodev skipped -- No module named linuxaudiodev
test test_locale skipped -- test locale en_US not supported
test test_longexp crashed -- exceptions.MemoryError:
test test_md5 skipped -- No module named md5
test test_minidom skipped -- No module named pyexpat
test test_mmap skipped -- No module named mmap
test test_nis skipped -- No module named nis
test test_ntpath skipped -- No module named nt
test test_openpty skipped -- No openpty() available.
test test_pep247 skipped -- No module named md5
test test_poll skipped -- select.poll not defined
test test_pty skipped -- Pseudo-terminals (seemingly) not
functional.
test test_pyexpat skipped -- No module named pyexpat
test test_re crashed -- exceptions.TypeError: 'NoneType' object is
not callable
test test_rgbimg skipped -- No module named rgbimg
test test_rotor produced unexpected output:
test test_sax skipped -- no XML parsers available
test test_sha skipped -- No module named sha
test test_socket crashed -- socket.error: unknown address family
test test_socket_ssl skipped -- Use of the `network' resource
not enabled
test test_socketserver skipped -- Use of the `network' resource
not enabled
test test_sre crashed -- exceptions.UnicodeError: \N escapes not
supported (can't load unicodedata module)
test test_sunaudiodev skipped -- No module named sunaudiodev
test test_sundry skipped -- No module named md5
test test_ucn skipped -- No module named unicodedata
test test_unicode crashed -- exceptions.UnicodeError: \N escapes
not supported (can't load unicodedata module)
test test_unicode_file skipped -- No Unicode filesystem semantics
on this platform.
test test_unicodedata skipped -- No module named sha
test test_urllib2 skipped -- No module named md5
test test_winreg skipped -- No module named winreg
test test_winsound skipped -- No module named winsound
test test_zipfile skipped -- No module named zlib
test test_zlib skipped -- No module named zlib
Not all of the test failures are of any significance, of course. (I
didn't really expect Windows registry operations to work.) But the
failure in test_re is a little worrying...
My immediate reason for installing Python on the Cray is that I want
to check out SCons, but it *must* work on the Cray to be of any use to
me. At the moment SCons won't work because it requires md5, which
couldn't be compiled because it requires a UINT4 data type. So I guess
I'll either have to get md5 working or persuade SCons to work without
md5 checksums. Oh well...
Michael Hudson
> "Michael Hudson" <m...@python.net> wrote in message
> news:lkhelg7...@pc150.maths.bris.ac.uk...
>
> > So does the resulting Python actually work after this? We got this
> > far with 2.2.1c1, but you never told me if the dang thing worked or
> > not...
>
> I'm not sure who this question was directed to,
Kalle Svensson. He tried to build on a Cray (a J90, I think) when
2.2.1c1 came out, but didn't have the time to finish the job off.
> but here are the results of my attempts to build Python 2.2.1 on a
> Cray T3E over the last few days. The summary is: it can be built and
> installed and it runs several simple scripts, but several modules
> are broken to a greater or lesser extent. It's not yet clear to me
> how useful it will be or how much work is required to fix things.
That's cool.
> The procedure is the usual "configure; make; make install" with the
> following variations and notes:
>
> - The compiler (found automatically by the configure script) is
> Cray's cc. We don't have gcc on the NIWA machine and a note from
> Konrad Hinsen in README warns against using it anyway.
That note is probably pretty old; I don't know if it's still relavent.
> - As has been discussed before on this group, Python's unicode
> module cannot be built because the Cray compiler lacks a 16-bit
> character datatype (something like that--don't trust me on the
> details here). So we disable it with "configure
> --enable-unicode=ucs4".
That doesn't actually disable unicode (you want --disable-unicode for
that). *Lots* of tests fail in --disable-unicode builds, mostly for
very shallow reasons (u'' literals not being supported for the most
part).
> - As noted earlier in this thread, there is a C-function name clash
> in the _sre module, which can be easily fixed. However, recall that
> sre is one of two alternative regular-expression engines for the re
> module, the other engine being pre. It turns out that sre does not
> pass its tests so the path of least resistance is is to modify
> Lib/re.py to use the pre engine (the change is trivial) and comment
> out the _sre entry in Modules/Setup.
How does it fail? Is it the "NoneType not callable" failure below.
That's an odd one.
That's a fairly impressive list.
Did the unicodedata module fail to build, or did you not try it?
> - With the above modules included, Python is built
> successfully. There are quite a few warnings, which I haven't
> checked out yet. I think that some of them come home to roost in the
> tests.
Quite likely. Though Python is more-or-less 64 bit clean (runs on
alphas, for isntance). Having 64-bit chars is probably a surprise to
some of the code, though...
> - After building Python, make runs setup.py, which attempts (among
> other things) to build extensions not already built & linked
> statically. This is a futile but harmless exercise; the inevitable
> failures do not cause make to abort. I have seen elsewhere (in
> connection with Cygwin Python) a reference to a "--disable-shared"
> option to configure that I presume would inhibit this. However this
> is ignored by configure in Python 2.2.1.
You can get it to not do this by just executing "make python" rather
than "make all" or "make". --disable-shared is a red herring in this
context, I think (even in 2.3 where it exists).
> - Testing is a somewhat depressing experience. On the NIWA machine,
> a simple "make test" command fails to complete after 12 hours and
> thrashes the swap space, with the side effect of inhibiting all
> network access for 10 seconds out of every 30.
! Is this test_longexp, by any chance?
> (They don't call these things supercomputers for nothing. I bet
> you can't do *that* on your $1000 Linux box!) By running "make
> test" for a while then completing the tests one at a time, I have
> established that the following tests fail:
Most of the skips are harmless.
> test test_al skipped -- No module named al
> test test_audioop skipped -- No module named audioop
> test test_binhex failed -- CRC error
> test test_bsddb skipped -- No module named bsddb
> test test_cd skipped -- No module named cd
> test test_cl skipped -- No module named cl
> test test_curses skipped -- No module named _curses
> test test_descr skipped -- No module named xxsubtype
> test test_dl skipped -- No module named dl
> test test_fcntl crashed -- IO Error
Hmm.
> test test_fpformat failed -- AssertionError: '-0' != '0'
Do Crays still have eccentric fp behaviour?
> test test_hmac failed -- No module named md5
> test test_hotshot skipped -- No module named _hotshot
> test test_imageop skipped -- No module named imageop
> test test_imgfile skipped -- No module named imgfile
> test test_linuxaudiodev skipped -- No module named linuxaudiodev
> test test_locale skipped -- test locale en_US not supported
> test test_longexp crashed -- exceptions.MemoryError:
Oh, so this isn't the one that thrashes for days...
I could imagine that this test would consume serious memory on a
cray...
> test test_md5 skipped -- No module named md5
> test test_minidom skipped -- No module named pyexpat
> test test_mmap skipped -- No module named mmap
> test test_nis skipped -- No module named nis
> test test_ntpath skipped -- No module named nt
> test test_openpty skipped -- No openpty() available.
> test test_pep247 skipped -- No module named md5
> test test_poll skipped -- select.poll not defined
> test test_pty skipped -- Pseudo-terminals (seemingly) not
> functional.
> test test_pyexpat skipped -- No module named pyexpat
> test test_re crashed -- exceptions.TypeError: 'NoneType' object is
> not callable
This is just wierd. Can you dig a little?
> test test_rgbimg skipped -- No module named rgbimg
> test test_rotor produced unexpected output:
> test test_sax skipped -- no XML parsers available
> test test_sha skipped -- No module named sha
> test test_socket crashed -- socket.error: unknown address family
> test test_socket_ssl skipped -- Use of the `network' resource
> not enabled
> test test_socketserver skipped -- Use of the `network' resource
> not enabled
> test test_sre crashed -- exceptions.UnicodeError: \N escapes not
> supported (can't load unicodedata module)
This is probably shallow. You could try hacking the unicode bits of
the test out.
> test test_sunaudiodev skipped -- No module named sunaudiodev
> test test_sundry skipped -- No module named md5
> test test_ucn skipped -- No module named unicodedata
> test test_unicode crashed -- exceptions.UnicodeError: \N escapes
> not supported (can't load unicodedata module)
> test test_unicode_file skipped -- No Unicode filesystem semantics
> on this platform.
> test test_unicodedata skipped -- No module named sha
> test test_urllib2 skipped -- No module named md5
> test test_winreg skipped -- No module named winreg
> test test_winsound skipped -- No module named winsound
> test test_zipfile skipped -- No module named zlib
> test test_zlib skipped -- No module named zlib
>
> Not all of the test failures are of any significance, of course. (I
> didn't really expect Windows registry operations to work.) But the
> failure in test_re is a little worrying...
>
> My immediate reason for installing Python on the Cray is that I want
> to check out SCons, but it *must* work on the Cray to be of any use to
> me. At the moment SCons won't work because it requires md5, which
> couldn't be compiled because it requires a UINT4 data type. So I guess
> I'll either have to get md5 working or persuade SCons to work without
> md5 checksums. Oh well...
Yeah, the md5 code that comes with Python is fairly 32 bit specific.
However, if you can find some md5-for-cray code in the wild, it's
probably not too much effort to hook it up to Python.
Cheers,
M.
--
Very clever implementation techniques are required to implement this
insanity correctly and usefully, not to mention that code written
with this feature used and abused east and west is exceptionally
exciting to debug. -- Erik Naggum on Algol-style "call-by-name"
Kalle Svensson
> "Mark Hadfield" <m.had...@niwa.co.nz> writes:
...
> > - The Cray compiler does not support dynamic linking, so extensions
> > must be explicitly listed in Modules/Setup. Here is the list of
> > extensions I have built & linked successfully:
Good to have, thanks. I'll check my Modules/Setup, I had some linking
problems.
> That's a fairly impressive list.
Indeed. Much better than I expected from UNICOS.
> Did the unicodedata module fail to build, or did you not try it?
It failed for me. I'll check out why.
...
> Yeah, the md5 code that comes with Python is fairly 32 bit specific.
> However, if you can find some md5-for-cray code in the wild, it's
> probably not too much effort to hook it up to Python.
I'll put this on my TODO as well. With an exam period coming up, I
should have some free time. <wink>
Med utmärkt högaktning,
Mark Hadfield
> "Mark Hadfield" <m.had...@niwa.co.nz> writes:
>> - As noted earlier in this thread, there is a C-function name
>> clash in the _sre module, which can be easily fixed. However,
>> recall that sre is one of two alternative regular-expression
>> engines for the re module, the other engine being pre. It turns
>> out that sre does not pass its tests so the path of least
>> resistance is is to modify Lib/re.py to use the pre engine (the
>> change is trivial) and comment out the _sre entry in
>> Modules/Setup.
>
> How does it fail? Is it the "NoneType not callable" failure below.
Oops. The test_sre failure occurred because it couldn't load the
unicodedata module. I thought I had tried and failed to build
unicodedata, but in fact I hadn't tried. I did try, it builds OK, and
test_re succeeds. So cancel my suggestion above. DO patch & build
_sre.c, DON'T modify Lib/re.py. DO make sure you build unicodedata.c.
The "NoneType not callable" failure comes from pre.
>> test test_re crashed -- exceptions.TypeError: 'NoneType' object
>> is not callable
>
> This is just weird. Can you dig a little?
OK, just a little so far. It's not Cray-related. It occurs on other
platforms if you modify re.py to select the pre engine. I've seen the
same exception elsewhere when a function argument is missing. So I'm
guessing that the problem is that pre does not present exactly the
same interface as sre.
It's a moot point for me now that I have sre working, but is the pre
module still supported? If so should I enter this failure as a bug in
pre?
>> - After building Python, make runs setup.py, which attempts
>> (among other things) to build extensions not already built &
>> linked statically. This is a futile but harmless exercise; the
>> inevitable failures do not cause make to abort. I have seen
>> elsewhere (in connection with Cygwin Python) a reference to a
>> "--disable-shared" option to configure that I presume would
>> inhibit this. However this is ignored by configure in Python
>> 2.2.1.
>
> You can get it to not do this by just executing "make python" rather
> than "make all" or "make". --disable-shared is a red herring in
> this context, I think (even in 2.3 where it exists).
Executing "make python" does indeed bypass attempts to build extension
modules for dynamic linking, but "make install" and "make test"
don't. My ad-hoc solution is to change the following in setup.py
(class PyBuildExt(build_ext), function build_extensions(self)):
# Remove modules that are present on the disabled list
self.extensions = [ext for ext in self.extensions
if ext.name not in disabled_module_list]
to this:
self.extensions = []
>> - Testing is a somewhat depressing experience. On the NIWA
>> machine, a simple "make test" command fails to complete after 12
>> hours and thrashes the swap space, with the side effect of
>> inhibiting all network access for 10 seconds out of every 30.
>
> ! Is this test_longexp, by any chance?
Not specifically that one. I think the problem is that running all the
tests in sequence without reloading the Python interpreter demands a
lot of memory. On our Compaq Unix machine it takes 67 MB and on the
Cray it takes 128 MB. This doesn't sound like an excessive amount but
Cray T3E's are not very smart about swapping and we seem to have some
sort of load-balancing problem that causes the single-CPU jobs to
flock to one processor.
>> My immediate reason for installing Python on the Cray is that I
>> want to check out SCons, but it *must* work on the Cray to be of
>> any use to me. At the moment SCons won't work because it requires
>> md5, which couldn't be compiled because it requires a UINT4 data
>> type. So I guess I'll either have to get md5 working or persuade
>> SCons to work without md5 checksums. Oh well...
>
> Yeah, the md5 code that comes with Python is fairly 32 bit specific.
> However, if you can find some md5-for-cray code in the wild, it's
> probably not too much effort to hook it up to Python.
I'll look into that. Alternatively it should be possible to persuade
SCons to use its TimeStamp module instead of its MD5 module for
file-change detection. But I haven't quite managed to get that going.
Still, it's all great fun!
---
Michael Hudson
[...]
> Oops. The test_sre failure occurred because it couldn't load the
> unicodedata module. I thought I had tried and failed to build
> unicodedata, but in fact I hadn't tried. I did try, it builds OK, and
> test_re succeeds. So cancel my suggestion above. DO patch & build
> _sre.c, DON'T modify Lib/re.py. DO make sure you build unicodedata.c.
Excellent!
[...]
> >> test test_re crashed -- exceptions.TypeError: 'NoneType' object
> >> is not callable
> >
> > This is just weird. Can you dig a little?
>
> OK, just a little so far. It's not Cray-related. It occurs on other
> platforms if you modify re.py to select the pre engine. I've seen the
> same exception elsewhere when a function argument is missing. So I'm
> guessing that the problem is that pre does not present exactly the
> same interface as sre.
>
> It's a moot point for me now that I have sre working, but is the pre
> module still supported? If so should I enter this failure as a bug in
> pre?
I'd guess it's more likely to be a buglet in the test, but I don't
really care :)
You could enter a bug in sf and see if anyone picks it up. Don't hold
your breath though.
[...]
> > You can get it to not do this by just executing "make python" rather
> > than "make all" or "make". --disable-shared is a red herring in
> > this context, I think (even in 2.3 where it exists).
>
> Executing "make python" does indeed bypass attempts to build extension
> modules for dynamic linking, but "make install" and "make test"
> don't.
Ah yes.
> My ad-hoc solution is to change the following in setup.py
> (class PyBuildExt(build_ext), function build_extensions(self)):
>
> # Remove modules that are present on the disabled list
> self.extensions = [ext for ext in self.extensions
> if ext.name not in disabled_module_list]
>
> to this:
>
> self.extensions = []
Probably fair enough. There should be a better mechanism for this.
I would have thought it would be possible to get the distutils
amchinery to build a static python, i.e. you build a minimal Python
that just contains the libraries needed for distutils (re, os,
os.path, whatever) and then it manages the building of the "real"
python. I think people doping cross compiles need this sort of
functionality too...
[...]
> Not specifically that one. I think the problem is that running all the
> tests in sequence without reloading the Python interpreter demands a
> lot of memory. On our Compaq Unix machine it takes 67 MB and on the
> Cray it takes 128 MB. This doesn't sound like an excessive amount but
> Cray T3E's are not very smart about swapping and we seem to have some
> sort of load-balancing problem that causes the single-CPU jobs to
> flock to one processor.
Hmm. Not our problem, then?
> >> My immediate reason for installing Python on the Cray is that I
> >> want to check out SCons, but it *must* work on the Cray to be of
> >> any use to me. At the moment SCons won't work because it requires
> >> md5, which couldn't be compiled because it requires a UINT4 data
> >> type. So I guess I'll either have to get md5 working or persuade
> >> SCons to work without md5 checksums. Oh well...
> >
> > Yeah, the md5 code that comes with Python is fairly 32 bit specific.
> > However, if you can find some md5-for-cray code in the wild, it's
> > probably not too much effort to hook it up to Python.
>
> I'll look into that. Alternatively it should be possible to persuade
> SCons to use its TimeStamp module instead of its MD5 module for
> file-change detection. But I haven't quite managed to get that going.
>
> Still, it's all great fun!
Glad you enjoying it!
Do you feel like drafting a new section for the README? I think the
Cray notes in there are a bit old. You might want to liase with
Konrad Hinsen about it, too.
Cheers,
M.
--
As it seems to me, in Perl you have to be an expert to correctly make
a nested data structure like, say, a list of hashes of instances. In
Python, you have to be an idiot not to be able to do it, because you
just write it down. -- Peter Norvig, comp.lang.functional
Mark Hadfield
> Do you feel like drafting a new section for the README? I think the
> Cray notes in there are a bit old. You might want to liase with
> Konrad Hinsen about it, too.
Sure.
Mark Hadfield
Previously, compilation of md5module.c and md5c.c failed because symbol
UINT4 was not defined. Cray T3Es *do* have an unsigned 4-byte integer data
type (short), but the code in md5.h wasn't finding it. So I changed it to
the following
/* UINT4 defines a four byte word */
#if SIZEOF_LONG == 4
typedef unsigned long int UINT4;
#elif SIZEOF_SHORT == 4
typedef unsigned short int UINT4;
#elif INT_MAX == 2147483647
typedef unsigned int UINT4;
#endif
Now the md5 module is built and passes its tests.
I'll enter this in the SourceForge bug tracker.