how to test for cython version
Ondrej Certik
our project (hermes2d) routinely fails due to users having old version
of Cython on their systems. What is the best way to test for the
minimum Cython version that is known to work? Then "cython --version"
is not enough. Something like a git version would be great.
The other option is to try to compile some essential features, that
are needed, but that's quite some more work on our side.
Ondrej
Ondrej Certik
> Hi,
>
> our project (hermes2d) routinely fails due to users having old version
> of Cython on their systems. What is the best way to test for the
> minimum Cython version that is known to work? Then "cython --version"
> is not enough. Something like a git version would be great.
Actually, I think we rather need the date of the commit, rather than
the git hash.
Or maybe we should go the "try to compile" route. What do you think?
These are the problematic features, that fail in older versions:
* new cppclass support
* new "bool" support
* new "complex" support
Ondrej
Robert Bradshaw
> On Mon, Nov 29, 2010 at 8:39 PM, Ondrej Certik <ond...@certik.cz> wrote:
>> Hi,
>>
>> our project (hermes2d) routinely fails due to users having old version
>> of Cython on their systems. What is the best way to test for the
>> minimum Cython version that is known to work? Then "cython --version"
>> is not enough. Something like a git version would be great.
>
> Actually, I think we rather need the date of the commit, rather than
> the git hash.
How about
python -c "from Cython.Compiler.Version import version; print version"
0.13
> Or maybe we should go the "try to compile" route. What do you think?
That's another option.
> These are the problematic features, that fail in older versions:
>
> * new cppclass support
> * new "bool" support
> * new "complex" support
Ideally they'll be happy to upgrade if they're notified.
- Robert
Robert Bradshaw
>
> our project (hermes2d) routinely fails due to users having old version
> of Cython on their systems. What is the best way to test for the
> minimum Cython version that is known to work? Then "cython --version"
> is not enough. Something like a git version would be great.
Oh, sorry, I didn't see that this wasn't enough. We don't really do
any versioning more granular than that--why isn't this sufficient? Or
are you expecting your users to be on the pre-release track of Cython?
- Robert
Dag Sverre Seljebotn
> On Mon, Nov 29, 2010 at 8:39 PM, Ondrej Certik<ond...@certik.cz> wrote:
>
>> Hi,
>>
>> our project (hermes2d) routinely fails due to users having old version
>> of Cython on their systems. What is the best way to test for the
>> minimum Cython version that is known to work? Then "cython --version"
>> is not enough. Something like a git version would be great.
>>
> Actually, I think we rather need the date of the commit, rather than
> the git hash.
>
> Or maybe we should go the "try to compile" route. What do you think?
>
Do you mean that our releases are not granular enough/timely enough and
you tell other users working off cython-devel trunk? Personally I'd put
up a Github fork with an added version number (0.13.ondrejrelease1),
just like distros do. If you don't want to do that, I think the
try-to-compile route is the only sane option.
Dag Sverre
Stefan Behnel
> Or maybe we should go the "try to compile" route. What do you think?
That sounds like the worst possible option. Even if it compiles, that
doesn't mean that you get (easily) reproducible results, just think of
optimisations that one version has and another one doesn't.
I think part of the problem is that we do not currently increase the
version number of the main branch after a release. cython-devel still
builds as Cython 0.13, even after lots of changes. Changing that to
"0.13.1-pre" or something similar would be helpful.
In lxml, I parse (or parsed, before the switch to hg-svn) the SVN revision
and append it to the version tuple. Not sure if git/hg provide anything
similar. After all, having the same latest commit at the top doesn't mean
that the history is identical.
Stefan
Dag Sverre Seljebotn
I don't know about hg, but with git, the commit SHA1 (or an abbreviated
version) uniquely identifies (full contents of the repository, history)
and is definitely useful for this. Of course, you don't know if it's
earlier or later than something else by looking at it, but it is useful
when inspecting logs etc.
However, the fact that many Cython devs will use hg-git may mean this is
not the correct choice for Cython.
Two different approaches with git: Assuming running from in-tree, this
code in fwrap dynamically creates a version attribute:
https://github.com/dagss/fwrap/blob/master/fwrap/version.py
Assuming one installs, there's code like this to generate a version.py:
https://github.com/dagss/Bento/blob/master/setup_common.py
Ideally one would combine these two I guess to cover both cases.
Dag Sverre
Robert Bradshaw
<da...@student.matnat.uio.no> wrote:
> On 11/30/2010 01:39 PM, Stefan Behnel wrote:
>>
>> Ondrej Certik, 30.11.2010 05:41:
>>>
>>> Or maybe we should go the "try to compile" route. What do you think?
>>
>> That sounds like the worst possible option. Even if it compiles, that
>> doesn't mean that you get (easily) reproducible results, just think of
>> optimisations that one version has and another one doesn't.
>>
>> I think part of the problem is that we do not currently increase the
>> version number of the main branch after a release. cython-devel still builds
>> as Cython 0.13, even after lots of changes. Changing that to "0.13.1-pre" or
>> something similar would be helpful.
Yes. Done. I don't like fixing the version number of the next release
right away, as that limits the flexibility of doing a major or minor
release.
>> In lxml, I parse (or parsed, before the switch to hg-svn) the SVN revision
>> and append it to the version tuple. Not sure if git/hg provide anything
>> similar. After all, having the same latest commit at the top doesn't mean
>> that the history is identical.
>
> I don't know about hg, but with git, the commit SHA1 (or an abbreviated
> version) uniquely identifies (full contents of the repository, history) and
> is definitely useful for this. Of course, you don't know if it's earlier or
> later than something else by looking at it, but it is useful when inspecting
> logs etc.
>
> However, the fact that many Cython devs will use hg-git may mean this is not
> the correct choice for Cython.
>
> Two different approaches with git: Assuming running from in-tree, this code
> in fwrap dynamically creates a version attribute:
>
> https://github.com/dagss/fwrap/blob/master/fwrap/version.py
>
> Assuming one installs, there's code like this to generate a version.py:
>
> https://github.com/dagss/Bento/blob/master/setup_common.py
>
> Ideally one would combine these two I guess to cover both cases.
We could do something like that, and live with the fact that we have
two hashes (git and hg) to identify the same version. Of course we
decided not to ship the revision control history with the actual
releases for space reasons.
- Robert
Robert Bradshaw
Of course, it would be nice to be releasing often enough that this is
a non-issue.
- Robert
Ondrej Certik
after I sent my emails, I just got this report, by a user using the
*latest Cython release 0.13":
[ 73%] Building CXX object
tests/integrals/domain-perimeter/CMakeFiles/integrals-domain-perimeter.dir/main.cpp.o
Linking CXX executable integrals-domain-perimeter
[ 73%] Built target integrals-domain-perimeter
[ 73%] Cythonizing hermes2d.pyx
Error converting Pyrex file to C:
------------------------------------------------------------
...
pass
cdef cppclass MeshFunction(Function):
#RefMap* get_refmap()
#c_Mesh* get_mesh()
scalar get_pt_value(double x, double y)
^
------------------------------------------------------------
/home/pavel/hermes/hermes2d/hermes2d/hermes2d_defs.pxd:15:15: Syntax
error in simple statement list
By installing the latest git version, it went away.
I understand, that trying to "compile some simple codes" to determine,
if the user's Cython is new enough, is not optimal, but all other
ways, that I can see, will not catch this at the "configure time", and
simply fail during the build, like for the above user.
Ondrej
Dag Sverre Seljebotn
But if you tested your software with Cython 0.13 before sending your
code off for someone else to build, you'd know that the code you just
wrote was, in essence, Cython 0.13-incompatible.
If you only test your code against Cython trunk, you really must be
prepared to tell users of your code to use the exact same trunk version,
or bundle Cython with your code, or help us make releases more often so
that this is less of an issue.
Dag Sverre
Dag Sverre Seljebotn
Whoops, I may have misunderstood the context of your post. But I believe
the gist of what I wrote is still true. When there's a show-stopper bug
like this, ideally a new release should be made
Dag Sverre
Ondrej Certik
I chose to use the latest cython git, because it had some things fixed
(in particular some "bool" issue in this case I believe, maybe it was
something else).
Well, in principle, you are right, that I could work around that, but
I chose to simply use the latest cython, where things are fixed. But
then we run into these problems, that users can't compile our code.
I chose to go the "try to compile" route. That way, in all the cases
that we encountered so far, it would tell the user right away,
"install latest cython". Yes, maybe even some newer cython would do a
better job, but my only concern is that people can compile our code.
Otherwise, yes, we also bundle the right Cython in femhub (as well as
all the other dependencies). For some people it's not an option
though.
Ondrej
Stefan Behnel
>>> On 11/30/2010 07:59 PM, Ondrej Certik wrote:
>>>> I understand, that trying to "compile some simple codes" to determine,
>>>> if the user's Cython is new enough, is not optimal, but all other
>>>> ways, that I can see, will not catch this at the "configure time", and
>>>> simply fail during the build, like for the above user.
>>>
>
> (in particular some "bool" issue in this case I believe, maybe it was
> something else).
>
> Well, in principle, you are right, that I could work around that, but
> I chose to simply use the latest cython, where things are fixed. But
> then we run into these problems, that users can't compile our code.
>
> I chose to go the "try to compile" route. That way, in all the cases
> that we encountered so far, it would tell the user right away,
> "install latest cython". Yes, maybe even some newer cython would do a
> better job, but my only concern is that people can compile our code.
>
> Otherwise, yes, we also bundle the right Cython in femhub (as well as
> all the other dependencies). For some people it's not an option
> though.
Why would your users need Cython at all? Aren't the generated C sources
enough? It's common (and I recommend) to ship a tested version of those and
to let users build without using Cython at all.
Stefan
Ondrej Certik
That's another option. We used to just ship the generated (and tested)
C files, but it's quite a pain that whenever you modify the .pyx
files, you suddenly have to workaround it with git (e.g. you can see
changes both in .pyx and C). So other contributors simply committed in
changes both in C and .pyx files, and because the C file is changed
all over the place, the commits (log -p) are really messy, 90% of them
are occupied by meaningless changes in the source paths hardwired by
cython in the C files. Since many of our contributors are not
profficient in git, it's a problem.
So my opinions on that oscillate, but now I prefer not to commit any
generated files in the repo.
Ondrej
Ondrej Certik
Plus there is another problem, that I was not able to setup cmake to
automatically pickup the changes in .pyx, but *not* to try to compile
the .pyx files when you switch branches in git (which would fail,
because users were having old cython installed). I believe that git
touches some files when switching branches, just to be sure. I don't
know a solution to this, and it's a show stopper.
Ondrej
Dag Sverre Seljebotn
.c files should not be checked into VCS, but there could be some
hook/script when making a source distribution that would generate them
for the source tarball (unless users get the software directly from git...)
Dag Sverre
Dag Sverre Seljebotn
Thanks for letting us know, I'll remember to not consider cmake for my
projects :-) (Some other build tools would use checksums in addition to
guard against this.)
Of course, you could write a dummy cython compiler that does the
checksumming and skips the compile if the checksum is up to date.
(It'd be nice to start dumping the SHA1 of the .pyx file to the
generated C file. Something like
/* This file was generated by .....
source: test.pyx a3882df2...
source: test.pxd 34aa434...
source: something.pxi 3434f434...
*/
Sounds like some work though, and redundant with what e.g. waf and scons
does already.)
Dag Sverre
Ondrej Certik
I think this particular thing is just an ignorance on my part, as I
doubt cmake would be that stupid not to allow md5/sha1 sums.
As to C files in the repo --- yes, our users use git directly, as our
users are the same as developers mostly, and our current cython setup
is hard for our developers (except me, all others use C++ as their
"main" language).
Ondrej
Sebastien Binet
> Plus there is another problem, that I was not able to setup cmake to
> automatically pickup the changes in .pyx, but *not* to try to compile
> the .pyx files when you switch branches in git (which would fail,
> because users were having old cython installed). I believe that git
> touches some files when switching branches, just to be sure. I don't
> know a solution to this, and it's a show stopper.
fyi,
waf computes a md5 checksum of the file (rather than relying on the
timestamp)
waf-trunk has support for cython :)
cheers,
sebastien.
Dag Sverre Seljebotn
It generates and relies on Makefiles, does it not? And make doesn't
support this.
Dag Sverre
Ondrej Certik
Well, the makefiles are clever, that if something changes in the
CMakeLists.txt, the "cmake" is rerun. So I can imagine, that the
makefiles *could* be clever enough to always run some md5 checks
whenever something changes, to determine whether it really changed or
not.
But I did some tests now, and if I just touch the file, it will get
recompiled. So indeed, this is imho a (serious?) deficiency of cmake.
Ondrej
Stefan Behnel
> But I did some tests now, and if I just touch the file, it will get
> recompiled. So indeed, this is imho a (serious?) deficiency of cmake.
Well, when I want to recompile something starting at a certain dependency,
the quickest way to do it is 'touch'. I wouldn't call this a deficiency,
it's just supporting a certain use case better than another one.
Stefan
Ondrej Certik
That's right. I really like cmake, it's very robust.
Ondrej
Ondrej Certik
to conclude this thread, I have implemented the following test:
https://github.com/hpfem/hermes/blob/master/hermes_common/cmake/FindCython.cmake
and here is the test file:
https://github.com/hpfem/hermes/blob/master/hermes_common/cmake/cython_test.pyx
it works like a charm, and it has already been tested quite a bit by
our users. So I am very happy with this solution.
Ondrej
