Cliquer (was Re: [sage-support] Re: Sage 7.0 crashes)

81 views
Skip to first unread message

William Stein

unread,
Jan 26, 2016, 11:09:00 AM1/26/16
to sage-support
On Tue, Jan 26, 2016 at 7:59 AM, Volker Braun <vbrau...@gmail.com> wrote:
> The cliquer "buildsystem" is a steaming pile of s**t, it clearly should not
> be a standard package. For starters, it shouldn't install a .so on darwin...

Agreed. In fact, here's a (was private) email I wrote in 2009 after
having lunch with Guido van Rossum; it's about Cython, Sage packaging
and how cliquer is a prime example of something that shouldn't be in
Sage: "I asked Guido about Cython going into core Python and he gave
me a general speech discouraging me from getting *anything* into core
Python. (I might use some of it when people try to propose packages
for Sage.) He said that only dead code in maintenance only mode
should be added, etc., etc. He didn't really answer specifically
about Cython though, except to say that modern software distribution
tools are getting so good (alluding to Distribute?) that being in the
core of Python isn't such a big deal anymore.

He also mentioned they have an official PEP or something for all new
contributions. E.g., they won't even consider a new component be
added to Python unless somebody clearly commits to supporting the
contribution for "five years". And of course the people making that
commitment have to be reputable. We should do the same for Sage --
e.g., when PIL got considered, we should have made sure somebody
*volunteers* to maintain it for some amount of time, before even
considering it. Same for cliquer, which got voted in, then sucked in
a million ways, and almost feels disowned already. I will bring
this up next time a package is considered for inclusion in Sage. I'm
actually really shocked that in all the discussion about getting
packages into Sage, we fixed on voting procedures and other stuff, but
totally missed by far the most important point -- getting somebody to
solidly commit to maintaining the package on all platforms for a given
amount of time. We missed the most important thing and instead got
caught up in stupid bureaucracy that doesn't solve anything."

William

Nathann Cohen

unread,
Jan 26, 2016, 12:02:55 PM1/26/16
to sage-support
contributions.  E.g., they won't even consider a new component be
added to Python unless somebody clearly commits to supporting the
contribution for "five years".  And of course the people making that
commitment have to be reputable.    We should do the same for Sage --

<sarcasm>
Don't you think we should start by doing the same for Sage's own source code? Anybody who proposes a patch should agree to do the debugging/maintenance of what (s)he added for the next 5 years? Looks weird to only have this high level of expectation for third-party code only, and not for our own.
</sarcasm>

In Sage, we are at a level where some files don't have a clear "regular maintainer". We would need to be 10x more developpers to enforce such rules.

For 'cliquer' in particular, maybe we could propose upstream a autotoolized build system, and see how it goes? We did it for 'planarity' not so long ago (and perhaps with others?). Our spkg-install file indeed contains several platform-specific instructions.

Nathann

William Stein

unread,
Jan 26, 2016, 12:21:07 PM1/26/16
to sage-support
On Tue, Jan 26, 2016 at 9:02 AM, Nathann Cohen <nathan...@gmail.com> wrote:
>> contributions. E.g., they won't even consider a new component be
>> added to Python unless somebody clearly commits to supporting the
>> contribution for "five years". And of course the people making that
>> commitment have to be reputable. We should do the same for Sage --
>
>
> <sarcasm>
> Don't you think we should start by doing the same for Sage's own source
> code? Anybody who proposes a patch should agree to do the
> debugging/maintenance of what (s)he added for the next 5 years? Looks weird
> to only have this high level of expectation for third-party code only, and
> not for our own.
> </sarcasm>

Hey I was just reporting on a conversation with Guido about what they
*already* do with Python.

> In Sage, we are at a level where some files don't have a clear "regular
> maintainer". We would need to be 10x more developpers to enforce such rules.
>

Indeed -- Python probably has 10x more developers... maybe someday we
will have way more developer time too. We don't now, as we are
massively under-resourced.

> For 'cliquer' in particular, maybe we could propose upstream a autotoolized
> build system, and see how it goes? We did it for 'planarity' not so long ago
> (and perhaps with others?). Our spkg-install file indeed contains several
> platform-specific instructions.

Huge +1. I'm all for Sage devs contributing upstream :-)

>
> Nathann
>
> --
> You received this message because you are subscribed to the Google Groups
> "sage-support" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sage-support...@googlegroups.com.
> To post to this group, send email to sage-s...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-support.
> For more options, visit https://groups.google.com/d/optout.



--
William (http://wstein.org)

Nathann Cohen

unread,
Jan 26, 2016, 12:35:38 PM1/26/16
to Sage Support
> Hey I was just reporting on a conversation with Guido about what they
> *already* do with Python.

It ended with "we should do the same in Sage"

> Huge +1. I'm all for Sage devs contributing upstream :-)

Are you one of them?

Nathann

William Stein

unread,
Jan 26, 2016, 12:40:03 PM1/26/16
to sage-support
On Tue, Jan 26, 2016 at 9:35 AM, Nathann Cohen <nathan...@gmail.com> wrote:
>> Hey I was just reporting on a conversation with Guido about what they
>> *already* do with Python.
>
> It ended with "we should do the same in Sage"

I was talking about packages not code.

>> Huge +1. I'm all for Sage devs contributing upstream :-)
>
> Are you one of them?

Rolls eyes...

Nathann Cohen

unread,
Jan 26, 2016, 12:42:50 PM1/26/16
to Sage Support
>> Are you one of them?
>
> Rolls eyes...

I can't help but notice that you say "we" when you say what should be
done, and you say "you" when there is actual work ahead.

You reported this problem concerning cliquer, and you are "all for
Sage developpers contributing upstream". Will you help?

Or will you wait for Sage developpers to do it?

Nathann

William Stein

unread,
Jan 26, 2016, 12:52:36 PM1/26/16
to sage-support
On Tue, Jan 26, 2016 at 9:42 AM, Nathann Cohen <nathan...@gmail.com> wrote:
>>> Are you one of them?
>>
>> Rolls eyes...
>
> I can't help but notice that you say "we" when you say what should be
> done, and you say "you" when there is actual work ahead.

I am too busy to work on core Sage development right now. If what
I'm doing right now pays off, there will be far more resources for
Sage development in the long run than anything I could ever personally
do. I have also already done a lot -- according to this, I've done
some work on Sage:

https://github.com/sagemath/sage/graphs/contributors

> You reported this problem concerning cliquer, and you are "all for
> Sage developpers contributing upstream". Will you help?

Nope, not on that. My time is better spent elsewhere.

> Or will you wait for Sage developpers to do it?

For now, yes.

I don't know how to say this, but I'm older than you have and have a
much bigger picture longterm strategic view of things. There is much
more to helping the Sage project than diving into the trenches and
writing code (though that is very valuable too). For example, Nicholas
Thierry's work on getting the DREAMKIT funding and managing that is an
extremely valuable contribution.

That said, it's not like I don't write code -- I've evidently made
over 3000 commits to open source code in the last year:

https://github.com/williamstein

I think I've done more programing in the last two years than at any
point in my life before. And it has been at times much more difficult
than any work I did on Sage.

I'm sorry if you feel like I'm "flame baiting" you with the above
off-topic message.

Back to work.

William

--
William (http://wstein.org)

Henri Girard

unread,
Jan 26, 2016, 1:05:05 PM1/26/16
to sage-s...@googlegroups.com
Wouldn't be possible to make a cliquer package ?

Nathann Cohen

unread,
Jan 26, 2016, 1:14:16 PM1/26/16
to Sage Support
> Wouldn't be possible to make a cliquer package ?

There is one already, installed in Sage by default.

This package is a copy of the cliquer tarball that you can download
from its official website. It contains a 'minimal' build system, which
does not work on all platforms that Sage supports. For this reason,
Sage adds a few instructions in order to make it work, but that's not
the 'clean way of doing things' for us.

The "clean way of doing things" would be for the original tarball to
compile properly on all platforms, without a need for additional
instructions.

Nathann

William Stein

unread,
Jan 26, 2016, 1:17:56 PM1/26/16
to sage-support
Nathann:

> I can't help but notice that you say "we" when you say what should be
done, and you say "you" when there is actual work ahead. Will you
help? Or will you wait for Sage developers to do it?

It occurred to me that another reason I've been doing this is because
it gives me a sense for what I *have* to do versus what other people
will do. There's no point in me doing something if I suggest it,and
other people are more than competent and willing to do it. As an
example, I've suggested numerous times that we should massively
refactor the sage library to be a bunch of smaller Python libraries,
develop them say on github (?), use Pypi and pip. If people would
realize how important it is that we revamp how Sage development is
done in a much less monolithic way, and better using existing tools,
then I would be happy and enjoy watching as people do the revamp
(e.g., like happened with switching from Mercurial to Git, which I
didn't do much on, but definitely supported). As it is, I sadly see
that nobody seems to "get it" regarding how broken our development
process is right now. So, I figure I'll have to do something.
Unfortunately, I can't do anything now, due to lack of time. I hope
that either I'm totally wrong, or that I do have time before Sage gets
into deeper trouble.

William


On Tue, Jan 26, 2016 at 10:05 AM, Henri Girard <henri....@gmail.com> wrote:
> Wouldn't be possible to make a cliquer package ?

Cliquer has already been a standard Sage package since at least 2009.
--
William (http://wstein.org)

Nathann Cohen

unread,
Jan 26, 2016, 1:36:45 PM1/26/16
to Sage Support
> Nathann:

William,

I'm sorry to say that the situation is not as simple as "You suggest,
and you are happy to see people doing it for you".

Sometimes, people do stuff because nobody would do it otherwise. They
feel responsible as members of the community, and so give it a try. I
swear: trust me. And by merely "suggesting stuff and waiting for
others to (happily) do it" the truth is that you sometimes *use* those
people.

Once, on a trac ticket, I saw something like that: "Of course this
implementation is unnecessarily slow, but someday somebody will see it
and improve it". For somebody like me, it comes very close to forcing
me to improve it: because I do not accept to see a very obviously slow
algorithm in Sage when I see a clear way to improve it: we deserve
better (*).

I can only pray that not too many people will grow this "much bigger
picture longterm strategic view of things" that you mentionned.
Because each time it happens (you cite Nicolas Thierry as an example)
we lose a developer and those they take with them.

And there is work to be done. Work which will only be done by those
who remain. Some key people who, like you, are hardly repleacable: how
many persons do you think have what it takes to change Sage's build
system?

I'd say three: Volker, Jeroen and you. Perhaps there are others: I
don't even know enough to give the full list.

But you cannot expect this to be done by one person, it takes a *lot*
of time and skill, and we are not enough.

So please, now that you have seen the "much bigger picture longterm
strategic view of things", come down here and get your hands dirty. We
can't do the job if those who know how to do it think that they are
above it.

Nathann

(*) On this ticket, they *knew* what was wrong. They just did not care
enough to spend the time.

Volker Braun

unread,
Jan 26, 2016, 3:36:24 PM1/26/16
to sage-support
For the record, the OSX binaries are broken because cliquer fails to install; the log contains:

error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: changing install names or rpaths can't be redone for: /Users/vbraun/Code/binary-pkg/source/SageMath/jc4b6yulaujayb9sr94ia88eourzeqip0oidmas391yaj24ng0bmdu/local/lib/libcliquer.dylib (for architecture x86_64) because larger updated load commands do not fit (the program must be relinked, and you may need to use -headerpad or -headerpad_max_install_names)

Of course errors are not caught in cliquer's spkg-install so the compilation continues from there. The binary then doesn't work, of course.

Volker Braun

unread,
Jan 26, 2016, 3:40:07 PM1/26/16
to sage-support
I created http://trac.sagemath.org/ticket/19967 for dealing with cliquer

Jeroen Demeyer

unread,
Jan 27, 2016, 3:55:17 AM1/27/16
to sage-s...@googlegroups.com
On 2016-01-26 17:08, William Stein wrote:
> He said that only dead code in maintenance only mode
> should be added, etc., etc.

So we can never add new features to Python? That's depressing...

> I'm
> actually really shocked that in all the discussion about getting
> packages into Sage, we fixed on voting procedures and other stuff, but
> totally missed by far the most important point -- getting somebody to
> solidly commit to maintaining the package on all platforms for a given
> amount of time.

That's not realistic. I am more in favour of making it easy to downgrade
a package from standard to experimental if problems appear. If cliquer
gives problems and nobody wants to fix it, then remove it from standard.
Not a big deal.

William Stein

unread,
Jan 27, 2016, 8:00:39 AM1/27/16
to sage-support
On Wed, Jan 27, 2016 at 12:55 AM, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:
> On 2016-01-26 17:08, William Stein wrote:
>>
>> He said that only dead code in maintenance only mode
>> should be added, etc., etc.
>
>
> So we can never add new features to Python? That's depressing...

Heh, I'm just reporting what he said to me at lunch (in 2009). And he
did indeed make me go from being excited at the prospect of getting
Cython into Python to me being depressed. Python is from the early
90's, so maybe by 2009, Guido had already had a lot of experience
being burned out by trouble with contributions not being maintained...
These things go through cycles.

>> I'm
>> actually really shocked that in all the discussion about getting
>> packages into Sage, we fixed on voting procedures and other stuff, but
>> totally missed by far the most important point -- getting somebody to
>> solidly commit to maintaining the package on all platforms for a given
>> amount of time.
>
>
> That's not realistic. I am more in favour of making it easy to downgrade a
> package from standard to experimental if problems appear. If cliquer gives
> problems and nobody wants to fix it, then remove it from standard. Not a big
> deal.

Many standard packages in Sage -- at least the ones we were adding in
2008 -- once added, get things built on them, and can't be removed
without massive pain and effort. That was the context back then.

- William



--
William (http://wstein.org)

Dima Pasechnik

unread,
Jan 27, 2016, 9:09:00 AM1/27/16
to sage-support


On Tuesday, 26 January 2016 20:40:07 UTC, Volker Braun wrote:
I created http://trac.sagemath.org/ticket/19967 for dealing with cliquer

ready for review, by the way...
Reply all
Reply to author
Forward
0 new messages