Re: [sage-devel] spkg trouble
William Stein
Bill Hart wrote me an enormous email strongly criticizing me and much
of the Sage project in great detail. With his permission I've
responded below, cc'ing this to sage-flame. I've also cc'd it to a
few people I think might find this interesting because they or their
projects are mentioned, or they might just enjoy reading somebody
criticizing the Sage project at Herculean (!) length.
On Mon, Aug 24, 2009 at 12:26 PM, Bill Hart<goodwi...@googlemail.com> wrote:
> I don't know why the message re spkg problems got sent three times.
> Some google problem. I certainly did not send it more than once, and
> only sent it to sage-devel.
>
> Wrt to the problem it is possible that I was looking at a binary, and
> hence the spkg problem I had. In fact I was working with someone
> else's computer at the time, not my own and had multiple windows open.
> I may have inadvertently been looking at the spkg file on his laptop
> instead of the source elsewhere. He definitely only has a binary of
> sage on his laptop. We had some screwy problems on his laptop. It is
> an Apple, which just seems to be "special" in that special kind of
> way.
>
> I agree shipping spkg's with binaries would not be a good idea.
> Obviously I don't advocate that. Yes, changing the name from spkg to
> .txt would be very helpful, if not a drain on someone's time. I assume
> there are also instructions in there about how to obtain the spkg for
> real by typing sage -something.
>
> But, with respect to the spkg concept, I really think they are a bad
> idea. I personally find them very frustrating. The rest of this email
> is one very long winded way of me expressing precisely why that is,
> and where that comes from. Sorry it is so ridiculously crazily long.
> But this obviously not about spkgs, per se, but something deeper.
>
> But let's start with spkgs. It is really annoying for example that
> browsing the source for sage online I cannot look inside spkgs. I have
> to download the spkg to a local file, try and extract it, etc. Same
> even if it is on my local hard drive instead of online. It also in my
> opinion discourages people digging into the source of libraries
> distributed with sage. That seems to be a bad thing wrt to getting
> people to contribute to the libraries. Also the only way someone can
> contribute to an spkg is to open the spkg, make the changes, learn how
> to make an spkg, repackage the entire thing, post the entire damn spkg
> to trac, etc. In addition there's the whole issue of needing to make
> an spkg every time an upstream library changes and you want the latest
> version in sage, which is all a lot of work for *someone*.
> Furthermore, how on earth does HG manage changes to the spkg's all the
> time? Surely this is a nightmare from the perspective of the version
> control software. It's designed to manage source code, not tarballs.
> Presumably they aren't even under version control. Another
> disadvantage: if I need to hunt down a bug across multiple spkg's,
> I've got to open each one individually. It just makes that whole area
> of Sage much more difficult to work on, which is very frustrating for
> someone like me, for obvious reasons!!
>
> The argument that .debs or .rpms are some kind of package file doesn't
> really make sense to me. There is always an option to download a
> source .deb or .rpm and then I get the source code. Also, a linux
> distribution is not a program or library, like sage, it is an
> operating system and large distribution of other programs and
> libraries. .deb's and .rpm's are a convenient way for a user to
> install new programs or libraries in their system. So if you translate
> that to Sage, indeed .spkg's are a convenient way to add new
> features/libraries/whatever to sage, and it is used for that in the
> sense of the standard vs optional spkg's. But is there a sage install
> flint-src option, to download a source spkg for flint within sage?
> Maybe there is and I just don't know about it. Obviously as a
> developer, it is the *source* I care about. As an OS user it is the
> program working on my machine *now*, that I care about. So the spkg
> model is actually better suited to the binary world, not the source
> world, at least the way it is currently set up.
>
> The analogy between sage and a linux distribution is limited. I am
> aware of no other program or library which distributes the source code
> for a component of that program or library, packaged in some
> non-standard package, or even in a tarball. It *is* a very odd thing
> to do. Maybe firefox extensions or GIMP modules. But again the analogy
> is pretty limited in both cases.
>
> Again, for me, it all comes back to this: is there a technical reason
> for having spkg's? I can only think of disadvantages - lots of them.
> One possible advantage:
>
> * The entire sage source tree doesn't take up as much space on
> people's hard drives.
>
> Maybe that is a non-trivial advantage. But really, apart from the
> explosion on sage.math, how many people is this really a problem for?
> I mean, who these days doesn't have a few GB free on their drive. And
> moreover, the source is distributed in one large tarball anyhow, so it
> isn't going to make a difference to the size of the source tarball.
>
> However let us suppose this really was a problem for some people. Then
> they should compress their drive, like the late 80's, when this was
> last an actual problem. Or switch to btrfs.
>
> Feel free to post any of the above to sage-devel. I'm just posting
OK, I will. I'm not afraid of any criticism that could possibly be
aimed at the Sage project. Note that I'm sending this to sage-flame
instead of sage-devel, since the readership of sage-flame is much more
likely to be interested in this email, it is written in an appropriate
style, etc.
> only to you in case there is a very simple and very valid reason that
> spkg's exist and you can tell me.
Wow, everything you wrote above is pretty negative. One thing I
always try to remind myself of is that the view looks a lot different
from the trenches. You might want to try being the Sage release
manager a few times to see what the view is like.
> Beware though, there is lots below
> this point you won't want on sage-devel. Please take care!
>
> Now, why do I care so much? Actually it is *you* who convinced me
> about these issues. You recall a while back you sent me an email about
> what stops people contributing to open source projects. Believe it or
> not, I really spent a *lot* of time thinking about this issue. I
> actually read through the very long documents you sent links to,
> watched the vids you sent me links to and even found a number of other
> very long docs and vids and processed those as well. I discovered one
> immutable truth (amongst a lot of superstition and bullshit):
>
> "People will not contribute to open source projects unless you make it
> absolutely trivial to do so, and *any* encumbrance, no matter how
> slight will prevent them from doing so".
>
> (BTW, in a few months time you will see my complete answer to this
> problem from the point of view of at least FLINT, and possibly MPIR.
> But I am remaining very tight lipped about that.)
>
> Anyhow, applying this maxim to Sage, if someone has to learn how to
> open an spkg, open it, modify it, learn how to make an spkg, post it
> to trac, etc, etc, (and their distributed version control software has
> no hope of handling this), just to contribute to those vital parts of
> Sage, called libraries, then people won't do it, because it is way too
> hard. Some people will do it, because they need to. But others will
> give up.
It will be interested if your "immutable truth" above were true. But
didn't you just disprove it, since there are over 200 sage trac
accounts, and the last Sage release had 56 distinct contributors, 13
of which contributed for the first time.
Again, consider your "immutable truth":
"People will not contribute to open source projects unless you make
it absolutely trivial to do so, and *any* encumbrance, no matter how
slight will prevent them from doing so".
It's a good thing mathematical research journals don't believe in a
similar immutable truth.
> I just spent the afternoon here at Warwick talking with a visitor who
> has precisely this problem.
>
> You keep making the argument that Sage is so successful, therefore
> what we are doing is by definition right. This is called superstition
> - the desire to not change anything because what we have has clearly
> worked so well for us in the past.
superstition = a belief or notion, not based on reason or knowledge.
Looking objectively at whether or not a workflow or process is working
by some objective measure, e.g., "number of contributors to a
release", does not seem to me to be the opposite of superstition.
> But you are not seeing the people who are *not* contributing to Sage
> because something put them off. Whether it be people not wanting to
> put their HG repos up for others to pull from for fear of
> embarrassment, or because people are unable to get patches in because
> the codebase changes in ways that make it difficult for them to
> rebase, or because of how daunting the code review process is, you may
> not see people turning away from Sage because of these issues, but I
> do. I see them all the time. In particular some of these issues really
> affect the number of permanent/tenured/faculty who could be
> contributing.
The goal of the Sage project -- which I started -- is not to maximize
the number of contributors. The goal of the Sage project is one
thing:
Mission Statement: Create a Viable Free Open Source Alternative to
Magma, Maple, Mathematica, and MATLAB.
You might be confusing two distinct goals: get big, accomplish
something; because, your projects -- MPIR and FLINT -- have few
contributors, so you are (rightly so) focused very much on that, which
you see as a problem.
> You might view all this as some flimsy attempt to divert attention
> from what is really the problem for me, which is lack of contribution
> to MPIR and FLINT, which is, as you put it, a problem with suboptimal
> things with regard to those projects from the point of view of
> contribution. The really odd thing for me is that one of the things
> you recommended was git. You had me watch Linus' vid about it. I
> assumed you had plans to go this way with Sage, as git is really
> technologically superior, and the entire model that it encourages
> "branch early, branch often and pull not push" is much, much, much,
> much, much, much better for open source projects. But when I asked
> about this issue, you said that Sage is not a big enough project yet,
> and too many people come and go. WTF!?
You specifically asked about having lieutenants responsible for each
subdomain in Sage, and having them each act as release managers for
their domains. I do not think *that* will work right now. Since I
think I already justified that at length, I won't go further.
By the way, though I recommended Linus's talk, it doesn't mean I
endorse everything he says. I always read the book "The Cathedral and
the Bizarre", which is a wonderful book, but again, I definitely do
not agree with everything it says. When I suggest that you look into
some resource, I'm not saying it is the bible with all the answers,
but something to read/listen to/consider, and then think through
carefully for yourself.
> I actually laughed, quite hard, when you suggested I had not "got" the git.
I'm sorry if I offended you. It wasn't in the least bit my intention.
> Anyhow, instead I've seen precisely why I'm not getting contribution
> to FLINT and MPIR. The main reason is that both projects do not have
> sufficient resources by themselves. Open source low level number
> theory software hacking is very, very specialised. It's not like
> writing an OS or a Matlab replacement, or a graphics program, where
> you have a large pool of potential high quality contributors. You know
> all the people who are going to contribute. Some of them don't like
> you. Some of them can't contribute because of restrictions their
> workplace places on them. Others are committed elsewhere. So, no
> quantity of FAQ's, changes to DVCS, development documentation, webpage
> resources, development groups, etc, etc, etc, is going to bring
> significant quantities of contribution to FLINT or MPIR. I could spend
> the next year writing such stuff and the net effect on FLINT and MPIR
> would be minimal. And my institution will not allow me to squander my
> time doing these things. I need to publish.
The first ever criticism of Sage was by Richard Fateman back in 2005
that Sage would fail because writing advanced mathematical software
requires too specialized of a skillset, and that there was no way we
would get enough contributions. It was valuable criticism, and I
learned a lot from it. Your argument above about FLINT/MPIR is
similar.
Sorry, but I just don't buy it for a second, because I'm incredibly
optimistic about people's abilities. I think this has a huge amount
to do with how I was raised -- I vaguely recall people pointing out to
my brother and I when we were young that we were vastly more
encouraging toward the other kids than they were used to seeing.
> You see the same thing in Pari, NTL, GMP, presumably Maxima and
> numerous other packages, even LiDIA died. Sage is in fact contributing
> to, not alleviating emaciation of these libraries and programs.
I don't buy that for a second, and Michael's argument below is spot
on. That's like saying the Linux kernel is emaciated by the existence
of Ubuntu.
> Michael used to say often that so many packages out there are
> benefiting from Sage, because of all the bug reports and patches that
> were going back upstream. But from the point of view of a library,
> this is not contribution, but merely more stuff in the inbox. Fix
> this, fix that, port this, port that, support this system, and why
> won't anyone help with the Sage port to Windows...
So ignore that email if your project doesn't have the organization to
handle it. My experience, and the experience of many other people
I've worked with, is that upstream projects are often thrilled to
learn of the existence of bugs and ideas for their fixes. I've never
heard of a quality software project that doesn't greatly value bug
reports.
> Here's my perspective. Sage is a community of volunteers all working
> toward the same aim - to build a mathematical software package.
> Contribution comes in many forms, but most significantly code
> contribution in libraries and in the Sage python and Cython code.
Maybe. This is a function of time and the specific mathematical area.
Robert Dodier and Andrej Vodopivec made a *huge* contribution to
symbolic manipulation in Sage today in the form of a 1-paragraph
email. http://trac.sagemath.org/sage_trac/ticket/6818/. This one
little thing will improve performance for people more than tons of
other work people have done.
> Those are the core aspects of Sage development. In many ways, Sage has
> subsumed things like NTL, Pari, Maxima and other libraries (even GMP),
> because their maintainers do not have the resources to maintain them.
>
> But instead of seeing these as all part of the "Sage project", only
> the Sage python library code is being given first class status (in
> some respects - yes I accept that in other respects they are given
> very first class treatment). One very influential Sage devel recently
> shared with me their vision of Sage: get rid of most of the C
> libraries and have everything implemented in Cython code. I mean,
> WTF!?
You are suggesting that design and development of Sage should proceed
with careful sensitivity to politics, "status", "class", etc. I
absolutely disagree with this. If a developer has a good idea that
leads to Sage being a technically better system, then go for it!
Sage is not some feel-good European Union project. It's about
technical excellence. If somebody can create a Cython replacement
for some existing C library and this new approach is technically
superior, I don't see what the problem is. Especially if they are
doing this work for free/fun! More power to them.
> One person bills Cython as "C without the semicolons", another bills
> it as "a dialect of python", another as a "smooth blend of python and
> C++". I've heard other people refer to it as a "Frankenlanguage" or as
> "glue code" or "what the 'eck is that".
"Cython is the ideal language for wrapping external C libraries, and
for fast C modules that speed up the execution of Python code." --
cython.org
> To me, cython exists for a couple of reasons (apart from the
> brilliance of its designers), most significantly because python is too
> slow for many purposes. But if cython really is just C without the
> semicolons, then why not encourage people to write straight C++? It's
> fast, apparently no more difficult that Cython, since you just have to
> add semicolons, and you can still apparently call C++ code from
> Python. Python can then be the "glue code".
>
> But as you put it, one of the reasons for the great success of Sage is
> the simplicity of the python language. People want to, and can
> contribute because python is so easy, and C++ is so "hard". So it is
> pretty clear that Cython is *not* the future of Sage, unless you have
> a whole load of people dedicated full time to going through python
> code and rewriting it in cython. Tough titties that this is never
> going to happen.
I can only conclude that you haven't written much Cython code.
Actually, have you contributed even one single line of code to the
Sage library? I'm not asking that to imply that you are somehow not
a "first class citizen", but to suggest that without experience you
might have some misconceptions regarding the use of Cython in the Sage
library. It is often a good idea to remind oneself that things look
a lot different from the trenches.
> People will always contribute python code, not cython if that is their
> level of their competence. Otherwise you are going to have people like
> me contributing C/C++/Assembly, because we can, and because it is
> faster and better than python for that kind of stuff. That being the
> case, ought not the C/C++ library part of Sage be split into
> discernible subprojects *of the Sage project* which receive equal
> attention and care to the rest of Sage, much like the core of Magma,
> but without the monolithic 2 million line bit?
What would this even mean? Do you want an account on
sage.math.washington.edu? Do you want access to "exclusive build
farms"? Do you want space on the sagemath.org webpage? Do you want
me to assign my "army of minions" to fixing bugs in MPIR?
I don't have an army of minions. I don't tell *anybody* what to do on
Sage, except people who work directly for me, and even then I hardly
do. I'm the opposite of John Cannon, who takes a lot of care and
pride in orchestrating much of Magma's development (and given what
Magma has accomplished, he has done a very impressive job). Right now
are four people getting paid to work on Sage: three UW undergrads
working part time on some coding projects:
1) a very nice units package in pure Python
2) a competitive implementation of Miller-Odlyzko-Lagarias for
computing pi(X), which he's deciding he is writing in C:
http://code.google.com/p/primecounter/
3) another who is polishing various things related to graphics,
including HTML canvas rendering related to 3d graphics.
Also, my NSF grant fully supported the development of the mpmath
Python library this summer.
> Cython has its place and do not misunderstand me. I am not suggesting
> that in some way Cython is not a vital part of Sage and the Sage
> community. I am only suggesting that people like me are not going to
> use it for everything because we are first of all not PFL (python
> first language) speakers and thus couldn't give a toss that Cython is
> "so much like python", and second of all because people like me can
> use C++ and occasionally need to do some optimisation in assembly, so
> we will.
My humble hope is merely to increase the chances that you will have
the choice to not use Magma or Mathematica if you don't want to.
> If that isn't a reasonable argument, and doesn't have consequences for
> the architecture and revision control and packaging of Sage wrt its
> libraries, then I just don't see how Sage plans to move forward in the
> future. I just see a train wreck coming in about 5-10 years. I see a
> whole pile of dead unmaintained libraries packaged along with a *lot*
> of python code, which someone one day says they are going to
> cythonise.
I've had numerous well-meaning people tell me they see a train wreck
coming for Sage in X years. I have greatly appreciated that they took
the time to share their concerns (e.g., Richard Fatemen was one such
person, but there have been many). I've been hearing this from
people since Sage was first released. For the first 3 years I
worked on Sage I actually believed it too, and told myself that I was
crazy putting so much of my time and energy into Sage. I knew for
years *myself* that Sage is impossible and would never go anywhere.
What the community has ended up creating so far has far surpassed my
wildest expectations.
> All the library writers will have moved on.... and sage will be full
> of libraries no one knows how to maintain.
There is tons of Cython code in Sage written by people who have now
moved on, and who don't read sage-devel anymore, don't pay any
attention to Sage right now, are finishing their thesis, etc. And
yet there were over 1,400 messages just on sage-devel last month, 56
contributors to the last sage release, and sage development just keeps
marching on. Instead of arguing as you are based on some secret
logic, I look at actual numbers and conclude that the Sage project is
healthier than ever, though of course it still isn't near succeeding
at its mission statement.
This constant "moving on" of people is natural. It is critical to
come to terms with this fact, because it has a big impact on design
decisions.
> The maxim that there are plenty more people out there to do the work,
> so it doesn't matter if people move on, is just not true when it comes
> to writing highly optimised assembly and C code. It is probably true
> of python code, but hacking low level number theory software is
> specialised and we know most of the people who do that.
I'm definitely far more optimistic than you are here.
> That means, it makes sense to also build a community around asm/C/C++
> library development, so that if library writers do move on, it won't
> matter, there's a community to get on with it.
The more communities the better!!
> I don't mean to be big headed and to seem self important, but you
> know, 80,000 lines of highly optimised C code is a pretty substantial
> chunk, along with MPIR which is what quarter of a million lines or
> something. I'm feeling the burden a little here...
>
> Now I don't want to speak out of turn here, but let me suggest
> something a little radical. Is it not true that the following
> architecture would alleviate any of the issues that one currently has
> with poor python performance, at least in our field - number theory:
>
> * At the bottom have your highly optimised C and assembly libraries -
> MPIR, FLINT, M4RI, IML, zn_poly, fflapack, linbox, MPFR - which in my
> opinion should pool resources and basically merge, within the sage
> community (I exclude Pari and NTL which eventually have to go - they
> are rubbish from a performance perspective and that is never going to
> change) - and who gives a shit about ...
[... censored offensive remarks about some people ... ]
> * Next level you have a bytecode interpreter written in C/C++
>
> * Top level you have a python interpreter which parses the input from
> the user and compiles it to bytecode as the user types and then sets
> the super fast bytecode interpreter running on it
>
> An empty _for_ loop in this scenario can do 100,000,000 iterations a
> second. Not so with Pari, Python, Ruby, any sufficiently high level
> interpreted language, which is probably minimum 25-100 times slower
> (Lua is an exception here). I don't know why Pari is so slow - it
> shouldn't be. It's just crap. But we knew that already.
For the record, it takes 6 seconds in Python (not 25-100 seconds):
sage: time for i in xrange(10^8): None
CPU times: user 6.18 s, sys: 0.00 s, total: 6.18 s
Wall time: 6.18 s
Interstingly, Pari also takes 6 second:
sage: gp.eval('gettime; for(i=1,10^8,); gettime/1000.0')
'6.1500000000000000000000000000000000000'
> Anyhow, the above model also makes generic programming really
> straightforward for things we care about, like arrays and matrices
> defined over other basic objects (and over themselves, etc) - and many
> times faster than python.
>
> If you think about it, this is what a user of Sage is going to really
> care about. It isn't whether Sage can compute the number of partitions
> of n really really fast. It is whether they can get their algorithm,
> which they've just implemented in the python/sage language to run
> really really fast. But they are mystified why it doesn't until they
> discover that python is actually really, really slow for basic things
> like iterating over the integers....
I have a much different view of what "a user of Sage is". That said,
let me emphasize yet again the goal of the Sage project: be a viable
free open source alternative to Magma, Maple, Mathematica, and Matlab.
Doing something new and different in the direction you suggest
doesn't really address that goal. Part of the goal of Sage is that it
is not supposed to be a "research system" (like Linbox was for years),
but a system that is ready for everyday usage ** as soon as possible
**.
> The above is more than just a thought, idly thrown out there. I've
> been thinking about this for months, and doing coding experiments. I
> even looked at other languages, other than python. Lua is really fast,
> but otherwise useless. Ruby is a lovely language, but as slow as
> python generally. C++ is really really fast, but somewhat pedantic to
> use when you do classes, which you need to use for generic
> programming. Cython is not yet a standard language with huge
> quantities of really well supported libraries (C++ and python are).
>
> (Again, please don't misunderstand me. Cython really does have its
> place in Sage, and is very, very important. I just think there is
> something better we can do for some limited subsection of what is in
> Sage.)
>
> The above also, by the way, makes it really trivial to have a front
> end for Sage which uses the Magma language (yuck). It just drives the
> bytecode interpreter using different commands from the user. But this
> is of course merely a pleasant advantage. Otherwise the proposal is to
> have a python front end, dealing with the user and providing a
> language and doing what it does best, dealing with text. Then
> underneath you have a very fast engine.
Maybe you should get involved in the awesome Mathemagix project!
http://www.mathemagix.org/www/main/index.en.html
Check it out:
" Mathemagix provides a new high level language, which is imperative,
strongly typed, with polymorphim and parametrized types. Mathemagix
can be used as an “extension language”, i.e. easy to embed into other
applications and to extend with existing libraries written in other
languages like C or C++. An interesting feature is that this extension
mechanism supports template types.
Currently, only a rather slow interpreter Mmx-light is available, but
a compiler is under development. All necessary type verifications are
done during the compilation phase. In addition, the Mathemagix
language provides powerful constructs for allowing the compiler to
generate extremely fast code (comparable to the speed of C or C++). "
It's built on the libraries you mention above, it will have a
compiler, etc. It sounds like what you want. Just keep in mind that
building something like this is really hard. I think the Mathemagix
project is *twice* as old as Sage! I would be much more interested
in that project if I wasn't so impatient (at this particular problem
-- I'm patient at math research).
> I have to confess that in the final analysis, when I come to use
> Sage/FLINT/MPIR/other libraries for my number theory research, if
> something is not implemented, if it is a basic type, I am going to
> implement it in assembly or C in FLINT or MPIR or some other library.
> If it is generic, I'm likely to implement it in C++ and stick a hook
> into the bytecode syntax. Only really high level stuff is going to be
> in python, where performance just doesn't matter. So doing something
> like what I propose above isn't going to get me using heaps of python
> probably. I don't think for example that I'm going to implement ideals
> of number fields in python for example. But I might type my high level
> algorithm which uses ideals into Sage in the Sage language, which
> might be a thin extension of the python language, as it is now, and it
> will call the ideal code, which will be written in C++, via the
> bytecode.
Just for the record, you are definitely not the typical target
audience I have in mind for Sage.
> Anyhow, obviously I envisage something different and perhaps much
> better than what we are currently doing. Python is right (actually
> Ruby is really nice, but I doubt it was around in the current form
> when Sage started). There isn't a better language, all things
> considered, to use for Sage than python. The focus on performance is
> right. Open source is right. The community, Sage Days, involvement of
> students, including undergrads, the notebook concept is right. So lots
> is right about Sage. But some things are wrong - very wrong, and are
> going to really bite in the long run. I've mentioned some here, some
> before.
At this point, I find myself unconvinced that anything is "very wrong"
with Sage. Obviously there are many technical aspects of the
software that are far from "best possible", but that's just life when
one has limited time.
> I really see some parts of Sage as lacking direction.
It's a big open source project with almost entirely volunteer effort.
Yes, there is a lack of top-down direction, since if I started telling
people what to do (much like you are trying to tell me what to do
right now), that wouldn't go over very well, to put it mildly. At
least you are expending the effort to careful explain and argue for
why I should do what you want me to do. But at the end of the day,
you're telling me: "do this or else your project is doomed". I don't
feel comfortable telling other people that, and if I gave you the
impression that I did that with MPIR, I sincerely apologize.
> And I am sure I
> can see why. The notebook, the Windows port, number theory in Sage,
> competing with Magma, even getting funding, all are suffering.
>
> The notebook : because its full potential has not yet been realised
> The windows port : lack of a GUI, DVCS issues, (and MSVC just sucks),
> numerous website issues
> Number theory in Sage (libraries) : library developers need to work
> together more and pool resources instead of building two-man empires,
> see above on spkgs, DVCS, general Sage architecture
> Number theory in Sage (python) : python actually sucks as a language
> for number theory development for performance reasons, see above
> Competing with Magma : lack of focus, no cohesive fast linear algebra
> package exists, fullstop, no decent open source multvariate polynomial
> library exists, fullstop, no decent polynomial factorisation or
> multivariate GCD exists, full stop and no one willing to tackle these
> head on, and see above re number theory.
> Getting funding : See above re Magma (from what I can gather, from
> what you've hinted from time to time)
Personally, I agree that some of the above are probably suffering
because, in fact I have barely worked on Sage in the last two months.
Why?
(1) I spent a few weeks doing serious research mathematics this
summer (which has led to some very interesting results about
Kolyvagin's Euler system for higher rank curves, which I'm very proud
of, along with several thousand lines of code for Heegner points that
I'll contribute to Sage soon).
(2) I spent several more weeks this summer writing a popular book on
the Riemann Hypothesis with Barry Mazur.
(3) I'm going to spend several more weeks working on databases of
modular forms and L-functions.
Why? Because I got an NSF grant that funds me to work full time for 2
months each summer on some specific number theory projects I applied
to work on, and so I'm doing just that.
In the meantime, Sage development chugs on amazingly, with the last
release having nearly 100 contributions by 56 people, etc., etc., and
I had almost nothing to do with that.
> I'm prepared to be part of the solution, not a growing group of people
> who keep winging about Sage. But there are some radical things needed.
>
> Would it be so impossibly hard to do an experiment. Put all the number
> theory related spkgs into a git repo and make it publicly accessible
> and encourage people to pull it and publish their own git repos. Make
> some noise on sage-devel and sage-nt about a fresh perspective or some
> other such nonsense, encouraging C library writers to think about a
> single library or perhaps a small number of libraries which satisfy
> Sage's high performance number theory goals (old, politically
> motivated power mongers will be discouraged from taking over so that
> younger contributors can be encouraged - but focus on trying to get
> professionals to oversee, i.e. people like me, clement, martin, craig
> - people who do, and intend to keep doing, maths, specifically NT and
> crypto for a living). Drop the review procedure for number theory
> contribution and the posting to trac crap and use a pull-pull model
> with a group of dedicated individuals overseeing that code and its
> overall quality and that it merges into Sage correctly. Float the idea
> of a bytecode interpreter and some localised optimisation within Sage
> nt python code to use it. Screw [censored XXX]. Discourage the
> restriction of "number theory" to elliptic curves only and let sage-nt
> be a place for number theorists, including those interested in trivial
> shit like polynomials and linear algebra to hang out.
Despite being project leader or whatever, I simply don't feel
comfortable telling people what to do, at least not at the level
suggested above. I don't feel comfortable using my power (as it were)
to do a blanket change in something the community introduced like
patch refereeing. If I had argued for patch refereeing and were the
reason that Sage has it, then maybe I would. But I'm not. I think
Craig Citro is the main reason Sage has patch refereeing and the 100%
doctest requirement.
You may want to create a git repo like you suggest above, argue
carefully to the community for why people should use it, and see what
happens. That said, watch out, since many important people don't read
sage-devel anymore, since it is such a high traffic list (and also
because some people's brain's are broken and they don't know proper
email netiquette).
> Anyhow, enough for now. For all I know, you've been reading this and
> thinking I'm off with the fairies, or just having a bad day (actually
> I've had a very, very good week and gotten all excited about where to
> next with FLINT and MPIR - you'll see what that is about in 3-6
> months). But anyhow, if you do think I'm crazy, please understand I've
> given this one hell of a lot of thought. I'm prepared to put in the
> effort to carry out what I'm proposing. But on the probability that
> I've completely lost you, I'll stop here, as more explanation is not
> going to help if that is the case.
Unfortunately you have completely lost me. I'm sorry. I wish it
weren't so. As I see it, the fundamental flaw in your argument is
that you vastly overestimate my relevance to the Sage project.
-- William
Robert Bradshaw
> Hi,
>
> Bill Hart wrote me an enormous email strongly criticizing me and much
> of the Sage project in great detail. With his permission I've
> responded below, cc'ing this to sage-flame. I've also cc'd it to a
> few people I think might find this interesting because they or their
> projects are mentioned, or they might just enjoy reading somebody
> criticizing the Sage project at Herculean (!) length.
Wow, that was a long, but interesting (to me at least) read. I'm just
going to comment on a couple of things here, I think the rest were
adequately addressed or not worth rehashing at the moment.
> On Mon, Aug 24, 2009 at 12:26 PM, Bill
> Hart<goodwi...@googlemail.com> wrote:
>> I agree shipping spkg's with binaries would not be a good idea.
>> Obviously I don't advocate that. Yes, changing the name from spkg to
>> .txt would be very helpful, if not a drain on someone's time. I
>> assume
>> there are also instructions in there about how to obtain the spkg for
>> real by typing sage -something.
I've been bitten by this too on the rare occasion I hit a sage binary
install. http://trac.sagemath.org/sage_trac/ticket/6822
>> But let's start with spkgs. It is really annoying for example that
>> browsing the source for sage online I cannot look inside spkgs.
Yep, that would be cool. Probably not even to hard to set up.
>> Another disadvantage: if I need to hunt down a bug across multiple
>> spkg's,
>> I've got to open each one individually. It just makes that whole area
>> of Sage much more difficult to work on, which is very frustrating for
>> someone like me, for obvious reasons!!
Suppose hunted down a bug that involved linbox, givaro, and mpir. I'd
still have to patch and upstream each of the packages individually.
>> The argument that .debs or .rpms are some kind of package file
>> doesn't
>> really make sense to me. There is always an option to download a
>> source .deb or .rpm and then I get the source code. Also, a linux
>> distribution is not a program or library, like sage, it is an
>> operating system and large distribution of other programs and
>> libraries. .deb's and .rpm's are a convenient way for a user to
>> install new programs or libraries in their system. So if you
>> translate
>> that to Sage, indeed .spkg's are a convenient way to add new
>> features/libraries/whatever to sage, and it is used for that in the
>> sense of the standard vs optional spkg's.
I think we need to be more clear on this--spkgs are ways to include
*independent upstream projects* in Sage. If someone wants to add a
feature, usually modifying the library directly is a much more
reasonable (and easier) route. Something like Flint and MPIR has a
life outside of Sage, its own releases, its own community, etc.
(though still much more closely tied to Sage than many upstreams),
and this is why the "distribution" model seems a good fit for me.
I think this will become more true as the notebook and infrastructure
becomes decoupled (SPD), and people start packaging distributions
that are slimmed down and/or domain specific.
>> But is there a sage install
>> flint-src option, to download a source spkg for flint within sage?
>> Maybe there is and I just don't know about it. Obviously as a
>> developer, it is the *source* I care about.
I don't think there even is a non-src flint spkg. As far as I know,
the only binary code we ship as part of an spkg is some .jar files
for 3d graphics in the notebook.
>> Anyhow, applying this maxim to Sage, if someone has to learn how to
>> open an spkg, open it, modify it, learn how to make an spkg, post it
>> to trac, etc, etc, (and their distributed version control software
>> has
>> no hope of handling this), just to contribute to those vital parts of
>> Sage, called libraries, then people won't do it, because it is way
>> too
>> hard. Some people will do it, because they need to. But others will
>> give up.
Yep, it's still too hard for some people. There are several ways in
which I wish the process would be improved. sage -merge is a big step
in that direction, but there's still too much manual overhead.
But despite that the community is thriving, people are submitting
patches faster than ever, and the biggest bottleneck is getting
things reviewed and merged fast enough.
>> One person bills Cython as "C without the semicolons", another bills
>> it as "a dialect of python", another as a "smooth blend of python and
>> C++". I've heard other people refer to it as a "Frankenlanguage"
>> or as
>> "glue code" or "what the 'eck is that".
>
> "Cython is the ideal language for wrapping external C libraries, and
> for fast C modules that speed up the execution of Python code." --
> cython.org
I had to laugh when I read this. Hey, at least people are talking
about it :) Yes, it's a bit of all of the above, but I think "a
dialect of Python" is the best characterization. Even closer to the
truth (at least, the direction it's headed now) is that Cython is a
Python compiler that takes type annotations.
>> To me, cython exists for a couple of reasons (apart from the
>> brilliance of its designers), most significantly because python is
>> too
>> slow for many purposes. But if cython really is just C without the
>> semicolons, then why not encourage people to write straight C++? It's
>> fast, apparently no more difficult that Cython, since you just
>> have to
>> add semicolons, and you can still apparently call C++ code from
>> Python. Python can then be the "glue code".
>>
>> But as you put it, one of the reasons for the great success of
>> Sage is
>> the simplicity of the python language. People want to, and can
>> contribute because python is so easy, and C++ is so "hard". So it is
>> pretty clear that Cython is *not* the future of Sage, unless you have
>> a whole load of people dedicated full time to going through python
>> code and rewriting it in cython. Tough titties that this is never
>> going to happen.
>
> I can only conclude that you haven't written much Cython code.
I would have to guess the same.
The point is that (with few exceptions, and they are going away as
Cython develops) Python code *is* Cython code. You don't have to port
it in the same sense that you have to port from Python to C, or even
C to assembly. Of course, to see massive speedups sometimes takes
some thought, but typically it's just a matter of declaring types
correctly.
There are a lot of cases where one person comes along, notices Sage
doesn't yet have what they need, and implements it in Python and
that's good enough. Weeks or years later, someone else comes along,
notices it isn't fast enough for what they need, and "ports" the
aforementioned code to Cython.
Of course for some stuff, like writing the fastest truncated, cache-
friendly, multi-core FFT for each brand of x86, Cython is
(probably :) the wrong tool for the job.
>> * At the bottom have your highly optimised C and assembly libraries -
>> MPIR, FLINT, M4RI, IML, zn_poly, fflapack, linbox, MPFR -
Sounds like what we have.
>> which in my opinion should pool resources and basically merge,
>> within the sage community
Sounds like a Herculean task on a social level, and I'm not sure if
it would be easier to manage as a single project, but I won't
discourage you from trying.
>> * Next level you have a bytecode interpreter written in C/C++
We have that--Python is compiled to bytecodes and run on a bytecode
interpreter. There are projects underway to make it even faster
(unladen swallow, parrot, ...) and of course anyone with the time and
motivation could try rolling their own.
>> * Top level you have a python interpreter which parses the input from
>> the user and compiles it to bytecode as the user types and then sets
>> the super fast bytecode interpreter running on it
See above.
>> An empty _for_ loop in this scenario can do 100,000,000 iterations a
>> second. Not so with Pari, Python, Ruby, any sufficiently high level
>> interpreted language, which is probably minimum 25-100 times slower
>> (Lua is an exception here). I don't know why Pari is so slow - it
>> shouldn't be. It's just crap. But we knew that already.
>
> For the record, it takes 6 seconds in Python (not 25-100 seconds):
>
> sage: time for i in xrange(10^8): None
> CPU times: user 6.18 s, sys: 0.00 s, total: 6.18 s
> Wall time: 6.18 s
Most of that time is spent storing 'i' into the global dictionary. On
sage.math with a local variable:
sage: def foo(n):
...: for i in xrange(n): None
...:
sage: time foo(10^8)
CPU times: user 2.98 s, sys: 0.00 s, total: 2.98 s
Wall time: 2.98 s
- Robert
William Stein
From: Bill Hart <goodwi...@googlemail.com>
Date: Mon, Aug 24, 2009 at 10:13 PM
Subject: Re: [sage-devel] spkg trouble
To: William Stein <wst...@gmail.com>
Ah, the stupid thing bounced. I don't care to figure out why. In case
it didn't make it there, here is my reply. It's mainly to you anyhow,
though to be honest I think you put a pretty negative spin on what I
wrote in the way that you characterised it and me, and maybe my
followup deals with that.
But I rather think that people who know me will just read that and
write to me and say WTF Bill, did you really say all that, and I'll
explain it to them. Much easier than explaining to the whole world.
Maybe one thing does require further comment. It bothers me that you
think I'm writing to you and saying do this or your project is fucked.
I think I failed to connect with you on any level at all if that is
what you think. And that surprises me. That seems to suggest to me
that you think I am saying that my proposal is the only means to avert
a danger I perceive.
What I am claiming is that there is specific expertise Sage is
currently not tapping.
Sage is entering a completely different phase of its evolution, where
it becomes "established". This can be good in some ways, but comes
with many dangers.
So let me add this. Sage did not begin by saying that we must be right
because of all the contributors we have. Remember the hard, lonely
work for over a year that you did. So Sage's success is not due to the
number of contributors it has. It is successful due to the hard
decisions that were made about things like architecture and the like,
and the hard work of you and lots of people over a long period of
time. That is the reason for the success Sage has. So the argument
that Sage is by definition right in the way it does things because of
all the current contributors is broken.
This is the danger of becoming established. Superstition and being
established in the old english sense of the word, like the wheel of a
carriage becoming established.
Bill.
2009/8/25 Bill Hart <goodwi...@googlemail.com>:
> People can follow or comment on sage-flame if they like. Hence I
> remove the CC's.
>
> 2009/8/25 William Stein <wst...@gmail.com>:
>> Hi,
>>
>> Bill Hart wrote me an enormous email strongly criticizing me and much
>> of the Sage project in great detail. With his permission I've
>> responded below, cc'ing this to sage-flame. I've also cc'd it to a
>> few people I think might find this interesting because they or their
>> projects are mentioned, or they might just enjoy reading somebody
>> criticizing the Sage project at Herculean (!) length.
>
> Not so much criticism of you. Nor of Sage really. Rather, some things
> which I think are suboptimal.
> Strangely, the first time you said this to me, I thought you were
> referring to me in the trenches, and maybe there were aspects of
> things you hadn't picked up on re things I had found difficult,
> working in those trenches.
> Yeah, there's a whole pile of contributors to FLINT and MPIR too,
> heaps of people signed up for the devel lists. Sage is a project with
> absolutely massive scope, covering an enormous number of areas. Of
> course you are going to have stats like this. This is a non-argument
> and does not change the "immutable truth".
>
>>
>> Again, consider your "immutable truth":
>> "People will not contribute to open source projects unless you make
>> it absolutely trivial to do so, and *any* encumbrance, no matter how
>> slight will prevent them from doing so".
>>
>> It's a good thing mathematical research journals don't believe in a
>> similar immutable truth.
>
> This is not about building prestige or discouraging unworthy
> contributions. It is about fostering an environment which is makes it
> easy for stellar contributors who may be very busy with many other
> things, to not have any excuses to do so.
>
>>
>>> I just spent the afternoon here at Warwick talking with a visitor who
>>> has precisely this problem.
>>>
>>> You keep making the argument that Sage is so successful, therefore
>>> what we are doing is by definition right. This is called superstition
>>> - the desire to not change anything because what we have has clearly
>>> worked so well for us in the past.
>>
>> superstition = a belief or notion, not based on reason or knowledge.
>
> Right.
>
>>
>> Looking objectively at whether or not a workflow or process is working
>> by some objective measure, e.g., "number of contributors to a
>> release", does not seem to me to be the opposite of superstition.
>
> So any particular thing related to the sage project is by definition
> right, just because of said number of volunteers having contributed.
> That's broken logic.
>
>>
>>> But you are not seeing the people who are *not* contributing to Sage
>>> because something put them off. Whether it be people not wanting to
>>> put their HG repos up for others to pull from for fear of
>>> embarrassment, or because people are unable to get patches in because
>>> the codebase changes in ways that make it difficult for them to
>>> rebase, or because of how daunting the code review process is, you may
>>> not see people turning away from Sage because of these issues, but I
>>> do. I see them all the time. In particular some of these issues really
>>> affect the number of permanent/tenured/faculty who could be
>>> contributing.
>>
>> The goal of the Sage project -- which I started -- is not to maximize
>> the number of contributors. The goal of the Sage project is one
>> thing:
>>
>> Mission Statement: Create a Viable Free Open Source Alternative to
>> Magma, Maple, Mathematica, and MATLAB.
>>
>> You might be confusing two distinct goals: get big, accomplish
>> something; because, your projects -- MPIR and FLINT -- have few
>> contributors, so you are (rightly so) focused very much on that, which
>> you see as a problem.
>
> Perhaps you didn't get the thread of my argument? In order to compete
> with those packages, Sage needs very strong C library support. This is
> an important part of Sage, just as Cython, the Sage library and other
> aspects are important. At present, numerous things are suboptimal with
> regard to supporting those libraries. You failed to respond to that
> critical point, nor did you seem to appreciate in any way the
> constructive suggestions I made regarding that.
>
>>
>>> You might view all this as some flimsy attempt to divert attention
>>> from what is really the problem for me, which is lack of contribution
>>> to MPIR and FLINT, which is, as you put it, a problem with suboptimal
>>> things with regard to those projects from the point of view of
>>> contribution. The really odd thing for me is that one of the things
>>> you recommended was git. You had me watch Linus' vid about it. I
>>> assumed you had plans to go this way with Sage, as git is really
>>> technologically superior, and the entire model that it encourages
>>> "branch early, branch often and pull not push" is much, much, much,
>>> much, much, much better for open source projects. But when I asked
>>> about this issue, you said that Sage is not a big enough project yet,
>>> and too many people come and go. WTF!?
>>
>> You specifically asked about having lieutenants responsible for each
>> subdomain in Sage, and having them each act as release managers for
>> their domains. I do not think *that* will work right now. Since I
>> think I already justified that at length, I won't go further.
>>
>> By the way, though I recommended Linus's talk, it doesn't mean I
>> endorse everything he says. I always read the book "The Cathedral and
>> the Bizarre", which is a wonderful book, but again, I definitely do
>> not agree with everything it says. When I suggest that you look into
>> some resource, I'm not saying it is the bible with all the answers,
>> but something to read/listen to/consider, and then think through
>> carefully for yourself.
>
> That's precisely what I am saying I did. I read and consulted further
> resources, as I mentioned, and considered the matter in depth for
> months before writing the above. I've had time to discuss it with
> people, including you, ruminate on it. What I put in this email was my
> considered response.
>
>>
>>> I actually laughed, quite hard, when you suggested I had not "got" the git.
>>
>> I'm sorry if I offended you. It wasn't in the least bit my intention.
>>
>>> Anyhow, instead I've seen precisely why I'm not getting contribution
>>> to FLINT and MPIR. The main reason is that both projects do not have
>>> sufficient resources by themselves. Open source low level number
>>> theory software hacking is very, very specialised. It's not like
>>> writing an OS or a Matlab replacement, or a graphics program, where
>>> you have a large pool of potential high quality contributors. You know
>>> all the people who are going to contribute. Some of them don't like
>>> you. Some of them can't contribute because of restrictions their
>>> workplace places on them. Others are committed elsewhere. So, no
>>> quantity of FAQ's, changes to DVCS, development documentation, webpage
>>> resources, development groups, etc, etc, etc, is going to bring
>>> significant quantities of contribution to FLINT or MPIR. I could spend
>>> the next year writing such stuff and the net effect on FLINT and MPIR
>>> would be minimal. And my institution will not allow me to squander my
>>> time doing these things. I need to publish.
>>
>> The first ever criticism of Sage was by Richard Fateman back in 2005
>> that Sage would fail because writing advanced mathematical software
>> requires too specialized of a skillset, and that there was no way we
>> would get enough contributions. It was valuable criticism, and I
>> learned a lot from it. Your argument above about FLINT/MPIR is
>> similar.
>
> My main point here is that I don't think Sage is getting enough
> contributors. I believe in some areas, critical areas, we are actually
> quite thin on the ground. And I think there are clear identifiable
> reasons for that.
>
> I do not identify with the criticisms of Fateman, on the whole. It
> worries me that the guy is planning an assault on Sage by writing the
> definitive critical article on why Sage is hopeless, as he has done
> for the other mathematical packages out there, as he boasts.
>
>>
>> Sorry, but I just don't buy it for a second, because I'm incredibly
>> optimistic about people's abilities. I think this has a huge amount
>> to do with how I was raised -- I vaguely recall people pointing out to
>> my brother and I when we were young that we were vastly more
>> encouraging toward the other kids than they were used to seeing.
>
> No argument from me there. You are a very positive encouraging person,
> and perhaps the usual psychology does not work with you. The standard
> approach in making an argument is to identify a problem, expand upon
> that problem showing how it can lead to negative things, then present
> a potential solution for consideration (and then maybe modification,
> rejection, whatever, and also possibly acceptance and adoption), then
> extol the virtues of the presented solution. In your case you perhaps
> view all negative as bad.
>
>>
>>> You see the same thing in Pari, NTL, GMP, presumably Maxima and
>>> numerous other packages, even LiDIA died. Sage is in fact contributing
>>> to, not alleviating emaciation of these libraries and programs.
>>
>> I don't buy that for a second, and Michael's argument below is spot
>> on. That's like saying the Linux kernel is emaciated by the existence
>> of Ubuntu.
>
> Michael's argument (no offense Michael), is rubbish. I can tell you
> from working in the trenches. It's rubbish.
>
> False analogy with the linux kernel. That's a many to one mapping,
> whereas Sage to libraries is a one to one mapping. It happens to be an
> important one for MPIR and FLINT. But that is unimportant.
>
>>
>>> Michael used to say often that so many packages out there are
>>> benefiting from Sage, because of all the bug reports and patches that
>>> were going back upstream. But from the point of view of a library,
>>> this is not contribution, but merely more stuff in the inbox. Fix
>>> this, fix that, port this, port that, support this system, and why
>>> won't anyone help with the Sage port to Windows...
>>
>> So ignore that email if your project doesn't have the organization to
>> handle it. My experience, and the experience of many other people
>> I've worked with, is that upstream projects are often thrilled to
>> learn of the existence of bugs and ideas for their fixes. I've never
>> heard of a quality software project that doesn't greatly value bug
>> reports.
>
> There is a value to bug reports. I can't respond to this seriously, as
> I don't think you were being serious. Of course we are talking about
> the level of contribution here.
>
>>
>>> Here's my perspective. Sage is a community of volunteers all working
>>> toward the same aim - to build a mathematical software package.
>>> Contribution comes in many forms, but most significantly code
>>> contribution in libraries and in the Sage python and Cython code.
>>
>> Maybe. This is a function of time and the specific mathematical area.
>> Robert Dodier and Andrej Vodopivec made a *huge* contribution to
>> symbolic manipulation in Sage today in the form of a 1-paragraph
>> email. http://trac.sagemath.org/sage_trac/ticket/6818/. This one
>> little thing will improve performance for people more than tons of
>> other work people have done.
>
> Thanks Robert and Andrej. This is a stellar contribution. I hope there
> will be much more of it!
>
> But let me cheekily apply your (William's) journal standard here.
> Looking forward to those one paragraph papers that change the face of
> mathematics.
>
>>
>>> Those are the core aspects of Sage development. In many ways, Sage has
>>> subsumed things like NTL, Pari, Maxima and other libraries (even GMP),
>>> because their maintainers do not have the resources to maintain them.
>>>
>>> But instead of seeing these as all part of the "Sage project", only
>>> the Sage python library code is being given first class status (in
>>> some respects - yes I accept that in other respects they are given
>>> very first class treatment). One very influential Sage devel recently
>>> shared with me their vision of Sage: get rid of most of the C
>>> libraries and have everything implemented in Cython code. I mean,
>>> WTF!?
>>
>> You are suggesting that design and development of Sage should proceed
>> with careful sensitivity to politics, "status", "class", etc. I
>> absolutely disagree with this. If a developer has a good idea that
>> leads to Sage being a technically better system, then go for it!
>
> Proceeding with careful sensitivity to politics, status, class, is
> imperative when dealing with human beings. That is especially true
> when your project relies on people giving their time for free and
> where such contribution is the lifeblood of the project. Sage is about
> people and tapping that human resource in an effective way.
>
>> Sage is not some feel-good European Union project. It's about
>> technical excellence. If somebody can create a Cython replacement
>> for some existing C library and this new approach is technically
>> superior, I don't see what the problem is. Especially if they are
>> doing this work for free/fun! More power to them.
>
> This is clearly flamebait. But I won't bite. But I can't prevent
> others from responding to that. :-)
>
>>
>>> One person bills Cython as "C without the semicolons", another bills
>>> it as "a dialect of python", another as a "smooth blend of python and
>>> C++". I've heard other people refer to it as a "Frankenlanguage" or as
>>> "glue code" or "what the 'eck is that".
>>
>> "Cython is the ideal language for wrapping external C libraries, and
>> for fast C modules that speed up the execution of Python code." --
>> cython.org
>
> Clearly advertising language, but I don't disagree with the sentiments
> behind that mission statement, if that is what it is.
>
>>
>>> To me, cython exists for a couple of reasons (apart from the
>>> brilliance of its designers), most significantly because python is too
>>> slow for many purposes. But if cython really is just C without the
>>> semicolons, then why not encourage people to write straight C++? It's
>>> fast, apparently no more difficult that Cython, since you just have to
>>> add semicolons, and you can still apparently call C++ code from
>>> Python. Python can then be the "glue code".
>>>
>>> But as you put it, one of the reasons for the great success of Sage is
>>> the simplicity of the python language. People want to, and can
>>> contribute because python is so easy, and C++ is so "hard". So it is
>>> pretty clear that Cython is *not* the future of Sage, unless you have
>>> a whole load of people dedicated full time to going through python
>>> code and rewriting it in cython. Tough titties that this is never
>>> going to happen.
>>
>> I can only conclude that you haven't written much Cython code.
>
> Correct.
>
>> Actually, have you contributed even one single line of code to the
>> Sage library?
>
> You mean Cython? No. I could also cheekily suggest that I didn't see
> your name high up there on the FLINT or MPIR contribution list though.
>
> Anyhow, this is again flamebait. You know precisely the quantity of C
> code I've contributed to Sage.
>
>> I'm not asking that to imply that you are somehow not
>> a "first class citizen", but to suggest that without experience you
>> might have some misconceptions regarding the use of Cython in the Sage
>> library. It is often a good idea to remind oneself that things look
>> a lot different from the trenches.
>
> Right. Not sure where you are going though. What are you actually
> responding to here? Are you arguing that replacing the C libraries in
> Sage with Cython actually is the future of Sage?
>
>>
>>> People will always contribute python code, not cython if that is their
>>> level of their competence. Otherwise you are going to have people like
>>> me contributing C/C++/Assembly, because we can, and because it is
>>> faster and better than python for that kind of stuff. That being the
>>> case, ought not the C/C++ library part of Sage be split into
>>> discernible subprojects *of the Sage project* which receive equal
>>> attention and care to the rest of Sage, much like the core of Magma,
>>> but without the monolithic 2 million line bit?
>>
>> What would this even mean? Do you want an account on
>> sage.math.washington.edu? Do you want access to "exclusive build
>> farms"? Do you want space on the sagemath.org webpage? Do you want
>> me to assign my "army of minions" to fixing bugs in MPIR?
>
> I spelled out what it could mean. I know you don't have some army of
> minions. I'm really puzzled by this. Is that what you think I am
> asking for!?
>
> My assumption is as yours. Build it and they will come.
>
>>
>> I don't have an army of minions. I don't tell *anybody* what to do on
>> Sage, except people who work directly for me, and even then I hardly
>> do. I'm the opposite of John Cannon, who takes a lot of care and
>> pride in orchestrating much of Magma's development (and given what
>> Magma has accomplished, he has done a very impressive job). Right now
>> are four people getting paid to work on Sage: three UW undergrads
>> working part time on some coding projects:
>>
>> 1) a very nice units package in pure Python
>
> Great. That will presumably be of interest to physicists? I think I
> saw some postings about this.
>
>>
>> 2) a competitive implementation of Miller-Odlyzko-Lagarias for
>> computing pi(X), which he's deciding he is writing in C:
>> http://code.google.com/p/primecounter/
>
> Is this the algorithm Miller used in his (ancient) primepi code? OK I
> see the link, I can check it out, thanks.
>
>>
>> 3) another who is polishing various things related to graphics,
>> including HTML canvas rendering related to 3d graphics.
>
> I know of people who are really looking forward to this.
>
>>
>> Also, my NSF grant fully supported the development of the mpmath
>> Python library this summer.
>
> Yep, Fredrick is a star.
>
>>
>>> Cython has its place and do not misunderstand me. I am not suggesting
>>> that in some way Cython is not a vital part of Sage and the Sage
>>> community. I am only suggesting that people like me are not going to
>>> use it for everything because we are first of all not PFL (python
>>> first language) speakers and thus couldn't give a toss that Cython is
>>> "so much like python", and second of all because people like me can
>>> use C++ and occasionally need to do some optimisation in assembly, so
>>> we will.
>>
>> My humble hope is merely to increase the chances that you will have
>> the choice to not use Magma or Mathematica if you don't want to.
>
> No offense to those other projects, but I will never use Mathematica,
> and whilst Magma has many impressive features, it is not expansible
> enough for me, and violates those principles of openness which I
> believe are vital for scientific endeavour. I am on record as having
> those opinions.
>
> That's the reason I care about Sage. But this humble dream won't be
> met without the right people making the right contributions. As I
> said, many things are right. I'm mentioning some things I believe are
> wrong with respect to fulfilling that dream.
>
>>
>>> If that isn't a reasonable argument, and doesn't have consequences for
>>> the architecture and revision control and packaging of Sage wrt its
>>> libraries, then I just don't see how Sage plans to move forward in the
>>> future. I just see a train wreck coming in about 5-10 years. I see a
>>> whole pile of dead unmaintained libraries packaged along with a *lot*
>>> of python code, which someone one day says they are going to
>>> cythonise.
>>
>> I've had numerous well-meaning people tell me they see a train wreck
>> coming for Sage in X years. I have greatly appreciated that they took
>> the time to share their concerns (e.g., Richard Fatemen was one such
>> person, but there have been many). I've been hearing this from
>> people since Sage was first released. For the first 3 years I
>> worked on Sage I actually believed it too, and told myself that I was
>> crazy putting so much of my time and energy into Sage. I knew for
>> years *myself* that Sage is impossible and would never go anywhere.
>> What the community has ended up creating so far has far surpassed my
>> wildest expectations.
>
> It is very large, very broad and has many pockets of really stellar
> stuff in it. However I believe I spot some gaps. I pointed out some
> very explicit ones:
>
> * Fast multivariate polynomial arithmetic
> * Decent multivariate GCD
> * Decent polynomial factoring of any kind
>
> I could add others. I am not the first developer to point out these
> deficiencies. But how to fix specific deficiencies like this? We need
> contributions from the right developers in the right way. But how to
> encourage that specific kind of contribution. Hmm.
>
>>
>>> All the library writers will have moved on.... and sage will be full
>>> of libraries no one knows how to maintain.
>>
>> There is tons of Cython code in Sage written by people who have now
>> moved on, and who don't read sage-devel anymore, don't pay any
>> attention to Sage right now, are finishing their thesis, etc. And
>> yet there were over 1,400 messages just on sage-devel last month, 56
>> contributors to the last sage release, and sage development just keeps
>> marching on.
>
> Many, many of these messages on sage-devel are about technological
> issues and bugs, not about mathematics per se. How does one even make
> a statement like that to an optimist? "Yes, it is fantastic that so
> many bugs are being sorted out and discussed, and it shows the healthy
> concern for the sage project that its many contributors and supporters
> have. That's great, and we should build on that by discussing lots of
> mathematics and algorithms". There, how did I do?
>
>> Instead of arguing as you are based on some secret
>> logic, I look at actual numbers and conclude that the Sage project is
>> healthier than ever, though of course it still isn't near succeeding
>> at its mission statement.
>
> I think you are an optimist, and possibly view me as a pessimist. In
> fact I am always somewhat of a realist and I am certainly not the only
> person with the concerns that I have, not by a long shot.
>
> And this really is about Sage achieving its mission. I think you do
> see that, but you aren't saying it explicitly.
>
> The numbers show that lots is right. But other numbers show that much
> more needs to be done.
>
>>
>> This constant "moving on" of people is natural. It is critical to
>> come to terms with this fact, because it has a big impact on design
>> decisions.
>
> Precisely the motivation of some of the suggestions I made.
>
>>
>>> The maxim that there are plenty more people out there to do the work,
>>> so it doesn't matter if people move on, is just not true when it comes
>>> to writing highly optimised assembly and C code. It is probably true
>>> of python code, but hacking low level number theory software is
>>> specialised and we know most of the people who do that.
>>
>> I'm definitely far more optimistic than you are here.
>
> Right, but I said we, not I.
> Interesting. I had 34s for python. Oh, wait, my for loop wasn't
> completely empty, in any of the cases I timed. So a factor of 34 for
> python. Pari I had 25s, but that was of course not the latest pari
> interpreter, which does use a bytecode interpreter and is obviously
> some substantial factor faster. My time of 1s for a bytecode
> interpeter was an upper limit. I haven't actually finished writing it
> yet.
>
> Lua meets the 1s, so I know it is possible. But yuck, they use doubles
> for integer arithmetic and have no native integer support, even on
> x86. Bizarre language.
>
>>
>>> Anyhow, the above model also makes generic programming really
>>> straightforward for things we care about, like arrays and matrices
>>> defined over other basic objects (and over themselves, etc) - and many
>>> times faster than python.
>>>
>>> If you think about it, this is what a user of Sage is going to really
>>> care about. It isn't whether Sage can compute the number of partitions
>>> of n really really fast. It is whether they can get their algorithm,
>>> which they've just implemented in the python/sage language to run
>>> really really fast. But they are mystified why it doesn't until they
>>> discover that python is actually really, really slow for basic things
>>> like iterating over the integers....
>>
>> I have a much different view of what "a user of Sage is". That said,
>> let me emphasize yet again the goal of the Sage project: be a viable
>> free open source alternative to Magma, Maple, Mathematica, and Matlab.
>> Doing something new and different in the direction you suggest
>> doesn't really address that goal.
>
> I certainly disagree with that.
>
>> Part of the goal of Sage is that it
>> is not supposed to be a "research system" (like Linbox was for years),
>> but a system that is ready for everyday usage ** as soon as possible
>> **.
>
> I am like you in that I need something for everyday use, as soon as
> possible. But there is a necessary process in achieving that. But
> there aren't easy answers to all problems, and we've certainly proved
> that, especially when performance is considered.
>
> I am not suggesting Sage become a "research system", and have never
> taken that attitude with maths software.
> This is a compiler they are talking about. A whole different kettle of
> fish. And correct me if I am wrong, but I am not sure there is much
> development on the compiler side itself at present. I have spoken with
> the mathemagix developers in the past, and there are some very
> impressive attributes. But what they are aiming at is a very long term
> project. Writing compilers, especially efficient ones is a very hard
> task. I'm *not* talking about a compiler.
>
> Don't get me wrong, an expert in using flex and yacc/bison could get a
> compiler up and running in a few days. I'm not such an expert. I could
> probably do it in a couple of months. But I don't have that much time,
> relative to the other things I am working on.
>
> The difficulty with a compiler is what you compile to. If you want it
> to be efficient and useful, you are going to compile to assembly, and
> you have a large number of architectures to cover. And then you've got
> keyhole optimisation to worry about and all sorts of other nasty
> things. Really, you need 100's of people to work on a compiler project
> for it to be really effective.
>
> Don't let me put anyone off, of course. And the expertise is out
> there, much more so than hacking on number theory software. But this
> is not something I am interested in.
>
> A bytecode interpreter can be a thousand lines of code and grow from
> there as needed. Give me a few weeks to finish my prototype.
>
>>
>> It's built on the libraries you mention above, it will have a
>> compiler, etc. It sounds like what you want. Just keep in mind that
>> building something like this is really hard. I think the Mathemagix
>> project is *twice* as old as Sage! I would be much more interested
>> in that project if I wasn't so impatient (at this particular problem
>> -- I'm patient at math research).
>
> Me too. Perhaps one day I will find the time to become more interested
> in the compiler they propose.
>
>>
>>> I have to confess that in the final analysis, when I come to use
>>> Sage/FLINT/MPIR/other libraries for my number theory research, if
>>> something is not implemented, if it is a basic type, I am going to
>>> implement it in assembly or C in FLINT or MPIR or some other library.
>>> If it is generic, I'm likely to implement it in C++ and stick a hook
>>> into the bytecode syntax. Only really high level stuff is going to be
>>> in python, where performance just doesn't matter. So doing something
>>> like what I propose above isn't going to get me using heaps of python
>>> probably. I don't think for example that I'm going to implement ideals
>>> of number fields in python for example. But I might type my high level
>>> algorithm which uses ideals into Sage in the Sage language, which
>>> might be a thin extension of the python language, as it is now, and it
>>> will call the ideal code, which will be written in C++, via the
>>> bytecode.
>>
>> Just for the record, you are definitely not the typical target
>> audience I have in mind for Sage.
>>
>
> That's definitely ok.
>
>>> Anyhow, obviously I envisage something different and perhaps much
>>> better than what we are currently doing. Python is right (actually
>>> Ruby is really nice, but I doubt it was around in the current form
>>> when Sage started). There isn't a better language, all things
>>> considered, to use for Sage than python. The focus on performance is
>>> right. Open source is right. The community, Sage Days, involvement of
>>> students, including undergrads, the notebook concept is right. So lots
>>> is right about Sage. But some things are wrong - very wrong, and are
>>> going to really bite in the long run. I've mentioned some here, some
>>> before.
>>
>> At this point, I find myself unconvinced that anything is "very wrong"
>> with Sage. Obviously there are many technical aspects of the
>> software that are far from "best possible", but that's just life when
>> one has limited time.
>>
>>> I really see some parts of Sage as lacking direction.
>>
>> It's a big open source project with almost entirely volunteer effort.
>> Yes, there is a lack of top-down direction, since if I started telling
>> people what to do (much like you are trying to tell me what to do
>> right now), that wouldn't go over very well, to put it mildly. At
>> least you are expending the effort to careful explain and argue for
>> why I should do what you want me to do. But at the end of the day,
>> you're telling me: "do this or else your project is doomed". I don't
>> feel comfortable telling other people that, and if I gave you the
>> impression that I did that with MPIR, I sincerely apologize.
>
> No, there's no sense that this is what I thought you did with MPIR.
>
> I'm not trying to tell you what to do. I'm making a proposal and
> trying to garner support. Obviously I did a very bad job of that. I
> only need to look at the name of the group I am now posting on to see
> that.
> Yep. One thing you can count on. Whenever I propose something, it is
> rarely because I want you personally to do it. OK, there are rare
> cases where I need something that *only* you can do and hopefully it
> consumes a few minutes of your time. But never with a proposal like
> this am I asking whether you want to do it.
> Firstly, you do have tremendous influence. Secondly I am not expecting
> you to use your influence to make a blanket change. It's a proposal.
> Proposals need discussion. What better person to discuss it with than
> the project leader.
>
>>
>> You may want to create a git repo like you suggest above, argue
>> carefully to the community for why people should use it, and see what
>> happens. That said, watch out, since many important people don't read
>> sage-devel anymore, since it is such a high traffic list (and also
>> because some people's brain's are broken and they don't know proper
>> email netiquette).
>>
>
> Sorry I don't know what the references to broken brains and email
> netiquette is.
>
> I obviously got a big -1 from you with regard to all of the
> suggestions above. What's really killing me is that it is really your
> influence on me that set me down this track in the first place. How
> ironic that it is you that seems to think it not worth exploring
> further.
>
> Anyhow, I've learned from experience not to pursue projects with
> negative press as community projects. So it's dead now. Well sort of
> dead.
>
> I will be using git and publishing repos for FLINT and MPIR. And I've
> obviously worked on code for a bytecode interpreter. Probably a couple
> more weeks to get the prototype working, depending on whatever else
> intervenes. Such a thing doesn't need a place in Sage. It can happily
> live on my hard drive and I can use it to impress girls at parties.
>
>>> Anyhow, enough for now. For all I know, you've been reading this and
>>> thinking I'm off with the fairies, or just having a bad day (actually
>>> I've had a very, very good week and gotten all excited about where to
>>> next with FLINT and MPIR - you'll see what that is about in 3-6
>>> months). But anyhow, if you do think I'm crazy, please understand I've
>>> given this one hell of a lot of thought. I'm prepared to put in the
>>> effort to carry out what I'm proposing. But on the probability that
>>> I've completely lost you, I'll stop here, as more explanation is not
>>> going to help if that is the case.
>>
>> Unfortunately you have completely lost me. I'm sorry. I wish it
>> weren't so. As I see it, the fundamental flaw in your argument is
>> that you vastly overestimate my relevance to the Sage project.
>>
>
> Nope. No flaw there. You just had a major influence.
>
> Bill.
>
--
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org
Glenn Tarbox, PhD
From: Glenn Tarbox, PhD <gl...@tarbox.org>
Date: Mon, Aug 24, 2009 at 10:26 PM
Subject: Re: [sage-devel] spkg trouble
To: William Stein <wst...@gmail.com>
<<< Snip everything but.... >>>
Unfortunately you have completely lost me. I'm sorry. I wish it
weren't so. As I see it, the fundamental flaw in your argument is
that you vastly overestimate my relevance to the Sage project.
Ah.
There are lots of things said about things said and having been proved wrong. And some discussion which essentially boils down to the last sentence... (or at least its the point I'll address)
My starting point is:
Sage == Stein
This isn't true, but lets see where it leads just for kicks.
First, clearly you did it, and did it in spite of the clear and simple "fact" that it couldn't be done.
(Sure, there's lots of help, huge amounts of third party code etc. But the audience of this email knows all the gory details... so lets suspend that and consider it a given... you also used some silicon you didn't process etc)
Second, you know more about Sage than anyone else. There are some with an intimate knowledge, but that is a small very very tight knit group and (I claim) pretty much in awe. Hell, I'm in awe. You da man.
Third, if you're not in agreement or behind something, it ain't gonna happen. This is really Second part 2 or an unavoidable consequence.
And, whats wrong with that? Ideally, you'd let lose your "mental baby" on the world and it would evolve in the ecosystem of the open-source plasma and, eventually, the goal of Sage would be realized. And, that may indeed happen.
But, for the time being, through no fault of your own, Sage == Stein. Hence, if Stein doesn't back it, and it isn't a bounded effort or something which doesn't affect core elements of Sage, it ain't gonna happen. Perhaps you'd have it be different, and I truly believe you wish it were different.
So there's a problem. Its hard to state clearly, and I'm not sure I follow all of Bill's issues, but some are clear to me (or I project some of his thoughts into my understanding of the issues). That problem, beginning to become important, is a projection of the past into the present and, hence, into the future.
A very important point you didn't address, in fact, one which you answered with an example of the problem, is that since Sage was destined to fail... and as everyone was wrong about that, anyone who identifies what appear to be major flaws could be wrong.
Thing is, you could be absolutely right. I suggest that however badass a programmer / mathematician you may be (and I think you're a pretty serious badass), you can't know everything. Not wrong, just not know. And, how could you? This ts not an attack on you or Sage (irrespective of the Sage == Stein formalism, it doesn't extend beyond reasonable limits of the Stein space)
My concern is simple, but orthogonal to Sage in its current form. Architecture is always present. Architecture exists whether it is designed with intent or evolves organically. Organic is great and is often the result of the Agile process (regardless of what flavor of agile) but Agile assumes that refactoring is OK... Otherwise, its Waterfall, and we know that doesn't work. But, Waterfall has something going for it, the design follows intent.
So Sage matures organically and gets pretty big... but it wasn't "intended" to be this or that. In my case, it wasn't intended to be a distributed component system able to handle asynchronous events. But it could be very easily re-factored to be that (in fact, I've done that a couplea times... hacks all, but easy nonetheless)
But that type of system design isn't just not in Sage, the concepts are entirely outside the Sage space (I'll leave Sage == Stein behind... its silly and probably not defense-able so I'll claim it isn't true and continue on)
So, whats a person to do? Clearly, Sage is full of smart guys... but here comes Tarbox talking all kinds of crap about asynchronous this, and threading == bad that... and badness with the notebook etc. Tarbox is kinda off balance. He's not offering an "opinion", he's expressing an "expert opinion" based on 15 years doing nothing but arguing the issues of distributed asynchronous systems and some things are just true.
Of course, it took years to get to where he's at, and given the audience he's worked with, no longer knows how to go back to square 1. All Stein needs to do is utter something almost trivially negative, and Tarbox is toast.
Process is another. I don't know where it came from or why the "community" chose it, but Sage's development process needs work. I'm not a genius and making this crap up, I'm basically distilling what I know to be true from years of working large programs and what it really takes to mash systems together. What doesn't work is centralized control of the CVS / SVN style... which is what Sage has in spades.
The two issues above are very very subtle when speaking to those who haven't been exposed to them. And its very very difficult to understand why DVCS works, and why asynchronous systems work... and why "best of breed" is less important than "what won". What Won is massively more important than what is technically superior (anyone have a Betamax?)
But, an outsider has no way to get a toe-hold unless Stein makes it so.
I claim that Sage is at a critical stage. To continue growing, issues beyond the core competence of the driving forces of Sage become increasingly important. Architecture and process are, in my opinion, two of those issues...
But, there's no way its gonna happen if Stein doesn't get behind it. Not with work... but with approval.
You're in charge, whether you wanna be or not.
Humbly (yea right :-)
-glenn
-- William
--
Glenn H. Tarbox, PhD || 206-274-6919
http://www.tarbox.org
--
Glenn H. Tarbox, PhD || 206-274-6919
http://www.tarbox.org
William Stein
> Maybe one thing does require further comment. It bothers me that you
> think I'm writing to you and saying do this or your project is fucked.
> I think I failed to connect with you on any level at all if that is
> what you think. And that surprises me. That seems to suggest to me
> that you think I am saying that my proposal is the only means to avert
> a danger I perceive.
Yes, you are right, that is exactly what I got from your message. It
did not connect with me. If you feel as you said above, I evidently
completely missed the point, and then some. The best I can suggest
is that we just take a breather and try again later, or try instead on
the phone or in person.
> I obviously got a big -1 from you with regard to all of the
> suggestions above. What's really killing me is that it is really your
> influence on me that set me down this track in the first place. How
> ironic that it is you that seems to think it not worth exploring
> further.
Look, if you really have a good idea and the whole problem is that I
don't immediately "get it" , maybe we should rethink how we are
communicating instead of just giving up?
-- William
William Stein
> Of course, it took years to get to where he's at, and given the audience
> he's worked with, no longer knows how to go back to square 1. All Stein
> needs to do is utter something almost trivially negative, and Tarbox is
> toast.
From my perspective, people constantly disagree with me all the time,
and frequently I loose in discussions. This happens every single day.
Glenn, I would like to very *strongly* encourage you to not be so
easily discouraged on sage-devel (or elsehwere) when I say something
"almost trivially negative" about an idea or suggestion you have. I
(and I'm sure much of the Sage community) really values your feedback
and contributions. I'm really sorry that the way I have posted my
concerns about your ideas has been so discouraging to you, and I'll
try to be more supportive of your suggestions in the future.
-- William
ghtdak
On Aug 24, 11:04 pm, William Stein <wst...@gmail.com> wrote:
>
> > I obviously got a big -1 from you with regard to all of the
> > suggestions above. What's really killing me is that it is really your
> > influence on me that set me down this track in the first place. How
> > ironic that it is you that seems to think it not worth exploring
> > further.
>
> Look, if you really have a good idea and the whole problem is that I
> don't immediately "get it" , maybe we should rethink how we are
> communicating instead of just giving up?
-glenn
Ondrej Certik
>
> Bill Hart wrote me an enormous email strongly criticizing me and much
> of the Sage project in great detail. With his permission I've
> responded below, cc'ing this to sage-flame. I've also cc'd it to a
> few people I think might find this interesting because they or their
> projects are mentioned, or they might just enjoy reading somebody
> criticizing the Sage project at Herculean (!) length.
Ok, I just spent 30 minutes just reading all of this. Few comments
William described how you did no Sage development over the summer
(doing real research instead) and then:
> In the meantime, Sage development chugs on amazingly, with the last
> release having nearly 100 contributions by 56 people, etc., etc., and
> I had almost nothing to do with that.
I think this is not true at all, that you had "almost nothing" to do
with that. Look here:
http://groups.google.com/group/sage-devel/about
and see the number of posts by different people and the top poster
(William Stein). In fact, it precisely matches my experience with
sympy. I didn't do any real development for 8 months already, just
browse the sympy log, who wrote most of the patches since April 4,
2009:
77 Fabian Pedregosa
58 Ondrej Certik
25 Aaron Meurer
22 Toon Verstraelen
17 Vinzent Steinberg
15 Priit Laes
7 Chris Smith
6 Alan Bromborsky
5 Luke Peterson
5 Nicolas Pourcelot
5 Tomasz Buchert
4 Florian Mickler
4 Jochen Voss
and then long tail of people fixing little things, with just a few
patches (about 10 more). My patches are mostly adding people to
credits and other book keeping things.
yet sympy is doing better than ever:
http://groups.google.com/group/sympy/about
in June we had 500 posts etc. We got a release, we had lots of people
actively working on sympy this summer, sponsored by google etc. We
have two big branches on review (ODE module, polys module)... But it
*does* require something from me --- I need to keep it going by
replying on the list, by managing the google summer of code stuff,
etc. I should spend more time pushing the reviews etc. I still spend
less time on sympy than william on sage, just judging from the number
of emails.
So I don't believe at all, that Sage would run as it is running now,
if you William stopped posting to the list and stop managing things.
Second comment is that yes, I also hear or heard negative and
discouraging comments from the Sage project (not from William) or Sage
users with regards SymPy, that it is slow, it will never get there to
be of "industrial strength", etc. But I don't care, I really do
believe in what I do and we will succeed to have a usable and fast
symbolic library in Python. My experience is that when I am positive
and full of energy, people just come and help out with sympy. And I
also have the same experience as William that old people leave and
lots of new people come. One example where I think we are doing way
better than Sage is that thanks to having core of sympy in Python,
even a high school student can fix the derivatives stuff (during
google GHOP). I watched that it took several weeks for a Sage
user/developer to fix derivatives in Sage, due to the fact that he had
to hack C++, Cython, etc. So in this area, it is way easier to
contribute to sympy than to sage. Sage is currently faster for some
operations, so sympy is not an option for some people. We will fix or
lower this speed difference eventually though.
Third comment is that I think Sage has alienated lots of people by
it's attitude towards other projects/companies, and I think it's
unfortunate. I think it will get better over time though. I hope,
because I depend on Sage with my other projects (FEMhub). I just came
from the scipy 09 conference, and people know Sage and use Sage and
it's not like that they hate it. But they are not terribly excited
either --- simply because they were discouraged by some remarks or
attitude of the Sage project. But as I said, I hope it will improve.
Ondrej
William Stein
but even so keeping up with email is really important. Thanks for
pointing out that you had the same experience.
> Second comment is that yes, I also hear or heard negative and
> discouraging comments from the Sage project (not from William) or Sage
> users with regards SymPy, that it is slow, it will never get there to
> be of "industrial strength", etc. But I don't care, I really do
> believe in what I do and we will succeed to have a usable and fast
> symbolic library in Python. My experience is that when I am positive
> and full of energy, people just come and help out with sympy. And I
> also have the same experience as William that old people leave and
> lots of new people come. One example where I think we are doing way
> better than Sage is that thanks to having core of sympy in Python,
> even a high school student can fix the derivatives stuff (during
> google GHOP). I watched that it took several weeks for a Sage
> user/developer to fix derivatives in Sage, due to the fact that he had
> to hack C++, Cython, etc. So in this area, it is way easier to
> contribute to sympy than to sage.
than Sympy.
> Sage is currently faster for some
> operations, so sympy is not an option for some people. We will fix or
> lower this speed difference eventually though.
>
> Third comment is that I think Sage has alienated lots of people by
> it's attitude towards other projects/companies, and I think it's
> unfortunate. I think it will get better over time though. I hope,
> because I depend on Sage with my other projects (FEMhub). I just came
> from the scipy 09 conference, and people know Sage and use Sage and
> it's not like that they hate it. But they are not terribly excited
> either --- simply because they were discouraged by some remarks or
> attitude of the Sage project. But as I said, I hope it will improve.
to improve things?
-- William
>
>
> Ondrej
Ondrej Certik
Well, only what I am doing already doing all the time, e.g. if you
watch my sympy presentation, I spend some time explaining what Sage
is, what it's benefits are etc. That you support derivatives projects,
like SPD or FEMhub. So that people can see that if someone from the
Sage project made some unfortunate remarks, it was just that:
unfortunate remarks.
Ondrej
Harald Schilly
wrote:
> *independent upstream projects* in Sage. If someone wants to add a
> feature, usually modifying the library directly is a much more
> reasonable (and easier) route.
as my personal opinion:
spkgs are not good at the moment and i understand why there are
problems with them. we have to document them more, automatize some
details and present them better. Only then, we have a "complete"
package system.
There are also two ways how to look at them ... as a part of Sage
(i.e. matplotlib) or as an optional addition, a bit independent of
sage core development. Maybe it's bad to mix this?
I want to point to the way "R" does it. They also have "library"
packages, they are in my eyes better than sage's, (they are an older
project and there are many more packages for a much longer time, so
they have more experience). They also have "core libraries" for core
functionality, but that's very very small .... in that respect it's
exactly as with Sage.
Basically:
1) there is a documentation how to create one + a trivial skeleton
2) they collect all of them at a central point, each package contains
build information, source documentation ("man-page" + latex style)
and
3) there is a way to test if they work
4) if you want to install one, inside R type install.packages("...")
or to search search.packages("...") (IIRC) ... that works at runtime,
and asks for a mirror, doesn't require you to visit a website.
5) to use it, library(name) and then all exported functions are in its
namespace and vignette(name) gives you the documentation
http://cran.r-project.org/web/packages/
checks: http://cran.r-project.org/web/checks/check_summary.html
And if you think that list above is long, there is also bioconductor:
http://www.bioconductor.org/packages/2.4/bioc/
notice, that many of those packages are maybe old and broken and it
depends how good they work, but they are pretty successful at
collecting them and many are submitted. For example, together with a
new paper and so on.
What Sage could learn is this:
1) documentation of spkgs, combine it with sphinx + pdf output could
be feasible.
2) an easy way to register all installed optional packages in the sage
library, i have read that it is not clear how this works and i don't
know it either. i think of something like a hook method that gets
called (__init__.py ?)
3) better website + mirroring ---- that's something for me, i know ;)
4) a good skeleton, that contains a small c++ and fortran program plus
some cython and python exporting this to sage if it is installed and a
documentation.
H
rjf
By comparison, the Maxima build/patch/ mechanism seems quite a bit
better.
Numbers of us try out our new features by starting up a Maxima system,
and doing something like
load("mynewpackage");
Which results in the loading of a file (which might also load in other
files etc.)
which can over-ride the definition of nearly anything in Maxima, from
the way
an FFT is computed, to the simplification of sin(n/m*%pi), to the way
expressions
are parsed.
The extra features (or the replacement of previous implementations)
can be
done with code that runs at full compiled speed, or (assuming it is
actually
lisp or "maxima language" code, ) in interpreted form. This is slower
but can
provide more information for debugging.
In recent years, people who were more familiar with or more
comfortable with "make" than with the lisp
tools for defining systems (defsystem, asdf) took over much of the
maxima "building"
responsibility, and this perhaps makes sense when some of the code
(like plotting, GUIs)
has been written in non-Lisp language.
Nevertheless, it is still possible for me (and I think the Scieneer
Lisp people) to build the core Maxima system
with a simple user interface, entirely in lisp, based entirely on lisp
data structures that
specify the order of compilation, and which even vary the contents of
the build depending
on factors like target architecture, target lisp implementation, etc.
I do not know if
anyone cares to do it, but I think that at one time it was possible to
do such things
as specify that we were compiling the code to run on machine
configuration X even
though the compiler was running on machine configuration Y. e.g. VAX
vs. M68010.
The source code lived (still lives!) in a source code tree that can be
browsed online,
or via CVS or by downloading a tarball.
Since this is Sage-flame, here is a flame-thrower blast.
A better way to build a viable FOSS alternative to MMM is to take
Maxima, and paste into it by
importing (optionally, and by auto-loading on first use. Maybe even
autoloading on
first use over the internet....) all those features that you would
like to have accessible in
Sage but which, frankly are unlikely individually to be used are would
almost certainly never
be used ALL TOGETHER except to run exhaustive test suites. Those
extra features
could be refreshed as executable downloadable packages as often as
needed,
though some care would be needed .. e.g. you load the current
subsystem X and it determines that
your subsystem Y is not recent enough.
These optional features could mimic Magma, or other non-free code,
and could include NTL, FLINT, GAP, and the other 83
packages.
They would have to be available in the right format for different
Maximas for download
from archives in object form.
But if you had to do this first for one architecture and then play
catchup with the others,
the obvious architecture would be Intel/Windows for 95% of the
audience (my guess).
and then a tossup between Mac OS X or some Linux. Maybe Solaris since
Oracle/Sun is
so generous.
This organization around Maxima can all be done with almost any Lisp
with foreign function calls.
Now maybe this can all be done around Python or Cython, but from what
I've seen, you have
such a hard time incorporating Maxima, that Python, pexpect, failures
of ECL, mysterious
errors that happen only from Sage, .... is not really such a winner.
As with many such projects,
time will tell if your approach works. Time will not tell if another
approach would have worked better.
kerosene has run out.
RJF
On Aug 25, 12:22 pm, Harald Schilly <harald.schi...@gmail.com> wrote:
>
....
>
> Sorry that I don't read the entire thread, but I just want to add this
> as my personal opinion:
>
>
RJF
Harald Schilly
> Numbers of us try out our new features by starting up a Maxima system,
> and doing something like
>
> load("mynewpackage");
>
> Which results in the loading of a file (which might also load in other
> files etc.)
> which can over-ride the definition of nearly anything in Maxima,
extension packages need enhancement on various levels to be able to
provide something similar. I disagree that it should be able to
override anything, i would rather like to see the usual python import
mechanism, that is, if you want to override functions, do "from
package-name import *" and if not, just "import package-name" to use a
namespace. But we are already there ...
> I've seen, you have
> such a hard time incorporating Maxima, that Python, pexpect, failures
> of ECL, mysterious
> errors that happen only from Sage, .... is not really such a winner.
talking to ecl through a socket, or a deeper integration) or kick
maxima. For example, maxima's behavior of asking questions during
calculations does not fit so well and can't be solved just by
enhancing the communication.
H
rjf
For example, maxima's behavior of asking questions during
> calculations does not fit so well and can't be solved just by
> enhancing the communication.
>
> H
like to eliminate the questions in Maxima regardless of Sage.
I view this as 2 separate issues.
1. The technically resolvable issues: sometimes Maxima asks question
where it should, in principle, be able to figure out the answers
itself.
Sometimes this is a matter of fixing a bug, and sometimes it is a
matter of implementing a substantial geometric deduction facility.
2. "Kicking the can down the road." That is, taking a problem that
you have "now" and converting it into a problem (perhaps much harder)
that you will have "later".
It is like being annoyed with your automobile. The fuel gauge keeps
on indicating that your fuel tank is getting empty. You can solve
this problem by installing a filler tube so that when the tank is low,
you just add water from the coolant system. Sure enough, the fuel
gauge now says the tank is always full.
When the car stops running ... well, that's a different problem.
When a solution becomes bifurcated by a decision point, you can
propose to continue working down both (or however many) paths there
are.
Unfortunately, this may not converge, and you may recursively generate
an infinite tree for a solution, just so you don't have to answer a
question
like "is realpart(n)>0?" [If you answer no, the next question might
be, is realpart(n+1)>0. etc]
William Stein
What does Mathematica or Maple or Mupad do in such cases? None of
them ask interactive questions, as far as I know.
William
Ondrej Certik
In sympy we'll use our assumptions system for this purpose.
Ondrej
William Stein
>>> like "is realpart(n)>0?" [If you answer no, the next question might
>>> be, is realpart(n+1)>0. etc]
>>
>> What does Mathematica or Maple or Mupad do in such cases? None of
>> them ask interactive questions, as far as I know.
>
> In sympy we'll use our assumptions system for this purpose.
>
> Ondrej
If the user has made no assumptions and Maxima *would* ask a question
-- in exactly that situation -- what would sympy do?
William
Ondrej Certik
I am not sure exactly when maxima is asking a question, but in the
integrals we'll just return it unevaluated if it can't be evaluated.
Ondrej
William Stein
Here's a typical interaction:
sage: var('v,c')
sage: integral((1+v^2/c^2)^3/(1-v^2/c^2)^(3/2),v)
Traceback (most recent call last):
...
TypeError: Computation failed since Maxima requested additional
constraints (try the command 'assume(c>0)' before integral or limit
evaluation, for example):
Is c zero or nonzero?
sage: assume(c!=0)
sage: integral((1+v^2/c^2)^3/(1-v^2/c^2)^(3/2),v)
-75/8*sqrt(c^2)*arcsin(sqrt(c^2)*v/c^2) - 17/8*v^3/(sqrt(-v^2/c^2 +
1)*c^2) - 1/4*v^5/(sqrt(-v^2/c^2 + 1)*c^4) + 83/8*v/sqrt(-v^2/c^2 + 1)
Are you telling me that in Sympy you would do:
sage: integral((1+v^2/c^2)^3/(1-v^2/c^2)^(3/2),v)
integral((1+v^2/c^2)^3/(1-v^2/c^2)^(3/2),v)
and just give the user no *hint* or clue that if they just do
"assume(c!=0)" then they can get an answer? That seems like very bad
design.
William
Ondrej Certik
What will you do in Sage?
Ondrej
William Stein
>>
>> sage: integral((1+v^2/c^2)^3/(1-v^2/c^2)^(3/2),v)
>> integral((1+v^2/c^2)^3/(1-v^2/c^2)^(3/2),v)
>>
>> and just give the user no *hint* or clue that if they just do
>> "assume(c!=0)" then they can get an answer? That seems like very bad
>> design.
>
> What will you do in Sage?
>
> Ondrej
Ideally return an answer with two cases, as RJF so opaquely suggested
above. I think that is what Maple/Mathematica do.
William
Ondrej Certik
Well, mathematica seems to just return one case:
http://www.wolframalpha.com/input/?i=integral((1%2Bv^2%2Fc^2)^3%2F(1-v^2%2Fc^2)^(3%2F2)%2Cv)
Ondrej
rjf
On Aug 27, 10:21 am, William Stein <wst...@gmail.com> wrote:
>
> What does Mathematica or Maple or Mupad do in such cases? None of
> them ask interactive questions, as far as I know.
might return some kind of conditional "branching" result.
Integrate[x^n,x] gives an answer that is wrong when n=-1. Oops.
In general, Mathematica seems comfortable giving answers that are
wrong over an infinite set of parameter values, so long as it seems
that the error set is of lower dimension in some sense, than the space
spanned by all the parameters. So a single-parameter integral could
be wrong on any finite set of points.
A physicist would check the answers, I guess. From my perspective,
the points where the general answers are WRONG may be the points of
most interest.
If you are happy for Sage to produce flakey answers in such cases,
perhaps you could get a volunteer to introduce appropriate bugs in
some version of Maxima, or you could write sympy that way.
To see an attempt trying to get right answers that was pasted on after
the fact in the Mathematica definite integration code, try
Integrate[x^n,{x,0,1},GenerateConditions->True]
The fact that this command produces an answer that (last I looked)
could not be used ( except trivially) as input to any other part of
Mathematica was one consequence of fundamentally ignoring a lesson
learned in building and evaluating Macsyma. Wolfram designed
Mathematica initially by imitating the easy parts of computer algebra,
and ignoring (or deferring) the knotty problems.
If sympy has an assumption system at its core, a powerful deduction
mechanism, and can deal with multiple possibilities for values, that
would be very nice. Reasoning geometrically about items like infinity,
'undefined', elementary functions, would represent, so far as I know,
breaking new ground scientifically and mathematically, especially
since some of the questions are undecidable. Even a partial solution
would likely be of some interest. Someone should write it up!
I agree with William in that I think it is not a good solution to
leave integrate(x^n,x) unevaluated unless assume(n != -1) is in your
database.
Question: Do you want to simplify x/x to 1 or to {1, assuming x!
=0}, which might be more appropriate sometimes. Can you deal with
representing that latter form?
Bjarke Hammersholt Roune
> =0}, which might be more appropriate sometimes. Can you deal with
> representing that latter form?
>
answer fundamentally depends on who the user is and what kind of
mathematics that user wants to do at the moment. I think an approach
of returning strictly speaking wrong but never-the-less mostly useful
answers is what is best for many users much of the time, so it makes
sense for, e.g., Mathematica to follow that path. At the same time
it's horrifying that important calculations are put in the hands of a
piece of software that's happy to give you wrong answers as long as
things *usually* work out. I'm not sure that Mathematica failed to
consider the problem of cases, it may just be that there isn't a
business case for Wolfram Research to implement them and integrate
them everywhere, especially if that would make Mathematica harder to
use for a non-mathematician.
I heard of the concept of proof=false some time ago, and I think it is
a nice solution to this dilemma. When proof=false, or some other
option, it would be nice to have the software keep things simple by
performing these kinds of wrong-but-useful calculations and guesses.
When proof=true, then everything must be correct. It might be nice to
have a compromise between the two available as well, saying "do this
calculation any way you can, and then tell me at the end exactly what
things you assumed along the way."
As for dealing with cases when proof=true, I think a good approach is
to try out all possibilities, until some adjustable cutoff is reached
as to the number of cases, memory consumed or time spent. At that
point return the partial result, with some cases solved and some
unevaluated, while indicating the partial failure somehow. The user
could then give up, see that the case he cares about succeeded, or
guide the computation by pulling out the unevaluated case he cares
about and rerunning the algorithm on that. Maxima's behavior is a
special case of this with the cutoff set at just 1 case, if we are
willing to consider returning two unevaluated cases as a question to
the user about which case (or both or none) to pursue further.
Harald Schilly
> When a solution becomes bifurcated by a decision point, you can
> propose to continue working down both (or however many) paths there
> are.
> Unfortunately, this may not converge, and you may recursively generate
Yes, it's a complicated situation, and it's probably not only about
integrals, also equations, ode/pdes and so on. It maybe also depends,
what type of branches it is doing. Just at a point? A countable set of
points? Or intervals?
I can think of several solutions, one option is to branch and include
it in the answer - like a logical expression associated with a
symbolic expression in a list or a more complex "answer" object:
{ (logical expression) : symbolic ), (( ...) : symbolic) }
But then it's necessary to have a "reduction" system, that picks the
answer you want to have -- because as you say, you want to reuse your
answer. Maybe, symbolic expressions in general need to have the
possibility to tag them with assumptions about their parameters or
even compose one symbolic expression out of several others, like a
piecewise defined function? There was something about this in the
sympy blog iirc.
Or an other way would be a flag for the command, say "principal
solution", that goes down one preferred way, and explicitly telling in
a warning message, that it has done assumptions.
H
rjf
On Aug 27, 4:01 pm, Bjarke Hammersholt Roune <bjarke.ro...@gmail.com>
wrote:
> > =0}, which might be more appropriate sometimes. Can you deal with
> > representing that latter form?
>
I'm not sure that Mathematica failed to
> consider the problem of cases, it may just be that there isn't a
> business case for Wolfram Research to implement them and integrate
> them everywhere, especially if that would make Mathematica harder to
> use for a non-mathematician.
design stages.
You seem to think it is a business case not to do it "right". I view
it as a
technical decision based on the likelihood that making this "right" is
too
hard to do after the system is so large.
Will Sage do the same thing? Have I changed anyone's mind on this
topic?
...
>. It might be nice to
> have a compromise between the two available as well, saying "do this
> calculation any way you can, and then tell me at the end exactly what
> things you assumed along the way."
DOOLEY, S. S. The use of domain restrictions in computer algebra
systems. Master's thesis, Computer Science Division, EECS Department,
University of California, Berkeley, 14 December 1988.
>
>
Also, a way of dealing with potentially infinite streams of results is
to return a function which can be resumed to produce a result (and a
function..).
This kind of operation is fairly well understood in a Scheme context
(and is taught to Freshman at UCB), and can be implemented in Common
Lisp.
I do not know if this can be done conveniently in Sage or python.
RJF
Harald Schilly
> see
> DOOLEY, S. S. The use of domain restrictions in computer algebra
> systems. Master's thesis, Computer Science Division, EECS Department,
> University of California, Berkeley, 14 December 1988.
>
>
>
> Also, a way of dealing with potentially infinite streams of results is
> to return a function which can be resumed to produce a result (and a
> function..)
it is possible in Python to return functions as objects. Here an
example where i create a function inside a function with a given
parameter. together with generators, i think that's all it needs?
In [1]: def k(i):
...: def f(x):
...: return x+i
...: return f
...:
In [2]: f21 = k(21)
In [3]: f21(1)
Out[3]: 22
In [4]: f8 = k(8)
In [5]: f8(0)
Out[5]: 8
In [6]: f21(0)
Out[6]: 21
In [7]: type(f8)
Out[7]: <type 'function'>
rjf
On Aug 28, 9:07 am, Harald Schilly <harald.schi...@gmail.com> wrote:
>
> Nice idea, i don't see how this would look in an real application, ..
several people have written up taylor series with a suspended
"function"
representing "the rest of the terms".
Axiom has this, I think.
The Abelson/Sussman book shows (to Freshman) how this can be done in
the Scheme dialect of lisp.
RJF
Is this still a flame?
Bjarke Hammersholt Roune
> it as a technical decision based on the likelihood that making this "right" is
> too hard to do after the system is so large.
>
motivated by what makes business sense. Sage and Maxima isn't run for
profit, so the basis for making decisions is different.
Nils Bruin
>
> > see
> > DOOLEY, S. S. The use of domain restrictions in computer algebra
> > systems. Master's thesis, Computer Science Division, EECS Department,
> > University of California, Berkeley, 14 December 1988.
>
> title sounds interesting, probably worth to look into it
>
>
>
> > Also, a way of dealing with potentially infinite streams of results is
> > to return a function which can be resumed to produce a result (and a
> > function..)
> > ....
> > I do not know if this can be done conveniently in Sage or python.
>
> Nice idea, i don't see how this would look in an real application, but
> it is possible in Python to return functions as objects. Here an
> example where i create a function inside a function with a given
> parameter. together with generators, i think that's all it needs?
could consider returning an iterator that will generate (answer,
assumptions) pairs. The problem would be to get maxima to backtrack to
try the other answer. Trivial code example:
def primes():
n=1
while true:
n+=1
if is_prime(n):
yield n
You can for instance do
>>> P=primes()
>>> P.next()
2
>>> P.next()
3
etc.
You could also do
>>> [n^2 for n in primes()]
but in this case it won't return an answer.
The "lazy" variant:
>>> (n^2 for n in primes())
does, though.
This implementation is probably even efficient enough to generate more
interesting input labels for Maxima (to honour the group name).
rjf
On Aug 28, 10:23 am, Bjarke Hammersholt Roune <bjarke.ro...@gmail.com>
wrote:
> > it as a technical decision based on the likelihood that making this "right" is
> > too hard to do after the system is so large.
>
> Wolfram Research is a business, so I expect much of what they do to be
> motivated by what makes business sense.
money.
I suspect there was a technical failure to realize that they were
digging themselves into a hole.
Now once you realize you are in a hole, you could make a business
decision to
(a) fix the situation properly
(b) ignore the situation
(c) declare that the situation is not a problem but just exactly what
you intended to do.
(d) make some kind of inadequate patch.
WRI chose some combination of b,c,d.
By the way, doing things "right" may get Sage into the domain of
Mathematical Knowledge Management (MKM).
I do not know if there is any overlap in personnel, but if the intent
for Sage is to link to everything, you need
to be looking at some MKM stuff.
> Sage and Maxima isn't run for
> profit, so the basis for making decisions is different.
decisions can be pretty complicated.
Google is "for profit" as is Microsoft, and some of those Unix support
companies.
The non-profitness of some organizations does not mean that they make
good decisions.
Consider, for example the National Knights of the Ku Klux Klan.
RJF
Bjarke Hammersholt Roune
> > motivated by what makes business sense.
> >
> I doubt there was ever a business decision to "do it wrong" to save
> money. I suspect there was a technical failure to realize that they were
> digging themselves into a hole.
>
calculations may be both cheaper to implement and sell more copies of
Mathematica then correct calculations. Then doing anything else is the
wrong thing to do for a business trying to make money. I don't think
Wolfram Research lacks people with the right kinds of skills to
identify and implement a reasonable and correct solution to these
issues given a lot of time to work on it. That they haven't done so
tells me that Wolfram Research believes that those people have more
profitable things to do than this. They might be wrong about that - I
don't know.
> The non-profitness of some organizations does not mean that they
> make good decisions.
>
such organizations have, which determines what the good decisions are
to meet those ambitions.
rjf
On Aug 28, 6:31 pm, Bjarke Hammersholt Roune <bjarke.ro...@gmail.com>
wrote:
> > > motivated by what makes business sense.
>
> > I doubt there was ever a business decision to "do it wrong" to save
> > money. I suspect there was a technical failure to realize that they were
> > digging themselves into a hole.
>
> That's entirely possible. What I'm saying is that wrong-but-useful
> calculations may be both cheaper to implement and sell more copies of
> Mathematica then correct calculations.
doing
the wrong calculations costs less than assigning someone to figure out
and
implement the right calculations. It is also less embarrassing to
Stephen
if his original incorrect design is allowed to remain.
> Then doing anything else is the
> wrong thing to do for a business trying to make money.
had a wider audience.
> I don't think
> Wolfram Research lacks people with the right kinds of skills to
> identify and implement a reasonable and correct solution to these
> issues given a lot of time to work on it.
involvement with
the wrong approach.
>That they haven't done so
> tells me that Wolfram Research believes that those people have more
> profitable things to do than this.
of the program.
>They might be wrong about that - I
> don't know.
>
> > The non-profitness of some organizations does not mean that they
> > make good decisions.
>
> True. The difference it does make is in the kinds of ambitions that
> such organizations have, which determines what the good decisions are
> to meet those ambitions.
make
good or bad decisions.