Re: [sage-devel] For french readers

[William Stein]

On Tue, Mar 31, 2015 at 3:11 AM, Jean-Pierre Flori <jpf...@gmail.com> wrote:
> Let's replace Python by Julia :p
>
> http://lavieestmaloptimisee.blogspot.fr/2015/03/les-informaticiens-meprisent.html?view=classic

Oops, April 1 isn't until tomorrow [1] ?!

[1] http://sage-devel.narkive.com/p7EcpMPd/python-and-lisp


Regarding:

On Tuesday, March 31, 2015 at 12:11:34 PM UTC+2, Jean-Pierre Flori wrote:
> And that Sage as it is written by mathematicians looks very amateuristic to a computer scientist.

The specific points in the blog post might be (who knows -- I'm
arguing with google translated french):

- "The interfaces between packages was by strings which made 10 ^ 6
times slower than what we got on Mathematica."

So computer scientists never use strings? (Yes, some things in Sage
are slower than in Mathematica, and some are much faster.) By
creating pexpect interfaces initially, we were able to establish an
API and build up things on top of it (e.g., compute the irreducible
components of a variety). Then later we wrote C library interfaces to
many of these same components, which migrate to use that new
interface, maintaining tests, etc. This sort of careful strategic
iterative development, which easily parallelizes to a large number of
people and gets big projects (GAP, Singular, PARI) to work together,
rather than compete, is not amateuristic -- it's very sensible. Of
course, often I feel I'm the only person in the world who believes
this; maybe that's why there wasn't a Sage before Sage.

- "The Windows version installs a Linux virtual machine."
- This is partly due to the fact that it is an open-source project,
led by mathematicians and that funding is somewhat limited.

R-Evolution was a company set up by academics with one of the main
goals to port just R to Windows properly. They raised about $40
million in funding, and eventually Microsoft just bought them this
year (see http://vator.tv/news/2015-01-23-microsoft-buys-revolution-analytics-for-big-data).
When we were first working on the Windows port of Sage (back in 2008)
we met with the R-evaluation guys and they told us about how Microsoft
was paying them millions to work on the port. Porting Sage fully
natively to Windows would be at least as difficult as R (as R is just
one of the many (and smaller) components of Sage). Sage has
basically no funding in comparison with this.
Anyways, "somewhat limited" is an understatement. Our budget is
less than 1% of something like Mathematica (in fact, right now I have
$0 in NSF support, since NSF rejected the last Sage grant proposal).

- But honestly, as a computer scientists it makes you cry.

Well I'm sure Sage has made many mathematicians cry too. Certainly me.

- Sage Development (at least initially) was provided in part by
undergrads and they had to refuse contributions...

Some undergraduates are extremely good programmers, often quickly
become graduate students, etc. We refuse contributions because we
have a referee process, just like a journal refuses contributions.

--

All that said, Julia seems really exciting. If people write major
packages of functionality in Julia that people doing mathematics
really need, and is better than what is already in Sage, we could
consider adding Julia to Sage... So far, the demand-from-end-users
scale hasn't tipped in that direction.

Many algorithms in computational pure mathematics are very, very deep.
Implementing major algorithms (at least in arithmetic geometry) can
take year(s) working on a relatively stable foundation, which itself
can take many years to build up, even with an ideal programming
language. So even if Julia were "perfect" right now, it might be a
while until it has the foundations and depth to implement say "ideal
theory in quaternion algebras over number fields". I continue to
be impressed by Magma...

-- William


--
William (http://wstein.org)

[rjf]

On Tuesday, March 31, 2015 at 6:19:03 AM UTC-7, William Stein wrote:

On Tue, Mar 31, 2015 at 3:11 AM, Jean-Pierre Flori <jpf...@gmail.com> wrote:
> Let's replace Python by Julia :p
>
> http://lavieestmaloptimisee.blogspot.fr/2015/03/les-informaticiens-meprisent.html?view=classic

Oops, April 1 isn't until tomorrow [1] ?!

[1]   http://sage-devel.narkive.com/p7EcpMPd/python-and-lisp


Regarding:

On Tuesday, March 31, 2015 at 12:11:34 PM UTC+2, Jean-Pierre Flori wrote:
> And that Sage as it is written by mathematicians looks very amateuristic to a computer scientist.

The specific points in the blog post might be (who knows -- I'm
arguing with google translated french):

- "The interfaces between packages was by strings which made 10 ^ 6
times slower than what we got on Mathematica."

I think the blogger and William (below) both miss the point that the cost of the

interface may be orders of magnitude slower, but what percent of the time is

spent in the interface, compared to the computation?   Of course if you ship off

a trivial computation, at great cost in the interface, and you do it many times

in a loop,  that really eats up time in the interface.   It is not a matter of

"do you use strings".

 

So computer scientists never use strings?   (Yes, some things in Sage
are slower than in Mathematica, and some are much faster.)

Indeed, some things were faster in Maxima than in Mathematica, when

tested in the past.  Dunno about today.  Both systems are moving targets.

 

 By
creating pexpect interfaces initially, we were able to establish an
API and build up things on top of it (e.g., compute the irreducible
components of a variety).  Then later we wrote C library interfaces to
many of these same components, which migrate to use that new
interface, maintaining tests, etc.  This sort of careful strategic
iterative development, which easily parallelizes to a large number of
people and gets big projects (GAP, Singular, PARI) to work together,
rather than compete, is  not amateuristic -- it's very sensible.  Of
course, often I feel I'm the only person in the world who believes
this; maybe that's why there wasn't a Sage before Sage.

I think the amateuristic (is that a word in English?) accusation is generally

correct in that code written by newbies tends to be amateurish. Code written

by mathematicians unacquainted with programming language techniques --

regardless of how much math they know -- tends to be amateurish.

 

- "The Windows version installs a Linux virtual machine."
- This is partly due to the fact that it is an open-source project,
led by mathematicians and that funding is somewhat limited.

R-Evolution was a company set up by academics with one of the main
goals to port just R to Windows properly.   They raised about $40
million in funding, and eventually Microsoft just bought them this
year (see http://vator.tv/news/2015-01-23-microsoft-buys-revolution-analytics-for-big-data).
When we were first working on the Windows port of Sage (back in 2008)
we met with the R-evaluation guys and they told us about how Microsoft
was paying them millions to work on the port.  Porting Sage fully
natively to Windows would be at least as difficult as R (as R is just
one of the many (and smaller) components of Sage).    Sage has
basically no funding in comparison with this.
Anyways, "somewhat limited" is an understatement.    Our budget is
less than 1% of something like Mathematica (in fact, right now I have
$0 in NSF support, since NSF rejected the last Sage grant proposal).

Interesting.  Presumably it was mathematicians who torpedoed that

proposal.  Unless you made the mistake of sending it to the computer

science directorate.    I guess you'll have to go back to that secret

agency.

- But honestly, as a computer scientists it makes you cry.

I find this unclear.  Is this person crying because of the display of

misdirected resources?

Well I'm sure Sage has made many mathematicians cry too.  Certainly me.

- Sage Development (at least initially) was provided in part by
undergrads and they had to refuse contributions...

Some undergraduates are extremely good programmers, often quickly
become graduate students, etc.   We refuse contributions because we
have a referee process, just like a journal refuses contributions.

--

All that said, Julia seems really exciting.  If people write major
packages of functionality in Julia that people doing mathematics
really need, and is better than what is already in Sage, we could
consider adding Julia to Sage...     So far, the demand-from-end-users
scale hasn't tipped in that direction.

How do you feel about javascript,  which according to the table in this article

http://www.admin-magazine.com/HPC/Articles/Julia-A-New-Language-For-Technical-Computing

is many times faster than Julia on numerous benchmarks.   

(And Julia in turn is much faster than Python)

 

Many algorithms in computational pure mathematics are very, very deep.

Eh,  the point of programming language subroutines is to make the

building of towers of algorithms that hide the complexity.  To make

things that (I expect) are the ones that you refer to as "deep"  as

shallow as possible.

This is a major advantage over mathematics --- or at least as

practiced or proposed by some mathematicians --  who claim that

to understand a theorem you must get it all into your head at

the very same time  (including presumably everything upward from

axioms?).

The major point of system engineering is to make hard things easy

by (to use your metaphor)  making deep things shallow.

 

Implementing major algorithms (at least in arithmetic geometry) can
take year(s) working on a relatively stable foundation, which itself
can take many years to build up, even with an ideal programming
language.

This is silly.   If you are spending years doing X, you should spend some

of that time building the ideal programming language to do X.   Not necessarily

X,Y,Z,    but X.

 

  So even if Julia were "perfect" right now, it might be a
while until it has the foundations and depth to implement say "ideal
theory in quaternion algebras over number fields".      I continue to
be impressed by Magma...

If you want to do what Magma does, sure.   I have not used it.  I looked at Cayley,

and was impressed by the poorly designed programming language  aspect.

Maybe Magma is better that way, but maybe it doesn't matter.

RJF

[William Stein]

On Tue, Mar 31, 2015 at 7:24 AM, rjf <fat...@gmail.com> wrote:
>
>
> On Tuesday, March 31, 2015 at 6:19:03 AM UTC-7, William Stein wrote:
>>
>> On Tue, Mar 31, 2015 at 3:11 AM, Jean-Pierre Flori <jpf...@gmail.com>
>> wrote:
>> > Let's replace Python by Julia :p
>> >
>> >
>> > http://lavieestmaloptimisee.blogspot.fr/2015/03/les-informaticiens-meprisent.html?view=classic
>>
>> Oops, April 1 isn't until tomorrow [1] ?!
>>
>> [1] http://sage-devel.narkive.com/p7EcpMPd/python-and-lisp
>>
>>
>> Regarding:
>>
>> On Tuesday, March 31, 2015 at 12:11:34 PM UTC+2, Jean-Pierre Flori wrote:
>> > And that Sage as it is written by mathematicians looks very amateuristic
>> > to a computer scientist.
>>
>> The specific points in the blog post might be (who knows -- I'm
>> arguing with google translated french):
>>
>> - "The interfaces between packages was by strings which made 10 ^ 6
>> times slower than what we got on Mathematica."
>
>
> I think the blogger and William (below) both miss the point that the cost of
> the

Sorry, I just didn't have time to go into that. You are of course
completely right about this additional point.

> interface may be orders of magnitude slower, but what percent of the time is
> spent in the interface, compared to the computation? Of course if you ship
> off
> a trivial computation, at great cost in the interface, and you do it many
> times
> in a loop, that really eats up time in the interface. It is not a matter
> of
> "do you use strings".
>
>>

>> By
>> creating pexpect interfaces initially, we were able to establish an
>> API and build up things on top of it (e.g., compute the irreducible
>> components of a variety). Then later we wrote C library interfaces to
>> many of these same components, which migrate to use that new
>> interface, maintaining tests, etc. This sort of careful strategic
>> iterative development, which easily parallelizes to a large number of
>> people and gets big projects (GAP, Singular, PARI) to work together,
>> rather than compete, is not amateuristic -- it's very sensible. Of
>> course, often I feel I'm the only person in the world who believes
>> this; maybe that's why there wasn't a Sage before Sage.
>
>
> I think the amateuristic (is that a word in English?) accusation is
> generally
> correct in that code written by newbies tends to be amateurish. Code written
> by mathematicians unacquainted with programming language techniques --
> regardless of how much math they know -- tends to be amateurish.

It's impossible to argue with that logic. The important thing to
keep in mind is that sometimes amateurs turn into pros.

> less than 1% of something like Mathematica (in fact, right now I have
>> $0 in NSF support, since NSF rejected the last Sage grant proposal).
>
>
> Interesting. Presumably it was mathematicians who torpedoed that
> proposal. Unless you made the mistake of sending it to the computer
> science directorate.

It was a proposal to the NSF part of education. It could be "lack of
enough money" that sunk it. Maybe we'll get it next time.

> I guess you'll have to go back to that secret agency.

I googled -- Are you suggesting this?
http://sam.msp.berkeley.edu/nsa-ams/about/program/guidelines.html

Good idea.

>>
>>
>> - But honestly, as a computer scientists it makes you cry.
>
>
> I find this unclear. Is this person crying because of the display of
> misdirected resources?

I think he means that the way Sage is written is "ugly"/painful/etc.

> How do you feel about javascript, which according to the table in this
> article
>
> http://www.admin-magazine.com/HPC/Articles/Julia-A-New-Language-For-Technical-Computing
>
> is many times faster than Julia on numerous benchmarks.
> (And Julia in turn is much faster than Python)

I love Javascript (and Julia, and a lot of other languages).

Javascript would be a terrible language in which to implement Sage,
but it's surprisingly good for some things that involve a lot of
asynchronous IO, e.g., web pages like https://cloud.sagemath.com. I
have written way more Javascript (actually CoffeeScript) than Python
during the last two years. With sufficient discipline, it's OK.

>> Many algorithms in computational pure mathematics are very, very deep.
>
>
> Eh, the point of programming language subroutines is to make the
> building of towers of algorithms that hide the complexity. To make
> things that (I expect) are the ones that you refer to as "deep" as
> shallow as possible.

Yes, but it takes a long time to build up that tower. It will indeed
not feel deep when you're working at the top. But to get to that point
can take a long time. Magma (and Sage and Mathematica) are all there
in various senses, and not in others.

> This is a major advantage over mathematics --- or at least as
> practiced or proposed by some mathematicians -- who claim that
> to understand a theorem you must get it all into your head at
> the very same time (including presumably everything upward from
> axioms?).

Actually it works pretty much the same way in mathematics. As a
mathematician you build up abstractions, learn proofs of lemmas, etc.
When you understand a complicated mathematical argument, it *seams* to
you at the time like a simple argument composed of several "simple"
pieces. Those pieces in turn...

> The major point of system engineering is to make hard things easy
> by (to use your metaphor) making deep things shallow.

Sure. My point is just that this takes significant time to do, which
is an issue for Julia, as it is fairly new.

>
>>
>> Implementing major algorithms (at least in arithmetic geometry) can
>> take year(s) working on a relatively stable foundation, which itself
>> can take many years to build up, even with an ideal programming
>> language.
>
>
> This is silly. If you are spending years doing X, you should spend some
> of that time building the ideal programming language to do X. Not
> necessarily
> X,Y,Z, but X.

It depends on X. No matter how good your ideal programming language
is, deep stuff is still deep.

>> So even if Julia were "perfect" right now, it might be a
>> while until it has the foundations and depth to implement say "ideal
>> theory in quaternion algebras over number fields". I continue to
>> be impressed by Magma...
>
>
> If you want to do what Magma does, sure.

I do. Sage's mission statement is "... viable alternative to Magma, ..."

> Maybe Magma is better that way, but maybe it doesn't matter.

I make no claims about the programming language of Magma being any
good. I don't like it. But it is impossible to dispute that Magma
contains implementations of several very, very deep mathematical
algorithms.

--
William (http://wstein.org)

[William Stein]

On Tuesday, March 31, 2015, Volker Braun <vbrau...@gmail.com> wrote:

On Tuesday, March 31, 2015 at 6:10:55 PM UTC+2, William wrote:

have written way more Javascript (actually CoffeeScript) than Python
during the last two years.

In other words you like Javascript so much that you'd rather write in a different language and transpile it. Sounds like you like the v8 runtime implementation, not the language ;-)

Yes. Definitely. 

 

 

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To post to this group, send email to sage-...@googlegroups.com.
Visit this group at http://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.

--
Sent from my massive iPhone 6 plus.

[Pierre]

>Anyways, "somewhat limited" is an understatement.

To be fair, the french "financement est pour le moins limité" means something like "funding is limited, to say the least". Don't trust google ! :-)

pierre

[William Stein]

On Wed, Apr 1, 2015 at 7:28 PM, Nicolas M. Thiery
<Nicolas...@u-psud.fr> wrote:

> On Tue, Mar 31, 2015 at 06:18:22AM -0700, William Stein wrote:
>> This sort of careful strategic iterative development, which easily
>> parallelizes to a large number of people and gets big projects (GAP,
>> Singular, PARI) to work together, rather than compete, is not
>> amateuristic -- it's very sensible. Of course, often I feel I'm the
>> only person in the world who believes this; maybe that's why there
>> wasn't a Sage before Sage.
>

> I can testify of at least another person believing in this :-)
> Probably many more.
>
> When we had our "Open source Computer Algebra Meeting" back in 2002,

A few years later, I think Paul Zimmerman gave me a bunch of notes
from that meeting (I wasn't there, of course). When I read them I
definitely felt like some people were thinking along the same lines as
me.

> the above ideas where very much in the air. I believe that what was
> missing is someone with the determination, qualification, and stamina
> to get it running -- with a fairly insane guts level to think «yes we
> can».

I only remember feeling "insane" when getting this thing going. I was
convinced for the first two years that it would be a complete failure,
but just kept doing it anyways...

>
> Cheers,
> Nicolas
> --
> Nicolas M. Thiéry "Isil" <nth...@users.sf.net>
> http://Nicolas.Thiery.name/

>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at http://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.



--

William (http://wstein.org)

[Bill Hart]

On 31 March 2015 at 15:18, William Stein <wst...@gmail.com> wrote:

<SNIP>

All that said, Julia seems really exciting.  If people write major
packages of functionality in Julia that people doing mathematics
really need, and is better than what is already in Sage, we could
consider adding Julia to Sage...     So far, the demand-from-end-users
scale hasn't tipped in that direction.

We are already writing new sparse linear algebra and class group computation code in Julia. 

Much of it is still slower than Magma, due to various things we are missing. But some parts are already orders (plural) of magnitude faster than Magma/Pari.

One of the big problems we have at present is that all the different packages we are using or plan to use (Flint, ANTIC, Pari, Arb, Singular, Factory, Julia, Gap, NTL, etc.) use different fundamental formats for basic things like integers. And they have incompatible memory managers. This results in either ugly hacks or 100x slowdowns in some things we need.

The other major issue is that some of these libraries are fundamentally dynamically typed, even at the C/C++ level. Agreed, it is difficult to do any terribly serious mathematics without generics, which are fundamentally dynamic. But the lesson learned is that all the high level generic code basically has to be in the same language if you want to eliminate massively uneven performance.

I spent months listening to people complain about how it was better to have a lot of code that worked than a small amount of code that worked really fast. Those same people are now coding stuff in Julia and complaining about "100x slowdowns" and "design issues" in my Julia code.

For the first time in my life, other people are complaining to me that things are not running fast enough for them, instead of it being the other way around.

Unfortunately, writing things in Julia and C/C++ makes it really obvious where the amateurish code is.  

I'm seriously considering splitting the Julia codebase that I have been working on into two parts:

1) A system where all the various packages are fully wrapped and data conversions just cost additional time.

2) A system where only fast statically typed code is wrapped from each package (not much in some cases, not mentioning any names), plus complex functionality from the various packages with high computational complexity where data conversion costs are almost irrelevant (Julia can even call python code using its pycall interface). Everything else (including all generics and everything built on it) will be reimplemented directly in Julia.

Number 2 would not have as much functionality, and would be targeted at number theorists, who apparently really care about smooth performance, and the other would be for just about everyone else.

Ultimately we are going to need much greater technical cooperation between the various packages, especially in the area of memory management, data conversion and C interfacing.

We'll also have to do quite some work to get all the packages working on Mingw64. We have a strategy for doing this, just not enough time to implement it as of now.

Of course there are some other issues we haven't resolved in the Julia stuff:

1) Julia integer literals have inconsistent semantics. Not very useful for a CAS.

2) The Julia division operators are only really useful for numerical people. They aren't the definitions algebraists want to see/use.

3) Julia matrices are optimised for column operations, not row operations, as various other packages are.

4) There are still some bugs in the Julia type system and garbage collector and we are still working with development branches of various things (llvm, cxx, julia, lldb) that are frequently broken.

5) Our generic power series and p-adic modules were implemented in a completely amateurish way (by me) and need to be completely re-implemented.

So the time is not right for the average developer just yet. But we are still getting serious work done. I anticipate things will look much better in another 6-12 months, after possibly another rewrite.

Bill.

[William Stein]

On Thu, Apr 2, 2015 at 12:09 AM, Bill Hart <goodwi...@googlemail.com> wrote:
>
>
> On 31 March 2015 at 15:18, William Stein <wst...@gmail.com> wrote:
>
> <SNIP>
>>
>>
>> All that said, Julia seems really exciting. If people write major
>> packages of functionality in Julia that people doing mathematics
>> really need, and is better than what is already in Sage, we could
>> consider adding Julia to Sage... So far, the demand-from-end-users
>> scale hasn't tipped in that direction.
>
>
> We are already writing new sparse linear algebra and class group computation
> code in Julia.
>
> Much of it is still slower than Magma, due to various things we are missing.
> But some parts are already orders (plural) of magnitude faster than
> Magma/Pari.

Can you provide some links so that people reading these threads can
easily try out and or look at what you're working on? Thanks. I just
got excited by everything you wrote below, and thought "heh, I want to
fire up Julia and try this out...! but I have to give a talk this
morning so I only have a few moments to spare..." I realize you've
posted or sent me a link or something before, but it'll take me 5+
minutes just to find it, and it might be out of date. (And of course,
thanks for posting.)

> --
> You received this message because you are subscribed to the Google Groups

> "sage-flame" group.

> To unsubscribe from this group and stop receiving emails from it, send an

> email to sage-flame+...@googlegroups.com.

> To post to this group, send email to sage-...@googlegroups.com.

> Visit this group at http://groups.google.com/group/sage-flame.

[Bill Hart]

On 2 April 2015 at 17:31, William Stein <wst...@gmail.com> wrote:

On Thu, Apr 2, 2015 at 12:09 AM, Bill Hart <goodwi...@googlemail.com> wrote:
>
>
> On 31 March 2015 at 15:18, William Stein <wst...@gmail.com> wrote:
>
> <SNIP>
>>
>>
>> All that said, Julia seems really exciting.  If people write major
>> packages of functionality in Julia that people doing mathematics
>> really need, and is better than what is already in Sage, we could
>> consider adding Julia to Sage...     So far, the demand-from-end-users
>> scale hasn't tipped in that direction.
>
>
> We are already writing new sparse linear algebra and class group computation
> code in Julia.
>
> Much of it is still slower than Magma, due to various things we are missing.
> But some parts are already orders (plural) of magnitude faster than
> Magma/Pari.

Can you provide some links so that people reading these threads can
easily try out and or look at what you're working on?  Thanks.  I just
got excited by everything you wrote below, and thought "heh, I want to
fire up Julia and try this out...! but I have to give a talk this
morning so I only have a few moments to spare..."   I realize you've
posted or sent me a link or something before, but it'll take me 5+
minutes just to find it, and it might be out of date.  (And of course,
thanks for posting.)

The link you have is indeed out of date.

I'm currently working on a massive rewrite and it is not really in a state for other people to use at present. As I mentioned, you have to have the latest development versions of Julia, LLVM, etc, which are all intermittently hard to compile. (In fact, just yesterday, someone told me the latest Julia nightly build completely breaks our code.)

Moreover, the new class group stuff I mentioned is not merged yet. I've been on holidays for over a month with hardly working internet access, so have had no time to clean up/merge the contributions written whilst I was away. Claus Fieker and Tommy Hoffman have been working on it. It's likely to be some time before our implementation is overall competitive with anything!

But you can get a view of the (largely broken) code that is committed, excluding the class group and sparse linear algebra stuff, as it currently stands, here:

https://github.com/wbhart/Nemo.jl/tree/rewrite

Note the test suite does not pass and the documentation is completely incorrect and out-of-date. The Pari stack overflows if it wants to. There is no attempt to intercept the signal handlers, so Pari currently does that if something crashes. Lots and lots of issues, definitely not usable by an end user in the current state.

Also, none of the stuff we've been working on to get Singular and Factory into Nemo is currently in a state to be committed anywhere. And it relies on Cxx, which is a (brilliant) experimental Julia package which currently takes significant effort to compile. I don't expect that to be stable enough for serious use for 6-12 months. It's absolutely brand new technology, developed by one of the core Julia developers.

As soon as I have completed the current rewrite of Nemo, I'll make an announcement. Assuming we are happy with the state of things at that point, and Julia/LLVM have stabilised, I'll let people know how they can make use of and possibly contribute to it.

As I mentioned, there remain quite a few design decisions and technical obstacles. The only thing I can say is that the remaining issues are resolving themselves sufficiently fast that we can be sure of something that works in a reasonable period of time. At least LLVM-3.6 has just been released, which is a major milestone for stability that we have been waiting for.

We are very deliberately targeting the leading/bleeding edge because there is just so much amazing, useful stuff in the works that we really can't ignore (dramatically improved gc, much better C struct support, staged functions, C++ interface support, including inline C++, many speedups and bug fixes). We have numerous local hacks so that we can keep working through all the chaos.

Our plans extend as much as 13 years into the future at present. So it's very early days.

Bill.

[rjf]

<snip all this>

The discussion about programming languages for X is largely a rehash of discussions that are inherently

non-convergent .

The language syntax and semantics tends to be pointless from a particular advanced standpoint,

which is that if you want to design a language to do some task T, you can design that language L

and implement it.   The language that provides the underpinning for that implementation could

be almost anything, but for prototyping and maybe for final implementation, Lisp seems to be

very convenient.  Some people prefer C or C++, but that's because they don't know Lisp

(reminder; this is posted on Sage-flame).

Then there is the run-time support, language optimization, data-structure and memory allocation

hacks.

These cannot always be done in Lisp, and so parts are written in assembler or C.  Some

Lisp implementations are largely or completely done via translation to C.

or have C runtime routines.

Some programs become so tied to their own unique style of memory or data structure that

they cannot be interfaced with other languages without expensive conversions at the

interfaces.  Sometimes this uniqueness buys special efficiencies.  I hope that's what

Bill Hart is doing.

But the language -- should it be Python, Ruby, Julia, C, C++, Fortran 90, Lisp, JavaScript (ECMA)

or the new languages proposed by Microsoft, Google, Apple ...

the discussion predictably will not converge.

Better and better implementations of Lisp compete too.   After all, if an algorithm precisely

can be expressed in a high level way, a "good enough" compiler should be able to

produce extremely efficient code.

RJF

[Bill Hart]

On 2 April 2015 at 18:23, rjf <fat...@gmail.com> wrote:

<snip all this>

The discussion about programming languages for X is largely a rehash of discussions that are inherently

non-convergent .

The language syntax and semantics tends to be pointless from a particular advanced standpoint,

which is that if you want to design a language to do some task T, you can design that language L

and implement it.   The language that provides the underpinning for that implementation could

be almost anything, but for prototyping and maybe for final implementation, Lisp seems to be

very convenient.  Some people prefer C or C++, but that's because they don't know Lisp

(reminder; this is posted on Sage-flame).

As I already mentioned, Julia compiles to a kind of Lisp (femptolisp by Jeff Bezanson) before using LLVM for optimisation.

 

Julia is heavily inspired by Lisp. It is homoiconic, provides macros and many other standard lisp features (though it is not considered a lisp as such).

Many of the Julia developers are extremely talented lisp hackers.

Then there is the run-time support, language optimization, data-structure and memory allocation

hacks.

These cannot always be done in Lisp, and so parts are written in assembler or C.  Some

Lisp implementations are largely or completely done via translation to C.

or have C runtime routines.

Some programs become so tied to their own unique style of memory or data structure that

they cannot be interfaced with other languages without expensive conversions at the

interfaces.  Sometimes this uniqueness buys special efficiencies.  I hope that's what

Bill Hart is doing.

But the language -- should it be Python, Ruby, Julia, C, C++, Fortran 90, Lisp, JavaScript (ECMA)

or the new languages proposed by Microsoft, Google, Apple ...

the discussion predictably will not converge.

Better and better implementations of Lisp compete too.   After all, if an algorithm precisely

can be expressed in a high level way, a "good enough" compiler should be able to

produce extremely efficient code.

RJF

[William Stein]

All I've got to say is:

- STRONG ENCOURAGEMENT

- I'm really glad you guys are working on this!

- Many thanks for sharing and the status report.

William

[Pablo Angulo]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

>
> - "The interfaces between packages was by strings which made 10 ^ 6
> times slower than what we got on Mathematica."

In most, if not all, web frameworks using a sql database, queries to the
database are made by building strings in sql.

> By creating pexpect interfaces initially, we were able to establish an
> API and build up things on top of it (e.g., compute the irreducible
> components of a variety). Then later we wrote C library interfaces to
> many of these same components, which migrate to use that new

> interface, maintaining tests, etc. This sort of careful strategic

> iterative development, which easily parallelizes to a large number of
> people and gets big projects (GAP, Singular, PARI) to work together,
> rather than compete, is not amateuristic -- it's very sensible.

I completely agree. Performance is great, sure, but being able to
perform a computation comes first. Having a simple and quick way to test
an idea is more important. To me, it's of great importance to have a
first working prototipe. If at the end I don't get enough performance, I
can rewrite the code, then code parts in cython, then trace the
libraries that get called and call them directly. The
effort-to-performance curve in Sage is very smooth. I can optimize only
up to the point when my computation gets done. It's very cool and R,
matlab, octave, scilab do not have such facilities... I don't understand
why Sage is not more popular.

Regarding contributions from undergrads, I recently found this book:

http://aosabook.org/

Many of the projects mentioned in that book have gone to great lenghts
to allow contributors to join, with different levels of experience. Many
of those projects have got serious work done from non-experts, having
them work in the right place. I personally like Sage's way. The open
process, the peer review, and the battery of tests helped
non-computer-scientists like myself contribute to the surface without
breaking the structure. The process is more complicated than in most
open source projects, but of course Sage is way more complex than the
rest, integrating tighly so many packages that were not designed to work
together. I would only change the way to contribute documentation, which
I don't think needs not be so complex.

I'd say you can't please everybody: simple enough to be used for
undergrad teaching, solving as big an array of mathematical problems as
possible, completely consistent, and with uniform performance... it's
important (and hard) not to pay attention to flamers. In an open source
project of mine, I've heard:
- - if you had used php instead of python, which everybody understands,
I would have contributed
- - if you had used ruby instead of python, which is more trendy for web
apps, I would have contributed
- - if you had a big array of tests, I would have contributed.
- - if X, I would have contributed.
and of course lots of users saying they would use the program if only it
would had this or that feature. Now I only listen to actual
users/contributors, for sanity.

Regarding funding, I think we, the mathematical community, have failed
to properly fund Sage. In Spain, it's hard to justify spending public
money on something you can get for free. It's easy to get a grant to
have undergrads contribute to a project, but only for a short time, so
usually they can't contribute code at all. In late years, it's also
incredibly difficult to have a permanent position, so only senior
researchers can afford to spend serious time on an open source project.
We should be ashamed that it's so easy to spend hugh amounts of money in
closed-source projects.

So William, it's a great project, and *it was necessary*. The pieces
were laying there, waiting for someone to grab them and join them.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIbBAEBAgAGBQJVIi1aAAoJEATsOw+FDrzIpRQP+P3VivDu2PN6y5zqS3Upof8I
oJzvehLZttLQJG9HbhMx7tWoJUC02fKTz+x6Gf4uS100XWsVJuLKNujjiPJEiI1Q
p17ST3+yXYL15lFN42xuothBOHsb6VXfLNh7Q+V99uAcXupRBMwwnKJlMr8m8qXb
QcghPkD8KLDXW1AQqG51NVjkqE31diHxaZpOV1RSZh+9QQkKzrbOhMOQSi8ABAPz
VcpEKgEsMwEviMJadpwEeXWFue4jdgr+eqIL0vlfDLvr4R2lbGtiZDpyon72QC39
IFu66ABq5O2NwZkXCO00Hpyik7raEEpvPVB8KfK5QPwOCxE7/MRxlReFmfr/WKjZ
A5BRmsAh1pgTg+SzCOFs+L8yvMAO8OSuKY2n5S4Adj53bkzKJ1fPuSPyilkTZMey
cBsRqvdrdLN9gU+a9pjTGvxY5uYMxvTQ22HRM6lyoL18a7H291B1TSLuX4u9Isgx
2+s8IhA0Je+jo2xXMtAy1r3BukBFPnEUDQKbpQ/0yrOA0LZYPCsM9s7O8c9r+11r
7Wp+Ul9p13tkxE4ZDTbpkhDEwEw0/bmp9i5HdEWIBMHqqy6UN88A52axqceLIBjx
F+o4n/hHsyD96Y9rmOfLI+QCi5spCgUFJ3rq490IfHS86592zc5bWcXH+kpjmkxt
0AU61WHop0CnMWA9gEY=
=eN8R
-----END PGP SIGNATURE-----