Python is slow?

101 views
Skip to first unread message

sturlamolden

unread,
Sep 23, 2008, 9:23:12 AM9/23/08
to
I have recently been playing with a kd-tree for solving the "post
office problem" in a 12-dimensional space. This is pure cpu bound
number crunching, a task for which I suspected Python to be
inefficient.

My prototype in Python 2.5 using NumPy required 0.41 seconds to
construct the tree from 50,000 samples. Unfortunately, searching it
felt a bit slow, finding the 11 nearest-neighbours of 1,000 points
took 29.6 seconds (and there were still 49,000 to go). Naturally, I
blamed this on Python. It would be 100 times faster if I used C++,
right?

After having a working Python prototype, I resorted to rewrite the
program in C++. The Python prototype took an hour to make, debug and
verify. The same thing in C++ took me almost a day to complete, even
with a working prototype as model. To my surprise, the resulting beast
of C++ required 64.3 seconds to construct the same kd-tree. Searching
the tree was not faster either, 1,000 points required 38.8 seconds. I
wasted a day, only to find my Python prototype being the faster.

We may conclude that I'm bad at programming C++, but I suspect that is
not the case here. Albeit micro-benchmarks may indicate that Python is
100-200 times slower than C++, they may not be applicable to the real
world. Python can be very efficient. And when combined with libraries
like NumPy, beating it's performance with hand-crafted C++ is
difficult. At least, my 10 years experience programming scientific
software in various languages was not sufficient to beat my own Python
prototype with C++.

That is not to say I have never seen C++ run a lot faster than Python.
But it tends to be very short pieces of CPU bound code, no more than a
function or two. But as the problem grows in complexity, C++
accumulates too much of its own bloat.


Robert Singer

unread,
Sep 23, 2008, 9:44:56 AM9/23/08
to
On Tue, 23 Sep 2008 06:23:12 -0700 (PDT), sturlamolden
<sturla...@yahoo.no> wrote:

>I have recently been playing with a kd-tree for solving the "post
>office problem" in a 12-dimensional space. This is pure cpu bound
>number crunching, a task for which I suspected Python to be
>inefficient.

Well, python is not a number crunching language. However much we would
like it to be (we would ? :-). No scripting language is.
Developing time is shorter, I agree, but when you have, for example a
problem which takes 8,31 minutes to go through in optimized fortran
code (like the one I had the other day), then that hardly matters.

>
>My prototype in Python 2.5 using NumPy required 0.41 seconds to
>construct the tree from 50,000 samples. Unfortunately, searching it
>felt a bit slow, finding the 11 nearest-neighbours of 1,000 points
>took 29.6 seconds (and there were still 49,000 to go). Naturally, I
>blamed this on Python. It would be 100 times faster if I used C++,
>right?


Not necessarily.
Before resorting to rewriting the problem try psyco. It speeds up
things sometimes.
Also, (I'm not that familiar with python yet, so I don't know how to
do it in python), try finding the bottlenecks of your calculation. Are
the loops where most of the processing time is wasted, or disk
accessing, or ... ?

>
>After having a working Python prototype, I resorted to rewrite the
>program in C++. The Python prototype took an hour to make, debug and
>verify. The same thing in C++ took me almost a day to complete, even
>with a working prototype as model. To my surprise, the resulting beast
>of C++ required 64.3 seconds to construct the same kd-tree. Searching
>the tree was not faster either, 1,000 points required 38.8 seconds. I
>wasted a day, only to find my Python prototype being the faster.

>
>We may conclude that I'm bad at programming C++, but I suspect that is
>not the case here. Albeit micro-benchmarks may indicate that Python is
>100-200 times slower than C++, they may not be applicable to the real
>world. Python can be very efficient. And when combined with libraries
>like NumPy, beating it's performance with hand-crafted C++ is
>difficult. At least, my 10 years experience programming scientific
>software in various languages was not sufficient to beat my own Python
>prototype with C++.
>
>That is not to say I have never seen C++ run a lot faster than Python.
>But it tends to be very short pieces of CPU bound code, no more than a
>function or two. But as the problem grows in complexity, C++
>accumulates too much of its own bloat.
>

Well, personally, I try to combine fortran (being a fortran programmer
by trade) with python (in the last few years), as I find fortran to
be, by two grades, more comfortable for solving scientific problems
then c (or python for that matter, although it has its merits).
Starting from ith his capabilities for "normal" array handling, to
optimisation and easy readability, to whatnot.


Best regards
Bob

Grant Edwards

unread,
Sep 23, 2008, 9:57:14 AM9/23/08
to
On 2008-09-23, sturlamolden <sturla...@yahoo.no> wrote:

[...]

> After having a working Python prototype, I resorted to rewrite the
> program in C++. The Python prototype took an hour to make, debug and
> verify. The same thing in C++ took me almost a day to complete, even
> with a working prototype as model. To my surprise, the resulting beast
> of C++ required 64.3 seconds to construct the same kd-tree. Searching
> the tree was not faster either, 1,000 points required 38.8 seconds. I
> wasted a day, only to find my Python prototype being the faster.
>
> We may conclude that I'm bad at programming C++,

AFAICT, _everybody_ is bad at programming C++.

One begins to suspect it's not the fault of the programmers.

--
Grant Edwards grante Yow! Finally, Zippy
at drives his 1958 RAMBLER
visi.com METROPOLITAN into the
faculty dining room.

George Sakkis

unread,
Sep 23, 2008, 10:13:16 AM9/23/08
to
On Sep 23, 9:57 am, Grant Edwards <gra...@visi.com> wrote:

> On 2008-09-23, sturlamolden <sturlamol...@yahoo.no> wrote:
>
> [...]
>
> > After having a working Python prototype, I resorted to rewrite the
> > program in C++. The Python prototype took an hour to make, debug and
> > verify. The same thing in C++ took me almost a day to complete, even
> > with a working prototype as model. To my surprise, the resulting beast
> > of C++ required 64.3 seconds to construct the same kd-tree. Searching
> > the tree was not faster either, 1,000 points required 38.8 seconds. I
> > wasted a day, only to find my Python prototype being the faster.
>
> > We may conclude that I'm bad at programming C++,
>
> AFAICT, _everybody_ is bad at programming C++.

+1 QOTW

sk...@pobox.com

unread,
Sep 23, 2008, 10:15:07 AM9/23/08
to Grant Edwards, pytho...@python.org

>> We may conclude that I'm bad at programming C++,

Grant> AFAICT, _everybody_ is bad at programming C++.

Grant> One begins to suspect it's not the fault of the programmers.

+1 QOTW...

Skip

bearoph...@lycos.com

unread,
Sep 23, 2008, 11:31:14 AM9/23/08
to
sturlamolden:

CPython is generally slow (you can see this from the huge amount of
solutions invented to solve the speed problem, like Cython, Numpy,
Psyco, ShedSkin, Weave, Inline, SIP, Boost Python, SWIG, etc etc), but
for most of the usages Python is used for, it's not a significant
problem. I know that sounds like a tautology :-)

Well written C++ code is generally faster or much faster than similar
Python code, but programming in Python is often simpler, and it
generally requires less time. So it may happen that to solve a problem
a Python program that runs in 1 hour that requires 1 hour to be
written allows you to find the solution in less time than a C++
program that runs in 5-10 minutes that requires you 3-4 hours to be
written :-)

Note that C++ is just one option, but there are other languages
around, like CLisp or D (or even a modern JavaVM), that are often an
acceptable compromise between Python and C/C++.

So you can show us a reduced/minimal working version of your Python
code, so I/someone may find ways to speed it up, translate it to C or C
++ or CLisp or D, etc.

Note that I have written a kd-tree in both Python (with Psyco
compulsively) and D, this is the Psyco version:
http://code.activestate.com/recipes/572156/

Bye,
bearophile

sturlamolden

unread,
Sep 23, 2008, 2:07:22 PM9/23/08
to
On Sep 23, 3:44 pm, Robert Singer <rsinger@____.com> wrote:

> Well, python is not a number crunching language. However much we would
> like it to be (we would ? :-).

> No scripting language is.

Not even Matlab, R, IDL, Octave, SciLab, S-PLUS or Mathematica?


> Before resorting to rewriting the problem try psyco. It speeds up
> things sometimes.

I did, Psyco did not help.


> Also, (I'm not that familiar with python yet, so I don't know how to
> do it in python), try finding the bottlenecks of your calculation.

I did use a profiler, there is no particular single bottle-neck.


> Well, personally, I try to combine fortran (being a fortran programmer
> by trade) with python

Good compilers are too expensive, and gfortran is not good enough yet.

sturlamolden

unread,
Sep 23, 2008, 2:16:54 PM9/23/08
to
On Sep 23, 5:31 pm, bearophileH...@lycos.com wrote:

> Well written C++ code is generally faster or much faster than similar
> Python code, but programming in Python is often simpler, and it
> generally requires less time. So it may happen that to solve a problem
> a Python program that runs in 1 hour that requires 1 hour to be
> written allows you to find the solution in less time than a C++
> program that runs in 5-10 minutes that requires you 3-4 hours to be
> written :-)

I could easily sit down and wait half an hour. I could even wait a
week for the calculation to complete. But this is a program that will
be used many times, and not just by me.


> Note that C++ is just one option, but there are other languages
> around, like CLisp or D (or even a modern JavaVM), that are often an
> acceptable compromise between Python and C/C++.

Yes, if I knew Lisp, I would have used SBCL. Java I can program. But
it is a pain in the ass for any scientific programming. F# and OCaml
look promising though.

> Note that I have written a kd-tree in both Python (with Psyco
> compulsively) and D, this is the Psyco version:http://code.activestate.com/recipes/572156/

Sure I could show you the code, Python and C++, if I had a place to
post it. It looks very different form yours though.


namekuseijin

unread,
Sep 23, 2008, 2:48:33 PM9/23/08
to
On Sep 23, 10:57 am, Grant Edwards <gra...@visi.com> wrote:
> AFAICT, _everybody_ is bad at programming C++.

Thankfully, at least Numpy developers are not bad at C programming.

bearoph...@lycos.com

unread,
Sep 23, 2008, 2:52:16 PM9/23/08
to
sturlamolden:

>F# and OCaml look promising though.<

I bet on the future of D and Haskell (and maybe Fortress) instead :-)
We'll see.


>Sure I could show you the code, Python and C++, if I had a place to post it.<

I think the Python version suffices. If it's not too much private you
may post the single minimal/reduced runnable Python module here, it
will be deleted in some time (if you want you can also use a private
paste):
http://codepad.org/


>It looks very different form yours though.<

Is this a good or bad thing? ;-)

Bye,
bearophile

sturlamolden

unread,
Sep 23, 2008, 3:08:47 PM9/23/08
to
On Sep 23, 8:52 pm, bearophileH...@lycos.com wrote:

> I think the Python version suffices. If it's not too much private you
> may post the single minimal/reduced runnable Python module here, it
> will be deleted in some time (if you want you can also use a private
> paste):http://codepad.org/

http://codepad.org/rh8GzzJT

Available 24 hours from now.


> Is this a good or bad thing? ;-)

It's just facinating how different people working on similar problems
come up with different looking code.

Robert Kern

unread,
Sep 23, 2008, 3:17:15 PM9/23/08
to pytho...@python.org
bearoph...@lycos.com wrote:
> sturlamolden:

>> Sure I could show you the code, Python and C++, if I had a place to post it.<
>
> I think the Python version suffices. If it's not too much private you
> may post the single minimal/reduced runnable Python module here, it
> will be deleted in some time (if you want you can also use a private
> paste):
> http://codepad.org/

You could also drop it on the scipy.org wiki in the Cookbook category.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

J Peyret

unread,
Sep 23, 2008, 3:23:26 PM9/23/08
to
On Sep 23, 8:31 am, bearophileH...@lycos.com wrote:

Guys, this looks like a great data structure/algo for something I am
working on.

But... where do I find some definitions of the original BK-tree idea?
I looked through Amazon
and only a few books mention something like BK-Tree and these are
mostly conference minutes books, at ungodly prices.

I also did a quick Google on it and there isn't that much about the
subject.

http://blog.notdot.net/archives/30-Damn-Cool-Algorithms,-Part-1-BK-Trees.html

is the one I mostly saw referred.

So... 2 questions:

1. More bk-tree references? I can follow the code, but some
understanding of the background would be nice.

2. What, if any, is a good book to understand the basic of fuzzy/
string matching? Proximity/affinity problems? Or, more generally, a
good book on advanced algorithms?

No, I don't wanna read Knuth's just yet, something more modern/easy to
follow maybe? Something like 'Programming Collective Intelligence',
ISBN 0596529325, would be very nice, though it is perhaps a bit too
specific in its applications. Books using Java or C are fine. Lisp,
hmmm, well... I have trouble reading its notation, sorry.

Cheers

JLuc

sturlamolden

unread,
Sep 23, 2008, 3:49:45 PM9/23/08
to
On Sep 23, 9:17 pm, Robert Kern <robert.k...@gmail.com> wrote:

> You could also drop it on the scipy.org wiki in the Cookbook category.

Yes, if I could figure out how to use it...

Robert Kern

unread,
Sep 23, 2008, 3:53:13 PM9/23/08
to pytho...@python.org
J Peyret wrote:
> On Sep 23, 8:31 am, bearophileH...@lycos.com wrote:
>
> Guys, this looks like a great data structure/algo for something I am
> working on.
>
> But... where do I find some definitions of the original BK-tree idea?

Uh, actually we're talking about kd-trees, not BK-trees. kd-trees are for
searching through point sets in a k-dimensional space.

Robert Singer

unread,
Sep 23, 2008, 4:05:55 PM9/23/08
to
On Tue, 23 Sep 2008 11:07:22 -0700 (PDT), sturlamolden
<sturla...@yahoo.no> wrote:

>On Sep 23, 3:44 pm, Robert Singer <rsinger@____.com> wrote:
>
>> Well, python is not a number crunching language. However much we would
>> like it to be (we would ? :-).
>
>> No scripting language is.
>
>Not even Matlab, R, IDL, Octave, SciLab, S-PLUS or Mathematica?
>

No. And just to avoid eventual useless discussions which might arise,
I ment to say that *in general* compiled languages are faster. We can
always have discussions whether or not some newer scripting languages
like some from the above list, come close, but that usually is just
wasted time.

Specifically, I cannot say about R, IDL or S-PLUS, since I never used
those (not even heard of IDL till now). Octave and Mathematica have
been with me for such a short time (we had a few licences for
Wolfram's child for one year, but not my part of the company, so ...)
that I would rather not give my opinion about those.
I've used Matlab and Scilab for a longer time (still do actually -
Matlab for measurement data acquisition, and Scilab ... well, it just
sits on the disk somewhere actually), and although Matlab is quite
fast when disk I/O is involved, it still comes far.


>> Also, (I'm not that familiar with python yet, so I don't know how to
>> do it in python), try finding the bottlenecks of your calculation.
>
>I did use a profiler, there is no particular single bottle-neck.

You're talking about your c or your python version of the program?

There is always a bottleneck - that's just the part which works most
slowly. Try to find the part which takes the longest to execute, try
to put it differently. If it cannot be done, go to the next slowest
part.


>Good compilers are too expensive, and gfortran is not good enough yet.
>

?
Gfortran is one of the better compilers on the market. There was, just
the other day, a nice discussion on comp.lang.fortran how it is
marvellous what a group of enthousiasts managed do in their time, what
commercial giants still didn't.
May I ask what are your main objections to it ?

Best regards
Bob

Kay Schluehr

unread,
Sep 23, 2008, 4:15:57 PM9/23/08
to
On 23 Sep., 21:23, J Peyret <jpey...@gmail.com> wrote:
> On Sep 23, 8:31 am, bearophileH...@lycos.com wrote:
>
> Guys, this looks like a great data structure/algo for something I am
> working on.
>
> But... where do I find some definitions of the original BK-tree idea?

*geometric data structures*. Just google for it.

Robert Kern

unread,
Sep 23, 2008, 4:16:07 PM9/23/08
to pytho...@python.org

What's confusing? You do have to create a profile:

http://www.scipy.org/UserPreferences

sturlamolden

unread,
Sep 23, 2008, 4:34:10 PM9/23/08
to
On Sep 23, 10:05 pm, Robert Singer <rsinger@____.com> wrote:

> May I ask what are your main objections to it ?

1. gfortran is not Absoft.

2. If I program the same in C99 and Fortran 95, and compile with gcc
and gfortran, the C99 code runs a lot faster (I've only tested with
wavelet transforms).

3. gfortran is not Absoft.


sturlamolden

unread,
Sep 23, 2008, 4:38:10 PM9/23/08
to
On Sep 23, 10:16 pm, Robert Kern <robert.k...@gmail.com> wrote:

> What's confusing? You do have to create a profile:

How do I add a new page to the wiki? I'm only able to edit the front
page of the cookbook. But it doesn't help to add link there if I have
no page to link. (I may be incredibly stupid though.)

Robert Kern

unread,
Sep 23, 2008, 5:38:25 PM9/23/08
to pytho...@python.org

You just navigate to the URL you want:

http://www.scipy.org/Cookbook/KDTree

This will show you a page saying:

"""
This page does not exist yet. You can create a new empty page, or use one of the
page templates. Before creating the page, please check if a similar page already
exists.

Create new empty page
"""

The latter is a link that you can click.

Looking at the other Cookbook page sources, I notice that you should add the
following to the bottom of your page to categorize it appropriately.

"""
----
. CategoryCookbook
"""

Then put the link on the main Cookbook page (or do that first, then navigate
through the link to create the page).

Robert Singer

unread,
Sep 23, 2008, 6:41:47 PM9/23/08
to
On Tue, 23 Sep 2008 13:34:10 -0700 (PDT), sturlamolden
<sturla...@yahoo.no> wrote:

>1. gfortran is not Absoft.

I find this comment absurd. What did you mean by it ?
Yes, gfortran is not Absoft, just as red is not blue (?!).

I also don't understand whether you're looking for a free or a
commercial compiler. I got the impression from your previous post that
money was an object.

>2. If I program the same in C99 and Fortran 95, and compile with gcc
>and gfortran, the C99 code runs a lot faster (I've only tested with
>wavelet transforms).

Hmm. Unfortunatelly, i have none whatsoever experience in that field.
Completely different area of study here, so without seeing at least
some code, I cannot comment anything on that part.

And 'a lot faster' is a very relative term.

>
>3. gfortran is not Absoft.
>

True.

Best regards
Bob

sturlamolden

unread,
Sep 23, 2008, 7:05:33 PM9/23/08
to
On Sep 23, 8:52 pm, bearophileH...@lycos.com wrote:

> Is this a good or bad thing? ;-)

It seems we have been implementing different algorithms. kd-trees are
not BK-trees.

http://www.scipy.org/Cookbook/KDTree

Robert Kern

unread,
Sep 23, 2008, 8:09:44 PM9/23/08
to pytho...@python.org
Robert Kern wrote:

> J Peyret wrote:
>> On Sep 23, 8:31 am, bearophileH...@lycos.com wrote:
>>
>> Guys, this looks like a great data structure/algo for something I am
>> working on.
>>
>> But... where do I find some definitions of the original BK-tree idea?
>
> Uh, actually we're talking about kd-trees, not BK-trees. kd-trees are
> for searching through point sets in a k-dimensional space.

My apologies. I did not actually follow bearophile's link, and thought he was
talking about kd-trees like Sturla was.

bearoph...@lycos.com

unread,
Sep 23, 2008, 8:15:24 PM9/23/08
to
sturlamolden:

> It seems we have been implementing different algorithms. kd-trees are
> not BK-trees.
> http://www.scipy.org/Cookbook/KDTree

Sorry for my silly mistake :-)

Note: in your code I don't know if the collections.deque data
structure may help (it's faster than list for appending), but I
presume that's not a bottleneck.

Bye,
bearophile

Eric Brunel

unread,
Sep 24, 2008, 4:19:35 AM9/24/08
to
On Tue, 23 Sep 2008 15:23:12 +0200, sturlamolden <sturla...@yahoo.no>
wrote:
[...]

Would it be possible to post this text to some "persistent" web page with
(links to) the code you wrote in both languages? This would be a very
interesting resource for people experiencing some resistence when they
suggest Python as a possible language because of the 'Python is slow'
myth...
--
python -c "print ''.join([chr(154 - ord(c)) for c in
'U(17zX(%,5.zmz5(17l8(%,5.Z*(93-965$l7+-'])"

Message has been deleted

sturlamolden

unread,
Sep 24, 2008, 2:12:15 PM9/24/08
to

I have updated the cookbook entry for yesterday to also include
parallel processing for large data sets. Even if you're not interested
in kd-trees, it is a good example of what the new multiprocessing
standard module can do. There are still people being scared by the
GIL, thinking it prevents Python from utilizing multicore processors.

http://www.scipy.org/Cookbook/KDTree

David Cournapeau

unread,
Sep 25, 2008, 11:45:14 PM9/25/08
to sturlamolden, pytho...@python.org
On Wed, Sep 24, 2008 at 3:07 AM, sturlamolden <sturla...@yahoo.no> wrote:
> On Sep 23, 3:44 pm, Robert Singer <rsinger@____.com> wrote:
>
>> Well, python is not a number crunching language. However much we would
>> like it to be (we would ? :-).
>
>> No scripting language is.
>
> Not even Matlab, R, IDL, Octave, SciLab, S-PLUS or Mathematica?

I am fairly experienced in matlab (have been using it extensively for
5 years in academical context), and now with numpy, and generally,
they are comparable speed-wise. Matlab has some niceties which makes
it faster in some simple cases (JIT for loops, function calls faster,
sometimes COW semantics means it faster), but numpy (at its core at
least) is much more powerful IMHO. Also, matlab is horrible when you
want to interface some C to it (the C api is basically broken; in
particular, there is no way to gurantee you won't leak memory when you
Ctrl+C custom C extensions because the C api does not have facility to
deal with signals). I totally gave up matlab for numpy 2 years ago,
and never regretted it.

I think speed is not the issue when comparing matlab, R and co.
Availability of functionalities matter much more. R is quite hard to
beat if you need to do advanced statistics, specially since it is the
tool of choice for most academic statisticians. I hope numpy/scipy
will be there sometime, but it is honestly still quite far in that
domain.

cheers,

David

Aahz

unread,
Sep 26, 2008, 12:23:05 AM9/26/08
to
In article <Y7CdnYRzU-8naEXV...@posted.visi>,

Grant Edwards <gra...@visi.com> wrote:
>
>AFAICT, _everybody_ is bad at programming C++.
>
>One begins to suspect it's not the fault of the programmers.

http://www.netfunny.com/rhf/jokes/98/May/stroustrup.html
--
Aahz (aa...@pythoncraft.com) <*> http://www.pythoncraft.com/

"Argue for your limitations, and sure enough they're yours." --Richard Bach

sturlamolden

unread,
Sep 26, 2008, 6:08:52 AM9/26/08
to
On Sep 26, 5:45 am, "David Cournapeau" <courn...@gmail.com> wrote:

> I am fairly experienced in matlab (have been using it extensively for
> 5 years in academical context), and now with numpy, and generally,
> they are comparable speed-wise. Matlab has some niceties which makes
> it faster in some simple cases (JIT for loops, function calls faster,
> sometimes COW semantics means it faster),

I've used Matlab for 10 years.

Matlab has a horrible pass-by-value semantics, which means that arrays
are copied in and copied out form function calls. Yes there is a copy-
on-write optimization. But unless you are not careful, it will kill
performance and make memory usage skyrocket. Also, slicing creates a
new array, whereas in numpy slicing gives you a view of an existing
array. This generally makes Matlab slow, and it wastes terrible
amounts of memory.

For example I once tested D4 wavelet transforms on a 64 MB array of
doubles (i.e. length 2**23). Matlab R14 Service Pack 2 did this in 27
seconds, whereas Python 2.4 with NumPy 1.0 one required 3.4 seconds.
That is an order of magnitude speed difference in favour of Python.

Matlab also has a strange habit of fragmenting the heap. This can be
so bad that you have to stop the script, run a special function called
pack() do defragement, and restart. I've never seen that with Python.

I more or less stopped using Matlab 3 years ago, and I have not
renewed my subscription for maintenance of my personal Matlab license
since. Matlab is a nice tool, but I have grown past it.

Matlab's strongest side is data visualization though. Although we have
matplotlib, mayavi and possibility of interfacing with gnuplot, it's
not anywhere near the capabilities of Matlab.


Fly Away

unread,
Sep 26, 2008, 10:22:27 AM9/26/08
to

> Matlab's strongest side is data visualization though. Although we have
> matplotlib, mayavi and possibility of interfacing with gnuplot, it's
> not anywhere near the capabilities of Matlab.

What particular Matlab visualization features are you referring to? I
can't think of anything that would justify using the "not anywhere
near" term.

Cheers,
Victor.

Lawrence D'Oliveiro

unread,
Sep 29, 2008, 5:05:44 AM9/29/08
to
In message
<02918eb6-c2fb-4908...@x35g2000hsb.googlegroups.com>,
sturlamolden wrote:

> ... and possibility of interfacing with gnuplot ...

Gnuplot is non-Free software.

Fly Away

unread,
Sep 29, 2008, 12:25:43 PM9/29/08
to
On Sep 29, 3:05 am, Lawrence D'Oliveiro <l...@geek-
central.gen.new_zealand> wrote:
> In message
> <02918eb6-c2fb-4908-923f-d878a1956...@x35g2000hsb.googlegroups.com>,

>
> sturlamolden wrote:
> > ... and possibility of interfacing with gnuplot ...
>
> Gnuplot is non-Free software.

Yes, it is.

Victor.

bearoph...@lycos.com

unread,
Sep 29, 2008, 12:57:31 PM9/29/08
to
Lawrence D'Oliveiro:
>> Gnuplot is non-Free software.

Fly Away:
> Yes, it is.

From:
http://www.gnuplot.info/faq/faq.txt

1.7 Does gnuplot have anything to do with the FSF and the GNU project?
[...]
Gnuplot is freeware in the sense that you don't have to pay for it.
However
it is not freeware in the sense that you would be allowed to
distribute a
modified version of your gnuplot freely. [...]

Bye,
bearophile

Victor Prosolin

unread,
Sep 29, 2008, 1:37:36 PM9/29/08
to

Yes, I did read this prior to posting.

Victor.

r0g

unread,
Sep 29, 2008, 1:52:28 PM9/29/08
to


Well, ish. You can only distribute modifications to gnuplot itself as
patches, but you can distribute it freely and they publish the source
so, while it's not GPL free it's tending towards it.

Roger.

Lawrence D'Oliveiro

unread,
Sep 29, 2008, 9:50:26 PM9/29/08
to
In message <gbr4ks$1vv$1...@aioe.org>, r0g wrote:

> You can only distribute modifications to gnuplot itself as

> patches, but you can distribute it freely ...

This must be some new definition of "freely" of which I'm unaware.

r0g

unread,
Sep 30, 2008, 4:15:14 AM9/30/08
to

As in beer.

Lawrence D'Oliveiro

unread,
Sep 30, 2008, 4:38:09 AM9/30/08
to

You get free beer?

Steven D'Aprano

unread,
Sep 30, 2008, 4:52:11 AM9/30/08
to

You're free to distribute the official release of gnuplot.

You're free to distribute patches to gnuplot.

You're even free to provide people with a script or program to apply
those patches to gnuplot.


Where's the non-free bit?


Personally, I don't get the whole "only distribute patches" requirement.
It's a bit like saying "You're free to distribute this software, but only
as a tarball". It seems silly to me. But I don't see it as non-free,
except in the sense that "only licences approved by the FSF are free".

--
Steven

Ben Finney

unread,
Sep 30, 2008, 5:04:41 AM9/30/08
to
Steven D'Aprano <ste...@REMOVE.THIS.cybersource.com.au> writes:

> On Tue, 30 Sep 2008 14:50:26 +1300, Lawrence D'Oliveiro wrote:
>
> > In message <gbr4ks$1vv$1...@aioe.org>, r0g wrote:
> >
> >> You can only distribute modifications to gnuplot itself as
> >> patches, but you can distribute it freely ...

[…]

> Where's the non-free bit?

You're not free to modify gnuplot and redistribute the result.

That you're free to distribute patches is nice, but it's not enough to
make the work free. The freedom to help people by giving them an
*already-modified* gnuplot is restricted by the copyright holder.

It's an artificial restriction on redistribution of derived works,
making them second-class for the prupose of getting them into people's
hands.

> Personally, I don't get the whole "only distribute patches"
> requirement. It's a bit like saying "You're free to distribute this
> software, but only as a tarball". It seems silly to me.

That, too, would be a non-free requirement.

> But I don't see it as non-free, except in the sense that "only
> licences approved by the FSF are free".

I try to judge freedom of a software work by the freedoms granted to
all recipients of the work, not by the approval of some organisation.

--
\ “When I turned two I was really anxious, because I'd doubled my |
`\ age in a year. I thought, if this keeps up, by the time I'm six |
_o__) I'll be ninety.” —Steven Wright |
Ben Finney

Steven D'Aprano

unread,
Sep 30, 2008, 6:04:18 AM9/30/08
to
On Tue, 30 Sep 2008 19:04:41 +1000, Ben Finney wrote:

> Steven D'Aprano <ste...@REMOVE.THIS.cybersource.com.au> writes:
>
>> On Tue, 30 Sep 2008 14:50:26 +1300, Lawrence D'Oliveiro wrote:
>>
>> > In message <gbr4ks$1vv$1...@aioe.org>, r0g wrote:
>> >
>> >> You can only distribute modifications to gnuplot itself as patches,
>> >> but you can distribute it freely ...
> […]
>
>> Where's the non-free bit?
>
> You're not free to modify gnuplot and redistribute the result.
>
> That you're free to distribute patches is nice, but it's not enough to
> make the work free. The freedom to help people by giving them an
> *already-modified* gnuplot is restricted by the copyright holder.
>
> It's an artificial restriction on redistribution of derived works,
> making them second-class for the prupose of getting them into people's
> hands.

Yes it is. It seems a strange, unnecessary restriction. But is it
sufficient to make it non-free? I don't think so.

In case you are thinking that gnuplot allows people to *only* distribute
the diffs, not the original source to apply the diffs onto, that is not
the case. I quote from gnuplot > help copyright

"Permission to distribute the released version of the source code along
with corresponding source modifications in the form of a patch file is
granted with same provisions 2 through 4 for binary distributions."

Those provisions aren't terribly onerous, although #3 may be considered a
privacy issue:

2. add special version identification to distinguish your version
in addition to the base release version number,
3. provide your name and address as the primary contact for the
support of your modified version, and
4. retain our contact information in regard to use of the base
software.


>> Personally, I don't get the whole "only distribute patches"
>> requirement. It's a bit like saying "You're free to distribute this
>> software, but only as a tarball". It seems silly to me.
>
> That, too, would be a non-free requirement.
>
>> But I don't see it as non-free, except in the sense that "only licences
>> approved by the FSF are free".
>
> I try to judge freedom of a software work by the freedoms granted to all
> recipients of the work, not by the approval of some organisation.

Yes, but you accept some restrictions as legitimate. For example, you
accept the restriction that the GPL makes that says you may not
redistribute a modified work without making the source code available.
That's a restriction, but it's not enough to disqualify it from being a
free software licence. In fact, that restriction is *necessary* to make
it a free software licence in the sense we're talking about. So "free"
does not mean "no restrictions", it merely means "none of some sorts of
restrictions, but other restrictions are okay". Likewise the restriction
that GPL software must be distributed with a copy of the appropriate
licence.

It is useful to compare the "diffs only" licence to two different GPL-
related scenarios. Scenario one is clearly against the spirit of the GPL,
and possibly (hopefully!) the letter as well. Scenario two is not.

(1) I distribute the modified source code encrypted and charge $1,000,000
for a NON-TRANSFERABLE licence to the encryption key. If you don't have
the encryption key, that's your bad luck.

(2) I distribute the modified source code archived in a tar file, and
refuse to offer it in any other format. If you don't have an untar
application, that's your bad luck.

It's my contention that the restriction of supplying diffs is closer to
Scenario 2 than to Scenario 1. The modified source is supplied, but it is
split into two pieces: the official source, plus a set of diffs.
Reversing that to get the modified source is not much more difficult than
untarring a tarball.


--
Steven

Ben Finney

unread,
Sep 30, 2008, 8:19:57 AM9/30/08
to
Steven D'Aprano <ste...@REMOVE.THIS.cybersource.com.au> writes:

> On Tue, 30 Sep 2008 19:04:41 +1000, Ben Finney wrote:
> > You're not free to modify gnuplot and redistribute the result.
> >
> > That you're free to distribute patches is nice, but it's not
> > enough to make the work free. The freedom to help people by giving
> > them an *already-modified* gnuplot is restricted by the copyright
> > holder.
> >
> > It's an artificial restriction on redistribution of derived works,
> > making them second-class for the prupose of getting them into
> > people's hands.
>
> Yes it is. It seems a strange, unnecessary restriction. But is it
> sufficient to make it non-free? I don't think so.

I do, because a natural, beneficial act (modify the work and
redistribute it) that has no technical reason to restrict, is
artifically restricted.

> In case you are thinking that gnuplot allows people to *only*
> distribute the diffs, not the original source to apply the diffs
> onto, that is not the case. I quote from gnuplot > help copyright
>
> "Permission to distribute the released version of the source code
> along with corresponding source modifications in the form of a patch
> file is granted with same provisions 2 through 4 for binary
> distributions."

That's what I refer to when I say that it artifically makes derived
works into second-class for the purpose of doing the beneficial act of
distributing them: the redistributor is artificially restricted from
making the work as useful as the original they received.

They have only the options to redistribute a work that is more
cumbersome for the recipient of that work, or not to redistribute at
all. That's not free redistribution.

> > I try to judge freedom of a software work by the freedoms granted
> > to all recipients of the work, not by the approval of some
> > organisation.
>
> Yes, but you accept some restrictions as legitimate. For example, you
> accept the restriction that the GPL makes that says you may not
> redistribute a modified work without making the source code available.

Yes, which is why I was careful to say "the freedoms granted to all
recipients of the work".

The power to restrict a recipient of one's work (by choosing not to
grant them the freedoms you yourself had when you received the work)
reduces the freedoms available to all recipients of the work, even
though one party's power may be increased.

This is where the useful "your freedom to swing your fist ends at the
tip of the other man's nose" applies: As soon as the act you wish to
perform is restricting the freedom of another, you're not
contemplating an act of freedom, but an act of power over another.
Freedoms should be protected, but only within the limits imposed by
the freedoms of others.

> That's a restriction, but it's not enough to disqualify it from
> being a free software licence.

Specifically because it upholds the freedom of the recipient of a
derived work from having power exerted over them.

> In fact, that restriction is *necessary* to make it a free software
> licence in the sense we're talking about.

Not really; it's necessary to make it a copyleft license, which is a
way of preserving freedom as the work gets passed along.

Works can still be free software without being copyleft-licensed,
though. A license allowing free redistribution and requiring only
attribution be preserved is less restrictive than a copyleft; yet,
because it allows any free act (even as it also allows acts of power
over others), the work is free software.

> So "free" does not mean "no restrictions", it merely means "none of
> some sorts of restrictions, but other restrictions are okay".
> Likewise the restriction that GPL software must be distributed with
> a copy of the appropriate licence.

That's right, and I've explained above what restrictions I consider
justified, and why, and how to tell the difference.

--
\ “Reichel's Law: A body on vacation tends to remain on vacation |
`\ unless acted upon by an outside force.” —Carol Reichel |
_o__) |
Ben Finney

Paul Boddie

unread,
Sep 30, 2008, 8:31:19 AM9/30/08
to
On 30 Sep, 14:19, Ben Finney <bignose+hates-s...@benfinney.id.au>
wrote:

>
> This is where the useful "your freedom to swing your fist ends at the
> tip of the other man's nose" applies: As soon as the act you wish to
> perform is restricting the freedom of another, you're not
> contemplating an act of freedom, but an act of power over another.
> Freedoms should be protected, but only within the limits imposed by
> the freedoms of others.

This is a very good explanation of what copyleft is all about. I
suppose one could regard copyleft as a means to preserve the "maximal
common freedom" in a system - if anyone else were to acquire more
power or privilege to do something, that would diminish the freedoms
of others.

Paul

Steven D'Aprano

unread,
Sep 30, 2008, 9:43:58 AM9/30/08
to
On Tue, 30 Sep 2008 22:19:57 +1000, Ben Finney wrote:

> Steven D'Aprano <ste...@REMOVE.THIS.cybersource.com.au> writes:
>
>> On Tue, 30 Sep 2008 19:04:41 +1000, Ben Finney wrote:
>> > You're not free to modify gnuplot and redistribute the result.
>> >
>> > That you're free to distribute patches is nice, but it's not enough
>> > to make the work free. The freedom to help people by giving them an
>> > *already-modified* gnuplot is restricted by the copyright holder.
>> >
>> > It's an artificial restriction on redistribution of derived works,
>> > making them second-class for the prupose of getting them into
>> > people's hands.
>>
>> Yes it is. It seems a strange, unnecessary restriction. But is it
>> sufficient to make it non-free? I don't think so.
>
> I do, because a natural, beneficial act (modify the work and
> redistribute it) that has no technical reason to restrict, is
> artifically restricted.

We agree that the restriction is artificial, and I think irrational
(although I'd be interested in hearing the gnuplot developers' reasoning
before making a final judgment).

But I just don't see the requirement that modified software be
distributed in form X (original source + diffs) versus form Y (modified
source in a tar ball) or form Z (an rpm) to be that big a deal. Not
enough to make it "non-free software".

I simply don't think that having to run some variation on

patch -i patchfile.patch

is a requirement so onerous that it makes the gnuplot licence non-free.
Perhaps I'm just more tolerant of eccentricities than you :)


--
Steven

George Sakkis

unread,
Sep 30, 2008, 11:04:35 AM9/30/08
to
On Sep 30, 9:43 am, Steven D'Aprano <st...@REMOVE-THIS-
cybersource.com.au> wrote:

What you're missing is that for Free Software (TM) zealots it's a
matter of philosophical principle, totally unrelated to how easy is to
overcome the restriction. There is not a "practicality beats purity"
clause in the FSF Bible.

George

José Matos

unread,
Sep 30, 2008, 2:14:08 PM9/30/08
to pytho...@python.org
On Tuesday 30 September 2008 16:04:35 George Sakkis wrote:
> What you're missing is that for Free Software (TM) zealots it's a
> matter of philosophical principle, totally unrelated to how easy is to
> overcome the restriction. There is not a "practicality beats purity"
> clause in the FSF Bible.

The gnuplot license is a free software according to FSF, what is the problem
here after all?

> George

--
José Abílio

Terry Reedy

unread,
Sep 30, 2008, 2:18:15 PM9/30/08
to pytho...@python.org
Steven D'Aprano wrote:
> On Tue, 30 Sep 2008 22:19:57 +1000, Ben Finney wrote:

>> I do, because a natural, beneficial act (modify the work and
>> redistribute it) that has no technical reason to restrict, is
>> artifically restricted.
>
> We agree that the restriction is artificial, and I think irrational
> (although I'd be interested in hearing the gnuplot developers' reasoning
> before making a final judgment).

I believe it is a matter of preserving clarity of authorship, just as is
the quoting mechanism we take for granted in posts like this. If I
removed the quote marks above and silently edited what Ben and you
wrote, I might upset someone and certainly could confuse readers.

tjr


Ben Finney

unread,
Sep 30, 2008, 6:59:09 PM9/30/08
to
Steven D'Aprano <st...@REMOVE-THIS-cybersource.com.au> writes:

> I simply don't think that having to run some variation on
>
> patch -i patchfile.patch
>
> is a requirement so onerous that it makes the gnuplot licence
> non-free. Perhaps I'm just more tolerant of eccentricities than you
> :)

The distinction here is that this command must be run by *every*
recipient of a modified work. A work where one must do that is more
onerous for *each* recipient than one where it's already been patched
for the recipient.

Thus there is value, and no loss of freedom, in you as a redistributor
doing that work *once* and then redistributing the work intact to any
recipient. Your freedom to do this useful, harmless action is
restricted artificially by copyright, and is not granted by the
license.

So, recipients of the 'gnuplot' code are artificially restricted from
performing an action useful to society that does no harm.

--
\ “The Bermuda Triangle got tired of warm weather. It moved to |
`\ Alaska. Now Santa Claus is missing.” —Steven Wright |
_o__) |
Ben Finney

Ben Finney

unread,
Sep 30, 2008, 7:06:08 PM9/30/08
to
Terry Reedy <tjr...@udel.edu> writes:

> Steven D'Aprano wrote:
> > We agree that the restriction is artificial, and I think
> > irrational (although I'd be interested in hearing the gnuplot
> > developers' reasoning before making a final judgment).
>
> I believe it is a matter of preserving clarity of authorship, just
> as is the quoting mechanism we take for granted in posts like this.
> If I removed the quote marks above and silently edited what Ben and
> you wrote, I might upset someone and certainly could confuse
> readers.

That, if it were to be prosecuted under law, would be a matter already
covered by laws other than copyright: fraud, libel, etc.

Note that I consider a work free even if it fails to grant “the right
to distribute misrepresentations of the author's words”, because that
act is an exercise of undue power over another person, and so falls
outside the limit imposed by the freedoms of others.

--
\ “What is it that makes a complete stranger dive into an icy |
`\ river to save a solid gold baby? Maybe we'll never know.” —Jack |
_o__) Handey |
Ben Finney

Steven D'Aprano

unread,
Sep 30, 2008, 7:54:18 PM9/30/08
to
On Wed, 01 Oct 2008 09:06:08 +1000, Ben Finney wrote:

> Terry Reedy <tjr...@udel.edu> writes:
>
>> Steven D'Aprano wrote:
>> > We agree that the restriction is artificial, and I think irrational
>> > (although I'd be interested in hearing the gnuplot developers'
>> > reasoning before making a final judgment).
>>
>> I believe it is a matter of preserving clarity of authorship, just as
>> is the quoting mechanism we take for granted in posts like this. If I
>> removed the quote marks above and silently edited what Ben and you
>> wrote, I might upset someone and certainly could confuse readers.
>
> That, if it were to be prosecuted under law, would be a matter already
> covered by laws other than copyright: fraud, libel, etc.
>
> Note that I consider a work free even if it fails to grant “the right to
> distribute misrepresentations of the author's words”, because that act
> is an exercise of undue power over another person, and so falls outside
> the limit imposed by the freedoms of others.


But distributing modified source code *does* misrepresent the author's
words, because you confuse authorship. Given only the modified version of
the source code, how is the recipient supposed to identify which parts of
the source code were written by the original authors and which parts
where written by you?

If that is why the gnuplot people do not allow you to distribute such
modified documents, then the only "freedom" they fail to grant is exactly
the one you don't consider necessary for a free licence: "the right to
distribute misrepresentations of the author's words".


--
Steven

Ben Finney

unread,
Sep 30, 2008, 8:52:40 PM9/30/08
to
Steven D'Aprano <st...@REMOVE-THIS-cybersource.com.au> writes:

> On Wed, 01 Oct 2008 09:06:08 +1000, Ben Finney wrote:
>
> > Note that I consider a work free even if it fails to grant “the
> > right to distribute misrepresentations of the author's words”,
> > because that act is an exercise of undue power over another
> > person, and so falls outside the limit imposed by the freedoms of
> > others.
>
> But distributing modified source code *does* misrepresent the
> author's words, because you confuse authorship.

That's a possibility. There are other ways to avoid it than to
restrict the freedom to redistribute; for example, some licenses state
that modified works must clearly state who, when, and what was
modified. I would not consider that a non-free restriction, since the
freedom of the original distributor *and* the freedom of the
redistributor is preserved.

> If that is why the gnuplot people do not allow you to distribute
> such modified documents, then the only "freedom" they fail to grant
> is exactly the one you don't consider necessary for a free licence:
> "the right to distribute misrepresentations of the author's words".

We're going around in circles. I've already pointed out another
freedom they fail to grant: the freedom to redistribute a modified
work *as modified*. It's not equivalent to the act of
misrepresentation.

--
\ “War is God's way of teaching geography to Americans.” —Ambrose |
`\ Bierce |
_o__) |
Ben Finney

greg

unread,
Oct 3, 2008, 2:00:43 AM10/3/08
to
Steven D'Aprano wrote:

> We agree that the restriction is artificial, and I think irrational

I think it's irrational for another reason, too -- it's
actually vacuous. There's nothing to prevent you creating
a set of patches that simply say "Delete all of the original
source and replace it with the following".

Then you're effectively distributing the modified source in
its entirety, just with a funny header at the top of each
source file that serves no useful purpose.

--
Greg

Terry Reedy

unread,
Oct 3, 2008, 4:00:27 PM10/3/08
to pytho...@python.org

The useful purpose is to show that you are distributing your work under
someone else's product name, instead of making up your own as you ought to.


Lawrence D'Oliveiro

unread,
Oct 5, 2008, 6:57:47 PM10/5/08
to
In message <mailman.1976.1223064...@python.org>, Terry
Reedy wrote:

Except that the approach Terry Reedy gets around that without violating the
licence.

Lawrence D'Oliveiro

unread,
Oct 5, 2008, 7:01:50 PM10/5/08
to
In message <mailman.1764.1222798...@python.org>, José Matos
wrote:

> The gnuplot license is a free software according to FSF ...

Not listed as one
<http://www.fsf.org/licensing/licenses/index_html/view?searchterm=license>.

Lawrence D'Oliveiro

unread,
Oct 5, 2008, 7:03:34 PM10/5/08
to
In message <87hc7xq...@benfinney.id.au>, Ben Finney wrote:

> Note that I consider a work free even if it fails to grant “the right
> to distribute misrepresentations of the author's words”, because that
> act is an exercise of undue power over another person, and so falls
> outside the limit imposed by the freedoms of others.

That's the difference between software and, say, an artistic work like a
novel, poem or illustration. Software is nearly always a work in progress.
That's why we have Free Software licences for the former, and Creative
Commons licences for the latter.

José Matos

unread,
Feb 15, 2009, 6:36:21 PM2/15/09
to pytho...@python.org
On Monday 06 October 2008 00:01:50 Lawrence D'Oliveiro wrote:
> Not listed as one
> <http://www.fsf.org/licensing/licenses/index_html/view?searchterm=license>.

Look further
http://directory.fsf.org/project/gnuplot/

--
José Abílio

Reply all
Reply to author
Forward
0 new messages