How to install Python package from source on Windows

[Deborah Swanson]

I want to install the recordclass package:
https://pypi.python.org/pypi/recordclass

But they've only released wheel files for two platforms, macosx and
win_amd64, neither of which will install on my system. I need win_x86 or
intel_x86, which they don't provide.

The only alternative to pip install that I know of is to install from
source, and the source I need is at:
https://bitbucket.org/intellimath/recordclass/src

I've never done this, so any help would be appreciated.
Deborah

[eryk sun]

On Sun, May 14, 2017 at 8:46 PM, Deborah Swanson
<pyt...@deborahswanson.net> wrote:
> I want to install the recordclass package:
> https://pypi.python.org/pypi/recordclass
>
> But they've only released wheel files for two platforms, macosx and
> win_amd64, neither of which will install on my system. I need win_x86 or
> intel_x86, which they don't provide.

The tag for 32-bit Windows is "win32". The PyPI page has win32 wheels
for 2.7-3.5.

If you're using 3.6, you'll have to build from source. The package has
a single C extension without external dependencies, so it should be a
straight-forward build if you have Visual Studio 2015+ installed with
the C/C++ compiler for x86. Ideally it should work straight from pip.
But I tried and it failed in 3.6.1 due to the new PySlice_GetIndicesEx
macro. Apparently MSVC doesn't like preprocessor code like this in
memoryslots.c:

#if PY_MAJOR_VERSION >= 3
if (PySlice_GetIndicesEx(item, Py_SIZE(self),
#else
if (PySlice_GetIndicesEx((PySliceObject*)item, Py_SIZE(self),
#endif
&start, &stop, &step, &slicelength) < 0) {

It fails with a C1057 error (unexpected end of file in macro
expansion). The build will succeed if you copy the common line with
`&start` to each case and comment out the original line, such that the
macro invocation isn't split across an #if / #endif. This is an ugly
consequence of making PySlice_GetIndicesEx a macro. I wonder if it
could be written differently to avoid this problem.

[Deborah Swanson]

eryk sun wrote, on Sunday, May 14, 2017 7:15 PM

>
> On Sun, May 14, 2017 at 8:46 PM, Deborah Swanson
> <pyt...@deborahswanson.net> wrote:
> > I want to install the recordclass package:
> > https://pypi.python.org/pypi/recordclass
> >
> > But they've only released wheel files for two platforms, macosx and
> > win_amd64, neither of which will install on my system. I
> need win_x86
> > or intel_x86, which they don't provide.
>
> The tag for 32-bit Windows is "win32". The PyPI page has
> win32 wheels for 2.7-3.5.

I'll look at the PyPi page again, it's true I skipped over the Python2
builds, but if that turns out tobe my best option, I can rewrite my
project for Python2.

> If you're using 3.6, you'll have to build from source. The
> package has a single C extension without external
> dependencies, so it should be a straight-forward build if you
> have Visual Studio 2015+ installed with the C/C++ compiler
> for x86.

Unfortunately I don't have Visual Studio 2015+ installed and I can't
install it on Windows XP SP2 (plus I really don't want to). Probably I
should have mentioned that, but I didn't know I'd need to build C/C++.

> Ideally it should work straight from pip. But I
> tried and it failed in 3.6.1 due to the new
> PySlice_GetIndicesEx macro. Apparently MSVC doesn't like
> preprocessor code like this in
> memoryslots.c:
>
> #if PY_MAJOR_VERSION >= 3
> if (PySlice_GetIndicesEx(item, Py_SIZE(self),
> #else
> if (PySlice_GetIndicesEx((PySliceObject*)item,
> Py_SIZE(self),
> #endif
> &start, &stop, &step,
> &slicelength) < 0) {
>
> It fails with a C1057 error (unexpected end of file in macro
> expansion). The build will succeed if you copy the common
> line with `&start` to each case and comment out the original
> line, such that the macro invocation isn't split across an
> #if / #endif. This is an ugly consequence of making
> PySlice_GetIndicesEx a macro. I wonder if it could be written
> differently to avoid this problem.

There isn't any solution or workaround for me if building C/C++ is
required.

[eryk sun]

On Mon, May 15, 2017 at 4:19 AM, Deborah Swanson
<pyt...@deborahswanson.net> wrote:
> Unfortunately I don't have Visual Studio 2015+ installed and I can't
> install it on Windows XP SP2 (plus I really don't want to). Probably I
> should have mentioned that, but I didn't know I'd need to build C/C++.

3.5+ doesn't work in XP, so the highest version you can run is 3.4.
For 32-bit, install recordclass-0.4.3-cp34-cp34m-win32.whl.

[Chris Angelico]

The best way to install this sort of thing is using pip. You should
have it with your Python 3.4, and it'll do all the work of figuring
out which one to download.

ChrisA

[Deborah Swanson]

Chris Angelico wrote, on Sunday, May 14, 2017 9:56 PM

> --

That was easy to try, "pip install recordclass", but it failed:

C:\Programs\Coding\Anaconda3\Scripts>pip install recordclass
You are using pip version 7.0.3, however version 9.0.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade
pip' comm
and.
Collecting recordclass
Downloading recordclass-0.4.3.tar.gz
Installing collected packages: recordclass
Running setup.py install for recordclass
Complete output from command C:\Programs\Coding\Anaconda3\python.exe
-c "imp
ort setuptools,
tokenize;__file__='E:\\temp\\pip-build-zhximvpd\\recordclass\\se
tup.py';exec(compile(getattr(tokenize, 'open',
open)(__file__).read().replace('\
r\n', '\n'), __file__, 'exec'))" install --record
E:\temp\pip-r7ky4fv1-record\in
stall-record.txt --single-version-externally-managed --compile:
running install
running build
running build_py
creating build
creating build\lib.win32-3.4
creating build\lib.win32-3.4\recordclass
copying lib\recordclass\record.py -> build\lib.win32-3.4\recordclass
copying lib\recordclass\__init__.py ->
build\lib.win32-3.4\recordclass
creating build\lib.win32-3.4\recordclass\test
copying lib\recordclass\test\test_memoryslots.py ->
build\lib.win32-3.4\reco
rdclass\test
copying lib\recordclass\test\test_record.py ->
build\lib.win32-3.4\recordcla
ss\test
copying lib\recordclass\test\__init__.py ->
build\lib.win32-3.4\recordclass\
test
running build_ext
building 'recordclass.memoryslots' extension
error: Microsoft Visual C++ 10.0 is required (Unable to find
vcvarsall.bat).


----------------------------------------
Command "C:\Programs\Coding\Anaconda3\python.exe -c "import setuptools,
tokenize
;__file__='E:\\temp\\pip-build-zhximvpd\\recordclass\\setup.py';exec(com
pile(get
attr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'),
__file__, '
exec'))" install --record E:\temp\pip-r7ky4fv1-record\install-record.txt
--singl
e-version-externally-managed --compile" failed with error code 1 in
E:\temp\pip-
build-zhximvpd\recordclass



Looks like lack of Visual C++ tripped me up again.

Strange how Python code can't install with purely Python code. I don't
plan to have Visual anything on Linux either.

It might be possible to do something with

lib\recordclass\test\test_record.py ->
build\lib.win32-3.4\recordclass\test and

lib\recordclass\record.py -> build\lib.win32-3.4\recordclass,

but I wouldn't know what.

Maybe I should go back to Python 2.7.8 for this.

Deborah

[Deborah Swanson]

eryk sun wrote, on Sunday, May 14, 2017 9:27 PM

>
> On Mon, May 15, 2017 at 4:19 AM, Deborah Swanson
> <pyt...@deborahswanson.net> wrote:
> > Unfortunately I don't have Visual Studio 2015+ installed
> and I can't
> > install it on Windows XP SP2 (plus I really don't want to).
> Probably I
> > should have mentioned that, but I didn't know I'd need to
> build C/C++.
>
> 3.5+ doesn't work in XP, so the highest version you can run
> is 3.4. For 32-bit, install recordclass-0.4.3-cp34-cp34m-win32.whl.
>

I have 3.4.3 installed. That was the last build it was possible to get
out of Anaconda for XP, and there's lots of goodies I have to do without
or work around.

But, "pip install recordclass-0.4.3-cp34-cp34m-win32.whl" fails with:

"recordclass-0.4.3-cp34-cp34m-win32.whl is not a supported wheel on this
platform."

Probably because it wants to use C++, or Visual something (higher than
version 4).

Unless you know how I can install recordclass without using pip, and
recordclass-0.4.3-cp34-cp34m-win32.whl won't fail if it can't find
Visual C++.

Again, maybe I should go back to Python 2 for this.

Deborah

[Chris Angelico]

On Mon, May 15, 2017 at 4:14 PM, Deborah Swanson
<pyt...@deborahswanson.net> wrote:
> Looks like lack of Visual C++ tripped me up again.
>
> Strange how Python code can't install with purely Python code. I don't
> plan to have Visual anything on Linux either.
>
> It might be possible to do something with
>
> lib\recordclass\test\test_record.py ->
> build\lib.win32-3.4\recordclass\test and
>
> lib\recordclass\record.py -> build\lib.win32-3.4\recordclass,
>
> but I wouldn't know what.
>

> Maybe I should go back to Python 2.7.8 for this.

That's the thing, though: it is NOT purely Python code. The module
you're trying to install is implemented in C, which is why it needs a
C compiler. It may be that Anaconda is breaking the wheels, which
means that you can't use the precompiled binaries; you could try using
conda to install the module. Otherwise, try installing vanilla CPython
3.4, and see if that helps.

Or get yourself off XP, because it's not supported any more anyway.

ChrisA

[Chris Angelico]

On Mon, May 15, 2017 at 4:28 PM, Deborah Swanson
<pyt...@deborahswanson.net> wrote:
>
> Again, maybe I should go back to Python 2 for this.

Won't help. The same problems will exist.

ChrisA

[Deborah Swanson]

eryk sun wrote on Sunday, May 14, 2017 9:27 PM

>
> On Mon, May 15, 2017 at 4:19 AM, Deborah Swanson
> <pyt...@deborahswanson.net> wrote:
> > Unfortunately I don't have Visual Studio 2015+ installed
> and I can't
> > install it on Windows XP SP2 (plus I really don't want to).
> Probably I
> > should have mentioned that, but I didn't know I'd need to
> build C/C++.
>
> 3.5+ doesn't work in XP, so the highest version you can run
> is 3.4. For 32-bit, install recordclass-0.4.3-cp34-cp34m-win32.whl.
>

Where did you find recordclass-0.4.3-cp34-cp34m-win32.whl? There
weren't any win32 builds on https://pypi.python.org/pypi/recordclass.

Maybe I can find an earlier 3 build that won't demand Visual C++.

[Deborah Swanson]

Chris Angelico wrote on Sunday, May 14, 2017 11:30 PM

Good to know. Maybe I just won't be able to try recordclass until I get
Linux set up one way or another. Right now it's more important to me to
spend the few hours a day I have making progress in Python.

(Now, if I can figure out how to make gettattr work in this one
namedtuple problem, I won't need anything like recordclass. I've
converted three more Excel spreadsheets to Python, but it's tricky
working with namedtuples when you want to group rows or take n colomns
at a time. I've done it, but this one is really tough.)

[Chris Angelico]

On Mon, May 15, 2017 at 4:48 PM, Deborah Swanson

<pyt...@deborahswanson.net> wrote:
> Chris Angelico wrote on Sunday, May 14, 2017 11:30 PM
>>
>> On Mon, May 15, 2017 at 4:28 PM, Deborah Swanson
>> <pyt...@deborahswanson.net> wrote:
>> >
>> > Again, maybe I should go back to Python 2 for this.
>>
>> Won't help. The same problems will exist.
>>
>> ChrisA
>>
>
> Good to know. Maybe I just won't be able to try recordclass until I get
> Linux set up one way or another. Right now it's more important to me to
> spend the few hours a day I have making progress in Python.
>
> (Now, if I can figure out how to make gettattr work in this one
> namedtuple problem, I won't need anything like recordclass. I've
> converted three more Excel spreadsheets to Python, but it's tricky
> working with namedtuples when you want to group rows or take n colomns
> at a time. I've done it, but this one is really tough.)

Hmm. If you're working heavily with tabular data, you may want to
consider numpy or pandas, both of which are designed for major
computational work and will happily manipulate rows or columns of
data. Both are well supported on Windows (and everywhere). They do
have their own learning curve though.

ChrisA

[Deborah Swanson]

Chris Angelico wrote, on Monday, May 15, 2017 1:02 AM

> --

Yes. I want to spend more time with numpy and learn pandas, but right
now I do have to consider the learning curves involved. Maybe in 4-5
months I'll pull out of this wretched state I'm in now, and I'll spread
my wings in dozens of directions then - for about a year, don't ask me
how I know. (I haven't calculated out beyond 2017 yet.)

But there's another issue too. Contrary to popular belief, just because
spreadsheets are what you build on a rectangular grid of cells, not many
advanced spreadsheet applications are straight-up tabular. Most
beginners start out tabular (though I didn't, a couple decades ago), and
it seems that most never get out of the tabular framework.

I've been finding that namedtuples covers a good 90-95% of my "not
exactly tabular" (and "not at all tabular") spreadsheet applications,
and I'm quite curious to keep extending my knowledge about what you can
do with namedtuples. And it probably would be surprising to some how
much that is. It's mainly a few awkward corner cases, like grouping rows
and taking columns n at a time that can really be tricky. (But I've done
both now.)

Eventually I plan to know several tools to convert spreadsheets to
Python with, including numpy and pandas. But considering the "tabular"
discipline both of those have, I doubt I'd be saving myself any grief by
switching to them.

Learning curves are a bitch in their own right, and I can think of worse
ways to spend a few broken months than getting to know one way really
well. It's my fallback position when things get bad, to just stick with
what I don't have to learn too much about and find out what I can do
with it. You'd be surprised!

Deborah

[bream...@gmail.com]

There are win32 builds on https://pypi.python.org/pypi/recordclass for 2.7 and 3.3/4/5.

Kindest regards.

Mark Lawrence.

[eryk sun]

On Mon, May 15, 2017 at 6:37 AM, Deborah Swanson
<pyt...@deborahswanson.net> wrote:
>
> Where did you find recordclass-0.4.3-cp34-cp34m-win32.whl? There
> weren't any win32 builds on https://pypi.python.org/pypi/recordclass.

It's in the middle of the file list:

recordclass-0.4.3-cp34-cp34m-win32.whl (md5) Python Wheel cp34
2017-04-17 17KB

Maybe you're looking at a cached page in your browser? But that
doesn't explain why pip doesn't see it.

> Maybe I can find an earlier 3 build that won't demand Visual C++.

The wheel doesn't need a compiler. It has an ABI tag because it
already includes the compiled extension module.

[Deborah Swanson]

eryk sun wrote, on Monday, May 15, 2017 5:52 AM

>
> On Mon, May 15, 2017 at 6:37 AM, Deborah Swanson
> <pyt...@deborahswanson.net> wrote:
> >
> > Where did you find recordclass-0.4.3-cp34-cp34m-win32.whl? There
> > weren't any win32 builds on
> https://pypi.python.org/pypi/recordclass.
>
> It's in the middle
> of the file list:
>
> recordclass-0.4.3-cp34-cp34m-win32.whl (md5) Python Wheel
> cp34 2017-04-17 17KB

Actually, I see now there are several win32.whl files on that page. I'm
not familiar with PyPi's link formatting and I didn't realize that both
the wheel and md5 files were available from each of those links, and I
only clicked on the md5s. (Truth be told I didn't even see the .whls. It
was late, and though I could read and write, I was a little bleary
around the edges.)

> Maybe you're looking at a cached page in your browser? But
> that doesn't explain why pip doesn't see it.

Actually, when I tried "pip install
recordclass-0.4.3-cp34-cp34m-win32.whl" it failed almost immediately

with:

"recordclass-0.4.3-cp34-cp34m-win32.whl is not a supported wheel on this
platform."

The one time I tried "pip install recordclass", it found
0.4.3-cp34-cp34m-win32 alright. It got as far as copying, building and
testing it, but failed in test:

Installing collected packages: recordclass
Running setup.py install for recordclass
Complete output from command C:\Programs\Coding\Anaconda3\python.exe
-c "imp
ort setuptools,
tokenize;__file__='E:\\temp\\pip-build-zhximvpd\\recordclass\\se
tup.py';exec(compile(getattr(tokenize, 'open',
open)(__file__).read().replace('\ r\n', '\n'), __file__, 'exec'))"
install --record E:\temp\pip-r7ky4fv1-record\in stall-record.txt
--single-version-externally-managed --compile:
running install
running build
running build_py
creating build
creating build\lib.win32-3.4
creating build\lib.win32-3.4\recordclass
copying lib\recordclass\record.py -> build\lib.win32-3.4\recordclass
copying lib\recordclass\__init__.py ->
build\lib.win32-3.4\recordclass
creating build\lib.win32-3.4\recordclass\test
copying lib\recordclass\test\test_memoryslots.py ->
build\lib.win32-3.4\reco rdclass\test
copying lib\recordclass\test\test_record.py ->
build\lib.win32-3.4\recordcla ss\test

copying lib\recordclass\test\__init__.py ->
build\lib.win32-3.4\recordclass\ test

running build_ext
building 'recordclass.memoryslots' extension
error: Microsoft Visual C++ 10.0 is required (Unable to find
vcvarsall.bat).

So it choked when it needed Visual C++ to test the installation, at
least that's what I'm assuming it was testing.

It continues to amaze me that Anaconda and Python.org, probably the two
biggest distributors of official Python builds, are now relying on
Visual C++. Why can't Python developers write the entire setup and
installation code in Python? Surely Python has the required
functionality, and it seems more than a little demeaning for Python to
be using Visual C++ as a crutch.

Relatedly, why is Python lacking in the basic build tools for users to
easily build and install packages and full releases from source?
Something for Python (loosely) akin to the earliest versions of Visual
Studio would do the job, and it wouldn't have to be real tricked out.
I've seen this lack show up in several threads on this list, including
my original question in this thread. I was fishing for hints about the
build tools in Python as a way to build recordclass from source, but
apparently there aren't any. Not that advanced users know about anyway.

Is the Python dev team so lacking in resources that they can't cover
this fundamental base? Microsoft had build teams in dev at least as
early as the first release of Windows NT, and probably much earlier. I'd
always assumed that major language developers would establish build and
installation teams very early on. Alhough, Microsoft's setup teams still
really sucked bananas as late as when the the first XP was released in
2001 (don't know anything about them since), but the build teams were
rock stars that everyone looked up to from the get go. Apparently Python
hasn't mirrored that sequence of development steps.

> > Maybe I can find an earlier 3 build that won't demand Visual C++.
>
> The wheel doesn't need a compiler. It has an ABI tag because
> it already includes the compiled extension module.

Ah, and you've got me there. I have no clue what an ABI tag is or why
it's significant, and I don't know a fraction enough about wheels to
know whether they'd use a compiler or not. Grist for the mill, so
thanks.

Deborah

[Chris Angelico]

On Tue, May 16, 2017 at 4:12 AM, Deborah Swanson
<pyt...@deborahswanson.net> wrote:
> It continues to amaze me that Anaconda and Python.org, probably the two
> biggest distributors of official Python builds, are now relying on
> Visual C++. Why can't Python developers write the entire setup and
> installation code in Python? Surely Python has the required
> functionality, and it seems more than a little demeaning for Python to
> be using Visual C++ as a crutch.

Are you suggesting that a C compiler should be written in Python?
Because you're dealing with an extension library. It's not written in
pure Python. That's why it needs a C compiler. While it is certainly
possible to write a C compiler in Python, it is unlikely to outperform
the existing popular compilers (gcc, clang, msvc, etc), which have had
many years of expertise poured into them.

None of this applies if you are actually depending on a pure Python
module, which *can* be installed without a C compiler. Visual C++ is
not a crutch but a critical part of the build process.

ChrisA

[MRAB]

On 2017-05-15 13:52, eryk sun wrote:

> On Mon, May 15, 2017 at 6:37 AM, Deborah Swanson
> <pyt...@deborahswanson.net> wrote:
>>
>> Where did you find recordclass-0.4.3-cp34-cp34m-win32.whl? There
>> weren't any win32 builds on https://pypi.python.org/pypi/recordclass.
>
> It's in the middle of the file list:
>
> recordclass-0.4.3-cp34-cp34m-win32.whl (md5) Python Wheel cp34
> 2017-04-17 17KB
>

> Maybe you're looking at a cached page in your browser? But that
> doesn't explain why pip doesn't see it.
>

>> Maybe I can find an earlier 3 build that won't demand Visual C++.
>
> The wheel doesn't need a compiler. It has an ABI tag because it
> already includes the compiled extension module.
>

I used pip to install into Python 3.4 (32-bit) from PyPI. It fetched
"recordclass-0.4.3.tar.gz" and compiled it instead of fetching
"recordclass-0.4.3-cp34-cp34m-win32.whl".

Why?

Come to think of it, what's that "cp34m" in the name? I don't have that
in the regex module's wheels, I have "none" instead.

Rename "recordclass-0.4.3-cp34-cp34m-win32.whl" to
"recordclass-0.4.3-cp34-name-win32.whl".

With the new name, it installs and seems to work!

[eryk sun]

On Mon, May 15, 2017 at 7:43 PM, MRAB <pyt...@mrabarnett.plus.com> wrote:
> On 2017-05-15 13:52, eryk sun wrote:
>>
>> The wheel doesn't need a compiler. It has an ABI tag because it
>> already includes the compiled extension module.
>>
> I used pip to install into Python 3.4 (32-bit) from PyPI. It fetched
> "recordclass-0.4.3.tar.gz" and compiled it instead of fetching
> "recordclass-0.4.3-cp34-cp34m-win32.whl".

I suspect you're using pip version 7.1. Update to version 9:

python -m pip install --upgrade pip

> Come to think of it, what's that "cp34m" in the name? I don't have that in
> the regex module's wheels, I have "none" instead.

"recordclass-0.4.3-cp34-cp34m-win32" is broken down as follows [1]:

Distribution: recordclass
Version: 0.4.3
Python: cp34 [CPython 3.4]
ABI: cp34m [CPython 3.4 w/ PyMalloc]
Platform: win32 [32-bit Windows]

Extension modules have to be built for a particular implementation of
Python, ABI, and platform.

[1]: https://www.python.org/dev/peps/pep-0491/#file-format

[Deborah Swanson]

Chris Angelico wrote, on Monday, May 15, 2017 11:22 AM

>
> On Tue, May 16, 2017 at 4:12 AM, Deborah Swanson
> <pyt...@deborahswanson.net> wrote:
> > It continues to amaze me that Anaconda and Python.org, probably the
> > two biggest distributors of official Python builds, are now relying
on
> > Visual C++. Why can't Python developers write the entire setup and
> > installation code in Python? Surely Python has the required
> > functionality, and it seems more than a little demeaning for Python
to
> > be using Visual C++ as a crutch.
>
> Are you suggesting that a C compiler should be written in
> Python? Because you're dealing with an extension library.
> It's not written in pure Python. That's why it needs a C
> compiler. While it is certainly possible to write a C
> compiler in Python, it is unlikely to outperform the existing
> popular compilers (gcc, clang, msvc, etc), which have had
> many years of expertise poured into them.

I don't know much about extensions yet, but I think I can see what
you're saying. It probably is a shortcut for volunteer developers to use
existing C/C++ code where they can, rather than writing all of it from
scratch in Python.

But I'm a little more mystified that official Python builds are leaning
on Visual C++ (and that's what the crutch comment was primarily aimed
at). Perhaps they're just as resource needy as the extension developers
are and new requirements have come up as Python 3 progresses that they
don't already have Python build and installation tools written and
tested for. I'd be willing to step into that void when my Python skills
are equal to the task (and my health permitting, of course). But that
would be years from now, and maybe someone else will step up before
then. I'm really thinking that build has languished far behind the rest
of the language's development for way too long.

> None of this applies if you are actually depending on a pure
> Python module, which *can* be installed without a C compiler.
> Visual C++ is not a crutch but a critical part of the build process.
>
> ChrisA

I'll still think of it as a crutch for official Python builds, but I can
see your point for extension developers.

Maybe I have more pride in the Python language than is right and proper,
but I do think it would be worth the relatively small investment to make
and start growing Python's own build and installation tools asap. Python
will be the reigning Queen of coding languages someday. Certainly seems
like she should have all her own working parts and be beholden to no
one. ;)

Microsoft saw the need for build and installation teams very early on,
and now everyone depends on what they made.

And yes, Windows and all Microsoft software from thence forward was
primarily made with C/C++, because MSDOS was appallingly incapable of
meeting the challenge and that's all they had before Windows.

Python has a lot more than MSDOS ever dreamed of, and that's what seems
demeaning to me that Python is piggybacking on a lesser language,
because their need to do so is so infinitesimally less than Microsoft's
need was to move on from MSDOS. And I think the day is coming that
Microsoft will be too much of a dinosaur (like IBM has become) to
continue meeting the needs of evolving languages. Better to get going on
our own tools, the sooner the better, in my opinion.

Deborah

[Nathan Ernst]

Deborah,

I get the feeling you don't understand the architecture/implementation of
Python. The (C)Python interpreter is written in C. A number of the built-in
modules are least partially written in C. As such, C is a natural
integration point for extensions. Many, many third-party extensions are
written completely or partially in C or C++. If you're going to install
Python or these modules from source, you need a C or C++ compiler. On
Windows, this tends to be MSVC. On Linux, GCC or clang. On OS X, clang.

You may consider C & C++ to be lesser languages, but Python's built upon
those "lesser" languages. Personally, I love both C++ & Python. You need
to know when to use the write tool for the job (hammer vs screwdriver).

Comparing Python to MSDOS is comparing apples to oranges. MSDOS is not a
language. It is an operating system (albeit a bare-bones, single-user one).

Regards,
Nathan

On Mon, May 15, 2017 at 4:01 PM, Deborah Swanson <pyt...@deborahswanson.net>
wrote:

> --
> https://mail.python.org/mailman/listinfo/python-list
>

[Deborah Swanson]

MRAB wrote, on Monday, May 15, 2017 12:44 PM

>
> On 2017-05-15 13:52, eryk sun wrote:

> > On Mon, May 15, 2017 at 6:37 AM, Deborah Swanson
> > <pyt...@deborahswanson.net> wrote:
> >>
> >> Where did you find recordclass-0.4.3-cp34-cp34m-win32.whl? There
> >> weren't any win32 builds on
> https://pypi.python.org/pypi/recordclass.
> >
> > It's in the
> middle of the file list:
> >
> > recordclass-0.4.3-cp34-cp34m-win32.whl (md5) Python Wheel cp34
> > 2017-04-17 17KB
> >
> > Maybe you're looking at a cached page in your browser? But that
> > doesn't explain why pip doesn't see it.
> >
> >> Maybe I can find an earlier 3 build that won't demand Visual C++.
> >

> > The wheel doesn't need a compiler. It has an ABI tag because it
> > already includes the compiled extension module.
> >
> I used pip to install into Python 3.4 (32-bit) from PyPI. It fetched
> "recordclass-0.4.3.tar.gz" and compiled it instead of fetching
> "recordclass-0.4.3-cp34-cp34m-win32.whl".
>

> Why?

Excellent question, though I have no clue what the answer is.

> Come to think of it, what's that "cp34m" in the name? I don't have
that
> in the regex module's wheels, I have "none" instead.

Another excellent question. "cp" in "cp34m" might stand for C++, but if
so, what's the rest of it mean? Or maybe "CPython 3.4" plus "m"
(whatever "m" stands for)?

What's "the regex module's wheels"? I certainly don't know, though I
know vague outines of what "regex" and "wheel" are.

> Rename "recordclass-0.4.3-cp34-cp34m-win32.whl" to
> "recordclass-0.4.3-cp34-name-win32.whl".
>
> With the new name, it installs and seems to work!

Most excellent and I will try it.

As a mutable alternative to namedtuples, recordclass certainly seems
worth the detour to look at and try, if I can get it to install! All of
the hairy points I've run into with namedtuples revolve around their
immutability. Getattr works and solves some of them, but it's helpless
and breaks with the "object has no attribute xxx" in others. So a
mutable version of namedtuple functionality would be extremely useful
indeed.

I'll also take a look at the contents of "recordclass-0.4.3.tar.gz" to
see what I can see. Maybe not much though. I'm real ignorant of how
extensions are made, but I might see something worth seeing.

Deborah

[Chris Angelico]

On Tue, May 16, 2017 at 7:01 AM, Deborah Swanson
<pyt...@deborahswanson.net> wrote:
> But I'm a little more mystified that official Python builds are leaning
> on Visual C++ (and that's what the crutch comment was primarily aimed
> at). Perhaps they're just as resource needy as the extension developers
> are and new requirements have come up as Python 3 progresses that they
> don't already have Python build and installation tools written and
> tested for. I'd be willing to step into that void when my Python skills
> are equal to the task (and my health permitting, of course). But that
> would be years from now, and maybe someone else will step up before
> then. I'm really thinking that build has languished far behind the rest
> of the language's development for way too long.
>
>

> Maybe I have more pride in the Python language than is right and proper,
> but I do think it would be worth the relatively small investment to make
> and start growing Python's own build and installation tools asap. Python
> will be the reigning Queen of coding languages someday. Certainly seems
> like she should have all her own working parts and be beholden to no
> one. ;)

You seem to be of the opinion that some day, binary executables will
be compiled using pure Python code. Maybe that's true; maybe it's not.
The point is, though, that high performance binaries can't be created
without a LOT of expertise relating to the OS, the standard library,
the CPU architecture, etc, etc, etc. That expertise has been coalesced
into existing C compilers (I mentioned gcc, clang, and msvc earlier;
others also exist but those are the most popular), and throwing that
out means having a subpar Python.

Perhaps a future version of gcc will be implemented in Python. Would
you then say that Python is beholden to no one? You would still need
to have a C compiler installed in order to compile extension modules.
It's too big to include with every single Python installation. So
you'd have the same situation even then.

ChrisA

[Pavol Lisy]

On 5/15/17, Chris Angelico <ros...@gmail.com> wrote:

> Perhaps a future version of gcc will be implemented in Python. Would
> you then say that Python is beholden to no one? You would still need
> to have a C compiler installed in order to compile extension modules.
> It's too big to include with every single Python installation. So
> you'd have the same situation even then.

Don't forget that 640 kB was too big too! ;)

(and compiler as service is also possibility)

PL.

[Chris Angelico]

Compiler As A Service is actually a very real possibility, and one
that's been explored at times. But it's more of a packaging question
than anything else; it's a means of simplifying the work of producing
those wheel files. Once the wheels are built, anyone can already
install them without a compiler (of course, you have to trust the
wheel builder, since you're now installing a binary blob on your
computer). There's not a lot of point having end users use CaaS; if
you're making custom software or an unusual system, it's easier to get
your own compiler than to match a host somewhere, and if you're using
something completely standard, there's no need to reinvent - or
rebuild - the wheel.

ChrisA

[MRAB]

On 2017-05-15 21:40, eryk sun wrote:
> On Mon, May 15, 2017 at 7:43 PM, MRAB <pyt...@mrabarnett.plus.com> wrote:

> > On 2017-05-15 13:52, eryk sun wrote:
> >>
> >> The wheel doesn't need a compiler. It has an ABI tag because it
> >> already includes the compiled extension module.
> >>
> > I used pip to install into Python 3.4 (32-bit) from PyPI. It fetched
> > "recordclass-0.4.3.tar.gz" and compiled it instead of fetching
> > "recordclass-0.4.3-cp34-cp34m-win32.whl".
>

> I suspect you're using pip version 7.1. Update to version 9:
>
> python -m pip install --upgrade pip
>

> > Come to think of it, what's that "cp34m" in the name? I don't have that in
> > the regex module's wheels, I have "none" instead.
>

> "recordclass-0.4.3-cp34-cp34m-win32" is broken down as follows [1]:
>
> Distribution: recordclass
> Version: 0.4.3
> Python: cp34 [CPython 3.4]
> ABI: cp34m [CPython 3.4 w/ PyMalloc]
> Platform: win32 [32-bit Windows]
>
> Extension modules have to be built for a particular implementation of
> Python, ABI, and platform.
>
> [1]: https://www.python.org/dev/peps/pep-0491/#file-format
>

I hadn't updated the pip for CPython 3.4 from the version it was
originally shipped with. Once upgraded, the wheel installed from PyPI
without a problem.

[Gregory Ewing]

Deborah Swanson wrote:
> It continues to amaze me that Anaconda and Python.org, probably the two
> biggest distributors of official Python builds, are now relying on
> Visual C++. Why can't Python developers write the entire setup and
> installation code in Python?

The setup and installation code *is* written in Python. It's
just that some of the source of the particular package you're
trying to install is written in C, so installing it *from
source* necessarily requires a C compiler.

Most people's Windows machines aren't set up for compiling C
code, and for that reason, prebuilt versions of such packages
are usually supplied for Windows.

So you have two reasonable options:

1) Find a prebuilt wheel that's compatible with your system.

2) Find another package written in pure Python that does what
you want.

It's also possible that you don't need the parts of recordclass
that are written in C. If that's the case you might be able to
install the pure Python parts manually. That's usually just a
matter of dropping the .py files in appropriate places.

> Relatedly, why is Python lacking in the basic build tools for users to
> easily build and install packages and full releases from source?
> Something for Python (loosely) akin to the earliest versions of Visual
> Studio would do the job,

Having the Python ecosystem stuck on some ancient version of
Visual Studio would be a bad situation for many reasons. One
of the main uses of C extensions is to interface with external
libraries, which will most likely have been compiled with a
recent toolset and rely on recent versions of Microsoft's
runtime libraries.

There's also the fact that Microsoft doesn't make the free
versions of Visual Studio available forever and forbids anyone
else distributing them, so sufficiently old versions are
simply not available to most people.

The alternative would be to use some other free compiler such
as MinGW as the standard compiler for Python and its extensions.
But this has been decided against on the grounds that it is
better to use the standard Windows tools to compile code for
Windows, for the same compatibility reasons as above, and that
means Visual Studio.

> Ah, and you've got me there. I have no clue what an ABI tag is or why
> it's significant,

ABI = Application Binary Interface. Essentially it's the
interface between the Python interpreter and code that's
been compiled from C, and it can change from one version
of Python to another.

The error message from pip could be more useful -- it would be
nice to know exactly *what* it is about the wheel that doesn't
match your system!

Some things it could be:

* Wrong version of Python

* Wrong CPU architecture (32 vs 64 bit)

--
Greg

[Deborah Swanson]

Nathan Ernst wrote, on Monday, May 15, 2017 2:14 PM

Deborah,


I get the feeling you don't understand the architecture/implementation
of Python. The (C)Python interpreter is written in C. A number of the
built-in modules are least partially written in C. As such, C is a
natural integration point for extensions. Many, many third-party
extensions are written completely or partially in C or C++. If you're
going to install Python or these modules from source, you need a C or
C++ compiler. On Windows, this tends to be MSVC. On Linux, GCC or clang.
On OS X, clang.

> You would be right, it has never occurred to me that built-in modules
to the > language would be partially written in C. I'll plan to have a C
compiler
> asap, but it probably won't be until I'm up and running on Linux
again.

You may consider C & C++ to be lesser languages, but Python's built upon
those "lesser" languages. Personally, I love both C++ & Python. You
need to know when to use the write tool for the job (hammer vs
screwdriver).

> That may well be, and probably I should tone down my snark about
C/C++. This > is why it's good to talk to people about these thoughts
that might be
> running amok and undetected inside your own head. That happens all too

> easily when you spend years reading and studying alone, so thanks, and
I'll > be looking more closely at C/C++ and it's role today.

Comparing Python to MSDOS is comparing apples to oranges. MSDOS is not a
language. It is an operating system (albeit a bare-bones, single-user
one).

Regards,
Nathan


On Mon, May 15, 2017 at 4:01 PM, Deborah Swanson
<pyt...@deborahswanson.net> wrote:

Chris Angelico wrote, on Monday, May 15, 2017 11:22 AM
>
> On Tue, May 16, 2017 at 4:12 AM, Deborah Swanson

> <pyt...@deborahswanson.net> wrote:
> > It continues to amaze me that Anaconda and Python.org, probably the
> > two biggest distributors of official Python builds, are now relying
on
> > Visual C++. Why can't Python developers write the entire setup and

> > installation code in Python? Surely Python has the required
> > functionality, and it seems more than a little demeaning for Python
to
> > be using Visual C++ as a crutch.
>
> Are you suggesting that a C compiler should be written in
> Python? Because you're dealing with an extension library.
> It's not written in pure Python. That's why it needs a C
> compiler. While it is certainly possible to write a C
> compiler in Python, it is unlikely to outperform the existing
> popular compilers (gcc, clang, msvc, etc), which have had
> many years of expertise poured into them.

I don't know much about extensions yet, but I think I can see what
you're saying. It probably is a shortcut for volunteer developers to use
existing C/C++ code where they can, rather than writing all of it from
scratch in Python.

But I'm a little more mystified that official Python builds are leaning
on Visual C++ (and that's what the crutch comment was primarily aimed
at). Perhaps they're just as resource needy as the extension developers
are and new requirements have come up as Python 3 progresses that they
don't already have Python build and installation tools written and
tested for. I'd be willing to step into that void when my Python skills
are equal to the task (and my health permitting, of course). But that
would be years from now, and maybe someone else will step up before
then. I'm really thinking that build has languished far behind the rest
of the language's development for way too long.

> None of this applies if you are actually depending on a pure
> Python module, which *can* be installed without a C compiler.
> Visual C++ is not a crutch but a critical part of the build process.
>
> ChrisA

I'll still think of it as a crutch for official Python builds, but I can
see your point for extension developers.

Maybe I have more pride in the Python language than is right and proper,
but I do think it would be worth the relatively small investment to make
and start growing Python's own build and installation tools asap. Python
will be the reigning Queen of coding languages someday. Certainly seems
like she should have all her own working parts and be beholden to no
one. ;)

[Chris Angelico]

On Tue, May 16, 2017 at 9:54 AM, Gregory Ewing
<greg....@canterbury.ac.nz> wrote:
> There's also the fact that Microsoft doesn't make the free
> versions of Visual Studio available forever and forbids anyone
> else distributing them, so sufficiently old versions are
> simply not available to most people.

Yes and no. The Python.org support policy is that Python X.Y will be
supported on Windows Z if and only if Windows Z was supported by
Microsoft as of when Python X.Y.0 was released. This means that, for
instance, Python 3.3 supported Windows XP, but Python 3.7 won't. If
you need a Win2K-compatible compiler, you're using a too-old version
of Python, and nobody's going to support that.

There is, however, one exception. Python 2.7 supports Windows XP, and
a specific version of MSVC (I don't remember which off-hand) that is
now unsupported by Microsoft. However, *specifically for Python
extension developers*, Microsoft offers the zero-dollar version of
that compiler still, and will for a decent while (again, I don't
remember the specifics, but it's something comparable to Py2.7's own
upstream support).

So in general, "sufficiently old versions" should actually still be available.

ChrisA

[Deborah Swanson]

Chris Angelico wrote, on Monday, May 15, 2017 2:58 PM

>
> On Tue, May 16, 2017 at 7:01 AM, Deborah Swanson
> <pyt...@deborahswanson.net> wrote:
> > But I'm a little more mystified that official Python builds are
> > leaning on Visual C++ (and that's what the crutch comment was
> > primarily aimed at). Perhaps they're just as resource needy as the
> > extension developers are and new requirements have come up as Python
3
> > progresses that they don't already have Python build and
installation
> > tools written and tested for. I'd be willing to step into that void
> > when my Python skills are equal to the task (and my health
permitting,
> > of course). But that would be years from now, and maybe someone else

> > will step up before then. I'm really thinking that build has
> > languished far behind the rest of the language's development for way

> > too long.
> >
> >

> > Maybe I have more pride in the Python language than is right and
> > proper, but I do think it would be worth the relatively small
> > investment to make and start growing Python's own build and
> > installation tools asap. Python will be the reigning Queen of coding

> > languages someday. Certainly seems like she should have all her own
> > working parts and be beholden to no one. ;)
>

> You seem to be of the opinion that some day, binary
> executables will be compiled using pure Python code.

Well, my naive opinion is that they should be. But it was just one
message back that I learned some Python built-in modules are at least
partially written in C. I didn't know that, nor did I know or suspect
that installation files for official Python releases likely contained C
compilers. This raises a lot of questions for me, though it probably
would be better for me to read and think about them on my own before
coming back here with anything.

> Maybe
> that's true; maybe it's not. The point is, though, that high
> performance binaries can't be created without a LOT of
> expertise relating to the OS, the standard library, the CPU
> architecture, etc, etc, etc. That expertise has been
> coalesced into existing C compilers (I mentioned gcc, clang,
> and msvc earlier; others also exist but those are the most
> popular), and throwing that out means having a subpar Python.

I'll accept that, though I still don't quite understand the recent
dependence on Visual C++, after Python has been building its releases
with the tools you and others mention from the beginning until shortly
after the first 3 build was released, and the requirement has been so
much deeper since 3.4.3, that Python (apparently) can no longer use the
Visual C++ compiler built into XP. These are just things I wonder about
since these new requirements put a real crimp in what I'm currently able
to do, and I wonder how they will change Linux too. Again, lots of stuff
for me to pursue on my own here.

> Perhaps a future version of gcc will be implemented in
> Python. Would you then say that Python is beholden to no one?
> You would still need to have a C compiler installed in order
> to compile extension modules. It's too big to include with
> every single Python installation. So you'd have the same
> situation even then.
>

> ChrisA

No, with what I've learned today, I certainly would agree that Python
has its beholdens, and likely always will. The bright side is that
Python at least doesn't appear to be unduly constrained in how it uses
the C/C++ code it does. I just hope that it continues to be kept to a
minimum, but I guess I'm just a stinking purist about some things.

Just another pretty fantasy smashed. So it goes. It's definitely better
to have broken fantasies to feel bad about having, and to know the
truth.

Deborah

[Chris Angelico]

On Tue, May 16, 2017 at 11:19 AM, Deborah Swanson
<pyt...@deborahswanson.net> wrote:
> Chris Angelico wrote, on Monday, May 15, 2017 2:58 PM

>> You seem to be of the opinion that some day, binary
>> executables will be compiled using pure Python code.
>
> Well, my naive opinion is that they should be. But it was just one
> message back that I learned some Python built-in modules are at least
> partially written in C. I didn't know that, nor did I know or suspect
> that installation files for official Python releases likely contained C
> compilers. This raises a lot of questions for me, though it probably
> would be better for me to read and think about them on my own before
> coming back here with anything.

It is possible to build a compiler for Python in Python. (See: PyPy.)
The most normal way to build extension libraries is with C, though,
because that's what most lower-level libraries depend on. For example,
suppose you want to build a GUI toolkit for Python, using the GTK
library. You can't easily do that in Python (technically you *can*,
using ctypes, but ctypes is itself implemented in C), so you'll end up
writing a C extension library.

> I'll accept that, though I still don't quite understand the recent
> dependence on Visual C++, after Python has been building its releases
> with the tools you and others mention from the beginning until shortly
> after the first 3 build was released, and the requirement has been so
> much deeper since 3.4.3, that Python (apparently) can no longer use the
> Visual C++ compiler built into XP. These are just things I wonder about
> since these new requirements put a real crimp in what I'm currently able
> to do, and I wonder how they will change Linux too. Again, lots of stuff
> for me to pursue on my own here.

An installed Python does not itself depend on MSVC. It's only when you
need to compile additional C code - another Python module that is
implemented in C - and you can't pick up someone else's precompiled
version. It's not a new requirement; although you could perhaps argue
that pip's presence in the standard library has made this a lot more
likely to come up, as previously, you wouldn't have been able to
install this module *at all* without fiddling around manually.

> No, with what I've learned today, I certainly would agree that Python
> has its beholdens, and likely always will. The bright side is that
> Python at least doesn't appear to be unduly constrained in how it uses
> the C/C++ code it does. I just hope that it continues to be kept to a
> minimum, but I guess I'm just a stinking purist about some things.

Actually, I'm glad that most of Python's core is implemented in C,
because it means my code runs faster :) It's all thoroughly
unit-tested, and I'm happy with it.

ChrisA

[Deborah Swanson]

Gregory Ewing wrote, on Monday, May 15, 2017 4:55 PM

>
> Deborah Swanson wrote:
> > It continues to amaze me that Anaconda and Python.org, probably the
> > two biggest distributors of official Python builds, are now relying
on
> > Visual C++. Why can't Python developers write the entire setup and
> > installation code in Python?
>
> The setup and installation code *is* written in Python. It's
> just that some of the source of the particular package you're
> trying to install is written in C, so installing it *from
> source* necessarily requires a C compiler.
>
> Most people's Windows machines aren't set up for compiling C
> code, and for that reason, prebuilt versions of such packages
> are usually supplied for Windows.
>
> So you have two reasonable options:
>
> 1) Find a prebuilt wheel that's compatible with your system.
>
> 2) Find another package written in pure Python that does what
> you want.
>
> It's also possible that you don't need the parts of
> recordclass that are written in C. If that's the case you
> might be able to install the pure Python parts manually.
> That's usually just a matter of dropping the .py files in
> appropriate places.

That might end up being the best choice for me, although I'm not
entirely sure what the appropriate places to drop the .py files are.
Probably the Scripts directory is one of them, but I don't know which
files go there and which files go other places, and I definitely don't
know what those other places are.

However, I think MRAB's success in pip installing recordclass after
upgrading pip might make workarounds unnecessary for me. I also haven't
upgraded pip since I last installed Python, and I did see a nag notice
that I should upgrade it. So that seems like the next step.

> > Relatedly, why is Python lacking in the basic build tools for users
to
> > easily build and install packages and full releases from source?
> > Something for Python (loosely) akin to the earliest versions of
Visual
> > Studio would do the job,
>
> Having the Python ecosystem stuck on some ancient version of
> Visual Studio would be a bad situation for many reasons.

Oh, and I was never thinking that should be done, that's why I said
"loosely akin to". Just something along those lines, updated for Python
and current realities, not the genuine original in the flesh!

My point then was to develop purely Python tools to do the job we're
leaning on Visual Studio to do, but I've learned in the last few
messages why that might be more complicated than I originally thought it
would be.

> One
> of the main uses of C extensions is to interface with
> external libraries, which will most likely have been compiled
> with a recent toolset and rely on recent versions of
> Microsoft's runtime libraries.

Another point I was unaware of, so thanks.

> There's also the fact that Microsoft doesn't make the free
> versions of Visual Studio available forever and forbids
> anyone else distributing them, so sufficiently old versions
> are simply not available to most people.

Another reason why it's sad that Python hasn't been making its own build
tools, or incorporating and developing open software tools all along.
This is a tail that could easily end up wagging the dog, and soon. But
likely there are stll many things I don't know or haven't fully
understood from what I've learned today.

> The alternative would be to use some other free compiler such
> as MinGW as the standard compiler for Python and its
> extensions. But this has been decided against on the grounds
> that it is better to use the standard Windows tools to
> compile code for Windows, for the same compatibility reasons
> as above, and that means Visual Studio.

Well, apparently that was the choice. I for one think it's sad, and that
it will lead to trouble for Python somewhere down the road. Probably the
trouble will be in its capacity to be available on all platforms, and I
think it's sad to have such binding ties with newer Windows. Time will
tell, but I think Windows is a slowly sinking ship, not unlike the
eventual fate of the behemoth IBM. It's the way of all things, and
Microsoft is no more immune to natural processes than anything else is.

> > Ah, and you've got me there. I have no clue what an ABI tag is or
why
> > it's significant,
>
> ABI = Application Binary Interface. Essentially it's the
> interface between the Python interpreter and code that's been
> compiled from C, and it can change from one version of Python
> to another.

Thanks! Another mysterious acronym demystified!

> The error message from pip could be more useful -- it would
> be nice to know exactly *what* it is about the wheel that
> doesn't match your system!

Ahem, yes that would be most useful.

> Some things it could be:
>
> * Wrong version of Python
>
> * Wrong CPU architecture (32 vs 64 bit)
>
> --
> Greg

Or it could be any number of other things, since neither of those two
are obviously the problem here.

I do know, from trying to get Microsoft setup devs to make their error
messages more helpful, that this problem too is more complicated than it
might appear at first glance. There are often a lot of things that can
cause a particular error to pop in a particular situation, and it's hard
for devs to write enough code to cover all the bases and give the end
user an appropriate response text. Most of the whacko error messages you
see are some developer's attempt to do something better, but they didn't
know all the trigger combinations for that particular message text.

Maybe some brilliant computer scientist will solve the error message
quagmire someday. Not holding my breath.

Deborah

[eryk sun]

On Tue, May 16, 2017 at 12:51 AM, Chris Angelico <ros...@gmail.com> wrote:
>
> The Python.org support policy is that Python X.Y will be
> supported on Windows Z if and only if Windows Z was supported by
> Microsoft as of when Python X.Y.0 was released.

Python 3.4 is the last to support XP (2014-04-08 EOL).
Python 3.6 is the last to support Vista (2017-04-11 EOL).

Estimates:
Windows 7, 2020-01-14 EOL: Python 3.8
Windows 8, 2023-01-20 EOL: Python 3.10
Windows 10, 2025-10-14 EOL: Python 3.11

> There is, however, one exception. Python 2.7 supports Windows XP,

The exception is 2.7's extended 10-year lifecycle (2010-2020). Given
the ABI compatibility requirement to use an old, unsupported compiler,
Python 2.7 is all but dead on Windows -- at least it should be.

> a specific version of MSVC (I don't remember which off-hand) that is
> now unsupported by Microsoft.

Python 2.7 is built using the VS 2008 (VC++ 9.0) compiler toolset.
This old compiler is required for compatibility with existing
extension modules. Note that the main project files in PCbuild/ have
been updated to VS 2010 format and require MSBuild from VS 2010 or
newer (plus of course the VS 2008 compiler). However, the PC/VS9.0
project files are still supported if you only have VS 2008 installed.

Getting VS 2008 requires an MSDN subscription. There is no free
version available.

> However, *specifically for Python extension developers*, Microsoft offers the
> zero-dollar version of that compiler still, and will for a decent while (again, I don't
> remember the specifics, but it's something comparable to Py2.7's own
> upstream support).

That's VC++ Compiler for Python 2.7:
https://www.microsoft.com/en-us/download/details.aspx?id=44266

Currently it can't build Python itself, but according to Steve Dower
it's possible to configure it to work.

[MRAB]

On 2017-05-16 02:19, Deborah Swanson wrote:
[snip]

> I'll accept that, though I still don't quite understand the recent
> dependence on Visual C++, after Python has been building its releases
> with the tools you and others mention from the beginning until shortly
> after the first 3 build was released, and the requirement has been so
> much deeper since 3.4.3, that Python (apparently) can no longer use the
> Visual C++ compiler built into XP. These are just things I wonder about
> since these new requirements put a real crimp in what I'm currently able
> to do, and I wonder how they will change Linux too. Again, lots of stuff
> for me to pursue on my own here.
>

CPython is being developed primarily on Linux. Linux and other Unix-like
operating systems come with a C compiler as standard (the histories of
Unix and C are closely linked).

Windows, on the other hand, doesn't come with a C compiler. If you've
ever wanted to write something in C on Windows, you've had to install a
C compiler yourself.

[Christian Gollwitzer]

Am 15.05.17 um 23:58 schrieb Chris Angelico:

> On Tue, May 16, 2017 at 7:01 AM, Deborah Swanson
> <pyt...@deborahswanson.net> wrote:
>> But I'm a little more mystified that official Python builds are leaning
>> on Visual C++ (and that's what the crutch comment was primarily aimed
>> at).

> You seem to be of the opinion that some day, binary executables will
> be compiled using pure Python code. Maybe that's true; maybe it's not.

More likely would be the option to ship a C compiler with Python written
in C. For C++ this is way too big, but a pure C compiler can be as small
as 1MB. tcc has a liberal license, supports many platforms and gives
reasonable (unoptimized) code. AFAIK Mathworks does that, they ship tcc
on Windows so that you can build .mex files without installing
additional software, though they recommend to get a decent compiler for
performance reasons

Christian

[Chris Angelico]

To do that, Python would itself have to be compiled with tcc, or else
all memory de/allocation would have to be funneled through a
Python-provided API. And that's going to kill performance, I suspect.

ChrisA

[bream...@gmail.com]

On Sunday, May 14, 2017 at 10:36:37 PM UTC+1, Deborah Swanson wrote:
> I want to install the recordclass package:
> https://pypi.python.org/pypi/recordclass
>
> But they've only released wheel files for two platforms, macosx and
> win_amd64, neither of which will install on my system. I need win_x86 or
> intel_x86, which they don't provide.
>
> The only alternative to pip install that I know of is to install from
> source, and the source I need is at:
> https://bitbucket.org/intellimath/recordclass/src
>
> I've never done this, so any help would be appreciated.
> Deborah

Not that it helps in this case but my first port of call if I can't install using pip is http://www.lfd.uci.edu/~gohlke/pythonlibs/

Kindest regards.

Mark Lawrence.

[Gregory Ewing]

MRAB wrote:
> CPython is being developed primarily on Linux. Linux and other Unix-like
> operating systems come with a C compiler as standard (the histories of
> Unix and C are closely linked).

Also, Linux (and other unix-like systems) don't tend to change
the compiler toolsets and libraries in incompatible ways with
every release like Microsoft is in the habit of doing, so it's
far less critical to have exactly the right version of the
compiler at hand when compiling Python modules.

--
Greg

[Mark Summerfield]

I think the problem that Deborah has encountered is a more general one on
Windows: many pip-installable packages assume that a C compiler is
available.

Now an "obvious" solution is for pip to recognise that a C compiler is
needed and give an appropriate error message. But while that may reduce
confusion, it won't actually help someone who wants to install a Python
package that needs a C compiler.

Of course, the error message could give a link to the appropriate
compiler. And then the user (who presumably isn't a Windows compiler
expert) will if they're really lucky get to download and install an
enormous Visual Studio compiler. Of course they may not be so lucky and
may discover that the installer tells them that they need some component
(and may or may not give the link to get it). And then they get that
component and, you guessed it, that component says it needs another
component, ... and eventually you end up with all the pieces. And even
then it doesn't necessarily work. (This has happened to me more than
once.)

Here's a fantasy:

C:\> pip install pkg_needs_c
Error: the pkg_needs_c package needs development tools which haven't been
found on this computer.
Run pip install --listtools pkg_needs_c for information on the required
tools or run pip install --useprebuilt pkg_needs_c to download prebuilt
components so no additional development tools are needed.

C:\> pip install --listtools pkg_needs_c
The pkg_needs_c needs Visual Studio 2015 (which itself may have additional
dependencies). This can be downloaded from http://www....

C:\> pip install --useprebuilt pkg_needs_c
...

This will send the minimum necessary details of the machine back to PyPI
which will then do a request on a Windows server farm which will send back
a matching pre-built package (building the package the first time it is
needed).

I have no idea if this is possible/practical!

But I do think that installing dev tools on Windows can be really difficult (esp. if you're on older versions of Windows, e.g., XP, Vista, or 7) and that this can make things hard for people coming to Python on Windows. I guess the best that can be done for now is to recommend Annaconda or a similar distro with lots of prebuilt stuff.

[bartc]

On 15/05/2017 19:21, Chris Angelico wrote:
> On Tue, May 16, 2017 at 4:12 AM, Deborah Swanson
> <pyt...@deborahswanson.net> wrote:
>> It continues to amaze me that Anaconda and Python.org, probably the two
>> biggest distributors of official Python builds, are now relying on
>> Visual C++. Why can't Python developers write the entire setup and
>> installation code in Python? Surely Python has the required
>> functionality, and it seems more than a little demeaning for Python to
>> be using Visual C++ as a crutch.
>
> Are you suggesting that a C compiler should be written in Python?
> Because you're dealing with an extension library. It's not written in
> pure Python. That's why it needs a C compiler. While it is certainly
> possible to write a C compiler in Python, it is unlikely to outperform
> the existing popular compilers (gcc, clang, msvc, etc), which have had
> many years of expertise poured into them.

In what way, compilation speed, or speed of the generated code?

In terms of compilation speed, efficient algorithms (and overall size)
are more important than implementation language, and gcc especially is
not hard to beat, even with an interpreted compiler (I've done that),
but anyway it probably wouldn't be too far behind.

And for a one-off task such as compiling an existing, working package,
build speed doesn't matter so much.

With output code, for running a substantial application, all these
compilation optimisations don't make as much difference as you might
think (you can get within a factor of 2x easily even with poor code).

But again, in order to get something working, when the alternative is
having nothing working, then that can be acceptable as a temporary
workaround.

(Note that I am talking about a compiler written in /any/ language, but
that would still generate native code.)

--
bartc

[Grant Edwards]

On 2017-05-16, Deborah Swanson <pyt...@deborahswanson.net> wrote:

> Well, my naive opinion is that they should be. But it was just one
> message back that I learned some Python built-in modules are at least
> partially written in C.

Many of the are, because there aren't any platforms that provide a
native Python API for doing things like network and file I/O,
interfacing with GUI layers, etc. And Python itself (the most widely
used implementation) is also written in C.

> I didn't know that, nor did I know or suspect that installation
> files for official Python releases likely contained C compilers.

What? That's news to me!

What "official python release" contains a C compiler?

--
Grant Edwards grant.b.edwards Yow! Is something VIOLENT
at going to happen to a
gmail.com GARBAGE CAN?

[Grant Edwards]

On 2017-05-16, Deborah Swanson <pyt...@deborahswanson.net> wrote:

> Another reason why it's sad that Python hasn't been making its own build
> tools, or incorporating and developing open software tools all along.

Python uses almost exclusively open-source tools on Linux and whatever
MacOS is called this year.

Feel free to build and distribute a python distro for Windows that's
based on a free C compiler (MinGW is probably your best option).

> This is a tail that could easily end up wagging the dog, and soon. But
> likely there are stll many things I don't know or haven't fully
> understood from what I've learned today.

Windows users have nobody but theirselves to blame for their
dependence on Microsoft. Whining about it to Linux users isn't going
to help much.

>> The alternative would be to use some other free compiler such as
>> MinGW as the standard compiler for Python and its extensions. But
>> this has been decided against on the grounds that it is better to
>> use the standard Windows tools to compile code for Windows, for the
>> same compatibility reasons as above, and that means Visual Studio.
>

> Well, apparently that was the choice. I for one think it's sad, and
> that it will lead to trouble for Python somewhere down the
> road. Probably the trouble will be in its capacity to be available
> on all platforms, and I think it's sad to have such binding ties
> with newer Windows. Time will tell, but I think Windows is a slowly
> sinking ship, not unlike the eventual fate of the behemoth IBM. It's
> the way of all things, and Microsoft is no more immune to natural
> processes than anything else is.

So Windows sinks and dies. Hooray!

Python will be fine.

--
Grant Edwards grant.b.edwards Yow! Why is everything made
at of Lycra Spandex?
gmail.com

[Chris Angelico]

On Tue, May 16, 2017 at 11:41 PM, bartc <b...@freeuk.com> wrote:
>> Are you suggesting that a C compiler should be written in Python?
>> Because you're dealing with an extension library. It's not written in
>> pure Python. That's why it needs a C compiler. While it is certainly
>> possible to write a C compiler in Python, it is unlikely to outperform
>> the existing popular compilers (gcc, clang, msvc, etc), which have had
>> many years of expertise poured into them.
>
>
> In what way, compilation speed, or speed of the generated code?

I'm talking mainly about the speed of the generated code, but the two
are somewhat linked. A slow compiler that produces amazing code will
take three parts of forever to complete, so if your code is slow,
you'll turn off some of the optimizations so it finishes sooner, which
means you get worse output.

It's possible that you could write a C compiler in Python that emits
something that back-ends into a machine code generator, but that just
moves the problem around ("why do we need this codegen?"). To build a
high quality C compiler *purely* in Python means knowing about every
CPU architecture that you can target. That's a daunting job if you
aren't going to piggy-back on something that already exists.

ChrisA

[bartc]

I can't test with Python because it's too complicated to compile,
especially on Windows.

But if I use tcc to compile /my/ byte-code interpreter, and then run the
interpreter on a computationally intensive benchmark (jpeg decoding), I
get these results for a given input file:

Tiny C 6.7 seconds
gcc -O3 3.6 seconds

So it is not that terrible. And there are also other, smaller compilers
which will generate better code than Tiny C. (I don't know why so many
bundle gcc with their language apps because is not exactly compact.)

(My own [not C] compiler, which has a rubbish code-generator, manages
4.9 seconds on this test. My own C compiler, with even worse code at the
minute, still does 5.6 seconds, but is not advanced enough to compile
CPython.)

Note that Tiny C is poor at code for switch-statements, which are used
quite a bit in this application, and doesn't handle label pointers. I
believe that CPython makes use of label pointers with gcc to get an
extra boost.

--
bartc

[bream...@gmail.com]

On Tuesday, May 16, 2017 at 5:09:34 PM UTC+1, bartc wrote:
> On 16/05/2017 08:53, Chris Angelico wrote:

> > On Tue, May 16, 2017 at 5:14 PM, Christian Gollwitzer wrote:
> >> Am 15.05.17 um 23:58 schrieb Chris Angelico:
> >>>
> >>> On Tue, May 16, 2017 at 7:01 AM, Deborah Swanson wrote:
> >>>>
> >>>> But I'm a little more mystified that official Python builds are leaning
> >>>> on Visual C++ (and that's what the crutch comment was primarily aimed
> >>>> at).
> >>>
> >>> You seem to be of the opinion that some day, binary executables will
> >>> be compiled using pure Python code. Maybe that's true; maybe it's not.
> >>
> >>
> >> More likely would be the option to ship a C compiler with Python written in
> >> C. For C++ this is way too big, but a pure C compiler can be as small as
> >> 1MB. tcc has a liberal license, supports many platforms and gives reasonable
> >> (unoptimized) code. AFAIK Mathworks does that, they ship tcc on Windows so
> >> that you can build .mex files without installing additional software, though
> >> they recommend to get a decent compiler for performance reasons
> >>
> >
> > To do that, Python would itself have to be compiled with tcc, or else
> > all memory de/allocation would have to be funneled through a
> > Python-provided API. And that's going to kill performance, I suspect.
>
> I can't test with Python because it's too complicated to compile,
> especially on Windows.
>

> --
> bartc

What is the problem with the documentation given here https://docs.python.org/devguide ?

Specifically:-

"and on Windows use:

PCbuild\build.bat -e -d"

Kindest regards.

Mark Lawrence.

[Chris Angelico]

On Wed, May 17, 2017 at 2:09 AM, bartc <b...@freeuk.com> wrote:
> But if I use tcc to compile /my/ byte-code interpreter, and then run the
> interpreter on a computationally intensive benchmark (jpeg decoding), I get
> these results for a given input file:
>
> Tiny C 6.7 seconds
> gcc -O3 3.6 seconds
>
> So it is not that terrible. And there are also other, smaller compilers
> which will generate better code than Tiny C. (I don't know why so many
> bundle gcc with their language apps because is not exactly compact.)

Does Tiny C compile to all the platforms that gcc does? And if you
want to support other platforms, what do you do?

ChrisA

[bartc]

On 16/05/2017 21:18, bream...@gmail.com wrote:
> On Tuesday, May 16, 2017 at 5:09:34 PM UTC+1, bartc wrote:

>> I can't test with Python because it's too complicated to compile,
>> especially on Windows.

> What is the problem with the documentation given here https://docs.python.org/devguide ?
>
> Specifically:-
>
> "and on Windows use:
>
> PCbuild\build.bat -e -d"

Everything?

That PCbuild line is step 3 of Quick Start. You have to get past steps 1
and 2 first. It talks about something called Git; I don't know what that
is or what I'm supposed to do with it, but it's a 35MB download. (Not
that that is very big these days, but 35MB (compressed size and there
may be more to come) suggests complexity that I don't want to get into.)

The Windows-specific link says I need to use MS VS2015. That's a BIG
application (I believe in the 1000s MB range for downloading) and my
track record with this stuff suggests that I will go to all this
trouble, spend a considerable amount of time, and then it'll fail at
some point and I won't have a clue why not.

So when I said it was too complicated to compile, I meant exactly that.

By contrast, the program I tested in place of CPython, builds
effortlessly with compilers in the 0.2MB size range, one of which is Tiny C.

--
bartc

[Chris Angelico]

On Wed, May 17, 2017 at 7:14 AM, bartc <b...@freeuk.com> wrote:
> That PCbuild line is step 3 of Quick Start. You have to get past steps 1 and
> 2 first. It talks about something called Git; I don't know what that is or
> what I'm supposed to do with it, but it's a 35MB download. (Not that that is
> very big these days, but 35MB (compressed size and there may be more to
> come) suggests complexity that I don't want to get into.)

It's 2017 and you're a programmer. Are you unfamiliar with source
control, or just unfamiliar with git? Either way, you should look into
correcting that.

ChrisA

[mm0fmf]

I smell trolling rather than ignorance.

[bream...@gmail.com]

On Tuesday, May 16, 2017 at 10:19:06 PM UTC+1, Chris Angelico wrote:

Please, be fair. Assuming of course that you can find out what "git" is, your 50 bps X25 link is going to take some time to download 35MB as broadband hasn't yet reached the UK. Then you've got to get Visual Studio Community Edition and follow the horrendously complicated instructions that I gave previously. It is impossible to do this unless you have at least five PhDs, all of which must be in completely different subjects. Come on you lot, own up, who didn't realise I was that smart?

Kindest regards.

Mark Lawrence.

[bartc]

On 16/05/2017 22:18, Chris Angelico wrote:

Sorry, no I don't. I like to do things the easy way. Other people can
tie themselves up in knots if they like, with all these complicated
tools. (Which, from what I can gather from the thread, don't always work
as expected.)

I understand that CPython is a C application, ie. a bunch of .c and .h
sources. So to build it, I need the sources, and a C compiler, yes?

So you download the sources (I've done that, actually; funnily enough I
didn't appear to need that Git thing). I've got several C compilers, but
not VS2015, and even managed to compile one module with two of them,
including Tiny C.

But now I need to know which files go where. This information I doubt
I'm going to get since developers like to hide this stuff in 'configure'
files and makefiles and project files. Sometimes it is necessary to run
a program to create a specific version of a file or some essential
header or whatever.

This sort of thing can be incredibly simple: download the sources,
compile modules X, Y and Z, link and run. If it's a bit more elaborate,
instructions can be provided in English (other languages are available).

We don't need to be locked in to these massive toolsets, where acquiring
them, learning them, and coaxing them to work is considerably more
effort than what we're trying to achieve in the first place.

--
bartc

[bartc]

There's another little flaw I forgot to mention.

The purpose of the test a few posts up was to get an idea of how fast
CPython might be when compiled with something like Tiny C, compared with
compiling it with gcc (on Linux) or MSVC (on Windows).

I've already got a working CPython binary to measure; I need the same
version built with Tiny C to compare.

Unfortunately downloading Visual Studio is not really going to help with
that. Good thing I remembered in time.

(Whether it is possible to build CPython with an alternative compiler,
I've no idea. I suspect that the source code has dependencies on both
gcc and MSVC. In other words, there will be conditional paths for gcc
(or clang which is a clone) and MSVC, with no other possibilities.)

--
bartc

[Chris Angelico]

On Wed, May 17, 2017 at 8:17 AM, bartc <b...@freeuk.com> wrote:
> On 16/05/2017 22:18, Chris Angelico wrote:
>>
>> On Wed, May 17, 2017 at 7:14 AM, bartc <b...@freeuk.com> wrote:
>>>
>>> That PCbuild line is step 3 of Quick Start. You have to get past steps 1
>>> and
>>> 2 first. It talks about something called Git; I don't know what that is
>>> or
>>> what I'm supposed to do with it, but it's a 35MB download. (Not that that
>>> is
>>> very big these days, but 35MB (compressed size and there may be more to
>>> come) suggests complexity that I don't want to get into.)
>>
>>
>> It's 2017 and you're a programmer. Are you unfamiliar with source
>> control, or just unfamiliar with git? Either way, you should look into
>> correcting that.
>
>
> Sorry, no I don't. I like to do things the easy way. Other people can tie
> themselves up in knots if they like, with all these complicated tools.
> (Which, from what I can gather from the thread, don't always work as
> expected.)

Then you are stuck in your own little bubble. That's fine as long as
you never try to foist your software OR your system on anyone else.
The rest of us collaborate with other people.

> I understand that CPython is a C application, ie. a bunch of .c and .h
> sources. So to build it, I need the sources, and a C compiler, yes?
>
> So you download the sources (I've done that, actually; funnily enough I
> didn't appear to need that Git thing). I've got several C compilers, but not
> VS2015, and even managed to compile one module with two of them, including
> Tiny C.

Sure, you can download a snapshot without git. How would you
contribute some changes upstream?

> But now I need to know which files go where. This information I doubt I'm
> going to get since developers like to hide this stuff in 'configure' files
> and makefiles and project files. Sometimes it is necessary to run a program
> to create a specific version of a file or some essential header or whatever.

How many combinations of compiler, standard library, operating system,
and CPU architecture do you support? Again, you're stuck in your own
petty little bubble and have no clue what it takes to build a big
project.

> We don't need to be locked in to these massive toolsets, where acquiring
> them, learning them, and coaxing them to work is considerably more effort
> than what we're trying to achieve in the first place.

You don't, because you've never used CPUs with different endianness,
or C standard libraries that vary in what functions they support, or
OSes that vary in the semantics of process starting and termination.
The rest of us do.

On Wed, May 17, 2017 at 7:35 AM, mm0fmf <no...@invalid.com> wrote:
> I smell trolling rather than ignorance.

Seems somewhere in between. It's wilful ignorance.

ChrisA

[bartc]

On 16/05/2017 23:33, Chris Angelico wrote:
> On Wed, May 17, 2017 at 8:17 AM, bartc <b...@freeuk.com> wrote:

> Then you are stuck in your own little bubble. That's fine as long as
> you never try to foist your software OR your system on anyone else.
> The rest of us collaborate with other people.

Other people who MUST have VS2015, otherwise nothing doing! Funny sort
of collaboration.

> Sure, you can download a snapshot without git. How would you
> contribute some changes upstream?

I thought the OP wanted to use a particular package not maintain it.
This is what those who supply open source seem to forget: most people
just want to build and use the software, not rewrite it!

And for that purpose, the method of distribution and building can be
considerably simpler.

> How many combinations of compiler, standard library, operating system,
> and CPU architecture do you support? Again, you're stuck in your own
> petty little bubble and have no clue what it takes to build a big
> project.

What do you mean what /I/ support?

I'm talking about opening up the build details of Python or a Python
component so it can be built with the nearest C compiler to hand. And C
compilers exist for pretty much any platform.

> You don't, because you've never used CPUs with different endianness,
> or C standard libraries that vary in what functions they support, or
> OSes that vary in the semantics of process starting and termination.
> The rest of us do.

And it sounds like the CPython developers have never used a compiler
other than gcc or MSVC, and the latter only reluctantly.

> On Wed, May 17, 2017 at 7:35 AM, mm0fmf <no...@invalid.com> wrote:
>> I smell trolling rather than ignorance.
>
> Seems somewhere in between. It's wilful ignorance.

On whose part? I can understand that (likely younger) people are so used
to working with large-scale tools where the details are hidden away,
then they find it inconceivable that it is possible to do without.

Anyway wasn't it you who suggested using TCC? This is a compiler that
was advertised as being only 100KB in size. (It might have been once,
but is nearer 200KB now.)

But since this is all apparently so easy for everyone else here accept
me, let one of them do this experiment: get CPython source code, compile
it with an alternative small compiler (Tiny C is one), and see how it
compares with CPython compiled with gcc or MSVC.

It should be a piece of cake, yes?

--
bartc

[Chris Angelico]

On Wed, May 17, 2017 at 9:01 AM, bartc <b...@freeuk.com> wrote:
> On 16/05/2017 23:33, Chris Angelico wrote:
>>
>> On Wed, May 17, 2017 at 8:17 AM, bartc <b...@freeuk.com> wrote:
>
>
>> Then you are stuck in your own little bubble. That's fine as long as
>> you never try to foist your software OR your system on anyone else.
>> The rest of us collaborate with other people.
>
>
> Other people who MUST have VS2015, otherwise nothing doing! Funny sort of
> collaboration.

No, I collaborate with people who use gcc and clang too. You don't
HAVE to use MSVC.

>> Sure, you can download a snapshot without git. How would you
>> contribute some changes upstream?
>
>
> I thought the OP wanted to use a particular package not maintain it. This is
> what those who supply open source seem to forget: most people just want to
> build and use the software, not rewrite it!

You're a programmer, not an end user, and you're saying that you don't
even know what git is.

> And for that purpose, the method of distribution and building can be
> considerably simpler.

You mean like wheel files? Yeah, whodathunk. They don't need a C
compiler or anything.

>> How many combinations of compiler, standard library, operating system,
>> and CPU architecture do you support? Again, you're stuck in your own
>> petty little bubble and have no clue what it takes to build a big
>> project.
>
>
> What do you mean what /I/ support?

Your programs. Anything you release. On how many of those combinations
do they work?

>> You don't, because you've never used CPUs with different endianness,
>> or C standard libraries that vary in what functions they support, or
>> OSes that vary in the semantics of process starting and termination.
>> The rest of us do.
>
>
> And it sounds like the CPython developers have never used a compiler other
> than gcc or MSVC, and the latter only reluctantly.

gcc, clang, msvc. That's three major compilers, at least one of which
is available on every major platform in use today. And at least one of
which is available on minor platforms too (gcc on OS/2, for instance,
which Python used to support until recently).

>> Seems somewhere in between. It's wilful ignorance.
>
>
> On whose part? I can understand that (likely younger) people are so used to
> working with large-scale tools where the details are hidden away, then they
> find it inconceivable that it is possible to do without.

How many CPUs have you compiled your code on? OSes? And how
complicated is the code?

> Anyway wasn't it you who suggested using TCC? This is a compiler that was
> advertised as being only 100KB in size. (It might have been once, but is
> nearer 200KB now.)

Not me, no. I'm too familiar with the hassles of mixing and matching
mallocs to recommend this.

> But since this is all apparently so easy for everyone else here accept me,
> let one of them do this experiment: get CPython source code, compile it with
> an alternative small compiler (Tiny C is one), and see how it compares with
> CPython compiled with gcc or MSVC.
>
> It should be a piece of cake, yes?
>

Well, let's see. What C standard does TCC support? What standard
library does it provide? If it supports all of C99 and links against
(say) GNU libc, then it's probably going to be fairly straight-forward
to compile CPython. If it supports C99 but has its own libc, you might
have to detect features to find out what you can and can't do with
it... yaknow, how the configure script does. That's what it's for. And
if TCC doesn't support C99, you may have major hassles. Specifically,
this is what you need:

https://www.python.org/dev/peps/pep-0007/#c-dialect

The specifications are laid out without naming compilers, other than a
passing reference to MSVC (since, as a general rule, the other
compilers support a lot more). You would be able to use TCC but only
if it's up to scratch.

ChrisA

[Michael Torrie]

On 05/16/2017 04:33 PM, Chris Angelico wrote:
> Then you are stuck in your own little bubble. That's fine as long as
> you never try to foist your software OR your system on anyone else.
> The rest of us collaborate with other people.

I don't collaborate with other people. However I still use git for my
little projects. Really, it's not that hard, and the benefits even to a
sole programmer are enormous. Delete a bunch of code only to find out
later you really want to revisit it? Just revert to an earlier commit.
Coming back to a project after a few weeks away and can't remember where
I was at? Review the commit logs, check the latest diff to see what I
was working on.

I really do recommend you take the few moments it takes to gain a casual
acquaintance with git or something like it, Bart.

[Michael Torrie]

On 05/16/2017 05:01 PM, bartc wrote:
> And it sounds like the CPython developers have never used a compiler
> other than gcc or MSVC, and the latter only reluctantly.

It's more a matter of choosing the best compiler to support the many
different platforms you wish to support. GCC is the defacto
cross-platform compiler. It makes sense to target it. If you stick to
the standards then you can can quite easily use different compilers on
certain platforms.

The real crux of the issue with Python on Windows comes down to a few
factors. First, nearly all developers use Visual Studio on Windows.
Second, all Windows programs must link against a version of the MS C
Standard library, which until recently was synced with releases of
Visual Studio. This is of particular importance to people who embed
Python in their own software, or write extension modules in C or C++.
The problem is that you really can't safely use more than one C runtime
DLL at a time. So if Python was built against MingW GCC, that for many
years was fixed to an older version of the MSVCRT.dll, if someone wanted
to write an extension for Python in VS, they'd have problems. Of course
this is an issue for different versions of VS as well, but now that MS
has released the universal C runtime DLL, you can now use a different
version of VS to build extensions than the one that build Python itself
(but not on XP).

Also there's the issue of C++ ABI compatibility across compilers (there
is none). So it makes sense to use the platform's dominant (default
even) compiler. GCC for *nix, Clang for OS X, VS for Windows, etc.

> But since this is all apparently so easy for everyone else here accept
> me, let one of them do this experiment: get CPython source code, compile
> it with an alternative small compiler (Tiny C is one), and see how it
> compares with CPython compiled with gcc or MSVC.
>
> It should be a piece of cake, yes?

If TCC implements the standard sufficiently, then yes it's possible.
However it won't be easy because the Python build tools are geared
towards the dominant compilers (GCC and VS), so you'd have to rework all
the build batch files first. Building an executable involves more than
just compiling a C file. There is dependency calculation, optional
compilation of features, linking, etc.

[Chris Angelico]

On Wed, May 17, 2017 at 9:31 AM, Michael Torrie <tor...@gmail.com> wrote:
> On 05/16/2017 04:33 PM, Chris Angelico wrote:
>> Then you are stuck in your own little bubble. That's fine as long as
>> you never try to foist your software OR your system on anyone else.
>> The rest of us collaborate with other people.
>
> I don't collaborate with other people. However I still use git for my
> little projects. Really, it's not that hard, and the benefits even to a
> sole programmer are enormous. Delete a bunch of code only to find out
> later you really want to revisit it? Just revert to an earlier commit.
> Coming back to a project after a few weeks away and can't remember where
> I was at? Review the commit logs, check the latest diff to see what I
> was working on.

This comes to mind:

http://www.catb.org/esr/writings/unix-koans/prodigy.html

ChrisA

[bartc]

Thank you, this is what I suggested several hours ago, that I was picked
up on:

"I can't test with Python because it's too complicated to compile,
especially on Windows.

But if I use tcc to compile /my/..." (implying I'd rather have used tcc
to compile CPython, if it had been trivial).

And actually my comments were also about building Python 'outside the
box' if that makes sense. Just manually going through the tasks because
there are certain to be issues that come up and I would need to know
where I am.

Although it seems the information needed to do this is either in the
18000 lines of configure script, or 7000 lines of VS project info, both
incomprehensible, so I doubt I would get far.

Trial and error might work, and tcc seems capable of compiling a few
sample modules I tried. But I don't have the inclination to battle all
the obstacles when all I wanted was a ball-park figure of how much
slower tcc would be than gcc at running CPython.

My much simpler test, on a similar application, suggested it might be
half the speed at worst.

--
bartc

[bartc]

On 17/05/2017 00:24, Chris Angelico wrote:
> On Wed, May 17, 2017 at 9:01 AM, bartc <b...@freeuk.com> wrote:

> You mean like wheel files? Yeah, whodathunk. They don't need a C
> compiler or anything.

I don't know if that's the same kind of thing. I'm not talking about
something like a binary distribution or something that self-installs.

I mean distributing actual source that needs to be built (so independent
of platform or compiler), but without having to download a sprawling
directory structure with thousands of files simply because that's the
layout needed during development. It's sort of in-between the
developer's sources, and a binary executable.

>> What do you mean what /I/ support?
>
> Your programs. Anything you release. On how many of those combinations
> do they work?

I'm not talking about my programs for a change. (I mentioned my compiler
as an example of one that is faster than tcc but not as fast as gcc.)

>> And it sounds like the CPython developers have never used a compiler other
>> than gcc or MSVC, and the latter only reluctantly.
>
> gcc, clang, msvc. That's three major compilers, at least one of which
> is available on every major platform in use today. And at least one of
> which is available on minor platforms too (gcc on OS/2, for instance,
> which Python used to support until recently).

In the CPython sources, there are references to GCC, clang, MSC (I
assume MSVC) and (oddly) WATCOM.

>> It should be a piece of cake, yes?
>>
>
> Well, let's see. What C standard does TCC support? What standard
> library does it provide? If it supports all of C99 and links against
> (say) GNU libc, then it's probably going to be fairly straight-forward
> to compile CPython. If it supports C99 but has its own libc, you might
> have to detect features to find out what you can and can't do with
> it... yaknow, how the configure script does. That's what it's for. And
> if TCC doesn't support C99, you may have major hassles.

OK. Now you understand what I meant that trying to compile CPython [with
TCC] was complicated, in order to get an idea of what performance would
be like.

--
bartc

[Chris Angelico]

On Wed, May 17, 2017 at 10:41 AM, bartc <b...@freeuk.com> wrote:
> On 17/05/2017 00:24, Chris Angelico wrote:
>>
>> On Wed, May 17, 2017 at 9:01 AM, bartc <b...@freeuk.com> wrote:
>
>
>> You mean like wheel files? Yeah, whodathunk. They don't need a C
>> compiler or anything.
>
>
> I don't know if that's the same kind of thing. I'm not talking about
> something like a binary distribution or something that self-installs.
>
> I mean distributing actual source that needs to be built (so independent of
> platform or compiler), but without having to download a sprawling directory
> structure with thousands of files simply because that's the layout needed
> during development. It's sort of in-between the developer's sources, and a
> binary executable.

*scratches head* What would that be? What is there between the
developer's sources and the binary? Are you implying that we have
messy development trees but we can run some program on them to produce
perfect, pristine, portable code?

ChrisA

[Terry Reedy]

On 5/16/2017 5:14 PM, bartc wrote:
> On 16/05/2017 21:18, bream...@gmail.com wrote:
>> On Tuesday, May 16, 2017 at 5:09:34 PM UTC+1, bartc wrote:
>
>>> I can't test with Python because it's too complicated to compile,
>>> especially on Windows.
>
>> What is the problem with the documentation given here
>> https://docs.python.org/devguide ?
>>
>> Specifically:-
>>
>> "and on Windows use:
>>
>> PCbuild\build.bat -e -d"
>
> Everything?
>
> That PCbuild line is step 3 of Quick Start. You have to get past steps 1
> and 2 first. It talks about something called Git; I don't know what that
> is or what I'm supposed to do with it, but it's a 35MB download. (Not
> that that is very big these days, but 35MB (compressed size and there
> may be more to come) suggests complexity that I don't want to get into.)

If you merely want to download and compile, you only need about 2 git
commands: 'clone' and the fetch-merge command. The devguide intros may
be enough. Creating patches is much more complex.

> The Windows-specific link says I need to use MS VS2015. That's a BIG
> application (I believe in the 1000s MB range for downloading) and my

Yes, one must download the Visual Studio Community Edition, 6-9
gigabytes. I had to install it on my terabyte auxiliary hard drive,
along with games, instead of my 72 gb SDD C: drive.

But once done, build.bat directly invokes the embedded C++ compiler.
Before it was added, one had to boot up Visual Studio (1/2 minute or
more) with the Python project files and hit the correct key. The .bat
file is *much* easier. I have it embedded in my own pull.bat that first
pulls updates and merges them into the local repository.

> track record with this stuff suggests that I will go to all this
> trouble, spend a considerable amount of time, and then it'll fail at
> some point and I won't have a clue why not.
>
> So when I said it was too complicated to compile, I meant exactly that.
>
> By contrast, the program I tested in place of CPython, builds
> effortlessly with compilers in the 0.2MB size range, one of which is
> Tiny C.

--

Terry Jan Reedy

[Terry Reedy]

The problem is that 'compiles with gcc' is not the same as 'compiles
with vc'. The CPython C sources have numerous ifdefs for Windows,
Darwin (Mac), Unix, and specific unix varieties. Package developers who
*have* successfully compiled on Windows should like provide wheels.

Christoph Gohlke, http://www.lfd.uci.edu/~gohlke/pythonlibs/
who has done us service of compiling over 300 packages on Windows,
offers patches back to developers who will accept them.

A server farm would need a person with a similar talent for adaptation.

--
Terry Jan Reedy

[Nathan Ernst]

MS used to, I'm not sure if they still do, provide a separate C++ SDK that
included the compiler, but not the full IDE. It was still quite a large
download at ~128MB. But, it included only the command-line compiler, linker
& std lib.

Starting with VS2017, the ABI is supposedly stable going foward (I've not
tested this) and the install is far more customizable. It's still quite a
hefty download, but you don't need the full 8GB just to be able to build a
python extension or python from source.

I don't think I've ever compiled python from source on windows (I most
certainly have on linux, and it's not hard), but I have compiled a number
of 3rd party libs. You simply need the requisite tools to do it. If you
don't have the tools, it's a futile effort. VS2015, VS2017 community
editions are free and provide the tools needed. They even include the
optimizing compiler, which used to be a premium. (VC6-VC7 days). The newest
runtimes only target windows 8 or later. With some effort to the build
scripts, one might be able to get it to build against an older C++ runtime,
but it certainly will not be supported.

Deborah, long term, you have to realize that if you insist on sticking to
WinXP, the rest of the world will, eventually, leave you behind.Software
vendors want to support the fewest platforms/configurations as possible. XP
is not supported by MS, so it makes no economic sense to an ISV to support
XP.

On Tue, May 16, 2017 at 8:56 PM, Terry Reedy <tjr...@udel.edu> wrote:

> On 5/16/2017 5:30 AM, Mark Summerfield via Python-list wrote:
>

> The problem is that 'compiles with gcc' is not the same as 'compiles with
> vc'. The CPython C sources have numerous ifdefs for Windows, Darwin (Mac),
> Unix, and specific unix varieties. Package developers who *have*
> successfully compiled on Windows should like provide wheels.
>
> Christoph Gohlke, http://www.lfd.uci.edu/~gohlke/pythonlibs/
> who has done us service of compiling over 300 packages on Windows,
> offers patches back to developers who will accept them.
>
> A server farm would need a person with a similar talent for adaptation.
>
>
> --
> Terry Jan Reedy
>

> --
> https://mail.python.org/mailman/listinfo/python-list
>

[Deborah Swanson]

Nathan Ernst wrote, on Tuesday, May 16, 2017 8:33 PM

> Deborah, long term, you have to realize that if you insist on
> sticking to WinXP, the rest of the world will, eventually,
> leave you behind.Software vendors want to support the fewest
> platforms/configurations as possible. XP is not supported by
> MS, so it makes no economic sense to an ISV to support XP.

Just to clarify, again (and again a dozen times), I will have a Linux
machine again as soon as it's possible within my constraints.

So please, all of you, stop beating me up about continuing to use XP. It
really is a fine operating system, but everyone else wants to get on
with their lives and stop supporting it. Not my fault, nor is it
something I should be continually beat around the head about.

I'm doing the best that I can. I'd like to see you know-it-alls do it
as well as I have, as crippled up with something closely related to
hydrocephalus for the last 15 years that the doctors gave up trying to
diagnose in 2005 and dirt poor as I am. But I diagnosed it and I'm
slowly getting well from it, and somehow I've managed to live in a
house with a nice garage and my trusty '76 Toyota pickup - and no it
wasn't my fairy godmother who gave it all to me. Some of us were born
thrifty and resourceful.

And I've been reading closely all the posts since I dropped out of this
thread. I saw no point in replying "that's nice, but I didn't know that"
to each one, but I'll definitely refer to this thread when I get this
far.

So thank you all.

Deborah

[Chris Angelico]

On Wed, May 17, 2017 at 2:14 PM, Deborah Swanson
<pyt...@deborahswanson.net> wrote:
> So please, all of you, stop beating me up about continuing to use XP. It
> really is a fine operating system, but everyone else wants to get on
> with their lives and stop supporting it. Not my fault, nor is it
> something I should be continually beat around the head about.
>

For the person who's using it, or for all the malware authors who are
gleefully using your computer as their own playground? It certainly is
a fine operating system for the latter.

ChrisA

[Deborah Swanson]

Chris Angelico wrote, on Tuesday, May 16, 2017 9:50 PM

Nobody's used my XP for a malware playground since about 2006. I used to
have about a dozen traps and traces that put a stop to all that. Now I
only have one, and it hasn't seen hide nor hair of a single trace or
sign of malware in the 6 or so years that I've had only that one sentry.

Not my problem if people can't use their computers and browse the web
sensibly. And I repeat, it's not something I should be continually beat
around the head about.

So please stop beating me up about it, you fools.

Deborah

[Gregory Ewing]

Dennis Lee Bieber wrote:
> I ended up having to download
> sources for 4.8, 4.7, and 4.6... And use GCC 4.9 to build 4.8, 4.8 to build
> 4.7, and 4.7 to build 4.6 -- I was NOW ready to build DragonEgg!

Could you not find a suitable binary package of 4.6 anywhere?

--
Greg

[Chris Angelico]

On Wed, May 17, 2017 at 4:31 PM, Deborah Swanson
<pyt...@deborahswanson.net> wrote:
> But the myth that modern Windows users are safe from malware is just
> that, a myth. I have plenty of Microsoftie friends who tell me privately
> that this is so.

Which is why people who care about security use BSD, not Windows.

ChrisA

[Deborah Swanson]

Chris Angelico wrote, on Tuesday, May 16, 2017 11:36 PM

Oh, I see. Now you're morphing your attack on Windows XP to all flavors
of Windows?

Not that I would entirely disagree. As I've said at least a dozen times
now, XP is merely an expediency for me until I can get resituated in
Linux, and I wouldn't have any Windows released after XP SP2.

So perhaps now you might be agreeing with me that investing in Windows
and Visual Studio now is a shortsighted move that will likely result in
long term grief for Python? Maybe reconsidering it?

Deborah

[Chris Angelico]

On Wed, May 17, 2017 at 4:59 PM, Deborah Swanson
<pyt...@deborahswanson.net> wrote:
> So perhaps now you might be agreeing with me that investing in Windows
> and Visual Studio now is a shortsighted move that will likely result in
> long term grief for Python? Maybe reconsidering it?

Not at all. I never said Windows wasn't important - just that it isn't
secure. Having Python able to run on Windows is a HUGE benefit for the
people who use Windows. Obviously it has a cost, and a quite
significant one, but it's not a bad decision on the part of the Python
core devs.

But if Windows were to die a quiet death (or, as some have predicted,
progressively morph into being a skin atop Linux, and then get spun
off as an open source project while Microsoft concentrates on Azure),
it would in my opinion be a great boon for the world.

ChrisA

[Deborah Swanson]

Chris Angelico wrote, on Wednesday, May 17, 2017 12:02 AM

Indeed.

And it made me glad to read earlier in this thread that Python devs are
leaving themselves a way out of this partnersip with Visual Studio, in
the form of only agreeing to use the VS for currently supported Windows
(or something like that).

I suppose that's at least one reason why XP has become a pariah in the
Python world, and I still predict grief coming from the Python-Visual
Studio alliance. But so it goes.

[wxjm...@gmail.com]

Le mercredi 17 mai 2017 09:23:10 UTC+2, Deborah Swanson a écrit :

Stick with Python 2 on XP. It's no so bad [*].
Python 3 on Windows just does not work as soon
as you leave the ascii world [*].

[*] Esp. for Western European Users.
[**] In fact, Python 3 just became more and
more disastrous release after release.

PS I'm finding this really funny.

[bartc]

Yes. This is done for sqlite for example:

https://www.sqlite.org/amalgamation.html

So people who want to just embed it in something else only have one main
.c file to worry about. The original multi-file structure and all the
gubbins needed to fit it all together are of no interest.

I do it on a smaller scale, combining dozens of source modules, which
are not even in C, plus other files requiring interpreted utilities to
set up (which then would require the sources of that interpreter),all
into ONE .c file. There's not even a header. Then it can be trivially
compiled.

You don't need a 35MB Git download (as I said, it's not the size, it's
getting it installed, finding out what to type, finding out why it
doesn't work...), you don't need a 6-8GB IDE, and you don't need to
download Cygwin or MSYS (ie. half of Linux) if you want to build
something that relies on Linux utilties.

Regarding portable code, there might be some tweaks needed to this model
to get that. It might be that you have to use to choose between a .c
file for Windows, and a .c file for Linux, similar to what happens now
when downloading binaries. Alternatively, there can be one .c file with
conditional sections.

There are ways of making things easier, simpler, more foolproof and with
fewer external dependencies. But little effort seems to be expended on
that; the developers don't care about having to inflict their own tools
on everyone else.

--
bartc

[Lele Gaifax]

Terry Reedy <tjr...@udel.edu> writes:

> If you merely want to download and compile, you only need about 2 git commands: 'clone' and the
> fetch-merge command. The devguide intros may be enough. Creating patches is much more
> complex.

Or just a few clicks on the github.com project, and download a plain zip
archive from https://github.com/python/cpython/archive/master.zip. Getting
a specific branch is left as an exercize for the reader.

ciao, lele.
--
nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri
real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia.
le...@metapensiero.it | -- Fortunato Depero, 1929.

[Rhodri James]

On 17/05/17 01:41, bartc wrote:
> On 17/05/2017 00:24, Chris Angelico wrote:
>> On Wed, May 17, 2017 at 9:01 AM, bartc <b...@freeuk.com> wrote:
>
>> You mean like wheel files? Yeah, whodathunk. They don't need a C
>> compiler or anything.
>
> I don't know if that's the same kind of thing. I'm not talking about
> something like a binary distribution or something that self-installs.
>
> I mean distributing actual source that needs to be built (so independent
> of platform or compiler), but without having to download a sprawling
> directory structure with thousands of files simply because that's the
> layout needed during development. It's sort of in-between the
> developer's sources, and a binary executable.

As a cross-platform developer, I find your naivity refreshing. If only
life were so simple.

When you develop code yourself, you can lay out your files however you
find most convenient, code to the foibles of your compiler, operating
system and indeed processor, and use whatever tools you choose. The
moment you expect anyone else to compile your code, all of those
conveniences go out of the window, and things become more complex, not
less. Arbitrary compilation environments require more support than
development, not less.

>>> What do you mean what /I/ support?
>>
>> Your programs. Anything you release. On how many of those combinations
>> do they work?
>
> I'm not talking about my programs for a change. (I mentioned my compiler
> as an example of one that is faster than tcc but not as fast as gcc.)

You should be. You are claiming something is easy. Chris invited you
to consider how easy it isn't for code you know well. That you keep
fighting shy of doing so does not speak well of you as a programmer.

>>> It should be a piece of cake, yes?
>>>
>>
>> Well, let's see. What C standard does TCC support? What standard
>> library does it provide? If it supports all of C99 and links against
>> (say) GNU libc, then it's probably going to be fairly straight-forward
>> to compile CPython. If it supports C99 but has its own libc, you might
>> have to detect features to find out what you can and can't do with
>> it... yaknow, how the configure script does. That's what it's for. And
>> if TCC doesn't support C99, you may have major hassles.
>
> OK. Now you understand what I meant that trying to compile CPython [with
> TCC] was complicated, in order to get an idea of what performance would
> be like.

Yes, it's complicated. That's the point: it *is* complicated. You seem
to think that it's been made complicated for you. It hasn't. The hard
work you had to put in to compile with TCC had to be put in by other
people for gcc, clang, MSVC, etc. That no one else put in the work for
TCC just indicates that no one else thought it was worth while.

--
Rhodri James *-* Kynesim Ltd

[bartc]

On 17/05/2017 13:35, Rhodri James wrote:
> On 17/05/17 01:41, bartc wrote:

> As a cross-platform developer, I find your naivity refreshing. If only
> life were so simple.
>
> When you develop code yourself, you can lay out your files however you
> find most convenient, code to the foibles of your compiler, operating
> system and indeed processor, and use whatever tools you choose. The
> moment you expect anyone else to compile your code, all of those
> conveniences go out of the window, and things become more complex, not
> less. Arbitrary compilation environments require more support than
> development, not less.

That's all true. But the answer is not to make it a nightmare for
everyone else as well as yourself. If the requirement is to get other
people to build your product from source for the purpose of using it or
testing it (and for various reasons using prebuilt binaries is not an
option), then the process ought to be as painless as possible.

Few people seem to see the point in making that effort.

>> I'm not talking about my programs for a change. (I mentioned my compiler
>> as an example of one that is faster than tcc but not as fast as gcc.)
>
> You should be. You are claiming something is easy. Chris invited you
> to consider how easy it isn't for code you know well. That you keep
> fighting shy of doing so does not speak well of you as a programmer.

It doesn't work. I can post something that is easy to compile, but
someone is going to say, Ah, but it doesn't do this, it doesn't do that,
it doesn't work on X...

But setting that aside, let's take something I know a little about:
compilers. Here's one project of mine, a C compiler:

https://github.com/bartg/langs/tree/master/bccproj

The actual compiler is the 'mcc64.c' link; just one C file.

(The project is not complete, but other than ending up somewhat bigger
than its current 200KB, its still only ever going to be one file.)

Compare ease of compiling with building other C compilers from source.
(And actually, this can compile itself, some 17Kloc, in a few tens of
milliseconds.)

Here's another project, a byte-code interpreter (not for Python!):

https://github.com/bartg/langs/blob/master/qlang/pcc64.c

Again one file. However, I've recently split this project into discrete
compiler and interpreter, as that suits me better. So it needs a compiler:

https://github.com/bartg/langs/blob/master/qlang/qcc64.c

Once more, a single .c source file. It should work with any C compiler;
the only stipulation is that a file with '64' in the name needs building
for 64-bits.

I believe these three examples are OS-neutral: they should work on
Windows or Linux. However, mcc64 targets Windows ABI.

Note: none of these three projects is written in C. C is used as an
intermediate language for the convenience, for portability, and for
taking advantage of optimised compiling.

> Yes, it's complicated. That's the point: it *is* complicated. You seem
> to think that it's been made complicated for you. It hasn't. The hard
> work you had to put in to compile with TCC had to be put in by other
> people for gcc, clang, MSVC, etc. That no one else put in the work for
> TCC just indicates that no one else thought it was worth while.

The hard work is because using the C language is such a minefield. It
can be made simpler, but people tend not to do that. That just add
layers of complexity: millions of #ifs and #ifdefs in source code, loads
of macros, elaborate IDEs to help find your way around thousands of
source files, scripting languages to help join all the dots because now
it's no longer practical to do it manually.

Look at the link for sqlite amalgamation I posted earlier today. On that
site, it says they extensively use TCL to generate some of the sources.
To build sqlite conventionally, would require someone to install a TCL
system. Their amalgamated version thankfully doesn't require that.


--
bartc

[bartc]

On 17/05/2017 14:53, bartc wrote:

> Note: none of these three projects is written in C. C is used as an

> intermediate language for the convenience, for ...

For the 'convenience of others' that should be before it was badly edited.

Not for me, as using C, even just generating C, is a pain in the arse.
But I make the effort when posting stuff to github. And actually, going
through that process, improves the quality of the code.

--
bartc

[Chris Angelico]

On Wed, May 17, 2017 at 11:53 PM, bartc <b...@freeuk.com> wrote:
> On 17/05/2017 13:35, Rhodri James wrote:
>>
>> On 17/05/17 01:41, bartc wrote:
>
>
>> As a cross-platform developer, I find your naivity refreshing. If only
>> life were so simple.
>>
>> When you develop code yourself, you can lay out your files however you
>> find most convenient, code to the foibles of your compiler, operating
>> system and indeed processor, and use whatever tools you choose. The
>> moment you expect anyone else to compile your code, all of those
>> conveniences go out of the window, and things become more complex, not
>> less. Arbitrary compilation environments require more support than
>> development, not less.
>
>
> That's all true. But the answer is not to make it a nightmare for everyone
> else as well as yourself. If the requirement is to get other people to build
> your product from source for the purpose of using it or testing it (and for
> various reasons using prebuilt binaries is not an option), then the process
> ought to be as painless as possible.

What, you mean like this?

./configure
make
sudo make install

I think the bulk of open source software can be built using those steps.

> Few people seem to see the point in making that effort.

Tell that to the folks behind automake. And scons. Etcetera.

>>> I'm not talking about my programs for a change. (I mentioned my compiler
>>> as an example of one that is faster than tcc but not as fast as gcc.)
>>
>>
>> You should be. You are claiming something is easy. Chris invited you
>> to consider how easy it isn't for code you know well. That you keep
>> fighting shy of doing so does not speak well of you as a programmer.
>
>
> It doesn't work. I can post something that is easy to compile, but someone
> is going to say, Ah, but it doesn't do this, it doesn't do that, it doesn't
> work on X...

....... yes. That's right. Because you live in your own little bubble,
and you don't even think about it not working for other people until
someone says "it doesn't work on my system". And then you blame that
person, because it's "easy to compile" for you on your system.

You're like the people who work with eight-bit character sets because
Unicode is "hard", and everything's just so easy because there's only
one code page that matters. And it's just called "ASCII" to you, even
though it's an eight-bit code. Does this sound at all familiar?

> But setting that aside, let's take something I know a little about:
> compilers. Here's one project of mine, a C compiler:
>
> https://github.com/bartg/langs/tree/master/bccproj
>
> The actual compiler is the 'mcc64.c' link; just one C file.
>
> (The project is not complete, but other than ending up somewhat bigger than
> its current 200KB, its still only ever going to be one file.)
>
> Compare ease of compiling with building other C compilers from source. (And
> actually, this can compile itself, some 17Kloc, in a few tens of
> milliseconds.)

Quotes from the README:

"""
(Both versions target x64 for Windows, although they should run under
Linux too.)

Note: this compiler is not guaranteed to compile any arbitrary C
programs, even within the limitations listed. It hasn't been tested on
enough sources. And the bundled headers are probably less than 5%
populated.
"""

In other words, it's only targeting one single CPU architecture and OS
API. Also, it emits nasm code, so if you actually want a binary, you
can't consider this complete; the size of nasm + linker needs to be
included. Yeah, it's not hard for a small program to produce valid
code for one target system. How good is the resultant code? Can you
outperform other popular compilers?

>> Yes, it's complicated. That's the point: it *is* complicated. You seem
>> to think that it's been made complicated for you. It hasn't. The hard
>> work you had to put in to compile with TCC had to be put in by other
>> people for gcc, clang, MSVC, etc. That no one else put in the work for
>> TCC just indicates that no one else thought it was worth while.
>
>
> The hard work is because using the C language is such a minefield. It can be
> made simpler, but people tend not to do that. That just add layers of
> complexity: millions of #ifs and #ifdefs in source code, loads of macros,
> elaborate IDEs to help find your way around thousands of source files,
> scripting languages to help join all the dots because now it's no longer
> practical to do it manually.

Again, you sound like the people who complain about Unicode, and how
it's so hard to work with. Actually, the difficulties are because *the
problem space* is hard to work with (either "compiling cross-platform
code for myriad CPUs/OSes/stdlibs" or "accurately representing the
vagaries of human language"), and what you're seeing is the vastly
simplified solution. Obviously it's much easier to solve a reduced
form of the problem, just as it's far easier to solve "2*x - 5 = 0"
than it is to solve "x^4 + 3*x^3 - 10*x^2 + 2*x - 5 = 0".
Congratulations.

> Look at the link for sqlite amalgamation I posted earlier today. On that
> site, it says they extensively use TCL to generate some of the sources. To
> build sqlite conventionally, would require someone to install a TCL system.
> Their amalgamated version thankfully doesn't require that.
>

Oh, you mean like how tkinter embeds tcl? You can't possibly do that
in a production language, can you.

ChrisA

[Deborah Swanson]

> Chris Angelico wrote, on Tuesday, May 16, 2017 9:50 PM

> > For the person who's using it, or for all the malware authors
> > who are gleefully using your computer as their own
> > playground? It certainly is a fine operating system for the latter.
> >
> > ChrisA
>
> Nobody's used my XP for a malware playground since about
> 2006. I used to have about a dozen traps and traces that put
> a stop to all that. Now I only have one, and it hasn't seen
> hide nor hair of a single trace or sign of malware in the 6
> or so years that I've had only that one sentry.
>
> Not my problem if people can't use their computers and browse
> the web sensibly. And I repeat, it's not something I should
> be continually beat around the head about.
>
> So please stop beating me up about it, you fools.
>
> Deborah

Can you imagine how much money and grief could have been saved with a
massive computer and internet use education blitz back in about 2005,
instead of the coddling and "saving from the malware monsters" that we
got instead? I figured out how to do it, surely others could. (I was way
too sick then to do any more than protect myself.)

People are barely little safer from malware today than they were in
2005. The evidence from that is the proliferation of identity theft,
fraudulent tax returns for refunds and the pervasive presence, power and
purloined wealth of the malware kings. And they don't just prey on
hapless XP users, follow the research yourself. Windows 7+ is
outstripping XP victims now, no doubt all the shaming to make XP users
upgrade has merely decimated their numbers and made the bandwagon
climbers-on vulnerable to all the Windows7+ tools the malware kings can
now employ (don't you guys read any of this stuff?). Windows 7+ is no
protection from malware, the malware kings are just stealthier and more
technically expert now. Just because your machine isn't behaving like a
malware-ridden XP did in the early 2000's doesn't mean that the malware
kings aren't getting you.

But the myth that modern Windows users are safe from malware is just
that, a myth. I have plenty of Microsoftie friends who tell me privately

that this is so. And if these new operating systems were designed from
the get go to be so safe from malware, why the need for the constant
stream of patches, and the thousands (or more) of zero-days they haven't
devised patches for yet, or they even know are out there to be patched?
Why are botnets still a flourishing enterprise, and now ransomware is on
the rise? Hint, the malware kings aren't raking in all their wealth and
power from XP machines alone.

It's a strange, strange world when people surrender their
responsibilities to the money makers who promise them safety and
security with no sweat required. And they get to let the NSA and other
government black actors spy on them for free in the bargain. It's a
strange, strange world.

Deborah

[Rhodri James]

Oh, the irony.

[bartc]

On 17/05/2017 15:13, Chris Angelico wrote:
> On Wed, May 17, 2017 at 11:53 PM, bartc <b...@freeuk.com> wrote:

>> That's all true. But the answer is not to make it a nightmare for everyone
>> else as well as yourself. If the requirement is to get other people to build
>> your product from source for the purpose of using it or testing it (and for
>> various reasons using prebuilt binaries is not an option), then the process
>> ought to be as painless as possible.
>
> What, you mean like this?
>
> ./configure
> make
> sudo make install

No, not like that. I mean genuinely simple. Your example:

(1) Doesn't work on Windows
(2) Usually seems to involve executing 20,000 to to 30,000 lines of
complete gobbledygook in that configuration script.

That can't possibly be justified.

One project I tried to compile comprised only a dozen modules, but still
required nearly 1000 lines of incantations in the combined configure and
makefile scripts to build it (which didn't work as it waS Windows).

> I think the bulk of open source software can be built using those steps.

Maybe, if within Unix or Linux.

What would happen if you were presented with a machine, of some unknown
OS except that it's not Linux, and only had an editor, a bare compiler
and linker to work with? Would you be completely helpless?

Suppose then the task was to run some Python, but you only had the bare
sources: .c and .h files, and whatever .py files come as standard; where
would you start? Would that 18,000-line configure script come in handy,
or would it be no use at all without your usual support systems?

And you're accusing /me/ of being in a bubble!

>> Compare ease of compiling with building other C compilers from source. (And
>> actually, this can compile itself, some 17Kloc, in a few tens of
>> milliseconds.)

> In other words, it's only targeting one single CPU architecture and OS
> API.

And? My gcc installation only generates code for x86, x64. So does my
Pelles C**. So does my lccwin**. So does my DMC (only x86). So does my
Tiny C**. So does my MSVC2008 (x86 only). All also only generate code
for Windows.

(** Those use separate installations to achieve those two targets.
lccwin is one installation but uses different sets of files.)

If I wanted to build gcc for example from sources, then I need to
download and grapple with a package containing 100,000 files, including
tens of thousands of source files, even if I'm only interested in one
target. /That/ is supposed to be better?

Anyway adding another target is no big deal. (I've targeted pdp10, z80,
8086, 80386[x86] on previous compilers, sometimes as asm, sometimes as
binary. Also C source code. I written assemblers for z80, 8051, 8086,
80186, 80386[partial] all generating binary.)

The Win64 ABI thing is a detail. Win64 API uses 4 registers to pass
parameters, and requires a shadow stack space; Linux uses 6 registers
and no shadow space, and handles XMM a bit differently. I think that's it...

> Also, it emits nasm code, so if you actually want a binary, you
> can't consider this complete; the size of nasm + linker needs to be
> included.

If I download Clang for Windows, then it doesn't emit anything at all!
In fact it can't compile anything. Because it relies on other compilers
(gcc or MSVC, but it has to be the right one) to supply header files,
linkers, everything it needs.

Clang is a 64MB executable.

But yes, Nasm is a poor match for my compiler, and unsatisfactory. The
next step is to eliminate that and the need for a linker. Then this
'mcc' project can run applications from source just like Python.

It's funny though that when /I/ stipulate third party dependencies
[small, self-contained ones like nasm or golink], then it's bad; when
other people do it [massive great ones like vs2015 or git] then that's good!

And I can only conclude from your comments, that CPython is also
incomplete because it won't work without a bunch of other tools, all
chosen to be the biggest and most elaborate there are.

Yeah, it's not hard for a small program to produce valid
> code for one target system. How good is the resultant code? Can you
> outperform other popular compilers?

For running my language projects, it might be 30% slower than gcc-O3,
about on a par with compilers such as DMC, lccwin, and PellesC, and
generally faster than Tiny C. [Here I'm mixing up results from my C
compiler and my non-C compiler; I've can't be bothered to do any
exhaustive testing because you'll simply find another way of belittling
my efforts.]

>> Look at the link for sqlite amalgamation I posted earlier today. On that
>> site, it says they extensively use TCL to generate some of the sources. To
>> build sqlite conventionally, would require someone to install a TCL system.
>> Their amalgamated version thankfully doesn't require that.
>>
>
> Oh, you mean like how tkinter embeds tcl? You can't possibly do that
> in a production language, can you.

No, I mean having to install some language system that is of no
relevance whatsoever to a developer, other than being a stipulation of
some third party add-on.

If you were paying someone by the hour to add sqlite to a project, would
you rather they just used the amalgamated file, or would you prefer they
spent all afternoon ******* about with TCL trying to get it to generate
those sources?

--
bartc

[Gene Heskett]

Pay attention to this lady, she has BTDT enough to make an astute
observation.

I'm hiding behind a locked down dd-wrt router, but sure as hell, some
black hat will find a way to get past it. No one has in a decade yet,
but... Deborah is correct. Everytime you lock a door, put another
differently locked door to back it up.

And its not going to change until the castle doctrine extends to ones net
presence.

Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

[bartc]

On 17/05/2017 16:06, Rhodri James wrote:

> On 17/05/17 14:53, bartc wrote:

>> It doesn't work. I can post something that is easy to compile, but
>> someone is going to say, Ah, but it doesn't do this, it doesn't do that,
>> it doesn't work on X...
>
> "Few people seem to see the point in making that effort."
>
> Oh, the irony.

You stopped reading too soon. I /have/ made the effort. I'm saying that
if I posted examples of how I make available my own code, then people
will piss all over them.

Which I did go on to do, and they did exactly that.

They're not looking for good points, but for anything that will make
them look bad. In which case it's always going to be futile, because
there's always going to be something to pick on.

--
bartc

[wxjm...@gmail.com]

Le mercredi 17 mai 2017 16:13:49 UTC+2, Chris Angelico a écrit :
>
> Again, you sound like the people who complain about Unicode, and how
> it's so hard to work with.

Sure it's very hard. Take Python 3, how many
years of development 6, 7, 8... and it is still
not working. In fact, "unicode" has never worked correctly,
Python 2 and since Python 3.0.

Maybe the problem is elsewhere.

:-)

[Chris Angelico]

On Thu, May 18, 2017 at 1:37 AM, bartc <b...@freeuk.com> wrote:
> On 17/05/2017 15:13, Chris Angelico wrote:
>>
>> On Wed, May 17, 2017 at 11:53 PM, bartc <b...@freeuk.com> wrote:
>
>
>>> That's all true. But the answer is not to make it a nightmare for
>>> everyone
>>> else as well as yourself. If the requirement is to get other people to
>>> build
>>> your product from source for the purpose of using it or testing it (and
>>> for
>>> various reasons using prebuilt binaries is not an option), then the
>>> process
>>> ought to be as painless as possible.
>>
>>
>> What, you mean like this?
>>
>> ./configure
>> make
>> sudo make install
>
>
> No, not like that. I mean genuinely simple. Your example:
>
> (1) Doesn't work on Windows
> (2) Usually seems to involve executing 20,000 to to 30,000 lines of complete
> gobbledygook in that configuration script.
>
> That can't possibly be justified.

[1] Does work on Windows. Install bash for Windows, or (on a
recent-enough Windows) just use the subsystem that Microsoft provides.
[2] So? You don't have to read it, just run it. Or do you read every
line of code that any of your programs executes?

> What would happen if you were presented with a machine, of some unknown OS
> except that it's not Linux, and only had an editor, a bare compiler and
> linker to work with? Would you be completely helpless?

I didn't even have an editor the first time I had that situation. And
it worked, because I was using the same C compiler that I was
accustomed to - it existed for that platform. I was specifically NOT
helpless because I am used to using good tools.

How about if I put you on a different CPU than you're used to? Can you
use your tiny C compiler? I doubt it, because it's emitting Intel byte
code.

> Suppose then the task was to run some Python, but you only had the bare
> sources: .c and .h files, and whatever .py files come as standard; where
> would you start? Would that 18,000-line configure script come in handy, or
> would it be no use at all without your usual support systems?
>
> And you're accusing /me/ of being in a bubble!

You're asking me to bootstrap Python. I would start by looking for the
nearest similar platform and trying to build a hybrid. I haven't done
this with Python itself, but a while ago, I wanted to port a
similarly-sized language to OS/2, and the process went like this:

1) Attempt to run the configure script, using bash and gcc (which
already existed for OS/2)
2) Locate the handful of places in the code where an ifdef checked for
"Windows vs POSIX" regarding path names and make it "Windows or OS/2
vs POSIX" (since Windows and OS/2 both use the MS-DOS style of path
names)
3) Fix exactly *one* flaw in the makefiles, where it wasn't compatible
with a .exe file name extension when running under a broadly POSIX
system.

And #3 resulted in an upstream patch (which wasn't accepted in its
original form, but a refined version did make it into trunk). Yes,
that configure script came in *VERY* handy, because it took care of
99.9999% of the potential differences between systems. (And by the
way, the real source code for configure is actually configure.in,
which is way shorter than the 18,000 lines you cite.) Once you've
supported Red Hat Linux, Debian Linux, Debian Hurd, Fedora PPC,
FreeBSD, OpenBSD, Windows, Solaris, OpenIndiana, HP-UX, AIX, Mac OS 9,
Mac OS X, SUSE on an s390, and whatever else I've forgotten, it's not
hard to add support for OS/2.

>> In other words, it's only targeting one single CPU architecture and OS
>> API.
>
>
> And? My gcc installation only generates code for x86, x64. So does my Pelles
> C**. So does my lccwin**. So does my DMC (only x86). So does my Tiny C**. So
> does my MSVC2008 (x86 only). All also only generate code for Windows.

But that's simply because you didn't choose to install other GCC back
ends. GCC itself supports plenty of CPUs.

> If I wanted to build gcc for example from sources, then I need to download
> and grapple with a package containing 100,000 files, including tens of
> thousands of source files, even if I'm only interested in one target. /That/
> is supposed to be better?

Yes. Yes, it is. For starters, GCC actually can compile to machine
code, instead of depending on nasm. So you don't have a fair
comparison. For seconds, GCC actually implements the entire C standard
(several versions of the C standard, in fact), plus C++, and Fortran.
Once you support the entire C99 standard (I won't even ask you to
support C11) and actual executable output, you can start comparing.

> Anyway adding another target is no big deal. (I've targeted pdp10, z80,
> 8086, 80386[x86] on previous compilers, sometimes as asm, sometimes as
> binary. Also C source code. I written assemblers for z80, 8051, 8086, 80186,
> 80386[partial] all generating binary.)

Then do it. Make your compiler able to target all of the above. See
how much ifdef mess you need.

> The Win64 ABI thing is a detail. Win64 API uses 4 registers to pass
> parameters, and requires a shadow stack space; Linux uses 6 registers and no
> shadow space, and handles XMM a bit differently. I think that's it...

Sure. That's the calling convention. What about having the full header
files so you can actually type-check your calls?

> It's funny though that when /I/ stipulate third party dependencies [small,
> self-contained ones like nasm or golink], then it's bad; when other people
> do it [massive great ones like vs2015 or git] then that's good!

Might that possibly be because your program is utterly useless without
them, but Python only needs them for compilation? A fully-built Python
doesn't depend on a C compiler or git or anything.

Plus, git shouldn't be considered an onerous requirement for a
developer. Go get it. Start using it. And I don't just mean uploading
stuff to github, I mean actually using it to track your changes during
development.

> And I can only conclude from your comments, that CPython is also incomplete
> because it won't work without a bunch of other tools, all chosen to be the
> biggest and most elaborate there are.

Exactly. The core devs reject small solutions, even if they're
perfect, in order to pick up a much larger and more elaborate one.
That's how they make technology decisions.

And you accuse me of belittling?

> For running my language projects, it might be 30% slower than gcc-O3, about
> on a par with compilers such as DMC, lccwin, and PellesC, and generally
> faster than Tiny C. [Here I'm mixing up results from my C compiler and my
> non-C compiler; I've can't be bothered to do any exhaustive testing because
> you'll simply find another way of belittling my efforts.]

You'll have to compare like for like, and on a serious benchmark. I'm
not going to believe "might be 30% slower" without some actual
numbers.

> If you were paying someone by the hour to add sqlite to a project, would you
> rather they just used the amalgamated file, or would you prefer they spent
> all afternoon ******* about with TCL trying to get it to generate those
> sources?

I would expect them to use a precompiled binary, frankly. Why use the
amalgamated file when you can just use your system's package manager
to grab a ready-to-go version? Unless, of course, you're stuck on a
platform that doesn't even HAVE a package manager, in which case your
options are (a) get a package manager, and (b) grab a precompiled
binary off the web and use that.

Rhodri is right. Your naivety is so charming. You seem to genuinely
think that life can be that simple.

ChrisA

[bartc]

On 17/05/2017 17:23, Chris Angelico wrote:
> On Thu, May 18, 2017 at 1:37 AM, bartc <b...@freeuk.com> wrote:
>> On 17/05/2017 15:13, Chris Angelico wrote:

> [1] Does work on Windows. Install bash for Windows, or (on a
> recent-enough Windows) just use the subsystem that Microsoft provides.

So the answer is, switch to Linux. In other words, a cop-out.

> [2] So? You don't have to read it, just run it. Or do you read every
> line of code that any of your programs executes?

Sometimes, if there's a problem. But usually the code is doing something
sensible. The stuff in configure is complete gobbledygook (if anyone
doesn't believe me, just have look).

It is impossible that all this is needed just to figure out what source
files need to be compiled. (If it generated CPython sources
fractal-style, then I might be impressed, but doesn't.)

> I didn't even have an editor the first time I had that situation. And
> it worked, because I was using the same C compiler that I was
> accustomed to - it existed for that platform. I was specifically NOT
> helpless because I am used to using good tools.
>
> How about if I put you on a different CPU than you're used to? Can you
> use your tiny C compiler? I doubt it, because it's emitting Intel byte
> code.

I'm used to it. I used to start off with machines with no software in
them at all. (Actually, I often needed to wire up the boards first.)

The point I'm making is that these things should still work when reduced
to the basics.

(As it happens, I'm done this on an original raspberry pi machine. It
will need an existing C compiler to get started, after which my
programs, interpreters and compilers, run fine.

But it's funny waiting for gcc to compile my mcc64.c file (taking 30 to
120 seconds depending on optimisation), and then it compiles itself in
0.7 seconds. Of course that would be targetting x64 so the output is not
much use.

What it does show is that it is practical to reduce a substantial
application to a single file, and trivially build it pretty much
everywhere, as C compilers are ubiquitous. And that it will run with
little trouble. The same source can run on ARM32, x86 and x64; Linux or
Windows. Probably elsewhere but that is the extent of my tests.)

>
> You're asking me to bootstrap Python. I would start by looking for the
> nearest similar platform and trying to build a hybrid. I haven't done
> this with Python itself, but a while ago, I wanted to port a
> similarly-sized language to OS/2, and the process went like this:
>
> 1) Attempt to run the configure script, using bash and gcc (which
> already existed for OS/2)

I said bash didn't exist.

(BTW I've gone down the route of installing MSYS etc many times,
something always went wrong. Why is it that people can't appreciate that
complicated thing are more likely to go wrong?)

You need hard information in the form of what exactly needs to be done
with the .c and .h files provided. If it's necessary that another
language is needed to generate some of those, then sources for that
program are needed too.

However, that would also suggest that the application (CPython etc) is
not a pure C application at all.

> Yes. Yes, it is. For starters, GCC actually can compile to machine
> code, instead of depending on nasm.

I don't think it does. gcc AFAIK generates assembly code (via pipes or
equivalent so usually you don't see it), then invokes an assembler
called 'as' to convert into object code.

I believe that the base gcc compiler is separate from the bin-utils such
as 'as' and 'ld' (the linker).

Downloading gcc tdm for Windows, these are separate download bundles. On
Linux, this stuff is pre-installed so it's easy to think it's all the
same thing.

When /my/ compiler does it properly, then it will be truly
self-contained. And still be one executable.

> Then do it. Make your compiler able to target all of the above. See
> how much ifdef mess you need.

My language doesn't have ifdef. There are other ways of managing this
switch. (Actually my current compiler for that language is interpreted.
It supports two targets.)

> Plus, git shouldn't be considered an onerous requirement for a
> developer. Go get it. Start using it.

I'm doing this for fun. It's not a job to be taken seriously. And doing
battle with other people's temperamental multi-GB applications is not my
idea of fun.

(I mean, even if I did so, what can I contribute? Nothing.)

>> And I can only conclude from your comments, that CPython is also incomplete
>> because it won't work without a bunch of other tools, all chosen to be the
>> biggest and most elaborate there are.
>
> Exactly. The core devs reject small solutions, even if they're
> perfect, in order to pick up a much larger and more elaborate one.
> That's how they make technology decisions.

You're being sarcastic, but you may have a hit upon the truth.

I mean, is there any bigger development system they could have chosen
than VS 2015? (And someone mentioned that VS 2017 needed 25GB or something.

(We are still talking about building a byte-code compiler, and byte-code
interpreter, aren't we?)

> You'll have to compare like for like, and on a serious benchmark. I'm
> not going to believe "might be 30% slower" without some actual
> numbers.

I've put some comments here: https://pastebin.com/hSCjNsA2 as it's
getting way off-topic.

>> If you were paying someone by the hour to add sqlite to a project, would you
>> rather they just used the amalgamated file, or would you prefer they spent
>> all afternoon ******* about with TCL trying to get it to generate those
>> sources?
>
> I would expect them to use a precompiled binary, frankly. Why use the
> amalgamated file when you can just use your system's package manager
> to grab a ready-to-go version? Unless, of course, you're stuck on a
> platform that doesn't even HAVE a package manager, in which case your
> options are (a) get a package manager, and (b) grab a precompiled
> binary off the web and use that.
>
> Rhodri is right. Your naivety is so charming. You seem to genuinely
> think that life can be that simple.

And some seem to genuinely think that it needs to be that complicated
just because there is plenty of memory to fill up.

I started computing 40 years ago: you couldn't have complicated because
the hardware wasn't up to it. Things still worked. Having more memory,
more storage and faster processors doesn't mean complexity has to
increase too.

--
bartc

[bream...@gmail.com]

On Wednesday, May 17, 2017 at 8:32:40 PM UTC+1, bartc wrote:
> On 17/05/2017 17:23, Chris Angelico wrote:
> > On Thu, May 18, 2017 at 1:37 AM, bartc <b...@freeuk.com> wrote:
> >> On 17/05/2017 15:13, Chris Angelico wrote:
>
> > [1] Does work on Windows. Install bash for Windows, or (on a
> > recent-enough Windows) just use the subsystem that Microsoft provides.
>
> So the answer is, switch to Linux. In other words, a cop-out.
>

> --
> bartc

Poor old Chris seems to be getting a little frustrated so I'll just repeat what I said previously.

<quote>
What is the problem with the documentation given here https://docs.python.org/devguide ?

Specifically:-

"and on Windows use:

PCbuild\build.bat -e -d"
</quote>

That is 24 key strokes, including ENTER, to do the build on Windows. What is so difficult about this?

Kindest regards.

Mark Lawrence.

[Michael Torrie]

On 05/17/2017 01:32 PM, bartc wrote:
> Sometimes, if there's a problem. But usually the code is doing something
> sensible. The stuff in configure is complete gobbledygook (if anyone
> doesn't believe me, just have look).

Well trying to edit an executable in a disassembler would look like
gobligook as well. configure is a generated script from a compiler
called autoconf. It is compiled from some high-level macros that are
somewhat easier to understand. The system is convenient because we can
abstract away differences in target OS platforms more easily.

> It is impossible that all this is needed just to figure out what
> source files need to be compiled. (If it generated CPython sources
> fractal-style, then I might be impressed, but doesn't.)

What a wonderful simplification! But yes, you're essentially correct.
All this gobbligook figures out how to best configure the features and
options you require, determines whether your system has the required
compilers and libraries, figures out what source files should be
compiled, and calculates the order in which to build the source files.
All of things things are very important. I'd rather have this and keep
my build times down to seconds and minutes than have to recompile all
million lines of code for just one change.

I admit autoconf is hard to wrap my head around, hard to use as a
programmer, and even harder to extend to handle new libraries and new
compilers.

You'd probably find scons more to your liking, and Qt's QMake facilities
are very nice and simple, and cross-platform. But all of these generate
Makefiles, which would likely be considered unnecessary gobbligook to
you, I suspect.

[bartc]

I'll repeat what I said before: I don't want to build CPython with
VS2015. I wanted to build it with Tiny C.

The purpose of doing the latter was to evaluate what the performance
might be, if for example Tiny C became the bundled C compiler so as to
quickly get a source distribution going. Bundling VS2015 with CPython
sources is not going to happen.

(Although I wouldn't really be surprised if someone tried it. People
here don't seem to see anything wrong with cracking a small nut with a
particularly massive sledgehammer.)

--
bartc

[Chris Angelico]

On Thu, May 18, 2017 at 5:32 AM, bartc <b...@freeuk.com> wrote:
> On 17/05/2017 17:23, Chris Angelico wrote:
>>
>> On Thu, May 18, 2017 at 1:37 AM, bartc <b...@freeuk.com> wrote:
>>>
>>> On 17/05/2017 15:13, Chris Angelico wrote:
>
>
>> [1] Does work on Windows. Install bash for Windows, or (on a
>> recent-enough Windows) just use the subsystem that Microsoft provides.
>
>
> So the answer is, switch to Linux. In other words, a cop-out.

No, I said "install bash". Since when does bash imply Linux? You can
get bash for any of the major platforms in use today. It's the default
user login shell on most Mac OS and Linux systems, as well as a good
smattering of other OSes.

>> [2] So? You don't have to read it, just run it. Or do you read every
>> line of code that any of your programs executes?
>
>
> Sometimes, if there's a problem. But usually the code is doing something
> sensible. The stuff in configure is complete gobbledygook (if anyone doesn't
> believe me, just have look).

That's because "configure" isn't source code. It's like someone
looking at your nasm files and moaning that they're unreadable. You
don't directly read or edit the configure script.

> It is impossible that all this is needed just to figure out what source
> files need to be compiled. (If it generated CPython sources fractal-style,
> then I might be impressed, but doesn't.)

I love how absolutely certain you are that this is *impossible*. Ahh,
that naivety again. Though in this case, you do happen to be correct:
it's doing more than merely figuring out which files to compile. It's
figuring out what libraries are available, what features are available
in those libraries, how the compiler produces object files, whether
the build environment is sane, and in some cases, enumerating all
available compilers so you can pick and choose. Imagine if your
compiler could detect whether nasm, tasm, masm, or gas is installed,
and emit appropriate assembly code for any of them. That's what you
need a configure script for.

>> How about if I put you on a different CPU than you're used to? Can you
>> use your tiny C compiler? I doubt it, because it's emitting Intel byte
>> code.
>
> I'm used to it. I used to start off with machines with no software in them
> at all. (Actually, I often needed to wire up the boards first.)
>
> The point I'm making is that these things should still work when reduced to
> the basics.

Define "basics". You seem to assume an Intel x86 CPU and nothing else,
or your program won't be any use. What's the point in reducing below
an OS if you're still depending on a specific line of CPUs? Are you
expecting to compile your own OS kernel? Type one in at the console?

> What it does show is that it is practical to reduce a substantial
> application to a single file, and trivially build it pretty much everywhere,
> as C compilers are ubiquitous. And that it will run with little trouble. The
> same source can run on ARM32, x86 and x64; Linux or Windows. Probably
> elsewhere but that is the extent of my tests.)

"substantial". Exactly how much code are you talking about? And how
portable does it need to be?

>> You're asking me to bootstrap Python. I would start by looking for the
>> nearest similar platform and trying to build a hybrid. I haven't done
>> this with Python itself, but a while ago, I wanted to port a
>> similarly-sized language to OS/2, and the process went like this:
>>
>> 1) Attempt to run the configure script, using bash and gcc (which
>> already existed for OS/2)
>
> I said bash didn't exist.
>
> (BTW I've gone down the route of installing MSYS etc many times, something
> always went wrong. Why is it that people can't appreciate that complicated
> thing are more likely to go wrong?)

Why would bash not exist? Or do you mean that it exists, but isn't installed?

>> Yes. Yes, it is. For starters, GCC actually can compile to machine
>> code, instead of depending on nasm.
>
>
> I don't think it does. gcc AFAIK generates assembly code (via pipes or
> equivalent so usually you don't see it), then invokes an assembler called
> 'as' to convert into object code.
>
> I believe that the base gcc compiler is separate from the bin-utils such as
> 'as' and 'ld' (the linker).

Check that. I think you'll find that you measured the size of the
compiler with the assembler.

> Downloading gcc tdm for Windows, these are separate download bundles. On
> Linux, this stuff is pre-installed so it's easy to think it's all the same
> thing.

No, it isn't pre-installed. Have you ever actually used Linux, or do
you just express assumptions as facts?

> When /my/ compiler does it properly, then it will be truly self-contained.
> And still be one executable.

Sure. Now tell me. What's your standard library? Let's suppose you
want to sort an array of numbers, for instance. Do you have a stdlib
function available to do that? And if so, do you include a complete
copy of your entire stdlib in every one of your output executables?

>> Then do it. Make your compiler able to target all of the above. See
>> how much ifdef mess you need.
>
> My language doesn't have ifdef. There are other ways of managing this
> switch. (Actually my current compiler for that language is interpreted. It
> supports two targets.)

Whatever you call it, you will have that switch, because platforms are
fundamentally different.

>> Plus, git shouldn't be considered an onerous requirement for a
>> developer. Go get it. Start using it.
>
> I'm doing this for fun. It's not a job to be taken seriously. And doing
> battle with other people's temperamental multi-GB applications is not my
> idea of fun.

If you aren't using git, you definitely won't be taken very seriously.
And git isn't temperamental; it's a perfectly well behaved servant
that we teach to our students in their *first day*. Why? Because it is
essential. (Actually, to be fair, they get more git workflow stuff
through their first week, but that's because we go beyond the basics
and discuss collaboration workflows, branching/merging, conflict
resolution, and more. But before the first day is over, every student
can handle the basics of creating a repository, adding files,
committing, pushing, pulling, etc.)

>> You'll have to compare like for like, and on a serious benchmark. I'm
>> not going to believe "might be 30% slower" without some actual
>> numbers.
>
>
> I've put some comments here: https://pastebin.com/hSCjNsA2 as it's getting
> way off-topic.

Not interested in clicking on a pastebin with all its ads.

>> Rhodri is right. Your naivety is so charming. You seem to genuinely
>> think that life can be that simple.
>
> And some seem to genuinely think that it needs to be that complicated just
> because there is plenty of memory to fill up.
>
> I started computing 40 years ago: you couldn't have complicated because the
> hardware wasn't up to it. Things still worked. Having more memory, more
> storage and faster processors doesn't mean complexity has to increase too.

Yeah. Things worked because IBM sold you a computer with its software,
and nobody ever expected that software to run on non-IBM hardware. Or
HP sold you a computer, and you didn't expect its software to run on
non-HP hardware. Even sharing data files wasn't easy, because one
computer used ASCII where another used EBCDIC, and they weren't
consistent with floating-point formats, and over in Japan they had
endless other differences, and nobody even thought about Chinese
characters in computers. Like I said, it's easy to solve a tiny
fraction of a problem.

ChrisA

[bream...@gmail.com]

On Wednesday, May 17, 2017 at 9:27:58 PM UTC+1, bartc wrote:

> On 17/05/2017 21:05, Mark Lawrence wrote:
> > On Wednesday, May 17, 2017 at 8:32:40 PM UTC+1, bartc wrote:
> >> On 17/05/2017 17:23, Chris Angelico wrote:
> >>> On Thu, May 18, 2017 at 1:37 AM, bartc <b...@freeuk.com> wrote:
> >>>> On 17/05/2017 15:13, Chris Angelico wrote:
> >>
> >>> [1] Does work on Windows. Install bash for Windows, or (on a
> >>> recent-enough Windows) just use the subsystem that Microsoft provides.
> >>
> >> So the answer is, switch to Linux. In other words, a cop-out.
> >>
> >> --
> >> bartc
> >
> > Poor old Chris seems to be getting a little frustrated so I'll just repeat what I said previously.
> >
> > <quote>
> > What is the problem with the documentation given here https://docs.python.org/devguide ?
> >
> > Specifically:-
> >
> > "and on Windows use:
> >
> > PCbuild\build.bat -e -d"
> > </quote>
> >
> > That is 24 key strokes, including ENTER, to do the build on Windows. What is so difficult about this?
>
> I'll repeat what I said before: I don't want to build CPython with
> VS2015. I wanted to build it with Tiny C.

Okay, so when you get it working please provide all the patches needed so that other people can do the same. You'll have to learn a few git commands to do that, but that should be a piece of cake for someone with 40 years programming experience. Incidentally which source control systems have you used during your career?

>
> The purpose of doing the latter was to evaluate what the performance
> might be, if for example Tiny C became the bundled C compiler so as to
> quickly get a source distribution going. Bundling VS2015 with CPython
> sources is not going to happen.
>

What makes you think that VS2015 needs to be bundled with cPython sources?

Kindest regards.

Mark Lawrence.

[Chris Angelico]

On Thu, May 18, 2017 at 6:27 AM, bartc <b...@freeuk.com> wrote:
> I'll repeat what I said before: I don't want to build CPython with VS2015. I
> wanted to build it with Tiny C.
>
> The purpose of doing the latter was to evaluate what the performance might
> be, if for example Tiny C became the bundled C compiler so as to quickly get
> a source distribution going. Bundling VS2015 with CPython sources is not
> going to happen.
>
> (Although I wouldn't really be surprised if someone tried it. People here
> don't seem to see anything wrong with cracking a small nut with a
> particularly massive sledgehammer.)

Before you can build CPython with Tiny C, you need to be sure that Tiny C can:

1) malloc and free arbitrary blocks of memory
2) open and close files and do other file system operations
3) perform trignometric calculations (sin/cos/tan and friends)
4) start threads and subprocesses
5) ascertain the time of day and any other clocks available
6) acquire and release semaphores
etc, etc, etc.

These features are provided generally by the C standard library. Does
Tiny C support all of those? If so, how? Do you include a full
standard library, or link against someone else's?

ChrisA

[bartc]

On 17/05/2017 21:17, Michael Torrie wrote:
> On 05/17/2017 01:32 PM, bartc wrote:
>> Sometimes, if there's a problem. But usually the code is doing something
>> sensible. The stuff in configure is complete gobbledygook (if anyone
>> doesn't believe me, just have look).
>
> Well trying to edit an executable in a disassembler would look like
> gobligook as well.

The content of configure is high-level human readable source code. But
it is meaningless.

> What a wonderful simplification! But yes, you're essentially correct.
> All this gobbligook figures out how to best configure the features and
> options you require,

Well, while we're here, why don't /you/ run configure and tell me what
the output is. Preferably in a form I can understand.

Here's the input data:

OS: Windows 7
Processor: Intel 64-bit

Um, that's it I guess. I only need enough info to build, say,
python.exe; just something to get to first base. (I don't know what the
other binary files are or where they live.)

(I have visions of you now feeding a punched card into a computer called
Multivac occupying a cubic mile of vacuum tubes and relays... That's
about the same degree of overkill!)

> determines whether your system has the required
> compilers and libraries, figures out what source files should be
> compiled, and calculates the order in which to build the source files.

Yes! That's what I need!

But either this information is highly classified, or nobody here
actually knows how to extract it from the output of configure or the
makefile or wherever it ends up.

Or, more importantly, how this list of files could be determined manually.

> All of things things are very important. I'd rather have this and keep
> my build times down to seconds and minutes than have to recompile all
> million lines of code for just one change.

Once again, this is not for development. It's a one-off build of a fresh
download, so everything will need compiling anyway.

> I admit autoconf is hard to wrap my head around, hard to use as a
> programmer, and even harder to extend to handle new libraries and new
> compilers.

Has someone thought about replacing it then? I would be nervous about
running 20000 lines of AI-generated code that manipulates my file system.

> are very nice and simple, and cross-platform. But all of these generate
> Makefiles, which would likely be considered unnecessary gobbligook to
> you, I suspect.

Yes. The end-result of all this might be something like:

gcc -c file1.c
gcc -c file2.c
...
gcc file1.o file2.o ... -opython.exe

Someone please explain why I can't just do that

--
bartc

[bartc]

On 17/05/2017 21:39, Chris Angelico wrote:
> On Thu, May 18, 2017 at 5:32 AM, bartc <b...@freeuk.com> wrote:

>> It is impossible that all this is needed just to figure out what source
>> files need to be compiled. (If it generated CPython sources fractal-style,
>> then I might be impressed, but doesn't.)
>
> I love how absolutely certain you are that this is *impossible*. Ahh,
> that naivety again. Though in this case, you do happen to be correct:
> it's doing more than merely figuring out which files to compile. It's
> figuring out what libraries are available, what features are available
> in those libraries, how the compiler produces object files, whether
> the build environment is sane, and in some cases, enumerating all
> available compilers so you can pick and choose. Imagine if your
> compiler could detect whether nasm, tasm, masm, or gas is installed,
> and emit appropriate assembly code for any of them. That's what you
> need a configure script for.

Please tell me you don't seriously think it takes 18,000 lines of script
just to do that.

And if you don't, what are the other 17,000 lines doing, or don't you care?

(Actually I've seen a 20,000-line configure just trying to compile the
GMP library.)

>> (BTW I've gone down the route of installing MSYS etc many times, something
>> always went wrong. Why is it that people can't appreciate that complicated
>> thing are more likely to go wrong?)
>
> Why would bash not exist? Or do you mean that it exists, but isn't installed?

Because it (and all the utilities the script intends to use) exist on a
different OS?

>> I've put some comments here: https://pastebin.XXX/hSCjNsA2 as it's getting

>> way off-topic.
>
> Not interested in clicking on a pastebin with all its ads.

OK, 30 minutes of effort down the drain. (I've deleted the link now.)

If you don't want to believe that my compilers are capable of competing
with gcc, then don't. It doesn't change the facts.

gcc can do astonishing things with small, tight, benchmarks or when
directed at specific bottlenecks. But its optimisations have less
dramatic effects on some real applications such as mine.

(And take another look at the figures I posted yesterday comparing gcc
with Tiny C, with an extra line:

Tiny C 6.7 seconds
gcc -O3 3.6 seconds
My comp: 1.2 seconds with ASM help

Yes, that last figure is three times the speed of gcc. If you won't
believe 30% slower, then I doubt you will 200% faster! Yes, one module
is ASM, but the others are HLL, and no C compiler is involved.)

--
bartc

[Chris Angelico]

On Thu, May 18, 2017 at 6:52 AM, bartc <b...@freeuk.com> wrote:
>> determines whether your system has the required
>> compilers and libraries, figures out what source files should be
>> compiled, and calculates the order in which to build the source files.
>
>
> Yes! That's what I need!
>
> But either this information is highly classified, or nobody here actually
> knows how to extract it from the output of configure or the makefile or
> wherever it ends up.
>

Have you looked in config.log?

configure:3651: checking for gcc
configure:3667: found /usr/bin/gcc
configure:3678: result: gcc

Or the less verbose form that you see on the console while it's
actually running?

checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking how to run the C preprocessor... gcc -E

And of course, there's all the headers that Python looks for:

checking spawn.h usability... yes
checking spawn.h presence... yes
checking for spawn.h... yes
checking util.h usability... no
checking util.h presence... no
checking for util.h... no
checking alloca.h usability... yes
checking alloca.h presence... yes
checking for alloca.h... yes
checking endian.h usability... yes
checking endian.h presence... yes
checking for endian.h... yes

and the standard library functions:

checking for kill... yes
checking for killpg... yes
checking for lchmod... no
checking for lchown... yes
checking for linkat... yes
checking for lstat... yes
checking for lutimes... yes
checking for mmap... yes

and some architecture-specific things:

checking whether C doubles are little-endian IEEE 754 binary64... yes
checking whether C doubles are big-endian IEEE 754 binary64... no
checking whether C doubles are ARM mixed-endian IEEE 754 binary64... no
checking whether we can use gcc inline assembler to get and set x87
control word... yes
checking whether we can use gcc inline assembler to get and set
mc68881 fpcr... no
checking for x87-style double rounding... no

For "highly classified" information, this is being very well hidden in
plain sight.

ChrisA

[Ian Kelly]

On Wed, May 17, 2017 at 2:52 PM, bartc <b...@freeuk.com> wrote:
>
> On 17/05/2017 21:17, Michael Torrie wrote:
>>
>> On 05/17/2017 01:32 PM, bartc wrote:
>>>
>>> Sometimes, if there's a problem. But usually the code is doing something
>>> sensible. The stuff in configure is complete gobbledygook (if anyone
>>> doesn't believe me, just have look).
>>
>>
>> Well trying to edit an executable in a disassembler would look like
>> gobligook as well.
>
>
> The content of configure is high-level human readable source code. But it
is meaningless.

configure is generated code. If you want to understand what it's doing then
you should really be looking at configure.ac. But that's a whole language
of its own.

>> What a wonderful simplification! But yes, you're essentially correct.
>> All this gobbligook figures out how to best configure the features and
>> options you require,
>
>
> Well, while we're here, why don't /you/ run configure and tell me what
the output is. Preferably in a form I can understand.
>
> Here's the input data:
>
> OS: Windows 7
> Processor: Intel 64-bit
>
> Um, that's it I guess. I only need enough info to build, say, python.exe;
just something to get to first base. (I don't know what the other binary
files are or where they live.)

The input data isn't just a description of your system. It's your actual
system, including the particulars of your exact compiler and libc version
as well as optional libraries you may have installed. Generally speaking,
the way configure checks for features is by writing a short program that
uses the feature and trying to compile it.

>> determines whether your system has the required
>> compilers and libraries, figures out what source files should be
>> compiled, and calculates the order in which to build the source files.
>
>
> Yes! That's what I need!
>
> But either this information is highly classified, or nobody here actually
knows how to extract it from the output of configure or the makefile or
wherever it ends up.

configure has two primary outputs: Makefile and pyconfig.h. The latter is
actually the more important one. You can probably figure out *what* to
build and in what order just by reading Makefile.pre.in which is
essentially a template for the actual Makefile. But you're not going to get
very far without the huge number of #defines in pyconfig.h which determine
things like "is pthread.h available on this system" or "how many bytes is
the double typle on this system".

> Or, more importantly, how this list of files could be determined manually.

You're funny. :-)

> Yes. The end-result of all this might be something like:
>
> gcc -c file1.c
> gcc -c file2.c
> ...
> gcc file1.o file2.o ... -opython.exe
>
> Someone please explain why I can't just do that

Because, as has already been stated, there's no way to make such a simple
process cross-platform.

[Michael Torrie]

On 05/17/2017 02:52 PM, bartc wrote:
> On 17/05/2017 21:17, Michael Torrie wrote:
>> On 05/17/2017 01:32 PM, bartc wrote:
>>> Sometimes, if there's a problem. But usually the code is doing something
>>> sensible. The stuff in configure is complete gobbledygook (if anyone
>>> doesn't believe me, just have look).
>>
>> Well trying to edit an executable in a disassembler would look like
>> gobligook as well.
>
> The content of configure is high-level human readable source code. But
> it is meaningless.

No, I assure you, it's compiled code. Happens to be readable to those
who know how to read it, but it was never written by a human.

> <snip>

> Yes. The end-result of all this might be something like:
>
> gcc -c file1.c
> gcc -c file2.c
> ...
> gcc file1.o file2.o ... -opython.exe
>
> Someone please explain why I can't just do that

You sure could if you were sure of the right order given dependencies
(.h files). But it would be tedious if you had thousands of source code
files. And what if you had a common header file depended on by hundreds
of other files? You'd have to either manually identify each affected
file and recompile it, or recompile the whole works again. For a large
project, or even a medium-sized one, that's just not sustainable. Sure
you could put more in a file, as you already do, and that works for
small projects such as your compilers and also sqlite. Wouldn't work at
all for the Linux kernel, of course.

That said, there are huge benefits to keeping your source files short,
modular, and as abstract as possible. And huge benefits to using a build
manager program like scons, qmake, or even Visual Studio solutions, even
to small programs.

I tend to do the same as you for small, one-off programs. I just compile
and link them with gcc manually. Since that fits your needs you can
keep doing it. Eventually you may encounter the limitations of that,
and then you can think about investigating a tool like scons.

[bartc]

On 17/05/2017 22:40, Chris Angelico wrote:
> On Thu, May 18, 2017 at 6:52 AM, bartc <b...@freeuk.com> wrote:
>>> determines whether your system has the required
>>> compilers and libraries, figures out what source files should be
>>> compiled, and calculates the order in which to build the source files.
>>
>>
>> Yes! That's what I need!
>>
>> But either this information is highly classified, or nobody here actually
>> knows how to extract it from the output of configure or the makefile or
>> wherever it ends up.
>>
>
> Have you looked in config.log?

Don't have to run it first?

> configure:3651: checking for gcc
> configure:3667: found /usr/bin/gcc
> configure:3678: result: gcc
>
> Or the less verbose form that you see on the console while it's
> actually running?

That sounds like the answer is yes!

OK, so maybe I can get an idea of what's involved if I run it on another
computer that has Linux. But I expect that the output will then be
slanted at that system rather than mine.

> and some architecture-specific things:
>
> checking whether C doubles are little-endian IEEE 754 binary64... yes
> checking whether C doubles are big-endian IEEE 754 binary64... no
> checking whether C doubles are ARM mixed-endian IEEE 754 binary64... no
> checking whether we can use gcc inline assembler to get and set x87
> control word... yes
> checking whether we can use gcc inline assembler to get and set
> mc68881 fpcr... no

It sounds more like CPython is written in assembly code rather than C.
Why don't the C language, C compilers and their optimisers take care of
these concerns?

(And what does the build system do with all this information, or is this
just routinely gathered anyway whether the application needs it or not?)

--
Bartc

[Christian Gollwitzer]

Am 16.05.17 um 09:53 schrieb Chris Angelico:
> On Tue, May 16, 2017 at 5:14 PM, Christian Gollwitzer <auri...@gmx.de> wrote:
>> More likely would be the option to ship a C compiler with Python written in
>> C. For C++ this is way too big, but a pure C compiler can be as small as
>> 1MB. tcc has a liberal license, supports many platforms and gives reasonable
>> (unoptimized) code.
>
> To do that, Python would itself have to be compiled with tcc, or else
> all memory de/allocation would have to be funneled through a
> Python-provided API. And that's going to kill performance, I suspect.

I don't understand this remark. Why would that be needed?
The C ABI is well defined on the usual systems, and tcc is compatible
with the respective compilers. You can link together objects from tcc,
gcc and icc on Linux /OSX or tcc, MSVC and gcc on Windows without problems.
tcc is *only* the compiler, not a library, so it will call out into the
C library that Python is using.

tcc even has a "JIT-mode" of operation (libtcc). For Tcl, there exists
an extension which compiles C code to memory and executes directly from
there. The same thing could be done for Python, too.

Christian