Consider refactor of testing / benchmarking?

[Matthew Brett]

Hi,

I've been running into trouble installing sympy with easy_install and
pip on python 2.7 and Python 3.3.

The fact that easy_install / pip is unlikely to work means that sympy
is a considerably heavier burden for our (nipy.org/nipy) users to
install, and I'd very much like to help fix that.

This led me to start editing the sympy setup.py to do the standard
thing of running 2to3 during setu.py install : (e.g numpy, scipy,
matplotlib, our projects).

I immediately found why you haven't done this: you depend on running

python setup.py run_tests

and

python setup.py run_benchkmarks

in the source tree. So, there's no way of running these for python 2
and python 3 in the same source tree (unless you go for python 3
compatibility in source - a bit much).

So - I'm writing to ask if you'd consider looking at a refactor that
would change these calls above to:

python setup.py install --user (or whatever)
cd /somewhere
python -c 'import sympy; sympy.test()'

or

python setup.py install --user (or whatever)
cd /somewhere
python -c 'import sympy; sympy.bench()'

This has three advantages:

1) You can run 2to3 in your install. This is generally very quick
after the first pass. E.g for our project nipy:

time python setup.py install
->
real 2m7.466s
user 1m51.458s
sys 0m6.254s

time python setup.py install # a second time, only processing modified files
->
real 0m3.428s
user 0m2.614s
sys 0m0.712s

2) You can use the workflow above to develop and test in python3 :

python setup.py install # some longish time say 2 minutes
<edit>
python setup.py install # A few seconds
<test>
etc

2) You can have compiled code in your project without breaking the
tests / benchmarks

3) easy_install / pip will work :)

So - would you consider this change? If so, I will write a pull
request for it.

Best,

Matthew

[Aaron Meurer]

On Nov 17, 2012, at 7:39 PM, Matthew Brett <matthe...@gmail.com> wrote:

> Hi,
>
> I've been running into trouble installing sympy with easy_install and
> pip on python 2.7 and Python 3.3.

What issues with pip are you having? I thought we fixed pip to do the
right thing.

>
> The fact that easy_install / pip is unlikely to work means that sympy
> is a considerably heavier burden for our (nipy.org/nipy) users to
> install, and I'd very much like to help fix that.
>
> This led me to start editing the sympy setup.py to do the standard
> thing of running 2to3 during setu.py install : (e.g numpy, scipy,
> matplotlib, our projects).
>
> I immediately found why you haven't done this: you depend on running
>
> python setup.py run_tests
>
> and
>
> python setup.py run_benchkmarks
>
> in the source tree. So, there's no way of running these for python 2
> and python 3 in the same source tree (unless you go for python 3
> compatibility in source - a bit much).
>
> So - I'm writing to ask if you'd consider looking at a refactor that
> would change these calls above to:
>
> python setup.py install --user (or whatever)
> cd /somewhere
> python -c 'import sympy; sympy.test()'

The problem is that setup.py test runs tests that aren't included in
an install, namely the Sphinx doctests.

>
> or
>
> python setup.py install --user (or whatever)
> cd /somewhere
> python -c 'import sympy; sympy.bench()'

Benchmarks from setup.py aren't very important. We can remove them if
they are a hinderance.

Aaron Meurer

> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To post to this group, send email to sy...@googlegroups.com.
> To unsubscribe from this group, send email to sympy+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/sympy?hl=en.
>

[Matthew Brett]

Hi,

On Sat, Nov 17, 2012 at 6:51 PM, Aaron Meurer <asme...@gmail.com> wrote:
> On Nov 17, 2012, at 7:39 PM, Matthew Brett <matthe...@gmail.com> wrote:
>
>> Hi,
>>
>> I've been running into trouble installing sympy with easy_install and
>> pip on python 2.7 and Python 3.3.
>
> What issues with pip are you having? I thought we fixed pip to do the
> right thing.

Ah - I have a strong memory that I had issues with pip, but not now I
test it on this machine. Broken for easy_install (via distribute):

mb312@mikesilver ~]$ mkvirtualenv py27 --no-site-packages
New python executable in py27/bin/python
Installing setuptools.............done.
Installing pip...............done.
[mb312@mikesilver ~]$ workon py27
(py27)[mb312@mikesilver ~]$ easy_install sympy
Searching for sympy
Reading http://pypi.python.org/simple/sympy/
Reading http://code.google.com/p/sympy
Reading http://sympy.org
Reading http://sympy.org/
Reading http://code.google.com/p/sympy/downloads/detail?name=sympy-0.7.0.tar.gz
Best match: sympy 0.7.2-py3.3
Downloading http://pypi.python.org/packages/source/s/sympy/sympy-0.7.2-py3.3.tar.gz#md5=4645f4bd3c50dac7486ccfaedbb29057
Processing sympy-0.7.2-py3.3.tar.gz
Running sympy-0.7.2/setup.py -q bdist_egg --dist-dir
/var/folders/YB/YBjJ+eaQGb0EZfa9PJfcSk+++Tg/-Tmp-/easy_install-n5AKga/sympy-0.7.2/egg-dist-tmp-WFXCnm
Traceback (most recent call last):
File "/Users/mb312/.virtualenvs/py27/bin/easy_install", line 8, in <module>
load_entry_point('setuptools==0.6c11', 'console_scripts', 'easy_install')()
File "/Users/mb312/.virtualenvs/py27/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py",
line 1712, in main
File "/Users/mb312/.virtualenvs/py27/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py",
line 1700, in with_ei_usage
File "/Users/mb312/.virtualenvs/py27/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py",
line 1716, in <lambda>
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py",
line 152, in setup
dist.run_commands()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py",
line 953, in run_commands
self.run_command(cmd)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py",
line 972, in run_command
cmd_obj.run()
File "/Users/mb312/.virtualenvs/py27/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py",
line 211, in run
File "/Users/mb312/.virtualenvs/py27/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py",
line 446, in easy_install
File "/Users/mb312/.virtualenvs/py27/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py",
line 476, in install_item
File "/Users/mb312/.virtualenvs/py27/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py",
line 655, in install_eggs
File "/Users/mb312/.virtualenvs/py27/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py",
line 930, in build_and_install
File "/Users/mb312/.virtualenvs/py27/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py",
line 919, in run_setup
File "/Users/mb312/.virtualenvs/py27/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/sandbox.py",
line 62, in run_setup
File "/Users/mb312/.virtualenvs/py27/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/sandbox.py",
line 105, in run
File "/Users/mb312/.virtualenvs/py27/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/sandbox.py",
line 64, in <lambda>
File "setup.py", line 36, in <module>
File "/var/folders/YB/YBjJ+eaQGb0EZfa9PJfcSk+++Tg/-Tmp-/easy_install-n5AKga/sympy-0.7.2/sympy/__init__.py",
line 27, in <module>
ImportError: It appears 2to3 has been run on the codebase. Use Python
3 or get the original source code.

We use easy_install much more than pip because pip will not install
binaries, and so is more or less useless for compiled packages.

>> The fact that easy_install / pip is unlikely to work means that sympy
>> is a considerably heavier burden for our (nipy.org/nipy) users to
>> install, and I'd very much like to help fix that.
>>
>> This led me to start editing the sympy setup.py to do the standard
>> thing of running 2to3 during setu.py install : (e.g numpy, scipy,
>> matplotlib, our projects).
>>
>> I immediately found why you haven't done this: you depend on running
>>
>> python setup.py run_tests
>>
>> and
>>
>> python setup.py run_benchkmarks
>>
>> in the source tree. So, there's no way of running these for python 2
>> and python 3 in the same source tree (unless you go for python 3
>> compatibility in source - a bit much).
>>
>> So - I'm writing to ask if you'd consider looking at a refactor that
>> would change these calls above to:
>>
>> python setup.py install --user (or whatever)
>> cd /somewhere
>> python -c 'import sympy; sympy.test()'
>
> The problem is that setup.py test runs tests that aren't included in
> an install, namely the Sphinx doctests.

Would it be an option to run these separately? For example with

python setup.py run_doctests

? Or maybe with

python setup.py run_tests

where ``run_tests`` does something like:

mkdir tmp
cd tmp

python -c 'import sympy; sympy.test()'

cd ..
cd doc
make.py doctest

- if you see what I mean. This would then fail unless there is an
installation somewhere, perhaps with a nice message to run ``python
setup.py install`` first.

I'm sorry for my ignorance - but how have y'all been doing the edit /
debug cycle in python 3?

Is it a concern that you can't use compiled code this way - or are you
committed to a compile free codebase for the long haul?

Cheers,

Matthew

[Aaron Meurer]

On Sat, Nov 17, 2012 at 8:46 PM, Matthew Brett <matthe...@gmail.com> wrote:

Hi,

On Sat, Nov 17, 2012 at 6:51 PM, Aaron Meurer <asme...@gmail.com> wrote:
> On Nov 17, 2012, at 7:39 PM, Matthew Brett <matthe...@gmail.com> wrote:
>
>> Hi,
>>
>> I've been running into trouble installing sympy with easy_install and
>> pip on python 2.7 and Python 3.3.
>
> What issues with pip are you having?  I thought we fixed pip to do the
> right thing.

Ah - I have a strong memory that I had issues with pip, but not now I
test it on this machine.  Broken for easy_install (via distribute):

That's because pip *was* broken, but we figured out how to fix it.  The trick was adding -py3.2 and so on to the tarball names so that it was sure to download the right one.

ARGH! I did not know this.  It seems that the same solution for pip is the problem for easy_install.  It thinks that sympy-0.7.2-py3.3 is the latest version.  

Any thoughts on how this can be fixed?  I've said this before, but it deserves reiterating: I as a package maintainer have surprisingly little control over what pip or easy_install install.  The best I can do is to manipulate what links are on PyPI and to use naming trickery to make it install the "latest" version.  

I created https://code.google.com/p/sympy/issues/detail?id=3511 for this.

  

We use easy_install much more than pip because pip will not install
binaries, and so is more or less useless for compiled packages.

My best suggestion for this is to use pip for SymPy and easy_install for the rest.  Or you could just pass the url manually to eas_install. If you can figure out how to name the tarballs so that pip *and* easy_install do the right thing, I'd love to hear it.  But I'm very doubtful that a solution exists.  The Python packaging ecosystem is irrevocably broken as far as I'm concerned.

setup.py test just runs all combinations of tests, which already have their individual scripts. I think right now it's equivalent to something like

./bin/test

./bin/doctest

<whatever the sage test script is>

./examples/all.py -q

Actually, it just runs sympy.utilities.runtests.run_all_tests(), which is itself a wrapper to the above tests.  So there is actually no need for anything in the setup.py script itself to do tests.  It's just a nice shortcut for us.

- if you see what I mean.  This would then fail unless there is an
installation somewhere, perhaps with a nice message to run ``python
setup.py install`` first.

I'm sorry for my ignorance - but how have y'all been doing the edit /
debug cycle in python 3?

I never use virtualenv to test SymPy, unless I am testing installation.  I just run ./bin/use2to3 and cd py3k-sympy and work locally.  I guess it's easy for us because we don't have any dependencies, especially dependencies that don't work unless installed.  

 

Is it a concern that you can't use compiled code this way - or are you
committed to a compile free codebase for the long haul?

I don't see us using compiled code, at least not any time soon.  

Aaron Meurer

 

Cheers,

[Matthew Brett]

Hi,

How about having a sympy3 pypi project?

>> We use easy_install much more than pip because pip will not install
>> binaries, and so is more or less useless for compiled packages.
>
>
> My best suggestion for this is to use pip for SymPy and easy_install for the
> rest. Or you could just pass the url manually to eas_install. If you can
> figure out how to name the tarballs so that pip *and* easy_install do the
> right thing, I'd love to hear it. But I'm very doubtful that a solution
> exists. The Python packaging ecosystem is irrevocably broken as far as I'm
> concerned.

Tell it brother :) And yes of course it throws the burden for the
workarounds onto each package maintainer, which is tiring and
wasteful. But nevertheless, it can be made to work, by doing 2to3 in
the setup.py.

RIght - I think I'm asking whether you'd consider moving from:

python setup.py run_tests

to

python setup.py install
python setup.py run_tests

in order to make it possible to processing the source tree (for
compiled code or 2to3) before running?

>> - if you see what I mean. This would then fail unless there is an
>> installation somewhere, perhaps with a nice message to run ``python
>> setup.py install`` first.
>>
>> I'm sorry for my ignorance - but how have y'all been doing the edit /
>> debug cycle in python 3?
>
>
> I never use virtualenv to test SymPy, unless I am testing installation. I
> just run ./bin/use2to3 and cd py3k-sympy and work locally. I guess it's
> easy for us because we don't have any dependencies, especially dependencies
> that don't work unless installed.

I probably wasn't clear; I was thinking that it might be tough to have to do:

./bin/use2to3
cd py3k-sympy
while not Finished:
<edit>
<test>
and then have to backport the changes in py3k-sympy to the sympy tree
- but I probably didn't get the workflow.

Best,

Matthew

[Aaron Meurer]

I hadn't thought of that.  If we did this, then it would never do the right thing in Python 3, i.e., the users would have to know to use "pip install sympy3" instead of "pip install sympy" in Python 3.  It might also complicated stuff for people whose travis.ini (or whatever) would otherwise just have a single pip install line.

Am I misunderstanding what you're asking for?  Would you run the tests against the installed version or the local version?

 

>> - if you see what I mean.  This would then fail unless there is an
>> installation somewhere, perhaps with a nice message to run ``python
>> setup.py install`` first.
>>
>> I'm sorry for my ignorance - but how have y'all been doing the edit /
>> debug cycle in python 3?
>
>
> I never use virtualenv to test SymPy, unless I am testing installation.  I
> just run ./bin/use2to3 and cd py3k-sympy and work locally.  I guess it's
> easy for us because we don't have any dependencies, especially dependencies
> that don't work unless installed.

I probably wasn't clear; I was thinking that it might be tough to have to do:

./bin/use2to3
cd py3k-sympy
while not Finished:
   <edit>
   <test>
and then have to backport the changes in py3k-sympy to the sympy tree
- but I probably didn't get the workflow.

Oh, we never develop against the Python 3 code (or at least I don't). I just make all changes against Python 2 and then 99% of the time 2to3 does the right thing and it works in Python 3 as well, and for the remaining 1%, I find out later and fix it using whatever workaround. I guess the take away is that I never personally use Python 3, except for testing to see if it work.  My day-to-day workflow is still Python 2.  I think I wrote one script for personal use in Python 3.  Maybe if some of the other SymPy developers use Python 3 more they do things differently.

Aaron Meurer

 

Best,

[Matthew Brett]

Hi,

Maybe it would be OK if there was a suitable error message from an
attempted pip install sympy in python 3? Sorry - I'm not sure that's
the best solution, just throwing it out there.

I agree it might be more complicated to sort out in travis, but people
in the position of writing a travis file for multiple pythons are
likely to be up to that task I would have thought.

Yes, the point would be to run the tests against the version on the
path, whatever that was, not the local version. So attempting to run
the tests without an install of some sort (including 'develop') would
get a message on the lines of

"Please install sympy on the python path before running tests; see
http:/blah/blah"

I guess at some point you will have to switch to a workflow that's
easy for developers on Python 3? I just wondering if doing that now,
along with fixing easy_install, might be worth the pain of losing the
ability to run the tests from the local tree.

See you,

Matthew

[Aaron Meurer]

There is, though admittedly it could be better:

$python3 setup.py # python 2 setup.py

Traceback (most recent call last):

  File "setup.py", line 36, in <module>

    import sympy

  File "/Users/aaronmeurer/Documents/Python/sympy/sympy/sympy/__init__.py", line 35, in <module>

    raise ImportError("This is the Python 2 version of SymPy. To use SymPy "

ImportError: This is the Python 2 version of SymPy. To use SymPy with Python 3, please obtain a Python 3 version from http://sympy.org, or use the bin/use2to3 script if you are using the git version.

$python2 setup.py # python2 setup.py

Traceback (most recent call last):

  File "setup.py", line 36, in <module>

    import sympy

  File "/Users/aaronmeurer/Documents/Python/sympy/sympy/py3k-sympy/sympy/__init__.py", line 27, in <module>

    raise ImportError("It appears 2to3 has been run on the codebase. Use "

ImportError: It appears 2to3 has been run on the codebase. Use Python 3 or get the original source code.

In particular, I don't think users care about use2to3, and it's just confusing.

OK.  Well, like I said, this won't work because we test things that aren't installed, namely, the Sphinx documentation doctests.  Actually, the examples are not installed (but they are included in the tarball).  Also, it seems like an unnecessary restriction to make people use virtualenv to test SymPy when it's not actually necessary.

I don't see people using Python 3 over Python 2 any time soon.  I guess if that ever happens, we can figure it out from there.  For the most part, we have to worry about the lowest common denominator, i.e., currently Python 2.5, because that's where the fewest Python features are supported.  

Aaron Meurer

See you,

[Matthew Brett]

Hi,

OK. I think I'm not seeing why it won't work, because I'm suggesting
`python setup.py run_tests` from the source tree, the only difference
being there would have to be some sort of install of the 'sympy'
library code first. It might be a virtualenv or it might be 'python
setup.py develop' for python 2, or it might be 'python setup.py
install --user' or 'python setup.py install
--prefix=/somewhere/on/pythonpath.

In my experience so far, most people who are up to checking out the
git repo and running tests are well capable of doing one of these.
However, the user who gets an error message on easy_install may not
find it easy to solve.

OK - well - I think I'm getting the message :) The offer is open of
a pull request if you would like to consider the alternative.

Cheers,

Matthew

[Matthew Brett]

Hi,

Sorry - I do see what you mean now - you mean that the sphinx doctests
and examples would not get 2to3 processed via a simple 'install' so
can't be tested in python 3. We've gone to requiring python 2.6 for
our examples and making them compatible with python 3 - not a big
problem, but a problem. I guess you could also use a numpy-like py3k
setup.py trick to process only the changed examples, sphinx docs into
a py3k-sympy directory and running tests from there. As I say, I'm
happy to try and work out how to make that happen if you're
interested.

Cheers,

Matthew

[Ondřej Čertík]

Matthew,

On Sat, Nov 17, 2012 at 5:45 PM, Matthew Brett <matthe...@gmail.com> wrote:
> Hi,
>
> I've been running into trouble installing sympy with easy_install and
> pip on python 2.7 and Python 3.3.
>
> The fact that easy_install / pip is unlikely to work means that sympy
> is a considerably heavier burden for our (nipy.org/nipy) users to
> install, and I'd very much like to help fix that.

Sorry for such problems, I would like to fix that too. SymPy should
behave just like any other
Python package, and if it doesn't, we have to fix it. So I am ok
with any reasonable solution that fixes it. Aaron is now the guru
on pip. ;)

Ondrej

[Aaron Meurer]

We do process the Sphinx docstrings with 2to3.  But they are not installed, so if you install sympy and then run "import sympy; sympy.doctest()", it doesn't include those doctests.  This is actually an issue also for us with Travis because it is currently setup to install SymPy and then test it, so it doesn't test Sphinx.

Aaron Meurer

 

Cheers,

[Aaron Meurer]

Actually, I still get the feeling that I am misunderstanding what you want to do, so if it's not too much work, go ahead and submit a pull request, and then I can see exactly what you mean.

Aaron Meurer

Cheers,

[Aaron Meurer]

Actually, I don't know hardly anything about pip.  All I know is that unless you are *very* careful how you name your tarballs, it will gladly install the wrong thing because of its agressive version searching algorithm that is almost impossible for package maintainers to control.  I think its done this for SymPy on at least three separate occasions now, with three separate issues.

Aaron Meurer

 

Ondrej

[Matthew Brett]

Hi,

This is just to explain my own developing understanding of the problem
to see if it helps.

I think the current situation is this:

In python2
* ``python setup.py run_tests`` runs tests from source tree
* tests include sphinx docstring tests, and examples and sage tests
* python3 tarball for e.g. pypi generated with `python setup.py sdist`

In python3:
* ./bin/use2to3 recreates whole source tree in py3-sympy subdirectory,
including sphinx doctests and examples
* running tests needs: 'cd py3k-sympy ; python setup.py run_tests
* python 3 tarball generated from `py3k-sympy` tree with `cd
py3k-sympy; python setup.py install`

Benefits of current setup:
* Users can run all of sympy tests from local tree without doing an install
* Complete porting of code / docs to python3

Problems with the current setup:
* Need different source (sdist) tarballs for python2 and python3.
This seems to make it impossible for both of easy_install and pip to
work by default on python2 and python3.
* Python3 not attractive as development environment because of need to
backport changes from py3k-sympy tree

Is this a reasonable summary?

See you,

Matthew

[Aaron Meurer]

setup.py test (not run_tests), but yeah.

 

* tests include sphinx docstring tests, and examples and sage tests

And potentially anything else that we want to test in the future. It's the "catch all" test command.  But it's only a shortcut.  It has nothing to do with setup.py build or setup.py install.

 

* python3 tarball for e.g. pypi generated with `python setup.py sdist`

Python 2 tarball you mean. 

 

In python3:
* ./bin/use2to3 recreates whole source tree in py3-sympy subdirectory,
including sphinx doctests and examples

Yes.

 

* running tests needs: 'cd py3k-sympy ; python setup.py run_tests
* python 3 tarball generated from `py3k-sympy` tree with `cd
py3k-sympy; python setup.py install`

bdist, but yeah.

 

Benefits of current setup:
* Users can run all of sympy tests from local tree without doing an install
* Complete porting of code / docs to python3

Problems with the current setup:
* Need different source (sdist) tarballs for python2 and python3.
This seems to make it impossible for both of easy_install and pip to
work by default on python2 and python3.

I haven't completely given up that there is no solution here.

 

* Python3 not attractive as development environment because of need to
backport changes from py3k-sympy tree

I don't know of anyone who wants to develop SymPy in Python 3.  I'm sure we could come up with ways to do it if it came up.  Maybe 3to2.  And some kind of mechanism to see what has changed (like a temporary git repo or something). 

Anyway, the only real way around this is to use a single code base for both Python 2 and Python 3, which I am opposed to. 

Is this a reasonable summary?

I still don't understand what you want to change in setup.py.  If it's not hard (and let me know if it is), just create a PR with it, and I'll see exactly what you mean.  When it comes to understand something precisely, nothing beats reading the code.

Aaron Meurer

See you,

[Matthew Brett]

Hi,

I would certainly prefer to avoid writing some complex code to an
outcome that y'all don't want, and that's why I'm trying to get the
range here. I'm sorry, I realize it's only getting there slowly.

I'm trying to work out a way of using the same source tree for python2
and python3. That is, all of

python setup.py test
python setup.py install
python setup.py sdist

work for both python2 and python3, from the sympy source directory,
and where the output from `python setup.py sdist` is a tarball that
pypi can install into python2 and python3.

I can solve being able to run tests and doctests on the library code
via python3 by insisting on an install of some sort (calling 2to3)
before running the test code.
I can imagine some solution to running the examples and the sphinx
doctests in python3 by essentially doing what you are doing now with
the examples and docs, from setup.py, into a temporary directory.

However, solving the 'sdist' problem is harder. I can think of a few solutions:

1) having py2/examples py3/examples py2/docs py3/docs directories in
the sdist tarball
2) for docs - running 2to3 within the `make html` or `make latexpdf`
build steps, but leaving the source as is for the tarball
3) for the examples - installing them via 2to3 from python setup.py
install into - say <site-packages>/sympy/examples where the
sympy/examples directory has no __init__.py

But I'm not sure whether any of these are acceptable to you, even if
they are doable.

Sorry if I am not being clear,

Cheers,

Matthew

[Aaron Meurer]

So maybe just modify those commands to run use2to3 first.  That level of automation would actually be nice (see https://code.google.com/p/sympy/issues/detail?id=3445).  But there is one more caveat that might be an issue for what you want: use2to3 currently uses git ls-files to determine what files to include.  This helps us avoid junk accumulating in py3k-sympy. 

However, solving the 'sdist' problem is harder.  I can think of a few solutions:

1) having py2/examples py3/examples py2/docs py3/docs directories in
the sdist tarball
2) for docs - running 2to3 within the `make html` or `make latexpdf`
build steps, but leaving the source as is for the tarball

2to3 has to be run on the source, because we doctest the source.  But probably something along the lines of what you are saying could be worked up.

Another potential issue: shipping the docs in the same tarball as the source would double the size of the tarball.  Including the pdf docs as well would triple the size (see the file sizes at https://code.google.com/p/sympy/downloads/list).

 

3) for the examples - installing them via 2to3 from python setup.py
install into - say <site-packages>/sympy/examples where the
sympy/examples directory has no __init__.py

The examples will eventually be changed from .py files to IPython notebooks (https://code.google.com/p/sympy/issues/detail?id=2940).  So that might not make as much sense at that point.  But they could probably be installed *somewhere* if need be.

 

Aaron Meurer

But I'm not sure whether any of these are acceptable to you, even if
they are doable.

Sorry if I am not being clear,

Cheers,

Matthew