Note that you can't just get rid of the the devel directory, as that's
where the compiled files sit as well as the sources. There's a lot of
hard and symbolic links laying around. Having the source around is
important for debugging and tracebacks (not just for development).
> I imagine that the reason this is not done is so development is "at
> hand" to any user, and so there would be more chance of them
> contributing. I think this is true.
+1, which has been essential to getting Sage where it is.
> After all, most users, given a
> choice of a normal version and a developer version which is double the
> size, would probably download the former.
The binaries do have empty spkgs, but I find it interesting that in
this day and age people would find, say (optimistically) 200MB small
enough but 350MB too big. Size is a concern, but mostly comes into
play when deciding whether it's worth including an spkg or some
pre-computed data.
- Robert
> Is there any recommended procedere to make the resulting package
> smaller? I checked the package and e.g. found that dynamic libraries
> are not stripped.
I think you'll find that stripping (dynamic) libraries is counter-productive. But try it and let us know how it works :-}
> Also I found duplicate file entries in the directory tree. e.g.
> /devel/sage-main/build/lib.linux-i686-2.6 and
> /devel/sage-main/build/sage
> /devel/sage-main/sage
>
> contain lots of identical files. This are 100 MB+ directories, so no
> peanuts.
I think the issue is that, without the devel directory set up as it is, actual development won't work.
I suppose it's worth considering a "really binary release" that is of use only for those that don't intend to add to the Sage project.
Justin
--
Justin C. Walker, Curmudgeon-At-Large
Institute for the Enhancement of the Director's Income
--------
When LuteFisk is outlawed,
Only outlaws will have LuteFisk
--------
This is wrong, as you might discover if you delete that directory,
then try to use Sage. There is a symlink from
SAGE_ROOT/local/lib/site-packages/sage to
SAGE_ROOT/devel/sage/build/sage
The best solution to the problem we're discussing in the context of
SAGE_ROOT/devel/sage would be to change the Sage library to build all
.so's in place, so that the SAGE_ROOT/devel/sage/build would in fact
be redundant, and there would only ever be exactly one copy of each
file (or .so) in SAGE_ROOT/devel/sage. This is what happens with
"python setup.py develop" or "setup.py build_ext --inplace <list of
extension names>", when using setuptools instead of distutils. For
example, with the Sage Notebook and PSAGE
(http://code.google.com/p/purplesage/), which both use setuptools, we
do all of our work this way.
Another plus to this change is that if foo is a function and you type:
sage: foo??
you'll see the actual location of the file you should edit to modify
foo, rather than some highly misleading file in SAGE_ROOT/local (which
is a major source of confusion at first).
That said, it could be significant and painfully disruptive work for
somebody to change the core Sage library to work this way. I hope it
happens at some point. Confusion will be reduced overall, as will
disk space usage.
William
--
William Stein
Professor of Mathematics
University of Washington
http://wstein.org
Yes. It would be a nontrivial project. I hope somebody does it sometime!
>
>>This is what happens with
>> "python setup.py develop" or "setup.py build_ext --inplace <list of
>> extension names>", when using setuptools instead of distutils. For
>> example, with the Sage Notebook and PSAGE
>> (http://code.google.com/p/purplesage/), which both use setuptools, we
>> do all of our work this way.
>
> I guess you use a core of sage for your PSAGE, does it mean
> work could concentrate on the packages you have excluded from psage?
By "PSAGE" I meant http://code.google.com/p/purplesage/, which is a
separate Python package, which itself has nothing to do with anything
currently in Sage. It's just a new Python package that depends on
Sage.
>
>> Another plus to this change is that if foo is a function and you type:
>>
>> sage: foo??
>>
>> you'll see the actual location of the file you should edit to modify
>> foo, rather than some highly misleading file in SAGE_ROOT/local (which
>> is a major source of confusion at first).
>>
>> That said, it could be significant and painfully disruptive work for
>> somebody to change the core Sage library to work this way.
>
> Do you have a rough estimate how long it takes for someone with no
> prior knowledge of setuptools?
It depends entirely on how good that person is, and what they know
about Sage. If they are sufficiently smart, maybe 1 week? It's
impossible to know what issues might arise though until one try.
>> I hope it
>> happens at some point. Confusion will be reduced overall, as will
>> disk space usage.
> A Rough estimate of size reduction?
I estimate 189MB, i.e., the size of SAGE_ROOT/devel/sage/sage/.
This would of course mean no reasonable tracebacks, if the source
couldn't be found....
- Robert
No, you're wrong. The source would be found, and the tracebacks would
be excellent.
Observe the following:
deep:sage wstein$ pwd
/Users/wstein/sd27/sage-4.6.1.rc1/devel/sage
deep:sage wstein$ ls -lh build/sage/rings/arith.py
-rw-r--r-- 1 wstein staff 127K Jan 5 00:22 build/sage/rings/arith.py
deep:sage wstein$ ls -lh sage/rings/arith.py
-rw-r--r-- 1 wstein staff 127K Jan 5 00:22 sage/rings/arith.py
Notice that there are two distinct but identical files. This is a
waste of disk space. This can be fixed with absolutely no reduction in
functionality or capabilities of Sage, e.g., by switching to using
setuptools and building the so modules in place.
I think I was wrong in my space saving estimate (189MB) above. The
SAGE_ROOT/devel/sage directory in sage-4.6.1.rc1 is 768MB:
deep:sage wstein$ du -sch *
4.0K MANIFEST.in
4.0K PKG-INFO
4.0K README.txt
367M build
4.0K bundle
4.5M c_lib
207M doc
4.0K export
4.0K install
68K module_list.py
40K module_list.pyc
4.0K pull
189M sage
4.0K sage-push
40K setup.py
4.0K spkg-delauto
4.0K spkg-dist
4.0K spkg-install
768M total
------
For Sage to run, I think all that's *needed* is:
SAGE_ROOT/devel/sage/build/sage
since that is exactly the thing that
SAGE_ROOT/local/lib/python/site-packages/sage is linked to, so it is
all that Python sees at runtime. This directory is 128MB:
deep:sage wstein$ du -sch build/sage
128M build/sage
128M total
-----
So one could strip 640MB away from SAGE_ROOT/devel/sage/build/sage and
still have a Sage that runs. And this would have docstring
introspection even for Cython code, since the .pyx files are in
SAGE_ROOT/devel/sage/build/sage.
The autogenerated C/C++ files are not there, so development ("sage
-br") would be painfully slow the first time, as these files would all
get regenerated. The SAGE_ROOT/devel/sage/doc/ directory would also
be missing, which is used by the notebook *only* when browsing the
manuals (not for function_name? introspection).
I didn't figure out the exact savings from just switching to
setuptools, but I think it would be at least 240MB.
-- William
and then we just use setuptools.
Francois
I was talking about a completely stripped binary (i.e no source).
> Observe the following:
>
> deep:sage wstein$ pwd
> /Users/wstein/sd27/sage-4.6.1.rc1/devel/sage
> deep:sage wstein$ ls -lh build/sage/rings/arith.py
> -rw-r--r-- 1 wstein staff 127K Jan 5 00:22 build/sage/rings/arith.py
> deep:sage wstein$ ls -lh sage/rings/arith.py
> -rw-r--r-- 1 wstein staff 127K Jan 5 00:22 sage/rings/arith.py
>
> Notice that there are two distinct but identical files. This is a
> waste of disk space. This can be fixed with absolutely no reduction in
> functionality or capabilities of Sage, e.g., by switching to using
> setuptools and building the so modules in place.
I agree 100% here.
I don't even think we need setuptools--just build extensions inplace
(there's an option for this) and put devel/sage in the python path
(this was what I did for enabling concurrent execution of distinct
branches, though that still needs a bit of work
http://trac.sagemath.org/sage_trac/ticket/9967). Then one wouldn't
even need to do a "sage -b" to work on (non-Cython) code.
- Robert
contain lots of identical files....
I'd be surprised if squashfs doesn't already do file/block deduping,
which would erase all potential gains. It'd also probably be a risky
think to do on an actively developed Sage build, but perhaps OK for a
non-development binary.
- Robert
I think that's definitely something we should look into! Could you
post a bash script? What somebody also has to check is, if it is still
possible to do development. i.e. apply some patches, changing cython
files, sage -b and then checking it again.
H
+100 There would have to be *significant* savings before I would want
to make it so people would have to download a new version of Sage to
create patches and otherwise develop Sage. Also, would this make
editing files and not doing sage -b have effects?
- Robert
Is this up on the wiki somewhere, and maybe in a readme on the live CD?
It looks like some great information, and it would be sad if it is
only buried in a post on sage-devel.
Thanks for all of your work on this!
Jason
Whoever implements this will be my hero of the week! PLEASE!!!
I like the other consequences of such a change, but that one would be
such a time saver in my everyday workflow! Beside, it would simplify
the explanations to newcomers, also a good thing.
Cheers,
Nicolas
--
Nicolas M. Thi�ry "Isil" <nth...@users.sf.net>
http://Nicolas.Thiery.name/
I would also really like to get that ticket finished...it would make
things like the patchbot much easier to work with. But beware, the
ticket is complex and I got stuck trying to work out exactly why certain
bits couldn't find other bits and so on. It's pretty tricky...but I
would like to see it working.
Dan
--
--- Dan Drake
----- http://mathsci.kaist.ac.kr/~drake
-------