Thoughts about getting rid of the extcode repo or resetting it.

[mabshoff]

Hello folks,

the extcode repo in Sage is somewhat of a dumping ground what doesn't
really fit anywhere else. Below is the disk use of all directories
that aren't more or less empty:

44K dist
36K gap
24K images
1.2M javascript
228K magma
52K maxima
3.0M notebook
1.9M pari
400K pickle_jar
204K sagebuild
16K singular
7.2M total

The 3.0 MB notebook is at least size-wise 95% javascript, too. We
(Jason Grout is doing all the hard work) are in the process of moving
all the javascript packages we use (jsmath, jquery, ...) to spkgs so
that the repo will be much smaller in the future while the .hg repo
information itself will grow by another couple MB. In the end the
compressed extcode.spkg will be just as large as before the removal.
And shipping such an amount of basically 3rd party code we checked in
and then removed seems like a waste of resources. But what to do?

(a) nuke the extrepo and move things into other places. The
pickle_jar could go into the main Sage repo, the pari scripts into the
pari.spkg, the images and the notebook code could also be part of the
main repo. That leaves certain code for Maxima, Singular and Magma in
search of a place to hang out.

(b) reset the extrepo. This is the less invasive option, but would
destroy the version history of some of the files. One option here is
to check in files in the name of the previous main author, but it
seems like a bad move.

Either way we should do something about the extocde repo. Since the
doc repo will die in 3.3. it seems like an excellent opportunity to
finish some long needed cleanup. I could certainly imagine some
combination of (a) and (b), i.e. the pari data seems to be large
untouched in a while and the pari.spkg would seem like a canonical
place for it. Putting those pari scripts and data under revision
control also seems rather pointless.

Thoughts?

Cheers,

Michael

[David Joyner]

I think I can rewrite the gap code in Sage now. However, it would be a
rather large
patch (read "patch bomb") though.I could also just delete it and the
method in the class which uses it.

[mabshoff]

On Dec 5, 4:49 am, "David Joyner" <wdjoy...@gmail.com> wrote:

Hi David,

> I think I can rewrite the gap code in Sage now. However, it would be a
> rather large
> patch (read "patch bomb") though.I could also just delete it and the
> method in the class which uses it.

What does the code do and why was it included in Sage?

Cheers,

Michael

[David Joyner]

It explicitly decomposes certain G-modules closely related to
Riemann-Roch spaces (and cohomology in special cases)
on modular and Hurwitz curves, where G is the automorphism group
of the curve, using character computations.

It was included because I was using it in several papers
and wanted to use Sage as much as possible. It the time
those character computations could not be done in Sage but
I think the recent patch of Franco on class functions
will allow this to be ported over now.


>
> Cheers,
>
> Michael
> >
>

[William Stein]

I am very opposed to moving any of the code in other languages into
spkg's. I think spkg's should as close as possible approximate the
upstream originals. This is very important for the Debian and other
packaging efforts, and is conceptually much better. For example, Nick
and I were working on the Sage/Magma interface yesterday, and had to
write lots of code that also goes in the data/extcode/magma directory.
Exactly the same thing could have happened with Gap, and it would
have sucked for half our patches to be patches against the Gap spkg.

However, I don't like the current situation either. I'm tempted to
actually propose that we move everything we need that is in extcode to
the main Sage library repo, just like Mike Hansen is moving the docs
of Sage to that repository. Then we delete the extcode spkg and
repository completely.
We would thus have a new subdirectory:

SAGE_ROOT/devel/sage/extcode

with subdirectories for various languages. A big plus to this is
that, e.g., the work Nick and I was doing would involve a single patch
against the main sage library instead of one patch for extcode and one
for the main library.

The main reason the repos are broken up now is just historical and
because our previous revision control system (darcs) seemed to not
scale very well. Mercurial scales fine, and seems to have no problem
at all dealing with the entire massive Sage history, so I think adding
the Sphinx docs and extcode to the main library would be a good move.

So that's my counter-proposal.

-- Wiliam

[mabshoff]

On Dec 5, 11:27 am, "William Stein" <wst...@gmail.com> wrote:

Hi,

> I am very opposed to moving any of the code in other languages into
> spkg's.

I didn't make myself clear on that point, but I have the same
intention as you do: I would claim that the pari data is special in
that regard since it consists on the the following scripts/data:

1.5M SEA
20K buzzard
108K cremona
84K dokchitser
160K simon

I think all of them are from some upstream source and we do rarely if
ever change those unlike say the Magma code which is written for Sage
specifically by us. Since I agree that sticking them into the
pari.spkg would cause trouble for packagers I would suggest we create
an pari_scripts.spkg and stick them in there. I see no point checking
those files into the main Sage devel repo since we don't actively
change them anyway and would only waste resources in tracking them.

I will comment more on each individual directory below.

>  I think spkg's should as close as possible approximate the
> upstream originals.  This is very important for the Debian and other
> packaging efforts, and is conceptually much better. For example, Nick
> and I were working on the Sage/Magma interface yesterday, and had to
> write lots of code that also goes in the data/extcode/magma directory.
>  Exactly the same thing could have happened with Gap, and it would
> have sucked for half our patches to be patches against the Gap spkg.

Absolutely.

> However, I don't like the current situation either.  I'm tempted to
> actually propose that we move everything we need that is in extcode to
> the main Sage library repo, just like Mike Hansen is moving the docs
> of Sage to that repository.  Then we delete the extcode spkg and
> repository completely.
> We would thus have a new subdirectory:
>
>           SAGE_ROOT/devel/sage/extcode
>
> with subdirectories for various languages.   A big plus to this is
> that, e.g., the work Nick and I was doing would involve a single patch
> against the main sage library instead of one patch for extcode and one
> for the main library.

Yes, I like that very much and am certainly in favor here.

> The main reason the repos are broken up now is just historical and
> because our previous revision control system (darcs) seemed to not
> scale very well.   Mercurial scales fine, and seems to have no problem
> at all dealing with the entire massive Sage history, so I think adding
> the Sphinx docs and extcode to the main library would be a good move.

I did not know that that was the reason for the existence of the doc
and ext repos, but that is good to know for future "obscure questions
about the history of Sage" quizzes at future Sage Days :)

> So that's my counter-proposal.
>
>  -- Wiliam

The following directories should be deleted - at least Tim Abbott
mentioned that the Debian people disliked empty directories. IIRC
William said they were an historical artifact, but maybe someone else
would come up with a reason to keep them. But if they are needed for
some interface we should just change those interfaces to use SAGE_TMP
or something along those lines.

0B MuPAD
0B QEPCAD
0B genus2reduction
0B gnuplot
0B kash
0B macaulay2
0B maple
0B mathematica
0B matlab
0B octave
0B sage
0B sobj

--------

Now on to the more interesting files and directories:

36K dist

Probably delete since that is the Debian code to package the extcode
repo.

--------

24K gap

Move to Sage Dev repo. David Joyner mentioned earlier he could now
implement that functionality in Sage, but I would suggest we move it
anyway to not hold up the move.

--------

20K images

Move to Sage Dev repo

--------

1.1M javascript

Move to independent spkgs - work in progress

--------

200K magma

Move to Sage Dev repo

--------

40K maxima

Move to Sage Dev repo

--------

4.0K mirror:

I am sure the following can be safely deleted: :)

cd $HOME/sage/web/dist/src/extcode-darcs/
darcs changes > changelog_darcs.txt
echo "uploading"
rsync -axLH --rsh=ssh --delete -r -v * modular:/home/was/www/sage/
dist/src/extcode-darcs

--------

4.0K mwrank

this contains the file primes with says: "19047851" I assume this
constant can be stuck either somewhere else or hardcoded in some
place.

--------

2.8M notebook

Mostly javascript which is being moved to spkgs, the other data should
go into the Sage Dev repo. Since the templates are over there anyway
it seems better to have one place for that data than two. It should be
audited which data are actually still in use so we can nuke the rest.

--------

1.8M pari

See above: I would like those to get moved to a pari_scripts.spkg

--------

392K pickle_jar

Move to Sage Dev repo

--------

192K sagebuild

We might have to hang on for that one for a while until -tp is
completely independent of pbuild, so move to Sage Dev repo

--------

8.0K singular

Move to Sage Dev repo
--------

The following four: Delete - no longer needed

4.0K spkg-debian
4.0K spkg-dist
4.0K spkg-install
4.0K sage-push

[John Cremona]

Two cents from me:

(1) the gp scripts called "cremona" are largely, but not entirely,
redundant now that what they do is either implemented directly in Sage
(e.g. group structure for elliptic curvesw over prime fields), never
wrapped anyway (Weil pairings for ditto), or better implemented
elsewhere (e.g. analyttic rank stuff for elliptic curves over Q).

If it would help I could go through all that systematically and work
out whether we would lose anything at all by deleting it now.

(2) the "primes" file for eclib is optional. When eclib stuff runs it
looks to see if there is such a file in the working directory, and if
there is, adds the numbers in it to an internal list of "useful"
primes. If there is no such file it should not complain. the
specific number in the file you have has no significance whatsoever.
I am therefore 99.999% sure that deleting that file would not make any
difference, so is safe to do.

John

2008/12/5 mabshoff <mabs...@googlemail.com>:

[William Stein]

I'm strongly in favor of this proposal.

William

[mabshoff]

On Dec 5, 2:53 pm, "William Stein" <wst...@gmail.com> wrote:

Hi William,

<SNIP>

> I'm strongly in favor of this proposal.

Cool, I am keeping an eye on this then, also in hindsight regarding
John's comment about "his" pari code. The plan for 3.2.2 is to get the
javascript spkgs broken out (which is mostly thanks to the hard work
of Jason Grout) and then we can tackle the rest of the pieces in 3.3.

One more thing about those empty directories:

0B MuPAD
0B QEPCAD
0B genus2reduction
0B gnuplot
0B kash
0B macaulay2
0B maple
0B mathematica
0B matlab
0B octave
0B sage
0B sobj

Can we get rid of them or will we have to fix something in the Sage
library? I can always try by deleting the directory and running some
optional doctests, but an informed opinion would be great here.

> William

Cheers,

Michael

[William Stein]

We can get rid of them. If it breaks something in the sage library,
then we may as well consider that a bug in Sage, and it'll get fixed
before the release.

-- William

[mabshoff]

On Dec 5, 5:12 pm, "William Stein" <wst...@gmail.com> wrote:

Cool, sounds like a game plan then. No need to open tickets yet, but I
have added the goal to eliminate to the sage-3.3 milestone in trac and
added a link back to this discussion so we will know what to do.

>  -- William

Cheers,

Michael