Building a deb for sage on Debian/Ubuntu

[Jan Groenewald]

Hi

I am looking for a Debian or Ubuntu packaging mentor.

I am not an experienced debian packager, I am learning everything from scratch.
But I am an experienced system administrator.

I am hoping to find a mentor familiar with both Debian/Ubuntu and sage to assist me.

I am trying to build a deb package for sage. I am doing this on launchpad for Ubuntu using
http://developer.ubuntu.com/packaging/html/packaging-new-software.html
I would not mind getting Sage into Debian first, but found the Ubuntu documentation
more accessible and am an Ubuntu user.

I am NOT trying to debianize sage fully -- i.e. I do not want to have sage depend
on debian/ubuntu versions of all the spkg'es, my goal is just to make Sage easy to
install on Ubuntu through a personal package archive, including all sage versions of
spkg'es.

I have succesfully built sage-4.8 on Ubuntu 12.04 (yes, the development version).
However, when running debuild, maxima fails to build:

l,-z,relro -L/home/jan/sage-4.8/sagemath/local/lib -Wl,--rpath,/home/jan/sage-4.8/sagemath/local/lib -L/home/jan/sage-4.8/sagemath/local/lib -lecl -lgmp -lgc -ldl -lm make[4]: *** [binary-ecl/maxima] Error 1
make[4]: Leaving directory `/home/jan/sage-4.8/sagemath/spkg/build/maxima-5.23.2.p3/src/src'
make[3]: *** [all-recursive] Error 1
make[3]: Leaving directory `/home/jan/sage-4.8/sagemath/spkg/build/maxima-5.23.2.p3/src'
***********************************************************
Error: Failed to build Maxima.
***********************************************************

real    4m21.449s
user    3m49.458s
sys    0m19.189s
************************************************************************
Error installing package maxima-5.23.2.p3
************************************************************************
Please email sage-devel (http://groups.google.com/group/sage-devel)
explaining the problem and including the relevant part of the log file
  /home/jan/sage-4.8/sagemath/spkg/logs/maxima-5.23.2.p3.log
Describe your computer, operating system, etc.
If you want to try to fix the problem yourself, *don't* just cd to
/home/jan/sage-4.8/sagemath/spkg/build/maxima-5.23.2.p3 and type 'make' or whatever is appropriate.
Instead, the following commands setup all environment variables
correctly and load a subshell for you to debug the error:
  (cd '/home/jan/sage-4.8/sagemath/spkg/build/maxima-5.23.2.p3' && '/home/jan/sage-4.8/sagemath/sage' -sh)
When you are done debugging, you can type "exit" to leave the subshell.
************************************************************************
make[2]: *** [installed/maxima-5.23.2.p3] Error 1
make[2]: Leaving directory `/home/jan/sage-4.8/sagemath/spkg'

real    145m34.953s
user    127m7.853s
sys    8m51.233s
Error building Sage.
make[1]: *** [build] Error 1
make[1]: Leaving directory `/home/jan/sage-4.8/sagemath'
dh_auto_build: make -j1 returned exit code 2
make: *** [build] Error 2
dpkg-buildpackage: error: debian/rules build gave error exit status 2
debuild: fatal error at line 1350:
dpkg-buildpackage -rfakeroot -D -us -uc failed
29 jan@osprey:~/sage-4.8/sagemath$




Regards,
Jan


--
  .~.
  /V\     Jan Groenewald
 /( )\    www.aims.ac.za
 ^^-^^

[Francois Bissey]

Hi Jan,

we'll need a bit more of your output, especially if you ran parallel make.
If you could link to the whole maxima build log (in skpg/log) that would
be most useful.

Francois

[Jan Groenewald]

Hi

On 25 February 2012 21:33, Francois Bissey <fbi...@slingshot.co.nz> wrote:

Hi Jan,

we'll need a bit more of your output, especially if you ran parallel make.
If you could link to the whole maxima build log (in skpg/log) that would
be most useful.

The maxima bild logs:

http://users.aims.ac.za/~jan/maxima-5.23.2.p3.log.lzma

That says 

/usr/bin/ld: unrecognized option '-Bsymbolic-functions -Wl'

I hear on #ubtuntu-motu that debuild sets these flags. I'm googling it,
but not finding much.

[Jan Groenewald]

Hi

As an aside, can I get a hold of the scripts that build the binaries available
for download on sagemath.org and mirrors (for Ubuntu LTS only). As a first
step to getting packaging skills and having a deb for people to install, I could
try make a virtual debian package that contains the sage prebuilt, and untars
it, sets permissions, and sets the SAGE_ROOT in the launch script.

Regards,
Jan

[Jan Groenewald]

Hi

Is it still necessary to run sage as root once to set some paths?

When one does that, will it create new files? This is important
for debian packaging.

Regards,
Jan

[P Purkayastha]

I am not sure how your packaging works. Do you install into some temporary directory first and then tar/bzip the resulting directory? Then, if the user changes the directory in which (s)he installs Sage (and this will obviously happen), then it will need to be run as root once.

Also see this ticket which might be relevant for you if you are using the "make install" command: http://trac.sagemath.org/sage_trac/ticket/12347

[Jeroen Demeyer]

On 2012-02-26 16:59, Jan Groenewald wrote:
> Is it still necessary to run sage as root once to set some paths?

It depends on how you build Sage. Let me elaborate: In order to compile
Sage, you don't need root privileges. In fact, it is strongly
recommended *not* to build Sage as root. During "make" (or "make all"),
Sage is ran once. This will create a few new files. After this, every
user on the system can run Sage from the directory it was compiled in
(if not, this is considered a bug). If you move your Sage tree after
building, you have to run sage-location again as the user who owns the
Sage files. As a final step, you can change the owner to root if you want.

[Jan Groenewald]

Hi

On 27 February 2012 09:55, Jeroen Demeyer <jdem...@cage.ugent.be> wrote:

On 2012-02-26 16:59, Jan Groenewald wrote:
> Is it still necessary to run sage as root once to set some paths?

It depends on how you build Sage.  Let me elaborate: In order to compile
Sage, you don't need root privileges.  In fact, it is strongly
recommended *not* to build Sage as root.

Yes, I am not compiling as root. I am making two packages

A deb from source, which will compile as some build user on launchpad build farms.
People can then install this from our PPA. I think this aids Sage adoption.

A deb from prebuilt binaries, which was built on sage build farms (not the right way,
but faster)

( Note I am not making a deb for debian where the components are separated out;
this is the Debian way, but too much work for now. The valid concern they raise is
that security updates will be hard as it involves all the components. This raised another
thought in my mind --- how does Sage deal with security updates to any of it's components
currently? )

 

 During "make" (or "make all"),
Sage is ran once.  This will create a few new files.  After this, every
user on the system can run Sage from the directory it was compiled in
(if not, this is considered a bug).  If you move your Sage tree after
building, you have to run sage-location again as the user who owns the
Sage files.  As a final step, you can change the owner to root if you want.

When building debs, for system-wide installs, sage is moved to a new location
and owned by root. So unlike above, it is changed to root during the move,
and sage-location will have to run as root.

It is important for deb whether this postinst step (running sage-location)
will create NEW files, as if it does, I have to remove them during package
removal with a postrm script.

As far as I can see there are 4200+ pyc files in lib/local/python, but I guess
those are all created during make or sage-location, and running it again
will only replace existing files.

Regards,
Jan
 

[Jeroen Demeyer]

On 2012-02-27 10:02, Jan Groenewald wrote:
> It is important for deb whether this postinst step (running sage-location)
> will create NEW files

That's easy to check with a ls -alR, diff the result before and after
running sage-location. My guess is that it doesn't create new files.

[Jan Groenewald]

Hi

I am making some progress with the first deb which just wraps the sage upstream binary.

I get this when I install it though:
http://pastebin.ubuntu.com/859462/

Any ideas?

Regards.,

[William Stein]

On Mon, Feb 27, 2012 at 9:47 AM, Jan Groenewald <j...@aims.ac.za> wrote:
> Hi
>
> I am making some progress with the first deb which just wraps the sage
> upstream binary.

Did the binary ever work for you? Do you have the appropriate ssl
libraries installed?

>
> I get this when I install it though:
> http://pastebin.ubuntu.com/859462/
>
> Any ideas?
>
> Regards.,
>
> Jan
>
> --
>   .~.
>   /V\     Jan Groenewald
>  /( )\    www.aims.ac.za
>  ^^-^^
>

> --
> To post to this group, send an email to sage-...@googlegroups.com
> To unsubscribe from this group, send an email to
> sage-devel+...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/sage-devel
> URL: http://www.sagemath.org

--
William Stein
Professor of Mathematics
University of Washington
http://wstein.org

[Jan Groenewald]

Hi

On 27 February 2012 18:55, William Stein <wst...@gmail.com> wrote:

On Mon, Feb 27, 2012 at 9:47 AM, Jan Groenewald <j...@aims.ac.za> wrote:
> Hi
>
> I am making some progress with the first deb which just wraps the sage
> upstream binary.

Did the binary ever work for you?  Do you have the appropriate ssl
libraries installed?

No, my other local install was from source, and works fine.

This binary via a deb needed openssl0.9.8 (openssl1.0.0 is default
on Ubuntu 12.04) and now seems to work!

I'll be continuing this (a from source deb), but it will be great to at least get a
from-binary deb online in a PPA that works.

Regards,
Jan

[William Stein]

On Mon, Feb 27, 2012 at 10:29 AM, Jan Groenewald <j...@aims.ac.za> wrote:
> Hi
>
> On 27 February 2012 18:55, William Stein <wst...@gmail.com> wrote:
>>
>> On Mon, Feb 27, 2012 at 9:47 AM, Jan Groenewald <j...@aims.ac.za> wrote:
>> > Hi
>> >
>> > I am making some progress with the first deb which just wraps the sage
>> > upstream binary.
>>
>> Did the binary ever work for you?  Do you have the appropriate ssl
>> libraries installed?
>
>
> No, my other local install was from source, and works fine.
>
> This binary via a deb needed openssl0.9.8 (openssl1.0.0 is default
> on Ubuntu 12.04) and now seems to work!
>
> I'll be continuing this (a from source deb), but it will be great to at
> least get a
> from-binary deb online in a PPA that works.

Yes, definitely.

I remember once in 2006 that an undergrad at UW -- Bobby Moretti --
made a PPA with Sage on Ubuntu, which built from source. I think he
did it in a "24 hour sprint". He added some fake "./configure" and
"make install" stuff to Sage so that this would work. Unfortunately,
he is no longer involved in Sage so can't easily give advice about
this.

-- William

>
> Regards,
> Jan
>
>
> --
>   .~.
>   /V\     Jan Groenewald
>  /( )\    www.aims.ac.za
>  ^^-^^
>

[Jan Groenewald]

Hi

By the way, as part of the workshop that is giving me the time to do this
sprint, http://cdsagenda5.ictp.it/full_display.php?email=0&ida=a11154 ,
we had a speaker, Steve Crouch, from http://www.software.ac.uk .

The Software Sustainability institute works with researchers to improve
their software. Attached is their software evaluation criteria. It seems
to me Sage is already very mature by these standards, but I could
not answer all of it.

Is someone more familiar with the whole of Sage
interested in filling this in?

Regards,
Jan

[Jan Groenewald]

Hi

The deb available below was not built from source on launchpad, instead
it is the sage upstream binary built for Ubuntu 10.04.3 hammered into a
PPA for lucid, maverick natty, oneiric, and precise in the ugliest possible way.

It is only 64bit at the moment.

To test it do this:

sudo apt-add-repository ppa:aims/sagemath
sudo apt-get update
sudo apt-get install sagemath-upstream-binary

If you have sage in /usr/local/bin/sage, that is most likely
in your path BEFORE /usr/bin/sage where this PPA will install
it.

Known bug:
Dragging the icon from the dash to the launcher in precise works
except the icon does not show in he launcher, just a transparent
square shows (the icon does show in the dash though).

I learnt a bit about packaging, and will continue to try to make a from-source
package in this PPA.

[Dan Drake]

On Tue, 28 Feb 2012 at 09:50PM +0100, Jan Groenewald wrote:
> The deb available below was not built from source on launchpad, instead
> it is the sage upstream binary built for Ubuntu 10.04.3 hammered into a
> PPA for lucid, maverick natty, oneiric, and precise in the ugliest possible
> way.

Three cheers for doing it in the ugliest possible way -- rather than not
doing it at all! Sage should definitely be available via a PPA.

I tried this in a Lucid virtual machine, and your dependencies may need
slight tweaking: it refuses to install because of a dpkg dependency:

The following packages have unmet dependencies:
sagemath-upstream-binary: PreDepends: dpkg (>= 1.15.6~) but
1.15.5.6ubuntu4.5 is to be installed

Would 1.15.5 work? Or, do you want to focus on the newer releases, since
Precise (a newer LTS) will be out soon?

Keep up the good work!

Dan

--
--- Dan Drake
----- http://mathsci.kaist.ac.kr/~drake
-------

[Jan Groenewald]

Hi Dan,

I want to focus on newer, and from-source, but until Friday I am only working on
this from-binary. I want to add i686 today, and fix any bugs I may find.

I will perhaps also try add a bug-reporting mailing list to the PPA page,
it seems launchpad can host them. I may be back more often asking for
help on sage-devel though!

I guess that dpkg dependency is because I built the deb on precise, and
there is a mechanism for Copying packages to save resources, where
they do not need rebuilding. I will look into it -- thanks for testing
and reporting.

Regards,
Jan

[Dan Drake]

On Tue, 28 Feb 2012 at 09:50PM +0100, Jan Groenewald wrote:

> The deb available below was not built from source on launchpad, instead
> it is the sage upstream binary built for Ubuntu 10.04.3 hammered into a
> PPA for lucid, maverick natty, oneiric, and precise in the ugliest possible
> way.
>
> It is only 64bit at the moment.
>
> To test it do this:
>
> sudo apt-add-repository ppa:aims/sagemath
> sudo apt-get update
> sudo apt-get install sagemath-upstream-binary

I tried this in a 11.10 virtual machine, and it created the user's .sage
directory as owned by root -- with 700 permissions, so when I try to
start Sage, it fails because it can't read that directory.

In another virtual machine, I got something similar, where the
matplotlib directory was owned by root.

This may be related to having used Sage previously on those machines,
but in any case we shouldn't create root-owned things in the user's
directory!

[Jan Groenewald]

Hi Dan

On 15 March 2012 02:05, Dan Drake <dr...@kaist.edu> wrote:

> sudo apt-add-repository ppa:aims/sagemath
> sudo apt-get update
> sudo apt-get install sagemath-upstream-binary

I tried this in a 11.10 virtual machine, and it created the user's .sage
directory as owned by root -- with 700 permissions, so when I try to
start Sage, it fails because it can't read that directory.

In another virtual machine, I got something similar, where the
matplotlib directory was owned by root.

This may be related to having used Sage previously on those machines,
but in any case we shouldn't create root-owned things in the user's
directory!

Thanks, I will look into it. Are the steps to reproduce 1) install new OS,
2) install Sage from PPA, 3) run sage as user?

I had previous .sage directories and the 64bit install on precise did not
cause this problem. I now moved my .sage out the way and ran sage,
and .sage was not created with root permissions.  I then removed my
.sage, aptitude reinstall sagemath-upstream-binary, ran sage,
but am not able to reproduce this problem.

I can't really think how the deb could cause this,
but will try reproduce on a fresh OS install in a VM
in a week or two.



An update on the previous dpkg version problem. I have dropped lucid support
on that PPA, as it is the only Ubuntu version requiring an older dpkg, and that
older dpkg does not support xz compression. The difference between xz compression
and gz compression is roughly 80M for the deb alone. dpkg 1.15.6 started supporting
this.

I did however add 32bit and changed to supporting natty, oneiric, and precise.

The 32bit is really compiled for i686 (not i386 as expected by debhelper)
so I had to disable stripping the binary, so the 32bit deb package is 70M
larger than the 64bit deb package. The latter is around 370M. The unpacked
space usage is 1.6G.

Anyway, the from-upstream-binary (that is the compiled binaries for Ubuntu 10.04.3
from sagemath.org) was just a hack to get something out quickly, I would like
to move efforts on to a from-source PPA soon. I will however fix bugs as I can.

The PPA has reached its 2G space limit on launchpad. I asked for more space on 6th
March:  https://answers.launchpad.net/launchpad/+question/189847
The project is pretty much standing still until that space problem is sorted out.

Regards,
Jan

[Dan Drake]

On Thu, 15 Mar 2012 at 07:13AM +0100, Jan Groenewald wrote:
> Thanks, I will look into it. Are the steps to reproduce 1) install new OS,
> 2) install Sage from PPA, 3) run sage as user?

What I have done is (this happens every time):

1. Remove $HOME/.sage
2. Reinstall Sage from PPA
3. New $HOME/.sage is root-owned.

If I have an existing $HOME/.sage owned by the user, I get a root-owned
file when reinstalling the PPA package: the "lazy_import_cache.pickle" file.

> An update on the previous dpkg version problem. I have dropped lucid
> support on that PPA, as it is the only Ubuntu version requiring an
> older dpkg, and that older dpkg does not support xz compression. The
> difference between xz compression and gz compression is roughly 80M
> for the deb alone. dpkg 1.15.6 started supporting this.

Dropping Lucid is a good idea. Focus on getting this to work on Precise,
and then see if we can do a source PPA.

[Jan Groenewald]

Hi Dan,

( I have been given a bit more space on launchpad (up from
2G to 4G now) so I can work slightly easier. )

On 16 March 2012 03:48, Dan Drake <dr...@kaist.edu> wrote:

On Thu, 15 Mar 2012 at 07:13AM +0100, Jan Groenewald wrote:
> Thanks, I will look into it. Are the steps to reproduce 1) install new OS,
> 2) install Sage from PPA, 3) run sage as user?

What I have done is (this happens every time):

1. Remove $HOME/.sage
2. Reinstall Sage from PPA
3. New $HOME/.sage is root-owned.

I have narrowed it down. There is a post-install script that runs sage once as
root to fix the paths.

root# dpkg-reconfigure sagemath-upstream-binary

does not reproduce your problem, which confused me. However,

user$ sudo dpkg-reconfigure sagemath-upstream-binary

does reproduce it, so I think that my run-once-as-root is run as root but
in the SUDOUSER /home directory! I guess maybe $HOME is wrong
in this case.

However, there are two possible solutions:
1) run the sage-change-paths command directly with an argument
/root/.sage (how?)
2) because I symlink (instead of copying and editing SAGE_ROOT)
/usr/bin/sage -> /usr/lib/sage/sage it might not be necessary at all to
run the sage-change-paths command?


This is the current bit of the postinst script:

0 jan@oystercatcher:~/src/sagemath-upstream-binary/sagemath-upstream-binary/debian$grep -A2 " configure)" postinst
    configure)
        echo "Running Sage once as root to set paths"
        echo 'exit' | /usr/bin/sage

[leif]

There's help coming I think:

#11932: Allow running Sage with a temporary DOT_SAGE directory

I.e., then run 'sage' with '--nodotsage' (which uses a temporary
'.sage/' directory).

(You could most probably also set DOT_SAGE "manually" when invoking
'sage' for the first time; it defaults to '$HOME/.sage/'.)


-leif

[1] http://trac.sagemath.org/sage_trac/ticket/11932

[Jan Groenewald]

Hi

On 20 March 2012 13:20, leif <not.r...@online.de> wrote:

(You could most probably also set DOT_SAGE "manually" when invoking
'sage' for the first time; it defaults to '$HOME/.sage/'.)

Thanks, I have done this and uploaded a new package to
launchpad. I now set DOT_SAGE=/root/.sage .

Regards,
Jan