"Copy&Paste&Symlink" vs "Refactor&Source"

[ph h]

Dear All,

While waiting for advice on the questions on the thread "Copy&Paste"

"""

BTW, is symlink a solution for "Copy & Paste"?

By another way, are there reasons that "Copy & Paste & Symlinks" is better than just symlinks.

"""

This is not meant "to look  a gift horse in the mouth", but  "straight from the horse's mouth", some advice were found (and appended below) and they beg the questions:

1/- Why "Copy&Passte&Symlink" was used instead of "Refactor&Source"? There are of course good reasons for this, it is believed.

2/- What "deserves to be factored out at some point in the future"? The "resolvelinks" function or the scripts, "sage/src/bin/sage-env", "sage/sage" and "sage/src/bin/sage"? 

3/- When will that "point in the future" come? If "the future" is now, how can these be best done? or just "que será, será"

Just curious minds wish to learn.

Thank you for your advice.

Regards,

phiho

<src/bin/sage-env>

###########################################################################
#
#  Set environment variables for building and/or running Sage.
#
#  NOTES:
#  - You must *source* this script instead of executing.
#  - Use "return" instead of "exit" to signal a failure.  Since this
#    file is sourced, an "exit" here will actually exit src/bin/sage,
#    which is probably not intended.
#  - All environment variables set here should be *exported*, otherwise
#    they won't be available in child programs.
#  - This script has a version number such that a newer version of
#    sage-env can be sourced when upgrading.  See below.
#
#  If you want to set all environment variables for your shell like
#  they are during the build of Sage packages, type
#
#             . src/bin/sage-env
#
#  from the SAGE_ROOT directory.
#
#  AUTHORS: William Stein, David Kirkby, Jeroen Demeyer,
#           J. H. Palmieri, Leif Leonhardy and others.
#
##########################################################################

</src/bin/sage-env>

and

<sage/sage>

#
# WARNING: this function is copy/pasted from src/bin/sage-env, and
# deserves to be factored out at some point in the future. In the
# meantime however the two should be kept synchronized.
resolvelinks() {

....

....

</sage/sage>

[Michael Orlitzky]

On Thu, 2022-04-21 at 14:28 -0700, ph h wrote:
> Dear All,
>
> While waiting for advice on the questions on the thread "Copy&Paste"
>
> """
> BTW, is symlink a solution for "Copy & Paste"?
> By another way, are there reasons that "Copy & Paste & Symlinks" is better
> than just symlinks.
> """
>
> This is not meant "to look a gift horse in the mouth", but "straight from
> the horse's mouth", some advice were found (and appended below) and they
> beg the questions:
>
> 1/- Why "Copy&Passte&Symlink" was used instead of "Refactor&Source"? There
> are of course good reasons for this, it is believed.

Refactor and source isn't as simple as it sounds. Sage is a mishmash of
subprojects with different build systems, and is constantly in
transition between high-level refactorings. During those neverending
transitions, we try to keep both the start and end points working.

At build- or run-time, some of those scripts may not exist, some may be
symlinked, some may have been moved from their original locations... if
everything was using autotools, you could just substitute in the
correct location of resolvelinks.sh. But some pieces of sage don't use
autotools, and the "sage" script itself is still invoked by many people
*before* building anything.

These aren't insurmountable problems, but copy/paste is relatively
straightforward.

> 2/- What "deserves to be factored out at some point in the future"? The
> "resolvelinks" function or the scripts, "sage/src/bin/sage-env",
> "sage/sage" and "sage/src/bin/sage"?

The function.

> 3/- When will that "point in the future" come? If "the future" is now, how
> can these be best done? or just "que será, será"
>

You could try it now if you want. To get an idea of what you're in for,
the last copy of resolvelinks() was introduced in Trac 30888:

https://trac.sagemath.org/ticket/30888

The comments reveal that,

1. The "sage" script needs to know its own absolute, link-free 
location.
2. It will be installed when ./configure has not been run.
3. Python doesn't make the final installation location available at 
setup.py time, so a fake-autotools solution is out.

To what extent those are true/necessary I can't really say, and I
wasn't super interested in learning the intricacies of a bad way to
install software just to avoid copy/pasting a shell function. If
there's any easy way to avoid it, we'd all be happy.

[Matthias Koeppe]

On Thursday, April 21, 2022 at 3:16:38 PM UTC-7 Michael Orlitzky wrote:

the last copy of resolvelinks() was introduced in Trac 30888:

https://trac.sagemath.org/ticket/30888

The comments reveal that,

1. The "sage" script needs to know its own absolute, link-free 
location.
2. It will be installed when ./configure has not been run.

Actually not installed, just executed directly out of the source tree when ./configure has not been run.

[ph h]

Hello,

> You could try it now if you want. To get an idea of what you're in for,
> the last copy of resolvelinks() was introduced in Trac 30888:
>
> https://trac.sagemath.org/ticket/30888

Thank you for the tips. 

It looks like there is a lot of information for a newbie to process without a "Hitchhiker's Guide to the SageMath Multiverse" (smiley)

She might try with some "low hanging fruit" first. (smiley)

<you wrote>

At build- or run-time, some of those scripts may not exist, some may be
symlinked, some may have been moved from their original locations... if
everything was using autotools, you could just substitute in the
correct location of resolvelinks.sh. But some pieces of sage don't use

autotools, .

</you wrote>

> and the "sage" script itself is still invoked by many people

> *before* building anything

Is this use case scenario, (running a script before building anything) documented or undocumented?

> "Make $SAGE_LOCAL/bin/sage work again from any directory, in an environment without SAGE_* variables, following symlinks"

> Trac #30888: resolve $0 links in src/bin/sage.

Please advise if this  issue can be reproduced with SageMath 9.2 and earlier only. 

Is it still available on GitHub and how to check it out.

BTW, it was asked:

> >BTW, is symlink a solution for "Copy & Paste"?

and the answer should be,  "Copy & Paste" is a solution to broken symlink?

A follow up question is "why 'Copy&Paste' works and 'source' does not?

Last but not least

>> 2/- What "deserves to be factored out at some point in the future"? The
>> "resolvelinks" function or the scripts, "sage/src/bin/sage-env",
>> "sage/sage" and "sage/src/bin/sage"?

> The function.

What are the problems with the current version of 'resolvelinks', how should it be factored?

What are the problems when just the function is sourced, not the whole 'sage/src/bin/sage-env' script

When it is properly done, can 'Copy&Past' be replaced with 'source'?

Thank you so much for your response. It is much appreciated

Regards,

phiho

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/2a6a7c87f54a056c4d22e1a84ebf5597c0b58f52.camel%40orlitzky.com.

[Michael Orlitzky]

On Fri, 2022-04-22 at 01:03 -0400, ph h wrote:
>
> Is this use case scenario, (running a script before building anything)
> documented or undocumented?
>

I'm not sure if it still is, but running things like "sage -b" and
"sage -i" was for 15+ years the documented way to build parts of sage.
These days "make build" and "make <package>" should be preferred,
precisely because they avoid a chicken and egg problem. But the old way
is still supported for now.


>
> *> "Make $SAGE_LOCAL/bin/sage work again from any directory, in an
> environment without SAGE_* variables, following symlinks"*

> > Trac #30888: resolve $0 links in src/bin/sage.
>
> Please advise if this issue can be reproduced with SageMath 9.2 and
> earlier only.

I think if you revert the commit from #30888 that the issue will
return.

>
> A follow up question is "why 'Copy&Paste' works and 'source' does not?

Roughly: you need to know where to source the file from, and that path
is relative to the script that wants to do the sourcing, but we don't
know where the script doing the sourcing is until we use resolvelinks()
on $0.

>
> What are the problems with the current version of 'resolvelinks', how
> should it be factored?

None, AFAIK.

> What are the problems when just the function is sourced, not the whole
> 'sage/src/bin/sage-env' script

If you can figure out how to put the function in its own file and
manage to source it, there shouldn't be any problems. It's the act of
sourcing itself that is problematic.

> When it is properly done, can 'Copy&Past' be replaced with 'source'?

Maybe =)

We've copy/pasted to avoid all of the work involved in determining the
answer to this question.

[ph h]

Dear All,

If the three files:

1. sage/sage
   
2. sage/src/bin/sage
   
3. sage/src/bin/sage-env

are to be factored out into

1. sage/resolvelinks.in
2. sage/sage.sage.in
3. sage/sage.src.bin.sage.in
4. sage/sage.src.bin.sage-env.in

and

1. sage/sage (sourcing sage/resolvelinks.in and sage/sage.sage.in)
2. sage/src/bin/sage (sourcing  sage/resolvelinks.in and sage/sage.src.bin.sage.in)
3. sage/src/bin/sage-env (sourcing  sage/resolvelinks.in and sage/sage.src.bin.sage-env.in)

Please advise how can this attempt be tested:

1. Before 'make configure' is run at $SAGE_ROOT
2. Before './configure' is run at $SAGE_ROOT
3. Before 'make' is run   at $SAGE_ROOT
4. After installation is completed

Thank you for your guidance.

Regards,

phiho

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/c9273b44d5fc213555e9420bf7a84c46b5afb8cc.camel%40orlitzky.com.

[Michael Orlitzky]

On Sat, 2022-04-23 at 08:28 -0400, ph h wrote:
> Dear All,
>
> If the three files:
>

> 1. sage/sage
> 2. sage/src/bin/sage
> 3. sage/src/bin/sage-env

>
> are to be factored out into
>

> 1. sage/resolvelinks.in
> 2. sage/sage.sage.in
> 3. sage/sage.src.bin.sage.in
> 4. sage/sage.src.bin.sage-env.in
>

Note: the ".in" suffix doesn't mean "include," by convention it roughly
means "a file that will be processed by ./configure". So, for example,
Makefile.in gets turned into Makefile when you run ./configure.

>
> Please advise how can this attempt be tested:
>

> 1. Before 'make configure' is run at $SAGE_ROOT
> 2. Before './configure' is run at $SAGE_ROOT
> 3. Before 'make' is run at $SAGE_ROOT
> 4. After installation is completed
>
>

In general: just go try it and see what breaks. Here there are two
problems:

1. The python build system apparently can't make the right path 
substitutions (like autotools could), and we use the python
build system for the bits relevant to these files.

2. Everyone wants the top-level "sage" script to work out of the box 
without being configured, built, or installed. Within "sage", 
then, how can you know where to look for the resolvelinks.sh 
file? You can't use $0 if $0 is a link unless you know how to
resolve it.

[ph h]

Hi,

Thank you for your response.

  > Note: the ".in" suffix doesn't mean "include," by convention it roughly
> means "a file that will be processed by ./configure". So, for example,
> Makefile.in gets turned into Makefile when you run ./configure.

So, to avoid confusion, ".in" should be ".src", just to be "sourced"?

> 2. Everyone wants the top-level "sage" script to work out of the box 
> without being configured, built, or installed. Within "sage", 
> then, how can you know where to look for the resolvelinks.sh 
> file? You can't use $0 if $0 is a link unless you know how to
> resolve it

WOW, without running even 'make configure'?

Is the distributed tarball already pre-configured ('configure" is included)?

> In general: just go try it and see what breaks.

Try what? 

Maybe experienced users in this camp can share some undocumented use case scenarios?

To be clear, what would they like to do with a copy of SageMath just freshly checked out from github and how to verify that "it works" out of the box

Maybe these people just use the pre-configured tarball because they may not be familiar with github?

Regards,

phiho

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/fdf2f4cb66ed9f5af12435c8425e3094f81e8660.camel%40orlitzky.com.

[Matthias Koeppe]

On Saturday, April 23, 2022 at 9:08:55 AM UTC-7 Michael Orlitzky wrote:

1. The python build system apparently can't make the right path 
substitutions (like autotools could), and we use the python
build system for the bits relevant to these files.

Yes, in modern Python packaging, all installation goes through building wheels -- so the final installation location is not known at build time. 

(Although in the Sage distribution, we are not quite there yet -- see https://trac.sagemath.org/ticket/32874)

[Dima Pasechnik]

On Sat, 23 Apr 2022, 19:00 ph h, <hohoa...@gmail.com> wrote:

Hi,

Thank you for your response.

  > Note: the ".in" suffix doesn't mean "include," by convention it roughly
> means "a file that will be processed by ./configure". So, for example,
> Makefile.in gets turned into Makefile when you run ./configure.

So, to avoid confusion, ".in" should be ".src", just to be "sourced"?

.in suffixes are standard (e.g. Makefile.in is the standard name for a file generated by automake). These files often contain templates, which are substituted by ./configure with actual values, they are seldom something one sources, in the unix shell sense.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CALZQg1azkPhWjnVuv9rD0PiEMRYdROr75e2WhdxrBj_J7BcYOA%40mail.gmail.com.

[ph h]

Hi,

Thank you for your responses.

> Yes, in modern Python packaging, all installation goes through building wheels -- so the final installation location is not known at build time. 

> (Although in the Sage distribution, we are not quite there yet -- see https://trac.sagemath.org/ticket/32874)

> in suffixes are standard (e.g. Makefile.in is the standard name for a file generated by automake). 

> These files often contain templates, which are substituted by ./configure with actual values, 

> they are seldom something one sources, in the unix shell sense.

Your guidance is much appreciated.

Regards,

phiho

 

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAAWYfq3jUC%3DY6r%2Bvy%3DA27xApKO0BNcOg-Lk9yhu_%3D%3Do%2BpkzZJA%40mail.gmail.com.

[ph h]

P.S:

"""

Everyone wants the top-level "sage" script to work out of the box 
without being configured, built, or installed
"""

> Maybe experienced users in this camp can share some undocumented use case scenarios?

While waiting for someone to share their secrets, It was found that the "advanced incantations"  is missing some important pieces,

(like 'sage -cbr"  "Configure&Build&Run" so that advanced users do not have to face questions like "Did you run configure?"

[Matthias Koeppe]

"configure" was deliberately made a separate, manual step because users need to read (and act upon) its output.

[ph h]

Good morning,

> "configure" was deliberately made a separate, manual step because users need to read (and act upon) its output.

Maybe, "sage -cbr" can take arguments the same way 'configure' does or it just takes one argument, like "sage -cbr @responsefile".

Maybe it just  configures and builds a minimal SageMath from there one can install object by object to build a custom SageMath

It is interesting to find out what would this minimal SageMath be, would this be the ultimate goal for the modularization of  SageMath?

Is  this minimal SageMath doable?

Best regards,

phiho

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/e190d6f4-105a-4f03-9e2d-1d84dbf35aa3n%40googlegroups.com.

[Matthias Koeppe]

On Thursday, April 28, 2022 at 4:14:11 AM UTC-7 hohoa...@gmail.com wrote:

[...] configures and builds a minimal SageMath from there one can install object by object to build a custom SageMath

It is interesting to find out what would this minimal SageMath be, would this be the ultimate goal for the modularization of  SageMath?

Is  this minimal SageMath doable?

Changing what is installed by default is not part of the modularization plan, but defining such "minimal" parts definitely is. 

For example, https://trac.sagemath.org/ticket/32432 works toward a package "sagemath-polyhedra", with very few dependencies.

If you want to help with modularization, here's a ticket that's easy to contribute to: https://trac.sagemath.org/ticket/33029

[Matthias Koeppe]

On Thursday, April 21, 2022 at 2:28:06 PM UTC-7 hohoa...@gmail.com wrote:

1/- Why "Copy&Passte&Symlink" was used instead of "Refactor&Source"? There are of course good reasons for this, it is believed.

2/- What "deserves to be factored out at some point in the future"? The "resolvelinks" function or the scripts, "sage/src/bin/sage-env", "sage/sage" and "sage/src/bin/sage"? 

3/- When will that "point in the future" come? If "the future" is now, how can these be best done? or just "que será, será"

I've updated the comment in the top-level "sage" script that led to these questions as part of https://trac.sagemath.org/ticket/33786 (needs review).