new source files doesn't work well with shadow directories

[Tony Mechelynck]

Dear Yegappan,

Recently you've been proposing (and Bram has accepted) to create a lot
of new source files to make the existing source easier to understand.
The notion is laudable but it doesn't work well with my shadow
directories: every time there is a new .c source (or .h; .pro sources
are taken care of automatically by virtue of their being in a special
src/proto directory for which a symlink has ben creted once and for
all in the shadow directory), I have to manually create a symlink
newsourcename.c -> ../newsourcename.c in each shadow directory,
otherwise the "make" step ends in error with "No rule to make
newsourcename.c. Stop.".

Do you think the following rule (added to the src/Makefile which is
soft-linked from every shadow directory) could take care of my problem
without undesired side-effects?

*: ../$@
ln -sv ../$@

The idea is to create a link if a needed file exist inthe parent (src)
directory but not in the current (shadow) directory. Maybe the target
sould be made less general to avoid using this rule out of turn.

I have proposed this to Bram but AFAIK he has neither answered my
proposal but taken action on it.

Best regards,
Tony.

[Yegappan Lakshmanan]

Hi Tony,

On Tue, Aug 27, 2019 at 11:17 PM Tony Mechelynck
<antoine.m...@gmail.com> wrote:
>
> Dear Yegappan,
>
> Recently you've been proposing (and Bram has accepted) to create a lot
> of new source files to make the existing source easier to understand.
> The notion is laudable but it doesn't work well with my shadow
> directories: every time there is a new .c source (or .h; .pro sources
> are taken care of automatically by virtue of their being in a special
> src/proto directory for which a symlink has ben creted once and for
> all in the shadow directory), I have to manually create a symlink
> newsourcename.c -> ../newsourcename.c in each shadow directory,
> otherwise the "make" step ends in error with "No rule to make
> newsourcename.c. Stop.".
>

I don't use the shadow directories, so I didn't see this problem before.

Do you just run "make shadow" to create the shadow directory
and build from there? How do you make use of the shadow
directory?

When I run "make shadow" in a newly pulled view, the symbolic links
to all the .c files from the src directory are created in the shadow directory.

Are you trying to update an existing shadow directory? Are the
symbolic links to the new .c files created if you clean the view and run
"make shadow" again?

Regards,
Yegappan

[Bram Moolenaar]

> Recently you've been proposing (and Bram has accepted) to create a lot
> of new source files to make the existing source easier to understand.
> The notion is laudable but it doesn't work well with my shadow
> directories: every time there is a new .c source (or .h; .pro sources
> are taken care of automatically by virtue of their being in a special
> src/proto directory for which a symlink has ben creted once and for
> all in the shadow directory), I have to manually create a symlink
> newsourcename.c -> ../newsourcename.c in each shadow directory,
> otherwise the "make" step ends in error with "No rule to make
> newsourcename.c. Stop.".
>
> Do you think the following rule (added to the src/Makefile which is
> soft-linked from every shadow directory) could take care of my problem
> without undesired side-effects?
>
> *: ../$@
> ln -sv ../$@
>
> The idea is to create a link if a needed file exist inthe parent (src)
> directory but not in the current (shadow) directory. Maybe the target
> sould be made less general to avoid using this rule out of turn.

What happens with "make foo"? Doesn't it execute "ln -sv ../foo" then?
Or say "../foo" does not exist"? Both are very confusing.

Perhaps we can the "ln" line from "make shadow" and add it with a
"shadowupdate" target:

shadowupdate:
ln -s ../*.[chm] ../*.in ../*.sh ../*.xs ../*.xbm ../gui_gtk_res.xml ../toolcheck ../proto ../libvterm ../vimtutor ../gvimtutor ../install-sh ../Make_all.mak .

Probably reports a lot of error for already existing links.

--
Statistics say that you can have a baby per month on average:
Just get nine women pregnant.

/// Bram Moolenaar -- Br...@Moolenaar.net -- http://www.Moolenaar.net \\\
/// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
\\\ an exciting new programming language -- http://www.Zimbu.org ///
\\\ help me help AIDS victims -- http://ICCF-Holland.org ///

[Tony Mechelynck]

1) Once and for all:
in the src directory: "make shadow" (or "SHADOWDIR=shadow-huge make -e
shadow" or similar) creates a shadow directory as a subdirectory of
the src directory and fills it with soft links to the sources in src
(see the "shadow" target ion src/Makefile for details)
2) Thereafter, I don't recreate the shadow directory every time: I cd
to the shadow directory and use it for incremental compiles, using the
Makefile in it (which is originally a copy of the src/Makefile but I
replace that by a symlink so updates to the Makefile are automatically
applied). If new source files were created _after_ I created the
shadow directory, they aren't automatically picked up.

Best regards,
Tony.

[Tony Mechelynck]

On Wed, Aug 28, 2019 at 10:19 PM Bram Moolenaar <Br...@moolenaar.net> wrote:
>
>
> > Recently you've been proposing (and Bram has accepted) to create a lot
> > of new source files to make the existing source easier to understand.
> > The notion is laudable but it doesn't work well with my shadow
> > directories: every time there is a new .c source (or .h; .pro sources
> > are taken care of automatically by virtue of their being in a special
> > src/proto directory for which a symlink has ben creted once and for
> > all in the shadow directory), I have to manually create a symlink
> > newsourcename.c -> ../newsourcename.c in each shadow directory,
> > otherwise the "make" step ends in error with "No rule to make
> > newsourcename.c. Stop.".
> >
> > Do you think the following rule (added to the src/Makefile which is
> > soft-linked from every shadow directory) could take care of my problem
> > without undesired side-effects?
> >
> > *: ../$@
> > ln -sv ../$@
> >
> > The idea is to create a link if a needed file exist inthe parent (src)
> > directory but not in the current (shadow) directory. Maybe the target
> > sould be made less general to avoid using this rule out of turn.
>
> What happens with "make foo"? Doesn't it execute "ln -sv ../foo" then?
> Or say "../foo" does not exist"? Both are very confusing.

"make foo" wouldn't work because there is no ../foo (i.e. no src/foo).
But maybe we should restrict the target to *.c and *.h. What I
wouldn't want is to add an unnecessary prerequisite for, for instance,
"phony" targets such as "install" or "reconfig".

I don't usually write Makefiles though I can more or less understand
those already written, so I suppose my proposed rule needs
fine-tuning. Normally (barring typing errors) this proposed rule would
be invoked _only_ as an implicit rule to make some prerequisite of
some *.o object needed for the link phase.

Once the softlink has been created, it will of course not be created
again, because thereafter the src/shadow/something.c exists (as a
symlink) and has the exact same date as its prerequisite, the
src/something.c to which the link points.

>
> Perhaps we can the "ln" line from "make shadow" and add it with a
> "shadowupdate" target:
>
> shadowupdate:
> ln -s ../*.[chm] ../*.in ../*.sh ../*.xs ../*.xbm ../gui_gtk_res.xml ../toolcheck ../proto ../libvterm ../vimtutor ../gvimtutor ../install-sh ../Make_all.mak .
>
> Probably reports a lot of error for already existing links.

Yeah, and BTW src/Make_all.mak is a very short file, only defining the
variable TAGS.

I think we can afford to assume that ../proto ../Makefile ../runtime
and some others are already correctly set (by virtue of "make shadow"
having been run at some point in the past). If possible, I'd like to
avoid "useless work" trying to recreate already existing links,
especially if it produces lots of error messages. What I propose is
that the make program will either (a) find that the *.c or *.h exists
in the shadowdir, has (by virtue of how the symlink was created) the
same date as its only prerequisite (the file by the same name one
level higher) and therefore doesn't need to be remade; or (b, less
often) find that it is not present in the shadowdir but is present one
level higher, and therefore can be remade by creating a symlink.

Best regards,
Tony.

[Tony Mechelynck]

P.S. The reason I use shadow directories is that it allows me to
compile several Vim builds with different config settiings (I started
with a Tiny "vi" build and a Huge "vim" build) without the one
throwing wrenches into the other's cogs — and also with the
possibility of incremental updates, i.e., rebuilding only what has
changed, with is what "make" is for.

Best regards,
Tony.

[Christian Brabandt]

using -f should force (re-creation) of the link and should not error.

Best,
Christian
--
Die Moral ist immer die letzte Zuflucht der Leute, welche die
Schönheit nicht begreifen.
-- Heidi Klum
(Erwin Pelzig: Falsch zugeordnete Zitate; eigentlich Oscar Wilde)

[Bram Moolenaar]

Assuming that the user hasn't replaced the link with something else, I
suppose it's not a problem to overwrite the link. I would prefer to
skip the existing files, but I don't see a flag for "ln" for that.

I can add the shadowupdate rule, but I do wonder how you keep Makefile
up-to-date, since that also needs to be changed.

--
All true wisdom is found on T-shirts.

[Tony Mechelynck]

OK, what about the following, which skips the existing files?

*.[chm] *.cc *.in *.sh *.xs *.xbm *.xml : ../$@
ln -sv ../$@

(links to directories are created by "make shadow" and don't need to
be created again; in the case of future directories we don't know what
they will be, so if new *subdirectories* are created I'll still have
to link them manually; but that happened much less often recently than
new .c sources). This won't recreate an already existing link because
"make" finds the link's date by following the link (otherwise
incremental compiles wouldn't work).

I intentionally set the -v (--verbose) switch to ln, in order to have
a trace of the link creation (which is not an everyday event) in the
log, so an alert user can check that it happened the way it should.

>
> I can add the shadowupdate rule, but I do wonder how you keep Makefile
> up-to-date, since that also needs to be changed.
>

I keep the src/shadow-*/Makefile up-to-date by replacing (once and for
all) the *copy* of the src/Makefile created by "make shadow" by a
*soft link* to ../Makefile. My config arguments are set by means of
environment variables, not by modifying the Makefile.

Best regards,
Tony.

[Bram Moolenaar]

AFAIK only GNU make accepts wildcards here, it doesn't work everywhere.

> (links to directories are created by "make shadow" and don't need to
> be created again; in the case of future directories we don't know what
> they will be, so if new *subdirectories* are created I'll still have
> to link them manually; but that happened much less often recently than
> new .c sources). This won't recreate an already existing link because
> "make" finds the link's date by following the link (otherwise
> incremental compiles wouldn't work).
>
> I intentionally set the -v (--verbose) switch to ln, in order to have
> a trace of the link creation (which is not an everyday event) in the
> log, so an alert user can check that it happened the way it should.
>
> >
> > I can add the shadowupdate rule, but I do wonder how you keep Makefile
> > up-to-date, since that also needs to be changed.
> >
> I keep the src/shadow-*/Makefile up-to-date by replacing (once and for
> all) the *copy* of the src/Makefile created by "make shadow" by a
> *soft link* to ../Makefile. My config arguments are set by means of
> environment variables, not by modifying the Makefile.

OK, that makes sense.

--
hundred-and-one symptoms of being an internet addict:
136. You decide to stay in a low-paying job teaching just for the
free Internet access.

[Tony Mechelynck]

I'm attaching a proposed patch. I wrote the recipe as "@test -e ./$@
|| ln -sv ../$@" rather than just the ln command to make 100% sure it
doesn't try to recreate an existing link. AFAICT it works for me. I
don't know why it rebuilds auto/pathdef.c, objects/pathdef.o and
objects/version.o even though they aren't supposed to have changed.

Best regards,
Tony.

[Bram Moolenaar]

Tony Wrote:

> I'm attaching a proposed patch. I wrote the recipe as "@test -e ./$@
> || ln -sv ../$@" rather than just the ln command to make 100% sure it
> doesn't try to recreate an existing link. AFAICT it works for me. I
> don't know why it rebuilds auto/pathdef.c, objects/pathdef.o and
> objects/version.o even though they aren't supposed to have changed.

As I mentioned before: wildcards in the target won't work with all make
programs. And also they usually only expand to existing files, not
missing files. BSD make mentions this explicitly:

Targets and sources may contain the shell wildcard values `?', `*', `[]',
and `{}'. The values `?', `*', and `[]' may only be used as part of the
final component of the target or source, and must be used to describe ex-
isting files.

--
hundred-and-one symptoms of being an internet addict:

146. You experience ACTUAL physical withdrawal symptoms when away
from your 'puter and the net.

[Tony Mechelynck]

On Sat, Aug 31, 2019 at 3:29 PM Bram Moolenaar <Br...@moolenaar.net> wrote:
>
>
> Tony Wrote:
>
> > I'm attaching a proposed patch. I wrote the recipe as "@test -e ./$@
> > || ln -sv ../$@" rather than just the ln command to make 100% sure it
> > doesn't try to recreate an existing link. AFAICT it works for me. I
> > don't know why it rebuilds auto/pathdef.c, objects/pathdef.o and
> > objects/version.o even though they aren't supposed to have changed.
>
> As I mentioned before: wildcards in the target won't work with all make
> programs. And also they usually only expand to existing files, not
> missing files. BSD make mentions this explicitly:
>
> Targets and sources may contain the shell wildcard values `?', `*', `[]',
> and `{}'. The values `?', `*', and `[]' may only be used as part of the
> final component of the target or source, and must be used to describe ex-
> isting files.
>

Ah, I didn't notice that.

Best regards,
Tony.