variants are dead - long live variants

[Mike Shal]

Hi all,

For those who use variants, and/or are interested in variants in Windows, can you give the 'explicit-variant' branch on github a try?

Some background:

The way variants were implemented has been a pain, because of the way it relied on the dependency checker to overlay the output directory with the source directory. This worked well with FUSE, but it's a pain to do in Windows and is preventing us from switching to LD_PRELOAD for OSX to avoid FUSE there.

The reason for the overlay was so that you wouldn't have to make any changes to your Tupfiles in order to support variants - if it worked as an in-tree build, it would work as a variant. However, if we relax that condition, then all of the requirements on the dependency checker go away. The downside is you may need to give tup hints when things need to be found in the source directory. For example, if you previously had a Tupfile like so:

sub/Tupfile:

: foreach *.c |> gcc -c %f -o %o -Iinclude |> %B.o

When gcc goes to open files in the 'include' directory, tup would use the overlay to pull files from the source directory. Now however, the '-Iinclude' only corresponds to the directory inside the variant. To tell tup that this is really in the source directory, use $(TUP_SRCDIR) -

: foreach *.c |> gcc -c %f -o %o -I$(TUP_SRCDIR)/include |> %B.o

This evaluates to a path like '../sub' inside a variant, or just '.' if no variants are used. Anything expanded with %-flags should work out of the box because tup knows if the file is a normal file (in the source directory), or a generated file (in the variant directory).

So I'm curious if just peppering $(TUP_SRCDIR) in the right places is sufficient to get this style of variants working in your setup. I'm sure there are still bugs or other cases I'm not considering, but most of the tests seem to work so far with fairly minimal changes. I don't know how that will scale to a live setup, though. Please let me know!

Thanks,

-Mike

[Ben Boeckel]

On Fri, Oct 02, 2015 at 21:23:14 -0400, Mike Shal wrote:
> The way variants were implemented has been a pain, because of the way it
> relied on the dependency checker to overlay the output directory with the
> source directory. This worked well with FUSE, but it's a pain to do in
> Windows and is preventing us from switching to LD_PRELOAD for OSX to avoid
> FUSE there.

An aside, but how is LD_PRELOAD going to work on 10.11? System binaries
(e.g., Python and likely the compiler) will no longer load unsigned
Mach-O files if SIP is enabled[1] (the default).

--Ben

[1]http://arstechnica.com/apple/2015/09/os-x-10-11-el-capitan-the-ars-technica-review/8/#h1

[Mike Shal]

On Tue, Oct 6, 2015 at 2:49 PM, Ben Boeckel <math...@gmail.com> wrote:

On Fri, Oct 02, 2015 at 21:23:14 -0400, Mike Shal wrote:
> The way variants were implemented has been a pain, because of the way it
> relied on the dependency checker to overlay the output directory with the
> source directory. This worked well with FUSE, but it's a pain to do in
> Windows and is preventing us from switching to LD_PRELOAD for OSX to avoid
> FUSE there.

An aside, but how is LD_PRELOAD going to work on 10.11? System binaries
(e.g., Python and likely the compiler) will no longer load unsigned
Mach-O files if SIP is enabled[1] (the default).

Hmm, good question. Though it sounds like OSXFUSE also won't work with SIP enabled, so I don't think we lose anything here by switching to a shared library shim:

"OS X users who depend upon unsigned KEXTs for extra functionality—be it third-party hardware with unsigned drivers or software like OSXFuse that depends upon unsigned kernel extensions—will have precisely two alternatives: stop using the applications or hardware until the developers provide signed KEXTs, or disable SIP altogether."

I haven't tried either yet, though.

-Mike

[Thomas Gahr]

Hello!

I just tried it and tup.exe warns me:
"Unable to retrieve WOW64 info"
...and subsequently errors out because it does not detect that the .o files are written.

The windows VM is Win7 64bit, the linux host is Arch Linux 64bit, where I have mingw-w64 cross-toolchain installed.
For building tup, I created tup.config with:

CONFIG_TUP_MINGW=x86_64-w64-mingw32
CONFIG_TUP_MINGW32=i686-w64-mingw32

I'm attaching the output of tup.exe -d as well as an archive of my test project

Hop this helps!

[Thomas Gahr]

Dammit, I think this was just my fault, my PATH was not set correctly. As soon as I add the folder with *.exe and *.dll of tup to PATH, it works like a charm - at least for my small test ;)

[Mike Shal]

On Thu, Nov 26, 2015 at 9:20 AM, Thomas Gahr <tom....@gmail.com> wrote:

Dammit, I think this was just my fault, my PATH was not set correctly. As soon as I add the folder with *.exe and *.dll of tup to PATH, it works like a charm - at least for my small test ;)

Thanks for the feedback! It seems not having tup32detect.exe in PATH is what was causing https://github.com/gittup/tup/issues/219, which I was having trouble reproducing. That should be fixed now though.

-Mike

[Pat Pannuto]

I had a little less success with this.

I'm trying to build https://github.com/lab11/M-ulator/tree/master/simulator

When I used the explicit-variant branch, the first complaint tup had came from this Tupfile: https://github.com/lab11/M-ulator/blob/master/simulator/cpu/common/private_peripheral_bus/Tupfile#L3

It said that the ppb.c and ppb.h files were unspecified output files, which I fixed by changing the output of that rule to:

: gen_registers.py | exceptions *.conf |> python %f *.conf |> $(TUP_VARIANTDIR)/ppb.c $(TUP_VARIANTDIR)/ppb.h

Next it identified that the explicitly name file ppb.h was scheduled to be deleted, easy fix, change the next line to:

: ppb.c | $(TUP_VARIANTDIR)/ppb.h |> !cc |> %B.o ../../../<objs>

Next, it had the same complaint about this Tupfile: https://github.com/lab11/M-ulator/blob/master/simulator/cpu/Tupfile#L3

That is, that common/private_peripheral_bus/ppb.h was scheduled to be deleted. So I changed that rule to

: foreach *.c | $(TUP_VARIANTDIR)/common/private_peripheral_bus/ppb.h |> !cc |> %B.o ../<objs>

Unfortunately, that gave:

tup error: Unable to use inputs from a generated directory (1899) that isn't written to by this Tupfile.

I'm not really sure how to reconcile this?

I thought it was a little bit weird that I had to specify TUP_VARIANTDIR on the outputs at all, felt like something that tup should've been able to figure out.

Tested off latest from explicit-variant branch.

Happy to try other things, let me know how else I can help.

-Pat

--
--
tup-users mailing list
email: tup-...@googlegroups.com
unsubscribe: tup-users+...@googlegroups.com
options: http://groups.google.com/group/tup-users?hl=en
---
You received this message because you are subscribed to the Google Groups "tup-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tup-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Freddie Chopin]

On Thursday 10 of December 2015 02:33:37 Pat Pannuto wrote:
> Unfortunately, that gave:
>
> tup error: Unable to use inputs from a generated directory (1899) that
> isn't written to by this Tupfile.
>
>
> I'm not really sure how to reconcile this?

You can solve this problem by using <groups> instead of explicitly named
files. Search the list - I wrote about that some time ago in a thread about
multiple output groups.

Regards,
FCh

[Mike Shal]

On Wed, Dec 9, 2015 at 9:33 PM, Pat Pannuto <ppan...@umich.edu> wrote:

I had a little less success with this.

I'm trying to build https://github.com/lab11/M-ulator/tree/master/simulator

When I used the explicit-variant branch, the first complaint tup had came from this Tupfile: https://github.com/lab11/M-ulator/blob/master/simulator/cpu/common/private_peripheral_bus/Tupfile#L3

It said that the ppb.c and ppb.h files were unspecified output files, which I fixed by changing the output of that rule to:

: gen_registers.py | exceptions *.conf |> python %f *.conf |> $(TUP_VARIANTDIR)/ppb.c $(TUP_VARIANTDIR)/ppb.h

Next it identified that the explicitly name file ppb.h was scheduled to be deleted, easy fix, change the next line to:

: ppb.c | $(TUP_VARIANTDIR)/ppb.h |> !cc |> %B.o ../../../<objs>

Next, it had the same complaint about this Tupfile: https://github.com/lab11/M-ulator/blob/master/simulator/cpu/Tupfile#L3

That is, that common/private_peripheral_bus/ppb.h was scheduled to be deleted. So I changed that rule to

: foreach *.c | $(TUP_VARIANTDIR)/common/private_peripheral_bus/ppb.h |> !cc |> %B.o ../<objs>

Unfortunately, that gave:

tup error: Unable to use inputs from a generated directory (1899) that isn't written to by this Tupfile.

I'm not really sure how to reconcile this?

I thought it was a little bit weird that I had to specify TUP_VARIANTDIR on the outputs at all, felt like something that tup should've been able to figure out.

Right, you shouldn't need to use TUP_VARIANTDIR in either the input or output list. However, the command needs to either use TUP_VARIANTDIR or the %-flags in some way in order for tup to communicate where the output file should go. Here's a simple example:

# Tupfile A

: |> touch foo.txt |> foo.txt

# Tupfile B

: |> touch %o |> foo.txt

Tupfile A won't work with the explicit-variants, since all commands run in the source directory, so 'touch foo.txt' creates foo.txt in the source dir instead of the variant dir. In Tupfile B, the '%o' flag is expanded to something like 'build/foo.txt', so the file is written inside the variant. You could also do:

: |> touch $(TUP_VARIANT)/foo.txt |> foo.txt

But that looks strange IMO. TUP_VARIANTDIR is still needed in some cases, such as for -I flags:

: |> generate.py %o |> foo.h

: foreach *.c | foo.h |> gcc -c %f -o %o -I$(TUP_VARIANTDIR) |> %B.o

This would add something like '-I../../build/sub/dir' so that it could find the generated foo.h (without a variant, TUP_VARIANTDIR is like TUP_CWD).

In your case, you'll need to pass something in to gen_registers.py to tell it where to write the files. I've attached a patch as an example that should work with explicit variants.

Though now that I'm writing this, maybe it makes sense to have explicit variants assume that the outputs will be written in the srcdir, and then just move them to the variant directory? So it'd be something like:

python gen_registers.py *.conf

(writes ppb.h and ppb.c as normal)

mv ppb.c ../../../../build/simulator/cpu/common/private_peripheral_bus/ppb.c
mv ppb.h ../../../../build/simulator/cpu/common/private_peripheral_bus/ppb.h

But that would mean variants couldn't be built in parallel easily on all platforms.

-Mike

[Pat Pannuto]

Ah, now I understand.

While in most cases I think that specifying the output is fine, I'm sure there are naive tools out there that someone will want to use that won't let you specify an output path. Naive tools notwithstanding, it would also be very surprising / confusing for users when something basic-looking like    : |> touch foo.txt |> foo.txt    suddenly doesn't work with variants.

I think having the non-parallel build/move path as a fallback is a good solution. Tup could emit a warning, something to the effect of, "WARN: <quote rule>, Rules with fixed output paths cannot build variants in parallel. To allow parallel builds, specify the output using %o in the rule, or explicitly using $(TUP_VARIANTDIR)/output_file"

--

[Mike Shal]

On Thu, Dec 10, 2015 at 11:14 AM, Pat Pannuto <ppan...@umich.edu> wrote:

Ah, now I understand.

While in most cases I think that specifying the output is fine, I'm sure there are naive tools out there that someone will want to use that won't let you specify an output path. Naive tools notwithstanding, it would also be very surprising / confusing for users when something basic-looking like    : |> touch foo.txt |> foo.txt    suddenly doesn't work with variants.

I think having the non-parallel build/move path as a fallback is a good solution. Tup could emit a warning, something to the effect of, "WARN: <quote rule>, Rules with fixed output paths cannot build variants in parallel. To allow parallel builds, specify the output using %o in the rule, or explicitly using $(TUP_VARIANTDIR)/output_file"

Thinking about this some more, would tools be likely to break if they output foo.o, but then tup moves the file to build/foo.o after the fact? I'd like to implement the generate & move function since I think it would work more out-of-the-box as compared to having to use %o everywhere, but I'd hate to be in the same position with broken gdb (or other tools) afterwards.

-Mike

[Pat Pannuto]

What if you went the other direction? i.e. copy / hard-link the source files into the variant directories and execute the commands in the variant directories? This would also allow things to run in parallel still.

--

[Mike Shal]

On Fri, Dec 18, 2015 at 2:57 PM, Pat Pannuto <ppan...@umich.edu> wrote:

What if you went the other direction? i.e. copy / hard-link the source files into the variant directories and execute the commands in the variant directories? This would also allow things to run in parallel still.

Hmm, that's a good idea, though I think the amount of copying it would have to do on Windows would make it prohibitive there. Especially since tup wouldn't know ahead of time what files it would need to copy, so effectively it would have to copy everything.

-Mike

[Pat Pannuto]

Modern windows does have support for symlinks (aka junctions) and hard links, though I've never worked with them. Could be an interesting experiment to try?

--

[Ben Boeckel]

On Fri, Dec 18, 2015 at 20:37:13 +0000, Pat Pannuto wrote:
> Modern windows does have support for symlinks (aka junctions) and hard
> links, though I've never worked with them. Could be an interesting
> experiment to try?

Needs admin privs to create them. Basically, to ensure that old Windows
apps aren't tripped up by the different behaviors caused by them.

Also, junctions and symlinks are different beasts (junction is closer to
mount while symlinks can't be relative and point outside the drive
letter).

https://msdn.microsoft.com/en-us/library/windows/desktop/aa365006%28v=vs.85%29.aspx

--Ben

[Pat Pannuto]

(exposing my Windows ignorance): Is there an analog of suid that tup could use so that it could create these without admin-prompting every run, or is this simply not a good path to try to go down on Windows?

[Ben Boeckel]

On Fri, Dec 18, 2015 at 20:58:14 +0000, Pat Pannuto wrote:
> (exposing my Windows ignorance): Is there an analog of suid that tup could
> use so that it could create these without admin-prompting every run, or is
> this simply not a good path to try to go down on Windows?

I don't think it does; apps ask for an admin user/password combo if they
need it.

--Ben