Variants on Windows

411 views
Skip to first unread message

Sergio Garces

unread,
Aug 30, 2012, 3:00:48 PM8/30/12
to tup-...@googlegroups.com
Any update on the status of variants on Windows?

What's the main problem to get this working?

This is one of the main obstacles preventing us from adopting tup, and we'd be happy to help get it implemented if we can.

Sergio.

Mike Shal

unread,
Sep 1, 2012, 3:59:33 PM9/1/12
to tup-...@googlegroups.com
Hi Sergio,

On Thu, Aug 30, 2012 at 3:00 PM, Sergio Garces <sga...@gmail.com> wrote:
> Any update on the status of variants on Windows?

Sorry, I haven't had time to make progress here - people keep finding
other things that are broken :).

>
> What's the main problem to get this working?

You can see my last attempt if you clone from github in the
tmp-file-tree branch. The way variants work in Linux/OSX is as
follows: we have a source tree, which is the root of your project and
all sub-directories, and variant trees, which are any top-level
directories that have a tup.config file in them. We may have a Tupfile
like so:

src/Tupfile:
: |> gcc -c foo.c -o foo.o |> foo.o

In a normal build, tup would cd to src/, and then run that gcc
command. In a variant build, it will cd to build-foo/src, and then run
the gcc command. The fuse fs knows that when gcc tries to open foo.c,
it may be either in build-foo/src (if it was generated by another
rule), or in src (for all normal files). It is setup to transparently
allow gcc to do open("foo.c") and have it pull the file from the
source tree as necessary. Additionally, output files are temporarily
mapped to the .tup/tmp/ directory, so if several sub-processes all try
to open a "tmp.o" file (think old-school yacc and the like), they will
each get their own copy and won't conflict. After dependency checking,
tup moves the temporary files to their real location in build-foo/.

I was trying to duplicate this behavior in Windows, which on the
surface didn't seem too tricky since we already wrap all of the file
open & creation routines for dependency detection. However, I ran into
some trouble when sub-processes do (I think) the equivalent of stat()
on UNIX. The last commit in tmp-file-tree was where I tried to pull
out the remapping code from fuse so it could be shared with Windows,
but the NtQueryDirectoryFile function was causing problems. With the
remapping code in place I think it would be easier to add variants,
since the "if file is not in build tree, go to source tree" logic can
be handled in the mapping code, rather than in each hooked function.

>
> This is one of the main obstacles preventing us from adopting tup, and we'd
> be happy to help get it implemented if we can.

I could definitely use some help here, as Windows programming is not
my best. If you aren't sure where to start, let me know and I can give
it another try and maybe get a more detailed question for you (like
why does NtQueryDirectoryFile do X and not Y?). Also I do ask for a
signed contributor license agreement if you want to contribute code:
http://gittup.org/tup/icla.txt

Thanks,
-Mike

Sergio Garces

unread,
Sep 5, 2012, 1:05:39 PM9/5/12
to tup-...@googlegroups.com
That makes sense. I don't have any experience with fuse but I'll look at your last submission, see if I can figure out how it works and if I can make it run.

I'll let you know.

Sergio.

Sergio Garces

unread,
Sep 6, 2012, 5:58:38 PM9/6/12
to tup-...@googlegroups.com
Hmm, I'm having trouble getting the code to compile on Windows. Not sure exactly how to go about it.

I tried MinGW, but it seems to be missing a bunch of stuff, including mmap.h and other POSIX libraries.
Cygwin also failed to compile. It reported some compile errors which I could try to look into, but I have a feeling the released binary is not using cygwin, as it doesn't depend on the .dll.

My Linux/Unix skills are pretty rusty, any guidance on how to get the Windows version to compile?

Sergio.

Mike Shal

unread,
Sep 14, 2012, 10:26:26 AM9/14/12
to tup-...@googlegroups.com
On Thu, Sep 6, 2012 at 5:58 PM, Sergio Garces <sga...@gmail.com> wrote:
> Hmm, I'm having trouble getting the code to compile on Windows. Not sure
> exactly how to go about it.
>
> I tried MinGW, but it seems to be missing a bunch of stuff, including mmap.h
> and other POSIX libraries.
> Cygwin also failed to compile. It reported some compile errors which I could
> try to look into, but I have a feeling the released binary is not using
> cygwin, as it doesn't depend on the .dll.
>
> My Linux/Unix skills are pretty rusty, any guidance on how to get the
> Windows version to compile?
>
> Sergio.

Hi Sergio,

I have never tried to build it natively on Windows. Tup's own Tupfiles
are configured to assume it is building on Linux/OSX, and can be
configured to additionally compile for Windows. So you can build:

1) Linux
2) Linux+Windows with mingw
3) OSX
(maybe OSX+Windows too? I've never tried it)

This is just how I've written tup's Tupfiles, and not necessarily
limitations of tup itself.

To build on Linux + Windows, you'll need mingw. In gentoo I use
crosstool, and I have gcc-4.5.2 installed (as I recall, I had issues
with a gcc-4.6.X version). This gives me executables like
'i686-pc-mingw32-gcc'. Then I put this in my tup.config:

CONFIG_TUP_MINGW=i686-pc-mingw32

Then when tup is built, it builds the executable for Linux, as well as
tup.exe and tup-dllinject.dll for Windows. I just scp those over to my
Windows VM to test out.

In other Linux distros there may already be mingw packages. For
example, ubuntu has a gcc-mingw32 package. You'll just want to try to
compile a simple hello-world.exe with it first to make sure it works
at all before trying tup (I've had some versions of mingw that only
generate exes that segfault immediately...)

In summary:

1) Install mingw in Linux
2) git clone git://github.com/gittup/tup.git
3) cd tup
4) echo 'CONFIG_TUP_MINGW=<your mingw prefix here>' > tup.config
5) ./bootstrap.sh
6) copy tup.exe & tup-dllinject.dll to Windows

Now you can edit files, and just run ./build/tup upd to rebuild your
current tree. I think that should be it - let me know if you still
have any issues getting it setup!

Thanks,
-Mike

Freddie Chopin

unread,
Mar 4, 2013, 10:58:26 AM3/4/13
to tup-...@googlegroups.com
Hi!

Any progress? Any plans? Any ideas? On github issues page someone is offering some help with that:

It would be great to have variants on Windows too (;

4\/3!!

Ben Robinson

unread,
Oct 22, 2014, 3:12:27 PM10/22/14
to tup-...@googlegroups.com
Any updates on this?  I am evaluating TUP for an existing large project, at a large company, where all development is done on Windows.

I know TUP will be an incredible replacement for our existing build system.  But if it doesn't work on Win64 with variants, TUP is unusable.

I really, really want to use it.  :)  Any updates?

Thank you.

Robert Pettersen

unread,
Nov 12, 2014, 4:36:29 PM11/12/14
to tup-...@googlegroups.com
See https://groups.google.com/forum/#!topic/tup-users/2Tw5YWh6wqE
There is a working version of tup for 64bit windows with variants.

--
Robert
Reply all
Reply to author
Forward
0 new messages