Tup on 64bit Windows

[Robert Pettersen]

Hi,

I've gotten Tup to work on 64bit Windows, even loading 32 bit applications.

It is not a very elegant solution, but it works. 

The reason 64bit loading on 64bit windows and 32bit loading on 32bit windows is straight forward is that Windows maps the kernel32.dll at the same address for all processes (until next reboot). So Tup can check what addresses the LoadLibraryA and GetProcAddress functions in that dll is and know that it will be the same for newly created processes, construct a small assembly stub and inject this into the new process forcing the new process to load the Tup dll using those function addresses.

This changes when loading 32 bits processes from a 64bit Tup. The way I found the 32bit addresses from the 64bit version of Tup, was to have a small program, compiled as 32bit, print them out. So when the 64bit tup needs to start a 32bit program, it first starts the small 32bit program and store the 32bit addresses to the functions in kernel32.dll. These are then used to load the 32bit version of the Tup dll.

Thus the 64bit version of Tup needs both the 32bit and 64bit version of the DLL, if both types of programs are to be started from Tup.

One caveat I haven't figured out yet is command chaining. (i.e. "cmd1 && cmd2") Tup encapsulates command chanining on Windows in a cmd.exe wrapper. But cmd.exe on 64bit Windows seems reluctant to injection, or would need some special handling. I have noticed that the process version for cmd.exe differs from the other 64bit applications, but according to documentation I have read this should only pertain to an extra MSDOS header saying that this application cannot run on ms-dos...

I solved this by having a python script handing command chaining with Popen for now.

Binaries are compiled on Ubuntu using x86_64-w64-mingw32 as 64bit compiler and i686-w64-mingw32 for the 32bit components.

[Mike Shal]

Hi Robert,

On Fri, Nov 8, 2013 at 10:54 AM, Robert Pettersen <nju...@gmail.com> wrote:

Hi,

I've gotten Tup to work on 64bit Windows, even loading 32 bit applications.

It is not a very elegant solution, but it works. 

That's awesome!! I've tried it out on my simple test case and it seems to work.

 

The reason 64bit loading on 64bit windows and 32bit loading on 32bit windows is straight forward is that Windows maps the kernel32.dll at the same address for all processes (until next reboot). So Tup can check what addresses the LoadLibraryA and GetProcAddress functions in that dll is and know that it will be the same for newly created processes, construct a small assembly stub and inject this into the new process forcing the new process to load the Tup dll using those function addresses.

This changes when loading 32 bits processes from a 64bit Tup. The way I found the 32bit addresses from the 64bit version of Tup, was to have a small program, compiled as 32bit, print them out. So when the 64bit tup needs to start a 32bit program, it first starts the small 32bit program and store the 32bit addresses to the functions in kernel32.dll. These are then used to load the 32bit version of the Tup dll.

Thus the 64bit version of Tup needs both the 32bit and 64bit version of the DLL, if both types of programs are to be started from Tup.

That sounds like a reasonable approach to me, though the Windows part was never my strong suit (which is why I haven't been able to get 64-bit programs working...)

 

One caveat I haven't figured out yet is command chaining. (i.e. "cmd1 && cmd2") Tup encapsulates command chanining on Windows in a cmd.exe wrapper. But cmd.exe on 64bit Windows seems reluctant to injection, or would need some special handling. I have noticed that the process version for cmd.exe differs from the other 64bit applications, but according to documentation I have read this should only pertain to an extra MSDOS header saying that this application cannot run on ms-dos...

I solved this by having a python script handing command chaining with Popen for now.

I run my tests on a 64-bit Windows 7 VM, but even there it seems cmd.exe is a 32-bit program. Is this the case on your machine as well? Are you loading the 32-bit DLL for it? Maybe it has something to do with the WoW64 layer?

I don't know if we need cmd.exe specifically, but we do need some way to support &&, ||, and the redirection operators < and >. Those are the cases where tup automatically adds cmd.exe into the mix. Can you explain a bit what the python script does? Is that effectively a mini-shell.py?

Let me know if you want to get this in the tree - I would need a license agreement (here: http://gittup.org/tup/icla.txt ) and of course a patch or github repo somewhere :)

Thanks!

-Mike

[Robert Pettersen]

Hi Mike,

Sorry for the late reply, beeing swamped with work here. We are currently porting our old makefile system to tup, for an operating system and all its applications. And unfortunately half our developers are on FreeBsd and half on Windows 8, so have taken some time porting all the tools. Just finished the auto-tools last night (generate tupfiles etc.). So we are switching today :)

One caveat I haven't figured out yet is command chaining. (i.e. "cmd1 && cmd2") Tup encapsulates command chanining on Windows in a cmd.exe wrapper. But cmd.exe on 64bit Windows seems reluctant to injection, or would need some special handling. I have noticed that the process version for cmd.exe differs from the other 64bit applications, but according to documentation I have read this should only pertain to an extra MSDOS header saying that this application cannot run on ms-dos...

I solved this by having a python script handing command chaining with Popen for now.

I run my tests on a 64-bit Windows 7 VM, but even there it seems cmd.exe is a 32-bit program. Is this the case on your machine as well? Are you loading the 32-bit DLL for it? Maybe it has something to do with the WoW64 layer?

We are running on Windows 8, and here cmd.exe is 64 bit. But the file format is "pei-x86-64", not "pe-x86-64" as is the case for the 64bit applications that work. I will investigate more when I get around.

 

I don't know if we need cmd.exe specifically, but we do need some way to support &&, ||, and the redirection operators < and >. Those are the cases where tup automatically adds cmd.exe into the mix. Can you explain a bit what the python script does? Is that effectively a mini-shell.py?

Attached the cmd.py script we are using at the moment.

 

Let me know if you want to get this in the tree - I would need a license agreement (here: http://gittup.org/tup/icla.txt ) and of course a patch or github repo somewhere :)

Yes, have to clean up the code a bit first, and fix the agreement and github repo.

--

Robert

 

[Mike Shal]

Hi Robert,

On Thu, Dec 5, 2013 at 5:13 AM, Robert Pettersen <nju...@gmail.com> wrote:

Hi Mike,

Sorry for the late reply, beeing swamped with work here. We are currently porting our old makefile system to tup, for an operating system and all its applications. And unfortunately half our developers are on FreeBsd and half on Windows 8, so have taken some time porting all the tools. Just finished the auto-tools last night (generate tupfiles etc.). So we are switching today :)

No worries - I have been pretty swamped as well. How did the switch go?

 

One caveat I haven't figured out yet is command chaining. (i.e. "cmd1 && cmd2") Tup encapsulates command chanining on Windows in a cmd.exe wrapper. But cmd.exe on 64bit Windows seems reluctant to injection, or would need some special handling. I have noticed that the process version for cmd.exe differs from the other 64bit applications, but according to documentation I have read this should only pertain to an extra MSDOS header saying that this application cannot run on ms-dos...

I solved this by having a python script handing command chaining with Popen for now.

I run my tests on a 64-bit Windows 7 VM, but even there it seems cmd.exe is a 32-bit program. Is this the case on your machine as well? Are you loading the 32-bit DLL for it? Maybe it has something to do with the WoW64 layer?

We are running on Windows 8, and here cmd.exe is 64 bit. But the file format is "pei-x86-64", not "pe-x86-64" as is the case for the 64bit applications that work. I will investigate more when I get around.

Any luck with cmd.exe? I think it would be great if that would work with the 64-bit support as well, though maybe the python script could be provided as a work-around in the meantime. People have been asking for 64-bit support for a while now, so any progress we can make there would be very welcome.

 

 

Let me know if you want to get this in the tree - I would need a license agreement (here: http://gittup.org/tup/icla.txt ) and of course a patch or github repo somewhere :)

Yes, have to clean up the code a bit first, and fix the agreement and github repo.

Thanks Robert - please keep me posted!

-Mike

[Tristan Rybak]

Hi Mike, Robert,
Any news on a patch for 64bit windows?
I am using the executable attached to first post and it's working great (very big thanks for that Robert!)
but I'd like to have new features and bugfixes introduced later too please.
Maybe I could help with integrating the fix into repository somehow?
Regards,
Tristan

[Mike Shal]

Hi Tristan,

I don't think I've seen the patch from Robert yet, though maybe I've missed it. I'd be happy to get it included if he's interested in getting it upstream.

Thanks,

-Mike

[Robert Pettersen]

Hi, 

Really sorry about not posting any updates, been swamped with work and just got back from vacation. I don't have time to setup a git repo and commit the code, means I'll actually have to clean-up and comment the code. But I have attached a diff between the current origin/HEAD and the changes I made to get it working on 64bit host windows. Haven't made any changes since the initial work, since it has been stable for our use. Be warned tho, I have not tested this on a 32bit host system, it will however execute 32bit binaries on a 64bit host system.

~/tup64$ cat tup.config

CONFIG_TUP_MINGW=x86_64-w64-mingw32 

I can probably answer some questions about the code and help out some here on the forum, but I can't guarantee their timelyness. Really swamped with work. 

Again, really sorry for not posting responses here sooner!

[Magnus Bodin]

Hi all,

I tried to run the provided files and even tried to compile tup myself without luck. Am I the only one not getting this to work? If I run tup from the PowerShell I get the printouts below. Running from the cmd.exe gives the same behavior as running tup compiled from the current branch.

PowerShell:

* 1) newmath: D:\mingw\bin\gcc -Wall -O2 -c square.c -o square.o

Code_Size: 160

SH Eip: 77B201B4

0028E770 | 68EFBEAD DE9C6068 EFBEADDE B8EFBEAD | h.....`h........

0028E770 | 68B401B2 779C6068 A0002B00 B864002B | h...w.`h..+..d.+

Code_Size: 160

SH Eip: 77B201B4

0028F100 | 68EFBEAD DE9C6068 EFBEADDE B8EFBEAD | h.....`h........

0028F100 | 68B401B2 779C6068 A0000B00 B864000B | h...w.`h.....d..

 *** tup errors ***

tup error: Expected to write to file 'square.o' from cmd 27 but didn't

Command Prompt:

* 1) cl -Wall -O2 -Inewmath -c hello.c -o hello.o

 *** tup errors ***

 *** Command ID=25 failed with return value -1073741819

[Magnus Bodin]

Seems like it doesn't matter what I try to run with tup on windows. No program can be started at all regardless of the combination of tup binaries (64-bit above or from current), Command Prompt, or Windows version (7-64-bit, Server 2003 32-bit, Windows 8 Preview). I always get one of 3 types of error:

1. tup error: failed to create child process: No such file or directory

2. *** tup errors ***

*** Command ID=27 failed with return value -1073741819

3. tup error: failed to inject dll: No such file or directory

The command I am trying to run is: "copy hello.c hello.o" (Can't get any simpler than that, right?)

I am running tup fine on Ubuntu but on Windows it's more or less impossible to get it to work. Anyone have any tip? What I am I doing wrong?

I have been running tup on Ubuntu just to test it out. It looks really promising but to make use of tup in our real build environment I need to get it to run on Windows.

[Spooge]

I have no idea what I'm talking about, but did you use mingw-w64 or the original mingw to build tup?  If you used the original, I think it only compiles 32 bit binaries and that may cause issues.

I used tup some on Windows but it was only with cl and link, using the vanilla tup windows binaries.

--
--
tup-users mailing list
email: tup-...@googlegroups.com
unsubscribe: tup-users+...@googlegroups.com
options: http://groups.google.com/group/tup-users?hl=en
---
You received this message because you are subscribed to the Google Groups "tup-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tup-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Magnus Bodin]

I have tried building myself with mingw-w64 + the patch above and also built vanilla tup with i686-w64-mingw32. Also tried with the binaries attached earlier in this thread and with the ones provided from http://gittup.org/tup/. Nothing works. I found that I get different errors depending on which command I test with, copy or xcopy. I have also tried cl and even gcc. That's were I started but when I ran into errors I tried to make it as simple as possible. 

Most likely I made some simple mistake but I can't find it so I think I have to give up and find something else to use other than tup. It bugs me since tup is what I want to use but Windows support is a deal breaker for me.

[Tristan Rybak]

Hi,

I managed to compile and make tup run with the 64bit patch. I did the following:

- compiled 32bit tup from git and saved generated tup-dllinject.dll as tup-dllinject32.dll in my 64bit tup folder

- copied tup32detect.exe from the original 64bit tup archive to my 64bit tup folder

- applied a patch to the tup code

- created tup.config with CONFIG_TUP_MINGW=x86_64-w64-mingw32

- compiled patched source and copied tup.exe and tup-dllinject.dll to my 64bit tup folder

with those four files I was able to use tup on 64bit windows with both 32 and 64 bit compilers/tools.

Hope it helps.

Cheers,

Tristan

On Fri, Aug 8, 2014 at 2:39 PM, Magnus Bodin <magnus....@gmail.com> wrote:

I have tried building myself with mingw-w64 + the patch above and also built vanilla tup with i686-w64-mingw32. Also tried with the binaries attached earlier in this thread and with the ones provided from http://gittup.org/tup/. Nothing works. I found that I get different errors depending on which command I test with, copy or xcopy. I have also tried cl and even gcc. That's were I started but when I ran into errors I tried to make it as simple as possible. 

Most likely I made some simple mistake but I can't find it so I think I have to give up and find something else to use other than tup. It bugs me since tup is what I want to use but Windows support is a deal breaker for me.

--

--
tup-users mailing list
email: tup-...@googlegroups.com
unsubscribe: tup-users+...@googlegroups.com
options: http://groups.google.com/group/tup-users?hl=en
---
You received this message because you are subscribed to the Google Groups "tup-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tup-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Tristan Rybak
Chief Technical Officer

Geniusprogrammer.net

[Freddie Chopin]

Just to clarify - standard binaries from the website work on Windows as long as the application being run is 32-bit. System can be 64-bit. I'm using tup for over a year on Windows 7 64-bit with gcc.

Regards,
FCh

[Robert Pettersen]

Ah yes, forgot to mention that on 64bit you will need the 32bit dll for 32bit applications, and the 64bit dll for 64bit applications. Just as Tristan explained below.

Source for tup32detect.exe is:

#include <windows.h>
int main(int argc, char** argv)
{
        int ll = (int) LoadLibraryA;
        int gpa = (int) GetProcAddress;
        printf("%x-%x\n", ll, gpa);
        ExitProcess(0);
}

The reason behind this is that the 32bit addresses for LoadLibraryA and GetProcAddress on 64bit windows changes between reboots, so we need to retrieve them somehow. As I said in the first post, this is not a very elegant solution, but it works.

[Magnus Bodin]

Thanks for the help everyone but nothing I try seems to work. I got a co-worker to compile tup as well (with no instructions from me) and he got the same errors. Tup only worked on 32-bits windows for him too. So, I will try to find some other build tool we can use. Maybe I'll dig into the dll-injection problem myself in the future if I get the time. I will at least keep an eye on tup, waiting for proper windows support.

[Robert Pettersen]

Hmm, I see you are using the i686 prefix on the compiler toolchain. I used x86_64.

 

CONFIG_TUP_MINGW=x86_64-w64-mingw32

Try if that helps. 

[Freddie Chopin]

W dniu 2014-08-27 13:46, Robert Pettersen pisze:

> Hmm, I see you are using the i686 prefix on the compiler toolchain. I
> used x86_64.
>
> CONFIG_TUP_MINGW=x86_64-w64-mingw32
>
>
> Try if that helps.

I suspect that is the main issue - 1686 toolchain produces 32-bit
binaries and to work with 64-bit tools tup binary needs to be 64-bits -
so it must be compiled with 64-bit toolchain x86_64.

Regards,
FCh

[Magnus Bodin]

Thanks for the suggestion but I have already tried that.

[Ben Robinson]

Hello,

Is there any progress on Windows64 support?  I was working through the examples on the website, and noticed that after adding unused.txt as an input to the rule which produces output.txt, it does not appear correctly in the graph of dependencies.  I suspect TUP did not correctly detect the file activity, due to my running on 64-bit Windows.

The online example claims that TUP will model this dependency, but because it detected that the file was not actually read when evaluating the rule, it will model it differently.  See the attached PNG showing that TUP is unaware of the unused.txt dependency on the rule.

I have attached a zip containing the example files as instructed to make on the website, with small changes to run on Windows (.sh -> .bat, etc...).

Feedback is greatly appreciated,

Ben

[tup_...@gmx.eu]

> From: Ben Robinson [icare...@gmail.com]
> Sent: Tuesday, October 07, 2014 1:08 AM

> Is there any progress on Windows64 support? I was working through the
> examples on the website, and noticed that after adding unused.txt as an
> input to the rule which produces output.txt, it does not appear correctly
> in the graph of dependencies. I suspect TUP did not correctly detect the
> file activity, due to my running on 64-bit Windows.

As I understand it, the tup.exe from http://gittup.org/tup/win32/ is a 32-bit
binary. This means on 64-bit versions of Windows it will run on top of the
WoW64 layer (Windows 32-bit on Windows 64-bit), see
http://en.wikipedia.org/wiki/WoW64. As long as your Tupfiles only call 32-bit
programs you should be safe. I'm using the vanilla 32-bit tup.exe from v0.7.3-3
with a 32-bit tool-chain on Windows 7 Enterprise 64-bit without issues like
yours.

Tup monitors file accesses by injecting a special DLL into to the created sub-
processes. Without changes to the tup code, using 32-bit tup to monitor 64-bit
programs won't work. The same would be valid for 64-bit tup trying to monitor
32-bit programs. Both versions would fail to inject the DLL.

I think what Robert tried to do, was to build a 64-bit tup.exe which would be
able to inject the 32-bit version of the DLL into 32-bit programs and the 64-bit
version of the DLL into 64-bit programs.

If you want to see which file accesses tup is able to see, try something like
"tup -d -j1". You should see many messages like this:
"[tup.updater:6208 file.c:88] received file 'C:\data\Workspace\tup_example\output.txt' in mode 1"

According to tup source code the numbers after "mode" signify the type of file
access: 0 = read, 1 = write, 2 = rename, 3 = delete.

> The online example claims that TUP will model this dependency, but because it
> detected that the file was not actually read when evaluating the rule, it will
> model it differently. See the attached PNG showing that TUP is unaware of the
> unused.txt dependency on the rule.

I'm not sure if the example at http://gittup.org/tup/ex_dependencies.html makes
any sense. What you try to do is adding a generated file as an input dependency
to a command which doesn't use this generated file at all. Why should tup care
about this link at all? There can't be any effect on output.txt if unused.txt
was changed.

If on the other hand you changed test.bat to read from unused.txt you would have
to add this new dependency on unused.txt. Tup will error out if you omit this.
As soon as you add the new dependency tup will know the correct order of
commands.

I don't think this behaviour has anything to do with 32-bit vs. 64-bit Windows.

Kind regards, Bernhard.

[Ben Robinson]

Bernhard,

Thank you for your reply.  You are correct, for the reasons you explained, TUP is not able to monitor file operations of 64-bit processes.  This is a serious bug since I cannot guarantee that I will never need to invoke a 64-bit process.

For me, one significant advantage of this build system is that it can produce provably correct builds.  Such a build system should not have a blind spot to a subset of file operations, in my humble opinion.

When is this bug expected to be fixed?

I will re-post my feedback on the example in another thread.  At the very least, the example should behave as the example claims it will.

Cheers,

Ben

[Mike Shal]

Hi Robert, thanks so much for working on this! I think we are fairly close.

I've tried to incorporate this patch and produce both the tup-dllinject.dll (64-bit) and tup-dllinject32.dll from the same tree. To build binaries from Linux, you'll need both a CONFIG_TUP_MINGW and CONFIG_TUP_MINGW32 in tup.config:

CONFIG_TUP_MINGW=x86_64-w64-mingw32
CONFIG_TUP_MINGW32=i686-w64-mingw32

I have a few questions so far, all pretty inter-related:

1) Why did you want to force have_shell=1 in windepfile.c? That causes many of the tests to fail, so if it is truly necessary I'll have to find a way to fix those tests (there are many). Or if there is some particular case that doesn't work without it, I'll want to add a test to check for it (and possibly fix it a different way)

2) Should it be possible to inject the 64-bit DLL into a 64-bit process from a 32-bit process? For example, if I run "sh -c './64-test.exe'", then sh (a 32-bit process) gets injected with a 32-bit dll, but 64-test.exe (a 64-bit process) doesn't get injected. (Whereas if I just run 64-test.exe directly, it gets injected fine since tup.exe is 64-bit). Is this why you added have_shell=1 above?

3) Am I compiling the dllinjection code correctly? I left in the @(TUP_32_BIT) code that you had commented out. (To be honest I'm not sure what it's used for - I can't find the email thread or issue that talks about it). Specifically, do I need to use both x86_64-w64-mingw32 and i686-w64-mingw32? Or is there some way with -m64 / -m32 / -somethingelse that we can just use x86_64-w64-mingw32?

4) The 64-bit DLL injection code won't compile with the 32-bit compiler, since the CONTEXT structure no longer has the .Rip field. I put an #ifdef around it, but I hit the "Error: Shouldn't be hooking here for the 32-bit dll" message (something I added to your patch) when using sh -c './64-test.exe' from question 2). Is there a better way to handle that?

Hopefully we can get these sorted out & in the master branch soon.

Thanks again,

-Mike

--

[Mike Shal]

I should have mentioned, the WIP is in the 'win64-2' branch: https://github.com/gittup/tup/tree/win64-2

[Robert Pettersen]

Hi Mike,

1) See 2)

2) 32bit processes can not inject into 64bit processes. cmd.exe on 64bit windows is a bit of magic, haven't really figured it out. But that was the reason I put have_shell=1 yes. We have a special case where we chain a 32bit compiler with a 64bit compiler, and tup fails to detect output written. But the expected behaviour is achieved.

3) You are right, compiling with the 64bit compiler with the -m32 flag seem to produce the expected output.

4) Think we have to ifdef the entire function, or atleast pepper it somewhat. Wow64 is loading 32bit binaries from 64bit host (yeah, naming is a bit confusing), and the other case is 64bit from 64bit, but as you have seen, the function does not contain a proper loading of 32bit from 32bit. See the attached patch-diff for a suggestion. Have tested it and it seems to work, with the exception of the cmd chaining. Maybe you have better luck now that you know more about it.

--

Robert

[Robert Pettersen]

Small followup, seems it doesnt matter if I chain commands or not, using cmd.exe to invoke commands will fail to detect output, even tho they execute correctly.

* 1) cmd /C gcc -c -o test.o test.c

 *** tup errors ***

tup error: Expected to write to file 'test.o' from cmd 20 but didn't

Perhaps cmd.exe starts sub-processes in a way we haven't hooked for...

And just noticed that one of the debug hooks provide erronous information in tup_inject_dll:

DEBUG_HOOK("%s is WOW64: %i\n", GetCommandLineA(), bWow64);

gives you the calling process, witch in most cases is tup.exe, to get the name of the process that tup starts up we need

TCHAR buffer[1024];

if (GetModuleFileNameEx(lpProcessInformation->hProcess,0,buffer,1024)){

        DEBUG_HOOK("Executable: %s\n", buffer);

}

[Robert Pettersen]

Ok, found it. Need to hook NtCreateUserProcess in ntdll.dll as well, since cmd.exe apparently uses this instead of the normal CreateProcess family for creating processes.

Attached is a diff with this hook implemented, have tested it and it works well, even with command chaining.

--

Robert

[Christian Åkerblom]

This is great news!

Could I ask you to upload the binary files since I don't have access to any system capable of compiling tup for Windows?

[Robert Pettersen]

Sure. This one have a partial implementation of run-scripts for Windows as well, so beware if you are using that.. It works, but dependency on the run-script itself is not present yet. So if the script changes, tup will not detect it.. Working on it.

Btw, anyone looked into variants? That is next on my list, so if anyone have started it I would love to join in.

--

Robert

[Christian Åkerblom]

Thank you!

I'm very interested in variants for Windows. I can probably support you with testing it out if you add the functionality.

[Robert Pettersen]

Sweet. Have sorted the run-scripts now, and almost done with variants. Just need a few more tests in the dll for detecting the tup environment correctly. Will probably have a version for you to test tomorrow.

--

Robert

[Mike Shal]

On Mon, Nov 3, 2014 at 8:51 AM, Robert Pettersen <nju...@gmail.com> wrote:

3) You are right, compiling with the 64bit compiler with the -m32 flag seem to produce the expected output.

Are you able to link tup32detect.exe and tup-dllinject32.dll with the 64bit compiler and -m32 as well? I was looking to avoid having to install & set the separately 32-bit compiler in tup.config entirely.

Thanks for working on the patch! I'm taking a look at it now.

-Mike

[Robert Pettersen]

No. Or, you could probably get it to work, but you would need the 32bit libraries from the 32bit compiler anyway. The linker stage will fail since x86_64-w64-mingw32-gcc has the 64bit versions of the libraries in its search path.

Just got variants on windows working now btw, just doing some code cleanup and some tests and I'll have another patch for you.

BTW, the fopen hook, doesn't it chain to NtCreateFile on Windows 7? It does so on 8 and 10 atleast, so for me it's redundant.

--

Robert

[Robert Pettersen]

Ok, variants for 64bit Windows ready for testing. These binaries will output some "ok.txt" files here and there with debug statements incase there are bugs. If it crashes, please post the file and possibly the Tupfile and I'll have a look at it. 

Run-scripts in variants does not work yet, but I don't have time to look at it right now, have to fix that tomorrow.

--

Robert

[Christian Åkerblom]

I did a quick test of the variant supporting version but unfortunately it crashes alot for me. All crashes seems to happen randomly and due to different reasons but no ok.txt files are outputted. It crashes during the output of the directory structure at random places. The most frequent errors seems to be c0000374 and c0000005.

It also outputs a lot of garbage characters every now and then in the console, as well as beeping randomly.

I'm running tup with Lua configuration files at the moment.

[Robert Pettersen]

Yeah, I'm working on it now. I only had time for a quick basic test with single source files on those binaries from friday before I had to run.

Have sorted most of the errors now, just one memory issue remaing.

[Spooge]

On 11/11/2014 03:25 AM, Robert Pettersen wrote:
> Yeah, I'm working on it now. I only had time for a quick basic test
> with single source files on those binaries from friday before I had to
> run.
> Have sorted most of the errors now, just one memory issue remaing.

I'm not doing any Windows development ATM, but I have had to do some in
the past and no doubt will again. Having all these Tup features working
would be a huge asset. A little late, but thanks for taking the time to
work on this stuff!

[Robert Pettersen]

Ok, testing round 2. Included are one which outputs debug information and one that doesn't, if it breaks use the one with debug and hopefully I'll get a hint where to fix stuff.

I have tried it on a fairly big project, utilizing most of Tup's features (and some hacks I think), with the exception of lua. And it works for me atleast.

I'm a bit worried that older versions of windows uses some other paths through the API that I might not have caught, but I think it should work. I hear that windows 7's cmd.exe calls CreateProcessA instead of Windows 8 and 10's NtCreateUserProcess, but in worst case it'll only be a bit slower. If you run Windows 7 and it breaks please post the debugging "ok.txt"s from the directory where it fails, and I'll look into it as well.

There is one issue which I'm not fully satisfied with tho. When working with variants, the dll needs to build some data structures to keep track of source files and path rewriting, and using malloc, or even allocating a separate heap causes heap corruptions which I'm not able to track down. Running it in debuggers causes no corruptions for some reason... I ended up writing my own memory allocator, and that worked.

--

Robert

[Magnus Bodin]

Sweeet!!! First time ever I got tup to work on Windows. Great! Thanks!

[Robert Pettersen]

Found a bug in Windows 8 and earlier when using chaining or 32bit applications, where injection sometimes failed. Attached a fixed version.

--

Robert

[Robert Pettersen]

Fixed some new bugs.

Added /D to cmdstr to avoid autoloading addons that might cause odd behavior.

Fixed compatibility with non-intel CPUs in the injected code, by removing a REX.W prefix.

Re-hauled run-scripts to locate dependencies when the script is located in another folder.

When the script has correct dependency, the run script will be re-evaluated if updated.

Global scripts are also supported, but will not get a dependency. Trying to ../../ out of

source directory will however fail, since it is probably not what you want.

Works, and will add dependency:

run folder/script.sh

run ../../scipts/script.sh

run ./script.sh

Works, but will assume global script, and no dependency added:

run script.sh

Made the dll smaller by coalescing all hooks pertaining to creating processes.

Source at https://github.com/njubee/tup/tree/win64-2 for those who wants.

--

Robert

[Zahary Karadjov]

I've managed to rebase Robert's code against the latest version of Tup and created a pull request here:
https://github.com/gittup/tup/pull/283

So far, I've only tested it with simple Tupfiles, but I'll be porting one fairly complicated project from Linux soon.