Copying Files on Windows

[bjornbytes]

Hello, I have a project that builds some smaller modules into shared libraries, and then copies those DLL files to the same folder as the final executable.  I'm having trouble copying those libraries on windows when they are nested in subfolders.

Here is a minimal project that illustrates the issue:

  > dir /B /S

  C:\test\a

  C:\test\a\b

  C:\test\a\b\c

  C:\test\Tupfile

  

  > more a\b\c

  hi

  

  > more Tupfile

  : a/b/c |> copy %f %o |> d/e/f

Attempting to build this results in the following:

  tup error: failed to create child process: No such file or directory

It took me some time to figure out that copy is not actually an exe file that exists in the PATH, but is instead a built in command.  I had to change the Tupfile to run the command inside cmd:

  : a/b/c |> cmd /c copy %f %o |> d/e/f

Now copy runs but it doesn't work properly because I'm using the wrong slashes.  I tried another change:

  : a\b\c |> cmd /c copy %f %o |> d\e\f

But tup invokes my command like this:

  cmd /c copy a\b/c d\e\f

Which understandably prints out "The syntax of the command is incorrect.".  I don't really fault tup for this or expect it to support the ugly windows slashes, but this could also be a bug, not sure.  At this point I think I'm stuck.  Questions/thoughts:

• Am I doing something wrong?  Is there an easier way to do this on windows or a way to work around it?  I could maybe install some packages (wsl/msys) that provide a sane unix cp, but I'm distributing this to others and would prefer it to minimize its dependencies.
• Should tup run commands inside a shell on windows?  I saw that there was a COMSPEC environment variable that has the path to cmd.exe, maybe that would be useful.
• I see that errno is used to try to provide better error messages when commands fail on windows, but perhaps this could be expanded.  I was stuck on the "no such file or directory" message for a while because I thought the paths provided to the copy command were invalid.
• Is this indeed an issue with path separators on windows?

I can submit pull requests for some of this stuff if that's helpful.

[Mike Shal]

On Sun, Feb 28, 2021 at 10:06 PM bjornbytes <bjorn...@gmail.com> wrote:

Hello, I have a project that builds some smaller modules into shared libraries, and then copies those DLL files to the same folder as the final executable.  I'm having trouble copying those libraries on windows when they are nested in subfolders.

Here is a minimal project that illustrates the issue:

  > dir /B /S

  C:\test\a

  C:\test\a\b

  C:\test\a\b\c

  C:\test\Tupfile

  

  > more a\b\c

  hi

  

  > more Tupfile

  : a/b/c |> copy %f %o |> d/e/f

Attempting to build this results in the following:

  tup error: failed to create child process: No such file or directory

It took me some time to figure out that copy is not actually an exe file that exists in the PATH, but is instead a built in command.  I had to change the Tupfile to run the command inside cmd:

  : a/b/c |> cmd /c copy %f %o |> d/e/f

Now copy runs but it doesn't work properly because I'm using the wrong slashes.  I tried another change:

  : a\b\c |> cmd /c copy %f %o |> d\e\f

But tup invokes my command like this:

  cmd /c copy a\b/c d\e\f

Which understandably prints out "The syntax of the command is incorrect.".  I don't really fault tup for this or expect it to support the ugly windows slashes, but this could also be a bug, not sure.  At this point I think I'm stuck.  Questions/thoughts:

• Am I doing something wrong?  Is there an easier way to do this on windows or a way to work around it?  I could maybe install some packages (wsl/msys) that provide a sane unix cp, but I'm distributing this to others and would prefer it to minimize its dependencies.

Nope, you're not doing something wrong here. Unix cp via msys/mingw (and probably wsl) should work, though I understand your concern of not wanting to require other dependencies, especially for something as minimal as copying a file. This is definitely something that should work with tup out-of-the box.

• Should tup run commands inside a shell on windows?  I saw that there was a COMSPEC environment variable that has the path to cmd.exe, maybe that would be useful.

It does run some commands using a cmd / bourne / bash shell depending on a few parameters:

1) Bash is used if the ^b flag is specified in the Tupfile (eg: |> ^b^ bashcmd |> )

2) Bourne shell (sh) is used if a ./ or a back-quote ` is in the command string

3) Windows command shell (cmd) is used if there are redirections/pipes with the &, |, <, or > characters

4) Otherwise, the command string is passed to CreateProcess() as-is with no shell added.

This is done in src/tup/server/windepfile.c in server_exec(). I think it would be reasonable here to look for built-in cmd-shell commands and make sure cmd is in the string as well, rather than rely on the developer specifying it in the Tupfile.

• I see that errno is used to try to provide better error messages when commands fail on windows, but perhaps this could be expanded.  I was stuck on the "no such file or directory" message for a while because I thought the paths provided to the copy command were invalid.

I agree this could use some work.

• Is this indeed an issue with path separators on windows?

Tup does a lot of work to make sure it is path-separator independent on Windows with regards to the paths it gets from file accesses. In other words, an open("foo/bar") and an open("foo\bar") in a subprogram would both result in tup getting a dependency on the same underlying file "bar" in the "foo" directory.

However, path-separators are more geared towards '/' in the parser, and '\' is used as an escape character to do things like escape spaces in paths. So you can't write a\b\c directly as a path in the Tupfile, it would need to be a\\b\\c. Although this is not ideal for native Windows users, I think it is reasonable for tup since its native internal path separator is '/'.

Whether or not paths are written with / or \ in the Tupfile itself, it still needs to work in the underlying command. Certainly a\b/c is obviously wrong :). Internally, tup separates filenames like a/b/c into a path part and a file part, so this would get split into a path of "a/b" and a file of "c". The '/' here gets put in by parser.c:set_path(), while the directory part is copied directly from what the Tupfile says. Probably what needs to happen is for tup to make sure that all paths are using '/' internally, and then when a cmd-shell invocation is detected, it can swap them for '\'. Does that sound reasonable?

I can submit pull requests for some of this stuff if that's helpful.

Sure, that would definitely be helpful to make sure it ends up working for your real project. I think most of the work of fixing \ would be in parser.c, while the error-handling and prefixing with cmd would be in windepfile.c (though prefixing with cmd could instead be detected and done in the parser, perhaps).

Thanks for the report!

-Mike