Re: tup and clang/llvm temporary files

[Christoph Harder]

Hello Mike,

1) the output of clang --version is:
clang version 10.0.0
Target: x86_64-pc-windows-msvc
Thread model: posix
InstalledDir: C:\Program Files\LLVM\bin

2) The output of lld-link --vrsion is:
LLD 10.0.0

3) cmd.exe
Microsoft Windows [Version 10.0.17763.1217]
Note, I'm just using the standard cmd.exe not the VC developer console or any other scripts to modify it.

Please find attached the minimal Tupfile.lua and the main.cc
On my System I put them under C:\tuptest\
Since clang on Windows doesn't ship it's own C++ STL it's a bit messy, especially the paths.

When running tup I do get the following output:

C:\tuptest>tup
[ tup ] [0.000s] Scanning filesystem...
[ tup ] [0.000s] Reading in new environment variables...
[ tup ] [0.000s] No Tupfiles to parse.
[ tup ] [0.000s] No files to delete.
[ tup ] [0.000s] Executing Commands...
* 0) "C:\Program Files\LLVM\bin\lld-link.exe" /dynamicbase /machine:x64 /subsystem:console /nodefaultlib /libpath:"C:\Program Files (x86)\Microsoft Visual Studio\2017\WDExpress\VC\Tools\MSVC\14.16.27023\lib\x64" /libpath:"C:\Program Files (x86)\Windows Kits\10\Lib\10.0.18362.0\ucrt\x64" /libpath:"C:\Program Files (x86)\Windows Kits\10\Lib\10.0.18362.0\um\x64" /out:"test.exe" main.obj libucrt.lib libvcruntime.lib libcmt.lib libcpmt.lib kernel32.lib user32.lib gdi32.lib comctl32.lib comdlg32.lib advapi32.lib hid.lib opengl32.lib
*** tup messages ***
tup error: File 'C:/tuptest/test.exe.tmp1a7550b' was written to, but is not in .tup/db. You probably should specify it as an output
-- Delete: C:/tuptest/test.exe.tmp1a7550b
*** Command failed due to errors processing the output dependencies.
[ ] 100%
*** tup: 1 job failed.


The test.exe file is created by clang.

By the way I'm using "Microsoft Windows 10 Enterprise 2019 LTSC" just in cas it's important.

Best regards,
Christoph


Am 11.05.2020 um 22:47 schrieb Mike Shal:
> Hi Christoph,
>
> There could be a / or \ issue somewhere, though tup should already
> normalize all the paths to use the same separators before doing
> comparisons. I did also notice that the error message contains E:, so that
> could potentially be an issue as well. Can you put together a small full
> example (Tupfiles and C files) that reproduces the issue? Also, can you let
> me know:
>
> 1) What does clang --version say?
> 2) lld-link --version?
> 3) What shell environment are you using? (Powershell, VS developer command
> prompt, cygwin, MinGW, etc)
>
> I'm having trouble producing the error now with the latest tup, so there's
> something else I'm missing that differs between our setups.
>
> Thanks,
> -Mike
>
> On Wed, May 6, 2020 at 11:25 PM Christoph Harder <shad...@arcor.de> wrote:
>
>> Hello Mike,
>>
>> thank you for your fast reply.
>>
>> I've tried tup-v0.7.8-37-ga8843c5c but still get the same message when
>> building when linking with lld-link.
>>
>> 3 warnings generated.
>> * 100% 0) System: "C:\Program Files\LLVM\bin\lld-link.exe" /dynamicbase
>> /machine:x64 /subsystem:windows /nodefaultlib /libpath:"C:\Program Files
>> (x86)\Microsoft Visual
>> Studio\2017\WDExpress\VC\Tools\MSVC\14.16.27023\lib\x64"
>> /libpath:"C:\Program Files (x86)\Windows Kits\10\Lib\10.0.18362.0\ucrt\x64"
>> /libpath:"C:\Program Files (x86)\Windows Kits\10\Lib\10.0.18362.0\um\x64"
>> /out:"lea.exe" main.obj lea_opengl.obj lea_splash_window.obj
>> versioninfo.res libucrt.lib libvcruntime.lib libcmt.lib libcpmt.lib
>> kernel32.lib user32.lib gdi32.lib comctl32.lib comdlg32.lib advapi32.lib
>> hid.lib opengl32.lib
>> *** tup messages ***
>> tup error: File 'E:/daten/projekte/lea/System/lea.exe.tmp9c02c97' was
>> written to, but is not in .tup/db. You probably should specify it as an
>> output
>> -- Delete: E:/daten/projekte/lea/System/lea.exe.tmp9c02c97
>> *** Command failed due to errors processing the output dependencies.
>> *** tup: 1 job failed.
>> *** Fehlgeschlagen ***
>>
>> Please note, the warnings are just unused parameter warnings, so they
>> shouldn't cause any problems.
>> Could the problem be related to / and \ ? E.g. could it be that tup
>> doesn't see the rename that lld-link is doing because of the different
>> directory seperators?
>>
>> Subscribing using the method you've provided (email to
>> tup-users...@googlegroups.com) did work without any problems,
>> thank you.
>>
>> Best regards,
>> Christoph
>>
>>
>> Am 07.05.2020 um 03:00 schrieb Mike Shal:
>>> Hi Christoph,
>>>
>>> Can you try the latest version here?
>>> http://gittup.org/tup/win32/tup-v0.7.8-37-ga8843c5c.zip
>>>
>>> The fix in the discussion you mentioned was included in the version you
>>> were testing, however, it appears that tup was not correctly matching up
>>> temporary files that used a different case in between. So for example,
>>> "touch tmp.txt; mv TMP.TXT bar.txt" would not be tracked properly.
>> Although
>>> I don't think clang was doing this directly, the way tup was tracking the
>>> current working directory could trigger this. Strangely, I could only
>>> reproduce this in a MinGW shell, whereas clang worked fine in a Cygwin
>>> shell.
>>>
>>> In any case, I added t4213 to help test this, and I think clang should
>> now
>>> work for you. Let me know if that is not the case.
>>>
>>> As far as the mailing list, I'm not completely sure if there's a way to
>>> join it without a Google account. From
>>>
>> https://www.mydigitallife.net/how-to-subscribe-or-join-google-groups-without-google-account/
>>> it sounds like you may be able to email
>> tup-users...@googlegroups.com
>>> to subscribe with your email address, though the Google help
>> documentation
>>> that the page links to doesn't confirm that suggestion so it may be out
>> of
>>> date. Also note that you don't need a GMail account, and can create a
>>> Google account with your existing email address. I don't recall that
>> there
>>> was this restriction when I setup the mailing list, any idea if it's
>>> something that changed? I likely would not have used Google Groups if I
>>> knew of that limitation in the first place.
>>>
>>> Thanks,
>>> -Mike
>>>
>>> On Tue, May 5, 2020 at 1:58 AM Christoph Harder <shad...@arcor.de>
>> wrote:
>>>
>>>> Hello,
>>>>
>>>> I'm trying to use tup to compile a project using clang and llvm on
>>>> windows, it does build, however tup always outputs an error message
>> because
>>>> an temporary file is used but not listed in the "Tupfile.lua".
>>>> My tup version is "tup v0.7.8-32-g335a1e15". When linking using
>>>> "lld-link.exe" a temporary file "<output>.exe.tmp<hex>" is generated
>> where
>>>> <hex> is a 7 digit hexadecimal code that changes each time.
>>>> After the build the temporary file no longer exists.
>>>>
>>>> The error message the tup gives is something like:
>>>> tup error: File '....exe.tmp5d3a8d5' was written to, but is not in
>>>> .tup/db. You probably should specify it as an output
>>>> -- Delete: ....exe.tmp5d3a8d5
>>>> *** Command failed due to errors processing the output dependencies.
>>>> *** tup: 1 job failed.
>>>>
>>>> I've found a similar problem on the mailling list
>>>> https://groups.google.com/forum/#!topic/tup-users/Q9_VgipYdpE but it'
>> for
>>>> an older version.
>>>> Is this fix incorporated in the newer builds?
>>>>
>>>> Also is there any way to use the mailling list without having to create
>> a
>>>> google user account?
>>>>
>>>> Thank you in advance.
>>>>
>>>> Best regards,
>>>> Christoph Harder
>>>>
>>>
>>
>

[Mike Shal]

On Wed, May 13, 2020 at 8:00 AM Christoph Harder <shad...@arcor.de> wrote:

Hello Mike,

1) the output of clang --version is:
clang version 10.0.0
Target: x86_64-pc-windows-msvc
Thread model: posix
InstalledDir: C:\Program Files\LLVM\bin

2) The output of lld-link --vrsion is:
LLD 10.0.0

Thanks, this seemed to be the part I was missing. I was still running clang 6...

The problem was that clang now creates 2 temporary files, one of which it sets delete-on-close with NtSetInformationFile using the FileDispositionInformation data, and the other one it renames to the final output with NtSetInformationFile using the FileRenameInformation data. Tup already handled the rename information (as part of supporting older versions of clang), but it did not handle the FileDispositionInformation as a method of deleting files. So it saw the file being created, but not that it was delete-on-close, and flagged it as an error.

I believe this is now handled properly in http://gittup.org/tup/win32/tup-v0.7.8-38-gd1a99c15.zip - can you give that a try? Hopefully we're getting closer :)

-Mike

[Christoph Harder]

Hello Mike,

thank you! That fixed it.

With the new version it works with the test project as well as my own project.
The error is no longer shown.

Best regards,
Christoph