Strange error when targeting x64 using Visual Studio 2010

[Jay]

Hi All,

Anyone have any luck in re-compiling CUDPP v2.0 to 64-bit along with CUDA v5.0?  I was able to compile CUDPP as 32-bit without any issues.

(1) Followed the link (http://code.google.com/p/cudpp/wiki/BuildingCUDPPwithCMake) to use Cmake to create a Visual Studio solution file.

(2) Followed the link below and made a few minor changes to cuda_util.h in order to fix any compiling and linking errors.

How to: Configure Visual C++ Projects to Target 64-Bit Platforms
http://msdn.microsoft.com/en-us/library/9yb4317s(v=vs.100).aspx


(3) After fixing the compile and link errors from step #2, when using "Build Solution" it now returns the error below.  It seems to be originating from inside the VS environment.

2>------ Build started: Project: cudpp, Configuration: Debug x64 ------
2>TRACKER : error TRK0002: Failed to execute command: "D:\apps\vs10\VC\bin\amd64\CL.exe "@C:\Users\Bob\AppData\Local\Temp\18dc0578852343efaae48b1b61846a65.rsp"". The handle is invalid.
2> 

Note: VS2010 is installed in D:\apps\vs10 instead of the usual default.  D:\apps\vs10\VC\bin\amd64 contains the 64-bit versions of the compiler and linker.

 
Found similar issues via Google, but ResGen.exe error workaround #2 did not work.  Still researching the other workarounds since I'm not familiar with Visual Studio.
 
Unable to build C/++ 64 Bit Projekt with Visual Studio 2010: TRK0002 ... The handle is invalid http://social.msdn.microsoft.com/Forums/vstudio/en-US/275fe355-478b-4b76-842c-91f94886d736/unable-to-build-c-64-bit-projekt-with-visual-studio-2010-trk0002-the-handle-is-invalid

ResGen.exe error: An attempt was made to load a program with an incorrect format
http://blogs.msdn.com/b/visualstudio/archive/2010/06/18/resgen-exe-error-an-attempt-was-made-to-load-a-program-with-an-incorrect-format.aspx?PageIndex=5#comments
 

My config:
Visual Studio 2010 Professional Service Pack1
Windows 7 Pro (64-bit)
i7-2500K (not overclocked)
Nvidia GTX 590

Thanks!

Jay

[Jay]

Finally able to compile CUDPP in VS2010 using CUDA 5.0 and 64-bit.  Many problems when trying to compile using Visual Studio's native 64-bit compiler (..\VC\bin\amd64\..) vs their cross compiler (..\VC\bin\x86_amd64\..).  I believe if the cross compiler was used it would avoid most of the problems below but there are some tradeoffs (http://stackoverflow.com/questions/3508173/when-compiling-x64-code-whats-the-difference-between-x86-amd64-and-amd64) using the cross-compiler.

Began by following MS instructions (http://msdn.microsoft.com/en-us/library/9yb4317s(v=vs.100).aspx) to target 64-bit CUDA and point everything to CUDA 5.0 *.dll and *.libs.  Unfortunately it was not that easy.  Below are are bunch of problems encountered and how to fix.  A detailed build config list has been included as well.  Note that I'm using some non-default directories for some files.


VS2010 Compile errors
===========================
237 error: identifier "IUnknown" is undefined reported in C:\Program Files(x86)\Microsoft SDKs\Windows\v7.0A\ObjBase.h.  It can be resolved by making the following modifications:

extern "C++"
{
/* Add the following includes to resolve */
#include <wtypes.h>
#include <unknwn.h>

    template<typename T> void** IID_PPV_ARGS_Helper(T** pp)
    {
        static_cast<IUnknown*>(*pp);    // make sure everyone derives from IUnknown
        return reinterpret_cast<void**>(pp);
    }
}

VS2010 Linker errors
=====================
msvcprtd.lib(MSVCP100D.dll) : fatal error LNK1112: module machine type 'x64' conflicts with target machine type 'X86'

Ensure there is no mix of x86 (32-bit) and x64 (64-bit) object code.  It is likely there is some 32-bit *.lib or *.dll being used.  The dumpbin.exe utility included with VS can confirm.  Enabling the Project properties->Linker /VERBOSE option will also list the libraries being used.

dumpbin /headers cudpp_generated_compact_app.cu.obj

Dump of file cudpp_generated_compact_app.cu.obj

File Type: COFF OBJECT

FILE HEADER VALUES
             14C machine (x86) /*32-bit, it would show x64 for 64-bit */
         
For some unknown reason the cmake files in the CUDPP source directory (..\src\cudpp\CMakeFiles\cudpp.dir\app\cudpp_generated_compact_app.cu.obj.make) generate 32-bit even though in the VS Project properties 64-bit is being explicitly specified. 

I'm not familar with Cmake.  Modifying all the cmake config files I could find but this did not help.  This was worked around by manually editing each of the cudpp_generated_*cu.obj.make files.  VS seems to call cmake for each rebuild and thus overwites the *cu.obj.make files.



VS Project Build Config
===================
CUDA 5.0 is installed in d:\devel\cuda\v5.0
VC++ Directories

Executable Directories
 $(VCInstallDir)bin\amd64
 $(WindowsSdkDir)bin\NETFX 4.0 Tools\x64
 $(WindowsSdkDir)bin\x64
 $(VSInstallDir)Common7\tools
 $(VSInstallDir)Common7\ide
 $(FrameworkSDKDir)\bin\x64
 $(FrameworkSDKDir)\lib\x64
 c:\windows\Microsoft.NET\Framework64
 c:\windows\Microsoft.NET\Framework64\v4.0.30319

Include Directories
 $(VCInstallDir)include
 $(VCInstallDir)atlmfc\include
 $(WindowsSdkDir)include
 $(FrameworkSDKDir)\include
 d:\devel\cuda\cudpp_src_2.0\include
 d:\devel\cuda\v5.0\include
 d:\devel\cuda\cudpp_src_2.0\src\cudpp_hash


Reference Directories
 $(WindowsSdkDir)include
 $(VCInstallDir)atlmfc\lib\amd64
 $(VCInstallDir)lib\amd64

Library Directories
 $(FrameworkSDKDir)\lib\x64
 $(VCInstallDir)lib\amd64
 $(VCInstallDir)atlmfc\lib\amd64
 $(WindowsSdkDir)lib\x64
 d:\devel\cuda\v5.0\lib\x64

Source Directories
 $(VCInstallDir)atlmfc\src\mfc
 $(VCInstallDir)atlmfc\src\mfcm
 $(VCInstallDir)atlmfc\src\atl
 $(VCInstallDir)crt\src\AMD64

Exclude Directories (defaults)
 $(VCInstallDir)include
 $(VCInstallDir)atlmfc\include
 $(WindowsSdkDir)include
 $(FrameworkSDKDir)\include
 $(MSBuildToolsPath)
 $(MSBuildToolsPath32)
 $(VCInstallDir)atlmfc\lib\amd64
 $(VCInstallDir)lib\amd64


General
  Target Extension: .dll
  Configuration Type: Dynamic Library (.dll)
 
C/C++ -> Preprocessor Definitions: Add "cudpp_EXPORTS"
CUDA C/C++ -> Target Machine Platform: 64-bit (--machine 64)

Linker
 Show Progress->Display all messages (/VERBOSE)
 Input
  Additional Dependencies: cudart.lib cuda.lib
 Additional Library Directories
  $(FrameworkSDKDir)\lib\x64
  $(VCInstallDir)lib\amd64
  $(WindowsSdkDir)lib\x64
  d:\devel\cuda\v5.0\lib\x64

 Advanced
  Import Library: $(TargetDir)$(TargetName).lib

 System
   Stack Reserve Size: 3
   Enable Large Address Aware: Yes
 Advanced
  Randomized Base ADdress:  /DYNAMICBASE:NO
  Target Machine /MACHINE:X64
 
 
CUDA Linker
  Additional Library Directories
  $(FrameworkSDKDir)\lib\x64
  $(VCInstallDir)lib\amd64
  $(WindowsSdkDir)lib\x64
  d:\devel\cuda\v5.0\lib\x64

Manifest Tool
 Input and Output
   Embed Manifest: No

[Jay]

Forgot to list the TRACKER errors

VS2010 Tracker errors
=======================
TRACKER : error TRK0002: Failed to execute command: "D:\apps\vs10\VC\bin\amd64\Lib.exe "@C:\Users\Bob \AppData\Local\Temp\tmpc80987c922614e18b19b8d0211d67562.rsp"". The operation identifier is not valid.
 
TRACKER : error TRK0002: Failed to execute command: "D:\apps\vs10\VC\bin\amd64\CL.exe "@C:\Users\Bob\AppData\Local\Temp\a7ebd297870b47e3b71359a51171c8d6.rsp"". The handle is invalid.

The errors above are a known MS bug documented below which MS will not fix (http://connect.microsoft.com/VisualStudio/feedback/details/505682/x64-c-c-projects-cannot-compile).  It can be worked around by following the instructions in the link.

1) Modify C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Platforms\x64\Microsoft.Cpp.x64.targets

2) Replace $(LinkTrackerFrameworkPath) with C:\Windows\Microsoft.NET\Framework64\v4.0.30319\
and replace $(LinkTrackerSdkPath) with C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\bin\NETFX 4.0 Tools\x64\
so these lines becomes:

  TrackerFrameworkPath             ="C:\Windows\Microsoft.NET\Framework64\v4.0.30319\"
  TrackerSdkPath                   ="C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\bin\NETFX 4.0 Tools\x64\"