can't build sdk examples on Windows - gcc hangs

[winte...@gmail.com]

I have a Windows 7, 64 bit machine and I am trying to build the examples from the SDK. It should be straightforward but it seems the compiler just hangs. The output in my console window  is as follows

c:\nacl_sdk\pepper_30\examples> make
c:/nacl_sdk/pepper_30/tools/make -C api all
make[1]: Entering directory 'c:/nacl_sdk/pepper_30/examples/api'
c:/nacl_sdk/pepper_30/tools/make -C audio all
make[2]: Entering directory 'c:/nacl_sdk/pepper_30/examples/api/audio'
    CXX newlib/Debug/audio_x86_32.o


In task manager, i686-nacl-g++.exe just sits there using 0 CPU time but totally hung. I can't even kill it.

Am I missing something? Any ideas, suggestions as to how to get around this or what could possibly be going on?

Thanks

[Don Winters]

This is still an issue for me. I can build the pnacl binaries but the newlib and glibc builds will hang. I've tried this on two Windows machines here at work and I get the same result. I've tried it outside my work network and it builds fine - could it be something with our network (??). Also I've tried with a Linux box, no problems.

I'm trying to do a proof of concept of this technology and right now I don't have much to report to my bosses. Any help or ideas on things to try or how to debug this would be greatly appreciated.

Thanks,

[Victor Khimenko]

On Mon, Oct 21, 2013 at 5:37 PM, Don Winters <winte...@gmail.com> wrote:

This is still an issue for me. I can build the pnacl binaries but the newlib and glibc builds will hang. I've tried this on two Windows machines here at work and I get the same result. I've tried it outside my work network and it builds fine - could it be something with our network (??).

Most likely some kind of BLODA: http://cygwin.com/faq/faq.html#faq.using.bloda

Do you know if CygWin works on your system at all? newlib and glibc toolchains are built with CygWIn, PNaCl is built with MinGW.

Also I've tried with a Linux box, no problems.

I'm trying to do a proof of concept of this technology and right now I don't have much to report to my bosses. Any help or ideas on things to try or how to debug this would be greatly appreciated.

Thanks,

On Wednesday, October 16, 2013 9:31:24 AM UTC-4, Don Winters wrote:

I have a Windows 7, 64 bit machine and I am trying to build the examples from the SDK. It should be straightforward but it seems the compiler just hangs. The output in my console window  is as follows

c:\nacl_sdk\pepper_30\examples> make
c:/nacl_sdk/pepper_30/tools/make -C api all
make[1]: Entering directory 'c:/nacl_sdk/pepper_30/examples/api'
c:/nacl_sdk/pepper_30/tools/make -C audio all
make[2]: Entering directory 'c:/nacl_sdk/pepper_30/examples/api/audio'
    CXX newlib/Debug/audio_x86_32.o


In task manager, i686-nacl-g++.exe just sits there using 0 CPU time but totally hung. I can't even kill it.

Am I missing something? Any ideas, suggestions as to how to get around this or what could possibly be going on?

Thanks

--
You received this message because you are subscribed to the Google Groups "Native-Client-Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to native-client-di...@googlegroups.com.
To post to this group, send email to native-cli...@googlegroups.com.
Visit this group at http://groups.google.com/group/native-client-discuss.
For more options, visit https://groups.google.com/groups/opt_out.

[Don Winters]

I've never had any problems with Cygwin - I don't use it extensively though. The security here at work is draconian - so this could very well be the problem. I'm not sure what to do about it though. But thanks a lot, at least I have an idea of what is going on.

[Don Winters]

Also, one workaround for me would be to just use the pnacl builds, but they don't seem to load in Chrome - I turned on the flag for pnacl support but still no joy. Is there a way to get the pnacl build to work in the Chrome browser?

Thanks,

[Victor Khimenko]

On Mon, Oct 21, 2013 at 6:31 PM, Don Winters <winte...@gmail.com> wrote:

Also, one workaround for me would be to just use the pnacl builds, but they don't seem to load in Chrome - I turned on the flag for pnacl support but still no joy. Is there a way to get the pnacl build to work in the Chrome browser?

They should work if you are using canary. Dev channel should also work by now - although you may need to wait a bit till PNaCl translator will be installed.

[Derek Schuff]

This is actually not true. PNaCl is built in the same cygwin sandbox that the x86 nacl-gcc is (arm-nacl-gcc is built with MinGW).

[Derek Schuff]

To load PNaCl .pexe files in Chrome, you need to have Chrome 31 (currently in beta channel) or better, and finalize the file using the pnacl-finalize tool (found in the same directory as pnacl-clang et al.). If you don't do that, you will see an error about the bitcode header in the Javascript console. If you did that and it still isn't working, it's possible that your Chrome install hasn't downloaded the translator yet (it's a separate component). Restart the browser and check about://nacl to see if it shows a path for the translator or if it reports not being installed.

You can use the pnacl-translate tool to generate .nexe files from pexe files. To translate a non-finalized pexe (e.g. with debug info), do 

pnacl-translate -arch x86-64 --allow-llvm-bitcode-input foo.pexe -o foo.nexe

(you can also do x86-32 and arm for the -arch flag). Then you can load, test, and debug the nexes locally through an unpacked extension or using the --enable-nacl flag, and distribute them through the Chrome Web Store (and you will be able to distribute finalized pexe files via the open web once Chrome 31 is pushed to the stable channel next month).

[Victor Khimenko]

On Mon, Oct 21, 2013 at 8:59 PM, Derek Schuff <dsc...@google.com> wrote:

This is actually not true. PNaCl is built in the same cygwin sandbox that the x86 nacl-gcc is (arm-nacl-gcc is built with MinGW).

Oops. This means it's not a cygwin issue and problem lies in our redirector.exe. This is tiny program which just adds few characters to the command line and then calls "real" cygwin-compiled program:

http://src.chromium.org/viewvc/native_client/trunk/src/native_client/tools/redirector.c

Could you call "x86_64-nacl-gcc --help" from libexec directory (NOT from bin directory)?

[Derek Schuff]

Sorry for the spam; I just noticed the wording here indicating you turned 'on' the flag for pnacl support; that means you're probably just using chrome 30 (in chrome 31, the flag turns *off* pnacl support, because it's on by default).

So, if you want to use pexe files in the browser, make sure you are using chrome 31 or better, and make sure you are using the Pepper 31 SDK or better. However if you just want to use offline translation to create nexe files, you should still be able to do that with the pnacl compiler in the pepper 30 SDK.

[Don Winters]

Thanks Derek, I can get pnacl examples to work. But I think I am more confused than ever about why the newlib and glibc builds are hanging. I will try to talk to our IT people and see if I can get a machine that is not loaded with their stuff - but I doubt if that will fly, and we have some mac guys in out group, I'll see if they can build.

[Don Winters]

khim, sorry, I missed this post - yes I can call "x86_64-nacl-gcc --help" from inside libexec. That works fine

--

--

[Victor Khimenko]

On Mon, Oct 21, 2013 at 11:29 PM, Don Winters <winte...@gmail.com> wrote:

khim, sorry, I missed this post - yes I can call "x86_64-nacl-gcc --help" from inside libexec. That works fine

Well, this means that software on your workstation somehow breaks the program which is literally 300 lines long and basically executes a single CreateProcessW function followed by WaitForSingleObject function.

You can use use cygwin-compiled programs directly if you'll take a look on this table:

  http://src.chromium.org/viewvc/native_client/trunk/src/native_client/tools/redirector.h

redirector.c primary purpose is to add one single argument to the command line of other, "real", tool (--32 or -m32 or something else - see details in the table above; some tools are called without change in the arguments at all). And it does that in pretty straightforward manner. Heck, it's so simple that it does not even use MSVC runtime! Apparently some of your security tools think it's TOO simple and break it.

We are using shell scripts for the purpose of command-line change on Linux and MacOS but on Windows it does not work since when you call .bat (or .cmd) file from another .bat (or .cmd) file it never returns back! Thus redirector.exe program. You can adjust your files to explicitly specify the required arguments instead (but this may cause problems if you'll try to call NaCl tools from cygwin).

[Don Winters]

Ok, I am going to look into this. BTW, we did test on a mac and there was no problem inside our network, it built fine - which was expected.

[Andrey Khalyavin]

You can also create a dump of the hang process using these instructions: http://blogs.msdn.com/b/debugger/archive/2009/12/30/what-is-a-dump-and-how-do-i-create-one.aspx and share it with us.

One more thing to try is pepper_31. We use different cygwin version there.

-- Andrey Khalyavin

[Don Winters]

The dump is about 14 Mb - too large to attach. How can I get it to you? Below is the relevant stack trace

    ntdll.dll!_ZwWaitForSingleObject@12 ()    Unknown
     ntdll.dll!_ZwWaitForSingleObject@12 ()    Unknown
     ntdll.dll!_ZwClose@4 ()    Unknown
     sysfer.dll!74937b12()    Unknown
     [Frames below may be incorrect and/or missing, no symbols loaded for sysfer.dll]   
     sysfer.dll!74938a90()    Unknown
     sysfer.dll!74938b20()    Unknown
     ntdll.dll!_ZwMapViewOfSection@40 ()    Unknown
     KernelBase.dll!755cec3d()    Unknown
     ntdll.dll!_ZwClose@4 ()    Unknown
     KernelBase.dll!755cc455()    Unknown
     ntdll.dll!_ZwClose@4 ()    Unknown
     KernelBase.dll!755cc455()    Unknown
     cygwin1.dll!610fe5fc()    Unknown
     cygwin1.dll!61031d59()    Unknown
     x86_64-nacl-g++.exe!004026e0()    Unknown
     x86_64-nacl-g++.exe!004163f6()    Unknown
     x86_64-nacl-g++.exe!00416633()    Unknown

[Don Winters]

BTW, pepper+31 hangs also.

[Andrey Khalyavin]

On Tuesday, October 22, 2013 4:24:06 PM UTC+4, Don Winters wrote:

The dump is about 14 Mb - too large to attach. How can I get it to you? Below is the relevant stack trace

 

You can try to pack it to zip/7z or share via google drive with halyavin at google dot com. BTW, it would be better to have a dump made with pepper_31.

Do you see other hang processes? You can use process explorer to see process tree.

This stacktrace belongs to gcc driver. There must be our redirector (x86_64-nacl-g++ process which launches different x86_64-nacl-g++ process) and there could be compiler (cc1.exe). If you see a compiler, its dump is more interesting that dump of gcc driver.

-- Andrey Khalyavin 

[Don Winters]

There are other process, in Process Explorer is looks like the following are in the cmd shell

Process     cmd.exe(4808)
Process     make.exe(2356)
Process     make.exe(5544)
Process     make.exe(4964)
Process     cmd.exe(6688)
Process     i686-nacl-g++.exe(4348)
Process     x86_64-nacl-g++.exe(2696)
Process     i686-nacl-g++.exe(1528)
Process     x86_64-nacl-g++.exe(2332)

but there is also a x86_64-nacl-as.exe running somewhere and resource monitor says it and PID 2332 (x86_64-nacl-g++.exe) are suspended. I don't see cc1.exe.

I've attached a debug dump zip file, I hope it helps.

BTW, google drive is off limits here at work - I can't get to any cloud storage from inside our network. Maybe I could get an exception.

Hopefully this will help.

[Andrey Khalyavin]

On Tuesday, October 22, 2013 10:22:50 PM UTC+4, Don Winters wrote:

There are other process, in Process Explorer is looks like the following are in the cmd shell

Process     cmd.exe(4808)
Process     make.exe(2356)
Process     make.exe(5544)
Process     make.exe(4964)
Process     cmd.exe(6688)
Process     i686-nacl-g++.exe(4348)
Process     x86_64-nacl-g++.exe(2696)
Process     i686-nacl-g++.exe(1528)
Process     x86_64-nacl-g++.exe(2332)

but there is also a x86_64-nacl-as.exe running somewhere and resource monitor says it and PID 2332 (x86_64-nacl-g++.exe) are suspended. I don't see cc1.exe.

 

Can you run make.exe with -j1? This will make clear which processes are hanged. i686-nacl-g++.exe is our redirector which launches x86_64-nacl-g++. x86_64-nacl-g++ is gcc driver which in turn launches compiler, assembler, linker and etc. You can see the chain more clearly using process explorer (http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx) instead of task manager which shows only process list instead of process tree.

I've attached a debug dump zip file, I hope it helps.

 

It opened successfully, I will take a look. 

Andrey Khalyavin

[Don Winters]

Andrey,

I've got to run today, but I ran make -j1 (I believe that was the request) and I am looking at it with Process Explorer, Resource Monitor and Task Manager.  The list from my previous post was from Process Explorer so there is nothing new to report there (maybe I am not using it right). But Resource Montitor is reporting that x86_64-nacl-as.exe is suspended. I don't know if that is helpful - I will look into this more tomorrow.

Thanks

[Andrey Khalyavin]

On Wednesday, October 23, 2013 12:45:48 AM UTC+4, Don Winters wrote:

Andrey,

I've got to run today, but I ran make -j1 (I believe that was the request) and I am looking at it with Process Explorer, Resource Monitor and Task Manager.  The list from my previous post was from Process Explorer so there is nothing new to report there (maybe I am not using it right). But Resource Montitor is reporting that x86_64-nacl-as.exe is suspended. I don't know if that is helpful - I will look into this more tomorrow.

I analyzed your x86_64-nacl-g++ dump. This process is waiting on as.exe to finish. as.exe is another redirector which in turn calls x86_64-nacl-as.exe. So the process tree should look like this (I replaced assembler with notepad++ to hang the assembler):

Can you confirm this? (I don't know why as.exe process is shown as the second i686-nacl-g++.exe process in process explorer). 

So the real problem must be in x86_64-nacl-as.exe process. I need his dump to understand the problem. It is better to send me a personal e-mail with the dump to save everyone who reads this list by e-mail from downloading large attachment.

-- Andrey Khalyavin

[Andrey Khalyavin]

Got you dump, looking into it.

-- Andrey Khalyavin

[Andrey Khalyavin]

Dump shows that x86_64-nacl-as.exe process is started under debugger. Since it hangs, this means that debugger doesn't continue its execution from initial breakpoint where none of x86_64-nacl-as.exe code is executed yet. I think it is antivirus software that started our process under debugger. It is still not clear why antivirus doesn't work correctly in our case though.

There is one unusual thing we are doing in NaCl SDK which may be related - we use hardlinks since all redirectors are the same executable. May be the following experiments will lit some light on the problem.

1. Try to run pepper_31\toolchain\win_x86_newlib\libexec\x86_64-nacl-as.exe --version.

2. If item 1 works, try to run pepper_31\toolchain\win_x86_newlib\x86_64-nacl\bin\as.exe --version.

3. If Item 2 works, execute the following script to break hardlink for as.exe:

cd pepper_31\toolchain\win_x86_newlib\x86_64-nacl\bin

move as.exe as.exe.tmp

copy as.exe.tmp as.exe

del as.exe.tmp

Then run make.bat for audio example and see whether it hangs in the same place. At least process explorer shows the process name correctly when I broke hard link.

If item 1 fails, you can just report x86_64-nacl-as.exe executable as false positive to your antivirus vendor. If item 2 fails, something really strange is going on. If build hangs in a different way after item 3, you should break all hard links in pepper_31\toolchain\win_x86_[newlib/glibc]\x86_64-nacl\bin and pepper_31\toolchain\win_x86_[newlib/glibc]\bin with move/copy/del. If it hangs the same way after item 3, I don't know what else to try.

-- Andrey Khalyavin

[Don Winters]

I guess it is a worst case scenario, item 2 is where it hangs.

Just to be clear, I created the dump using Visual Studio - attaching to the process and then dumping. I don't know if that has anything to do with the process being started in the debugger.

Maybe I can work with our IT people to see if we can figure out what software on our side is causing the problems - but they are not thrilled about this sort of stuff. It may take a while.

[Andrey Khalyavin]

On Thursday, October 24, 2013 4:00:06 PM UTC+4, Don Winters wrote:

I guess it is a worst case scenario, item 2 is where it hangs.

This is not the worst, it is the most strange scenario ;).  Does other programs (ar/c++/g++/gcc/ld) in the same folder hang too?

Just to be clear, I created the dump using Visual Studio - attaching to the process and then dumping. I don't know if that has anything to do with the process being started in the debugger.

That explains the dump. In this case this means that x86_64-nacl-as.exe process was suspended at startup because it is not possible to attach to process which is already debugged by someone else. 

-- Andrey Khalyavin

[Don Winters]

Well not everything hangs in that folder, but most do. c++.exe and g++.exe did not hang, everything else did.

As for the dump, again to be clear, I attached to the process after it had hung and then dumped from there.

[Andrey Khalyavin]

On Thursday, October 24, 2013 5:42:01 PM UTC+4, Don Winters wrote:

Well not everything hangs in that folder, but most do. c++.exe and g++.exe did not hang, everything else did.

I found that most redirectors in this folder (except c++/g++/gcc) are needed only to save space and handle the case where our tools are called by their short name from another cygwin distrubution. You probably will not need the later feature for a while, so you can override redirectors with real executables:

cd pepper_31\toolchain\win_x86_newlib

copy libexec\cyggcc_s-1.dll x86_64-nacl\bin\cyggcc_s-1.dll

copy libexec\cygiconv-2.dll x86_64-nacl\bin\cygiconv-2.dll

copy libexec\cygintl-8.dll x86_64-nacl\bin\cygintl-8.dll

copy libexec\cygwin1.dll x86_64-nacl\bin\cygwin1.dll

move x86_64-nacl\bin\as.exe x86_64-nacl\bin\as.exe.tmp

copy libexec\x86_64-nacl-as.exe x86_64-nacl\bin\as.exe

move x86_64-nacl\bin\ar.exe x86_64-nacl\bin\ar.exe.tmp

copy libexec\x86_64-nacl-ar.exe x86_64-nacl\bin\ar.exe

move x86_64-nacl\bin\ld.exe x86_64-nacl\bin\ld.exe.tmp

copy libexec\x86_64-nacl-ld.exe x86_64-nacl\bin\ld.exe

move x86_64-nacl\bin\nm.exe x86_64-nacl\bin\nm.exe.tmp

copy libexec\x86_64-nacl-nm.exe x86_64-nacl\bin\nm.exe

move x86_64-nacl\bin\objcopy.exe x86_64-nacl\bin\objcopy.exe.tmp

copy libexec\x86_64-nacl-objcopy.exe x86_64-nacl\bin\objcopy.exe

move x86_64-nacl\bin\objdump.exe x86_64-nacl\bin\objdump.exe.tmp

copy libexec\x86_64-nacl-objdump.exe x86_64-nacl\bin\objdump.exe

move x86_64-nacl\bin\ranlib.exe x86_64-nacl\bin\ranlib.exe.tmp

copy libexec\x86_64-nacl-ranlib.exe x86_64-nacl\bin\ranlib.exe

move x86_64-nacl\bin\strip.exe x86_64-nacl\bin\strip.exe.tmp

copy libexec\x86_64-nacl-strip.exe x86_64-nacl\bin\strip.exe

The same thing should be done for glibc toolchain, if hangs will gone.

-- Andrey Khalyavin

[Don Winters]

I copied the files per your instructions below, I don't think I missed anything. There was some success.  It builds the 32 bit newlib nexe file for the audio sample (audio_x86_32.nexe) but it still hangs when it tries to build the 64 bit version.

[Andrey Khalyavin]

On Thursday, October 24, 2013 7:54:39 PM UTC+4, Don Winters wrote:

I copied the files per your instructions below, I don't think I missed anything. There was some success. 

I am glad to hear it. 

It builds the 32 bit newlib nexe file for the audio sample (audio_x86_32.nexe) but it still hangs when it tries to build the 64 bit version.

What is the process chain that hangs? If you haven't brake hardlinks yet, use process properties to see correct process names in

process explorer.

My hypothesis is that it is behaviour analyzer that blocks starting new processes. It recognizes some processes like as.exe by name and

forbids them to start new processes since gnu assembler doesn't do that. It doesn't explain why 64 bit version doesn't work though.

Anyway, I think it worth to report this problem to your antivirus/firewall vendor. Do you know which program is doing behaviour analysis

on your system?

-- Andrey Khalyavin

[Don Winters]

x86_64-nacl-g++ is hanging - I'll send along a screenshot of process explorer.

We are using Symantec Point Protection as well as Safend End and LANDesk. The IT guys turned off Symantec and Safend on my machine but that did not help. The had to get approval to turn off LANDesk so I haven't tried that yet.

[Andrey Khalyavin]

2013/10/30 Don Winters

Sorry, is this what you need? It is showing up as as.exe.

Yes, it is what I need. This is strange though, I thought you already replaced bin\as.exe with x86_64-nacl-as.exe. 

BTW, I have trouble getting gmail at work. 

Also, it seems to me you need to be able to reproduce this to fix it. Is there any chance you guys could send us a laptop that our IT guys could image and we would send it back to you.

I live in Russia and our MTV guys are less experienced (and willing to work) with Windows. I don't think it will work.

If this is false positive in behavior analysis as I suspect, we can't fix it on our side other than moving to mingw from cygwin. This is a lot of work and so is in our distant plans. Alternatively, if there is a way to find which security product is responsible, they can be contacted and asked to fix the issue. 

I have an idea recently how this might be done. You can search for handles in process explorer. If you type the name of the suspended process, you can see which processes hold its handles. Look only to handles of "Process" type. If you see an antivirus among them, it is most likely responsible.

-- Andrey Khalyavin

[Don Winters]

Ok. We work in both Mac and Windows here. I prefer Windows but I am going to switch to Mac for this. I don't what else to do.

[Eric Lan]

Did you guys managed to solve this problem? I am having the exact same problem on my Windows 7 64bit. Following the earlier post. I arrived with a the 32-bit .nexe successful but still hangs on 64 bit version.

[vineet jain]

Any Antivirus installed on that windows 7 machine?

[Kira]

Yes it is installed with antivirus which is impossible to disable =< (office machine)

--
You received this message because you are subscribed to a topic in the Google Groups "Native-Client-Discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/native-client-discuss/BS98ol5ouKY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to native-client-di...@googlegroups.com.

[Don Winters]

Just to update this. It was never solved. My machine is an office machine also. There is some bad synergy here, probably LANDesk or Symantec. I switched to using a Mac machine.