LOCAL_SHORT_COMMANDS and slow make.exe/builds on Windows
361 views
Skip to first unread message
James Swift
Apr 18, 2016, 9:57:51 AM4/18/16
to android-ndk
Hi all,
This is really two questions.
As recommended I activated LOCAL_SHORT_COMMANDS because my link step generates a command that's over the Windows limit.
However, as noted by many people, this results in extremely long build times when you have a large project.
Would it be possible to only enable it for linking and not for compilation?
Whilst investigating this issue I noticed that make.exe seems to pause for long periods doing apparently little between commands.
E.g. (with LOCAL_SHORT_COMMANDS set to false)
If I start make like something like this (with paths shortened for readability)...
ndk-bundle\prebuilt\windows-x86_64\bin\make.exe -f ndk-bundle\build\core\build-local.mk SHELL=cmd NDK_PROJECT_PATH=c:\path_to_my_project -j8 APP_OPTIM=debug NDK_OUT=obj-debug NDK_LIBS_OUT=libs-debug "APP_ABI=x86 armeabi-v7a-hard"
Make is outputting each of the following lines slowly.
[x86] Compile++ : mylibrary <= somefile.cpp
[x86] Compile++ : mylibrary <= somefile1.cpp
[x86] Compile++ : mylibrary <= somefile2.cpp
ProcessExplorer indicates it's not doing much. It's as if it's sleeping or waiting for some event.
It seems to slowly prepare 8 tasks as specified by the -j8 and then in one blast 8 clang++.exe processes are started and they execute very fast as expected.
Anyone have and idea how or why the steps before clang++.exe is started are so slow?
kind regards,
James
This is really two questions.
As recommended I activated LOCAL_SHORT_COMMANDS because my link step generates a command that's over the Windows limit.
However, as noted by many people, this results in extremely long build times when you have a large project.
Would it be possible to only enable it for linking and not for compilation?
Whilst investigating this issue I noticed that make.exe seems to pause for long periods doing apparently little between commands.
E.g. (with LOCAL_SHORT_COMMANDS set to false)
If I start make like something like this (with paths shortened for readability)...
ndk-bundle\prebuilt\windows-x86_64\bin\make.exe -f ndk-bundle\build\core\build-local.mk SHELL=cmd NDK_PROJECT_PATH=c:\path_to_my_project -j8 APP_OPTIM=debug NDK_OUT=obj-debug NDK_LIBS_OUT=libs-debug "APP_ABI=x86 armeabi-v7a-hard"
Make is outputting each of the following lines slowly.
[x86] Compile++ : mylibrary <= somefile.cpp
[x86] Compile++ : mylibrary <= somefile1.cpp
[x86] Compile++ : mylibrary <= somefile2.cpp
ProcessExplorer indicates it's not doing much. It's as if it's sleeping or waiting for some event.
It seems to slowly prepare 8 tasks as specified by the -j8 and then in one blast 8 clang++.exe processes are started and they execute very fast as expected.
Anyone have and idea how or why the steps before clang++.exe is started are so slow?
kind regards,
James
David Turner
Apr 20, 2016, 12:02:04 PM4/20/16
to andro...@googlegroups.com
The implementation of LOCAL_SHORT_COMMANDS is slow by design (mostly due to GNU Make limitations), but what you describe seems to be way over the expected slowdown.
Before anything else, have you tried disabling your anti-virus during the build to see if this speeds things up? We've seen really bad interactions between some of them and Cygwin/MSys/NDK on Windows in the (distant) past.
--
You received this message because you are subscribed to the Google Groups "android-ndk" group.
To unsubscribe from this group and stop receiving emails from it, send an email to android-ndk...@googlegroups.com.
To post to this group, send email to andro...@googlegroups.com.
Visit this group at https://groups.google.com/group/android-ndk.
For more options, visit https://groups.google.com/d/optout.
James Swift
Apr 21, 2016, 10:03:05 AM4/21/16
to android-ndk
Hi,
Anti-virus isn't a problem as far as I can tell. I've build with it off and no change.
However, I got some really good improvements by building Make 4.1 myself and replacing 3.8 in the NDK. (Half an hour instead of days!
It's still really slow compared to OSX build for example, but the apparently needless pauses are gone.
Now it's just taking a long time because of the splitting up into short commands which, as you pointed, out is by design.
If there was a way to not use short commands for anything except the link phase?
This would bring the compile speed in line with other platforms.
It's probably true for almost everyone that they encounter the cmd.exe character limit only when linking.
regards,
James
Anti-virus isn't a problem as far as I can tell. I've build with it off and no change.
However, I got some really good improvements by building Make 4.1 myself and replacing 3.8 in the NDK. (Half an hour instead of days!
It's still really slow compared to OSX build for example, but the apparently needless pauses are gone.
Now it's just taking a long time because of the splitting up into short commands which, as you pointed, out is by design.
If there was a way to not use short commands for anything except the link phase?
This would bring the compile speed in line with other platforms.
It's probably true for almost everyone that they encounter the cmd.exe character limit only when linking.
regards,
James
Reply all
Reply to author
Forward
0 new messages