That's my first post here so let's I don't get flamed for telling something stupid :) If anybody from Google working on the NDK could read this and give feedback, it would be great.
We use Android NDK a lot at my company (DotEmu SAS, a French videogame company) and I decided to build my own from AOSP master for many reasons (mostly critical bugfixes we need for our projects). I tried to build NDK from AOSP master branch, from 2012-11-07, on OS X Lion 10.7.5. XCode 4.5.2 installed with command-line tools, and environment setup as AOSP guidelines require it.
I ran into two major issues :
1. Building the NDK itself Without adding --systems=darwin-x86 it to make-release.sh command line the build fails. Still, the documentation says it should be the default on OS X (which makes a lot of sense). Still, after scratching my head a bit, I discovered the switch was being filtered by make-release.sh itself by this line : HOST_SYSTEMS_FLAGS=$(echo "$HOST_SYSTEMS_FLAGS" | sed -e 's/darwin-x86//') I had to comment it to get the NDK compiling fine.
After a good sleep, I had a nice tar.bz2 file containing my NDK, which was building our projects quite well. Then comes the second and most problematic issue.
2. My NDK keeps rebuilding all files at each build, even untouched It took me a while to figure out why, but the reason was in fact each directory being "touched" by the OS, therefore bringing a timestamp later than my source files. So GNU make complains for each file it's out of date due to the timestamp its own directory being later than the file itself.
I had a deep look between r8b (working fine) and master, and discovered a lot of changes were done in directory handling, especially regarding host-mkdir command. I finally managed to tweak it to work as it should, namely rebuilding only changed files between to builds, and all files if I request a clean. Please find attached a (dirty) patch which is mostly a rollback to r8b system. I am aware this is not going to be merged in AOSP but I would love to understand what's going "under the hood" and why I had to make such a dirty hack.
The first issue seems to be a bug. We don't package the darwin NDK on OS X
anymore, instead we launch the build from Linux, and use the
--darwin-ssh=<hostname> option to build only the host NDK binaries on a
remote machine. I guess that's why this change to make-release.sh wasn't
tested properly. It should be easy to fix though.
The second issue is stranger to me. The "generate-file-dir" improvement has
been written specifically to ensure that directories containing each object
file are only created once per build. This is in contrast to NDK r8b and
before, which performed a forced "mkdir -p <dirname>" for every object file
being built. The directories themselves should only be touched/created if
they don't already exist.
I can't reproduce the problem on Linux, I'll have to check on an OS X
machine.
Thanks for reporting this.
- David
On Thu, Nov 8, 2012 at 12:28 AM, Romain Tisserand <
> That's my first post here so let's I don't get flamed for telling
> something stupid :)
> If anybody from Google working on the NDK could read this and give
> feedback, it would be great.
> We use Android NDK a lot at my company (DotEmu SAS, a French videogame
> company) and I decided to build my own from AOSP master for many reasons
> (mostly critical bugfixes we need for our projects). I tried to build NDK
> from AOSP master branch, from 2012-11-07, on OS X Lion 10.7.5. XCode 4.5.2
> installed with command-line tools, and environment setup as AOSP guidelines
> require it.
> I ran into two major issues :
> 1. Building the NDK itself
> Without adding --systems=darwin-x86 it to make-release.sh command line the
> build fails. Still, the documentation says it should be the default on OS X
> (which makes a lot of sense). Still, after scratching my head a bit, I
> discovered the switch was being filtered by make-release.sh itself by this
> line :
> HOST_SYSTEMS_FLAGS=$(echo "$HOST_SYSTEMS_FLAGS" | sed -e 's/darwin-x86//')
> I had to comment it to get the NDK compiling fine.
> After a good sleep, I had a nice tar.bz2 file containing my NDK, which was
> building our projects quite well. Then comes the second and most
> problematic issue.
> 2. My NDK keeps rebuilding all files at each build, even untouched
> It took me a while to figure out why, but the reason was in fact each
> directory being "touched" by the OS, therefore bringing a timestamp later
> than my source files. So GNU make complains for each file it's out of date
> due to the timestamp its own directory being later than the file itself.
> I had a deep look between r8b (working fine) and master, and discovered a
> lot of changes were done in directory handling, especially regarding
> host-mkdir command. I finally managed to tweak it to work as it should,
> namely rebuilding only changed files between to builds, and all files if I
> request a clean. Please find attached a (dirty) patch which is mostly a
> rollback to r8b system.
> I am aware this is not going to be merged in AOSP but I would love to
> understand what's going "under the hood" and why I had to make such a dirty
> hack.
> Looking forward from feedback :)
> Thanks,
> Romain Tisserand
> --
> You received this message because you are subscribed to the Google Groups
> "android-ndk" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/android-ndk/-/sqsn8uXZkSYJ.
> To post to this group, send email to android-ndk@googlegroups.com.
> To unsubscribe from this group, send email to
> android-ndk+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/android-ndk?hl=en.
My various OSX setups at work and home have various MacPorts binaries installed, I thought that would be wise to add. Maybe some of them, since in the path, is conflicting with the new generate-file-dir behaviour. Will try on a virgin OSX 10.8 install and try to reproduce the issue using ndk-r8c.
On Tuesday, November 13, 2012 9:36:14 AM UTC+1, Digit wrote:
> Hello,
> First, sorry for your troubles,
> The first issue seems to be a bug. We don't package the darwin NDK on OS X > anymore, instead we launch the build from Linux, and use the > --darwin-ssh=<hostname> option to build only the host NDK binaries on a > remote machine. I guess that's why this change to make-release.sh wasn't > tested properly. It should be easy to fix though.
> The second issue is stranger to me. The "generate-file-dir" improvement > has been written specifically to ensure that directories containing each > object file are only created once per build. This is in contrast to NDK r8b > and before, which performed a forced "mkdir -p <dirname>" for every object > file being built. The directories themselves should only be touched/created > if they don't already exist.
> I can't reproduce the problem on Linux, I'll have to check on an OS X > machine.
> Thanks for reporting this.
> - David
> On Thu, Nov 8, 2012 at 12:28 AM, Romain Tisserand <romain.t...@gmail.com<javascript:> > > wrote:
>> Hello,
>> That's my first post here so let's I don't get flamed for telling >> something stupid :) >> If anybody from Google working on the NDK could read this and give >> feedback, it would be great.
>> We use Android NDK a lot at my company (DotEmu SAS, a French videogame >> company) and I decided to build my own from AOSP master for many reasons >> (mostly critical bugfixes we need for our projects). I tried to build NDK >> from AOSP master branch, from 2012-11-07, on OS X Lion 10.7.5. XCode 4.5.2 >> installed with command-line tools, and environment setup as AOSP guidelines >> require it.
>> I ran into two major issues :
>> 1. Building the NDK itself >> Without adding --systems=darwin-x86 it to make-release.sh command line >> the build fails. Still, the documentation says it should be the default on >> OS X (which makes a lot of sense). Still, after scratching my head a bit, I >> discovered the switch was being filtered by make-release.sh itself by this >> line : >> HOST_SYSTEMS_FLAGS=$(echo "$HOST_SYSTEMS_FLAGS" | sed -e 's/darwin-x86//') >> I had to comment it to get the NDK compiling fine.
>> After a good sleep, I had a nice tar.bz2 file containing my NDK, which >> was building our projects quite well. Then comes the second and most >> problematic issue.
>> 2. My NDK keeps rebuilding all files at each build, even untouched >> It took me a while to figure out why, but the reason was in fact each >> directory being "touched" by the OS, therefore bringing a timestamp later >> than my source files. So GNU make complains for each file it's out of date >> due to the timestamp its own directory being later than the file itself.
>> I had a deep look between r8b (working fine) and master, and discovered a >> lot of changes were done in directory handling, especially regarding >> host-mkdir command. I finally managed to tweak it to work as it should, >> namely rebuilding only changed files between to builds, and all files if I >> request a clean. Please find attached a (dirty) patch which is mostly a >> rollback to r8b system. >> I am aware this is not going to be merged in AOSP but I would love to >> understand what's going "under the hood" and why I had to make such a dirty >> hack.
>> Looking forward from feedback :)
>> Thanks, >> Romain Tisserand
>> -- >> You received this message because you are subscribed to the Google Groups >> "android-ndk" group. >> To view this discussion on the web visit >> https://groups.google.com/d/msg/android-ndk/-/sqsn8uXZkSYJ. >> To post to this group, send email to andro...@googlegroups.com<javascript:> >> . >> To unsubscribe from this group, send email to >> android-ndk...@googlegroups.com <javascript:>. >> For more options, visit this group at >> http://groups.google.com/group/android-ndk?hl=en.
On Tuesday, November 13, 2012 9:36:14 AM UTC+1, Digit wrote:
> Hello,
> First, sorry for your troubles,
> The first issue seems to be a bug. We don't package the darwin NDK on OS X > anymore, instead we launch the build from Linux, and use the > --darwin-ssh=<hostname> option to build only the host NDK binaries on a > remote machine. I guess that's why this change to make-release.sh wasn't > tested properly. It should be easy to fix though.
> The second issue is stranger to me. The "generate-file-dir" improvement > has been written specifically to ensure that directories containing each > object file are only created once per build. This is in contrast to NDK r8b > and before, which performed a forced "mkdir -p <dirname>" for every object > file being built. The directories themselves should only be touched/created > if they don't already exist.
> I can't reproduce the problem on Linux, I'll have to check on an OS X > machine.
> Thanks for reporting this.
> - David
> On Thu, Nov 8, 2012 at 12:28 AM, Romain Tisserand <romain.t...@gmail.com<javascript:> > > wrote:
>> Hello,
>> That's my first post here so let's I don't get flamed for telling >> something stupid :) >> If anybody from Google working on the NDK could read this and give >> feedback, it would be great.
>> We use Android NDK a lot at my company (DotEmu SAS, a French videogame >> company) and I decided to build my own from AOSP master for many reasons >> (mostly critical bugfixes we need for our projects). I tried to build NDK >> from AOSP master branch, from 2012-11-07, on OS X Lion 10.7.5. XCode 4.5.2 >> installed with command-line tools, and environment setup as AOSP guidelines >> require it.
>> I ran into two major issues :
>> 1. Building the NDK itself >> Without adding --systems=darwin-x86 it to make-release.sh command line >> the build fails. Still, the documentation says it should be the default on >> OS X (which makes a lot of sense). Still, after scratching my head a bit, I >> discovered the switch was being filtered by make-release.sh itself by this >> line : >> HOST_SYSTEMS_FLAGS=$(echo "$HOST_SYSTEMS_FLAGS" | sed -e 's/darwin-x86//') >> I had to comment it to get the NDK compiling fine.
>> After a good sleep, I had a nice tar.bz2 file containing my NDK, which >> was building our projects quite well. Then comes the second and most >> problematic issue.
>> 2. My NDK keeps rebuilding all files at each build, even untouched >> It took me a while to figure out why, but the reason was in fact each >> directory being "touched" by the OS, therefore bringing a timestamp later >> than my source files. So GNU make complains for each file it's out of date >> due to the timestamp its own directory being later than the file itself.
>> I had a deep look between r8b (working fine) and master, and discovered a >> lot of changes were done in directory handling, especially regarding >> host-mkdir command. I finally managed to tweak it to work as it should, >> namely rebuilding only changed files between to builds, and all files if I >> request a clean. Please find attached a (dirty) patch which is mostly a >> rollback to r8b system. >> I am aware this is not going to be merged in AOSP but I would love to >> understand what's going "under the hood" and why I had to make such a dirty >> hack.
>> Looking forward from feedback :)
>> Thanks, >> Romain Tisserand
>> -- >> You received this message because you are subscribed to the Google Groups >> "android-ndk" group. >> To view this discussion on the web visit >> https://groups.google.com/d/msg/android-ndk/-/sqsn8uXZkSYJ. >> To post to this group, send email to andro...@googlegroups.com<javascript:> >> . >> To unsubscribe from this group, send email to >> android-ndk...@googlegroups.com <javascript:>. >> For more options, visit this group at >> http://groups.google.com/group/android-ndk?hl=en.
> My various OSX setups at work and home have various MacPorts binaries
> installed, I thought that would be wise to add.
> Maybe some of them, since in the path, is conflicting with the new
> generate-file-dir behaviour.
> Will try on a virgin OSX 10.8 install and try to reproduce the issue using
> ndk-r8c.
> Romain
> On Tuesday, November 13, 2012 9:36:14 AM UTC+1, Digit wrote:
>> Hello,
>> First, sorry for your troubles,
>> The first issue seems to be a bug. We don't package the darwin NDK on OS
>> X anymore, instead we launch the build from Linux, and use the
>> --darwin-ssh=<hostname> option to build only the host NDK binaries on a
>> remote machine. I guess that's why this change to make-release.sh wasn't
>> tested properly. It should be easy to fix though.
>> The second issue is stranger to me. The "generate-file-dir" improvement
>> has been written specifically to ensure that directories containing each
>> object file are only created once per build. This is in contrast to NDK r8b
>> and before, which performed a forced "mkdir -p <dirname>" for every object
>> file being built. The directories themselves should only be touched/created
>> if they don't already exist.
>> I can't reproduce the problem on Linux, I'll have to check on an OS X
>> machine.
>> Thanks for reporting this.
>> - David
>> On Thu, Nov 8, 2012 at 12:28 AM, Romain Tisserand <romain.t...@gmail.com>wrote:
>>> Hello,
>>> That's my first post here so let's I don't get flamed for telling
>>> something stupid :)
>>> If anybody from Google working on the NDK could read this and give
>>> feedback, it would be great.
>>> We use Android NDK a lot at my company (DotEmu SAS, a French videogame
>>> company) and I decided to build my own from AOSP master for many reasons
>>> (mostly critical bugfixes we need for our projects). I tried to build NDK
>>> from AOSP master branch, from 2012-11-07, on OS X Lion 10.7.5. XCode 4.5.2
>>> installed with command-line tools, and environment setup as AOSP guidelines
>>> require it.
>>> I ran into two major issues :
>>> 1. Building the NDK itself
>>> Without adding --systems=darwin-x86 it to make-release.sh command line
>>> the build fails. Still, the documentation says it should be the default on
>>> OS X (which makes a lot of sense). Still, after scratching my head a bit, I
>>> discovered the switch was being filtered by make-release.sh itself by this
>>> line :
>>> HOST_SYSTEMS_FLAGS=$(echo "$HOST_SYSTEMS_FLAGS" | sed -e
>>> 's/darwin-x86//')
>>> I had to comment it to get the NDK compiling fine.
>>> After a good sleep, I had a nice tar.bz2 file containing my NDK, which
>>> was building our projects quite well. Then comes the second and most
>>> problematic issue.
>>> 2. My NDK keeps rebuilding all files at each build, even untouched
>>> It took me a while to figure out why, but the reason was in fact each
>>> directory being "touched" by the OS, therefore bringing a timestamp later
>>> than my source files. So GNU make complains for each file it's out of date
>>> due to the timestamp its own directory being later than the file itself.
>>> I had a deep look between r8b (working fine) and master, and discovered
>>> a lot of changes were done in directory handling, especially regarding
>>> host-mkdir command. I finally managed to tweak it to work as it should,
>>> namely rebuilding only changed files between to builds, and all files if I
>>> request a clean. Please find attached a (dirty) patch which is mostly a
>>> rollback to r8b system.
>>> I am aware this is not going to be merged in AOSP but I would love to
>>> understand what's going "under the hood" and why I had to make such a dirty
>>> hack.
>>> Looking forward from feedback :)
>>> Thanks,
>>> Romain Tisserand
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "android-ndk" group.
>>> To view this discussion on the web visit https://groups.google.com/d/** >>> msg/android-ndk/-/sqsn8uXZkSYJ<https://groups.google.com/d/msg/android-ndk/-/sqsn8uXZkSYJ>
>>> **.
>>> To post to this group, send email to andro...@googlegroups.com.
>>> To unsubscribe from this group, send email to android-ndk...@**
>>> googlegroups.com.
> To post to this group, send email to android-ndk@googlegroups.com.
> To unsubscribe from this group, send email to
> android-ndk+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/android-ndk?hl=en.
> Sorry about the annoyance. I hope we'll be able to release the fix shortly.
> On Thu, Nov 15, 2012 at 11:09 AM, Romain Tisserand
> <romain.tisser...@gmail.com> wrote:
>> Hello David,
>> Thank you for your feedback and explanation.
>> My various OSX setups at work and home have various MacPorts binaries
>> installed, I thought that would be wise to add.
>> Maybe some of them, since in the path, is conflicting with the new
>> generate-file-dir behaviour.
>> Will try on a virgin OSX 10.8 install and try to reproduce the issue using
>> ndk-r8c.
>> Romain
>> On Tuesday, November 13, 2012 9:36:14 AM UTC+1, Digit wrote:
>>> Hello,
>>> First, sorry for your troubles,
>>> The first issue seems to be a bug. We don't package the darwin NDK on OS
>>> X anymore, instead we launch the build from Linux, and use the
>>> --darwin-ssh=<hostname> option to build only the host NDK binaries on a
>>> remote machine. I guess that's why this change to make-release.sh wasn't
>>> tested properly. It should be easy to fix though.
>>> The second issue is stranger to me. The "generate-file-dir" improvement
>>> has been written specifically to ensure that directories containing each
>>> object file are only created once per build. This is in contrast to NDK r8b
>>> and before, which performed a forced "mkdir -p <dirname>" for every object
>>> file being built. The directories themselves should only be touched/created
>>> if they don't already exist.
>>> I can't reproduce the problem on Linux, I'll have to check on an OS X
>>> machine.
>>> Thanks for reporting this.
>>> - David
>>> On Thu, Nov 8, 2012 at 12:28 AM, Romain Tisserand <romain.t...@gmail.com>
>>> wrote:
>>>> Hello,
>>>> That's my first post here so let's I don't get flamed for telling
>>>> something stupid :)
>>>> If anybody from Google working on the NDK could read this and give
>>>> feedback, it would be great.
>>>> We use Android NDK a lot at my company (DotEmu SAS, a French videogame
>>>> company) and I decided to build my own from AOSP master for many reasons
>>>> (mostly critical bugfixes we need for our projects). I tried to build NDK
>>>> from AOSP master branch, from 2012-11-07, on OS X Lion 10.7.5. XCode 4.5.2
>>>> installed with command-line tools, and environment setup as AOSP guidelines
>>>> require it.
>>>> I ran into two major issues :
>>>> 1. Building the NDK itself
>>>> Without adding --systems=darwin-x86 it to make-release.sh command line
>>>> the build fails. Still, the documentation says it should be the default on
>>>> OS X (which makes a lot of sense). Still, after scratching my head a bit, I
>>>> discovered the switch was being filtered by make-release.sh itself by this
>>>> line :
>>>> HOST_SYSTEMS_FLAGS=$(echo "$HOST_SYSTEMS_FLAGS" | sed -e
>>>> 's/darwin-x86//')
>>>> I had to comment it to get the NDK compiling fine.
>>>> After a good sleep, I had a nice tar.bz2 file containing my NDK, which
>>>> was building our projects quite well. Then comes the second and most
>>>> problematic issue.
>>>> 2. My NDK keeps rebuilding all files at each build, even untouched
>>>> It took me a while to figure out why, but the reason was in fact each
>>>> directory being "touched" by the OS, therefore bringing a timestamp later
>>>> than my source files. So GNU make complains for each file it's out of date
>>>> due to the timestamp its own directory being later than the file itself.
>>>> I had a deep look between r8b (working fine) and master, and discovered
>>>> a lot of changes were done in directory handling, especially regarding
>>>> host-mkdir command. I finally managed to tweak it to work as it should,
>>>> namely rebuilding only changed files between to builds, and all files if I
>>>> request a clean. Please find attached a (dirty) patch which is mostly a
>>>> rollback to r8b system.
>>>> I am aware this is not going to be merged in AOSP but I would love to
>>>> understand what's going "under the hood" and why I had to make such a dirty
>>>> hack.
>>>> Looking forward from feedback :)
>>>> Thanks,
>>>> Romain Tisserand
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "android-ndk" group.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msg/android-ndk/-/sqsn8uXZkSYJ.
>>>> To post to this group, send email to andro...@googlegroups.com.
>>>> To unsubscribe from this group, send email to
>>>> android-ndk...@googlegroups.com.
>> To post to this group, send email to android-ndk@googlegroups.com.
>> To unsubscribe from this group, send email to
>> android-ndk+unsubscribe@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/android-ndk?hl=en.
> --
> You received this message because you are subscribed to the Google Groups
> "android-ndk" group.
> To post to this group, send email to android-ndk@googlegroups.com.
> To unsubscribe from this group, send email to
> android-ndk+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/android-ndk?hl=en.
