Cygwin issues, still...

[Doug Schaefer]

I'm sorry if someone already pointed this out, but I just ran into it
and missed a lot of e-mails on this list lately.

If you're build has a compile error, and then you fix it and then you
build again, it seems you get an infamous:

/cygdrive/c/Eclipse/workspaces/android-sdk/.test/Test/obj/local/armeabi/objs/test2//test.o.d:1:
*** target pattern contains no `%'. Stop.

Looking at test.o.d, I see Windows paths.

So I removed my objs directory and launched the build again (simply
doing a clean resulted in the same error). And low and behold, the .d
file contains cygwin paths. eh?

Is this considered a bug? I really, really want to support the
official NDK with the CDT, but fighting cygwin isn't in my plans, at
least not this much. I won't get into how much hassle it's been just
to get the CDT to call make appropriately.

Sorry for the rant.
Doug.

[Doug Schaefer]

Now the good news :). I've been able to use MSYS in place of Cygwin.
All you need is a dummy cygpath script in your path that returns that
file name unchanged (i.e. as it's natural Windows path). Everything
seems to work except for clean which doesn't seem to remove the object
files. Good enough for me, for now. I'll make sure the CDT works with
both.

D.

[Doug Schaefer]

Ah, clean doesn't work with cygwin either.

[Doug Schaefer]

One more :). let me be clear, clean isn't working in my CDT tests. It
is working fine from the command line.

[David Turner]

On Mon, Dec 13, 2010 at 3:31 AM, Doug Schaefer <cdt...@gmail.com> wrote:

I'm sorry if someone already pointed this out, but I just ran into it
and missed a lot of e-mails on this list lately.

If you're build has a compile error, and then you fix it and then you
build again, it seems you get an infamous:

   /cygdrive/c/Eclipse/workspaces/android-sdk/.test/Test/obj/local/armeabi/objs/test2//test.o.d:1:
*** target pattern contains no `%'.  Stop.

Looking at test.o.d, I see Windows paths.

This is very strange, the compiler doesn't know anything about cygwin, and the .d file normally only contains headers and sources that have been included during a compilation step (i.e. it doesn't contain any paths provided through -I/cygdrive/foo/ if no corresponding header was included).

Are you sure your build didn't use a cygwin compiler by mistake?

 

So I removed my objs directory and launched the build again (simply
doing a clean resulted in the same error). And low and behold, the .d
file contains cygwin paths. eh?

It's be interesting to see the exact list of commands that led to this situation.

 

Is this considered a bug? I really, really want to support the
official NDK with the CDT, but fighting cygwin isn't in my plans, at
least not this much. I won't get into how much hassle it's been just
to get the CDT to call make appropriately.

It could be considered a bug, but you should first provide a way to reproduce it with the standalone NDK, just to be sure.

 

Sorry for the rant.
Doug.

--
You received this message because you are subscribed to the Google Groups "android-ndk" group.
To post to this group, send email to andro...@googlegroups.com.
To unsubscribe from this group, send email to android-ndk...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/android-ndk?hl=en.

[Raymond.Wen]

I can reproduce this issue.
In normal case that the code compiles fine, the gcc generates a .d file using windows path like:
d:/foo.o : d:/.../foo.c
After this step, the cmd-convert-deps-to-cygwin is called to convert the path above to:
/cygdrive/d/foo.o : cygdrive/d/..../foo.c

But in the case where the compilation fails, the gcc process that generates .d file returns a non-zero value, so that make process stops immediately. The consequence is cmd-convert-deps-to-cygwin won't run, and the .d file is left in a incorrect state. After that, make can't process the .d file anymore, unless it's cleanly manually.

If the cmd-convert-deps-to-cygwin can be made to run even if the gcc process fails, I think the issue can be resolved.


Best Regards,
Raymond Wen

[alan]

happens to me too:
http://groups.google.com/group/android-ndk/browse_thread/thread/20046f28dabca2d1#

On Dec 13, 8:27 am, "Raymond.Wen" <rx.wen...@gmail.com> wrote:
> I can reproduce this issue.
> In normal case that the code compiles fine, the gcc generates a .d file
> using windows path like:
> d:/foo.o : d:/.../foo.c
> After this step, the cmd-convert-deps-to-cygwin is called to convert the
> path above to:
> /cygdrive/d/foo.o : cygdrive/d/..../foo.c
>
> But in the case where the compilation fails, the gcc process that generates
> .d file returns a non-zero value, so that make process stops immediately.
> The consequence is cmd-convert-deps-to-cygwin won't run, and the .d file is
> left in a incorrect state. After that, make can't process the .d file
> anymore, unless it's cleanly manually.
>
> If the cmd-convert-deps-to-cygwin can be made to run even if the gcc process
> fails, I think the issue can be resolved.
>
> Best Regards,
> Raymond Wen
>
> On Mon, Dec 13, 2010 at 4:02 PM, David Turner <di...@android.com> wrote:
>

> >> android-ndk...@googlegroups.com<android-ndk%2Bunsu...@googlegroups.com>
> >> .

> >> For more options, visit this group at
> >>http://groups.google.com/group/android-ndk?hl=en.
>
> >  --
> > You received this message because you are subscribed to the Google Groups
> > "android-ndk" group.
> > To post to this group, send email to andro...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > android-ndk...@googlegroups.com<android-ndk%2Bunsu...@googlegroups.com>
> > .

[David Turner]

Thanks, that's exactly what I needed, preparing a fix to come with the update being prepared...

[Raymond.Wen]

David,

Since you mentioned that a update is coming soon. May I know will the this update includes sip support for emulator?
Thanks.

Best Regards,
Raymond Wen
http://rxwen.blogspot.com

[David Turner]

On Mon, Dec 13, 2010 at 11:41 AM, Raymond.Wen <rx.w...@gmail.com> wrote:

David,

Since you mentioned that a update is coming soon. May I know will the this update includes sip support for emulator?

 

I'm talking about an NDK update here, nothing more.

[Peter Mackay]

Thanks guys for investigating this.

I also get the issue (both in CDT and through a Bash session) and it's
annoying to have to occasionally delete the .d files.

I just started with NDK, so I couldn't say if this is a new bug in NDK
r5 or not.

[Raymond.Wen]

Yes, it's really annoying. But at least it gives us a chance to use our own brain other than the compiler to find out errors before we perform actual compilation. :)
As a convenience, I created a makefile in the project root directory to perform clean up.

clean:
 find ./ -name "*.d" -exec rm {} \;

Best Regards,
Raymond Wen

--

[Wex]

I have a fix for this problem. It requires two changes, one to build/
core/definitions.mk.save:

930c930
< $(hide) $$(PRIVATE_CC) -MMD -MP -MF $$(PRIVATE_DEPS) $$
(PRIVATE_CFLAGS) $$(call host-path,$$(PRIVATE_SRC)) -o $$(call host-
path,$$(PRIVATE_OBJ))
---
> $(hide) $$(PRIVATE_CC) -MM -MP -MF $$(PRIVATE_DEPS) $$(PRIVATE_CFLAGS) $$(call host-path,$$(PRIVATE_SRC))
933a934
> $(hide) $$(PRIVATE_CC) $$(PRIVATE_CFLAGS) $$(call host-path,$$(PRIVATE_S
RC)) -o $$(call host-path,$$(PRIVATE_OBJ))

The bug in definitions.mk is that the command that converts the
Windows-style paths into Cygwin formatted paths will not be run if g++
fails when compiling a file. This ends up leaving the dependency file
with Windows drive letter path names that, in turn, causes g++ to barf
out the "target pattern contains no `%'" error. The way I chose to
fix this is to split the g++ command into two separate passes, one
that creates the dependency file and another that compiles the file.
The existing code tries to do both of these in a single pass. I also
slightly modified the dependency generation command line from -MMD to -
MM, which forces g++ to exit after the preprocessing stage (so you
don't double your compile time with this fix). Then, before doing the
object code compile step, we insert the conversion to cygwin for
Windows.

The second half of the fix is to build/awk/convert-deps-to-cygwin.awk
which was written to assume that each file was on a separate line.
This is not always the case with shorter path names. Here's a better
version of the awk program that tests each token along all lines. It
has the benefit of not needing two separate conditionals since the
tokenization will handle the prepended space that was handled by the
second conditional in the original code. Apologies, I'm sure the awk
experts out there could probably find a slightly more elegant
solution, but at least this one works (I had a dependency file that
had the target and first dependency on the first line which was missed
by the old version):

41,51c41,51
< # First case: D:/Stuff/source.o: \
< if ( $0 ~ /^[A-Za-z]:/ ) {
< print CYGDRIVE_PREFIX tolower(substr($0,1,1)) "/"
substr($0,4)
< }
< # Second case: <space>D:/Stuff/source.h
< else if ( $0 ~ /^ [A-Za-z]:/ ) {
< print " " CYGDRIVE_PREFIX tolower(substr($0,2,1)) "/"
substr($0,5)
< }
< else {
< print $0
< }
---
> for (n = 1; n <= NF; n++) {
> if ($n ~ /^[A-Za-z]:/) {
> printf("%s%s/%s", CYGDRIVE_PREFIX, tolower(substr($n,1,1)), substr($n,4))
> } else {
> printf("%s", $n)
> }
> if (n < NF) {
> printf(" ")
> }
> }
> printf("\n")


David, I hope this helps and gives you another version for
comparison. Obviously, I've only tested this under Windows, but I
don't think this will break other platforms. In the meantime, this
should help windows developers continue to work under Cygwin until the
new patch is ready.


Dan

> > android-ndk...@googlegroups.com<android-ndk%2Bunsubscribe@googlegr oups.com>
> > .

[Doug Schaefer]

I'd also like to see a fix for msys. If cygpath isn't in your path,
don't call it. Everything else seems to be working fine.

> To unsubscribe from this group, send email to android-ndk...@googlegroups.com.

[SChaser]

Thanks! This solved it for me.

[David Turner]

On Tue, Dec 28, 2010 at 10:36 PM, Doug Schaefer <cdt...@gmail.com> wrote:

I'd also like to see a fix for msys. If cygpath isn't in your path,
don't call it. Everything else seems to be working fine.

I'm doubtful, but if you say so, I'll add some code to ignore the path translation if cygpath is not in the path (sorry, don't have the time to test that myself, I'm trying to get rid of Cygwin and its bastard children :-))

[Doug Schaefer]

Failing that, I've got the CDT to put a dummy cygpath into the PATH.
It's ugly but it's working :).

[Alex Cohn]

For cygwin, you don't need that awk(ward) script at all. I suggest the
simple fix for definitions.mk:

cmd-convert-deps-to-cygwin = \
sed "s/\([A-Za-z]\):\//\/cygdrive\/\l\1\//g" -i $1

To be more precise,

907,909c907
< mv $1 $1.org && \
< $(HOST_AWK) -f $(BUILD_AWK)/convert-deps-to-cygwin.awk $1.org >
$1 && \
< rm -f $1.org
---
> sed "s/\([A-Za-z]\):\//\/cygdrive\/\l\1\//g" -i $1
930c928

< $(hide) $$(PRIVATE_CC) -MMD -MP -MF $$(PRIVATE_DEPS) $$
(PRIVATE_CFLAGS) $$(call host-path,$$(PRIVATE_SRC)) -o $$(call host-
path,$$(PRIVATE_OBJ))
---
> $(hide) $$(PRIVATE_CC) -MM -MP -MF $$(PRIVATE_DEPS) $$(PRIVATE_CFLAGS) $$(call host-path,$$(PRIVATE_SRC))

933a932
> $(hide) $$(PRIVATE_CC) $$(PRIVATE_CFLAGS) $$(call host-path,$$(PRIVATE_SRC)) -o $$(call host-path,$$(PRIVATE_OBJ))

Enjoy,
Alex

[Vince Castellano]

Not sure where to reply to on this thread, but with NDK r5, I have an
interesting reproduction case.

Occasionally, my .o.d files will have 2 files on one line, and when
this happens, only the first file is attempted to be converted to
cygdrive. The second is left as a windows path, and breaks future
compilation without deleting this file first.

I do not know what causes the .o.d file to get 2 files on one line,
but either the awk script should be extended to support this, the
generator of .o.d file should not allow more that one per line, or use
Alex's suggestion of a sed script instead. The sed script is actually
the fix I put on my ndk, and it's now working fine.

Thanks,
Vince

> > that creates thedependencyfile and another that compiles the file.

> > The existing code tries to do both of these in a single pass.  I also

> > slightly modified thedependencygeneration command line from -MMD to -

> > MM, which forces g++ to exit after the preprocessing stage (so you
> > don't double your compile time with this fix).  Then, before doing the
> > object code compile step, we insert the conversion to cygwin for
> > Windows.
>
> > The second half of the fix is to build/awk/convert-deps-to-cygwin.awk
> > which was written to assume that each file was on a separate line.
> > This is not always the case with shorter path names.  Here's a better
> > version of the awk program that tests each token along all lines.  It
> > has the benefit of not needing two separate conditionals since the
> > tokenization will handle the prepended space that was handled by the
> > second conditional in the original code.  Apologies, I'm sure the awk
> > experts out there could probably find a slightly more elegant

> > solution, but at least this one works (I had adependencyfile that
> > had the target and firstdependencyon the first line which was missed

[David Turner]

This is fixed in the coming update NDK r5b. In the meantime, you can grab the following file from the AOSP tree and replace the one in your installation:

http://android.git.kernel.org/?p=platform/ndk.git;a=blob;f=build/awk/convert-deps-to-cygwin.awk;h=5dc36463c2341e8b311e690da838467d94f592e3;hb=refs/heads/master

To unsubscribe from this group, send email to android-ndk...@googlegroups.com.

[Wex]

I've noticed another dependency related problem, even with the fixes
to the dependency file generation mentioned above.

When a header file is changed, the associated .cpp file is not
recompiled, even when it is properly listed in the .d file as a
dependency. Has anyone else had this problem?

The .o file is correctly recompiled when the .cpp file is changed, but
it is not recompiled when any of the headers are changed.

One thing about our source layout is that the source files do not live
within the jni subdirectory in the android build tree. We refer to
the external source files in Android.mk and everything builds and
links correctly. The resuling .o, .d and other files generated during
the build all go to their normal places in the android build tree --
just the source files are external. I'm not sure why this would cause
a problem, or if it is the problem.

Kudos to AlexC for his improvement on my awk fixes. The sed script
looks much cleaner.


Wex

On Jan 21, 11:07 am, David Turner <di...@android.com> wrote:
> This is fixed in the coming update NDK r5b. In the meantime, you can grab
> the following file from the AOSP tree and replace the one in your
> installation:
>

> http://android.git.kernel.org/?p=platform/ndk.git;a=blob;f=build/awk/...
> <http://android.git.kernel.org/?p=platform/ndk.git;a=blob;f=build/awk/...>

[David Turner]

On Wed, Jan 26, 2011 at 2:58 AM, Wex <danw...@gmail.com> wrote:

I've noticed another dependency related problem, even with the fixes
to the dependency file generation mentioned above.

When a header file is changed, the associated .cpp file is not
recompiled, even when it is properly listed in the .d file as a
dependency.  Has anyone else had this problem?

The .o file is correctly recompiled when the .cpp file is changed, but
it is not recompiled when any of the headers are changed.

Do you have an example .d file to see its content. Maybe the issue is related to relative paths.

Feel free to file a bug at b.android.com and attach the files or a sample to reproduce the problem. This will help a lot.

Thanks

 

To unsubscribe from this group, send email to android-ndk...@googlegroups.com.

[Wex]

I will file a bug and submit source examples shortly if I don't find a
solution. In the meantime, further debugging has discovered a bug in
the fixes to definitions.mk mentioned earlier in this thread which
broke dependencies for local files. The -MT option is required on the
dependency generation line. So, please use:

$(hide) $$(PRIVATE_CC) -MM -MP -MT '$$(call host-path,$$
(PRIVATE_OBJ))' -MF $$(PRIVATE_DEPS) $$(PRIVATE_CFLAGS) $$(call host-
path,$$(PRIVATE_SRC))

instead of the suggestion above in definitions.mk:930. The -MT forces
a full path to the object file instead of just a local filename.

This still does not fix dependencies for external sources. Looking at
the output from 'ndk-build -p' it seems as though external files are
listed multiple times, with different paths, which may account for the
dependency problems. After I try a few more things, I'll create a bug
with an example if I can't figure out a solution.


Wex

On Jan 26, 3:53 am, David Turner <di...@android.com> wrote:

> ...
>
> read more »

[Wex]

One more fix, this time with gusto!

The -MT solution above was very close, but the following minor change,
which uses -MT '$$(PROJECT_OBJ)' instead of using the host-path macro,
fixes all known issues, including proper rebuilding of EXTERNAL source
files when a header changes:

$(hide) $$(PRIVATE_CC) -MM -MP -MT '$$(PRIVATE_OBJ)' -MF $$
(PRIVATE_DEPS) $$(PRIVATE_CFLAGS) $$(call host-path,$$(PRIVATE_SRC))


The host-path macro converts from Windows to cygwin paths (see
definitions.mk:118), but the $$(PROJECT_OBJ) variable is already a
cygwin path, works fine as input to gcc, and, regardless, we convert
Windows to cygwin paths using the sed (or awk) script anyway.

As a result, I don't think I need to file a bug on this issue. David,
please let me know if you would still like me to file a bug along with
the suggested fix for tracking.

Hope this helps folks out there. I know we've seen this issue in
multiple production code bases.


Wex

> ...
>
> read more »

[David Turner]

On Wed, Jan 26, 2011 at 7:51 PM, Wex <danw...@gmail.com> wrote:

One more fix, this time with gusto!

The -MT solution above was very close, but the following minor change,
which uses -MT '$$(PROJECT_OBJ)' instead of using the host-path macro,
fixes all known issues, including proper rebuilding of EXTERNAL source
files when a header changes:

$(hide) $$(PRIVATE_CC) -MM -MP -MT '$$(PRIVATE_OBJ)' -MF $$

(PRIVATE_DEPS) $$(PRIVATE_CFLAGS) $$(call host-path,$$(PRIVATE_SRC))

The host-path macro converts from Windows to cygwin paths (see
definitions.mk:118), but the $$(PROJECT_OBJ) variable is already a
cygwin path, works fine as input to gcc, and, regardless, we convert
Windows to cygwin paths using the sed (or awk) script anyway.

I see, I think the cygwin path is *not* fine as input to gcc, but that what you mean is that it doesn't matter which form is used in the generated dependency file since it will later be translated by the awk script, which is true. I still don't understand why the header dependency stuff doesn't work or is fixed by that change, but will test.

 

As a result, I don't think I need to file a bug on this issue.  David,
please let me know if you would still like me to file a bug along with
the suggested fix for tracking.

Oh, for God's sake, please file a bug. There are so many things going on that I'll probably forget about the problem if you don't !

Or better, submit a patch to r.android.com :-)

 

Thanks in advance.

> ...
>
> read more »

--
You received this message because you are subscribed to the Google Groups "android-ndk" group.
To post to this group, send email to andro...@googlegroups.com.

To unsubscribe from this group, send email to android-ndk...@googlegroups.com.

[Wex]

> you mean is that it doesn't matter which form is used in the generated
> dependency file since it will later be translated by the awk script, which
> is true.

Yeah, that's what I meant.

> I still don't understand why the header dependency stuff doesn't
> work or is fixed by that change, but will test.

External headers are fixed by this change because the -MT option
specifies the exact text that is entered into the .d file for the
target. The problem with external dependencies is that make was
referring to the same file using two different absolute paths. The
main rule for the .cpp file had relative directories within an
absolute path (e.g. /cygpath/c/foo/bar/../../bam/boom/goo.cpp) whereas
the cygpath -m done by the host-path macro in definitions.mk resulted
in /cygpath/c/bam/boom/goo.cpp. By switching to $$(PROJECT_OBJ)
instead of '$$(call host-path,$$(PRIVATE_OBJ)), the .d file now used
the former, absolute-with-relative path.

For future reference, and as a tip for ndk-build and make debugging, I
tracked this down using the -p option, which prints out the make
database, including dependency rules. By careful examination of what
was going on for local vs. external files, I was able to see that
there were two separate dependency entries for external files, and
only one for internal files. Usually, I just pass the -p output
through grep to minimize the verbosity.

> Oh, for God's sake, please file a bug. There are so many things going on
> that I'll probably forget about the problem if you don't !
> Or better, submit a patch to r.android.com :-)

Will do. Happy to help. Thanks,


Wex

[David Turner]

On Wed, Jan 26, 2011 at 11:59 PM, Wex <danw...@gmail.com> wrote:

> you mean is that it doesn't matter which form is used in the generated
> dependency file since it will later be translated by the awk script, which
> is true.

Yeah, that's what I meant.

> I still don't understand why the header dependency stuff doesn't
> work or is fixed by that change, but will test.

External headers are fixed by this change because the -MT option
specifies the exact text that is entered into the .d file for the
target.  The problem with external dependencies is that make was
referring to the same file using two different absolute paths.  The
main rule for the .cpp file had relative directories within an
absolute path (e.g. /cygpath/c/foo/bar/../../bam/boom/goo.cpp) whereas
the cygpath -m done by the host-path macro in definitions.mk resulted
in /cygpath/c/bam/boom/goo.cpp.  By switching to $$(PROJECT_OBJ)
instead of '$$(call host-path,$$(PRIVATE_OBJ)), the .d file now used
the former, absolute-with-relative path.

Ah, thanks, that makes sense: The dependencies were not attached to the same GNU Make target (which is a simple string, internally).

 

For future reference, and as a tip for ndk-build and make debugging, I
tracked this down using the -p option, which prints out the make
database, including dependency rules.  By careful examination of what
was going on for local vs. external files, I was able to see that
there were two separate dependency entries for external files, and
only one for internal files.  Usually, I just pass the -p output
through grep to minimize the verbosity.

> Oh, for God's sake, please file a bug. There are so many things going on
> that I'll probably forget about the problem if you don't !
> Or better, submit a patch to r.android.com :-)

Will do.  Happy to help.  Thanks,


Wex

--

[Doug Schaefer]

BTW, I can confirm that they cygpath trick in r5b works well with
MSYS. Thanks, David!

Doug.

[Wex]

Uploaded patch I4ec8292a with these fixes with DavidT as reviewer.


W00t! My very first official contribution to an Open Source project!
Whew, had to set up a 64-bit Ubuntu box and learn about git/repo/ssh,
but now I'm good to go.

Compiling Android apps under Ubuntu is *much* faster than
Win7+Cygwin. Like 3-4x.


On Jan 26, 2:38 pm, David Turner <di...@android.com> wrote:

> > > > > > > > > <         print CYGDRIVE_PREFIX...
>
> read more »