Failure to build apk after unexpected format of call to "ant" in new p4a

[B Fiwoc]

I'm using the new (master) branch of python-for-android (at commit 93728aa), on top of a clean, up-to-date install of Debian jessie; all package dependencies are installed as listed in the p4a readme docs. I'm finding the error shown below.

I see the same error (below) whether I call p4a directly, or if I use buildozer to call it (via "android_new").  The old-toolchain of python-for-android works fine on this system to produce APKs.  The error below seems strange to me, as the command being called by p4a: 

  /opt/android-sdk-linux/tools/ant debug

doesn't seem right to me (more below).

Here's my call to p4a:

  p4a apk --sdk_dir /opt/android-sdk-linux --ndk_dir /opt/android-ndk-r10e \

          --android_api 15 --ndk_ver r10e-rc4 --private /root/HelloWorld \

 --package=org.hw.helloworld --name="HelloWorld" \

 --version=0.5 --bootstrap=pygame --requirements=sdl,python2 \

 --dist_name=helloworld

Here's some initial logging output from p4a showing things going well:

  [INFO]:    This python-for-android revamp is an experimental alpha release!

  [INFO]:    It should work (mostly), but you may experience missing features or bugs.

  [INFO]:    Will compile for the following archs: armeabi

  [INFO]:    Getting Android API version from user argument

  [INFO]:    Available Android APIs are (15)

  [INFO]:    Requested API target 15 is available, continuing.

  [INFO]:    Getting NDK dir from from user argument

  [INFO]:    Got NDK version from from user argument

  [INFO]:    Using Google NDK r10e-rc4

  [INFO]:    Found virtualenv at /usr/local/bin/virtualenv

  [INFO]:    Found the following toolchain versions: ['4.8', '4.9', 'clang3.5', 'clang3.6']

  [INFO]:    Picking the latest gcc toolchain, here 4.9

  [INFO]:    Of the existing distributions, the following meet the given requirements:

  [INFO]:       helloworld: includes recipes (hostpython2, pygame_bootstrap_components, python2, sdl, six, pygame, pyjnius, android, kivy), built for archs (armeabi)

  [INFO]:    helloworld has compatible recipes, using this one

  [INFO]:    Of the existing distributions, the following meet the given requirements:

  [INFO]:       helloworld: includes recipes (hostpython2, pygame_bootstrap_components, python2, sdl, six, pygame, pyjnius, android, kivy), built for archs (armeabi)

  [INFO]:    helloworld has compatible recipes, using this one

  [INFO]:    -> directory context /root/.local/share/python-for-android/dists/helloworld

  <snip>

Here's the relevant snippet of log output showing the error:

  [INFO]:    -> running ant debug

  [INFO]:    STDOUT:                                                          

    Traceback (most recent call last):    

    File "/usr/local/bin/p4a", line 9, in <module>    

      load_entry_point('python-for-android==0.3', 'console_scripts', 'p4a')() 

    File "/usr/local/lib/python2.7/dist-packages/pythonforandroid/toolchain.py", line 708, in main

      ToolchainCL()       

    File "/usr/local/lib/python2.7/dist-packages/pythonforandroid/toolchain.py", line 323, in __init__

      getattr(self, args.command)(unknown)       

    File "/usr/local/lib/python2.7/dist-packages/pythonforandroid/toolchain.py", line 106, in wrapper_func

      func(self, args)       

    File "/usr/local/lib/python2.7/dist-packages/pythonforandroid/toolchain.py", line 556, in apk

      shprint(sh.ant, 'debug', _tail=20, _critical=True)       

    File "/usr/local/lib/python2.7/dist-packages/pythonforandroid/logger.py", line 164, in shprint

      output = command(*args, **kwargs)    

    File "/usr/lib/python2.7/dist-packages/sh.py", line 769, in __call__    

      return RunningCommand(cmd, call_args, stdin, stdout, stderr)    

    File "/usr/lib/python2.7/dist-packages/sh.py", line 327, in __init__    

      self.call_args, pipe=pipe)  

    File "/usr/lib/python2.7/dist-packages/sh.py", line 891, in __init__

      os.execv(cmd[0], cmd)  

  OSError: [Errno 13] Permission denied

  [INFO]:    STDERR:

    

  [INFO]:    COMMAND:

  cd /root/.local/share/python-for-android/dists/helloworld && /opt/android-sdk-linux/tools/ant debug

  

  [WARNING]: ERROR: /opt/android-sdk-linux/tools/ant failed!

But, the command being executed ("/opt/android-sdk-linux/tools/ant") is referencing what is the ant *DIRECTORY*, not any ant executable (so the permission denied error message is unsurprising).  At least that's the internal structure of the Android SDK (r24.4.1) I downloaded and extracted earlier today:

  root@laptop:~# ls -d /opt/android-sdk-linux/tools/ant/*

  /opt/android-sdk-linux/tools/ant/build.xml

  /opt/android-sdk-linux/tools/ant/NOTICE

  /opt/android-sdk-linux/tools/ant/uibuild.xml

I've tried to run p4a as both root, and as a normal user, just to see if it mattered (it did not, as I'd suspected).

I searched online, both in Kivy/P4A google groups and elsewhere, and couldn't find any mention of this issue.  Any input is very welcome.

Thanks!

[Alexander Taylor]

It looks like python-for-android just tries to call the ant executable, and works for others because ant is on the path (for me, it's in /usr/bin/ant which is before the sdk tools dir in my $PATH). Do you have an ant executable somewhere else?

This seems like a p4a bug anyway, are you able to make a github issue about it?

[B Fiwoc]

Thanks for your reply.

I should have perhaps been more clear in my original post.  Yes, python-for-android is definitely trying to call the ant executable, that's clear --- what strikes me as strange, though, are two aspects of this call:

 (1) the call to ant uses a fully-canonicalized path (shown above, in log from my run as: "/opt/android-sdk-linux/tools/ant").  So, irrespective of my particular $PATH setting, or location of ant on my system, it's not going to search any further locations than that one.  Incidentally, though not mentioned above (I should have), my version of ant was installed via standard Debian (Jessie) package, so the primary ant executable lives at normal location of "/usr/bin/ant".  (If the solution were as simple as more careful construction of my $PATH env variable, I probably wouldn't have even posted here.)

 (2) the fully-canonicalized path that python-for-android calls ("/opt/android-sdk-linux/tools/ant") just happens to correspond to that of a directory for ant in the android SDK as extracted (r24.4.1), so there is indeed an "ant" object at that location, it's just that it's an ant directory, not the expected ant binary executable.  Weird to see this coincidence (?), and not sure what to make of it.

I tried to poke around the python-for-android source to see if I could identify where this fully-canonicalized path ("/opt/android-sdk-linux/tools/ant") was coming from, but nothing stood out.

Any further thoughts in this forum, before I convert this to a github bug report?

[Alexander Taylor]

Regarding (1), p4a calls ant using `sh.ant`, and the sh module locates
this executable in the $PATH and runs it via this location. I don't know
why it would get it wrong if you have it in /usr/bin - if you run
`whereis ant` outside p4a what order do the results appear in? p4a does
also possibly manipulate the PATH in a way that could affect things though.

Regarding (2), I think it's locating a directory becuase directories
have the executable bit set and therefore appear in e.g. `whereis`.

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Python For Android" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/python-android/2fLGqp6Bcec/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> python-androi...@googlegroups.com
> <mailto:python-androi...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[B Fiwoc]

Yes, I found `sh.ant` in p4a source, but didn't track down it's complete resolution process to canonicalize "ant".  I agree that it seems weird that "sh.ant" found the wrong ant (a directory, and not a binary, in "/opt" of all places.  See below for output from my system.  Just a reminder that I installed python-for-android (+ dependencies, as listed in p4a readthedocs.org) on a completely virgin Debian jessie (to reduce side-effects of other stuff in identifying bugs such as this).

Anyway, here's the output on my system from a few key commands:

root@laptop:~# whereis ant

ant: /usr/bin/ant /usr/share/ant /usr/share/man/man1/ant.1.gz

root@laptop:~# which ant

/usr/bin/ant

root@laptop:~# ls -l /usr/bin/ant /usr/share/ant

lrwxrwxrwx 1 root root   20 Oct  7  2014 /usr/bin/ant -> ../share/ant/bin/ant

/usr/share/ant:

total 12

drwxr-xr-x 2 root root 4096 Mar  7 21:02 bin

drwxr-xr-x 3 root root 4096 Mar  7 21:02 etc

drwxr-xr-x 2 root root 4096 Mar  7 21:02 lib

root@laptop:~# ls -l /usr/bin/ant

lrwxrwxrwx 1 root root 20 Oct  7  2014 /usr/bin/ant -> ../share/ant/bin/ant

root@laptop:~# whereis -l

bin: /usr/bin

bin: /usr/sbin

bin: /usr/lib

bin: /bin

bin: /sbin

bin: /etc

bin: /lib

bin: /lib64

bin: /usr/games

bin: /usr/local/bin

bin: /usr/local/sbin

bin: /usr/local/etc

bin: /usr/local/lib

bin: /usr/local/games

bin: /usr/include

bin: /usr/local

bin: /usr/share

man: /usr/share/man/fr

man: /usr/share/man/man7

man: /usr/share/man/es

man: /usr/share/man/sr

man: /usr/share/man/pl

man: /usr/share/man/sl

man: /usr/share/man/cs

man: /usr/share/man/lt

man: /usr/share/man/gl

man: /usr/share/man/man2

man: /usr/share/man/man5

man: /usr/share/man/da

man: /usr/share/man/man6

man: /usr/share/man/ar

man: /usr/share/man/man3

man: /usr/share/man/uk

man: /usr/share/man/ug

man: /usr/share/man/id

man: /usr/share/man/hu

man: /usr/share/man/sk

man: /usr/share/man/zh_TW

man: /usr/share/man/ru

man: /usr/share/man/ko

man: /usr/share/man/el

man: /usr/share/man/de

man: /usr/share/man/man4

man: /usr/share/man/ja

man: /usr/share/man/pt

man: /usr/share/man/sv

man: /usr/share/man/fi

man: /usr/share/man/tr

man: /usr/share/man/zh_CN

man: /usr/share/man/nl

man: /usr/share/man/it

man: /usr/share/man/ca

man: /usr/share/man/man8

man: /usr/share/man/man1

man: /usr/share/man/pt_BR

root@laptop:~# 

It's interesting to note that "whereis ant" does not actually find the "/opt/android-sdk-linux/tools/ant" command invocation, as identified by "sh.ant" in p4a.  Instead, it correctly identifies "/usr/bin/ant" (among others).  Further, it's apparent that whereis does not even *look* in the "/opt" directory (see "whereis -l" above), so "sh.ant" must be getting that from somewhere else.  One possibility to consider is that you'll note (from my previous message here) that I use the "--sdk" command parameter to pass the location of the Android SDK to p4a, rather than using an environmental variable; according to p4a docs, that's fully supported, though.

I'll turn this into a bug report on github...

Thanks.

[B Fiwoc]

My bug report was very quickly resolved on github issue tracking (Thanks!), with the more explicit requirement that p4a depend on python-sh>1.10, rather than just python-sh.  So far, so good.  Unfortunately, we seem to be triggering a new bug in the same toolchain now.  I'm attempting still to manually create an apk by directly invoking p4a as follows:

  p4a apk --sdk_dir /opt/android-sdk-linux \

          --ndk_dir /opt/android-ndk-r10e \

          --android_api 15 --ndk_ver r10e-rc4 \

          --private /root/HelloWorld \

          --package=org.example.helloworld \

          --name="HelloWorld" --version=0.5 \

          --bootstrap=pygame --requirements=sdl,python2 \

          --dist_name=helloworld

P4a (or, more specifically, python-sh) is now properly resolving "ant"; however, ant is failing due to an error in the build.xml file:

  Buildfile: /root/.local/share/python-for-android/dists/helloworld/build.xml

    

  BUILD FAILED

  /root/.local/share/python-for-android/dists/helloworld/build.xml:18: Cannot find /opt/android-sdk/tools/ant/build.xml imported from /root/.local/share/python-for-android/dists/helloworld/build.xml 

Specifically, my dist's build.xml seems to be trying to import the Android SDK's build.xml from the wrong fully canonicalized file location.  Above, note that it looks for it in:

  /opt/android-sdk/tools/ant/build.xml

However, the p4a invocation clearly passes the SDK directory parameter as:

  /opt/android-sdk-linux

Somehow, p4a is going from "/opt/android-sdk-linux" to "/opt/android-sdk" and thus failing to find the file.

After some digging, I note that this is really due to use of the pygame bootstrap; more specifically, "/root/.local/share/python-for-android/dists/helloworld/build.xml" includes "/root/.local/share/python-for-android/dists/helloworld/local.properties", which itself contains the incorrect SDK directory parameter ("/opt/android-sdk", rather than "/opt/android-sdk-linux").

If we look at "/usr/local/lib/python2.7/dist-packages/pythonforandroid/bootstraps/pygame/__init__.py", we see this code snippet:

  shprint(sh.cp, '-a', join(src_path, 'local.properties'), '.')

So, the pygame bootstrap is simply copying the default "local.properties" (as in the git repo for python-for-android) into my particular dist.  I'm assuming that the intention here is that the "local.properties" file is supposed to be a fallback default location for SDK directory, and that my command line parameter ("--sdk_dir /opt/android-sdk-linux" above) at p4a invocation should supersede this default.  Clearly, that's not happening.

In fact, though, if I manually change the "local.properties" file to match the correct SDK directory, everything works fine.  Still, there's a bug somewhere here, and I'd like to resolve it, so things work properly.

Thanks!

[Alexander Taylor]

Feel free to open a github issue about this. Does it also affect the
sdl2 bootstrap?

[B Fiwoc]

Done.  Listed as issue #684.  As detailed in that bug report, the SDL2 bootstrap does *NOT* trigger this issue (alas it triggers a different one), and it is also not triggered when pygame is used with buildozer.  Thanks.