Buildozer APK crashes: Caused by: java.io.FileNotFoundException: /proc/net/xt_qtaguid/stats: open failed: EACCES (Permission denied)

[Shoumik Das]

Hi. I have compiled an apk with default values for android sdk, ndk and minimum api version. The app installs on the device but crashes when launched. Following is the logcat output. It appears as if the permissions are not set correctly. This is what is specified in my buildozer spec file:

android.permissions = INTERNET,ACCESS_NETWORK_STATE,ACCESS_WIFI_STATE,CHANGE_WIFI_STATE,CHANGE_WIFI_MULTICAST_STATE,READ_EXTERNAL_STORAGE,WRITE_EXTERNAL_STORAGE,MANAGE_DOCUMENTS,MANAGE_EXTERNAL_STORAGE,READ_PHONE_STATE

I have also include runtime permissions inside my main Python file:

from android.permissions import request_permissions, Permission

request_permissions([Permission.WRITE_EXTERNAL_STORAGE,Permission.INTERNET])

The requirements are specified as follows:

requirements = python3,kivy==2.0.0rc3,tweepy,kivmob,mysql_connector

A similar app with slightly less requirements (only Python and Kivy) has compiled and worked fine on the device. I am not sure why this one is not working. Logcat and spec file are attached for reference.

Logcat output:

--------- beginning of crash

11-28 10:20:35.120 13466 13466 E AndroidRuntime: FATAL EXCEPTION: main

11-28 10:20:35.120 13466 13466 E AndroidRuntime: Process: com.google.android.gm, PID: 13466

11-28 10:20:35.120 13466 13466 E AndroidRuntime: android.app.RemoteServiceException: Context.startForegroundService() did not then call Service.startForeground(): ServiceRecord{8311ffa u0 com.google.android.gm/com.android.email.service.AttachmentService}

11-28 10:20:35.120 13466 13466 E AndroidRuntime: at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2137)

11-28 10:20:35.120 13466 13466 E AndroidRuntime: at android.os.Handler.dispatchMessage(Handler.java:107)

11-28 10:20:35.120 13466 13466 E AndroidRuntime: at android.os.Looper.loop(Looper.java:237)

11-28 10:20:35.120 13466 13466 E AndroidRuntime: at android.app.ActivityThread.main(ActivityThread.java:7948)

11-28 10:20:35.120 13466 13466 E AndroidRuntime: at java.lang.reflect.Method.invoke(Native Method)

11-28 10:20:35.120 13466 13466 E AndroidRuntime: at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:493)

11-28 10:20:35.120 13466 13466 E AndroidRuntime: at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1075)

If I search with "Caused by", this is what I get:

26060 26194 E fb4a.FbTrafficStats: Caused by: java.io.FileNotFoundException: /proc/net/xt_qtaguid/stats: open failed: EACCES (Permission denied)

11-29 21:40:34.231 26060 26194 E fb4a.FbTrafficStats: at libcore.io.IoBridge.open(IoBridge.java:496)

11-29 21:40:34.231 26060 26194 E fb4a.FbTrafficStats: at java.io.FileInputStream.<init>(FileInputStream.java:159)

11-29 21:40:34.231 26060 26194 E fb4a.FbTrafficStats: at X.1Ln.A00(:15)

11-29 21:40:34.231 26060 26194 E fb4a.FbTrafficStats: ... 17 more

11-29 21:40:34.231 26060 26194 E fb4a.FbTrafficStats: Caused by: android.system.ErrnoException: open failed: EACCES (Permission denied)

11-29 21:40:34.231 26060 26194 E fb4a.FbTrafficStats: at libcore.io.Linux.open(Native Method)

11-29 21:40:34.231 26060 26194 E fb4a.FbTrafficStats: at libcore.io.ForwardingOs.open(ForwardingOs.java:167)

11-29 21:40:34.231 26060 26194 E fb4a.FbTrafficStats: at libcore.io.BlockGuardOs.open(BlockGuardOs.java:252)

11-29 21:40:34.231 26060 26194 E fb4a.FbTrafficStats: at libcore.io.ForwardingOs.open(ForwardingOs.java:167)

11-29 21:40:34.231 26060 26194 E fb4a.FbTrafficStats: at android.app.ActivityThread$AndroidOs.open(ActivityThread.java:7825)

11-29 21:40:34.231 26060 26194 E fb4a.FbTrafficStats: at libcore.io.IoBridge.open(IoBridge.java:482)

11-29 21:40:34.231 26060 26194 E fb4a.FbTrafficStats: ... 19 more

11-29 21:40:34.234 26060 26194 E fb4a.FbTrafficStats: Unable to parse data usage from system file

11-29 21:40:34.234 26060 26194 E fb4a.FbTrafficStats: X.1Of: java.io.FileNotFoundException: /proc/net/xt_qtaguid/stats: open failed: EACCES (Permission denied)

[planckp...@gmail.com]

com.google.android.gm is gmail, ..........?

java.io.FileNotFoundException: /proc/net/xt_qtaguid/stats: open failed: EACCES (Permission denied)

Google search finds:  https://source.android.com/devices/tech/datausage/kernel-overview

As I read that it is related to sockets.

A similar app with slightly less requirements (only Python and Kivy) has compiled and worked fine on the device. I

Step by step, start looking at what is different.

[Shoumik Das]

Hi. I have been trying numerous compile options and debugging throughout the day and this is what I found:

1. From the kernel overview link, I tried using android.permission.MODIFY_NETWORK_ACCOUNTING as a permission but it didn't quite work. 'buildozer android clean' indicated an error with this permission.

2. Took a simple single screen app and kept on adding custom modules until it broke (like you suggested).

3. Seems like pysftp is breaking the build. I am using pysftp as a module in my code. Even if I include pysftp in the requirements, the apk crashes upon on launch.

I did some digging around and found the following issue:

https://github.com/kivy/buildozer/issues/1050

The Python package CFFI cannot be added as a Buildozer requirement. As a minimal reproducible example on Ubuntu 18.04

Someone commented: Got same error recently, it s currently fixed in develop branch of pythonforandroid.

Another link points out the following: Bumps cffi==1.13.2 fixes under Python 3.8

https://github.com/kivy/python-for-android/commit/10cf15868c08acc755d421a26309ffcdb6c0aeac

How should I proceed in this case? a) Use the development branch of p4a or b) use cffi==1.13.2  in the requirements section?

Please advise the recommended approach. Does my investigation look correct?

Thanks

[planckp...@gmail.com]

pysftp:

2. Took a simple single screen app and kept on adding custom modules until it broke (like you suggested).

Next step by step remove all the other stuff so you can see that it really is that last module.

If you want to take a wild jump write an app that uses only pysftp. Really simple send or receive a file at a button push, no screens just a button.

Even if I include pysftp in the requirements, the apk crashes upon on launch.

It seems likely you would have to do this, but maybe there is something about the project that I don't know.

A quick look https://pypi.org/project/pysftp/ shows it tested up to Python 3.4 or 3.5 (depending where on the page I look!). I don't know the Python version used there on the desktop, but Android is using 3.8. I'm not saying this is the issue, but when starting the project if would have been a reason to avoid or try a test case with pysftp. A lesson for the future. If the author is not maintaining it, then you have to maintain your copy.

cffi:

I did some digging around and found the following issue:

https://github.com/kivy/buildozer/issues/1050

Looks like the cffi version fix went into p4a release in April

https://github.com/kivy/python-for-android/blob/v2020.04.29/pythonforandroid/recipes/cffi/__init__.py

I'd say that probably not the issue.

[Shoumik Das]

Hello. I am listing below the results from several rounds of testing and compiling. I found some missing dependencies and specified them in the Buildozer requirements. Apaprently, pysftp needs paramiko and paramiko needs ecdsa and cryptography which, in turn, needs bcrypt and pynacl.

requirements = python3,kivy==2.0.0rc3,cffi==1.13.2,ecdsa,cryptography,bcrypt,pynacl,paramiko,pysftp

Runtime Permissions are as follows:

from android.permissions import request_permissions, Permission

request_permissions([Permission.WRITE_EXTERNAL_STORAGE,Permission.INTERNET,Permission.ACCESS_NETWORK_STATE,Permission.ACCESS_WIFI_STATE,Permission.MANAGE_DOCUMENTS,Permission.READ_PHONE_STATE])

Now, the app does not crash on launch. It opens up successfully. When I press on the button this is what happens:

Sequence of steps:

1. Creates a new log file and writes a line of text. --- Successfully done. This validates that file create and write permissions are working fine on Android.

2. Attach the newly created log file and email to my GMail address. --  Validates read and write access again.

3. Use pysftp to upload a hard-code sample file to a remote server - The app crashes. So this clearly proves that the app is crashing at the pysftp command execution step.

I got a logcat output: The only error that I am getting is this: 

12-01 16:28:56.882  8719  8851 E msgr.FbTrafficStats: Caused by: java.io.FileNotFoundException: /proc/net/xt_qtaguid/stats: open failed: EACCES (Permission denied)

12-01 16:28:56.882  8719  8851 E msgr.FbTrafficStats: at libcore.io.IoBridge.open(IoBridge.java:496)

12-01 16:28:56.882  8719  8851 E msgr.FbTrafficStats: at java.io.FileInputStream.<init>(FileInputStream.java:159)

12-01 16:28:56.882  8719  8851 E msgr.FbTrafficStats: at X.194.A00(:15)

12-01 16:28:56.882  8719  8851 E msgr.FbTrafficStats: ... 17 more

12-01 16:28:56.882  8719  8851 E msgr.FbTrafficStats: Caused by: android.system.ErrnoException: open failed: EACCES (Permission denied)

12-01 16:28:56.882  8719  8851 E msgr.FbTrafficStats: at libcore.io.Linux.open(Native Method)

12-01 16:29:26.415  4717  7689 E Parcel  : Caused by: java.lang.ClassNotFoundException: com.google.android.apps.docs.download.DownloadSpec

12-01 16:29:26.415  4717  7689 E Parcel  : ... 23 more

There is no error/warning related to any missing module. This indicates a permission issue at Android level. I am so close to getting this right but it still doesn't give any clear error message. What am I doing wrong? The latest logcat is attached.

[planckp...@gmail.com]

This indicates a permission issue at Android level

This seems like a huge leap.

In the previous thread I 1) suggested making very small pysftp app to verify that using this package was the issue.

2) pysftp is a risky choice because its documentation only claims usage up to Python 3.5 and Android uses 3.8.

Did you try a pysftp (and dependencies) only test case?

No way for me to know what run time permissions your app requires, there is at least one (INTERNET) that is a manifest permission not a run time permission.

[Shoumik Das]

I made a very simple app using only a button and pysftp. The app did not crash upon launch. It crashed when I clicked the button to launch the actual sftp operation. The code works fine on my desktop. I have verified that the file being transferred is present in the same location as main.py. The error that I get in the logcat is this:

12-02 16:55:44.342 16961 16961 W System.err: Caused by: android.system.ErrnoException: open failed: ENOENT (No such file or directory)

12-02 16:55:44.342 16961 16961 W System.err: at libcore.io.Linux.open(Native Method)

12-02 16:55:44.342 16961 16961 W System.err: at libcore.io.ForwardingOs.open(ForwardingOs.java:167)

12-02 16:55:44.342 16961 16961 W System.err: at libcore.io.BlockGuardOs.open(BlockGuardOs.java:252)

12-02 16:55:44.343 16961 16961 W System.err: at libcore.io.ForwardingOs.open(ForwardingOs.java:167)

12-02 16:55:44.343 16961 16961 W System.err: at android.app.ActivityThread$AndroidOs.open(ActivityThread.java:7825)

12-02 16:55:44.343 16961 16961 W System.err: at libcore.io.IoBridge.open(IoBridge.java:482)

and

12-02 16:45:23.234 14957 15411 E fb4a.FbTrafficStats: Caused by: java.io.FileNotFoundException: /proc/net/xt_qtaguid/stats: open failed: EACCES (Permission denied)

12-02 16:45:23.234 14957 15411 E fb4a.FbTrafficStats: at libcore.io.IoBridge.open(IoBridge.java:496)

12-02 16:45:23.234 14957 15411 E fb4a.FbTrafficStats: at java.io.FileInputStream.<init>(FileInputStream.java:159)

12-02 16:45:23.234 14957 15411 E fb4a.FbTrafficStats: at X.1Ln.A00(:15)

12-02 16:45:23.234 14957 15411 E fb4a.FbTrafficStats: ... 8 more

12-02 16:45:23.234 14957 15411 E fb4a.FbTrafficStats: Caused by: android.system.ErrnoException: open failed: EACCES (Permission denied)

12-02 16:45:23.234 14957 15411 E fb4a.FbTrafficStats: at libcore.io.Linux.open(Native Method)

Couple of questions:

1) Is it necessary to set the current directory in Android at runtime? If so, how do I achieve it?

2) Are permissions specific to the api version? I am using defaults in the specfile.

3) If not pysftp, what other option do I have to securely transfer files from an Android app to a remote server? Passwordless transfer over ssh? Please suggest.

Here is the code I used:

import kivy

# kivy.require('1.11.1') # replace with your current kivy version

from kivy.app import App

from kivy.uix.button import Button

import sivasftp2 as sf

#from android.permissions import request_permissions, Permission

#request_permissions([Permission.READ_EXTERNAL_STORAGE,Permission.WRITE_EXTERNAL_STORAGE,Permission.INTERNET,Permission.ACCESS_NETWORK_STATE,Permission.ACCESS_WIFI_STATE,Permission.MANAGE_DOCUMENTS,Permission.READ_PHONE_STATE])

def sftp_upload(a):

print("Upload initiated...")

sf.upload_file('bot-96.png')

class MyApp(App):

def build(self):

btn1=Button(text='Upload',size_hint_x=0.2,size_hint_y=0.2, pos=(100,100))

btn1.bind(on_press=sftp_upload)

return btn1

if __name__=='__main__':

MyApp().run()

Thanks

[planckp...@gmail.com]

1) Is it necessary to set the current directory in Android at runtime? If so, how do I achieve it?

I've never tried to change cwd, the install directory is '.', the Android way is to specify the file path.

Attached a small runnable example of file access (just like your example above this is the way to test understanding)

2) Are permissions specific to the api version? I am using defaults in the specfile.

Generally not, but the need for say  WRITE_EXTERNAL_STORAGE has a history of changing.

FYI request_permissions() needs to be in the build() method, else you will get problems on_resume()

3) If not pysftp, what other option do I have to securely transfer files from an Android app to a remote server? Passwordless transfer o

No suggestion, I expect there are lots of alternatives.

[Shoumik Das]

Hi. I shall be running your code to understand how paths work. But before that, I tried a file copy operation with the standard shutil module. It seems that any shutil operation is not working and the app crashes upon button click with the same permission denied error. The app can successfully read and write files in the same location as main.py, which is ".". But, if I try to copy a file from the program's root directory to a folder in the same path, it fails. So, it seems that the issue is not just with pysftp but an overall access issue with the file system. I cannot seem to access sub-directories at the moment. I even tried using "." at the beginning of the sub-directory but even that didn't work. The code worked fine on the desktop, though.

Following is my code:

import shutil

import sivamail as sm

def siva_copy():

shutil.copyfile('bot-96.png', './tmpmedia/bot-96.png')

sm.send_email('copied_image','./tmpmedia/bot-96.png') # Email copied file. Proof of copy action within app.

[planckp...@gmail.com]

Yes absolutely, Android is not POSIX compliant, the file system not even close.

POSIX is so deep in out assumptions about computers, most people don't know it exists.

Don't know if I've ever used shutil, there is no Bash shell and I don't know about dependency.

Work in the app local storage.

[Shoumik Das]

Thanks for the information. That was really helpful. I shall work out something with the app local storage and post the results here.

--
You received this message because you are subscribed to the Google Groups "Kivy users support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kivy-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kivy-users/cd202c46-93da-4e9a-915c-a0ade5a46eb5n%40googlegroups.com.

[Shoumik Das]

I have a question: As per the documents, app_storage_path() gives you Android’s so-called “internal storage” which is specific to your app and cannot seen by others or the user. It compares best to the AppData directory on Windows.

primary_external_storage_path() returns Android’s so-called “primary external storage”, often found at /sdcard/ and potentially accessible to any other app. It compares best to the Documents directory on Windows. Requires Permission.WRITE_EXTERNAL_STORAGE to read and write to.

It is possible that a device may not use an external SD card. In such a case, is it advisable to use the app_storage_path() only? Also, if I retrieve the path values inside my App class and store them in some variables, the variables are local in scope to that class only. However, if I want to use these path variables in other classes and screens outside the App class, would it be advisable to declare them as global variableso or is there any other recommended technique?

class TestCryptApp(App):

def build(self):

#request_permissions([Permission.WRITE_EXTERNAL_STORAGE,Permission.READ_PHONE_STATE])

global app_path

app_path = app_storage_path

global log_path

log_path = os.path.join(app_path,'logs/sta.log')

global media_path

media_path = os.path.join(primary_external_storage_path(),'tmpmedia')

Thanks

[planckp...@gmail.com]

Android storage terminology is confusing as it has evolved. Just use the api calls as in the example. Running the example shows what they mean on your phone, almost certainly different paths on some other phone.

In the context of the Kivy life cycle, I would not use globals except as constants, so no global keyword.

Your example could be rewritten with 'logs/sta.log' as a static constant outside a class and no global statements.

Alternatively pass variables between classes, assumes class instantiation, outside the scope here see some Python tutorial.

Also there are Kivy properties (not the same as Python properties), I never use so I have nothing to say on the matter.

[Shoumik Das]

Hi! I worked on your suggestions and finally got my app working. Thanks a lot for the Android storage tips. I have managed to get most of my modules working as well except pysftp. Even the file copy operation with shutil is working fine now. For pysftp, I will have to look for other alternatives. Another important thing to note is that any use of the global keyword seemed to crash my app. However, if I declare global constants at the beginning of my code without using the global keyword, the app is working fine. Here is the working code:

from kivy.app import App

from kivy.uix.boxlayout import BoxLayout

from kivy.uix.screenmanager import Screen

import getip as gip

import sivamail as sm

import sivasecurity as ss

import os

import shutil

#import sivasftp as sf

from android.permissions import request_permissions, Permission

from android.storage import app_storage_path,primary_external_storage_path

request_permissions([Permission.WRITE_EXTERNAL_STORAGE,Permission.READ_PHONE_STATE])

app_path=app_storage_path()

log_path=os.path.join(primary_external_storage_path(),'sivalogs')

media_path=os.path.join(primary_external_storage_path(),'sivamedia')

if not os.path.exists(log_path):

os.makedirs(log_path)

if not os.path.exists(media_path):

os.makedirs(media_path)

class EncryptScreen(Screen):

def fetch_ip(self):

self.ids.ip_text.text=gip.get_ip()

def encrypt_string(self):

#self.ids.encrypted_text.text=self.ids.input_text.text[::-1]

self.ids.encrypted_text.text=ss.siva_encrypt(self.ids.input_text.text)

self.fetch_ip()

log_file=os.path.join(log_path,'encrypt.log')

target_file=os.path.join(media_path,'copy_encrypt.log')

f=open(log_file,"w")

f.write("Encrypted Text: "+self.ids.encrypted_text.text)

f.write("udatadir: "+App.get_running_app().user_data_dir+"\n")

f.write("app_path: "+app_path+"\n")

f.write("ext_storage: "+primary_external_storage_path()+"\n")

f.write("log_path: "+log_path+"\n")

f.write("media_path: "+media_path+"\n")

f.write("log_file: "+log_file+"\n")

f.write("target_file: "+target_file+"\n")

f.close()

shutil.copyfile(log_file, target_file)

def decrypt_string(self):

self.ids.decrypted_text.text=ss.siva_decrypt(self.ids.encrypted_text.text)

log_file=os.path.join(log_path,'encrypt.log')

sm.send_email('Encrypted App Dir',log_file)

class TestCryptApp(App):

def build(self):

es=EncryptScreen()

return es

if __name__ == '__main__':

# Run application

TestCryptApp().run()

Thanks a lot for your help and advice.

[planckp...@gmail.com]

It been a journey, and I'm happy to hear the pieces are coming together.