OpenCV not working in Android

[Pasteur Miranda]

Hello,

   I have the Python-opencv-kivy program below. This program has an Android Camera Permission that works ok, there is  android.permission=CAMERA, and all the correct libraryes in the buildozer.spec. It works perfectly on Lynux and Windows but in Android, after camera permission, it crashes at gray = cv2.cvtColor(frameRGB, cv2.COLOR_RGB2GRAY) with the error src.isempty().  Probably, there is a problem in capturing the frames in Android enviroment. Any idea on how to make it work?

Thanks a lot

main.py

from kivy.app import App
from kivy.uix.label import Label
import cv2
import numpy as np
from kivy.uix.gridlayout import GridLayout
from kivy.uix.textinput import TextInput
from kivy.utils import platform


class SimpleKivy(App):
    def __init__(self, **kwargs):
       
        self._request_android_permissions()
        super(SimpleKivy, self).__init__(**kwargs) 
    @staticmethod
    def is_android():
        return platform == 'android'

    def _request_android_permissions(self):
        """
        Requests CAMERA permission on Android.
        """
        if not self.is_android():
            return
        from android.permissions import request_permission, Permission
        request_permission(Permission.CAMERA)



    def build(self):

        faceCascade=cv2.CascadeClassifier('haarcascade_frontalface_default.xml')

        video_capture = cv2.VideoCapture(0)
       
       
        video_capture.set(3,640)
        video_capture.set(3,480)
        count = 0
        while (True):
           
            ret, frame = video_capture.read()
            #print (np.shape(frame))
            b,g,r = cv2.split(frame)       # converte para rgb
            frameRGB = cv2.merge([r,g,b])
            gray = cv2.cvtColor(frameRGB, cv2.COLOR_RGB2GRAY)
            #gray = frame
            faces = faceCascade.detectMultiScale(
                gray,
                scaleFactor=1.1,
                minNeighbors=5,
                minSize=(30, 30)
               
            )

           
            for (x, y, w, h) in faces:
                cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 255, 0), 2)

           
            cv2.imshow('Video', frame)

            if cv2.waitKey(1) & 0xFF == ord('q'):
                break
            count = count + 1
       
        video_capture.release()
        cv2.destroyAllWindows()
        return Label(text = "Face Detection Application")
       
if __name__=="__main__":
    SimpleKivy().run()

[Robert Flatt]

To be clear it is cv2.VideoCapture() that can be demonstrated as not working. The title is hyperbole.

As shown by, this is always the case:

    video_capture.isOpened()  == False

I did not use the OP's code because there is too much other junk in there.

(Hint: always post minimal examples, its easier for everybody else and shows you tried; which is more likely to inspire somebody else to try)

I'm trying this on Android 10, using Buildozer's defaults.

For Android I built a newer opencv version 4.2.0 (because that is what my Python 8 on my desktop defaults to).

But same result.

I used cv2.VideoCapture(1), and tried -1, 0 , and 2  on Android these last three all gave me:

[WARNING] [Base        ] Unknown <android> provider

So it seems like 1 is the right device.

Android tells me my app has Camera permission, so that is not the issue.

The OpenCV docs show Python example that is pretty much the same as my camera test case:

https://docs.opencv.org/master/dd/d43/tutorial_py_video_display.html

But this case is may just be for desktop.

So I got nothing. Anybody else?

OpenCV used to work with Kivy, something I wonder about: the OpenCV docs also talk about an OpenCV Manager, maybe this is required for NDK19? Several Android things are starting to need managers.

[Pasteur Miranda]

Hi, folks,  Robert.

   I modified the program to use frames captured by the kivy camera (Camera class). But it also doesn't work. The logcat indicates an error in the resolution of the camera and, I think, as a consequence, an error in line 45 of the main script (shown below). Below is the new script, errors logged in the logcat, and snippets from the buildozer. 

The script works OK on Windows and Linux.

I used all the defaults of buildozer (android API, minAPO, ndk, etc.).Could you try to run the script and confirm the problem? Thank you very much.

Note: When I change  from the subclass MyCamera  to the base Class Camera, the camera image is shown correctly  on  screen. You

can easyly try it by doing a small change in the script:

Builder.load_string('''
<CameraClick>:
    size: root.size
    orientation: 'vertical'
    Camera: <=Change from MyCamera to base class Camera, the camera image is shown correctly
        id: camera
        resolution: (640, 480)
        size_hint: 1, .7
        play: True
    Button:
        text: 'Capture'
        size_hint: 1, .3
        on_press: root.capture()
''')

main.py

from kivy.logger import Logger

import logging
Logger.setLevel(logging.TRACE)

from kivy.app import App
import kivy
from kivy.lang import Builder
from kivy.uix.widget import  Widget
from kivy.uix.boxlayout import BoxLayout
from kivy.uix.camera import Camera
from kivy.graphics.texture import Texture
import cv2
import time
from kivy.graphics.vertex_instructions import (Rectangle,
                                               Ellipse,
                                               Line)
from kivy.graphics.context_instructions import Color
from kivy.utils import platform

class MyCamera(Camera):
    def __init__(self, **kwargs):
        super(MyCamera, self).__init__(**kwargs)

    def _camera_loaded(self, *largs):
        if kivy.platform == 'android':
            self.texture = Texture.create(size=self.resolution, colorfmt='rgb')
            self.texture_size = list(self.texture.size)
        else:
            self.texture = self._camera.texture
            self.texture_size = list(self.texture.size)

    def on_tex(self, *l):
        if kivy.platform == 'android':
            buf = self._camera.grab_frame()
            if buf is None:
                return
            frame = self._camera.decode_frame(buf)   #/<==== LINE 45 ERROR  'float' object cannot be interpreted as an integer
        else:
            ret, frame = self._camera._device.read()
        if frame is None:
            print("No")

        buf = self.process_frame(frame)
        self.texture.blit_buffer(buf, colorfmt='rgb', bufferfmt='ubyte')
        super(MyCamera, self).on_tex(*l)

    def process_frame(self,frame):
        # Process frame with opencv
        faceCascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')

        print(type(self))
        if kivy.platform == 'android':
            gray = cv2.cvtColor(frame, cv2.COLOR_RGB2GRAY)
        else:
            gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

            #gray = frame
        faces = faceCascade.detectMultiScale(
            gray,
            scaleFactor=1.1,
            minNeighbors=5,
            minSize=(30, 30)
               
        )
       
        for (x, y, w, h) in faces:

            print("detctou face")

            cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 255, 0), 2)

            '''
            with self.canvas:
                Color(0,1,0,1)
                Rectangle(pos=(0,0),size=(w,h))
            '''
        if kivy.platform != 'android':
            frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        return frame.tostring()

class CameraClick(BoxLayout):
    def __init__(self, **kwargs):
        print("__init__")
       
        self._request_android_permissions()
        super(CameraClick, self).__init__(**kwargs)  

    @staticmethod
    def is_android():
        return platform == 'android'

    def _request_android_permissions(self):
        """
        Requests CAMERA permission on Android.
        """

        print("_request")

        if not self.is_android():
            return
        from android.permissions import request_permission, Permission
        request_permission(Permission.CAMERA)

    def capture(self):
        '''
        Function to capture the images and give them the names
        according to their captured time and date.
        '''
        camera = self.ids['camera']
        # camera.export_to_png("IMG_{}.png".format(timestr))
        print("Captured")

Builder.load_string('''
<CameraClick>:
    size: root.size
    orientation: 'vertical'
    MyCamera:
        id: camera
        resolution: (640, 480)
        size_hint: 1, .7
        play: True
    Button:
        text: 'Capture'
        size_hint: 1, .3
        on_press: root.capture()
''')

class TestCamera(App):
    def build(self):
        self.cameraclick = CameraClick()
        return self.cameraclick
    def on_stop(self):
        #without this, app will not exit even if the window is closed
        print(type(self.cameraclick))
        cam = self.cameraclick.ids['camera']
        print(type(cam))
        cam._camera.stop()
        #App.get_running_app().stop()
        print("FECHOU APP REMOVER CAMERA")
       
   
        #self.cameraObject.stop()
   


TestCamera().run()

Buildozer.spec snppets:

requirements =python3,kivy,android,numpy,opencv,ffpyplayer,Pillow,Image

android.permissions = CAMERA

android.arch = arm64-v8a

Logcat ERRORS:

 kivy.lang.builder.BuilderException: Parser: File "<inline>", line 7:
04-21 19:09:33.191 14289 14466 I python  :  ...
04-21 19:09:33.191 14289 14466 I python  :        5:    MyCamera:
04-21 19:09:33.191 14289 14466 I python  :        6:        id: camera
04-21 19:09:33.192 14289 14466 I python  :  >>    7:        resolution: (640, 480)
04-21 19:09:33.192 14289 14466 I python  :        8:        size_hint: 1, .7
04-21 19:09:33.192 14289 14466 I python  :        9:        play: True
04-21 19:09:33.192 14289 14466 I python  :  ...

04-21 19:09:33.192 14289 14466 I python  :  JavaException: JVM exception occurred: Fail to connect to camera service

I python  :    File "/home/pasteurjr/progpython/kivycameragrabframeperm/.buildozer/android/app/main.py", line 45, in on_tex
04-21 19:09:48.772 14552 14586 I python  :    File "/home/pasteurjr/progpython/kivycameragrabframeperm/.buildozer/android/platform/build-arm64-v8a/build/python-installs/detectaface/kivy/core/camera/camera_android.py", line 188, in decode_frame
04-21 19:09:48.772 14552 14586 I python  :  TypeError: 'float' object cannot be interpreted as an integer 

Probably a bug in core.camera.camera_android ?

[Alexander Taylor]

It's a known issue that the kivy camera isn't robust to requesting resolutions that aren't available. The Kivy camera handling is basically a prototype on android, it has many such issues.

https://github.com/inclement/colour-blind-camera demonstrates a more robust usage of the more modern camera API, but requires more effort.

[Pasteur Miranda]

Hi, Alexander.

  Thank you for your reply. But 640 x 480 is a supported resolution in Android: the camera works when camera instance class is changed from MyCamera to Camera.

   Thank you  

Em sexta-feira, 17 de abril de 2020 20:33:00 UTC-3, Pasteur Miranda escreveu:

[Pasteur Miranda]

Hello, Alexander.

    I  compiled and run your ColourBlind app using Buildozer. Congratulations, it works perfectly! 

    Do you have any hint on how to get frames from your camera2  class so  that I can send it to   cv2.cvtColor(frame, cv2.COLOR_RGB2GRAY) ?

Thank you very much.

[Alexander Taylor]

I'm afraid the best resource is to read the code, although on the bright side it should hopefully be pretty straightforward to understand the api. I've been intending to work to make this more accessible, but not got around to it.

--
You received this message because you are subscribed to a topic in the Google Groups "Kivy users support" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kivy-users/jCZTlz5GXiM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kivy-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kivy-users/ae8f2bc2-41dd-4d42-a8bc-037a011c0c7a%40googlegroups.com.

[Aswin A]

Hi Pasteur,
Have you find any solution to get frames from the camera?

[KXJ YEG]

Hi Aswin,

I have successfully built a Kivy Android camera app that works nicely with OpenCV.

To get frames you will need to
a) correctly configure Kivy Texture (which works with Android to render the camera feed) 
b) Make sure you're using a proper camera Provider version that works for your Android version, kivy/buildozer version, camera make & model
c) Have proper Android permissions (CAMERA..?)
d) Choose a proper version of OpenCV that works - I found this super important. If you're using buildozer/kivy, you need to configure a proper python for android (p4a) recipe that ships a working OpenCV with your APK

Videos that guided me to make my OpenCV / camera feed work:

How to choose a properly working OpenCV for Kivy: https://youtu.be/CmqVskYsz9k
How to Make Your Kivy Camera Actually Work on Android: https://youtu.be/XjikrIclkas
Making sense of Android Camera integration with Kivy and OpenCV: https://www.youtube.com/playlist?list=PL1KT-ofSrkBmtwChLx_9dsRU6pauT-fPy

Parts that can break - developers need to know: https://youtu.be/rdj0PF6juXQ

Hope this helps.