Catching errors

[Aaron Hawryluk]

I'm playing with a modified Carousel class from the demos to create a slideshow player that does static slides, a GL overlay HUD, and movies in the framebuffer. This is for a kiosk-type display I'm working on running under Python 3.5. I'd eventually like to get the whole thing running as a subprocess in node.js, and have a basic REST control wrapper for it, but I have so many problems with the Python program that I figure I'd better solve that before getting any more complex.

Sometimes pi3d's Display class either fails to initialize or freezes, and I don't get any error output, the whole thing just hangs on a black screen. If I kill the running python3 process and try to re-run it, it can't grab the display and fails to initialize. I end up having to reboot.

I've tried setting the Logger's loglevel to DEBUG, but that a) stops all program framebuffer functionality due to commandeering STDOUT/STDERR and b) hangs the entire system because it dumps ALL running programs' debug output to the tty.

Is there any way to tell Python to log errors to file or something so that I can debug my issue?

On a related note, is there any way to run pi3d detached? If I try to run it detached using the demo files, I lose all STDOUT and STDERR, and the framebuffer never kicks in (is the framebuffer redirecting STDOUT or something? Any way around that?) If I try to run the python3 program under node,js, I have to run the thing with stdio set to "inherit," otherwise I get no screen output. This is annoying on many levels since it means I have to use an ssh session to run the node program and leave it open. Would much rather have it running on its own and not requiring a shell.

TIA for any help anyone can provide.

--Aaron

[Paddy]

@Aaron, If there's an error caused by pi3d I'd like to fix it. Are you running on the Raspberry Pi? I've struggled with error and debug info in the past when I've used the Keyboard class as it depends on curses which does seem to intercept output to stdout/stderr. The two ways I got round it are a) comment out the keyboard bits b) if that's not feasible, write output to file.

with open('temp.txt','a') as f:

  f.write('got to here with x={}\n'.format(x))

I'm not sure about running detached. The opengl function calls have to be made from the main thread so maybe that's why the graphics don't start. I had a similar issue outputting to the GPU while communicating using Flask and I think I got it working in the end by running the web server as a subprocess. Not sure if this is any help but you might be able to glean something https://github.com/davidedc/devart-template/blob/master/project_code/AutoVJ.py

The most serious thing sound to be the Display not initializing or freezing. In the past this has been due to gpu memory, shader bugs or trying to start from a non-main thread. Let me know as you get more info and I will try and help.

Paddy

[Paddy]

Now I look at it. It was multiprocessing.Process I think threading and subprocess didn't work

[Aaron Hawryluk]

Thanks Paddy, I'm on a Pi2 running jessie. I'll see if I can get errors logged to a file and get an idea of what's going on. Right now I'm "logging" to stdout with Logger.info and the process seems to die somewhere in the middle of the alpha increments for the slides (randomly, of course, never predictably, that would make things too easy). I'm relatively new to both Python and OpenGL so I'm sure a great deal of reading is in order.

[Paddy]

Thanks Paddy, I'm on a Pi2 running jessie...

 Which reminds me that the most common way I debug things is actually by running the whole thing on the laptop first! When the keyboard stuff is done by X11 or pygame the stdout is helpfully displayed in the terminal

[Aaron Hawryluk]

Thanks, I'll give that a go. I have yet to get pi3d running on Ubuntu, Windows, or anything besides the Pi2, really, due to lack of effort. ;-)

[Aaron Hawryluk]

On a side note, what Python versions do you have the best results with on "big" machines? I've tried 64-bit 3.5.1 (Anaconda, since I couldn't get numpy binaries for vanilla Python, they didn't release them for the latest version due to... natch... compile-time errors) on Windows, but I get access violations when pygame tries to initialize the display. Am I better off with 32-bit? 2.7? Maybe using Ubuntu instead of Windows, since the environment is closer? 

[Paddy]

On this laptop's windows I have 32 2.7 because that looked to be the simplest wrt pygame. But when my son was home during university vacation I persuaded him to do a test install and he had 64 bit 2.7 Anaconda. pygame isn't one of the standard Anaconda packages and he had found a way to install it by doing the 'normal' install then copying everything into the location Anaconda expected, when he did that and put the 64 bit windows binaries I compiled for Angle into the same folder as the demos it seemed to work. I can't remember if we were getting access violation errors before any of those steps.

With Ubuntu (which is what I generally have running on this machine) or lubuntu in a vmware machine on windows I think it was pretty easy to install pi3d following the instructions on pi3d.github.com The main difference from the RPi is that the drawing surface is provided by x11 or pygame but I've tried to keep behaviour as close as possible. On the RPi jessie numpy is 1.8 (I think) and has blas and lapack available which you don't get with vanilla apt-get install python3-numpy (you can apt-get install libopenblas-dev liblapack-dev then pip3 install numpy but the speed increase won't be discernible with pi3d)

So, long answer summarised - yes, use ubuntu instead of windows unless you have any objections to doing that!

[Aaron Hawryluk]

Great, I did manage to get it running on Windows 10 using Python.org 3.5.1. I ended up using a number of pre-compiled pip wheels from the "unofficial" packages at http://www.lfd.uci.edu/~gohlke/pythonlibs/ for any binaries I couldn't get pip support for (with 3.5.1 on Windows, that is apparently almost everything, since there doesn't seem to be support for much). I'm going to re-try using the same methods for 64-bit and see how that goes.

[Aaron Hawryluk]

Ok, all seems to be well in both 32- and 64-bit. I now need to make a few adjustments to get the program running properly and will try testing in various environments to see if I can replicate the hang (and possibly get some kind of error out of it).

Thanks for your help, Paddy! Will keep you posted on what I find.

[Aaron Hawryluk]

Hi again, Paddy!

After a while off the problem and a while on, I've come to the conclusion that the black screen issue seems to be caused by.... running out of GPU memory.
It works fine on Windows with nearly-unlimited GPU memory (by comparison). But if I reduce the amount of memory available to the GPU on the Pi, I also reduce the number of images slides I can display before it locks up. So now I'll need figure a way to (free up texture memory | do some garbage collection) when things aren't actually displaying... if that actually is my problem. If I limit the number of slides I use, everything works fine and I can even run it in a node.js supervisor without issue.

On a side note, I'm experimenting with displaying video on Sprites and adding that to the slideshow as well. I lifted some code from VideoWalk.py (the thread handler/ffmpeg bits), but naturally the Pi 2 I'm using chokes on 1080p video frames being mapped onto full-screen OpenGL surfaces. I'm wondering if implementing OpenMax (maybe borrowing from pyOpenMax which I saw on github) to read the frames and map them to the OGL texture instead of using ffmpeg/NumPy might improve performance by using the GPU instead of the CPU, any thoughts?

Right now it takes a good 1/10th of a second to map a frame, and that framerate looks pretty slow. I'm going to try it on a Pi3 and see if that improves things, if it does I'm thinking it's definitely a CPU bottleneck and I need to find a way to push that work onto the GPU.

On Friday, March 4, 2016 at 10:02:23 AM UTC-7, Aaron Hawryluk wrote:

[Paddy]

Hi, there is a free_after_load argument to Texture.__init__() does that help? There should be a system of releasing GPU memory used for Texture instances once they are deferenced (there is a Texture.__del__() method and a list is kept in Display for clearing out each frame), are you sure there isn't some lingering reference somewhere? There's a little app in the pi3d/experiments directory that has a skeleton leak test structure that I have used in the past to see where memory was going.

The ffmpeg streaming of video to numpy arrays is only really viable for small surfaces. If you are rendering full screen I think you would be better to use a video player and put the pi3d shapes on top of that.

[Aaron Hawryluk]

I'll try fiddling with the leak test and see where I get. I could be completely wrong about the GPU memory, but it seems a logical conclusion at this stage.

I'll also look into rendering on top of a video player, not quite sure how I'd approach that at this stage but it seems more efficient than mapping videos as textures, especially when the videos don't need to be 3d.

[Paddy]

I'm not sure exactly what you're wanting to do but @swehner contributed some code to pi3d as he was sorting out his foos video replay system. https://github.com/swehner/foos This is based on overlaying pi3d gpu sprites on a background video.

[Aaron Hawryluk]

Thanks, that looks very similar to what I'd like to do, I'll have a play and see what I can learn.

[Aaron Hawryluk]

Update:
Thanks for the layering concept! It led me to the dispmanx documentation. I found can actually overlay pi3d over something like omxplayer just by setting the dispmanx layers (set pi3d Display to layer 1, set omxplayer to layer 0) and setting the alpha of the background in pi3d's Display to 0.0. A preliminary test suggests this works fairly well with only a minimal increase in CPU usage, so I'll pursue it a little further.