Debugging gstreamer issues with stb-tester

[ekelly]

I am running multiple scripts with differing noise parameters in order to discover the most appropriate noise thresholds for our set up. 

The scripts run successfully most of the time, but after running them continually in a loop I eventually get the following error from gstreamer:

Error: Could not read from resource.: Could not enqueue buffers in device '/dev/video0'.

v4l2src_calls.c(84): gst_v4l2src_buffer_pool_activate (): /GstPipeline:stb-tester/GstBin:bin234/GstV4l2Src:v4l2src117:

enqueing buffer 0/2 failed: Device or resource busy

I assume that this is due to something not freeing up the device when finished. I am looking for advice in running gstreamer in debug mode with stb-tester to try and characterise the issue. Are you guys able to help or would I be better off on the gstreamer mailing lists?

I ran the scripts with the following command (which I pilfered from a blog somewhere):

GST_DEBUG=3,python:5,gnl*:5 ./test_enc.py

 

But it gave me a ton of debug which I'm currently trying to make sense of. Any help or advice about getting to the bottom of this would be great.

[David Röthlisberger]

Before you can ask on the GStreamer mailing list you'll have to get
stb-tester out of the equation. Start by running your stbt
source-pipeline and sink-pipeline with `gst-launch-0.10`. For example if
your stbt source-pipeline is "v4l2src device=/dev/video0 ! mpegtsdemux !
video/x-h264 ! decodebin2" and your sink-pipeline is "ximagesink
sync=false" then run this:

gst-launch-0.10 \
v4l2src device=/dev/video0 ! mpegtsdemux ! video/x-h264 ! decodebin2 \
! ffmpegcolorspace \
! ximagesink sync=false

Note that I inserted an "ffmpegcolorspace" element between the source
and sink pipelines -- you may need to add this depending on the
colorspace output by and expected by your source and sink elements,
respectively. When you use `stbt run`, stbt inserts this for you
automatically.

To test this gst-launch command line in a loop, you could specify
v4l2src's "num-buffers" parameter so that the gst-launch command exits
after a certain amount of frames, to better simulate the behaviour you
are observing under stbt.

Regarding debug output: setting the GST_DEBUG environment variable
affects the debug output of gst-launch or any other GStreamer-based
program (including stbt). GST_DEBUG=3 enabled debug level 3 (the levels
are 0 to 5; 4 is ridiculously verbose.) GST_DEBUG=v4l2src:3 enables
debug level 3 just for that element. As well as element names, you can
specify some broader "debug categories": Use
`gst-launch-0.10 --gst-debug-help` to see the available categories.

GST_DEBUG (and other environment variables that affect GStreamer) are
documented in the man page for gst-launch-0.10.

We're always happy to help, especially when the request is clear,
includes relevant and thoughtfully-cropped logs, and it's obvious that
you've done some homework (all of which are the case here).
Though of course the people on the GStreamer mailing list will have more
expertise on GStreamer, V4L, etc., than most of us. If you do ever post
there, be sure to include your full gst-launch command line.

A final hint regarding command-line usage of gst-launch: Consider
installing the tab-completion script here:
https://raw.github.com/drothlis/gstreamer/bash-completion-0.10/tools/gstreamer-completion-0.10
to /etc/bash_completion.d/ (assuming you have the bash-completion
package installed).

Cheers,
Dave.

[ekelly]

Hi Dave,
Thanks for your response.

I ran a test overnight where I called gst-launch with my pipeline a few thousand times and seen no problems. Previously with my other method I was seeing issues before 200 iterations or so.
The main difference I can spot:
    - With gst-launch the debug refers to a new v4l2src0 for each iteration.
    - With my python script, the debug refers to v4l2src0, v4l2src1, v4l2src2...

My python script was simple enough, it contained an alteration of the stbt-run script as a function, and it called this function in a loop numerous times. From the debug it looked as if there may have been an issue when gstreamer tried to access the incorrect v4l2src object.
i.e. Trying to set v4l2src(X-1) to NULL after v4l2srcX was already in use.

I think the error is due to me using stbt in a manner it wasn't designed to. I'm happy enough that I can carry out my investigations into the noise threshold using gst-launch. Or by using the command line stbt-run. But if you're interested in this error then I can give you more details/debug some more for you.

Thanks,
Emmett

[David Röthlisberger]

On 9 Jan 2013, at 10:23, ekelly wrote:
>
> Hi Dave,
> Thanks for your response.
>
> I ran a test overnight where I called gst-launch with my pipeline a few thousand times and seen no problems. Previously with my other method I was seeing issues before 200 iterations or so.
> The main difference I can spot:
> - With gst-launch the debug refers to a new v4l2src0 for each iteration.
> - With my python script, the debug refers to v4l2src0, v4l2src1, v4l2src2...
>
> My python script was simple enough, it contained an alteration of the stbt-run script as a function, and it called this function in a loop numerous times. From the debug it looked as if there may have been an issue when gstreamer tried to access the incorrect v4l2src object.
> i.e. Trying to set v4l2src(X-1) to NULL after v4l2srcX was already in use.

Perhaps your own implementation of stbt-run is failing to clean up
resources somehow. If you can post the source code I'll have a quick
look, though I can't promise I'll spot anything. :-)

> I'm happy enough that I can carry out my investigations into the noise threshold using gst-launch.

That's great. Presumably you're inserting stbt-motiondetect into the
pipeline and checking its output (you can use gst-launch's "--messages"
flag to print the messages each element posts to the GStreamer bus).
Just remember that stbt's wait_for_motion performs additional filtering
that stbt-motiondetect doesn't (in particular, wait_for_motion waits for
x _consecutive_ frames with "has_motion" reported from
stbt-motiondetect).

> Or by using the command line stbt-run.

That's an even better idea. From the bash shell command line:

for i in {70..99}; do
stbt run <(printf 'wait_for_motion(noise_threshold=0.%02d)\n' $i);
done

(The "<(...)" is a bash feature called process substitution: It runs the
commands inside the parentheses and then passes "stbt run" a filename;
that file's content is the output of the printf command. See
http://www.gnu.org/software/bash/manual/html_node/Process-Substitution.html )

Of course you can use python instead of shell, and use subprocess.call
to run "stbt run".

Or even simpler, run a single stbt script that contains:

wait_for_motion(noise_threshold=0.70)
wait_for_motion(noise_threshold=0.71)
wait_for_motion(noise_threshold=0.72)
...

Of course the appropriate solution depends on the particular details of
what you need to do, which I don't know. These are just suggestions. :-)

Dave.

[ekelly]

Thanks again for the response. 

Perhaps your own implementation of stbt-run is failing to clean up 

resources somehow. If you can post the source code I'll have a quick
look, though I can't promise I'll spot anything. :-)

Not much has changed other than the dirty hack of passing in all the local variables as the globals to the script. (I had issues where the imports (MotionTimeout etc) were not available within functions within the scripts)

    def run_script(self, script):

        """

        A function to run stb-tester scripts.

        """

        stbt.init_run("v4l2src", "fakesink", "lirc:/var/run/lirc/lircd:" + self.model)

        from stbt import press, press_until_match, wait_for_match, wait_for_motion, \

        detect_match, MatchResult, Position, detect_motion, MotionResult, \

        UITestError, UITestFailure, MatchTimeout, MotionTimeout, ConfigurationError

        try:

            execfile(script, locals())

        except stbt.MatchTimeout as e:

            traceback.print_tb(sys.exc_traceback)

            print ("FAIL: %s: Didn't find match for '%s' after %d seconds." % (

                script, e.expected, e.timeout_secs))

            if e.screenshot:

                stbt.save_frame(e.screenshot, "screenshot.png")

                print ("Saved screenshot to '%s'." % ("screenshot.png"))

            sys.exit(1)

        finally:

            stbt.teardown_run()

> Or by using the command line stbt-run.

That's an even better idea. From the bash shell command line:

    for i in {70..99}; do
        stbt run <(printf 'wait_for_motion(noise_threshold=0.%02d)\n' $i);
    done

(The "<(...)" is a bash feature called process substitution: It runs the
commands inside the parentheses and then passes "stbt run" a filename;
that file's content is the output of the printf command. See
http://www.gnu.org/software/bash/manual/html_node/Process-Substitution.html )

 Thanks for the advice, I think I'll go with something like this. I forgot about the consecutive frames etc making it more awkward to do with gst-launch. 

Emmett