Re: port gst-dsp into ffmpeg

[Felipe Contreras]

On Thu, Nov 15, 2012 at 5:22 PM, Selso Liberado
<selso.l...@gmail.com> wrote:

> I "need" to implement in the dspbridge acceleration in ffmpeg.
> I've seen that Felipe did make a try in with ti-dsp, but it seems that
> patches have not been accepted until now (no tidsp in the last git).

Yeah, I haven't tried to push them again.

> I would like to re-work on gst-dsp to have ideally things that way :
> a standalone lib ffmpeg/gstreamer independant. In that way we could test
> directly implementation from h264 coded file (not muxed) and avoid source
> duplication for dsp-tools
> a wrapper for gstreamer integration, the code for fmmpeg integration should
> be directly coded in ffmpeg.
>
> In that way ffmpeg guy would not have to look inside dspbridge stuff, but at
> least the interface would be maintained by ffmpeg developpers.
>
> the difficulty is the current gst-dsp using low-level objects from gstreamer
> (gstbuffer, gstmemory).
> I wonder what would be simpler :
>
> modify this for only use of C stuffs. Faisable if there aren't too many...
> modify for use of a "transition layer" that implements equivalent
> gstreamer/ffmpeg low-level object (if there are). But that way we would keep
> a dependancy on gstreamer/ffmpeg API

Yes, we would need to get rid of GStreamer API. I started to do that,
but didn't manage to finish. Another option is to just link libti-dsp
to libgstreamer for internal purposes, while the public API of
libti-dsp is completely independent. This would not look good to
FFmpeg developers, but we would slowly remove the GStreamer stuff.

> I am still unaware to the ffmpeg API (I just know about avcontext, and code
> integration) so I'm not sure on the right thing to do.
> The other difficulty for me is to handle the whole code that have little
> documentation and I sometimes it is hard to see what is written exclusiveley
> for dspbridge or gstreamer. Also it is hard for me to read the data flow
> (asyncquue, dmm_fuffer, td_buffers...)
> About LGPL licence details I'm like newbie.
>
> I've made a first code review :
> dspbridge.h and dmmbuffer.h are low-level code for controlling dsp and pass
> data. they're base only on ANCI code
> sem.h and Asyncqueue a low-level object base on glib. Their names are
> relevant. they are dependant on glib. I should search for any ffmpeg
> equivalent.

These are not needed.

> gstdspbase is containes base gstreamer element class. It implements :
> _ structure "GstDspBase" : it gathers gstreamer elements/pads, dpsbridge
> flow container (td_buffers) and methods, and dspside codec representation
> (td_codec).
> _ td_codec definition, td_buffer definition, ts_item (timestamp). Well some
> little description about theses structs would be welcome.

td_codec is the structure to contain a codec (decoder, encoder)
td_buffer holds a DSP buffer (which contains multiple td_buffers)

These are mainly for the USN type codecs.

ts_item is not needed.

> _ td_codec (td_h264dec,...) struct implementation : define where is located
> dsp code file to load and very close related information (uuid,...) for
> earch supported codec.
> In these structure we're using low-level gstreamer api objects like
> gstbuffers.
>
> I wonder now if the current interface structuration is compatible with
> ffmpeg or if I'll have to make an common sub-layer.

You can take a look at the patches I sent to the FFmpeg mailing list.

Basically, most of the code in gst-dsp is not needed if we use a
technique used in FFmpeg; delayed decoding. Each time we push a buffer
to the DSP, we check if there are any output buffers ready, and push
them to the next GStreamer element. This way we don't need a dedicated
output thread which makes things so much more complicated. And then we
don't need asyncqueue, or semaphores, or a queue of timestamps.

I implemented it this way when I sent the patches to FFmpeg, and I was
testing the code with gst-av, which is like gst-ffmpeg, but simpler,
and probably more efficient. And it was working just fine.

Cheers.

--
Felipe Contreras