Migrating to Gstreamer 1.x

[Sergio Matone]

Let's start to collect info about architecture, roadmap and related topics about migration to Gst 1.x.
This is related to the discussion issue (#136) and to the stub of new version hosted @Github.
The idea is to move the discussion away form issue tracking.

Main topics that come to my mind are:
- project name (e.g. gstreamer-1.x-java)
- project location (I'll vote definitely for Github)
- backward compatibility with Gst 0.1x
- reference platform (whether cross compatibility cannot be achieved)
- milestones to achieve

For the latter I think it will beuseful to discuss
- API java generation
- Maven project definition
- setup of codebase
- java class generation from gobject-introspect data

Comments are welcome.
Sergio

[Neil C Smith]

Hi,

On 22 January 2014 13:04, Sergio Matone <sw36...@gmail.com> wrote:
> Let's start to collect info about architecture, roadmap and related topics

> about migration to Gst 1.x. ... The idea is to move the discussion away form issue tracking.
>

IMO, good move! :-)

> Main topics that come to my mind are:
> - project name (e.g. gstreamer-1.x-java)

I'd agree with that.

> - project location (I'll vote definitely for Github)

Yes, though I suggest we set up an organisation for this, rather than
hosting through any individual GitHub account.

I'm happy to set up an organisation called GStreamer-Java, or someone
else can? Thoughts? That naming is deliberate btw, so that we can
maintain bug fixes for the 0.10 branch there too if necessary.

> - backward compatibility with Gst 0.1x

Personally, I don't think this is a high-priority, or necessarily
achievable. It could be that once we have the bare bones of a native
wrapping it is possible to create a somewhat API compatible library
that delegates to the new bindings.

As GStreamer hasn't retained API compatibility in order to facilitate
onward progression and improvement, I'd suggest that neither do we.

> - reference platform (whether cross compatibility cannot be achieved)

OTOH, cross-platform compatibility is a definite requirement if you
ask me, and I see no reason it cannot be achieved. That's not to say
that the JNAerator configuration has to be cross-platform capable - as
stated elsewhere, I'd be in favour of an initial automated binding
created against the GStreamer headers, and manually massaging it from
there.

> - API java generation

Just taking this for now, the reason I'm suggesting an initial
automated binding with manual addition is that we need to be careful
how we treat any additions to the GStreamer API if we want the Java
bindings to remain backwards compatible. For this reason,
incidentally, I think we should probably be building against 1.0
headers. IIRC the direct bindings in JNA will fail at library load if
a native function cannot be found (unlike interface mapping which
happens at runtime). I emphasise the IIRC there though - need to
relook at this to see what the best way of approaching possibly
missing functions is.

Best wishes,

Neil

--
Neil C Smith
Artist : Technologist : Adviser
http://neilcsmith.net

Praxis LIVE - open-source intermedia development - www.praxislive.org
Digital Prisoners - interactive spaces and projections -
www.digitalprisoners.co.uk
OpenEye - the web, managed - www.openeye.info

[Thomas Greenwood]

Hi

I was pointed at this thread by some folks on github as I have a view on generation of Java APIs.  I have a version if gstreamer-java working on gstreamer1.0/1.2, though it's far from complete as it's been done manually (if you try to use things that haven't been before it often throws exceptions or even SEGFAULTs).  I am trying to get it pushed onto github - which might happen in the next week or so.  I think that doing it manually is too error prone for a high quality implementation.

When I was originally looking at this I did try other approaches that involved generation but had issues that I couldn't fix in the timescales.  I did try JNAerator which is great but doesn't understand gobject and so cannot do as good a job as something based on gobject-introspection.  I then found JGIR which is pretty good, but had some issues with GStreamer - I spent quite some time debugging, but found the approach of generating Java byte code isn't easily customisable or easy to debug.  I found that most of the issues were things that wouldn't really be issues with generating source code - for example managing the stack.

I think that the best way forward is to use some of the JGIR code for gobject-introspection and then generate Java source code.  That will make it easier to customise and also mean that it's easier to see what is going on when there are issues.  In the worst case if there are little parts that don't work it's easy to patch generated code until bugs in the generator can be fixed.

I've started looking into what could work here - https://github.com/armouroflight/java-gobject. It doesn't work at all yet, but I hope it will in the end.

[Neil C Smith]

Hi Thomas,

Looks interesting!

On 26 January 2014 15:36, Thomas Greenwood <tcdgre...@gmail.com> wrote:
> I think that the best way forward is to use some of the JGIR code for
> gobject-introspection and then generate Java source code.
>

> I've started looking into what could work here -
> https://github.com/armouroflight/java-gobject. It doesn't work at all yet,
> but I hope it will in the end.

As mentioned somewhere else (too many conversations going on recently!
:-)) when JGIR was first mentioned, I noticed that JGIR is based on a
fork of the GLib bindings that were originally in GStreamer-Java
There have been numerous changes since (I noticed you adding in the
field orders to structs for one), and probably lots of other bug
fixes. OTOH, they've probably got a few additional fixes / features
too.

I'd suggest a careful diff of the sources from JGIR and
http://code.google.com/p/gstreamer-java/source/browse/ , particularly
the org.gstreamer.lowlevel package.

[Thomas Greenwood]

I noticed pretty much the same thing - though I think the JGIR code has been changed a little.  At the moment I'm just looking at getting a prototype of producing the Java code and compiling it.  After I've got to that point I was going to review the reference handling code in the main.  I suspect I'll be able to get a simple pipeline working (perhaps videotestsrc ! fakesink or something) before I'd need to change it.

I'm still hoping that I can work out a way of sharing some of the gstreamer-java 1.2 port that I have done which includes changes to the low level bindings.  The issue is that some of the work was done for my day job and they want to check that it's something they are happy with.  I have managed to share much of the other gstreamer work I've done - so I don't think it will be an issue but it's just about how quickly the cogs turn!

What do people think about using velocity templates for the Java classes?  It potentially allows for easier customisation and I think it's easier than directly generating Java code from Java!

I think if I get to the point of a simple pipeline working others may be help if they consider it the right way to go.  If folks have concerns I'd be happy enough to help look at something else or we could try a number of approaches and then decide on the best.

HTH

[Sergio Matone]

Hi Neil,
sorry for disappearing for few days...
I can see that a lot of people complaining and asking for Gst-java 1.x has neither commented or
gave an opinion about what it's going on. I can imagine what we can excpect on the coding side...

Anyway, what about creating an organization on Github and go for a name that sounds good to you [gstreamer-1.x-java]
(I think we can just go on, since we received only silence as answer to our questions/proposals).
I'm still trying to understand if actually it will be better to work on JGIR or keep up with JNAerator.

Let me know and thanks for being collaborative ;)

[Neil C Smith]

Hi Sergio,

On 27 January 2014 14:19, Sergio Matone <sw36...@gmail.com> wrote:
> sorry for disappearing for few days...

Not a problem! I'm aware that I might be one of the more vocal here
at the moment, but I'm also more talk than action right now! :-) As I
mentioned in an email off-list to Andres, I'm fairly tied up with work
until early March, so unlikely to be able to spend much time on this
until then. I'm also wary of talking more than longer-standing
project members ...

> I can see that a lot of people complaining and asking for Gst-java 1.x has
> neither commented or
> gave an opinion about what it's going on. I can imagine what we can excpect
> on the coding side...

There's already more action than when this last came up a year ago -
I'm going to be quietly optimistic!

> Anyway, what about creating an organization on Github and go for a name that
> sounds good to you [gstreamer-1.x-java]

I've done that, but need to set up a few more things. Actually chose
gstreamer-java - as stated in my earlier email, I'm in favour of
maintaining (or at least cloning) the existing 0.10 project there as
well. GStreamer 0.10 ain't dead yet.

If anyone on list has actually used svn2git and is happy to push the
existing code, let me know, or I'll try and get my head around to it

in the next week or so.

> (I think we can just go on, since we received only silence as answer to our
> questions/proposals).
> I'm still trying to understand if actually it will be better to work on JGIR
> or keep up with JNAerator.

JGIR wouldn't preclude JNAerator - they have different purposes. JGIR
could be the answer to building bindings automatically based on
GObject introspection. That still requires a JNA binding to the
actual C library. Currently JGIR seems to use an old fork taken from
the GStreamer-Java project. It might be possible to update the GLib
parts manually. OTOH it might make sense to fork the JGIR code to
work with a new JNAerator binding to GLib and GStreamer 1.0.

I didn't know JGIR until someone mentioned it recently, and haven't
had a chance to look at it much. This needs some input from someone
who's spent more time working with it.

[ac]

Ok, I did some reading, after which it strikes me that, given one of the main goals of GObject being "automatic transparent API bindings to other compiled or interpreted languages" (https://developer.gnome.org/gobject/stable/chapter-intro.html) and widespread of use of Java, there after all no robust GObject binding mechanism for Java...  At least, the official java-object-introspection project (https://wiki.gnome.org/Projects/JGIR) seems abandoned at least since 2010. Python on the other hand, does have a very stable and up-to-date support for GObject and GStreamer (https://wiki.ubuntu.com/Novacut/GStreamer1.0)... I guess that not that many people are at the intersection of Java and GObject!

Leaving those considerations aside, it looks to me that JNAerator won't be useful at all since autogeneration of the GStreamer bindings would require gobject introspection, which JNAerator does not support. Am I wrong here? 

Now, specifically about gobject-introspection, I found this tutorial https://wiki.gnome.org/Projects/GObjectIntrospection/FromHereToThere and diagram https://wiki.gnome.org/Projects/GObjectIntrospection/Architecture from the gnome wiki that helped me a bit understanding how it works. So, a couple of questions for Thomas: the code you uploaded to the java-gobject repo is basically an adapted/trimmed down version of the original JGIR project, right? And the .typelib files should still be generated with g-ir-compiler? Even more basic: how do you actually run java-gobject?

I also found a project which offers (JNI-based) bindings for various gnome components, including GObject: http://java-gnome.sourceforge.net/. It seems to be somewhat active https://github.com/afcowie/java-gnome, and a couple years ago someone tried to add gstreamer bindings to it, although didn't go too far: https://launchpad.net/java-gnome-gstreamer

[Neil C Smith]

Hi Andres,

Some useful links in that email.

On 27 January 2014 23:14, ac <andres....@gmail.com> wrote:
> Leaving those considerations aside, it looks to me that JNAerator won't be
> useful at all since autogeneration of the GStreamer bindings would require
> gobject introspection, which JNAerator does not support. Am I wrong here?

Yes, no, maybe! :-)

JGIR uses an old version of the GLib / GObject bindings from
GStreamer-Java. I'm assuming we still need to map GLib, etc. for the
GObject introspection stuff to work against, and possibly a few bits
of GStreamer itself. From what I can tell the PyGI code still seems
to have a lot of C interfacing outside of auto-generated introspection
code. We could use / update the existing bindings we have for this
incrementally and use the JGIR code against those, or use JNAerator to
map the C API anew and in an updated JNA fashion, and then adapt JGIR
code to work with that, or ... ?

> I also found a project which offers (JNI-based) bindings for various gnome
> components, including GObject: http://java-gnome.sourceforge.net/. It seems
> to be somewhat active https://github.com/afcowie/java-gnome,

I had a look at that earlier this week. The generator code could also
be interesting, though it works against JNI bindings rather than JNA
ones, and seems to work with .defs files. What I'm completely unsure
about is what relation the .defs files have to GObject
introspection???

[ac]

> We could use / update the existing bindings we have for this
> incrementally and use the JGIR code against those, or use JNAerator to
> map the C API anew and in an updated JNA fashion, and then adapt JGIR
> code to work with that, or ... ?

That's seems very tricky, what do you exactly mean by "use JNAerator to map the C API anew"... The main problem here is that, at least from what I understand, JNAerator is not able to parse any code based on GObject in order to understand the class structure, because it simply doesn't understand how the structs, the functions operating on those structs, and the boilerplate preprocessor code defines classes in GLib. Perhaps we can contact JNAerator's author directly (Olivier Chafik), and ask him?

Maybe once Thomas finishes uploading his code, we can pass gstreamer through the g-ir-compiler and then feed java-gobject with the resulting .typelib files?

[Neil C Smith]

Hi,

On 29 January 2014 16:55, ac <andres....@gmail.com> wrote:
>
>> We could use / update the existing bindings we have for this
>> incrementally and use the JGIR code against those, or use JNAerator to
>> map the C API anew and in an updated JNA fashion, and then adapt JGIR
>> code to work with that, or ... ?
>
> That's seems very tricky, what do you exactly mean by "use JNAerator to map
> the C API anew"...

Maybe not explaining this very well. My understanding is that JGIR,
and presumably the result of Thomas' effort, will rely on this runtime
code -

https://github.com/armouroflight/java-gobject/tree/master/src/gobject/internals
https://github.com/armouroflight/java-gobject/tree/master/src/gobject/runtime

That code was forked from GStreamer-Java. It's old, so if we keep it
we'd want to make sure changes on either side are reflected. At the
same time, it's interface and not direct mapped, hence it might be
good to try to use JNAerator to recreate this runtime. Or we just
patch what's there in a more modern JNA style.

> The main problem here is that, at least from what I
> understand, JNAerator is not able to parse any code based on GObject in
> order to understand the class structure,

Exactly, but that's what I mean by there being two jobs here as far as
I understand - wrapping the basics of GLib, then running the GObject
introspection stuff over GStreamer.

[ac]

Ok, got it now.

In fact, my main basis to state that JNAerator is not able to parse any code using GLib is simply what Thomas wrote earlier ("I did try JNAerator which is great but doesn't understand gobject and so cannot do as good a job as something based on gobject-introspection"), and this long-standing issue in nativelibs4java: https://github.com/ochafik/nativelibs4java/issues/132

But going through the comments in that issue again, the core problem seems to be the "Failed to convert define 'G_CSET_LATINS" error, which perhaps is not as serious after all. Someone just did a pull request in the nativelibs4java repo to solve that problem https://github.com/ochafik/nativelibs4java/pull/473, and the change looks fairly trivial. I will contact Olivier Chafik directly to get his input, and maybe we can do some more concrete tests soon.

[Thomas Greenwood]

Sorry I missed alot of this - too busy.  I have made some progress on the JNA generation.

I think folks have misunderstood what I was saying.  I think that JNAerator could well parse all the headers correctly and generate valid JNA interfaces - barring bugs!  What it wouldn't do is be able to understand about GObject - which is extra information that GObject introspection provides and used by the GStreamer python bindings.  This doesn't mean JNAerator couldn't be used for various things:

• Generating the low level JNA code with GObject introspection layer on top (though I think GObject introspection gives you the information required here)
• Generate code for accessing the introspection API itself
  
• Generate code for some of the low level stuff required for runtime e.g. https://github.com/armouroflight/java-gobject/tree/master/src/gobject/internals (though I think the generation for this part is relatively small in comparison to the logic required for binding into Java correctly)

When it comes down to it gstreamer is C-code and there is no fundamental reason why JNAerator shouldn't understand that, but I understand it doesn't know GObject so some extra information is lost.

[ac]

Hi Thomas, thanks a lot for the clarifications.

Just want to briefly mention that Olivier Chafik (JNAerator's author) is looking into some errors people came across when running JNAerator on GLib, see: https://github.com/ochafik/nativelibs4java/issues/132#issuecomment-34541908

So we should probably build JNAerator directly from the nativelibs4java repo to make sure we are using Olivier's latest fixes.

I also like to bring up the news that we are looking for projects proposals to submit for Google Summer of Code 2014, under the Processing organization (http://processing.org/). One of the project ideas for GSOC is to update the video library in Processing to use gstreamer 1.x, which would require the new gstreamer-java bindings. If there is any student around here who is interested on working on this, please let me know! The video library in Processing is in fact a very thin layer on top of gstreamer-java, so most of the work would consist mainly in getting the new bindings ready.

[Farkas Levente]

On 02/08/2014 06:52 PM, ac wrote:
> I also like to bring up the news that we are looking for projects
> proposals to submit for Google Summer of Code 2014, under the Processing
> organization (http://processing.org/). One of the project ideas for GSOC
> is to update the video library in Processing to use gstreamer 1.x, which
> would require the new gstreamer-java bindings. If there is any student
> around here who is interested on working on this, please let me know!
> The video library in Processing is in fact a very thin layer on top of
> gstreamer-java, so most of the work would consist mainly in getting the
> new bindings ready.

we've got a student who can do it but has limited free time if he can't
win the gsoc..

--
Levente "Si vis pacem para bellum!"

[ac]

Hi Levente, that sounds great, let's continue the discussion about GSOC over email to avoid polluting the conversation on this thread too much :-)

About the JNAerator tests, I started using a new config.jnaerator file suggested by Olivier and a version of jnaerator.jar that fixed the GLib parsing issues. With that, I'm able to parse glib.h without errors and generate the corresponding java files, take a look at the updates in the https://github.com/codeanticode/gstreamer-1.x-java repo.

@Thomas, once you update your java-object code, perhaps we can discuss the specific of how to integrate it with the automatic parsing using jnaerator.

[Sergio Matone]

Hi ac,
I've seen your pushed code in repo...it's not very clear to me what we are trying to achieve... Then why don't we start from glib source code compiled instead of using MacPorts stuff,
that would be longer more flexible IMHO...

[ac]

Hi, I first tried to compile glib from the source packages by hand, but seems is not trivial. Macports already includes all the patches and tricks needed to do that, so I prefere not to solve things again by myself. Also, seems to be that Olivier is also using macports, so I can copy his jnaerator files directly into the build system the master branch of gstreamer-1.x... Those should work on Linux only with minor changes to use the corresponding Linux paths.

[ac]

This thread on the native4libs repo might also be useful to keep track of the updates:

https://github.com/ochafik/nativelibs4java/issues/132#issuecomment-35811824

[ac]

Hey all, a quick update on the gstreamer 1.x migration.

Levente Farkas has been mentoring Elek Roland, a CS student from the University of Szeged, during this year's Google Summer of Code.

Roland looked into different alternatives to implement fully automated binding generation for GLib, GObject and GStreamer. He put together two different mechanisms, and although we are still actively working on them, you are encouraged to take a look:

1) JNAerator -based: https://github.com/octachoron/gstreamer-1.x-java

2) GIR-to-Java translation: https://github.com/octachoron/gir2java

We originally planned to generate the low-level bindings (GLib/GObject) using JNAerator, and then the GStreamer bindings by parsing the .gir files that come with the development packages and fully specify the classes siganture, and using the JNAerator-generated bindings to fulfill the low-level dependencies. Now we are leaning towards doing everything with the GIR parser, since even the structure of the GObject classes in specified in .gir files. Anyways, this is work in progress, so any suggestion/help is welcome!

Also, I did some quick and dirty changes to Thomas Greenwood's fork of gstreamer-java, in order to get a version of the video library for Processing that works with the latest official release of the GStreamer libs on OSX (1.4.0):

https://github.com/codeanticode/gstreamer1.x-java

https://github.com/processing/processing-video/tree/gstreamer1.x 

Although the plan is to replace the old gstreamer-java bindings with automatically-generated ones (either with JNAerator or gir-to-java or a combination of both), the ones I hacked to work with 1.4.0 could be useful as a reference as we finish the new bindings.

Again, any help will be enormously appreciated!

[Sergio Matone]

Hi Andreas,
good news, that sounds really good !

Congrat, you are doing a really good job, I am willing to collaborate in the near future,
next month, after I'll be back from my holidays.

One question, are you planning to merge the repositories or they will remain different projects,
such as different points of view?

Best
Sergio

[ac]

Right now we are in the process of determining which mechanism will be used to generate the new bindings: either JNAerator or gir2java, or a combination of both. Depending on the results of the current tests, we will probably put the stable code in a new repo, but not sure at this point.