gstreamer-java and processing

126 views
Skip to first unread message

acolubri

unread,
Dec 3, 2007, 11:51:56 AM12/3/07
to gstreamer-java
Hi Wayne,

Thanks for updating the trunk, it solves the compilation and execution
problems.

So, I started making some small changes in order to grab the video
data of each frame. I added a public function to GstVideoComponent
called getCurrentImage() that just returns currentImage. Seems to be
almost working, because I can start playing the video from my
application (more on this in a second) and grab the first frame just
by calling currentImage.getRGB(i, j). I know it is very inefficient,
but it is just an initial test.

But the problem is that I can get only the first frame of video, the
content of currentImage remains the same throughout the playback of
the file, although I can hear the sound playing normally.

My suspicion is that I need to create a GMainLoop object and call to
gloop.run(), is that correct?

But now the problem is that gloop.run() interferes with the drawing
loop of my app. The thing is that I'm trying to use gstreamer-java
from within processing (processing.org). I don't know if you are
familiar with processing, it is a language for visual artists, based
on java. It greatly simplifies the handling of the canvas drawing.
Making things short, the processing application is basically an AWT
component (doesn't use swing) with its own internal handling of the
screen refresh. Whenever I add the gloop.run() call, the processing
application stops updating the screen.

Any suggestions on how to update the video frame without using
gloop.run()? In a previous post you mentioned that I should create my
own gstreamer pipeline. Would this solve the issues that I have when
using the image buffer from GstVideoComponent? However, this is
approach is flying over my head right now, I looked at the
sinkRender() in the CustomSink class, but there is no implementation
of sinkRender in GstVideoComponent, so I'm not sure what to do next.

Sorry for the long post, but I guess that I'm trying to do something
fairly complicated, and still don't have a good understanding of how
gstreamer-java works.

Thanks,
Andres

Wayne Meissner

unread,
Dec 4, 2007, 5:35:02 AM12/4/07
to gstream...@googlegroups.com
You can also run the GMainLoop in the background.
e.g.
   GMainLoop loop = new GMainLoop();
   new Thread(loop).start();

or more simply:

   loop.startInBackground();

Yep, its a common enough thing that its already encapsulated in a method.  The lack of javadoc is a bit of a hindrance to discovering some of the easier ways to do things though.

As for the data extraction, I had to roll back to an older version of GstVideoComponent for windows - it didn't like the CustomSink based version.  Have a look at the HandoffListener in GstVideoComponent and how it gets setup.  That was based on some code I found somewhere (might even have been on the gstreamer tutorial) on how to get data out of a pipeline.

I'll see if I have enough time tonight to hack together a RGBDataExtractor example or something, that just puts together a pipeline, and allows you to register a simplified listener that gets called whenever it gets a frame.

Wayne Meissner

unread,
Dec 5, 2007, 7:29:43 PM12/5/07
to gstream...@googlegroups.com
You should now be able to use a RGBDataSink() as the video-sink element for a playbin, and get the data easily from that.  You need to provide a RGBDataSink.Listener which gets given an RGB frame everytime one is decoded.

Something like:

    GMainLoop loop = new GMainLoop();
    PlayBin player = new PlayBin("Video Data Extractor");
    RGBDataSink videoSink = new RGBDataSink("rgb",
        new RGBDataSink.Listener () {
            public void rgbFrame(int w, int h, IntBuffer rgbPixels) {
                // do whatever you want with the data here
            }
        }
     );
    player.setVideoSink(videoSink);
    player.setInputFile(new File(args[0]));
    loop.startInBackground();
    // go and do whatever else is needed/run some other loop



On 04/12/2007, acolubri <andres....@gmail.com> wrote:

acolubri

unread,
Dec 6, 2007, 4:25:49 AM12/6/07
to gstreamer-java
It works!

Thanks a lot, now I can play videos in Processing with gstreamer!

Just one minor issue:

When using the new version of the library, I got this error:

Exception in thread "main" java.lang.NoSuchMethodError:
com.sun.jna.Memory.setString(JLjava/lang/String;Z)V
at org.gstreamer.NativeArgs.<init>(Gst.java:157)
at org.gstreamer.Gst.init(Gst.java:109)
at example.Player.main(Player.java:39)

It went away when I used the jna.jar downloaded from the
jna.dev.java.net website, instead of the jna-20070909-r283.jar you
provide in the downloads section.

Andres


On Dec 5, 4:29 pm, "Wayne Meissner" <wmeiss...@gmail.com> wrote:
> You should now be able to use a RGBDataSink() as the video-sink element for
> a playbin, and get the data easily from that. You need to provide a
> RGBDataSink.Listener which gets given an RGB frame everytime one is decoded.
>
> Something like:
> GMainLoop loop = new GMainLoop();
> PlayBin player = new PlayBin("Video Data Extractor");
> RGBDataSink videoSink = new RGBDataSink("rgb",
> new RGBDataSink.Listener() {
> public void rgbFrame(int w, int h, IntBuffer rgbPixels) {
> // do whatever you want with the data here
> }
> }
> );
> player.setVideoSink(videoSink);
> player.setInputFile(new File(args[0]));
> loop.startInBackground();
> // go and do whatever else is needed/run some other loop
>

Wayne Meissner

unread,
Dec 6, 2007, 5:36:44 AM12/6/07
to gstream...@googlegroups.com
Ahh, yeh.  The rough rule of thumb is: when using code from svn, use the libraries under lib/ in the tree.  The binary releases of jna on gstreamer-java download page match up with the binary releases of gstreamer-java.

Great to see that its working for you.  Send a screenshot of it doing something cool.

Ravi Syamala

unread,
Dec 6, 2007, 12:59:25 PM12/6/07
to gstreamer-java
Sorry, I was following this and had a bit more trouble just writing
that out. For now, all I want to do is just print out each frame that
I reach in the movie. This class was a tester that I came up with
based on your example code above:

import java.awt.BorderLayout;
import java.awt.Dimension;
import java.awt.image.BufferedImage;
import java.io.File;
import java.io.IOException;
import java.nio.IntBuffer;

import javax.swing.JFrame;
import javax.swing.SwingUtilities;

import org.gstreamer.GMainLoop;
import org.gstreamer.Gst;
import org.gstreamer.PlayBin;
import org.gstreamer.elements.RGBDataSink;
import org.gstreamer.media.MediaPlayer;
import org.gstreamer.swing.GstVideoPlayer;

public class GStreamSample {
public GStreamSample()
{

}
public static void main(String[] args) {
//System.setProperty("sun.java2d.opengl", "True");
final GMainLoop loop = new GMainLoop();
final String file = args[0];

PlayBin player = new PlayBin("Video Data Extractor");
RGBDataSink videoSink = new RGBDataSink("rgb",
new RGBDataSink.Listener() {
public void rgbFrame(int w, int h, IntBuffer
rgbPixels) {
System.out.println("Got to a frame.");
File outputFile = new File(System.currentTimeMillis()
+".png"); // create a File Object
BufferedImage bufferedImage = new BufferedImage(w, h,
BufferedImage.TYPE_INT_ARGB);
bufferedImage.setRGB(0, 0, w, h, rgbPixels.array(), 0, w);
try {
javax.imageio.ImageIO.write(bufferedImage, "PNG",
outputFile);
} catch (IOException e) {
e.printStackTrace();
}
}
}
);
player.setVideoSink(videoSink);
player.setInputFile(new File(file));
loop.startInBackground();
}
}

However, this program just exits and doesn't produce any PNG files,
instead it just dies quietly. When I used a debugger to see what was
happening, I saw that it was falling right through. I think there is
something obvious that is missing. I am new to this package and
haven't worked with it before so any help would be appreciated.


On Dec 6, 5:36 am, "Wayne Meissner" <wmeiss...@gmail.com> wrote:
> Ahh, yeh. The rough rule of thumb is: when using code from svn, use the
> libraries under lib/ in the tree. The binary releases of jna on
> gstreamer-java download page match up with the binary releases of
> gstreamer-java.
>
> Great to see that its working for you. Send a screenshot of it doing
> something cool.
>

Wayne Meissner

unread,
Dec 6, 2007, 6:20:45 PM12/6/07
to gstream...@googlegroups.com
Change loop.startInBackground() to loop.run()

The startInBackground() is there to make it easier to work with other event loops (e.g the AWT/Swing EDT).


> > > > But now the problem is that gloop.run () interferes with the drawing

> > > > loop of my app. The thing is that I'm trying to use gstreamer-java
> > > > from within processing (processing.org ). I don't know if you are

acolubri

unread,
Dec 6, 2007, 10:27:11 PM12/6/07
to gstreamer-java
A couple of questions:

* Is it possible to change the speed of the playback?
* Does gstreamer support input from webcams?
* Is it possible to create video files with gstreamer?

Andres
> > > > > > But now the problem is that gloop.run() interferes with the
> > drawing
> > > > > > loop of my app. The thing is that I'm trying to use gstreamer-java
> > > > > > from within processing (processing.org). I don't know if you are

Wayne Meissner

unread,
Dec 7, 2007, 2:02:13 AM12/7/07
to gstream...@googlegroups.com
On 07/12/2007, acolubri <andres....@gmail.com> wrote:

A couple of questions:

* Is it possible to change the speed of the playback?

I think so.  You use a capsfilter (like RGBDataSink does), but instead of changing the output format, you change the framerate.  I can vaguely remember trying this and it worked erratically for me - but I wasn't really worried about it, so didn't try to work out the kinks.
 

* Does gstreamer support input from webcams?

Again, I think so.  You might want to ask on one of the gstreamer lists and/or google.

* Is it possible to create video files with gstreamer?

Sure is.  You just need to do the reverse of what RGBDataSink does - you want a RgbDataSource that you throw RGB frames at, and it sends them down a pipe.  I might look at it later if I get time.




acolubri

unread,
Dec 9, 2007, 12:00:59 AM12/9/07
to gstreamer-java
Hey Wayne,

Great to hear that it is possible to create videos with gstreamer. Let
me know when you have something more about this.

Now, I'll show you what I have done so far. Basically, I wrote a
"wrapper" for gstreamer-java so it can be used
as a library within processing. Works on both windows and linux,
didn't have time to test it on OSX. My intention is to
create an alternative to the built-in video library of processing,
based on quicktime: http://processing.org/reference/libraries/video/index.html

So basically, I implemented a class called GSMovie within the package
gsvideo, which has exactly the same API as the original
Movie class. I compared both libraries, the built-in and the gstreamer-
based, using an HD mov file, and gstreamer is much more faster
than the built-in library.

Here you can download the current version of the gsvideo library for
processing:
http://users.design.ucla.edu/~acolubri/processing/gsvideo/gsvideo.zip
Source code is here:
http://users.design.ucla.edu/~acolubri/processing/gsvideo/source-0.1.zip

and some examples:
http://users.design.ucla.edu/~acolubri/processing/gsvideo/gstreamer5.zip
- moves a little video across the screen using the mouse coordinates
http://users.design.ucla.edu/~acolubri/processing/gsvideo/gstreamer6.zip
- video scratching
http://users.design.ucla.edu/~acolubri/processing/gsvideo/gstreamer7.zip
- same as 5, but with two videos

In order to run these examples, you need to install processing on your
machine, and then the gsvideo library inside the processing/library
folder.
Processing can also creates applets to run from the web, but for some
reason the gstreamer applets don't work. Seems to be
some sort of security issue with jna:

Caused by: java.security.AccessControlException: access denied
(java.lang.RuntimePermission loadLibrary.jnidispatch)
at java.security.AccessControlContext.checkPermission(Unknown Source)
at java.security.AccessController.checkPermission(Unknown Source)
at java.lang.SecurityManager.checkPermission(Unknown Source)
at java.lang.SecurityManager.checkLink(Unknown Source)
at java.lang.Runtime.loadLibrary0(Unknown Source)
at java.lang.System.loadLibrary(Unknown Source)
at com.sun.jna.Native.<clinit>(Native.java:77)

Any ideas?

Andres

On Dec 6, 11:02 pm, "Wayne Meissner" <wmeiss...@gmail.com> wrote:

acolubri

unread,
Dec 17, 2007, 11:13:36 AM12/17/07
to gstreamer-java
Hi guys, I posted the work I'm doing with gstreamer and processing on
the processing.org website:
http://processing.org/discourse/yabb_beta/YaBB.cgi?board=os_libraries_tools;action=display;num=1197475141
People are quite interested in it.
However, many users of Processing work with Mac, anyone here has
installed gstreamer on OSX?


On Dec 9, 2:00 am, acolubri <andres.colu...@gmail.com> wrote:
> Hey Wayne,
>
> Great to hear that it is possible to create videos with gstreamer. Let
> me know when you have something more about this.
>
> Now, I'll show you what I have done so far. Basically, I wrote a
> "wrapper" for gstreamer-java so it can be used
> as a library within processing. Works on both windows and linux,
> didn't have time to test it on OSX. My intention is to
> create an alternative to the built-in video library of processing,
> based on quicktime:http://processing.org/reference/libraries/video/index.html
>
> So basically, I implemented a class called GSMovie within the package
> gsvideo, which has exactly the same API as the original
> Movie class. I compared both libraries, the built-in and the gstreamer-
> based, using an HD mov file, and gstreamer is much more faster
> than the built-in library.
>
> Here you can download the current version of the gsvideo library for
> processing:http://users.design.ucla.edu/~acolubri/processing/gsvideo/gsvideo.zip
> Source code is here:http://users.design.ucla.edu/~acolubri/processing/gsvideo/source-0.1.zip
>
> and some examples:http://users.design.ucla.edu/~acolubri/processing/gsvideo/gstreamer5.zip
> - moves a little video across the screen using the mouse coordinateshttp://users.design.ucla.edu/~acolubri/processing/gsvideo/gstreamer6.zip
> - video scratchinghttp://users.design.ucla.edu/~acolubri/processing/gsvideo/gstreamer7.zip

Wayne Meissner

unread,
Dec 17, 2007, 7:32:04 PM12/17/07
to gstream...@googlegroups.com
I use gstreamer from macports under OSX - but that only works without useful codecs like MP3, mpeg4, etc, since gst-plugins-bad fails to compile.  The macports version is a bit out of date, but it is sufficient for ogg/vorbis playback.

I'm attempting to get gstreamer compiled from cvs now.


>         at java.security.AccessControlContext.checkPermission (Unknown Source)

>         at java.security.AccessController.checkPermission(Unknown Source)
>         at java.lang.SecurityManager.checkPermission(Unknown Source)
>         at java.lang.SecurityManager.checkLink (Unknown Source)

dev....@gmail.com

unread,
Dec 17, 2007, 11:37:22 PM12/17/07
to gstreamer-java
Hi guys. i used to try install gstreamer under OSX (tiger) from
macport. but it was not installed.
especially, gst-plugins-bad fail, as like as wayne.
so i tried to compile gstreamer from source (maybe 0.10.15?)
the source compiled, but excepted many plugins. (
i think about that install it from FINK port.
but FINK support lower than distributed version ( the version was
0.10.01)

if i have time I'll have plan that try to build from lower version
source code.
when i try to build then report result. = )


On 12월18일, 오전9시32분, "Wayne Meissner" <wmeiss...@gmail.com> wrote:
> I use gstreamer from macports under OSX - but that only works without useful
> codecs like MP3, mpeg4, etc, since gst-plugins-bad fails to compile. The
> macports version is a bit out of date, but it is sufficient for ogg/vorbis
> playback.
>
> I'm attempting to get gstreamer compiled from cvs now.
>
> On 18/12/2007, acolubri <andres.colu...@gmail.com> wrote:
>
>
>
>
>
> > Hi guys, I posted the work I'm doing with gstreamer and processing on
> > the processing.org website:
>
> >http://processing.org/discourse/yabb_beta/YaBB.cgi?board=os_libraries...
> > > at java.lang.SecurityManager.checkLink(Unknown Source)
> > > at java.lang.Runtime.loadLibrary0(Unknown Source)
> > > at java.lang.System.loadLibrary(Unknown Source)
> > > at com.sun.jna.Native.<clinit>(Native.java:77)
>
> > > Any ideas?
>
> > > Andres
>
> > > On Dec 6, 11:02 pm, "Wayne Meissner" <wmeiss...@gmail.com> wrote:
>
> > > > On 07/12/2007, acolubri <andres.colu...@gmail.com> wrote:
>
> > > > > A couple of questions:
>
> > > > > * Is it possible to change the speed of the playback?
>
> > > > I think so. You use a capsfilter (like RGBDataSink does), but instead
> > of
> > > > changing the output format, you change the framerate. I can vaguely
> > > > remember trying this and it worked erratically for me - but I wasn't
> > really
> > > > worried about it, so didn't try to work out the kinks.
>
> > > > * Does gstreamer support input from webcams?
>
> > > > Again, I think so. You might want to ask on one of the gstreamer
> > lists
> > > > and/or google.
>
> > > > * Is it possible to create video files with gstreamer?
>
> > > > Sure is. You just need to do the reverse of what RGBDataSink does -
> > you
> > > > want a RgbDataSource that you throw RGB frames at, and it sends them
> > down a
> > > > pipe. I might look at it later if I get time.- 따온 텍스트 숨기기 -
>
> - 따온 텍스트 보기 -

acolubri

unread,
Dec 18, 2007, 10:48:34 AM12/18/07
to gstreamer-java
Is darwinports the same as macports? Here are all the gstreamer
packages available on darwinports.com

http://gstreamer.darwinports.com/ (0.10.14)

http://gst-plugins-base.darwinports.com/ (0.10.13)

http://gst-plugins-good.darwinports.com/ (0.10.6)

http://gst-plugins-bad.darwinports.com/ (0.10.3)

http://gst-plugins-ugly.darwinports.com/ (0.10.3)

Wayne Meissner

unread,
Dec 19, 2007, 2:19:48 AM12/19/07
to gstream...@googlegroups.com
Yes, macports is the successor to darwin ports.

If you just install gstreamer, gst-plugins-base, gst-plugins-good and gst-plugins-ugly, its good enough to play with gstreamer-java (thats what I develop on now), but to play most media you need -bad for mpeg4, x264, etc.

Wayne Meissner

unread,
Dec 19, 2007, 6:34:24 AM12/19/07
to gstream...@googlegroups.com
Ok, after a bit of experimentation, I have the latest stable releases of gstreamer and most of the plugins running.  Its a bit slow and laggy when playing 720P stuff, but at least it'll play xvid, mp3, etc stuff fine.

These are the steps I followed (note, I installed gstreamer into /usr/local to separate it from macports /opt hierarchy):

1) Install gstreamer gst-plugins-base gst-plugins-good gst-plugins-ugly from macports. 
This is just used to suck in a lot of dependencies, so after they are installed, uninstall them again.

2) Install these additional ports from macports:

x264  cdparanoia libcdio  cdrdao cdrtools vcdimager lame id3v2 id3lib id3tool libid3tag a52dec libdvdcss libdvdread libmpeg2 libmad  faac faad2 libmusicbrainz XviD libdts

3) Extract all the source for the latest gstreamer from the gstreamer download site.

4) Build each of gstreamer gst-plugins-base and gst-plugins-bad  with:

  LDFLAGS="-L/usr/local/lib -L/opt/local/lib" CFLAGS="-I/usr/local/include -I/opt/local/include"  \
  ./configure --prefix=/usr/local --with-pkg-config-path=/usr/local/lib/pkgconfig:/opt/local/lib/pkgconfig && make && sudo make install

5) Now build gst-plugins-good with the above configure line, but run make like so:

make DEPRECATED_CFLAGS="" && sudo make install

6) When configuring gst-ffmpeg, add '--disable-mmx' to the configure flags, then make && sudo make install

You might also want to add /usr/local/bin to $PATH.
i.e. export PATH=$PATH:/usr/local/bin

acolubri

unread,
Dec 19, 2007, 8:38:09 AM12/19/07
to gstreamer-java
Great!! Thanks a lot, Wayne!

Just two questions. Did you need any special options to compile the
base library?
And what are the dependencies that macports solve when you install the
gst-plugins-base, gst-plugins-good and gst-plugins-ugly packages? Is
it possible to install them separately?

Andres

Wayne Meissner

unread,
Dec 19, 2007, 9:39:28 AM12/19/07
to gstream...@googlegroups.com
On 19/12/2007, acolubri <andres....@gmail.com> wrote:

Great!! Thanks a lot, Wayne!

Just two questions. Did you need any special options to compile the
base library?

No, the base gstreamer library compiles using the same options as gst-plugins-base and gst-plugins-bad. 

And what are the dependencies that macports solve when you install the
gst-plugins-base, gst-plugins-good and gst-plugins-ugly packages? Is
it possible to install them separately?

I'm not sure which packages it sucks in, because:
a) I already had them installed, and have a heap of other macports stuff installed
b) I wasn't going to go through the pain of removing macports and redoing it all from scratch.

I figure someone else who has the time, can figure out what individual packages are needed, but I'd at least get a method to get a working setup done, even if it means double installing gstreamer.

Reply all
Reply to author
Forward
0 new messages