Various MediaTools implementations with variable FPS output

[je...@jungletech.com]

We have created several MediaTools implementations for capturing audio
from a microphone and video from the screen.

One implementation is very closely modeled after the Xuggler demo
(GenerateAudioAndVideo.java) and the other is a custom implementation
that implements the MediaTools.addVideoStream method which moderates
the frame rate using an IRational.

What we notice is that with the first demo is the frame rate is
correct however the video goes at super speed and finishes very
quickly. The later demo's playback is correct but the frame rate
(which we expect to be around 30 FPS which is set in the
MediaTools.addVideoStream method with the IRational) is typically much
lower (~7-8 FPS which we found out by inspecting the video using
players like VLC, Quicktime, etc.).

For the first demo: why is the playback so quick? Is it because there
isn't enough information between segments to allocate 30 FPS so the
timestamps in Xuggler are regulating the speed and thus the video
plays back quicker? Is it possible for Xuggler to not be able to run
fast enough to capture enough images in a single thread? Do we need to
push this to another thread?

For the second demo: Why isn't the MediaWriter regulating the FPS as
we expected? What is Xuggler doing on the back end to regulate this?

I've included sample run methods for both demos, we haven't included
all of our classes but all of our names are explicative and the
relevant constants that we use to create our Xuggler and Java objects
are there.

--------------------Demo 1-------------------------------
public void run ()
{
try
{
// Instantiate Objects
final double FPS = 30; // Video Settings
final NUMBER_OF_PICTURES_IN_GROUP_OF_PICTURES = 30;
final BITRATE = 25000;
final long FRAME_CAPTURE_RATE = DEFAULT_TIME_UNIT.convert(30,
MILLISECONDS);
final int AUDIO_STREAM_ID = 1; // Encoding Settings
final int VIDEO_STREAM_ID = 0;
final float SAMPLE_RATE = 44100.0F; // Audio Settings
final int SAMPLE_RATE_AS_INT = (int)SAMPLE_RATE;
final int SAMPLE_SIZE_IN_BITS = 16;
final int NUMBER_OF_CHANNELS = 1;
final boolean SIGNED = true;
final boolean BIG_ENDIAN = false;
boolean isCapturing = true; // Capture Variables
byte tempBuffer[] = new byte[1000];
long nextFrameTime = 0;
long totalSampleCount = 0;
long numberOfAudioSamplesInDefaultPoints = 0;
ScreenManager screenManager = CustomXugglerFactory.makeScreenManager
(); // Xuggler Objects
TargetDataLine targetDataLine =
CustomXugglerFactory.makeTargetDataLineWrapper().getTargetDataLine();
IMediaWriter mediaWriter = ToolFactory.makeWriter
("sample_xuggler_capture.mp4");

// Set Properties
mediaWriter.open();
mediaWriter.addVideoStream(0, 0, ICodec.guessEncodingCodec(null,
null, "sample_xuggler_capture.mp4", null,
ICodec.Type.CODEC_TYPE_VIDEO), _screenWidth, _screenHeight);
mediaWriter.addAudioStream(AUDIO_STREAM_ID, AUDIO_STREAM_ID,
ICodec.guessEncodingCodec(null, null, "sample_xuggler_capture.mp4",
null, ICodec.Type.CODEC_TYPE_AUDIO), NUMBER_OF_CHANNELS,
SAMPLE_RATE_AS_INT);
mediaWriter.setForceInterleave(true);
mediaWriter.addListener(ToolFactory.makeDebugListener());

while (isCapturing)
{
while (numberOfAudioSamplesInDefaultPoints >= nextFrameTime)
{
mediaWriter.encodeVideo(VIDEO_STREAM_ID,
ConverterFactory.convertToType(screenManager.returnSingleSnapshot(),
BufferedImage.TYPE_3BYTE_BGR), nextFrameTime, DEFAULT_TIME_UNIT);

nextFrameTime += FRAME_CAPTURE_RATE;
}

if (targetDataLine.read(tempBuffer, 0, tempBuffer.length) > 0)
{
IBuffer audioBuffer = IBuffer.make(null, tempBuffer, 0,
tempBuffer.length);
audioBuffer.setType(IBuffer.Type.IBUFFER_SINT16);

IAudioSamples audioSamples = IAudioSamples.make(audioBuffer,
NUMBER_OF_CHANNELS, IAudioSamples.Format.FMT_S16);
audioSamples.setComplete(true, audioBuffer.getSize(),
SAMPLE_RATE_AS_INT, NUMBER_OF_CHANNELS, IAudioSamples.Format.FMT_S16,
Global.NO_PTS);

numberOfAudioSamplesInDefaultPoints =
IAudioSamples.samplesToDefaultPts(totalSampleCount,
SAMPLE_RATE_AS_INT);
totalSampleCount += (int)audioSamples.getNumSamples();

mediaWriter.encodeAudio(AUDIO_STREAM_ID, audioSamples);
}
}
}
catch (Exception $exception)
{
// DROP THIS FRAME OF CAPTURE
}
}
--------------------------------------------------------------
--------------------Demo 2-------------------------------
public void run ()
{
try
{
// Instantiate Objects
final double FPS = 30; // Video Settings
final NUMBER_OF_PICTURES_IN_GROUP_OF_PICTURES = 30;
final BITRATE = 25000;
final int AUDIO_STREAM_ID = 1; // Encoding Settings
final int VIDEO_STREAM_ID = 0;
final float SAMPLE_RATE = 44100.0F; // Audio Settings
final int SAMPLE_RATE_AS_INT = (int)SAMPLE_RATE;
final int SAMPLE_SIZE_IN_BITS = 16;
final int NUMBER_OF_CHANNELS = 1;
final boolean SIGNED = true;
final boolean BIG_ENDIAN = false;
boolean isCapturing = true; // Capture Variables
long startTime = System.currentTimeMillis();
byte [] tempBuffer = new byte[1000];
IConverter converter; // Xuggler Objects
ScreenManager screenManager = CustomXugglerFactory.makeScreenManager
();
TargetDataLine targetDataLine =
CustomXugglerFactory.makeTargetDataLineWrapper().getTargetDataLine();
IMediaWriter mediaWriter = ToolFactory.makeWriter
("sample_xuggler_capture.mp4");

// Set Properties
mediaWriter.open();
mediaWriter.addVideoStream(0, 0, ICodec.guessEncodingCodec(null,
null, "sample_xuggler_capture.mp4", null,
ICodec.Type.CODEC_TYPE_VIDEO), IRational.make(3000, 100) ,
_screenWidth, _screenHeight);
mediaWriter.addAudioStream(AUDIO_STREAM_ID, AUDIO_STREAM_ID,
ICodec.guessEncodingCodec(null, null, "sample_xuggler_capture.mp4",
null, ICodec.Type.CODEC_TYPE_AUDIO), NUMBER_OF_CHANNELS,
SAMPLE_RATE_AS_INT);
mediaWriter.setForceInterleave(true);
mediaWriter.addListener(ToolFactory.makeDebugListener());

while (_isCapturing)
{
if (targetDataLine.read(tempBuffer, 0, tempBuffer.length) > 0)
{
IBuffer audioBuffer = IBuffer.make(null, tempBuffer, 0,
tempBuffer.length);
audioBuffer.setType(IBuffer.Type.IBUFFER_SINT16);

IAudioSamples audioSamples = IAudioSamples.make(audioBuffer, 1,
IAudioSamples.Format.FMT_S16);
audioSamples.setComplete(true, audioBuffer.getSize(), SAMPLE_RATE,
1, IAudioSamples.Format.FMT_S16, Global.NO_PTS);

mediaWriter.encodeAudio(AUDIO_STREAM_ID, audioSamples);
}
BufferedImage postConvertImage = ConverterFactory.convertToType
(_screenManager.returnSingleSnapshot(), BufferedImage.TYPE_3BYTE_BGR);

if (converter == null)
{
converter = ConverterFactory.createConverter(postConvertImage,
VideoConstants.VIDEO_STREAM_PIXEL_FORMAT);
}
IVideoPicture videoPicture = _converter.toPicture(postConvertImage,
(long)((System.currentTimeMillis() - startTime) * 1000));
videoPicture.setQuality(VideoConstants.QUALITY);
videoPicture.setComplete(true, IPixelFormat.Type.YUV420P,
_screenWidth, _screenHeight, (long)((System.currentTimeMillis() -
startTime) * 1000));

mediaWriter.encodeVideo(VIDEO_STREAM_ID, videoPicture);
}
}
catch (Exception $exception)
{
// DROP THIS FRAME OF CAPTURE
}
}
--------------------------------------------------------------

TIA,
---Jeff

[je...@jungletech.com]

We've created another test to help debug this issue. It appears like
Java is unable to capture the screen via the native libraries
(robot.createScreenCapture) with a 1024x768 resolution fast enough to
capture the frame rate desired (29.97 FPS). See the below code sample
for the test that we ran to find the average:

---------------- Frame Rate Demo -------------------
public void run ()
{
double timeToCaptureScreen = 0;
double beforeCaptureTime = 0;
double currentFPS = 0;

while (/* condition for capturing is true*/)
{
beforeCaptureTime = System.nanoTime();
_writer.encodeVideo(0, ConverterFactory.convertToType
(_screenManager.returnSingleSnapshot(), BufferedImage.TYPE_3BYTE_BGR),
System.nanoTime(), TimeUnit.NANOSECONDS);
timeToCaptureScreen += (System.nanoTime() -
beforeCaptureTime);
_counter++;

currentFPS = ((timeToCaptureScreen / _counter) * 0.000000001);
System.out.println(currentFPS);
_totalAverageFPS += currentFPS;
}
System.out.println("TOTAL AVERAGE FPS FOR CAPTURE IS: " +
(_totalAverageFPS / _counter));
}
----------------------------------------

On each of our systems this comes out to be somewhere around ~8-10
FPS.

Is there a way to increase the FPS? Perhaps use the Java OpenGL
library to capture images (http://kenai.com/projects/jogl/pages/Home)?
Implement a screen capture that only uses OpenGL? Or could we change
or image container type (that is instead of capturing 3_BYTE_BGR
capture another type that is possibly faster)? Capture the screen at a
lower quality?

Thanks again for the support,
--Jeff

[Art Clarke]

Jeff, I would suggest you keep measuring where the performance bottleneck is and optimize from there.  I suspect you'll find that it's the java.robot method that is taking the most time.

If it is the robot class, look into other ways to capture screens.  JOGL might do it (I'm not sure).  If JOGL does, if it allows captures in YUV space, that'll be faster, but if not, you'll have to convert from RGB24 to YUV420P.  For that, try to use the JOGL Native ByteBuffers (if accessible) and pass those to the IVideoResampler object in Xuggler's.  In that way the images wouldn't need to be copied to and from Java byte[] arrays which will speed things up.  Xuggler's IVideoResampler object will resample from one area of native memory to another using MMX assembly code so it'll be pretty fast.

If it's xuggler causing the performance problems, find out where (again measure).  If it's colorspace conversion, use the IVideoResampler interface and cache objects.  If it's encoding, try tweaking setProperty() settings on your IStreamCoders to trade off lower-quality or high-bandwidth for less CPU time.

But in short... measure, measure, measure.  You'll find things move much faster (and I can be more helpful) if you actually have performance numbers.

Hope that helps.

- Art

--Jeff

--

You received this message because you are subscribed to the Google Groups "xuggler-users" group.
To post to this group, send email to xuggle...@googlegroups.com.
To unsubscribe from this group, send email to xuggler-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/xuggler-users?hl=en.

--
http://www.xuggle.com/
xu‧ggle (zŭ' gl) v. To freely encode, decode, and experience audio and video.

Use Xuggle to get the power of FFmpeg in Java.

[je...@jungletech.com]

Thanks for the quick reply and for the advice in implementing JOGL,

We are currently looking into implementing JOGL (since based on some
of our test we found robot is causing at least one bottleneck).

We'll be back shortly with some additional demos with performance
measurements.

---Jeff