Trying to capture screen + combine it with audio from microphone

[abwayaxkauchaomai]

I'm trying to use Xuggler to record a video of the user's monitor and
combine it with audio from microphone input. I'm following the
MediaTools screen grabber example combined with code using the Java
Sound API to access the microphone. The capture class runs in a
separate thread along with a GUI that features a stop/start capture
button.



I have searched this group, the wiki, and the JavaDocs for information
that might help. All the examples I found dealt either with using an
existing sound file or generating sound from scratch, not with using
sound data gathered from a microphone.



I have also considered running a separate thread for capturing audio
into an audio file and merging the two after ending the capture, but I
felt that for larger captures this would not be efficient.



I find that when I try to compile and I create the finalized video
output in mp4, I am only getting video; no audio. I think this is
because the audio isn't correctly being encoded into the mp4
container. Can you please help me debug this problem; where in my code
am I messing up?



************************************************************************************************************

Source code for the capture class below:

************************************************************************************************************

--

/**

* @(#)CaptureThread.java

*

*

* @author

* @version 1.00 2009/10/2

*/



import com.xuggle.xuggler.*;

import com.xuggle.ferry.*;

import com.xuggle.mediatool.*;

import java.awt.*;

import java.awt.image.*;

import java.util.concurrent.TimeUnit;



import javax.sound.sampled.*;



public class CaptureThread extends Thread {



private boolean inCapture = false;

private TargetDataLine line;

private AudioFileFormat.Type targetType;

private AudioInputStream audioInputStream;

private String captureDir;



public CaptureThread() {

// Audio format

AudioFormat audioFormat = new AudioFormat(

AudioFormat.Encoding.PCM_SIGNED,

44100.0F, 16, 2, 4, 44100.0F, false);



/* Now, we are trying to get a TargetDataLine. The

TargetDataLine is used later to read audio data from it.

If requesting the line was successful, we are opening

it (important!).

*/

DataLine.Info info = new DataLine.Info(TargetDataLine.class,
audioFormat);

try

{

this.line = (TargetDataLine) AudioSystem.getLine(info);

this.line.open(audioFormat);

}

catch (LineUnavailableException e)

{

System.out.println("unable to get a recording line");

e.printStackTrace();

System.exit(1);

}



/* Again for simplicity, we've hardcoded the audio file

type, too.

*/

this.targetType = AudioFileFormat.Type.WAVE;



this.audioInputStream = new AudioInputStream(line);

}



public void startCapture() {

this.inCapture = true;

this.start();

}



public void setCaptureDir(String path) {

captureDir = path;

}



public void stopCapture() {

this.inCapture = false;

}



public void run() {

try {

this.line.start();

this.capture();

} catch(Exception e) {

e.printStackTrace();

}

}



public void capture() throws Exception {

final IRational FRAME_RATE = IRational.make(30);

final int SECONDS_TO_RUN_FOR = 20;



final Robot robot = new Robot();

final Toolkit toolkit = Toolkit.getDefaultToolkit();

final Rectangle screenBounds = new Rectangle(toolkit.getScreenSize
());



// First, let's make a IMediaWriter to write the file.

final IMediaWriter writer = ToolFactory.makeWriter(captureDir + "/
output.mp4");



// Add a debugger to the writer

writer.addListener(ToolFactory.makeDebugListener());



// We tell it we're going to add one video stream, with id 0,

// at position 0, and that it will have a fixed frame rate of

// FRAME_RATE.

writer.addVideoStream(0, 0,

FRAME_RATE,

screenBounds.width, screenBounds.height);



// Add audio stream

writer.addAudioStream(1, 0,

1, 44100);



// audio buffer

byte[] audioBuf = new byte[192000];



// Now, we're going to loop

long startTime = System.nanoTime();

while(inCapture) {

//for (int index = 0; index < FRAME_RATE.getDouble(); index++)

//{

// take the screen shot

BufferedImage screen = robot.createScreenCapture(screenBounds);



// convert to the right image type

BufferedImage bgrScreen = xugtest.convertToType(screen,

BufferedImage.TYPE_3BYTE_BGR);



long nanoTs = System.nanoTime()-startTime;



// encode the image to stream #0

writer.encodeVideo(0,bgrScreen,

nanoTs, TimeUnit.NANOSECONDS);

//System.out.println("encoded image: " +index);



// get and encode audio

int nBytesRead = this.line.read(audioBuf, 0, 192000);

System.out.println("Read "+nBytesRead);

// encode audio to stream #1

IBuffer iBuf = IBuffer.make(null,audioBuf,0,192000);

/*System.out.println(iBuf);

for(int i = 0; i < 40960; i++) {

System.out.print((char) audioBuf[i]);

if(i % 100 == 0) System.out.println();

}*/

IAudioSamples smp = IAudioSamples.make(iBuf,
1,IAudioSamples.Format.FMT_S16);

smp.setComplete(true,1,44100,1,IAudioSamples.Format.FMT_S16,
nanoTs / 1000);

smp.put(audioBuf,0,0,192000);

writer.encodeAudio(1,smp);



// increment counter

xugtest.window.increment();



// sleep for framerate milliseconds

Thread.sleep((long) (1000 / FRAME_RATE.getDouble()));

//}

}

// Finally we tell the writer to close and write the trailer if

// needed

this.line.stop();

//this.line.close();

writer.close();



}

}

************************************************************************************************************

************************************************************************************************************



************************************************************************************************************

and here's the contents of test.log:

************************************************************************************************************

2009-10-06 14:16:37,897 [Thread-3] DEBUG
c.x.mediatool.MediaDebugListener - onOpen()

2009-10-06 14:16:37,926 [Thread-3] DEBUG
c.x.mediatool.MediaDebugListener - onAddStream(0)

2009-10-06 14:16:37,928 [Thread-3] DEBUG
c.x.mediatool.MediaDebugListener - onAddStream(1)

2009-10-06 14:16:38,021 [Thread-3] DEBUG
c.x.mediatool.MediaDebugListener - onOpenStream(0)

2009-10-06 14:16:38,026 [Thread-3] DEBUG
c.x.mediatool.MediaDebugListener - onOpenStream(1)

2009-10-06 14:16:38,027 [Thread-3] DEBUG
c.x.mediatool.MediaDebugListener - onWriteHeader()

2009-10-06 14:16:38,096 [Thread-3] DEBUG
c.x.mediatool.MediaDebugListener - onWritePacket
(com.xuggle.xuggler.IPacket@81375544[complete:true;dts:2;pts:2;size:
243965;key:true;flags:1;stream index:0;duration:-1;position:-1;time
base:1/30;])

2009-10-06 14:16:38,099 [Thread-3] DEBUG
c.x.mediatool.MediaDebugListener - onVideoPicture
(com.xuggle.xuggler.IVideoPicture@81375920[pixel type:YUV420P;width:
1366;height:768;time stamp:87610;complete:true;size:
1573632;key:true;time base:1/1000000;], BufferedImage@29428e: type = 5
ColorModel: #pixelBits = 24 numComponents = 3 color space =
java.awt.color.ICC_ColorSpace@d0a5d9 transparency = 1 has alpha =
false isAlphaPre = false ByteInterleavedRaster: width = 1366 height =
768 #numDataElements 3 dataOff[0] = 2, 0)

2009-10-06 14:16:38,730 [Thread-3] DEBUG
c.x.mediatool.MediaDebugListener - onAudioSamples
(com.xuggle.xuggler.IAudioSamples@81375544[sample rate:44100;channels:
1;format:FMT_S16;time stamp:87610;complete:true;num samples:1;size:
2;key:true;time base:1/1000000;], 1)

2009-10-06 14:16:38,888 [Thread-3] DEBUG
c.x.mediatool.MediaDebugListener - onWritePacket
(com.xuggle.xuggler.IPacket@81376480[complete:true;dts:27;pts:27;size:
139679;key:false;flags:0;stream index:0;duration:-1;position:-1;time
base:1/30;])

2009-10-06 14:16:38,900 [Thread-3] DEBUG
c.x.mediatool.MediaDebugListener - onVideoPicture
(com.xuggle.xuggler.IVideoPicture@81375624[pixel type:YUV420P;width:
1366;height:768;time stamp:915825;complete:true;size:
1573632;key:true;time base:1/1000000;], BufferedImage@72ffb: type = 5
ColorModel: #pixelBits = 24 numComponents = 3 color space =
java.awt.color.ICC_ColorSpace@d0a5d9 transparency = 1 has alpha =
false isAlphaPre = false ByteInterleavedRaster: width = 1366 height =
768 #numDataElements 3 dataOff[0] = 2, 0)

2009-10-06 14:16:39,777 [Thread-3] DEBUG
c.x.mediatool.MediaDebugListener - onAudioSamples
(com.xuggle.xuggler.IAudioSamples@81375808[sample rate:44100;channels:
1;format:FMT_S16;time stamp:915825;complete:true;num samples:1;size:
2;key:true;time base:1/1000000;], 1)

2009-10-06 14:16:39,982 [Thread-3] DEBUG
c.x.mediatool.MediaDebugListener - onWritePacket
(com.xuggle.xuggler.IPacket@81376480[complete:true;dts:60;pts:60;size:
101445;key:false;flags:0;stream index:0;duration:-1;position:-1;time
base:1/30;])

2009-10-06 14:16:39,982 [Thread-3] DEBUG
c.x.mediatool.MediaDebugListener - onVideoPicture
(com.xuggle.xuggler.IVideoPicture@81375808[pixel type:YUV420P;width:
1366;height:768;time stamp:2025722;complete:true;size:
1573632;key:true;time base:1/1000000;], BufferedImage@1193779: type =
5 ColorModel: #pixelBits = 24 numComponents = 3 color space =
java.awt.color.ICC_ColorSpace@d0a5d9 transparency = 1 has alpha =
false isAlphaPre = false ByteInterleavedRaster: width = 1366 height =
768 #numDataElements 3 dataOff[0] = 2, 0)

2009-10-06 14:16:40,984 [Thread-3] DEBUG
c.x.mediatool.MediaDebugListener - onAudioSamples
(com.xuggle.xuggler.IAudioSamples@81376048[sample rate:44100;channels:
1;format:FMT_S16;time stamp:2025722;complete:true;num samples:1;size:
2;key:true;time base:1/1000000;], 1)

2009-10-06 14:16:41,141 [Thread-3] DEBUG
c.x.mediatool.MediaDebugListener - onWritePacket
(com.xuggle.xuggler.IPacket@83306416[complete:true;dts:95;pts:95;size:
132549;key:false;flags:0;stream index:0;duration:-1;position:-1;time
base:1/30;])

2009-10-06 14:16:41,142 [Thread-3] DEBUG
c.x.mediatool.MediaDebugListener - onVideoPicture
(com.xuggle.xuggler.IVideoPicture@81376080[pixel type:YUV420P;width:
1366;height:768;time stamp:3184314;complete:true;size:
1573632;key:true;time base:1/1000000;], BufferedImage@1f4689e: type =
5 ColorModel: #pixelBits = 24 numComponents = 3 color space =
java.awt.color.ICC_ColorSpace@d0a5d9 transparency = 1 has alpha =
false isAlphaPre = false ByteInterleavedRaster: width = 1366 height =
768 #numDataElements 3 dataOff[0] = 2, 0)

2009-10-06 14:16:42,020 [Thread-3] DEBUG
c.x.mediatool.MediaDebugListener - onAudioSamples
(com.xuggle.xuggler.IAudioSamples@81375640[sample rate:44100;channels:
1;format:FMT_S16;time stamp:3184314;complete:true;num samples:1;size:
2;key:true;time base:1/1000000;], 1)

2009-10-06 14:16:42,230 [Thread-3] DEBUG
c.x.mediatool.MediaDebugListener - onWritePacket
(com.xuggle.xuggler.IPacket@83306464[complete:true;dts:128;pts:
128;size:15156;key:false;flags:0;stream index:
0;duration:-1;position:-1;time base:1/30;])

2009-10-06 14:16:42,231 [Thread-3] DEBUG
c.x.mediatool.MediaDebugListener - onVideoPicture
(com.xuggle.xuggler.IVideoPicture@81376048[pixel type:YUV420P;width:
1366;height:768;time stamp:4274253;complete:true;size:
1573632;key:true;time base:1/1000000;], BufferedImage@1ce2dd4: type =
5 ColorModel: #pixelBits = 24 numComponents = 3 color space =
java.awt.color.ICC_ColorSpace@d0a5d9 transparency = 1 has alpha =
false isAlphaPre = false ByteInterleavedRaster: width = 1366 height =
768 #numDataElements 3 dataOff[0] = 2, 0)

2009-10-06 14:16:43,108 [Thread-3] DEBUG
c.x.mediatool.MediaDebugListener - onAudioSamples
(com.xuggle.xuggler.IAudioSamples@81375640[sample rate:44100;channels:
1;format:FMT_S16;time stamp:4274253;complete:true;num samples:1;size:
2;key:true;time base:1/1000000;], 1)

2009-10-06 14:16:43,213 [Thread-3] DEBUG
c.x.mediatool.MediaDebugListener - onFlush()

2009-10-06 14:16:43,218 [Thread-3] DEBUG
c.x.mediatool.MediaDebugListener - onWriteTrailer()

2009-10-06 14:16:43,220 [Thread-3] DEBUG
c.x.mediatool.MediaDebugListener - onCloseStream(0)

2009-10-06 14:16:43,220 [Thread-3] DEBUG
c.x.mediatool.MediaDebugListener - onCloseStream(1)

2009-10-06 14:16:43,222 [Thread-3] DEBUG
c.x.mediatool.MediaDebugListener - onClose()

************************************************************************************************************

************************************************************************************************************

[Robert Harris]

On Tue, Oct 6, 2009 at 1:58 PM, abwayaxkauchaomai <abw...@gmail.com> wrote:

> I find that when I try to compile and I create the finalized video
> output in mp4, I am only getting video; no audio. I think this is
> because the audio isn't correctly being encoded into the mp4
> container. Can you please help me debug this problem; where in my code
> am I messing up?

when you call setComplete():

>
>                         smp.setComplete(true,1,44100,1,IAudioSamples.Format.FMT_S16,
> nanoTs / 1000);

you are specifying a numSamples (2nd param) value of 1. i assume you
want to put way more then 1 audio sample in this object. :)

-trebor

--
http://www.xuggle.com/
xu‧ggle (zŭ' gl) v. To freely encode, decode, and experience audio and video.

Use Xuggle to get the power of FFMPEG in Java.

[abwayaxkauchaomai]

On Oct 6, 8:05 pm, Robert Harris <tre...@xuggle.com> wrote:

> On Tue, Oct 6, 2009 at 1:58 PM, abwayaxkauchaomai <abwa...@gmail.com> wrote:
> > I find that when I try to compile and I create the finalized video
> > output in mp4, I am only getting video; no audio. I think this is
> > because the audio isn't correctly being encoded into the mp4
> > container. Can you please help me debug this problem; where in my code
> > am I messing up?
>
> when you call setComplete():
>
>
>
> >                         smp.setComplete(true,1,44100,1,IAudioSamples.Format.FMT_S16,
> > nanoTs / 1000);
>
> you are specifying a numSamples (2nd param) value of 1.  i assume you
> want to put way more then 1 audio sample in this object. :)
>

Thanks. I changed the numSamples to a larger value and it -seemed- to
work, except now the audio isn't synced up with the video correctly.
Also I noticed that when I would seek while the video was playing back
it would play the audio correctly for a little while. I also tried
adjusting the size of the byte buffer holding the microphone data,
thinking this would have an effect. It didn't seem to work. Currently
I have the buffer size at 40000 bytes and the numSamples set equal to
the sampling rate (44100).

Is there an example app I can look at that does something similar to
this? i.e. encodes input from a microphone for the audio and does a
desktop screen capture and interlaces the two correctly?

[Robert Harris]

On Fri, Oct 9, 2009 at 10:07 AM, abwayaxkauchaomai <abw...@gmail.com> wrote:

> Thanks. I changed the numSamples to a larger value and it -seemed- to
> work, except now the audio isn't synced up with the video correctly.

just to be certain, it shouldn't be any arbitrary number, but the
actual number of samples you've collected from the microphone.

is the audio getting ahead or behind? in general timestamps on video
are honored by players. the audio is assumed to be complete, and
played as fast as it arrives. if any audio is missing, the video will
get ahead.

i notice that your computing image time stamps from clock time. you
might instead use the amount of time which has passed on the audio to
drive screen capture and video time stamp computation.

> Is there an example app I can look at that does something similar to
> this? i.e. encodes input from a microphone for the audio and does a
> desktop screen capture and interlaces the two correctly?

no we do not have an example of doing this. if you like you can add
it to the list of desired demos in the wiki, but it's not likely to be
top priority unless we see other folks calling for it.