how to resolve pixel jittering in javacv generated video

Vikram Jain

unread,

Sep 16, 2024, 5:54:36 PM9/16/24

to javacv

I am trying to generate video using images, where image have some overlays text and png icons. I am using javacv library for this. Final output video seems pixelated, i don't understand what is it since i do not have video processing domain knowledge, i am beginner to this. I know that video bitrate and choice of video encoder are important factor which contributes to video quality and there are many more factors too.

I am providing you two output for comparison, one of them is generated using javacv and another one is from moviepy library

Please watch it in full screen since the problem i am talking about only gets highlighted in full screen, you will see the pixel dancing in javacv generated video, but python output seems stable

https://imgur.com/a/aowNnKg - javacv generated

https://imgur.com/a/eiLXrbk - Moviepy generated

I am using same encoder in both of the implementation

Encoder - libx264 bitrate - 800 Kbps for javacv 500 Kbps for moviepy frame rate - 24fps for both of them output video size -> 7MB (javacv) 5MB (Moviepy)

generated output size from javacv is bigger then moviepy generated video.

here is my java configuration for FFmpegFrameRecorder

FFmpegFrameRecorder recorder = new FFmpegFrameRecorder(this.outputPath, this.screensizeX, this.screensizeY);

if(this.videoCodecName!=null && "libx264".equals(this.videoCodecName)) { recorder.setVideoCodec(avcodec.AV_CODEC_ID_H264); }

recorder.setFormat("mp4");

recorder.setPixelFormat(avutil.AV_PIX_FMT_YUV420);

recorder.setVideoBitrate(800000);

recorder.setImageWidth(this.screensizeX);

recorder.setFrameRate(24);

and here is python configuration for writing video file

Full_final_clip.write_videofile( f"{video_folder_path}/{FILE_ID}_INTERMEDIATE.mp4", codec="libx264", audio_codec="aac", temp_audiofile=f"{FILE_ID}_INTER_temp-audio.m4a", remove_temp=True, fps=24, )

as you can see i am not specifying bitrate in python, but i checked that bitrate of final output is around 500 kbps, which is lower then what i specified in java, yet java generated video quality seems poor.

I have tried setting crf value also , but it seems it does not have any impact when used.

increasing bitrate improve quality somewhat but at the cost of file size, still generated output seems pixelated.

Can someone please highlight what might be the issue, and how python is generating better quality video, when both of the libraries use ffmpeg at the backend.

Edit 1 : also, I am adding code which is being used to make zoom animation for continuous frames, As somewhere i read that this might be the cause for pixel jitter, please see and let me know if there is any improvement we can do to remove pixel jittering

private Mat applyZoomEffect(Mat frame, int currentFrame, long effectFrames, int width, int height, String mode, String position, double speed) {

long totalFrames = effectFrames;

double i = currentFrame;

if ("out".equals(mode)) {

i = totalFrames - i;

}

double zoom = 1 + (i * ((0.1 * speed) / totalFrames));

double originalCenterX = width/2.0;

double originalCenterY = height/2.0;

// Resize image

opencv_imgproc.resize(frame, resizedMat, new Size(newWidth, newHeight));

// Determine crop region based on position

double x = 0, y = 0;

switch (position.toLowerCase()) {

case "center":

// Adjusting for center

x = originalCenterX - originalCenterX * zoom;

y = originalCenterY - originalCenterY * zoom;

break;

}

double[][] rowData = {{zoom, 0, x},{0,zoom,y}};

double[] flatData = flattenArray(rowData);

// Create a DoublePointer from the flattened array

DoublePointer doublePointer = new DoublePointer(flatData);

// Create a Mat object with two rows and three columns

Mat mat = new Mat(2, 3, org.bytedeco.opencv.global.opencv_core.CV_64F); // CV_64F is for double type

// Put the data into the Mat object

mat.data().put(doublePointer);

Mat transformedFrame = new Mat();

opencv_imgproc.warpAffine(frame, transformedFrame, mat, new Size(frame.cols(), frame.rows()),opencv_imgproc.INTER_LANCZOS4,0,new Scalar(0,0,0,0));

return transformedFrame; }

Samuel Audet

unread,

Sep 16, 2024, 10:11:54 PM9/16/24

to jav...@googlegroups.com, Vikram Jain

If you're expecting to get the exact same results from the exact same encoder, you'll need to make sure you give it the exact same settings in both cases...

Vikram Jain

unread,

Sep 17, 2024, 3:01:05 PM9/17/24

to javacv

Hi Samuel,

Thanks for quick response

I am attaching the complete details of both video , please have a look, i think parameters which have major impact are almost same.

Java Video Details

{'format': {'bit_rate': '939570',
'duration': '117.166667',
'filename': '/Users/vikramkumar/Downloads/472_WEB.mp4',
'format_long_name': 'QuickTime / MOV',
'format_name': 'mov,mp4,m4a,3gp,3g2,mj2',
'nb_programs': 0,
'nb_stream_groups': 0,
'nb_streams': 2,
'probe_score': 100,
'size': '13760788',
'start_time': '0.000000',
'tags': {'compatible_brands': 'isomiso2avc1mp41',
'encoder': 'Lavf60.16.100',
'major_brand': 'isom',
'minor_version': '512'}},
'streams': [{'avg_frame_rate': '24/1',
'bit_rate': '872130',
'bits_per_raw_sample': '8',
'chroma_location': 'left',
'closed_captions': 0,
'codec_long_name': 'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10',
'codec_name': 'h264',
'codec_tag': '0x31637661',
'codec_tag_string': 'avc1',
'codec_type': 'video',
'coded_height': 720,
'coded_width': 1280,
'disposition': {'attached_pic': 0,
'captions': 0,
'clean_effects': 0,
'comment': 0,
'default': 1,
'dependent': 0,
'descriptions': 0,
'dub': 0,
'forced': 0,
'hearing_impaired': 0,
'karaoke': 0,
'lyrics': 0,
'metadata': 0,
'non_diegetic': 0,
'original': 0,
'still_image': 0,
'timed_thumbnails': 0,
'visual_impaired': 0},
'duration': '117.166667',
'duration_ts': 1439744,
'extradata_size': 30,
'field_order': 'progressive',
'film_grain': 0,
'has_b_frames': 0,
'height': 720,
'id': '0x1',
'index': 0,
'is_avc': 'true',
'level': 31,
'nal_length_size': '4',
'nb_frames': '2812',
'pix_fmt': 'yuv420p',
'profile': 'Constrained Baseline',
'r_frame_rate': '24/1',
'refs': 1,
'start_pts': 0,
'start_time': '0.000000',
'tags': {'handler_name': 'VideoHandler',
'language': 'und',
'vendor_id': '[0][0][0][0]'},
'time_base': '1/12288',
'width': 1280},
{'avg_frame_rate': '0/0',
'bit_rate': '65075',
'bits_per_sample': 0,
'channel_layout': 'mono',
'channels': 1,
'codec_long_name': 'AAC (Advanced Audio Coding)',
'codec_name': 'aac',
'codec_tag': '0x6134706d',
'codec_tag_string': 'mp4a',
'codec_type': 'audio',
'disposition': {'attached_pic': 0,
'captions': 0,
'clean_effects': 0,
'comment': 0,
'default': 1,
'dependent': 0,
'descriptions': 0,
'dub': 0,
'forced': 0,
'hearing_impaired': 0,
'karaoke': 0,
'lyrics': 0,
'metadata': 0,
'non_diegetic': 0,
'original': 0,
'still_image': 0,
'timed_thumbnails': 0,
'visual_impaired': 0},
'duration': '116.959002',
'duration_ts': 5157892,
'extradata_size': 5,
'id': '0x2',
'index': 1,
'initial_padding': 0,
'nb_frames': '5038',
'profile': 'LC',
'r_frame_rate': '0/0',
'sample_fmt': 'fltp',
'sample_rate': '44100',
'start_pts': 0,
'start_time': '0.000000',
'tags': {'handler_name': 'SoundHandler',
'language': 'und',
'vendor_id': '[0][0][0][0]'},
'time_base': '1/44100'}]}

Python video details

{'format': {'bit_rate': '763186',
'duration': '124.150000',
'filename': '/Users/vikramkumar/Downloads/472_WEB (1).mp4',
'format_long_name': 'QuickTime / MOV',
'format_name': 'mov,mp4,m4a,3gp,3g2,mj2',
'nb_programs': 0,
'nb_stream_groups': 0,
'nb_streams': 2,
'probe_score': 100,
'size': '11843694',
'start_time': '0.000000',
'tags': {'compatible_brands': 'isomiso2avc1mp41',
'encoder': 'Lavf58.29.100',
'major_brand': 'isom',
'minor_version': '512'}},
'streams': [{'avg_frame_rate': '24/1',
'bit_rate': '632570',
'bits_per_raw_sample': '8',
'chroma_location': 'left',
'closed_captions': 0,
'codec_long_name': 'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10',
'codec_name': 'h264',
'codec_tag': '0x31637661',
'codec_tag_string': 'avc1',
'codec_type': 'video',
'coded_height': 720,
'coded_width': 1280,
'disposition': {'attached_pic': 0,
'captions': 0,
'clean_effects': 0,
'comment': 0,
'default': 1,
'dependent': 0,
'descriptions': 0,
'dub': 0,
'forced': 0,
'hearing_impaired': 0,
'karaoke': 0,
'lyrics': 0,
'metadata': 0,
'non_diegetic': 0,
'original': 0,
'still_image': 0,
'timed_thumbnails': 0,
'visual_impaired': 0},
'duration': '123.166667',
'duration_ts': 1513472,
'extradata_size': 42,
'field_order': 'progressive',
'film_grain': 0,
'has_b_frames': 2,
'height': 720,
'id': '0x1',
'index': 0,
'is_avc': 'true',
'level': 31,
'nal_length_size': '4',
'nb_frames': '2956',
'pix_fmt': 'yuv420p',
'profile': 'High',
'r_frame_rate': '24/1',
'refs': 1,
'start_pts': 0,
'start_time': '0.000000',
'tags': {'handler_name': 'VideoHandler',
'language': 'und',
'vendor_id': '[0][0][0][0]'},
'time_base': '1/12288',
'width': 1280},
{'avg_frame_rate': '0/0',
'bit_rate': '129388',
'bits_per_sample': 0,
'channel_layout': 'stereo',
'channels': 2,
'codec_long_name': 'AAC (Advanced Audio Coding)',
'codec_name': 'aac',
'codec_tag': '0x6134706d',
'codec_tag_string': 'mp4a',
'codec_type': 'audio',
'disposition': {'attached_pic': 0,
'captions': 0,
'clean_effects': 0,
'comment': 0,
'default': 1,
'dependent': 0,
'descriptions': 0,
'dub': 0,
'forced': 0,
'hearing_impaired': 0,
'karaoke': 0,
'lyrics': 0,
'metadata': 0,
'non_diegetic': 0,
'original': 0,
'still_image': 0,
'timed_thumbnails': 0,
'visual_impaired': 0},
'duration': '124.150000',
'duration_ts': 5475015,
'extradata_size': 5,
'id': '0x2',
'index': 1,
'initial_padding': 0,
'nb_frames': '5348',
'profile': 'LC',
'r_frame_rate': '0/0',
'sample_fmt': 'fltp',
'sample_rate': '44100',
'start_pts': 0,
'start_time': '0.000000',
'tags': {'handler_name': 'SoundHandler',
'language': 'und',
'vendor_id': '[0][0][0][0]'},
'time_base': '1/44100'}]}

Samuel Audet

unread,

Sep 18, 2024, 8:08:52 PM9/18/24

to jav...@googlegroups.com, Vikram Jain

You need to check the encoder settings. Those don't appear in the
encoded file. In the case of libx264, they get displayed on the console
when requesting debug information

Reply all

Reply to author

Forward