VDPAU/VAAPI h.264 NAL(U) to VSTRM for Encoding

[Trevor Rudolph]

Would it be possible to replace the x264 calls with native hardware accelerated libraries to free up CPU time. Yes the baseline profile doesn't take up many resources but it would be nice to have hardware back-ends too. I was planning on approaching this by replacing the calls and doing some extra work on the NAL packets. However I fear that the "planar intra prediction" feature is implemented in the hardware encoder depending on the company (Intel/NVIDIA/AMD/OMX) and not on VAAPI initialization. If anybody has thoughts or research on this topic I would appreciate some insight. 

[Trevor Rudolph]

It seems that the VAAPI driver would need to be modified to produce a NAL unit every 6 macro-blocks instead of it's normal settings. I also think the NAL header needs to be removed? That produces a NALU slice I believe. Lastly the graphics driver (in my case intel-drive) would need to be modified to stub intra-planar prediction subroutines, that code would refer to the mb_intra_struct structure. It's a bit more complicated than just editing x264 especially when you would probably need to edit the encoding driver based on the system you're using. 

[Trevor Rudolph]

https://01.org/sites/default/files/documentation/intel_os_gfx_prm_vol8_-_media_vdbox_0.pdf good guide on where to find macro-block modes, I believe you only need to disable 16x16 intra prediction (according to drc-x264 patch).