Wow, thank for the pointers here.
I was able to test it and it runs very quickly and does seem to cut down the size of the output.
Unfortunately the output is a bit corrupted.
I also tried to add audio track and it does seem to crop but still have a slightly corrupt output.
ffmpeg is reporting these kind of errors:
[h264 @ 0xa0e2970]Missing reference picture
[h264 @ 0xa0e2970]decode_slice_header error
[h264 @ 0xa0e2970]Missing reference picture
[h264 @ 0xa0e2970]decode_slice_header error
[h264 @ 0xa0e2970]Missing reference picture
[h264 @ 0xa0e2970]decode_slice_header error
[h264 @ 0xa0e2970]Missing reference picture
[h264 @ 0xa0e2970]decode_slice_header error
[h264 @ 0xa0e2970]Missing reference picture
[h264 @ 0xa0e2970]decode_slice_header error
My hope was to end up with a valid output file with audio included.