Ahmed,
Now that I understand that you are using a stereo camera, then the right thing to do is to model the camera differently instead of trying to do it as two cameras with their own extrinsics.
I recommend modeling the two cameras's extrinsics as two rigid transformations. The first one say, models the rotation and translation to say the center of your stereo camera rig. And then based on how the stereo rig is structured, you can have another rigid transformation, which goes from the center of the rig to each camera but it only has say the distance from the center of the stereo rig as a parameter that can be changed.
So your parameter block then will consist of a rotation, a translation and a scalar that represents the distance from the center of the rig to each of the two stereo cameras.
Sameer