it appears that the camera you get is very rough, which is quite expected for a first pass for KH4B.
I am not sure what you mean by weird projection error. The lon and lat appear legit. The height is 163 km above ground, which is wild. The median pixel reprojection error of 72 pixels is not bad at all at this stage as we do a lot of guessing.
I guess you can check first with the DEM, if the heights are in meters. Then can see if there is any mix up in the order of lon and lat and order of corners. Then you can see if modifying the focal length for the camera (say by 0.5x - 2x or in smaller increments) makes any difference.
You can also try mapprojecting the image with the produced camera onto the DEM. If you at least get the right orientation, that means the corners are good. Then you can play more with focal length or optical center. Otherwise, there can be a mix-up with lon and lat or order, especially if you get a flipped result.
It is advised that it is quite tricky to get this right. I think after following the latest doc mentioned above you may end up with a plausible solution, but going from there to a very good solution will not be easy. I also don't know now how differnt KH-4B is different from panoramic KH-9 which we tested a lot more.
Historical images require a lot of guesswork, have warping, mechanical camera model, etc, which can be quite complicated.