After some time's simulation, I'm almost sure the singularity error appears due to numerics. The error happens when truncation photon number N is large and excitation weak. Both situations imply essentially empty occupation of high rank Fock states, then I guess the too small occupation can't be distinguished by computer from exact zero and somehow leads to singularity.
If the above reasoning makes sense, the appearance of the error simply means the truncation has been made at unnecessarily high photon numbers. Lowering N will solve the problem.
Cheers!