I recommend using distsamp instead. Yes gdistsamp allows you to incorporate the repeated visits explicitly, but in practice I don't think it's achieving what you hoped. First, the gdistsamp formula is in the order 1) lambda (abundance) 2) phi (availability) and 3) p (detection) and you have put the observer covariate in the detection function, when according to your explanation I would think it should be a factor in availability?
In practice, unless there is a significant reason why you think availability would change between visits (e.g. temporary or annual migration) then gdistsamp I think has trouble attributing non-detections among the p and phi models. I would expect also that your lack of covariates on lambda has led to difficulty assigning variable abundance values among sites, try changing observer to the availability model and adding a few covariates to abundance.
If you don't think there are major shifts in availability between visits, and even if there is a large difference in skill level among your observers, then if every site got three visits and three observers then all have roughly identical effort assuming you used standardized field sampling protocols. Therefore gdistsamp is probably not necessary and I'm guessing will continue to give you unsatisfactory results.