I have a similar setup using xvfb, but I have not had much issues with clicking on elements. I just have to ensure they are there and visible (so I implemented a waitForVisible(type, selector) method. My click wrapper method calls waitForVisible, and all is good.
Occasionally, I will come across issues with clicking on elements that appear when another element is hovered over. Sometimes the browser shows the elements, but the DOM doesn't register them as being visible. Thus, I had to write javascript injection that forces the elements to register as visible while being hovered over.
Hopefully, you are not running into one of these type of issues. I would recommend that in your teardown you take a screenshot if there is a failure so you can see what the browser's state was at that time to ensure the elements you are attempting to click are indeed visible.
Brian