I'm using vncserver but it should function like xvfb. I did once have
the issue where I got this exact behavior whenever a dialog appaered
in a location such that the JButton I was trying to click() as outside
of the virtual screen's resolution. When I changed the resolution to
1600x1200, that particular problem went away. The issue I got was
always timeout, never component not found or multiple components
found.
Specifically, I am using the following in tomcat's catalina.sh
DISPLAY=:2
vncserver -kill $DISPLAY
vncserver $DISPLAY -geometry 1600x1200
...
-Djava.awt.headless=true