There was some mention of doing regression tests with a buildbot or something. Buildbot can be a little difficult, but I've heard good things about Hudson:
http://wiki.hudson-ci.org/display/HUDSON/Meet+Hudson -- this is from Python people, so don't let the Java put you off, it sounds like it's mostly a black box (in a good way). I haven't tried it myself yet, but the that page does make it sound really easy.