Try the latest ODE version or SVN revision and compile with OPCODE 'new' or GIMPACT as trimesh library.
I've found GIMPACT to be very stable and efficient for trimesh-trimesh collisions. OPCODE 'new' is a
big improvement, but it still has the occasional jumps you describe in my own simulation, and worse
performance than GIMPACT or at least in my simulation. But that may vary between
applications/geometry complexity, so better try both and see for yourself.
/Martijn
Also, try posting your collision/contact generating code, rather than
the body set-up code. It's contact generation that matters. For good
trimesh/trimesh intersections, you should extract as many contacts as
you can (say, pass in an array of 100 items), and then filter them down
to a set of 10 or so using various contact manifold filter methods
(point thinning, coincidence merging, convex hull, etc).
Sincerely,
jw