A couple of years ago I found that the problem mentioned here (distro
packages are optimised for pessimal machines) made the default OpenCV
packages several times slower than a re-compiled version. That was on
32 bit x86. When I switched to x86_64, the difference disappeared.

The reason is of course that the x86_64 baseline includes SSE2 and the
compiler can assume a fairly modern CPU structure with caches and
pipelines and so forth. The 32 bit default has to expect to run, if
not on a 386 sans floating point coprocessor, at least on a mid-90s
MMX-less Pentium.

My bet is recompiling will give you some improvement on x86_64, but it
will be in the range of a few percent rather than a few times.

Douglas