Do you have JIT enabled or not? There's a lot of potential for JIT, but it is very experimental for now, and not so good for Rails yet. Otherwise it sounds like a garbage collection (GC) issue.
I would recommend reading this:
https://pragprog.com/book/adrpo/ruby-performance-optimization
There's some good stuff there on how to bench and tune things (albeit, slightly out of date).
A few simple examples is to use direct SQL queries when possible instead of full model loads, and using the BANG (!) methods when possible (in other words, using, say "reject!" instead of "myvar = myvar.reject ...")
Hope this helps.
Phil