Hi Justin,
With Monte-Carlo, I would just reduce max.mem. If you don't OOM immediately, it should iterate until it can't any more and then return what it has.
If you are using Riemann, then max.mem is not used - the memory usage shouldn't be that high for range-resident individuals.
Another thing that can help is to subset the rasters down to an area around the extent of the home-range before running rsf.fit/rsf.select.
Best,
Chris