I had the same issue, and I think it is do with linking to dynamic lib.
If you look at the dependency of btk modules, by "otool -L btkGetAnalogs.mexmaci64”(you have to be in that folder)
Then you probably see @loader_path rather than @rpath. So you have to change this to @rpath by
"install_name_tool -change "@loader_path/libmex.dylib @rpath/libmex.dylib [name of btk module].
But as some modules are linked to each other, all these files need to be changed which is an effort and I have not yet figured out a way of doing this.
This is because the change of system in Mac OS and btk was built quite a long time ago.
Hope this helps, and if you somehow managed to find out a way to change the @loader_path to @rpath for all btk modules please let me know!