I think the problem here is coming from numba, which we use to compile some functions at load-time for efficiency. Numba will want to save the compiled objects in the same place as the module, but if you don't have write permission to that location (eg because it's an administrator-installed package), that would cause this error.
Something like:
import os
import IPython
os.environ['NUMBA_CACHE_DIR'] = IPython.paths.get_ipython_cache_dir()
import librosa
ought to do the trick.