This is usually caused by some version incompatibility between the requested dependencies (for librosa) and what's currently on your system.
Conda uses a satisfiability solver to determine a mutually consistent set of installed versions of packages. In theory, that's an np-hard problem and can take quite a while to compute; in practice, it's usually pretty easy, and you don't notice it.
In my experience, when a conda install takes a long time saying "solving environment", it's often because it needs to wind all the way back to the python version and then rebuild the entire installed package collection. For instance, I have a python 3.7 environment for which there's no tensorflow package available. If I try to install a conda package that requires tensorflow, conda will basically try to scrap the entire environment and rebuild it as a 3.6 env. That's an extreme case, but it might give you some clue as to what's going on.
In your specific case, I would suspect either numba or scikit-learn as the culprit. You can see the full list of dependencies in our conda recipe here:
https://github.com/librosa/librosa-feedstock/blob/master/recipe/meta.yaml#L26-L37 and check against what's currently installed in your environment. Perhaps try installing one of those at a time until you hit the "solving environment" issue again, and that will tell you which package is to blame.