After wresting with it in a few days, I figured out this - all turns out to be not up-to-date brew and conda packages. you may have a problem to use Caffe in python or python without upgrading brew packages and conda packages.
I had problem with using opencv and caffe with python in upgraded El Capitan; 1) segmentation fault in ipython, 2) can’t compile caffe 3) brew’ed opencv didn’t allow me to compile my opencv C++ projects.
All turns out to be not up-to-date brew and conda packages. When you update the OS, please make sure update brew packages and conda packages too for making sure all things work fine. Especially to use Caffe in ipython, please make sure python and ipython in your conda should be up-to-date and install only one libpng library either in conda or with brew.
Hope it help!