Cannot PyPlot / Matplotlib broken (warnings about font cache?)

[Daniel Carrera]

Hi everyone,

I am getting some cryptic errors when I try to load PyPlot. It seems to be a problem on the Python end, but I'm not sure. I tried to use the "Conda" package so that Julia would have its own (hopefully not-broken) copy of Python, but that didn't help as PyPlot is still looking at the system-wide install of Python. Here are the messages in full (and some comments after ward). I decided to start from a completely clean install of Julia 0.5 (no packages) to make sure that there was no problem there:

-----------------------------------

$ mv ~/.julia/v0.5 ~/.julia/old-v0.5

$ julia

               _

   _       _ _(_)_     |  A fresh approach to technical computing

  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org

   _ _   _| |_  __ _   |  Type "?help" for help.

  | | | | | | |/ _` |  |

  | | |_| | | | (_| |  |  Version 0.5.0 (2016-09-19 18:14 UTC)

 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release

|__/                   |  x86_64-pc-linux-gnu

julia> Pkg.add("Conda")

INFO: Initializing package repository /home/daniel/.julia/v0.5

INFO: Cloning METADATA from https://github.com/JuliaLang/METADATA.jl

INFO: Installing BinDeps v0.4.5

INFO: Installing Compat v0.9.3

INFO: Installing Conda v0.3.2

INFO: Installing JSON v0.8.0

INFO: Installing SHA v0.2.1

INFO: Installing URIParser v0.1.6

INFO: Package database updated

julia> Pkg.add("PyPlot")

INFO: Installing ColorTypes v0.2.12

INFO: Installing Colors v0.6.9

INFO: Installing FixedPointNumbers v0.2.1

INFO: Installing LaTeXStrings v0.2.0

INFO: Installing MacroTools v0.3.2

INFO: Installing PyCall v1.7.2

INFO: Installing PyPlot v2.2.4

INFO: Installing Reexport v0.0.3

INFO: Building PyCall

INFO: PyCall is using python (Python 2.7.12) at /usr/bin/python, libpython = libpython2.7

INFO: Package database updated

julia> using PyPlot

INFO: Recompiling stale cache file /home/daniel/.julia/lib/v0.5/PyPlot.ji for module PyPlot.

/usr/lib/python2.7/dist-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.

  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')

/usr/lib/python2.7/dist-packages/matplotlib/__init__.py:1352: UserWarning:  This call to matplotlib.use() has no effect

because the backend has already been chosen;

matplotlib.use() must be called *before* pylab, matplotlib.pyplot,

or matplotlib.backends is imported for the first time.

  warnings.warn(_use_error_msg)

/usr/lib/python2.7/dist-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.

  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')

/usr/lib/python2.7/dist-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.

  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')

/usr/lib/python2.7/dist-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.

  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')

/usr/lib/python2.7/dist-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.

  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')

WARNING: No working GUI backend found for matplotlib.

/usr/lib/python2.7/dist-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.

  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')

ERROR: InitError: PyError (:PyImport_ImportModule) <type 'exceptions.IOError'>

IOError(13, 'Permission denied')

  File "/usr/lib/python2.7/dist-packages/matplotlib/pyplot.py", line 29, in <module>

    import matplotlib.colorbar

  File "/usr/lib/python2.7/dist-packages/matplotlib/colorbar.py", line 34, in <module>

    import matplotlib.collections as collections

  File "/usr/lib/python2.7/dist-packages/matplotlib/collections.py", line 27, in <module>

    import matplotlib.backend_bases as backend_bases

  File "/usr/lib/python2.7/dist-packages/matplotlib/backend_bases.py", line 62, in <module>

    import matplotlib.textpath as textpath

  File "/usr/lib/python2.7/dist-packages/matplotlib/textpath.py", line 15, in <module>

    import matplotlib.font_manager as font_manager

  File "/usr/lib/python2.7/dist-packages/matplotlib/font_manager.py", line 1421, in <module>

    _rebuild()

  File "/usr/lib/python2.7/dist-packages/matplotlib/font_manager.py", line 1406, in _rebuild

    fontManager = FontManager()

  File "/usr/lib/python2.7/dist-packages/matplotlib/font_manager.py", line 1059, in __init__

    self.ttflist = createFontList(self.ttffiles)

  File "/usr/lib/python2.7/dist-packages/matplotlib/font_manager.py", line 581, in createFontList

    font = ft2font.FT2Font(fpath)

 in pyerr_check at /home/daniel/.julia/v0.5/PyCall/src/exception.jl:56 [inlined]

 in pyerr_check at /home/daniel/.julia/v0.5/PyCall/src/exception.jl:61 [inlined]

 in macro expansion at /home/daniel/.julia/v0.5/PyCall/src/exception.jl:81 [inlined]

 in pyimport(::String) at /home/daniel/.julia/v0.5/PyCall/src/PyCall.jl:387

 in __init__() at /home/daniel/.julia/v0.5/PyPlot/src/PyPlot.jl:235

 in _include_from_serialized(::String) at ./loading.jl:150

 in _require_from_serialized(::Int64, ::Symbol, ::String, ::Bool) at ./loading.jl:187

 in require(::Symbol) at ./loading.jl:394

during initialization of module PyPlot

-----------------------------------

Ok. So first of all, PyPlot is grabbing the system-wide installation of Python. I don't know how to tell PyPlot to look at the one that was installed with the Conda package. Second, the errors are saying something about fonts, but I can't figure out what they are saying. I am also confused by the "permission denied" error:

ERROR: InitError: PyError (:PyImport_ImportModule) <type 'exceptions.IOError'>

IOError(13, 'Permission denied')

All the files seem to be present and have sensible permissions:

$ ls -l /usr/lib/python2.7/dist-packages/matplotlib/pyplot.py

-rw-r--r-- 1 root root 123661 Mar 21  2016 /usr/lib/python2.7/dist-packages/matplotlib/pyplot.py

$ ls -l /usr/lib/python2.7/dist-packages/matplotlib/font_manager.py

-rw-r--r-- 1 root root 47824 Jan 10  2016 /usr/lib/python2.7/dist-packages/matplotlib/font_manager.py

So... I'm stuck. I would appreciate any help you might offer.

Cheers,

Daniel.

[Isaiah Norton]

Ok. So first of all, PyPlot is grabbing the system-wide installation of Python. I don't know how to tell PyPlot to look at the one that was installed with the Conda package.

 

https://github.com/JuliaPy/PyCall.jl#specifying-the-python-version

To force Julia to use its own Python distribution, via Conda, rather than whatever is installed on your system, simply set ENV["PYTHON"] to the empty string "" as described above.

Second, the errors are saying something about fonts, but I can't figure out what they are saying. I am also confused by the "permission denied" error:

Hard to say. Try checking the permissions of the files listed by `fc-list`... Maybe something was installed with `sudo` that should not have been. 

[Steven G. Johnson]

For the font-cache, see

    https://github.com/matplotlib/matplotlib/issues/5836

...there is some ~/.cache/matplotlib directory that you can delete to get rid of this.

To tell PyPlot to use the Conda Python, you need to set the PYTHON environment variable and re-build PyCall:

    ENV["PYTHON"]=""

    Pkg.build("PyCall")

[Daniel Carrera]

On Wednesday, 2 November 2016 15:20:07 UTC+1, Isaiah wrote:

Ok. So first of all, PyPlot is grabbing the system-wide installation of Python. I don't know how to tell PyPlot to look at the one that was installed with the Conda package.

 

https://github.com/JuliaPy/PyCall.jl#specifying-the-python-version

To force Julia to use its own Python distribution, via Conda, rather than whatever is installed on your system, simply set ENV["PYTHON"] to the empty string "" as described above.

Thanks! I've realized that Conda.jl doesn't actually install Python. But I could use Anaconda to install Python in /opt and point Julia to that version.

 

Second, the errors are saying something about fonts, but I can't figure out what they are saying. I am also confused by the "permission denied" error:

Hard to say. Try checking the permissions of the files listed by `fc-list`... Maybe something was installed with `sudo` that should not have been. 

That wouldn't surprise me. The other day I was messing around with pip3 when trying to setup Jupyter and the Octave kernel, and I did run some commands with sudo. Thanks.

[Daniel Carrera]

Thanks!  Ok... getting close, but not quite there:

Deleting ~/.cache/matplotlib and even ~/.cache/fontconfig didn't work for me, but I figured out how to use Conda.jl to install matplotlib separately. So I did this:

$ sudo apt-get remove python-matplotlib

$ julia

julia> Pkg.add("Conda")

julia> using Conda

julia> Conda.add("matplotlib")

julia> ENV["PYTHON"] = "/home/daniel/.julia/v0.5/Conda/deps/usr/bin/python"

julia> Pkg.add("PyPlot")

julia> using PyPlot

Up to this point there are no errors!  But this is what happens next:

julia> plot( [1,2,3], [4,5,6] )

ERROR: ArgumentError: haskey of NULL PyObject

 in haskey(::PyCall.PyObject, ::String) at /home/daniel/.julia/v0.5/PyCall/src/PyCall.jl:282

 in #plot#85(::Array{Any,1}, ::Function, ::Array{Int64,1}, ::Vararg{Array{Int64,1},N}) at /home/daniel/.julia/v0.5/PyPlot/src/PyPlot.jl:396

 in plot(::Array{Int64,1}, ::Vararg{Array{Int64,1},N}) at /home/daniel/.julia/v0.5/PyPlot/src/PyPlot.jl:396

I partially understand the error. I'll experiment a bit more to see if I can resolve it, but I thought you'd like to know about it.

Cheers,

Daniel.

 so I think I'll switch to Conda Python. I figured out how to use Conda.jl to get a separate matplotlib install

[Daniel Carrera]

This is depressing... Installing a new matplotlib did not fix the problem at all. After I exited and restarted Julia the old error came right back, even with the new Python install:

------------------

Orion ~ % julia        

               _

   _       _ _(_)_     |  A fresh approach to technical computing

  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org

   _ _   _| |_  __ _   |  Type "?help" for help.

  | | | | | | |/ _` |  |

  | | |_| | | | (_| |  |  Version 0.5.0 (2016-09-19 18:14 UTC)

 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release

|__/                   |  x86_64-pc-linux-gnu

julia> using PyPlot

/home/daniel/.julia/v0.5/Conda/deps/usr/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.

  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')

WARNING: No working GUI backend found for matplotlib.

/home/daniel/.julia/v0.5/Conda/deps/usr/lib/python2.7/site-packages/matplotlib/__init__.py:1357: UserWarning:  This call to matplotlib.use() has no effect

because the backend has already been chosen;

matplotlib.use() must be called *before* pylab, matplotlib.pyplot,

or matplotlib.backends is imported for the first time.

  warnings.warn(_use_error_msg)

/home/daniel/.julia/v0.5/Conda/deps/usr/lib/python2.7/site-packages/matplotlib/font_manager.py:273: UserWarning: Matplotlib is building the font cache using fc-list. This may take a moment.

  warnings.warn('Matplotlib is building the font cache using fc-list. This may take a moment.')

ERROR: InitError: PyError (:PyImport_ImportModule) <type 'exceptions.IOError'>

IOError(13, 'Permission denied')

  File "/home/daniel/.julia/v0.5/Conda/deps/usr/lib/python2.7/site-packages/matplotlib/pyplot.py", line 29, in <module>

    import matplotlib.colorbar

  File "/home/daniel/.julia/v0.5/Conda/deps/usr/lib/python2.7/site-packages/matplotlib/colorbar.py", line 34, in <module>

    import matplotlib.collections as collections

  File "/home/daniel/.julia/v0.5/Conda/deps/usr/lib/python2.7/site-packages/matplotlib/collections.py", line 27, in <module>

    import matplotlib.backend_bases as backend_bases

  File "/home/daniel/.julia/v0.5/Conda/deps/usr/lib/python2.7/site-packages/matplotlib/backend_bases.py", line 62, in <module>

    import matplotlib.textpath as textpath

  File "/home/daniel/.julia/v0.5/Conda/deps/usr/lib/python2.7/site-packages/matplotlib/textpath.py", line 15, in <module>

    import matplotlib.font_manager as font_manager

  File "/home/daniel/.julia/v0.5/Conda/deps/usr/lib/python2.7/site-packages/matplotlib/font_manager.py", line 1421, in <module>

    _rebuild()

  File "/home/daniel/.julia/v0.5/Conda/deps/usr/lib/python2.7/site-packages/matplotlib/font_manager.py", line 1406, in _rebuild

    fontManager = FontManager()

  File "/home/daniel/.julia/v0.5/Conda/deps/usr/lib/python2.7/site-packages/matplotlib/font_manager.py", line 1059, in __init__

    self.ttflist = createFontList(self.ttffiles)

  File "/home/daniel/.julia/v0.5/Conda/deps/usr/lib/python2.7/site-packages/matplotlib/font_manager.py", line 581, in createFontList

    font = ft2font.FT2Font(fpath)

 in pyerr_check at /home/daniel/.julia/v0.5/PyCall/src/exception.jl:56 [inlined]

 in pyerr_check at /home/daniel/.julia/v0.5/PyCall/src/exception.jl:61 [inlined]

 in macro expansion at /home/daniel/.julia/v0.5/PyCall/src/exception.jl:81 [inlined]

 in pyimport(::String) at /home/daniel/.julia/v0.5/PyCall/src/PyCall.jl:387

 in __init__() at /home/daniel/.julia/v0.5/PyPlot/src/PyPlot.jl:235

 in _include_from_serialized(::String) at ./loading.jl:150

 in _require_from_serialized(::Int64, ::Symbol, ::String, ::Bool) at ./loading.jl:187

 in _require_search_from_serialized(::Int64, ::Symbol, ::String, ::Bool) at ./loading.jl:217

 in require(::Symbol) at ./loading.jl:371

during initialization of module PyPlot

------------------

Notice that it's now looking at my brand-new locally installed Python in ~/.julia/v0.5/Conda/deps

What's perhaps more surprising is that I still see an IO error and a "Permission denied". How is that possible? I'm pretty sure I never typed "sudo" when I did any of this. Incidentally, I already tried removing the ~/.cache directory and it didn't help.

Cheers,

Daniel.

[Daniel Carrera]

Hi guys,

I am really sorry for the spam. I just want to say that I am cautiously optimistic that the problem has been solved. I started Python to see the error from within Python, and this is what I saw:

>>> import matplotlib.pyplot as p

... blah blah blah...

IOError: [Errno 13] Permission denied: u'/usr/share/fonts/opentype/FrutigerLTStd/FrutigerLTStd-Light.otf'

So the IO error was referring to a font file!!! I didn't see this in the error output in Julia and I foolishly thought that it was complaining about one of the .py files. So I checked the permissions of the Frutiger font files and they were indeed wrong. I fixed the permissions and now PyPlot seems to work as before. I don't know why the permissions were wrong, but I guess it no longer matters.

Here is an idea for the future: Is it possible to get PyCall to print the rest of Python's error message so that dummies like me aren't led astray... It's weird that most of the error was printed but the path to the file was not.

Cheers,

Daniel.

[Steven G. Johnson]

On Wednesday, November 2, 2016 at 10:39:58 AM UTC-4, Daniel Carrera wrote:

On Wednesday, 2 November 2016 15:20:07 UTC+1, Isaiah wrote:

Thanks! I've realized that Conda.jl doesn't actually install Python. But I could use Anaconda to install Python in /opt and point Julia to that version.

No, Conda.jl does actually install its own Python. 

[Steven G. Johnson]

On Wednesday, November 2, 2016 at 11:35:20 AM UTC-4, Daniel Carrera wrote:

Here is an idea for the future: Is it possible to get PyCall to print the rest of Python's error message so that dummies like me aren't led astray... It's weird that most of the error was printed but the path to the file was not.

Could you file an issue, maybe with test code via pyeval or something, that illustrates the problem?   Maybe there is a separate Python API function that I should be using to display exception output.