upgrading matplotlib and ignoring sage's matplotlibrc

[Jason Grout]

I just finished upgrading the matplotlib spkg to the newest version.
See http://trac.sagemath.org/sage_trac/ticket/4774

This version of matplotlib deprecates some of the constructs found in
Sage's matplotlibrc (which is located in $SAGE_ROOT and automatically
copied to $DOT_SAGE if needed every time sage starts up). The result is
a bunch of deprecation warnings every time Sage starts up (when
matplotlib loads Sage's matplotlibrc file). In trying to figure out
what to do about this with several other developers, one option that
came up was just throwing away/ignoring the special Sage matplotlibrc
and using the normal, standard defaults for matplotlibrc (including the
standard location for a customized matplotlibrc).

In investigating things more deeply, there were only a few real changes
we made to the default behavior of matplotlib. IIRC, a few font choices
were reordered, legends were changed to display a bit more of the
function, and the dpi of saved images was bumped up from 80 dpi to 100
dpi (but this should be set when Sage saves an image anyway, so I don't
know that this changes anything).

So here's a proposal: Should Sage stop distributing a custom
matplotlibrc, and ignore matplotlibrc files that already exist in the
$DOT_SAGE directories?

Note that if people really want to customize the matplotlib settings,
they can always use the standard location for matplotlibrc (i.e.,
~/.matplotlib/matplotlibrc, I think). This will clean code out of the
bin repository and reduce startup time for sage as well. Patches which
do this are posted to #4774.

I vote yes, provided some sort of note is made in the release notes
about the ignored matplotlibrc file.

Thanks,

Jason

[Nick Alexander]

> So here's a proposal: Should Sage stop distributing a custom
> matplotlibrc, and ignore matplotlibrc files that already exist in the
> $DOT_SAGE directories?

I am +0, since I don't plot, but one could distribute a custom
matplotlibrc saying that it is no longer used and pointing to likely
locations to check.

Nick

[William Stein]

I vote no, since I created the $DOT_SAGE/matplotlib directory
*precisely* because of problems that happen if you were to do what you
describe above. For starters, people often also used a system-wide
Python with their own version of matplotlib -- then end result was
that if they tried to switch back and forth between sage and
python/ipython/matplotlib, they would get tons of deprecation
warnings, since the systemwide version of matplotlib is often
different than the sage version.

Second, how will what you suggest solve any problems? All you do is
move the problem from $DOT_SAGE/matplotlib/matplotlibc to
$HOME/.matplotlibrc. It's exactly the same problem. You just
temporarily put it off for a while.

I *wish* matplotlib would replace their stupid deprecation warnings by
something that just updates the matplotlibrc file, and say makes a
copy of the old one. Is there any way we could catch the warnings and
if they occur move $DOT_SAGE/matplotlib/matplotlibrc to another file,
then put a new matplotlibrc in place and print a message that this
just happened? You could do that by making slightly patching
matplotlib as well. I think matplotlib's behavior of emitting
warnings but doing nothing helpful to resolve them is just obnoxious.

-- william

[Ondrej Certik]

Maybe matplotlib developers would accept a patch fixing this.

Ondrej

[Jason Grout]

Yes, since they use deprecation warnings, we should be able to do this
with the standard deprecation warning functionality in python.

Jason

[William Stein]

I don't think they consider it a bug. I think they view it as a
feature. I should probably ask John though.

-- William

[Jason Grout]

John Hunter wrote:

> On Fri, Jan 30, 2009 at 9:30 AM, Ondrej Certik <ond...@certik.cz> wrote:
>
>>> I *wish* matplotlib would replace their stupid deprecation warnings by
>>> something that just updates the matplotlibrc file, and say makes a
>>> copy of the old one. Is there any way we could catch the warnings and
>>> if they occur move $DOT_SAGE/matplotlib/matplotlibrc to another file,
>>> then put a new matplotlibrc in place and print a message that this
>>> just happened? You could do that by making slightly patching
>>> matplotlib as well. I think matplotlib's behavior of emitting
>>> warnings but doing nothing helpful to resolve them is just obnoxious.
>> Maybe matplotlib developers would accept a patch fixing this.
>

> To address this problem, we went to an all commented out rc file.
> That way, when people change their mpl version, they don't get
> deprecation warnings. Only people who make specific changes to their
> rc file will get deprecation warnings. Those people will know what rc
> is and how to change it. I'm disinclined to overwrite files people
> have changed -- they may be running multiple versions of mpl under
> different scenarios, and each version would be competing with one
> another to overwrite the file. mpl has a general philospohy of not
> trying to be too helpful -- we want to make it easy for users to
> customize, we don't want to customize it for them.
>
> The verbose deprecation warnings are in my opinion mostly a solved
> problem: get a new rc file which is all commented out. Just change
> the things you want to change, and you will get very few warnings
> going forward. When you get them, fix the problem and you will get no
> more warnings.

Okay, how about this strategy proposal instead:

1. Add a warnings handler which traps the Matplotlib deprecation
warning. This handler will compare the matplotlibrc with the one that
was distributed in the last release (maybe by checking a hash value).
If it is identical (i.e., the user hasn't modified the matplotlibrc),
then replace the matplotlibrc with a commented-out version. This is a
bit tricky, since the sage default matplotlibrc is not under version
control (it's in $SAGE_ROOT), so we don't know what old copies looked
like. (hehe...yet another reason to really like mabshoff for the
library of old builds). If the file is not identical to the sage
default matplotlibrc, then throw a warning with a helpful error message
about updating the matplotlibrc (and where to find the file).

2. Put the sage default matplotlibrc under version control inside the
matplotlib spkg and copy it to someplace in $SAGE_ROOT/local for
installation. Delete the matplotlibrc in $SAGE_ROOT.

What do people think?

Jason

[Carl Witty]

On Thu, Jan 29, 2009 at 5:49 PM, William Stein <wst...@gmail.com> wrote:
>

I don't understand this. As I understand it, the proposal is to not
do anything with matplotlibrc (don't ship any "Sage" version at all),
so almost everybody will not have a matplotlibrc. The problem is
fixed permanently for those people. We're left with the people who
actually read the matplotlib documentation, notice the existence of
matplotlibrc, and create their own matplotlibrc file. They may have a
problem if they switch back and forth, but since they personally
created the matplotlibrc file that's causing the problem, they're also
well-placed to fix it.

Carl

[William Stein]

IIRC matplotlib *creates* or copies a matplotlibrc file into place the
first time it is run if one isn't there. This is the case at least
with ipython, and I think with matplotlibrc. Everything I said above
was under that assumption. If it isn't true for matplotlibrc than I
take it back.

OK, as a test I deleted the matplotlibrc file I have, started up sage,
and drew a plot. No matplotlibrc is created.

So never mind what I said :-)

[Jason Grout]

Well, there is something to be said for what you said. We ship our own
version of matplotlib, which mostly likely is different than the system
install. That means that their perfectly good default matplotlibrc file
might cause deprecation warnings for us. For that reason, I think we
should still set the MATPLOTLIBRC environment variable to point to a
location within $DOT_SAGE.

For that reason, how about yet another proposal:

Add a warnings handler which traps the Matplotlib deprecation
warning. This handler will compare the matplotlibrc with the one that
was distributed in the last release (maybe by checking a hash value).
If it is identical (i.e., the user hasn't modified the matplotlibrc),

then *delete* the matplotlibrc. This is a

bit tricky, since the sage default matplotlibrc is not under version

control (it's in $SAGE_ROOT), so we don't know what older copies looked

like. (hehe...yet another reason to really like mabshoff for the

library of old builds). If the file is not identical to the current

sage default matplotlibrc, then throw a warning with a helpful error
message about updating the matplotlibrc (and where to find the file).

Jason

[Jason Grout]

Jason Grout wrote:
>
> Well, there is something to be said for what you said. We ship our own
> version of matplotlib, which mostly likely is different than the system
> install. That means that their perfectly good default matplotlibrc file
> might cause deprecation warnings for us. For that reason, I think we
> should still set the MATPLOTLIBRC environment variable to point to a
> location within $DOT_SAGE.
>
> For that reason, how about yet another proposal:
>
> Add a warnings handler which traps the Matplotlib deprecation
> warning. This handler will compare the matplotlibrc with the one that
> was distributed in the last release (maybe by checking a hash value).
> If it is identical (i.e., the user hasn't modified the matplotlibrc),
> then *delete* the matplotlibrc. This is a
> bit tricky, since the sage default matplotlibrc is not under version
> control (it's in $SAGE_ROOT), so we don't know what older copies looked
> like. (hehe...yet another reason to really like mabshoff for the
> library of old builds). If the file is not identical to the current
> sage default matplotlibrc, then throw a warning with a helpful error
> message about updating the matplotlibrc (and where to find the file).

I should note that the file we would be deleting lives in $DOT_SAGE, and
so should not affect the user's other usage of matplotlib outside of Sage.

Jason

[Nils Bruin]

On Jan 30, 10:46 am, Jason Grout <jason-s...@creativetrax.com> wrote:

> Add a warnings handler which traps the Matplotlib deprecation
> warning. This handler will compare the matplotlibrc with the one that
> was distributed in the last release (maybe by checking a hash value).
> If it is identical (i.e., the user hasn't modified the matplotlibrc),
> then *delete* the matplotlibrc. This is a
> bit tricky, since the sage default matplotlibrc is not under version
> control (it's in $SAGE_ROOT), so we don't know what older copies looked
> like.

I have no opinion on whether the idea itself is worth implementing,
but if you want to know if the file has changed from
the original, you should probably include a hash of the file into
itself. matplotlibrc's format allows for comments, so this is
straightforward to do.
The person editing sage's default matplotlibrc should remember to
update the hash.

If a user edits matplotlibrc, the hash is almost surely not going to
match. If the user updates the hash .... well that's his own fault
then.

Some proof of concept shell script to do this:

----------------------------- sample for testing the hash
----------------------
#!/bin/sh
FILE=$1
REALHASH="#MD5-HASH: "`grep -v "^#MD5-HASH:" $FILE | md5sum`
STOREDHASH=`grep "^#MD5-HASH:" $FILE`
echo "R:" $REALHASH
echo "S:" $STOREDHASH
if [ "${REALHASH}" = "${STOREDHASH}" ]; then
echo "match"
else
echo "no match"
fi
-----------------------------------------------------------------------------------------

------------------ code for updating the stored hash
---------------------
#!/bin/sh
TMP=`mktemp`
FILE=$1
grep -v "^#MD5-HASH:" $FILE >> $TMP
HASH=`md5sum < $TMP`
mv $TMP $FILE
echo "#MD5-HASH: $HASH" >> $FILE
----------------------------------------------------------------------------------------

I'm sure the corresponding routines in Python would be both more
elegant and faster.

[William Stein]

I'm just going to make a call on this, so that the new matplotlib gets into
Sage ASAP. Jason's original proposal is:

"So here's a proposal: Should Sage stop distributing a custom
matplotlibrc, and ignore matplotlibrc files that already exist in the
$DOT_SAGE directories?"

After the above discussion, and responses to my concerns about this, I
say we do exactly what Jason originally proposed.

William

[William Stein]

Just to be clear, since Jason wasn't. We should stop distributing a
custom matplotlibrc, and ignore the $DOT_SAGE/matplotlibrc that we
ship, by not setting that environment variable when Sage starts up.
To implement this is as simple as deleting this code from
SAGE_ROOT/local/bin/sage-sage and anything in there that called it:

matplotlib_setup() {
MATPLOTLIBRC="$DOT_SAGE/" && export MATPLOTLIBRC
if [ ! -f "$MATPLOTLIBRC/matplotlibrc" ]; then
mkdir -p "$DOT_SAGE"
cp "$SAGE_ROOT/matplotlibrc" "$DOT_SAGE/"
fi
}

If users want to have custom matplotlibrc's they can always set
MATPLOTLIBRC however they want.

-- William

[Jason Grout]

William Stein wrote:

>> I'm just going to make a call on this, so that the new matplotlib gets into
>> Sage ASAP. Jason's original proposal is:
>>
>> "So here's a proposal: Should Sage stop distributing a custom
>> matplotlibrc, and ignore matplotlibrc files that already exist in the
>> $DOT_SAGE directories?"
>>
>> After the above discussion, and responses to my concerns about this, I
>> say we do exactly what Jason originally proposed.
>
> Just to be clear, since Jason wasn't. We should stop distributing a
> custom matplotlibrc, and ignore the $DOT_SAGE/matplotlibrc that we
> ship, by not setting that environment variable when Sage starts up.
> To implement this is as simple as deleting this code from
> SAGE_ROOT/local/bin/sage-sage and anything in there that called it:
>
> matplotlib_setup() {
> MATPLOTLIBRC="$DOT_SAGE/" && export MATPLOTLIBRC
> if [ ! -f "$MATPLOTLIBRC/matplotlibrc" ]; then
> mkdir -p "$DOT_SAGE"
> cp "$SAGE_ROOT/matplotlibrc" "$DOT_SAGE/"
> fi
> }
>
> If users want to have custom matplotlibrc's they can always set
> MATPLOTLIBRC however they want.

The patch up on #4774 along with the spkg does exactly this, IIRC (the
patch also takes care of a few other places in the bin directory that
need to change to implement this). So with this decision, I think that
#4774 is ready to be reviewed.

The new spkg up on #4774 deletes the custom arrow-drawing code that I
put in a few months ago since much better functionality has now been
written in Matplotlib. There is a patch on #4774 that also modifies the
arrow drawing in Sage to use the new arrow-drawing code included in
Matplotlib.

We should provide instructions somewhere on how to set a MATPLOTLIBRC
variable just for Sage. Maybe it would be enough to add something to a
person's init.sage?

Thanks,

Jason

P.S. wait, I see that rlm posted a comment about a problem with deleting
the matplotlibrc files, so apparently the ticket is ready to be fixed
maybe? Or at least the problem should be verified. It looks like it
might be one of those dreaded OSX libpng issues. Michael Abshoff: care
to comment?