Scaling & Stats Bug

[Sala]

Hey all

I continue to run into problems with logs. Let's say I have a simple
data frame, like

df = data.frame(x=c(1,1,1),y=c(1,0,0.5))

The mean of df$y is 0.5:

> mean(df$y)
[1] 0.5

Plotting this with a log10 transformed y axis shows an incorrect value
(somewhere around 10^-0.33):

> ggplot(df,aes(x,y)) + stat_summary(fun.y=mean, geom="point") + scale_y_continuous(trans="log10")

but it should be 10^-0.30103 since

> log10(mean(df$y))
[1] -0.30103

Now even stranger, when I add 0.01 to the y values:

> df$y = df$y+0.01
> mean(df$y)
[1] 0.51

I would now expect the mean value to be at 10^-0.2924298 since

> log10(mean(df$y))
[1] -0.2924298

but

> ggplot(df,aes(x,y)) + stat_summary(fun.y=mean, geom="point") + scale_y_continuous(trans="log10")

shows it completely off, around 10^-0.75

Repeat the entire exercise without scale_y_continuous(trans="log10"),
and everything works perfectly fine.

After reading up on scales and statistical transformations, it seems
clear to me that scale transforms the data BEFORE the statistical
summary. So what is does is take the mean of log10(0.01), log10(0.51)
and log10(1.01) which is -0.7627027. Fair enough, but why is the mean
now shown as 10^-0.7627027 (rather than just -0.7627027)? If this
behavior is intended, I'd love to hear the reasoning behind it.
Otherwise it's probably a bug.

Thanks!
Marcel

--
You received this message because you are subscribed to the ggplot2 mailing list.
Please provide a reproducible example: http://gist.github.com/270442

To post: email ggp...@googlegroups.com
To unsubscribe: email ggplot2+u...@googlegroups.com
More options: http://groups.google.com/group/ggplot2

[Charlotte Wickham]

Regarding the second example. The scale has been transformed but the
plotted raw data haven't. So, the original point y = 0.01 is plotted
at 10^{-2}, and explains why the mean is at 10^(-0.76) - it's the
backtransformed mean of log10(y).

The problem with the first example is that infinity gets involved.
> log10(df$y)
[1] 0.00000 -Inf -0.30103
> mean(log10(df$y))
[1] -Inf

So, in the plot coordinate system the mean gets the y-coordinate -Inf
which refers to the bottom most margin of the plot about 10^-0.33
(often a useful trick but here it is misleading). An illustration of
this behaviour:

ggplot(df,aes(x, y)) + stat_summary(fun.y=mean, geom="point", size = 3) +
scale_y_continuous(trans="log10") + geom_point(colour = "red")

ggplot(df,aes(x, y)) + stat_summary(fun.y=mean, geom="point", size = 3) +
scale_y_continuous(trans="log10", limits = c(0.000001, 1)) +
geom_point(colour = "red")

I think I've explained the behavior you've seen but not the philosophy
behind it. Perhaps Hadley will way in.

Charlotte

[Marcel Salathé]

Hi Charlotte

Thanks for your response. Regarding the first example, you're right, and you're example is a very nice demonstration of the problem. I'm curious to hear what the philosophy is - I would think the more intuitive approach would be not to plot any Inf values, rather than an incorrect value.

Regarding the second example, I am still confused - I probably misunderstand scale transformations. If we do:

> df = data.frame(x=c(1,1,1),y=c(1,0,0.5))
> df$y = df$y+0.01

> ggplot(df,aes(x, y)) + stat_summary(fun.y=mean, geom="point", size = 3) + scale_y_continuous(trans="log10") + geom_point(colour = "red")

The data are plotted correctly, but the mean isn't. So the mean get's backtransformed? Why? That doesn't seem to make sense, because again, an incorrect value is shown at the end.

I wonder what I'm missing?

Thanks
Marcel

[Hadley Wickham]

On Fri, May 7, 2010 at 5:17 PM, Sala <salathe...@gmail.com> wrote:
> Hey all
>
> I continue to run into problems with logs. Let's say I have a simple
> data frame, like
>
> df = data.frame(x=c(1,1,1),y=c(1,0,0.5))
>
> The mean of df$y is 0.5:
>
>> mean(df$y)
> [1] 0.5
>
> Plotting this with a log10 transformed y axis shows an incorrect value
> (somewhere around 10^-0.33):
>
>> ggplot(df,aes(x,y)) + stat_summary(fun.y=mean, geom="point") + scale_y_continuous(trans="log10")

It's a bit hard to see what's going wrong here. Adding some more
information and tweaking values makes things a bit easier:

df <- data.frame(x = 1,y = c(50,100,150))

ggplot(df,aes(x,y)) +
geom_point() +
stat_summary(fun.y=mean, geom="point", color = "red") +
scale_y_log10()

Why doesn't the red dot lie at 10^2? Well, the scale transformation
occurs before the statistical transformation - so we're computing the
mean of the log transformed values:

> mean(log10(c(50, 100, 150)))
[1] 1.958354

It works this way because the basic idea is that these two plots
should appear identical (apart from the labels on the scale):

ggplot(df,aes(x,y)) +
geom_point() +
stat_summary(fun.y=mean, geom="point", color = "red") +
scale_y_log10()

and

ggplot(df,aes(x, log10(y))) +
geom_point() +
stat_summary(fun.y=mean, geom="point", color = "red")

Does that help?

Hadley

--
Assistant Professor / Dobelman Family Junior Chair
Department of Statistics / Rice University
http://had.co.nz/