Quantile-quantile plot
Stavros Macrakis
I can do this for a single case very simply:
# Distributions to compare
dist1 <- c(rnorm(100,1,1),rnorm(100,3,1))
dist2 <- c(rnorm(50,2,1),rnorm(300,3,1))
# Calculated quantiles
q1 <- quantile(dist1,0:100/100)
q2 <- quantile(dist2,0:100/100)
ggplot(data=data.frame(a=q1,b=q2)) +
geom_line(aes(x=a,y=b)) +
geom_abline(intercept=0,slope=1)
This works fine.
However, I can't figure out how to do the usual beautiful ggplot slice-and-dice (colors, facets) with different subsets of the distributions.
One thought I had was to define a scale -- perhaps this is the intent of the undocumented scale_xx_prob? But even if I could do that, I don't see how I'd get the data in, since ggplot requires observations to come in as a single data frame where x's and y's are linked. In the case of comparing distributions, the count may be different both for the full data set and for subsets, and the data points aren't linked.
Ideas?
-s
Stavros Macrakis
On Thu, May 21, 2009 at 4:19 PM, Stavros Macrakis <macr...@gmail.com> wrote:
I am trying to create a quantile-quantile plot of two sample distributions. ...
However, I can't figure out how to do the usual beautiful ggplot slice-and-dice (colors, facets) with different subsets of the distributions....
What I ended up doing was building a data frame with the qq values precalculated. This is not satisfactory in general because every possible combination of conditions that could be displayed needs to be precalculated.
What I am thinking of doing now is to put all the data in a single data frame and write a stat_ which takes an argument specifying the subsets to compare, something like this:
# Distributions to compare
dist1 <- data.frame( day = sample(7,200,replace=TRUE), traffic = c(rnorm(100,1,1),rnorm(100,3,1)) )
df <- cbind( rbind(dist1,dist2), idx = c( rep(1:2, c(200,350) ) )
If I want to compare these distributions using a density plot, I can now do:
ggplot(data=df) +
stat_density(aes(x=traffic,colour=idx),
geom="path",
position='identity') + # btw, why do I get a warning if this is missing?
facet_grid(. ~ day,margins=T)
One possible syntax for stat_qq (the current stat_qq should be renamed as stat_probability or something):
ggplot(data=df) +
stat_qq(aes(x=traffic,y=traffic, axis=idx)) +
facet_grid(. ~ day,margins=T)
but that has its own limitations. For example, what if I want to compare N distributions against a reference distribution? Or N against M? Then again, that seems to be a more general problem -- which I'll write up as a separate email.
-s
hadley wickham
> This is not the same as stat_qq, which is a sample/theoretical plot
> (apparently the standard name for that is not a quantile-quantile plot, but
> a "probability plot" --
> http://www.itl.nist.gov/div898/handbook/eda/section3/probplot.htm).
Oh, good point. Maybe stat_qp would be more appropriate.
> One thought I had was to define a scale -- perhaps this is the intent of the
> undocumented scale_xx_prob? But even if I could do that, I don't see how I'd
> get the data in, since ggplot requires observations to come in as a single
> data frame where x's and y's are linked. In the case of comparing
> distributions, the count may be different both for the full data set and for
> subsets, and the data points aren't linked.
Yes, that's true. I think you'd need to precompute a fixed set of
quantiles beforehand, and then plot those. I don't see why you need
to precompute every possible combination.
Hadley
Stavros Macrakis
> One thought I had was to define a scale -- perhaps this is the intent of theYes, that's true. I think you'd need to precompute a fixed set of
> undocumented scale_xx_prob? But even if I could do that, I don't see how I'd
> get the data in, since ggplot requires observations to come in as a single
> data frame where x's and y's are linked. In the case of comparing
> distributions, the count may be different both for the full data set and for
> subsets, and the data points aren't linked.
quantiles beforehand, and then plot those. I don't see why you need
to precompute every possible combination.
Suppose for example I'm doing q-q plotting of rents in different neighborhoods in some city, but then I want to slice-and-dice by size of apartment (studio, 1br, 2br, etc.), amenities (doorman? central air conditioning?), age of building, etc. Every combination of conditions (colour=apartment_size, shape=doorman) will have different quantiles which cannot be calculated from the aggregated quantiles.
But I think you'll see in a later email that I have some thoughts on how to do this with a stat. All the data are in one data frame, and the quantiles are calculated by a stat which has as an argument the specifications of the two subsets (e.g. neighborhoods) to compare. This could be a two-valued column (Upper East Side or Williamsburg?) or an n-valued column with an additional argument specifying which values, e.g. axes=neighbourhood, axis_values=c("Upper East Side","Williamsburg"). The second approach would be especially attractive if there were some way to combine this with faceting, so that you could get an nxn table of the cross-comparisons.
-s
hadley wickham
> different distribution.
Oh, of course. Still, this isn't terribly hard to do (i.e. one line of code)
> Suppose for example I'm doing q-q plotting of rents in different
> neighborhoods in some city, but then I want to slice-and-dice by size of
> apartment (studio, 1br, 2br, etc.), amenities (doorman? central air
> conditioning?), age of building, etc. Every combination of conditions
> (colour=apartment_size, shape=doorman) will have different quantiles which
> cannot be calculated from the aggregated quantiles.
In this case, what would you have on the x axis, and what on the y
axis? Or would it more like a scatterplot matrix?
> But I think you'll see in a later email that I have some thoughts on how to
> do this with a stat.
I either didn't get this or it got lost in another thread - could you
resend please?
> All the data are in one data frame, and the quantiles
> are calculated by a stat which has as an argument the specifications of the
> two subsets (e.g. neighborhoods) to compare. This could be a two-valued
> column (Upper East Side or Williamsburg?) or an n-valued column with an
> additional argument specifying which values, e.g. axes=neighbourhood,
> axis_values=c("Upper East Side","Williamsburg"). The second approach would
> be especially attractive if there were some way to combine this with
> faceting, so that you could get an nxn table of the cross-comparisons.
So it would work something like this:
ggplot(rents, aes(sample = price)) +
stat_summary(aes(compare = Neighbourhood == "Upper East Side"))
(I'm assuming it would be better to use a logical column)
This would give you upper east side on the y axis, and everything else
on the x axis.
To get all combinations, I think we need a new high level function
that works like plot matrix, but splits based on the value of a
variable, rather than the different variables.
q <- function(x, probs = seq(0.01, 0.99, length = 100)) {
data.frame(q = probs, value = quantile(x, probs))
}
quants <- ddply(rents, "Neighbourhood", function(df) q(df$rent))
xxx(quants)
quants <- ddply(rents, c("Neighbourhood", "Rooms"), function(df) q(df$rent))
xxx(quants[, -1], colour = "rooms")
The xxx function could probably do the actual computation too.
Hadley
Stavros Macrakis
> Because every subset (as defined by colour/facet/shape/etc.) will have aOh, of course. Still, this isn't terribly hard to do (i.e. one line of code)
> different distribution.
But you have to generate *all* subsets that you might display.
> Suppose for example I'm doing q-q plotting of rents in differentIn this case, what would you have on the x axis, and what on the y
> neighborhoods in some city, but then I want to slice-and-dice by size of
> apartment (studio, 1br, 2br, etc.), amenities (doorman? central air
> conditioning?), age of building, etc. Every combination of conditions
> (colour=apartment_size, shape=doorman) will have different quantiles which
> cannot be calculated from the aggregated quantiles.
axis? Or would it more like a scatterplot matrix?
I'd have the same subset of different neighborhoods on each axis. e.g. x axis is 1br doorman buildings on the Upper West Side and y axis is 1br doorman buildings in Williamsburg.
> But I think you'll see in a later email that I have some thoughts on how to
> do this with a stat.
I either didn't get this or it got lost in another thread - could you
resend please?
I've put a copy at the end of this email.
> All the data are in one data frame, and the quantilesSo it would work something like this:
> are calculated by a stat which has as an argument the specifications of the
> two subsets (e.g. neighborhoods) to compare. This could be a two-valued
> column (Upper East Side or Williamsburg?) or an n-valued column with an
> additional argument specifying which values, e.g. axes=neighbourhood,
> axis_values=c("Upper East Side","Williamsburg"). The second approach would
> be especially attractive if there were some way to combine this with
> faceting, so that you could get an nxn table of the cross-comparisons.
ggplot(rents, aes(sample = price)) +
stat_summary(aes(compare = Neighbourhood == "Upper East Side"))
(I'm assuming it would be better to use a logical column)
This would give you upper east side on the y axis, and everything else
on the x axis.
To get all combinations, I think we need a new high level function
that works like plot matrix, but splits based on the value of a
variable, rather than the different variables.
Yes.
q <- function(x, probs = seq(0.01, 0.99, length = 100)) {
data.frame(q = probs, value = quantile(x, probs))
}
quants <- ddply(rents, "Neighbourhood", function(df) q(df$rent))
xxx(quants)
quants <- ddply(rents, c("Neighbourhood", "Rooms"), function(df) q(df$rent))
xxx(quants[, -1], colour = "rooms")
The xxx function could probably do the actual computation too.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
On Thu, May 21, 2009 at 4:19 PM, Stavros Macrakis <macr...@gmail.com> wrote:
I am trying to create a quantile-quantile plot of two sample distributions. ...
However, I can't figure out how to do the usual beautiful ggplot slice-and-dice (colors, facets) with different subsets of the distributions....
What I ended up doing was building a data frame with the qq values precalculated. This is not satisfactory in general because every possible combination of conditions that could be displayed needs to be precalculated.
What I am thinking of doing now is to put all the data in a single data frame and write a stat_ which takes an argument specifying the subsets to compare, something like this:
Stavros Macrakis
In this case, what would you have on the x axis, and what on the y
axis? Or would it more like a scatterplot matrix?
I'd have the same subset of different neighborhoods on each axis. e.g. x axis is 1br doorman buildings on the Upper West Side and y axis is 1br doorman buildings in Williamsburg.
Or I might have
x axis is Upper West Side, y axis is Williamsburg
color maps to size of apartment (studio, 1br, 2br, 3+br)
shape maps to has-doorman (y/n)
So there are 8 qq plots per pane.
-s