Smoothed quantile plot

[Stavros Macrakis]

I'm looking for a way of calculating and plotting smoothed quantiles.

Suppose I teach archery, and I want to plot the progress of students as they get more and more practice.  I want to plot not only the fitted improvement line, but also see what the distribution of marksmanship is like at any given experience level.  So I collect snapshot data for all my students: as of today, how many times have you shot at the target? And how many times have you hit it? (Yes, there are issues around drop-outs and so on, but let's leave that aside for now.) So I make a scatterplot of (hit count)/(try count) against (try count).  Oh, and by the way, the distribution of (try count) is highly skewed -- most of the students have only tried 1-3 times, though a few students have tried as many as 1000 times.

A first look at the data shows that a linear quantile regression doesn't capture the data very well, so I want to plot some sort of smoothed quantile. I could certainly bin the data and calculate the quantiles for each bin, but I'd rather have a smooth estimate.

What would be a sensible way of smoothing this data so that I can visualize it?  If expect a priori that the quality of marksmanship will improve monotonically with experience, can I use that fact to smooth better?

What I have in mind is some sort of running quantile calculation weighted by a kernel function.  If there's nothing standard along those lines, I'll try my hand at it, but I'm sure my solution would take much longer to write, run inefficiently, and probably not be statistically/mathematically as sound as something that exists already!

Thanks!

           -s

[hadley wickham]

On Sun, Apr 5, 2009 at 11:34 AM, Stavros Macrakis <macr...@gmail.com> wrote:
> I'm looking for a way of calculating and plotting smoothed quantiles.

Have you looked at the quantreg package? This is what powers the
stat_quantile. Unfortunately due to bugs in quantreg (which I have
reported to the maintainer but he has never responded) you can't use
the smoothed quantile estimation (qss) within stat_quantile, but you
should be able to run it yourself and then display the results with
geom_line.

Hadley

--
http://had.co.nz/

[Stavros Macrakis]

Hadley,

Thanks for the pointer -- no, I wasn't aware of quantreg or for that matter that what I was looking for (or at least the statistically well-founded version of what I was looking for) is called nonparametric quantile regression estimation.  A lot to learn.

It appears there was also a package called nprq doing similar things, but it is no longer on CRAN (superceded by quantreg?).

Now to study.  Or is there a simple approach which might actually be meaningful?

               -s

[hadley wickham]

On Sun, Apr 5, 2009 at 9:20 PM, Stavros Macrakis <macr...@alum.mit.edu> wrote:
> Hadley,
>
> Thanks for the pointer -- no, I wasn't aware of quantreg or for that matter
> that what I was looking for (or at least the statistically well-founded
> version of what I was looking for) is called nonparametric quantile
> regression estimation.  A lot to learn.
>
> It appears there was also a package called nprq doing similar things, but it
> is no longer on CRAN (superceded by quantreg?).
>
> Now to study.  Or is there a simple approach which might actually be
> meaningful?

I'd start with rqss:

" Fitting function for additive quantile regression models with
possible univariate and/or bivariate nonparametric terms estimated
by total variation regularization."

Hadley

--
http://had.co.nz/

[Stavros Macrakis]

Thanks, that looks very useful....

            -s