sample size in propensity score matching

[Martin Holt]

Dear all.

 

I am working on a project in which the most natural analysis would be logistic regression.

 

I understand that the usual rule of thumb in logistic regression (based on simulation studies....see reference below) is that the frequency of the rarest outcome (0 or 1) should be at least 10 times the number of parameters. So if you have 1000 observations but only 60 1's and 940 0's you should probably not have more than 6 parameters in your model. We have insufficient observations, however, to support that rule of thumb.

 

So I am currently researching analysis by propensity score matching. Here I could do with some help on two fronts.

 

(a) From what I've read there is no need to perform multiple imputation when using propensity score matching. I'd welcome comments on this, please.

 

(b) Again, from what I've read, the sample size required is very much smaller. Is that right? If so, is there some rule of thumb regarding what sample size is required? Comments would be very much appreciated.

 

Best Wishes

 

Martin P. Holt

 

Ref: Peduzzi, P., Concato, J., Kemper, E., Holford, T.R., Feinstein, A.R. (1996) A simulation study of the number of events per variable in logistic regression analysis. Journal of Clinical Epidemiology, 49, 1373-9.

[Mehwish Hussain]

Good Question Halt! Though, I also need answer for the same.

Well, can you please, provide the reference for your sample size description for logistic regression as I know this rule of thumb too but I have different figures.

--
To post a new thread to MedStats, send email to MedS...@googlegroups.com .
MedStats' home page is http://groups.google.com/group/MedStats .
Rules: http://groups.google.com/group/MedStats/web/medstats-rules

--

Regards

Mehwish Hussain, PhD*
Senior Lecturer of Biostatistics
Manager, Research Development

Office of Research, Innovation and Commercialization

Dow University of Health Sciences

Karachi, Pakistan
LinkedIn: http://www.linkedin.com/pub/mehwish-hussain/17/a8a/449

Research Gate: https://www.researchgate.net/profile/Mehwish_Hussain/

[Martin Holt]

Hello.

The reference you ask for is at the end of my earlier email:

Ref: Peduzzi, P., Concato, J., Kemper, E., Holford, T.R., Feinstein, A.R. (1996) A simulation study of the number of events per variable in logistic regression analysis. Journal of Clinical Epidemiology, 49, 1373-9.

I'll keep you informed should I get any offline replies.

Best Wishes,

Martin

---

From: Mehwish Hussain <mehvish....@gmail.com>
To: meds...@googlegroups.com
Sent: Tuesday, 27 November 2012, 17:07
Subject: Re: {MEDSTATS} sample size in propensity score matching

[Mehwish Hussain]

Thank you Martin, I was pleased for your kindness.

[Steve Simon, P.Mean Consulting]

On 11/27/2012 10:57 AM, Martin Holt wrote:> Dear all.

An interesting counterpoint is

Vittinghoff E, McCulloch CE. Relaxing the Rule of Ten Events per
Variable in Logistic and Cox Regression. Am. J. Epidemiol.
2007;165(6):710-718. Available at:
http://aje.oxfordjournals.org/cgi/content/abstract/165/6/710 [Accessed
February 18, 2009].

This article examines the rule that you need 10 events per independent
variable. Some sources cite 15 events and other 20 events per
independent variable. The authors argue that in the context of adjusting
for confounders, this rule might be relaxed a bit.

As far as sample size for propensity score matching goes, I think that
the propensity score counts as a single variable and thus can be used
when the rarer of the two outcomes is as little as 15 or 10. The
rationale is that the rule of 10 or 15 events per variable was justified
using variable selection methods such as stepwise. You're not interested
in variable selection here, but rather in getting a good predicted value
for the propensity score.

I haven't seen a formal justification of this, however.

Steve Simon, n...@pmean.com, Standard Disclaimer.
Sign up for the Monthly Mean, the newsletter that
dares to call itself average at www.pmean.com/news

[Steve Simon, P.Mean Consulting]

One more comment.

On 11/27/2012 10:57 AM, Martin Holt wrote:> Dear all.

> (a) From what I've read there is no need to perform multiple
> imputation when using propensity score matching. I'd welcome
> comments on this, please.

Are you asking whether single imputation is okay? It should be. The
reason you have to impute multiple times is to get the correct amount of
sampling variation. Single imputation understates sampling variation.
Since the only thing you need in propensity score matching is the
propensity score itself, which is a predicted value, you don't care
about sampling variation.

It makes intuitive sense, but, alas, I do not have a reference for this.
If you could let us know the sources for "what I've read" that would be
helpful. I've not done a lot of propensity score matching, so I should
probably read more from the experts.

[Martin Holt]

Dear Steve,

 

Thank you for your coming back to me. 

 

The sources I referred to I have as powerpoint slides so I can't send them out via MedStats. I will try and find where I got them from. In particular, I need to find a source relating to sample size.

 

In the meantime I am sending one source referring to propensity score matching and multiple imputation. It takes some reading but is very thorough.

 

http://faculty.smu.edu/Millimet/classes/eco7377/papers/mattei.pdf

 

Best Regards,

 

Martin P. Holt

From: "Steve Simon, P.Mean Consulting" <n...@pmean.com>
To: meds...@googlegroups.com
Cc: Martin Holt <martin...@yahoo.com>
Sent: Wednesday, 28 November 2012, 12:51
Subject: Re: {MEDSTATS} sample size in propensity score matching

[Martin Holt]

Dear Steve,

 

I've located the sources I referred to.

 

Slide 15 of www.chrp.org/love/ASACleveland2003Propensity.pdf says that "Thou shalt have 10 times as many subjects as predictors" so if I use 10 predictors I shall have to have 100 participants. This is much more achievable than the rules of thumb for logistic regression. Am I missing something?

 

With respect to how propensity score matching and imputation of missing data interact, Slide 21 of http://ssw.unc.edu/VRC/lectures/PSM_SSWR_2004.pdf

discusses "Stratification, one of several methods developed for missing data imputation. (1) Group sample into five categories based on propensity score (quintiles). (2) Within each quintile there are r participants and n non-participants.Use "approximate Bayesian bootstrap method to conduct matching or resampling.

 

Any further comments?

 

Best Regards,

 

Martin P. Holt

 

From: "Steve Simon, P.Mean Consulting" <n...@pmean.com>
To: meds...@googlegroups.com
Cc: Martin Holt <martin...@yahoo.com>
Sent: Wednesday, 28 November 2012, 12:51
Subject: Re: {MEDSTATS} sample size in propensity score matching
 

[Steve Simon, P.Mean Consulting]

A couple more thoughts. The rule of 15 is only half of the problem. That
rule insures that your models are stable and repeatable. But you also
have to insure adequate power and precision. The very fact that you need
to use propensity scores is an indication that you expect to lose some
power because of multicollinearity between your treatment variable and
various covariates. This loss of power manifests itself in a large
number of observations that can't be matched properly. If you stratify
by your propensity score, you lose power by the imbalance in sample
sizes in the various strata.

You can and should account for this in your sample size justification.
The formulas are tedious but not difficult.

Another point is that matching is almost always a bad choice unless you
have a lot of data that you're willing to throw away. Serious matching
will leave a lot of your data unmatched, especially if the propensity
score matching is needed. The only time I would match is if you have
lots of controls for every possible treated patient. Then you're losing
from the group that has "too many" in it anyway, so the loss doesn't
sting as much.

And those people who use a rule of 10 versus a rule of 15 have nothing
really to back themselves up with other than a fear that the rule of 15
is too harsh. I've actually heard that it might be better to move in the
opposite direction and that you should strive for 20 events per
independent variable.

Finally, from what I've read, there's not a lot of consensus in the
research community about how to compute and use propensity scores.
Whenever there is lack of consensus, that gives you the green light to
do what you think is best, as there is no definitive source that
everyone uses. Be ready to adapt to a different approach though, as the
peer reviewers are unpredictable and are likely to ask for changes no
matter which approach you choose.

[Marc Schwartz]

One other thought to put forth and perhaps I missed this or mis-read it in the thread along the way.

Rather than engaging in the actual matching and/or stratification on the PS between groups, which can reduce your effective sample size as Steve notes, consider using the PS (or the logit of the PS) as a continuous covariate in the regression model, to provide for covariate adjustment on the PS.

In combination with the above approach, strongly consider using a cubic regression spline on the PS covariate, rather than presuming a linear relationship.

I recall some work done on this in the past, suggesting a reduction in bias when using a cubic spline transformation on the PS, albeit I can't recall the source at the moment. A Google search with the requisite keywords is likely to come up with something.

Regards,

Marc Schwartz

[Bruce Weaver]

Thank you for raising that point, Marc.  I think that people are often far too eager to match rather than to keep all cases in the analysis and control for the propensity score (or whatever variables would be used for matching).  IIRC, Hennekens & Buring talk about this issue in their book Epidemiology in Medicine, and come down largely in favour of controlling for covariates.

Bruce


On Monday, December 3, 2012 12:04:37 PM UTC-5, Marc Schwartz wrote:

--- snip ---

[Steve Simon, P.Mean Consulting]

On 12/3/2012 11:04 AM, Marc Schwartz wrote:

> One other thought to put forth and perhaps I missed this or mis-read
> it in the thread along the way.
>
> Rather than engaging in the actual matching and/or stratification on
> the PS between groups, which can reduce your effective sample size as
> Steve notes, consider using the PS (or the logit of the PS) as a
> continuous covariate in the regression model, to provide for
> covariate adjustment on the PS.
>
> In combination with the above approach, strongly consider using a
> cubic regression spline on the PS covariate, rather than presuming a
> linear relationship.
>
> I recall some work done on this in the past, suggesting a reduction
> in bias when using a cubic spline transformation on the PS, albeit I
> can't recall the source at the moment. A Google search with the
> requisite keywords is likely to come up with something.

You still pay a penalty. In any model where propensity scores are
needed, the propensity score will be correlated with your treatment
variable, and the resulting multi-collinearity will cause a loss in power.

Even so, this is a very reasonable approach and may be superior to
matching and/or stratification.

[Martin Holt]

Hi Mehwish.

I seem to remember that you sent me an email outside of MedStats recommending a text (I remember the message saying that there was a chapter all about how to use propensity scoring). If I'm wrong, I'm sorry for the intrusion, but I'm hoping I'm right.

Best Wishes,

Martin

---

From: Mehwish Hussain <mehvish....@gmail.com>
To: meds...@googlegroups.com

Sent: Wednesday, 28 November 2012, 6:44