Effect sizes based on R-squared included-excluded, problems
Ned Kock
Good discussion going on here folks.
Let me preface this message by saying that I don’t want to be a spoiler, or doggedly push a biased point of view. At the same time, I am fascinated by the mathematics underlying PLS-based SEM; some argue it is simple, but in fact it is not that simple when you delve deeply into it.
So, with the provisos that I may be wrong, and that I am open to debate, here goes my perspective on this.
Calculating effect sizes for paths based on R-squared included and R-squared excluded may lead to unreliable estimates in PLS-based SEM. Just to be clear, the formula I am referring to is shown below; in it, R2i is the R-squared when the path is included in the model, and R2e is the R-squared when the path is excluded from the model.
( R2i - R2e ) / ( 1 - R2i )
The formula in question has been derived from a similar one provided by Cohen in the context of hierarchical multiple regression, not PLS-based SEM. There are a number of reasons why this formula may lead to values that probably deviate from the actual contribution to the R-squared by a given path (latent variable), which is what the effect size is supposed to reflect, particularly in complex models.
One of the reasons is that if multiple predictor latent variables are correlated in a latent variable block, removing one of them will significantly alter the contributions to the R-squared by the other latent variables. This is in part due to the fact that the path coefficients associated with the other latent variables will change.
This is particularly problematic when PLS modes A or B are used for outer model coefficients and latent variable scores estimation (as opposed to straight PLS regression), because in these modes the latent variable scores depend on the links between latent variables.
In other words, if you remove a link to or from a latent variable, the latent variable scores change in these modes. This follows directly from what Lohmöller called the “good neighbor assumption” in PLS modes A and B. Here is a blog post on this, for those who are interested:
Again, Cohen’s formula was proposed as an estimate of effect size in the context of hierarchical multiple regression analysis, where latent variables are not used. Often PLS-based SEM is equated with multiple regression analysis (a.k.a. OLS), but that is a misconception that can lead to problems.
Ned
Christian Ringle
You are right: Many of the criteria such as f² and q² have not been developed for PLS-SEM but for OLS regression. That's why I usually do not report f² and q² values in my papers. Looking into effect size and predictive relevance topics may be a research opportunity. However, I guess that nobody is taking these opportunities because the results would not differ from those approximations obtained from the OLS regression methods.
To avoid the issue of changes in the LV scores in the PLS-SEM for the estimation with and with out a construct, you can "freeze-in" the LV scores; then, just run simple regressions:
- Estimate the PLS-SEM
- Copy the latent variable scores for this model (SmartPLS includes provides the LV scores in a results table).
- Copy/import the LV scores into a stats software the allows you conducting a multiple regression analysis (e.g., Statistica, SPSS or R).
(- Run the regression model with the selected dependent LV and those that explain it as indented variables; the results are exactly the same as those for the path coefficients and the R²(included) value in SmartPLS)
- Delete the indicator of interest and run the regression model (based on the LVs from the full model) to obtain R(excluded).
This approach may solve your doubts about the changes in the LV scores. However, the effect size results are almost identical compared with the computation that was previously suggested...
Best
CR
--
--
You received this message because you are subscribed to the Google
Groups "PLS-SEM" group.
To post to this group, send email to pls...@googlegroups.com
To unsubscribe from this group, send email to
pls-sem+u...@googlegroups.com
---
You received this message because you are subscribed to the Google Groups "PLS-SEM" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pls-sem+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
Ned Kock
Hi Christian.
First of all, thanks for posting here. Your contributions and opinions are very welcome, always. I am sure that most subscribers truly appreciate the intellectual richness that your contributions bring to the discussion list.
On the effect sizes calculation topic, Cohen’s formula itself may lead to biased estimates under a number of conditions, even in the context of linear multiple regression analysis. Not only did Cohen acknowledge that, but also provided workarounds, mathematical procedures etc.
On a side note, the situation here with Cohen is similar to that with Lohmöller. The latter discussed a variety of PLS-based SEM algorithms, in detail. The end result is that 10 percent (if not less) of what he discussed was put in practice, and the rest was largely forgotten.
In my opinion, the ideal approach to calculate effect sizes as meant by Cohen in the context of PLS-based SEM is roughly the following: first calculate an R-squared, then “freeze” (nearly) all parameters, then remove the variable of interest, then re-calculate the R-squared (with the parameters “frozen”), and then take the unsigned difference in R-squared coefficients. That is the effect size.
Mathematically speaking, it is easy to “freeze” any individual parameter, or group of parameters, in any calculation. This would be similar to your example of manually “freezing” latent variable scores, but done computationally.
The procedure I briefly described above is the one implemented by WarpPLS for calculation of effect sizes. It leads to the estimation of the exact contribution of each predictor latent variable to the R-squared of the criterion latent variable, not an approximation of that value, even in nonlinear analyses.
The procedure employed by WarpPLS also allows for the calculation of effect sizes for indirect and total effects, in which case individual or grouped paths with multiple segments are manipulated while “freezing” parameters.
All of this is done without one having to resort to “corrections” for possible variations in path coefficients. This is what the denominator of Cohen’s formula essentially is, a “correction”.
Come to think of it, perhaps we should not call the estimates produced by either approach (the formula-based one’s or WarpPLS’s), as Cohen’s f-squared coefficients. Strictly speaking they aren’t, since f-squared coefficients have not been defined in the context of PLS-based SEM.
I am planning on publishing on these and other related issues in the future, but there are other things that need to be published as well, and time is finite.
Ned
Maria Daskalakis
I really appreciate your discussion which clarifies some topics about effect sizes, thank you very much!
Ned, I read that you plan to publish about how the calculation of effect sizes is implemented in WarpPLS in the future, however I would like very much to make reference by now - can you give me already some reference?
Best regards,
Maria
--
--
You received this message because you are subscribed to the Google
Groups "PLS-SEM" group.
To post to this group, send email to pls...@googlegroups.com
To unsubscribe from this group, send email to
pls-sem+u...@googlegroups.com
---
You received this message because you are subscribed to the Google Groups "PLS-SEM" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pls-sem+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
-- Dipl.-Oec. Maria Daskalakis Universität Kassel - University of Kassel FB Wirtschaftswissenschaften - Faculty of Economics Fachgebiet Umwelt- und Verhaltensökonomik - Environmental Economics and Behavioral Economics Nora-Platiel-Str. 4 34109 Kassel fon +49 (0)561-804-3052 fax +49 (0)561 804-3882 mailto:daska...@wirtschaft.uni-kassel.de http://www.ivwl.uni-kassel.de/beckenbach/mitarbeiter/daskalakis.html Projekt "Eine akteursbasierte dynamische Analyse und Bewertung von umweltpolitischen Instrumenten am Beispiel des Immissionsschutzes - Ein Beitrag zur Nachhaltigkeitsgovernance" (DABEI) Project "Agent-based dynamic analysis and evaluation of economic instruments: the example of immission control - A contribution to the governance of sustainability" http://www.uni-kassel.de/beckenbach/index.php?option=com_content&view=article&id=43&Itemid=5&lang=de Projekt "Innovative Ansätze zur Verbesserung der Anreizwirkungen umweltpolitischer Instrumente: Bestandsaufnahme innovativer Erklärungsansätze" Project "Innovative approaches for improving the incentives of environmental instruments: state of the art" http://www.uni-kassel.de/beckenbach/index.php?option=com_content&view=article&id=61&Itemid=5&lang=de Projekt "Gender & Gründungsinteressen: Vorschläge für eine gendersensible Entrepreneurship Education an der Universität Kassel" Project "Gender & Selfemployment: Proposals for a gender-sensitiv entrepreneurship education at the University of Kassel" http://www.uni-kassel.de/beckenbach/index.php?option=com_content&view=article&id=62&Itemid=5&lang=de Projekt "Sidestep" (Studentische Ideen Stärken die eigenen Potentiale); Eine Untersuchung zur den unternehmerischen Potentialen von Studierenden Project "Sidestep" - Exploring the entrepreneurial potential of students http://www.uni-kassel.de/beckenbach/index.php?option=com_content&view=article&id=56&Itemid=5&lang=de Diese E-Mail könnte vertrauliche und/oder rechtlich geschützte Informationen enthalten. Diese Informationen sind ausschließlich für die bezeichnete/-n Person/-en oder Einrichtung/-en bestimmt. Sollten Sie nicht der für diese E-Mail bestimmte Adressat sein, ist Ihnen jede Veröffentlichung, Vervielfältigung oder Weitergabe untersagt. Haben Sie diese E-Mail irrtümlich erhalten, bitte ich Sie, mich darüber in Kenntnis zu setzen, die E-Mail zurückzusenden und Ihr Exemplar zu vernichten.
Ned Kock
Hi Maria. At this point, I can only suggest that you cite the WarpPLS User Manual (citation below). I am not aware of any article that incorporates the details of the discussion that we are having here.
Kock, N. (2012). WarpPLS 3.0 User Manual. Laredo, Texas: ScriptWarp Systems.
Ned
Maria Daskalakis
thanks a lot!
If the article is accepted, I´ll send you the details.
Kind regards,
Maria
Kock, N. (2012). WarpPLS 3.0 User Manual. Laredo, Texas: ScriptWarp Systems.