standardized analysis output

[Daniel Lakens]

Dear Psych Data Standards team,

I need your help. I am working on a standardized analysis approach where users upload a datafile, specify some choices (e.g., their alpha level), and knit a document that will report the full analysis following best practices. I started this in 2015, and made an example of a 'perfect t-test'. All code is at:

https://github.com/Lakens/perfect-t-test

The output of the RMarkdown document is at:

https://github.com/Lakens/perfect-t-test/blob/master/perfect_independent_t-test_2.md

The goal is to make data analysis easy for users. But *if* users choose to use a standardized analysis procedure, we should be able to save all the results, the variable names, the codebook, etc, in a standardized format. The idea is that the standard data output format can subsequently be read in be the standardized meta-analysis script, for example, which will run a meta-analysis using best standards.

It would be a way for users to very easily analyze their data, while at the same time automatically providing all the meta-data underlying their analysis.

What I need help with is how I should store the data that exists in the RMarkdown file. Basically I have statistical data such as:

m_diff
ci_l_m_diff
ci_u_m_diff
d
d_unb
sd_pooled
ci_l_d
ci_u_d
ci_l_d_unb
ci_u_d_unb
n1
n2

But users also specify variable names. The code know (badly) uses Ruben Arslan's codebook package - which could be another part of the output.

I would like to get a alpha version ready before SIPS. I think such a standard analysis approach with standard data output, would allow us to link together several other SIPS projects - for example pre-reg 2.0, where the specific design choices in the pre-reg document are linked to in the final report (in RMarkdown).

I would love to hear your suggestions.

Thanks,

Daniel

[Ruben Arslan]

HI Daniel,

I'm excited about this sort of thing.

I recently tried to get the authors of the "easyverse" packages (formerly known

as sjPlot, sjstats etc) interested in the approach I took in codebook.

https://github.com/easystats/report/issues/17

I think this approach (rmarkdown partials) is powerful, because it allows

us to write standardised output combining tables, text, plots, and even 

hidden metadata – which can then be transformed into HTML, PDF, Docx.

Basically you could take your perfect t test rmarkdown and put it into

an R function – which may make it easier to use.

I think what you're asking for is basically a standardised way to report results

right? I think making standardised tables (with var labels?) would be great. 

Hope this helps a little. Happy to talk more at SIPS,

Ruben

PS.:

The report package and similar pkgs make tables with proper variable labels

(used in the same way as codebook) etc. It's probably very related to what you're

interested in.

--
You received this message because you are subscribed to the Google Groups "Psych-Data-Standards" group.
To unsubscribe from this group and stop receiving emails from it, send an email to psych-data-stand...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/psych-data-standards/6b8ae468-7009-43e0-baff-d3c250610f37%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ruben Arslan]

Hi Peder,

I'm not so sure how helpful it is to standardise the input

– asking people to write a JSON file of their hypothesis might be harder than using an R function.

But YOU could given standardised input (such as in a function or an Rmd file with fixed names) write such a JSON file.

You could even start with a t.test object in R and make it "perfect". That would be comparable to what the report package does.

Best,

Ruben

On Fri, Mar 15, 2019 at 3:40 PM Peder Isager <peder...@gmail.com> wrote:

Thanks Ruben! 

Just to add to Daniel's suggestion, I think the perfect-t-test is a nice example of data analysis apps that could take psych-DS open data as input. The challenge seems to be to figure out how to set up the meta-data so that one could automatically run the t-test (e.g. could would one specify a statistical hypothesis as a json object, in a way that variables, test type, and parameters are made clear?), and to figure out best output format for the test given the goal of machine-readability, the recommendations of the psych-DS spec, and the fact that the output should later be usable as input in an automated meta-analysis. 

In general, as a current user of the Psych-DS standard, I am very interested in learning more about how I should/could structure my meta-data. Would be very interested in talking about this during SIPS

Best, 

Peder

To view this discussion on the web visit https://groups.google.com/d/msgid/psych-data-standards/CALv3xzvaz_%3Dg4a%2BCX98%2BrZ6p6G50gQ3zXKzBnBb5tVdXDMEY5Q%40mail.gmail.com.

[Daniel Lakens]

Hi,

Thanks, glad to see this sounds interesting! I would for now like to focus on, as Ruben calls it, the 'hidden meta-data' output of a t-test. However, I think it would be cool to make the 'perfect meta-analysis' RMarkdown script that then can use the JSON t-test output, just to demonstrate one potential use of good meta-data output.

I am open to any way the output should look, and not knowledgeable about JSON. I would also be happy to turn this into a function (is that possible? A function that outputs an RMarkdown doc? That sounds perfect to me).

It would be great if we could get other popular packages on board. For now, If we start with a t-test, correlation, and then meta-analyses of these, I think we could get a nice initial userbase. I am working on a new MOOC, and happy to introduce these tools there - which could help spreading the word.

I would love to hang out at SIPS and actually make this (but also happy to come to SIPS with a decent alpha version).

Daniel

[Peder Isager]

Thanks Ruben! 

Just to add to Daniel's suggestion, I think the perfect-t-test is a nice example of data analysis apps that could take psych-DS open data as input. The challenge seems to be to figure out how to set up the meta-data so that one could automatically run the t-test (e.g. could would one specify a statistical hypothesis as a json object, in a way that variables, test type, and parameters are made clear?), and to figure out best output format for the test given the goal of machine-readability, the recommendations of the psych-DS spec, and the fact that the output should later be usable as input in an automated meta-analysis. 

In general, as a current user of the Psych-DS standard, I am very interested in learning more about how I should/could structure my meta-data. Would be very interested in talking about this during SIPS

Best, 

Peder

Den fre. 15. mar. 2019 kl. 15:32 skrev Ruben Arslan <ruben...@gmail.com>:

To view this discussion on the web visit https://groups.google.com/d/msgid/psych-data-standards/CALv3xzvaz_%3Dg4a%2BCX98%2BrZ6p6G50gQ3zXKzBnBb5tVdXDMEY5Q%40mail.gmail.com.

[Tal Yarkoni]

If the idea is to automate the transformation from dataset into statistical result, it may be worth looking at the (draft) BIDS-StatsModels spec:

https://docs.google.com/document/d/1bq5eNDHTb6Nkx3WUiOBgKvLNnaa5OMcGtD0AZ9yms2M/edit?usp=sharing

There's a lot in there that's not relevant outside of a neuroimaging context, but the core of it is basically a standardized way of representing a GLM, supporting a closed set of data transformations, and feeding forward results to additional analysis levels.

With respect to representing the outputs of statistical tests, I'd be happy to collaborate on a standard for this. I think it would be worth spending some time thinking about core principles before rushing to pick out specific naming conventions (maybe at SIPS?). One general issue that springs to mind is whether to represent each possible procedure as its own little world (with its own controlled vocabulary), or to try to find a common generative representation from which various statistics can be deterministically extracted. For example, the GLM framework can accommodate nearly all t-tests, ANOVAs, etc., so arguably it would be much simpler and more maintainable to come up with a spec for representing the structure of a GLM and the obtained parameter estimates than to have a giant dictionary of OneSampleTTest, TwoSampleTTest, RepeatedMeasuresANOVA, etc.

Put differently, if you have a good standard for machine-readable representation of (common) statistical models, adding support for results should be relatively straightforward, and most of the complexity in reporting can be off-loaded onto utility libraries that takes the common representation and spit out a wide variety of derivative statistics (or tables, etc.) as requested.

Tal

To view this discussion on the web visit https://groups.google.com/d/msgid/psych-data-standards/CALv3xzvYhd%2Bv0yywCyRj5KyhzWiOmPwEYZo%3DoYoUn66jtCx1%3Dg%40mail.gmail.com.

[Ruben Arslan]

Hey,

this is really cool. I didn't know BIDS was working on this. 

I think the highest-impact approach to making such metadata widespread in the R
world would be to identify packages that already summarise GLMs (e.g. broom, report)

and send pull requests for them to include the right metadata.

Happy to talk at SIPS.

Daniel, if you want to get going now, feel free to reach out off list and I can try

to explain how I would apply the approach I took with codebook to your t-test.

Basically, it's a function that validates its input, puts out an rmarkdown

component (which can be repeated many times in a document) that 

has the plots, explanations and (hidden) metadata.

Best,

Ruben

To view this discussion on the web visit https://groups.google.com/d/msgid/psych-data-standards/CAMQtfP%2BqT1QoknQgprkhVNpDtPKws8BNVNXTyhUTM-LPMdXPXA%40mail.gmail.com.

[Rickard Carlsson]

I’m *very much* interested in the perfect meta-analysis part. This is something that’s on my todo list regardless...

I might even be so lucky to have an assistant in the fall who could help with this. I’m 89 % confident... :)

--
You received this message because you are subscribed to the Google Groups "Psych-Data-Standards" group.
To unsubscribe from this group and stop receiving emails from it, send an email to psych-data-stand...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/psych-data-standards/3f2fc223-b555-4c92-984b-ad8528e3fc47%40googlegroups.com.

[Erin Buchanan]

Late to this party, but I would also be interested – given I’ve been working on the apps for the meta-data, I’d be happy to think about how to standardize meta-data into the right psych-ds format. Also got some experience on the package front (self promote, our effect size package MOTE is now in cran! Woo!). This sounds like an excellent hackathon. Might be useful to figure out the flow that you are interested in creating, which would allow us to create the right outputs for the user, etc.

 

erin

To view this discussion on the web visit https://groups.google.com/d/msgid/psych-data-standards/9022FD09-5D86-4438-95B0-4ADA5627643B%40gmail.com.

[Lisa Debruine]

Thanks for reminding me to get back to this thread, Erin.

I am also really interested in this but from the perspective of linking data and analyses to study structure. I’ve been working on a way to represent a full study such that you could use the JSON files to recreate the study exactly (e.g., in a version of https://github.com/debruine/experimentum, which is what the PSA study runs on, or something like Ruben’s formR), to automatically generate a description of the methods, and to provide a source for extracting meta-data from a literature.

I attached two examples of structure files from the PSA faces study (I didn’t send the whole study file because it includes 28 versions of the exp file and is a pain to try to view). Ruben, you and I should talk about how similar/different this is to formR experiment files and what we can do to coordinate.

I’d like to link this to psych-ds (since the PSA is adopting that standard anyways) and that to Daniël’s analysis structure (totally agree with Tal that GLM is better than specific functions for ttest/anova/etc), and then I think we’d have a full pipeline of things you’d want to have meta-data on to describe an experiment. 

I hope that made sense (I’m starting a new strategy of not worrying if I send half-assed ideas instead of just never responding to anything until my response is fully polished since it never actually gets there).

Cheers,

Lisa

To view this discussion on the web visit https://groups.google.com/d/msgid/psych-data-standards/BN6PR08MB3409CDC48B0546317C61DA1BF35F0%40BN6PR08MB3409.namprd08.prod.outlook.com.

[Ruben Arslan]

Hi Lisa,

lab.js also does full JSON exports. 

For formr, here is an example from one our diary studies:

https://osf.io/8qzrc/

Basically, ensuring full reproducibility of all settings in one file is the goal, right?
But to follow some common standard (except using an open format like JSON)? 

That seems very hard to achieve and I'm not sure whether the gains justify the 

effort..

Best,

Ruben

To view this discussion on the web visit https://groups.google.com/d/msgid/psych-data-standards/3EE5D5CC-347F-4BFE-B9F7-12DFD1EDDCE6%40glasgow.ac.uk.

[Ruben Arslan]

I very much agree with not letting the perfect be the enemy of the good. 

Formr export make studies completely reproducible (if you keep the platform constant).

I also implemented the XLSform standard which a number of other survey engines 

use.

But, when I run into the limits of the existing standards (happened a lot with XLSform),
I go for reproducibility, not compliance with the standard.

I'm not sure I'm up for finding a common standard for representing longitudinal surveys in formr

and reaction time experiments in lab.js – I don't see what for. 

For data, I totally see the gains.

Stefan and I are actually in the same workgroup, great guy :-) One of our local policy goals is

to get the MPI folks to store and share data in standardised ways.

On Thu, Mar 28, 2019 at 3:23 PM Russ Poldrack <pold...@gmail.com> wrote:

Achieving full exact reproducibility is likely impossible, since it depends not just on the code but also on the platform.  Sharing containers (as in BIDS Apps) gets you closer, but even there it's possible for differences to arise if things change in the OS kernel. But it's really important not to let the perfect be the enemy of the good (one of our mantras around here). Our experience in BIDS has shown that the goal of reproducibility is still very worthwhile even if you can only get to 80% coverage.    Unfortunately, figuring out what needs to be represented in order to get there and how to represent it in a useful way is really hard work.  Tal has the best insight into this from his recent work coming up with a BIDS statistical models standard for fMRI analysis, and we are having a meeting at Princeton next week to begin working on a BIDS standard for representing computational models.  BIDS has taken several years of work by dozens of people, and it's nowhere near its goal yet, but it has already shown its utility (e.g. in allowing automated data sharing via OpenNeuro).

BTW Ruben - Stefan Appelhof, the new BIDS maintainer, is at the MPI in Berlin, would be great for you to talk with him about your concerns if you haven't already.

cheers

russ

To view this discussion on the web visit https://groups.google.com/d/msgid/psych-data-standards/CALv3xztdoZaNJFPRTrs9CETtP4AJFHhZepZxWc%2BNdQJQYFom0A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Russell A. Poldrack
Albert Ray Lang Professor of Psychology
Professor (by courtesy) of Computer Science
Bldg. 420, Jordan Hall
Stanford University
Stanford, CA 94305

pold...@stanford.edu
http://www.poldracklab.org/

[Russ Poldrack]

Achieving full exact reproducibility is likely impossible, since it depends not just on the code but also on the platform.  Sharing containers (as in BIDS Apps) gets you closer, but even there it's possible for differences to arise if things change in the OS kernel. But it's really important not to let the perfect be the enemy of the good (one of our mantras around here). Our experience in BIDS has shown that the goal of reproducibility is still very worthwhile even if you can only get to 80% coverage.    Unfortunately, figuring out what needs to be represented in order to get there and how to represent it in a useful way is really hard work.  Tal has the best insight into this from his recent work coming up with a BIDS statistical models standard for fMRI analysis, and we are having a meeting at Princeton next week to begin working on a BIDS standard for representing computational models.  BIDS has taken several years of work by dozens of people, and it's nowhere near its goal yet, but it has already shown its utility (e.g. in allowing automated data sharing via OpenNeuro).

BTW Ruben - Stefan Appelhof, the new BIDS maintainer, is at the MPI in Berlin, would be great for you to talk with him about your concerns if you haven't already.

cheers

russ

On Thu, Mar 28, 2019 at 2:15 AM Ruben Arslan <ruben...@gmail.com> wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/psych-data-standards/CALv3xztdoZaNJFPRTrs9CETtP4AJFHhZepZxWc%2BNdQJQYFom0A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

[Russ Poldrack]

gotcha - agreed that one should try to adhere to existing standards as much as possible, but sometimes they are either too far off or too bloated. It should always be determined by concrete use cases. but I think that adhering to common file formats (json, tsv, etc) is really important so that anyone can easily read them.

glad to hear that you are already in touch with Stefan.

cheers

rp