My first comment on EPA's proposed transparency regulation

[David Wojick]

For those who might be interested. Note that while this is a US action it is potentially a precedent for other Governments, so it deserves some international attention.

In particular, at the end I begin to discuss the actual open science procedures needed for implementation. Below is that section, which assumes some familiarity with the EPA proposal. This will not be my last submission so I am wide open to ideas.

The proposal and other stuff are here: https://www.regulations.gov/docket?D=EPA-HQ-OA-2018-0259

David
http://insidepublicaccess.com/

EPA usability and use procedures

Not using a given research result looks to be procedurally easy. EPA simply does not cite the research in its justification documents or statements. They might even create a new support document that specifically identifies the research used to justify the regulation.

Determining which research is not usable is the difficult part. This looks very much like what is called a "certification" procedure. That is, EPA will need to certify that each piece of research it wants to use is properly available for attempted replication. Note that neither attempted or successful replication is required under this procedural transparency rule. All that is required for EPA use is that the research be properly available for possible attempted replication.

The best way, perhaps the only way, to do this is for EPA to contact the researchers and determine that the research is properly available. Another possibility is to establish a practice of self certification among those researchers who desire that their research be usable by EPA. Or certification of EPA usability might be done by third parties, such as scholarly societies. Such certification might even become a condition of funding with some funders.

In any case, what EPA needs to do is specify the requirements of proper availability as needed for usability. This might be a simple checklist of questions, or perhaps something more complex. There is already considerable discussion and action within the open science community, as to just what availability for the purpose of replication requires. EPA should draw on this existing work and expertise.
 
The basic requirements are not that mysterious. The data and codes need to be available and properly documented. The procedures followed and decisions made along the way need to be properly explained. (There is already a relevant literature on this topic in some fields, especially biomedical.) Research that cannot be certified is then excluded.