metrics for PLOS

[Alexander Garcia Castro]

http://scholarlykitchen.sspnet.org/2013/06/20/the-rise-and-fall-of-plos-ones-impact-factor-2012-3-730/

Perhaps it is time to re-think metrics like those from JCR and
understand the quality of research as something else. probably related
to reproducibility of science.

--
Alexander Garcia
http://www.alexandergarcia.name/
http://www.usefilm.com/photographer/75943.html
http://www.linkedin.com/in/alexgarciac

[rebholz]

Hi,

it is funny to see that Thompson still sticks to the 2 years. Automatic
processing can pull out several metrices. You can be very negative
about Google Scholar, but it delivers already 4 measures at least, other
tools even more (Harzing). One problem remains: who oversees all
content to make the index calculation at least complete.

Ciao,
-drs-

On 09.12.2013 15:17, Alexander Garcia Castro wrote:
> http://scholarlykitchen.sspnet.org/2013/06/20/the-rise-and-fall-of-plos-ones-impact-factor-2012-3-730/
>
> Perhaps it is time to re-think metrics like those from JCR and
> understand the quality of research as something else. probably related
> to reproducibility of science.
>

--

D. Rebholz-Schuhmann, MD, PhD - Visiting Research Group Leader
EBI, Wellcome Trust Genome Campus, Cambridge CB10 1SD (UK)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Twitter: jbiomedsem

[Alexander Garcia Castro]

http://www.voxeu.org/article/our-uneconomic-methods-measuring-economic-research

this one is more interesting. this one goes straight into what I think
is a key part of the problem, authorship contribution.


a good paper is one that can be reproduced. a lot of papers, probably
the vast majority, cant be reproduced. there is no index of
reproducible science as far as I know. no index of reprodicible
papers.

> --
> You received this message because you are subscribed to the Google Groups
> "Beyond the PDF" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to beyond-the-pd...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Paul Groth]

I think the interesting is not reproducibility but what Cameron Neylon
identifies as "use".

Did this result get used in someone else's science? for some product?
for someone's thinking?

In some since citation is a proxy for this but is not detailed enough.

We need finer resolution details about use and about contribution -
it's not the number of authors a paper has but what everybody has
contributed and how their products were used. (This is where I
disagree with Liebowitz)

Writing about something is important, but so is coding, and running
the experiment, and talking to policy makers, and shaping the research
theme, and distilling the information in a blog, and convincing the
public that it's worthwhile. Let's make it explicit.

regards
Paul

On Mon, Dec 9, 2013 at 4:02 PM, Alexander Garcia Castro
<alexg...@gmail.com> wrote:
> http://www.voxeu.org/article/our-uneconomic-methods-measuring-economic-research
>
> this one is more interesting. this one goes straight into what I think
> is a key part of the problem, authorship contribution.
>
>
> a good paper is one that can be reproduced. a lot of papers, probably
> the vast majority, cant be reproduced. there is no index of
> reproducible science as far as I know. no index of reprodicible
> papers.
>
>
>
> On Mon, Dec 9, 2013 at 9:50 AM, rebholz <reb...@ebi.ac.uk> wrote:

>> --
>> You received this message because you are subscribed to the Google Groups
>> "Beyond the PDF" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to beyond-the-pd...@googlegroups.com.
>> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

> --
> Alexander Garcia
> http://www.alexandergarcia.name/
> http://www.usefilm.com/photographer/75943.html
> http://www.linkedin.com/in/alexgarciac
>

> --
> You received this message because you are subscribed to the Google Groups "Beyond the PDF" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to beyond-the-pd...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.



--

-----------------------------------------------------------------------------------
Dr. Paul Groth (p.t....@vu.nl)
http://www.few.vu.nl/~pgroth/
Assistant Professor
- Web & Media Group | Department of Computer Science
- The Network Institute
VU University Amsterdam

[rebholz]

Sounds like a pretty comprehensive tracking system. I guess, you are
thinking not only about some counting of "items" but some kind of
control, i.e. for a peer review whenever somebody is for example talking
to policy makers. This is not about some teatime talk, this is about
some impact and progress.

Publications are unique since they give a pretty comprehensive statement
about a subject including a final judgment.
Data is unique since it gives the results linked to an experimental
setting, well the semantics of the experimental setting has to be clear.
Software & Ontologies have a functional component.

It may be difficult to find similar qualities for other type of "items".

Cheers,
-drs-

On 09.12.2013 18:11, Paul Groth wrote:
> Writing about something is important, but so is coding, and running
> the experiment, and talking to policy makers, and shaping the research
> theme, and distilling the information in a blog, and convincing the
> public that it's worthwhile. Let's make it explicit.

[Paul Groth]

I guess I like what I've seen in many papers where people just write what they contributed - if we could have some common vocabulary for this that would get used we could at least see, which would be a good start.

If you look at impact story we're starting to see some of this. There's also some nice work on people metrics going on in the same line.

Paul

> On Dec 9, 2013, at 19:28, rebholz <reb...@ebi.ac.uk> wrote:
>
> Sounds like a pretty comprehensive tracking system. I guess, you are
> thinking not only about some counting of "items" but some kind of
> control, i.e. for a peer review whenever somebody is for example talking
> to policy makers. This is not about some teatime talk, this is about
> some impact and progress.
>
> Publications are unique since they give a pretty comprehensive statement
> about a subject including a final judgment.
> Data is unique since it gives the results linked to an experimental
> setting, well the semantics of the experimental setting has to be clear.
> Software & Ontologies have a functional component.
>
> It may be difficult to find similar qualities for other type of "items".
>
> Cheers,
> -drs-
>
>> On 09.12.2013 18:11, Paul Groth wrote:

>> Writing about something is important, but so is coding, and running
>> the experiment, and talking to policy makers, and shaping the research
>> theme, and distilling the information in a blog, and convincing the
>> public that it's worthwhile. Let's make it explicit.
>

[Gudmundur A. Thorisson]

The SCoRO ontology is a pretty good attempt at capturing many of the nuances of contributorship, without not being overly complex (common sense stuff, basically). Importantly, it teases apart a person’s role vs. the type of contribution he/she made to the scholarly work:

The Scholarly Contributions and Roles Ontology - http://purl.org/spar/scoro

> ontology for use by authors and publishers for describing the contributions that may be made and the roles that may be held by a person with respect to a journal article or other publication, and by research administrators for describing contributions and roles with respect to other aspects of scholarly research.

Some examples from SCoRO:
* authorship roles: article guarantor, consortium author, corresponding author, illustrator, photographer, principal author, pro:author, senior author
* types of authorship contribution: approves final manuscript, prepares illustrations, prepares supplementary information, publishes data, revises manuscript, writes manuscript draft
* types of intellectual contribution: analyses data, conceives project, designs experiments, formulates research questions, interprets results, leads investigation, provides advice, undertakes modelling


PS Personal observation: I continue to be amazed the solid SPAR stuff and other related ontology work doesn’t get used and built upon, and instead folks are reinventing the wheel left, right and center (e.g. CASRAI http://dictionary.casrai.org).



Mummi

--
Gudmundur A. Thorisson
http://gthorisson.name | http://orcid.org/0000-0001-5635-1860 | http://twitter.com/gthorisson

[Sarven Capadisli]

On 12/09/2013 10:04 PM, Paul Groth wrote:
> I guess I like what I've seen in many papers where people just write
> what they contributed - if we could have some common vocabulary for
> this that would get used we could at least see, which would be a good
> start.
>
> If you look at impact story we're starting to see some of this.
> There's also some nice work on people metrics going on in the same
> line.

I don't want to sound like an old record here and elsewhere, but how
about we start with the basics and get a machine friendly (and no PDF is
not something that's "machine friendly") out door regardless of how
imperfect it may be?

<Insert past discussions on starting with basic XHTML+RDFa>

There are plenty of vocabularies in existence to help us accomplish all
sorts of stuff. Today! Like, towards your next paper for instance! ;)

Until then, I don't see what good of such hypothetical common vocabulary
(that's so fantastically cool for mining "influences" in research papers
at such a granular level) will do.

-Sarven

[Rebholz-Schuhmann]

Automatic Style analysis + Discourse analysis + Markup of contributions
=> statistical analysis ...

How does this sound. But, I am afraid this goes into the AI corner.

Well, the bigger problem: apart from the scientific interest (= to make
this work und use it) and the scientist's interest (= hey look what I
have contributed), what is the standard use case and what would be the
killer App?
Publishers want it: more data more clicks/attention/money (including OA
journals)

What user would profit how?
I don't want to be negative but apart from some want-to-be-Professors,
there may not be many users.

Have a nice day,
-drs-

[Sarven Capadisli]

Hi Rebholz-Schuhmann,

Let me ask you this: if you write papers, will you contribute by
publishing your next paper in HTML(+RDFa)? otherwise, will you encourage
the ones around you to do it?

-

The importance or at the very least, the curiosity to make research
manuscripts more machine-friendly (without sacrificing the UX for the
humans) is fairly well-documented and worked on by different communities.

There are of course some communities out there still struggling or
resisting to adapt. At the core of it, the real challenges were never
about the technology. It is still very much a social problem.

So, I strongly disagree with you on investing any resources to talk
about or building a "killer app" to prove a point.

If *this* or communities like the SemWeb don't eat their own dogfood, to
make an honest effort to move things forward, why should we expect any
publisher to go out of their to bother with it?

We don't need killer apps or a vocabulary to rule all other vocabularies
to start somewhere. If we want to make a difference, now, we already
have the technology stack, tools, and the expertise to do plenty. We can
surely figure out the finer details or build on the "killer apps" as we go.

-Sarven

[rebholz]

Hi Sarven,

not sure which point you are making. Now you are talking about some
open formats, but before we discussed in which way the impact factor
should be calculated and - in a wider context - how this could be done
with automatic means. I guess open formats contribute, but won't
automatically generate reliable impact factors. A more general
argument: a killer app always helps to get something going, but is not a
necessary condition. I won't comment formats now, but my preferences lie
with XML coupled to whatever transformation to Html, Pdf, Doc, whatever,
and I have done so in the past (check out PaperMaker).

My personal opinion: we have enough open formats, we are - only slowly -
working towards an efficient infrastructure that covers all components
of the publication pipeline (including archiving and "marketing" our own
publications). The next big trouble could be the increasing number of
want-to-be-publishers providing as much as possible at as little of a
quality with as little costs possible.

Hope this helps,
-drs-