Re: AE. Artifact available

[Fabrice Rastello]

Dear Michel

I wrongly thought you were AE co-chair for CGO this year and when we postponed the discussion to later I did not figure out that it would become urgent for CGO earlier than expected...

I am adding Jubi and Roland (CGO-22 AE co-chair) to the thread + the mailing list Grigori created some years ago (did not receive any email since a long time -- is it still used?) as I believe this does not only concern CGO.

all the best

--Fabrice

Le 19/11/2021 à 17:42, Fabrice Rastello a écrit :

Dear Michel,

I just submitted the artifact for our paper and the requirement that was made by CGO's AE to get the "Archived badge" is not consistent with the requirement provided by ACM anymore:

"Archived?: Note that the author-created artifacts relevant to this paper will receive the ACM "artifact available" badge *only
if* they have been placed on a publicly accessible archival repository such as Zenodo, FigShare or Dryad. A DOI will be then assigned to their artifacts and must be provided here!
Personal web pages, Google Drive, GitHub, GitLab and BitBucket are not accepted for this badge. Authors can provide this link at the end of the evaluation." 

As far as I understand, DOI is not anymore imposed by ACM: https://www.acm.org/publications/policies/artifact-review-and-badging-current, and I think this is a good thing...

As an example, Inria suggests to use Sofware Heritage that *does not* provide a DOI. This is I believe much better:

1. dedicated howto page, explaining how to archive and reference your source code in a few clicks, and how to cite it using biblatex-software
2. comparison of a source code as archived on different platforms
• in Software Heritage: source code with full version history, browsable and refereancable, possibility to highlight the key algorithm in the code
• in Zenodo: zip file for a given version

As providing a DOI was not required for the initial submission (but only to get the "Archivable" badge), don't you think we should accept other platforms than Zenedo like, as allowed by ACM?

If you agree, I think we should suggest other people who are handling AE for other conferences to also update the rules related to this badge.

All the best

--Fabrice

--

Fabrice Rastello
Inria Research Director
CORSE Inria research team leader
---
Antenne Inria GIANT
DRT/LETI/DACLE - Batiment 50C - Bur. C339
Minatec Campus
17 Rue des Martyrs
38054 GRENOBLE cedex
Office: +33 4 38-78-16-97
Mobile: +33 6 43-98-34-57

[Tatiana Shpeisman]

Jubi and Roland,

Do you have any thoughts on Fabrice question?

Thanks,

Tatiana

[Michel Steuwer]

Hi all,

Sorry for answering late.

Fabrice question was if we require a DOI for the artifacts.

The rules of ACM have not changed and do not strictly require a DOI.

ACM says that:

“Author-created artifacts relevant to this paper have been placed on a

   publically accessible archival repository. A DOI or link to this repository

   along with a unique identifier for the object is provided.” 

The Software Heritage service that Fabrice mentioned is clearly acceptable as

a “publically accessible archival repository” and they assign a “PID” which is

the “unique identifier for the object” that ACM demands.

I think for the future we should update our formulation slightly to account for

this option. Maybe we can even recommend Software Heritage as our preferred

option — it certainly looks like a very good service.

In regards to how the available badge is assigned in practice, this is the job of the

publisher. I can’t remember from the past if we had to provide a DOI or a link to

the artifact in the form.

We should communicate with the publishing team and see how they handle the process.

Best,

Michel

[Jubi Taneja]

Hi Michel,

Thanks for your email. I overlooked the group of email ids that were included originally in the email and just replied initially to Roland and Fabrice. I will copy over the email here for you all that I sent to both of them yesterday.

I also agree with you that we should update AE guidelines to sync with the ACM policy for archiving the source code and open up more options for the authors.

I am unclear on who sets the rules for this part of the available badge policy and do we have to abide by this rule as well?

The below text is from https://www.acm.org/publications/policies/artifact-review-and-badging-current 

• Artifacts do not need to have been formally evaluated in order for an article to receive this badge. In addition, they need not be complete in the sense described above. They simply need to be relevant to the study and add value beyond the text in the article. Such artifacts could be something as simple as the data from which the figures are drawn, or as complex as a complete software system under study.

I have expressed my thoughts in the email below.
---------------------------------------------------
Dear Fabrice,

Thanks for bringing our attention to this important point.

I was out of the office and thus delayed in getting back to your email.

I was not aware of the updated guidelines regarding the Available badge criterion. I had a discussion with the conference publishing team a while ago but the updated rules never came up in that brief discussion. It would be hard to reset the criterion for CGO 2022 as the badges are already assigned. However, I agree with you that moving forward we should adopt the new rules and make the availability of artifacts as easy as possible.

A few questions/thoughts that came to my mind:

1. ACM guidelines also allow institutional repository. What would be a good example of that? For instance, Google has a public Github repo and all open-source projects in google are released under that repo. However, Github, Gitlab is strictly not allowed, and personal webpages are also not allowed. Sometimes research groups in the university have their web pages and they release their projects on that website. Do we allow that? This is a bit ambiguous.

2. Another criterion for an available badge is not to pass any formal evaluation from the Artifact Evaluation Committee. I cannot go and change ACM rules but just wanted to share my own thoughts on this. I believe the criterion that we followed until now (CGO 2022 AE process) was much better where the artifact at least needs to pass the basic functionality test to be considered for the available badge. The idea is if the artifacts are broken and not working, what's the point to keep them available? Are we trying to say that the rest of the research community has to chase down the authors in order to use their artifact for their own purposes? What are your thoughts on this? I would also need to discuss this with the General Chair of CGO 2022 and other PCs if we want to adopt this criterion for CGO 2023 or not.

However, I agree with you on adopting different platforms to release the source code and spread the word to other conferences as well.

Thanks,

Jubi 

---------------------------------------------------

Jubi

[Michel Steuwer]

Hi Jubi,

I think the situation grading the available badge is pretty clear:

ACM sets the rules and they say that for the available badge there has to be no review - and this is also what we have implemented in the prior years at CGO:

The available badge is assigned by the publisher and this is open to all papers, not just the once that went though artifact evaluation.

I personally think that this can lead to confusing understanding of the badges.

Especially it is currently not possible to distinguish if an artifact has been attempted to be evaluated (but didn’t get the functional badge) or not.

But this is not something that we will be able to change alone.

We can try to help educate the community what the green badge on its own means (i.e. not that much).

Cheers,

Michel

[Jubi Taneja]

Hi Michel,

Thanks for clarifying the available badge criterion. My impression was that we as AE Co-Chairs send out the list of badges awarded for each paper that submitted their artifact for evaluation and the conference publishing team reaches out to that list of authors only. However, I am not sure if they reach out to all the authors including the ones who did not submit any artifacts for evaluation. I would need to check with the publishing team on that to understand what's going on so far. To my knowledge, I have not seen any paper with just a Green (Available) Badge.

I exactly agree with your thoughts that it is very confusing to only award a paper with a Green badge because it does not say explicitly if it failed the other badges criterion or it didn't go for any evaluation at all. What are your thoughts on pursuing this with the publishing team and figuring out how to update these policies regarding the Available badges? I can work on figuring out the appropriate point of contact on the ACM side as well because they set the rules. My thoughts are that we should at least bring up our concerns and see what the ACM team thinks.

There was a small situation in PLDI SRC last year where judges wanted to award Gold Medal to two students and the rules did not align with this decision, but, discussing the issue with ACM folks helped move past that barrier. So, I am hopeful here with the Artifact Evaluation process as well. :) 

Thanks,

Jubi

[Roland Leißa]

Hi all,

sorry for answering late, but I've been on vacation.

I agree with Jubi here and I also find the current badging scheme highly
confusing.

Best regards,
Roland

On 08.01.22 00:03, Jubi Taneja wrote:
> Hi Michel,
>

>> <mailto:ju...@cs.utah.edu>> wrote:
>>
>> Hi Michel,
>>
>> Thanks for your email. I overlooked the group of email ids that
>> were included originally in the email and just replied initially
>> to Roland and Fabrice. I will copy over the email here for you all
>> that I sent to both of them yesterday.
>>
>> I also agree with you that we should update AE guidelines to sync
>> with the ACM policy for archiving the source code and open up more
>> options for the authors.
>>
>> I am unclear on who sets the rules for this part of the
>> available badge policy and do we have to abide by this rule as well?
>>
>> The below text is from
>> https://www.acm.org/publications/policies/artifact-review-and-badging-current

>> <https://www.acm.org/publications/policies/artifact-review-and-badging-current>
>>
>>
>> *
>>
>> /Artifacts do not need to have been formally evaluated in

>> order for an article to receive this badge. In addition, they
>> need not be complete in the sense described above. They simply
>> need to be relevant to the study and add value beyond the text
>> in the article. Such artifacts could be something as simple as
>> the data from which the figures are drawn, or as complex as a

>> complete software system under study./

>>
>> I have expressed my thoughts in the email below.
>> ---------------------------------------------------
>> Dear Fabrice,
>>
>> Thanks for bringing our attention to this important point.
>>
>> I was out of the office and thus delayed in getting back to your
>> email.
>>
>> I was not aware of the updated guidelines regarding the Available
>> badge criterion. I had a discussion with the conference publishing
>> team a while ago but the updated rules never came up in that brief
>> discussion. It would be hard to reset the criterion for CGO 2022
>> as the badges are already assigned. However, I agree with you that
>> moving forward we should adopt the new rules and make the
>> availability of artifacts as easy as possible.
>>
>> A few questions/thoughts that came to my mind:
>> 1. ACM guidelines also allow institutional repository. What would
>> be a good example of that? For instance, Google has a public

>> Github repo <https://github.com/google> and all open-source

>> <mailto:Michel....@glasgow.ac.uk>> wrote:
>>
>> Hi all,
>>
>> Sorry for answering late.
>>
>> Fabrice question was if we require a DOI for the artifacts.
>>
>> The rules of ACM have not changed and do not strictly require
>> a DOI.
>> ACM says that:
>> “Author-created artifacts relevant to this paper have been
>> placed on a

>>    publically accessible archival repository. /A DOI or link
>> to this repository/
>> /   along with a unique identifier for the object is provided./”

>>
>> The Software Heritage service that Fabrice mentioned is
>> clearly acceptable as
>> a “publically accessible archival repository” and they assign
>> a “PID” which is
>> the “unique identifier for the object” that ACM demands.
>>
>> I think for the future we should update our formulation
>> slightly to account for
>> this option. Maybe we can even recommend Software Heritage as
>> our preferred
>> option — it certainly looks like a very good service.
>>
>> In regards to how the available badge is assigned in practice,
>> this is the job of the
>> publisher. I can’t remember from the past if we had to provide
>> a DOI or a link to
>> the artifact in the form.
>>
>> We should communicate with the publishing team and see how
>> they handle the process.
>>
>>
>> Best,
>> Michel
>>
>>> On 5 Jan 2022, at 06:21, Tatiana Shpeisman

[Michel Steuwer]

Hi Jubi,

My impression was that we as AE Co-Chairs send out the list of badges awarded for each paper that submitted their artifact for evaluation and the conference publishing team reaches out to that list of authors only.

The process you describe is correct for the functional and evaluated badges, but I believe that for the green available badge there is a field for a link (or DOI) in the form that all authors can fill out as part of the camera ready process. You could check this with conference publishing if you want to.

I can work on figuring out the appropriate point of contact on the ACM side as well because they set the rules. My thoughts are that we should at least bring up our concerns and see what the ACM team thinks.

I think that would be helpful. Grigori Fursin (cc’d in these mails) might know how to ask.

Best,

Michel

--
You received this message because you are subscribed to the Google Groups "Artifact Evaluation" group.
To unsubscribe from this group and stop receiving emails from it, send an email to artifact-evalua...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/artifact-evaluation/CABxY-SKHa42Fj_rB6wobfVx3dqm%2Bsxf0tYPrNDSYJ-51go2rJQ%40mail.gmail.com.

[Fabrice Rastello]

Don't you think we can just, as a conference, decide not to attribute a given badge to any paper of the conference that we believe would me more confusing/counterproductive than useful?

Concerning the DOI for ACM DL, we need to update the web page of CGO accordingly to make it clear that DOI are *not* required and that actually Software Heritage ID is recommended. ACM DL seems to have limitations when it is not a DOI (did not understood all the details from my exchanges with CPS) but I am optimistic this will change in a near future.

--fab

Le 12/01/2022 à 10:44, Michel Steuwer a écrit :

Hi Jubi,

My impression was that we as AE Co-Chairs send out the list of badges awarded for each paper that submitted their artifact for evaluation and the conference publishing team reaches out to that list of authors only.

The process you describe is correct for the functional and evaluated badges, but I believe that for the green available badge there is a field for a link (or DOI) in the form that all authors can fill out as part of the camera ready process. You could check this with conference publishing if you want to.

I can work on figuring out the appropriate point of contact on the ACM side as well because they set the rules. My thoughts are that we should at least bring up our concerns and see what the ACM team thinks.

I think that would be helpful. Grigori Fursin (cc’d in these mails) might know how to ask.

3.