SMT-LIB Benchmarks on Zenodo

[Mathias Preiner]

Dear all,

As decided last year at the SMT Workshop, we created an SMT-LIB
community on Zenodo for publishing and archiving SMT-LIB benchmark
releases. Zenodo provides an easy way to archive releases with a proper
DOI. All future and past releases will be uploaded to this community.

You can find the SMT-LIB community here:

https://zenodo.org/communities/smt-lib

We've started with publishing the most recent release from 2023, which
you can access here:

- incremental: https://zenodo.org/records/10607775
- non-incremental: https://zenodo.org/records/10607722

More releases will follow soon.

The GitLab server that hosted the benchmark repositories had to be taken
down due to security concerns, which was also the reason for the server
not being available over the past couple of weeks. We are currently in
the process of migrating the repositories to a different server, but at
the current time it is unclear whether access to the git repositories
will be available to everyone again.


Cheers,
Mathias

[BOBOT Francois]

Hi Mathias,

Le mardi 13 février 2024 à 12:25 -0800, Mathias Preiner a écrit :

As decided last year at the SMT Workshop, we created an SMT-LIB

community on Zenodo for publishing and archiving SMT-LIB benchmark

releases. Zenodo provides an easy way to archive releases with a proper

DOI. All future and past releases will be uploaded to this community.

Thank you for the work, it is great!

We've started with publishing the most recent release from 2023, which

you can access here:

- incremental: https://zenodo.org/records/10607775

- non-incremental: https://zenodo.org/records/10607722

Do you ponder between using multiple version (year) of one record (SMTLIB Benchmarks) or one record by year? It seems you went with the second, no?

Best,

-- 

François

[Mathias Preiner]

Hi Francois,

On 2/14/24 02:22, BOBOT Francois wrote:
>
> Do you ponder between using multiple version (year) of one
> record (SMTLIB Benchmarks) or one record by year? It seems you went with
> the second, no?
>

That's correct, we went with one record per year (one for incremental
and non-incremental each). This allows us to change involved maintainers
from year to year and also makes it easier for us to publish older
releases for archival purposes.

Cheers,
Mathias

[BOBOT Francois]

Le mercredi 14 février 2024 à 10:10 -0800, Mathias Preiner a écrit :

That's correct, we went with one record per year (one for incremental

and non-incremental each). This allows us to change involved maintainers

from year to year and also makes it easier for us to publish older

releases for archival purposes.

That's interesting, I though that a community would allow to change
curators and so would allow to change who can create new versions.

In https://zenodo.org/help/versioning , record versioning is expressly
describe as a way to manage tool versions (of course solver versions!).
But what you are describing seems to show that it would not be a good
fit for developer teams that can change. 

Are you sure that maintainers of a record in a community can't be changed?

Since we plan to use zenodo for smtcomp I would prefer to have clear
best practice.

Best,

-- 

François

[Mathias Preiner]

On 2/15/24 01:44, BOBOT Francois wrote:
>
> That's interesting, I though that a community would allow to change
> curators and so would allow to change who can create new versions.
>
> In https://zenodo.org/help/versioning
> <https://zenodo.org/help/versioning> , record versioning is expressly
> describe as a way to manage tool versions (of course solver versions!).
> But what you are describing seems to show that it would not be a good
> fit for developer teams that can change.
>
> Are you sure that maintainers of a record in a community can't be changed?
>
> Since we plan to use zenodo for smtcomp I would prefer to have clear
> best practice.
>

The main issue I see here is that once a version is uploaded you cannot
upload older versions for the same DOI. Since we started uploading the
2023 release we cannot upload older releases under the same DOI since
they'd show up as newer versions.

Further, if users cite the benchmarks release and use the "catch-all"
DOI it's less precise what they actually used since this could be any
version of SMT-LIB uploaded under this DOI.

For SMT-COMP I'd suggest you upload a dataset per year. I find it
strange do have one DOI for all SMT-COMP results and then you have to
sift through the different uploaded versions to actually find the
results for a specific year.


Mathias

[Clark Barrett]

One DOI per year sounds like a good best practice to me.

-Clark

--
You received this message because you are subscribed to the Google Groups "SMT-LIB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to smt-lib+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/smt-lib/26a2a2b8-6a0b-44bd-89bd-e4e6dd4765c0%40gmail.com.

[BOBOT Francois]

Le jeudi 15 février 2024 à 10:31 -0800, Clark Barrett a écrit :

One DOI per year sounds like a good best practice to me.

It is the case with one entry by year or one version by year. Each version has a specific DOI and page.

However, the possibility to change the list of authors for each version is still not clear for me.

Best,

-- 

François