Problems with the download count on homepage (Dataverse v5.10.1)

[Data CUHK]

Hi everyone,

Currently we have three figures:

1. Download count on DataCite Commons - DataCite claims it is counting unique download, calculated from MDC reports.
2. Download count on dataset page - we think it is counting total download.
3. Download count on repository homepage - not sure if it is counting total or unique.

We obtained the total download counts from all dataset pages using web-scraping and it shows a total of 34,000+ downloads; while on the homepage, it shows 83,000+ downloads (see the screenshot below).

Where does this download count come from?

Thanks,

Jack

[Julian Gautier]

Hi Jack,

Could the 83,000+ downloads in your screenshot and on the homepage of the CUHK Research Data Repository be the regular Dataverse download counts and not the MDC download counts? That's what I suspect.

I think the first two things on your list - about what a dataset's download count on DataCite Commons means versus a dataset's download count shown on your repository's dataset page - are also what we discussed in the Google Groups thread at https://groups.google.com/g/dataverse-community/c/eXX1LN0rLrw/m/ituQDQY6EgAJ. Could you check that thread out? I'm very curious what you think.

[Data CUHK]

Hi Julian,

According to my understanding (please feel free to correct any misunderstanding from me), MDC logs events and replace the regular counter (out-of-box) from Dataverse. 

These two figures are the only two figures we have to consider after enabling MDC:

• MDC unique: reported to DataCite
• MDC total: updated and displayed on Dataverse interface (especially dataset pages). 

Currently we enabled uploading to DataCite hub again and I have checked the MDC logs and monthly database against the increment on usage metrics shown on the DataCite commons profile as well as the DataCite report via API. 

The event counts in the MDC logs are reflected exactly the same on the DataCite report via API and on the DataCite commons profile.

I have checked the Comparing MDC download counts from Dataverse repositories to MDC download counts from DataCite API - Google Sheets you created and found the datasets in our CUHK RDR (thanks for creating this spreadsheet, it is helpful).

Since I have counted the download count of all published datasets via web-scrapping, it should show the MDC total of download count. Yet the download count on the homepage is way bigger than this web-scrapped figure. So, I wonder if the regular counter is still running and has been updating the homepage metrics with its data / with the sum of it data + MDC total? And if so, where should I locate the log for regular counter?

Jack

[Julian Gautier]

Hey Jack,

Definitely, that's my understanding, too, and what I've seen so far. By default, with MDC enabled, Dataverse logs MDC events and replaces the regular counter (out-of-box), specifically the download counts. We write that in this section of the guides too: https://guides.dataverse.org/en/5.10.1/admin/make-data-count.html#enable-or-disable-display-of-make-data-count-metrics

About the two figures you have to consider after enabling MDC, I think you're right too. I'd call what you wrote a simplified version of the differences discussed in the Google Groups thread at https://groups.google.com/g/dataverse-community/c/eXX1LN0rLrw/m/ituQDQY6EgAJ.

Even when we enable MDC counts in a Dataverse repository, the regular, out-of-box download counter is always running, since it's associated with the guestbooks. The API call https://researchdata.cuhk.edu.hk/api/info/metrics/downloads returns a count that roughly all regular download counts of datasets in the CUHK Research Data Repository. The count there is 81,027 at the time of this writing, and I think it's different from the 83,027 we see on the homepage at https://researchdata.cuhk.edu.hk/dataverse/root because of the caching that Dataverse is doing so that these /metrics API endpoints return metrics faster.

So I'm even more certain that the downloads count at the top of the homepage is only the regular, out-of-box download counts, even for Dataverse repositories that have enabled MDC counts. The QDR repository also has MDC enabled, and its home page shows download counts that I believe are only the regular counts. The download count on the homepage at https://data.qdr.syr.edu roughly matches what's returned by the API call https://data.qdr.syr.edu/api/info/metrics/downloads.

You can use the /metrics APIs to get the regular download counts. They're described at https://guides.dataverse.org/en/5.10.1/api/metrics.html.

You can also query the repository's database if you have access to it. The Dataverse community maintains a Google Doc with example database queries, and its "File downloads, subsets and explorations" section includes examples that mostly use the database's "guestbookresponse" table to get regular download counts in different ways.

Let me know if you have any questions about this!

And thanks for encouraging discussion with this thread! I plan to continue helping test the design of how Dataverse records and communicates these MDC counts and regular download counts, as we plan to enable MDC counts on Harvard Dataverse and later on as we consider how to include them on the Modern UI version of Dataverse, and this discussion has been helpful, especially about what the homepage shows.