MDC fails to update usage metrics to DataCite Commons (Dataverse v5.10.1)

[Data CUHK]

Hi everyone,

I have obtained the Json Web Token from DataCite (stored in secrets.yaml, referenced as a string-type parameter in config.yaml) and set "upload_to_hub" to true. 

But when I checked the DataCite commons profile after 24 hours, it still shows the same usage metrics (same views and downloads).

What could be missing here? Is there any verification I can do to see if the counter processor actually sending the correct info to DataCite?

Thanks,

Jack

[Data CUHK]

Update:

It seems that the main.py of counter processor keeps generating the same make-data-count-report.json, but I couldn't locate the script that generates this JSON file.

Jack

[James Myers]

Jack,

 

In general, several things have to be correctly configured and running for MDC to work:

• MDC Logging has to be enabled and producing logs
• Counter_processor has to be configured and running (usually via the counter_daily.sh script running as a daily cron job)
• Dataverse and DataCite need to be receiving the reports

 

It may be that DataCite Commons also has delays in how frequently it updates – I haven’t checked, but I believe they only expect complete monthly reports from some sources and so may only be updating their display monthly. If everything else is working, that might be a question for DataCite.

 

Nominally, if counter_daily.sh is being run, the local MDC counts in Dataverse should be updating daily. If that’s true, you can also check the counter_processor logs to see if the call to DataCite is being made and  didn’t have an error. The cal to push the report to DataCite uses their API. You should also be able to use the GET equivalent of that to retrieve the report from DataCite to confirm it.

 

(If the report you’ve included is what is getting produced, it looks like you’ve got counter_processor configured to process June 2023 repeatedly and not to do the current month – see the counter_daily.sh script for how the date gets set to be the current month/day.)

 

Note that there have been updates to counter_processor and Dataverse over time to handle large log files better and to fix some bugs too.

 

Hope that helps,

--Jim

 

 

From: dataverse...@googlegroups.com <dataverse...@googlegroups.com> On Behalf Of Data CUHK
Sent: Friday, August 7, 2026 5:24 AM
To: Dataverse Users Community <dataverse...@googlegroups.com>
Subject: [Dataverse-Users] Re: MDC fails to update usage metrics to DataCite Commons (Dataverse v5.10.1)

 

Update:

 

It seems that the main.py of counter processor keeps generating the same make-data-count-report.json, but I couldn't locate the script that generates this JSON file.

 

Jack

 

On Friday, August 7, 2026 at 3:08:26 PM UTC+8 Data CUHK wrote:

Hi everyone,

 

I have obtained the Json Web Token from DataCite (stored in secrets.yaml, referenced as a string-type parameter in config.yaml) and set "upload_to_hub" to true. 

 

But when I checked the DataCite commons profile after 24 hours, it still shows the same usage metrics (same views and downloads).

 

What could be missing here? Is there any verification I can do to see if the counter processor actually sending the correct info to DataCite?

 

Thanks,

Jack

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/dataverse-community/22f389ac-1c5f-4f7a-a289-e52c65777498n%40googlegroups.com.

[Data CUHK]

Jim,

Thanks for the suggestions. We have contacted DataCite and they say the latest report is generated in June 2023. For the Views count on DataCite commons, it shows 4130 views only, while we should have >1,000,000 views, (but the Works count is updating daily though).

The MCD logs are generated, which may indicates that the counter_daily.sh and counter_processor are running daily. The event time is recent, plus some datasets are deposited later than June 2023, which may indicates that the MDC part is running properly. 

However, only August logs (starting 1st Aug) are found. Is there a mechanism in Dataverse to auto-clean / move the mdc logs? If not, will the logs get ingested to generate the DataCite hub report on a daily basis even if DataCite is calculating on a monthly basis?

This is the counter_daily.sh, it doesn't seem to be hard-coded to process June 2023:

#! /bin/bash

COUNTER_PROCESSOR_DIRECTORY="/usr/local/counter-processor-0.1.04"
MDC_LOG_DIRECTORY="/usr/local/payara6/glassfish/domains/domain1/logs/mdc"

# counter_daily.sh

cd $COUNTER_PROCESSOR_DIRECTORY

echo >>/tmp/counter_daily.log
date >>/tmp/counter_daily.log
echo >>/tmp/counter_daily.log

# "You should run Counter Processor once a day to create reports in SUSHI (JSON) format that are saved to disk for Dataverse to process and that are sent to the DataCite hub."

LAST=$(date -d "yesterday 13:00" '+%Y-%m-%d')
# echo $LAST
YEAR_MONTH=$(date -d "yesterday 13:00" '+%Y-%m')
# echo $YEAR_MONTH
d=$(date -I -d "$YEAR_MONTH-01")
#echo $d
while [ "$(date -d "$d" +%Y%m%d)" -le "$(date -d "$LAST" +%Y%m%d)" ];
do
  if [ -f "$MDC_LOG_DIRECTORY/counter_$d.log" ]; then
#       echo "Found counter_$d.log"
  else
        touch "$MDC_LOG_DIRECTORY/counter_$d.log"
  fi
  d=$(date -I -d "$d + 1 day")
done

#run counter-processor as counter user

sudo -u counter YEAR_MONTH=$YEAR_MONTH python3 main.py >>/tmp/counter_daily.log

curl -X POST "http://localhost:8080/api/admin/makeDataCount/addUsageMetricsFromSushiReport?reportOnDisk=/tmp/make-data-count-report.json"

Jack

[Data CUHK]

Is my following understanding regarding the DataCite usage metrics updating procedure correct?

1. Counter processor enabled on Dataverse.
2. Some page landing / downloading events occurred for some datasets.
3. Counter processor records these events into the MDC logs,
4. On a daily basis, an MDC log is compiled into "make-data-count-report.json" and this JSON is sent to DataCite.
5. On a daily / monthly basis, DataCite updates the usage metrics according to the latest "make-data-count-report.json" received.

Jack

[Data CUHK]

Update:

I edited the config.yaml and changed the year_month to 2026-08:

An updated "make-data-count-report.json" (2026-08) is generated, but there is an exception in the JSON file, which was not seen in the old one (2023-06):

Does the JSON exception imply any important error?

Jack

[James Myers]

That exception is ~expected – you’re processing this month and it’s not over.

 

Note that if you change config.yaml, you need to update it every month. The counter_daily.sh script handles that automatically by getting the current date and finding the current month/year every time it runs. That said, setting the date yourself would be the way to process past months – as long as ou have the original logs, you can process any past months that haven’t been reported.

 

-- Jim

 

From: dataverse...@googlegroups.com <dataverse...@googlegroups.com> On Behalf Of Data CUHK
Sent: Monday, August 10, 2026 12:02 AM
To: Dataverse Users Community <dataverse...@googlegroups.com>
Subject: Re: [Dataverse-Users] Re: MDC fails to update usage metrics to DataCite Commons (Dataverse v5.10.1)

 

Update:

I edited the config.yaml and changed the year_month to 2026-08:

An updated "make-data-count-report.json" (2026-08) is generated, but there is an exception in the JSON file, which was not seen in the old one (2023-06):

To view this discussion visit https://groups.google.com/d/msgid/dataverse-community/66792c50-1b37-4b5e-9026-6854d100cbc6n%40googlegroups.com.

[James Myers]

Some notes about your emails:

 

Dataverse automatically creates the MDC logs of you have MDCLogging enabled. That happens regardless of whether you run counter_processor (and is also independent of whether you display the MDC counts.

 

When you run counter-processor, it reads the logs, creates a report and sends the results both to Dataverse and, if configured, to DataCite. Thus you’ll only see updates to the counts in Dataverse when counter-processor runs (and runs for the current date).

 

MDC values are usually much smaller than the local download counts as MDC ignores counts by known robots and counts multiple downloads of one item with a few seconds as one count, etc. A low MDC count could also indicate that counter-processor didn’t run for all months. That sounds like your issue as DataCite doesn’t have reports after 2023. Dataverse has MDC APIs that report by month which would allow you to see if there are many months with 0 counts.

 

Works counts are (I assume) from the individual DOIs being registered and is not connected to MDC/counter-processor at all.

 

 

-- Jim

To view this discussion visit https://groups.google.com/d/msgid/dataverse-community/c4fd557d-6fe1-4185-b8eb-1be4bc2d11b9n%40googlegroups.com.

[Data CUHK]

Dear Jim,

I'm manually changing the config.yaml to retrospectively generate the missing reports and upload to DataCite. 

I'm not sure why config.yaml was stuck at 2023-06 as counter_daily.sh is supposed to update that automatically.

Jack

[Data CUHK]

Dear Jim,

Is it correct that the mdc log directory only keeps the log in the current month? 

I couldn't see the logs for previous months, yet I see a bunch of database files (e.g. counter_db_2021-10.sqlite3) in the state directory in the counter processor directory that seem to be related to previous months.

Thanks,

Jack

[James Myers]

The raw daily logs that get processed are stored in whatever directory you assign for the :MDCLogPath setting (e.g. an mdc subdir below where Payara’s server.log sits). If you have those logs, you have all you need to report.

 

Counter-processor creates reports that it stores wherever the output_file is set in its config.yaml file. In my case, that’s  tmp/make-data-count-report so it is within the counter-processor directory tree. It’s possible that’s normally set to /tmp/make-data-count-report – the global temp directory which may get cleared periodically.

 

In general, those reports aren’t needed after they’ve been created, as the counter-daily script uses them immediately to send results to Dataverse and DataCite.

 

If you need to recreate/resend a given month, I believe you can edit the state file to remove that month. (I don’t think you have to delete the corresponding sqlite3 file, but I think you can. I think you can also just decrement the lastdayprocessed in the state file by 1 to make it only reprocess the last day (getting all the other days from the sqlite3 file). The last day’s report is cumulative for the month, so just reporocessing the last day and resending the report is all that’s needed to update Dataverse and DataCite.

 

Hope that helps,

To view this discussion visit https://groups.google.com/d/msgid/dataverse-community/fe6f7bd1-6f97-45c0-800b-a83d414ebb60n%40googlegroups.com.