Can Dataverse 4.x send notifications outside of Dataverse when a dataset is published

39 views
Skip to first unread message

MM - ADA

unread,
Jul 30, 2018, 7:08:10 PM7/30/18
to Dataverse Users Community
Hi - In a post from 2013, a user asked about RSS feeds from Dataverse. (https://groups.google.com/forum/#!msg/dataverse-community/MFNSc2EicTo/1fshLLptdm0J)

In his response, Phil Durbin mentioned "You might also consider configuring automatic tweets from your DVN installation: http://guides.thedata.org/book/4e-twitter-setup".



Is it possible in version 4.x to configure automatic tweets?

ADA would like to automatically send notifications to various accounts outside of Dataverse (ADA's Twitter account being one of them) when a dataset is published (new or new version). 

Is there functionality already in Dataverse that can be used to achieve this? If not directly (configuration), then indirectly (API call? something else?)

Thanks!

danny...@g.harvard.edu

unread,
Jul 31, 2018, 11:03:53 PM7/31/18
to Dataverse Users Community
This isn't something that's currently available in Dataverse 4. This could be neat as a new API endpoint for the use that you describe or for powering widgets or other places where recently updated content may be of interest. 

Marina McGale

unread,
Aug 1, 2018, 12:10:20 AM8/1/18
to dataverse...@googlegroups.com
I thought as much - thanks for letting us know.

We will do it a different way, then.

Cheers.
M.

From: dataverse...@googlegroups.com <dataverse...@googlegroups.com> on behalf of danny...@g.harvard.edu <danny...@g.harvard.edu>
Sent: Wednesday, 1 August 2018 1:03:52 PM
To: Dataverse Users Community
Subject: [Dataverse-Users] Re: Can Dataverse 4.x send notifications outside of Dataverse when a dataset is published
 
--
You received this message because you are subscribed to a topic in the Google Groups "Dataverse Users Community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dataverse-community/P9D28UFdcqI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dataverse-commu...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/bb2c18b4-020b-4c64-9c0f-bde57e1fae6c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Durand, Gustavo

unread,
Aug 1, 2018, 2:53:08 AM8/1/18
to dataverse...@googlegroups.com
One idea could be to use the publication workflows.

On Tue, Jul 31, 2018, 21:10 Marina McGale <Marina...@anu.edu.au> wrote:
I thought as much - thanks for letting us know.

We will do it a different way, then.

Cheers.
M.

From: dataverse...@googlegroups.com <dataverse...@googlegroups.com> on behalf of danny...@g.harvard.edu <danny...@g.harvard.edu>
Sent: Wednesday, 1 August 2018 1:03:52 PM
To: Dataverse Users Community
Subject: [Dataverse-Users] Re: Can Dataverse 4.x send notifications outside of Dataverse when a dataset is published
 
--
You received this message because you are subscribed to a topic in the Google Groups "Dataverse Users Community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dataverse-community/P9D28UFdcqI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dataverse-commu...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/bb2c18b4-020b-4c64-9c0f-bde57e1fae6c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

Steven McEachern

unread,
Aug 1, 2018, 5:51:58 AM8/1/18
to Dataverse Users Community
Hi Gustavo

Can you expand on this? (Where do we look for more on the workflows??)

Cheers
Steve

Michael Bar-sinai

unread,
Aug 1, 2018, 10:12:58 AM8/1/18
to Dataverse Users Community
Dataverse supports publications workflows - a series of steps that happen before or after a dataset is published. One of the built-in steps is a HTTP REST client. So, you could create a single-step workflow that notifies about a dataset being published by sending a, say,  HTTP POST request. Details and sample code are here:

http://guides.dataverse.org/en/4.9.1/developers/big-data-support.html#workflows

I'd use a post-publication workflow here, so that failures in notifying the server won't block publication. If you would like them to block the publication, run the workflow at the PrePublishDataset trigger.


Note that this is a global installation setting, and not per-dataverse. If that's too coarse, you could maybe create a simple HTTP server that gets the notifications from Dataverse and decides how to treat them using custom logic.

-- Michael

MM - ADA

unread,
Aug 1, 2018, 10:54:35 PM8/1/18
to Dataverse Users Community
That's a great idea. ADA is on v4.6.1 (I should have specified that instead of just 4.x) due to in-house code implementation but will leverage this workflow functionality once we move to the latest version.

Thanks for the feedback and advice.

M.

Pete Meyer

unread,
Aug 3, 2018, 5:28:13 PM8/3/18
to Dataverse Users Community
I *believe* that 4.6 was the first version to include pre/post publication workflows - so this may be something available in a 4.6.1 based branch.

Best,
Pete

Philip Durbin

unread,
Aug 6, 2018, 4:37:41 PM8/6/18
to dataverse...@googlegroups.com
Workflows were introduced in 4.8: https://github.com/IQSS/dataverse/pull/4116

My other thought is that you could poll Dataverse for changes, for newly published datasets or whatever. There's some discussion around polling for new stuff using the Search API at https://github.com/IQSS/dataverse/issues/2291

Or you could poll using OAI-PMH but I'm not very familiar with that protocol.


--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/4164ac45-7b7a-4731-aedc-28cfe2df4f29%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

MM - ADA

unread,
Aug 6, 2018, 9:34:22 PM8/6/18
to Dataverse Users Community
Thanks for all the ideas and feedback.

I did look at the search API but it doesn't seem to work with the publication date at the moment (I logged an issue on github:  https://github.com/IQSS/dataverse/issues/4908).

As another idea, we use metabase for querying the Dataverse database for reporting/stats. After writing the initial post to the community, I discovered that metabase can be used to create a "Pulse" that can be set to run hourly/daily/weekly to send an email with the results of a particular query appended as a csv file to whoever. 

So I set up a query (with the help of Julian: https://groups.google.com/forum/#!topic/dataverse-community/PyiaergrmCs) to get the new datasets (not harvested and not DEACCESSIONED) published in the previous 7 days. Then I set up metabase to run this every Friday to send the results to major stakeholders. So that is an easy solution for weekly emails to stakeholders.

We still need to send newly published datasets to twitter and can perhaps leverage metabase->email->twitter by setting up an hourly pulse but there are potential drawbacks to that solution. So instead we may run a (non-metabase) process hourly to poll the dataverse database for newly published datasets (using the queries Julian assisted with) and tweet those. It won't be immediate but might be the best solution at the moment.

Will update this post when we decide on best approach.

M.
Reply all
Reply to author
Forward
0 new messages