Can Dataverse 4.x send notifications outside of Dataverse when a dataset is published

[MM - ADA]

Hi - In a post from 2013, a user asked about RSS feeds from Dataverse. (https://groups.google.com/forum/#!msg/dataverse-community/MFNSc2EicTo/1fshLLptdm0J)

In his response, Phil Durbin mentioned "You might also consider configuring automatic tweets from your DVN installation: http://guides.thedata.org/book/4e-twitter-setup".

Is it possible in version 4.x to configure automatic tweets?

ADA would like to automatically send notifications to various accounts outside of Dataverse (ADA's Twitter account being one of them) when a dataset is published (new or new version). 

Is there functionality already in Dataverse that can be used to achieve this? If not directly (configuration), then indirectly (API call? something else?)

Thanks!

[danny...@g.harvard.edu]

This isn't something that's currently available in Dataverse 4. This could be neat as a new API endpoint for the use that you describe or for powering widgets or other places where recently updated content may be of interest. 

[Marina McGale]

I thought as much - thanks for letting us know.

We will do it a different way, then.

Cheers.

M.

---

From: dataverse...@googlegroups.com <dataverse...@googlegroups.com> on behalf of danny...@g.harvard.edu <danny...@g.harvard.edu>
Sent: Wednesday, 1 August 2018 1:03:52 PM
To: Dataverse Users Community
Subject: [Dataverse-Users] Re: Can Dataverse 4.x send notifications outside of Dataverse when a dataset is published

 

--
You received this message because you are subscribed to a topic in the Google Groups "Dataverse Users Community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dataverse-community/P9D28UFdcqI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dataverse-commu...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/bb2c18b4-020b-4c64-9c0f-bde57e1fae6c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Durand, Gustavo]

One idea could be to use the publication workflows.

On Tue, Jul 31, 2018, 21:10 Marina McGale <Marina...@anu.edu.au> wrote:

I thought as much - thanks for letting us know.

We will do it a different way, then.

Cheers.

M.

---

From: dataverse...@googlegroups.com <dataverse...@googlegroups.com> on behalf of danny...@g.harvard.edu <danny...@g.harvard.edu>
Sent: Wednesday, 1 August 2018 1:03:52 PM
To: Dataverse Users Community

Subject: [Dataverse-Users] Re: Can Dataverse 4.x send notifications outside of Dataverse when a dataset is published

 

--

You received this message because you are subscribed to a topic in the Google Groups "Dataverse Users Community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/dataverse-community/P9D28UFdcqI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to dataverse-commu...@googlegroups.com.
To post to this group, send email to dataverse...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/bb2c18b4-020b-4c64-9c0f-bde57e1fae6c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-commu...@googlegroups.com.

To post to this group, send email to dataverse...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/MEXPR01MB12076F9B666D341F6E60AFE1D72D0%40MEXPR01MB1207.ausprd01.prod.outlook.com.

[Steven McEachern]

Hi Gustavo

Can you expand on this? (Where do we look for more on the workflows??)

Cheers
Steve

[Michael Bar-sinai]

Dataverse supports publications workflows - a series of steps that happen before or after a dataset is published. One of the built-in steps is a HTTP REST client. So, you could create a single-step workflow that notifies about a dataset being published by sending a, say,  HTTP POST request. Details and sample code are here:

http://guides.dataverse.org/en/4.9.1/developers/big-data-support.html#workflows

I'd use a post-publication workflow here, so that failures in notifying the server won't block publication. If you would like them to block the publication, run the workflow at the PrePublishDataset trigger.

http://guides.dataverse.org/en/4.9.1/api/native-api.html#workflows

Note that this is a global installation setting, and not per-dataverse. If that's too coarse, you could maybe create a simple HTTP server that gets the notifications from Dataverse and decides how to treat them using custom logic.

-- Michael

[MM - ADA]

That's a great idea. ADA is on v4.6.1 (I should have specified that instead of just 4.x) due to in-house code implementation but will leverage this workflow functionality once we move to the latest version.

Thanks for the feedback and advice.

M.

[Pete Meyer]

I *believe* that 4.6 was the first version to include pre/post publication workflows - so this may be something available in a 4.6.1 based branch.

Best,

Pete

[Philip Durbin]

Workflows were introduced in 4.8: https://github.com/IQSS/dataverse/pull/4116

My other thought is that you could poll Dataverse for changes, for newly published datasets or whatever. There's some discussion around polling for new stuff using the Search API at https://github.com/IQSS/dataverse/issues/2291

Or you could poll using OAI-PMH but I'm not very familiar with that protocol.

--

You received this message because you are subscribed to the Google Groups "Dataverse Users Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to dataverse-community+unsub...@googlegroups.com.
To post to this group, send email to dataverse-community@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dataverse-community/4164ac45-7b7a-4731-aedc-28cfe2df4f29%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Philip Durbin
Software Developer for http://dataverse.org
http://www.iq.harvard.edu/people/philip-durbin

[MM - ADA]

Thanks for all the ideas and feedback.

I did look at the search API but it doesn't seem to work with the publication date at the moment (I logged an issue on github:  https://github.com/IQSS/dataverse/issues/4908).

As another idea, we use metabase for querying the Dataverse database for reporting/stats. After writing the initial post to the community, I discovered that metabase can be used to create a "Pulse" that can be set to run hourly/daily/weekly to send an email with the results of a particular query appended as a csv file to whoever. 

So I set up a query (with the help of Julian: https://groups.google.com/forum/#!topic/dataverse-community/PyiaergrmCs) to get the new datasets (not harvested and not DEACCESSIONED) published in the previous 7 days. Then I set up metabase to run this every Friday to send the results to major stakeholders. So that is an easy solution for weekly emails to stakeholders.

We still need to send newly published datasets to twitter and can perhaps leverage metabase->email->twitter by setting up an hourly pulse but there are potential drawbacks to that solution. So instead we may run a (non-metabase) process hourly to poll the dataverse database for newly published datasets (using the queries Julian assisted with) and tweet those. It won't be immediate but might be the best solution at the moment.

Will update this post when we decide on best approach.

M.