Keeping Contact Databases Updated

[Michael Morisy]

Hi all,

Michael from MuckRock, a US FOI platform (not Alaveteli, alas). As we've hit a larger scale, we've found it challenging to keep our contact database systems regularly updated. We've been thinking of taking a step back and working on a more holistic way to check in and task out research, verification, and updating. Curious if other folks have seen this problem, found solutions, and/or would be interested in collaborating around some potential solutions?

Michael

--

Michael Morisy

Founder & Chief Executive, MuckRock
Phone/Signal: 1-650-656-3417

[Mattias Axell]

Dear Michael,

We experience the same in Sweden and also want to the same. For OpenDataDay 2022 this Sat 5th of March there are hopes that we can do a "sprint" about this: https://opendataday.org/

There is a Github issue about this here: https://github.com/okfn/publicbodies/issues/138

Main thing I see needed for, apart from what you write, is to internationalize and harmonize the datamodel for public bodies contacts database so it can be easily updated to Alaveteli platforms, MuckRock and connected to entities on Wikidata.org.

I imagine this is a larger project than a fix on Open Data Day and that we potentially collaborate on funding applications = Open Knowledge network + MuckRock + MySociety and also find interest from more organizations.

What do you think? Feel free to join the discussion on https://github.com/okfn/publicbodies and chat https://matrix.to/#/#publicbodies.org:matrix.org

Best,

Mattias

Handlingar.se (Alaveteli platform in Sweden)

--
You received this message because you are subscribed to the Google Groups "Alaveteli Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alaveteli-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/alaveteli-users/CAH7N8YkFnXMe4Ok2ObNd60_%3DfyD%2BmArMSDUfPG2dcwPTwwLfGA%40mail.gmail.com.

[Gareth Rees]

At WhatDoTheyKnow we've had a bit of a mixed experience.

We regularly get crowdsourced email address updates and new body suggestions through the built in Alaveteli public body change requests.

We recently spoke to an Alaveteli site runner who wasn't familiar with these, so I've recorded some short screencasts. Sorry about the very poor editing where I was trying to redact an internal email address 🙈. These are unlisted on YouTube – I'll probably redo them at some point for more public publishing.

* Updating addresses through public body change requests: https://youtu.be/x8D5Aj4N44I

* Adding authorities through public body change requests: https://youtu.be/oWro-w5X94s

Searching our support inbox, in the last month we've had ~45 suggestions. We're currently in the very early stages of working on a pitch around improving the visibility of this workflow in Alaveteli and adding more capability, like having a public list of bodies missing an email address for example, and trying to build more of a "community" around this aspect of the platform.

What hasn't worked so well in recent times is crowdsourcing large batches of authorities. When WhatDoTheyKnow started we shared a Google Sheet and asked our community to fill in the blanks. We've tried this a few times recently, but only got a handful of contributions. I suspect this is down to listing increasingly obscure groups of authorities, rather than central government and local councils that everyone is familiar with. To tackle this we're trying a more structured volunteering role that gets set projects around this – adding a new group of authorities; checking our coverage of an existing group, etc. We've just recruited our first volunteer with this new aim, so we'll see how it goes.

Best,

Gareth

[Laurent Savaëte]

Hi Michael,

At madada.fr, we're trying a few different things:

- we've setup a daily automated task that fetches an official list of FOI officers, and notifies us when changes happen. https://gitlab.com/madada-team/moniteur-annuaire-cada/ it's mostly in French, but you can probably figure it out. Overall, this has been of limited use so far, as the FOI authority publishes updates only once or twice a quarter, so they end up being huge and hard to navigate. This is also useless for community provided updates.

- We do get 1-2 updates a day on average from users, and those follow the process Gareth described just below. Our goal now is to find a couple of volunteers who could take care of validating these requests, as very often our users simply suggest adding a public body without providing the contact emails (because French authorities seem to like keeping public information private :) and we end up having to dig a fair bit to find the missing bits.

- We're also pondering adding an API endpoint to alaveteli so that these addition/update requests could be created automatically from various sources, and then have our volunteers review/accept them. https://github.com/mysociety/alaveteli/issues/6766

Laurent

[WhatDoTheyKnow Support]

There is a related discussion at:

https://groups.google.com/g/alaveteli-dev/c/S9hw3VJf4T8/m/lBimvabIAQAJ

about linking an Alavetli site's public body database with official databases. This helps more with keeping the lists of public bodies up to date rather than keeping contact details up to date though as contact details aren't typically on the official public databases.  Removing bodies which have ceased to exist does though consequentially remove outdated contact details for those bodies. 

On WhatDoTheyKnow we have an alert set up for phrases which appear in error/bounce emails. We update our contact details if we're notified of a problem via that alert, or by users getting in touch. The alert doesn't rely on users to inform us of issues. Our alert is for:

Track query         "Recipient address rejected"
Track query         "SMTP error"
Track query         "A message that you sent could not be delivered"
Track query         "delivery status notification" -"THIS IS A WARNING MESSAGE ONLY"
Track query         "MIME format"
Track query         ukcod

(ukcod is, or was, in our mailserver domain).

 

Richard

--
WhatDoTheyKnow Support Team

WhatDoTheyKnow is a project of mySociety run by a small team of staff and dedicated volunteers.

If you’d like to help us maintain the site you can make a donation. Your contributions, however small, really help: https://www.mysociety.org/donate

Find out more about how we handle your data at: https://www.whatdotheyknow.com/help/privacy

mySociety is a registered charity in England and Wales (1076346) and a limited company (03277032).

[Michael Morisy]

Thanks so much everyone for chiming in! This was really helpful discussion, and the PublicBodies.org site is very cool! We've been looking for a standard like that to contribute our data to and link our database.

We've been exploring trying to set up a more generalized task management/verification flow, with the first area to tackle being a lot of these related problems. It's very much in the early exploratory phase, but I'll drop a note here if and as we move forward, and might reach out with additional questions or resources I find.

Mattias, thanks for suggesting that Open Data Day event — I've been backlogged but it's definitely a good discussion and if it moves forward we'd be interested as well.

--
You received this message because you are subscribed to the Google Groups "Alaveteli Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alaveteli-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/alaveteli-users/e96e3af6-1d97-4bf6-949c-cbd9b24b7d11n%40googlegroups.com.

[Mattias Axell]

Thanks Gareth for your reply! I would like to schedule a call in end of April/May to talk about this crowdsourcing part of the project and pitch? I would like to give it a try. It looks like we're having traction here and OKFN, publicbodies.org, OK Nepal, Muckrock, - https://github.com/okfn/publicbodies/issues/138#issuecomment-1072668037

To view this discussion on the web visit https://groups.google.com/d/msgid/alaveteli-users/c71d3a29-931f-4a31-b1ac-bd307abb898fn%40googlegroups.com.

[Gareth Rees]

Happy to join a call – get in touch nearer the time to coordinate.

I don't think we'll be able to commit to work at the moment, but at least trying to figure out what we all need from each other might push our understanding forward, and then we can ticket things up and integrate them into any funding proposals we might make.

[Mattias Axell]

Hi! Great to hear.

I completely understand and I will get back to the ones who showed interest along with a Framadate: https://framadate.org/abc/en/ for virtual call in April or May.

Best,

Mattias

To view this discussion on the web visit https://groups.google.com/d/msgid/alaveteli-users/542f5e76-5a8f-47ce-8241-904df7de5d02n%40googlegroups.com.