Sharing data and visualisations across the alaveteli network

[laurent...@gmail.com]

Hello everyone,

Ma Dada is starting work on a series of technical topics, and as part of our funding, we are aiming to share what we
build with the wider alaveteli network. We talked with mySociety earlier this week, and agreed to use this mailing list
as a starting point for various conversations, typically around upstreaming some of our local changes into alaveteli
itself if it appears interesting to other sites.

As a sort of test for this workflow, I'd like to share a bit about our datavisualisation tool, as it seems that some
other site owners have expressed interest in reusing it.

In France, we have been struggling with very low response rates since the beginning (all our calculations land around
18-20% response rate, even when including late replies). So we are trying to show metadata to put numbers on a fact that
most people familiar with the problem here feel without being able to quantify it.

We built https://madada.fr/stats. The statistics in there are mostly the same as under alaveteli's admin page, but we
think they look a bit neater, and they're public! Also, it's very easy to add new metrics to this, which we will be
doing very soon.

It works in two steps:
- Every night, a script makes a copy of our live database and anonymises it by removing all sorts of personal
  information, etc... This copy of the database can then be made public, and used for a variety of purposes (dataviz
  like we did, but also research, etc...)
- We use a tool called metabase (https://metabase.com) to calculate and display metrics based on the copy of our
  database. Because it only has access to the anonymised copy of our database, we can easily give access to new people
  for them to explore the data and find information they're interested in.

We think the first part of this could be integrated into alaveteli, which would allow all sorts of metadata reuses
across the network.

Technically speaking, the data export is a simple SQL script which duplicates the database and then wipes out anything
that needs to stay confidential. It runs on a cron job. It requires read access to the live database, so it needs to
run pretty close to your main server. For better security, it should probably run in reverse: positively copying data
that we actually want, to remain safe even with changes that each site might have made to their database.

The dataviz is a docker container that runs metabase. We just give it its own database and access to the anonymised
clone. This can easily be replaced by any tool.

We think it would be super interesting to compare "live" data across countries. Our guess is that France is
among the bad countries, and we'd love to be able to show evidence around this to put pressure on our administration :)

If you find this interesting, please let me know. We have a bit of funding to work on this, so you might not need to do
much more than express interest :)

Laurent for Ma Dada