[FixMyStreet] Open311 integration documentation RFC
Rikard Fröberg
We are working with a vendor of a system for issue tracking and error
reporting, which offer some similarities with FMS.
This vendor has quite a few municipalities as their customers, which
is why we approached them regarding the possibility for them to
implement Open311 in their product so that we could
1. Send reports using Open311 directly to the municipality's system
(bypassing email and putting reports in the same place as those which
come in via the product interface)
2. Collect reports from the municipality's system to see if there are
any updated statuses so that we could reflect that on our instance of
FixMyStreet
The general idea is of course to promote Open311 as being an open
standard and suitable for reporting issues such as those sent via
FixMyStreet and at the same time allow for us to send reports directly
into the system which the municipalities (that are customers to this
vendor) already use. FixMyStreet already is prepared for using
Open311.
The vendor was interested in this, as they see this as a good thing
which adds value to their product as well as broadens the market for
their product to other countries than Sweden which have a version of
FixMyStreet in place.
To the point now ;-) In the process of setting up the testing
environment for them (setting up endpoints etc so that they can test
getting reports via Open311 and publishing updates the same way), we
ran into some problems understanding exactly what needs to be in place
in order for FMS to actually go about and send the reports as Open311
(and look for updates also via Open311).
So we embarked on an archaeological investigation mission tracing how
the internals of the FMS system handled this, in order for us to set
up the endpoints etc correctly.
We decided that what we excavated might be of interest to the list (we
couldn't find this documented anywhere but if it is, please share a
link!) so here is a link of what we found:
(sorry about the long URL, here's the UR1: http://ur1.ca/cuk4p )
Now, it is of course possible that we have misunderstood some parts of
the process, so if anyone who knows this part by heart would have time
to read it and help us improve it, that would be great.
So please read (if this is relevant for any of you) and comment if you
want. This might be translated to a blog post or something to be
merged with the official documentation set. We'd be glad to help also
with that.
Kind regards
Rikard
FixaMinGata, Sweden
--
Rikard Fröberg, Rådgivare offentlig sektor
Morus konsult AB | http://morus.se | rik...@morus.se
0700 - 90 69 64 | 031 385 88 93
Matthew Somerville
Just a quick note to say that we've never used the bin/open311-update-reports script (I removed it from the repo 10 weeks ago). It is fetch-comments and send-comments that we use for fetching/sending Open311 updates. Sorry for the confusion caused there.
Since you started using the codebase, we have improved the backend and implemented 'bodies', which separates the concept of the administrative area from the body that reports are sent to (so a body can cover more than one MapIt administrative area, or vice-versa, and it makes the Open311 stuff clearer, as that is then associated with a body directly). This has been tested and is now live for us for some weeks now, but I still need to document and write this up properly in a blog post. Obviously for new users of the code it all seems straightforward, it is only for those who have used the old code that may need some explanation :-) The database schema changes needed are all in db/ , the only thing that you need to do after running them is to give bodies names, and make sure the database contains all the bodies you need.
FixMyStreet's code now works by using MapIt to look up the administrative areas of the point clicked, and then looking up which bodies cover those areas to display their categories to the user for selecting for the new report. You probably shouldn't now need to use 'devolved' contacts at all, because you can create new bodies with the right category/categories and apply them to the right areas. This is conceptually clearer and makes more sense than assuming a MapIt administrative area is a thing that reports are sent to.
I'll update your page, but I'd like to do so based on the current codebase of course (so e.g. get_council_sender is now the clearer get_body_sender etc). Do you think you'll be able to update your codebase in the near future at all? As more people start using FixMyStreet code, we probably need to think about how best to keep people up to date with changes and improvements made, I realise we have not done a lot to help there as yet.
Other changes have been made that helps with some of the points on your page - e.g. you can now have STAGING_SITE on and still send reports to Open311 contacts using the new SEND_REPORTS_ON_STAGING configuration variable, and we have a --debug flag to send-reports which might be of use.
ATB,
Matthew
> _______________________________________________
> fixmystreet mailing list
> fixmy...@lists.mysociety.org
> https://secure.mysociety.org/admin/lists/mailman/listinfo/fixmystreet
Rikard Fröberg
<mat...@mysociety.org> wrote:
> Hi Rikard,
>
Hi!
> Just a quick note to say that we've never used the bin/open311-update-reports script (I removed it from the repo 10 weeks ago). It is fetch-comments and send-comments that we use for fetching/sending Open311 updates. Sorry for the confusion caused there.
>
OK!
> Since you started using the codebase, we have improved the backend and implemented 'bodies', which separates the concept of the administrative area from the body that reports are sent to (so a body can cover more than one MapIt administrative area, or vice-versa, and it makes the Open311 stuff clearer, as that is then associated with a body directly). This has been tested and is now live for us for some weeks now, but I still need to document and write this up properly in a blog post. Obviously for new users of the code it all seems straightforward, it is only for those who have used the old code that may need some explanation :-) The database schema changes needed are all in db/ , the only thing that you need to do after running them is to give bodies names, and make sure the database contains all the bodies you need.
>
Bodies seem to be a good feature!
> FixMyStreet's code now works by using MapIt to look up the administrative areas of the point clicked, and then looking up which bodies cover those areas to display their categories to the user for selecting for the new report. You probably shouldn't now need to use 'devolved' contacts at all, because you can create new bodies with the right category/categories and apply them to the right areas. This is conceptually clearer and makes more sense than assuming a MapIt administrative area is a thing that reports are sent to.
>
As I understand you, MapIt hasn't changed, rather the way FMS
considers where to send reports? The bodies are stored apart from
MapIt?
> I'll update your page, but I'd like to do so based on the current codebase of course (so e.g. get_council_sender is now the clearer get_body_sender etc).
Feel free to copy the text for use elsewhere, or add comments! Since
we are behind in updating, I'd like to have both versions if possible
(one for ourselves, and one for the current version of the code base,
when we upgrade).
> Do you think you'll be able to update your codebase in the near future at all? As more people start using FixMyStreet code, we probably need to think about how best to keep people up to date with changes and improvements made, I realise we have not done a lot to help there as yet.
>
We are honestly not sure. We might be a bit undermanned for this. All
suggestions and help appreciated (as I also hinted off-list since I
had some off-topic questions).
> Other changes have been made that helps with some of the points on your page - e.g. you can now have STAGING_SITE on and still send reports to Open311 contacts using the new SEND_REPORTS_ON_STAGING configuration variable, and we have a --debug flag to send-reports which might be of use.
>
Sounds great! We are missing out, being late in updating.
Cheers
Rikard
Matthew Somerville
> As I understand you, MapIt hasn't changed, rather the way FMS
> considers where to send reports? The bodies are stored apart from
> MapIt?
Yes, that's correct.
> Feel free to copy the text for use elsewhere, or add comments!
I'm not able to edit the page itself, it seems, but I have edited the Talk page to have some more details.
> Since we are behind in updating, I'd like to have both versions if possible
> (one for ourselves, and one for the current version of the code base,
> when we upgrade).
There's no real difference in the fundamentals, mostly in renames (get_body_sender/get_council_sender). The basics of how the Open311 process works remains the same.
> We are honestly not sure. We might be a bit undermanned for this. All
> suggestions and help appreciated (as I also hinted off-list since I
> had some off-topic questions).
There was this recent related offer we made on the DIY blog: http://diy.mysociety.org/2013/01/22/new-get-free-developer-time-to-set-up-fixmystreet-or-alaveteli-in-your-country-city-or-region/
The issue here is not identical, as it's about keeping up to date rather than with setting up a new installation, but I personally would say that both are important, and we certainly do need to think about how we can encourage and help keep people's installations up to date and include the changes they've made that will be of help to other uses of the code.
ATB,
Matthew
Rikard Fröberg
> Just a quick note to say that we've never used the bin/open311-update-reports script (I removed it from the repo 10 weeks ago). It is fetch-comments and send-comments that we use for fetching/sending Open311 updates. Sorry for the confusion caused there.
>
Correct me if I'm wrong, but it seems that fetch_comments use the
"open311 comments extension". Our vendor has not implemented that, and
I can't find where that extension is documented.
However, we decided to use the old script, open311-update-reports,
instead since it works with our vendor (who complies with the
GeoReport v2 of Open311).
Now, this lead to more questions. Sorry about this ;-)
Using the old script worked fine except for one part. As you can see here:
https://github.com/MorusAB/fixmystreet/commit/95122d764f568daf924a1d99ecfed4fb85f5df88
we had to skip the call to gettext translation. This seems to be
because gettext is not available when called via crontab or the
command line. So we had to hardcode our Swedish translation instead.
Cheers
Rikard
[snip]
Matthew Somerville
> Correct me if I'm wrong, but it seems that fetch_comments use the
> "open311 comments extension". Our vendor has not implemented that, and
> I can't find where that extension is documented.
It is documented at https://github.com/mysociety/fixmystreet/wiki/Open311-FMS---Proposed-differences-to-Open311 (there is a comment in send-comments to this affect, but not fetch-comments).
> However, we decided to use the old script, open311-update-reports,
> instead since it works with our vendor (who complies with the
> GeoReport v2 of Open311).
Okay; as I said we've not really used it in anger so it might have bugs, but hopefully it will work for you. I note it uses the old 'fixed' state rather than 'fixed - council' as an example.
> Using the old script worked fine except for one part. As you can see here:
> https://github.com/MorusAB/fixmystreet/commit/95122d764f568daf924a1d99ecfed4fb85f5df88
>
> we had to skip the call to gettext translation. This seems to be
> because gettext is not available when called via crontab or the
> command line. So we had to hardcode our Swedish translation instead.
No, you don't have to do that, and gettext definitely is available when called by crontab or the command line, you just have to set it up appropriately. See e.g. how send-reports, send-alerts, or other cron scripts do it - e.g. send_reports() calls, for each report, $cobrand->set_lang_and_domain($row->lang, 1) to switch to the right language for that report. It does this on a per-report basis because we have installations where reports can be in more than one language, so isn't done on a site-wide basis.
ATB,
Matthew
Dave Whiteland
> On 28 Feb 2013, at 14:42, Rikard Fröberg wrote:
>> Correct me if I'm wrong, but it seems that fetch_comments use the
>> "open311 comments extension". Our vendor has not implemented that, and
>> I can't find where that extension is documented.
>
> It is documented at https://github.com/mysociety/fixmystreet/wiki/Open311-FMS---Proposed-differences-to-Open311 (there is a comment in send-comments to this affect, but not fetch-comments).
Incidentally, for a non-techie explanation of why this extension
exists, we recently blogged about it:
http://www.mysociety.org/2013/02/20/open311-extended/
Yours
Dave