Reworking the event importer

[Igal Koshevoy]

A few days ago, I had to disable the "import" feature of calagator.org because it was being abused to create thousands of unwanted events -- all of which were swiftly deleted. I've dug further into the data to identify specific problems and some possible solutions so we can re-enable the import functionality. Please read and respond if you think this makes sense, how it could be better, etc.

FACTS

Based on the last 12 months of activity on calagator.org:

* 16% of all events we kept were imported, therefore we should keep this functionality.

* ~1.3 edits were needed per imported event, so every imported event needed to be edited at least once on average to fix its contents, and almost all of these edits were done hours later by gardeners rather than the person that imported the event, therefore we should figure out how to make the person importing do the edits themselves before they're published to our users.

* 93% of events we deleted came from imports, yet most of these seem to be excessively broad bulk imports by well-meaning people, therefore we should make it more obvious what events are appropriate and make the user specify which individual events to import.

PROBLEMS AND SOLUTIONS

1. How to discourage imports of inappropriate events? I propose changing the "Import event(s)" form to display a prominent message explaining the site's definition of appropriate content. This message will be configurable per theme so that each Calagator instance can have their own, e.g. calagator.org say that appropriate events are those of interest to the Portland metro area technical community and also show a picture of a sad kitten urging imports of only appropriate events.

2. How to import specific events from a URL? I propose changing the "Import event(s)" system so that after the user submits a URL for importing, they get a form showing all events extracted from that source. The user then needs to tick a checkbox next to each event they want imported. If they don't specifically import an individual event, it won't be published to the feed, search or any other listing. This extra step will make it more obvious what events are being imported so the user can be more selective, and make it harder to bulk import many unwanted events.

3. How to improve the quality of imported events? I propose changing the "Import event(s)" system so that the user submits a URL for importing, and then gets a big form containing all the events and venues extracted from their source. They can then edit the content on this form and publish the edited events they selected. Making the user that imports the events do the editing will improve the quality of the data and reduce the load on gardeners to fix the imports. This form could also be more strict than our "Add event" form, for example, by demanding that all event "description" fields has some content.

IMPLEMENTATION DETAILS

The site-specific text describing appropriate content can be a partial in the theme, similar to the partial used to display the homepage welcome message.

The Event and Venue models will be altered to include a new "publish" boolean, which is true for any existing record or those created through the "Add event" and "Add venue" forms. Imported events and venues will be initially created with "publish" set to false, and will be excluded from the feed, search and other listings. When the user edits the list of records extracted from their import source, they can mark which records should be published.

The form used to add and edit events will be altered to display a venue form inline if that venue is new. This will allow the user to specify all the information about a venue on the same form as the events. This improvement can also be used as part of the regular "Add event" form, so that if a user specifies an existing event, everything looks the same as it does currently, but if they specify an unknown event, an in-line venue form will appear for them to enter the address and other details. This inline form will improve the quality of our venues by making sure that the user edits them as part of their event, rather than choosing to submit the event and then ignore the edit venue form's request to specify details about the venue.

The inline venue form on the importer should also provide a way of selecting another venue specified in the form, e.g., if the user describes a new venue as part of their first event, the second event should be able to also use this same new venue without requiring the user retype everything.

-igal