City of Vancouver Looking for Open Data Feedback

14 views
Skip to first unread message

Darrin

unread,
Jul 9, 2010, 4:53:11 p.m.2010-07-09
to Vancouver Data
The City of Vancouver is continually looking for ways to improve the
accessibility and usability of its open data catalogue. One option
that has been considered is to provide access to the catalogue through
APIs based on open standards such as REST and web services.

The city currently provides data via bulk downloads of datasets
whereby developers download an entire dataset and create custom API’s
in order to interact with the data and display it in meaningful ways.
Those who want to look at the raw data currently have no easy way of
filtering that data and listing only the records that are relevant to
them. As our goal is to provide as much data as possible in open
formats and in ways that meet the needs of the community we would like
to understand if providing this level of access would be beneficial to
you and help us achieve our goal. So, what we would like to know from
you is:

Has anyone had any experience using data from open data sites in this
way and if so can you talk a little about your experience? Did it make
life easier? Harder? Any opinions?

Does anyone have any experience with a specific solution or platform
that provides this kind of access?

For anyone who has built applications that use the City’s data
catalogue, what would the impacts to those applications be if
Vancouver were to go this direction? Could Vancouver do anything to
minimize those impacts?

Do you as users have any preference as to what platform or mechanism
the City uses to provide its Data?

Is anyone interested in seeing other features on the City’s Open Data
website such as social media tags, dataset rankings, etc?

On behalf of the Open Data Team, thanks in advance for your feedback.

Darrin Fast
darri...@vancouver.ca
(604) 873-7033

Kaitlin Duck Sherwood

unread,
Jul 10, 2010, 5:39:23 p.m.2010-07-10
to vancouv...@googlegroups.com
Speaking purely for myself, for a small sample size of applications, I generally do not use APIs for incremental work.  I grab all the data, munge it locally, and then display the information that I want.  This means that I don't care if I get the data via a static file or via an API.

My applications are pretty static, however: I get the data once and then don't get it again.  If I wanted to do some sort of mashup with data that changed frequently, an API would be important.



--
You received this message because you are subscribed to the Google Groups "Vancouver Data" group.
To post to this group, send email to vancouv...@googlegroups.com.
To unsubscribe from this group, send email to vancouver-dat...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/vancouver-data?hl=en.




--
Kaitlin Duck Sherwood

Luke Closs

unread,
Jul 12, 2010, 2:13:08 p.m.2010-07-12
to vancouv...@googlegroups.com
Hello Darrin,

When building VanTrash, we wanted to host the whole dataset (in our
case, garbage zones and schedules) so our application could be fast
and optimized. So we wouldn't have used RESTful APIs if they were
around. We ended up taking the raw data and then building a RESTful
API around the data for our app. (http://vantrash.ca/api.html)

I've used the OGDI data browser on Edmonton's data site and it was
pretty cool. Not sure I'd use it in an application though. For small
hacks, I think an API can be useful, but for more substantive projects
I think people would typically grab the whole dataset and then carve
it up specific to their app.

My preference is for data.vancouver.ca to keep with the raw data
downloads. APIs for those datasets would be cool, but maybe expensive
to build & maintain. In my experience, the key to building a good API
is to be a consumer of that API - so you're building based on a real
need. If you build an API just guessing at what people will want,
it's easy to build features people won't use and to miss features that
are useful. (Of course this would all iterate over time towards
something workable...)

One thing I would like would be some kind of "social features" around
the datasets. For instance, I'd like to get some kind of notification
when a dataset (eg: garbage zones) are updated. Or I'd like to see
what other community projects are using each dataset. For example,
people browsing to data.vancouver.ca and looking at the garbage zones
dataset would not know that VanTrash has already built and hosts a
RESTful API to that data. This could be a good resource to share.

Glad to see y'all practicing kaizen on the city data!

Luke

On Fri, Jul 9, 2010 at 1:53 PM, Darrin <darri...@vancouver.ca> wrote:

Matt Laird

unread,
Jul 12, 2010, 2:33:53 p.m.2010-07-12
to vancouv...@googlegroups.com
While I haven't actually used any of the data yet I must agree with
Luke that IF I were to build an app I think I'd use a local copy of
the raw data. Both for the reason Luke said, about being able to
munge it for my needs but also to deal with update issues.

In other projects I work on professionally, when our upstream data
sources put out a new release it often means a week or two for me to
figure out how they've broken the dataset and update my parser. Not
that I'm saying the City would ever do such things, but an API
definitely has the danger of making a large number of applications
stop working by either changing the API or if the API should go down
for some reason.

An update notification for data sets as Luke said would also be
fantastic, even if its just a machine queriable version number so we
can know when we should grab a new version and try parsing it in to
existing applications. Yes it might mean new data isn't instantly
available in community created applications (a definite point in
favour of an API), but just based on experience it can be so much more
fragile a setup.

Gorilla APIs

unread,
Jul 14, 2010, 10:33:08 a.m.2010-07-14
to Vancouver Data
Great discussion. I have a few points to consider in favour of APIs.
Before I get started, when I'm referring to APIs I specifically
talking about a RESTful implementation where all calls to the API make
use of standard HTTP verbs. Also, I am coming from the perspective of
a mobile application developer as opposed to a web application
developer that has the benefit of an underlying app server to help
with the work (although I think a number of my points apply to both).

Flat files have some limitations that constrain application
developers. This can be manageable for web applications that have the
benefit of a server-side to handle pre-processing of the data, but for
mobile applications that live entirely on the client side (for which
must of the data in question has high value) we do not have the luxury
of nightly jobs to pickup and munge data. The mobile client needs
data on demand.

Flat files are sufficient for fairly static, read-only data sets but
if it is fair to say that Vancouver has designs to mature their open
data services over time, then APIs open great opportunities for
manageable, secure read/write capabilities. As the need arises for
more real-time data, the flat file solution will weaken. An API on
the other hand will allow apps to consume just the up-to-date data
that they need at the time it is needed. Vancouver, as the provider,
also has the opportunity to better monitor and manage the performance
and of course the developer still has the option of doing their own
optimization (i.e. server side caching).

Speaking to Matt's point, where an API is implemented with proper
versioning, the API is frozen from the time it is made available to
the time it is deprecated. The API breaking should never be an issue
for the consumer. Conversely, with a flat file, you as the consumer
are tightly coupled to the implementation and the risk is high that
your app breaks if anything changes about its format.

I wholly agree with Luke's point that if Vancouver provides an API,
that it should be delivered in a iterative manner driven by active
dialogue with the consumers of the API to ensure that it matures based
on real world needs. Never build for requirements you don't actually
have.

Summarizing, I would like to see a RESTful API implementation because
it allows a simpler application architecture. My client application
can be built to be more self-sufficient; I don't have to deal with
munging data is which is frustrating, fragile work; Mashing data for
disparate sources is easier, and for mobile applications where
performance is a big issue, I can get exactly what I need just when I
need it.

Matt Laird

unread,
Jul 14, 2010, 1:24:19 p.m.2010-07-14
to vancouv...@googlegroups.com
Oh I completely agree that an API is definitely something that should
be looked at and done right with proper versioning to prevent breakage
(the Google Maps API is a great example of this). Particularly as you
say for real time data a flat file simply won't do. The API breakage
that I deal with and give me a bad taste about web APIs is simply an
artifact of biologists trying to write large software projects. :)

I don't think this is an either/or choice, for some data sets a flat
file may be more appropriate, for some an API. And for more static
ones there's certainly nothing preventing and API along side the
existing flat file, once the infrastructure is in place upkeep should
be minimal. Hell, if done right the flat file could be generated on
the fly from the dynamic data if you really wanted (with proper
versioning previsions of course to allow users to consistently get a
desired point in time), we already do this in some projects I work on
professionally. The user never knows its not actually a file sitting
on our server.

As to where the crunching is done, well I guess that's a debate on if
the mobile app developer should also be responsible for munging then
hosting the data their app accesses. I've seen apps where you get
regular data updates from the app developer's server which does have
the ability to download and munge updated data to a form optimized for
the mobile app's needs. However if you can save a step by having a
RESTful API on the City's server, and they don't mind the extra
infrastructure costs of hosting it for mobile apps to access on the
fly, that's great too! As you've said, yes for real time data that
might be the optimal choice anyhow, since who would want the time
delay of it having to bounce through the app developer's server first.

So final answer, yes. We want it all! Gimme. Gimme. Gimme. :)

Darrin

unread,
Jul 14, 2010, 1:31:48 p.m.2010-07-14
to Vancouver Data
Just wanted to clarify that this would not be an all or nothing type
of thing. If the City were to take the API direction we would
certainly like to ensure that the 'flat-file' download option is also
available. Our goal is to widen the range of access and potential use
of the datasets in whatever way is practical and possible.

Darrin
> > For more options, visit this group athttp://groups.google.com/group/vancouver-data?hl=en.- Hide quoted text -
>
> - Show quoted text -

Meng

unread,
Jul 23, 2010, 3:44:34 p.m.2010-07-23
to Vancouver Data
My personal preference at least for the current phase of the City's
Open Data initiative is to focus on exposing the data rather than on
exposing data operations and interactions (via REST or other web
services) which could be very expensive for the city to build. Private
parties such as MS and Google, as well as the open data commnuity
would be in a better position to munge the data and maximize the
public value of the city's data in a variety of user-preferred ways by
using whatever open APIs and platforms such as OGDI. Strategically,
this can be viewed as a public-private partnership initiative (without
a contract).

Meng
> > > For more options, visit this group athttp://groups.google.com/group/vancouver-data?hl=en.-Hide quoted text -
>
> > - Show quoted text -- Hide quoted text -

Luke Closs

unread,
Jul 23, 2010, 5:00:28 p.m.2010-07-23
to vancouv...@googlegroups.com
Case in point: http://vantrash.ca/api.html
Reply all
Reply to author
Forward
0 new messages