Finding data on hamilton.ca
Joey Coleman
the City website.
Here's a listing of Crossing Guard locations in Hamilton which could
be fairly easily converted into an open dataset and mapped:
http://www.hamilton.ca/CityDepartments/PlanningEcDev/Divisions/ParkingBylawServices/SchoolCrossing/SchoolCrossingGuardLocations.htm
Let's see what else is out there asking to be converted into Open Data.
Ryan McGreal, Raise the Hammer
> Here's a listing of Crossing Guard locations in Hamilton which could
> be fairly easily converted into an open dataset and mapped:
> http://www.hamilton.ca/CityDepartments/PlanningEcDev/Divisions/ParkingBylawServices/SchoolCrossing/SchoolCrossingGuardLocations.htm
Okay, I'll bite: this looks like a fun and fairly easy conversion. This
week (probably after Wednesday evening's talk), I will:
* Get the data into a database and host on the RTH data resource [1].
* Create a Google Map plotting the crossing guard locations.
* Share the code I used to convert the data on github with a FOSS licence
so others can easily replicate it.
Kind Regards,
Ryan
[1] http://raisethehammer.org/data - each data set is available in HTML
and JSON format
Joey Coleman
You're amazing.
I'm working on Ward Boundaries for the WardMaps project tonight. Also
the logistics of planning hackfest and the workshop. I should nail
down the workshop date tomorrow so you can announce it on Wednesday.
I was actually going to discuss the municipal election finance data
hosting in a thread I'm working on for this afternoon. In terms of
hosting data, I'd like to see use get the data on both RTH's election
data API and OGDI.
Could we place crossing guard data on both RTH and OGDI? I'd like to
get it on OGDI as that ties it into datadotgc.ca which hopefully
provides the opportunity for people to find Hamilton data using the
search function there.
- Joey
On Mar 21, 11:00 am, "Ryan McGreal, Raise the Hammer"
> On Mon, March 21, 2011 10:11 am, Joey Coleman wrote:
> > Here's a listing of Crossing Guard locations in Hamilton which could
> > be fairly easily converted into an open dataset and mapped:
> Okay, I'll bite: this looks like a fun and fairly easy conversion. This
> week (probably after Wednesday evening's talk), I will:
>
> * Get the data into a database and host on the RTH data resource [1].
> * Create a Google Map plotting the crossing guard locations.
> * Share the code I used to convert the data on github with a FOSS licence
> so others can easily replicate it.
>
> Kind Regards,
> Ryan
>
> and JSON format
Joey Coleman
There has been discussion of getting all mapping data into
OpenStreetMaps as a layer towards the eventual Hamilton mapping app.
> > Here's a listing of Crossing Guard locations in Hamilton which could
> > be fairly easily converted into an open dataset and mapped:
> Okay, I'll bite: this looks like a fun and fairly easy conversion. This
> week (probably after Wednesday evening's talk), I will:
>
> * Get the data into a database and host on the RTH data resource [1].
> * Create a Google Map plotting the crossing guard locations.
> * Share the code I used to convert the data on github with a FOSS licence
> so others can easily replicate it.
>
> Kind Regards,
> Ryan
>
> and JSON format
Richard Weait
Being a new addition to this list, perhaps I've missed the part where
the city has said, "Yeah, okay, have your way with the data on our web
site".
From their web site terms of use, I find :
"You may not reproduce, publish, copy, link to, frame, tag, embed,
merge, modify, recompile, license, distribute, sell, store in an
electronic retrieval system, download (except by the browser of a
single user) or transmit, in while or in part, in any form or by any
means whatsoever, be they physical, electronic or otherwise, the
Portal and/or the Content."
Which, to me says, "Hands off".
Ryan McGreal, Raise the Hammer
> Could we place crossing guard data on both RTH and OGDI?
The data on RTH will be available to download in JSON format, so you
should be able to drop it easily into any other data store.
Update: I wrote a script that grabs geocodes for each location (using the
Google Maps API) and formats the whole thing into an SQL insert statement.
Here's the script (in Python) if anyone wants to play around (warning -
the script is not pretty): https://gist.github.com/879619
I've run the script myself and dumped the results into the RTH database.
As soon as I get a chance this evening, I'll expose it via the RTH data
API.
Cheers,
Ryan
Ryan McGreal, Raise the Hammer
about having data in a structured format is that it becomes trivially
easy, instead of slow and manually cumbersome, to perform summary
operations on it.
* Numbers of crossing guards by ward (in decreasing order):
* ward 9 has 19 crossing guards
* ward 8 has 18 crossing guards
* ward 7 has 17 crossing guards
* ward 6 has 16 crossing guards
* ward 1 has 14 crossing guards
* ward 3 has 14 crossing guards
* ward 2 has 11 crossing guards
* ward 10 has 8 crossing guards
* ward 5 has 7 crossing guards
* ward 12 has 7 crossing guards
* ward 4 has 6 crossing guards
* ward 13 has 3 crossing guards
* ward 11 has 2 crossing guards
* ward 15 has 1 crossing guards
* Intersection types:
* 61 records do not specify intersection type
* 37 are full signal
* 18 are 3 way stops
* 10 are PPS
* 9 are 4 way stops
* 8 are mid-block crossings
* School types:
* 65 are public schools
* 17 are separate schools
* 61 are both public and separate schools
Cheers,
Ryan
NikG
Ryan McGreal, Raise the Hammer
> Have you already geo-coded the intersections
Yes - lat and long for each location.
> can you share as a CSV (or any other table format)?
Sure thing. I'll post a follow-up once it's available.
> Also, wonder if you can share more on the RTH Data API?
If I can get it, I can share it. :)
James McKinney
Sent from my iPod
Joey Coleman
absurdity of their license.
I'm confident the City will not go after anyone for producing data in
the public interest from information on their website. If they did,
the courts have been rather generous towards the production of
information in the public interest.
Richard does make an important point, we don't have the blessing of
the City for what we are doing. We'll need to get focused on
pressuring them to adopt the Open Data motion. I want to wait until
after they are done the budget meetings which means we're looking at
late April/early May to go before Council.
- Joey
On Mar 21, 1:33 pm, James McKinney <oxford.tux...@gmail.com> wrote:
> "You may not ... link to"? This is the web, folks. Anyway, facts are not generally protectable by licenses. You can copy facts without worrying.
>
> Sent from my iPod
>
>
>
>
>
>
>
> > On Mon, Mar 21, 2011 at 10:11 AM, Joey Coleman <j...@joeycoleman.ca> wrote:
> >> Let's have an early Easter Egg hunt and go searching for quick wins on
> >> the City website.
>
> >> Here's a listing of Crossing Guard locations in Hamilton which could
> >> be fairly easily converted into an open dataset and mapped:
Joey Coleman
- Joey
On Mar 21, 12:57 pm, "Ryan McGreal, Raise the Hammer"
NikG
Richard Weait
> "You may not ... link to"? This is the web, folks. Anyway, facts are not generally protectable by licenses. You can copy facts without worrying.
That isn't the entire story about factual data, James. "The law" as
it applies to data is evolving quickly (for law).
"Sweat of the Brow", that is, the effort expended in compiling
information, earns copyright protection in the UK.
http://en.wikipedia.org/wiki/Sweat_of_the_brow
And in the European Union, the European Database Directive gives sui
generis protection to database works even if they are not entitled to
copyright.
So, to publish Open Data, data that is really open and does not
discriminate against users in, say the UK and Europe, one needs to
publish that data with a license granting the rights to use that data
as open data.
Your rights to use published data vary by jurisdiction. Yours and the
jurisdiction of the publisher should they choose to mess with you.
;-)
It's worth encouraging Hamilton and other municipalities to take a
modern and sustainable approach with their Open Data initiatives. As
challenging as it may be to get the city to adopt an Open Data
initiative, it will be tougher to get them to change it afterwards.
Witness the failings of the Vancouver / Toronto / Edmonton / London /
Ottawa data license, and the difficulty those municipalities are
having in righting their ships. Hamilton can learn from Surrey, BC,
and adopt the ODC PDDL* for their Open Data. That insures that their
data may be used by those in any jurisdiction
James McKinney
"generally" as a nod to "sweat of the brow" which is a legally
untested concept in Canada. In the US, it has been rejected. Also,
"sweat of the brow" in some jurisdictions means that I can take a
copyrighted dataset, and if I sufficiently sweat over it, I get
copyright over my "new" dataset. Given that, as Joey wrote, legal
action is very unlikely, I am still confident in saying that "you can
copy facts without worrying."
Richard
On Mar 21, 11:39 am, Richard Weait <rich...@weait.com> wrote:
> > Let's have an early Easter Egg hunt and go searching for quick wins on
> > the City website.
>
> > Here's a listing of Crossing Guard locations in Hamilton which could
> > be fairly easily converted into an open dataset and mapped:
> > Let's see what else is out there asking to be converted into Open Data.
>
> Being a new addition to this list, perhaps I've missed the part where
> the city has said, "Yeah, okay, have your way with the data on our web
> site".
>
> From their web site terms of use, I find :
>
> "You may not reproduce, publish, copy, link to, frame, tag, embed,
> merge, modify, recompile, license, distribute, sell, store in an
> electronic retrieval system, download (except by the browser of a
> single user) or transmit, in while or in part, in any form or by any
> means whatsoever, be they physical, electronic or otherwise, the
> Portal and/or the Content."
>
> Which, to me says, "Hands off".
data.
If we decide to ignore the licensing from the City of Hamilton then we
are going to seriously damage our reputation within the City of
Hamilton and not only set ourselves back but any other group that
wants to open the data from their local governments.
As the creator of the data the City of Hamilton is perfectly legal in
determining what licensing it wants to impose on that data. They can
make it as restrictive or as open as they wish. The fact that it is
readily accessible on the Internet is not part of the discussion. Nor
is the fact that we want to access portions of it.
That we want to have access to large portions of the data created for,
and by, the City of Hamilton is not a consideration nor in what
interesting forms we want to be able to present and use the data.
Until we are granted access to the data in ways that we want to use it
it will remain hands off.
We have two choices; we can get the City of Hamilton to change their
licensing to allow us to access, and use, the data in as many
interesting ways as we want or we can ask, and only act when they
offer it, for an exemption from the City to use at least portions of
their data to demonstrate what might be possible. Not doing so will
seriously impact our credibility and any chance to work with the City
to open the data.
We do not want to be regarded as rouge "hackers", unfortunately in the
negative sense, who are not willing to abide by legal restrictions
that have been imposed by the City. If we cannot abide by the legal
restrictions imposed by the licensing how can we seriously expect to
be able to deal with the City of Hamilton? How would we be able to
negotiate with Councilors and the City's administration if the first
impression of this group is that we do not abide by the licensing
terms that are currently in place? Why should the City change
anything, and especially be co-operative with us, when their
impression is that we do what we want?
By using the data sets that the City of Hamilton has available on
their site in any way that is not permitted by the licensing then we
are going to destroy any good will and any co-operation we may have
from the City. Change the licensing, or get exemptions for us, to
something that allows us to access and use the data in ways that we
want before we use the data and nobody will be able to complain. Our
statements that the data really belongs to us, the citizens of
Hamilton, does not stand up to much scrutiny if the licensing
restricts our access or use of the data. Nor does the fact that the
data is readily accessible on the Internet, it still belongs to the
City of Hamilton.
As an aside; although we are looking to have the City of Hamilton
Council change the licensing through a motion in Council is that
really necessary? Can the change come from an administrative action
within the City, meaning that senior bureaucrats within the City may
have the power to change the licensing themselves without requiring
that Council pass a motion to do so? If they can do so that may be
the way to approach the change in licensing.
Richard Degelder
Joey Coleman
Open Hamilton without changing the entire purpose and mandate of Open
Hamilton. We exist to facilitate the use of data by anyone, not to
have any control over it or the process of getting it.
Further, why have information available to the public if we cannot use
or link to it. It's clear that the problem here is with the City and
not us. The City is the "rogue" element with its demands that people
not link to their website. I doubt anyone in City Hall is even aware
of their ridiculous restriction.
We will not lose any allies by using information to provide the public
with information. While the opponents of open data may cite the fact
that the City has a closed data policy right now and that citizens are
defying it by providing election finance disclosures, vacant
properties listings, and the locations of school crossing zones, their
arguments will not hold any influence with the general public.
If we choose to stop all activity, we will lose momentum behind open
data in Hamilton and all creditability with the general public. We are
creating datasets in response to the very critics we'd be seeking to
appease. They are stating open data serves no purposes, costs lots of
money, and will be a "weapon of accountability." They are entrenched
in their positions. We need to counter their attacks and the only
method of doing so is to act.
The option of sitting around for the next few months hoping those
opposed to open data decide to change their minds is not a path of
likely success.
I say we continue to move forward.
If there are many who wish to stop in obedience to city's closed data
policy, we can focus our efforts on solely lobbying and pointing to
examples elsewhere.
As well, we will have to cancel the hackfest for Monday as the
information will be pulled from the city website.
Not that I don't see your point Richard. You make a good argument.
- Joey
Randy Kay
Randy Kay
"A free society cannot be the substitution of a 'new order' for the old order; it is the extension of spheres of free action until they make up most of social life."
Paul Goodman
Ryan McGreal
First: I'm not quite sure I'm reading the City's terms of use correctly,
so I sent an email to the city manager and head of legal to ask for
clarification.
If it is indeed the city's policy that no one is allowed to link to the
city website without permission, I intend to write about it for RTH to
draw public attention to how ridiculous the city's website terms are.
If we want them to change a bad policy, publicity and public attention
are critical.
Second: IANAL but even if that is the city's terms of use, that doesn't
mean the terms are legally enforceable. The problem, in Canada, is that
there is no legislation governing the legality of various
internet-specific activities, like hyperlinking and so on. That has left
the courts to decide how to handle conflicts over what is allowed.
A case in front of the Supreme Court of Canada right now [1] turns
around whether linking to a web page constitutes citation or
republication. In this case, the plaintiff is accusing the defendant of
defamation for linking to web pages the plaintiff argues are defamatory.
[1] http://www.scc-csc.gc.ca/case-dossier/cms-sgd/sum-som-eng.aspx?cas=33412
The Supreme Court of British Columbia had decided that simply linking to
a defamatory web page does not constitute defamation, unless the link is
provided for the purpose of endorsing the defamatory material.
It seems to me that if linking = citation, a web site owner can't
legally prevent others from linking to it; whereas if linking =
republication, every link becomes a potential copyright infringement.
What a disaster that would be!
Third: in terms of linking to and using public data, another issue that
may limit the City's ability to restrict what third parties do with the
data is municipal freedom of information law [2] that asserts the right
of the public to access government information that is not specifically
exempted for reasons of privacy or has an overriding public interest.
[2]
http://www.e-laws.gov.on.ca/html/statutes/english/elaws_statutes_90m56_e.htm
It seems ridiculous from a legal standpoint that one might have to file
an FOI under MFIPPA just to publish legally public information that is
already published on a municipal website. Certainly newspapers publish
information obtained under FOI requests without legal consequence.
Information "owned" by the City of Hamilton that is not specifically
exempted from publication due to privacy issues is BY LAW accessible to
the public. I would be most surprised if a use licence on the part of
the City that tries to restrict what the public can do with public
information would hold up to a legal challenge.
Regards,
Ryan
Joey Coleman
I've been planning to get at this dataset with an FOI request this
summer. I can't get the data to show in Google Docs right now, but
will download it when I get home and get back to you.
I suspect it's an easy set to clean-up and at the latest map during
our mapping workshop in early April.
- Joey
On Mar 21, 8:38 pm, Randy Kay <dundast...@gmail.com> wrote:
> Hi - I have a spreadsheet from the city on pedestrian accident locations
> (attached) - what does it need to have to be useful? I want to see this on a
> map!
> I love what you folks are doing, I'm hoping to learn a few things!
>
> Randy
>
> > Let's have an early Easter Egg hunt and go searching for quick wins on
> > the City website.
>
> > Here's a listing of Crossing Guard locations in Hamilton which could
> > be fairly easily converted into an open dataset and mapped:
>
> > Let's see what else is out there asking to be converted into Open Data.
>
> --
> Randy Kay
>
> "A free society cannot be the substitution of a 'new order' for the old
> order; it is the extension of spheres of free action until they make up most
> of social life."
> Paul Goodman
>
> 73KViewDownload
Randy Kay
Ryan McGreal
The crossing guard data is now up on RTH:
http://raisethehammer.org/data/crossing_guards.html
The HTML view plots the crossing guard locations on a Google map,
including links to a JSON format and a CSV format so you can download it
for other uses.
Cheers,
Ryan
Joey Coleman
I'll write something on the Open Hamilton website this afternoon
linking to the data.
- Joey
Ryan McGreal, Raise the Hammer
> I'll write something on the Open Hamilton website this afternoon
> linking to the data.
I had actually never coded against the google maps API before, so it was a
useful exercise. It turns out to be easier than I expected to feed
geocoding markers onto a map - I'll definitely be doing more of this in
future.
Note also that the raw data are available in JSON and CSV format, so it
should be easy to incorporate into other data stores.
Regards,
Ryan
Joey Coleman
Milton Friesen
_____________
Milton Friesen
Cardus :: research for the common good
Ingenuity Arts :: adaptive leadership
Profile
Recent Article
Adaptive Cities
Twitter: @ingenuityarts
Skype: ingenuityskype
James McKinney
> As the creator of the data the City of Hamilton is perfectly legal in
>
> We do not want to be regarded as rouge "hackers", unfortunately in the
> negative sense, who are not willing to abide by legal restrictions
> that have been imposed by the City. If we cannot abide by the legal
> restrictions imposed by the licensing how can we seriously expect to
> be able to deal with the City of Hamilton? How would we be able to
> negotiate with Councilors and the City's administration if the first
> impression of this group is that we do not abide by the licensing
> terms that are currently in place? Why should the City change
> anything, and especially be co-operative with us, when their
> impression is that we do what we want?
>
> By using the data sets that the City of Hamilton has available on
> their site in any way that is not permitted by the licensing then we
> are going to destroy any good will and any co-operation we may have
This assumes the City even knows its own licensing terms. The current
terms bear all the signs of what is common practice on the web:
1) an organization figures it needs a terms of use
2) it hires a law firm that does not have relevant experience
3) the law firm drafts the most aggressive possible terms of use
4) the city rubberstamps the terms of use and publishes it on its site
This sort of thing happens because insufficient consideration is given
to what the license should include, and thus insufficient instructions
are given to the lawyers, and so the lawyers of course write the most
protective license possible. AOL in particular has made headlines with
its terms of service, which it has subsequently modified. The
licensing on the recent data.gc.ca had some particularly wrong-headed
clauses, and, after they were brought to his attention, Minister Day
said they will be struck. I don't believe anyone in the City has
strong feelings about its licensing terms, and I think most people are
unaware of them.
This group's reputation with the City will be more strongly affected
by other concerns, e.g.:
1) Are you reaching out to the City? Are you writing to people in
relevant positions at the City? Are you inviting the City to public
meetings? In short, are you promoting a dialogue?
2) Are you approaching the City as a friend and partner, or as an
obstacle that you must push and pressure? Is your approach to educate
people about open data to get them on your side, or to bully them with
arguments?
It's important to understand politicians. Politicians move slowly.
They like to make announcements. They like to be in control. The
approach I favor for open data groups is to prepare the terrain for
politicians. If you push them, they will push back; things will drag
out. An open data policy can be achieve without pushing, and it will
probably happen sooner without it. For example, it is better to
propose a draft motion to sympathetic councillors, who may then take
initiative in council, than to arrive a dozen strong at question
period and demand they pass a motion you prepared.
James McKinney
at the City are prepared to offer you if you ask. For example,
Montreal publishes budget and election information in PDFs. A first
thought by many is "it's going to suck to scrape those." But with a
little effort, it was not hard to find someone at the City willing to
share the Excel files with the same data.
Ryan McGreal, Raise the Hammer
> This assumes the City even knows its own licensing terms.
I'm in correspondence with city legal manager Peter Barkwell. He's
preparing a response/explanation of the city's TOS and will send it as
soon as he can.
Regards,
Ryan
Joey Coleman
fulfilled as been done with a request asking if an FOI would be
required. This gives city staff a "cover your ass" paper trail.
On Mar 22, 12:16 pm, James McKinney <oxford.tux...@gmail.com> wrote:
> Last thought - sometimes, you will be surprised by what people working
> at the City are prepared to offer you if you ask. For example,
> Montreal publishes budget and election information in PDFs. A first
> thought by many is "it's going to suck to scrape those." But with a
> little effort, it was not hard to find someone at the City willing to
> share the Excel files with the same data.
>
> On Tue, Mar 22, 2011 at 12:11 PM, James McKinney
>
>
>
>
>
>
>
Ryan McGreal
> Let's have an early Easter Egg hunt and go searching for quick wins on
> the City website.
Here's another easy conversion:
http://www.hamilton.ca/CultureandRecreation/Recreation/CentresPoolsArenas/recCenterListing.htm
Data hosted on RTH in HTML, JSON and CSV format:
http://raisethehammer.org/data/recreation_centres.html
After getting through the coding for the crossing guard data, this one
was dead easy - less than half an hour total. Other tabular HTML data on
the site will be similarly easy to convert.
Regards,
Ryan
Joey Coleman
- Joey
Ryan McGreal
> I have to get that update to the OpenHamilton website done. :) Quick
> too, in 30 mins you'll have me doing a rewrite.
The speed and technical ease of many of these data conversions -
particularly the low-lying fruit - weakens the claim of open data
opponents that it will be 'too costly' to undertake.
Regards,
Ryan
NikGarkusha
geo-coded CSV.
Here it is in OGDI:
http://datadotgc.cloudapp.net/DataBrowser/def/OpenHamiltonCrossingGuards#param=NOFILTER--DataView--Results
I played around with filtering the data, use the Bar Charts tab to
visualize essentially the same limited tid-bits of information that
Ryan already shared on # of guards per specific ward, etc... The
dataset really could use some mashing up with some other data (i.e.
public school locations, traffic accidents)
Randy shared pedestrian accident data, I think it's probably possible
to geocode it with some effort, but I dont have the programming skills
to do that. If we had lat/lon for Randy's dataset, we could easily
mash up and see whether crossing guards are in the areas with most
accidents & proximity to public schools..... i.e. if there are many
accident around a particular school with fewer crossing guards,
perhaps that's an insight.
thoughts?
On Mar 22, 10:28 am, Joey Coleman <j...@joeycoleman.ca> wrote:
> Looking at the data, we could plot the locations with some creative
> scripting. However, we need to know what the time period covered is.
>
> It would be useful to know the dates of the incidents, number of cars and
> pedestrians involved, and extent of injuries. Further to that, it would be
> useful to know what direction the vehicles and pedestrians were travelling
> at the time of incident.
>
> - Joey
>
>
>
>
>
>
>
>
> > Hi - I have a spreadsheet from the city on pedestrian accident locations
> > (attached) - what does it need to have to be useful? I want to see this on a
> > map!
> > I love what you folks are doing, I'm hoping to learn a few things!
>
> > Randy
>
>
> >> Let's have an early Easter Egg hunt and go searching for quick wins on
> >> the City website.
>
> >> Here's a listing of Crossing Guard locations in Hamilton which could
> >> be fairly easily converted into an open dataset and mapped:
>
Joey Coleman
- Joey
NikG
Aaron McGowan
Joey Coleman
- Joey
NikG
Randy Kay
Ryan
Policy.
http://raisethehammer.org/article/1344/
They recognize that it needs to be updated and state clearly that they
have no intention of enforcing it.
Regards,
Ryan