Major upgrade to planningalerts
Matthew Landauer
Just wanted to let you know that the Ruby on Rails port of
planningalerts is "done". I released it for testing at
test.planningalerts.org.au a few weeks ago, asked for feedback and
since I received none, I assumed it's perfect and ready for release to
the wider world. :-)
This morning the main site http://planningalerts.org.au is now using
the Ruby on Rails code.
As always please let the list know if you see anything wrong or file a
bug report at http://tickets.openaustralia.org
Matthew
Henare Degan
Cheers,
Henare
Ps. also a great way to double the number of alerts we've sent :P #sorry
--
You received this message because you are subscribed to the Google Groups "OpenAustralia Community" group.
To post to this group, send email to openaust...@googlegroups.com.
To unsubscribe from this group, send email to openaustralia-...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/openaustralia-dev?hl=en.
Matthew Landauer
"date_scraped" field on all the planning applications getting updated
(to the current time) which resulted in _all_ the old applications
getting resent.
Oops!
I fixed that pretty quickly...
Henare Degan
In a vaguely related point, can you define what the various date fields do (i.e. date_scraped, date_received, on_notice_to, on_notice_from)? I'm sure I could work it out, I just haven't had the time.
The reason I ask is I want to log tickets for all the scrapers that aren't full compliant with the spec[0], i.e. ones that pull every application, every time resulting in an "Application already exists in database" message from the parser.
Then on some dark, rainy weekend I might look at fixing some of them, including the Marrickville scraper, which could get the dates but the original developer was too lazy to bother including that functionality.
Cheers,
Henare
[0] http://tickets.openaustralia.org/browse/PA-110
Matthew Landauer
date_scraped is the time when the application is added to the
database. i.e. when it is scraped from the council website for the
first time.
date_received is the date when the council received the application -
that's figured out by the scraper, though not all councils give this
information.
on_notice_from and on_notice_to is the date range when the application
is "on notice" or "on display" or "on exhibition" which is when it is
officially open to feedback from the community. This information comes
from the council website (through the scraper) and is often not given
by specifically by the council. Also, we're currently not using this
information for anything on planningalerts though there are tickets
for making us of it: http://tickets.openaustralia.org/browse/PA-65
When you get "Application already exists in database" messages in the
scraper log:
As you can see on http://planningalerts.org.au/getinvolved the
scrapers accept a date and should do a search for all applications
that were submitted on that date to the council (i.e. with
date_received == date). For councils that don't give the date_received
info the best the scraper can do is just return a list of all the
development applications that it can find. And so that's why you'll
get the same applications being received on multiple days which are
then ignored as they match already pre-existing applications.
As for figuring out which scrapers have which data it might be a good
ideas to write a little admin interface that figures that out from the
applications in the database. Feel like adding a ticket at
http://tickets.openaustralia.org for that?
Matthew
Henare Degan
As for figuring out which scrapers have which data it might be a good
ideas to write a little admin interface that figures that out from the
applications in the database. Feel like adding a ticket at
http://tickets.openaustralia.org for that?
Cheers,
Henare
[0] http://tickets.openaustralia.org/browse/PA-140