street address database for australia
4,021 views
Skip to first unread message
cij
Jul 26, 2011, 10:06:44 PM7/26/11
to Silicon Beach Australia
hi, just reaching out to the community for tips on how to quickly
source an australian national street address/suburb database for use
in a website startup. No luck yet going through aussie post and
sensis. Other suggestions welcome. Thanks!
source an australian national street address/suburb database for use
in a website startup. No luck yet going through aussie post and
sensis. Other suggestions welcome. Thanks!
Craig
Jul 27, 2011, 6:42:58 AM7/27/11
to Silicon Beach Australia
VB
Jul 27, 2011, 6:58:37 AM7/27/11
to Silicon Beach Australia
Do you actually want to populate a database with each Australian
address? Or do you want to validate/sanitise data as it comes in?
I'm not aware of anywhere that just lists all addresses. Besides, I
reckon that list would be out of date before it's finished
downloading. It would be very expensive to maintain, which is probably
why Sensis isn't prepared to just hand their data over. If you do find
a source please be sure to post it back here.
I know of these address validation products:
Sensis TotalCheck
http://www.sensisdata.com.au/products/white-pages-totalcheck.html
Intech Solutions
http://www.intechsolutions.com.au/
Cheers,
Vincent
address? Or do you want to validate/sanitise data as it comes in?
I'm not aware of anywhere that just lists all addresses. Besides, I
reckon that list would be out of date before it's finished
downloading. It would be very expensive to maintain, which is probably
why Sensis isn't prepared to just hand their data over. If you do find
a source please be sure to post it back here.
I know of these address validation products:
Sensis TotalCheck
http://www.sensisdata.com.au/products/white-pages-totalcheck.html
Intech Solutions
http://www.intechsolutions.com.au/
Cheers,
Vincent
Message has been deleted
Yoo-Jin
Jul 27, 2011, 7:53:49 AM7/27/11
to silicon-bea...@googlegroups.com
Hi,
This probably won't be that helpful since you mentioned the word quick but you also mentioned startup. So I assume cheap is good too.
The states are slowly opening up address information for free. Unfortunately, it's very fragmented. Each state is responsible for the delivery method - if they open it at all. This means even if every state opened up their info you would still have to piece it all together. A bit of a nightmare. Hopefully this changes with the push to open data in the government.
NSW has an address validation service - free. I don't know about the other states though. You can each state's gis website (if they have one).
Source Website: https://six.nsw.gov.au/wps/portal/!ut/p/b1/04_SjzQ0NTczs7A0NDHTj9CPykssy0xPLMnMz0vMAfGjzOKDnZxCTR0NzQwNzJ1dDDwNzfxMnEMNvfx9DYEKIvEocDYkTr8BDuBoQEh_uH4UXiUgF4AV4LEiNyrHzdIzXREAaCwqTg!!/
Online Validation Service: http://address.maps.nsw.gov.au/AVSWebClient/faces/index.jsp
Web Service WSDL: http://maps.nsw.gov.au/wsdl/AddressSearch.wsdl
Good luck. Maybe you can post the results of your search too?
- Yoo-Jin
This probably won't be that helpful since you mentioned the word quick but you also mentioned startup. So I assume cheap is good too.
The states are slowly opening up address information for free. Unfortunately, it's very fragmented. Each state is responsible for the delivery method - if they open it at all. This means even if every state opened up their info you would still have to piece it all together. A bit of a nightmare. Hopefully this changes with the push to open data in the government.
NSW has an address validation service - free. I don't know about the other states though. You can each state's gis website (if they have one).
Source Website: https://six.nsw.gov.au/wps/portal/!ut/p/b1/04_SjzQ0NTczs7A0NDHTj9CPykssy0xPLMnMz0vMAfGjzOKDnZxCTR0NzQwNzJ1dDDwNzfxMnEMNvfx9DYEKIvEocDYkTr8BDuBoQEh_uH4UXiUgF4AV4LEiNyrHzdIzXREAaCwqTg!!/
Online Validation Service: http://address.maps.nsw.gov.au/AVSWebClient/faces/index.jsp
Web Service WSDL: http://maps.nsw.gov.au/wsdl/AddressSearch.wsdl
Good luck. Maybe you can post the results of your search too?
- Yoo-Jin
Babak M
Jul 27, 2011, 8:33:44 AM7/27/11
to Silicon Beach Australia
Can I ask what you want to do with it?
Rich
Jul 27, 2011, 8:39:30 AM7/27/11
to Silicon Beach Australia
It depends on what data you want and what you want to do with it, but
openstreetmap may have what you need.
See here: http://wiki.openstreetmap.org/wiki/Planet.osm#Processing_the_File
for some ideas.
You can get Aus only subsets from
http://downloads.cloudmade.com/oceania/australia_and_new_zealand/australia#downloads_breadcrumbs
openstreetmap may have what you need.
See here: http://wiki.openstreetmap.org/wiki/Planet.osm#Processing_the_File
for some ideas.
You can get Aus only subsets from
http://downloads.cloudmade.com/oceania/australia_and_new_zealand/australia#downloads_breadcrumbs
angus
Jul 28, 2011, 12:30:00 AM7/28/11
to Silicon Beach Australia
The most timely, and complete national coverage is from PSMA. You deal
with a reseller rather than them direct. Your options include
licensing the raw data and dealing with the rather complex data model
(call point spatial is one supplier I can vouch for) or you can deal
with a value added reseller who will shield you from the complex data
model and possibly provide web services or other value add.
Depending on the application most people just use the second rate
google API. Second rate only since they switched from PSMA data to
Sensis data. The PSMA data is high quality but you pay for it. PSMA
data has the quality as it sources the data from each state government
who in turn get it from the 600 odd local governments in Australia.
It all depends, as others have mentioned what you need it for.
with a reseller rather than them direct. Your options include
licensing the raw data and dealing with the rather complex data model
(call point spatial is one supplier I can vouch for) or you can deal
with a value added reseller who will shield you from the complex data
model and possibly provide web services or other value add.
Depending on the application most people just use the second rate
google API. Second rate only since they switched from PSMA data to
Sensis data. The PSMA data is high quality but you pay for it. PSMA
data has the quality as it sources the data from each state government
who in turn get it from the 600 odd local governments in Australia.
It all depends, as others have mentioned what you need it for.
cij
Jul 28, 2011, 4:01:50 AM7/28/11
to Silicon Beach Australia
Many thanks for all the suggestions. really appreciate it.
specifically, m starting with simple requirements i.e. looking for
list of street names, suburbs and postcodes. Not looking to validate
specific street addresses. geocoding requirements may come later.
@Rich, i also started going down the openstreetmaps path. As it is
community effort to maintain, not clear on frequency of updates.
yeah most of the services i enquired about are expensive! e.g. access
to GNAF which has geocoding (also offered via PSMA, thanks Angus)
specifically, m starting with simple requirements i.e. looking for
list of street names, suburbs and postcodes. Not looking to validate
specific street addresses. geocoding requirements may come later.
@Rich, i also started going down the openstreetmaps path. As it is
community effort to maintain, not clear on frequency of updates.
yeah most of the services i enquired about are expensive! e.g. access
to GNAF which has geocoding (also offered via PSMA, thanks Angus)
cij
Jul 28, 2011, 9:06:33 PM7/28/11
to Silicon Beach Australia
hi folks,
ok, so my downgraded requirements of using street names only didn't
sit well with my product manager : ).. had a debate with her last
night. her and the developers insist on a highly detailed database of
street addresses. i'm concerned about cost to business and whether we
need it ; )
the goal is to validate delivery addresses that customers enter into
website.. to avoid phantom addresses and prank orders. we are planning
on offering a COD service for customised low cost products.
i'm looking now at AMAS address matching providers linked to the
national Postal Address File. not sure if they can interact with a
website in real time to check addresses
@Yoo-Jin, thanks for the NSW address validation link!
back to my debate with the product manager. My take is that address
validation of that sophistication may be an overkill/overengineering?
and what if we reject customers that made minor typos in their address
that a human postie can figure out, but machine logic rejects ; ) (i
will check field structure and tolerances in AMAS)?
interested to hear your takes on this debate? : ) what's the
experience in oz regarding how common prank orders are for website
businesses? or is this problem normally addressed through deposit or
prepayment i.e. don't trust the customer ; )
ok, so my downgraded requirements of using street names only didn't
sit well with my product manager : ).. had a debate with her last
night. her and the developers insist on a highly detailed database of
street addresses. i'm concerned about cost to business and whether we
need it ; )
the goal is to validate delivery addresses that customers enter into
website.. to avoid phantom addresses and prank orders. we are planning
on offering a COD service for customised low cost products.
i'm looking now at AMAS address matching providers linked to the
national Postal Address File. not sure if they can interact with a
website in real time to check addresses
@Yoo-Jin, thanks for the NSW address validation link!
back to my debate with the product manager. My take is that address
validation of that sophistication may be an overkill/overengineering?
and what if we reject customers that made minor typos in their address
that a human postie can figure out, but machine logic rejects ; ) (i
will check field structure and tolerances in AMAS)?
interested to hear your takes on this debate? : ) what's the
experience in oz regarding how common prank orders are for website
businesses? or is this problem normally addressed through deposit or
prepayment i.e. don't trust the customer ; )
David
Jul 29, 2011, 1:06:03 AM7/29/11
to silicon-bea...@googlegroups.com
AusPost addresses are very accurate, updated quarterly and a solid solution for existing businesses. I found that shopping around with Australia Post or their many resellers will give you their current address database with updates for not much less than $16k. The actual price depends on your reseller.
If that's too much then per-request validation is a good option. You can get per-request access for input validation for as little as $2k in some cases, including the choice between post-entry validation and type-ahead suggestions.
You can get quite reasonable pricing from a reseller like DataTools. I recommend getting in touch with them about your options.
> Interested to hear your takes on this debate? : ) what's the xperience in oz regarding how common prank orders are for website businesses? or is this problem normally addressed through deposit or prepayment i.e. don't trust the customer ; )
If no one on your team already has a security background to guide you on this, you should look into having someone hone their knowledge of security best-practices and/or pay a consultant to help lock your system down. The main concerns are usually credit card fraud, especially from suspicious overseas IPs, plus if you're unlucky or profitable enough, hackers will regularly turn your site upside down looking for security holes. Sometimes a real customer will put incorrect details in their order as well. My experience with online ordering is that malicious orders backed by real payments, eg an ex husband paying for pizza to be delivered at 4am to his ex-wife's house, are pretty rare.
Niki Scevak
Jul 29, 2011, 7:19:46 PM7/29/11
to Silicon Beach Australia
Google Maps have a free api that includes 2,500 geocoder calls a day:
http://code.google.com/apis/maps/documentation/geocoding/
Geocoding, although slightly different, also includes scrubbing and
cleaning address inputs.
http://code.google.com/apis/maps/documentation/geocoding/
Geocoding, although slightly different, also includes scrubbing and
cleaning address inputs.
Babak M
Jul 29, 2011, 8:58:34 PM7/29/11
to Silicon Beach Australia
To me it seems like you are overthinking it (however I don't know how
many orders you are going to get per day, how much the average order
is going to be, etc...).
From a security point of view these are the points you should be
concerned about:
1. Human hackers submitting malicious input trying to find XSS and SQL
injection vulnerabilities
2. Spam bots making random submissions
3. Someone just trying to have fun with your form, targetted mass
submission.
4. Fake paid orders with incorrect details.
The first three apply to almost all online forms. A combination of
client-side and robust server-side input validation will take care of
most of the problems.
(also: captcha, ip blacklist, x submissions per ip per y minutes, form-
tokens)
4 as David said is very rare (also consider the nature of the
product).
If it was me I would start with some basic input validation just so to
prevent security issues.
Then you can make a better decision based on the actual orders you
receive.
I'd rather deal with those 1% fake orders than pay thousands of
dollars to prevent them.
many orders you are going to get per day, how much the average order
is going to be, etc...).
From a security point of view these are the points you should be
concerned about:
1. Human hackers submitting malicious input trying to find XSS and SQL
injection vulnerabilities
2. Spam bots making random submissions
3. Someone just trying to have fun with your form, targetted mass
submission.
4. Fake paid orders with incorrect details.
The first three apply to almost all online forms. A combination of
client-side and robust server-side input validation will take care of
most of the problems.
(also: captcha, ip blacklist, x submissions per ip per y minutes, form-
tokens)
4 as David said is very rare (also consider the nature of the
product).
If it was me I would start with some basic input validation just so to
prevent security issues.
Then you can make a better decision based on the actual orders you
receive.
I'd rather deal with those 1% fake orders than pay thousands of
dollars to prevent them.
Jeromy Evans
Aug 1, 2011, 8:41:55 AM8/1/11
to Silicon Beach Australia
Hi cjs,
Forcing validation of addresses introduces friction for your users.
Unnecessary friction will result in lost sales.
It's certainly a waste of your resources to develop this before the
problem of fake/phantom addresses even exists for you. If it's an
assumption, you should have a really high cost-risk of fake addresses
to justify it. There may simpler alternatives to mitigate that risk.
The best databases don't always work too. My business pays for an MBE
post box (Suite 294, 4 Young Street...) which is a valid mail &
courier address but not AFAIK not accepted by AMAS. Similar problems
occur with virtual offices/suites addresses. Likewise I used to work
on a defence site that had it's own postcode which was often
(stupidly) rejected.
You're likely better off developing a helpful address validator.
Perhaps it could flag "risky" addresses for you. You should argue
with your PM about the risk/cost analysis and other strategies to
mitigate that risk (credit card number, send an SMS validation code
etc). Hard to say without knowing more about the business.
Finally, Niki's pointer to the Google Maps goecoding API is
excellent. It does such a good job parsing addresses.
regards,
Jeromy Evans
Forcing validation of addresses introduces friction for your users.
Unnecessary friction will result in lost sales.
It's certainly a waste of your resources to develop this before the
problem of fake/phantom addresses even exists for you. If it's an
assumption, you should have a really high cost-risk of fake addresses
to justify it. There may simpler alternatives to mitigate that risk.
The best databases don't always work too. My business pays for an MBE
post box (Suite 294, 4 Young Street...) which is a valid mail &
courier address but not AFAIK not accepted by AMAS. Similar problems
occur with virtual offices/suites addresses. Likewise I used to work
on a defence site that had it's own postcode which was often
(stupidly) rejected.
You're likely better off developing a helpful address validator.
Perhaps it could flag "risky" addresses for you. You should argue
with your PM about the risk/cost analysis and other strategies to
mitigate that risk (credit card number, send an SMS validation code
etc). Hard to say without knowing more about the business.
Finally, Niki's pointer to the Google Maps goecoding API is
excellent. It does such a good job parsing addresses.
regards,
Jeromy Evans
cij
Aug 2, 2011, 9:40:39 PM8/2/11
to Silicon Beach Australia
Thanks all for the feedback
@David, i was quoted upwards of $20K pa by one provider to license the
database!! and they mentioned that Aus Post also won't give you access
until you pass a rigorous/expensive certification. Was this your
experience too?
@Niki, how did you find the speed of the Google Maps geocoder calls?
@Babak, i like your comment "I'd rather deal with those 1% fake orders
think it's their role to manage this.
as a compromise, we may be trialling an AMAS cloud-hosted solution to
check speed and accuracy. Though the per validation cost will add up
for this kind of solution! again should be back to risk assessment and
cost/benefit analysis.
cheers,
CJ.
@David, i was quoted upwards of $20K pa by one provider to license the
database!! and they mentioned that Aus Post also won't give you access
until you pass a rigorous/expensive certification. Was this your
experience too?
@Niki, how did you find the speed of the Google Maps geocoder calls?
@Babak, i like your comment "I'd rather deal with those 1% fake orders
than pay thousands of
dollars to prevent them".
@Jeromy, Similarly good point: "Forcing validation of addresses
dollars to prevent them".
introduces friction for your users.
Unnecessary friction will result in lost sales"
i'm trying to align the PM on business requirements thought you would
Unnecessary friction will result in lost sales"
think it's their role to manage this.
as a compromise, we may be trialling an AMAS cloud-hosted solution to
check speed and accuracy. Though the per validation cost will add up
for this kind of solution! again should be back to risk assessment and
cost/benefit analysis.
cheers,
CJ.
Clifford Heath
Aug 2, 2011, 9:52:50 PM8/2/11
to silicon-bea...@googlegroups.com
On 03/08/2011, at 11:40 AM, cij wrote:
> @Niki, how did you find the speed of the Google Maps geocoder calls?
> @Niki, how did you find the speed of the Google Maps geocoder calls?
IME, it's fine, but don't make your users wait for it. Do it
asynchronously,
if only because you don't want to fail the transaction if Google is
slow or
says no.
Clifford Heath.
Tim Clark
Aug 2, 2011, 9:50:26 PM8/2/11
to Silicon Beach Australia
Hey mate,
If you are shipping in the US, and are using UPS, they have a free address validation service - https://www.ups.com/upsdeveloperkit/downloadresource?loc=en_US. It works in other countries too, but I'm not sure of the accuracy (and UPS want you to use their services, which might be a goer, depending on what you are sending / how fast it needs to get there.)
We ship about 12 000 things around the world every month (sending stuff via Aus Post and UPS), and the number of addresses that needed to be changed or are undeliverable is about 1.3% (though this includes things like changing 101st Street to 101st Avenue), so the number of fake addresses has to be lower than this.
Most of our outrageous fake addresses are associated with stolen credit cards - and our fraud system picks these up before we get to the needing to deliver stage.
Hope this helps!
Tim
On 03/08/2011, at 11:40 AM, cij wrote:
> --
> You received this message because you are subscribed to the Silicon Beach Australia mailing list. Visting http://siliconbeachaustralia.org for more
>
> Forum rules
> 1) No lurkers! It is expected that you introduce yourself.
> 2) No jobs postings. You can use http://siliconbeachaustralia.org/jobs
>
>
> To post to this group, send email to
> silicon-bea...@googlegroups.com
> To unsubscribe from this group, send email to
> silicon-beach-aus...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/silicon-beach-australia?hl=en?hl=en
Matthieu Stone
Aug 2, 2011, 10:03:34 PM8/2/11
to silicon-bea...@googlegroups.com
You can try Google maps geocoding with this example.
--
You received this message because you are subscribed to the Silicon Beach Australia mailing list. Visting http://siliconbeachaustralia.org for more
Forum rules
1) No lurkers! It is expected that you introduce yourself.
2) No jobs postings. You can use http://siliconbeachaustralia.org/jobs
To post to this group, send email to
To unsubscribe from this group, send email to
Sai Lavu
Aug 2, 2011, 11:07:28 PM8/2/11
to Silicon Beach Australia
Hi CJ,
We provide AMAS PAF data as a RESTful web service which allows you to
integrate type as you go functionality quite quickly. Check out
http://www.datasan.com.au/
Regarding the comment about forcing users - yes I agree too. Plus even
AMAS data isn't up to date. So we provide advice/best matches as the
user types. For an internet facing app this is better approach. For
intranet facing app you can perhaps mandate.
We provide probably the most fair usage and pricing options if you are
interested. Plus we are L2 providers so we get the data directly from
Aus Post allowing us complete control over implementation & cost
passed on to the customer.
cheers,
Sai.
We provide AMAS PAF data as a RESTful web service which allows you to
integrate type as you go functionality quite quickly. Check out
http://www.datasan.com.au/
Regarding the comment about forcing users - yes I agree too. Plus even
AMAS data isn't up to date. So we provide advice/best matches as the
user types. For an internet facing app this is better approach. For
intranet facing app you can perhaps mandate.
We provide probably the most fair usage and pricing options if you are
interested. Plus we are L2 providers so we get the data directly from
Aus Post allowing us complete control over implementation & cost
passed on to the customer.
cheers,
Sai.
VB
Aug 3, 2011, 8:50:23 AM8/3/11
to Silicon Beach Australia
Thanks Sai, I'd like to get more information on the DataSan product,
is it production ready? I couldn't get much information from the
website as the link to the brochure seems to be broken. I'm looking at
alternatives for TotalCheck (to integrate into NetSuite) for one of my
clients.
Also, this one's for CIJ, just today, I've come across a product that
claims to have 12.6 Million Australian addresses and their geocode:
http://www.psma.com.au/products/gnaf.html
However, I don't think you can get direct access to it though, you
need to use a value-add reseller such as http://www.qas-experian.com.au/
Cheers,
Vincent
On Aug 3, 1:07 pm, Sai Lavu <sai.l...@systempartners.com> wrote:
> Hi CJ,
>
> We provide AMAS PAF data as a RESTful web service which allows you to
> integrate type as you go functionality quite quickly. Check outhttp://www.datasan.com.au/
is it production ready? I couldn't get much information from the
website as the link to the brochure seems to be broken. I'm looking at
alternatives for TotalCheck (to integrate into NetSuite) for one of my
clients.
Also, this one's for CIJ, just today, I've come across a product that
claims to have 12.6 Million Australian addresses and their geocode:
http://www.psma.com.au/products/gnaf.html
However, I don't think you can get direct access to it though, you
need to use a value-add reseller such as http://www.qas-experian.com.au/
Cheers,
Vincent
On Aug 3, 1:07 pm, Sai Lavu <sai.l...@systempartners.com> wrote:
> Hi CJ,
>
> We provide AMAS PAF data as a RESTful web service which allows you to
Reply all
Reply to author
Forward
0 new messages