Big List of Data Sources
Richard Milton
========================================================================================
London Underground Trains (TfL, Trackernet System)
http://cloud.tfl.gov.uk/TrackerNet/PredictionSummary/B
(B=Bakerloo line, C=Central line etc)
Line status for all lines i.e. Good Service, Minor Delays + Branch Disruptions
http://cloud.tfl.gov.uk/Trackernet/LineStatus
========================================================================================
London Buses and River Services (TfL, Countdown System)
Returns all arrival times for buses at every stop they will visit. Can filter by bus number.
========================================================================================
UK Trains (Network Rail)
http://www.networkrail.co.uk/data-feeds/
The feed is a stream using the STOMP protocol. Need to register with Network Rail, subscribe to the required feeds and use a STOMP library like the Apache one to receive messages.
Current feeds include real-time movements of all trains by train operating company, speed restrictions, signals and timetables.
========================================================================================
Aircraft ADS-B
Sites like flightradar24.com show real-time planes, but it is possible to buy your own ADB-B receiver and collect the data yourself.
========================================================================================
Boats AIS
London river services are covered by the TfL Countdown API, but sites like www.marinetraffic.com/ais and www.shipfinder.co plot the information on the web. Boats have a collision avoidance system
called AIS in a similar way to the aircraft, so you can buy receivers to collect your own data.
========================================================================================
Weather Data
UK Met Office: DataPoint Service http://www.metoffice.gov.uk/datapoint
Observations
(NOTE: all datapoint requests can use either json or xml by switching the xml or json part of the REST URL /wxobs/all/json/capabilities)
Make a get capabilities request:
http://datapoint.metoffice.gov.uk/public/data/val/wxobs/all/xml/capabilities?res=hourly&key=<APIkey>
Then request data for all UK stations (it is possible to filter by a specific station):
http://datapoint.metoffice.gov.uk/public/data/val/wxobs/all/xml/all?res=hourly&key=<API key>
SURFACE PRESURE
Make a capabilities request to get the available timesteps first, then enter the relevant timestep into the data request:
http://datapoint.metoffice.gov.uk/public/data/image/wxfcs/surfacepressure/gif?timestep=0&key=<mykey>
RAINFALL (and others)
Make a capabilities request first to find out what data is available;
http://datapoint.metoffice.gov.uk/public/data/layer/wxfcs/all/xml/capabilities?key=<mykey>
Then the format of the following requests follows this pattern:
Rainfall PPN Rate
http://datapoint.metoffice.gov.uk/public/data/layer/wxfcs/Precipitation_Rate/png?RUN=2013-09-25T09:00:00Z&FORECAST=0&key=<mykey>
Total Cloud Cover
http://datapoint.metoffice.gov.uk/public/data/layer/wxfcs/Total_Cloud_Cover/png?RUN=2013-09-25T09:00:00Z&FORECAST=0&key=<mykey>
Pressure (fronts) (note run time)
http://datapoint.metoffice.gov.uk/public/data/layer/wxfcs/Atlantic/png?RUN=2013-09-25T00:00:00Z&FORECAST=0&key=<mykey>
Weather data: synoptic, radar, satellite (UK Met Office DataPoint, GTS Feed)
Alternative sources of weather data exist in GTS format:
http://www.met.fsu.edu/index.pl/wxdata/reports
http://weather.cod.edu/digatmos/
(check NOAA site as you can g
These require understanding the WMO coding format outlined in the coding manual: http://www.wmo.int/pages/prog/www/WMOCodes/Manual/WMO306_Vol-I-1-PartA.pdf
(very technical and you also need parts B and C). There is code on the Internet that will decode a synop, metar or buoy message though.
Weather Underground
http://www.wunderground.com/
This also has an API for requesting weather data. There is no quality control though.
http://www.wunderground.com/weather/api/
========================================================================================
Air Quality
Use this to get a list of sites and lat/lon coordinates
http://uk-air.defra.gov.uk/ajax/import_map_data?doajax=true&n=aurn
Then get the data by scraping this html page
http://uk-air.defra.gov.uk/latest/currentlevels
Defra list of sites:
http://uk-air.defra.gov.uk/networks/aurn-sites
and data download:
http://uk-air.defra.gov.uk/data/
Other sources of data:
London Air Quality Network (LAQN, run by Kings College, contains some very good real-time pollution maps)
http://www.londonair.org.uk/LondonAir/Default.aspx
National Atmospheric Emissions Inventory (NAEI)
http://naei.defra.gov.uk/
========================================================================================
Census and ONS
UK 2001 and 2011 releases from CASWEB and NOMIS at OA, LSOA and MSOA levels
ONS Website:
http://www.ons.gov.uk/ons/guide-method/census/2011/index.html
NOMIS Census Homepage:
http://www.nomisweb.co.uk/census/2011
ONS Boundary File download (this is not easy to find):
https://geoportal.statistics.gov.uk/geoportal/catalog/content/filelist.page
If you're interested in all the Census tables, then NOMIS has a bulk upload of everything processed so far by ONS:
http://www.nomisweb.co.uk/census/2011/bulk/r3
========================================================================================
Geology
The British Geological Survey have various maps that can be downloaded:
http://www.bgs.ac.uk/
========================================================================================
OS Open Data
http://www.ordnancesurvey.co.uk/business-and-government/products/opendata-products.html
Primarily used for map data like roads, rivers, railways and building outlines. The vector datasets are the most useful for GIS. "Vector Map District" or "Boundary Line"
are the two most useful ones.
If you want to build a 3D building map, you can use the outlines from Vector Map District buildings, but you still need an additional source of height data to extrude the
building outlines. A LIDAR DEM file or some other source of heights is required, which is a lot harder to source.
========================================================================================
National Grid, power and gas
This is the root of the National Grid website that allows you to download cable and pylon shapefiles for electricity and gas:
http://www.nationalgrid.com/uk/Electricity/Balancing/services/
This is the top page relating to the data (not easy to find):
http://www.nationalgrid.com/uk/LandandDevelopment/DDC/
and data from here:
http://www.nationalgrid.com/uk/LandandDevelopment/DDC/GasElectricNW/
Don't get the PDFs from there, click the shapefile link to this page:
http://www.nationalgrid.com/uk/LandandDevelopment/DDC/GasElectricNW/undergroundcables/shape/
and download "all files", or "Gas Site", "Cable", "Line", "Substation Site", "Gas Pipe" or "Tower"
Also, there are "realtime"? files relating to amounts of electricity traded and demanded:
http://www.nationalgrid.com/uk/Electricity/Data/
(not really figured out what these are yet)
========================================================================================
Telecoms
General broadband statistics and maps of telephone exchanges:
http://www.samknows.com
BT OpenZone
You can often find data designed for uploading to a GPS on some commercial and retail sites:
http://www.btwifi.com/find/directories.jsp
Some of the GARMIN file types are just plain text, or use the Excel one.
========================================================================================
Other Geographic Data
OpenStreetMap: http://www.openstreetmap.org/ contains lots of free downloadable geographic data. There is also an API for querying specific features.
The Global Administrative Areas Project is very good for free international boundaries:
http://www.gadm.org/
========================================================================================
Datastores and other general sources of data
Data.gov.uk
London.data.gov.uk
Guardian Data Blog: http://www.theguardian.com/news/datablog (Also contains a list of other data stores around the world and an API to query it for data)
The US Data.GOV.UK site has a more accessible list: http://www.data.gov/opendatasites
It's worth looking at the UN, World Bank and
========================================================================================