Facebook Road AI data!!!!!!!!!!

[Justin]

free up a lot of space on your machine - all the roads in India missing from OSM

https://mapwith.ai/country_exports/IN_PK_mapwithai_road_data.gpkg.tar.gz

Have fun!

[Deepak Sharda]

can you please help me how to open this data as QGIS hunged when i tried to open this big file 

[Justin]

@Deepak, open it, and walk away for about 30 minutes. It will open. Try to have your area of interested centered on your screen, zoom into it. Then select and export the section you need, otherwise it is a bit difficult to work with. 

[Deepak Sharda]

thank a lot.

one more question. As osm doesnt have so many rural roads  in its data for India.

Does this file has all India rural road Data kindly tell is it worth.

u being so kind and helpful

thank you

On Tue, 28 Apr, 2020, 10:51 pm Justin, <justinell...@gmail.com> wrote:

@Deepak, open it, and walk away for about 30 minutes. It will open. Try to have your area of interested centered on your screen, zoom into it. Then select and export the section you need, otherwise it is a bit difficult to work with. 

--
Datameet is a community of Data Science enthusiasts in India. Know more about us by visiting http://datameet.org
---
You received this message because you are subscribed to the Google Groups "datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email to datameet+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/datameet/d6563043-1155-494b-8c85-1bbe20d94cbe%40googlegroups.com.

[Nikhil VJ]

Hi Devdatta, Ujaval

Thanks! Both ways look good. I got the ogr2ogr way working, but QGIS processing toolbox way looks promising too.

Hi Deepak,

That would be an unfair question to ask as the intention for FB to share the dataset is "this is what some of our programs have come up with by looking at lots of satellite imagery data etc. It might help you map roads that may have been missed so far, please see and check for yourself". 

So this is at an initial stage in a very long process. The reason why such initiatives exist is because there is no definitive "all rural roads" proper source. 

And whether the data does help or not may change from place to place with local condtions like soil color, tree cover etc impacting the ability of the programs in interpreting satellite imagery. So there can't even be any definite answer at centralized level - it may be helpful at some place and not at others - people will have to dig in and find out.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in

To view this discussion on the web visit https://groups.google.com/d/msgid/datameet/CA%2B4SsQBLaOqJAB%2BeOTpLe7czE4uB4PZkY%2BLzkzxOK%2B7ebX8uUQ%40mail.gmail.com.

[Deepak Sharda]

thanks a lot to all of you putting so much thought helping newbie like me.

i loaded the data with the help of ogr2ogr in my pgsql admin 4. all tables are there and i can see data there with vrt lat long set.

when i tried load into qgis my computer become unresponsive.

i tried to clip the data with ogr2ogr method but it didnt work it gave error no geometry

ogr2ogr -clipsrc 70.1 18.2 75.5 20.6 output.gpkg input.gpkg

will try new method and update. they data has pakistan coordinates in it when i checked some random points.

thank again

To view this discussion on the web visit https://groups.google.com/d/msgid/datameet/CAH7jeuMH2MVwmuBimUg_eo2w9LR9KTXZaBOuHPJOnHXNA-nndA%40mail.gmail.com.

[Nikhil VJ]

Hi, I learned from GIS stackexchange that "-spat" may be better than "-clipsrc" as it doesn't bother to cut features that extend beyond our rectangle. Which I'm fine with, and it saves a lot on processing time.

So this command worked better for me:

ogr2ogr -spat 70.1 18.2 75.5 20.6 output.gpkg IN_PK_mapwithai_road_data.gpkg

PFA a screenshot : I loaded OSM road data (highway=*) in white and this data (let's call it FB-AI) in orange over a satellite view. 

My observation: The FB-AI data has excluded existing roads that are already there on OSM it seems, and is marking possible roads that don't existi in OSM yet. And It seems to be doing a pretty good job of detecting unmapped roads.

Caveats: Inevitable consequence of trying to catch em all: False positives. It's marking any scratch it finds and many are just dirt tracks in the fields or even dried canals that may or may not be getting used as general thoroughfare. While local tracks used by tractors to go to fields are fine for mapping as dirt tracks (pls don't mark them as residential roads or unclassified), there is a greater priority to mark roads that can be used by regular vehicles without damaging those vehicles; roads than can be used in times of emergencies, for delivery of essentials and the likes. And you wouldn't want to send an ambulance trying to get to a landlslide affected area into an impasse while thinking they were taking a shortcut as shown on map.

For this reason, IMHO the current approach of not declaring this data as roads straightaway but rather feeding it in as suggestions and letting human mappers decide (see links below), is better. That being said, there can be purposes other than precise mapping, such as using this data as proxy for indicating other things, or aggregating over areas to compare between areas, or using this to map irrigation canals (:D), or using it to decide where to focus the next mapping efforts. This data can be useful and fill in (do mention the caveats in your summary!) for areas where OSM mapping hasn't had proper coverage yet.

I'm sharing source links:

Github page where all countries' data download links are posted:

https://github.com/facebookmicrosites/Open-Mapping-At-Facebook/wiki/Available-Countries

Ah, even there they're mentioning "Country exports contain only the AI predicted roads that are missing from OpenStreetMap."

The RapID editor that brings this data as pink suggestions into OSM's online editor for mapping:

https://mapwith.ai/rapid

The main website where there's explanation etc: https://mapwith.ai/

PS: Satellite imagery brought in to QGIS by copying over tile urls from http://leaflet-extras.github.io/leaflet-providers/preview/ under "XYZ Tiles" in left pane where we find the OpenStreetMap layer.

Regards

Nikhil VJ

https://nikhilvj.co.in

[Deepak Sharda]

thanks a lot you are a true gem 

i have not opened it yet but fingers crossed 

On Friday, November 22, 2019 at 9:30:29 AM UTC+5:30, Justin wrote:

[Deepak Sharda]

Now as I know how to extract data for any region. A new challenge is how to refine data as it does have lots of line data which are not roads in my case in Barmer rajasthan we have sand dunes which has shadow property with contrast in google earth. I am thinking to minimize this data with accuracy in three steps

1. First to clean data upto  village level 

   

   Get list of all villages of India with Lat Log address (most probably middle of village) 

2. create an buffer of 500 Mtr or 1 km or 2 km as per population index 

3. select Facebook ai data in that buffer region only and delete rest of lines 

4. merge that data with OSM line data ( only lines which are connected to an existing osm lines ) in other words if a line is not connected with any other line(road ) with OSM data with the help of query tool it will be deleted 

I believe this can be a method to refine data. If you agree or have better idea kindly guide with some instruction video or page as number of villages in india are more then 15 lak and manually this is next to impossible 

If we can publish that data on this group this will be ahead all the data source available even with bhuvan wms rj_roads doesnt have that much data.

Note - I have a shape file data that distinguish between urban and rural areas of India but accuracy is upto 85 % (area larger then actual area)