2018 India Village GIS Data Project

[Justin]

I plan on trying to update individual states to 2018 standards/ amounts/ etc. I know this is a crazy idea, but I am going to start.

I have started with Goa, it has been a challenge. I thought I was going to finish over two weeks ago, but just haven't had time and have been doing more tracing of villages that what I thought I was going to have to do.

I am trying to align amounts with http://lgdirectory.gov.in/ and http://nlrmp.nic.in/faces/masterReport/masterReport.xhtml

Sometimes these may disagree and may be missing towns, but I am trying to make sure I account for those somehow. Once I finish, I will upload hoping for feedback as per what I did wrong, am missing, or anything else.

I think a state-by-state way is the only way to successfully do this. Trying to do all 6-8 lakh would be a nightmare, so breaking it down to several hundred - a few thousand would be easiest and manageable.

Please let me know if you would like to help, start mapping a state, or give any feedback, ideas, data, solutions, etc.

Thank you all so much!

Justin

[Dilip Damle]

HI, 

As already discussed I am in.

Would be available for checking, verifying  and specific digitization polygon checking etc.

I also go off and on when I have other priorities (of assignment work). 

My plan is to use a database and compare and accumulate the whole of India  as one/linked database (WITHOUT POLYGONS) 

I expect after the exercise I would have identified and remarked on every village.

Will do it independently from Census 1991, 2001, 2011 and the above two as later references  using them in REverse chronological order for Priority.
There is no point in saying where I will end up and when but it will happen. Keeping my fingers crossed.

[Justin]

Dilip,

This is great news! I knew you wanted to do some of this, I just wanted to start a new post about in instead of continuing on the other thread. Let me know what resources you need or if you need anything from me.

Thanks!!

Justin

[Justin]

okay; started to upload the data:

https://github.com/justinelliotmeyers/INDIA_2018_SHAPEFILE_BOUNDARIES

please give any feedback, poit out errors, what I did wrong, best way to move forward, etc. This is v1, so I expect issues to exist, and missing locations, bad spellings, code issues, etc.

I tried to clean it up as much as possible - in order to get this pushed out before the weekend I dissolved to a grid to keep geometry size down. We can review geometry after initial review of attributes.

I just threw everything in a repo; we can upload to a more formal project after this first stae, just wanted to start.

Thanks!!!!!

[Nikhil VJ]

Hi Justin and others,

All the very best for this effort. 

I want to share about our (Pune chapter's) experiences regarding Maharashtra Villages data. This might also help resolve some questions filed as issues our github repo.

To start with, we had quite some mismatch between the shapefile we got and the census 2011 village data (which we are considering as benchmark as it is a unique, well-ordered and government-ratified dataset that is published openly, whereas the shapefile is sourced informally). And we went with 2011 because with all things considered, 2011 is closer to present reality than 2001. 

After a few attempts by me at "fixing" things, we decided it was best not to mess with the shapes, as we were crossing many point-of-no-returns in the process and the output was looking more like Swiss cheese than a map. Rather, we could just add new columns/attributes to indicate the recommended changes. The agenda shifted to not necessarily fix and make a proper map, but to document what was wrong and where the corrections are needed, and hopefully someday send the feedback upstream and get the appropriate government agency (MRSAC in this case) to fix it. Or at least if we do publish a "fixed" version at some point, we need to have full documentation of exactly what changes have been done so that there is traceability.

And so we set up this tracking sheet.

It's a bit chaotic, there are several worksheets there. I won't be able to explain further here and it is a suspended work-in-progress.. . we have set some things up but left the tasks pending for potential volunteers and interns to take up. For those who want to know more about this, please reply with a different subject line or please post your queries on the #pune channel in datameet.slack.com .

We did have one immediate requirement of producing a shapefile for a web interface that Namita was developing that dynamically combines shapefile and census csv data. We needed the shapefile to have a census code column/attribute that is non-null and unique, to act as the primary key to match with census data. So, I created a version with a new column added where the repeating and null codes were suffixed with serial numbers. (that would render them unmatchable with the census data, but at least they would not interfere with the program). I have documented the process here.

So my suggestion for the all-states effort, would be to document all the changes, fixes done and keep ways of tracing back. (adding new columns and making changes there is one such way). It's much tougher, but it will help set up a situation where the fixes can be integrated back into official sources rather than being isolated forks. We can be skeptical of that ever happening, but the alternative is to have to repeat the entire exercise post 2021 census or any time the govt agencies republish the shapes. In any case keeping documentation of changes will speed up the next round as mistakes we find now will likely be repeated later.

PS: By "we" I am referring to myself, Craig, Devdatta, Namita, Riddhi, Jinda and some more people from Pune who contributed to the process (apologies for missing names!). We did a few meetups and then made a smaller focus group.

--
Cheers,
Nikhil VJ
+91-966-583-1250
Pune, India
Website <http://nikhilvj.co.in>

DataMeet Pune chapter <https://datameet-pune.github.io/>
Self-designed learner at Swaraj University <http://www.swarajuniversity.org>

Contribute <https://www.instamojo.com/@nikhilvj/>

--
Datameet is a community of Data Science enthusiasts in India. Know more about us by visiting http://datameet.org
---
You received this message because you are subscribed to the Google Groups "datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email to datameet+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dilip Damle]

Hi, Will have a look at things including Goa after about a week.

[J M]

Thank you! And if you know any colleagues familiar with Goa, please let me/ them know so we can get the best feedback!

Cheers,

Justin

--

Datameet is a community of Data Science enthusiasts in India. Know more about us by visiting http://datameet.org
---

You received this message because you are subscribed to a topic in the Google Groups "datameet" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/datameet/sWbFO8eAeDk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to datameet+u...@googlegroups.com.

[Justin]

I'm thinking of doing all the smaller states first; as per the list here: http://nlrmp.nic.in/faces/masterReport/masterReport.xhtml

Anyone have data, maps, resources they can share to help me get started on another state? Once I get feedback on Goa, I will go back and update where I can.

Thanks!!

Justin

[Justin]

Just wanted to check and see if there is any feedback on the Goa data. Idid not formally type up a recap as per what was edited or individual edits. I just jumped in and was trying to make sure I accounted for the list from http://lgdirectory.gov.in/ . There are still a few loose ends, but I will try to take care of them this week.

Any feedback on the actual data, geometry, attributes, etc. Or where villages are that I am missing would be greatly appreciated!

Thanks!

Justin

[Dilip Damle]

HI, 

I will work on it. Also send to some friends. But am stuck at the finishing stages of another project. (You know) 
Can you send a separate list of missing Villages only

[Dilip Damle]

Hi, 

I have done some correction about the missing information. Mostly Urban areas. Also attached the Names in the attribute table.

[Justin]

Dilip,

Thanks for sending this; I'm finalizing the detailed boundaries today/ tomorrow. Once finished, I will finalize the names fields. The LGD data doesn't have all town/ village names, But I believe they are in the wards tables, so some attributes can come from there. Also, I believe the 2011 census actually creates bad names for census towns. IE Canacona (Village) and Canacona (Town). I believe Canacona (Town) is actually Chaudi and Nagorcem - Palolem. So I believe the India census names locations based on an important city, or its parent subdivision for census towns just for reference or fromal govt services. But local people would consider the location a different name. They would understand Canacona, but not think of the main urban area as that, but I could be wrong (*probably am!?). But just want to make sure I am not creating a census names dataset, this is the actual villages (*not settlements either).

Thanks for all of your help!  Talk soon,

Justin

[Dilip Damle]

I am sending to some of my friends in Goa for review. I had to put names to make PDFs plus my own viewing, hence I put them. There will always be specific cases. of doubtful names. Which need to be discussed case by case.

[Justin]

Dilip, all my names in my final dataset match your except:

 Sanguem (M Cl) (74.004066  15.175658 ) I have Cuncolim or Cuncolim (M Cl)

Siolim (CT) (73.795308  15.626771) **This is because I split out Sodiem (or Sudiem) and have Siolim and Sodiem separate. This is a very tricky area because Siolim is split into numerous subdivisions (https://en.wikipedia.org/wiki/Siolim#Subdivisions )

On Monday, April 9, 2018 at 6:41:51 AM UTC-4, Dilip Damle wrote:

[Pithani Venkatesh]

Hi,

If u have remaining village boundaries of all Indian states please share for me or guide me how can I get it from online

[Rahul Basu]

Hi, I'm in Goa, could look at the data. 

BTW, Goa had a Regional Plan 2021 where the whole state was mapped down to houses in layers using GIS. We only have images but at the village panchayat level as very detailed JPGs. Happy to share them, will help with your shapes as well.

Rahul

[J M]

here is the data: https://github.com/justinelliotmeyers/INDIA_2018_SHAPEFILE_BOUNDARIES/tree/master/GOA_2018

Please do share!

Justin

--

Datameet is a community of Data Science enthusiasts in India. Know more about us by visiting http://datameet.org
---
You received this message because you are subscribed to a topic in the Google Groups "datameet" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/datameet/sWbFO8eAeDk/unsubscribe.

To unsubscribe from this group and all its topics, send an email to datameet+unsubscribe@googlegroups.com.

[Justin]

Rahul,

Do you have any data you could share? I would like to review as per the data I have created. Thanks!

Justin

[Naraina aka Dilip Damle]

Hello, 

I was coming back to this project 

There were basically 3 independent Sources for Village lists 

1. Census ... Various years  http://www.censusindia.gov.in/DigitalLibrary/Tables.aspx

2. Local Government Body directory  : http://lgdirectory.gov.in/

3. Rural Ministry Village data : I have the data but do not recollect the soyurce URL now 



Now I see a fourth different data  from Water Ministry 

4. Habitation directory, National Drinking water Programme  :  http://indiawater.gov.in/IMISWeb/DataEntry/HabitationDirectory/Reports/Rep_ViewStatewiseDirectory.aspx?Rep=Y 

I do not know if 3 and 4 are same will need to study 

[Justin]

Rahul,

Any updates? Thanks again!

Justin

On Friday, May 11, 2018 at 11:55:06 AM UTC-4, Rahul Basu wrote:

[Sharad Lele]

Dear Dilip:

Firstly, thanks for sharing this 4th source. Having the list of hamlets (so-called "Settlements") is really useful.

Secondly, I am guessing that the second last column (village names) should match the 2011 Census village list. But have not done a thorough check.

Sharad

[Justin]

Just wanted to check back and see if there are any updates as per mapping new states. I've been super busy and havent had a chance to start another state yet, but wanted to see if anyone else has started mapping villages?

Thanks!!

Justin

[Dhaval Shukla]

Hey

Naraina had done something with Gujarat district and village shape files some time ago, but for the life of me I am not able to find it on this group today. I downloaded it at the time about a week ago.

This village and district data hasn't been uploaded to the datameet github nor yours.

Naraina, I wanted to know if there is a reason why. It is most accurate I have found of Gujarat till date (although there are some errors). 

On Wednesday, 21 March 2018 20:04:16 UTC+5:30, Justin wrote:

[Naraina Damle]

Hi Dhaval, 

Actually I keep doing things in an Intermittent manner. 
Then I move to something else. 

However I keep all my data safe. 

Here is the link to the Datameet folder on My Google Drive. 

There could be other things at other places. But Gujarat Data is here. 

https://drive.google.com/drive/folders/0Bw9Ef2_V5qDeOTFwV1U1WDA5Rnc?usp=sharing

[Dhaval Shukla]

Great thanks.

I later found it on another post. 

Would it be possible to upload this on the Github here or the Datameet maps portal with the disclaimer that they are not 100% accurate, so that people could use it at their own risk?

[Naraina Damle]

Yes, 

But I think it is better that people who handle that should do it.
I do not tamper with the GITHUB Uploading. I am not that proficient in its working.