Mapping Local Government Directory to WikiData

[Thejesh GN]

LGD publishes some important IDs, that can be useful. I also think WikiData item Id as a primary key. I just started syncing both of them locally so, I can update the WikiData with missing Census Location IDs. States was easy, but districts turned out to be not so easy.

I have blogged here

https://thejeshgn.com/2021/03/20/mapping-local-government-directory-to-wikidata/

But here are the differences. Let me know what do you guys think.

WikiDataId	Label	Description	Comments
Q955977	South Arcot	Former district in Tamil Nadu, India	Needs to be marked as dissolved in WikiData
Q1900496	Bangalore	Former district in Karnataka, India	Needs to be marked as dissolved in WikiData
Q1606061	Andaman	Former district of the Andaman and Nicobar Islands	Needs to be marked as dissolved in WikiData
Q24949801	Shahbazwan	District of Bihar in India	is this same as GOPALGANJ district? Marked by mistake in WikiData. Should be removed as a district.
Q6007135	Imphal	Wikimedia disambiguation page	is ex-district. Was split. Needs to be marked as dissolved in WikiData
Q48731903	Noklak	District in India, Nagaland	New district. LGD needs update. January 20, 2021.
Q61746013	Narayanapet	District of Telangana, India	There seem to be a duplicate Narayanpet district (Q85787759); but Q61746013 was created earlier. DataCommons also uses the same. It also has
Q29025081	East Karbi Anglong	District of Assam, India	When KARBI ANGLONG was split. The western part became the new "West Karbi Anglong" and the rest remained part of "Karbi Anglong". There is no "East Karbi Anglong" as such. Should be removed in WikiData?
Q101088203	Bajali	district of Assam India	New district formed in 12 January 2021. LGD needs an update
DONT KNOW	Vijayanagara	district of Karnataka in India	New district formed in 2020/21. Needs an addition to LGD. May be mark Q1611788 as district in WikiData?
DONT KNOW	Chachaura	district of mp	Missing on LGD, WikiData and OSM. No gazette yet
DONT KNOW	Maihar	district of mp	Missing on LGD, WikiData and OSM. No gazette yet
DONT KNOW	Nagda	district of mp	Missing on LGD and WikiData. No gazette yet.
Q61439260	Pakke-Kessang	district of Arunachal Pradesh in India	It was missing from WikiData query results. Because it was not tagged as district. I updated WikiData.

Thej
--
Thejesh GN ⏚ ತೇಜೇಶ್ ಜಿ.ಎನ್
http://thejeshgn.com
GPG ID :  0xBFFC8DD3C06DD6B0

[Bodhisattwa Mandal]

Hi Thejesh,

The best place to discuss this is here - https://www.wikidata.org/wiki/Wikidata_talk:WikiProject_India

There are Wikidata contributors who had been working on this, who might respond there.

Thanks,

Bodhisattwa

--
Datameet is a community of Data Science enthusiasts in India. Know more about us by visiting http://datameet.org
---
You received this message because you are subscribed to the Google Groups "datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email to datameet+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/datameet/CAABnYsUTfHZnmisWitBKBAGRBkYQ2OA8%2BuuK46MwRy8uNqiWTg%40mail.gmail.com.

[Arun Ganesh]

Very cool Thejesh! The LGD dataset is definitely super useful to help reconcile various other datasets that reference any territory.

Have been maintaining a dump of all the other LGD lookups here https://github.com/planemad/india-local-government-directory . Would be great to have it merged with the datmeet repo and see how we can maintain an easy to access dump of https://lgdirectory.gov.in

To view this discussion on the web visit https://groups.google.com/d/msgid/datameet/CAHyrfgb4AybH9okfQ2S9-r71agRi5kkMGueTdt_hOQs%2B1e1eDw%40mail.gmail.com.

[Thejesh GN]

Bodhisattwa - Thank you. Added a note - https://www.wikidata.org/wiki/Wikidata_talk:WikiProject_India#Syncing_Local_Government_Data_to_WikiData

Arun - Sure. How do we proceed?

I also have the udise_districts and udise_blocks in the same SQLITE. udise_districts uses a completely different  udise_dist_code. I will try and map wikiDataId to this as well.

https://india-local-government-directory.glitch.me/india-local-government-directory/udise_districts

udise_blocks are completely different from blocks as a geographical area. I am not going to pick it up as of now.

https://india-local-government-directory.glitch.me/india-local-government-directory/udise_blocks

My plan to pickup sub-district after this.

Thej
--
Thejesh GN ⏚ ತೇಜೇಶ್ ಜಿ.ಎನ್
http://thejeshgn.com
GPG ID :  0xBFFC8DD3C06DD6B0

To view this discussion on the web visit https://groups.google.com/d/msgid/datameet/CA%2BGKQr0YAKYDsTHyo7H0X0CqOAtWGsvjAsp9eDDTt2QPcowH9g%40mail.gmail.com.

[Naveen Francis]

Hello 

To maintain the country subdivision data model, there is a task force in Wikidata. 

https://www.wikidata.org/wiki/Wikidata:Country_subdivision_task_force/India

Thanks,

naveenpf

[sreeram kandimalla]

Just an FYI, LGD mappings have been asserted in wikidata till the district level based on Thejesh's work and I verified them independently. 

Moving onto lower divisions( Tehsils/CD blocks ) now. The wikidata hierarchy for these is unclear and needs to be cleaned up. 

[Thejesh GN]

Thank you for letting us know Sreeram.

I had started working on Taluks. Its not that straightforward. I will keep the list informed.

Thej
--

Thejesh GN ⏚ ತೇಜೇಶ್ ಜಿ.ಎನ್

http://thejeshgn.com
GPG ID :  0xBFFC8DD3C06DD6B0

To view this discussion on the web visit https://groups.google.com/d/msgid/datameet/4a7f72db-d74e-4302-ab03-6abecd3970aan%40googlegroups.com.

[Arun Ganesh]

Sharing some of the LGD-Wikidata mapping that I had done from two years ago. Hopefully its of some use and can be a start.

https://docs.google.com/spreadsheets/d/1FhXYDgenJ9rzIrSfXbnajGW1jiMrzcsdvYkkslyFKow/edit#gid=0

To view this discussion on the web visit https://groups.google.com/d/msgid/datameet/CAABnYsV_Zv%2By43Vq%3DGywTGFBX74fNRJKsNXfXSZwJNvRb%2B3v7w%40mail.gmail.com.

[sreeram kandimalla]

@Thejesh: Calling it not straightforward would be an understatement :)

@Arun : I will try and use the data.

BTW, I did add LGD codes to Tehsils in Wikidata based on their existing census codes in Wikidata. I need to verify the names. 

Still around 800(Tehsils) + 1000(CD blocks) of the close to 4000 entries already in Wikidata left to be reconciled.

I have been using Paul Novosad's masala-merge to reconcile names. I wonder if libindic's inexactsearch can also be used for this. 

Also, this is the current status of the Subdistrict/Tehsil/CD Block entity class hierarchy in Wikidata.  

I think this hierarchy is incorrect and needs to be fixed. 

Out of curiosity, I also ran an analysis to check if blocks are co-terminus with tehsils in any of the states, based on data in LGD.

Here is the link to the gist if anyone is interested - https://gist.github.com/ramSeraph/66f4b4a8780e3e47932467776731416a

The results are still confusing mostly because block mapping in LGD is probably incomplete.

To view this discussion on the web visit https://groups.google.com/d/msgid/datameet/CA%2BGKQr1WE3jPNnF%3DToD1mZSNcWgyz-eg7KUk%3Da8YZoXLjR-S%3Dw%40mail.gmail.com.

[Arun Ganesh]

The results are still confusing mostly because block mapping in LGD is probably incomplete.

This is part of the problem, generally it seems LGD is still a WIP from subdistricts onwards. Only sometime in the last year did they update the missing taluks for Mumbai Suburban and Chennai districts even though it was always in existence. So the LGD unfortunately cannot be trusted to be current even though the creation of new entities seem to be quite prompt.

Coverage of subdistrict items in wikidatata is quite low. Most of the items that exist with the same name would be the item for the town. There are also cases where the Wikidata item may be missing the English label (example) making name matching a bit of a puzzle..

[sreeram kandimalla]

One more update here.. 

Wikidata syncing with LGD is done till subdistrict level. 

You can query them at https://w.wiki/8sWm

Also, I tried to add all alternate names I could find as aliases in wikidata.

Automating this syncing and updating periodically is something I might take up at a future date.

But I suspect manual intervention and review is going to be required for this.

If someone wants to have a go at it, I will try to provide support.

--

Datameet is a community of Data Science enthusiasts in India. Know more about us by visiting http://datameet.org
---
You received this message because you are subscribed to the Google Groups "datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email to datameet+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/datameet/CA%2BGKQr0jj9AWDanBrmUiWHXJ5WDOK3PUCd_AF7fKzUVehEwhVg%40mail.gmail.com.

[sreeram kandimalla]

Correction: query link is https://w.wiki/8sXT

[Arun Ganesh]

This is the kind of painful work that can drive most people insane. Kudos to you Sreeram, next level stuff!

To view this discussion on the web visit https://groups.google.com/d/msgid/datameet/5e924323-054a-4495-aad1-09251f1e3147n%40googlegroups.com.

[Sabarish, KSITM]

The UDISE uses a totally different blocks in Kerala We have educational districts and sub districts  and blocks and educational grouping is totally different from that of Revenue grouping.

Regards

Sabarish

To view this discussion on the web visit https://groups.google.com/d/msgid/datameet/37642410-34e5-44e4-9125-61575ad8be82n%40googlegroups.com.

[Thejesh GN]

Its not just in Kerala, all over India. Educational (UDISE) geographic boundaries of districts and below, don't match with revenue ones .

Thej
--

Thejesh GN ⏚ ತೇಜೇಶ್ ಜಿ.ಎನ್

http://thejeshgn.com
GPG ID :  0xBFFC8DD3C06DD6B0

To view this discussion on the web visit https://groups.google.com/d/msgid/datameet/CAKo_Kpj1SEaDLjvv2NR8HFN-EuumB%3DfdhEcuU0msYxfj10rYgQ%40mail.gmail.com.