UK Census Data

[Gary Stanley]

A somewhat general posting, but I'm looking to store some census data in a database and not 100% sure how to break it down table wise (I'm suffering from brain fade I think!) ... does anyone here currently work with UK Census data/store it in a database, and if so, would you be willing to share a pseudo schema?

[Ben Brumfield]

Hi Gary,

I've been working on getting FreeCEN's database of UK census data into a new search engine on and off for a few months now.  Our goal is to create a search engine a la http://freereg2.freereg.org.uk/ and our starting point is an existing system of VLD and DAT files which has been in use by the project for a couple of decades.

Take a look at the models beginning with "freecen" here:
https://github.com/FreeUKGen/MyopicVicar/tree/freecen_parsing/app/models

Models beginning with "freecen1_" are what we load the VLD and DAT files into -- our raw, un-transformed data.

Models beginning with "freecen_" are the working models we transform that raw data into, and what the search engine actually displays -- this should be of more interest to you.

The "search_record" and "place" models are what the search engine actually searches, and are pretty generic, so they can be common to FreeCEN and FreeREG.

I'd be happy to answer any questions about the data model or to point you to the code that actually uses it.

Ben

On Tue, Apr 12, 2016 at 3:24 AM, Gary Stanley <gmsp...@gmail.com> wrote:

A somewhat general posting, but I'm looking to store some census data in a database and not 100% sure how to break it down table wise (I'm suffering from brain fade I think!) ... does anyone here currently work with UK Census data/store it in a database, and if so, would you be willing to share a pseudo schema?

--

---
You received this message because you are subscribed to the Google Groups "rootsdev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rootsdev+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gary Stanley]

Excellent, thanks for this Ben ... I keep forgetting that FreeCEN exists (which is ridiculous as I use FreeREG all this time!) but what FreeCEN does is exactly what I am looking to achieve, but with a very specific record set.

I'm not much of a Ruby coder, and I'm working with PHP/MySQL ... but on first glance it seems easy enough to work through what is going on and the structure of the models. If I have any questions I'll be sure to ask, thanks for the offer of assistance, it's much appreciated. :o)

Gary

[Doug Blank]

Gramps 3.x has a third-party census addon that defines a collection of census formats in XML, and allows the entry of the data into a Gramps database, storing the census info as text attributes on the object. Starting page:

https://gramps-project.org/wiki/index.php?title=Census_Addons

and the census XML are stored here I believe:

https://sourceforge.net/p/gramps-addons/code/HEAD/tree/trunk/contrib/Census/census.xml

Gramps 4.x uses the "Form addon". Hope that is useful,

-Doug

On Tue, Apr 12, 2016 at 4:24 AM, Gary Stanley <gmsp...@gmail.com> wrote:

A somewhat general posting, but I'm looking to store some census data in a database and not 100% sure how to break it down table wise (I'm suffering from brain fade I think!) ... does anyone here currently work with UK Census data/store it in a database, and if so, would you be willing to share a pseudo schema?

[Tony Proctor]

There are no piece/folio/page/etc in this XML Doug. That's very important for citations of the England & WALES census, and I believe TNA stipulate that access be provided by those parameters when census data is indexed.

 

    Tony

[Doug Blank]

On Tue, Apr 12, 2016 at 8:18 AM, Tony Proctor <to...@proctor.net> wrote:

There are no piece/folio/page/etc in this XML Doug. That's very important for citations of the England & WALES census, and I believe TNA stipulate that access be provided by those parameters when census data is indexed.

I'd be glad to pass that bit of info onto the author. Do note that as far as I understand the system, anyone can edit the XML to add or remove fields.

-Doug