Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
A genealogical commons
24 views
Skip to first unread message
Richard Smith
Dec 29, 2011, 9:10:04 AM12/29/11
to
A recent thread on soc.genealogy.methods under the subject "Genealogical
Estate Planning" has started me thinking about the problem of preserving
the results of our genealogical research so that it will still be
accessible to future generations, and I'm coming to the view that there
is specific hole in the genealogy marketplace for a service that does
this. What follows is quite a long summary of the sort of service I
could envisage someone providing quite easily, but so far as I know,
no-one is currently doing. I'd be interested if anyone knows of any
project along these lines, and whether you agree with my analysis of how
best to preserve your research.
(I did initially post this to soc.genealogy.methods, but it seems either
the moderator(s) is/are away, or there's a problem with the moderation
software and/or my news feed. In any case, I think this is the more
appropriate group.)
DIGITAL DATA
The most reliable way of ensuring that something still exists several
centuries in the future is to make sure that as many copies as possible
exist in different places. Losses can happen in even the most secure of
archives, but with many copies, if some get destroyed, others still
stand a chance of surviving. Publishing research in book form is one
way of achieving this, but this can be costly, take a lot of time, and
many people are reluctant to publish things they consider still to be
works in progress. What is needed is an easier way to publish research:
a way so easy that every little breakthrough can be published.
Almost all our research can be converted into digital format:
speculation, inferences and deductions can be written down; texts can be
transcribed; photos and certificates can be scanned; memorials, statues,
paintings and medals can be digitally photographed; audio and video are
often already digitised these days and can be made so if they're not.
Digital information can easily be shared over the Internet, and it does
not degrade as it is copied and recopied. (There are problems when
formats become obsolete, and serious though they are, I don't want to
dwell on them in this post.) Putting digitised research online seems
easiest way of publishing it.
Online data, however, often has no permanence. If you publish on a
personal website, who will pay the hosting costs long after you've died?
If it's a free hosting provider, will it still exist in twenty years'
time? In the mid-90s geocities.com was the most popular free provider,
but sites created there then no longer exist. What about ancestry.com?
That has a better chance of surviving because has an obvious, viable
business plan. But its ultimate aim is to make money, not to preserve
your data. You have no guarantee that they won't delete your data, and
even if they don't they might start charging prohibitively to access it.
Trusting the preservation of your research exclusively to one company
is a bad strategy. (Throughout this post I use ancestry.com as an
example of a commercial genealogy site. My comments about it could be
equally be considered as general comments on an arbitrary commercial
genealogy site.)
A GENEALOGICAL COMMONS
The best way to ensure the long-term preservation of digital data has to
be to make sure it is continually being copied to new places, and not
tied to fortunes of a few present-day companies. When a new genealogy
company sets up, you need them to be legally free to import your
research onto their site. That freedom to copy in perpetuity is
essential. If I download someone's research from ancestry.com, I find
it marked with ancestry.com's copyright statement and I am seemingly not
free to publish it elsewhere. (In fact, my understanding is that
copyright typically remains with the original researcher, despite the
notice seemingly to the contrary, but either way, I do not have
permission to publish research downloaded from ancestry.com.) If the
original data had been clearly and unambiguously made available under a
"share-alike" or "copyleft" licence, such as the Creative Commons
CC-BY-SA licence, this situation couldn't arise. If you publish your
research under a licence like CC-BY-SA, a company like ancestry.com
would be free to import the data on to their site, but not free to
prevent their competitors, current or future, from copying it from them.
However, the licence is only one part of the problem, and a copyleft
licence introduces problems of its own. We still need to find a place
to put copies initially. A company like ancestry.com will not collate
research from individual websites as it does not have the resources,
and, to the best of my knowledge, most of the existing genealogical
websites do not allow copyleft uploads -- rather they require you to
accept their licence conditions. What is needed is an organisation
whose primary objective is to collect researchers' work and make it
easily accessible. In a word, what we need is a *commons*, a collected
body of research that is accessible to all, hopefully in perpetuity.
The idea of a vast, online commons has already been used extensively by
the Wikimedia project, in the form of the Wikimedia Commons, a
collection of over 11 million media files (mostly images), free for use
by anyone, providing they acknowledge the original creator. What we
need is a genealogical version of that: a well-indexed, freely-available
repository of research. And in the short term, that needs an
organisation to manage it and website to access it.
It is worth clarifying, lest there is any doubt on this point, that this
genealogical commons would not be managed remotely like Wikipedia. The
original, unmodified version of your research would always be there, and
other people would not be modifying it. Others may cite your research,
may quote it, may produce research derived from it, or even produce a
new version "correcting" what an errors that the later researcher feels
you may have made, but your original research is always there, separate
from any derivative work.
FINANCING IT
The Wikimedia Commons exists thanks to donations from users and from
companies who wish to support it. Another model is to make people pay
to access the research in the commons. (This is not necessarily
incompatible with a copyleft licence, depending on which one is used.)
But I don't see that either of these models will work for genealogical
research because a lot of research will probably get accessed very
infrequently. However, I think it is quite feasible for the
organisation running a genealogical commons to provide indefinite
storage of research for a single, fairly modest up-front fee. Some
rough calculation suggest that £20 per gigabyte (€25, US$30) could be
enough, which seems good value to me. This assumes two things: first,
that there will always be more researchers coming along wanting to store
their research, and second that storage continues to get cheaper.
Specifically, we are assuming that the cost of storing data forever is
finite because the unit cost of storage is falling exponentially, and
that cost of making this ever-increasing body of research available to
the public (e.g. the electricity and bandwidth costs, administrative
overheads, and so on) can always be met by the up front fees from the
current generation of researchers.
"Anyone who believes exponential growth can go on forever in a finite
world is either a madman or an economist", or so the saying goes. On
the face of it, this is what we are doing. However, we do not require
it to go on for ever: merely for long enough that the commons is well
enough known to have been incorporated into other companies' databases.
If it survives after then, so much the better; if not, it's still
served it's purpose. The key is that it does survive that long, and
that its content is tempting enough that it does get combined into other
databases. In this case "tempting enough" means large enough -- a large
commercial site like ancestry.com are not going to go to the hassle of
incorporating data from a single person's research, but if they can
include many in a single go, they will.
--
Richard Smith
Estate Planning" has started me thinking about the problem of preserving
the results of our genealogical research so that it will still be
accessible to future generations, and I'm coming to the view that there
is specific hole in the genealogy marketplace for a service that does
this. What follows is quite a long summary of the sort of service I
could envisage someone providing quite easily, but so far as I know,
no-one is currently doing. I'd be interested if anyone knows of any
project along these lines, and whether you agree with my analysis of how
best to preserve your research.
(I did initially post this to soc.genealogy.methods, but it seems either
the moderator(s) is/are away, or there's a problem with the moderation
software and/or my news feed. In any case, I think this is the more
appropriate group.)
DIGITAL DATA
The most reliable way of ensuring that something still exists several
centuries in the future is to make sure that as many copies as possible
exist in different places. Losses can happen in even the most secure of
archives, but with many copies, if some get destroyed, others still
stand a chance of surviving. Publishing research in book form is one
way of achieving this, but this can be costly, take a lot of time, and
many people are reluctant to publish things they consider still to be
works in progress. What is needed is an easier way to publish research:
a way so easy that every little breakthrough can be published.
Almost all our research can be converted into digital format:
speculation, inferences and deductions can be written down; texts can be
transcribed; photos and certificates can be scanned; memorials, statues,
paintings and medals can be digitally photographed; audio and video are
often already digitised these days and can be made so if they're not.
Digital information can easily be shared over the Internet, and it does
not degrade as it is copied and recopied. (There are problems when
formats become obsolete, and serious though they are, I don't want to
dwell on them in this post.) Putting digitised research online seems
easiest way of publishing it.
Online data, however, often has no permanence. If you publish on a
personal website, who will pay the hosting costs long after you've died?
If it's a free hosting provider, will it still exist in twenty years'
time? In the mid-90s geocities.com was the most popular free provider,
but sites created there then no longer exist. What about ancestry.com?
That has a better chance of surviving because has an obvious, viable
business plan. But its ultimate aim is to make money, not to preserve
your data. You have no guarantee that they won't delete your data, and
even if they don't they might start charging prohibitively to access it.
Trusting the preservation of your research exclusively to one company
is a bad strategy. (Throughout this post I use ancestry.com as an
example of a commercial genealogy site. My comments about it could be
equally be considered as general comments on an arbitrary commercial
genealogy site.)
A GENEALOGICAL COMMONS
The best way to ensure the long-term preservation of digital data has to
be to make sure it is continually being copied to new places, and not
tied to fortunes of a few present-day companies. When a new genealogy
company sets up, you need them to be legally free to import your
research onto their site. That freedom to copy in perpetuity is
essential. If I download someone's research from ancestry.com, I find
it marked with ancestry.com's copyright statement and I am seemingly not
free to publish it elsewhere. (In fact, my understanding is that
copyright typically remains with the original researcher, despite the
notice seemingly to the contrary, but either way, I do not have
permission to publish research downloaded from ancestry.com.) If the
original data had been clearly and unambiguously made available under a
"share-alike" or "copyleft" licence, such as the Creative Commons
CC-BY-SA licence, this situation couldn't arise. If you publish your
research under a licence like CC-BY-SA, a company like ancestry.com
would be free to import the data on to their site, but not free to
prevent their competitors, current or future, from copying it from them.
However, the licence is only one part of the problem, and a copyleft
licence introduces problems of its own. We still need to find a place
to put copies initially. A company like ancestry.com will not collate
research from individual websites as it does not have the resources,
and, to the best of my knowledge, most of the existing genealogical
websites do not allow copyleft uploads -- rather they require you to
accept their licence conditions. What is needed is an organisation
whose primary objective is to collect researchers' work and make it
easily accessible. In a word, what we need is a *commons*, a collected
body of research that is accessible to all, hopefully in perpetuity.
The idea of a vast, online commons has already been used extensively by
the Wikimedia project, in the form of the Wikimedia Commons, a
collection of over 11 million media files (mostly images), free for use
by anyone, providing they acknowledge the original creator. What we
need is a genealogical version of that: a well-indexed, freely-available
repository of research. And in the short term, that needs an
organisation to manage it and website to access it.
It is worth clarifying, lest there is any doubt on this point, that this
genealogical commons would not be managed remotely like Wikipedia. The
original, unmodified version of your research would always be there, and
other people would not be modifying it. Others may cite your research,
may quote it, may produce research derived from it, or even produce a
new version "correcting" what an errors that the later researcher feels
you may have made, but your original research is always there, separate
from any derivative work.
FINANCING IT
The Wikimedia Commons exists thanks to donations from users and from
companies who wish to support it. Another model is to make people pay
to access the research in the commons. (This is not necessarily
incompatible with a copyleft licence, depending on which one is used.)
But I don't see that either of these models will work for genealogical
research because a lot of research will probably get accessed very
infrequently. However, I think it is quite feasible for the
organisation running a genealogical commons to provide indefinite
storage of research for a single, fairly modest up-front fee. Some
rough calculation suggest that £20 per gigabyte (€25, US$30) could be
enough, which seems good value to me. This assumes two things: first,
that there will always be more researchers coming along wanting to store
their research, and second that storage continues to get cheaper.
Specifically, we are assuming that the cost of storing data forever is
finite because the unit cost of storage is falling exponentially, and
that cost of making this ever-increasing body of research available to
the public (e.g. the electricity and bandwidth costs, administrative
overheads, and so on) can always be met by the up front fees from the
current generation of researchers.
"Anyone who believes exponential growth can go on forever in a finite
world is either a madman or an economist", or so the saying goes. On
the face of it, this is what we are doing. However, we do not require
it to go on for ever: merely for long enough that the commons is well
enough known to have been incorporated into other companies' databases.
If it survives after then, so much the better; if not, it's still
served it's purpose. The key is that it does survive that long, and
that its content is tempting enough that it does get combined into other
databases. In this case "tempting enough" means large enough -- a large
commercial site like ancestry.com are not going to go to the hassle of
incorporating data from a single person's research, but if they can
include many in a single go, they will.
--
Richard Smith
Ian Goddard
Dec 29, 2011, 11:07:37 AM12/29/11
to
Richard Smith wrote:
%><
Much more, likewise, snipped.
> A GENEALOGICAL COMMONS
>
> The best way to ensure the long-term preservation of digital data has to
> be to make sure it is continually being copied to new places, and not
> tied to fortunes of a few present-day companies. When a new genealogy
> company sets up, you need them to be legally free to import your
> research onto their site. That freedom to copy in perpetuity is
> essential.
Agsin, couldn't agree more.
> If I download someone's research from ancestry.com, I find it
> marked with ancestry.com's copyright statement and I am seemingly not
> free to publish it elsewhere. (In fact, my understanding is that
> copyright typically remains with the original researcher, despite the
> notice seemingly to the contrary, but either way, I do not have
> permission to publish research downloaded from ancestry.com.)
Things can get complex here. There are fact, which are, AIUI, not
copyrightable as such, database copyright which I believe has been
somewhat shaken by court decisions and copyright arising out of
individual effort.
Of the latter one aspect is imaging and another is transcription.
Instance of the first: West Yorks Archives have produced micro-fiche of
my local parish records. Some of these are now available as scanned
images on Ancestry. Both involve cost and effort by these
organisations. So the fiches are copyright of the Archives and what I
see on Ancestry is presumably the copyright of both with Ancestry
sub-licensing from the archive.
Instance of the second: Yorks. Arch. Soc. have published transcriptions
of Almondbury PRs. Without digging out a copy I don't know off-hand who
holds the copyright these publications. However there have been
instances where there is some doubt as to the accuracy of a
transcription or where the transcriber has failed to read a name and
where I have then consulted the fiche to provide my own transcription
which might differ from the original; those new transcripts are my own
efforts and, should I wish to do so, I could publish them with my own
copyright.
With regard to Richard's point in parentheses; I'd expect it to depend
on the small print of the individual website. Clearly the site owner
needs some degree of clearance in order to publish contributions. If
the material is the result of the site's own scanning or transcription
work then they own the copyright & can dictate what should be done. If
it's a user submission resulting from the user's own efforts then whilst
one might consider it morally wrong for the site to grab ownership of
the entire rights (which far exceed what's practically necessary) I'd be
surprised if it was legally invalid if the T&Cs of the site made that a
condition of posting.
> If the original data had been clearly and unambiguously made available under a
> "share-alike" or "copyleft" licence, such as the Creative Commons
> CC-BY-SA licence, this situation couldn't arise. If you publish your
> research under a licence like CC-BY-SA, a company like ancestry.com
> would be free to import the data on to their site, but not free to
> prevent their competitors, current or future, from copying it from them.
Needs some care. Some such licenses permit copying and modifying. I
think we need a licence which permits - encourages even - copying but
forbids modifying. If we return to the original point, that copying is
the best chance for long-term survival of information, information only
truly survives if it is copied in its original form. (Ancient texts
have come down to us because they were copied. But where multiple
copies exist they can differ because of copying errors, even conscious
editing or even because they exist only as translations. There can be
doubt as to what the original document actually said.) If a researcher
wishes to add a commentary to provide an additional reading then this
should be done as a new item.
> However, the licence is only one part of the problem, and a copyleft
> licence introduces problems of its own. We still need to find a place to
> put copies initially. ... What is needed is an organisation whose
single point of failure just as, say, geocities, was. There could be
any number of places where copies would be originally posted including
personal sites, one name studies and one-place studies. If copies are
collected and redistributed by individual researchers any number of the
site(s) of original posting could fail without loss. The commons would
be preserved by its users as a whole, not by some central organisation.
However where an organisation is needed is to specify formats to
exchange data. Such an organisation would create the commons but it
wouldn't have & wouldn't need to have the responsibility for
perpetuating it.
%><
>
> FINANCING IT
>
The issues I see here are rather different to those Richard discusses. A
commons which exists by individual users collecting and sharing data
objects doesn't need some central site to be financed.
The problem I do see is that of archives. Our raw material is largely
in local or national archives and their finance is an ongoing problem.
For instance my local paper is reporting that the local council is
considering shutting its museums for three months in the winter to cut
costs.
At present I can buy a copy of, say, a will from an archive. The
payment contributes to the archive's ongoing cost of conserving
material, extracting what it needs, scanning it and supplying it to me.
In order for a genealogical commons to exist I should then be able to
share the copy freely. However the archive will wish to remain
copyright of the scan; free sharing would cut the archive's potential
source of funding.
What's needed is a consideration of how the initial fees continue to s
--
Ian
The Hotmail address is my spam-bin. Real mail address is iang
at austonley org uk
%><
>
> DIGITAL DATA
>
> The most reliable way of ensuring that something still exists several
> centuries in the future is to make sure that as many copies as possible
> exist in different places. Losses can happen in even the most secure of
> archives, but with many copies, if some get destroyed, others still
> stand a chance of surviving.
Couldn't agree more
> DIGITAL DATA
>
> The most reliable way of ensuring that something still exists several
> centuries in the future is to make sure that as many copies as possible
> exist in different places. Losses can happen in even the most secure of
> archives, but with many copies, if some get destroyed, others still
> stand a chance of surviving.
Much more, likewise, snipped.
> A GENEALOGICAL COMMONS
>
> The best way to ensure the long-term preservation of digital data has to
> be to make sure it is continually being copied to new places, and not
> tied to fortunes of a few present-day companies. When a new genealogy
> company sets up, you need them to be legally free to import your
> research onto their site. That freedom to copy in perpetuity is
> essential.
> If I download someone's research from ancestry.com, I find it
> marked with ancestry.com's copyright statement and I am seemingly not
> free to publish it elsewhere. (In fact, my understanding is that
> copyright typically remains with the original researcher, despite the
> notice seemingly to the contrary, but either way, I do not have
> permission to publish research downloaded from ancestry.com.)
copyrightable as such, database copyright which I believe has been
somewhat shaken by court decisions and copyright arising out of
individual effort.
Of the latter one aspect is imaging and another is transcription.
Instance of the first: West Yorks Archives have produced micro-fiche of
my local parish records. Some of these are now available as scanned
images on Ancestry. Both involve cost and effort by these
organisations. So the fiches are copyright of the Archives and what I
see on Ancestry is presumably the copyright of both with Ancestry
sub-licensing from the archive.
Instance of the second: Yorks. Arch. Soc. have published transcriptions
of Almondbury PRs. Without digging out a copy I don't know off-hand who
holds the copyright these publications. However there have been
instances where there is some doubt as to the accuracy of a
transcription or where the transcriber has failed to read a name and
where I have then consulted the fiche to provide my own transcription
which might differ from the original; those new transcripts are my own
efforts and, should I wish to do so, I could publish them with my own
copyright.
With regard to Richard's point in parentheses; I'd expect it to depend
on the small print of the individual website. Clearly the site owner
needs some degree of clearance in order to publish contributions. If
the material is the result of the site's own scanning or transcription
work then they own the copyright & can dictate what should be done. If
it's a user submission resulting from the user's own efforts then whilst
one might consider it morally wrong for the site to grab ownership of
the entire rights (which far exceed what's practically necessary) I'd be
surprised if it was legally invalid if the T&Cs of the site made that a
condition of posting.
> If the original data had been clearly and unambiguously made available under a
> "share-alike" or "copyleft" licence, such as the Creative Commons
> CC-BY-SA licence, this situation couldn't arise. If you publish your
> research under a licence like CC-BY-SA, a company like ancestry.com
> would be free to import the data on to their site, but not free to
> prevent their competitors, current or future, from copying it from them.
think we need a licence which permits - encourages even - copying but
forbids modifying. If we return to the original point, that copying is
the best chance for long-term survival of information, information only
truly survives if it is copied in its original form. (Ancient texts
have come down to us because they were copied. But where multiple
copies exist they can differ because of copying errors, even conscious
editing or even because they exist only as translations. There can be
doubt as to what the original document actually said.) If a researcher
wishes to add a commentary to provide an additional reading then this
should be done as a new item.
> However, the licence is only one part of the problem, and a copyleft
> licence introduces problems of its own. We still need to find a place to
> primary objective is to collect researchers' work and make it easily
> accessible.
This is where we part company to some extent. Such an organisation is a
> accessible.
single point of failure just as, say, geocities, was. There could be
any number of places where copies would be originally posted including
personal sites, one name studies and one-place studies. If copies are
collected and redistributed by individual researchers any number of the
site(s) of original posting could fail without loss. The commons would
be preserved by its users as a whole, not by some central organisation.
However where an organisation is needed is to specify formats to
exchange data. Such an organisation would create the commons but it
wouldn't have & wouldn't need to have the responsibility for
perpetuating it.
%><
>
> FINANCING IT
>
The issues I see here are rather different to those Richard discusses. A
commons which exists by individual users collecting and sharing data
objects doesn't need some central site to be financed.
The problem I do see is that of archives. Our raw material is largely
in local or national archives and their finance is an ongoing problem.
For instance my local paper is reporting that the local council is
considering shutting its museums for three months in the winter to cut
costs.
At present I can buy a copy of, say, a will from an archive. The
payment contributes to the archive's ongoing cost of conserving
material, extracting what it needs, scanning it and supplying it to me.
In order for a genealogical commons to exist I should then be able to
share the copy freely. However the archive will wish to remain
copyright of the scan; free sharing would cut the archive's potential
source of funding.
What's needed is a consideration of how the initial fees continue to s
--
Ian
The Hotmail address is my spam-bin. Real mail address is iang
at austonley org uk
Richard Smith
Dec 29, 2011, 5:21:18 PM12/29/11
to
Ian Goddard wrote:
countries' copyright laws are involved.
> Of the latter one aspect is imaging and another is transcription.
> Instance of the first: West Yorks Archives have produced micro-fiche of
> my local parish records. Some of these are now available as scanned
> images on Ancestry. Both involve cost and effort by these organisations.
> So the fiches are copyright of the Archives and what I see on Ancestry
> is presumably the copyright of both with Ancestry sub-licensing from the
> archive.
I am not a lawyer, but assuming Ancestry have digitised the micro-fiche,
I expect the copyright of the images to exclusively be Ancestry's, but
the usage of them to be governed by whatever licensing agreement was
made between Ancestry and the West Yorks Archive. It was the West Yorks
Archive's copyright on the micro-fiche that allowed them to force
Ancestry to licence them under certain terms. The difference is subtle,
but important in certain cases.
> Instance of the second: Yorks. Arch. Soc. have published transcriptions
> of Almondbury PRs. Without digging out a copy I don't know off-hand who
> holds the copyright these publications. However there have been
> instances where there is some doubt as to the accuracy of a
> transcription or where the transcriber has failed to read a name and
> where I have then consulted the fiche to provide my own transcription
> which might differ from the original; those new transcripts are my own
> efforts and, should I wish to do so, I could publish them with my own
> copyright.
What I'm not quite sure about is whether you might be infringing the W.
Yorks Archive's copyright on the micro-fiches if you transcribe the
whole thing. In this case, it's academic as any archive would of course
give you permission to publish a transcript in such a case; but
supposing you transcribed the whole of a series of images from Ancestry?
Would your transcription infringe Ancestry's copyright? (I have just
posted a question to this effect on uk.legal.moderated.)
> With regard to Richard's point in parentheses; I'd expect it to depend
> on the small print of the individual website. Clearly the site owner
> needs some degree of clearance in order to publish contributions. If the
> material is the result of the site's own scanning or transcription work
> then they own the copyright & can dictate what should be done. If it's a
> user submission resulting from the user's own efforts then whilst one
> might consider it morally wrong for the site to grab ownership of the
> entire rights (which far exceed what's practically necessary) I'd be
> surprised if it was legally invalid if the T&Cs of the site made that a
> condition of posting.
On none of the sites I'm familiar with do you assign copyright to the
site when uploading, say, a GEDCOM file. But you do grant licence to
use the work; in the case of Ancestry "you grant Ancestry a
non-exclusive, transferable, sublicenseable, royalty-free license to
host, store, copy, publish, distribute, provide access to and otherwise
use such material." Of course, you can only do that if you either you
own the copyright, if it is public domain, or if you have it under a
licence that allows you to do this.
In particular, you cannot typically upload a scan of a photocopy made in
an archive (because the copyright of photocopy is held by the archive),
or a GEDCOM file that is licensed under a "copyleft" style licence.
>> If the original data had been clearly and unambiguously made available
>> under a
>> "share-alike" or "copyleft" licence, such as the Creative Commons
>> CC-BY-SA licence, this situation couldn't arise. If you publish your
>> research under a licence like CC-BY-SA, a company like ancestry.com
>> would be free to import the data on to their site, but not free to
>> prevent their competitors, current or future, from copying it from them.
>
> Needs some care. Some such licenses permit copying and modifying. I
> think we need a licence which permits - encourages even - copying but
> forbids modifying.
That was a something I gave quite a bit of thought to. I should say
straight away that I'm not sure CC-BY-SA is the right choice of licence,
merely that it seemed the most appropriate of the standard ones.
CC-BY-ND is closer to what you have in mind. I can see situations when
this would be good. I absolutely don't like the idea of someone taking
my research, rewording it, and republishing it.
However, consider this case. I have a collection of old family photos.
I want to digitise them and make them part of the commons. If I make
them CC-BY-ND, would this allow someone to publish a PDF (or a physical
book) using my photos as illustration? I would hope so, but I'm not
sure. What about combining several images together, perhaps to make a
compound graphic of all eight g.grandparents? What about cropping an
image, changing the colour balance or contrast, or removing some defect
from it? I think most of these constitute creating a derivative work,
and so are prohibited by most licences like CC-BY-ND.
So I think it's important that any media in the commons *does* permit
modification. But perhaps I was concentrating too hard on that case
when I suggested CC-BY-SA.
> If we return to the original point, that copying is
> the best chance for long-term survival of information, information only
> truly survives if it is copied in its original form. (Ancient texts have
> come down to us because they were copied. But where multiple copies
> exist they can differ because of copying errors, even conscious editing
> or even because they exist only as translations. There can be doubt as
> to what the original document actually said.) If a researcher wishes to
> add a commentary to provide an additional reading then this should be
> done as a new item.
I agree with most of your points, but my conclusion is different to yours.
You mention translation. Translation constitutes creating a derivative
work under copyright law. A license that does not permit any
modification will therefore not permit translation. Is this what we
want? My maternal grandmother was from a family of Prussian immigrants,
I could imagine someone might wish to translate my (rather modest)
research on that family into German, and I wouldn't wish to stop it.
My eventual conclusion was that modification should be permitted, but a
mechanism should be in place to establish whether modification has
occurred. Cryptographic signing is perhaps a good technical strategy;
maybe using a licence that requires all contributions (or contributors)
to be recorded is a good legal strategy.
>> However, the licence is only one part of the problem, and a copyleft
>> licence introduces problems of its own. We still need to find a place to
>> put copies initially. ... What is needed is an organisation whose
>> primary objective is to collect researchers' work and make it easily
>> accessible.
>
> This is where we part company to some extent. Such an organisation is a
> single point of failure just as, say, geocities, was. There could be any
> number of places where copies would be originally posted including
> personal sites, one name studies and one-place studies. If copies are
> collected and redistributed by individual researchers any number of the
> site(s) of original posting could fail without loss. The commons would
> be preserved by its users as a whole, not by some central organisation.
There are advantages and disadvantages to both approaches. The
disadvantages of the distributed approach you advocate is that it is
hard to locate things, and that if a company like Ancestry wants to take
a copy of the whole commons (and we have to hope that they will want
to), it is almost impossible to do so.
Reconsidering the matter, I think you are right to say distributed
primary storage is the way forward, but that you have a central index of
the commons, including metadata on the individual resources. The
central index knows where to find entries, what each resource is, who
owns its copyright, how it is licensed, when it was added to the
commons, and a hash of the resource (to locate copies). But it also
knows more genealogically-relevant information. For a photo, it might
know who the photo is of. For a family tree, it might know the relevant
surname(s), place(s) and period.
Some of this information would be mandatory: for example, the licence
conditions and copyright holder. It may desirable to make a lot of the
information mandatory. For example, there's little use in a photo
without any information on who it is of or where it came from, so should
it be possible to add such a thing to a commons?
> However where an organisation is needed is to specify formats to
> exchange data. Such an organisation would create the commons but it
> wouldn't have & wouldn't need to have the responsibility for
> perpetuating it.
I think I do disagree slightly there. Without some central index, the
commons is too nebulous a entity to be perpetuated in bulk, and without
that things will gradually get lost. We cannot rely on repeated
small-scale copying perpetuating it because so much of the commons will
be of interest to a very tiny number of people, and may go years without
being accessed.
Of course, there are distributed indexing strategies, and the index
itself should be licensed in a permissive way so that people can (and
are encouraged to) take copies. It may be best not to have a single
index server that (necessarily) knows about all of the commons. These
strategies are effectively the ones used to locate resources on
peer-to-peer networks, and quite a lot of research has been conducted in
this field.
>> FINANCING IT
> The issues I see here are rather different to those Richard discusses. A
> commons which exists by individual users collecting and sharing data
> objects doesn't need some central site to be financed.
I now agree. A central index (if you have one) is much smaller
undertaking and can almost certainly be run by someone's good will.
> The problem I do see is that of archives. Our raw material is largely in
> local or national archives and their finance is an ongoing problem. For
> instance my local paper is reporting that the local council is
> considering shutting its museums for three months in the winter to cut
> costs.
>
> At present I can buy a copy of, say, a will from an archive. The payment
> contributes to the archive's ongoing cost of conserving material,
> extracting what it needs, scanning it and supplying it to me. In order
> for a genealogical commons to exist I should then be able to share the
> copy freely. However the archive will wish to remain copyright of the
> scan; free sharing would cut the archive's potential source of funding.
This is a good point. However, if I ask an archive for permission to
put the will of one of my ancestors online, they will probably say yes;
at least, that's my experience. But in saying that, they haven't given
me permission to place it under a copyleft licence (or indeed any
licence that permits copying). So the will hasn't entered the commons.
To be honest, it's not things like wills that are my primary concern. A
will isn't all that likely to go missing, and an archive is about the
safest place where something can be deposited. As soon as I know where
to find the will, I can pay a copy from the relevant archive. My aim
isn't to allow people can access a resource for free, but to ensure that
people can access it at all in a century's time.
> What's needed is a consideration of how the initial fees continue to s
I think something got lost there...
Anyway, thank you for your thoughts.
--
Richard Smith
> Richard Smith wrote:
>> If I download someone's research from ancestry.com, I find it
>> marked with ancestry.com's copyright statement and I am seemingly not
>> free to publish it elsewhere. (In fact, my understanding is that
>> copyright typically remains with the original researcher, despite the
>> notice seemingly to the contrary, but either way, I do not have
>> permission to publish research downloaded from ancestry.com.)
>
> Things can get complex here. There are fact, which are, AIUI, not
> copyrightable as such, database copyright which I believe has been
> somewhat shaken by court decisions and copyright arising out of
> individual effort.
I agree that this can be a complex area, and even more so if several
>> If I download someone's research from ancestry.com, I find it
>> marked with ancestry.com's copyright statement and I am seemingly not
>> free to publish it elsewhere. (In fact, my understanding is that
>> copyright typically remains with the original researcher, despite the
>> notice seemingly to the contrary, but either way, I do not have
>> permission to publish research downloaded from ancestry.com.)
>
> Things can get complex here. There are fact, which are, AIUI, not
> copyrightable as such, database copyright which I believe has been
> somewhat shaken by court decisions and copyright arising out of
> individual effort.
countries' copyright laws are involved.
> Of the latter one aspect is imaging and another is transcription.
> Instance of the first: West Yorks Archives have produced micro-fiche of
> my local parish records. Some of these are now available as scanned
> images on Ancestry. Both involve cost and effort by these organisations.
> So the fiches are copyright of the Archives and what I see on Ancestry
> is presumably the copyright of both with Ancestry sub-licensing from the
> archive.
I expect the copyright of the images to exclusively be Ancestry's, but
the usage of them to be governed by whatever licensing agreement was
made between Ancestry and the West Yorks Archive. It was the West Yorks
Archive's copyright on the micro-fiche that allowed them to force
Ancestry to licence them under certain terms. The difference is subtle,
but important in certain cases.
> Instance of the second: Yorks. Arch. Soc. have published transcriptions
> of Almondbury PRs. Without digging out a copy I don't know off-hand who
> holds the copyright these publications. However there have been
> instances where there is some doubt as to the accuracy of a
> transcription or where the transcriber has failed to read a name and
> where I have then consulted the fiche to provide my own transcription
> which might differ from the original; those new transcripts are my own
> efforts and, should I wish to do so, I could publish them with my own
> copyright.
Yorks Archive's copyright on the micro-fiches if you transcribe the
whole thing. In this case, it's academic as any archive would of course
give you permission to publish a transcript in such a case; but
supposing you transcribed the whole of a series of images from Ancestry?
Would your transcription infringe Ancestry's copyright? (I have just
posted a question to this effect on uk.legal.moderated.)
> With regard to Richard's point in parentheses; I'd expect it to depend
> on the small print of the individual website. Clearly the site owner
> needs some degree of clearance in order to publish contributions. If the
> material is the result of the site's own scanning or transcription work
> then they own the copyright & can dictate what should be done. If it's a
> user submission resulting from the user's own efforts then whilst one
> might consider it morally wrong for the site to grab ownership of the
> entire rights (which far exceed what's practically necessary) I'd be
> surprised if it was legally invalid if the T&Cs of the site made that a
> condition of posting.
site when uploading, say, a GEDCOM file. But you do grant licence to
use the work; in the case of Ancestry "you grant Ancestry a
non-exclusive, transferable, sublicenseable, royalty-free license to
host, store, copy, publish, distribute, provide access to and otherwise
use such material." Of course, you can only do that if you either you
own the copyright, if it is public domain, or if you have it under a
licence that allows you to do this.
In particular, you cannot typically upload a scan of a photocopy made in
an archive (because the copyright of photocopy is held by the archive),
or a GEDCOM file that is licensed under a "copyleft" style licence.
>> If the original data had been clearly and unambiguously made available
>> under a
>> "share-alike" or "copyleft" licence, such as the Creative Commons
>> CC-BY-SA licence, this situation couldn't arise. If you publish your
>> research under a licence like CC-BY-SA, a company like ancestry.com
>> would be free to import the data on to their site, but not free to
>> prevent their competitors, current or future, from copying it from them.
>
> Needs some care. Some such licenses permit copying and modifying. I
> think we need a licence which permits - encourages even - copying but
> forbids modifying.
straight away that I'm not sure CC-BY-SA is the right choice of licence,
merely that it seemed the most appropriate of the standard ones.
CC-BY-ND is closer to what you have in mind. I can see situations when
this would be good. I absolutely don't like the idea of someone taking
my research, rewording it, and republishing it.
However, consider this case. I have a collection of old family photos.
I want to digitise them and make them part of the commons. If I make
them CC-BY-ND, would this allow someone to publish a PDF (or a physical
book) using my photos as illustration? I would hope so, but I'm not
sure. What about combining several images together, perhaps to make a
compound graphic of all eight g.grandparents? What about cropping an
image, changing the colour balance or contrast, or removing some defect
from it? I think most of these constitute creating a derivative work,
and so are prohibited by most licences like CC-BY-ND.
So I think it's important that any media in the commons *does* permit
modification. But perhaps I was concentrating too hard on that case
when I suggested CC-BY-SA.
> If we return to the original point, that copying is
> the best chance for long-term survival of information, information only
> truly survives if it is copied in its original form. (Ancient texts have
> come down to us because they were copied. But where multiple copies
> exist they can differ because of copying errors, even conscious editing
> or even because they exist only as translations. There can be doubt as
> to what the original document actually said.) If a researcher wishes to
> add a commentary to provide an additional reading then this should be
> done as a new item.
You mention translation. Translation constitutes creating a derivative
work under copyright law. A license that does not permit any
modification will therefore not permit translation. Is this what we
want? My maternal grandmother was from a family of Prussian immigrants,
I could imagine someone might wish to translate my (rather modest)
research on that family into German, and I wouldn't wish to stop it.
My eventual conclusion was that modification should be permitted, but a
mechanism should be in place to establish whether modification has
occurred. Cryptographic signing is perhaps a good technical strategy;
maybe using a licence that requires all contributions (or contributors)
to be recorded is a good legal strategy.
>> However, the licence is only one part of the problem, and a copyleft
>> licence introduces problems of its own. We still need to find a place to
>> put copies initially. ... What is needed is an organisation whose
>> primary objective is to collect researchers' work and make it easily
>> accessible.
>
> This is where we part company to some extent. Such an organisation is a
> single point of failure just as, say, geocities, was. There could be any
> number of places where copies would be originally posted including
> personal sites, one name studies and one-place studies. If copies are
> collected and redistributed by individual researchers any number of the
> site(s) of original posting could fail without loss. The commons would
> be preserved by its users as a whole, not by some central organisation.
disadvantages of the distributed approach you advocate is that it is
hard to locate things, and that if a company like Ancestry wants to take
a copy of the whole commons (and we have to hope that they will want
to), it is almost impossible to do so.
Reconsidering the matter, I think you are right to say distributed
primary storage is the way forward, but that you have a central index of
the commons, including metadata on the individual resources. The
central index knows where to find entries, what each resource is, who
owns its copyright, how it is licensed, when it was added to the
commons, and a hash of the resource (to locate copies). But it also
knows more genealogically-relevant information. For a photo, it might
know who the photo is of. For a family tree, it might know the relevant
surname(s), place(s) and period.
Some of this information would be mandatory: for example, the licence
conditions and copyright holder. It may desirable to make a lot of the
information mandatory. For example, there's little use in a photo
without any information on who it is of or where it came from, so should
it be possible to add such a thing to a commons?
> However where an organisation is needed is to specify formats to
> exchange data. Such an organisation would create the commons but it
> wouldn't have & wouldn't need to have the responsibility for
> perpetuating it.
commons is too nebulous a entity to be perpetuated in bulk, and without
that things will gradually get lost. We cannot rely on repeated
small-scale copying perpetuating it because so much of the commons will
be of interest to a very tiny number of people, and may go years without
being accessed.
Of course, there are distributed indexing strategies, and the index
itself should be licensed in a permissive way so that people can (and
are encouraged to) take copies. It may be best not to have a single
index server that (necessarily) knows about all of the commons. These
strategies are effectively the ones used to locate resources on
peer-to-peer networks, and quite a lot of research has been conducted in
this field.
>> FINANCING IT
> The issues I see here are rather different to those Richard discusses. A
> commons which exists by individual users collecting and sharing data
> objects doesn't need some central site to be financed.
undertaking and can almost certainly be run by someone's good will.
> The problem I do see is that of archives. Our raw material is largely in
> local or national archives and their finance is an ongoing problem. For
> instance my local paper is reporting that the local council is
> considering shutting its museums for three months in the winter to cut
> costs.
>
> At present I can buy a copy of, say, a will from an archive. The payment
> contributes to the archive's ongoing cost of conserving material,
> extracting what it needs, scanning it and supplying it to me. In order
> for a genealogical commons to exist I should then be able to share the
> copy freely. However the archive will wish to remain copyright of the
> scan; free sharing would cut the archive's potential source of funding.
put the will of one of my ancestors online, they will probably say yes;
at least, that's my experience. But in saying that, they haven't given
me permission to place it under a copyleft licence (or indeed any
licence that permits copying). So the will hasn't entered the commons.
To be honest, it's not things like wills that are my primary concern. A
will isn't all that likely to go missing, and an archive is about the
safest place where something can be deposited. As soon as I know where
to find the will, I can pay a copy from the relevant archive. My aim
isn't to allow people can access a resource for free, but to ensure that
people can access it at all in a century's time.
> What's needed is a consideration of how the initial fees continue to s
Anyway, thank you for your thoughts.
--
Richard Smith
Ian Goddard
Dec 29, 2011, 6:34:17 PM12/29/11
to
Richard Smith wrote:
> Ian Goddard wrote:
%><
>> Needs some care. Some such licenses permit copying and modifying. I
>> think we need a licence which permits - encourages even - copying but
>> forbids modifying.
>
> That was a something I gave quite a bit of thought to. I should say
> straight away that I'm not sure CC-BY-SA is the right choice of licence,
> merely that it seemed the most appropriate of the standard ones.
> CC-BY-ND is closer to what you have in mind. I can see situations when
> this would be good. I absolutely don't like the idea of someone taking
> my research, rewording it, and republishing it.
>
> However, consider this case. I have a collection of old family photos. I
> want to digitise them and make them part of the commons. If I make them
> CC-BY-ND, would this allow someone to publish a PDF (or a physical book)
> using my photos as illustration? I would hope so, but I'm not sure. What
> about combining several images together, perhaps to make a compound
> graphic of all eight g.grandparents? What about cropping an image,
> changing the colour balance or contrast, or removing some defect from
> it? I think most of these constitute creating a derivative work, and so
> are prohibited by most licences like CC-BY-ND.
>
> So I think it's important that any media in the commons *does* permit
> modification. But perhaps I was concentrating too hard on that case when
> I suggested CC-BY-SA.
I think there are two different issues. I'm thinking primarily of a
core *digital* commons. Such a system would be a distributed database
of multiple copies of unique objects, unique in that any copy would have
the same content and the same unique identifier. Maybe the better term
to apply here would be replication. An object should be replicated
unchanged. If anyone wanted to extract a picture from its replicated
wrapper and use it cropped in another context that wouldn't be a
problem. Neither would creating a *new* object with the cropped picture
under a new ID, preferably with a link back to the original.
>> If we return to the original point, that copying is
>> the best chance for long-term survival of information, information only
>> truly survives if it is copied in its original form. (Ancient texts have
>> come down to us because they were copied. But where multiple copies
>> exist they can differ because of copying errors, even conscious editing
>> or even because they exist only as translations. There can be doubt as
>> to what the original document actually said.) If a researcher wishes to
>> add a commentary to provide an additional reading then this should be
>> done as a new item.
>
> I agree with most of your points, but my conclusion is different to yours.
>
> You mention translation. Translation constitutes creating a derivative
> work under copyright law. A license that does not permit any
> modification will therefore not permit translation. Is this what we
> want? My maternal grandmother was from a family of Prussian immigrants,
> I could imagine someone might wish to translate my (rather modest)
> research on that family into German, and I wouldn't wish to stop it.
I think you've missed my point. If the *only* copy has come via
translation we don't know exactly what the original said. An example
might be an ancient Greek mathematical text surviving only in Arabic
translation.
In the case you cited the German translation would be fine but it would
be a *new* object which, like the cropped picture example, would have
its own ID, point back to the original, and be replicated independently.
%><
> maybe using a licence that requires all contributions (or contributors)
> to be recorded is a good legal strategy.
An object without a provenance is pretty useless. The commons would
have to include a provenance in the form of some chain of IDs leading
back the original source.
%><
> Reconsidering the matter, I think you are right to say distributed
> primary storage is the way forward, but that you have a central index of
> the commons, including metadata on the individual resources. The central
> index knows where to find entries, what each resource is, who owns its
> copyright, how it is licensed, when it was added to the commons, and a
> hash of the resource (to locate copies). But it also knows more
> genealogically-relevant information. For a photo, it might know who the
> photo is of. For a family tree, it might know the relevant surname(s),
> place(s) and period.
>
> Some of this information would be mandatory: for example, the licence
> conditions and copyright holder. It may desirable to make a lot of the
> information mandatory. For example, there's little use in a photo
> without any information on who it is of or where it came from, so should
> it be possible to add such a thing to a commons?
As regards licence, this is surely part & parcel of your original
proposal. Source information is covered by providing a provenance. I'm
not sure about a photo without information; one reason for publishing
might be to try to gather information from someone who knows!
>> However where an organisation is needed is to specify formats to
>> exchange data. Such an organisation would create the commons but it
>> wouldn't have & wouldn't need to have the responsibility for
>> perpetuating it.
>
> I think I do disagree slightly there. Without some central index, the
> commons is too nebulous a entity to be perpetuated in bulk, and without
> that things will gradually get lost. We cannot rely on repeated
> small-scale copying perpetuating it because so much of the commons will
> be of interest to a very tiny number of people, and may go years without
> being accessed.
>
> Of course, there are distributed indexing strategies, and the index
> itself should be licensed in a permissive way so that people can (and
> are encouraged to) take copies. It may be best not to have a single
> index server that (necessarily) knows about all of the commons. These
> strategies are effectively the ones used to locate resources on
> peer-to-peer networks, and quite a lot of research has been conducted in
> this field.
A distributed index was one thing I had in mind.
One point to consider is that if the commons consists of a series of
objects with unique IDs and objects can also point to related objects
then a good deal of some subset might be navigated simply by following
such pointers Some of the most useful objects might be essentially
pairs of pointers to objects a researcher believes to be connected.
Many servers offering objects from the commons might simply list the
IDs of the objects they offer with other sites providing a meta-index of
such servers. More general exploration might depend on the extent to
which material was exposed to general search engines. You might
originally find great-uncle Fred through Google & then follow a
crumb-trail of link numbers to other material. However, bearing in mind
the spelling variations of our ancestors a more practical indexing
approach might be value-added sites offering smarter Soundex or whatever
searches. A useful commons would undoubtedly grow its own ecosystem as
current terminology would have it <original palaeoecological self
shudders at such abuse of vocabulary>.
>>> FINANCING IT
%><
> To be honest, it's not things like wills that are my primary concern. A
> will isn't all that likely to go missing, and an archive is about the
> safest place where something can be deposited. As soon as I know where
> to find the will, I can pay a copy from the relevant archive. My aim
> isn't to allow people can access a resource for free, but to ensure that
> people can access it at all in a century's time.
I think a degree of pessimism is needed about even archival storage.
Not only do disasters, natural (e.g. Cologne) or man-made (e.g. the Four
Courts fire in Dublin), destroy huge amounts of material which existed
in only the single copy; they can also have funding problems. My local
paper is reporting today that the council is considering closing its
museums for 3 months in winter as a cost-cutting exercise. And some
years ago when on holiday I wandered into a bookshop & found it crammed
with bound volumes of Nature that the county library had decided to
clear from its shelves. You can't trust nobody!
>> What's needed is a consideration of how the initial fees continue to s
>
> I think something got lost there...
>
Yup. I had a bit of a bother with posting & hadn't noticed that when I
reposted it wasn't a complete copy.
What I had in mind is that although on the one hand an archive will make
some of its income from sales of images it may also be incurring
on-going costs in generally making some of its information freely
available on the net, e.g. summaries in A2A. Launching data into a
commons scheme could reduce those ongoing costs. Setting the fees for a
copy which would then be put into the commons were sufficient to offset
the possible loss of repeat sales (how many wills are copied more than
once?) then the overall balance could work in the archive's favour.
OK. Lets hope this one goes be
--
Ian
> Ian Goddard wrote:
%><
>> Needs some care. Some such licenses permit copying and modifying. I
>> think we need a licence which permits - encourages even - copying but
>> forbids modifying.
>
> That was a something I gave quite a bit of thought to. I should say
> straight away that I'm not sure CC-BY-SA is the right choice of licence,
> merely that it seemed the most appropriate of the standard ones.
> CC-BY-ND is closer to what you have in mind. I can see situations when
> this would be good. I absolutely don't like the idea of someone taking
> my research, rewording it, and republishing it.
>
> However, consider this case. I have a collection of old family photos. I
> want to digitise them and make them part of the commons. If I make them
> CC-BY-ND, would this allow someone to publish a PDF (or a physical book)
> using my photos as illustration? I would hope so, but I'm not sure. What
> about combining several images together, perhaps to make a compound
> graphic of all eight g.grandparents? What about cropping an image,
> changing the colour balance or contrast, or removing some defect from
> it? I think most of these constitute creating a derivative work, and so
> are prohibited by most licences like CC-BY-ND.
>
> So I think it's important that any media in the commons *does* permit
> modification. But perhaps I was concentrating too hard on that case when
> I suggested CC-BY-SA.
core *digital* commons. Such a system would be a distributed database
of multiple copies of unique objects, unique in that any copy would have
the same content and the same unique identifier. Maybe the better term
to apply here would be replication. An object should be replicated
unchanged. If anyone wanted to extract a picture from its replicated
wrapper and use it cropped in another context that wouldn't be a
problem. Neither would creating a *new* object with the cropped picture
under a new ID, preferably with a link back to the original.
>> If we return to the original point, that copying is
>> the best chance for long-term survival of information, information only
>> truly survives if it is copied in its original form. (Ancient texts have
>> come down to us because they were copied. But where multiple copies
>> exist they can differ because of copying errors, even conscious editing
>> or even because they exist only as translations. There can be doubt as
>> to what the original document actually said.) If a researcher wishes to
>> add a commentary to provide an additional reading then this should be
>> done as a new item.
>
> I agree with most of your points, but my conclusion is different to yours.
>
> You mention translation. Translation constitutes creating a derivative
> work under copyright law. A license that does not permit any
> modification will therefore not permit translation. Is this what we
> want? My maternal grandmother was from a family of Prussian immigrants,
> I could imagine someone might wish to translate my (rather modest)
> research on that family into German, and I wouldn't wish to stop it.
translation we don't know exactly what the original said. An example
might be an ancient Greek mathematical text surviving only in Arabic
translation.
In the case you cited the German translation would be fine but it would
be a *new* object which, like the cropped picture example, would have
its own ID, point back to the original, and be replicated independently.
%><
> maybe using a licence that requires all contributions (or contributors)
> to be recorded is a good legal strategy.
have to include a provenance in the form of some chain of IDs leading
back the original source.
%><
>
> There are advantages and disadvantages to both approaches. The
> disadvantages of the distributed approach you advocate is that it is
> hard to locate things, and that if a company like Ancestry wants to take
> a copy of the whole commons (and we have to hope that they will want
> to), it is almost impossible to do so.
I think it very likely that this would happen.
> There are advantages and disadvantages to both approaches. The
> disadvantages of the distributed approach you advocate is that it is
> hard to locate things, and that if a company like Ancestry wants to take
> a copy of the whole commons (and we have to hope that they will want
> to), it is almost impossible to do so.
> Reconsidering the matter, I think you are right to say distributed
> primary storage is the way forward, but that you have a central index of
> the commons, including metadata on the individual resources. The central
> index knows where to find entries, what each resource is, who owns its
> copyright, how it is licensed, when it was added to the commons, and a
> hash of the resource (to locate copies). But it also knows more
> genealogically-relevant information. For a photo, it might know who the
> photo is of. For a family tree, it might know the relevant surname(s),
> place(s) and period.
>
> Some of this information would be mandatory: for example, the licence
> conditions and copyright holder. It may desirable to make a lot of the
> information mandatory. For example, there's little use in a photo
> without any information on who it is of or where it came from, so should
> it be possible to add such a thing to a commons?
proposal. Source information is covered by providing a provenance. I'm
not sure about a photo without information; one reason for publishing
might be to try to gather information from someone who knows!
>> However where an organisation is needed is to specify formats to
>> exchange data. Such an organisation would create the commons but it
>> wouldn't have & wouldn't need to have the responsibility for
>> perpetuating it.
>
> I think I do disagree slightly there. Without some central index, the
> commons is too nebulous a entity to be perpetuated in bulk, and without
> that things will gradually get lost. We cannot rely on repeated
> small-scale copying perpetuating it because so much of the commons will
> be of interest to a very tiny number of people, and may go years without
> being accessed.
>
> Of course, there are distributed indexing strategies, and the index
> itself should be licensed in a permissive way so that people can (and
> are encouraged to) take copies. It may be best not to have a single
> index server that (necessarily) knows about all of the commons. These
> strategies are effectively the ones used to locate resources on
> peer-to-peer networks, and quite a lot of research has been conducted in
> this field.
One point to consider is that if the commons consists of a series of
objects with unique IDs and objects can also point to related objects
then a good deal of some subset might be navigated simply by following
such pointers Some of the most useful objects might be essentially
pairs of pointers to objects a researcher believes to be connected.
Many servers offering objects from the commons might simply list the
IDs of the objects they offer with other sites providing a meta-index of
such servers. More general exploration might depend on the extent to
which material was exposed to general search engines. You might
originally find great-uncle Fred through Google & then follow a
crumb-trail of link numbers to other material. However, bearing in mind
the spelling variations of our ancestors a more practical indexing
approach might be value-added sites offering smarter Soundex or whatever
searches. A useful commons would undoubtedly grow its own ecosystem as
current terminology would have it <original palaeoecological self
shudders at such abuse of vocabulary>.
>>> FINANCING IT
%><
> To be honest, it's not things like wills that are my primary concern. A
> will isn't all that likely to go missing, and an archive is about the
> safest place where something can be deposited. As soon as I know where
> to find the will, I can pay a copy from the relevant archive. My aim
> isn't to allow people can access a resource for free, but to ensure that
> people can access it at all in a century's time.
Not only do disasters, natural (e.g. Cologne) or man-made (e.g. the Four
Courts fire in Dublin), destroy huge amounts of material which existed
in only the single copy; they can also have funding problems. My local
paper is reporting today that the council is considering closing its
museums for 3 months in winter as a cost-cutting exercise. And some
years ago when on holiday I wandered into a bookshop & found it crammed
with bound volumes of Nature that the county library had decided to
clear from its shelves. You can't trust nobody!
>> What's needed is a consideration of how the initial fees continue to s
>
> I think something got lost there...
>
reposted it wasn't a complete copy.
What I had in mind is that although on the one hand an archive will make
some of its income from sales of images it may also be incurring
on-going costs in generally making some of its information freely
available on the net, e.g. summaries in A2A. Launching data into a
commons scheme could reduce those ongoing costs. Setting the fees for a
copy which would then be put into the commons were sufficient to offset
the possible loss of repeat sales (how many wills are copied more than
once?) then the overall balance could work in the archive's favour.
OK. Lets hope this one goes be
--
Ian
Ian Goddard
Dec 29, 2011, 6:49:40 PM12/29/11
to
As an additional point to this discussion a wider commons would be
better still. Local history can provide valuable material for the
family historian & vice versa. A corpus of family history material of
milling families might provide material for an economic historian
researching milling and such research would be useful to me in
understanding my milling ancestors. And so it goes on...
better still. Local history can provide valuable material for the
family historian & vice versa. A corpus of family history material of
milling families might provide material for an economic historian
researching milling and such research would be useful to me in
understanding my milling ancestors. And so it goes on...
Kerry Raymond
Dec 29, 2011, 7:01:33 PM12/29/11
to
> The best way to ensure the long-term preservation of digital data has to
> be to make sure it is continually being copied to new places, and not tied
> to fortunes of a few present-day companies.
Agreed.
> be to make sure it is continually being copied to new places, and not tied
> to fortunes of a few present-day companies.
> The idea of a vast, online commons has already been used extensively by
> the Wikimedia project, in the form of the Wikimedia Commons
Wikimedia run many projects, Wikipedia and Commons are just two of them. See
here for the complete list:
http://en.wikipedia.org/wiki/Wikimedia_Foundation#Projects.2C_initiatives_and_chapters
We could consider whether existing projects such as Commons and WikiSource
could be used for these purposes, or whether we would need a more
purpose-specific project.
I think there are two types of digital data that we are interested in as
genealogists, our personal family history (GEDs, photos, etc) and family
history sources (church records, contents of archives, etc). I think family
history sources would fit within the current purpose of Commons and
WikiSource (you could start storing them today). The storage of personal
family histories however is not such a good fit to current Wikimedia
projects, and would suggest the need for a separate Wikimedia project (which
would have the advantage of allowing for a specialised tools that understand
GED and searching for people etc).
If Wikimedia was willing, this strategy has the advantages of building on an
existing successful organisation with existing web sites and tools which is
committed to open access and with an apparently successful financing method
(annual donation cycle). It is (mostly) based on a CC-BY-SA licence which is
compatible with what you are talking about. I can't see much benefit in
re-inventing all of this, *if* Wikimedia were willing to create a new
project.
On the subject of licensing (looking at some of the other replies), I am not
sure it is worth getting too worried about it. You will be dead and buried
so who's going to pursue anyone who misuses your data (particularly in the
absence of any effective way to enforce licensing until there is economic
value sufficient to justify the legal costs involved). Just pick a license
that allows people who intend to Do The Right Thing to be allowed to do
that; don't worry what the bad guys will do because you can't stop them in
practice.
Kerry
Richard Smith
Dec 29, 2011, 8:05:52 PM12/29/11
to
On 30/12/11 00:01, Kerry Raymond wrote:
> Could we consider Wikimedia as the place to build a genealogical
> commons? Wikimedia run many projects, Wikipedia and Commons are just two
> of them. See here for the complete list:
>
> http://en.wikipedia.org/wiki/Wikimedia_Foundation#Projects.2C_initiatives_and_chapters
>
> We could consider whether existing projects such as Commons and
> WikiSource could be used for these purposes, or whether we would need a
> more purpose-specific project.
>
> I think there are two types of digital data that we are interested in as
> genealogists, our personal family history (GEDs, photos, etc) and family
> history sources (church records, contents of archives, etc). I think
> family history sources would fit within the current purpose of Commons
> and WikiSource (you could start storing them today).
They certainly fit within the scope of the Wikimedia Commons and
> Could we consider Wikimedia as the place to build a genealogical
> commons? Wikimedia run many projects, Wikipedia and Commons are just two
> of them. See here for the complete list:
>
> http://en.wikipedia.org/wiki/Wikimedia_Foundation#Projects.2C_initiatives_and_chapters
>
> We could consider whether existing projects such as Commons and
> WikiSource could be used for these purposes, or whether we would need a
> more purpose-specific project.
>
> I think there are two types of digital data that we are interested in as
> genealogists, our personal family history (GEDs, photos, etc) and family
> history sources (church records, contents of archives, etc). I think
> family history sources would fit within the current purpose of Commons
> and WikiSource (you could start storing them today).
Wikisource, but is it the best place for them? You can't upload
spreadsheets or similar to these projects, nor can you upload structured
data. An old baptism register can be correctly represented by simple
transcription:
1673
Thomas Hutchinson sonn of Tho. and Dorathy Hutchinson buried Aprill 1.
Walter Mercer buried Aprill 7
John Goward son of Tho. and Ann Goward buried May 5
...
But a more recent register (and some older ones) has columns for things
like date, name, surname, mother and father. And you want to preserve
that in a *standard* machine readable way. First, Wikisource has no
easy means for standardising the format, and second it doesn't support
most formats you might want to use to convey the information.
Further, by using such a broad project such as Wikisource or the
Wikimedia Commons you are at the mercy of project administrators who
mostly have no idea on genealogy, and anyone can edit the resources.
I'm afraid I have little enthusiasm for using one of the established
Wikimedia projects.
> The storage of
> personal family histories however is not such a good fit to current
> Wikimedia projects, and would suggest the need for a separate Wikimedia
> project (which would have the advantage of allowing for a specialised
> tools that understand GED and searching for people etc).
support this because the storage is so large and the likely readership
so small.
> If Wikimedia was willing, this strategy has the advantages of building
> on an existing successful organisation with existing web sites and tools
> which is committed to open access and with an apparently successful
> financing method (annual donation cycle). It is (mostly) based on a
> CC-BY-SA licence which is compatible with what you are talking about. I
> can't see much benefit in re-inventing all of this, *if* Wikimedia were
> willing to create a new project.
to be entrusted to the stewardship of a single organisation, the
Wikimedia Foundation isn't a bad choice. But is it a better choice than
the sort of distributed architecture that Ian Goddard is talking about?
> On the subject of licensing (looking at some of the other replies), I am
> not sure it is worth getting too worried about it. You will be dead and
> buried
research to *now*, before I die. After I die is too late; I can't
upload it then, and who's to know whether my next of kin will do, or
will even know how to find it all?
> Just pick a license that allows people who intend to Do The
> Right Thing to be allowed to do that; don't worry what the bad guys will
> do because you can't stop them in practice.
a large part, than action of some sort is entirely possible. But
mostly, yes, I agree.
--
Richard Smith
singhals
Dec 29, 2011, 8:27:09 PM12/29/11
to gen...@rootsweb.com
Ian Goddard wrote:
> As an additional point to this discussion a wider commons would be
> better still. Local history can provide valuable material for the
> family historian& vice versa. A corpus of family history material of
> As an additional point to this discussion a wider commons would be
> better still. Local history can provide valuable material for the
> milling families might provide material for an economic historian
> researching milling and such research would be useful to me in
> understanding my milling ancestors. And so it goes on...
>
Ian, Richard,
> researching milling and such research would be useful to me in
> understanding my milling ancestors. And so it goes on...
>
Lift yourselves out of the weeds for a minute and explain to
me (us?) how a Genealogical Commons would significantly
differ from (a) the LDS patron-submitted data (b)
OneWorldTree (c) WorldConnect or even (d) all the internet?
However valuable it would be to be certain your research and
mine is preserved for eternity (all of us here being of
course the very model of perfect researchers who make
perfect sourcing and provenences and conclusions), I'm not
sure Cousin Iff's data is equally deserving. So, What's the
mechanism for ensuring that the Commons won't turn into
another pit of bad research, retained forever?
Down in the knee-high weeds, updates within a 12-month
calendar period ought not be allowed, to prevent 8 different
people having 8 different 2010 versions of the same
pedigree/data.
Just sayin' ...
Cheryl
Richard Smith
Dec 29, 2011, 8:59:13 PM12/29/11
to
Cheryl Singhals wrote:
> Lift yourselves out of the weeds for a minute and explain to me (us?)
> how a Genealogical Commons would significantly differ from (a) the LDS
> patron-submitted data (b) OneWorldTree (c) WorldConnect or even (d) all
> the internet?
It differs from the first three because it is resource-oriented, not
> Lift yourselves out of the weeds for a minute and explain to me (us?)
> how a Genealogical Commons would significantly differ from (a) the LDS
> patron-submitted data (b) OneWorldTree (c) WorldConnect or even (d) all
> the internet?
tree-oriented, and because you are permitted to copy the whole commons
which you are not permitted to do in many (all?) of these instances.
It differs from the Internet as a whole because it is customised
specifically to genealogy. Though if something like Ian's distributed
architecture is adopted, then really it is just the Internet, but with a
custom index engine to accessing it. But really, if it's like anything
that's currently around, then it's the Wikimedia Commons. We cannot use
that project, however, because modern genealogical research is outwith
its scope.
> However valuable it would be to be certain your research and mine is
> preserved for eternity (all of us here being of course the very model of
> perfect researchers who make perfect sourcing and provenences and
> conclusions), I'm not sure Cousin Iff's data is equally deserving. So,
> What's the mechanism for ensuring that the Commons won't turn into
> another pit of bad research, retained forever?
not a big communal tree like One World Tree or similar because it is not
a tree at all. It's a about resources, not trees. The commons provides
guarantees on the provenance of a resource, at least back to the point
when it entered the commons. It's clear who produced a given resource,
and it's your job as a researcher to arrive at an opinion on the
reliability of the author of the resource.
Or maybe someone will build a service on top of the commons that will
somehow rate different researchers, perhaps a bit like the seller
feedback mechanism on Ebay. But that's beyond the scope of the commons.
> Down in the knee-high weeds, updates within a 12-month calendar period
> ought not be allowed, to prevent 8 different people having 8 different
> 2010 versions of the same pedigree/data.
much as you can produce a second edition of a book to supersede the
first edition. But then both editions exist, and both can still be
read. Once you've published something into the commons, it is there for
good.
--
Richard Smith
Kerry Raymond
Dec 29, 2011, 9:48:44 PM12/29/11
to
> They certainly fit within the scope of the Wikimedia Commons and
> Wikisource, but is it the best place for them? You can't upload
> spreadsheets or similar to these projects, nor can you upload structured
> data.
I prefer to think in terms of incremental solutions to problems instead of
> Wikisource, but is it the best place for them? You can't upload
> spreadsheets or similar to these projects, nor can you upload structured
> data.
Big Bang solutions. If we want a genealogical commons to provide longevity
to family history data, then I think we should ask ourselves what things
could we do this week / this month / this year to take the *first steps*
towards that vision.
Right now, this very minute, we could start scanning documents such as
church registers etc and store with with associated text transcription on
Wikisources. OK, you can't upload a spreadsheet or a GED, but surely it's
better to preserve something than nothing. And if a superior project comes
along, then the Wikisource material can be migrated to it.
Given that WMF depends on volunteers, if genealogists were seen to be out
there contributing genealogical resources into Wikisource, then I suspect
that would create a willingess within WMF to expand its support of
genealogists which might then enable the upload spreadsheets, GEDs or
whatever. Or perhaps the creation of a WikiPeople project with tools and
formats specific to genealogy.
I think coming up with a Grand Plan involving a new organisation, the need
to raise funds, etc is unlikely to get off the ground. While WMF might not
be ideal as it is, at least it has an already established the organisation,
the technical infrastructure, and the funding mechanism. WMF projects grow
and change all the time through the effort of volunteers to write the code
etc.
But, hey, that's just one strategy. Put more on the table.
Kerry
Wes Groleau
Dec 30, 2011, 1:05:38 AM12/30/11
to
On 12-29-2011 19:01, Kerry Raymond wrote:
> Could we consider Wikimedia as the place to build a genealogical
> commons? Wikimedia run many projects, Wikipedia and Commons are just two
> of them. See here for the complete list:
WeRelate.org
> Could we consider Wikimedia as the place to build a genealogical
> commons? Wikimedia run many projects, Wikipedia and Commons are just two
> of them. See here for the complete list:
(This is an acknowledgment that it exists, not an endorsement)
--
Wes Groleau
Hispanics want immigration reform but…
http://Ideas.Lang-Learn.us/russell?itemid=1493
Ian Goddard
Dec 30, 2011, 6:45:42 AM12/30/11
to
Also, in my version, provenance.
Richard Smith
Dec 30, 2011, 8:14:54 AM12/30/11
to
On 30/12/11 02:48, Kerry Raymond wrote:
>> They certainly fit within the scope of the Wikimedia Commons and
>> Wikisource, but is it the best place for them? You can't upload
>> spreadsheets or similar to these projects, nor can you upload
>> structured data.
>
> I prefer to think in terms of incremental solutions to problems instead
> of Big Bang solutions.
Me too. And that's why I much prefer Ian's distributed architecture to
>> They certainly fit within the scope of the Wikimedia Commons and
>> Wikisource, but is it the best place for them? You can't upload
>> spreadsheets or similar to these projects, nor can you upload
>> structured data.
>
> I prefer to think in terms of incremental solutions to problems instead
> of Big Bang solutions.
my own initial suggestion. Ian's suggestion is, as I understand it,
that you can host the content wherever you like. (At least, anywhere
that allows a suitable licence which those projects both do.) That
might be Wikimedia Commons; it might be your personal website; it might
be an established genealogical website (though many of them have
incompatible T&Cs, or require logging in, which renders them
incompatible with our commons).
Once you've uploaded your research, you add it to the index. This index
is the new bit, the thing that makes the commons more than just a
nebulous collection of unrelated resources. The index tells you how to
locate the resource in such a way that a computer can be set to download
the whole commons, given enough time. The index tells you about the
provenance of the resource, and allows you to verify that it really is
the version originally added to commons. The index contains information
about the places, names and dates relevant to the resource in a
structured manner. This means that someone can download all of the
resources pertaining to a given surname, or a given county, or whatever.
It may be that there is an existing service that can be used for the
index. If so, and if it seems suitable, I would definitely advocate
using it, as I fully agree with your points about not reinventing what's
already out there.
> Right now, this very minute, we could start scanning documents such as
> church registers etc and store with with associated text transcription
> on Wikisources. OK, you can't upload a spreadsheet or a GED, but surely
> it's better to preserve something than nothing. And if a superior
> project comes along, then the Wikisource material can be migrated to it.
upload them to Wikisource. Obviously this needs to be done by the owner
of the transcript (or at least, with their permission), as it will be
copyright to them.
> Given that WMF depends on volunteers, if genealogists were seen to be
> out there contributing genealogical resources into Wikisource, then I
> suspect that would create a willingess within WMF to expand its support
> of genealogists which might then enable the upload spreadsheets, GEDs or
> whatever. Or perhaps the creation of a WikiPeople project with tools and
> formats specific to genealogy.
resources (such as tabular baptism registers, and so on). But I'm still
sceptical that they will support recent research because the readership
will be so low. But I may be proved wrong. Maybe a project like
http://familypedia.wikia.com/ might be better suited to this. (Despite
their similar names, Wikia is nothing to do with the Wikimedia Foundation.)
I would like to think that a commons would encourage a "publish soon and
publish often" model. If I spend a weekend researching one branch of my
family and finally make a small breakthrough, I want to be able to jot
down a few paragraphs on the subject and add it to the commons -- a
short research note, if you like. I think it's only by actively
encouraging such things that our research will actually get preserved in
a meaningful way. Preserving a GEDCOM file or a PDF of a tree is all
well and good, but it frequently won't explain *why* you've come to the
conclusions you have. (Yes, I know GEDCOM allows for notes with the
NOTE field, and there's no reason why the same research note shouldn't
be added to your GEDCOM.) And waiting until we've completed that area
of research to upload a polished history is also no good as we generally
never do complete it.
> I think coming up with a Grand Plan involving a new organisation, the
> need to raise funds, etc is unlikely to get off the ground.
money is needed. No central organisation needs to be formed.
--
Richard Smith
Richard Smith
Dec 30, 2011, 8:45:12 AM12/30/11
to
On 30/12/11 06:05, Wes Groleau wrote:
> On 12-29-2011 19:01, Kerry Raymond wrote:
>> Could we consider Wikimedia as the place to build a genealogical
>> commons? Wikimedia run many projects, Wikipedia and Commons are just two
>> of them. See here for the complete list:
>
> WeRelate.org
Or similarly, http://familypedia.wikia.com/.
> On 12-29-2011 19:01, Kerry Raymond wrote:
>> Could we consider Wikimedia as the place to build a genealogical
>> commons? Wikimedia run many projects, Wikipedia and Commons are just two
>> of them. See here for the complete list:
>
> WeRelate.org
And to some extent, sites like these are already part of the commons.
They already have suitable licences (CC-BY-SA in both cases), support
adding media, and can be accesed transparently. It would be easy to
write a script to automatically add all this content to an index of the
commons.
I certainly don't want to discourage people from using these sites if
they want to. However, some people will not want to because of the
uncontrolled collaborative nature of it. So far as I know, you cannot
control who edits your research. Some people will be happy with that,
others will not, and I can understand both views.
So what does the commons achieve that these sites don't? For one thing,
it is an index of many such sites, and it is a way of locating
duplicates between sites. However there is a much more fundamental
difference. WeRelate and Familypedia are both person-oriented, the
commons I'm envisaging is resource-oriented. A resource might be a
photograph of a person, a scan of a will, a transcript of a parish
register, a family tree or pedigree, an essay on the history of family,
a short research note explaining a breakthrough or musing over an
apparent dead-end. A resource may pertain to a person, or several
people, or to a village, or to another resource. But the resource isn't
the person.
As such, the commons should be seen as complementing existing projects
such as WeRelate and Familypedia, not competing with them. The commons
helps you find the evidence; those sites help you interpret and present
that information.
--
Richard Smith
Ian Goddard
Dec 30, 2011, 9:58:59 AM12/30/11
to
some organisation to bootstrap things.
One of the central concepts would be a wrapper for content. The wrapper
would allow multiple "pages" so that it could hold, say a scan and its
transcription or a scan of each side of a physical page or scans and/or
transcripts of multiple page docs.
Each page would have its content type defined as a mime-type in
accordance with the current RFCs & registrations. This would allow any
type of content which is a recognised mime-type such as a spreadsheet.
I think there might be a problem with GEDCOM as I can't find it
registered at http://www.iana.org/assignments/media-types/index.html and
there may be issues with getting it registered. An unregistered type
might have to be adopted by convention.
The wrapper would have a unique ID, a UUID. It would also quote the
UUID of the preceeding link, the parent, of a source chain which would
ultimately link to an archive, publication or whatever which would
establish the provenance of the record.
This wrapper should be replicated as-is but extracted contents used as
required. If it were required to amend the contents the mechanisms
would be either to amend the contents and republish with a new UUID
quoting the original UUID as the parent or, if the intent were to
supplement by adding, say, a translation, to generate a new object,
again with its own UUID and quoting the original as a parent.
Another type of object would be an object providing an analysis of the
content. For instance the content might be the passage from Jessop's
diary relating where he went to Greens with John Goddard. This would
give rise to analysis objects listing the places, e.g. Greens, and
people, e.g. John Goddard. Each such analysis record would have its
UUID so there would be a UUID for the mention of John Goddard in this
particular tale and others for each of the other mentions of John
Goddard (including "young John Goddard" reporting on his visit to
Manchester at the time of the 1745 rising), each with their UUID.
In building my own family tree I would have entities representing
various people including my 5xggfather John and various other named
individuals and entities representing the relationships. I might then
publish entities representing 5xggfather, another John G and a
father-son link of the first to the second. Having identified several
instances of John Goddard in Jessop as 5xggfather John G I might also
publish links to this effect and another link between young John &
5xggfather's son John. I'd also publish links to the BMD information
for all members of the family. If nobody else had done so I might also
publish a place record for the Duke of Leeds public house and a link
between Jessop's "Greens" and that.
All these object types would need definitions and those definitions,
together with licences mandating the usages in regard to replication
would need to be defined. For that purpose an organisation would be
needed but it would publish standards, not material.
The system thus envisaged is rich in UUIDs identifying objects and also
in linking them. Some objects would consist of little but UUIDs.
This allows scope for indexes of object content, e.g. the name John
Goddard, UUIDs of objects offered by a site and UUIDs of objects linked
to a given UUID.
It's easy to envisage how anyone then finding one of the objects indexed
under John Goddard would be able to follow the links and retrieve other
objects relating not just to BMD data about the family but also to any
other published records which, to quote a current thread on s.g.britain,
put flesh on the begattery skeleton. And, because the main evidential
objects are the terminations of provenance chains, they'd be able to
verify why particular inferences were drawn from the data.
It's easy to envisage applications where the user, encountering a UUID
in data presented by an application, would be able to click on that UUID
and have the application automagically go out to search the indexes,
find the object identified by that UUID and download it. And, if
desired, the user could have the application then present that and other
his other objects on his own site, advertised under their UUIDs.
Ian Goddard
Dec 30, 2011, 10:03:33 AM12/30/11
to
Kerry Raymond wrote:
> Just pick a license that allows people who intend to Do The
> Right Thing to be allowed to do that
I like that. "Do The Right Thing" would be a good name for a licence.
> Just pick a license that allows people who intend to Do The
> Right Thing to be allowed to do that
It immediately labels anyone who objects as intending to do the wrong thing.
J. Hugh Sullivan
Dec 30, 2011, 11:36:57 AM12/30/11
to
On Thu, 29 Dec 2011 20:27:09 -0500, singhals <sing...@erols.com>
wrote:
>Lift yourselves out of the weeds for a minute and explain to
>me (us?) how a Genealogical Commons would significantly
>differ from (a) the LDS patron-submitted data (b)
>OneWorldTree (c) WorldConnect or even (d) all the internet?
That's a better question than "To be, or not to be."
My genealogy can whip your genealogy will always be a problem.
When, and if, I decide to share my entire tree I'll get my own web
site. Anybody can look and e-mail but control would reside in one
person. That's how it would differ from the aforementioned (a), (b),
(c) and (d).
Hugh
wrote:
>Lift yourselves out of the weeds for a minute and explain to
>me (us?) how a Genealogical Commons would significantly
>differ from (a) the LDS patron-submitted data (b)
>OneWorldTree (c) WorldConnect or even (d) all the internet?
My genealogy can whip your genealogy will always be a problem.
When, and if, I decide to share my entire tree I'll get my own web
site. Anybody can look and e-mail but control would reside in one
person. That's how it would differ from the aforementioned (a), (b),
(c) and (d).
Hugh
Richard Smith
Dec 30, 2011, 1:23:32 PM12/30/11
to
On 30/12/11 14:58, Ian Goddard wrote:
> Can I clarify what I have in mind to explain why there would have to be
> some organisation to bootstrap things.
>
> One of the central concepts would be a wrapper for content. The wrapper
> would allow multiple "pages" so that it could hold, say a scan and its
> transcription or a scan of each side of a physical page or scans and/or
> transcripts of multiple page docs.
>
> Each page would have its content type defined as a mime-type in
> accordance with the current RFCs & registrations. This would allow any
> type of content which is a recognised mime-type such as a spreadsheet. I
> think there might be a problem with GEDCOM as I can't find it registered
> at http://www.iana.org/assignments/media-types/index.html and there may
> be issues with getting it registered. An unregistered type might have to
> be adopted by convention.
>
> The wrapper would have a unique ID, a UUID. It would also quote the UUID
> of the preceeding link, the parent, of a source chain which would
> ultimately link to an archive, publication or whatever which would
> establish the provenance of the record.
[...]
> Can I clarify what I have in mind to explain why there would have to be
> some organisation to bootstrap things.
>
> One of the central concepts would be a wrapper for content. The wrapper
> would allow multiple "pages" so that it could hold, say a scan and its
> transcription or a scan of each side of a physical page or scans and/or
> transcripts of multiple page docs.
>
> Each page would have its content type defined as a mime-type in
> accordance with the current RFCs & registrations. This would allow any
> type of content which is a recognised mime-type such as a spreadsheet. I
> think there might be a problem with GEDCOM as I can't find it registered
> at http://www.iana.org/assignments/media-types/index.html and there may
> be issues with getting it registered. An unregistered type might have to
> be adopted by convention.
>
> The wrapper would have a unique ID, a UUID. It would also quote the UUID
> of the preceeding link, the parent, of a source chain which would
> ultimately link to an archive, publication or whatever which would
> establish the provenance of the record.
I agree with most of your points, though I think with a few tweaks we
can remove the requirement for a central server. And I think that would
be a good thing.
Let me explain by way of an example. I have a scan, S, of a will, and a
transcript, T. You would have these both attached to some wrapper and a
UUID assigned to it. The problem here is who's job is it to assign that
UUID? If two index servers know about S and T, they might both generate
wrappers with distinct UUIDs. That's one reason why your scheme seems
to need a central server.
But instead of thinking of the wrapper as some persistent object with a
UUID that contains S and T, let's instead consider it just as a view of
index generated on the fly by that index server. The index server has
been told and stores the fact that T and S are related to each other in
a specific way: namely that T is a transcription of S. If I ask about
S, it'll show me S and its related documents, T in this case; and
similarly, if I ask about T, it'll show me both S and T. I never assign
a UUID to either the wrapper, or to S or T. I simply use the URL of S
or of T.
Now suppose someone makes a copy of S, let's call it S', and let's say
they write a short research note, N, about S'. The index server is told
that N is a note about S', but not that any relationship exists between
S and S'. However, the index server can immediately infer that S and S'
are identical (probably by comparing hashes) and consider the two URLs
to be interchangeable. So now if I ask the index server to show me its
information on S, it'll generate the set of identical documents {S, S'}
and find all of their related documents, {T, N}. It then displays a
page containing S, T and N.
But a second index server may still only know about S an T, or perhaps
only know about S' and N, and will give a different view.
--
Richard Smith
Richard Smith
Dec 30, 2011, 2:21:07 PM12/30/11
to
Ian Goddard wrote:
> Each page would have its content type defined as a mime-type in
> accordance with the current RFCs & registrations. This would allow any
> type of content which is a recognised mime-type such as a spreadsheet. I
> think there might be a problem with GEDCOM as I can't find it registered
> at http://www.iana.org/assignments/media-types/index.html and there may
> be issues with getting it registered. An unregistered type might have to
> be adopted by convention.
The de facto standard appears to be "application/x-gedcom". I'm not
> Each page would have its content type defined as a mime-type in
> accordance with the current RFCs & registrations. This would allow any
> type of content which is a recognised mime-type such as a spreadsheet. I
> think there might be a problem with GEDCOM as I can't find it registered
> at http://www.iana.org/assignments/media-types/index.html and there may
> be issues with getting it registered. An unregistered type might have to
> be adopted by convention.
sure I agree it should be in the application type rather than text type,
but Googling the them both finds 23,100 hits for "application/x-gedcom"
and only 3 for "text/x-gedcom". (Presumably "+xml" should be appended
for GEDCOM 6.)
> Another type of object would be an object providing an analysis of the
> content. For instance the content might be the passage from Jessop's
> diary relating where he went to Greens with John Goddard. This would
> give rise to analysis objects listing the places, e.g. Greens, and
> people, e.g. John Goddard. Each such analysis record would have its UUID
> so there would be a UUID for the mention of John Goddard in this
> particular tale and others for each of the other mentions of John
> Goddard (including "young John Goddard" reporting on his visit to
> Manchester at the time of the 1745 rising), each with their UUID.
passage simply reads
Today I met Mr John Goddard and we went to Greens.
Instead of, or as well as, this purely textual transcription, you can
write something like
<span property="ns:dateWritten"
datatype="xsd:date" content="1820-01-20">Today</span>
<span property="ns:originalAuthor"><span typeof="ns::Person"
about="urn:uuid:4c359533-aa5b-492c-956f-a2651ec886f4"
property="ns:surname" content="Jessop">I</span></span>
met <span typeof="ns:Person"
about="urn:uuid:d8e6a531-5dee-47a1-a0e2-ca5dbffd87c0"
property="ns:name">Mr John Goddard</span>
and we went to <span typeof="ns:Place"
about="urn:uuid:f2ded1ce-05c3-45f5-95c0-e54f559c1fbc">Greens</span>.
In a text editor that looks a mess, but it a XML/HTML viewer will
display the same basic text. And, modulo any syntactic errors I may
have made (and there almost certainly are some), it now conveys
additional semantic information. It says that the document was written
on 20th Jan 1820 and that's what 'today' means. It says that 'I' refers
to the author of the document, assigns a UUID to him, and states that
his surname is Jessop. It says that 'Mr John Goddard' is a person, and
assigns a UUID to him. And if says that 'Greens' is a place and gives
that a UUID too.
If you prefer to use URIs instead of UUIDs, you can. An indexing engine
can now read this, and note which people, places, dates and so on are
mentioned in it. And if it turns out that unbeknownst to you another
researcher has assigned their own UUID to Jessop, then (or someone else
who notices this) can state this, e.g. in short piece of RDF:
<rdf:Description rdf:about="...their uuid...">
<owl:sameAs rdf:resource="...your uuid..."/>
</rdf:Description>
(The difference between RDF and RDFa is that the former is more powerful
and is for stand-alone statements; the latter is for embedding in other
content, such as HTML.)
--
Richard Smith
Tony Proctor
Dec 31, 2011, 7:31:36 AM12/31/11
to
"Richard Smith" <ric...@ex-parrot.com> wrote in message
news:9m6dt6...@mid.individual.net...
persistent. Unfortunately, many people have committed their data to
proprietary databases and pushing one of those into this commons is assuming
that the associated product will be there forever. Exporting to GEDCOM (even
if the format were registered) won't help as it is currently incapable of
fully & accurately representing everyone's data.
At the time of writing, their is no universally accepted machine-readable
data format that would represent the core aspects of our data (i.e. ignoring
attachments), including lineage-based data, event-based data, evidence-based
data, and full use of source citations in a standard fashion.
As for UUIDs and URIs (rather than URLs), they are obviously not
interchangeable. A UUID (aka GUID) is a randomised identifier that is
designed to be spatially and temporally unique. There is no meaning in the
resultant identifier - it is simply an identifier that is unique. A URI, on
the other hand, has semantics visible in the value. It is based around an
issuing authority - but not a restrictive centralised one - and that is also
visible in the value. The value is also extensible and can include
versioning and parameters if necessary. Hence, I much favour the use of URIs
:-)
Tony Proctor
Ian Goddard
Dec 31, 2011, 8:45:26 AM12/31/11
to
Tony Proctor wrote:
> As for UUIDs and URIs (rather than URLs), they are obviously not
> interchangeable. A UUID (aka GUID) is a randomised identifier that is
> designed to be spatially and temporally unique. There is no meaning in the
> resultant identifier - it is simply an identifier that is unique. A URI, on
> the other hand, has semantics visible in the value. It is based around an
> issuing authority - but not a restrictive centralised one - and that is also
> visible in the value. The value is also extensible and can include
> versioning and parameters if necessary. Hence, I much favour the use of URIs
Version 1 UUIDs depended on central authority for their uniqueness as
> As for UUIDs and URIs (rather than URLs), they are obviously not
> interchangeable. A UUID (aka GUID) is a randomised identifier that is
> designed to be spatially and temporally unique. There is no meaning in the
> resultant identifier - it is simply an identifier that is unique. A URI, on
> the other hand, has semantics visible in the value. It is based around an
> issuing authority - but not a restrictive centralised one - and that is also
> visible in the value. The value is also extensible and can include
> versioning and parameters if necessary. Hence, I much favour the use of URIs
part of the UUID was essentially a supposedly unique token. However as
this token comes as part of the computer - it's the MAC of the network
interface the user generating the UUID doesn't have to take any specific
steps to get such a token other than getting access to the computer itself.
Version 4 UUIDs depend on generating a random number sufficiently large
to ensure it's sufficiently unlikely to be duplicated by any other UUID.
No authority is involved.
I can simply type "uuid" or "uuid -v 4" at my computer's command line
asmany times as I like and get a supply of UUIDs
I don't know, however, how I would generate a unique URI so freely.
Could you please explain how this would work?
Tony Proctor
Dec 31, 2011, 9:12:27 AM12/31/11
to
"Ian Goddard" <godd...@hotmail.co.uk> wrote in message
news:9m8hvm...@mid.individual.net...
your own URIs freely from that domain with the knowledge that no one else's
will clash, e.g. http://www.me.com/mydata/etc/v1. The use of the http://
scheme prefix, here, is not implying that this is dereferencable - it is
simply indicating the nature of the domain. Although I cannot point to an
example, deriving URIs from some other registered entity would work, e.g. an
email address.
An example of a very common use of such URIs is in the identification of a
namespace within an XML document.
Tony Proctor
Ian Goddard
Dec 31, 2011, 9:47:59 AM12/31/11
to
ISTM that this scheme is restricted to text. Could it be applied to
other formats - including formatted text such as WP documents, PDFs,
spreadsheets & GEDCOMS?
There may be a case for embedding analysis objects in evidence objects.
It makes for an economy of transmission. But the analysis objects
have their own existence. When the transmitted XML is stored internally
in some S/W package it's likely going to have to be picked apart so that
the individual objects can be presented to the user. It might be
difficult to ensure that it gets put back together correctly for export
and if it isn't the software will fail the criterion of replication
integrity.
> If you prefer to use URIs instead of UUIDs, you can. An indexing engine
> can now read this, and note which people, places, dates and so on are
> mentioned in it. And if it turns out that unbeknownst to you another
> researcher has assigned their own UUID to Jessop, then (or someone else
> who notices this) can state this, e.g. in short piece of RDF:
>
> <rdf:Description rdf:about="...their uuid...">
> <owl:sameAs rdf:resource="...your uuid..."/>
> </rdf:Description>
>
of different classes. As you observed Jessop is special in this context
because he's the author and should be represented as a source object at
the head of the provenance chain. He can also be represented by a
number of analysis objects. And if we're drawing up a Jessop family
tree he's Arthur Jessop and he's a represented by a reconstruction
object - or interpretation object if you prefer. All these are objects
of different classes because some of the other attributes will be
different even if they share string attributes of the same value (or not
necessarily the same when you take into account historical spellings!).
Nevertheless objects of the type you represent here as XML are important
because they represent our understanding of the raw data. But it would
be necessary to make provision for a measure of confidence in the
identification and an explanation for it.
J. Hugh Sullivan
Dec 31, 2011, 10:14:54 AM12/31/11
to
running. :)
Hugh
Ian Goddard
Dec 31, 2011, 10:20:07 AM12/31/11
to
need for a domain name.
pblair
Jan 1, 2012, 12:02:12 AM1/1/12
to
Yep, I love all these techniques. And I'm sure that there are a lot of
honest people currently posting information in various places.
But that may or may not have value, just as this new location may or
may not have value. Creating a repository is not difficult. But there
will be just as many erratic posts (or just plain mischievous ones) as
we have now. Where's the control and what's the benefit?
Paul
honest people currently posting information in various places.
But that may or may not have value, just as this new location may or
may not have value. Creating a repository is not difficult. But there
will be just as many erratic posts (or just plain mischievous ones) as
we have now. Where's the control and what's the benefit?
Paul
Richard Smith
Jan 1, 2012, 7:21:52 AM1/1/12
to
Ian Goddard wrote:
UUIDs are valid URNs, which are a type of URI. For example, this is a URI:
urn:uuid:d8e6a531-5dee-47a1-a0e2-ca5dbffd87c0
--
Richard Smith
> Tony Proctor wrote:
>> An example of a very common use of such URIs is in the identification
>> of a namespace within an XML document.
>
> OK, ISWYM now. But I think it has a few problems revolving round the
> need for a domain name.
Me too. Fortunately, URIs needn't involve a domain name, and 'wrapped'
>> An example of a very common use of such URIs is in the identification
>> of a namespace within an XML document.
>
> OK, ISWYM now. But I think it has a few problems revolving round the
> need for a domain name.
UUIDs are valid URNs, which are a type of URI. For example, this is a URI:
urn:uuid:d8e6a531-5dee-47a1-a0e2-ca5dbffd87c0
--
Richard Smith
Tony Proctor
Jan 1, 2012, 7:33:49 AM1/1/12
to
news:9mb1f0...@mid.individual.net...
A true URN identifies an object in a hierarchical namespace Richard. It
therefore has a restricted form of the full URI syntax (as you've shown) and
the 'schemes' have to authorised ones - I admit I didn't realise uuid was an
authorised one. As result, there is still a loss of visible semantics in
the names, and you can't extend them in the same way as a full URI
Tony Proctor
therefore has a restricted form of the full URI syntax (as you've shown) and
the 'schemes' have to authorised ones - I admit I didn't realise uuid was an
authorised one. As result, there is still a loss of visible semantics in
the names, and you can't extend them in the same way as a full URI
Tony Proctor
Lesley Robertson
Jan 1, 2012, 10:27:37 AM1/1/12
to
"Richard Smith" wrote in message
news:9m3alt...@mid.individual.net...
A recent thread on soc.genealogy.methods under the subject
"Genealogical
Estate Planning" has started me thinking about the problem of
preserving
the results of our genealogical research so that it will still be
accessible to future generations, and I'm coming to the view that
there
is specific hole in the genealogy marketplace for a service that does
this. What follows is quite a long summary of the sort of service I
could envisage someone providing quite easily, but so far as I know,
no-one is currently doing. I'd be interested if anyone knows of any
project along these lines, and whether you agree with my analysis of
how
best to preserve your research.
*******************
A point that doesn't seem to have been dealt with thus far is quality
control. If you don't want the system to be dismissed as "good for
clues but not to be trusted" (cf the IGI) than you have to have some
sort of measure of quality - only fully sourced, only if based on
primary resources, or something else.
That then tends to imply some sort of moderation.
Wikipedia seems to operate on an "if anyone can alter the entries,
they’ll eventually become correct. Doesn't work - Despite the fact
that I hold one famous Professor's death cert, copies of every known
obit, and a telegram from his sister saying "my brother died this
morning", every time I correct his death date on Wikipedia, the same
person changes it back to what he fondly considers to be correct.
In that case, it's not over-important, but the scope offered to
someone with a bee in their bonnet by a genealogical dtabase of some
sort is immense.
Lesley Robertson
news:9m3alt...@mid.individual.net...
A recent thread on soc.genealogy.methods under the subject
"Genealogical
Estate Planning" has started me thinking about the problem of
preserving
the results of our genealogical research so that it will still be
accessible to future generations, and I'm coming to the view that
there
is specific hole in the genealogy marketplace for a service that does
this. What follows is quite a long summary of the sort of service I
could envisage someone providing quite easily, but so far as I know,
no-one is currently doing. I'd be interested if anyone knows of any
project along these lines, and whether you agree with my analysis of
how
best to preserve your research.
*******************
A point that doesn't seem to have been dealt with thus far is quality
control. If you don't want the system to be dismissed as "good for
clues but not to be trusted" (cf the IGI) than you have to have some
sort of measure of quality - only fully sourced, only if based on
primary resources, or something else.
That then tends to imply some sort of moderation.
Wikipedia seems to operate on an "if anyone can alter the entries,
they’ll eventually become correct. Doesn't work - Despite the fact
that I hold one famous Professor's death cert, copies of every known
obit, and a telegram from his sister saying "my brother died this
morning", every time I correct his death date on Wikipedia, the same
person changes it back to what he fondly considers to be correct.
In that case, it's not over-important, but the scope offered to
someone with a bee in their bonnet by a genealogical dtabase of some
sort is immense.
Lesley Robertson
Tony Proctor
Jan 1, 2012, 10:41:15 AM1/1/12
to
"Lesley Robertson" <l.a.ro...@tnw.tudelft.nl> wrote in message
news:oLmdnVdiG-P35p3S...@infopact.nl...
upper hand? I notice Wikipedia are stepping up the need for citations and I
have a task to provide some myself on one of my contributions. If he undoes
your change without providing verifiable citations then that would be
grounds for a dispute requiring arbitration.
Tony Proctor
singhals
Jan 1, 2012, 11:17:34 AM1/1/12
to gen...@rootsweb.com
Tony Proctor wrote:
>
> "Lesley Robertson"<l.a.ro...@tnw.tudelft.nl> wrote in message
> news:oLmdnVdiG-P35p3S...@infopact.nl...
>> "Richard Smith" wrote in message news:9m3alt...@mid.individual.net...
>>
>> A recent thread on soc.genealogy.methods under the subject "Genealogical
>> Estate Planning" has started me thinking about the problem of preserving
>> the results of our genealogical research so that it will still be
>> accessible to future generations, and I'm coming to the view that there
>> is specific hole in the genealogy marketplace for a service that does
>> this. What follows is quite a long summary of the sort of service I
>> could envisage someone providing quite easily, but so far as I know,
>> no-one is currently doing. I'd be interested if anyone knows of any
>> project along these lines, and whether you agree with my analysis of how
>> best to preserve your research.
>>
>> *******************
>>
>> A point that doesn't seem to have been dealt with thus far is quality
>> control. If you don't want the system to be dismissed as "good for clues
>> but not to be trusted" (cf the IGI) than you have to have some sort of
>> measure of quality - only fully sourced, only if based on primary
>> resources, or something else.
>>
>> That then tends to imply some sort of moderation.
>>
>> Wikipedia seems to operate on an "if anyone can alter the entries, they�ll
>
> "Lesley Robertson"<l.a.ro...@tnw.tudelft.nl> wrote in message
> news:oLmdnVdiG-P35p3S...@infopact.nl...
>> "Richard Smith" wrote in message news:9m3alt...@mid.individual.net...
>>
>> A recent thread on soc.genealogy.methods under the subject "Genealogical
>> Estate Planning" has started me thinking about the problem of preserving
>> the results of our genealogical research so that it will still be
>> accessible to future generations, and I'm coming to the view that there
>> is specific hole in the genealogy marketplace for a service that does
>> this. What follows is quite a long summary of the sort of service I
>> could envisage someone providing quite easily, but so far as I know,
>> no-one is currently doing. I'd be interested if anyone knows of any
>> project along these lines, and whether you agree with my analysis of how
>> best to preserve your research.
>>
>> *******************
>>
>> A point that doesn't seem to have been dealt with thus far is quality
>> control. If you don't want the system to be dismissed as "good for clues
>> but not to be trusted" (cf the IGI) than you have to have some sort of
>> measure of quality - only fully sourced, only if based on primary
>> resources, or something else.
>>
>> That then tends to imply some sort of moderation.
>>
>> eventually become correct. Doesn't work - Despite the fact that I hold one
>> famous Professor's death cert, copies of every known obit, and a telegram
>> from his sister saying "my brother died this morning", every time I
>> correct his death date on Wikipedia, the same person changes it back to
>> what he fondly considers to be correct.
>> In that case, it's not over-important, but the scope offered to someone
>> with a bee in their bonnet by a genealogical dtabase of some sort is
>> immense.
>>
>> Lesley Robertson
>>
>
> If your correction provides citations Lesley then doesn't that give you the
> upper hand? I notice Wikipedia are stepping up the need for citations and I
> have a task to provide some myself on one of my contributions. If he undoes
> your change without providing verifiable citations then that would be
> grounds for a dispute requiring arbitration.
Arbitration, by definition, requires a hierarchy that
>> famous Professor's death cert, copies of every known obit, and a telegram
>> from his sister saying "my brother died this morning", every time I
>> correct his death date on Wikipedia, the same person changes it back to
>> what he fondly considers to be correct.
>> In that case, it's not over-important, but the scope offered to someone
>> with a bee in their bonnet by a genealogical dtabase of some sort is
>> immense.
>>
>> Lesley Robertson
>>
>
> If your correction provides citations Lesley then doesn't that give you the
> upper hand? I notice Wikipedia are stepping up the need for citations and I
> have a task to provide some myself on one of my contributions. If he undoes
> your change without providing verifiable citations then that would be
> grounds for a dispute requiring arbitration.
enforces obedience. IOW, moderation.
With nothing but a file-holding "Commons" there is no
governing hierarchy with moderation/arbitration rights.
With those rights, you need workers, labor-hours, and $$$.
Without them, you've got a data-dump site, where the "facts"
may be right, may be wrong, but sure are fun to read.
Cheryl
Ian Goddard
Jan 1, 2012, 11:45:02 AM1/1/12
to
Lesley Robertson wrote:
> "Richard Smith" wrote in message news:9m3alt...@mid.individual.net...
>
> A recent thread on soc.genealogy.methods under the subject "Genealogical
> Estate Planning" has started me thinking about the problem of preserving
> the results of our genealogical research so that it will still be
> accessible to future generations, and I'm coming to the view that there
> is specific hole in the genealogy marketplace for a service that does
> this. What follows is quite a long summary of the sort of service I
> could envisage someone providing quite easily, but so far as I know,
> no-one is currently doing. I'd be interested if anyone knows of any
> project along these lines, and whether you agree with my analysis of how
> best to preserve your research.
>
> *******************
>
> A point that doesn't seem to have been dealt with thus far is quality
> control. If you don't want the system to be dismissed as "good for clues
> but not to be trusted" (cf the IGI) than you have to have some sort of
> measure of quality - only fully sourced, only if based on primary
> resources, or something else.
That's why I said that an evidential record should sit at the end of an
> "Richard Smith" wrote in message news:9m3alt...@mid.individual.net...
>
> A recent thread on soc.genealogy.methods under the subject "Genealogical
> Estate Planning" has started me thinking about the problem of preserving
> the results of our genealogical research so that it will still be
> accessible to future generations, and I'm coming to the view that there
> is specific hole in the genealogy marketplace for a service that does
> this. What follows is quite a long summary of the sort of service I
> could envisage someone providing quite easily, but so far as I know,
> no-one is currently doing. I'd be interested if anyone knows of any
> project along these lines, and whether you agree with my analysis of how
> best to preserve your research.
>
> *******************
>
> A point that doesn't seem to have been dealt with thus far is quality
> control. If you don't want the system to be dismissed as "good for clues
> but not to be trusted" (cf the IGI) than you have to have some sort of
> measure of quality - only fully sourced, only if based on primary
> resources, or something else.
evidence chain and suggested a format which could contain an image and a
transcription. I also separated the transcript from its analysis and
interpretation.
One of the problems with IGI is that it makes no provision for
transcripts. It provides analysis or interpretation. One instance I've
quoted before is one where the text reads something like "Wife of John
Goddard Ch" which someone of limited understanding has recorded as a
baptism of the wife of John Goddard and given her the name Christiana.
And then compounded this by taking an entry "Jonathan son of John
Goddard bapt" which makes no mention of the mother's name and listing
the mother, not mentioned in the original as Christiana. A proper
transcription would have made it clear that the word Christiana wasn't
mentioned, a proper provenance chain would have shown that the title of
the source was "Register of Christenings and Churchings" and an image of
the page would have made shown that the normal form was to put the verb
at the end, as in the second example.
> Wikipedia seems to operate on an "if anyone can alter the entries,
> they’ll eventually become correct. Doesn't work - Despite the fact that
> I hold one famous Professor's death cert, copies of every known obit,
> and a telegram from his sister saying "my brother died this morning",
> every time I correct his death date on Wikipedia, the same person
> changes it back to what he fondly considers to be correct.
> In that case, it's not over-important, but the scope offered to someone
> with a bee in their bonnet by a genealogical dtabase of some sort is
> immense.
found. Amendments should be offered only as attached records. In your
case you could have provided images of each of the resources. If
someone still wanted to post an unsourced and incorrect claim you
wouldn't need to correct anything. The evidence would be there to be
read and readers would form their own opinion. It might even convince
the bee-keeper ;)
Wes Groleau
Jan 1, 2012, 1:45:23 PM1/1/12
to
On 01-01-2012 10:27, Lesley Robertson wrote:
> If you don't want the system to be dismissed as "good for clues but not
> to be trusted" (cf the IGI)
And cf ancestral file and ancestry.com and familysearch.org and
> If you don't want the system to be dismissed as "good for clues but not
> to be trusted" (cf the IGI)
worldconnect.rootsweb.com and .....
--
Wes Groleau
People would have more leisure time if it weren't
for all the leisure-time activities that use it up.
— Peg Bracken
singhals
Jan 1, 2012, 10:23:09 PM1/1/12
to gen...@rootsweb.com
users.
As Hugh says, many people who are doing genealogy haven't a
clue how to do what you're talking about, among those who
DO, a fair percentage won't care to make the effort of doing it.
If the "commons" is to be open to ALL and not just to the
techno-few, then it needs to be as simplistic as possible
and as easy to use as possible, even if that means forgoing
some desirable options.
Few of us care HOW a program stores data so long as it
yankee-well gives it back to us the way we put it in.
Cheryl
Lesley Robertson
Jan 2, 2012, 5:21:28 AM1/2/12
to
"Tony Proctor" <tony@proctor_NoMore_SPAM.net> wrote in message
news:jdpuij$um7$1...@reader01.news.esat.net...
>
>
> If your correction provides citations Lesley then doesn't that give you
> the upper hand? I notice Wikipedia are stepping up the need for citations
> and I have a task to provide some myself on one of my contributions. If he
> undoes your change without providing verifiable citations then that would
> be grounds for a dispute requiring arbitration.
>
> Tony Proctor
I gave up about a year back - far too much hassle! Anyone doing serious
>
> If your correction provides citations Lesley then doesn't that give you
> the upper hand? I notice Wikipedia are stepping up the need for citations
> and I have a task to provide some myself on one of my contributions. If he
> undoes your change without providing verifiable citations then that would
> be grounds for a dispute requiring arbitration.
>
> Tony Proctor
research on the guy can quite easily find the accurate data anywhere BUT
Wikipedia. It's his achievements that are important rather than his DOD,
anyway.
Thanks for the suggestion though - maybe one day when I run out of things to
do.
Lesley Robertson
Ian Goddard
Jan 2, 2012, 5:43:27 AM1/2/12
to
companies. My users also didn't care how data was stored. That was MY
job. Because if I hadn't cared they might not have got it back.
But let's take Hugh who you quoted as an example. We know from what
he's said in the past when we've discussed spelling variations that he's
spent a lot of time collecting records with all sorts of spelling
variations of Sullivan. He must have a lot of data which would be
useful to other Sullivan researchers but not necessarily connected into
trees. Let's assume he wishes to make this data - and preferably his
raw data, complete with full details as to where each record came from -
available to other Sullivan researchers. How does he do it? (Remember
that it was the genealogical estate planning thread on s.g.m that
inspired Richard to start this thread.)
At the moment his only option would be to find some site to which he
could upload his collection in GEDCOM format. Apart from the fact that
has a number of shortcomings which limit what GEDCOM can usefully convey
this isn't necessarily a recipe for safekeeping of that data. Even if
the site makes the data permits downloading it probably won't permit
wholesale copying to other sites, it might well remove any traceability
beyond itself and its owner might decide to simply take it offline &
dump the data at some point in the future, cf Geocities.
What we're considering is ways & means of providing a better way of
doing this. From Hugh's POV he would need some additional S/W which
would gather up data from whatever program he's using now and repackage
it into a suitable form and then upload it to some site which would
/encourage/ sharing.
Yup, this S/W would need to be user-friendly and so would the S/W which
would enable others to find it & use it. But that user-friendly S/W
doesn't get built unless there's a sound data model, designed to
facilitate sharing, on which to base it.
Ian Goddard
Jan 2, 2012, 12:00:52 PM1/2/12
to
Last Saturday the Times published an article by Tim Berners-Lee & Nigel
Shadbolt, co-directors of HMG's projected Open Data Institute about
opening up government data with the general approach that we've already
paid for it so it should be free.
Unfortunately I can't provide a link as this paean of praise to open
data is imprisoned behind the Times' paywall. Let's hope that this is
no more than one of life's little ironies.
They're in favour of just making stuff available and letting third
parties add value such as indexing. One of the data sources they list
is census data.
Here's a scenario based on the assumption that they actually succeeded
and that a scheme such as Richard & I have outlined existed & was used
to publish this data.
I visit the website where the data is published. The data isn't indexed
but I can drill down geographically to a location I think might be
interesting & browse through the images of census sheets. I find a
useful sheet & click on a link. My genealogical S/W fires up and asks
if I want to import this data. I click "OK" and my S/W appears to grab
the image & incorporate it into my database & present it along with all
the geographical references.
In fact what's happened is that the link is to an object of a type which
is registered with my browser as having to be opened by my S/W (your
browser probably does something similar every time you click on a link
to a PDF). The object is, as I described elsewhere, a wrapper for the
image & has a link to a parent object which is the last level of
geographical information. This link is in the form of a UUID (or a URI)
as discussed elsewhere in the thread. The S/W has stored this object
and then started to follow the links and grabbed those. At some stage
it finds one of the links points back to some object it already has,
maybe just the object that describes the publishing authority, maybe the
census year object. Maybe the remote site packaged all these objects
together in the download in which case my S/W won't have had to search
for them but will have discarded the duplicates which it can recognise
because it already has objects of the same UUID. Whatever. It now has a
complete chain of authority from the census page back to the publisher.
It's extracted the data, including the image, and presented them
nicely on the screen. The details, of course, are totally hidden; all I
did was click on a web link & see the results.
Of course I'm still just looking at an image. In order to get the data
as text I open a data entry form which allows me to transcribe the names
as written - if the census says Geo or Wm I enter this rather than
expand it up to the full name. As it's a census I can enter
relationships, separating households, indicating head, relationship to
head etc. As I'm feeling community-minded I list all the households,
not just the family I'm interested in. As it happens my page includes a
Sullivan & a Cressop.
What I'm doing behind the scenes is creating a small network of objects
representing an analysis of the census page linked by UUIDs. But I'm
not directly aware of this. I'm just filling in a form on the screen.
Because we're operating a genealogical commons there will be servers to
which I can upload my latest discoveries. When I've entered all my data
I click a button to sync with my chosen server.
My software packages up the data into a similar type of external
representation as I downloaded. Of course I don't see this. All I've
done is select a server at some time & just clicked a button now.
My uploaded data will include my analysis. It might also include a copy
of what I've downloaded. Because we're operating a commons there's no
problem; provided my S/W adheres to the requirement of replicating the
data correctly the copyright licence allows me to do this.
When Cheryl next runs her S/W after a few seconds it pops up a message
to tell her that there's a new record she might be interested in. She
clicks a button and is presented with the contents of my analysis object
of the Cressop entry from the census. She clicks a button and the
census image and its geographical references are shown. She clicks
another button to import the data. She recognises the Wm Cressop as a
William Cressap for whom she already has a record in her database. She
finds that record and drags a copy of my Wm Cressop listing onto it.
What's happened is that there are indexing sites which are interested in
our genealogical records. Several of them have scanned the server to
which I uploaded my objects and, because they now contain indexable
text, have indexed names against UUIDs. Maybe my server is also one of
these. Other indexes, on the same or other sites, will have indexed
object UUIDs against sites which offer the objects for download. Cheryl
has set up her S/W to check a couple of these name index sites for
Cressap and variants. It did so when it started up, discovered the new
index entries, recognised that they pointed to the same object, checked
and found that she didn't have a copy of the object and then raised a
single message.
She has successively downloaded and viewed firstly the contents of my
analysis and then the evidence object and its provenance chain and
viewed their contents. Her S/W will have consulted the indexes of UUIDs
against sites to find where it can download from. She's then imnported
copies into her database and finally created a link object within her
database between my analysis object naming Wm Cressop and her object
representing the historical William Cressap. Through this link she can
then navigate back to the image of the census whenever she wishes. All
these internal links are mediated by UUIDs. If she wishes to upload her
William Cressap object and the link object into the commons it will all
make sense to anyone who downloads them because those UUIDs uniquely
identify this particular set of objects and their copies. But she
doesn't need to bother with these details. As far as she's concerned
all she's done is list a few index sites and then clicked and dragged &
dropped.
When Hugh runs his genealogical S/W nothing happens. Someone who's
concentrating on Sullivans has discovered this census page a month ago
but only posted an analysis of the Sullivan entry. Hugh's S/W
discovered the page by that post. As Hugh now has a copy of the census
page object his S/W recognised that and didn't bother telling him what
he already knows.
Now, if the underpinning was in place is there anything in this scenario
which would be beyond the average genealogical S/W user?
Shadbolt, co-directors of HMG's projected Open Data Institute about
opening up government data with the general approach that we've already
paid for it so it should be free.
Unfortunately I can't provide a link as this paean of praise to open
data is imprisoned behind the Times' paywall. Let's hope that this is
no more than one of life's little ironies.
They're in favour of just making stuff available and letting third
parties add value such as indexing. One of the data sources they list
is census data.
Here's a scenario based on the assumption that they actually succeeded
and that a scheme such as Richard & I have outlined existed & was used
to publish this data.
I visit the website where the data is published. The data isn't indexed
but I can drill down geographically to a location I think might be
interesting & browse through the images of census sheets. I find a
useful sheet & click on a link. My genealogical S/W fires up and asks
if I want to import this data. I click "OK" and my S/W appears to grab
the image & incorporate it into my database & present it along with all
the geographical references.
In fact what's happened is that the link is to an object of a type which
is registered with my browser as having to be opened by my S/W (your
browser probably does something similar every time you click on a link
to a PDF). The object is, as I described elsewhere, a wrapper for the
image & has a link to a parent object which is the last level of
geographical information. This link is in the form of a UUID (or a URI)
as discussed elsewhere in the thread. The S/W has stored this object
and then started to follow the links and grabbed those. At some stage
it finds one of the links points back to some object it already has,
maybe just the object that describes the publishing authority, maybe the
census year object. Maybe the remote site packaged all these objects
together in the download in which case my S/W won't have had to search
for them but will have discarded the duplicates which it can recognise
because it already has objects of the same UUID. Whatever. It now has a
complete chain of authority from the census page back to the publisher.
It's extracted the data, including the image, and presented them
nicely on the screen. The details, of course, are totally hidden; all I
did was click on a web link & see the results.
Of course I'm still just looking at an image. In order to get the data
as text I open a data entry form which allows me to transcribe the names
as written - if the census says Geo or Wm I enter this rather than
expand it up to the full name. As it's a census I can enter
relationships, separating households, indicating head, relationship to
head etc. As I'm feeling community-minded I list all the households,
not just the family I'm interested in. As it happens my page includes a
Sullivan & a Cressop.
What I'm doing behind the scenes is creating a small network of objects
representing an analysis of the census page linked by UUIDs. But I'm
not directly aware of this. I'm just filling in a form on the screen.
Because we're operating a genealogical commons there will be servers to
which I can upload my latest discoveries. When I've entered all my data
I click a button to sync with my chosen server.
My software packages up the data into a similar type of external
representation as I downloaded. Of course I don't see this. All I've
done is select a server at some time & just clicked a button now.
My uploaded data will include my analysis. It might also include a copy
of what I've downloaded. Because we're operating a commons there's no
problem; provided my S/W adheres to the requirement of replicating the
data correctly the copyright licence allows me to do this.
When Cheryl next runs her S/W after a few seconds it pops up a message
to tell her that there's a new record she might be interested in. She
clicks a button and is presented with the contents of my analysis object
of the Cressop entry from the census. She clicks a button and the
census image and its geographical references are shown. She clicks
another button to import the data. She recognises the Wm Cressop as a
William Cressap for whom she already has a record in her database. She
finds that record and drags a copy of my Wm Cressop listing onto it.
What's happened is that there are indexing sites which are interested in
our genealogical records. Several of them have scanned the server to
which I uploaded my objects and, because they now contain indexable
text, have indexed names against UUIDs. Maybe my server is also one of
these. Other indexes, on the same or other sites, will have indexed
object UUIDs against sites which offer the objects for download. Cheryl
has set up her S/W to check a couple of these name index sites for
Cressap and variants. It did so when it started up, discovered the new
index entries, recognised that they pointed to the same object, checked
and found that she didn't have a copy of the object and then raised a
single message.
She has successively downloaded and viewed firstly the contents of my
analysis and then the evidence object and its provenance chain and
viewed their contents. Her S/W will have consulted the indexes of UUIDs
against sites to find where it can download from. She's then imnported
copies into her database and finally created a link object within her
database between my analysis object naming Wm Cressop and her object
representing the historical William Cressap. Through this link she can
then navigate back to the image of the census whenever she wishes. All
these internal links are mediated by UUIDs. If she wishes to upload her
William Cressap object and the link object into the commons it will all
make sense to anyone who downloads them because those UUIDs uniquely
identify this particular set of objects and their copies. But she
doesn't need to bother with these details. As far as she's concerned
all she's done is list a few index sites and then clicked and dragged &
dropped.
When Hugh runs his genealogical S/W nothing happens. Someone who's
concentrating on Sullivans has discovered this census page a month ago
but only posted an analysis of the Sullivan entry. Hugh's S/W
discovered the page by that post. As Hugh now has a copy of the census
page object his S/W recognised that and didn't bother telling him what
he already knows.
Now, if the underpinning was in place is there anything in this scenario
which would be beyond the average genealogical S/W user?
Wes Groleau
Jan 2, 2012, 12:17:40 PM1/2/12
to
On 01-02-2012 12:00, Ian Goddard wrote:
> My uploaded data will include my analysis. It might also include a copy
> of what I've downloaded. Because we're operating a commons there's no
> problem; provided my S/W adheres to the requirement of replicating the
> data correctly the copyright licence allows me to do this.
Requires compatible software on your computer and the server. As soon
> My uploaded data will include my analysis. It might also include a copy
> of what I've downloaded. Because we're operating a commons there's no
> problem; provided my S/W adheres to the requirement of replicating the
> data correctly the copyright licence allows me to do this.
as some person, group, or company produces such software, people will
start using it. When the number of such people gets large enough to be
noticed, several commercial and one or more open-source competitors will
appear, most of them incompatible with the original and each other. Or
at best, as compatible as today's GEDCOM implementations.
Oops, is my cynicism showing?
--
Wes Groleau
Thinking It Through
http://Ideas.Lang-Learn.us/WWW?itemid=476
Richard Smith
Jan 2, 2012, 1:27:03 PM1/2/12
to
On 01/01/12 12:33, Tony Proctor wrote:
> A true URN identifies an object in a hierarchical namespace Richard. It
> therefore has a restricted form of the full URI syntax (as you've shown) and
> the 'schemes' have to authorised ones- I admit I didn't realise uuid was an
> A true URN identifies an object in a hierarchical namespace Richard. It
> therefore has a restricted form of the full URI syntax (as you've shown) and
> authorised one.
It's a relatively recent registration; RFC 4122 is the relevant standard.
In any case, UUID URNs have the advantage of not requiring a domain
name, and if we're concerned about long term survival of data, that may
be relevant. However, in general I prefer using http: URLs as URI, and
I think the problems to do with domain name expiry can easily be managed.
Suppose I upload some sort of resource to a website, say
http://mywebsite.example.com/Wills/John_Smith.png
I now notify an index server about this resource. The notification
probably contains some sort of information on the licensing conditions
of the resource, and some cryptographic stuff to determine who (is
claiming to have) produced the resource. And, of relevance to this
discussion, it also contains a hash (say SHA1) of the resource.
The index server periodically checks that this URL still accesses the
resource and that its hash has not changed. At some point it may become
aware of a duplicate copy at
http://willsgalore.example.com/24498.png
(The index server suspected it was a duplicate because its hash matched,
and then confirmed this by fetching copies to check they were bitwise
identical.)
Some time later, mywebsite.example.com expires or otherwise no longer
under my control, and the content hosted on it is no longer accessible.
The former address is now simply a URI. If I want to get the document
it points to, I can no longer simply dereference the URL with DNS and
HTTP. Instead I can ask the index server for an alternative URL for it
and use that.
There's a danger that when ownership of mywebsite.example.com changes,
instead of getting a '404 Not Found', it might start serving different
content; conceivably content that looks a bit like the a genealogical
resource, but not the one I'm expecting. (This is most likely if the
domain name is something valuable like 'family-tree.com'.) To avoid
this resulting in any problems, whenever I want to fetch the source
document, I should go to the index server and ask what its hash ought to
be and then verify that it is indeed that.
Finally, we say that once a URL has been used in adding a resource to
the Commons, it cannot ever be used again. So almost certainly I
wouldn't want to use a URL like:
http://mywebsite.example.com/Wills/John_Smith.png
Instead, I would be better using something that potentially allows for
versioning, such as:
http://mywebsite.example.com/Wills/John_Smith.png?version=1
This doesn't mean the server needs to understand versioning: if it
serves same content irrespective of the version=N query, that's fine,
because the index server will simply drop the server as valid place to
find the earlier versions.
--
Richard Smith
Wes Groleau
Jan 2, 2012, 1:30:39 PM1/2/12
to
On 01-02-2012 13:27, Richard Smith wrote:
> This doesn't mean the server needs to understand versioning: if it
> serves same content irrespective of the version=N query, that's fine,
> because the index server will simply drop the server as valid place to
> find the earlier versions.
The index server might have to drop that server as being a reliable
> This doesn't mean the server needs to understand versioning: if it
> serves same content irrespective of the version=N query, that's fine,
> because the index server will simply drop the server as valid place to
> find the earlier versions.
place to get the latest version. If it doesn't recognize the version
request, how can you determine whether the one returned is the latest?
Richard Smith
Jan 2, 2012, 1:35:12 PM1/2/12
to
On 02/01/12 18:30, Wes Groleau wrote:
> On 01-02-2012 13:27, Richard Smith wrote:
>> This doesn't mean the server needs to understand versioning: if it
>> serves same content irrespective of the version=N query, that's fine,
>> because the index server will simply drop the server as valid place to
>> find the earlier versions.
>
> The index server might have to drop that server as being a reliable
> place to get the latest version. If it doesn't recognize the version
> request, how can you determine whether the one returned is the latest?
Because the index server stores the hash it was sent when it was
> On 01-02-2012 13:27, Richard Smith wrote:
>> This doesn't mean the server needs to understand versioning: if it
>> serves same content irrespective of the version=N query, that's fine,
>> because the index server will simply drop the server as valid place to
>> find the earlier versions.
>
> The index server might have to drop that server as being a reliable
> place to get the latest version. If it doesn't recognize the version
> request, how can you determine whether the one returned is the latest?
initially notified of the resource. If the hash changes, the resource
must have done.
--
Richard Smith
Richard Smith
Jan 2, 2012, 2:36:56 PM1/2/12
to
Lesley Robertson wrote:
> A point that doesn't seem to have been dealt with thus far is quality
> control.
A fair observation, and the reason I haven't mentioned it is because I
> A point that doesn't seem to have been dealt with thus far is quality
> control.
envisage hardly any quality control. Obviously a very small amount is
needed to catch things like copyright infringement, pornographic
content, and what have you. But beyond that I don't see the need for
it. Instead of quality control we have two things that are much more
valuable. We have persistence, and we have reliable attribution.
By persistence, I mean that once something has entered the Commons it
cannot ordinarily be removed (the special cases noted above excepted),
and it can never be modified, either by the author or by anyone else.
If the author notices some egregious mistake in his/her research, he/she
can produce a new version of it, or issue a corrigenda, or something
else on those lines. But he/she cannot delete the old one. In this
respect it is like writing a book: once it has been published, you
cannot undo that.
Reliable attribution means that we always know who entered a given
resource into the Commons. We cannot completely prevent pseudonymous
contributions or people with multiple accounts (though I would be in
favour of taking whatever steps are feasible to reduce the likelihood of
this happening), but we can prevent (almost all) impersonation of other
Commons users. This means that if I find something apparently uploaded
by a researcher I trust, I can be sure it really was uploaded by them.
The persistence aspect means I can also be sure it has not subsequently
been altered by anyone else.
Obviously if I search around the Commons and locate a resource uploaded
by complete stranger I have no idea as to how competent a researcher
they are, or generally how trustworthy they are. (These are two
separate things. Trustworthy is about not lying about where information
came from: for example, stealing someone else's resource and passing it
off as your own. Competency is about drawing valid conclusions in their
genealogical research. It is quite possible, indeed, probably quite
common, to be highly trustworthy but not especially competent.)
So how do we determine trustworthiness and competency? I favour a
rating system a bit like the seller feedback system on E-bay. I haven't
thought through the details fully, but I would certainly modify E-bay's
system somewhat. For example, I would count the opinions of a
trustworthy person more highly than those of an untrustworthy person
when considering trustworthiness, and similarly for competency. This
would give you a general idea of how much to trust a resource uploaded
by a stranger.
> If you don't want the system to be dismissed as "good for clues
> but not to be trusted" (cf the IGI) than you have to have some sort of
> measure of quality - only fully sourced, only if based on primary
> resources, or something else.
don't routinely properly source your work, you're unlikely to end up
with a high 'competency' rating. Presumably authors with a low
'competency' score will often be 'dismissed as "good for clues but not
to be trusted"', as you phrase it.
> That then tends to imply some sort of moderation.
>
> Wikipedia seems to operate on an "if anyone can alter the entries,
> they’ll eventually become correct. Doesn't work - Despite the fact that
> I hold one famous Professor's death cert, copies of every known obit,
> and a telegram from his sister saying "my brother died this morning",
> every time I correct his death date on Wikipedia, the same person
> changes it back to what he fondly considers to be correct.
person's research, and he cannot edit yours. But suppose for the sake
of argument, this other person had contributed a biography of the
professor and that you were aware of the error in the death date.
Perhaps your first step would be to contact the author to see whether it
was simply a typo. He may well acknowledge it as a typo an issue a
corrigenda to that effect.
If the author sticks to his guns, or simply doesn't respond, you have
another course open to you. It will also be possible to create related
resources. These can be anything: a transcription of an image, an
summary of a document, an interpretation or contextual notes, or a
commentary or critique. In this case, you've found a piece of dodgy
research, so you can write a critique of it, reference whatever evidence
you have, and upload that.
Now whenever anyone looks at the original resource, your critique will
be listed in a section of related resources where any competent
researcher will surely read it and come to their own conclusions. The
author of the professor's biography may, of course, respond to your
critique, and if he does that response will be linked as a related
document to your critique.
--
Richard Smith
Richard Smith
Jan 2, 2012, 3:39:07 PM1/2/12
to
Tony Proctor wrote:
> A fail-point is in the choice of electronic data that is both meaningful and
> persistent. Unfortunately, many people have committed their data to
> proprietary databases and pushing one of those into this commons is assuming
> that the associated product will be there forever. Exporting to GEDCOM (even
> if the format were registered) won't help as it is currently incapable of
> fully& accurately representing everyone's data.
> A fail-point is in the choice of electronic data that is both meaningful and
> persistent. Unfortunately, many people have committed their data to
> proprietary databases and pushing one of those into this commons is assuming
> that the associated product will be there forever. Exporting to GEDCOM (even
> if the format were registered) won't help as it is currently incapable of
>
> At the time of writing, their is no universally accepted machine-readable
> data format that would represent the core aspects of our data (i.e. ignoring
> attachments), including lineage-based data, event-based data, evidence-based
> data, and full use of source citations in a standard fashion.
Obviously there are major problems with dated, obscure, proprietary
> At the time of writing, their is no universally accepted machine-readable
> data format that would represent the core aspects of our data (i.e. ignoring
> attachments), including lineage-based data, event-based data, evidence-based
> data, and full use of source citations in a standard fashion.
formats, and it *may* be desirable to limit the formats supported by the
Commons, perhaps to non-proprietary ones, or maybe to those for which
open source translation programs exist.
But equally, imagine you inherit a collection of Word Perfect files
containing a cousin's genealogical research, and let's assume you have
no means of reading the files. Isn't it better to upload them on the
off-chance that someone may be able to read them?
I envisage some people writing 'conversion bots', programs that search
out content in particular obscure formats, convert them into more
accessible formats, and upload the converted files. The converted file
is just another type of related document, and the original still exists.
There's no monopoly on doing this: anyone can create such a 'bot'.
That might result in two translations. If they're bitwise identical,
they'll be created as copies of the same file; if they're slightly
different, they'll coexist. One 'bot' may gain a better reputation [see
my other post about rating trustworthiness and competency] than another,
and presumably that one's translations will then be preferred.
Over time, the old version will probably be carried by fewer servers,
and eventually might only exist in a few deep storage locations: sites
like archive.org with vast capacities, but that are very slow to fetch
up their data.
--
Richard Smith
Wes Groleau
Jan 2, 2012, 4:49:40 PM1/2/12
to
On 01-02-2012 14:36, Richard Smith wrote:
> Now whenever anyone looks at the original resource, your critique will
> be listed in a section of related resources where any competent
> researcher will surely read it and come to their own conclusions. The
> author of the professor's biography may, of course, respond to your
> critique, and if he does that response will be linked as a related
> document to your critique.
What if my critique is utterly groundless (but plausible) and the
> Now whenever anyone looks at the original resource, your critique will
> be listed in a section of related resources where any competent
> researcher will surely read it and come to their own conclusions. The
> author of the professor's biography may, of course, respond to your
> critique, and if he does that response will be linked as a related
> document to your critique.
original author is deceased, or no longer paying attention, or ....
For example: http://findagrave.com will allow me to create a memorial,
and then say that the person's father is George Washington. That will
automatically cause George Washington's memorial to say that his son is
that person. The only way for that to go away is for someone to
persuade the findagrave admins to remove it. The creator of the G.W.
entry can't remove it, though if he/she notices it he/she could add text
saying that it is búshì (Chinese for "not so").
--
Wes Groleau
Don’t Keep Dying Languages Alive!
http://Ideas.Lang-Learn.us/russell?itemid=1436
Richard Smith
Jan 2, 2012, 4:57:21 PM1/2/12
to
Ian Goddard wrote of RDFa:
> Your original post proposed a commons of all types of digital data. ISTM
> that this scheme is restricted to text. Could it be applied to other
> formats - including formatted text such as WP documents, PDFs,
> spreadsheets & GEDCOMS?
RDFa only applies to XML (and probably also non-XML HTML). Obviously
plain text can easily be made into XML, as I did in that example:
effectively just slap an HTML <pre> tag around it all. Some other
formats are possible too. The .docx and .xlsx formats introduced by
Word 2007 and Excel 2007 are both just zipped XML, as are the formats
used in Open Office (and Star Office, Libre Office, and other variants).
In principle you could add RDFa attributes to these formats, though I
don't know whether doing so would result in compatibility problems. PDF
and most older word processor formats are not XML-based and so don't
support RDFa, nor does GEDCOM (except for the new XML versions).
But I'm really just giving RDFa as an example of one way in which people
might choose to encode such information. A piece of standalone RDF is
another way. In twenty years time, quite likely people will be using
some new technology that doesn't exist today, and I don't want to
prevent that from being used either. The key is that the format should
be convertible to RDF. Fortunately there is another technology (well,
set of conventions would be a better term for it) called GRDDL that
provides a way of extracting RDF from documents.
> There may be a case for embedding analysis objects in evidence objects.
> It makes for an economy of transmission. But the analysis objects have
> their own existence. When the transmitted XML is stored internally in
> some S/W package it's likely going to have to be picked apart so that
> the individual objects can be presented to the user. It might be
> difficult to ensure that it gets put back together correctly for export
> and if it isn't the software will fail the criterion of replication
> integrity.
I think this is another case where Commons shouldn't impose a particular
view. If people want to embed this analysis they should be able to use
it. If they want a standalone analysis document, they should be able to
do that. And if they want to write the analysis simply as an essay in
English (or Serbo-Croat or Quechua or whatever), that's fine too, though
obviously it then won't be accessible to machine reasoning in the
foreseeable future.
However I don't want to get too far into the details of this, as I think
we need to be careful to avoid biting off too big a task at this stage.
What is the minimum the Commons actually needs? I would suggest we
need a way of saying that certain place names, people's names, or dates
are mentioned in the resource is probably sufficient. So, for example,
we might say
http://example.com/Wills/1234.png mentions
a person called "John Smith",
a place called "Dunny-on-the-Wold", and
a date of "12 January 1620"
This isn't wholly satisfactory for a number of reasons. How do we know
which John Smith it is? In what capacity is John Smith mentioned? Is
the date in the Gregorian or Julian calendar, and which year ending
convention is used? Is that Dunny-on-the-Wold, Dunshire, England? Or
Dunny-on-the-World, Massachusetts? (A more careful author would
probably have appended some county- or state-level entity in the place
name text.)
This should be a point where extensions are allowed (and encouraged);
analysis documents may provide more detailed information, index servers
may support searching using those more advanced features, and maybe
Commons 2.0 will define more required behaviour here. We may want to
add a bit more to the initial Commons, but to avoid our initial Commons
being overly complex, we shouldn't require too much additional
understanding. Fortunately this is exactly the sort of extensibility
that RDF is excellent at.
The one extension beyond simple textual values that I would propose
mandating would be to allow the use of the XML Schema xsd:date type as a
machine readable version of the date. (That format has difficulties: it
doesn't distinguish unknown timezones from UTC. It assumes all dates
are in the Gregorian calendar. It doesn't cope with unknown
information, or "about" or "circa". It doesn't handle regnal years.
But that format is standard and widely implemented.)
--
Richard Smith
> Your original post proposed a commons of all types of digital data. ISTM
> that this scheme is restricted to text. Could it be applied to other
> formats - including formatted text such as WP documents, PDFs,
> spreadsheets & GEDCOMS?
plain text can easily be made into XML, as I did in that example:
effectively just slap an HTML <pre> tag around it all. Some other
formats are possible too. The .docx and .xlsx formats introduced by
Word 2007 and Excel 2007 are both just zipped XML, as are the formats
used in Open Office (and Star Office, Libre Office, and other variants).
In principle you could add RDFa attributes to these formats, though I
don't know whether doing so would result in compatibility problems. PDF
and most older word processor formats are not XML-based and so don't
support RDFa, nor does GEDCOM (except for the new XML versions).
But I'm really just giving RDFa as an example of one way in which people
might choose to encode such information. A piece of standalone RDF is
another way. In twenty years time, quite likely people will be using
some new technology that doesn't exist today, and I don't want to
prevent that from being used either. The key is that the format should
be convertible to RDF. Fortunately there is another technology (well,
set of conventions would be a better term for it) called GRDDL that
provides a way of extracting RDF from documents.
> There may be a case for embedding analysis objects in evidence objects.
> It makes for an economy of transmission. But the analysis objects have
> their own existence. When the transmitted XML is stored internally in
> some S/W package it's likely going to have to be picked apart so that
> the individual objects can be presented to the user. It might be
> difficult to ensure that it gets put back together correctly for export
> and if it isn't the software will fail the criterion of replication
> integrity.
view. If people want to embed this analysis they should be able to use
it. If they want a standalone analysis document, they should be able to
do that. And if they want to write the analysis simply as an essay in
English (or Serbo-Croat or Quechua or whatever), that's fine too, though
obviously it then won't be accessible to machine reasoning in the
foreseeable future.
However I don't want to get too far into the details of this, as I think
we need to be careful to avoid biting off too big a task at this stage.
What is the minimum the Commons actually needs? I would suggest we
need a way of saying that certain place names, people's names, or dates
are mentioned in the resource is probably sufficient. So, for example,
we might say
http://example.com/Wills/1234.png mentions
a person called "John Smith",
a place called "Dunny-on-the-Wold", and
a date of "12 January 1620"
This isn't wholly satisfactory for a number of reasons. How do we know
which John Smith it is? In what capacity is John Smith mentioned? Is
the date in the Gregorian or Julian calendar, and which year ending
convention is used? Is that Dunny-on-the-Wold, Dunshire, England? Or
Dunny-on-the-World, Massachusetts? (A more careful author would
probably have appended some county- or state-level entity in the place
name text.)
This should be a point where extensions are allowed (and encouraged);
analysis documents may provide more detailed information, index servers
may support searching using those more advanced features, and maybe
Commons 2.0 will define more required behaviour here. We may want to
add a bit more to the initial Commons, but to avoid our initial Commons
being overly complex, we shouldn't require too much additional
understanding. Fortunately this is exactly the sort of extensibility
that RDF is excellent at.
The one extension beyond simple textual values that I would propose
mandating would be to allow the use of the XML Schema xsd:date type as a
machine readable version of the date. (That format has difficulties: it
doesn't distinguish unknown timezones from UTC. It assumes all dates
are in the Gregorian calendar. It doesn't cope with unknown
information, or "about" or "circa". It doesn't handle regnal years.
But that format is standard and widely implemented.)
--
Richard Smith
Richard Smith
Jan 2, 2012, 5:09:55 PM1/2/12
to
On 02/01/12 21:49, Wes Groleau wrote:
> On 01-02-2012 14:36, Richard Smith wrote:
>> Now whenever anyone looks at the original resource, your critique will
>> be listed in a section of related resources where any competent
>> researcher will surely read it and come to their own conclusions. The
>> author of the professor's biography may, of course, respond to your
>> critique, and if he does that response will be linked as a related
>> document to your critique.
>
> What if my critique is utterly groundless (but plausible) and the
> original author is deceased, or no longer paying attention, or ....
Sooner or later someone will check some of the facts you rely on and
> On 01-02-2012 14:36, Richard Smith wrote:
>> Now whenever anyone looks at the original resource, your critique will
>> be listed in a section of related resources where any competent
>> researcher will surely read it and come to their own conclusions. The
>> author of the professor's biography may, of course, respond to your
>> critique, and if he does that response will be linked as a related
>> document to your critique.
>
> What if my critique is utterly groundless (but plausible) and the
> original author is deceased, or no longer paying attention, or ....
find they don't support your case. Or they may catch you doing
something similar to another document entirely. Either way, once you've
been caught, they'll alter their trustworthiness or competence rating of
you (depending whether they think you were being malicious or
incompetent). The contributor's ratings will be displayed in some
fashion along side the list of related documents, and the user may
choose to configure the interface they are using to hide contributions
from poorly-rated users. (Indeed, that may be the default.)
Even if you use a brand new account for each malicious contribution, the
default rating for a new user will probably be poor enough that a
researcher will be wary of the contribution, and perhaps look to see
what else you've contributed.
--
Richard Smith
Ian Goddard
Jan 2, 2012, 5:58:55 PM1/2/12
to
1. XML, schemas, validation.
2. Communication works by adherence to standards. Who'd buy a program
that didn't work?
3. Enforcement of the standard brand. If it doesn't comply, you can't
claim it does.
The problem with GEDCOM is that it was intended for a limited purpose &
stretched out of its original scope & AFAIA nobody has been leaned on
for using the name for stretched versions.
Wes Groleau
Jan 2, 2012, 6:10:54 PM1/2/12
to
On 01-02-2012 16:57, Richard Smith wrote:
>
> http://example.com/Wills/1234.png mentions
> a person called "John Smith",
> a place called "Dunny-on-the-Wold", and
> a date of "12 January 1620"
>
> This isn't wholly satisfactory for a number of reasons. How do we know
> which John Smith it is? In what capacity is John Smith mentioned? Is
> the date in the Gregorian or Julian calendar, and which year ending
> convention is used? Is that Dunny-on-the-Wold, Dunshire, England? Or
> Dunny-on-the-World, Massachusetts?
There is admittedly all those uncertainties. However, you are not
>
> http://example.com/Wills/1234.png mentions
> a person called "John Smith",
> a place called "Dunny-on-the-Wold", and
> a date of "12 January 1620"
>
> This isn't wholly satisfactory for a number of reasons. How do we know
> which John Smith it is? In what capacity is John Smith mentioned? Is
> the date in the Gregorian or Julian calendar, and which year ending
> convention is used? Is that Dunny-on-the-Wold, Dunshire, England? Or
> Dunny-on-the-World, Massachusetts?
likely to have that document cluttering your results if you are looking
for Stasz Słomiany, Żołynia¹, 5 March 1873
¹Pow. Łańcucki, Podkarpackie, Poland
--
Wes Groleau
Film Review: The Blue Butterfly
http://Ideas.Lang-Learn.us/russell?itemid=1565
Wes Groleau
Jan 2, 2012, 6:14:02 PM1/2/12
to
On 01-02-2012 17:58, Ian Goddard wrote:
> 2. Communication works by adherence to standards. Who'd buy a program
> that didn't work?
Define "doesn't work." Aren't people still buying semi-compatible
> 2. Communication works by adherence to standards. Who'd buy a program
> that didn't work?
genealogy programs?
--
Wes Groleau
Obama Changing “Latin” Policies
http://Ideas.Lang-Learn.us/russell?itemid=1515
Ian Goddard
Jan 2, 2012, 6:46:53 PM1/2/12
to
Wes Groleau wrote:
> On 01-02-2012 17:58, Ian Goddard wrote:
>> 2. Communication works by adherence to standards. Who'd buy a program
>> that didn't work?
>
> Define "doesn't work." Aren't people still buying semi-compatible
> genealogy programs?
>
Fails to import good files.
> On 01-02-2012 17:58, Ian Goddard wrote:
>> 2. Communication works by adherence to standards. Who'd buy a program
>> that didn't work?
>
> Define "doesn't work." Aren't people still buying semi-compatible
> genealogy programs?
>
Attempts to upload files rejected.
Sits in a little island only able to communicate with other victims.
Richard Smith
Jan 2, 2012, 7:07:01 PM1/2/12
to
On 02/01/12 17:00, Ian Goddard wrote:
> Last Saturday the Times published an article by Tim Berners-Lee & Nigel
> Shadbolt, co-directors of HMG's projected Open Data Institute about
> opening up government data with the general approach that we've already
> paid for it so it should be free.
[long snip]
> Last Saturday the Times published an article by Tim Berners-Lee & Nigel
> Shadbolt, co-directors of HMG's projected Open Data Institute about
> opening up government data with the general approach that we've already
> paid for it so it should be free.
> Now, if the underpinning was in place is there anything in this scenario
> which would be beyond the average genealogical S/W user?
had in mind, and I agree with everything Ian has written here.
(And if my earlier comments about not wanting to initially require too
much of the person-level logic Ian discusses were taken as meaning I
don't want it all, let me correct that impression. I certainly do want
to see that sort of stuff implemented, but pragmatically I thought it
better to focus on the lower-level document stuff first.)
--
Richard Smith
Ian Goddard
Jan 2, 2012, 7:25:36 PM1/2/12
to
They're the basis of everything. And getting the source repositories to
make stuff available like this would kick-start the whole process.
At least as far as UK is concerned something like the ODI might just do
it. But I'm not holding my breath; they'll have to do some strong-arm
stuff on Whitehall to succeed.
Wes Groleau
Jan 2, 2012, 9:07:53 PM1/2/12
to
On 01-02-2012 18:46, Ian Goddard wrote:
> Wes Groleau wrote:
>> On 01-02-2012 17:58, Ian Goddard wrote:
>>> 2. Communication works by adherence to standards. Who'd buy a program
>>> that didn't work?
>>
>> Define "doesn't work." Aren't people still buying semi-compatible
>> genealogy programs?
>
> Fails to import good files.
Check
> Wes Groleau wrote:
>> On 01-02-2012 17:58, Ian Goddard wrote:
>>> 2. Communication works by adherence to standards. Who'd buy a program
>>> that didn't work?
>>
>> Define "doesn't work." Aren't people still buying semi-compatible
>> genealogy programs?
>
> Fails to import good files.
> Attempts to upload files rejected.
> Sits in a little island only able to communicate with other victims.
Yep, people are still buying them. Unless they have stopped very recently!
--
Wes Groleau
Free speech has its limits
http://Ideas.Lang-Learn.us/WWW?itemid=99
singhals
Jan 2, 2012, 9:35:44 PM1/2/12
to gen...@rootsweb.com
Ian Goddard wrote:
> Wes Groleau wrote:
>> On 01-02-2012 17:58, Ian Goddard wrote:
>>> 2. Communication works by adherence to standards. Who'd buy a program
>>> that didn't work?
>>
>> Define "doesn't work." Aren't people still buying semi-compatible
>> genealogy programs?
>>
>
> Fails to import good files.
An argument /could/ be made that if Program Wonderful fails
> Wes Groleau wrote:
>> On 01-02-2012 17:58, Ian Goddard wrote:
>>> 2. Communication works by adherence to standards. Who'd buy a program
>>> that didn't work?
>>
>> Define "doesn't work." Aren't people still buying semi-compatible
>> genealogy programs?
>>
>
> Fails to import good files.
to import a certain file, then Program Wonderful doesn't
consider that a "good file."
>
> Attempts to upload files rejected.
to upload stuff, I've no clue which program(s) this references.
>
> Sits in a little island only able to communicate with other victims.
There's the sort of communicate a 3-wk old baby manages, the
sort I manage with my non-English-speaking relatives, the
sort between adult speakers of the same language, and the
sort between siblings. Given what a poor Virginia gal can
get overseas when she orders up a biscuit, I'm not sure
you're in a position to stow thrones on this island.
Cheryl
Tony Proctor
Jan 3, 2012, 7:28:45 AM1/3/12
to
"Richard Smith" <ric...@ex-parrot.com> wrote in message
news:9meni1...@mid.individual.net...
If each contribution had some sort of manifest file that identified all the
names to be indexed, the relations between them, and the nature of all the
associated attachments (images etc) then I could see how it might work.
Personally, I would also want to see proper source citations too, though,
otherwise I wouldn't know how much credence to give to something a Search
throws up.
However, that's gradually moving towards the age-old issue of a universal
description for family-history data. It just doesn't exist. If it did then
the genealogy world would be a much better place and there would be so many
new things we could do. :-)
Tony Proctor
Richard Smith
Jan 3, 2012, 8:38:58 AM1/3/12
to
Tony Proctor wrote:
> It still sounds like a dumping ground for unstructured data to me Richard.
> If each contribution had some sort of manifest file that identified all the
> names to be indexed, the relations between them, and the nature of all the
> associated attachments (images etc) then I could see how it might work.
I think we're confusing two separate things. First there is the data
> It still sounds like a dumping ground for unstructured data to me Richard.
> If each contribution had some sort of manifest file that identified all the
> names to be indexed, the relations between them, and the nature of all the
> associated attachments (images etc) then I could see how it might work.
that storage node stores. This can be as broad and as structured as you
want. Then there is the data that the index server *must* understand
and store locally. It is the latter that I am saying should small and
probably fairly unstructured. An index server *may* understand more
metadata, and may allow you to search on that, but it is not required
to. If we require the index server to store too much content, it'll
make it hard to implement.
Let me give an example. I upload a scan of a will. And I produce an
interpretation document listing all of the people mentioned in it, their
roles in the will (testator, witness, executor, beneficiary, and so on),
how they are related (when this is stated), their occupations and where
they live (again, where stated), the probate jurisdiction, the dates of
the will and of any codicils, the record office where the will is
located, and so on.
The index server reads this interpretation document. It may store all
of this information, and if so, it may allow highly structured queries
such as "find all wills proved in the Consistory Court of Ely which were
witnessed by a woman whose surname was Jones". The search API will make
it possible to ask such questions (using the SPARQL query language), but
requiring all index servers to be answer such questions seems a step too
far, at least at first.
Instead we say that the index server *must* at least store the people's
names, place names, and dates from the interpretation document. So
having tried and failed to get results from the very structured question
above, I can then fall back to "find all wills mentioning the place Ely
and the name Jones". Obviously this is less good, but it's better than
nothing.
We can argue about exactly what we require the index servers to store,
and maybe what I've proposed isn't quite right, but the key point is not
to require too much. Too much will make it hard to implement and result
in the index consuming too much storage space. We can always add
additional storage requirements at a later stage. And we may want to
provide a means for index servers to list their storage policy so that a
user can make an informed choice about which index to use when.
> Personally, I would also want to see proper source citations too, though,
> otherwise I wouldn't know how much credence to give to something a Search
> throws up.
to know about.
--
Richard Smith
tms
Jan 3, 2012, 1:36:56 PM1/3/12
to
On Monday, January 2, 2012 3:39:07 PM UTC-5, Richard Smith wrote:
>
> Obviously there are major problems with dated, obscure, proprietary
> formats, and it *may* be desirable to limit the formats supported by the
> Commons, perhaps to non-proprietary ones, or maybe to those for which
> open source translation programs exist.
I would think that the use of open formats would be an absolute requirement, if only so the data could still be read 50 years from now. As I see it, the requirements for formats for such a project are:
>
> Obviously there are major problems with dated, obscure, proprietary
> formats, and it *may* be desirable to limit the formats supported by the
> Commons, perhaps to non-proprietary ones, or maybe to those for which
> open source translation programs exist.
* Open --- There must be no legal or financial obstacles for using the format, and the specification must be freely available.
* Stable --- Preferably there would be no further changes in the format for the foreseeable future. Both forward and backward compatibility are necessary.
* Open implementations --- Not just to reduce costs for users, but also so that developers can reuse existing code in new implementations for new platforms or languages.
* Widely available implementations --- This would make it more likely that the implementations are portable enough to work on platforms of the future. Preferably the implementations would be in multiple widely-used languages, or at least in one very-widely-used language (such as C). Any format that exists in only one implementation in one language that works only on one platform must be rejected.
--
T.M. Sommers
tms
Jan 3, 2012, 1:58:00 PM1/3/12
to Grolea...@freeshell.org
On Sunday, January 1, 2012 1:45:23 PM UTC-5, Wes Groleau wrote:
> On 01-01-2012 10:27, Lesley Robertson wrote:
> > If you don't want the system to be dismissed as "good for clues but not
> > to be trusted" (cf the IGI)
>
> And cf ancestral file and ancestry.com and familysearch.org and
> worldconnect.rootsweb.com and .....
... and your local library. Just because something is in a book does not mean that it is necessarily correct. One must always use one's judgment when using any source.
> On 01-01-2012 10:27, Lesley Robertson wrote:
> > If you don't want the system to be dismissed as "good for clues but not
> > to be trusted" (cf the IGI)
>
> And cf ancestral file and ancestry.com and familysearch.org and
> worldconnect.rootsweb.com and .....
--
T.M. Sommers
Wes Groleau
Jan 3, 2012, 9:37:36 PM1/3/12
to
On 01-03-2012 13:36, tms wrote:
> at least in one very-widely-used language (such as C).
The most error-prone widely-used language in existence.
> at least in one very-widely-used language (such as C).
--
Wes Groleau
Items of dialect heard …
http://Ideas.Lang-Learn.us/barrett?itemid=1522
Wes Groleau
Jan 3, 2012, 9:42:01 PM1/3/12
to
of producing it meant that it happened less often (and was more often
intentional).
Current technology makes it super easy to reproduce garbage and people
do just that.
Richard Smith
Jan 3, 2012, 9:47:01 PM1/3/12
to
I thought I'd mock up a simple example of how the details of Commons
upload process might work. But first a warning: this email is quite
long on technical detail and short on genealogical content. Obviously
in any real application these details would be happening behind the scenes.
First off, I need a GPG key. I can use an existing one if I have one,
but I imagine most applications will choose to create a new key for user
for this purpose. This key needs to uploaded to an agreed keyserver, or
maybe several. For the purpose of this example, I have created myself a
new key with ID 78B6EA81 and have uploaded it to the keys.gnupg.net
keyserver.
Next I need to create and publish short bit of RDF/XML with a few bits
of housekeeping information about me. The contents of this are
completely extensible. For example, I might choose to link to home page
or a page of my genealogical research interests. Or if I felt so
inclined, I could put in something totally irrelevant like my favourite
single malt. We would probably wish to require a few details to be
given, and I would suggest the researcher's real name and email address
are a sensible minimum. The researcher (or his software) then publishes
this RDF/XML somewhere on the web. For my example, I have created a
minimal example which I have uploaded here:
http://commons.genmine.com/Users/Richard_Smith.foaf
Everything in this document is written using established technologies
(mostly FOAF) that are frequently used together in this manner, and so
plenty of tools and libraries exist for manipulating such files. The
document contains five pieces of information. Two are about me: my
name, and the SHA1 hash of my email address. I could have written my
actual email address (using the foaf:mbox attribute), but to avoid too
many spambots harvesting my address, I've only given a hash of it.
The other three pieces of information are about the document itself. I
state that I wrote the document and that the document (trivial though it
is) is licensed under the CC-ND licence (which says people can copy it
but not change it). And I link to a GPG signature of the file which
verifies that I (or at least someone with access to the GPG key
purporting to be mine) really produced the document. The GPG signature
was trivially generated with the following command (where 78B6EA81 is
the ID of my key):
gpg -sbau '78B6EA81' Richard_Smith.foaf
[The next three paragraphs are on arcane details of URI usage in RDF.
Feel free to jump past them to the *** marker.]
The rdf:about and rdf:resource attributes in this are actually relative
URIs, so rdf:about="" means a statement about the current document.
A subtly with RDF is that it is conventional to use http: URIs as both
URLs and URNs -- that is both as a way of referencing an additional
resource, and as a way of naming abstract or physical concepts. In this
example,
http://commons.genmine.com/Users/Richard_Smith
... is a URN that identifiers me, in my capacity as a Commons user, and
so in this piece of RDF/XML, rdf:about="Richard_Smith" means a statement
about me, the user. A further subtlety arises from the fact that it is
conventional to simultaneously be able to use this URI as a a URN to
represent me, and also as a URL to locate information about me. A first
that probably sounds horribly confused, but in practice it does work
well, *provided* that accessing the URN results in a redirect (typically
a 303 "See Other"). In this case, I have configured the server to do:
GET /Users/Richard_Smith HTTP/1.1
Host: commons.genmine.com
HTTP/1.1 303 See Other
Location: http://commons.genmine.com/Users/Richard_Smith.foaf
This redirection is necessary so that in Richard_Smith.foaf references
to the current document URI (by means of the empty URI reference, e.g.
in rdf:about="") resolve to the .foaf URL representing the document,
rather than the unsuffixed URI representing me. This is the standard
way of doing things, and it really isn't as ghastly as it sounds once
you've got used to it.
[*** End of discussion on URIs in RDF.]
I, or my software, would then hand pass this URL (either, it doesn't
matter) to an index server that will verify the GPG signature. You can
check this yourself by downloaing both the FOAF file and the GPG
signature and then running:
gpg --verify --keyserver-options auto-key-retrieve
--keyserver keys.gnupg.net Richard_Smith.foaf.asc
... which will fetch my GPG key from the keyserver. This command will
also tell you that the signature was created by
Richard Smith (Genealogical Commons) <ric...@ex-parrot.com>
The bit in parentheses is a comment and can be ignored. The first and
last bits are my name and email address as stored in the GPG key. The
index server should check that the name is the same as the one in the
FOAF, and that when you take the SHA1 hash of
mailto:ric...@ex-parrot.com
you get the value quoted in the FOAF. (This means that machines in the
Commons can decipher your email address, but it'll should be a long time
before spambots evolve to the stage where they can addresses this way.)
There are various standard ways we can increase a confidence that the
name and email address in the key are correct -- e.g. with the standard
web of trust ideas, or with an OpenID. I don't propose to go into
details of these now, but we may wish to require additional steps to
allow us to be surer of the users' real identity. Once we're confident
the key is really owned by whoever it claims to be owned by, we can be
pretty sure that the FOAF document and any other documents signed by
that key really written by written by that user.
I may have taken a dozen or so paragraphs to describe this initial set
up, but the whole process can be hidden from the user and done in a
matter of seconds. Anyway, I'm finally ready to do some genealogy, and
I decide to start by uploading a scan of my great great grandparents'
marriage certificate:
http://commons.genmine.com/Test/Thomas_Smith_marriage.png
There's absolutely nothing special about the way I've uploaded that, and
it could be on any website, anywhere. Perhaps I didn't upload it myself
but discovered that a cousin had already uploaded it to his website.
That's fine too.
If I want to add that to the commons, the first question I should be
asking myself is: is it copyrighted? And do I have permission to copy
it into the commons. Marriage certificates are usually Crown Copyright,
and are covered by the Open Government Licence which allows people to
copy them (with a few restrictions).
http://www.nationalarchives.gov.uk/doc/open-government-licence/
(In this case, the certificate is the original one that has been in my
family for 140 years. It may be that Crown Copyright has therefore
elapsed, but for the sake of argument let's assume it still holds.)
Happy that I'm allowed to upload, it I produce a piece of RDF/XML with
some basic metadata about the scanned certificate, which I upload here:
http://commons.genmine.com/Test/Thomas_Smith_marriage.rdf
(There's no requirement for it to be in the same directory or even on
the same server as the image.) This has a number of similarities to the
earlier FOAF file I produced about me. As with the FOAF file, I state
that I (or the software I'm using) wrote the RDF and I link to a GPG
signature. When I state that I wrote the FOAF, I do it by way of the URI
http://commons.genmine.com/Users/Richard_Smith
which we noted above represents me, in my capacity as a user. When the
index server comes to read the RDF/XML, if it hasn't yet encountered me
(e.g. if I've just started using a new index server), it can use that as
a URL to fetch the FOAF file and then run through the user checks given
above. Obviously the index server will also verify the GPG signature of
the marriage certificate RDF/XML.
The interesting part of the RDF is <commons:GenealogicalResource>
element, which is a custom class introduced for this project. This is
saying that the certificate PNG, referenced in the rdf:about attributed,
is a genealogical resource of some sort. It's my current thought that
this term should be interpreted in the widest of ways. Clearly a scan
of a mid-19th century marriage certificate is a genealogical resource,
but so too is a piece of explanatory text, or a list of parishes in a
given county. Even a hoax genealogy might constitute a valid
genealogical resource if the hoax has propagated widely, simply so that
it can be included in the Commons are carefully noted as a hoax.
We also state the MIME type of the resource (image/png in this case) and
its SHA1 hash. The hash is so that we know whether or not the URL in
the rdf:about attribute is still pointing to the resource the user had
in mind.
Finally, the marriage certificate RDF contains two <xhtml:licence>
elements, one referring to the the RDF document, and one referring to
the image. (The earlier FOAF file contained one too.) This states that
the RDF itself is licensed under the CC-ND licence, meaning you can copy
but not modify it; and that the PNG is licensed under the Open
Government Licence. The index server needs to verify that the licence
is compatible with the Commons: in particular, that we are allowed to
copy the resource. This is done with a technology called CC-REL:
http://wiki.creativecommons.org/Ccrel
This allows the index server to look up the rights granted and
prohibitions imposed by the licence, and the licence will only be
accepted if the rights are sufficient and the prohibitions are all
understood and considered acceptable. (In some cases, a licence will
not support CC-REL and the index server will have to know about the
licence. But many of the common open source licences do support it,
including the Creative Commons licences, the GNU ones, and the UK Open
Government Licence.) Whenever anything is passed to the index server,
whether it is a piece of FOAF or RDF, or a genealogical resource, it
must always have a licence that permits at least copying, or it will be
rejected.
Once the index server has determined that the RDF/XML is well-formed, is
correctly signed by a verified user, and has suitable licensing
information, it will download the image, to verify that it does yield a
resource with the specified MIME type and SHA1 hash. This check is
important because it verifies that the resource is accessible without
logging in. It also checks that the resource has not changed since the
version the user intended to upload to the Commons.
The index server will store the date and time that it accepted the
upload, and the <foaf:maker> element of the RDF/XML, which references
the user who created the RDF/XML is now taken to be the user who
uploaded the contents to the Commons. In other words, by GPG signing an
<commons:GenealogicalResource> element with a key registered to a
Commons user, the user is effectively consenting to it being uploaded to
the commons, even if they themselves don't pass the RDF to the index server.
More generally, by signing a statement you are guaranteeing its truth.
This is the reason why I did not sign the marriage certificate, but
rather put a SHA1 hash into a document that I signed. Had I GPG signed
the marriage certificate, I would have been guaranteeing its contents to
be true. I don't want to do that because I'm pretty sure the groom was
actually 24, not 22, and the bride was only 19 rather than 20. Even if
I didn't know of any errors, I wouldn't wish to make a guarantee about
something so long ago.
Obviously, uploading is just one small part of the process. I've not
discussed how the index servers deal with them once they're uploaded.
I've not discussed searching or indexing. Or replicating resources
across servers. I've not discussed genealogical metadata (such as
providing the names of the parties to the marriage), or related
documents (such as transcripts). These are all topics for another day.
Complex thought this upload process may sound, the whole thing is built
from existing technologies, and implementing this process should be
quite a quick job. (And indeed, I implemented some parts of it while
writing this post in order to test them.)
--
Richard Smith
upload process might work. But first a warning: this email is quite
long on technical detail and short on genealogical content. Obviously
in any real application these details would be happening behind the scenes.
First off, I need a GPG key. I can use an existing one if I have one,
but I imagine most applications will choose to create a new key for user
for this purpose. This key needs to uploaded to an agreed keyserver, or
maybe several. For the purpose of this example, I have created myself a
new key with ID 78B6EA81 and have uploaded it to the keys.gnupg.net
keyserver.
Next I need to create and publish short bit of RDF/XML with a few bits
of housekeeping information about me. The contents of this are
completely extensible. For example, I might choose to link to home page
or a page of my genealogical research interests. Or if I felt so
inclined, I could put in something totally irrelevant like my favourite
single malt. We would probably wish to require a few details to be
given, and I would suggest the researcher's real name and email address
are a sensible minimum. The researcher (or his software) then publishes
this RDF/XML somewhere on the web. For my example, I have created a
minimal example which I have uploaded here:
http://commons.genmine.com/Users/Richard_Smith.foaf
Everything in this document is written using established technologies
(mostly FOAF) that are frequently used together in this manner, and so
plenty of tools and libraries exist for manipulating such files. The
document contains five pieces of information. Two are about me: my
name, and the SHA1 hash of my email address. I could have written my
actual email address (using the foaf:mbox attribute), but to avoid too
many spambots harvesting my address, I've only given a hash of it.
The other three pieces of information are about the document itself. I
state that I wrote the document and that the document (trivial though it
is) is licensed under the CC-ND licence (which says people can copy it
but not change it). And I link to a GPG signature of the file which
verifies that I (or at least someone with access to the GPG key
purporting to be mine) really produced the document. The GPG signature
was trivially generated with the following command (where 78B6EA81 is
the ID of my key):
gpg -sbau '78B6EA81' Richard_Smith.foaf
[The next three paragraphs are on arcane details of URI usage in RDF.
Feel free to jump past them to the *** marker.]
The rdf:about and rdf:resource attributes in this are actually relative
URIs, so rdf:about="" means a statement about the current document.
A subtly with RDF is that it is conventional to use http: URIs as both
URLs and URNs -- that is both as a way of referencing an additional
resource, and as a way of naming abstract or physical concepts. In this
example,
http://commons.genmine.com/Users/Richard_Smith
... is a URN that identifiers me, in my capacity as a Commons user, and
so in this piece of RDF/XML, rdf:about="Richard_Smith" means a statement
about me, the user. A further subtlety arises from the fact that it is
conventional to simultaneously be able to use this URI as a a URN to
represent me, and also as a URL to locate information about me. A first
that probably sounds horribly confused, but in practice it does work
well, *provided* that accessing the URN results in a redirect (typically
a 303 "See Other"). In this case, I have configured the server to do:
GET /Users/Richard_Smith HTTP/1.1
Host: commons.genmine.com
HTTP/1.1 303 See Other
Location: http://commons.genmine.com/Users/Richard_Smith.foaf
This redirection is necessary so that in Richard_Smith.foaf references
to the current document URI (by means of the empty URI reference, e.g.
in rdf:about="") resolve to the .foaf URL representing the document,
rather than the unsuffixed URI representing me. This is the standard
way of doing things, and it really isn't as ghastly as it sounds once
you've got used to it.
[*** End of discussion on URIs in RDF.]
I, or my software, would then hand pass this URL (either, it doesn't
matter) to an index server that will verify the GPG signature. You can
check this yourself by downloaing both the FOAF file and the GPG
signature and then running:
gpg --verify --keyserver-options auto-key-retrieve
--keyserver keys.gnupg.net Richard_Smith.foaf.asc
... which will fetch my GPG key from the keyserver. This command will
also tell you that the signature was created by
Richard Smith (Genealogical Commons) <ric...@ex-parrot.com>
The bit in parentheses is a comment and can be ignored. The first and
last bits are my name and email address as stored in the GPG key. The
index server should check that the name is the same as the one in the
FOAF, and that when you take the SHA1 hash of
mailto:ric...@ex-parrot.com
you get the value quoted in the FOAF. (This means that machines in the
Commons can decipher your email address, but it'll should be a long time
before spambots evolve to the stage where they can addresses this way.)
There are various standard ways we can increase a confidence that the
name and email address in the key are correct -- e.g. with the standard
web of trust ideas, or with an OpenID. I don't propose to go into
details of these now, but we may wish to require additional steps to
allow us to be surer of the users' real identity. Once we're confident
the key is really owned by whoever it claims to be owned by, we can be
pretty sure that the FOAF document and any other documents signed by
that key really written by written by that user.
I may have taken a dozen or so paragraphs to describe this initial set
up, but the whole process can be hidden from the user and done in a
matter of seconds. Anyway, I'm finally ready to do some genealogy, and
I decide to start by uploading a scan of my great great grandparents'
marriage certificate:
http://commons.genmine.com/Test/Thomas_Smith_marriage.png
There's absolutely nothing special about the way I've uploaded that, and
it could be on any website, anywhere. Perhaps I didn't upload it myself
but discovered that a cousin had already uploaded it to his website.
That's fine too.
If I want to add that to the commons, the first question I should be
asking myself is: is it copyrighted? And do I have permission to copy
it into the commons. Marriage certificates are usually Crown Copyright,
and are covered by the Open Government Licence which allows people to
copy them (with a few restrictions).
http://www.nationalarchives.gov.uk/doc/open-government-licence/
(In this case, the certificate is the original one that has been in my
family for 140 years. It may be that Crown Copyright has therefore
elapsed, but for the sake of argument let's assume it still holds.)
Happy that I'm allowed to upload, it I produce a piece of RDF/XML with
some basic metadata about the scanned certificate, which I upload here:
http://commons.genmine.com/Test/Thomas_Smith_marriage.rdf
(There's no requirement for it to be in the same directory or even on
the same server as the image.) This has a number of similarities to the
earlier FOAF file I produced about me. As with the FOAF file, I state
that I (or the software I'm using) wrote the RDF and I link to a GPG
signature. When I state that I wrote the FOAF, I do it by way of the URI
http://commons.genmine.com/Users/Richard_Smith
which we noted above represents me, in my capacity as a user. When the
index server comes to read the RDF/XML, if it hasn't yet encountered me
(e.g. if I've just started using a new index server), it can use that as
a URL to fetch the FOAF file and then run through the user checks given
above. Obviously the index server will also verify the GPG signature of
the marriage certificate RDF/XML.
The interesting part of the RDF is <commons:GenealogicalResource>
element, which is a custom class introduced for this project. This is
saying that the certificate PNG, referenced in the rdf:about attributed,
is a genealogical resource of some sort. It's my current thought that
this term should be interpreted in the widest of ways. Clearly a scan
of a mid-19th century marriage certificate is a genealogical resource,
but so too is a piece of explanatory text, or a list of parishes in a
given county. Even a hoax genealogy might constitute a valid
genealogical resource if the hoax has propagated widely, simply so that
it can be included in the Commons are carefully noted as a hoax.
We also state the MIME type of the resource (image/png in this case) and
its SHA1 hash. The hash is so that we know whether or not the URL in
the rdf:about attribute is still pointing to the resource the user had
in mind.
Finally, the marriage certificate RDF contains two <xhtml:licence>
elements, one referring to the the RDF document, and one referring to
the image. (The earlier FOAF file contained one too.) This states that
the RDF itself is licensed under the CC-ND licence, meaning you can copy
but not modify it; and that the PNG is licensed under the Open
Government Licence. The index server needs to verify that the licence
is compatible with the Commons: in particular, that we are allowed to
copy the resource. This is done with a technology called CC-REL:
http://wiki.creativecommons.org/Ccrel
This allows the index server to look up the rights granted and
prohibitions imposed by the licence, and the licence will only be
accepted if the rights are sufficient and the prohibitions are all
understood and considered acceptable. (In some cases, a licence will
not support CC-REL and the index server will have to know about the
licence. But many of the common open source licences do support it,
including the Creative Commons licences, the GNU ones, and the UK Open
Government Licence.) Whenever anything is passed to the index server,
whether it is a piece of FOAF or RDF, or a genealogical resource, it
must always have a licence that permits at least copying, or it will be
rejected.
Once the index server has determined that the RDF/XML is well-formed, is
correctly signed by a verified user, and has suitable licensing
information, it will download the image, to verify that it does yield a
resource with the specified MIME type and SHA1 hash. This check is
important because it verifies that the resource is accessible without
logging in. It also checks that the resource has not changed since the
version the user intended to upload to the Commons.
The index server will store the date and time that it accepted the
upload, and the <foaf:maker> element of the RDF/XML, which references
the user who created the RDF/XML is now taken to be the user who
uploaded the contents to the Commons. In other words, by GPG signing an
<commons:GenealogicalResource> element with a key registered to a
Commons user, the user is effectively consenting to it being uploaded to
the commons, even if they themselves don't pass the RDF to the index server.
More generally, by signing a statement you are guaranteeing its truth.
This is the reason why I did not sign the marriage certificate, but
rather put a SHA1 hash into a document that I signed. Had I GPG signed
the marriage certificate, I would have been guaranteeing its contents to
be true. I don't want to do that because I'm pretty sure the groom was
actually 24, not 22, and the bride was only 19 rather than 20. Even if
I didn't know of any errors, I wouldn't wish to make a guarantee about
something so long ago.
Obviously, uploading is just one small part of the process. I've not
discussed how the index servers deal with them once they're uploaded.
I've not discussed searching or indexing. Or replicating resources
across servers. I've not discussed genealogical metadata (such as
providing the names of the parties to the marriage), or related
documents (such as transcripts). These are all topics for another day.
Complex thought this upload process may sound, the whole thing is built
from existing technologies, and implementing this process should be
quite a quick job. (And indeed, I implemented some parts of it while
writing this post in order to test them.)
--
Richard Smith
singhals
Jan 3, 2012, 10:41:46 PM1/3/12
to gen...@rootsweb.com
Richard Smith wrote:
> I thought I'd mock up a simple example of how the details of Commons
> upload process might work. But first a warning: this email is quite
> long on technical detail and short on genealogical content. Obviously
> in any real application these details would be happening behind the scenes.
>
> First off, I need a GPG key. I can use an existing one if I have one,
> but I imagine most applications will choose to create a new key for user
> for this purpose. This key needs to uploaded to an agreed keyserver, or
> maybe several. For the purpose of this example, I have created myself a
> new key with ID 78B6EA81 and have uploaded it to the keys.gnupg.net
> keyserver.
>
WHO is the "I" who needs to do this? The DBA, the
> I thought I'd mock up a simple example of how the details of Commons
> upload process might work. But first a warning: this email is quite
> long on technical detail and short on genealogical content. Obviously
> in any real application these details would be happening behind the scenes.
>
> First off, I need a GPG key. I can use an existing one if I have one,
> but I imagine most applications will choose to create a new key for user
> for this purpose. This key needs to uploaded to an agreed keyserver, or
> maybe several. For the purpose of this example, I have created myself a
> new key with ID 78B6EA81 and have uploaded it to the keys.gnupg.net
> keyserver.
>
programmer, or the person doing the uploading?
> Next I need to create and publish short bit of RDF/XML with a few bits
> of housekeeping information about me. The contents of this are
Cheryl
Richard Smith
Jan 4, 2012, 4:45:40 AM1/4/12
to
On 04/01/12 03:41, singhals wrote:
> Richard Smith wrote:
>> I thought I'd mock up a simple example of how the details of Commons
>> upload process might work. But first a warning: this email is quite
>> long on technical detail and short on genealogical content. Obviously
>> in any real application these details would be happening behind the
>> scenes.
>>
>> First off, I need a GPG key. I can use an existing one if I have one,
>> but I imagine most applications will choose to create a new key for user
>> for this purpose. This key needs to uploaded to an agreed keyserver, or
>> maybe several. For the purpose of this example, I have created myself a
>> new key with ID 78B6EA81 and have uploaded it to the keys.gnupg.net
>> keyserver.
>>
>
> WHO is the "I" who needs to do this? The DBA, the programmer, or the
> person doing the uploading?
This is something that the person uploading needs to do (or have done
> Richard Smith wrote:
>> I thought I'd mock up a simple example of how the details of Commons
>> upload process might work. But first a warning: this email is quite
>> long on technical detail and short on genealogical content. Obviously
>> in any real application these details would be happening behind the
>> scenes.
>>
>> First off, I need a GPG key. I can use an existing one if I have one,
>> but I imagine most applications will choose to create a new key for user
>> for this purpose. This key needs to uploaded to an agreed keyserver, or
>> maybe several. For the purpose of this example, I have created myself a
>> new key with ID 78B6EA81 and have uploaded it to the keys.gnupg.net
>> keyserver.
>>
>
> WHO is the "I" who needs to do this? The DBA, the programmer, or the
> person doing the uploading?
for them) before they upload anything. Now obviously most such people
won't have any idea how to do this, and that's fine: in practice it will
be done behind the scenes by their software the first time it uploads
anything to the Commons, or perhaps when the software is installed.
Nor does it need to be the primary genealogical program. It could be an
additional program they use solely for interacting with the Commons.
This would likely be the case initially because established genealogical
programs would presumably not support it from day zero. It could even
be a web service that the researcher is using.
To the non-technical genealogist, all they will see at this stage is a
page asking them to enter their name and email address (and any other
information that we may make mandatory).
The reason for detailing the underlying process is to allow multiple
programs by different vendors to interact seamlessly with the Commons.
And the reason for raising it here and now, i.e. on this newsgroup, is
(i) so that other technically-minded people have a clearer idea of the
sort of infrastructure I have in find, and (ii) so that any glaring
problems that I've overlooked will hopefully be noticed.
--
Richard Smith
Richard Smith
Jan 4, 2012, 5:13:39 AM1/4/12
to
absolutely right that only by choosing a format that satisfies all of
the above will will you maximise the chances of your data surviving.
My biggest horror isn't with common proprietary formats such as (the
old, non-XML) Word format. They are now so common that your third and
fourth points are both true, even if the first two aren't, and I've give
reasonable odds on them still being more-or-less readable with effort in
fifty years' time. No, my real horror is the plethora of non-GEDCOM
formats used by today's genealogy software. What are the odds that
anyone will be able to read a FTM 2010 file in fifty years? Virtually
zero I'd say.
But suppose I have a resource in some proprietary format that I can no
longer read (or at least convert), is it better that I can upload it on
the off-chance that someone can convert it before it's too late? Or is
it best to prevent such uploading on the basis that most people doing so
will be doing it because they're too lazy and/or ill-informed to convert
it to a better format? I'm not sure what I think and would be
interested to hear other people's views.
--
Richard Smith
Ian Goddard
Jan 4, 2012, 5:39:49 AM1/4/12
to
singhals wrote:
> Richard Smith wrote:
> Richard Smith wrote:
>> First off, I need a GPG key. I can use an existing one if I have one,
>> but I imagine most applications will choose to create a new key for user
>> for this purpose. This key needs to uploaded to an agreed keyserver, or
>> maybe several. For the purpose of this example, I have created myself a
>> new key with ID 78B6EA81 and have uploaded it to the keys.gnupg.net
>> keyserver.
>>
>
> WHO is the "I" who needs to do this? The DBA, the programmer, or the
> person doing the uploading?
>
>> Next I need to create and publish short bit of RDF/XML with a few bits
>> of housekeeping information about me. The contents of this are
>
> See above: who is the "I" who needs to do this?
It's worth remembering what "you" do any time you use the internet. The
>> but I imagine most applications will choose to create a new key for user
>> for this purpose. This key needs to uploaded to an agreed keyserver, or
>> maybe several. For the purpose of this example, I have created myself a
>> new key with ID 78B6EA81 and have uploaded it to the keys.gnupg.net
>> keyserver.
>>
>
> WHO is the "I" who needs to do this? The DBA, the programmer, or the
> person doing the uploading?
>
>> Next I need to create and publish short bit of RDF/XML with a few bits
>> of housekeeping information about me. The contents of this are
>
> See above: who is the "I" who needs to do this?
process probably runs along lines such as:
You check your cache to find if you already know the IP address of the
server you want. The next few steps assume you don't have that.
You check the IP address of your nameserver(s).
You check the IP address of your gateway.
You send a message to the gateway to be forwarded to your nameserver
containing the name of server you wish to connect to.
Your gateway may well only have one server to which it can send your
message and if that server isn't the nameserver itself it will have to
forward the message to some other server which may have to forward it to
some other server etc. until it reaches the nameserver.
Your nameserver returns the IP address of that server, possibly via a
number of intermediate steps as above to your gateway which forwards it
to you (on your behalf it may well have had to query another nameserver
which in turn may well have had to query another etc.).
You cache the IP address.
You send a message to your gateway to be forwarded to the server.
and so on.
Of course you're not aware that "you" did this. You may not be aware of
the fact that you have a nameserver (it's probably provided by your ISP)
although you probably are aware that you have a gateway; it's your
cable/ADSL/whatever box. All you personally did was to type in a URL,
click on a link or whatever. Lots of pieces of software written by lots
of programmers did the rest.
Broadly, what Richard describes would be follow the same pattern: you'd
have to fill in on-screen forms and the S/W would do the rest.
I have a reservation which you may well share.
New users are often required to fill in, live on line, forms, maybe
several screensworth, to join a system about which, at that stage, they
know little or nothing. And as they know little or nothing about the
system or, indeed, about what they might be asked on the next screen,
what they enter might not be what, in retrospect, they'd like to have
entered. I'd like to think that a system such as Richard envisages
would give the user a much better understanding of what's happening
before stuff happens on the net. Having to fill all this in before you
can proceed to use your new application really shouldn't be acceptable.
Richard Smith
Jan 4, 2012, 6:21:32 AM1/4/12
to
Ian Goddard wrote:
> I have a reservation which you may well share.
>
> New users are often required to fill in, live on line, forms, maybe
> several screensworth, to join a system about which, at that stage, they
> know little or nothing. And as they know little or nothing about the
> system or, indeed, about what they might be asked on the next screen,
> what they enter might not be what, in retrospect, they'd like to have
> entered. I'd like to think that a system such as Richard envisages would
> give the user a much better understanding of what's happening before
> stuff happens on the net. Having to fill all this in before you can
> proceed to use your new application really shouldn't be acceptable.
A good point. Perhaps the best interface is a menu option (or similar)
> I have a reservation which you may well share.
>
> New users are often required to fill in, live on line, forms, maybe
> several screensworth, to join a system about which, at that stage, they
> know little or nothing. And as they know little or nothing about the
> system or, indeed, about what they might be asked on the next screen,
> what they enter might not be what, in retrospect, they'd like to have
> entered. I'd like to think that a system such as Richard envisages would
> give the user a much better understanding of what's happening before
> stuff happens on the net. Having to fill all this in before you can
> proceed to use your new application really shouldn't be acceptable.
called "Connect to the Genealogical Commons" that you can do whenever
you're ready to. It should first give you a brief overview of what the
Commons is, and any caveats that such as that once you've released a
file into the Commons there's no guarantee you can delete it.
Obviously it needs to be possible to edit the information in your public
Commons profile (i.e. in the FOAF file). I need to give some thought as
to allow this.
--
Richard Smith
Ian Goddard
Jan 4, 2012, 7:04:01 AM1/4/12
to
Richard Smith wrote:
%><
>
> First off, I need a GPG key. ... I have created myself a
As things stand you have two servers, keys.gnupg.net and
commons.genmine.com which represent potential single points of failure.
The scheme would need to be extended so the data can be replicated.
%><
>
> If I want to add that to the commons, the first question I should be
> asking myself is: is it copyrighted? And do I have permission to copy it
> into the commons. Marriage certificates are usually Crown Copyright, and
> are covered by the Open Government Licence which allows people to copy
> them (with a few restrictions).
>
> http://www.nationalarchives.gov.uk/doc/open-government-licence/
I think this may need a little more work. What you uploaded was a scan
of an old cert with a plain background. If you buy one of the current
A4 certs it will have a background security print which is on the base
stationery. The base stationery will have its own copyright & that
might well not be covered by the OGL.
%><
completely detached from the file to which it refers. What I'd suggest
is that you upload one file which contains the metadata and the image.
This file would be covered by the CC-ND licence which means it can be
replicated but must be unchanged when this is done. The image itself
might be covered by some other licence when extracted. For instance, as
was suggested elsewhere in the thread, the image might be a photograph
which someone might wish to publish in cropped form. This would require
a more liberal licence.
%><
> More generally, by signing a statement you are guaranteeing its truth.
> This is the reason why I did not sign the marriage certificate, but
> rather put a SHA1 hash into a document that I signed. Had I GPG signed
> the marriage certificate, I would have been guaranteeing its contents to
> be true. I don't want to do that because I'm pretty sure the groom was
> actually 24, not 22, and the bride was only 19 rather than 20. Even if I
> didn't know of any errors, I wouldn't wish to make a guarantee about
> something so long ago.
But what would you be guaranteeing? The document itself bears a
signature saying it's a true copy of the register. If you signed the
upload you'd be saying nothing more than that the image was a true copy
of that piece of paper.
We need to bear in mind that there are a whole lot of layers here. The
document is evidence.
If you extract what it says, including ages, that's analysis.
If you consider the content - for instance, if you evaluate the truth of
the alleged ages or if you identify the people named with people named
in other documents - that's interpretation. In fact your evaluation of
the truth of the ages depends on your identification of the people; they
could, in fact, be different people whose true ages were as stated.
Because these are different types of information I consider they should
belong in different classes of object in any system that stores them.
This is a further reason for wrapping the image inside the same XML that
contains its metadata: when an application downloads a file it will be
clear from the wrapper what internal class of object should be created
to hold it. This, in passing, is one of the major problems with GEDCOM
- it makes no such distinction.
%><
>
> First off, I need a GPG key. ... I have created myself a
> new key with ID 78B6EA81 and have uploaded it to the keys.gnupg.net
> keyserver.
>
> Next I need to create and publish short bit of RDF/XML with a few bits
> of housekeeping information about me. ... For my example, I have created a
> keyserver.
>
> Next I need to create and publish short bit of RDF/XML with a few bits
> minimal example which I have uploaded here:
>
> http://commons.genmine.com/Users/Richard_Smith.foaf
%><
>
> http://commons.genmine.com/Users/Richard_Smith.foaf
As things stand you have two servers, keys.gnupg.net and
commons.genmine.com which represent potential single points of failure.
The scheme would need to be extended so the data can be replicated.
%><
> seconds. Anyway, I'm finally ready to do some genealogy, and I
> decide to start by uploading a scan of my great great grandparents'
> marriage certificate:
>
> http://commons.genmine.com/Test/Thomas_Smith_marriage.png
>
> There's absolutely nothing special about the way I've uploaded that,
I'll come back to that one.
> decide to start by uploading a scan of my great great grandparents'
> marriage certificate:
>
> http://commons.genmine.com/Test/Thomas_Smith_marriage.png
>
> There's absolutely nothing special about the way I've uploaded that,
>
> If I want to add that to the commons, the first question I should be
> asking myself is: is it copyrighted? And do I have permission to copy it
> into the commons. Marriage certificates are usually Crown Copyright, and
> are covered by the Open Government Licence which allows people to copy
> them (with a few restrictions).
>
> http://www.nationalarchives.gov.uk/doc/open-government-licence/
of an old cert with a plain background. If you buy one of the current
A4 certs it will have a background security print which is on the base
stationery. The base stationery will have its own copyright & that
might well not be covered by the OGL.
%><
> Happy that I'm allowed to upload, it I produce a piece of RDF/XML with
> some basic metadata about the scanned certificate, which I upload here:
>
> http://commons.genmine.com/Test/Thomas_Smith_marriage.rdf
> (There's no requirement for it to be in the same directory or even on
> the same server as the image.)
OK, going back to the point I skipped earlier. The licence is
> some basic metadata about the scanned certificate, which I upload here:
>
> http://commons.genmine.com/Test/Thomas_Smith_marriage.rdf
> (There's no requirement for it to be in the same directory or even on
> the same server as the image.)
completely detached from the file to which it refers. What I'd suggest
is that you upload one file which contains the metadata and the image.
This file would be covered by the CC-ND licence which means it can be
replicated but must be unchanged when this is done. The image itself
might be covered by some other licence when extracted. For instance, as
was suggested elsewhere in the thread, the image might be a photograph
which someone might wish to publish in cropped form. This would require
a more liberal licence.
%><
> More generally, by signing a statement you are guaranteeing its truth.
> This is the reason why I did not sign the marriage certificate, but
> rather put a SHA1 hash into a document that I signed. Had I GPG signed
> the marriage certificate, I would have been guaranteeing its contents to
> be true. I don't want to do that because I'm pretty sure the groom was
> actually 24, not 22, and the bride was only 19 rather than 20. Even if I
> didn't know of any errors, I wouldn't wish to make a guarantee about
> something so long ago.
signature saying it's a true copy of the register. If you signed the
upload you'd be saying nothing more than that the image was a true copy
of that piece of paper.
We need to bear in mind that there are a whole lot of layers here. The
document is evidence.
If you extract what it says, including ages, that's analysis.
If you consider the content - for instance, if you evaluate the truth of
the alleged ages or if you identify the people named with people named
in other documents - that's interpretation. In fact your evaluation of
the truth of the ages depends on your identification of the people; they
could, in fact, be different people whose true ages were as stated.
Because these are different types of information I consider they should
belong in different classes of object in any system that stores them.
This is a further reason for wrapping the image inside the same XML that
contains its metadata: when an application downloads a file it will be
clear from the wrapper what internal class of object should be created
to hold it. This, in passing, is one of the major problems with GEDCOM
- it makes no such distinction.
Tony Proctor
Jan 4, 2012, 7:43:33 AM1/4/12
to
news:9min2o...@mid.individual.net...
Ideally, Richard, the raw data should be in a "source format" rather than in
discrete databases or proprietary binary formats. In other words, something
textual, including XML. An advantage of a "source format" is that it will
always be possible to parse the content - even in 50 years time - as long as
there's at least a specification document for it.
A "source format" would the format you'd use for long-term preservation of
your own data. It isn't the same as the database or other indexed system
that your software works from. This distinction often gets glossed over.
OK, so there's GEDCOM - which is sadly lacking these days - and nothing else
that has anywhere near as much recognition.
Tony Proctor
discrete databases or proprietary binary formats. In other words, something
textual, including XML. An advantage of a "source format" is that it will
always be possible to parse the content - even in 50 years time - as long as
there's at least a specification document for it.
A "source format" would the format you'd use for long-term preservation of
your own data. It isn't the same as the database or other indexed system
that your software works from. This distinction often gets glossed over.
OK, so there's GEDCOM - which is sadly lacking these days - and nothing else
that has anywhere near as much recognition.
Tony Proctor
Tony Proctor
Jan 4, 2012, 7:50:46 AM1/4/12
to
news:9mhst6...@mid.individual.net...
RE; "http://commons.genmine.com/Users/Richard_Smith... is a URN that
identifiers me"
Technically speaking, Richard, that isn't a URN - which is a more restricted
form of URI for identifying an object in a hierarchical namespace. However,
the use of a general URI as a "name" is generally accepted and informally
referred to as a URN. [Sorry about nick-picking. I use the same terms myself
but it's just for anyone else following up on things]
Tony Proctor
identifiers me"
Technically speaking, Richard, that isn't a URN - which is a more restricted
form of URI for identifying an object in a hierarchical namespace. However,
the use of a general URI as a "name" is generally accepted and informally
referred to as a URN. [Sorry about nick-picking. I use the same terms myself
but it's just for anyone else following up on things]
Tony Proctor
Richard Smith
Jan 4, 2012, 8:21:29 AM1/4/12
to
Ian Goddard wrote:
> As things stand you have two servers, keys.gnupg.net and
> commons.genmine.com which represent potential single points of failure.
> The scheme would need to be extended so the data can be replicated.
There are lots of keyservers out there. pgp.mit.edu is another popular
> As things stand you have two servers, keys.gnupg.net and
> commons.genmine.com which represent potential single points of failure.
> The scheme would need to be extended so the data can be replicated.
one. The index servers may wish to run their own too. Users would be
encouraged to upload their keys to multiple keyservers, and index
servers would be encouraged to try multiple keyservers until the key is
located even if they are running one locally. We should not mandate the
use of any specific keyservers as the current ones may all have
disappeared in a few decades.
commons.genmine.com (in this example) ceases to be a single point of
failure as soon as the content has entered the Commons. Data, including
FOAF and RDF metadata and GPG signatures, gets copied on to multiple
storage nodes. The remaining way in which this domain may be a single
point of failure is in its use in URNs for users. I think I have a
solution to this which I'll explain when I explain how I envisage the
index server storing data.
>> If I want to add that to the commons, the first question I should be
>> asking myself is: is it copyrighted? And do I have permission to copy it
>> into the commons. Marriage certificates are usually Crown Copyright, and
>> are covered by the Open Government Licence which allows people to copy
>> them (with a few restrictions).
>>
>> http://www.nationalarchives.gov.uk/doc/open-government-licence/
>
> I think this may need a little more work. What you uploaded was a scan
> of an old cert with a plain background. If you buy one of the current A4
> certs it will have a background security print which is on the base
> stationery. The base stationery will have its own copyright & that might
> well not be covered by the OGL.
http://www.nationalarchives.gov.uk/documents/information-management/copying-bmd-certificates.pdf
But even if that does not apply here, either copying is permitted under
some explicit or implicit licence (including the possibility of it being
public domain) in which case this must be documented and the resource
can enter the Commons, or copying is not permitted and the resource may
not enter the Commons.
>> Happy that I'm allowed to upload, it I produce a piece of RDF/XML with
>> some basic metadata about the scanned certificate, which I upload here:
>>
>> http://commons.genmine.com/Test/Thomas_Smith_marriage.rdf
>
>> (There's no requirement for it to be in the same directory or even on
>> the same server as the image.)
>
> OK, going back to the point I skipped earlier. The licence is completely
> detached from the file to which it refers. What I'd suggest is that you
> upload one file which contains the metadata and the image. This file
> would be covered by the CC-ND licence which means it can be replicated
> but must be unchanged when this is done. The image itself might be
> covered by some other licence when extracted. For instance, as was
> suggested elsewhere in the thread, the image might be a photograph which
> someone might wish to publish in cropped form. This would require a more
> liberal licence.
PNG, RDF and GPG signature into a ZIP file, and referenced the parts
using the jar: URL scheme. (This applies to arbitrary ZIP files, even
though it was originally created with Java's JAR archives in mind.)
Another possibility if I don't have the ability to publish the resources
myself is that I can email them to a combined index/storage server. I
use the cid: URL scheme to reference the parts (attachments, if you
like) to the email, and the index/storage server publishes the multipart
MIME message somewhere. (It cannot separate them because that would
mean rewriting the cid: URLs inside the signed document.)
However I don't want to lose the ability to upload someone else's
document without duplicating it myself. For example, I might browse
through the Wikimedia Commons looking for resources of potential
interest to a genealogist. I think it would be very useful if I could
upload them without uploading the RDF metadata to the Wikimedia Commons
and without copying the resource elsewhere.
A combination of the last two examples may also be useful. I might
email the RDF and GPG key to a combined index/storage server, but
reference a remote document on, say, the Wikimedia Commons.
A cid: URL has some issues in that there's no way of locating a copy of
the email containing the resource, however this is precisely the same
problem as that of coping with domains that have expired, and hopefully
the mechanism I have in mind to cope with that will cope with this too.
More on that in a later email.
As I see it, the key point is to provide flexibility. You may not want
to interact with the server using email, but that may be useful to
someone else. Not all index servers would need to support that
functionality. And I hope that I've designed it such that only clients
and servers that want to use email functionality need to including
functionality to support it (for example, to understand cid: URLs).
> But what would you be guaranteeing?
now.
--
Richard Smith
Richard Smith
Jan 4, 2012, 9:03:07 AM1/4/12
to
Tony Proctor wrote:
> RE; "http://commons.genmine.com/Users/Richard_Smith... is a URN that
> identifiers me"
>
> Technically speaking, Richard, that isn't a URN - which is a more restricted
> form of URI for identifying an object in a hierarchical namespace. However,
> the use of a general URI as a "name" is generally accepted and informally
> referred to as a URN. [Sorry about nick-picking. I use the same terms myself
> but it's just for anyone else following up on things]
Yes, you're quite right to pick me up on that. To have been accurate I
> RE; "http://commons.genmine.com/Users/Richard_Smith... is a URN that
> identifiers me"
>
> Technically speaking, Richard, that isn't a URN - which is a more restricted
> form of URI for identifying an object in a hierarchical namespace. However,
> the use of a general URI as a "name" is generally accepted and informally
> referred to as a URN. [Sorry about nick-picking. I use the same terms myself
> but it's just for anyone else following up on things]
should have written "non-URL URI" or "URN-like URI" or something like
that. (In fact, I do have in mind using real URNs as an alternative to
unresolvable http: URIs, but more of that later.)
--
Richard Smith
Richard Smith
Jan 4, 2012, 9:10:46 AM1/4/12
to
Tony Proctor wrote:
> Ideally, Richard, the raw data should be in a "source format" rather than in
> discrete databases or proprietary binary formats. In other words, something
> textual, including XML. An advantage of a "source format" is that it will
> always be possible to parse the content - even in 50 years time - as long as
> there's at least a specification document for it.
I fully agree with everything you've written here.
> Ideally, Richard, the raw data should be in a "source format" rather than in
> discrete databases or proprietary binary formats. In other words, something
> textual, including XML. An advantage of a "source format" is that it will
> always be possible to parse the content - even in 50 years time - as long as
> there's at least a specification document for it.
But there will be some situations where a user cannot get a suitable
"source format". For instance, if they've inherited a bunch of files
from a relative, but don't have access to the software used to create
them. (Or perhaps have access to a free or somehow limited viewer
program, but not to the full program that allows export in another format.)
In such a situation is it better to allow upload in an obscure
proprietary format, when the alternative is almost certainly that the
file gets lost?
Or is it better to prohibit it on the basis that if allowed it would be
overused by people who could easily have converted the data and would
have done if there was no other option?
--
Richard Smith
singhals
Jan 4, 2012, 10:11:01 AM1/4/12
to gen...@rootsweb.com
Richard Smith wrote:
> Ian Goddard wrote:
>
>> I have a reservation which you may well share.
>>
>> New users are often required to fill in, live on line, forms, maybe
>> several screensworth, to join a system about which, at that stage, they
>> know little or nothing. And as they know little or nothing about the
>> system or, indeed, about what they might be asked on the next screen,
>> what they enter might not be what, in retrospect, they'd like to have
>> entered. I'd like to think that a system such as Richard envisages would
>> give the user a much better understanding of what's happening before
>> stuff happens on the net. Having to fill all this in before you can
>> proceed to use your new application really shouldn't be acceptable.
>
> A good point. Perhaps the best interface is a menu option (or similar)
> called "Connect to the Genealogical Commons" that you can do whenever
> you're ready to. It should first give you a brief overview of what the
> Commons is, and any caveats that such as that once you've released a
> file into the Commons there's no guarantee you can delete it.
One private password protected site I visit does something good.
> Ian Goddard wrote:
>
>> I have a reservation which you may well share.
>>
>> New users are often required to fill in, live on line, forms, maybe
>> several screensworth, to join a system about which, at that stage, they
>> know little or nothing. And as they know little or nothing about the
>> system or, indeed, about what they might be asked on the next screen,
>> what they enter might not be what, in retrospect, they'd like to have
>> entered. I'd like to think that a system such as Richard envisages would
>> give the user a much better understanding of what's happening before
>> stuff happens on the net. Having to fill all this in before you can
>> proceed to use your new application really shouldn't be acceptable.
>
> A good point. Perhaps the best interface is a menu option (or similar)
> called "Connect to the Genealogical Commons" that you can do whenever
> you're ready to. It should first give you a brief overview of what the
> Commons is, and any caveats that such as that once you've released a
> file into the Commons there's no guarantee you can delete it.
The user clicks on the tab wanted; a brief most-of-a-screen
description of the content is provided and an [ENTER]
button. Upon clicking [ENTER], you get a new box with the
SAME description, subtly changed and a 2nd [ENTER] button.
Click that one and you get a 3rd box with a cavaet and
another [ENTER] button. Upon clicking that one too, you
get, finally, to the search boxes. The 2nd appearance of
what appears to be the same information generally makes
people read it at least once. Which cuts down on complaints
of "I didn't know that!"
Cheryl
singhals
Jan 4, 2012, 10:03:19 AM1/4/12
to gen...@rootsweb.com
include all that. One says "https://my.bank.com/" not
"dial/type/autofill if possible/confirm connex/check DNS/wait".
Perhaps it's a grammatics issue; it may take more chr$ but
saying "the program needs to" rather than "I need to" would
have had sufficient clarity to avoid the question.
> Broadly, what Richard describes would be follow the same pattern: you'd
> have to fill in on-screen forms and the S/W would do the rest.
>
> I have a reservation which you may well share.
>
> New users are often required to fill in, live on line, forms, maybe
> several screensworth, to join a system about which, at that stage, they
> know little or nothing. And as they know little or nothing about the
> system or, indeed, about what they might be asked on the next screen,
> what they enter might not be what, in retrospect, they'd like to have
> entered. I'd like to think that a system such as Richard envisages
> would give the user a much better understanding of what's happening
> before stuff happens on the net. Having to fill all this in before you
> can proceed to use your new application really shouldn't be acceptable.
>
Word-of-mouth and street-buzz might be enough in the
beginning, but 3, 4, 7 years later, prolly not.
A menu page with
*Who We Are
*What We Can Do For You
*Thanks for Visiting
*Sounds Good, What Next?
with What Next having subs for
*Information we need and why
*What we're going to do with it
*What we'll not do with it
*Sign me up.
*Include me out
Or whatever terms seem equally clear and better phrased.
Cheryl
singhals
Jan 4, 2012, 10:18:16 AM1/4/12
to gen...@rootsweb.com
surely it can ASK what you're planning to upload then check
that against a list of known-obsolete-formats and reporting
to the uploader that the format can be and must be
standardized to plain-text.
OTOH, you'll find an awful lot of people who don't know
plain-text from Sanskrit ... most of those are over 60 or
under 30.
Cheryl
Richard Smith
Jan 4, 2012, 10:33:19 AM1/4/12
to
Cheryl Singhals wrote:
> Since our hypothetical commons is doing other magical stuff, surely it
> can ASK what you're planning to upload then check that against a list of
> known-obsolete-formats and reporting to the uploader that the format can
> be and must be standardized to plain-text.
Yes, it can do that. But what if the uploader doesn't have the ability
> Since our hypothetical commons is doing other magical stuff, surely it
> can ASK what you're planning to upload then check that against a list of
> known-obsolete-formats and reporting to the uploader that the format can
> be and must be standardized to plain-text.
to convert it and cannot easily get hold of software to do it? (Imagine
the software can only run on some archaic machine that few people still
have, or the conversions software costs a vast sum of money.)
In such a circumstance do we grudgingly allow people to upload the
content anyway? Or do we say "tough luck; come back in the event that
you can get someone to convert it for you"?
> OTOH, you'll find an awful lot of people who don't know plain-text from
> Sanskrit ... most of those are over 60 or under 30.
--
Richard Smith
Ian Goddard
Jan 4, 2012, 10:35:02 AM1/4/12
to
Consider for a moment a stand alone application's database. This
probably uses some auto-incrementing int as surrogate keys to stitch
itself together.
To recall Cheryl's question "I" don't even have to be aware of any of
this. It's buried deep within the S/W. Whatever S/W Cheryl is
currently using may well be doing just this, However, if two users of
such data wish to exchange data they can't use these keys to identify
such exchanged data as the same values will be in use on each system to
identify different data items.
UUIDs are essentially an extension of this except that the key-length is
almost certainly longer and the auto-increment is replaced by the need
to generate a long random number (assuming it's a v4 UUID). However I
think any platform that a user is likely to be using has support for
UUIDs (under some name or other ;) with bindings in most if not all
languages likely to be used for development.
Again, returning to Cheryl's question, this would be equally invisible.
But now users could use such keys to exchange information as any
particular value will be unique across all systems, not just on the one
that generated it. Also if user A gives B a copy of some record, B
gives C a copy and C gives A a copy A's S/W will recognise the duplicate
& discard it.
FWIW I don't know what's in use in any genealogical S/W other than
Gramps & Gramps does sort of use UUIDs because they're exposed when data
is exported as XML except that they seem to be a little shorter than the
OSI defined versions.
Now at its simplest a commons is a general scheme for sharing data and
UUIDs are sufficient for that purpose. The commons becomes a
distributed replicate database and each user's database is potentially a
sub-set of that. Or to put it another way, the commons is just an
extension of what the user might be using already.
If, however, you use URIs (or URNs) as the glue for the commons the
commons is no longer a straightforward extension unless you also use
URIs as internal keys. URIs have disadvantages as keys. Firstly there
is this external registration requirement. Secondly they are now
variable length text. Thirdly they only differ in the less significant
bytes which isn't ideal for indexing. If, however, you retain some
integer or UUID internally and URIs externally you have to map between
the two somehow.
I can see the advantage for handling trust etc. but this is a different
function & I think the mechanism for handling such issues is best kept
separate from structural mechanism.
Richard Smith
Jan 4, 2012, 10:47:42 AM1/4/12
to
Cheryl Singhals wrote:
> Perhaps it's a grammatics issue; it may take more chr$ but saying "the
> program needs to" rather than "I need to" would have had sufficient
> clarity to avoid the question.
I can see what you mean, but what I'm doing at the moment is designing
> Perhaps it's a grammatics issue; it may take more chr$ but saying "the
> program needs to" rather than "I need to" would have had sufficient
> clarity to avoid the question.
the infrastructure that underlies the Commons, not a program that access
it. So when I say "I need to", "I" is what I mean. From an
infrastructural point of view, I really don't care whether the person is
doing it themselves, or a program is doing it on their behalf.
For some of the tasks I described, exactly how much of the process is
automated will depend on the program the person is used. Some people
will prefer a program that automates the whole process, hiding the
details as completely as possible. Others, particularly the more
technically advanced users while doing something out of the ordinary,
will actively want certain details them entirely.
To reiterate the main point, the Commons isn't a genealogical program:
it is a collection of online resources together with a standardised
protocol for accessing them that developers can use to the expose these
resources in their applications or web services.
--
Richard Smith
singhals
Jan 4, 2012, 10:54:59 AM1/4/12
to gen...@rootsweb.com
Richard Smith wrote:
> Cheryl Singhals wrote:
>
>> Since our hypothetical commons is doing other magical stuff, surely it
>> can ASK what you're planning to upload then check that against a list of
>> known-obsolete-formats and reporting to the uploader that the format can
>> be and must be standardized to plain-text.
>
> Yes, it can do that. But what if the uploader doesn't have the ability
> to convert it and cannot easily get hold of software to do it? (Imagine
> the software can only run on some archaic machine that few people still
> have, or the conversions software costs a vast sum of money.)
Then, it would be on the list of known-obsolete-formats. I
> Cheryl Singhals wrote:
>
>> Since our hypothetical commons is doing other magical stuff, surely it
>> can ASK what you're planning to upload then check that against a list of
>> known-obsolete-formats and reporting to the uploader that the format can
>> be and must be standardized to plain-text.
>
> Yes, it can do that. But what if the uploader doesn't have the ability
> to convert it and cannot easily get hold of software to do it? (Imagine
> the software can only run on some archaic machine that few people still
> have, or the conversions software costs a vast sum of money.)
have data in a dandy little program called Eucalyptus; ran
in DOS, no export facility. I have some data (duplicate of
the Eucalyptus content) in a different program I was
beta-testing, again no export facility; I have PAF2.0
databases, FTM, Legacy, Ultimate Family Tree -- and some of
those would have to be known-obsoletes as well, because to
move data from PAF2.0 to anything known today, you need _2_
conversion steps.
>
> In such a circumstance do we grudgingly allow people to upload the
> content anyway? Or do we say "tough luck; come back in the event that
> you can get someone to convert it for you"?
>
also list the steps in that op so the Uploader could do it.
No need to advertise that you'll take 24hrs later.
If like PAF2.0 files, it isn't, then yes, let them upload
it. I suppose you COULD create a separate "area" where the
uploader could place these uploads and give consent for
someone else (as an Act of Genealogical Kindness) to convert.
Cheryl
Ian Goddard
Jan 4, 2012, 11:27:44 AM1/4/12
to
Richard Smith wrote:
> But suppose I have a resource in some proprietary format that I can no
> longer read (or at least convert), is it better that I can upload it on
> the off-chance that someone can convert it before it's too late? Or is
> it best to prevent such uploading on the basis that most people doing so
> will be doing it because they're too lazy and/or ill-informed to convert
> it to a better format? I'm not sure what I think and would be interested
> to hear other people's views.
My background, way back, is in science. Because of that the starting
> But suppose I have a resource in some proprietary format that I can no
> longer read (or at least convert), is it better that I can upload it on
> the off-chance that someone can convert it before it's too late? Or is
> it best to prevent such uploading on the basis that most people doing so
> will be doing it because they're too lazy and/or ill-informed to convert
> it to a better format? I'm not sure what I think and would be interested
> to hear other people's views.
point in my thinking is always evidence. Before coming to genealogy I'd
already been round the block a couple of times in that I've worked in
two areas which were essentially investigations of the past. The
essence of both areas was that you mostly had to use what evidence you'd
already got; you couldn't go out and take another water sample, turn on
a LHC and generate a few billion more particles or whatever. This leads
to a great respect for the primacy of original evidence.
So if what you've got is a file in proprietary or even obsolete format
that is your original evidence so you should share that.
If you've no means of translating it to some other format SKS might have
the means may do so if you share it but if you don't they can't. OTOH
if you have some means of translating it into some a preferable format
why not share both?
If what you have is an actual document to be scanned then share several
different formats of the scan.
Tony Proctor
Jan 4, 2012, 11:37:52 AM1/4/12
to
"Ian Goddard" <godd...@hotmail.co.uk> wrote in message
news:9mj9t7...@mid.individual.net...
tell you that issues such as variable-length and commonality of the
left-hand (or right-hand) portion are not relevant to indexing. A good hash
algorithm would not be sensitive to these. The Java algorithm (visible in
the String class) is a good example of a bad one - if you know what I mean.
It uses a multiply-and-add approach (which is hugely CPU-intensive when
compared to say an XOR) and *does* favour the left-hand side. There are
well-documented ones that use a rotate-and-XOR approach which avoids both
disadvantages but Java couldn't change theirs because the original
openly-visible algorithm is now assumed by a significant amount of s/w.
Even when comparing two identifiers, as on a hash collision, there are some
very clever algorithms which do not perform a simple left-to-right literatim
match.
URIs only need a unique root part to have been registered but derivations of
that root do not have to be registered. This root is usually an Internet
domain name but that's not the only possibility. It could involve an email
address for instance.
Tony Proctor
Richard Smith
Jan 4, 2012, 12:09:50 PM1/4/12
to
Ian Goddard wrote:
> So if what you've got is a file in proprietary or even obsolete format
> that is your original evidence so you should share that.
>
> If you've no means of translating it to some other format SKS might have
> the means may do so if you share it but if you don't they can't. OTOH if
> you have some means of translating it into some a preferable format why
> not share both?
That was my initial thought, too. Maybe it's not a coincidence that my
> So if what you've got is a file in proprietary or even obsolete format
> that is your original evidence so you should share that.
>
> If you've no means of translating it to some other format SKS might have
> the means may do so if you share it but if you don't they can't. OTOH if
> you have some means of translating it into some a preferable format why
> not share both?
background is also in science.
--
Richard Smith
Ian Goddard
Jan 4, 2012, 12:20:33 PM1/4/12
to
%><
issue can make for an inefficient B-tree index in an RDBMS. Which I can
tell you as a DBA for many years :)
The issue with variable -and worse, undefined - length isn't so much
indexing but the need for a longish column length to be specified just
to be on the safe side (you don't want to truncate your key!), which
again is an issue for RDBMS.
>> URIs have disadvantages as keys. Firstly there is this
>> external registration requirement. Secondly they are now variable length
>> text. Thirdly they only differ in the less significant bytes which isn't
>> ideal for indexing. If, however, you retain some integer or UUID
>> internally and URIs externally you have to map between the two somehow.
%><
>> external registration requirement. Secondly they are now variable length
>> text. Thirdly they only differ in the less significant bytes which isn't
>> ideal for indexing. If, however, you retain some integer or UUID
>> internally and URIs externally you have to map between the two somehow.
>
> Sorry to pick you up here Ian but as a compiler-writer for many years I can
> tell you that issues such as variable-length and commonality of the
> left-hand (or right-hand) portion are not relevant to indexing.
It depends on what sort of index you're thinking of. The commonality
> Sorry to pick you up here Ian but as a compiler-writer for many years I can
> tell you that issues such as variable-length and commonality of the
> left-hand (or right-hand) portion are not relevant to indexing.
issue can make for an inefficient B-tree index in an RDBMS. Which I can
tell you as a DBA for many years :)
The issue with variable -and worse, undefined - length isn't so much
indexing but the need for a longish column length to be specified just
to be on the safe side (you don't want to truncate your key!), which
again is an issue for RDBMS.
tms
Jan 4, 2012, 12:27:09 PM1/4/12
to
On Jan 3, 9:42 pm, Wes Groleau <Groleau+n...@FreeShell.org> wrote:
> On 01-03-2012 13:58, tms wrote:
>
> > On Sunday, January 1, 2012 1:45:23 PM UTC-5, Wes Groleau wrote:
> >> On 01-01-2012 10:27, Lesley Robertson wrote:
> >>> If you don't want the system to be dismissed as "good for clues but not
> >>> to be trusted" (cf the IGI)
>
> >> And cf ancestral file and ancestry.com and familysearch.org and
> >> worldconnect.rootsweb.com and .....
>
> > ... and your local library. Just because something is in a book does not mean that it is necessarily correct. One must always use one's judgment when using any source.
>
> There was false information in "the old days" too. But the difficulty
> of producing it meant that it happened less often (and was more often
> intentional).
The difficulty of producing it also meant that there was less good
> On 01-03-2012 13:58, tms wrote:
>
> > On Sunday, January 1, 2012 1:45:23 PM UTC-5, Wes Groleau wrote:
> >> On 01-01-2012 10:27, Lesley Robertson wrote:
> >>> If you don't want the system to be dismissed as "good for clues but not
> >>> to be trusted" (cf the IGI)
>
> >> And cf ancestral file and ancestry.com and familysearch.org and
> >> worldconnect.rootsweb.com and .....
>
> > ... and your local library. Just because something is in a book does not mean that it is necessarily correct. One must always use one's judgment when using any source.
>
> There was false information in "the old days" too. But the difficulty
> of producing it meant that it happened less often (and was more often
> intentional).
information, too. Just because Aristotle had to have his books copied
by hand does not mean that he got physics right.
> Current technology makes it super easy to reproduce garbage and people
> do just that.
previously-difficult-to-find sources, that is also easy to keep up to
date.
I'd agree that in the old days it took real dedication to track down
data, so few dilettantes published anything, but dedication is not the
same as competence. There are plenty of very dedicated crackpots out
there (take a look at sci.physics.relativity).
--
T.M. Sommers
tms
Jan 4, 2012, 12:38:22 PM1/4/12
to
On Jan 4, 5:13 am, Richard Smith <rich...@ex-parrot.com> wrote:
> On 03/01/12 18:36, tms wrote:
>
> But suppose I have a resource in some proprietary format that I can no
> longer read (or at least convert), is it better that I can upload it on
> the off-chance that someone can convert it before it's too late?
Best not to get into that situation in the first place. Routinely
> On 03/01/12 18:36, tms wrote:
>
> But suppose I have a resource in some proprietary format that I can no
> longer read (or at least convert), is it better that I can upload it on
> the off-chance that someone can convert it before it's too late?
backup your data to a more portable format, if possible, or at least
to Gedcom.
> Or is
> it best to prevent such uploading on the basis that most people doing so
> will be doing it because they're too lazy and/or ill-informed to convert
> it to a better format? I'm not sure what I think and would be
> interested to hear other people's views.
service. There are other ways of converting old data, such as asking
for help in this newsgroup.
--
T.M. Sommers
tms
Jan 4, 2012, 12:32:02 PM1/4/12
to
On Jan 3, 9:37 pm, Wes Groleau <Groleau+n...@FreeShell.org> wrote:
The language is not error-prone, careless programmers are. Agreed,
though, that C requires more care than some other languages. However,
it is nearly ubiquitous, and it is highly probably that whatever
systems are around in 50 years will have C compilers that can compile
legacy code, even from K&R days.
--
T.M. Sommers
> On 01-03-2012 13:36, tms wrote:
>
> > at least in one very-widely-used language (such as C).
>
> The most error-prone widely-used language in existence.
>
> > at least in one very-widely-used language (such as C).
>
The language is not error-prone, careless programmers are. Agreed,
though, that C requires more care than some other languages. However,
it is nearly ubiquitous, and it is highly probably that whatever
systems are around in 50 years will have C compilers that can compile
legacy code, even from K&R days.
--
T.M. Sommers
Wes Groleau
Jan 4, 2012, 8:23:09 PM1/4/12
to
On 01-04-2012 12:32, tms wrote:
> The language is not error-prone, careless programmers are. Agreed,
> though, that C requires more care than some other languages. However,
All programmers are error-prone. Some languages protect against this,
> The language is not error-prone, careless programmers are. Agreed,
> though, that C requires more care than some other languages. However,
others exaggerate it. C is one of the worst of the latter.
--
Wes Groleau
Items of dialect heard …
http://Ideas.Lang-Learn.us/barrett?itemid=1522
Wes Groleau
Jan 4, 2012, 8:26:27 PM1/4/12
to
On 01-04-2012 05:13, Richard Smith wrote:
> fifty years' time. No, my real horror is the plethora of non-GEDCOM
> formats used by today's genealogy software. What are the odds that
> anyone will be able to read a FTM 2010 file in fifty years? Virtually
No secret that I have a lot of complaints about GEDCOM, but if ever it
> fifty years' time. No, my real horror is the plethora of non-GEDCOM
> formats used by today's genealogy software. What are the odds that
> anyone will be able to read a FTM 2010 file in fifty years? Virtually
happens that there are NO programs that can read it, and NO documents to
explain it, a human of average intelligence and a reasonable amount of
motivation could figure it out.
Richard Smith
Jan 4, 2012, 9:08:01 PM1/4/12
to
On 05/01/12 01:26, Wes Groleau wrote:
> On 01-04-2012 05:13, Richard Smith wrote:
>
>> fifty years' time. No, my real horror is the plethora of non-GEDCOM
>> formats used by today's genealogy software. What are the odds that
>> anyone will be able to read a FTM 2010 file in fifty years? Virtually
>
> No secret that I have a lot of complaints about GEDCOM, but if ever it
> happens that there are NO programs that can read it, and NO documents to
> explain it, a human of average intelligence and a reasonable amount of
> motivation could figure it out.
Err... I said "non-GEDCOM formats", specifically the binary formats. As
> On 01-04-2012 05:13, Richard Smith wrote:
>
>> fifty years' time. No, my real horror is the plethora of non-GEDCOM
>> formats used by today's genealogy software. What are the odds that
>> anyone will be able to read a FTM 2010 file in fifty years? Virtually
>
> No secret that I have a lot of complaints about GEDCOM, but if ever it
> happens that there are NO programs that can read it, and NO documents to
> explain it, a human of average intelligence and a reasonable amount of
> motivation could figure it out.
you say, for all its faults GEDCOM can easily be figured out.
--
Richard Smith
Bob Melson
Jan 4, 2012, 9:12:13 PM1/4/12
to
On Wednesday 04 January 2012 18:23, Wes Groleau
(Grolea...@FreeShell.org) opined:
Years ago one of my profs likened C to a Formula-I race car - built low and
close to the ground, able to corner well but deadly to the novice or
careless driver. Same guy said it was a high-level assembly language.
I've found no reason in the past mumble years to dispute either
observation.
Senescent Ol' Bob
--
Robert G. Melson | Rio Grande MicroSolutions | El Paso, Texas
-----
The greatest tyrannies are always perpetrated
in the name of the noblest causes -- Thomas Paine
(Grolea...@FreeShell.org) opined:
close to the ground, able to corner well but deadly to the novice or
careless driver. Same guy said it was a high-level assembly language.
I've found no reason in the past mumble years to dispute either
observation.
Senescent Ol' Bob
--
Robert G. Melson | Rio Grande MicroSolutions | El Paso, Texas
-----
The greatest tyrannies are always perpetrated
in the name of the noblest causes -- Thomas Paine
Wes Groleau
Jan 4, 2012, 9:40:35 PM1/4/12
to
On 01-04-2012 21:12, Bob Melson wrote:
> Years ago one of my profs likened C to a Formula-I race car - built low and
> close to the ground, able to corner well but deadly to the novice or
Here's an interesting comparison between two languages:
> Years ago one of my profs likened C to a Formula-I race car - built low and
> close to the ground, able to corner well but deadly to the novice or
" During the first six years that the real-time systems course was
offered, students developed their control code in C. As shown in Figure
3, no team successfully implemented minimum project requirements when
the C language was used. To ease student and teacher frustrations I made
an increasing amount of my solutions available to the teams. Figure 3
shows that even when I provided nearly 60 percent of the project code,
no team was successful in implementing the minimum requirements.
Along with the new hardware provided by the NSF funding was a
collection of DEC compilers. Thinking that the low level of tasking
provided through semaphores was the major contributor to the problem, I
selected a language with a much higher level of tasking
abstractions—Ada. I expected a disaster the first year with the new
equipment and new language. As in a real-life embedded systems project,
I was building the hardware while my students were writing the software.
I finished the hardware with only four weeks remaining in the semester.
But to my amazement, nearly 50 percent of the student teams had their
projects working before the end of the semester. I had only supplied
them with two sample device drivers. As shown in Figure 4, when I
supplied some additional software components (simple window packages not
relevant to the real-time aspect of the project), more than 75 percent
of my teams routinely completed their projects.
The only difference between the years in which teams succeeded in
implementing their projects and those in which no team succeeded was the
implementation language. The project specification, design, and unit
testing techniques did not vary."
Wes Groleau
Jan 4, 2012, 9:42:40 PM1/4/12
to
preferable for LONG-TERM availability.
tms
Jan 5, 2012, 1:13:52 AM1/5/12
to
On Wednesday, January 4, 2012 8:23:09 PM UTC-5, Wes Groleau wrote:
> On 01-04-2012 12:32, tms wrote:
> > The language is not error-prone, careless programmers are. Agreed,
> > though, that C requires more care than some other languages. However,
>
> All programmers are error-prone.
But some are more so than others.
> On 01-04-2012 12:32, tms wrote:
> > The language is not error-prone, careless programmers are. Agreed,
> > though, that C requires more care than some other languages. However,
>
> All programmers are error-prone.
> Some languages protect against this,
> others exaggerate it. C is one of the worst of the latter.
But to return to the present subject, the major selling point of C implementations of file formats for the Commons is its portability, not just across platforms today, but through time as well. You can bet that in 50 years there will still be C compilers, and that you will be able to compile today's code without too much trouble.
--
T.M. Sommers
Bob Melson
Jan 5, 2012, 2:15:44 AM1/5/12
to
On Wednesday 04 January 2012 19:40, Wes Groleau
(Grolea...@FreeShell.org) opined:
> On 01-04-2012 21:12, Bob Melson wrote:
>> Years ago one of my profs likened C to a Formula-I race car - built low
>> and close to the ground, able to corner well but deadly to the novice or
>
> Here's an interesting comparison between two languages:
>
<snip>
I'm certainly NOT going to argue that C - or any language - is ideal for
every application - it isn't true and we all know it.
At the same time, my (limited) exposure to ADA has convinced me of two
things - it was the PL-I of the 90s and suffered, as did PL-I, from having
been designed by a committee attempting to create something to satisfy
everybody. (What's the saying? A camel's a horse designed by a
committee - same thing applies to ADA, seems to me.)
Where we might have a meeting of the minds is over C++, which, IMO, has
bloated a decent language out of recognition and usability. But that's MY
prejudice.
The truth is that there's no one perfect programming language. And, having
said that, I'll now fold my tent and fade back into the shadows.
Swell Ol' Bob
(Grolea...@FreeShell.org) opined:
> On 01-04-2012 21:12, Bob Melson wrote:
>> Years ago one of my profs likened C to a Formula-I race car - built low
>> and close to the ground, able to corner well but deadly to the novice or
>
> Here's an interesting comparison between two languages:
>
I'm certainly NOT going to argue that C - or any language - is ideal for
every application - it isn't true and we all know it.
At the same time, my (limited) exposure to ADA has convinced me of two
things - it was the PL-I of the 90s and suffered, as did PL-I, from having
been designed by a committee attempting to create something to satisfy
everybody. (What's the saying? A camel's a horse designed by a
committee - same thing applies to ADA, seems to me.)
Where we might have a meeting of the minds is over C++, which, IMO, has
bloated a decent language out of recognition and usability. But that's MY
prejudice.
The truth is that there's no one perfect programming language. And, having
said that, I'll now fold my tent and fade back into the shadows.
Swell Ol' Bob
Ian Goddard
Jan 5, 2012, 4:30:23 AM1/5/12
to
tms wrote:
> On Jan 4, 5:13 am, Richard Smith<rich...@ex-parrot.com> wrote:
>> On 03/01/12 18:36, tms wrote:
>>
>> But suppose I have a resource in some proprietary format that I can no
>> longer read (or at least convert), is it better that I can upload it on
>> the off-chance that someone can convert it before it's too late?
>
> Best not to get into that situation in the first place.
You start with what you've got, not what you'd like to have.
> On Jan 4, 5:13 am, Richard Smith<rich...@ex-parrot.com> wrote:
>> On 03/01/12 18:36, tms wrote:
>>
>> But suppose I have a resource in some proprietary format that I can no
>> longer read (or at least convert), is it better that I can upload it on
>> the off-chance that someone can convert it before it's too late?
>
> Best not to get into that situation in the first place.
> Routinely backup your data to a more portable format, if possible, or at least
> to Gedcom.
format. As I've said in a different reply that file becomes your
primary evidence. You need to preserve it as such in just the same way
that your local archive preserves a crumbling Tudor parish register.
Richard Smith
Jan 5, 2012, 7:44:16 AM1/5/12
to
I've been trying to find a good website that gives a good introduction
to the types of technologies I'm proposing using in the Commons and I
have just stumbled across one.
http://linkeddatabook.com/
(You can buy it as a book from Amazon if you prefer.)
This answers a number of questions like why URIs rather than UUIDs (see
§2 in general and §2.2 in particular), as well as being a fairly
readable, non-technical overview of the subject.
I have been envisaging the Genealogical Commons as a big collection of
Linked Data conforming to the conventions and standards given in that
website / book, which will allow us to use the many existing tool for
dealing with Linked Data. The Commons isn't *just* Linked Data because
we need to deal carefully with issues of trust and provenance, as well
as needing specific vocabulary to talk about genealogy. But if we can
use the existing Linked Data infrastructure, we're well on our way.
Hopefully this will explain some of the design decisions I've been
making in cases where a priori there are several similarly good
decisions (of which URIs vs UUIDs is one).
--
Richard Smith
to the types of technologies I'm proposing using in the Commons and I
have just stumbled across one.
http://linkeddatabook.com/
(You can buy it as a book from Amazon if you prefer.)
This answers a number of questions like why URIs rather than UUIDs (see
§2 in general and §2.2 in particular), as well as being a fairly
readable, non-technical overview of the subject.
I have been envisaging the Genealogical Commons as a big collection of
Linked Data conforming to the conventions and standards given in that
website / book, which will allow us to use the many existing tool for
dealing with Linked Data. The Commons isn't *just* Linked Data because
we need to deal carefully with issues of trust and provenance, as well
as needing specific vocabulary to talk about genealogy. But if we can
use the existing Linked Data infrastructure, we're well on our way.
Hopefully this will explain some of the design decisions I've been
making in cases where a priori there are several similarly good
decisions (of which URIs vs UUIDs is one).
--
Richard Smith
Richard Smith
Jan 5, 2012, 8:05:03 AM1/5/12
to
Ian Goddard wrote:
>> More generally, by signing a statement you are guaranteeing its truth.
>> This is the reason why I did not sign the marriage certificate, but
>> rather put a SHA1 hash into a document that I signed. Had I GPG signed
>> the marriage certificate, I would have been guaranteeing its contents to
>> be true. I don't want to do that because I'm pretty sure the groom was
>> actually 24, not 22, and the bride was only 19 rather than 20. Even if I
>> didn't know of any errors, I wouldn't wish to make a guarantee about
>> something so long ago.
>
> But what would you be guaranteeing? The document itself bears a
> signature saying it's a true copy of the register. If you signed the
> upload you'd be saying nothing more than that the image was a true copy
> of that piece of paper.
OK. So what if instead of scanning the image, I had in fact downloaded
the image from the web or elsewhere in the Commons, digitally enhanced
it somehow, and uploaded the new copy. In that case, shouldn't signing
the image mean that I am stating it is a true copy of the other image,
rather than a true copy of original certificate? But given just the
image and a signature I cannot tell which of these possibilities is
meant and may well come to an incorrect conclusion.
If we want to indicate that the image is a true copy of the physical
certificate, then I should put a statement to that effect in an RDF file
and sign that. This way it's unambiguous what the signature applies to,
instead of being a slightly nebulous implicit statement.
Whether statements about provenance belong the same or a separate RDF
document and the precise format for making the statement are probably a
topic for a later discussion.
From a cryptographic point of view, signing an SHA1 hash of the data
and signing the original data are both valid ways of ensuring the data
is as it was produced. So we don't lose anything by hashing and signing
instead of directly signing.
> We need to bear in mind that there are a whole lot of layers here. The
> document is evidence.
>
> If you extract what it says, including ages, that's analysis.
>
> If you consider the content - for instance, if you evaluate the truth of
> the alleged ages or if you identify the people named with people named
> in other documents - that's interpretation. In fact your evaluation of
> the truth of the ages depends on your identification of the people; they
> could, in fact, be different people whose true ages were as stated.
My mental picture uses slightly different terminology: what you call
analysis, I call extraction and what you call interpretation I call
analysis. But that aside, I agree.
> Because these are different types of information I consider they should
> belong in different classes of object in any system that stores them.
Agreed.
> This is a further reason for wrapping the image inside the same XML that
> contains its metadata: when an application downloads a file it will be
> clear from the wrapper what internal class of object should be created
> to hold it. This, in passing, is one of the major problems with GEDCOM -
> it makes no such distinction.
But I don't agree here. You see, I think it's more important that a
user can download and view the file itself (without the wrapper) using
tools that don't know about the Commons. For example, a web browser.
In any case, I'm not happy putting the image inside the XML largely
because there are no good technologies to do that. Yes, I could base-64
encode it and stick it in data: scheme URL attribute (or in a custom
element I invent for the purpose). But that's very verbose and makes
life hard for anyone wanting to view or edit the XML in a text editor.
If we want to use existing sites such as the Wikimedia Commons as
storage servers for our Commons then we need the resources to be in a
format that suitable for them too. They are not going to accept images
embedded in XML, but they will accept the images themselves assuming
they're within the scope of the project. As storage is likely to be the
expensive part of the Commons, being able to use existing projects to do
it is vital.
--
Richard Smith
>> More generally, by signing a statement you are guaranteeing its truth.
>> This is the reason why I did not sign the marriage certificate, but
>> rather put a SHA1 hash into a document that I signed. Had I GPG signed
>> the marriage certificate, I would have been guaranteeing its contents to
>> be true. I don't want to do that because I'm pretty sure the groom was
>> actually 24, not 22, and the bride was only 19 rather than 20. Even if I
>> didn't know of any errors, I wouldn't wish to make a guarantee about
>> something so long ago.
>
> But what would you be guaranteeing? The document itself bears a
> signature saying it's a true copy of the register. If you signed the
> upload you'd be saying nothing more than that the image was a true copy
> of that piece of paper.
OK. So what if instead of scanning the image, I had in fact downloaded
the image from the web or elsewhere in the Commons, digitally enhanced
it somehow, and uploaded the new copy. In that case, shouldn't signing
the image mean that I am stating it is a true copy of the other image,
rather than a true copy of original certificate? But given just the
image and a signature I cannot tell which of these possibilities is
meant and may well come to an incorrect conclusion.
If we want to indicate that the image is a true copy of the physical
certificate, then I should put a statement to that effect in an RDF file
and sign that. This way it's unambiguous what the signature applies to,
instead of being a slightly nebulous implicit statement.
Whether statements about provenance belong the same or a separate RDF
document and the precise format for making the statement are probably a
topic for a later discussion.
From a cryptographic point of view, signing an SHA1 hash of the data
and signing the original data are both valid ways of ensuring the data
is as it was produced. So we don't lose anything by hashing and signing
instead of directly signing.
> We need to bear in mind that there are a whole lot of layers here. The
> document is evidence.
>
> If you extract what it says, including ages, that's analysis.
>
> If you consider the content - for instance, if you evaluate the truth of
> the alleged ages or if you identify the people named with people named
> in other documents - that's interpretation. In fact your evaluation of
> the truth of the ages depends on your identification of the people; they
> could, in fact, be different people whose true ages were as stated.
My mental picture uses slightly different terminology: what you call
analysis, I call extraction and what you call interpretation I call
analysis. But that aside, I agree.
> Because these are different types of information I consider they should
> belong in different classes of object in any system that stores them.
Agreed.
> This is a further reason for wrapping the image inside the same XML that
> contains its metadata: when an application downloads a file it will be
> clear from the wrapper what internal class of object should be created
> to hold it. This, in passing, is one of the major problems with GEDCOM -
> it makes no such distinction.
But I don't agree here. You see, I think it's more important that a
user can download and view the file itself (without the wrapper) using
tools that don't know about the Commons. For example, a web browser.
In any case, I'm not happy putting the image inside the XML largely
because there are no good technologies to do that. Yes, I could base-64
encode it and stick it in data: scheme URL attribute (or in a custom
element I invent for the purpose). But that's very verbose and makes
life hard for anyone wanting to view or edit the XML in a text editor.
If we want to use existing sites such as the Wikimedia Commons as
storage servers for our Commons then we need the resources to be in a
format that suitable for them too. They are not going to accept images
embedded in XML, but they will accept the images themselves assuming
they're within the scope of the project. As storage is likely to be the
expensive part of the Commons, being able to use existing projects to do
it is vital.
--
Richard Smith
Tony Proctor
Jan 5, 2012, 8:17:07 AM1/5/12
to
"Richard Smith" <ric...@ex-parrot.com> wrote in message
news:9mlk90...@mid.individual.net...
clash markedly with the XML standard in the area of URI namespaces.
I would say there's a grey scale from entity "addresses" (i.e. URLs) to
entity "names" (i.e. URNs and name-like URIs) that includes entity
"descriptions" (as in their section 2.3). Their "descriptions" are part-way
between the two since the URI acts as a URL locating a secondary document
entity which describes the primary entity (e.g. a Person).
W3C are a way off still from incorporating this :-)
Tony Proctor
Richard Smith
Jan 5, 2012, 8:28:43 AM1/5/12
to
Ian Goddard wrote:
>> Sorry to pick you up here Ian but as a compiler-writer for many years
>> I can tell you that issues such as variable-length and commonality of
>> the left-hand (or right-hand) portion are not relevant to indexing.
>
> It depends on what sort of index you're thinking of. The commonality
> issue can make for an inefficient B-tree index in an RDBMS. Which I can
> tell you as a DBA for many years :)
Fortunately some databases support HASH indexes, and they are probably
>> Sorry to pick you up here Ian but as a compiler-writer for many years
>> I can tell you that issues such as variable-length and commonality of
>> the left-hand (or right-hand) portion are not relevant to indexing.
>
> It depends on what sort of index you're thinking of. The commonality
> issue can make for an inefficient B-tree index in an RDBMS. Which I can
> tell you as a DBA for many years :)
better suited to indexing things like URLs.
> The issue with variable -and worse, undefined - length isn't so much
> indexing but the need for a longish column length to be specified just
> to be on the safe side (you don't want to truncate your key!), which
> again is an issue for RDBMS.
variable length. If there are specific inefficiencies with variable
length database records that you need to avoid, which there may well be,
you can stick it in a separate URL table and link to it via an internal
auto-incrementing integer.
--
Richard Smith
Richard Smith
Jan 5, 2012, 9:29:49 AM1/5/12
to
Tony Proctor wrote:
> Interesting. The recommendations for URIs being deferencable (2.3) seems to
> clash markedly with the XML standard in the area of URI namespaces.
Not really. All the XML namespace spec says is that it makes no
> Interesting. The recommendations for URIs being deferencable (2.3) seems to
> clash markedly with the XML standard in the area of URI namespaces.
requirement that the namespace URI should be dereferencable. Current
good practice is to make them dereferencable, and most do dereference to
something useful both to a human and a machine. Give it a go.
Take the XML namespace for XHTML, <http://www.w3.org/1999/xhtml>, for
instance. That ordinarily dereferences to an HTML page directing a
human reader to various related resources defining the language. That
HTML header also contains GRDDL attributes in the header to allow a
machine to get RDF about the namespace. But if I use HTTP content
negotiation to request application/xml from the URL, I get an XML schema
for the domain (which right at the moment is a placeholder schema).
> I would say there's a grey scale from entity "addresses" (i.e. URLs) to
> entity "names" (i.e. URNs and name-like URIs) that includes entity
> "descriptions" (as in their section 2.3). Their "descriptions" are part-way
> between the two since the URI acts as a URL locating a secondary document
> entity which describes the primary entity (e.g. a Person).
>
> W3C are a way off still from incorporating this :-)
are now W3C recommendations: XML Schema datatypes, RDF, RDF Schema,
XHTML+RDF, GRDDL, OWL, SPARQL, and a bunch of others. The main points
where that Linked Data document goes beyond the W3C idea of the Semantic
Web is that it discusses how to locate more information about resources.
However those additional recommendations are almost precisely the ones
given in this W3C note:
http://www.w3.org/TR/swbp-vocab-pub/
--
Richard Smith
It is loading more messages.
0 new messages