Re-writing Genealogy::Gedcom::Date

[Ron Savage]

Hi All

I've re-written Genealogy::Gedcom::Date, and V 2.00 will be released
next week. I'm about to start updating the docs.

There are massive internal changes, and hence the V 2.00.

Gone are these methods: parse_approximate_date(), parse_date_escape(),
parse_datetime(), parse_date_period(), parse_date_range(),
parse_date_value(), parse_interpreted_date(), debug(), method_index(),
months_in_gregorian() and style().

All calls are now thru parse(). And calendar() replaces and extends style().

So, why do this? Because I have finally used Marpa [1] to specify the
BNF of dates as per the Gedcom V 5 spec.

However, the latter only includes a tiny number of calendars: french,
gregorian, hebrew and julian. I've decided to accept suggestions re
other languages.

So, if you're interested in support for others, send me a list of the
abbreviated month names in your language. Include the day month year
order, and how dates BC are written.

E.g.: This is part of the new Marpa-style BNF [2]. Put very simply,
'::=' declares a rule and '~' declares a lexeme.

french_date ::= year_bc
| year
| french_r_month year
| day french_r_month year

year_bc ::= year bc

year ::= number

bc ~ 'bc'
| 'b.c.'

day ~ digit
| digit digit

digit ~ [0-9]

french_r_month ~ 'vend' | 'brum' | 'frim' | 'nivo' | 'pluv'
| 'vent' | 'germ' | 'flor' | 'prai' | 'mess'
| 'ther' | 'fruc' | 'comp'

number ~ digit+

:discard ~ whitespace
whitespace ~ [\s]+

The special lexeme :discard hopefully explains why the definition of bc
does not need 'B C' and 'B. C.'. And, no, discarding whitespace does
/not/ affect recognition of tokens which have nothing but whitespace
between them.

You don't have to provide BNF, just enough for me to construct the BNF.

As for existing code, I plan to release new versions of Gedcom::Date V
0.90 and Genealogy::Gedcom::Date V 1.17 next week. But, no bugs will be
fixed, ever, in those modules. The update will only say those modules
are deprecated, and will be removed from CPAN some time around
2017-01-01 (sic). So you have 1+ year's notice. Of course the code will
still be available via backpan [3].

If you don't want to upgrade, them let me know. I'll preserve
Gedcom::Date on CPAN only if it's of monumental significance, but be
warned, Genealogy::Gedcom::Date V 2.00 will be released, and eventually
V 1.17 will be deleted.

[1] http://savage.net.au/Marpa.html. No, I did not write Marpa.

[2] Another warning! Marpa's BNF only bears a slight resemblance to the
original BNF.

See https://metacpan.org/pod/distribution/Marpa-R2/pod/Scanless/DSL.pod
for an introduction to the vast array of new features.

See also http://savage.net.au/Marpa/html/Marpa.R2.DSL.Structure.html.

[3] http://gitpan.integra.net/

--
Ron Savage - savage.net.au

[Michael Ionescu]

Hi Ron,

perhaps someone else can improve upon this, but here's a firt shot at German.


german_date ::= any_year
| german_r_month dot any_year
| day dot german_r_month dot any_year
| german_r_month any_year

dot ~ '.'

any_year ::= year | year_bc

year_bc ::= year german_bc

year ::= number

german_bc ~ 'vc'
| 'v.c.'
| 'v.chr.'
| 'vchr'
| 'vuz'
| 'v.u.z.'

day ~ digit
| digit digit

digit ~ [0-9]

german_r_month ~ 'jan' | 'feb' | 'mär' | 'maer' | 'mrz' | 'apr' | 'mai'
| 'jun' | 'jul' | 'aug' | 'sep' | 'sept' | 'okt'
| 'nov' | 'dez'

[Ron Savage]

Hi Nokolay

On 14/10/15 13:58, Nikolay Mishin wrote:
> Ron, very good job, thanks,
> do you have converters from gedvom to csv?
> Something wrong with mine (going with Ftree
> https://metacpan.org/source/MISHIN/FamilyTreeInfo-2.3.26/script/convertFormat.pl)

Sorry. I've don't believe I've ever heard of gedvom. I'll investigate,
but I won't promise anything!

[Ron Savage]

Hi Michael

Excellent!

PS Everybody
I've subscribed to the perl-gedcom list, so no need to duplicate 'To'.

[Ron Savage]

Hi Michael

On 14/10/15 15:19, Michael Ionescu wrote:

> german_r_month ~ 'jan' | 'feb' | 'mär' | 'maer' | 'mrz' | 'apr' | 'mai'
> | 'jun' | 'jul' | 'aug' | 'sep' | 'sept' | 'okt'
> | 'nov' | 'dez'

Is there some particular reason you've used 'german_r_month' and not
'german_month'?

I used _r_ for French because that's what the Gedcom spec uses.

[Mike Elston]

… because it's the French _Republican_ Calendar — as I'm sure you all know, they're actually different months from the Gregorian/Julian months, not just different names for the G/J months.

/m

[Ron Savage]

Hi Mike

On 14/10/15 21:05, Mike Elston wrote:
> … because it's the French _Republican_ Calendar — as I'm sure you all know, they're actually different months from the Gregorian/Julian months, not just different names for the G/J months.

Yes. That's for French. But German?

[Mike Elston]

Hi Ron,

There is some confusion here between /calendars/ and /names of months/.

The French Republican (or French Revolutionary) calendar was adopted in October 1793 during the French Revolution, and was abandoned in January 1806. The year is divided into 12 months of 30 days (3 weeks of 10 days) each. The remaining 5 to 6 days in the year are grouped at the end if the year, known as Jours Complementairs, and are holidays. Only years 1 to 14 (22 September 1792 to 22 September 1806 in the Gregorian calendar) are supported by the Gedcom definition. The Jours Complementairs are treated as a 13th month. These are the 13 months you have named as 'french_r_month'.

So, there is no "BC' in the French Revolutionary calendar. The only valid years are 1 to 14. For dates before year 1 in the FRC (ie before 22 Sep 1792, which in the new calendar was 1 Vendémiaire an I), or after year 14, one uses the usual (Julian or) Gregorian calendar.

What you're asking contributors for is quite a different thing: /localizations/ of the 12 months of the Julian and Gregorian calendars.

As a French localization, for example, you might have:

french_month ~ 'jan' | 'fév' | 'mar' | 'avr' | 'mai' | 'jun' | 'jul' | 'aou' | 'sep' | 'oct' | 'nov' | 'déc'

or (if you don't want to use accented letters):

french_month ~ 'jan' | 'fev' | 'mar' | 'avr' | 'mai' | 'jun' | 'jul' | 'aou' | 'sep' | 'oct' | 'nov' | 'dec'

or, as some French sources prefer not to restrict the abbreviations to 3 letters:

french_month ~ 'jan' | 'fév' | 'mar' | 'avr' | 'mai' | 'juin' | 'juil' | 'aout' | 'sept' | 'oct' | 'nov' | 'déc'

Hope I haven't just confused things even more...

/mike

[Ron Savage]

Hi Mike

Since I know nothing about the detail you supply, I'm wholly dependent
on this type of feedback.

More below.

On 15/10/15 09:33, Mike Elston wrote:
> Hi Ron,
>
> There is some confusion here between /calendars/ and /names of months/.
>
> The French Republican (or French Revolutionary) calendar was adopted in October 1793 during the French Revolution, and was abandoned in January 1806. The year is divided into 12 months of 30 days (3 weeks of 10 days) each. The remaining 5 to 6 days in the year are grouped at the end if the year, known as Jours Complementairs, and are holidays. Only years 1 to 14 (22 September 1792 to 22 September 1806 in the Gregorian calendar) are supported by the Gedcom definition. The Jours Complementairs are treated as a 13th month. These are the 13 months you have named as 'french_r_month'.
>
> So, there is no "BC' in the French Revolutionary calendar. The only valid years are 1 to 14. For dates before year 1 in the FRC (ie before 22 Sep 1792, which in the new calendar was 1 Vendémiaire an I), or after year 14, one uses the usual (Julian or) Gregorian calendar.
>
> What you're asking contributors for is quite a different thing: /localizations/ of the 12 months of the Julian and Gregorian calendars.

It's a juggling act. I want to support what's literally in the Gedcom
doc, and simultaneously follow the Perl dictum: Be liberal in what you
accept and strict in what you send.

Of course, ultimately, what matters is what the code must accept as
presumably being from a Gedcom file, and hence compatible (supply own
definition thereof) - to some degree - with other Gedcom programs.

> As a French localization, for example, you might have:
>
> french_month ~ 'jan' | 'fév' | 'mar' | 'avr' | 'mai' | 'jun' | 'jul' | 'aou' | 'sep' | 'oct' | 'nov' | 'déc'
>
> or (if you don't want to use accented letters):
>
> french_month ~ 'jan' | 'fev' | 'mar' | 'avr' | 'mai' | 'jun' | 'jul' | 'aou' | 'sep' | 'oct' | 'nov' | 'dec'
>
> or, as some French sources prefer not to restrict the abbreviations to 3 letters:
>
> french_month ~ 'jan' | 'fév' | 'mar' | 'avr' | 'mai' | 'juin' | 'juil' | 'aout' | 'sept' | 'oct' | 'nov' | 'déc'
>
> Hope I haven't just confused things even more...

Nope. Well, not much :-). The question remains: What do users of the
code intend to supply as data? I can add all these (Marpa handles
ambiguous input), but it requires more code be written (if the calendar
is not provided by the Gedcom escape syntax).

Can you provide a few dates I can add to the test data? I only have
Gregorian data ATM. This request applies to any language which the code
purports to support.

[Ron Savage]

Hi

It's taken longer than expected to write the grammar to match the Gedcom
doc, and this is just for dates....

Nevertheless, I've almost certainly finished the Gregorian and Julian
parts, and will next do the calendar escapes. After that comes the new
docs and tests.

Later, I'll add the German and French dates. I may release a version to
CPAN before adding these 2 languages.

[Ron Savage]

Hi Mike

On 14/10/15 21:05, Mike Elston wrote:

> … because it's the French _Republican_ Calendar — as I'm sure you all know, they're actually different months from the Gregorian/Julian months, not just different names for the G/J months.
>
> /m
>
> On 14 Oct 2015, at 10:04, Ron Savage wrote:
>
>> Hi Michael
>>
>> On 14/10/15 15:19, Michael Ionescu wrote:
>>
>>> german_r_month ~ 'jan' | 'feb' | 'mär' | 'maer' | 'mrz' | 'apr' | 'mai'
>>> | 'jun' | 'jul' | 'aug' | 'sep' | 'sept' | 'okt'
>>> | 'nov' | 'dez'
>>
>> Is there some particular reason you've used 'german_r_month' and not 'german_month'?

To repeat my question: These are German /republican/ names, right?

[Mike Elston]

Hi Ron,

As I said before in another message, your question is the result of confusion between /calendars/ and /names of months/.

There's no such thing as "German republican names" for months, any more than there are "German republican months" (or even "German months"). There are just the German names for the usual months that most of the Western world use, the months of the Julian and Gregorian calendars. Whereas the "French Republican names" are the names of the months of the French Republican calendar.

How can I make this clear?

The abbreviations supplied by Michael Ionescu are simply the common German-language abbreviations for their names for the months of the Julian and Gregorian calendars. In other words, the date which is written "1 Mar 1970" by an English speaker might be written "1 Mrz 1970" by someone speaking German. But they're the /same/ date.

It's quite different from your 'french_r_month' array, which is a list of abbreviations of the names of the months of the French Revolutionary or French Republican /calendar/, and bear no similarity at all to the months of the Gregorian and Julian calendars. They're different months. Vendémiaire, for example, in the French Republican calendar, covered roughly the last week of September and the first 3 weeks of October in the Julian calendar.

Same goes for the Hebrew calendar (which, unlike the French Republican calendar, is still in use). It's a different /calendar/.

To try another analogy, the German for 'cat' is 'Katze'. They're the /same/ animal. But a 'dog' is a quite different animal.

HTH,

/mike

[Marcel S. Henselin]

Hey guys

I am German and I could give some support. Only the French had during the time of the Revolution a different calendar. So did the Russian.

German month names are quite easy and nearly the same as in English. Because of the umlauts the abbreviations have been internationalized.

March would be März and the abbr. would be Mär but because other countries don't have umlauts the Mrz is used.

Regards
Marcel

Von meinem Sony Xperia™-Smartphone gesendet

---- Ron Savage schrieb ----

[Mike Elston]

Hi Marcel,

I wan't aware that the Russians ever had a different calendar, except that they continued to use the Julian calendar longer than almost every other Western country.

The Gregorian calendar — which simply has a different scheme of leap years from the old Julian calendar, but the same months, and the same lengths of the different months — was instituted in 1582; England didn't adopt it until 1751, by which time the Julian calendar was 11 days out compared to the Gregorian calendar (it was in use in Scotland before that, which meant that the date was different in England from what it was in Scotland until 1751!); Russia continued to use the Julian calendar until 1918, when it converted to the Gregorian calendar, which was by then used by (nearly) all of the rest of Europe.

The numbering of the years has varied too, but that doesn't need to concern us here.

The Russians did have some strange ways of using the calendar (like 5-day weeks and 6-day weeks between the two World Wars), and a Russian Revolutionary calendar modelled on the old French one was once proposed (but rejected), but as far as I know, the dates in the year (the months and the number of days within them) have always been those of the Julian or Gregorian calendar.

Unless you know different, in which case I will be happy to be corrected!

/mike

[Michael Ionescu]

Hi all,

sorry about the confusion I caused by keeping the _r_ in some of the names without realizing its meaning.

There is no German Republican calendar and I did not intend to give German translations for the month names of the French Republican calendar. I simply meant to aid with the German syntax and month names for the Gregorian calendar typically used in the western world today.

The Month names for this calendar may differ from the ones I gave, for variations or dialects of the German language such as Swiss German or Austrian German or even Bavarian or Alsatian. Heck, even the syntax might.

Given this, it might be a good idea to revise your naming scheme. I would suggest using one element for the language and one for the location of language use (as are used in a typical unix locale name) and an additional element for the calendar.
Ron's example might thus be named fr_FR_rv and mine de_DE_gr or some such.

Ron, it might be helpful if you explained whether you are just aiming for syntax checking or whether you intend to parse these dates and possibly convert them or do other calculations on them. For the former the concept of different calendars is relatively negligible, however for the latter it is a must.

Michael

[Mike Elston]

On 4 Nov 2015, at 12:29, Michael Ionescu wrote:

> Ron, it might be helpful if you explained whether you are just aiming for syntax checking or whether you intend to parse these dates and possibly convert them or do other calculations on them. For the former the concept of different calendars is relatively negligible, however for the latter it is a must.

Very good point. Thanks Michael.

[Ron Savage]

Hi

I should add: Thanx to all who have been replying to this thread.

[Ron Savage]

Hi Mike

On 04/11/15 21:30, Mike Elston wrote:
> Hi Ron,
>
> As I said before in another message, your question is the result of confusion between /calendars/ and /names of months/.
>
> There's no such thing as "German republican names" for months, any more than there are "German republican months" (or even "German months"). There are just the German names for the usual months that most of the Western world use, the months of the Julian and Gregorian calendars. Whereas the "French Republican names" are the names of the months of the French Republican calendar.

Thanx for the detail.

The above ("There's no such thing...") is the only thing that matters.

You used 'german_r_month' (orginally, below) and not 'german_month'.
That confused me.

It's a matter of what I put in the grammar, and I've gone with
'german_month' :-).

I just did not wish to find out one day that I needed both a
german_month list and a german_r_month list! That would just mean an
update, but rather worse would be if both existed but the only one I
implemented had the 'wrong' name.

So, the French calendar escape is 'French r' or '@#dFrench r@', both
case-insensitive.

Using 'French r' leaves 'French' free for later adoption.

And for German it is 'German' or '@#dGerman@', likewise case-insensitive.

And I /do/ realize the simple (@#d...@-free) versions will presumably
not be compatible with other software.

[snip]

>>>>
>>>>> german_r_month ~ 'jan' | 'feb' | 'mär' | 'maer' | 'mrz' | 'apr' | 'mai'
>>>>> | 'jun' | 'jul' | 'aug' | 'sep' | 'sept' | 'okt'
>>>>> | 'nov' | 'dez'

This gives me an opportunity for a progress report:

shell> prove -l t/
t/English.t .. ok
t/French.t ... ok
t/German.t ... ok
All tests successful.
Files=3, Tests=304, 2 wallclock secs ( 0.07 usr 0.00 sys + 1.96 cusr
0.03 csys = 2.06 CPU)
Result: PASS

The English tests use both Gregorian and Julian. The French and German
tests mix their respective calendars with both Gregorian and Julian.

This means there are so far 304 pairs of input and output for you to study.

I'll do Hebrew today or tomorrow. After that, I'll release the module
since the docs are 98% complete.

Now, from your other message:

> Ron, it might be helpful if you explained whether you are just aiming
> for syntax checking or whether you intend to parse these dates and >
> possibly convert them or do other calculations on them. For the > > >
> former the concept of different calendars is relatively negligible, >
> however for the latter it is a must.

The code parses dates, and accepts the Gedcom defined calendar escapes.

Gedcom mentions the German language but not the German calendar.
Nevertheless, I have extended the grammar to accept German dates.

The output from the parse() method can be passed into the normalize()
method which will reconstruct the date input, with various bits
normalized. E.g. 'Int' or 'Interpreted' comes back as 'INT'.

As for calculations, you'll have to build up a date string from the
output of parse() and pass that to DateTime (or whatever).

In Perl there is a perhaps-unfortunate plethora of date-oriented
modules. My module's 'See Also' section lists a few, and has a set of
links to various articles which mention many others (especially in the
comments sections of the articles).

[John Washburn]

Ron:

 

You may have to consider support for Astronomical Julian Days.  Hear me out

 

I have worked in logistics software for some many years and date calculations are the bedevilment of anything that touches the notion of date/time.  For example, the implementations of native DateTime objects in the major databases (Oracle, Sybase, Ingress, SQL Server, Informix, DB2, etc.) are all subtle difference; e.g. date time model for these is the number of seconds since second ZERO but the resolution on the second (3 or 6 places) varies and the exact date time of the second zero (in the proleptic Gregorian Calendar) varies. Also is NULL (no date information) supported or not varies from DB to DB. Similarly the underlying implementation of date time for Perl, Java, C#, etc has the same Modeling and NULL-bility variations.

 

It took me some time to realized that I was confusing date with calendar.  Date is a particular point in time.  Calendar is how to describe that point in time. 

 

We found the most uniform way to deal with date and times was pick a core calendar and to store the string representation on the database.  We chose to store a string on the data base in ISO_8601 format of the proleptic Gregorian Calendar with UTC as the date portion down to the second.  This sufficed for our dates; lots, holds, receipts, shipment, orders, etc. and it supported NULLs (e.g. the ship date is NULL because the order has not shipped yet)

 

Dates math was then

1)  convert from proleptic Gregorian date to Astronomical Julian Days,

2)  Do your date math (e.g. subtract two dates or add 17 days)

3)  Covert from Astronomical Julian Days to proleptic Gregorian date

4)  Store ISO ISO_8601 representation of the proleptic Gregorian date

 

 

Perhaps GEDCOM dates needs to have at its heart dates in the form of Astronomical Julian Days (with suitable qualifiers for (abt, circa, before, etc) and suitable qualifiers for resolution (e.g. nearest decade, nearest year, nearest month). Converting into, out of, and between calendars becomes simplified, because the astronomical community has wrestled with this issue for more than 2 centuries and developed algorithms to convert all of the know calendars of man into Astronomical Julian Days and back to a local calendar.

 

I propose the internal structure of a Gedcom date be an object with Astronomical Julian Days and two kinds of qualifiers: Certainty (exact, abt, before, after, unknown, etc.) and Resolution (NULL, Nearest century, decade, year, month, day, etc.).

 

Parsing a string then is to convert from a string representation from a local calendar (Gregorian, Julian, French republican, Hebrew, Islamic, Mayan, etc.) to the internal Julian day format.

 

Date math (e.g. life span) is done with internal representation of genealogically qualified Juliann days.  Example Died before ZZZZ minus born after YYYY is and either an age of about AAAA or and age of more than BBBB

 

Printing a date string is to convert from the internal Julian day format to a string representation using the specified local calendar (Gregorian, Julian, French republican, Hebrew, Islamic, Mayan, etc.).

 

There is no notion of parsing or printing a date without also specifying the local calendar by which the string is parse or printed.  Also NULL dates (e.g. empty string) are supported so you can signify the absence of date information; e.g.  have a birth date and a NULL death date indicates (possibly) a living person

 

This presents more foundational work up front for GEDCOM Dates, but it permit the expansion to more local calendars over time and provides for uniform date math regardless of the calendar used to describe the date.  Date, date math, creating a string representation, parsing a string representation, and supported local calendars are decoupled from each other.

 

I hope this helps more than confuses.  I have wrestled with dates and date math off and on since 1996 and the above flows from the mistakes and successes I have had.

[Ron Savage]

Hi John

Thanx for your informative email.

Luckily my code is not intended to do any calculations. It simply
accepts, parses and returns date strings - which must be provided in a
given format.

I did read various parts from your references and understand they are
complex for good reason. The whole topic is a marvellous example of a
can of worms.

Regarding writing new code:

As for ISO 8601, I could buy a copy of the doc for $AUD235, or start
with a similar standard, RFC 3339 for free.

As for the BNF on p 12 of RFC 3339, it resembles the BNF for "RFC4514:
Lightweight Directory Access Protocol (LDAP): String Representation of
Distinguished Names." [1] for which I have already written a
Marpa::R2-based parser [2]. So, definitely doable, but fortuitously not
needed at this point in time.

[1] https://www.ietf.org/rfc/rfc4514.txt

[2] https://metacpan.org/pod/X500::DN::Marpa

[John Washburn]

O yeah. Dates and time are a can a worms. Moreover the worms under study
wiggle all the more fiercely the more closely you focus on them (i.e.
examine the details of dates and calendar representations). The key
revelation for me 15 years ago was that date and calendar are not the same.

Sorry I mis-understood the scope of the date package. Removing date math
does simplify things greatly and eliminates the need to adopt a core
calendar representation for the internal description of dates.

I should have been more clear in my email that parsing or printing a string
representation of a date requires a both a calendar and a culture. For
example it is possible to have date represented by the Gregorian calendar in
French, German, or English just as it would be possible to use the Hijri
calendar in English or Tamil to represent or parse a date. The names of the
months have to be translated from Arabic to English or Tamil.

-----Original Message-----
From: Ron Savage [mailto:r...@savage.net.au]
Sent: Friday, November 06, 2015 5:22 PM
To: jo...@washburnresearch.org; perl-...@perl.org
Subject: Re: Re-writing Genealogy::Gedcom::Date

[Ron Savage]

Hi

Genealogy::Gedcom::Date V 2.01 is now on CPAN.

Docs: https://metacpan.org/pod/Genealogy::Gedcom::Date.

[Ron Savage]

Hi

I doubt anyone's using it :-) but I've just updated Genealogy::Gedcom to
use the newly re-written Genealogy::Gedcom::Data V 2.01.

See https://metacpan.org/release/Genealogy-Gedcom.

Also, there's a new UTF-8 sample in data/sample.7.ged. This file was
constructed by parsing the ANSEL sample of Tamura Jones' web site. So
it's not in ANSEL, but it is in UTF-8.