ADB export file format - partly public domain

[Alois Treindl]

On the Swiss Ephemeris mailing list this question was asked:

On 20.06.16 16:07, Bart Lidofsky  wrote:

What is the current legal status of the ADB file format? Please note
that I am talking about the format itself, and not any data that is
stored using that format. I am especially interested if it is public
domain or GPL'd. The reason is that I have been thinking about, for
several years, creating a flexible data standard for transferring
astrological data, including calculated data (notably to be used for
piping data from a more general astrological utility to a more
specialized one), and the ADB format would make an excellent starting
point, provided that it is currently an open, and therefore extensible,
standard.

Bart

I have directed the discussion to this ADB-users mailing list.

The file format was invented by me, for the purpose of exporting ADB data.
It is described here:
http://www.astro.com/astro-databank/Help:XML_export_format

It should be noted that this format was created exactly for that purpose:to export ADB data. The format was not designed as an 'astrological exchange format', It is a simple ad-hoc creation and as such subject to change at will.

-- quote:

An <adb_entry> element consists of four sections:

1. <timestamp itst="1453464001">2016-01-22T12:00:01Z</timestamp>
2. <public_data> ... </public_data>
3. <text_data> ... </text_data>
4. <research_data> ... </research_data>

-- end quote

I declare that <pubic_data> segment of the xml-format to be in the public domain. But not the <text_data> format and not the <research_data> format.

Please note that this refers simple to the xml structure description, not to the actual data in the ADB export files.

The parts in the public domain are the tags

 <name>
 <sflname>
 <birthname>   optional
 <gender>
 <roddenrating>
 <datatype>
 <bdata>
   <sbdate>
   <sbdate_dmy>
   <sbtime>
   <place>
   <country>
   <positions>
 </bdata>
 <bdata_alt>  optional, alternative birth information
 </bdata_alt>
 <scollector>
 <seditor>
 <sbiographer>
 <screationdate>
 <slasteditdate>

the description of these tags and attributes on the ADB help page linked above, and one example record given there.

[Bart Lidofsky]

OK, that sounds reasonable. As the format is subject to change without notice, it would make more sense to just extract the public_data part, and use it as a basis of a new format, rather than making an extension of the ADB file format. Although there is no legal requirement to do so, it would be a good idea to make disconnect the resulting standard from any individual or company by either making it public domain, or GPL'd by a not-for-profit foundation.

Bart

On Monday, June 20, 2016 at 12:19:56 PM UTC-4, Alois Treindl wrote:

On the Swiss Ephemeris mailing list this question was asked:

On 20.06.16 16:07, Bart Lidofsky  wrote:

What is the current legal status of the ADB file format? Please note
that I am talking about the format itself, and not any data that is
stored using that format. I am especially interested if it is public
domain or GPL'd. The reason is that I have been thinking about, for
several years, creating a flexible data standard for transferring
astrological data, including calculated data (notably to be used for
piping data from a more general astrological utility to a more
specialized one), and the ADB format would make an excellent starting
point, provided that it is currently an open, and therefore extensible,
standard.

 

The file format was invented by me, for the purpose of exporting ADB data.

[Roberto Scerni]

Bart
at this moment the only format most compatible with other professional astrological program is the .QCK

I've asked to different software house to adopt the ADB format but they seem not interested (ore they are too lazy), so I think the only way is create an utility that can convert the Public section or the ADB .XML file in .QCK file an import it.

Please keep us informed about your work

Roberto

[Ray Murphy]

Bart, the Astrodatabank PROGRAM exporting format is far too complex to be treated as an export format because it is actually a full MS Access database with an array of interlocked sections. The only way to adequately extract all of the data is to have, well..... an Astrodatabank program.

An alternative (for Astrodatabank program owners) for extracting parts of it is to simply open the whole database in MS Access and write code in any computer language to process all the visible tab-delimited text which can be copy/pasted in a few seconds.

This may not be addressing the central issues here but it is important to know. I routinely do that with OTHER databases I create in the ADB program.

----

Now, this new great thing that Alois has produced - Where do I start, to begin converting that XML data into CSV files?

I write in VB6 and could write spaghetti code somehow, but are there any tips anyone can give?

Regarding this XML format - I feel like i did when I first got involved with PC's after being a Mac user. I was in a shop and asked .. "Will this program work on my PC?"... The assistant replied "Have you got Windows?"... I answered "What's Windows?"... He said "If you don't know what Windows is, you haven't got it - so no, don't buy it"

[Alois Treindl]

Am 21.06.2016 um 04:08 schrieb Ray Murphy <raym...@tpg.com.au>:

Now, this new great thing that Alois has produced - Where do I start, to begin converting that XML data into CSV files?

I write in VB6 and could write spaghetti code somehow, but are there any tips anyone can give?

Both, in the ADB help page describing the xml format, and in the download directory, is an example script in XSL format which extracts the data frm the xml file and outputs them as tab separated text lines.

See http://www.astro.com/adbexport/00_readme.htm 

adbx_dma.xsl 

 A xsl script, which can extract birth data from the xml file into 
	a TAB separated text file. This file serves as an example to show
	how conveniently data from an xml file can be extracted for processing.
	It is used like this: 

xsltproc adbx_dma.xsl  adb_export_sample.xml

For running this on windows, pleas google: xslt on Windows

Because some of the data elements like <name> and <birthname> can contain commata in the text, a comma separated text line is a bit more tricky. The commata would have to be escaped or quoted. In the latter case, quotes become the problem, as the text can also quote characters.

The typical pitfalls of a comma-separated data format. The ADB data do not contains tab characters, this is why tab separation is simpler. I thnk nearly everywhere where you can import CSV, you can also import TSV.

SEE ALSO https://en.wikipedia.org/wiki/Tab-separated_values

[Bart Lidofsky]

On 6/20/2016 8:37 PM, Roberto Scerni wrote:
> Bart
> at this moment the only format most compatible with other professional
> astrological program is the .QCK
>
> I've asked to different software house to adopt the ADB format but
> they seem not interested (ore they are too lazy), so I think the only
> way is create an utility that can convert the Public section or the
> ADB .XML file in .QCK file an import it.

.qck is highly static and extremely limited. Once a standard is created,
software can be written independently to adapt it. Since Astrodienst has
become the de facto standard for ephemeris data, I thought that it would
be a good idea to take the already useful ADB format and extend it to
include additional data. The big problem, of course, would be to
construct the schema (well, there may be some licensing issues, as I
want to make absolutely sure that any format will not be usable to
violate the spirit of Astrodienst's licenses).

Bart

[Bart Lidofsky]

On 6/20/2016 10:08 PM, Ray Murphy wrote:
> Regarding this XML format - I feel like i did when I first got
> involved with PC's after being a Mac user. I was in a shop and asked
> .. "Will this program work on my PC?"... The assistant replied "Have
> you got Windows?"... I answered "What's Windows?"... He said "If you
> don't know what Windows is, you haven't got it - so no, don't buy it"

I worked with SGML files in the early 1990's. XML is a simplification of
SGML, and, in addition, is extremely well supported by many free
computer language platforms.

Bart

[Ray Murphy]

Bart, how does one begin to extract the required data from XML lines.

For example, in Visual Basic do we simply read in an XML text file and stop
at every heading we want and then save whatever is between < >

I have no idea waht SGML is.

Ray

[Bart Lidofsky]

On 6/21/2016 9:34 PM, Ray Murphy wrote:
> Bart, how does one begin to extract the required data from XML lines.
>
> For example, in Visual Basic do we simply read in an XML text file and
> stop at every heading we want and then save whatever is between < >
>
> I have no idea waht SGML is.

SGML is standardized method for creating markup systems; HTML was
created using SGML (although the extensions pushed through by Netscape
and Microsoft violated a bunch of rules). XML was created to be a sort
of SGML "Junior", giving a lot of the power with much less of the
complexity. XML is an excellent way of storing heterogeneous data that
doesn't fit neatly into fixed length rows and columns, especially when
records don't contain all the data. Here's a link on how to use XML in
Visual Basic: https://msdn.microsoft.com/en-us/library/bb384833.aspx

Bart

[Roberto Scerni]

Hi Bart

I know that the .qck format is not the best  .. but seems to be one of the format in common to the most famous astrology program.

There is also another problem. for example in Solar Fire you can't create (or I less the program has some limit to read) a file with all the 53000+ data on the ADB database, I had to split in smaller file (on the order of 10000 data for file)

I hope that the ADB format will be adopted/read from different program, but for now the reality is different

Roberto

 

Bart

--
-- You received this message because you are subscribed to the Google Groups ADBusers group. To post to this group, send email to adbu...@googlegroups.com. To unsubscribe from this group, send email to adbusers+u...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/adbusers
--- You received this message because you are subscribed to the Google Groups "ADBusers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to adbusers+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^-^
Roberto Scerni

[Alois Treindl]

Ray,

not all data in xml are as text in elements, like
<name>Stuart, Matthew</name>

Some data are in element attributes, like
<sbtime sbtime_ampm="07:28 AM" ctimetype="s" stimetype="standard time"
stmerid="h5w" ctzauto="a" sznabbr="EST">07:28</sbtime>

This tag has text inside: 07:28
but it also has 6 different attributes.

This is familiar from html, which in its modern expressions a dialect of
xml.

Older html is not xml compatible, because it allowed tags which were not
closed, like
<br> or <p> or <img src="...">
instead of enforcing
<br/> or <p>blabla</p> or <img src=".." />

If your goal is simply to convert to TSV (tab separated text lines), the
simplest approach is as I described: use an xsl script and the xslt tool.

xml is not simple for the programmer.

If you want to work with xml data programmatically, in software, reading
and writing them: use a xml library tool for your programming
environment. Doing it 'by hand' is not advisable. The parsing of input
is the hard part. The writing is simpler, it can be done 'by hand'.

Its advantage is elsewhere: it is a standard, and there exists lots of
toos to deal with it. But it can really be awkward and complicated.

This is why many programmers prefer JSON, which is also a standard way
of encoding data, but more 'human'.

I chose XML because at Astrodienst, we have been forced to use it a lot,
e.g.
- bring Astrodatabank data into the Wiki format (Mediawiki has good xml
import)
- for data exchange with our payment providers

We now use it also to generate a lot of our screen content: our programs
write xml data, which we then convert with xsl/xslt into html for
display. This is much cleaner than creating html corrrectly.

xml does not tolerate any syntax errors, it breaks immediately of there
is an error.

html (and browsers) tolerate all kind of syntax errors, which can create
a lot of device-dependent display problems on the user's side.

When we generate html out of xml, the generated html is free of such
errors, by design.

For similar reasons companies like Microsoft went to xml for storing
MS-Word or Excel data. These file formats are extremely complicated, but
stable and much more error-free by design than the previously used
binary data formats.

For your purpose: look at the xsl script we provide in
http://www.astro.com/adbexport/


PS: please note that the audience of this adbusers mailing list is
extremely small. There are just a few old ADB software users subscribed
to it.

I don't know what platform would be better. I do not want this
discussion in the SE mailing list because it has nothing to do with
Swiss Ephemeris, except that I as a person and Astrodienst as a company
are involved in both.

I do not know of a general mailing list for 'Astrological programming'.
Maybe we should start one.

[Webmaster Astrodienst]

Am 21.06.2016 um 04:08 schrieb Ray Murphy <raym...@tpg.com.au>:

Now, this new great thing that Alois has produced - Where do I start, to begin converting that XML data into CSV files?

I write in VB6 and could write spaghetti code somehow, but are there any tips anyone can give?

[Webmaster Astrodienst]

Am 21.06.2016 um 04:08 schrieb Ray Murphy <raym...@tpg.com.au>:

Now, this new great thing that Alois has produced - Where do I start, to begin converting that XML data into CSV files?

I write in VB6 and could write spaghetti code somehow, but are there any tips anyone can give?

[alois]

Sorry, one of my postings has appeared twice, and with delay.

The reason is that I had used a sender address which was until now not an official group member.

I face the problem that I use multiple email addresses, some intentionally, some for historical reasons. Some devices I read and write mail on do not let me select which sender address they use. This can cause some problems.

[Ray Murphy]

Alois,

Thanks for the usual extremely clear advice.

I looked at the facilities for Visual Basic that Bart mentioned earlier, but
it is all far to complex for me, so prior to your writing just now I began
writing VB code to extract a limited number of fields from the test XML data
file and I can now easily assemble single Jigsaw data files for categories
like traits and occupations, and aslo "double-datafiles" as I call them, for
Synastry and Events, where the natal and event data are in pairs..

This now means I can utilise the Astrodienst XML files for all of my
standard work after I make application for acquire them.

The things you have outlined below are handy to know for when I begin
expanding and making use of a lot more of the information in the XML files.

There is a very large amount in the XML files that I am not using because
most of my categories were made from the 27,000 charts in ADB3, plus a lot
of separate chart data.

Routinely I might run between 5 and 50 big searches a week, where 120+
categories are compared simultaneously from a percentage point of view and a
stats p.o.v, so you can be sure a very large part of the XML data will be
processed in various ways before long.

Ray

[Bart Lidofsky]

On 6/22/2016 2:26 AM, Alois Treindl wrote:
> I do not know of a general mailing list for 'Astrological
> programming'. Maybe we should start one.

I created one a few years ago, for similar reasons; a place for
general astrology programming questions that didn't belong on the SE
list. Hasn't been active in a while, but here it is:
https://groups.yahoo.com/neo/groups/astrology_programming/info

Bart

[Astrodienst Webmaster]

Looks good to me. I forgot about it.

The only point which worries me (also in respect to the Swiss Ephemeris
mailing list): Yahoo has an uncertain future.

It is quite possible that the new owner will stop the groups. They are
not a profitable part of Yahoo.

The bad thing is that the groups have no export feature, i.e. there is
no way, except copy/paste or printing into PDFs, to get he group content
out for transfer into another system.

I think Google groups have a brighter future, as Google continue to be
controlled by Software geeks.

I think we are all interested that information we exchange in such a
mailing list remains accessible to the public for a long future, and
never gets deleted or removed.

[Ray Murphy]

As someone who has worked almost non-stop in research since 2002 I can say
that what we need is a small program that reads the Swiss Ephemeris and it
accepts Jigsaw csv files for input.

The output text file can have an unlimited number of categories in the 17th
field, with simple separations like this: "Manager | 4 years | Male |
English | ........." (inserted before run time).

The output csv file would be identical to the input file, but with the
standard factors following it- roughly the same or identical to, the
Astrodatabank program default settings which generate the data for
"tblPositions" (in Vers.3 of ADB).

Once such output is available astrologers everywhere can get in the research
loop in a more serious way and get answers to many questions very rapidly
via software that simply counts hits.

[Bart Lidofsky]

I agree with everything mentioned. I'll look into starting one on Google
Groups.

Bart

[Ray Murphy]

An ordinary forum like this can easily handle file exchanges, but ideally
with an independent address.
This style is idea for ongoing technical discussion and unlimited graphics.
http://astrologyresearch.forumatic.com/index.php

A secure future could be established by key members holding recent copies of
the forum data in case a new address was needed.

[Bart Lidofsky]

Just created it. I should probably get a co-owner, in case my account
gets hacked: https://groups.google.com/forum/#!forum/astrology-programming

Bart

On 6/22/2016 9:35 AM, Astrodienst Webmaster wrote:

[Bart Lidofsky]

This should be moved to the new group, but are you familiar with
"piping"? That is a concept that was originally from UNIX, and was built
into MS/DOS as well, but got kind of lost in MSWindows and Mac. It is
essentially using the output of one program as the input for another.
The format that I was talking about, which started this, is based on the
Free Software concept: Do one thing and do it well. The idea is that if
a standard data format can be created to contain astrological data,
including calculated data, then it could be used as a method of
transferring data from one program to another. So, for example, given
the standard format and your needs, you could have:

Program 1: Converts jigsaw csv to standard format, and sends it to:
Program 2: Takes the standard format as input and, using the SE, creates
the output you describe in the standard format
Program 3: Takes the standard format as input and outputs csv data.

The advantage is that one person doesn't have to write all 3 parts, and
Program 1 and 3 can be recycled by many programmers who want to take
input from Jigsaw csv and/or output data to a csv file.

The major problem (as Alois can tell you) is designing a schema is not
at all trivial (well designing a GOOD schema).

Bart

[Ray Murphy]

Yes, I agree. I once had such a system for " EGG" files as I called them
Everything was in the EGG file and it was the common format that all other
programs could translate to and from -a bit like how we can do converting
via the Astrodatabank program:
-----