structured string use cases - as requested by Jack/Fred on tech conf call 2009-11-10

[nitin]

Jack, Fred

Over the past year and in multiple earlier iterations of the BibJSON
spec we (Jim and I) have identified 4 major use cases for structured
strings which I explicate below :-

(I will use a $$ prefix for now to identify structured strings ($ ==
"like S" ) $$ not to be interpreted as in Perl or in PHP or in Shell
etc.)

Here are the 4 use cases.

a) standalone as a container for attributes during entity extraction
and enhancement of a single entity. Here the "text" is what Fred is
calling prefLabel.
e.g

SanFrancisco = { text: "SanFrancisco, CA", city: "SanFrancisco",
state: "CA", latitude: ...., longitude: .... }

In the simplest usage the above string may be used in a string as
follows

"My favorite city in the world is " + SanFrancisco['text']

If SanFrancisco is a Structured String you MUST ALWAYS be able to do
this.

Another usage could be "My favorite city in the world is " +
SanFrancisco['city'] + ", " + SanFrancisco['state']" but this may not
work for all structured strings.


b) in a list for a multivalued BibTeX field e.g.authors where authors
= [ $$author1, $$conjunction, $$author2 ]

where $$conjunction is a "trivial" structured string i.e. has only the
"text" key and is something like " and " or " with " etc.and $$authorN
is a structured string with potentially many more keys such as MR_ID,
affiliation etc. Note that a single author can have a much more
complex representation which we avoid in this use case but is
described at the end.

This list is naively rendered as the concatenation of the text attrs
of all the members giving say

"Jim Pitman and David Aldous".


c) in a list that represents the content of an abstract e.g.

[ $$chunk1, $$paper1, $$chunk2, $$author1, $$chunk2, $$author2 ]

here $$chunkN are (trivial) structured strings with only a "text" key
and the other $$xyz represent non-trivial structured strings with
attributes identifying a published paper, and two authors.

It is naively rendered as the concatenation of the text attrs of all
the members.

d) as a collection of alternate names or personas for a singe entity

[ $$preferredIdentity, $$alternateIdentity1, $$alternateIdentity2 ]

This is naively rendered as the "text" attribute of the first element,
with the text attributes of the others say in parens indicating they
are not the primary identifier.
Richer renderings can render them as above but with HTML links all
leading to e.g. the same web page.

e) finally a richer rendering of b) will have such "secondary-name
lists" instead of the singe structured string for each author. But
this is just a composition of b) and d) above so is not considered an
independent use case

Hope this shows the richness and non-trivial nature of these
structured strings.

Nitin Borwankar,
Project Manager, BKN.

[Jack Alves]

What you refer to as a structured string looks like a very general type. This kind of defeats the purpose of schema. Whenever possible it is better to have meaningful labels for attributes.

In case a), I would expect for there to be a type "address" with attributes for text, city, state, ... I recommend against using "text" as an attribute name unless it really doesn't represent a specific attribute. For the address example, "text" is meant to be the display name of the object.


In case b), I don't understand the purpose of $$conjunction. Why does that need to be stored? Can the value represent distinct meanings? It doesn't seem necessary in the example because there is an implicit "and" when listing authors. The example mentions that $$author1 "is a structured string with potentially many more keys such as MR_ID,
affiliation etc". In the example is seems like you are using $$author1 as a variable.

authors = [ $$author1, $$conjunction, $$author2 ]

If the content of the structured strings are shown would this look like an ordered list of objects?

authors =
[
 {
   "name": "Jim Pitman",
   "MRid": "6785565"
  },
{
   "conjunction": "and"
},
{
   "name": "David Aldous",
   "MRid": "87855" 
}]

Or is the structured string meant to be the definitive object for the author? In the latter case you are saying every attribute of Jim Pitman should be displayed. That would mean listing all publications because they would be attributes of the Jim Pitman object.


For case c), I recommend using one text field with embedded references to objects. Like an HTML anchor tag in a text block. I don't see the advantage of chunking.


For case d), BibJSON needs to support a list for altLabel. An object might look like,

{
  "prefLabel": "Jim Pitman",
   "altLabel":  {
            "0": "J. Pitman",
            "1": "James Pitman",
            "2": "jpitman",
       " }
}


Case e) represents the same issues as case b.

[Nitin Borwankar]

For "other things on my plate"-reasons  I'll have to wait till after next week to respond - in the meanwhile I hope Jim can continue this thread.

Nitin 

 

[Benjamin]

Hi Nitin,

I'm afraid I am confused by much of your discussion and notation. I
remember the spec before last allowed an array of objects to be used
in place of a string, with the value of the string being taken as the
concatenation of any "text" attributes in the objects. Is this what
you are referring to?

Ben

[Frederick Giasson]

Hi all,

As I tried to explain in the past and as I commented in the past
versions of the spec: I think there is a problem of concerns with this
proposition at some different levels. Lets enumerate them here, and then
lets talk about them.

(1) Storing of information related to specific citation styles; and
trying to outline documents structures and sentences ( related to point
(b) and (c) )
(2) Modifying the data model for programming language purposes (related
to points (a) )


The problem I see with (b) and (c) is one of purpose and focus. I
personally don't see BibJSON as an document & text outline language
since we have been talking about publication of bibliographic related
data (authors, institutions, publishers, documents information, etc),
and not about the publication of internal document and citations
structures. There are two totally different purposes that cannot be
reconciled in a single vocabulary/ontology. (at least, I won't ever
suggest that).

What I consider important for BibJSON is to have the good granular level
to be able to describe each entity implied in any bibliographic process
(authors, documents, editors, publishers, events (conferences, etc),
etc). Once this expressiveness is reached, consumers of data can display
they ingested from different sources, the way they want, with the style
they want.

Lets take an example to demonstrate what I mean here. Lets take a
"citations generation application". This application would ingest
BibJSON and generate different bibliographies using different citation
styles (Chicago, etc). What is important for this application is to have
all the information at hands to display the *same bibliographic data*
according to different styles. Such an application won't care about
conjunctions specified by the data publisher.

In fact, who the data publisher is to tell people how they should
display information?

As for point (c), why should BibJSON try to describe the outline of a
text corpus? (at least, it is what I understand from the explanation)

So, this raises a scoping issue with BibJSON.


Now, for the point (2) above, programming considerations impact the
development of the BibJSON spec. A distinction has to be made between:
(1) a data representation format which aim to be easily used by data
publisher to describe information its own, and to help him to easily
transmit this data to external agents; and (2) parsed structure created
from (1).

(2) should never impact the good development of (1). The goal is
two-fold: (1) we want to have to create a format that is the perfect
tradeoff between simplicity and expressiveness, and (2) having
programming API that parse this structure in easily re-usable data
structure to enable what you demonstrate in (a). And this should be true
with *any* programming language: Python, JavaScript, PHP, C, C#, etc.

We shouldn't ever say: I want to change the data format (bibjson) in
that way because it is easier to implement it in the programming
language X. We have to say: I want to make that change to the data
format because it is less confusing, make it simpler and easier to use
(to publish data) and it keeps its expressiveness power. Hoo and by the
way, it becomes easier to parse too :)


To answer what Nitin said in his email:

(a) is already possible with the current spec. It is the description of
a record.
(b) I won't propose to do that
(c) If my understand is right; I won't propose to do that neither
(d) We have two choices: we create multiple entities for the same entity
and links them together with some attribute. Or we do what Jack
suggested below.


Now:

> In case a), I would expect for there to be a type "address" with
> attributes for text, city, state, ... I recommend against using "text"
> as an attribute name unless it really doesn't represent a specific
> attribute. For the address example, "text" is meant to be the display
> name of the object.

I agree. But at the end, this is really a vocabulary consideration, and
not a notational one.

> In case b), I don't understand the purpose of $$conjunction. Why does
> that need to be stored? Can the value represent distinct meanings? It
> doesn't seem necessary in the example because there is an implicit
> "and" when listing authors. The example mentions that $$author1 "is a
> structured string with potentially many more keys such as MR_ID,
> affiliation etc". In the example is seems like you are using $$author1
> as a variable.
>
> authors = [ $$author1, $$conjunction, $$author2 ]
>
> If the content of the structured strings are shown would this look
> like an ordered list of objects?
>
> authors =
> [
> {
> "name": "Jim Pitman",
> "MRid": "6785565"
> },
> {
> "conjunction": "and"
> },
> {
> "name": "David Aldous",
> "MRid": "87855"
> }]
>
> Or is the structured string meant to be the definitive object for the
> author? In the latter case you are saying every attribute of Jim
> Pitman should be displayed. That would mean listing all publications
> because they would be attributes of the Jim Pitman object.

I agree. And what I don't like is that the code above suggest that we
have three authors, and that the conjunction is one of them. Otherwise,
a structure, like the lists we talked about in another thread, could be
introduced. Like a list, it won't have any semantic meaning other than
that it is a structure that order things in some ways. Maybe we could
endup with something like:


"authors": {
"textOutline": {
"0": {
"text": "Jim Pitman",
"ref": "@jpitman"
},
"conjuntion": " and ",
"1": {
"text": "David Aldous",
"ref": "@daldous"
}
}
}

Here, "textOutline" and "conjunction" would be reserved processing
keywords; and at least this would be consistent with the general ideas
of structures such as lists, etc.

In the example above it would make sense to use the attribute "text"
since we are talking about the outline structure of a text corpus
("text" + "conjuntion" + "text").

The semantic of the above would mean something like: There are two
authors @jpitman and @daldous. Additionally, we have a textOutline for
these two authors that would generate the literal "Jim Pitmand and David
Aldous".


The definition of the "author" attribute would remain the same:


"author": {
"prefLabel": "author",
"description": "The name(s) of the author(s) (in the
case of more than one author, separated by and)",

"allowedType": "Document",
"allowedValue": ["String", "Person"],

"minCardinality": "1"
},


Any structure can be used to as a value of the attribute "author"
(sequence, bag, textOutline, etc). They doesn't impact the validation of
the data according to the allowedValue of the "author" attribute.

However, all objects belonging to these structures ("0" and "1" above)
have to comply with the allowedValue specified in the structure schema
for "author".

The logic being this is that allowed values of an attribute can be
structured in any ways without impacting on the semantic of the
relationships. So, for "author", we could have have another object ("2")
which would be of time "Document" for example.

> For case d), BibJSON needs to support a list for altLabel. An object
> might look like,
>
> {
> "prefLabel": "Jim Pitman",
> "altLabel": {
> "0": "J. Pitman",
> "1": "James Pitman",
> "2": "jpitman",
> " }
> }
>

In fact it would be:

{
"prefLabel": "Jim Pitman",
"altLabel": [
"J. Pitman",
"James Pitman",
"jpitman",
]
}


(according to the current spec)

Or we could think about:


{

"prefLabel": "Jim Pitman",
"altLabel": {

"unorderedList": {

"0": "J. Pitman",
"1": "James Pitman",
"2": "jpitman"
}
}
}

I think I would enable both methods to be used and make them
semantically equivalent.



Thanks!


Take care,


Fred

[pitman]

Generally, my idea is that we must be able to accomodate multiple
formattings,
say
"latex", "bibtex","html","stuct_text", "list_text", ....
where there is no limit in principle to the number of these formats,
for any one of a number of fields, especially
"author","editor","title", "note","abstract"
and that this should be nicely embedded in JSON rather than an ugly
"author_latex", "author_struct_text","author_latex_list" etc.
A complication is that is the format of e.g an author field is
"stuct_text",
then the "text" field must be formatted somehow, it could be just
utf-8, or latex,
or bibtex (note latex and bibtex are different!). One way to indicate
how the
"text" values of a "struct_text" field are formatted would be to put
this in
an initial object of the "struct_text" array, which is like a metdata
object for the
array, with empty "text" value. I think this a very effective way to
record information about the entire field, like how the "text"
components are formatted, and
its origin/provenace, indications of doubt or correctness of the data
field, etc.
This is one of many reasons why I like "struct_text" that it allows
this level
of field-specific commentary and control.

Bottom line. Lets try this. Allow fields like "author" etc to be
objects,
some of whose keys may be formats "latex",
"bibtex","html","stuct_text", "list_text", ....
and the value of each key is the "author" attribute in this format.
There is an editorial issue raised by Micah, which is the root format?
But this can easily be specified, e.g. by giving the "author" object
an
attribute like {"source":"latex"}, and what the data owner does with
this information
is up to the data owner. It is not a problem for the specification.
--Jim

[Benjamin Kalish]

Jim, there is one big problem with specifying formats with keys: it
means you can only provide one string for each format. What if you had
two titles, one in German and one in English, both of which were
formatted using LaTeX markup? See my new post "Supporting multiple
string formats" (which is a further development of an idea I've
mentioned to you before) for my idea of how we might go about handling
this.

Benjamin Kalish

[Benjamin Kalish]

"author_statement" : [
{ "text": "Jim Pitman", "href": "@jpitman" },
{ "text": "and" },
{ "text": "David Aldous", "href": "@daldous" }
],
"author" : [
{
"prefLabel": "Aldous, David",
"ref": "@aldous"
},
{
"prefLabel": "Pitman, Jim",
"prefURL": "http://www.stat.berkeley.edu/~pitman/",
"ref": "@jpitman"
}
]

In this example structured text is used only in the "author_statement"
attribute and it isn't used to give information about
authors---instead it is enhancing the value of the "author_statement"
in the same way that pure html enhances text without giving it
meaning.

What confuses me about the more RFDa like approach , in which
structured text could be used not just to markup text but also to
imply relationships between entities and give meaning to the text
(which isn't what Fred is suggesting, but which has been suggested by
others, I believe), is that I don't see why we would need the
structure of BibJSON at all if we took such approach. If any string of
text could imply relationships between any pair of entities, then why
would we bother to put the text in an "author" attribute, or a "title"
attribute, or any other attribute?

Benjamin Kalish

p.s. A question about the use of "prefLabel" in the above example. Is
the prefLabel only meant to affect the display of the author in this
record, or is it meant to effect other records where the same author
appears. This is, I suppose, a general question about augmenting
metadata and reified metadata. It seems to me that if it is meant to
effect other records it could make for some very messy and hard to
maintain datasets.

[Jack Alves]

I'm still not clear why a new kind of structured text type is better than using references embedded in a text block like a standard HTML anchor,

"author_statement": {"html": "<a href='@jpitman' >Jim Pitman</a> and <a href='@daldous'>David Aldous</a>"}

I guess anchor tags might be harder to parse. If a structured text type is necessary it could be made a little cleaner by dropping the "text" key and using a mixed type array like,

"author_statement":
      {"stext": [

                   "Jim Pitman",{"ref":"@jpitman"},

                   "and David Aldous", {"ref":"@daldous"}
                   ]

[Benjamin Kalish]

A structured text type would have the "advantage" of being pure JSON.
Whether or not this is truly an advantage is debatable. Either way, I
think our priority should be to figure where we need to allow
structured text and what type of information it would convey. Once
that is established we can worry about the syntax. The requirements
will be clearer then and we can make a more informed decision.

Benjamin Kalish

[Nitin Borwankar]

Hello all,

I have been following the discussion with great interest - but only have a short time to reply.

There is a modeling chasm here in my opinion and we are at the bottom of it.

 

Fred's BibJSON spec is for strongly typed data as is irON and friends. If I understand correctly,  it also has the notion of wrapping loosely typed data like CSV in a strongly typed wrapper.

All of this machinery is very valuable once we have data that has evolved to the stage where it has a well defined tree of types or at least a well defined set of base types.

The whole discussion Jim and I had late last year and early this year leading to structured strings is about data that is of necessity loosely typed and we need to allow that data to remain that way because it reflects the state of the data in *an interim stage* in the modeling - i.e. entity extraction from "bibliographic soup".

However the discussion so far has not focused on this distinction nor has it recognized this distinction.

This, i.e. in BibTex processing,  is where structured strings arose and they are a *very powerful* lightweight rapid prototyping tool for doing linked data inside a single HTML file.  Once you look at the Schramm dataset as processed by Jim into a single HTML file with all links nicely resolving internally - a light bulb goes off - it did for me.   Structured strings provide  workflow support to generate these kind of datasets and we should look at them and see if the powerful machinery we bring to bear can generate these kind of HTML datasets simply and with evolutionary flexibility.

Structured strings are a workflow support tool and as Jack correctly points out they are a "typeless" type.

Be that as it may they are essential to the processing of BibTex in Jim's world and we need to formalize the concept and fit it into some sort of document that communicates to processors of BibTeX how to do this sort of thing.

A good test for any proposed spec would be if one can go from BibTeX to the Schramm.html of Jim using that spec alone.

The discussion so far has implicity assumed somehow that there is a mandate to fit these structured strings into Fred's BibJSON spec and irON.  This of course leads to statements like a) fits into the spec and b) c) etc don't and here's how you should do b) c) and d) better using irON and BibJSON etc etc.

I personally don't think structured strings and irON can and should be reconciled as one is for data in-process and one is for finished data.  ( I consider the fully linked schramm.html of Jim to be data in process because it is not how we inted to save it in final form). 

We need to find a set of docs so that all that has been learnt by BKN so far is effectively captured - I would suggest expanding on structured strings and their usage in creating HTML linked datasets - this should be a HOWTO doc and eschew modeling and schematic rigor, trading it for exploratory freedom and Just Do It.

Complementary to this we should have a parallel discussion on what do we mean by an author - what attributes does one have ( address, organization, affiliations ...etc) what do we mean by an organization  and focus on the BibJSON type system or vocabulary.

This should be the set of types that *finished* bibdata should look like. A list of author-as-dictionaries or deeply structured dictionaries with fluid attributes etc does not fit into this discussion.  That should be a parallel discussion - how to transform flat BibTeX data to the structured vocabularies of irON and BibJSON. I do not think we are, collectively, at the same level of understanding about formalizing structured strings as we are about the rest of the issues so some leeway in time and concepts need to be given so that these things can be formalized.  I believe Jim has advanced understanding about the workflow but we collectively do not.

That HOWTO discussion should explore and explicate the power of structured strings as a bridge structure between unstructured data and strongly structured data.  Trying to force fit structured strings into irON and BibJSON is likely to be frustrating IMHO because it is based on a foundational mismatch, conceptually.  

Nitin Borwankar

P.S. All opinions stupid, brilliant or otherwise are entirely my own :-)

37% of all statistics are made up on the spot
-------------------------------------------------------------------------------------
Nitin Borwankar
nborw...@gmail.com

[Benjamin Kalish]

Hi Nitin,

Where can I find this 'Schramm.html' file? I'm trying to envision what
it would look like to have bibliographic data embedded in HTML and I'm
not sure that what I am imaging is actually what you are talking
about.

Also, are you suggesting that we need two separate file formats, one
with structured strings to serve as an intermediary and one with
strong types as a destination format? I'm not trying to put words in
your mouth, I just want to be sure I understand what you are saying.

Thanks!

Benjamin Kalish

[Benjamin Kalish]

Hi Nitin,

I'm looking at http://yinas.org/display/schramm.html now, which must
be HTML file to which you were referring. It's not at all what I
expected! I'll have to give it a closer look before I can respond.

Benjamin Kalish

[Nitin Borwankar]

Hi Benjamin,

Jim is the best source for the file.  I had one a long while ago and he is always refining his files so best to get it from him.

It is a standalone HTML file with links which have HTML # anchors and local refs that use those anchors in links so when you click on an author link it takes you to the author record in the same file.  So I am not sure the one at ...display/schramm.html is the right one.

All of that is processed from bibtex --> HTML using workflow that involves structured strings. Or at least that is my understanding.

I am not the one suggesting we have two different formats - I am pointing out we already have two different ones and we are trying to shoehorn one into the other.  My suggestion is that structured strings need better documentation and externalization of Jim's knowledge about them so we can all discuss them better.  It is possible that we may find a conceptual bridge in which case we have just one format.  It is possible we don't - in which case we need two.  But either way we need to document structured strings better before we can talk about them at the same level of understanding as we can irON etc.

Nitin

37% of all statistics are made up on the spot
-------------------------------------------------------------------------------------
Nitin Borwankar
nborw...@gmail.com

[Frederick Giasson]

Hi Jack!

> I'm still not clear why a new kind of structured text type is better
> than using references embedded in a text block like a standard HTML
> anchor,
>
> "author_statement": {"html": "<a href='@jpitman' >Jim Pitman</a> and
> <a href='@daldous'>David Aldous</a>"}

There are already many such winning serializations that does just this
(RDFa, Microformats, etc), so why no re-using them for these specific
use-cases of embedding structured content within textual corpus (XHTML
files in these examples)?

> I guess anchor tags might be harder to parse. If a structured text
> type is necessary it could be made a little cleaner by dropping the
> "text" key and using a mixed type array like,
>
> "author_statement":
> {"stext": [
> "Jim Pitman",{"ref":"@jpitman"},
> "and David Aldous", {"ref":"@daldous"}
> ]
> }

I personally don't like this because we have to keep the order in memory
for *each* text string you will define in a dataset, even if the order
is not normally needed. But if you wrap that in an object, then the
order of objects in the array is not important, except if specified.

Thanks!


Fred

[Frederick Giasson]

Hi Benjamin!

> What confuses me about the more RFDa like approach , in which
> structured text could be used not just to markup text but also to
> imply relationships between entities and give meaning to the text
> (which isn't what Fred is suggesting, but which has been suggested by
> others, I believe), is that I don't see why we would need the
>

This is what Jim wanted (given our latest exchanges), and this is what I
was not keen about (it can be done, but this add a whole lot of
underlying complexity).

> structure of BibJSON at all if we took such approach. If any string of
> text could imply relationships between any pair of entities, then why
> would we bother to put the text in an "author" attribute, or a "title"
> attribute, or any other attribute?
>

Well, this is why it is quite important to make a distinction between
the Serialization (RDFa) and the description framework (RDF). RDFa is a
really good *serialization* for embedding RDF description in XHTML
files. Considering that each RDF description are attached to XHTML
elements such as SPAN and DIV, we can achieve Jim's aims as stated above.

RDFa is quite good for this kind of task. However, it is no good to
simply describe structured content. But N3 is good at it.

The good tool for the good job.

This is a quite important moto to keep in mind. You *can't* have a
serialization or a notation that is good at everything. You have to make
compromises to make certain usecases easier to handle given your
notation/serialization.

It is exactly for this reason (making compromises) that you have a
endless list of programming language for example. The same apply for
data description languages such as BibJSON.

And it is for that reason that I was not keen on doing this for BibJSON.


Thanks!


Take care,


Fred

[Benjamin Kalish]

Right. What I am afraid would be confusing is a situation analogous
to including RDFa withing N3 literals. This is what I was afraid Jim
and Nitin might be talking about, however, having looked at the
schramm.html file, I think that what they are actually doing is much
simpler than that. I didn't see any examples where they implied any
relationship other than those where the object was the current record,
and the predicate was the current attribute. What they have done is, I
think, to provide an alternative method for specifying the objects of
a relationship which allows them to specify some information about how
that relationship is displayed at the same time. So long as the
implied relationships remain of this type I don't think it will become
excessively complicated.

I do wonder however, what the real value of such a system is. If it is
meant to allow, say, the statement of responsibility and the
enumeration of authors to be combined in one attribute, then it will
not achieve its purpose; there is not necessarily a one-to-one
correspondence between the authors of a work and the names listed on
the document, and so these must sometimes be given separately. (For
simplicity, I believe they should *always* be given separately.) If
this is not its purpose, though, then what is it? If it is simply so
that dataset authors may have more control over the display of their
records, then why must it define a relationship at the same time?

Benjamin Kalish