Linked Data and a new Browser API event

[Benjamin Francis]

Hi,

In Gaia the Systems Front End team is implementing the Pinning the Web
design concept [1] which amongst other things represents "pinned" web pages
as cards on the homescreen. The goal is that where possible these cards
should not just be a thumbnail screenshot of the page, but should be a
meaningful representation of key metadata associated with the page based on
a particular "type", and even associated actions.

The idea is that these cards could be generated from the Linked Data [2]
that many web pages already include for the consumption of search engines
(to create "rich snippets" in search results) but which browsers currently
ignore.

All the big search engines agree on schemas for this data at schema.org [3]
but they support multiple formats for encoding this information in a web
page including Microdata, RDFa and JSON-LD. In short, there's a format war
going on.

In the protoype we created [4] we supported some basic use cases by parsing
Open Graph meta tags from web pages [5], Open Graph [6] being a kind of
simplified form of RDFa.

For the real implementation I suggest we investigate supporting one or more
formats for Linked Data in web pages (based on level of adoption) and
surface them to Gaia through a linkeddatachange event on the Browser API. I
propose that the payload of this event should be the contents of the Linked
Data expressed in JSON-LD [7].

JSON-LD is a W3C recommendation and can be used to express data from any of
the other Linked Data formats. The JSON encoding is particularly suitable
for the use cases in Gaia as it can easily be parsed in JavaScript and
stored in the places database in IndexedDB/DataStore.

The best data I've found so far on adoption of these formats is from the
University of Mannheim [8] and maybe suggests that Microdata is on the
increase while RDFa is staying relatively constant, but it's far from
clear. And this study doesn't include data for JSON-LD usage.

In an ideal world we'd support RDFa, Microdata and JSON-LD and convert them
all into JSON-LD for consumption in Gaia, but we could also pick a side in
the format war based on usage data.

Has support for Linked Data been looked into before? Is this something we
can get into the platform?

Thanks

Ben

1.
https://docs.google.com/presentation/d/17CGWPwu59GB7miyY1ErTjr4Wb-kS-rM7dB3MAMVO9HU/pub#slide=id.p
2. http://en.wikipedia.org/wiki/Linked_data
3. http://schema.org/
4. https://www.youtube.com/watch?v=FiLnRoRjD5k
5. https://gist.github.com/mikehenrty/6c506767b0fb15aaa2d4
6. http://ogp.me/
7. http://www.w3.org/TR/json-ld/
8. http://webdatacommons.org/structureddata/index.html

[Benjamin Francis]

On 28 May 2015 at 18:13, Benjamin Francis <bfra...@mozilla.com> wrote:

> For the real implementation I suggest we investigate supporting one or
> more formats for Linked Data in web pages (based on level of adoption) and
> surface them to Gaia through a linkeddatachange event on the Browser API. I
> propose that the payload of this event should be the contents of the Linked
> Data expressed in JSON-LD [7].
>

Actually what might make more sense is a getLinkedData() method on the API
which returns a Promise that resolves with the JSON data, as I think we're
also going to need a getManifest() method which could work in a similar way.

[Anne van Kesteren]

On Fri, May 29, 2015 at 2:55 AM, Benjamin Francis <bfra...@mozilla.com> wrote:
> Actually what might make more sense is a getLinkedData() method on the API
> which returns a Promise that resolves with the JSON data, as I think we're
> also going to need a getManifest() method which could work in a similar way.

We've bitten ourselves before going down the RDF rathole (see
extensions et al). Not sure we should so rapidly start again. Why
can't you use the Microdata API?


--
https://annevankesteren.nl/

[Benjamin Francis]

On 30 May 2015 at 00:56, Anne van Kesteren <ann...@annevk.nl> wrote:

> We've bitten ourselves before going down the RDF rathole (see
> extensions et al). Not sure we should so rapidly start again. Why
> can't you use the Microdata API?
>

Is this already supported in Gecko? I can't find it documented anywhere,
except a partial implementation in bug 591467 and a suggestion to remove it
again in bug 909633.

If it was then that would help, but it wouldn't quite solve the problem
we're trying to solve here. We need to get the Linked Data from an embedded
mozbrowser iframe for use in the system app and we don't have access the
Document object (hence the Browser API). If there was an implementation of
the Microdata DOM API I guess we could hook up a getLinkedData() method to
that inside Gecko.

But Microdata is only one of the formats widely used on the web today. I'd
like to see some evidence-based discussion on which format(s) we should
support to get the most possible value out of what already exists on the
web. The examples we used in our prototype all use Open Graph, which seems
quite widely used (mainly due to Facebook and Twitter) and is based on RDFa.

JSON-LD seems like a convenient format which can express them all, and is
useful in its own right. We could quite easily detect script tags in the
DOM like <script type="application/ld+json">.

Examples are provided for RDFa, Microdata and JSON-LD on all the schema.org
schemas, but I'm not sure what weighting these are given by the various
search engines. If anyone has data on that it would be really helpful!

Ben

[Jonas Sicking]

We should use whatever formats people are using to mark up pages. If that
is microdata we should use that. If it's RDF we should use that. If its
JSONLD we should use that.

The API that is used to extract the data is irrelevant. That will be an
internal API anyway. Effectively we should think of the browser api as an
internal api. There is no way it will be standardized in any relevant
timeframe.

/ Jonas

On May 29, 2015 16:57, "Anne van Kesteren" <ann...@annevk.nl> wrote:

> On Fri, May 29, 2015 at 2:55 AM, Benjamin Francis <bfra...@mozilla.com>
> wrote:
> > Actually what might make more sense is a getLinkedData() method on the
> API
> > which returns a Promise that resolves with the JSON data, as I think
> we're
> > also going to need a getManifest() method which could work in a similar
> way.
>

> We've bitten ourselves before going down the RDF rathole (see
> extensions et al). Not sure we should so rapidly start again. Why
> can't you use the Microdata API?
>
>

> --
> https://annevankesteren.nl/
> _______________________________________________
> dev-webapi mailing list
> dev-w...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-webapi
>

[Gordon Brander]

We should consider a series of fallbacks for this internal API.

The metadata story for things like icon, title, description, hero images, is complicated. Implementation in the follows real-world use cases like posting rich snippets to Facebook or getting an image to show up on Twitter, rather than some standard.

I think it would be best to think of this kind of API as a sort of light "scraper" that crawls through a collection of known in-the-wild patterns to provide a good-enough answer.

---

Gordon Brander
Sr Design Strategist
Mozilla

[Anne van Kesteren]

On Sun, May 31, 2015 at 5:09 AM, Jonas Sicking <jo...@sicking.cc> wrote:
> We should use whatever formats people are using to mark up pages. If that is
> microdata we should use that. If it's RDF we should use that. If its JSONLD
> we should use that.
>
> The API that is used to extract the data is irrelevant. That will be an
> internal API anyway. Effectively we should think of the browser api as an
> internal api. There is no way it will be standardized in any relevant
> timeframe.

Sure, but if our project has any success any competitor would have to
reverse engineer this mess. Which seems sad.


--
https://annevankesteren.nl/

[Karl Dubost]

Benjamin,

Le 30 mai 2015 à 21:35, Benjamin Francis <bfra...@mozilla.com> a écrit :
> But Microdata is only one of the formats widely used on the web today. I'd
> like to see some evidence-based discussion on which format(s) we should
> support to get the most possible value out of what already exists on the
> web. The examples we used in our prototype all use Open Graph, which seems
> quite widely used (mainly due to Facebook and Twitter) and is based on RDFa.

if not done yet, you might want to talk with Dan Brickley. He is working at Google on everything related to schema.org. danbri -AT- google.com

He might have access on how popular for each of those markup.
It might be possible to have a conversion tables in between the different markups so making it easier to start with one markup and build up little by little.

Whatever you adopt as an internal API. I would recommend you publish along on how to write down these meta in Web pages, so it encourages authors to add those if not done yet. It will help enter into a virtuous circle with a very quick feedback loop ala:

1. You see you put data like this in the page
2. This is the result for the pinned page.



--
Karl Dubost, Mozilla
http://www.la-grange.net/karl/moz

[Jonas Sicking]

I think we're already talking about reverse-engineering what search
engines and twitter/facebook/etc do.

But I'm still all for proper standardization. Including driving
towards good technical solutions.

But given how small marketshare browsers in general have as metadata
consumers, I think any standardization efforts would have to be driven
by the current matadata consumers, like search engines and social
networks.

/ Jonas

[Gordon Brander]

On June 1, 2015 at 17:34:48 , Jonas Sicking (jo...@sicking.cc) wrote:
> I think we're already talking about reverse-engineering what search
> engines and twitter/facebook/etc do.
> 
> But I'm still all for proper standardization. Including driving
> towards good technical solutions.

Yup. We’re really talking about 2 things in parallel:

1. Defining a standards-based approach to marking these things up (using pre-existing patterns where it makes sense). Encouraging authors to use it.

2. Creating internal APIs that will leverage this metadata, and in cases where the standards-based metadata does not exist, scraping reasonable results from other common metadata or markup patterns.

[Tantek Çelik]

Gordan has the right general idea to approaching this problem space.

On Mon, Jun 1, 2015 at 5:42 PM, Gordon Brander <gbra...@mozilla.com> wrote:
> On June 1, 2015 at 17:34:48 , Jonas Sicking (jo...@sicking.cc) wrote:
>> I think we're already talking about reverse-engineering what search
>> engines and twitter/facebook/etc do.

The summary among all the myriad proprietary (read: single corp /
oligopoly controlled) proposals is that Facebook OGP meta tags have a
strong lead over all the other proprietary approaches (for various
reasons we can get into offline if desired), while among the "open
standards community" options - i.e. per Mozilla open web principles,
microformats have the lead.

>> But I'm still all for proper standardization. Including driving
>> towards good technical solutions.
>
> Yup. We’re really talking about 2 things in parallel:
>
> 1. Defining a standards-based approach to marking these things up (using pre-existing patterns where it makes sense). Encouraging authors to use it.
>
> 2. Creating internal APIs that will leverage this metadata, and in cases where the standards-based metadata does not exist, scraping reasonable results from other common metadata or markup patterns.

This analysis and conclusion matches what we've been figuring out with
implementations and deployments in the IndieWebCamp community as well
(which has several use-cases similar to pins/cards for providing
summaries/link-previews of pages on the web). In short, the general
approach involves parsing for two general sets of published data /
markup:

1. Pragmatic parsing of what's most out there:
a) according to anecdotal sampling by the indieweb community,
Facebook OGP, and
b) according to studies / open/common crawl datasets: classic microformats

2. Simplest open standards based approach (so we can recommend the
easiest, least work, and most openly developed/maintained approach to
authors and site owners) - microformats2.

I'm happy to provide citations/specs for these, as well as follow-up
on any detailed questions.

Thanks,

Tantek Çelik
Web Standards Lead
Mozilla

[Benjamin Francis]

Thanks for all the responses so far! Comments inline...

On 30 May 2015 at 21:09, Jonas Sicking <jo...@sicking.cc> wrote:

> We should use whatever formats people are using to mark up pages. If that
> is microdata we should use that. If it's RDF we should use that. If its
> JSONLD we should use that.
>

Agree, that's what I'd like to find out.

On 30 May 2015 at 21:31, Gordon Brander <gbra...@mozilla.com> wrote:

> We should consider a series of fallbacks for this internal API.
>
> The metadata story for things like icon, title, description, hero images,
> is complicated. Implementation in the follows real-world use cases like
> posting rich snippets to Facebook or getting an image to show up on
> Twitter, rather than some standard.
>
> I think it would be best to think of this kind of API as a sort of light
> "scraper" that crawls through a collection of known in-the-wild patterns to
> provide a good-enough answer.
>

Agree, I think we will at least initially need to support multiple formats,
ideally translating them all internally into a single format that Gaia can
consume via the Browser API.

On 2 June 2015 at 00:53, Karl Dubost <kdu...@mozilla.com> wrote:

> if not done yet, you might want to talk with Dan Brickley. He is working
> at Google on everything related to schema.org. danbri -AT- google.com
>

Thanks, I will!

It might be possible to have a conversion tables in between the different
> markups so making it easier to start with one markup and build up little by
> little.
>

The JSON-LD spec gives examples of how Turtle, RDFa, Microformats and
Microdata can be expressed in JSON-LD [1]. Given JSON is an ideal format
for us to use in Gaia, I like the idea of internally translating into that.

On 2 June 2015 at 01:34, Jonas Sicking <jo...@sicking.cc> wrote:

> I think we're already talking about reverse-engineering what search
> engines and twitter/facebook/etc do.
>

Exactly, this is about getting more value out of the content that already
exists on the web, not defining new ways to create content.

But given how small marketshare browsers in general have as metadata
> consumers, I think any standardization efforts would have to be driven
> by the current matadata consumers, like search engines and social
> networks.
>

Agree. Though that doesn't prevent us from contributing to that discussion
if we have something to say.

On 2 June 2015 at 01:42, Gordon Brander <gbra...@mozilla.com> wrote:

> Yup. We’re really talking about 2 things in parallel:
>
> 1. Defining a standards-based approach to marking these things up (using
> pre-existing patterns where it makes sense). Encouraging authors to use it.
>
> 2. Creating internal APIs that will leverage this metadata, and in cases
> where the standards-based metadata does not exist, scraping reasonable
> results from other common metadata or markup patterns.
>

Agree, except I don't want to solve the problem of multiple formats by
creating another format, I'd like to either pick one of the existing
formats or (more likely) hedge our bets and support multiple popular
formats, giving developer warnings for non-standard usage where necessary.
If we find we have suggestions of how to improve the existing formats, then
we should participate in the groups that already exist to make that happen.

On 3 June 2015 at 00:45, Tantek Çelik <tan...@cs.stanford.edu> wrote:

> The summary among all the myriad proprietary (read: single corp /
> oligopoly controlled) proposals is that Facebook OGP meta tags have a
> strong lead over all the other proprietary approaches

That seems to match our anecdotal experience in building a prototype. Open
Graph is quite primitive in comparison to other formats in terms of what
can be expressed (and it's not clear to me whether it validates as either
valid HTML5 or valid RDFa), but it does seem like a clear contender.

> (for various
> reasons we can get into offline if desired),

I would like to understand those reasons. Are there reasons beyond "
Facebook and Twitter make use of this data so people add it to their web
pages"?

while among the "open
> standards community" options - i.e. per Mozilla open web principles,
> microformats have the lead.
>

That is the answer I would expect from the person whose name happens to be
used as an hCard example in a W3C spec under the heading of "Microformats"
[2] ;)

> This analysis and conclusion matches what we've been figuring out with
> implementations and deployments in the IndieWebCamp community as well
> (which has several use-cases similar to pins/cards for providing
> summaries/link-previews of pages on the web). In short, the general
> approach involves parsing for two general sets of published data /
> markup:
>
> 1. Pragmatic parsing of what's most out there:
> a) according to anecdotal sampling by the indieweb community,
> Facebook OGP, and
> b) according to studies / open/common crawl datasets: classic
> microformats
>
> 2. Simplest open standards based approach (so we can recommend the
> easiest, least work, and most openly developed/maintained approach to
> authors and site owners) - microformats2.
>

I'd like to understand more about the lessons learned here, please email me
off thread if that makes more sense.

I'm happy to provide citations/specs for these, as well as follow-up
> on any detailed questions.
>

This is what I'd really like to get more of, particularly usage data.

Thanks

Ben

1. http://www.w3.org/TR/json-ld/#relationship-to-other-linked-data-formats
2. http://www.w3.org/TR/json-ld/#microformats

[Michael[tm] Smith]

Benjamin Francis <bfra...@mozilla.com>, 2015-06-03 19:42 +0100:
...

> Open Graph is quite primitive in comparison to other formats in terms of
> what can be expressed (and it's not clear to me whether it validates as
> either valid HTML5 or valid RDFa)

It’s not valid HTML(5), because the HTML spec doesn’t allow the meta element
to have a `property` attribute. But it is valid HTML+RDFa (Lite), because
the HTML+RDFA spec extends the HTML spec to add the `property` attribute to
the meta element (and as far as I understand it least, Open Graph markup
entirely consists just of using meta[property] and meta[content]).

As far as how HTML-checking tools deal with that markup, the shared backend
code for validator.nu and the W3C Nu HTML Checker at the HTML5 facet of the
legacy W3C validator by default doesn’t report errors for meta[property]
(or for any other RDFa Lite attributes in HTML documents). In other words,
it essentially treats it (and all other RDFa Lite markup) as being valid.

We made that behavior the default specifically because we were getting
enough bug reports from users about the checker reporting errors for Web
documents containing Open Graph meta[property] that we decided it’d waste
less user time if we were to allow it rather than emitting errors for it.

And that sorta necessarily meant also allowing all other RFDa Lite
attributes, because there is no actual Open Graph spec, and the HTML+RDFa
spec is the only spec that does actually define a meta[property], and RDFa
Lite was the smallest defined subset of RDFa which would get us that.

Beyond that I think we don’t (can’t) do any useful datatype/microsyntax
checking of the actual value of meta[property] due to the fact the RDFa
specs define it as being allowed to contain CURIEs, while at the same
defining CURIES in such a way that any arbitrary string can be a valid
CURIE. So in practice authors can put anything they want into
meta[property] without the checker reporting any errors.

—Mike

--
Michael[tm] Smith https://people.w3.org/mike

[Michael[tm] Smith]

[cc’ing Marcos] Marcos, if you’ve not been following along already, full context
starts at https://lists.mozilla.org/pipermail/dev-platform/2015-May/010149.html

Anne van Kesteren <ann...@annevk.nl>, 2015-06-02 08:31 +0900:

> On Sun, May 31, 2015 at 5:09 AM, Jonas Sicking <jo...@sicking.cc> wrote:
> > We should use whatever formats people are using to mark up pages. If that is
> > microdata we should use that. If it's RDF we should use that. If its JSONLD
> > we should use that.
> >

> > The API that is used to extract the data is irrelevant. That will be an
> > internal API anyway. Effectively we should think of the browser api as an
> > internal api. There is no way it will be standardized in any relevant
> > timeframe.
>
> Sure, but if our project has any success any competitor would have to
> reverse engineer this mess. Which seems sad.

As came up in some off-list discussion with Anne, is the “Manifest for a
web application” spec at https://w3c.github.io/manifest/ not relevant here?
(Nothing to reverse engineer, since it has an actual spec—with defined
processing requirements—and at least one other browser-engine project is
also contributing to it and implementing it.)

[Michael[tm] Smith]

[re-sending with Marcos actually Cc’ed this time]

[Benjamin Francis]

On 4 June 2015 at 03:27, Michael[tm] Smith <mi...@w3.org> wrote

> As came up in some off-list discussion with Anne, is the “Manifest for a
> web application” spec at https://w3c.github.io/manifest/ not relevant
> here?
> (Nothing to reverse engineer, since it has an actual spec—with defined
> processing requirements—and at least one other browser-engine project is
> also contributing to it and implementing it.)
>

Yes, we already support W3C web app manfiests in our prototype, and it's a
key part of the implementation.

A manifest provides metadata for a website as a whole, whereas Linked Data
provides metadata to a particular web page.

When you pin a whole site we use the manifest (and fall back to other
metadata when not available), and when you pin a page we use Linked Data
(and fall back to other metadata when not available).

Ben

[Robin Berjon]

Pinning based on Linked Data also makes a lot of sense because it's
already massively deployed (to millions of domains), whereas manifest isn't.

To reinforce Benjamin's point imagine a flight booking site. The
manifest would get you to pin the site as an app with which you could
book flights in the future; LD would allow you to pin a ticket you've
bought in such a way that it displays just the essential time/location
information you want to see at a glance. The use cases are totally
different.

--
Robin Berjon - http://berjon.com/ - @robinberjon

[Benjamin Francis]

On 3 June 2015 at 19:42, Benjamin Francis <bfra...@mozilla.com> wrote:

> This is what I'd really like to get more of, particularly usage data.
>

I've reached out to a few people at Yahoo, Google and a couple of
universities and have managed to turn up a few studies with useful data
[1][2][3][4].

My conclusions so far are:

- Microformats are used on a large number of web sites but are limited
by their case by case syntax and more fixed vocabulary and are less
formally defined.
- Microdata and RDFa are vocabulary agnostic which makes them inherently
more extensible, they're increasing in popularity due to schema.org and
consumption by major search engines, whilst the use of Microformats has
remained relatively constant over time.
- Microdata is a bit more concise than RDFa but doesn't allow for the
mixing of vocabularies.
- Open Graph is a simplistic form of RDFa with a limited vocabularly and
limited usefulness in comparison to other formats, but is very widely used
due to Facebook and Twitter being major consumers.
- Microformats is used by more websites (domains) but Microdata is used
by more web pages (more URLs, more typed entities and more triples) and is
growing the fastest. Microformats has the breadth, but Microdata has the
depth. In our case I think what we care about is the latter - the amount of
pinnable content.
- JSON-LD is the newest format, the main difference being that it isn't
intended to be embedded in with HTML markup, but is included separately in
a script tag. It's also useful as a canonical JSON-based format to
represent all of the other formats.

That leads me to recommend that we do the following:

- Parse Microdata and RDFa (including Open Graph) from web pages in Gecko
- Expose all of this data to Gaia via a single getLinkedData() or
getStructuredData() method on the Browser API which returns a Promise that
resolves with the data in a canonical JSON-LD format
- Also consider supporting JSON-LD directly as no parsing is required,
we just need to detect a script tag

If anyone finds any more usage data, or has a different interpretation of
the data below, then please do share.

Thanks

Ben

1. Web Data Commons website based on Common Crawl corpus (2009-2014)
http://webdatacommons.org/
2. Web Data Commons Paper based on Common Crawl Corpus (2009-2012)
http://events.linkeddata.org/ldow2012/papers/ldow2012-inv-paper-2.pdf
3. Yahoo post based on Yahoo corpus (2011)

https://tripletalk.wordpress.com/2011/01/25/rdfa-deployment-across-the-web/
4. Yahoo paper based on Bing corpus (2012)
http://events.linkeddata.org/ldow2012/papers/ldow2012-inv-paper-1.pdf

[Benjamin Francis]

To follow up on this, there is resistance against implementing the more
complex Microdata or RDFa specifications in Gecko.

We definitely now need some form of Linked Data support for Firefox OS 2.5
so I'm suggesting the following: We should support Open Graph (because of
its wide usage by existing web content) and JSON-LD (because it supports
Gaia's more complex use cases).

Both of these should be simple to implement in Gecko as events on the
Browser API, without requiring any complex parsing on the Gecko side.

Open Graph just requires firing metachange events (see bug 962626 for an
example) for all meta tags which specify a property attribute. (This would
be a crude subset of RDFa. We don't need to specify particular vocabularies
in Gecko, just include the value of the property attribute in the payload
of the event).

<meta property="og:title" content="The Rock" />

JSON-LD just requires firing a new linkeddatachange event whenever a
JSON-LD script tag is encountered, sending the contents of the script tag
in the payload of the event.

<script type="application/ld+json">

We can then easily parse the JSON in Gaia and even directly store it
directly in our Places database.

If there's resistance against implementing the more complex Microdata and
RDFa specifications in Gecko then I don't think we should implement
Microformats either, the data I have and our experience through prototyping
just don't justify it.

Unless there's a really good reason not to do so, I'm going to file the
bugs and look towards getting this implemented on the Browser API as soon
as possible.

Thanks

Ben

[Jonas Sicking]

On Thu, Jun 25, 2015 at 7:19 PM, Benjamin Francis <bfra...@mozilla.com> wrote:
> and JSON-LD (because it supports Gaia's more complex use cases).

Hi Ben,

My only concern here is that if you pin a contact, it seems to me that
it would be good if the name and picture of that homescreen UI should
be quickly updated if the user changes the name/picture of the
contact.

So I think that for pins of contacts, we should call into the contacts
API rather than rely on the metadata extracted from the webpage.

I'm not sure if we might still need JSON-LD for other gaia use cases
though, like pinning a song from the music app or an image from the
gallery?

/ Jonas