In-browser annotation

146 views
Skip to first unread message

David Bokan

unread,
Sep 30, 2021, 1:25:54 PM9/30/21
to dev-pl...@mozilla.org
Hi there!

A few of us over in Chromium are thinking of how we can enable annotation, directly in the browser, without need for extensions. In a sentence: to enable shared commentary over web content. We're very early in the stages of thinking about this, there's no prototype or code written yet but we did gather some thoughts and ideas in an explainer

This is something that many folks have considered in the past, Mozilla included. I know that link is ancient but I'm wondering if there's anyone at Mozilla today who's working or interested in a related space and would like to get involved -- or even just provide early feedback.

I think this is functionality that really relies on working across browsers so would really appreciate getting early perspective and/or involvement from Firefox.

Thanks!
David

Tantek Çelik

unread,
Sep 30, 2021, 2:29:07 PM9/30/21
to David Bokan, dev-pl...@mozilla.org
Hi David,

Interesting to see annotations popping-up again, it feels like one of
those cyclical things/efforts on the web. On this in particular:

On Thu, Sep 30, 2021 at 10:25 AM David Bokan <bo...@chromium.org> wrote:
>
> Hi there!
>
> A few of us over in Chromium are thinking of how we can enable annotation, directly in the browser, without need for extensions.

Since you brought up "without need for extensions", I'd ask the
obvious question:

Are there any browser extensions that do it *well*?

(my experience to date has been no, for various reasons, which isn't a
reason not to do it, however does question that kind of framing, at
least as evidence of feasibility)

> In a sentence: to enable shared commentary over web content. We're very early in the stages of thinking about this, there's no prototype or code written yet but we did gather some thoughts and ideas in an explainer. https://github.com/bokand/web-annotations/

I took a quick look and there's a lot of good background there! Great start.

There's at least 1-2 more examples of distributed web annotation prior
art (some semi-active) that I can think of, like
https://indieweb.org/marginalia. I can suggest these as pull requests
to the explainer if you'd like.


> This is something that many folks have considered in the past, Mozilla included. I know that link is ancient https://wiki.mozilla.org/Firefox/Feature_Brainstorming:Notes_and_Annotations

Great find (TIL). Good to see some of that historical thinking :)

> but I'm wondering if there's anyone at Mozilla today who's working or interested in a related space and would like to get involved -- or even just provide early feedback.

I'm at least tangentially interested, more from the perspectives of
having seen over the (many years) :

1. Lots of web annotation (including in browser) attempts / services /
sites try & fail (often repeating mistakes)

2. The few web annotation services that gain traction eventually
turning into yet another harassment vector for the web (pretty much
every such attempt designed/built by highly privileged individuals who
conceive more of the optimistic possibilities without any real
understanding of the threats to safety on the web, again repeating
mistakes of every social media site)

3. The overly complex attempt at standardization at W3C, which barely
has (some) interop, but at a high enough barrier as to cause more
centralization (yet another silo, Hypothesis) than decentralization.

4. Potential for misframing / loss of meaning. When you focus on "web
annotation", everything published to the web starts to look like a
"web annotation", at which point the phrase or even word "annotation"
becomes meaningless.


> I think this is functionality that really relies on working across browsers

Maybe? I'm not sure about "relies on" (at least for
prototyping/incubation), and even in the case of small communities
wanting to interact amongst themselves, they're exploring peer-to-peer
methods for doing so, that are essentially differently styled web
comments.

> so would really appreciate getting early perspective and/or involvement from Firefox.

Hopefully the above is helpful. There's also a community of folks
actively publishing distributed "annotations" to the web based on
fairly simple composable open standards in the IndieWeb community that
has done a lot of analysis and documenting of their work:
https://indieweb.org/annotation, so you may want to also ask there:
https://chat.indieweb.org/dev

Thanks,

Tantek

David Bokan

unread,
Sep 30, 2021, 3:11:54 PM9/30/21
to dev-pl...@mozilla.org, tan...@cs.stanford.edu, dev-pl...@mozilla.org, David Bokan
Thanks for the reply!

On Thursday, September 30, 2021 at 2:29:07 PM UTC-4 tan...@cs.stanford.edu wrote:
Since you brought up "without need for extensions", I'd ask the
obvious question:

Are there any browser extensions that do it *well*?
(my experience to date has been no, for various reasons, which isn't a
reason not to do it, however does question that kind of framing, at
least as evidence of feasibility)

This is, of course, subjective -- I've been using hypothes.is for the last few months and actually used it to decent effect to have a conversation about my explainer. I find the tool itself -- the UI and usage patterns in it -- to be quite intuitive and works well for the most part.

The major down side I see is that it requires both parties interested in a discussion to be using this same tool and to know about it. Because of this (my assumption) - most of the web feels pretty barren using it. It's also service-locked -- meaning it's one broad tent for discussion. When I do finally stumble across a comment, it's usually irrelevant and from some far flung corner, not someone I know or interesting.

I expect this would be dramatically improved by having this functionality built-in in the browser which would provide more feedback, discovery, incentive to annotate. Also allowing smaller ("denser") circles so commentary is more likely from people the user knows or at least is interested in.

There are smaller issues with it too -- more technical in nature. e.g. annotations on feed based sites (Twitter, etc.) don't work at all. These are things I think that are fixable but not necessarily show-stoppers.

But I'm curious if you or others have thoughts on the existing landscape and where it's lacking.
  
There's at least 1-2 more examples of distributed web annotation prior
art (some semi-active) that I can think of, like
https://indieweb.org/marginalia. I can suggest these as pull requests
to the explainer if you'd like.

Ah, yes - I'm somewhat familiar with it from work around text-fragments/fragmentation. I'll take a closer look. I'd be happy to merge in any PRs. 
 
1. Lots of web annotation (including in browser) attempts / services /
sites try & fail (often repeating mistakes)

2. The few web annotation services that gain traction eventually
turning into yet another harassment vector for the web (pretty much
every such attempt designed/built by highly privileged individuals who
conceive more of the optimistic possibilities without any real
understanding of the threats to safety on the web, again repeating
mistakes of every social media site)

3. The overly complex attempt at standardization at W3C, which barely
has (some) interop, but at a high enough barrier as to cause more
centralization (yet another silo, Hypothesis) than decentralization.

4. Potential for misframing / loss of meaning. When you focus on "web
annotation", everything published to the web starts to look like a
"web annotation", at which point the phrase or even word "annotation"
becomes meaningless.

+1 to all those. In particular #2 is top of mind for us and the biggest challenge we have to solve / pitfall to avoid.

Also particularly curious if there's any thoughts/advice Re:W3C standard. I'm weary of trying to reinvent the wheel. My thinking was there's some useful technology standardized there, though it tries to be everything to everyone. The approach I was thinking to take is to find the minimal functionality we'd want to support and cherry-pick the bits of that standard needed to make it work, build on those.
 
> I think this is functionality that really relies on working across browsers

Maybe? I'm not sure about "relies on" (at least for
prototyping/incubation), and even in the case of small communities
wanting to interact amongst themselves, they're exploring peer-to-peer
methods for doing so, that are essentially differently styled web
comments.

For prototyping and running a trial it's maybe not a deal breaker. The problem is it's the kind of feature where I want to send a link to a friend to respond to my comments/annotations on a page -- I don't know (and shouldn't have to) what browser they're using. If it fails and causes confusion instead, even 1/10 times, it seems like it really kills the motivation to use it.
 
Hopefully the above is helpful. There's also a community of folks
actively publishing distributed "annotations" to the web based on
fairly simple composable open standards in the IndieWeb community that
has done a lot of analysis and documenting of their work:
https://indieweb.org/annotation, so you may want to also ask there:
https://chat.indieweb.org/dev

 It is, thank you! Will definitely poke around and ask around there.

Tantek Çelik

unread,
Sep 30, 2021, 4:39:08 PM9/30/21
to David Bokan, dev-pl...@mozilla.org, tan...@cs.stanford.edu
On Thu, Sep 30, 2021 at 12:11 PM David Bokan <bo...@chromium.org> wrote:
>
> Thanks for the reply!

No worries, this is a subject that touches on a lot!


> On Thursday, September 30, 2021 at 2:29:07 PM UTC-4 tan...@cs.stanford.edu wrote:
>>
>> Since you brought up "without need for extensions", I'd ask the
>> obvious question:
>>
>> Are there any browser extensions that do it *well*?
>>
>> (my experience to date has been no, for various reasons, which isn't a
>> reason not to do it, however does question that kind of framing, at
>> least as evidence of feasibility)
>
>
> This is, of course, subjective -- I've been using hypothes.is for the last few months and actually used it to decent effect to have a conversation about my explainer. I find the tool itself -- the UI and usage patterns in it -- to be quite intuitive and works well for the most part.
>
> The major down side I see is that it requires both parties interested in a discussion to be using this same tool and to know about it.

Indeed.

> Because of this (my assumption) - most of the web feels pretty barren using it. It's also service-locked -- meaning it's one broad tent for discussion. When I do finally stumble across a comment, it's usually irrelevant and from some far flung corner, not someone I know or interesting.

Service-locked and "one broad tent for discussion", hmmm... sounds
like another example of the Social Media silo problem.


> I expect this would be dramatically improved by having this functionality built-in in the browser which would provide more feedback, discovery, incentive to annotate.

Yes I agree with the goals of "more feedback, discovery, incentive to
annotate" (hopefully in a constructive & safe way).

However, I find it is a major leap of faith to assert "dramatically
improved by having this functionality built-in in the browser".

That is, the hard problems of "feedback, discovery, incentive" (and
others) IMO have very little to do with where annotations are being
implemented, whether in a browser, in a service, in a server, in a
federation of servers, or some combination thereof.

> Also allowing smaller ("denser") circles so commentary is more likely from people the user knows or at least is interested in.

This goal I also agree with!

And again, it is a major leap of faith that "built-in in the browser"
would have any impact on that goal whatsoever, or would be positioned
to do so. (Exception: there are some peer-to-peer browser experiments
that allow for different interactions among people on the same local
wifi, or within bluetooth range, that could be interesting to
consider.)

"smaller/denser circles [of people]" requires intimate social network
(or "graph" if preferred) knowledge that browsers don't have. Who does
have that knowledge are big Social Media silos.

Similarly "people the user knows or at least is interested in" is
literally an existing popular social media silo feature (and certainly
not a built-in feature of any browser I know of). A few examples
documented here:
https://indieweb.org/followers_you_know
https://indieweb.org/People_You_May_Know
https://indieweb.org/who_to_follow


>
> There are smaller issues with it too -- more technical in nature. e.g. annotations on feed based sites (Twitter, etc.) don't work at all.

Not sure what you mean?

There's an entire feature on Twitter that is essentially annotations:
Quote tweets.

This in addition to "replies" being a form of annotations themselves,
as are comments in general (more on that later).

Even retweets/likes of tweets are micro-annotations of a sort.


> These are things I think that are fixable but not necessarily show-stoppers.

I would disagree, there are at least two big show stoppers that you've
identified:

* user identity
* user social network knowledge (both explicit and implicit)

Both of which on their own are massive efforts in standardization (or
more than one effort).


> But I'm curious if you or others have thoughts on the existing landscape and where it's lacking.

Tbh, not really interested in any single-site based solutions whether
explicitly (Genius et al, see
https://www.niemanlab.org/2021/09/true-genius-how-to-go-from-the-future-of-journalism-to-a-fire-sale-in-a-few-short-years/
for the latest) or implicitly by way of too complex a "standard" for
anyone else to implement (Hypothesis).

I think there is promise in solutions which give a user ownership over
their annotations, whether public or private, and the permalinks to
those.

The #1 thing it's lacking it seems like you and I agree on (see below).


>> There's at least 1-2 more examples of distributed web annotation prior
>> art (some semi-active) that I can think of, like
>> https://indieweb.org/marginalia. I can suggest these as pull requests
>> to the explainer if you'd like.
>
>
> Ah, yes - I'm somewhat familiar with it from work around text-fragments/fragmentation. I'll take a closer look. I'd be happy to merge in any PRs.

Will do!


>> 1. Lots of web annotation (including in browser) attempts / services /
>> sites try & fail (often repeating mistakes)
>>
>> 2. The few web annotation services that gain traction eventually
>> turning into yet another harassment vector for the web (pretty much
>> every such attempt designed/built by highly privileged individuals who
>> conceive more of the optimistic possibilities without any real
>> understanding of the threats to safety on the web, again repeating
>> mistakes of every social media site)
>>
>>
>> 3. The overly complex attempt at standardization at W3C, which barely
>> has (some) interop, but at a high enough barrier as to cause more
>> centralization (yet another silo, Hypothesis) than decentralization.
>>
>> 4. Potential for misframing / loss of meaning. When you focus on "web
>> annotation", everything published to the web starts to look like a
>> "web annotation", at which point the phrase or even word "annotation"
>> becomes meaningless.
>
>
> +1 to all those. In particular #2 is top of mind for us and the biggest challenge we have to solve / pitfall to avoid.

This and so much this.

IMO solving 2 is "just" a special case of the problem of
comments/harassment on the web in general. Solving the latter IMO is a
prerequisite.

To be clear, this is not a "standards" problem or a "technical"
problem. This a massive human design problem that *may* imply some
technical architectural aspects/directions.

Jumping into technical proposals/brainstorms without actually
addressing this problem first will only create yet another way to
propagate (and potentially amplify) harm, especially at scale, as
we've seen from the existing social media hellscape and its impacts on
large scale "real world" things like elections, to pandemic/vax denial
etc.


> Also particularly curious if there's any thoughts/advice Re:W3C standard.

It's a bloated monolith that tried to do too much. I wouldn't waste
any time attempting to implement any of it. It's got *some* good
concepts, and may be useful as prior research.

> I'm weary of trying to reinvent the wheel.

Strongly agreed!

> My thinking was there's some useful technology standardized there,

I'd disagree. There's a prototype standardized there that enabled
*one* "useful" service, Hypothesis.

> though it tries to be everything to everyone.

Agreed, which is of course an impossible goal, since new use-cases and
interactions will always be emerging in the future.


> The approach I was thinking to take is to find the minimal functionality we'd want to support and cherry-pick the bits of that standard needed to make it work, build on those.

I'd see it more as a warning roadmap. If you find yourself
re-inventing W3C Web Annotations with different bits/bytes/syntax,
you've gone wrong. Start over.

There may be some useful concepts and vocabulary (like English
vocabulary, not values of properties) to re-use, but even most of
those pre-date W3C Web Annotations.


>> > I think this is functionality that really relies on working across browsers
>>
>> Maybe? I'm not sure about "relies on" (at least for
>> prototyping/incubation), and even in the case of small communities
>> wanting to interact amongst themselves, they're exploring peer-to-peer
>> methods for doing so, that are essentially differently styled web
>> comments.
>
>
> For prototyping and running a trial it's maybe not a deal breaker. The problem is it's the kind of feature where I want to send a link to a friend to respond to my comments/annotations on a page

Yes, this is a good example of a use-case, and gets at my point above
about prioritizing:
* user ownership over their annotations, whether public or private,
and the permalinks to those (that they can share with friends to allow
viewing, responding etc.)

> -- I don't know (and shouldn't have to) what browser they're using. If it fails and causes confusion instead, even 1/10 times, it seems like it really kills the motivation to use it.

Agreed, and this is part of "URLs should just work because people
expect to be able to share them and have them work" (in any browser,
device).

However, there are interesting opportunities to design systems that
allow for good fallback, e.g.
* send a link to a friend to respond to my comments/annotations on a page
could fallback to:
* send a link to a friend to view my comments/annotations about a page

Which gets back to my point about annotations being a *publishing*
problem at a technical level (in addition to the larger harassment
problem noted previously).

For that particular modular technical problem (how do we allow a user
to *publish* something to the web, public or private, at a permalink
they control), there's the existing Micropub spec (which has quite the
growing ecosystem):
https://www.w3.org/TR/micropub

Per your previous point about "minimal functionality we'd want to
support", this is where I'd start as a foundation for creating
annotations ("send a link to a friend to view my comments/annotations
about a page"), as it's a protocol that's designed for user identity &
ownership of content/permalinks.

>> Hopefully the above is helpful. There's also a community of folks
>> actively publishing distributed "annotations" to the web based on
>> fairly simple composable open standards in the IndieWeb community that
>> has done a lot of analysis and documenting of their work:
>> https://indieweb.org/annotation, so you may want to also ask there:
>> https://chat.indieweb.org/dev
>
>
> It is, thank you! Will definitely poke around and ask around there.

Great! I'm active there too, and look forward to chats there.

Thanks,

Tantek

Karl Dubost

unread,
Sep 30, 2021, 9:32:58 PM9/30/21
to David Bokan, dev-pl...@mozilla.org, Tantek Çelik
David,

To second what Tantek said:

Le 2021/10/01 à 05:38, Tantek Çelik <tan...@cs.stanford.edu> a écrit :

> IMO solving 2 is "just" a special case of the problem of comments/harassment on the web in general. Solving the latter IMO is a prerequisite.

This is the first question to address.

* As a site owner, I do (not) want to see random annotations
* As a recipient of someone else’s annotation, I do (not) want to read about random annotations.

Everything which is related to
Decentralized moderation in the explainer.

Another angle which is also complex to solve is the balance in between identification (for avoiding harassment) and privacy for people who are already targeted (one browser/one person).


--
Karl Dubost, mozilla 💡 Webcompat
https://www.la-grange.net/karl/moz






Reply all
Reply to author
Forward
0 new messages