Intelligent clipboards that enables better web browser <==> desktop application communication
fizk
This morning I had an idea for "intelligent" clipboards that enables
better web browser <==> desktop application communication. A
"clipboard" is what allows you to copy/paste between different
applications.
Here's the main example:
1. Chat client to web browser:
• When you copy/paste an IM message (eg. a Yahoo Chat message) into
your browser, the browser should see something like:
<xml>
<copypaste id="xxx123" type="im_chat">
<network>Yahoo Chat</network>
<from>john...@yahoo.ca</from>
<to>yon...@gmail.com</to>
<timestamp>1232883043</timestamp>
</copypaste>
• The IM client would broadcasts this information over D-Bus (http://
www.freedesktop.org/wiki/Software/dbus)
• The web browser listens for D-Bus copy/paste messages on your
computer
• The web browser relays the message to open windows via javascript
2. Web browser to chat client:
• When you copy/paste a specially crafted <div> element on a webpage,
extra contextual information will be available
• The web browser sends copy/paste notification over D-Bus
• As the user drags the content over his IM window, the content may
change in appearance
-- other software plugins may automatically add pictures, personal
information, fetch extra information (like artist, etc), or just
format the message in a certain way
• The user finally drops the message in the chat window and sends it
to his friend
It's like enabling drag and drop for anything on a webpage. Imagine
drag 'n dropping a YouTube video to another website or another desktop
application and without any effort, it still works and displays as a
YouTube video. Wohooo! :)
By using this extra contextual information, users will not be dealing
with separate pieces of text and separate images anymore. The
information's intelligence is kept intact through transit over
multiple applications and multiple users!
Copy and pasting need not degenerate into simple words and image
objects anymore. We need intelligent clipboards. As of today,
clipboards allow one application to be the source of the clipboard
data, and another application to choose which format it wants to get
that data in.
But there's no standardization beyond that.
No standard way for developers to write plugins that would
automatically enhance your clipboard data.
No standard way to specify contextual information to clipboard
information so that a website can readily make use of that extra
contextual information.
We should be creating a standard to specify the format of rich
contextual information so that javascript libraries could parse the
contextual information for the website to easily consume and vice
versa.
I'm sure businesses would love the idea of having their sales
information readily available and continually presented beautifully as
the user takes it from their website to their blog, from that blog to
this IRC channel, and from.....etc.
More examples:
You could show the user a button "Share this to Digg, Facebook,
MySpace, etc" and post it to whatever social network they have an
account on and whatever is posted would __retain__ the rich
contextual information that came with it! What would that
information be?
Let's say you were on IRC....the contextual information, specified in
a standard XML format, could be formatted as
MrRoboto42 in #diso on Freenode IRC said, "How about those Yankees!"
at 12:34 PM, Jan. 28, 2010
which would have been parsed from XML such as:
<xml>
<copypaste id="xxx123" type="irc_chat">
<network>Freenode</network>
<from>MrRoboto42</from>
<to>#diso</to>
<timestamp>1232883043</timestamp>
</copypaste>
- Yonas
David Karger
recall that took advantage of microformat data representations in html.
On 1/28/2010 10:55 AM, fizk wrote:
> Hi everyone,
>
>
> This morning I had an idea for "intelligent" clipboards that enables
> better web browser <==> desktop application communication. A
> "clipboard" is what allows you to copy/paste between different
> applications.
>
>
> Here's the main example:
>
>
> 1. Chat client to web browser:
>
> � When you copy/paste an IM message (eg. a Yahoo Chat message) into
> your browser, the browser should see something like:
>
> <xml>
> <copypaste id="xxx123" type="im_chat">
> <network>Yahoo Chat</network>
> <from>john...@yahoo.ca</from>
> <to>yon...@gmail.com</to>
> <timestamp>1232883043</timestamp>
> </copypaste>
>
> � The IM client would broadcasts this information over D-Bus (http://
> www.freedesktop.org/wiki/Software/dbus)
> � The web browser listens for D-Bus copy/paste messages on your
> computer
> � The web browser relays the message to open windows via javascript
>
>
> 2. Web browser to chat client:
>
> � When you copy/paste a specially crafted <div> element on a webpage,
> extra contextual information will be available
> � The web browser sends copy/paste notification over D-Bus
> � As the user drags the content over his IM window, the content may
> change in appearance
> -- other software plugins may automatically add pictures, personal
> information, fetch extra information (like artist, etc), or just
> format the message in a certain way
>
> � The user finally drops the message in the chat window and sends it
fizk
This site supposedly represents Live Clipboard, but all the links are
dead:
http://www.liveclipboard.org/
I have a feeling that Ray Ozzie didn't go as far as enabling plugin's
to be notified of a copy/paste operation (eg. via D-Bus) and modify
the data.
On Jan 28, 10:59 am, David Karger <david.kar...@gmail.com> wrote:
> Ray Ozzie's "Live Clipboard" from a few years back seems relevant. If I
> recall that took advantage of microformat data representations in html.
>
> On 1/28/2010 10:55 AM, fizk wrote:
>
> > Hi everyone,
>
> > This morning I had an idea for "intelligent" clipboards that enables
> > better web browser <==> desktop application communication. A
> > "clipboard" is what allows you to copy/paste between different
> > applications.
>
> > Here's the main example:
>
> > 1. Chat client to web browser:
>
> > When you copy/paste an IM message (eg. a Yahoo Chat message) into
> > your browser, the browser should see something like:
>
> > <xml>
> > <copypaste id="xxx123" type="im_chat">
> > <network>Yahoo Chat</network>
> > <from>john....@yahoo.ca</from>
> > <to>yona...@gmail.com</to>
> > <timestamp>1232883043</timestamp>
> > </copypaste>
>
> > The IM client would broadcasts this information over D-Bus (http://
> >www.freedesktop.org/wiki/Software/dbus)
> > The web browser listens for D-Bus copy/paste messages on your
> > computer
> > The web browser relays the message to open windows via javascript
>
> > 2. Web browser to chat client:
>
> > When you copy/paste a specially crafted <div> element on a webpage,
> > extra contextual information will be available
> > The web browser sends copy/paste notification over D-Bus
> > As the user drags the content over his IM window, the content may
> > change in appearance
> > -- other software plugins may automatically add pictures, personal
> > information, fetch extra information (like artist, etc), or just
> > format the message in a certain way
>
Stephan Sokolow
it'd be a good, interoperable way to implement various functionality
currently implemented by browser extensions. (eg. copying snippets of
content with a citation URL and optional metadata)
However, I think it'd have to be two separate specs: A native desktop
spec, extending the clipboard in a manner similar to POSIX xattrs or
MacOS resource forks and a web spec extending the API the browser
exposes.
For the former, it shouldn't be too difficult (aside from getting
adoption by apps) but I'm not sure about D-Bus. Even omitting the
synchronization issues using an out-of-band channel would inevitably
cause, at present, it's primarily used on desktops that already use
X11. Given the lack of a cross-platform usability advantage, it'd
probably be more architecturally appropriate to implement said
functionality by extending the existing clipboard mechanism somehow.
At the very least, it would avoid a lot of potential headaches that
come from having to keep multiple transmission channels in sync. (eg.
bugs or resource starvation causing one channel to drop the message or
backlog traffic long enough to confuse things or cause GUI latency in
an application which wasn't written with that in mind. I've had half
of KDE 3.5 freeze up because dcopserver hung and the apps were written
using blocking reads)
As for the web side of things, that also shouldn't be too difficult.
The Javascript drag-and-drop and clipboard-access APIs should be easy
to extend, but I'm not sure how you'd implement it for things like
hitting Ctrl+V on a <textarea>. Probably by defining a new Javascript
event on elements which accept pastes that receives the metadata and
can then handle it as appropriate for the web app in question. Maybe
also with some kind of additional attribute for contentEditable
elements which instructs the browser to convert the metadata to
attributes on a <span> for graceful degradation with JS disabled.