Sharing Server

[Josh Marinacci]

This is the initial specs for the Leonardo Sharing server.

The vision:

Leonardo is about bridging the web with desktop apps, and providing useful features to 21st century users.  What better way to help people than letting them share reusable symbols. This sharing server will let you share symbol sets you've created with other users, and search for sets they've created to use in your own documents.  This isn't meant to replace the share via Twitter & Flickr commands. Those are for sharing finished works. The sharing server is for sharing symbols and eventually other reusable assets such as icon, fonts, color swatches, and scripts; all without having to leave the editor.  If you are working on a network diagram, you could search for a set of network diagram symbols someone else has created. You could also add your own symbols and share back the improved version.  That's the beauty of the web, brought to the desktop.

Requirements:  

This is a general list of features, not necessarily what will go into the first version.  

* upload a binary payload with metadata. initially this payload with be a leoz file, and the metadata will be:

     all types:

     email/username of creator

     date of creation

     mimetype of entity (just set or symbol for now)

     unique id of binary payload

     symbols:

     name of symbol

     tags of symbol

     description of symbol 

     sets:

     name of set 

     list of symbols in the set

     description of set

     tags of set

* search through all symbols and sets by name, description, and tags.

* show a thumbnail preview of what's in the symbol set before downloading it

* download a set and add it to your symbol panel

* receive updates to a downloaded set if it's updated / forked

* modify a symbol set you've downloaded and reupload it as a fork

* rate a symbol set after you've used it

* track the author & copyright of a symbol set (initially we will allow only creative commons licensed symbols with derivative use allowed)

* let the user create an account from within Leo

* let the user use OpenID or OpenAuth so they don't have to create yet another account

* make the webservice have a JSON feed so that others can build frontends to it

* track usage of a symbol so you can see how many people like the stuff you've created

* support other binary payloads such as color swatches, photos, fonts, plugins, and scripts

For now, let's keep all discussion of the sharing server on this thread.

thx,

J

Blasting forth in three part harmony!

[Juliano]

Hi Josh,

Seems to me like this is going to be something like a SaaS offering -
there will be a public sharing server where all Leonardo users can
upload/download content form. Am I right? In this case, do you think
people should also be able to use private sharing servers?
It seems to me this service could (and should) be developed as a cloud
service. I believe Google AppEngine would be the perfect candidate for
hosting it, but I would be interested in developing an architecture
that can be deployed on different cloud providers if needed.

[Josh Marinacci]

Yes, the idea is there will be a shared public server. I like the idea of having private servers too, though. It would be handy for people working inside of a company. In fact, the client side UI should probably let you hook into multiple servers if you want. I'd love for the server to work with multiple cloud providers.

- Josh

[Ivo Limmen]

That is a big pile of features!

It would be easier to implement if you try to divide the work into
several milestones. For instance:

Fase 1: The ability to upload/download and simplistic search on name.
Add account control. But for this release use the (almost) complete
data and meta-data.

Fase 2: Add more searching options.

Fase 3: Add the ability to rate.

Fase 4: Add the ability to check for updates.

I personally would recommend to use JSON as communication protocol for
everything. Same reason you say to want to add it as an extra feature.
A different server can be implemented in another language and replace
the current very easily.

I would recommend to design it so that you can add 'repositories' such
as private ones. That is also why JSON would be a great protocol.

Best regards,
Ivo Limmen

[filsanet]

Josh,

Some comments:

- Google App Engine +1
- multiple cloud providers +1

Thanks for laying out the vision, it is good to know where you want to
go with the sharing server. It is a big feature list, though. Can we
define the minimum viable product for the sharing server?

Regards
Phil Suh

[Collin Fagan]

I think apache solr would be a good way to get a lot of the functionality you want.

http://lucene.apache.org/solr/

From their web site:

Solr is the popular, blazing fast open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, and rich document (e.g., Word, PDF) handling. Solr is highly scalable, providing distributed search and index replication, and it powers the search and navigation features of many of the world's largest internet sites.

The indexing, searching and rich document handling seems to fill many of your requirements. Unfortunately all I can do is suggest as work has kept me from contributing the way I'd like to.

Good luck!

Collin

[Josh Marinacci]

Oh yes. This is just a vision doc. To start with we have to build something smaller. I'm thinking the Minimum Viable Product would be:

client hook and prototype server that can:
* create a new user account
* upload a set with a name and description
* search for sets by name and description
* download a set

That should be enough to start designing a JSON API.

-j

Blasting forth in three part harmony!

[Ivo Limmen]

Phil,

Trouble with Google AppEngine is that the load expected in the future
might be costly. This is also true for other cloud services. Google is
by far the cheapest and is even free for the first x request per day
but it is difficult to say what the load will be.
If the application and this feature becomes populair and billing is
off people will get errors. With billing on... who knows that the
costs will be.

Best regards,
Ivo Limmen

[Ivo Limmen]

Collin,

That is just a small subset of the required features and can not be
the basis of the tooling criteria. Storage isn't even discussed so how
we will implement searching is not that important.

Best regards,
Ivo Limmen

On Dec 10, 3:22 am, Collin Fagan <collin.fa...@gmail.com> wrote:
> I think apache solr would be a good way to get a lot of the functionality
> you want.
>
> http://lucene.apache.org/solr/
>
> From their web site:
>

> *Solr is the popular, blazing fast open source enterprise search platform

> from the Apache Lucene project. Its major features include powerful
> full-text search, hit highlighting, faceted search, dynamic clustering,
> database integration, and rich document (e.g., Word, PDF) handling. Solr is
> highly scalable, providing distributed search and index replication, and it
> powers the search and navigation features of many of the world's largest

> internet sites. *

>
> The indexing, searching and rich document handling seems to fill many of
> your requirements. Unfortunately all I can do is suggest as work has kept me
> from contributing the way I'd like to.
>
> Good luck!
>
> Collin
>

[Ivo Limmen]

Josh,

I agree. I do want to start designing the protocol and writing the
server software. But we also need to discus the storage. Do we go
cloud/nosql or use a normal database. We can always scale-out using a
database or even convert to a nosql solution in the future (but it is
costly in time).
I am interested in discussing the possibilities in hosting. Where is
the service going to run? Is there some server already available? If
there is a server already available we could look at solutions that
could utilize the available server. We might even consider writing the
service in PHP. It's not my thing but I wouldn't mind. I am a
practical kind of guy. I mainly develop in Java, C# and Javascript.

Best regards,
Ivo Limmen

[filsanet]

Hi Ivo!

I should have explained more:

- I would be interested in making the client-server protocol for
Leonardo work for multiple clouds/platforms/servers.
- I personally would like to do an implementation in AppEngine. (but
would not want to foot the bill for it in the future :-)

Cheers, Phil

[beise]

Another aspect

I understand, that this is a technical discussion, but I'm from the
old school, and for me Privacy is sacred.
Please do not forget.


Andreas

[Ivo Limmen]

Phil,

I have experience in developing AppEngine applications and would like
to assist if required.

We could design the protocol to be platform/cloud/etc. independent.

Best regards,
Ivo Limmen

[Ivo Limmen]

Andreas,

I might be relatively young (35) but I can assure you that my
upbringing is also very old-school. My country (the Netherlands) is
also very strict on privacy so that would be no problem.

Best regards,
Ivo Limmen

[Collin Fagan]

Solr is search + storage + http api + rich document indexing. It's more nosql then just searching. Add security and you have most of Josh's requirements for the back end. This is just a suggestion, but I have a feeling that it will make what you are trying to do trivial.
 
Collin

[beise]

hi Ivo,

Yes, I believe this.
I am not the voice in the wilderness, but I like to have control of
my data. And better opening my mouth before .
On the other hand, I am 53 and see so much more critical than others.

Other Topic:
If anyone needs QA support for Leo and environment, please contact.

best regards
Andreas

[Juliano Viana]

I believe that the hosting cost problem is going to arise no matter what the platform. GaE only makes it more explicit.
If you select a "unlimited" hosting provider and starts having a lot of traffic the provider will just cap the bandwidth and performance will suffer.
The way to tackle this would be to let people donate bandwidth to Leonardo by hosting Shared Servers. This could be achieved on a p2p fashion (like BitTorrend/GnuTella) but I´m afraid p2p would not provide the level of predictability people would expect.
The other would be to allow mirrors of the Shared Server content. Since most of the time people would be reading content (or searching) instead of writing, we could design an architecture that would have a "master"  server (that would allow writes) and "slave"  servers that would be read-only from the user point of view. The master server can then have permission to replicate its content to slave servers when someone uploads a new symbol set.
The load balancing in this case could be handled on the client, that would retrieve a mirror list from the master server and connect to one at random (or to the closest one if geolocation is available).

Regards,
  - Juliano

[filsanet]

>> upload a binary payload with metadata. initially this payload with be a leoz file, and the metadata will be

JSON seems like a great match for the metadata, but the leoz file will be binary, and JSON doesnt seem great for embedding binary data. I mean, you can do it, but it is a little weird. Because the leoz file is zipped, and then to survive in the json container it needs to be converted to string, (Base64 encoding?) Feels clumsy to me.

otoh, for search and getting result listings, JSON is just fine. For downloading the payload, the search result can contain the url for the download, and do a second connection to get the payload.

just thinking out loud...

[Josh Marinacci]

How's this for an initial spec. 

Get Server Info

/info

Returns info on the server as a JSON struct. Mainly used to let the user know they have successfully connected to their server of choice and that it is live. JSON struct would contain the name of the server, a description, and a timestamp to help estimate latency.

Searching

/search?text=<free form text search>

Returns a list of results as a JSON struct. The query searches the name and description of all available sets. The struct contains a list of results. For each result it has the name, author, creation date, description, tags, size in bytes, and a payload id.

Downloading

/download?id=<payload id>

Returns the binary LeoZ file with the specified ID.

Uploading

/upload?metadata=<metadata>&payload=<payload>&author=<authorid>&password=<password>

Uploads a LeoZ file via POST. The metadata is a JSON struct containing the name, author, description, keywords, and creation date.  The author is the user's authorid.  The password is the user's password.  

In the interest of security all requests are POST requests over HTTPS. That way everything is secure and no info will show up in proxy logs.  Only uploading requires an authorid and password. Everything else is unauthenticated, but still over HTTPS.

- Josh

[Josh Marinacci]

I hadn't thought that far yet. :)

For now it will run on the server I already have, a virtual linux box, probably as a J2EE app in Tomcat, though I'm open to anything. If we get so popular that we outgrow the server, then we'll simply have to move. That's a problem I'm not going to worry about just yet. We only have maybe 20 users so far. :)

- J

[Josh Marinacci]

Can you explain to me what you mean by privacy in the context of the Sharing Server?

[Andreas Beisemann]

yes, I'll try that in short words. Not another facebook, please. More right's for me as user of my data.

And I know that the discussion about sharing is different than in Europe.

Andreas

 

2010/12/10 Josh Marinacci <jos...@marinacci.org>

[Josh Marinacci]

:) Could you try longer words then?  I'm still not sure what you mean.

The sharing server is for sharing Leonardo assets from within Leonardo.  It will not even have a website (at least not in the current plans).  You will need to have a valid email address registered with the site, but it won't be exposed to anyone else using the service.   Of course the drawings you share will be shared with the world, but that's sort of the point.  We well definitely need to design the UI to make it explicit what you are currently sharing and what is private.

Does that address the concerns? What other privacy issues should we be aware of in our design?

- Josh

[beise]

The sharing server is for sharing Leonardo assets from within
Leonardo.  It will not even have a website (at least not in the
current plans).  You will need to have a valid email address
registered with the site, but it won't be exposed to anyone else using
the service.   Of course the drawings you share will be shared with
the world, but that's sort of the point.  We well definitely need to
design the UI to make it explicit what you are currently sharing and
what is private.

Does that address the concerns? What other privacy issues should we be
aware of in our design?

>>Yes.

Other examples:
In Bug 45 I've criticized that Leo stores data without permission.
That was not ok. Let leo ask.
OK is that Leo ask (for tracking e.g.) and gives information about.

Andreas

[beise]

> We only have maybe 20 users so far. :)

Maybe that's help to become more users, if we translate nontechnical
parts of the webside?

Andreas

[Josh Marinacci]

Yes, but that's a larger issue. The website right now is really meant for early adopters. We need to redesign the website for real authors, including tutorials and help text.

- Josh

>
> Andreas

[Josh Marinacci]

>
>>> Yes.
>
> Other examples:
> In Bug 45 I've criticized that Leo stores data without permission.
> That was not ok. Let leo ask.

The data you referred to is the image cache for the Flickr search, similar to a browser cache. What information should we tell / ask the user about regarding caching. Would it be okay if we stored the cache in a new place? What about a button in the prefs to clear the cache?

- Josh

> OK is that Leo ask (for tracking e.g.) and gives information about.
>
> Andreas

Blasting forth in three part harmony!

[Andreas Beisemann]

Am 10.12.2010, 20:06 Uhr, schrieb Josh Marinacci <jos...@marinacci.org>:

>>
>>>> Yes.
>>
>> Other examples:
>> In Bug 45 I've criticized that Leo stores data without permission.

My first reaction was angry because I did not know what happened there.

Would it be okay if we stored the cache in a new place? What about a
button in the prefs to clear the cache?

Your solution was to create a temporary directory and delete the data
afterwards. That was great. But you must communicate that.

> The data you referred to is the image cache for the Flickr search,
> similar to a browser cache. What information should we tell / ask the
> user about regarding caching.

What about a button in the prefs to clear the cache?

The button is one nice idea. Or you use blank space as billboard.

Andreas

[filsanet]

On Friday, December 10, 2010 11:50:52 PM UTC+9, filsanet wrote:

>> upload a binary payload with metadata. initially this payload with be a leoz file, and the metadata will be

JSON seems like a great match for the metadata, but the leoz file will be binary, and JSON doesnt seem great for embedding binary data. I mean, you can do it, but it is a little weird. Because the leoz file is zipped, and then to survive in the json container it needs to be converted to string, (Base64 encoding?) Feels clumsy to me.

After some sleep, I realized that the POST can be a multipart/form-data  media type which allows for a content type setting for the file data, and the meta data sent in a different part of the response. So no Base64 armoring is necessary.

[TJB]

Definitly, Solr would be the best option. Take some time to investigate as it does what you want and more when you want it.

Thom

[Ivo Limmen]

John,

Well for one thing: it might be an option to remove all your account
and published data.

And the ability to flag other uploads as inappropriate (instead of
only rating them) is also a much used feature that is welcome.

Best regards,
Ivo Limmen

[filsanet]

I am making headway on a reference implementation for the sharing server.

I had hoped to get it done tonight but ran into the usual probably getting it all warred up and ready to go.

I have some thoughts on the api as well but that should wait till i get more sleep.

hopefully will have something out in a couple days...

phil

[Ivo Limmen]

Josh,

I would add:

/createAccount?...

/deleteAccount?removeData=true|false

Then there also should be an option for the administrators to remove illegal uploaded content. Content uploaded by someone that does not own the content. So account are either user or admin types.

Best regards,
Ivo Limmen

[Ivo Limmen]

What if we implement two type of servers?

Index server:

/registerServer?url=...&type=mirror|localized&country=...
=> returns id

url: public url of the server
type: mirror=mirror of public server, localized means country specific content (Dutch flags, Dutch Logo's all in country=nl).
country: ISO standard country code.

/unregisterServer?id=[id]

/getServer?type=mirror|localized&country=...
=returns server based on request


We would need to create another protocol for syncing the data cross mirrors. But that has been done before...

This server could be implemented in Google AppEngine and would be less costly as it only indexes other servers or simply spreads the load...

Best regards,
Ivo Limmen

[filsanet]

Obviously, I'm late with this. Due to holidays and family obligations, I probably wont have anything till next week.

Sorry about that.