I have the answer you're all looking for.

[jez0990]

...or at least answer that is compatible with everything I've seen
discussed here.

I need to get the attention of specialised developers to accelerate
how quickly I can make this happen - does anybody in this forum have
personal connections relevant to CouchDB?

[Mark Roest]

Hello Jez,

We discuss a few different topics and projects here. Could you please be more specific about what you want to do, and what you plan to ask when you "get the attention" of developers?

Regards,

Mark

[Jeremy Taylor]

Hi Mark, thanks for replying. I guess I'm talking about
decentralisation of everything - which is why I think my message is
generally applicable to the group. So decentralisation of the
Internet, of economy, and society at large...

It will happen through the development of a new paradigm in software
technology (I'd argue as being the next level 'above' software and
hardware) - in some ideal use of the phrase a truly new "operating
system". I understand not only where we must head (integration of the
species into a harmonious organism), but also how we actually get
there from a 1's and 0's practical perspective. I'd happily go into
further detail with people that can help me develop this using CouchDB
but I hope you can understand that I don't want to risk
over-propagating all these new ideas too early. Once I'm a little
further a long and I have set up a governing non-profit organisation
to act as a more capable mouthpiece I will definitely get back in
touch with this forum to seek your input :-)


Jeremy

[Poor Richard]

Jeremy,

If you search the past messages in this group for "couchdb" you'll find several members who have mentioned it in some context and you may be able to contact them by email. But I don't know if you can generate much interest without "propagating" your idea a bit more than you have thus far.

Since you've read some of the PeerPoint proposal I'd be very interested in a brief synopsis of how you think CouchDB might fit in as a possible component of that project. I haven't been able to investigate it in any detail yet.

Good luck with your project,

Richard

[Nathan Rixham]

Likewise - very interested to hear what you'd propose, and FWIW I'm a
half decent developer who also knows couchdb.

Best, Nathan

[Poor Richard]

Nathan,

Any thoughts about how well CouchDB would fit the PeerPoint requirements as compared to other distributed data stores?

Richard

[Melvin Carvalho]

On 14 July 2012 15:14, Poor Richard <poor.r...@gmail.com> wrote:

Nathan,

Any thoughts about how well CouchDB would fit the PeerPoint requirements as compared to other distributed data stores?

CouchDB isnt a bad RESTful js store.  Advantages are that it's fast, scalable, reliable.

However iirc it doesnt let you store freeform javascript.  It's limited to exactly one object (js/json normally allows multiple objects) and it inserts _rev and _id for versioning that your software needs to keep track of.

That makes some RESTful aspects of the web harder achieve.  e.g. the idea of multiple objects in a document which is key to flexibility the web, needs to be rearchitected, and the representation that you put is, is not what you get out, so you cant do something as simple as saving a file (HTTP PUT) without getting a bespoke format. 

I'm not sure it's possible to turn of versioning, so while couch is awesome for some purposes, it makes it a challenge to scale massively, imho.
 

[Mark Roest]

Hello Jeremy,

I have given about 25+ years of thought to this (the information technology side of it, as well as how to design sustainable economies, which I started with a lecture course with Bucky Fuller in 1966), and you can find some of my recent work by searching for my inputs here and in Coalition of the Willing's Open Stewardship forum (about a year ago), and focusing on the word knowledgebase. Also, you might check DesignEarth (dot net).

Regards,

Mark 

[Nathan Rixham]

Hi,

Melvin has kindly covered some of the technical detail of couchdb
already, thanks!

The main thing I'd like to mention, is that I strongly feel that tightly
coupling to any specific data stores will pretty much doom this project
from the beginning.

However if you were to take the approach of documenting and
standardizing the intersection of several (distributed) datastores, and
create a generic storage API/interface/protocol, then you would have a
key component of this project generalized and made universal, thus
allowing implementers to use whatever technology they choose behind the
interface, couchdb or otherwise.

HTTP is a very good protocol and would server as a good base for
whatever you do, it's certainly ubiquitous enough. The REST dissertation
covers the properties needed for standardized async network
communication within a project like this. The CREST (Computational REST)
goes one step further and outlines how the REST architectural style can
be used to implement distributed service compositions in continuation
style, something which I also feel would aid this project. The
approaches of SIP, with SUBSCRIBE and NOTIFY cover another side of
things needed.

Ultimately all of this is achievable via the standardization of
interfaces, protocols and media types. Augment what already exists, and
where necessary decouple things which are already centralized with a
single point of control/failure (e.g. DNS).

The core of all of this are items 1-8 from "First tier services &
applications" within the PeerPoint document - and they correspond
directly to the work being done to create the Read Write Web of Data.

If this group and the people in the RWW group (of which Melvin is chair)
were to put our heads together and cover items 1-8 over the next year or
so, I'm positive that we'd make a markable impact on the shape of the
web/internet, and be a huge part of the way to enabling a more P2P world.

Best,

Nathan

[Poor Richard]

Melvin,

Thanks, that's very useful information.

PR

On Saturday, July 14, 2012 8:34:15 AM UTC-5, melvincarvalho wrote:

[Poor Richard]

On Saturday, July 14, 2012 11:52:06 AM UTC-5, nathan wrote:

Hi,

Melvin has kindly covered some of the technical detail of couchdb
already, thanks!

Ditto!

 

The main thing I'd like to mention, is that I strongly feel that tightly
coupling to any specific data stores will pretty much doom this project
from the beginning.

However if you were to take the approach of documenting and
standardizing the intersection of several (distributed) datastores, and
create a generic storage API/interface/protocol, then you would have a
key component of this project generalized and made universal, thus
allowing implementers to use whatever technology they choose behind the
interface, couchdb or otherwise.

Great suggestion, Nathan. Any suggestions as to the other dist data stores (dds)? What of these (or others):

Miscellaneous:

• OpenLink Data Spaces
• HBase
• Freenet

Distributed data stores (Wikipedia)

• Apache Cassandra, former data store of Facebook
• BigTable, the data store of Google
• Dynamo of Amazon
• HBase, current data store of Facebook's Messaging Platform
• Riak
• VelocityDB (.NET)
• Voldemort

Peer network node data stores

• BitTorrent
• Chord project
• GNUnet
• Freenet
• NNTP (the distributed data storage protocol used for Usenet news)
• Mnet
• Storage@home
• Tahoe-LAFS

Distributed hash tables (per wikipedia)

DHT protocols and implementations

• Apache Cassandra
• BitTorrent DHT (based on Kademlia as provided by Khashmir^[13])
• CAN (Content Addressable Network)
• Chord
• Kademlia
• Pastry
• P-Grid
• Tapestry
• TomP2P

Applications employing DHTs

• BTDigg: BitTorrent DHT search engine
• CloudSNAP: a decentralized web application deployment platform
• Codeen: Web caching
• Coral Content Distribution Network
• Dijjer: Freenet-like distribution network
• FAROO: Peer-to-peer Web search engine
• Freenet: A censorship-resistant anonymous network
• GNUnet: Freenet-like distribution network including a DHT implementation
• JXTA: Opensource P2P platform
• maidsafe: C++ implementation of Kademlia, with NAT traversal and crypto libraries. On its home page listed as "Available as a technology licence and a software solution written in cross platform C++."^[14]
• Oracle Coherence: An In Memory Data Grid built on a Java DHT implementation
• Retroshare: a Friend-to-friend network^[15]
• WebSphere eXtreme Scale: proprietary DHT implementation by IBM,^[16] used for object caching
• YaCy: distributed search engine
• i2p-Bote: Decentralised email service which stores encrypted messages in a DHT.

 

HTTP is a very good protocol and would server as a good base for
whatever you do, it's certainly ubiquitous enough. The REST dissertation
covers the properties needed for standardized async network
communication within a project like this. The CREST (Computational REST)
goes one step further and outlines how the REST architectural style can
be used to implement distributed service compositions in continuation
style, something which I also feel would aid this project. The
approaches of SIP, with SUBSCRIBE and NOTIFY cover another side of
things needed.

Thanks for connecting so many dots!

 

Ultimately all of this is achievable via the standardization of
interfaces, protocols and media types. Augment what already exists, and
where necessary decouple things which are already centralized with a
single point of control/failure (e.g. DNS).

The core of all of this are items 1-8 from "First tier services &
applications" within the PeerPoint document - and they correspond
directly to the work being done to create the Read Write Web of Data.

Thanks for the confirmation. Unfortunately, that big picture is about as far as my background gets me on my own. I appreciate the added layers of detail you provided.

 

If this group and the people in the RWW group (of which Melvin is chair)
were to put our heads together and cover items 1-8 over the next year or
so, I'm positive that we'd make a markable impact on the shape of the
web/internet, and be a huge part of the way to enabling a more P2P world.

Best,

Nathan

I love the optimism and the confirmation that PeerPoint is making some sense.
Maybe I should separate 1-8 and 9-17 into separate tiers. 

I wonder if you and Melvin could make a copy of the PeerPoint doc and pare it down to the parts that would most apply and appeal to the RWW group, adding whatever makes sense to round out that particular "view" of the project? I'll help in any way I can, but my progress is pretty slow due to the amount of research I have to do.

Again, thanks for the input and encouragement!

Richard

[Nathan Rixham]

Poor Richard wrote:
> On Saturday, July 14, 2012 11:52:06 AM UTC-5, nathan wrote:
> Great suggestion, Nathan. Any suggestions as to the other dist data stores
> (dds)? What of these (or others):
>
> Miscellaneous:

> - OpenLink Data Spaces <http://ods.openlinksw.com/wiki/ODS/>

ODS + Virtuoso are by far the most advanced products on the web today,
they're light years ahead of everybody else, indeed using Virtuoso_ODS
only you could build most of PeerPoint ;)

> Distributed data stores (Wikipedia)<http://en.wikipedia.org/wiki/Distributed_data_store>
>
> - Apache Cassandra <http://en.wikipedia.org/wiki/Apache_Cassandra>,
> former data store of Facebook <http://en.wikipedia.org/wiki/Facebook>
> - BigTable <http://en.wikipedia.org/wiki/BigTable>, the data store of
> Google <http://en.wikipedia.org/wiki/Google>
> - Dynamo <http://en.wikipedia.org/wiki/Dynamo_%28storage_system%29> of
> Amazon <http://en.wikipedia.org/wiki/Amazon.com>
> - HBase <http://en.wikipedia.org/wiki/HBase>, current data store of
> Facebook's Messaging Platform
> - Riak <http://en.wikipedia.org/wiki/Riak>
> - VelocityDB (.NET) <http://www.velocitydb.com>
> - Voldemort<http://en.wikipedia.org/wiki/Voldemort_%28distributed_data_store%29>

BigTable is proprietary, the only access is through AppEngine - however
HBase is based on BigTable.

Dynamo is also proprietary and used internally by amazon, although
Cassandra, Riak and Voldemort are all implementations of it. DynamoDB is
also an implementation of it, but only accessible via the managed web
services of AWS.

HBase is part of the Hadoop project, and Cassandra has Hadoop
integration (and support for Hive).

Cassandra I would say is the most advanced, followed by Riak, then
Voldemort.

It's also worth considering that Dynamo, Voldemort, Cassandra, and Riak
are both Highly Available and Partition Tolerant. Whilst HBase is
consistent and partition tolerant (but not HA).

HBase and Cassandra are Column Oriented/Tabular. (like vertica and
hypertable, and of course bigtable on which hbase is based).

Dynamo and Voldemort are KV stores. (like tokyo cabinet, redis,
berkeley, and memcached)

Riak is documented oriented (like couchdb and mongodb)

So really, what you pick depends on what works for you and what suits
the challenges of the particular domain, there is no "one size fits all".

Previously I've found it to be advantageous to use multiple solutions in
projects, and also to layer them on top of each other, for example
positioning Redis (KV) as a fast cache and queue system in front of
something beefier like Couch or Cassandra.

> Peer network node data stores

> Distributed hash tables (per wikipedia)<http://www.google.com/url?q=http%3A%2F%2Fen.wikipedia.org%2Fwiki%2FDistributed_hash_tables%23Examples&sa=D&sntz=1&usg=AFQjCNEtEj9s1rEIVeCGCivvJM60Oa6SCQ>
> DHT protocols and implementations
> Applications employing DHTs

Sadly I've not had enough time yet to dive really heavily in to
applications using DHTs and P2P, other than the usual general techy
level of knowledge, thus I feel it would be more pertinent if somebody
more qualified were to comment on these.

I will say that it's nice to consider a P2P web where DNS is replaced by
open DHTs, and where each node on the network is both client and server
(peer) serving and responding to HTTP requests. If you think about it,
HTTP is a nigh on perfect async communication method for peers to
communicate with each other, POSTing back replies when they are ready.
Tip: the last section of the REST dissertation has a complimentary note
on this, where Roy mentions that adding a simple message identifier
header would allow async communication with messages being returned in
whatever order was fastest, rather than whatever order they were
requested in.

> Thanks for the confirmation. Unfortunately, that big picture is about as
> far as my background gets me on my own. I appreciate the added layers of
> detail you provided.

NP, the big picture is what's important here, so many don't take the
time to consider it or have an overview - so glad to see somebody else
who takes the time to consider it!

>> If this group and the people in the RWW group (of which Melvin is chair)
>> were to put our heads together and cover items 1-8 over the next year or
>> so, I'm positive that we'd make a markable impact on the shape of the
>> web/internet, and be a huge part of the way to enabling a more P2P world.

> I love the optimism and the confirmation that PeerPoint is making some
> sense.
> Maybe I should separate 1-8 and 9-17 into separate tiers.

Now you're talking.. although it's a slight paradox.

You see 9-17 need to be built on top of 1-8, so require 1-8 to be
created first.

However, the feature requirements and best approaches for 1-8 can only
be gained by the process of creating and reviewing 9-17, thus 9-17 need
to be created first.

This is why I'd suggest strongly, that PeerPoint takes the approach of
looking at what has already been created in the FOSS world for 9-17, and
use the details, real working projects to get the requirements for 1-8.

It's really just the standardization process. Find the common
requirements, create something universal/generic on which existing 9-17
software can work, then back port it all on to the new generalized specs
created for 1-8. Get even half way there and you've already created most
of the PeerPoint dream, and have many already existing, aged, and
proven projects all interoperable and working in a P2P fashion :)

> I wonder if you and Melvin could make a copy of the PeerPoint doc and pare
> it down to the parts that would most apply and appeal to the RWW group,
> adding whatever makes sense to round out that particular "view" of the
> project? I'll help in any way I can, but my progress is pretty slow due to
> the amount of research I have to do.

Sounds like a good idea, I'll discuss with Melvin out of band and I'm
sure we'll get to it in the next day or two.

> Again, thanks for the input and encouragement!

NP, and keep up the good work!

Best,

Nathan

[Poor Richard]

On Saturday, July 14, 2012 7:01:10 PM UTC-5, nathan wrote:

Poor Richard wrote:

>    - OpenLink Data Spaces <http://ods.openlinksw.com/wiki/ODS/>

ODS + Virtuoso are by far the most advanced products on the web today,
they're light years ahead of everybody else, indeed using Virtuoso_ODS
only you could build most of PeerPoint ;)

Holy bejusus! I had not looked at the ODS/Virtuoso site before. I just caught the link in my dragnet recently. I have not seen anything else comparable to their framework/platform. We not only could probably build most of PeerPoint with it, but a shitload of it is already there.

I'm also going to keep the following on the short list for now (summarizing):
 

HBase is based on BigTable and is ... part of the Hadoop project

 

Apache Cassandra I would say is the most advanced...and has Hadoop integration (and support for Hive).

 

It's also worth considering that ... Cassandra ...is Highly Available and Partition Tolerant.

 

Whilst HBase is consistent and partition tolerant (but not HA).

HBase and Cassandra are Column Oriented/Tabular. (like vertica and
hypertable, and of course bigtable on which hbase is based).

 

So that gives us ODS, Cassandra, Hbase, and CouchDB to work with, and chances are that ODS can do a lot by itself.

 

Previously I've found it to be advantageous to use multiple solutions in
projects, and also to layer them on top of each other, for example
positioning Redis (KV) as a fast cache and queue system in front of
something beefier like Couch or Cassandra.

Redis is an open-source, networked, in-memory, key-value data store with optional durability. It is written in ANSI C. OK, Reddis or another in-memory KV or DHT store sounds good as a fast front-end and lightweight data store with beefier stuff behind it. <http://redis.io/>
Wikipedia's Redis page lists these (in-memory also going by the names) as "see also":

• Berkeley DB
• Cassandra
• Coherence
• Hazelcast
• Membase
• Memcached
• MemcacheDB
• Mnesia
• MySQL Cluster

Sadly I've not had enough time yet to dive really heavily in to
applications using DHTs and P2P, other than the usual general techy
level of knowledge, thus I feel it would be more pertinent if somebody
more qualified were to comment on these.

 

Aren't DHTs similar to KV data stores? (Cassandra is also called a KV store). I just found this list/comparison: http://en.wikipedia.org/wiki/Key-value_data_store#Key-value_store

I will say that it's nice to consider a P2P web where DNS is replaced by
open DHTs, and where each node on the network is both client and server
(peer) serving and responding to HTTP requests. If you think about it,
HTTP is a nigh on perfect async communication method for peers to
communicate with each other, POSTing back replies when they are ready.
Tip: the last section of the REST dissertation has a complimentary note
on this, where Roy mentions that adding a simple message identifier
header would allow async communication with messages being returned in
whatever order was fastest, rather than whatever order they were
requested in.

That sounds awesome for a p2p environment.

 

> Maybe I should separate 1-8 and 9-17 into separate tiers.  

Now you're talking.. although it's a slight paradox.

You see 9-17 need to be built on top of 1-8, so require 1-8 to be
created first.

However, the feature requirements and best approaches for 1-8 can only
be gained by the process of creating and reviewing 9-17, thus 9-17 need
to be created first.

This is why I'd suggest strongly, that PeerPoint takes the approach of
looking at what has already been created in the FOSS world for 9-17, and
use the details, real working projects to get the requirements for 1-8.

It's really just the standardization process. Find the common
requirements, create something universal/generic on which existing 9-17
software can work, then back port it all on to the new generalized specs
created for 1-8. Get even half way there and you've already created most
  of the PeerPoint dream, and have many already existing, aged, and
proven projects all interoperable and working in a P2P fashion :)

 

I completely agree. Assuming ODS + Virtuoso in the picture early on, it shouldn't be too much of a stretch. They made their bones with middleware, making existing stuff inter-operate.

Before I looked up the ODS site, I was mulling over your comments and thinking "OK, we sandwich existing apps between a middleware/data-store back-plane and a new front-end GUI framework." I thought about Integrated Development Environments (IDE) and wondered if we could call the front-end an integrated user environment (IUE) or maybe a Graphical Layer User Environment (GLUE).  My impression is that the ODS-Virtuoso user environment is still pretty sparse at this point --mostly admin utilities. But I'm not very sure about that side of it yet.

One more thought I wanted to run by you--part of my vision of PeerPoint is that it comes with an integrated development environment as standard equipment. The GLUE front-end might belong to the apps and the development tools, too. The "object inspector" metaphor could apply to both. The user version would drill down to rights, tags, trust metrics, and other user info and the developer version would drill down to object properties, protocols, docs, and code.

> I wonder if you and Melvin could make a copy of the PeerPoint doc and pare
> it down to the parts that would most apply and appeal to the RWW group,
> adding whatever makes sense to round out that particular "view" of the
> project? I'll help in any way I can, but my progress is pretty slow due to
> the amount of research I have to do.

Sounds like a good idea, I'll discuss with Melvin out of band and I'm
sure we'll get to it in the next day or two.

 

I think we are actually moving the ball down the field. With ODS-Virtuoso we may even be close to a first down. That feels pretty good.

Richard
 

[Melvin Carvalho]

On 15 July 2012 04:58, Poor Richard <poor.r...@gmail.com> wrote:

On Saturday, July 14, 2012 7:01:10 PM UTC-5, nathan wrote:

Poor Richard wrote:

>    - OpenLink Data Spaces <http://ods.openlinksw.com/wiki/ODS/>

ODS + Virtuoso are by far the most advanced products on the web today,
they're light years ahead of everybody else, indeed using Virtuoso_ODS
only you could build most of PeerPoint ;)

Holy bejusus! I had not looked at the ODS/Virtuoso site before. I just caught the link in my dragnet recently. I have not seen anything else comparable to their framework/platform. We not only could probably build most of PeerPoint with it, but a shitload of it is already there.

I'm also going to keep the following on the short list for now (summarizing):

+1 to openlink ... it certainly is a long way ahead of anything else out there in terms of functionality, it's also part of KDE/nepomuk ... I'll cc Kingsley here, who is the CEO

I should mention data.fm which is developed at Tim Berners-Lee's lab at MIT, I run this locally as my "personal data store" and it can handle 1 million hits a month no problem.
 

One system I really like technically is RetroShare.  It's open source, has first class developers, who really know their stuff. and an active, working, community.  One team has already ported libretroshare into a browser.  Imagine reatlime, secure, encrypted, chat straight in your browser, plus a ton of other features.  There's even  little chess game you can plug in to the framework so you can challenge your friends.  Once you see this working it's a real paradigm shift, that makes you think 'why doesnt every browser do this?'.  CC: drbob (lead dev)

It's currently applying for some microfinance so that more of the developers can try to go full time.  If anyone on this list has some ideas on that side of things, would love to hear! :)
 

There's a ton of projects out there.  The ones that are great technically are not always the ones that are best at promotion and vice versa.  Definitely the right person to talk to is Kinsley he's been doing this longer than 20 years.  On P2P the hardest challenge is busing through the myriad firewall and proxy systems, which the retroshare people have done some great work on.
 

Richard
 

[Nathan Rixham]

Melvin Carvalho wrote:
> On 15 July 2012 04:58, Poor Richard <poor.r...@gmail.com> wrote:
>> On Saturday, July 14, 2012 7:01:10 PM UTC-5, nathan wrote:
>>> Poor Richard wrote:

>>>> - OpenLink Data Spaces <http://ods.openlinksw.com/**wiki/ODS/<http://ods.openlinksw.com/wiki/ODS/>>

>>>
>>> ODS + Virtuoso are by far the most advanced products on the web today,
>>> they're light years ahead of everybody else, indeed using Virtuoso_ODS
>>> only you could build most of PeerPoint ;)
>>>
>>
>> Holy bejusus! I had not looked at the ODS/Virtuoso site before. I just
>> caught the link in my dragnet recently. I have not seen anything else
>> comparable to their framework/platform. We not only could probably build
>> most of PeerPoint with it, but a shitload of it is already there.
>>
>> I'm also going to keep the following on the short list for now
>> (summarizing):
>>
>
> +1 to openlink ... it certainly is a long way ahead of anything else out
> there in terms of functionality, it's also part of KDE/nepomuk ... I'll cc
> Kingsley here, who is the CEO

Yes, Kingsley is possibly the only person on the planet who truly
understands data (and data integration) inside out, for a couple of
decades now him and his team have really led the field.

Virtuoso is the core, I'd encourage you to take a look at:
http://virtuoso.openlinksw.com/virtuoso-technical-architecture/
Even with a tertiary glance at the page it's clear to see you're dealing
with a different class of middleware.

ODS is layered on top of virtuoso, see:
http://ods.openlinksw.com/wiki/ODS/
Each module is not only already packaged with existing UI's, but due to
it's heritage, each module is also available via SOAP and REST, meaning
you can build your own applications and UIs over the top of it - as
browser apps, on client, server or on peers. IMHO ODS-Briefcase is one
of the most wonderful modules available for it, it's basically a really
nice RESTful WEBDAV enabled data store package, with full support for
multiple auth* protocols right up to WebID, and which recognises
different data types. For instance it allows RDF that's been PUT/POSTed
to be sponged straight in to the very powerful SPARQL-enabled triple
running behind the scenes. E.G. it understands your data and serves as
both a CRUD store, and a more advanced store which you can query
extremely fast, using v powerful query languages like SPARQL.

> I should mention data.fm which is developed at Tim Berners-Lee's lab at
> MIT, I run this locally as my "personal data store" and it can handle 1
> million hits a month no problem.

Yes definitely, data.fm is "the other project" which is truly way ahead
of the field at the minute, it's a RESTful, multi-auth* enabled store
which supports querying, CRUD, automatic media type transformation, data
browsers and even tabulator panes to view data. It's also open source
and you can run your own instances very easily. Highly highly recommended.

Tabulator is also worth mentioning here, it's one of TimBL long running
code based projects and is simply wonderful too - very well designed,
and extensible in every way - Tim of course also understands data inside
out, and the webizing of systems.

The three projects above are very much complimentary, all interlinked,
Kingsley (openlink) knows TimBL (tabulator/wem/semweb) knows Joe
(creator of data.fm, from Tim's team at MIT). It may be fair to say that
each of the projects wouldn't be quite what they are today without the
presence of the others.

IMHO, the most valuable thing anybody in this group can do is to take
the time to fully understand:
1) Virtuoso+ODS and Kingsley's blog posts
2) Tabulator + TimBLs Design Issues
3) Data.fm and it's correlations to 1+2

Those three can be seen as the reference implementations of the next
generation of the web, one which can easily be P2P too, and which
continues to be built, standardized and innovated around.

Really, there are people doing this every day (most are members of the
RWW group), they are fortunate, are years ahead of everybody else on the
web IMHO, have unparalleled understanding, are warm, welcoming,
extremely knowledgeable and full of golden insight, and I'm sure every
person hear could have a really constructive time by getting involved in
any capacity. Not least because they are a group of people who innovate
and create, whilst taking the time to communicate and standardize
between them.

Richard, the RWW group really is the definition of what you want to
achieve with PeerPoint, its 50+ highly skilled, devoted, people who
rapidly innovate and standardize around FOSS on the web. IMHO they're
also the only people currently capable of realizing the PeerPoint dream
of your 1-8, and who are already working on and have implementations
many things from your 9-17.

>> So that gives us ODS, Cassandra, Hbase, and CouchDB to work with, and
>> chances are that ODS can do a lot by itself.

Virtuoso covers 1-8, ODS is a healthy start at 9-17.
But then data.fm is also a healthy start at 1-8, and projects like
MyProfile are from 9-17.

The bigger point is, that if you have standardized interfaces, then
there's no need to consider whether you'll use Cassandra or HBase or
Couch or ODS or data.fm or (insert anything here, and multiples thereof)
- that's just implementation details and different people will want
different things for different purposes.

How can you build a full P2P web on one selected datastore? you can't,
that's not free and not in the P2P ethos as you know and have written -
hence why the interfaces need standardized, and the choice as to which
software to use for what purpose should be left to each individual user
/ developer / creator / org.

>> Aren't DHTs similar to KV data stores? (Cassandra is also called a KV
>> store). I just found this list/comparison:
>> http://en.wikipedia.org/wiki/Key-value_data_store#Key-value_store

Yes, HT = Hash Table, a Hash Table is pretty much just a KV store, so a
distributed hash table is pretty much a distributed kv store.

If you think of the web as a big three-tier system:
http://en.wikipedia.org/wiki/Multitier_architecture#Three-tier_architecture
and then think of HTTP (and related protocols) as the interface between
each layer, and think of media types as the universally understood input
and output types. (in the same way that turning made computers universal
by standardizing the input and output to 1s and 0s).

Then the data tier is made up of web accessible data, the web of data,
powered by anything, including data.fm and virtuoso.

The logic tier can either be web accessible services (CREST style
computations and continuations, or APIs, or web services), or "client"
side libraries (generic, domain specific, application specific).

Then the presentation tier can be handled by js application, using cross
origin requests to web mounted data stores and web accessible services,
and presenting in HTML+CSS, with functionality via JS (the GLUE).

It already all exists and is ubiquitous across devices, servers, clients
and peers.

The Data tier and web service tier become "one", as when the interface
is standardized (HTTP) it doesn't matter whether you are consulting a
store or a service - and services are just data transformation apps, and
some stores (ODS + data.fm) have data transformation in them - so the
lines are blurred and barriers broken down. I know people usually think
of the web as MVC, but I quite like this 3-tier view of it to grok
what's going on initially.

Tabulator, mentioned earlier, can be seen as the perfect GLUE model, as
it handles "uniform data" (rdf in various forms) as input, then provides
a framework on which you can hook and listen for certain types of data,
and provide widgets / composable modules to offer UIs for the data, and
context dependent ones to boot. So in one case it may show you a view of
a full view of a person or product, in another a simplified view with a
list of actions that can be taken (like web intents but even more webby
via linked data) and.. I may be loosing you here, but trust me it's all
there for anybody who cares to look.

>> One more thought I wanted to run by you--part of my vision of PeerPoint is
>> that it comes with an integrated development environment as standard
>> equipment. The GLUE front-end might belong to the apps and the development
>> tools, too. The "object inspector" metaphor could apply to both. The user
>> version would drill down to rights, tags, trust metrics, and other user
>> info and the developer version would drill down to object properties,
>> protocols, docs, and code.

Yes, exactly - and Tabulator == object inspector.

>>> I wonder if you and Melvin could make a copy of the PeerPoint doc and
>>> pare
>>>> it down to the parts that would most apply and appeal to the RWW group,
>>>> adding whatever makes sense to round out that particular "view" of the
>>>> project? I'll help in any way I can, but my progress is pretty slow due
>>> to
>>>> the amount of research I have to do.
>>> Sounds like a good idea, I'll discuss with Melvin out of band and I'm
>>> sure we'll get to it in the next day or two.
>>>
>>
>> I think we are actually moving the ball down the field. With ODS-Virtuoso
>> we may even be close to a first down. That feels pretty good.
>>
>
> There's a ton of projects out there. The ones that are great technically
> are not always the ones that are best at promotion and vice versa.
> Definitely the right person to talk to is Kinsley he's been doing this
> longer than 20 years. On P2P the hardest challenge is busing through the
> myriad firewall and proxy systems, which the retroshare people have done
> some great work on.

Yes definitely - although I would say that Melvin himself is very very
good to talk to, and has connected the dots between the ton of projects
out there, I wouldn't be aware of half of what I am, if it wasn't for
him, including this group ;)

ps: retroshare is awesome, but I don't know enough to comment greatly on
it, this is the only reason I haven't went in to more detail about it -
still v recommended and worth taking the time.

Best,

Nathan

[Poor Richard]

Thanks, Melvin. I captured your recommendations and links to the PeerPoint doc.

PR

On Sunday, July 15, 2012 3:18:01 AM UTC-5, melvincarvalho wrote:

[Poor Richard]

Sebastian, Kingsley, Melvin, Nathan, et al

Synopsis: PeerPoint is an embyonic requirements definition. What is different about it, as far as I can tell, is that it aims to encompass all of web 3.0 in scope, starting at the topmost level of user requirements, predicated on the urgent imperatives for greater social justice, sustainability, and an open society. In short, PeerPoint aims to describe the full compilation of applications we desperately need for a new economy and a new culture. The big corporations like Google and Facebook are working towards greater enclosure (more walled gardens), more user surveillance, more user exploitation, etc. so the internet actually becomes more centralized and less free by the month. An internet platform for implementing a free, just, and sustainable society must come from the indy FOSS community, and it must be designed for free (or very low cost) turnkey provisioning to masses of generic, non-technical internet users.

But such "master plans" as PeerPoint are generally considered naive in FOSS land because non-commercial development is self-motivated and anarchistic. Thus few have taken on the job of planning beyond their own technical spheres of interest. And nobody (as far as I can tell) has been assigned or taken on themselves the job of defining or aggregating all the user requirements over all the major application domains under one umbrella. Not finding a coherent universal mapping between the user population's requirements and current technical capabilities, I appointed myself and anyone I can recruit to do this.

PeerPoint isn't intended to replace existing requirements definitions or specifications. It is meant to connect dots and fill in gaps in the hope of more rapid convergence and more comprehensive solutions. The resource in shortest supply is time.

If you open the PeerPoint doc, please let me know at what page you stop reading.

Thanks for your interest and encouragement,

Richard

PeerPoint: https://docs.google.com/document/d/1TkAUpUxdfKGr_5Qio2SlZcnBu_sgnZWdoVTZuD_Regs/edit?pli=1#

On Mon, Jul 16, 2012 at 9:05 AM, Sebastian Trueg <tr...@openlinksw.com> wrote:

Kingsley, thank you for including me.

I am in charge of ODS at OpenLink Software. Could you please recap for me the requirement for PeerPoint, so I can get an idea of how we could come together.

Thanks,
Sebastian Trueg

On 07/15/2012 09:28 PM, Kingsley Idehen wrote:

On 7/15/12 4:18 AM, Melvin Carvalho wrote:

On 15 July 2012 04:58, Poor Richard <poor.r...@gmail.com

<mailto:poor.r...@gmail.com>> wrote:

    On Saturday, July 14, 2012 7:01:10 PM UTC-5, nathan wrote:

        Poor Richard wrote:

        > - OpenLink Data Spaces <http://ods.openlinksw.com/wiki/ODS/>

        ODS + Virtuoso are by far the most advanced products on the
        web today,
        they're light years ahead of everybody else, indeed using
        Virtuoso_ODS
        only you could build most of PeerPoint ;)



    Holy bejusus! I had not looked at the ODS/Virtuoso site before. I
    just caught the link in my dragnet recently. I have not seen
    anything else comparable to their framework/platform. We not only
    could probably build most of PeerPoint with it, but a shitload of
    it is already there.

    I'm also going to keep the following on the short list for now
    (summarizing):


+1 to openlink ... it certainly is a long way ahead of anything else
out there in terms of functionality, it's also part of KDE/nepomuk ...
I'll cc Kingsley here, who is the CEO

I've added Sebastian to the cc list as we are making the RESTful
interfaces to ODS cleaner. Our ultimate goal is to have endevors such as
PeerPoint leapfrog into Linked Data and WebID adoption and exploitation
via our powerful framework for personal and distributed data spaces.

Sebastian: please pick this up.

--

Regards,

Kingsley Idehen

Founder&  CEO

OpenLink Software
Company Web:http://www.openlinksw.com
Personal Weblog:http://www.openlinksw.com/blog/~kidehen
Twitter/Identi.ca handle: @kidehen
Google+ Profile:https://plus.google.com/112399767740508618350/about
LinkedIn Profile:http://www.linkedin.com/in/kidehen

--
Sebastian Trueg
Technical Consultant
OpenLink Software
tr...@openlinkws.com
http://openlinksw.com
http://trueg.wordpress.com
http://www.linkedin.com/in/trueg

[Poor Richard]

Nathan,

I copied your brilliant comments and links to various places in the PeerPoint doc.

On Sunday, July 15, 2012 7:21:21 AM UTC-5, nathan wrote:
<snip>

Richard, the RWW group really is the definition of what you want to
achieve with PeerPoint, its 50+ highly skilled, devoted, people who
rapidly innovate and standardize around FOSS on the web. IMHO they're
also the only people currently capable of realizing the PeerPoint dream
of your 1-8, and who are already working on and have implementations
many things from your 9-17.

I joined the RWW group. I looked at the task list and don't think I have any of the necessary expertise. My functional skillset at this point is mainly writing, informed by past IT experience (corporate WAN/LAN and web 1.0 environments) and past technical writing experience.

I have taken the technical detail in the PeerPoint doc to about as deep a level as I can, although I still have a lot of horizontal ground to cover.

I want to include enough technical discussion at this point for the project to be taken seriously by software engineers and designers, but my largest contribution should ultimately be at the top end, coherently describing the end-to-end scope and functionality required by various kinds internet users, most of whom won't want to know what's under the hood of an all-in-one internet application compilation or appliance. An important target class is social entrepreneurs, activists, and citizens who need better collaboration tools but who won't want to research, pick, integrate, and extensively configure a bunch of separate technologies.

I don't know the RWW etiquette and site layout -- is it appropriate for me to introduce myself and the PeerPoint project somewhere?

The bigger point is, that if you have standardized interfaces, then
there's no need to consider whether you'll use Cassandra or HBase or
Couch or ODS or data.fm or (insert anything here, and multiples thereof)
- that's just implementation details and different people will want
different things for different purposes. 

How can you build a full P2P web on one selected datastore? you can't,
that's not free and not in the P2P ethos as you know and have written -
hence why the interfaces need standardized, and the choice as to which
software to use for what purpose should be left to each individual user
/ developer / creator / org.

I completely agree, almost. I want the PeeerPoint spec to include a default turnkey application set (sort of like KDE) for technically challenged users, while at the same time allowing for infinite mix-and-mash variations and customization. That has been a key to Windows, Apple, and Linux adoption, and I want PeerPoint to be more so. Hence my interest in comparing strengths and weaknesses of potential components in each category, such as front-end (fast & lite) datastore and back-end (heavy-duty) datastore, for example.

If you think of the web as a big three-tier system:

( . . . )

lines are blurred and barriers broken down. I know people usually think
of the web as MVC, but I quite like this 3-tier view of it to grok
what's going on initially.

 

3-T (or n-T) is easier for me grasp in a multi-app environment than MVC, but I'll mention both.

ciao for niao,

Richard
----------------------

On Sunday, July 15, 2012 7:21:21 AM UTC-5, nathan wrote:

Melvin Carvalho wrote:

On Sunday, July 15, 2012 7:21:21 AM UTC-5, nathan wrote:

Melvin Carvalho wrote:

[Melvin Carvalho]

On 16 July 2012 21:48, Poor Richard <poor.r...@gmail.com> wrote:

Nathan,

I copied your brilliant comments and links to various places in the PeerPoint doc.

On Sunday, July 15, 2012 7:21:21 AM UTC-5, nathan wrote:
<snip>

Richard, the RWW group really is the definition of what you want to
achieve with PeerPoint, its 50+ highly skilled, devoted, people who
rapidly innovate and standardize around FOSS on the web. IMHO they're
also the only people currently capable of realizing the PeerPoint dream
of your 1-8, and who are already working on and have implementations
many things from your 9-17.

I joined the RWW group. I looked at the task list and don't think I have any of the necessary expertise. My functional skillset at this point is mainly writing, informed by past IT experience (corporate WAN/LAN and web 1.0 environments) and past technical writing experience.

I have taken the technical detail in the PeerPoint doc to about as deep a level as I can, although I still have a lot of horizontal ground to cover.

I want to include enough technical discussion at this point for the project to be taken seriously by software engineers and designers, but my largest contribution should ultimately be at the top end, coherently describing the end-to-end scope and functionality required by various kinds internet users, most of whom won't want to know what's under the hood of an all-in-one internet application compilation or appliance. An important target class is social entrepreneurs, activists, and citizens who need better collaboration tools but who won't want to research, pick, integrate, and extensively configure a bunch of separate technologies.

I don't know the RWW etiquette and site layout -- is it appropriate for me to introduce myself and the PeerPoint project somewhere?

Absolutely! :)

RWW is a W3C Community Group.  Community Groups are the most informal of the W3C groups, and the idea is bringing like minded people together, to create standards and solve problems (preferably in a way that can scale to the whole Web).  It's a very open and flat structure, so there are no fees, anyone is free to join, and it's as much your group as anyone else's.

If you got a chance to introduce yourself and PeerPoint on the mailing list, that would we awesome. 

We have a lot of techies, hackers and standards people.  People with ability to write, and present ideas, are absolutely invaluable, as this space matures.
 

[Nathan Rixham]

Couldn't agree more Melvin.

Richard your skillset is as valuable addition to the RWW CG as the
technical skills of the members are to PeerPoint :)

[Poor Richard]

I hesitate to "relply all" on these emails but I wanted everyone to know I appreciate the friendly reception and if any writing tasks come up that I mighht be able to contribute to just let me know.

I'll pop a brief description of PeerPoint up on the RWW mail list.

I look forward to  participating in the community.

Richard

[Nathan Rixham]

Poor Richard wrote:
> I hesitate to "relply all" on these emails but I wanted everyone to know I
> appreciate the friendly reception and if any writing tasks come up that I
> mighht be able to contribute to just let me know.

wouldn't worry, I can't remember the last time I *didn't* click reply all!

[Om G]

Richard, Nathan.. and the rest..

let's do it :-)

it would be a good start to quell the naysayers.

Working on standards of communication (RDF, JSON, FOAF, APML) is a great place to begin because it doesn't require a 'single' tool.

I think that's one of the main failings of most 'Let's build a cool new thing' - things. We can't mandate that everyone use the same tool, but we certainly can suggest a protocol.

As for 'us' and the new DNS, or UnderNet-like behavior we shouldn't hesitate. This should be discussed seriously as quickly as possible. This capability should be the first thing that put onto any freedom box/beaglebone etc. and I can't wait to get an Alpha running now. This seems like the perfect use for Tor [https://www.torproject.org/] which is damnably slow for regular browsing but could serve up pseudo-DNS all day. In conjunction with some fancy bouncing we could hybrid a pretty functional, quite anonymous system of systems. {must slow down.. brain moving faster than fingers}

Naming one's own resources should start with a teeny bit of forethought; but seriously.. if we build it...

I'd also advocate some level of coordination with the Occupy diaspora as a group of like minds with deep reserves of talent and a growing capacity/need for PeerPoint.

Andiamo!

Om

[Mark Roest]

Hello Om,

While you're at it, how about the empire's diaspora too?!?

I have been plugging for a knowledgebase which combines digital earth imaging (a la Google Earth /Keyhole) with geographic information systems and expert systems for a long time now. I'm like Poor Richard -- a writer, not a programmer, and I design methods for designing sustainable economies which can be shaped to the eco-region, ecosystems, and cultures which co-evolved in each place on our planet. 

It seems that Open Geospatial Consortium is the strongest open-source GIS toolkit, so when your Fearless Fifty-plus set this up, could you please integrate it / provide a robust API? There are profound implications if you do. And if someone asks, I will be glad to re-share some of what I have been writing about it, to the appropriate place.

Regards,

Mark Roest

[Poor Richard]

om,

feel free to post links or upload/email copies of PeerPoint to any and all occupy sites.

PR

[Poor Richard]

Thanks for the reference, Mark. I put it in.

PR

[Fabio Barone]

Hi

I am following this list since its inception, and have been fascinated
by the high quality content ever since.

I had to drop out in reading each post for a while now, I work over the Internet
and I recently moved to a totally new country, so I need from time to time
to leave the computer in order to actually engage with the locals here and
have a decent life :)

Just wanted to drop a line saying how marvellous the work you're doing
looks to me. Wish I'd be able to contribute somehow, but you guys know your stuff,
I am a developer and know what it takes to really master something,
and time and financial issues are pressing here.

Hope I'll be able to catch up with latest developments, especially on PeerPoint,
but just wanted to leave a note of awe, support and encouragement.

2012/7/16 Nathan Rixham <nri...@gmail.com>

[Sepp Hasslberger]

good work, PR 

I am interestedly lurking only at this time as I have little leeway for this, but am very happy to see this proceeding as well as it does. It is a very needed action - to introduce some planning in FoSS land, especially if we want to move towards a p2p world, not just isolated instances of peer collaboration. 

Great stuff. Sorry for not chiming in for now. 

Sepp

[Poor Richard]

On Tuesday, July 17, 2012 11:02:24 AM UTC-5, Sepp wrote:

good work, PR 

Thanks, Sepp!
 

I am interestedly lurking only at this time as I have little leeway for this, but am very happy to see this proceeding as well as it does. It is a very needed action - to introduce some planning in FoSS land, especially if we want to move towards a p2p world, not just isolated instances of peer collaboration. 

Great stuff. Sorry for not chiming in for now. 

Keep lurking but please do chime in when you get a chance -- I'd love to have your input!

PR

[Poor Richard]

tawhuac (Fabio),

Thanks for the note of encouragement -- I sure appreciate it!

Don't hesitate to comment on, question, or criticize PeerPoint whenever you get settled and have the time.

Best wishes in your new location,

PR