Social Networking (And Generic Ideas found within)

[Lee Olayvar]

In response to an email from Chris Babcock about building a social network plugin for Tortuga, i would like to open it up for public discussion a bit.

I am a big fan of generic plugins which can be bound together to make lots of functionality, and although these are complex, i will most likely be putting most of my effort into these. Keep in mind though i have never developed for a project this big, so my contribution will not carry anything.. i just wanted to discuss this. Now ofcourse we can't do much direct planning until we are able to see a bit more of Tortuga flesh out, but i'd like to atleast talk about how to generalize all of the concepts that make up a social network.

A good question is though, what makes up a social networking site? What key features can be generalized and implemented by themselves as plugins?

To start, custom content form generation and custom queries would be a good places to start. That would allow for everything from blogs to forums to gallerys, etc. custom queries might also be able to handle content relations, allowing you to have a user view 10 other users with similar attributes as him (location, hobbies, religion, whatever).


What else?
--
Lee Olayvar

[Chris B - JK at asciiking dot com]

> A good question is though, what makes up a social networking site? What key
> features can be generalized and implemented by themselves as plugins?

The three essential elements of SNS are membership, relationship and
interaction.

Membership minimally entails a single permission set for the entire
site, i.e. UserX has PermissionY for all content of TypeZ throughout
the site. An expected feature of any SNS is that UserX would be able
to create and own content on the site in a way that enables him to
control access to that content.

Minimally, relationships consist of either a friendship meme or
follower meme. Friendship involves involves negotiation of a
reciprocal relationship, using resources to enforce a relationship
model that is less robust. As a consequence of this, the follower meme
is becoming more prevalent. The only problem with this is that sites
generally want to send only positive notifications, so if a user
wishes to enforce reciprocity then he is denied the tools to do so
effectively. Best implementation would be a site that gives
administrators the options of friends, followers or user-selected
friendship behavior with an administration set default.

Another desirable for SNS relationships in most applications is
groups. In its purest implementation, groups would be virtual members
with group membership mechanisms simply renamed. Robust groups permit
multiple owners and fine-grained permissions with sane defaults. I
would like to enable groups to own and join groups in order to enable
the formation of organic group structures. (Note to Ian: Work groups
as well as work flow inhibit user participation in CMS applications.
Currently available software doesn't implement the rich variety of
committees, sub-committees, joint committees, etc. and the fluid
movement in and out of these groups in real time in a realistic way.)
I think that Tortuga's i18n model contains some seed ideas for
implementation. A user registration is a generic object. His primary
profile, aliases and groups inherit the generic information, with
multiple inheritance for groups being a desirable.

Interaction happens over content, which is most often text. Most text
content consists of a title and a body. The primary difference between
text content types is in presentation. Presentation constraints may
consist of length or structural elements that are socially imposed,
imposed by the view, imposed by the data model or follow naturally
from the medium - For example mailing lists don't have an inherent per
message size limit, but for most topics there is a 4 paragraph cap on
constructive communications. Google groups is a good example of how
presentations are merging to reveal a fundamental unity in the
underlying content structure for Newsgroups, mailing list archives,
and forums. This trend is generally useful for concentrating content,
providing users the ability to interact seamlessly with others whose
choice of content tools might otherwise separate them, but it's still
useful to identify the input tools and permit the user to filter the
views by the source because a chat log may be structurally similar to
a Wiki discussion page, but there's a dramatic difference in the
quality of the content.

There are any number of ways to handle non-text content, the most
popular being embedding it in text and creating specialized content
handlers - for examples MIME and Facebook apps.

> To start, custom content form generation and custom queries would be a good
> places to start. That would allow for everything from blogs to forums to
> gallerys, etc. custom queries might also be able to handle content
> relations, allowing you to have a user view 10 other users with similar
> attributes as him (location, hobbies, religion, whatever).
>
> What else?

Decide whether SNS plugins on Tortuga should communicate via ODD or
OpenSocial. ODD is XML-only and fairly young, while OpenSocial
includes both XML and JSON specs and is backed by Google. I think that
the desirability or not of having a JSON definition is a more
important factor than whether to take the Google Kool-Aid, but I may
be alone at that table.

Once there is a tool (wizard?) for defining member profiles, the next
step is defining permissions. The default authorization structure from
TG2 (where a permission is a verb that applies to all the content of
of a given type on the site) may be sufficient for a CMS, but SNS
requires finer grained permissions. If it's not already done in
Tortuga, the authorization model has to be extended to include realms.
Ian, what is the auth model for Tortuga?

I also happen to think that user administration tools should be an
integral part of the design prior to release. The principle of least
privilege still applies even though we are trying to be expansive with
capabilities. It's OK to allow group owners or even individual users
to have access to form generation and custom queries through a web
interface, but allowing users to upload templates or define structures
(whether views or underlying data) in raw text could lead to injection
attacks. Most standard web-based textual SNS content can be derived
through the API that has already been exposed once the tools for
defining memberships and relationships are built (preferably to the
same API or one substantially similar), in other words the role of the
base SNS plug-in would simplify the creation of content-types web-
member and web-group. (The purpose of the 'web-' prefix being to
distinguish public content sources from sources of meta-content.)

Some extensions for SNS apps will probably fall back to the core
TurboGears functionality, for example the API exposed for document
creation would probably handle creating an RSS feed, but would not
handle mailing list integration.

Chris

[Ian Charnas]

Chris, you asked what was the Auth model for tortuga. Tortuga will do
both authentication and authorization itself, and will also allow for
external authentication (I especially want to allow apache's many auth
modules tell Tortuga 'who' a user is). One day it may also support
external authorization also.

I should be clear that I'm not building a web app framework, I think
Turbogears and Pylons are fine for that. Tortuga will integrate as a
WSGI app, which means it will play VERY nicely with TG2 apps. It may
be possible to do what you're suggesting however, as a Tortuga plugin
should the need arise. I'll do my best to keep things as flexible as
possible, so you can do great things as plugins.
ian

[Lee Olayvar]

On Mon, Nov 17, 2008 at 7:28 PM, Ian Charnas <ian.c...@gmail.com> wrote:

I should be clear that I'm not building a web app framework, I think
Turbogears and Pylons are fine for that.  Tortuga will integrate as a
WSGI app, which means it will play VERY nicely with TG2 apps.  It may
be possible to do what you're suggesting however, as a Tortuga plugin
should the need arise.  I'll do my best to keep things as flexible as
possible, so you can do great things as plugins.
ian

Not knowing TG that well, i'm very interested to see how Tortuga itself will play into things. That is to say, given that TG will provide the majority of the framework, i'm curious to see what exactly tying into Tortuga will be doing/providing. Im guessing database and users at the very least.


--
Lee Olayvar

[Chris B - JK at asciiking dot com]

> Chris, you asked what was the Auth model for tortuga. Tortuga will do
> both authentication and authorization itself, and will also allow for
> external authentication (I especially want to allow apache's many auth
> modules tell Tortuga 'who' a user is). One day it may also support
> external authorization also.

I wasn't so much expecting Tortuga to *provide* auth as to *use* an
authorization model that I might be able to exploit. TG2 is using the
Repoze WSGI plug-ins for auth, but they've only completed the
implementation this month. I was basically wondering whether you'd
rolled your own, planned to use TG2's or used some middleware
independently. More specifically, I was wondering if you'd already
solved my problem. :-)

So Tortuga handles it's own authorization and it will either work with
what's available for authentication or handle it internally. Even
though that still leaves the 'SNS team' with the issue of handling
authorization for member content, it's a good security model.
Credentials are centralized, but permissions to modify site content
will be managed separately from permissions for user content.

> I should be clear that I'm not building a web app framework, I think
> Turbogears and Pylons are fine for that. Tortuga will integrate as a
> WSGI app, which means it will play VERY nicely with TG2 apps. It may
> be possible to do what you're suggesting however, as a Tortuga plugin
> should the need arise. I'll do my best to keep things as flexible as
> possible, so you can do great things as plugins.

I've been thinking of TurboGears and Tortuga as a stack like Zope and
Plone, only young and beautiful. There's plenty of fun stuff to do on
the naked framework, but if the purpose of the site happens to involve
managing content then that's a job for a CMS. I know you're designing
Tortuga as an application and I'm looking at it as infrastructure, but
I'll do my best not to provide undue pressure towards feature-creep. I
don't expect to influence development on Tortuga as much as I want to
determine how much help I'm going to need when you've done what you
want.

The point of this thread is that Social Networking Software is really
a specialized CMS. For site level content management, you want to
enable a 2-3 stage workflow and user-defined content types for a
single domain that coincides with the site. In SNS, you want a 1-2
stage workflow for a number of user domains on the site and generally
you want to limit the available content types to a pre-defined set.
I'm not content to confuse the matter to this degree, however. I want
an architecture that actively encourages the site team to add content
on the core site as well as building widget collections to publish the
user content. I think that the "first gear" missing from most Social
Networking sites is core content.

In terms of exposing the authorization structure that I'm looking for,
what's the core mechanism in Tortuga for keeping the sales force out
of the HR docs?

Chris

[Chris B - JK at asciiking dot com]

I hope that I understand this and express it well enough so that it's
a net savings of time for Ian to correct any mistakes as opposed to
answering himself...

On Nov 17, 9:51 pm, "Lee Olayvar" <leeolay...@gmail.com> wrote:
> Not knowing TG that well, i'm very interested to see how Tortuga itself will
> play into things. That is to say, given that TG will provide the majority of
> the framework, i'm curious to see what exactly tying into Tortuga will be
> doing/providing. Im guessing database and users at the very least.

Here's a link to the TG2 docs - http://turbogears.org/2.0/docs/index.html

A simple TG2 app implements the MVC pattern like this:

Model: Data structure in SQL Alchemy ORM
View: Genshi XML templates
Controller: URLs map to controller classes which pass data from the
model into the template

Tortuga essentially provides a meta-structure for manipulating and
managing documents in that pattern. Instead of manually creating
documents and leaving nice details like i18n to the foresight of the
user, Tortuga prompts the user for the basic model and a template then
builds i18n, workflow and the rest of it automatically. In keeping
with modern design principles, the layout and the content of a
document are stored separately. Because of this, it is necessary to be
able to do revision control in both a relational database (content)
and the file system (layout). That is the role of the Brainfreeze and
Checkpoint components.

A CMS means that you don't have to learn a framework to manage
documents. It also exposes some nice libraries for developers for
those tasks other than managing documents that go back to the APIs
provided by the programming language and the framework. There are all
kinds of hooks in TG. I don't see Tortuga closing any of those off for
developers wanting to run an app alongside Tortuga, even integrating
with a Tortuga-based site to varying degrees, but I do see some value-
added direction being provided by Tortuga for applications that
publish content - For example, a tag plug-in that makes the
infrastructure for content tags available to any content type that is
created on the site. This wouldn't necessarily be part of the core
functionality of Tortuga, which was conceived for generating business
docs, but it would be core for most public web content. A plug-in for
Tortuga is primarily a way of adding functionality to content types
defined by the system, whether it's something simple like tagging or
complex like submitting content to a Spam filter. The plug-in
architecture means that the details of any added functionality are
largely transparent to the person who creates the content-type.

The models for users and groups for a minimal SNS are at least as
complex as those for the most robust CMS although the workflow is less
demanding. If the CMS libraries provide enough goodies then it's worth
the complexity cost of incorporating the API for the CMS into the
design. In this case, I would like the whole workflow for the
development of the core sites. Also, Ian's implementation is sensible
enough to leave it intact for members as well. Your typical blogger
doesn't need the whole draft/review/publish cycle, but that's OK
because it's easy to skip the parts he doesn't need. At the very least
you get revisioning, i18n and a plug-in interface for free with the
workflow, so it's worth including even without considering the future
availability of specific plug-ins or what tweaking may be necessary to
make authorization work for this context.

What we're talking about here is an 'SNS-base' plug-in that would
enable a Tortuga-based site to work with content-types typical of
social networking sites, for example member profiles, that aren't easy
to manage as documents.

Chris

[Lee Olayvar]

On Tue, Nov 18, 2008 at 8:21 AM, Chris B - JK at asciiking dot com <swift2...@gmail.com> wrote:

So Tortuga handles it's own authorization and it will either work with
what's available for authentication or handle it internally. Even
though that still leaves the 'SNS team' with the issue of handling
authorization for member content, it's a good security model.
Credentials are centralized, but permissions to modify site content
will be managed separately from permissions for user content.

As far as the SNS client authentication, what is wrong with it entirely using Tortuga's methods (whatever those end up being)? The way i view it you only have to have the user login and assure that the correct user is logged in, after that you can simply base all allow/disallowed actions off of a set of permissions given to that user group (or individual user). All this would be handled by Tortuga i would assume.

Isn't that the basic means which most plugable software use for permission based client management?

What we're talking about here is an 'SNS-base' plug-in that would
enable a Tortuga-based site to work with content-types typical of
social networking sites, for example member profiles, that aren't easy
to manage as documents.

Well the more we can take from the 'SNS-Base' and put it into generic separate plugins, the better, imo. Whether its just an API plugin, or a full generic plugin (perhaps a user group plugin, which internally can be used for security roles, or it can be exposed as an SNS type user group), etc. That way work on one goal (SNS) can propagate to much more than that of just the SNS. Pretty basic ofcourse, i just really think its important.. specially in the beginning. The more we can all work on code together, to achieve our separate desires, the better. Ie, i have no interest in SNS, but if we made some user group plugin to handle basic grouped user permissions, etc, that would be very useful to me, and i'd contribute. Etc.

But really we wont know how much we can separate these generic concepts into plugins themselves.. a lot of that relies on Tortuga. We may just end up sharing libraries heh.

--
Lee Olayvar

[Ian Charnas]

Chris, I am glad to see such interest at this early stage. I'll look
into the Repoze auth stuff, truth be told I haven't seen it yet.

The permissions system isn't written yet, but it will involve the
standard users/roles/permissions scheme. Plugins will have to enforce
the permissions, because without an object-oriented database like
ZODB, there's no real way to protect the data on a case-by-case basis.

ian

[Ian Charnas]

I've been waiting to use this quote, it's a quote from my all-time
favorite movie, the Big Lebowski.

"Well dude, we just don't know."

I'm talking with Mark Ramm over in the TG group for the past few
months, and we will find a good fit between the two projects. The
details will have to wait until there's more of a Tortuga to speak of!
Right now this is just a preview so you can see the status.

[Chris B - JK at asciiking dot com]

> As far as the SNS client authentication, what is wrong with it entirely
> using Tortuga's methods (whatever those end up being)? The way i view it you
> only have to have the user login and assure that the correct user is logged
> in, after that you can simply base all allow/disallowed actions off of a set
> of permissions given to that user group (or individual user). All this would
> be handled by Tortuga i would assume.

It's possible, but not a safe assumption. The main issue is that,
compared to CMS, SNS wants to give more power to more people in whom
there is less reason to place trust. It's a little tricky to permit
users liberal ability to create groups and directories in a content
area and prevent them from tinkering with one another's content in the
same area. There's a significant overlap between high end CMS and low
end SNS in how robustly they handle permissions, but there's no
guarantee that Tortuga will debut at that level even with Ian at the
helm.

> Isn't that the basic means which most plugable software use for permission
> based client management?

The TG2 authorization model, at least as currently documented, relies
on an SQL authorization model that doesn't map well relationally. The
@authorize.has_permission decorator appears to take one parameter, a
verb - "@authorize.require(authorize.has_permission('manage'))".
Authorization for a CMS should take an object along with a verb in its
predicate - "authorize.has_permission('manage', 'sales')". SNS
requires both direct and indirect objects along with the verb, edit
blog in joe's stuff - "authorize.has_permission('edit', 'blog',
'joe')". I've seen evidence that the second form will be supported in
TG2 and it is possible to have a sane enough SNS with permissions
based on TG2's authorize.is_user method, but that imposes certain
undesirables like forcing a single owner for groups. In order to have
a scalable SNS, you need the 3rd form. If you use
"authorize.has_permission('edit_blog_joe')" then you'll be in trouble
when the site outgrows the local cluster, if not sooner. Next upgrade,
someone *will* normalize the database and leave you kicking in the
breeze.

> What we're talking about here is an 'SNS-base' plug-in that would
>
> > enable a Tortuga-based site to work with content-types typical of
> > social networking sites, for example member profiles, that aren't easy
> > to manage as documents.
>
> Well the more we can take from the 'SNS-Base' and put it into generic
> separate plugins, the better, imo. Whether its just an API plugin, or a full
> generic plugin (perhaps a user group plugin, which internally can be used
> for security roles, or it can be exposed as an SNS type user group), etc.
> That way work on one goal (SNS) can propagate to much more than that of just
> the SNS. Pretty basic ofcourse, i just really think its important..
> specially in the beginning. The more we can all work on code together, to
> achieve our separate desires, the better. Ie, i have no interest in SNS, but
> if we made some user group plugin to handle basic grouped user permissions,
> etc, that would be very useful to me, and i'd contribute. Etc.

I'm sorry, I though you *were* interested in Social Networking
Software. I confused your interest in social networking content types
for interest in the core implementation.

> But really we wont know how much we can separate these generic concepts into
> plugins themselves.. a lot of that relies on Tortuga. We may just end up
> sharing libraries heh.

Most of your standard social content types are pretty straight forward
to derive in the CMS. User profiles are complex enough that a wizard
would be helpful. There's a fair chance that authorization would need
tweaking in order to scale to dynamic group creation. Those two
'enhancements' would also benefit the larger commercial installs. A
tagging plug-in, used for a number of social media types, would be
useful for sites that are primarily public content. When someone gets
around to writing a second e-commerce package for Python, they'll want
to implement it on top of a CMS rather than a bare framework in order
to make it easier to generate content-rich product pages. Plug-in SNS
functionality is a desirable for e-commerce, because it enables user
communities. You want reviews - even bad ones - and other user content
to leverage your customer base for increasing sales and reducing the
overhead on customer service. It also helps on-line sales to provide
incentive for users to create profiles before they checkout. You don't
need SNS plug-ins to implement blogs, forums and photo galleries for a
group with existing social ties. You need them if you want to open up
content generation - and user/group creation - to the general public.

Chris

[Chris B - JK at asciiking dot com]

> The permissions system isn't written yet, but it will involve the
> standard users/roles/permissions scheme. Plugins will have to enforce
> the permissions, because without an object-oriented database like
> ZODB, there's no real way to protect the data on a case-by-case basis.

OK, that's the 3-part predicate that I'm looking for. It's unclear
from docs how that would be implemented in TG2 as the docs there focus
on a simpler implementation.

Enforcing permissions at the controller level is perfectly fine. I was
more concerned about the database scaling and being able to factor out
groups that grow large enough to be implemented on separate sites.

Chris

[Lee Olayvar]

On Tue, Nov 18, 2008 at 9:54 PM, Chris B - JK at asciiking dot com <swift2...@gmail.com> wrote:

It's possible, but not a safe assumption. The main issue is that,
compared to CMS, SNS wants to give more power to more people in whom
there is less reason to place trust. It's a little tricky to permit
users liberal ability to create groups and directories in a content
area and prevent them from tinkering with one another's content in the
same area. There's a significant overlap between high end CMS and low
end SNS in how robustly they handle permissions, but there's no
guarantee that Tortuga will debut at that level even with Ian at the
helm.

Oh i agree, i was just seeing permissions levels as something which would be enforced by the plugins themselves, like Ian mentioned *i believe*. Anything outside of this common permissions design is something i've never seen hehe (such as that mentioned about database level permissions, etc).
 

I'm sorry, I though you *were* interested in Social Networking
Software. I confused your interest in social networking content types
for interest in the core implementation.

Well im interested in it all, but no matter what the implementation, the more one plugin can help another, the more im interested :). But yes, i will not be making an SNS (in the normal sense) :)
 

Most of your standard social content types are pretty straight forward
to derive in the CMS. User profiles are complex enough that a wizard
would be helpful.

 I agree, and especially Group baseded permissions with fine grained control is something i would be very interested in, for the immediate.

--
Lee Olayvar

[Jorge Vargas]

On Tue, Nov 18, 2008 at 10:04 PM, Ian Charnas <ian.c...@gmail.com> wrote:
>
> Chris, I am glad to see such interest at this early stage. I'll look
> into the Repoze auth stuff, truth be told I haven't seen it yet.
>
> The permissions system isn't written yet, but it will involve the
> standard users/roles/permissions scheme. Plugins will have to enforce
> the permissions, because without an object-oriented database like
> ZODB, there's no real way to protect the data on a case-by-case basis.
>

I really encourage you to relay this to TG with repoze.who and
repoze.what, they in themself are meta-frameworks withs tons of
plugins, so the user (read: admin) is able to store auth into in
whatever means he wants, all the apache stuff you suggested is already
in repoze.who, and plugins for repoze.what (formerly tg.authorization)
will be written.

IMO the best thing tortuga can do is provide it's own repoze.what
plugin to handle all things which will give it the most integration
with other tg2 apps.

[Mark Ramm]

>> The permissions system isn't written yet, but it will involve the
>> standard users/roles/permissions scheme. Plugins will have to enforce
>> the permissions, because without an object-oriented database like
>> ZODB, there's no real way to protect the data on a case-by-case basis.
>>
> I really encourage you to relay this to TG with repoze.who and
> repoze.what, they in themself are meta-frameworks withs tons of
> plugins, so the user (read: admin) is able to store auth into in
> whatever means he wants, all the apache stuff you suggested is already
> in repoze.who, and plugins for repoze.what (formerly tg.authorization)
> will be written.
>
> IMO the best thing tortuga can do is provide it's own repoze.what
> plugin to handle all things which will give it the most integration
> with other tg2 apps.

Agreed, and I think the respoze.what folks are actually looking into
row-level permissions already, because that really is a common need.
Alberto Vargas wrote something called TurboPeakSecurity a long time
ago, and those ideas seem to be coming up again in the repoze.what
discussion. I think Jorge is right on about using repoze.who/what
because they are going to support all the external authentication and
metadata provision systems, and I really don't think Tortuga should go
down that road on it's own.

--Mark Ramm

[Chris B - JK at asciiking dot com]

On Nov 18, 11:05 pm, Chris B - JK at asciiking dot com

<swift2plun...@gmail.com> wrote:
> > The permissions system isn't written yet, but it will involve the
> > standard users/roles/permissions scheme. Plugins will have to enforce
> > the permissions, because without an object-oriented database like
> > ZODB, there's no real way to protect the data on a case-by-case basis.
>
> OK, that's the 3-part predicate that I'm looking for.

Except that it's not. They are the is_user, is_member, and
has_permission methods for authorize. (Insert elaborate curse here.)

The has_permission('verb') method is perfectly adequate for statically
defined content types. As long as you have a hand-coded chain of
controller methods with a lookup at the end then the whole thing falls
apart at the code level long before failure to normalize the database
bites you. It's when you start dynamically defining content types and
nesting lookup methods in the controllers that you find X users/groups
* Y number of content types * Z actions = "In the time this page was
generated, the CEO made enough money to feed a third world nation" for
some values of X and Y low enough to put a consultant on the customer
side of the soup line.

In order to normalize the database and parse URLs in a sane way there
needs to be a 1:1 correspondence between calls to the lookup method
and keywords in the authorize.has_permission call. I've asked a couple
times about this in the TG group, but I haven't hit upon the right
combination of words to communicate that effectively. At least I think
that's the problem. I have a tendency to flip-flop between genius and
moron. When I'm not understood, it gets difficult to tell whether it's
because of genius or, well, some other reason. It could well be that
the method is loaded so that it expects the right number of keywords
for the number of columns in the permissions database. That's the way
it should be, but I'm not seeing any sign that there was a design
decision to that effect. I'm hoping that the CMS application is
complex enough to illustrate the desirability of generalizing the
function in this way. Please, if you grok this then explain it to Mark
(and if I'm having a medicine dream then tell me to go to my tent).
You're going to have situations in a company CMS where business units
get spun off and you'll wish that you could factor out permissions for
that content with a simple select query instead of using wildcard or
regex searches.

Chris

[Chris B - JK at asciiking dot com]

> Agreed, and I think the respoze.what folks are actually looking into
> row-level permissions already, because that really is a common need.
> Alberto Vargas wrote something called TurboPeakSecurity a long time
> ago, and those ideas seem to be coming up again in the repoze.what
> discussion. I think Jorge is right on about using repoze.who/what
> because they are going to support all the external authentication and
> metadata provision systems, and I really don't think Tortuga should go
> down that road on it's own.

OK, now we're there. "Row-level permissions" is the vocabulary I was
looking for.

The repoze discussion fell off my Google map, but that's where I
should be poking around for info on this issue.

Chris