Federation Ideas

[stwf]

After having a few days to mull this over...

I noticed 2 real problems:

1) a real limiting factor in running d* on smaller hardware is the federation issue. A slower server could get overwhelmed if it subscribed to some popular tags on a huge server.

2) This stuff is hard.

So why not solve both problems at once and make federation purely (maybe not purely) a polling process. I would actually propose having the files served up statically (since thats very easy), written out as xml at predefined times. This way all of your server code just goes away.

You can just put all of the smarts into the polling code. This way when my pod decides, it can connect, grab a file of all of the action in the public arena, possibly a different file for any interaction specific to my pod, and import these into my local database. In the case of a comment on a post from a third pod, a url could be included and the third pod could be added to the polling. Maybe include a ping mechanism to initiate a poll, but simple, a send and forget.

I do have some ideas as to the mechanics of it all, but that is a complicated issue. But my main point is that by having federation be a (mostly) polling issue it greatly simplifies the code, and with outgoing federation be statically served files you night be able to significantly shrink the resource needs.

[Steven Hancock]

In general, I like the idea.. but then the question becomes, what about the pods (including joindiaspora.com) hosted on Heroku? As you may know, Heroku gives the running app read-only access to the filesystem (all except Rails.root/tmp which is not shared between instances for an app running more than one dyno). Thanks to CarrierWave the issue of file uploads in this kind of environment has already been addressed, so hosting on Heroku is at least possible if not a little difficult.. by making the app write out and serve static files, we're right back to a situation where you have to rent a VPS or dedicated server (or at least find shared Rails hosting that gives you both mysql and redis.. trust me I'd rather just rent a VPS and set it up myself lol) just to host your own personal pod and federate with the rest of the network.

[Tom Scott]

I don't believe static files are necessary for this at all. We simply need an XML route on the Rails app somewhere (e.g. /status.xml) that looks up a single row in the database representing the latest federation data. This could either be a fully-built XML file saved in a :text column or a bunch of data in :string columns that we insert into an XML Builder template on the fly. The point is, anything you can store in the file system can also be stored (usually more securely) in the database, for the database is nothing more than organized, tracked files on the file system. 

The concepts make sense to me though: we cache outbound data for other pods into a single resource to cut down on load times and complex/constant HTTP requests.

-T

--
You received this message because you are subscribed to the Google Groups "diaspora-dev" group.
To view this discussion on the web visit https://groups.google.com/d/msg/diaspora-dev/-/slcy1kWrN84J.
To post to this group, send email to diaspo...@googlegroups.com.
To unsubscribe from this group, send email to diaspora-dev...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/diaspora-dev?hl=en.

[Sarah Mei]

It's a bit complicated to cache the denormalized federation data, because it has to be constantly updated with comments, likes, deletions, etc. Ultimately we'll probably want to do that on a per-user basis, probably in some sort of in-memory db, so at that point we could think about extending it to federation.

[stwf]

Actually I was thinking that each item would really be just a diff of what happened during that time period.

So if I commented on a post you made (on your pod) 3 weeks ago, the federated data would just contain my comment, identified by the guid of your post, along with the url where it is federated (which I remember from when I grabbed it). So if someone on a third pod grabs my public posts they will know to then go and grab the original data from your pod if they don't already have it.

In that case where it was on a pod they hadn't known about they can grab a bunch of public data, possibly providing lots of good new content to the pods people and making everyone happy.

On Monday, March 5, 2012 9:53:06 AM UTC-5, Sarah Mei wrote:

It's a bit complicated to cache the denormalized federation data, because it has to be constantly updated with comments, likes, deletions, etc. Ultimately we'll probably want to do that on a per-user basis, probably in some sort of in-memory db, so at that point we could think about extending it to federation.

On Monday, March 5, 2012, Tom Scott <tu...@psychedeli.ca> wrote:
> I don't believe static files are necessary for this at all. We simply need an XML route on the Rails app somewhere (e.g. /status.xml) that looks up a single row in the database representing the latest federation data. This could either be a fully-built XML file saved in a :text column or a bunch of data in :string columns that we insert into an XML Builder template on the fly. The point is, anything you can store in the file system can also be stored (usually more securely) in the database, for the database is nothing more than organized, tracked files on the file system. 
> The concepts make sense to me though: we cache outbound data for other pods into a single resource to cut down on load times and complex/constant HTTP requests.
> -T
> On Mar 4, 2012, at 11:41 PM, Steven Hancock <steve...@gmail.com> wrote:
>
> In general, I like the idea.. but then the question becomes, what about the pods (including joindiaspora.com) hosted on Heroku? As you may know, Heroku gives the running app read-only access to the filesystem (all except Rails.root/tmp which is not shared between instances for an app running more than one dyno). Thanks to CarrierWave the issue of file uploads in this kind of environment has already been addressed, so hosting on Heroku is at least possible if not a little difficult.. by making the app write out and serve static files, we're right back to a situation where you have to rent a VPS or dedicated server (or at least find shared Rails hosting that gives you both mysql and redis.. trust me I'd rather just rent a VPS and set it up myself lol) just to host your own personal pod and federate with the rest of the network.
>
> On Sunday, March 4, 2012 12:02:50 PM UTC-8, stwf wrote:
>>
>> After having a few days to mull this over...
>> I noticed 2 real problems:
>> 1) a real limiting factor in running d* on smaller hardware is the federation issue. A slower server could get overwhelmed if it subscribed to some popular tags on a huge server.
>> 2) This stuff is hard.
>> So why not solve both problems at once and make federation purely (maybe not purely) a polling process. I would actually propose having the files served up statically (since thats very easy), written out as xml at predefined times. This way all of your server code just goes away.
>> You can just put all of the smarts into the polling code. This way when my pod decides, it can connect, grab a file of all of the action in the public arena, possibly a different file for any interaction specific to my pod, and import these into my local database. In the case of a comment on a post from a third pod, a url could be included and the third pod could be added to the polling. Maybe include a ping mechanism to initiate a poll, but simple, a send and forget.
>> I do have some ideas as to the mechanics of it all, but that is a complicated issue. But my main point is that by having federation be a (mostly) polling issue it greatly simplifies the code, and with outgoing federation be statically served files you night be able to significantly shrink the resource needs.
>>
>>
>>
>>
>>
>>
>>
>>
>>
> --
> You received this message because you are subscribed to the Google Groups "diaspora-dev" group.
> To view this discussion on the web visit https://groups.google.com/d/msg/diaspora-dev/-/slcy1kWrN84J.
> To post to this group, send email to diaspo...@googlegroups.com.

> To unsubscribe from this group, send email to diaspora-dev+unsubscribe@googlegroups.com.

> For more options, visit this group at http://groups.google.com/group/diaspora-dev?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups "diaspora-dev" group.
> To post to this group, send email to diaspo...@googlegroups.com.

> To unsubscribe from this group, send email to diaspora-dev+unsubscribe@googlegroups.com.