ShiftSpace in the cloud
Mushon | Shual
Greetings from the land of eternal sunshine for the blindspot minds, I am getting accustomed to my (re)new(ed) space and would love to shift some of my gears back to §.
Two weeks ago I attended a workshop run by Amazon CTO Dr. Werner Vogel. The event was obviously promotional but it was also very informative and has taught me a lot about Amazon's cloud approach. As we have been discussing much in the § team, cloud services are a problematic paradigm of the social web. They often lead to users shedding every expectance of privacy, to lock-in syndromes and to reduced of user agency in the platforms they are invested in. For me this has led to a quite negative blanket approach to the hype around cloud services. Amazon itself has been guilty of this problematic approach with Amazon Mechanical Turk being the poster-child for user-generated-exploited-labor.
But the truth is, ShiftSpace is already a cloud service. And we're already running our infrastructure (the Metatron server) in a server farm we cant physically reach and closely monitor (in comparison to running § of off a server in Doron's home. We've been paying $90 a month for hosting our own box somewhere in Long Island. Now, when our server needs some fundamental attention we're in trouble, since we only have one box and we need to take it down. Beyond that, we have not used Metatron's full cycle capacity in the past two years or so (to say the least) as while working on §1.0 getting new users was not a priority. But when 1.0 is out it is quite reasonable we will get a lot of attention that would translate to a request load that can easily overwhelm Metatron, resulting in some unfortunate fail-metawhales…
These concerns can be met by transition of the § server-side to AWS. I have talked with Dr. Vogel who has also explained to me their approach to privacy. While they are required to abide by the laws of the countries they are running their business in, they are also adamant about not caving in and protecting their users privacy. For example they do not accept security letters (CIA unwarranted secret search, Patriot Act BS). And in the case of a warranted inquiry they make sure to inform the client as soon as they get the call. Since they don't want to know or keep any record of what takes place on any instance, this is probably as much privacy as one can get. And with Amazon's high-end security level, it probably beats even what we can get running Metatron from under Doron's pillow. AWS's model is infrastructure as a service, they want their services to be as low level as electricity and they are working to make them as accessible. Amazon sees AWS becoming a possibly bigger business to them than online retail. It is far from being a side project for them and they stand a lot to lose from disappointed customers.
One last reason I think AWS is good for us is the potential of teams to build §-based business models. If we can easily serve and maintain custom §-installations for clients, we can soon make the project sustain itself with low overhead for us + save some extra money for hiring deva on client/core assignments. This is obviously a complicated issue and means starting a separate start up based on § (similar model to OSM/CloudMade) but it is important to make options like this available for us in the long run. Especially since we've already seen a demand for such a model and a willingness to pay for it. If we can lower the price and complexity of deploying any social/UGC interface in any website (whether you own it or not) then we are offering something valuable that people would want to pay for. This also plays nice with our plans for a decentralized, federated and finally distributed model as it makes running § servers more accessible.
I have already discussed this with David & Dan over IRC. Both agreed AWS is a good option for ShiftSpace and should be something to consider sooner rather than later. I haven't run this by Doron yet. Being the Metatron-master he is the #1 person from whom to get feedback, blessing or opposition from. In a case we move fwd with this, he will also be the main person to lead this transition.
I would love for us to discuss this options and to raise whatever issues need to be raised here.
Thanks again, I'm off to the beach!
Mushon Zer-Aviv
Mushon.com | Shual.com | @Mushon
Doron Ben-avraham
I think its an excellent idea, but i am wondering if we are getting
ahead of ourselves.
my concerns for the platform have always been scalability and
reliability, our current setup does not meet these goals, this is due in
part to our single server store, but also to the platform current (pre
1.0) limitation in ability to scale.
I would be hesitant to label our current setup a cloud in the sense it
is most commonly applied, because there is no mechanism for our system
to be scale aware vertically or horizontally, if it dies it dies, and
the role it serves cant be transferred to another node. as such it is no
cloud, but a box in a farm. There are elements in 1.0 design that might
enable this capability, we need to figure out how to properly implement
them if we are to use the cloud services to their full extent. and scale
on EC2 if we are required to do so, otherwise we will just move to
another box in a farm (albeit virtual) put simply, our ability to scale
needs to be inherent to the software we build, reliance on the elastic
block might be a bad and costly design choice ( for a platform that
generates no income).
There is the issue of overall pricing, primarily of traffic that we will
have to account for, as well as the additional services we will need to
commit to so we can maximize our use scenario, lastly, and this might be
a departure from this subject.
IMO shiftspace will need to become some sort of legal entity (nonprofit
or forprofit, we need to figure it out)
D
Dan Phiffer
David, in the Couch.IO case study text you mentioned something about searching for document IDs in the properties (using Lucene? or Couch views?). Is that written in more detail somewhere? Is the plan to use the built-in CouchDB replication functionality? If so, how do we prevent non-public data from being shared openly? I haven't been paying close attention to the discussion about this so far, so apologies if this has been covered on the list already...
-Dan
> --
> You received this message because you are subscribed to the Google Groups "ShiftSpace-dev" group.
> To post to this group, send email to ShiftSp...@googlegroups.com.
> To unsubscribe from this group, send email to ShiftSpace-de...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/ShiftSpace-dev?hl=en.
>
David Nolen
> I'm curious what it would take for us to actually replicate the data in some distributed way (although I'm not sure what horizontal vs. vertical means, as Doron described).
Not much to do. We would pretty much use CouchDB replication but we do need some kind of authentication step.
> David, in the Couch.IO case study text you mentioned something about searching for document IDs in the properties (using Lucene? or Couch views?). Is that written in more detail somewhere? Is the plan to use the built-in CouchDB replication functionality? If so, how do we prevent non-public data from being shared openly? I haven't been paying close attention to the discussion about this so far, so apologies if this has been covered on the list already...
>
> -Dan
The only way to prevent non-public data from being shared is for it to never leave the client's machine.
David
Dan Phiffer
On Oct 23, 2010, at 3:01 AM, Mushon | Shual wrote:
> These concerns can be met by transition of the § server-side to AWS. I have talked with Dr. Vogel who has also explained to me their approach to privacy. While they are required to abide by the laws of the countries they are running their business in, they are also adamant about not caving in and protecting their users privacy. For example they do not accept security letters (CIA unwarranted secret search, Patriot Act BS).
I'm curious how the team feels about Amazon cloud services post-Wikileaks shut-down? Related: does anyone have a Diaspora invite to spare?
Doron Ben Avraham
certain scalability that, should we need it is there. but retaining a copy
we can move to and keep is essential. the problem with Amazon is a problem
with any and all ISP, the ideal solution would be user based distribution.
D
Mushon Zer-Aviv
For better or worse ShiftSpace is not Wikileaks, and for better or worse it doesn't seem to get as much people as passionate.
Mushon Zer-Aviv
Shual.com
- design studio
§ ShiftSpace.org - an
opensource layer above any website
¶ Mushon.com - blog
× @mushon - Tweet me
Dan Phiffer
On Dec 9, 2010, at 10:31 AM, Doron Ben Avraham wrote:
> Certainly a very valid point, i would contend that a amazon offers a
> certain scalability that, should we need it is there.
Until you get DDoS'd, which apparently is against their ToS.
> the problem with Amazon is a problem
> with any and all ISP, the ideal solution would be user based distribution.
Yeah, user-based distribution was what I was getting at. I do understand that it's a hard problem, and that maybe ShiftSpace is not in a position to be taking on more hard problems.
-Dan
Dan Phiffer
On Dec 9, 2010, at 10:41 AM, Mushon Zer-Aviv wrote:
> I agree with Doron. Amazon is more likely to contest government intrusion to its web services than most ISPs exactly because it's taking some heat for doing this with Wikileaks.
I come to a different conclusion -- Amazon has shown they will drop a customer for (understandable) business reasons and perhaps the government has discovered that technological solutions are cheaper than Bush-era secret lawyering.
> For better or worse ShiftSpace is not Wikileaks, and for better or worse it doesn't seem to get as much people as passionate.
Obviously I wouldn't imply otherwise. To me it's more of a perception problem -- a cloud-based ShiftSpace is not an attractive choice for controversial speech.
-Dan
> Mushon Zer-Aviv
> <shual_gray.gif> Shual.com - design studio
David Nolen
>
> Obviously I wouldn't imply otherwise. To me it's more of a perception problem -- a cloud-based ShiftSpace is not an attractive choice for controversial speech.
>
> -Dan
At the moment, cloud-based or no, it's not even an attractive choice for non-controversial speech. First it needs to allow people to speak in a compelling new way. Then someone might actually say something controversial.
David