Prop Web Service Implemented in OpenPalace Client

[Brian McKelvey]

Alright... the openpalace client now works with the openpalace web service for props.

I've got it running on my dev server, storing data to my personal amazon s3 account.

Try it out at this private beta url:

http://www.openpalace.org/demo_webservice

With this change, OpenPalace will request information about props from the OpenPalace Web Service first.  It will request props from the palace server only if the OpenPalace Web Service doesn't have them.  Once the client downloads a new prop from the Palace server, it will convert it to a PNG image and upload it to the web service for others to access.  Each prop requires between 0.08 and 6.0 KB to store, depending on how complex the image is.  For props that are available on the web service, this massively speeds up prop download and rendering, while simultaneously lightening the load on the palace server.

Try it out, let me know what you think, and if you run into any issues with it.

Brian

[Heini]

I tried the demo and it works wonderful.

What about the growing datasize of the props on amazon? Our Palazzo.at
server-propfile grows to several hundred megabytes each week, and if
it gets to big, it slows down everything. If in future the maiority of
the users use the flex-client, a lot of data will flow over your
webservice.

Greets
Heini

[Brian McKelvey]

It's currently occupying 78MB on Amazon.  But the lovely thing about Amazon is that it's such a huge-scale cloud architecture, that the volume of data doesn't affect the retrieval speed.  Hundreds of gigabytes shouldn't be a problem on that end.

The Metadata about the props is stored in a MySQL database on my server that currently has just over 160,000 rows (one row per prop).  It consumes about 72MB for just the metadata and table indexes.

The Web service just provides the metadata and provides the client with a URL to the location on Amazon S3 where it can download the image.

Fortunately, the web service is a regular LAMP architecture that should be able to scale fairly well... more front-end web application servers accessing a single MySQL instance, and eventually, more MySQL instances replicating from the master databases to enhance the read performance... maybe upgrade to beefier hardware, and throw in a memcached layer for ram caching across the cluster, and it should be able to handle whatever load as far as that goes.

Brian

[Brian McKelvey]

I suppose that might answer the question though... I'm expecting that if this becomes wildly popular, an entire cluster of servers may be required for the web service eventually.

Brian

[Heini]

So if i understood right, on amazon are just ordinary pictures, and
the whole metadata is in a mysql database. Would it be possible to run
a "local" mysql on the same server as the palace runs, and make
something like a cache for metadata which only looks up the data from
the main database when its not locally available? That would reduce
the serverload on the main server again, i think.

Im think its important, that a single server can run standalone even
if the main server is not reachable somehow (we had that problems from
time to time, when routing problems between europe and america
occured)

On 12 Sep., 02:05, Brian McKelvey <tur...@openpalace.org> wrote:
> I suppose that might answer the question though... I'm expecting that if
> this becomes wildly popular, an entire cluster of servers may be required
> for the web service eventually.
> Brian
>
> On Fri, Sep 11, 2009 at 5:03 PM, Brian McKelvey <tur...@openpalace.org>wrote:
>
> > It's currently occupying 78MB on Amazon.  But the lovely thing about Amazon
> > is that it's such a huge-scale cloud architecture, that the volume of data
> > doesn't affect the retrieval speed.  Hundreds of gigabytes shouldn't be a
> > problem on that end.
> > The Metadata about the props is stored in a MySQL database on my server
> > that currently has just over 160,000 rows (one row per prop).  It consumes
> > about 72MB for just the metadata and table indexes.
>
> > The Web service just provides the metadata and provides the client with a
> > URL to the location on Amazon S3 where it can download the image.
>
> > Fortunately, the web service is a regular LAMP architecture that should be
> > able to scale fairly well... more front-end web application servers
> > accessing a single MySQL instance, and eventually, more MySQL instances
> > replicating from the master databases to enhance the read performance...
> > maybe upgrade to beefier hardware, and throw in a memcached layer for ram
> > caching across the cluster, and it should be able to handle whatever load as
> > far as that goes.
>
> > Brian
>

[Brian McKelvey]

Having a lot of geographically disparate local MySQL replicants is not feasible, but one or two may be.  It would probably be feasible to have a replicated datacenter located in Europe to handle that sort of problem.  It depends on the frequency of changes to the database that would have to flow from the master to the replicants to keep them up to date.  Also, replicants can only be used for reading data, not updating it, so if the connection was severed, anything that depended on updating data would not be possible.  Facebook did something similar.  Their main datacenter is in California, but they enhanced responsiveness for European users by opening a new read-only data center on the east coast of the United States.  However, if the main datacenter in california becomes unreachable, the site can only be read-only from the east-coast datacenter.

Multi-master replication is an extremely hard problem and the only viable solutions that exist to solve it for a SQL database model are high-end, extremely expensive proprietary extensions for commercial databases, like Oracle.

One of my goals is actually to work toward a lot more centralization in the palace community.  It's true that having palace servers be completely stand-alone and independent has preserved the small remnant of the palace community this long, but it is that very same fragmentation that has prevented the Palace from regaining widespread popularity.

I intend to extend this web service to keep people's profiles including Palace registration codes, Prop Bags, Saved Avatars, etc., so that a user can log onto their own account from any computer and have all the same capabilities that they would have when they were at home using their local client.  Once the features are complete, the highest things that will be on my priority list are integration with social networks, using tools such as Facebook Connect, so that users can invite their Facebook friends to join them in the palace where they're currently chatting, and organically spread interest in the Palace.

I also intend to make the specifications for the web service open, so that other client authors can include the web service features in their existing clients.  These desktop clients would potentially be able to keep locally cached copies of a users profile and prop bag in case of loss of connectivity to the web service.  Additionally, the palace server will still have all props that people have used there in its own prop file, so even without the web service, it shouldn't really interrupt normal operation, at least for the desktop clients.  OpenPalace, being a web application, is of course much more dependent on a web service to store extended user data like props, since there are no capabilities to store data on a user's hard drive via a web application.

Brian

[Michael Heiniger]

Ok, i see you have big plans and its exactly what i'm thinking too: integration in some social networks like Facebook will help to repopulate palaces again.
I tried once to integrate instantpalace into a portal which displays fotos of partys (like www.tillate.com), but without user-login and profile page, the owner of that portal didn't want it.

2009/9/13 Brian McKelvey <tur...@openpalace.org>

[Heini]

Any news about:
"I also intend to make the specifications for the web service open, so that other client authors can include the web service features in their existing clients" ?

Heini

[Brian McKelvey]

Sorry for the late response, but this is pretty much defunct now.  It's been supplanted by a new API that has a smaller scope, implemented by PalaceChat.  Reach out to Jameson Heesen (Pawn) for more info.

Brian

--

You received this message because you are subscribed to the Google Groups "OpenPalace Developers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to openpalace-devel...@googlegroups.com.
To post to this group, send email to openpalace...@googlegroups.com.
Visit this group at http://groups.google.com/group/openpalace-developers.
For more options, visit https://groups.google.com/groups/opt_out.