mediaserver brainstorming
43 views
Skip to first unread message
dodo
Apr 30, 2012, 12:33:18 PM4/30/12
to buddycl...@googlegroups.com
here is a my chatlog with rodrigods about the media server architecture design.
please feel free to criticize :)
[12:07:23 AM] Status is now: Available
[12:14:52 AM] rodrigods: ping
[12:15:00 AM] dodo: pong
[12:15:04 AM] rodrigods: i did a sketch of how i visualize the media server
[12:15:05 AM] rodrigods: http://dl.dropbox.com/u/60827853/mediaserver.png
[12:16:36 AM] dodo: what kind of storage controller do you have in mind?
[12:16:59 AM] rodrigods: if we use java, we can have a layer
[12:17:06 AM] rodrigods: that is independent of technology
[12:17:15 AM] rodrigods: but... for the reference implementation
[12:17:23 AM] rodrigods: i'm thinking about using mongodb
[12:17:28 AM] rodrigods: with gridfs
[12:19:04 AM] dodo: what do you think what kind of data will be stored
in the media server?
[12:19:26 AM] rodrigods: binary data
[12:19:26 AM] rodrigods: with their metadata
[12:20:45 AM] rodrigods: (and also previews, for the case of images and videos)
[12:20:53 AM] rodrigods: (also, binary data)
[12:21:14 AM] dodo: hm .. yes binary data .. isnt the best way to
store these to put them just in a filesystem?
[12:21:35 AM] rodrigods: gridfs is a file system
[12:21:46 AM] rodrigods: but we also have the option to store only the metadata
[12:21:52 AM] rodrigods: with the path
[12:21:54 AM] rodrigods: for a fs
[12:22:22 AM] dodo: ah .. ok.. good, that,, because i thougth you
wanted to store the files in mongodb too :)
[12:22:35 AM] dodo: s/that//
[12:22:49 AM] rodrigods: gridfs is like a fs the mongodb uses to store
binary files
[12:23:00 AM] rodrigods: it automatically
[12:23:06 AM] rodrigods: generates a hash of the file
[12:23:14 AM] rodrigods: and also have fields just only metadata
[12:23:27 AM] rodrigods: i thought that it would be easier to handle
[12:24:01 AM] dodo: the files can have metadata in gridfs?
[12:25:06 AM] rodrigods: yes
[12:25:31 AM] dodo: so you want to store metadata twice? :}
[12:25:43 AM] dodo: (in gridfs and in mongodb)
[12:26:30 AM] rodrigods: gridfs doesn't generate all metadatas that we need
[12:26:38 AM] rodrigods: and it has a field
[12:26:41 AM] rodrigods: just to add it
[12:28:10 AM] rodrigods:
http://www.mongodb.org/display/DOCS/GridFS+Specification
[12:31:10 AM] dodo: what else does gridfs offer, that is neccessary
for the media server?
[12:31:48 AM] rodrigods: it scales very well
[12:31:58 AM] rodrigods: (people say, i haven't experience with it)
[12:32:42 AM] rodrigods: but... if the latency of adding the file to
it, becomes a problem
[12:32:47 AM] rodrigods: we can change the approach
[12:32:53 AM] dodo: but is it neccessary?
[12:32:54 AM] rodrigods: and stores directly on a data base
[12:33:49 AM] rodrigods: hmm
[12:34:00 AM] rodrigods: yes... in a first moment
[12:34:22 AM] rodrigods: i don't see why we would have a big
scalability requisite
[12:35:07 AM] rodrigods: but... what in my head
[12:35:16 AM] rodrigods: the media server should be able to be used
for a lot of users
[12:35:26 AM] rodrigods: of different bc deploys
[12:35:38 AM] rodrigods: we can have one media server
[12:35:51 AM] rodrigods: for more than one bc deploy
[12:36:48 AM] dodo: i think there will be two kind of people that will
install a buddycloud instance with a media server: sysadmins ad
average nerds
[12:37:31 AM] dodo: the average nerd has a small number of users ..
with low scalability problems (probably none)
[12:38:14 AM] dodo: the sysadmin is an "experienced" user that will
probably want to handle the scalability problem on his own
[12:39:19 AM] dodo: which more or less means for us: dont handle
scalability of the storage and use a simple default (simple in setup
_and_ maintainance)
[12:39:44 AM] dodo: do you agree with that? or do you think that im
total nuts? :D
[12:40:02 AM] rodrigods: heheh
[12:40:03 AM] rodrigods: i agree
[12:40:17 AM] dodo: ok, cool
[12:41:08 AM] dodo: so .. im my opinion a simple solution would be to
just use the filesystem available on the machine serving the media
server
[12:43:16 AM] rodrigods: yes... maybe i was too much concerned with
the wrong thing
[12:43:23 AM] dodo: this solves both cases quiet well .. if a sysadmin
wants to scale the storage he can do it with the tools that she is
already familiar with (the day to day fs tools) .. if a average nerd
wants to install a media server, all she needs to add is a root path
for the folder containing the data and the credentials for the
metadata database
[12:45:13 AM] rodrigods: so we have to implement a good way for
organizing the files
[12:45:28 AM] rodrigods: a clever way
[12:45:38 AM] dodo: and especially syncing the metadata with the storage
[12:45:47 AM] rodrigods: we must be aware about some particularities of fs
[12:45:52 AM] dodo: yes
[12:46:19 AM] dodo: but i think they boil down to just a couple of ground rules
[12:46:43 AM] rodrigods: yep
[12:46:49 AM] rodrigods: we do that later
[12:46:59 AM] dodo: agreed :)
[12:47:39 AM] rodrigods: so, you agree about building a java storage
controller that communicates with a java xmpp component?
[12:47:58 AM] rodrigods: (the lower layer of the drawing)
[12:50:56 AM] dodo: im still uncomfortable with the idea if using java :P
[12:51:14 AM] rodrigods: heheh
[12:51:20 AM] rodrigods: which are our options
[12:51:25 AM] rodrigods: for this layer
[12:51:25 AM] rodrigods: i mean
[12:55:06 AM] dodo: well .. since you're keen to use java, i asume you
know the language quiet well, which means this is still an option ..
the other options are: luvit, nodejs, erlang, python, ruby, haskell, perl, etc…
[12:55:49 AM] rodrigods: ok...
[12:55:58 AM] rodrigods: and about the http layer
[12:56:52 AM] dodo: i like software that is easy to hack to code ..
especially in software for federations, because that encorages people
to start add new and awesome features
[12:58:32 AM] dodo: about the http layer i think, that projects like
nginx and apache a quiet good in serving files .. all that we have to
figure out is how to set permissions for files .. but i think
something NIH (own http server) cant we that bad :P
[12:59:49 AM] dodo: i would like to prefer nginx :}
[01:00:50 AM] rodrigods: that's your area i think
[01:01:02 AM] rodrigods: this part i'm very (totally) naive
[01:02:06 AM] dodo: the permissions thingy is the main reason why we
need our own media server solution
[01:03:57 AM] rodrigods: in my head
[01:04:02 AM] rodrigods: the flow is like that:
[01:04:24 AM] rodrigods: client (web, etc) sends http reqs for the http layer
[01:04:33 AM] rodrigods: 1 - client (web, etc) sends http reqs for the
http layer
[01:05:25 AM] rodrigods: 2 - the http layer transmits the req details
for the storage controller (like user@server, and where to they want
to put the file)
[01:05:52 AM] rodrigods: 3 - the storage controller asks the xmpp
component if that user can do what he is intending to do
[01:06:16 AM] rodrigods: 4 - the xmpp server asks the bc server about
this user permissions and responds yes or no the storage controller
[01:07:16 AM] rodrigods: 5 - depending on the xmpp component response,
the storage controller stores or no the file and returns a preview for
the http layer
[01:07:35 AM] rodrigods: 6 - the http layer responds with the preview
for the client
[01:07:55 AM] rodrigods: for getting a file, a similar flow would work
[01:08:42 AM] dodo: so ,this is the flow for saving data on the media serveR?
[01:09:02 AM] rodrigods: that is what i imagine, at least
[01:10:58 AM] rodrigods: any thoughts?
[01:11:09 AM] rodrigods: i'm completely nuts?
[01:11:11 AM] rodrigods: heheh
[01:17:09 AM] dodo: im currently reading
https://buddycloud.org/wiki/Buddycloud_media_server
[01:17:50 AM] dodo: is that what you used as draft?
[01:18:22 AM] rodrigods: i used this doc:
https://docs.google.com/document/d/14mU8eXNGpzSn4axR36-doOfr-cq726KXwM-xwi8WD54/edit?pli=1
[01:19:00 AM] dodo: i mean your flow makes sense .. and suits that
description .. but i must say, its not very well thought trough
[01:19:25 AM] rodrigods: sure...
[01:19:32 AM] rodrigods: i'm the student here!
[01:19:42 AM] rodrigods: what do you think
[01:19:45 AM] rodrigods: that would be better?
[01:20:12 AM] dodo: sorry ..did mean your flow .. its the wiki page
that is not very useful :P
[01:20:22 AM] dodo: s/did/didnt/
[01:21:00 AM] rodrigods: i always was the beckend guy
[01:21:07 AM] rodrigods: implementing database
[01:21:12 AM] rodrigods: components
[01:21:17 AM] rodrigods: etc
[01:21:30 AM] rodrigods: sorry if sometimes i appear a little confused
[01:21:36 AM] rodrigods: with http stuff
[01:21:45 AM] rodrigods: maybe is better use nodejs for everything
[01:21:54 AM] rodrigods: and i'm not aware about it
[01:23:48 AM] dodo: no its alright .. its just that it doesnt make
sense to me to make first a http request to the mediaserver, that wait
until the media server asks back over xmpp and so on .. but thats what
the wiki pages says
[01:24:50 AM] rodrigods: i think that is because would use the
channels permissions
[01:24:52 AM] rodrigods: based on pubsub
[01:25:13 AM] rodrigods: with a http api for doing the outside requests
[01:28:07 AM] rodrigods: sorry, the http api would be provided for the
media server, so we can have an independent way to build clients
[01:28:26 AM] rodrigods: simon cited the flickr http api
[01:29:41 AM] rodrigods: as example
[02:01:21 AM] dodo: alright .. i think i got something more useful
[02:02:56 AM] dodo: the media server consists out of nginx (serving
public data), filestsystem (storage) and an http server with the xmpp
component running
[02:03:27 AM] dodo: well dont has to be neccessarily a http server
[02:03:42 AM] dodo: but at least the xmpp component has to be there
[02:04:36 AM] dodo: what a bc enabled client has to do is:
[02:04:56 AM] dodo: 1. discover the channels media server
[02:05:56 AM] dodo: 2. negotiage a backend .. i think the simplest
one would be http (rest api)
[02:06:51 AM] dodo: 3. get an access token (appended to the url) or a
cookie from the http backend for private data
[02:07:57 AM] dodo: to get the data the client just has to get the
urls (if its public) and if the channel is private the client has to
append the token to url or to send the cookie to the server to
authenticate
[02:08:21 AM] dodo: sry .. i mean the client has to GET request the url
[02:09:01 AM] dodo: uploading data would be a POST or PUT request
[02:09:30 AM] dodo: what the media server has to do:
[02:10:12 AM] dodo: - serve a xmpp component
[02:10:26 AM] dodo: - discoverable via xmpp
[02:11:11 AM] dodo: - serve different backends like http rest api server
[02:12:36 AM] dodo: i think it would be best if the http is integrated
into the media server since the http part would have to query the xmpp
component for channel permissions
[02:13:27 AM] rodrigods: i understand
[02:13:41 AM] rodrigods: what i was envioning
[02:13:46 AM] rodrigods: was something differente
[02:13:47 AM] dodo: all the metadata would be accessable via xmpp
[02:14:06 AM] rodrigods: where the xmpp component itself would provide
a http api
[02:14:14 AM] rodrigods: envisioning*
[02:15:05 AM] dodo: with media server i meant the xmpp component
[02:16:42 AM] rodrigods: so the guy that is serving the files
[02:16:53 AM] rodrigods: is nginx
[02:17:41 AM] dodo: only for public data .. because i dont know how to
let nginx know the data's permissions
[02:18:08 AM] dodo: nginx or apache .. doesnt matter
[02:18:24 AM] dodo: or anything else that can serve static files
[02:19:48 AM] rodrigods: hmm
[02:19:58 AM] rodrigods: i'll make a draw
[02:20:06 AM] rodrigods: of this approach later =)
[02:20:14 AM] rodrigods: now i'm trying to imagine
[02:20:18 AM] rodrigods: how to handle private files
[02:20:29 AM] rodrigods: private channel files
[02:20:38 AM] dodo: they get served via http too
[02:20:46 AM] dodo: but not by nginx or whateva
[02:21:05 AM] dodo: i see two ways to do the authentication
[02:21:12 AM] dodo: one way is to use tokens
[02:22:07 AM] dodo: a token can be requested via xmpp and will be used
in the url like https://bcmedia.com/file?token
[02:22:20 AM] dodo: the token should expire somehow
[02:22:43 AM] dodo: the other way is to use cookies
[02:23:35 AM] dodo: get an cookie init page url from the media server
via xmpp and open that url .. the mediaserver will send a cookie to
the client
[02:24:06 AM] dodo: now for every data request over http the client
will send this cookie to the server
[02:24:36 AM] dodo: the cookie could contain an expiring token
[02:25:47 AM] dodo: im not sure if this whole approach is a good idea
.. because the security has to be implement for every different
backend in a specific way
[02:26:17 AM] dodo: for example a jingle backend would use the xmpps
allready established authentication
[02:27:49 AM] rodrigods: who is backend in this context? =P
[02:28:22 AM] dodo: oh .. just noticing that the token in url approach
is dumb, because that would descroy the browser capability to cache
files
[02:29:55 AM] dodo: right .. the backend would have the same problem
like security .. every backend would have to implement the right
storage usage on they own.
[02:31:21 AM] dodo: so how to generalise that?
[02:31:39 AM] dodo: or even security?
[02:32:23 AM] rodrigods: the backend is the guy serving the files?
[02:32:46 AM] dodo: yes, serving and receiving
[02:34:53 AM] rodrigods: i haven't understand an earlier step
[02:35:02 AM] rodrigods: that was the translating of the URL
[02:35:07 AM] rodrigods: into a file
[02:36:04 AM] rodrigods: sorry...
[02:36:08 AM] rodrigods: into a file permission
[02:36:23 AM] dodo: the token approach?
[02:36:28 AM] rodrigods: yep
[02:36:35 AM] rodrigods: for public channels
[02:37:23 AM] dodo: tfor public channels you dont need a token ..
because its pulbic, the client dont has to authenticate
[02:38:27 AM] rodrigods: sorry
[02:38:32 AM] rodrigods: i knew that
[02:38:34 AM] rodrigods: and somehow
[02:39:10 AM] rodrigods: have mixed the things
[02:39:36 AM] dodo: no problem :)
[02:41:02 AM] rodrigods: i thought in a keypair approach
[02:41:08 AM] rodrigods: or something like that
[02:41:17 AM] rodrigods: so the backend would know
[02:41:22 AM] dodo: the key would be the url
[02:42:07 AM] dodo: so when uploading the client would request a new
url on the server to post to
[02:42:24 AM] dodo: (request via xmpp)
[02:42:39 AM] dodo: (upload data via backend)
[02:43:28 AM] rodrigods: the url would be composed
[02:43:35 AM] rodrigods: by the channel
[02:43:44 AM] rodrigods: right?
[02:43:48 AM] rodrigods: something like
[02:43:49 AM] dodo: makes sense
[02:43:54 AM] dodo: yes
[02:45:42 AM] rodrigods: (sorry dodo, if is late there... where do you
live, btw?)
[02:45:43 AM] dodo:
https://media.buddycloud.org/ca...@topics.buddycloud.org/cutykitty.mkv
[02:45:52 AM] dodo: im from germany
[02:46:16 AM] dodo: its 2:45 in the morning ^^
[02:46:37 AM] rodrigods: the conversation is being very very productive
[02:46:41 AM] rodrigods: this*
[02:46:55 AM] rodrigods: anytime that you want to go to bed
[02:46:59 AM] rodrigods: just tell me =)
[02:47:10 AM] dodo: im not tired yet ^^
[02:47:34 AM] rodrigods: great
[02:47:40 AM] rodrigods: =)
[02:47:54 AM] rodrigods: hmm thinking now
[02:48:07 AM] rodrigods: when the client
[02:48:13 AM] rodrigods: asks the xmpp component
[02:48:15 AM] rodrigods: for a url
[02:48:19 AM] rodrigods: to get or to post
[02:48:54 AM] rodrigods: if the user doesn't have permission
[02:49:10 AM] rodrigods: the xmpp component just have to respond with an error
[02:49:15 AM] rodrigods: right?
[02:49:17 AM] dodo: normaly it doesnt has to query an url to GET ..
because the links would be in the posts
[02:49:29 AM] dodo: yes
[02:50:04 AM] rodrigods: we can hide the channel url
[02:50:06 AM] rodrigods: with a hash
[02:50:14 AM] rodrigods: that only the xmpp would know
[02:50:28 AM] rodrigods: and, aways the client
[02:50:32 AM] rodrigods: have to ask the url
[02:50:38 AM] rodrigods: if is public
[02:50:39 AM] rodrigods: or not
[02:50:43 AM] rodrigods: the channel
[02:50:56 AM] rodrigods: and it wouldn't be able to 'guess'
[02:50:59 AM] dodo: urls are public .. always :)
[02:51:00 AM] rodrigods: urls from private channels
[02:51:07 AM] rodrigods: the file path
[02:51:09 AM] rodrigods: i mean
[02:51:28 AM] rodrigods: instead of being
[02:51:30 AM] rodrigods:
https://media.buddycloud.org/ca...@topics.buddycloud.org/cutykitty.mkv
[02:51:32 AM] rodrigods: it would be
[02:51:42 AM] rodrigods:
https://media.buddycloud.org/hash(ca...@topics.buddycloud.org)/cutykitty.mkv
[02:51:56 AM] rodrigods: the hash, only the xmpp component would know
[02:52:18 AM] rodrigods: so... the client would never be able to guess
[02:52:20 AM] rodrigods: where a file is
[02:52:38 AM] rodrigods: thats make sense?
[02:53:32 AM] dodo: its like hiding your homes key under your doormat :)
[02:53:59 AM] dodo: i mean there is no way that you can garanty the
privacy of the url
[02:54:43 AM] dodo: so the only reason to do this is to obviouscate
the information in the url
[02:56:23 AM] dodo: i mean
https://media.buddycloud.org/a309ffba7e7b4ca266cb0935e2f1a312e4455631/c1f22dc0a5972b997a8f9196cfc9844ea9bea904
is still better than
https://media.buddycloud.org/hard...@topics.porn.com/under16.mp4
[02:56:51 AM] rodrigods: sure...
[02:58:11 AM] dodo: so obviouscating urls in private channels makes
sense but .. it doesnt matter that the xmpp component is the only one
that knows the url
[02:58:46 AM] dodo: the url should be usable in posts without concerns
[02:59:03 AM] rodrigods: yes
[02:59:16 AM] rodrigods: i haven't thought this through
[02:59:29 AM] rodrigods: its like facebook photos
[02:59:36 AM] rodrigods: once i have the url
[02:59:47 AM] rodrigods: doesn't matter if i'm friends with you
[03:00:01 AM] dodo: no .. not ecactly
[03:00:18 AM] dodo: because you can only get the urls data if you send
the right cookie
[03:00:34 AM] rodrigods: but...
[03:00:38 AM] rodrigods: once i get the url
[03:00:42 AM] rodrigods: i can access the file
[03:01:20 AM] dodo: you have to send your authentication cookie with a
token that you got via xmpp along with that url
[03:01:30 AM] rodrigods: ahh
[03:01:31 AM] rodrigods: ok
[03:01:52 AM] rodrigods: just one question
[03:01:56 AM] rodrigods: not related at al
[03:01:56 AM] rodrigods: all
[03:02:01 AM] rodrigods: with this most recent subject
[03:02:18 AM] rodrigods: the client would speak only http
[03:02:22 AM] rodrigods: right?
[03:02:50 AM] dodo: only when the http api server is ready ..
[03:03:04 AM] dodo: (the one that denis will build)
[03:03:29 AM] dodo: at least that is my idea :P
[03:03:30 AM] rodrigods: just received email from code jam
[03:03:41 AM] rodrigods: but i can publish a http api
[03:03:42 AM] rodrigods: rest api
[03:03:50 AM] rodrigods: can i?
[03:04:12 AM] dodo: you need your own
[03:04:26 AM] rodrigods: if the client
[03:04:31 AM] rodrigods: will speak xmpp
[03:04:37 AM] rodrigods: why do we need
[03:04:40 AM] rodrigods: a xmpp component
[03:04:45 AM] rodrigods: handling the media server?
[03:05:14 AM] rodrigods: ahh
[03:05:18 AM] rodrigods: for the url stuff?
[03:05:28 AM] rodrigods: only for that?
[03:05:35 AM] dodo: for the cookie
[03:06:06 AM] dodo: it can only request a valid cookie if its
authenticated via xmpp
[03:07:00 AM] dodo: oh .. and for serving the metadata the component
is also needed
[03:07:38 AM] rodrigods: sure
[03:07:42 AM] rodrigods: after this conversation
[03:07:48 AM] rodrigods: everything is much much clearer
[03:07:51 AM] rodrigods: in my head =)
[03:07:52 AM] rodrigods: thanks dodo
[03:08:13 AM] dodo: this is fun :)
[03:08:47 AM] rodrigods: code jam has just started!
[03:08:59 AM] rodrigods: i think that i'll only participate from the
furthers rounds
[03:09:16 AM] rodrigods: anyway
[03:09:36 AM] rodrigods: now that i have a bigger understanding
[03:09:44 AM] rodrigods: of the whole architecture
[03:10:00 AM] rodrigods: i will do the file serving stuff
[03:10:03 AM] rodrigods: with node.js
[03:10:06 AM] rodrigods: just a toy
[03:10:12 AM] rodrigods: to see if i learn a little
[03:10:19 AM] dodo: n!ce <3
[03:10:26 AM] dodo: :D
[03:10:35 AM] rodrigods: i can use coffee script
[03:10:41 AM] rodrigods: right?
[03:11:18 AM] dodo: coffeescript is just a fancy way to write
javascript .. it mostly macros a lot boilerplate code away
[03:11:28 AM] dodo: yes :>
[03:11:47 AM] rodrigods: great
[03:12:13 AM] rodrigods: the xmpp component
[03:12:19 AM] rodrigods: i insist in java
[03:12:22 AM] rodrigods: heheh
[03:13:09 AM] dodo: so you want a "real" seperation of backend and
server (xmpp component)?
[03:13:23 AM] rodrigods: yes
[03:13:34 AM] rodrigods: now i don't see why they would be together
[03:13:47 AM] rodrigods: if they don't talk with each other
[03:14:13 AM] dodo: they talk with each other else the http server
would know if a cookie is valid
[03:14:30 AM] dodo: but that can be done with redis or somthing
[03:14:53 AM] rodrigods: yep... i've forgot that
[03:15:01 AM] rodrigods: but... do you agree with this separation?
[03:15:12 AM] dodo: or they simply talk via http, tcp or whatevery
directly with each other
[03:15:20 AM] dodo: yes makes sense
[03:15:23 AM] dodo: at some point
[03:15:52 AM] dodo: i mean a faster way to get things done is to put
them together
[03:16:10 AM] dodo: and then try to implement more backends into it
[03:16:46 AM] dodo: and look at it again and decide what worked out
and what not and what can be shared and what can be seperated
[03:17:19 AM] rodrigods: the tricky thing would talk xmpp
[03:17:23 AM] rodrigods: in my opinion
[03:17:28 AM] rodrigods: or there is a xmpp lib
[03:17:30 AM] rodrigods: for nodejs?
[03:17:37 AM] dodo: node-xmpp by astro :D
[03:17:51 AM] dodo: he is sitting to my right ^^
[03:17:56 AM] rodrigods: ahhhh
[03:17:59 AM] rodrigods: hi astro!
[03:18:00 AM] rodrigods: heheh
[03:18:11 AM] dodo: hi back ^^
[03:18:35 AM] rodrigods: great
[03:18:37 AM] dodo: https://github.com/astro/node-xmpp
[03:18:41 AM] rodrigods: i'll give nodejs a try
[03:18:50 AM] rodrigods: don't promise that i'll love i
[03:18:51 AM] rodrigods: it
[03:18:53 AM] dodo: sweet :3
[03:18:53 AM] rodrigods: but...
[03:19:29 AM] dodo: i can give you a tutorial into nodejs and
coffeescript via skype
[03:20:01 AM] dodo: and you can test it first of course and then still
decide wether to go with it or to use java
[03:20:16 AM] rodrigods: i found here
[03:20:17 AM] rodrigods: a video
[03:20:34 AM] rodrigods: thanks for your ....
[03:20:45 AM] rodrigods: i don't know the word in english
[03:20:53 AM] rodrigods: =)
[03:21:32 AM] dodo: just type it in your native language (is it
spanish or portoguese?) :)
[03:22:00 AM] rodrigods: portuguese
[03:22:41 AM] rodrigods: now i'm not remembering in portuguese
[03:22:43 AM] rodrigods: anyway
[03:22:45 AM] rodrigods: heheh
[03:22:53 AM] dodo: h:D
[03:23:15 AM] rodrigods: ahh
[03:23:21 AM] rodrigods: 'prestatividade'
[03:23:46 AM] rodrigods: the translator says that is "helpfulness"
[03:24:13 AM] dodo: (:
please feel free to criticize :)
[12:07:23 AM] Status is now: Available
[12:14:52 AM] rodrigods: ping
[12:15:00 AM] dodo: pong
[12:15:04 AM] rodrigods: i did a sketch of how i visualize the media server
[12:15:05 AM] rodrigods: http://dl.dropbox.com/u/60827853/mediaserver.png
[12:16:36 AM] dodo: what kind of storage controller do you have in mind?
[12:16:59 AM] rodrigods: if we use java, we can have a layer
[12:17:06 AM] rodrigods: that is independent of technology
[12:17:15 AM] rodrigods: but... for the reference implementation
[12:17:23 AM] rodrigods: i'm thinking about using mongodb
[12:17:28 AM] rodrigods: with gridfs
[12:19:04 AM] dodo: what do you think what kind of data will be stored
in the media server?
[12:19:26 AM] rodrigods: binary data
[12:19:26 AM] rodrigods: with their metadata
[12:20:45 AM] rodrigods: (and also previews, for the case of images and videos)
[12:20:53 AM] rodrigods: (also, binary data)
[12:21:14 AM] dodo: hm .. yes binary data .. isnt the best way to
store these to put them just in a filesystem?
[12:21:35 AM] rodrigods: gridfs is a file system
[12:21:46 AM] rodrigods: but we also have the option to store only the metadata
[12:21:52 AM] rodrigods: with the path
[12:21:54 AM] rodrigods: for a fs
[12:22:22 AM] dodo: ah .. ok.. good, that,, because i thougth you
wanted to store the files in mongodb too :)
[12:22:35 AM] dodo: s/that//
[12:22:49 AM] rodrigods: gridfs is like a fs the mongodb uses to store
binary files
[12:23:00 AM] rodrigods: it automatically
[12:23:06 AM] rodrigods: generates a hash of the file
[12:23:14 AM] rodrigods: and also have fields just only metadata
[12:23:27 AM] rodrigods: i thought that it would be easier to handle
[12:24:01 AM] dodo: the files can have metadata in gridfs?
[12:25:06 AM] rodrigods: yes
[12:25:31 AM] dodo: so you want to store metadata twice? :}
[12:25:43 AM] dodo: (in gridfs and in mongodb)
[12:26:30 AM] rodrigods: gridfs doesn't generate all metadatas that we need
[12:26:38 AM] rodrigods: and it has a field
[12:26:41 AM] rodrigods: just to add it
[12:28:10 AM] rodrigods:
http://www.mongodb.org/display/DOCS/GridFS+Specification
[12:31:10 AM] dodo: what else does gridfs offer, that is neccessary
for the media server?
[12:31:48 AM] rodrigods: it scales very well
[12:31:58 AM] rodrigods: (people say, i haven't experience with it)
[12:32:42 AM] rodrigods: but... if the latency of adding the file to
it, becomes a problem
[12:32:47 AM] rodrigods: we can change the approach
[12:32:53 AM] dodo: but is it neccessary?
[12:32:54 AM] rodrigods: and stores directly on a data base
[12:33:49 AM] rodrigods: hmm
[12:34:00 AM] rodrigods: yes... in a first moment
[12:34:22 AM] rodrigods: i don't see why we would have a big
scalability requisite
[12:35:07 AM] rodrigods: but... what in my head
[12:35:16 AM] rodrigods: the media server should be able to be used
for a lot of users
[12:35:26 AM] rodrigods: of different bc deploys
[12:35:38 AM] rodrigods: we can have one media server
[12:35:51 AM] rodrigods: for more than one bc deploy
[12:36:48 AM] dodo: i think there will be two kind of people that will
install a buddycloud instance with a media server: sysadmins ad
average nerds
[12:37:31 AM] dodo: the average nerd has a small number of users ..
with low scalability problems (probably none)
[12:38:14 AM] dodo: the sysadmin is an "experienced" user that will
probably want to handle the scalability problem on his own
[12:39:19 AM] dodo: which more or less means for us: dont handle
scalability of the storage and use a simple default (simple in setup
_and_ maintainance)
[12:39:44 AM] dodo: do you agree with that? or do you think that im
total nuts? :D
[12:40:02 AM] rodrigods: heheh
[12:40:03 AM] rodrigods: i agree
[12:40:17 AM] dodo: ok, cool
[12:41:08 AM] dodo: so .. im my opinion a simple solution would be to
just use the filesystem available on the machine serving the media
server
[12:43:16 AM] rodrigods: yes... maybe i was too much concerned with
the wrong thing
[12:43:23 AM] dodo: this solves both cases quiet well .. if a sysadmin
wants to scale the storage he can do it with the tools that she is
already familiar with (the day to day fs tools) .. if a average nerd
wants to install a media server, all she needs to add is a root path
for the folder containing the data and the credentials for the
metadata database
[12:45:13 AM] rodrigods: so we have to implement a good way for
organizing the files
[12:45:28 AM] rodrigods: a clever way
[12:45:38 AM] dodo: and especially syncing the metadata with the storage
[12:45:47 AM] rodrigods: we must be aware about some particularities of fs
[12:45:52 AM] dodo: yes
[12:46:19 AM] dodo: but i think they boil down to just a couple of ground rules
[12:46:43 AM] rodrigods: yep
[12:46:49 AM] rodrigods: we do that later
[12:46:59 AM] dodo: agreed :)
[12:47:39 AM] rodrigods: so, you agree about building a java storage
controller that communicates with a java xmpp component?
[12:47:58 AM] rodrigods: (the lower layer of the drawing)
[12:50:56 AM] dodo: im still uncomfortable with the idea if using java :P
[12:51:14 AM] rodrigods: heheh
[12:51:20 AM] rodrigods: which are our options
[12:51:25 AM] rodrigods: for this layer
[12:51:25 AM] rodrigods: i mean
[12:55:06 AM] dodo: well .. since you're keen to use java, i asume you
know the language quiet well, which means this is still an option ..
the other options are: luvit, nodejs, erlang, python, ruby, haskell, perl, etc…
[12:55:49 AM] rodrigods: ok...
[12:55:58 AM] rodrigods: and about the http layer
[12:56:52 AM] dodo: i like software that is easy to hack to code ..
especially in software for federations, because that encorages people
to start add new and awesome features
[12:58:32 AM] dodo: about the http layer i think, that projects like
nginx and apache a quiet good in serving files .. all that we have to
figure out is how to set permissions for files .. but i think
something NIH (own http server) cant we that bad :P
[12:59:49 AM] dodo: i would like to prefer nginx :}
[01:00:50 AM] rodrigods: that's your area i think
[01:01:02 AM] rodrigods: this part i'm very (totally) naive
[01:02:06 AM] dodo: the permissions thingy is the main reason why we
need our own media server solution
[01:03:57 AM] rodrigods: in my head
[01:04:02 AM] rodrigods: the flow is like that:
[01:04:24 AM] rodrigods: client (web, etc) sends http reqs for the http layer
[01:04:33 AM] rodrigods: 1 - client (web, etc) sends http reqs for the
http layer
[01:05:25 AM] rodrigods: 2 - the http layer transmits the req details
for the storage controller (like user@server, and where to they want
to put the file)
[01:05:52 AM] rodrigods: 3 - the storage controller asks the xmpp
component if that user can do what he is intending to do
[01:06:16 AM] rodrigods: 4 - the xmpp server asks the bc server about
this user permissions and responds yes or no the storage controller
[01:07:16 AM] rodrigods: 5 - depending on the xmpp component response,
the storage controller stores or no the file and returns a preview for
the http layer
[01:07:35 AM] rodrigods: 6 - the http layer responds with the preview
for the client
[01:07:55 AM] rodrigods: for getting a file, a similar flow would work
[01:08:42 AM] dodo: so ,this is the flow for saving data on the media serveR?
[01:09:02 AM] rodrigods: that is what i imagine, at least
[01:10:58 AM] rodrigods: any thoughts?
[01:11:09 AM] rodrigods: i'm completely nuts?
[01:11:11 AM] rodrigods: heheh
[01:17:09 AM] dodo: im currently reading
https://buddycloud.org/wiki/Buddycloud_media_server
[01:17:50 AM] dodo: is that what you used as draft?
[01:18:22 AM] rodrigods: i used this doc:
https://docs.google.com/document/d/14mU8eXNGpzSn4axR36-doOfr-cq726KXwM-xwi8WD54/edit?pli=1
[01:19:00 AM] dodo: i mean your flow makes sense .. and suits that
description .. but i must say, its not very well thought trough
[01:19:25 AM] rodrigods: sure...
[01:19:32 AM] rodrigods: i'm the student here!
[01:19:42 AM] rodrigods: what do you think
[01:19:45 AM] rodrigods: that would be better?
[01:20:12 AM] dodo: sorry ..did mean your flow .. its the wiki page
that is not very useful :P
[01:20:22 AM] dodo: s/did/didnt/
[01:21:00 AM] rodrigods: i always was the beckend guy
[01:21:07 AM] rodrigods: implementing database
[01:21:12 AM] rodrigods: components
[01:21:17 AM] rodrigods: etc
[01:21:30 AM] rodrigods: sorry if sometimes i appear a little confused
[01:21:36 AM] rodrigods: with http stuff
[01:21:45 AM] rodrigods: maybe is better use nodejs for everything
[01:21:54 AM] rodrigods: and i'm not aware about it
[01:23:48 AM] dodo: no its alright .. its just that it doesnt make
sense to me to make first a http request to the mediaserver, that wait
until the media server asks back over xmpp and so on .. but thats what
the wiki pages says
[01:24:50 AM] rodrigods: i think that is because would use the
channels permissions
[01:24:52 AM] rodrigods: based on pubsub
[01:25:13 AM] rodrigods: with a http api for doing the outside requests
[01:28:07 AM] rodrigods: sorry, the http api would be provided for the
media server, so we can have an independent way to build clients
[01:28:26 AM] rodrigods: simon cited the flickr http api
[01:29:41 AM] rodrigods: as example
[02:01:21 AM] dodo: alright .. i think i got something more useful
[02:02:56 AM] dodo: the media server consists out of nginx (serving
public data), filestsystem (storage) and an http server with the xmpp
component running
[02:03:27 AM] dodo: well dont has to be neccessarily a http server
[02:03:42 AM] dodo: but at least the xmpp component has to be there
[02:04:36 AM] dodo: what a bc enabled client has to do is:
[02:04:56 AM] dodo: 1. discover the channels media server
[02:05:56 AM] dodo: 2. negotiage a backend .. i think the simplest
one would be http (rest api)
[02:06:51 AM] dodo: 3. get an access token (appended to the url) or a
cookie from the http backend for private data
[02:07:57 AM] dodo: to get the data the client just has to get the
urls (if its public) and if the channel is private the client has to
append the token to url or to send the cookie to the server to
authenticate
[02:08:21 AM] dodo: sry .. i mean the client has to GET request the url
[02:09:01 AM] dodo: uploading data would be a POST or PUT request
[02:09:30 AM] dodo: what the media server has to do:
[02:10:12 AM] dodo: - serve a xmpp component
[02:10:26 AM] dodo: - discoverable via xmpp
[02:11:11 AM] dodo: - serve different backends like http rest api server
[02:12:36 AM] dodo: i think it would be best if the http is integrated
into the media server since the http part would have to query the xmpp
component for channel permissions
[02:13:27 AM] rodrigods: i understand
[02:13:41 AM] rodrigods: what i was envioning
[02:13:46 AM] rodrigods: was something differente
[02:13:47 AM] dodo: all the metadata would be accessable via xmpp
[02:14:06 AM] rodrigods: where the xmpp component itself would provide
a http api
[02:14:14 AM] rodrigods: envisioning*
[02:15:05 AM] dodo: with media server i meant the xmpp component
[02:16:42 AM] rodrigods: so the guy that is serving the files
[02:16:53 AM] rodrigods: is nginx
[02:17:41 AM] dodo: only for public data .. because i dont know how to
let nginx know the data's permissions
[02:18:08 AM] dodo: nginx or apache .. doesnt matter
[02:18:24 AM] dodo: or anything else that can serve static files
[02:19:48 AM] rodrigods: hmm
[02:19:58 AM] rodrigods: i'll make a draw
[02:20:06 AM] rodrigods: of this approach later =)
[02:20:14 AM] rodrigods: now i'm trying to imagine
[02:20:18 AM] rodrigods: how to handle private files
[02:20:29 AM] rodrigods: private channel files
[02:20:38 AM] dodo: they get served via http too
[02:20:46 AM] dodo: but not by nginx or whateva
[02:21:05 AM] dodo: i see two ways to do the authentication
[02:21:12 AM] dodo: one way is to use tokens
[02:22:07 AM] dodo: a token can be requested via xmpp and will be used
in the url like https://bcmedia.com/file?token
[02:22:20 AM] dodo: the token should expire somehow
[02:22:43 AM] dodo: the other way is to use cookies
[02:23:35 AM] dodo: get an cookie init page url from the media server
via xmpp and open that url .. the mediaserver will send a cookie to
the client
[02:24:06 AM] dodo: now for every data request over http the client
will send this cookie to the server
[02:24:36 AM] dodo: the cookie could contain an expiring token
[02:25:47 AM] dodo: im not sure if this whole approach is a good idea
.. because the security has to be implement for every different
backend in a specific way
[02:26:17 AM] dodo: for example a jingle backend would use the xmpps
allready established authentication
[02:27:49 AM] rodrigods: who is backend in this context? =P
[02:28:22 AM] dodo: oh .. just noticing that the token in url approach
is dumb, because that would descroy the browser capability to cache
files
[02:29:55 AM] dodo: right .. the backend would have the same problem
like security .. every backend would have to implement the right
storage usage on they own.
[02:31:21 AM] dodo: so how to generalise that?
[02:31:39 AM] dodo: or even security?
[02:32:23 AM] rodrigods: the backend is the guy serving the files?
[02:32:46 AM] dodo: yes, serving and receiving
[02:34:53 AM] rodrigods: i haven't understand an earlier step
[02:35:02 AM] rodrigods: that was the translating of the URL
[02:35:07 AM] rodrigods: into a file
[02:36:04 AM] rodrigods: sorry...
[02:36:08 AM] rodrigods: into a file permission
[02:36:23 AM] dodo: the token approach?
[02:36:28 AM] rodrigods: yep
[02:36:35 AM] rodrigods: for public channels
[02:37:23 AM] dodo: tfor public channels you dont need a token ..
because its pulbic, the client dont has to authenticate
[02:38:27 AM] rodrigods: sorry
[02:38:32 AM] rodrigods: i knew that
[02:38:34 AM] rodrigods: and somehow
[02:39:10 AM] rodrigods: have mixed the things
[02:39:36 AM] dodo: no problem :)
[02:41:02 AM] rodrigods: i thought in a keypair approach
[02:41:08 AM] rodrigods: or something like that
[02:41:17 AM] rodrigods: so the backend would know
[02:41:22 AM] dodo: the key would be the url
[02:42:07 AM] dodo: so when uploading the client would request a new
url on the server to post to
[02:42:24 AM] dodo: (request via xmpp)
[02:42:39 AM] dodo: (upload data via backend)
[02:43:28 AM] rodrigods: the url would be composed
[02:43:35 AM] rodrigods: by the channel
[02:43:44 AM] rodrigods: right?
[02:43:48 AM] rodrigods: something like
[02:43:49 AM] dodo: makes sense
[02:43:54 AM] dodo: yes
[02:45:42 AM] rodrigods: (sorry dodo, if is late there... where do you
live, btw?)
[02:45:43 AM] dodo:
https://media.buddycloud.org/ca...@topics.buddycloud.org/cutykitty.mkv
[02:45:52 AM] dodo: im from germany
[02:46:16 AM] dodo: its 2:45 in the morning ^^
[02:46:37 AM] rodrigods: the conversation is being very very productive
[02:46:41 AM] rodrigods: this*
[02:46:55 AM] rodrigods: anytime that you want to go to bed
[02:46:59 AM] rodrigods: just tell me =)
[02:47:10 AM] dodo: im not tired yet ^^
[02:47:34 AM] rodrigods: great
[02:47:40 AM] rodrigods: =)
[02:47:54 AM] rodrigods: hmm thinking now
[02:48:07 AM] rodrigods: when the client
[02:48:13 AM] rodrigods: asks the xmpp component
[02:48:15 AM] rodrigods: for a url
[02:48:19 AM] rodrigods: to get or to post
[02:48:54 AM] rodrigods: if the user doesn't have permission
[02:49:10 AM] rodrigods: the xmpp component just have to respond with an error
[02:49:15 AM] rodrigods: right?
[02:49:17 AM] dodo: normaly it doesnt has to query an url to GET ..
because the links would be in the posts
[02:49:29 AM] dodo: yes
[02:50:04 AM] rodrigods: we can hide the channel url
[02:50:06 AM] rodrigods: with a hash
[02:50:14 AM] rodrigods: that only the xmpp would know
[02:50:28 AM] rodrigods: and, aways the client
[02:50:32 AM] rodrigods: have to ask the url
[02:50:38 AM] rodrigods: if is public
[02:50:39 AM] rodrigods: or not
[02:50:43 AM] rodrigods: the channel
[02:50:56 AM] rodrigods: and it wouldn't be able to 'guess'
[02:50:59 AM] dodo: urls are public .. always :)
[02:51:00 AM] rodrigods: urls from private channels
[02:51:07 AM] rodrigods: the file path
[02:51:09 AM] rodrigods: i mean
[02:51:28 AM] rodrigods: instead of being
[02:51:30 AM] rodrigods:
https://media.buddycloud.org/ca...@topics.buddycloud.org/cutykitty.mkv
[02:51:32 AM] rodrigods: it would be
[02:51:42 AM] rodrigods:
https://media.buddycloud.org/hash(ca...@topics.buddycloud.org)/cutykitty.mkv
[02:51:56 AM] rodrigods: the hash, only the xmpp component would know
[02:52:18 AM] rodrigods: so... the client would never be able to guess
[02:52:20 AM] rodrigods: where a file is
[02:52:38 AM] rodrigods: thats make sense?
[02:53:32 AM] dodo: its like hiding your homes key under your doormat :)
[02:53:59 AM] dodo: i mean there is no way that you can garanty the
privacy of the url
[02:54:43 AM] dodo: so the only reason to do this is to obviouscate
the information in the url
[02:56:23 AM] dodo: i mean
https://media.buddycloud.org/a309ffba7e7b4ca266cb0935e2f1a312e4455631/c1f22dc0a5972b997a8f9196cfc9844ea9bea904
is still better than
https://media.buddycloud.org/hard...@topics.porn.com/under16.mp4
[02:56:51 AM] rodrigods: sure...
[02:58:11 AM] dodo: so obviouscating urls in private channels makes
sense but .. it doesnt matter that the xmpp component is the only one
that knows the url
[02:58:46 AM] dodo: the url should be usable in posts without concerns
[02:59:03 AM] rodrigods: yes
[02:59:16 AM] rodrigods: i haven't thought this through
[02:59:29 AM] rodrigods: its like facebook photos
[02:59:36 AM] rodrigods: once i have the url
[02:59:47 AM] rodrigods: doesn't matter if i'm friends with you
[03:00:01 AM] dodo: no .. not ecactly
[03:00:18 AM] dodo: because you can only get the urls data if you send
the right cookie
[03:00:34 AM] rodrigods: but...
[03:00:38 AM] rodrigods: once i get the url
[03:00:42 AM] rodrigods: i can access the file
[03:01:20 AM] dodo: you have to send your authentication cookie with a
token that you got via xmpp along with that url
[03:01:30 AM] rodrigods: ahh
[03:01:31 AM] rodrigods: ok
[03:01:52 AM] rodrigods: just one question
[03:01:56 AM] rodrigods: not related at al
[03:01:56 AM] rodrigods: all
[03:02:01 AM] rodrigods: with this most recent subject
[03:02:18 AM] rodrigods: the client would speak only http
[03:02:22 AM] rodrigods: right?
[03:02:50 AM] dodo: only when the http api server is ready ..
[03:03:04 AM] dodo: (the one that denis will build)
[03:03:29 AM] dodo: at least that is my idea :P
[03:03:30 AM] rodrigods: just received email from code jam
[03:03:41 AM] rodrigods: but i can publish a http api
[03:03:42 AM] rodrigods: rest api
[03:03:50 AM] rodrigods: can i?
[03:04:12 AM] dodo: you need your own
[03:04:26 AM] rodrigods: if the client
[03:04:31 AM] rodrigods: will speak xmpp
[03:04:37 AM] rodrigods: why do we need
[03:04:40 AM] rodrigods: a xmpp component
[03:04:45 AM] rodrigods: handling the media server?
[03:05:14 AM] rodrigods: ahh
[03:05:18 AM] rodrigods: for the url stuff?
[03:05:28 AM] rodrigods: only for that?
[03:05:35 AM] dodo: for the cookie
[03:06:06 AM] dodo: it can only request a valid cookie if its
authenticated via xmpp
[03:07:00 AM] dodo: oh .. and for serving the metadata the component
is also needed
[03:07:38 AM] rodrigods: sure
[03:07:42 AM] rodrigods: after this conversation
[03:07:48 AM] rodrigods: everything is much much clearer
[03:07:51 AM] rodrigods: in my head =)
[03:07:52 AM] rodrigods: thanks dodo
[03:08:13 AM] dodo: this is fun :)
[03:08:47 AM] rodrigods: code jam has just started!
[03:08:59 AM] rodrigods: i think that i'll only participate from the
furthers rounds
[03:09:16 AM] rodrigods: anyway
[03:09:36 AM] rodrigods: now that i have a bigger understanding
[03:09:44 AM] rodrigods: of the whole architecture
[03:10:00 AM] rodrigods: i will do the file serving stuff
[03:10:03 AM] rodrigods: with node.js
[03:10:06 AM] rodrigods: just a toy
[03:10:12 AM] rodrigods: to see if i learn a little
[03:10:19 AM] dodo: n!ce <3
[03:10:26 AM] dodo: :D
[03:10:35 AM] rodrigods: i can use coffee script
[03:10:41 AM] rodrigods: right?
[03:11:18 AM] dodo: coffeescript is just a fancy way to write
javascript .. it mostly macros a lot boilerplate code away
[03:11:28 AM] dodo: yes :>
[03:11:47 AM] rodrigods: great
[03:12:13 AM] rodrigods: the xmpp component
[03:12:19 AM] rodrigods: i insist in java
[03:12:22 AM] rodrigods: heheh
[03:13:09 AM] dodo: so you want a "real" seperation of backend and
server (xmpp component)?
[03:13:23 AM] rodrigods: yes
[03:13:34 AM] rodrigods: now i don't see why they would be together
[03:13:47 AM] rodrigods: if they don't talk with each other
[03:14:13 AM] dodo: they talk with each other else the http server
would know if a cookie is valid
[03:14:30 AM] dodo: but that can be done with redis or somthing
[03:14:53 AM] rodrigods: yep... i've forgot that
[03:15:01 AM] rodrigods: but... do you agree with this separation?
[03:15:12 AM] dodo: or they simply talk via http, tcp or whatevery
directly with each other
[03:15:20 AM] dodo: yes makes sense
[03:15:23 AM] dodo: at some point
[03:15:52 AM] dodo: i mean a faster way to get things done is to put
them together
[03:16:10 AM] dodo: and then try to implement more backends into it
[03:16:46 AM] dodo: and look at it again and decide what worked out
and what not and what can be shared and what can be seperated
[03:17:19 AM] rodrigods: the tricky thing would talk xmpp
[03:17:23 AM] rodrigods: in my opinion
[03:17:28 AM] rodrigods: or there is a xmpp lib
[03:17:30 AM] rodrigods: for nodejs?
[03:17:37 AM] dodo: node-xmpp by astro :D
[03:17:51 AM] dodo: he is sitting to my right ^^
[03:17:56 AM] rodrigods: ahhhh
[03:17:59 AM] rodrigods: hi astro!
[03:18:00 AM] rodrigods: heheh
[03:18:11 AM] dodo: hi back ^^
[03:18:35 AM] rodrigods: great
[03:18:37 AM] dodo: https://github.com/astro/node-xmpp
[03:18:41 AM] rodrigods: i'll give nodejs a try
[03:18:50 AM] rodrigods: don't promise that i'll love i
[03:18:51 AM] rodrigods: it
[03:18:53 AM] dodo: sweet :3
[03:18:53 AM] rodrigods: but...
[03:19:29 AM] dodo: i can give you a tutorial into nodejs and
coffeescript via skype
[03:20:01 AM] dodo: and you can test it first of course and then still
decide wether to go with it or to use java
[03:20:16 AM] rodrigods: i found here
[03:20:17 AM] rodrigods: a video
[03:20:34 AM] rodrigods: thanks for your ....
[03:20:45 AM] rodrigods: i don't know the word in english
[03:20:53 AM] rodrigods: =)
[03:21:32 AM] dodo: just type it in your native language (is it
spanish or portoguese?) :)
[03:22:00 AM] rodrigods: portuguese
[03:22:41 AM] rodrigods: now i'm not remembering in portuguese
[03:22:43 AM] rodrigods: anyway
[03:22:45 AM] rodrigods: heheh
[03:22:53 AM] dodo: h:D
[03:23:15 AM] rodrigods: ahh
[03:23:21 AM] rodrigods: 'prestatividade'
[03:23:46 AM] rodrigods: the translator says that is "helpfulness"
[03:24:13 AM] dodo: (:
Tuomas Koski
Apr 30, 2012, 3:36:09 PM4/30/12
to buddycl...@googlegroups.com
Cool,
looking good. I'll Read it better in airplane tomorrow night.
Cheers,
--
tuomas
ps. great communication guys. Thanks for sharing this dodo!
Ringo De Smet
Apr 30, 2012, 3:53:11 PM4/30/12
to buddycl...@googlegroups.com
Gents,
On 30 April 2012 18:33, dodo <dodo.t...@googlemail.com> wrote:
here is a my chatlog with rodrigods about the media server architecture design.
please feel free to criticize :)
I have a few remarks regarding the storage aspect of the media server. While I agree to keep a first implementation simple, don't cut back on thinking about an massively scalable design. My main request is not to start your design with the assumption that the media server will be a single process, running on one computer and having a local file system for the storage, whether it be MongoDB+GridFS or something else.
The media server will become a *very* important part of BuddyCloud. Look at the amount of media that is shared daily on Twitter, Facebook, Google+, ... So before going of and start coding, let's make sure that we start designing from the base idea that the media server will be a clustered architecture, fault tolerant, running on distributed storage. Implementing the minimal single machine, single process media server (= shortest path to get something working), will probably end up looking very different within this setup than starting the other way around (= start implement the single machine version, than evolve in a clustered architecture).
My request from the beginning is to use a storage abstraction layer from the start. When using Java, have a look at JClouds (http://jclouds.org). For Ruby, there is the fog library (http://fog.io) and for Python, there exists the libcloud library (http://libcloud.apache.org). For a simple setup, the storage layer can be configured to use the local file system (also for developer testing), but for production deployment, I could as easily configure it to use Amazon S3 (1) and the related CDN feature CloudFront (2).
Supporting GridFS is done by implementing a new storage provider for the chosen cloud library.
Hope this starts a productive discussion. :-)
Ringo
Tuomas Koski
May 3, 2012, 5:35:38 AM5/3/12
to buddycl...@googlegroups.com
Hi All and cheers to Ringo to join the discussion,
On 30 April 2012 21:53, Ringo De Smet <ringo....@gmail.com> wrote:
> My request from the beginning is to use a storage abstraction layer from the
> start. When using Java, have a look at JClouds (http://jclouds.org). For
> Ruby, there is the fog library (http://fog.io) and for Python, there exists
> the libcloud library (http://libcloud.apache.org). For a simple setup, the
> storage layer can be configured to use the local file system (also for
> developer testing), but for production deployment, I could as easily
> configure it to use Amazon S3 (1) and the related CDN feature CloudFront
> (2).
>
> (1) http://aws.amazon.com/s3/
> (2) http://aws.amazon.com/cloudfront/
+1
Cheers,
--
Tuomas
On 30 April 2012 21:53, Ringo De Smet <ringo....@gmail.com> wrote:
> My request from the beginning is to use a storage abstraction layer from the
> start. When using Java, have a look at JClouds (http://jclouds.org). For
> Ruby, there is the fog library (http://fog.io) and for Python, there exists
> the libcloud library (http://libcloud.apache.org). For a simple setup, the
> storage layer can be configured to use the local file system (also for
> developer testing), but for production deployment, I could as easily
> configure it to use Amazon S3 (1) and the related CDN feature CloudFront
> (2).
>
> (1) http://aws.amazon.com/s3/
> (2) http://aws.amazon.com/cloudfront/
Cheers,
--
Tuomas
imaginator
May 3, 2012, 9:43:06 AM5/3/12
to buddycl...@googlegroups.com
media server (and any other project): http://theleanstartup.com/principles
- none of us know how the media server will be used. (We think it will be really useful). But we can eliminate this uncertantity by building v0.1-mostly-works and seeing how people use it.
- get the minimum viable product out as quick as possible and if necessary rewrite based on what we learn. (this could look like file upload, associate to a channel, display quick and dirty). The next version could have auth and the rest.
S.
- none of us know how the media server will be used. (We think it will be really useful). But we can eliminate this uncertantity by building v0.1-mostly-works and seeing how people use it.
- get the minimum viable product out as quick as possible and if necessary rewrite based on what we learn. (this could look like file upload, associate to a channel, display quick and dirty). The next version could have auth and the rest.
S.
dodo
May 3, 2012, 4:20:20 PM5/3/12
to buddycl...@googlegroups.com
On 30 April 2012 21:53, Ringo De Smet <ringo....@gmail.com> wrote:
> Gents,
>
> On 30 April 2012 18:33, dodo <dodo.t...@googlemail.com> wrote:
>>
>> here is a my chatlog with rodrigods about the media server architecture
>> design.
>>
>> please feel free to criticize :)
>
>
> I have a few remarks regarding the storage aspect of the media server. While
> I agree to keep a first implementation simple, don't cut back on thinking
> about an massively scalable design. My main request is not to start your
> design with the assumption that the media server will be a single process,
> running on one computer and having a local file system for the storage,
> whether it be MongoDB+GridFS or something else.
i agree.
>
> On 30 April 2012 18:33, dodo <dodo.t...@googlemail.com> wrote:
>>
>> here is a my chatlog with rodrigods about the media server architecture
>> design.
>>
>> please feel free to criticize :)
>
>
> I have a few remarks regarding the storage aspect of the media server. While
> I agree to keep a first implementation simple, don't cut back on thinking
> about an massively scalable design. My main request is not to start your
> design with the assumption that the media server will be a single process,
> running on one computer and having a local file system for the storage,
> whether it be MongoDB+GridFS or something else.
> The media server will become a *very* important part of BuddyCloud. Look at
> the amount of media that is shared daily on Twitter, Facebook, Google+, ...
buddycloud builds a distributed platform, which means it should be
more easily install-able than scale-able.
> So before going of and start coding, let's make sure that we start designing
> from the base idea that the media server will be a clustered architecture,
> fault tolerant, running on distributed storage. Implementing the minimal
> single machine, single process media server (= shortest path to get
> something working), will probably end up looking very different within this
> setup than starting the other way around (= start implement the single
> machine version, than evolve in a clustered architecture).
i'm writing nodejs for some time now (and am therefor quiet used to
write low level clustered systems) and must admit, the difference is
only in the backend of the server if you're doing it right.
so if you write code for one machine, its not that hard to just change
everything that is not scaling to something different (at least in the
nodejs world, where everything is a small module and all it takes is
to find it and to make the interface work with the existing code).
> My request from the beginning is to use a storage abstraction layer from the
> start. When using Java, have a look at JClouds (http://jclouds.org). For
> Ruby, there is the fog library (http://fog.io) and for Python, there exists
> the libcloud library (http://libcloud.apache.org). For a simple setup, the
> storage layer can be configured to use the local file system (also for
> developer testing), but for production deployment, I could as easily
> configure it to use Amazon S3 (1) and the related CDN feature CloudFront
> (2).
> (1) http://aws.amazon.com/s3/
> (2) http://aws.amazon.com/cloudfront/
>
> Supporting GridFS is done by implementing a new storage provider for the
> chosen cloud library.
well, but would you buy an external storage for your private use (like
sharing naked pictures with your GF/wife)?
we will keep cloud storage in mind, but i'm pretty sure it wont be
part of the first code
-- dodo
dodo
May 3, 2012, 4:23:32 PM5/3/12
to buddycl...@googlegroups.com
> we will keep cloud storage in mind, but i'm pretty sure it wont be
> part of the first code
i just added a trello board to keep track of those ideas :)
> part of the first code
https://trello.com/board/mediaserver/4fa2e89d3056846b6d1ff7ab
-- dodo
Reply all
Reply to author
Forward
0 new messages