Perlbal MogileFS plugin proof of concept
dormando
I wanted to wrap this up a bit better today but I'm probably out of time
and will be AWOL for a week or two. So, here's a demo of a perlbal plugin
that handles MogileFS lookups.
This is *NOT PRODUCTION READY*. It's example code and uses the existing
blocking client. I'm in the process of writing a basic async client for
use in lookups. Upload support will happen later.
The idea is to attempt to make this compatible with S3, or at least give
it a mode to be S3 compat.
The goal is to make MogileFS a lot easier to get along with out of the
box. We've fixed a lot of the memory leaks that caused people to devalue
mogilefs early on, and I think this module should help people get files
serving a lot more quickly. It's similar to what the Nginx module does but
will come packaged with mogilefs and will likely have more features.
For example, this plugin already supports perlbal's path cache, allowing
much faster responses. In general this should be faster than the normal
path of passing the request back to an app server, but adds restrictions
on key naming and things like access control.
Apologies for the crappy work in progress source :P (think I left a STDERR
debug print or two :P) I'd like to hear comments on the approach and any
specific feature requests so I can prioritize what to do.
-Dormando
Jon Skarpeteig
want to use mysql-galera for a multimaster setup for failover, ideally
I want two trackers connected to two different mysql instances. This
is not possible now, due to locks not replicating.
> 9KVisLast ned
Ask Bjørn Hansen
How were you thinking the authentication should be implemented?
- ask
dormando
>
> How were you thinking the authentication should be implemented?
Initially, probably not at all.
My thoughts.. I think I want to either extend get_paths' options or add a
new command. The command would allow requesting more information
(filesize, etc) about the fid, but also allow you to send in metadata
(such as a cookie header). You could then use a tracker plugin to hook
into your authentication system.
Perhaps some caching could be done by informing the cache that a
header/cookie must match as well as the uri. Otherwise it's just too
difficult to access a database/whatever from perlbal asyncronously.
-Dormando
dormando
On Tue, 29 Jun 2010, Jon Skarpeteig wrote:
> Is there a way for the tracker to avoid using get_lock() for mysql? I
> want to use mysql-galera for a multimaster setup for failover, ideally
> I want two trackers connected to two different mysql instances. This
> is not possible now, due to locks not replicating.
That's pretty unrelated for this thread...
No, there were a set of patches to move the functionality to ddlockd, but
that daemon isn't in a shape where it'd be usable for the general public.
traditional mysql HA, aggressive caching, and DC-localized slaves are the
best options right now.