Large files in mogilefs
runnerup
reading through google and the mailing list for tips before I start.
A few questions I have is about large files in mogilefs.
1.
It seems large files are broken into 64 MB chunks when inserting into
mogilefs, if I am not mistaken.
If they are chunked like that, are they distributed through multiple
servers in 64 MB chunks or are they on one server + replication
count-1.
2.
Another question is when using client APIs(Perl and PHP) to put the
file in mogilefs, do I have to use special options for the file to be
chunked or is it usually automatic, and would there be any temporary
memory issues (i.e. putting a 2 GByte file in mogilefs taking 2 GBytes
of memory).
3.
Also I use nginx to reverse proxy to apache and directly serve static
files.
I wish to use nginx to server the files via a "X-Accel-Redirect" header
(http://wiki.nginx.org/NginxXSendfile),
would this work with the Nginx mogilefs module, http://www.grid.net.ru/nginx/mogilefs.en.html
And I need support of http ranged requests, anyone knows if this would
work with nginx + X-Accel-Redirect + mogilefs module?
Lastly a more unrelated question
4.
Is there anyway/setup to directly/indirectly ftp to mogilefs and
upload/download fiiles?
Thanks.
I would appreciate an answer even partially to any of the above
questions.
Jonathan Steinert
> 1.
> It seems large files are broken into 64 MB chunks when inserting into
> mogilefs, if I am not mistaken.
This is incorrect, I think you're looking at 'mogtool'... which isn't part of the mogilefs api... just a command line tool included with it that can do some stuff.
> If they are chunked like that, are they distributed through multiple
> servers in 64 MB chunks or are they on one server + replication
> count-1.
>
> 2.
> Another question is when using client APIs(Perl and PHP) to put the
> file in mogilefs, do I have to use special options for the file to be
> chunked or is it usually automatic, and would there be any temporary
> memory issues (i.e. putting a 2 GByte file in mogilefs taking 2 GBytes
> of memory).
There are no options in the perl client, and I'm pretty sure I can say in any client as well, that allow you to automatically chunk the upload into pieces. Mogilefs is just designed to have an arbitrary key string (maximum length 255 imposed) be associated with a arbitrary file kept on one or more logical disks. To support chunking you need to have another layer to handle mapping a single key to multiple chunks in a specific order.
As for a memory issue. The perl client has multiple ways of pushing data into a file. Check out the 'largefile' option in the perl client's 'new_file' method.
>
> 3.
> Also I use nginx to reverse proxy to apache and directly serve static
> files.
> I wish to use nginx to server the files via a "X-Accel-Redirect" header
> (http://wiki.nginx.org/NginxXSendfile),
> would this work with the Nginx mogilefs module, http://www.grid.net.ru/nginx/mogilefs.en.html
> And I need support of http ranged requests, anyone knows if this would
> work with nginx + X-Accel-Redirect + mogilefs module?
Reading the documentation on the NginxXSendfile feature, that looks like it only handles files stored on the nginx server... you're looking for something more like perlbal's X-REPROXY-URL feature... which is part of the defacto mogilefs installation.
I'm not an expert on nginx though, so this could be wrong.
>
> Lastly a more unrelated question
> 4.
> Is there anyway/setup to directly/indirectly ftp to mogilefs and
> upload/download fiiles?
There's a FUSE driver, and a DAV driver (which is kinda barebones) that can be used to mount mogilefs and then put a normal ftp server on top of it.
I wouldn't use this for anything more than an experiment though... for larger things I would consider writing an FTP server to handle things, but I don't have any recommendations on ways to do that.
>
> Thanks.
> I would appreciate an answer even partially to any of the above
> questions.
------------------------------------------------------------------------
"She smiled again, shrugged her shoulders, and became a perfect mirror."
------------------------------------------------------------------------
Justin Azoff
> > Another question is when using client APIs(Perl and PHP) to put the
> > file in mogilefs, do I have to use special options for the file to be
> > chunked or is it usually automatic, and would there be any temporary
> > memory issues (i.e. putting a 2 GByte file in mogilefs taking 2 GBytes
> > of memory).
>
> There are no options in the perl client, and I'm pretty sure I can say in any
> client as well, that allow you to automatically chunk the upload into pieces.
> Mogilefs is just designed to have an arbitrary key string (maximum length 255
> imposed) be associated with a arbitrary file kept on one or more logical
> disks. To support chunking you need to have another layer to handle mapping a
> single key to multiple chunks in a specific order.
This is implemented in the python client. I used the same method as mogtool,
so chunked files uploaded with mogtool would be readable. I have no idea if it
still works, or if anyone is even using the python client anymore..
Back then mogilefs had some issues with storing huge files, but I think things
are better now. lighttpd/nginx as a backends and range support make this less
of a problem.
--
-- Justin Azoff
-- Network Performance Analyst
Tomas Doran
On 15 Nov 2009, at 15:17, runnerup wrote:
> 3.
> Also I use nginx to reverse proxy to apache and directly serve static
> files.
> I wish to use nginx to server the files via a "X-Accel-Redirect"
> header
> (http://wiki.nginx.org/NginxXSendfile),
> would this work with the Nginx mogilefs module, http://www.grid.net.ru/nginx/mogilefs.en.html
Yes, this will work.
> And I need support of http ranged requests, anyone knows if this would
> work with nginx + X-Accel-Redirect + mogilefs module?
This however, I have no idea I'm afraid.
Cheers
t0m
rajan chandi
Regards
Rajan