mogilefs statistics, similar projects?

[sanados]

Our current setup:
files total: 57TB in use: 53TB free: 4TB
files count: 130.533.240 fids (in 8 domains, 31 classes (not counted
the default ones))

currently we have:

9 storage server. ( additional 4 in shipment)
7 lvs balanced tracker -> asked by
3 uploader (for accepting new files, using php
extension: http://projects.usrportage.de/index.fcgi/php-mogilefs)
35 www-server doing additional operations on files

4 fscp (as i call em, fileserver cluster proxies) to deliver files to user.
(using perlbal as proxy and a selfwritten c++ programm doing caching
(memcached) and location lookup over local running trackers)

2 dedicated memcached server (á 8GB RAM)

1 DB ( currently single machine as it has a load at 0.5 with serving
avg. of 500queries per sec up to a peak of 2143 queries/sec (mogadm
drain not included))

"mogadm stats" not working after second weeks in production.

Would like to exchange experience with other mogilefs users and discuss
issues with their systems.

main problems:
* massive active db queries (mostly due to our 11 running trackers)
( 590 Queries aktiv!)
* trackers can handle only about 100queries / sec
* mogstored keeps dying unter high load
* mogilefsd shows devices as ? (not writeable) at some states and
needs to be resetted
* drain of devices causes massive load on db, no status infos, ....
(basically every cleanup, statistic has to be calculated by ourselves)

lg
Jürgen

[Clint Byrum]

On Jun 29, 2009, at 3:21 AM, sanados wrote:

              3 uploader (for accepting new files, using php extension: http://projects.usrportage.de/index.fcgi/php-mogilefs)

How stable has this extension been for you?

When I was evaluating options for using mogilefs from PHP, I was worried that this one was too new and untested, and went with Erik Ostermann's MogileFS PHP class. It works fine with some tweaks like pfsockopen for tracker cons, but an extension usually will trump a class library for latency.

[sanados]

Clint Byrum wrote:
>
> On Jun 29, 2009, at 3:21 AM, sanados wrote:
>
>> 3 uploader (for accepting new files, using php
>> extension: http://projects.usrportage.de/index.fcgi/php-mogilefs)

>> <http://projects.usrportage.de/index.fcgi/php-mogilefs%29>

>
> How stable has this extension been for you?
>
> When I was evaluating options for using mogilefs from PHP, I was
> worried that this one was too new and untested, and went with Erik
> Ostermann's MogileFS PHP class. It works fine with some tweaks like
> pfsockopen for tracker cons, but an extension usually will trump a
> class library for latency.

it had some glitches up version 0.7.1 ... but then got a complete rewrite.
up to version 0.7.3 it had some memissues.

We are currently using 0.7.5b3 and it is working fine.
Though we occasionally get some

exception 'MogileFsException' with message 'No devices found to store file'
Stack trace:
#0 lib/File/Cluster.php(165): MogileFs->put('/tmp/sess/48645...', '486459_62084341...', 'original')
#1 lib/File/Cluster.php(240): File_Cluster->saveFile()
#2 application/controllers/UseruploadController.php(272): File_Cluster::savePhoto(Object(SplFileInfo), '486459_62084341', 'user', 'original')
#3 lib/Zend/Controller/Action.php(498): UseruploadController->uploadAction()
#4 lib/Zend/Controller/Dispatcher/Standard.php(294): Zend_Controller_Action->dispatch('uploadAction')
#5 lib/Zend/Controller/Front.php(914): Zend_Controller_Dispatcher_Standard->dispatch(Object(Controller_Request_Http), Object(Zend_Controller_Response_Http))
#6 index.php(346): Zend_Controller_Front->dispatch()
#7 {main}

but i blame the tracker for that ones.

And we use it in production with 30.000 users online.
Would say that the extension could go stable. :)

lg
Jürgen

[Derek Spranger]

we use 0.7.5b3 and haven't spotted any issues yet, works rather well.

[adamgerst]

My question is, with the compiled extension, does it allow to upload
streamed (as in plain text) data? All of my data consists of
serialized json data and I believe that extension required I save it
to a temporary file, which is a huge waste of resources. Also, could
anyone link the Erik Osterman library, I'm interested in checking it
out. His blog where he announced the release of it doesn't seem to
have a link to it any where....

On Jun 29, 8:23 am, sanados <sana...@failure.at> wrote:
> Clint Byrum wrote:
>
> > On Jun 29, 2009, at 3:21 AM, sanados wrote:
>
> >>               3 uploader (for accepting new files, using php
> >> extension:http://projects.usrportage.de/index.fcgi/php-mogilefs)
> >> <http://projects.usrportage.de/index.fcgi/php-mogilefs%29>
>
> > How stable has this extension been for you?
>
> > When I was evaluating options for using mogilefs from PHP, I was
> > worried that this one was too new and untested, and went with Erik
> > Ostermann's MogileFS PHP class. It works fine with some tweaks like
> > pfsockopen for tracker cons, but an extension usually will trump a
> > class library for latency.
>
> it had some glitches up version 0.7.1 ... but then got a complete rewrite.
> up to version 0.7.3 it had some memissues.
>
> We are currently using 0.7.5b3 and it is working fine.
> Though we occasionally get some
>
> exception 'MogileFsException' with message 'No devices found to store file'
> Stack trace:
> #0 lib/File/Cluster.php(165): MogileFs->put('/tmp/sess/48645...', '486459_62084341...', 'original')
> #1 lib/File/Cluster.php(240): File_Cluster->saveFile()
> #2 application/controllers/UseruploadController.php(272): File_Cluster::savePhoto(Object(SplFileInfo), '486459_62084341', 'user', 'original')
> #3 lib/Zend/Controller/Action.php(498): UseruploadController->uploadAction()
> #4 lib/Zend/Controller/Dispatcher/Standard.php(294): Zend_Controller_Action->dispatch('uploadAction')

> #5 lib/Zend/Controller/Front.php(914): Zend_Controller_Dispatcher_Standard->dispatch(Object(Controller_Request_Htt p), Object(Zend_Controller_Response_Http))

[sanados]

i used the extension in first place for bigfiles.
on of the early version kept a local copy which became unusable pretty
fast because of mem issues.

I sent in an patch, for right now it uses streams for sending data,
which makes it able to store files of infinite length.

stream = php_stream_open_wrapper(filename, "rb", USE_PATH
| ENFORCE_SAFE_MODE, NULL)
At first sight this would work for input streams aswell. I have not
tested this kind of usage though.
you would still have to parse your data through a linux fifo stream but
it does sound doable.

Erik Osterman:
http://lists.danga.com/pipermail/mogilefs/attachments/20070630/995c3387/MogileFS.class.bin
(no idea if that is the latest version though)

lg
Jürgen

[sanados]

I just rechecked the source code ... seems my patch never got commited.
It wont work in the current version they still use
ne_set_request_body_buffer(req, file_buffer, file_buffer_len) on a memcopy;

you might want to try the streaming version. (is the patched version
0.7.3 from capoune)

lg