mogilefs 2.5M files read only

[Wheel]

Hello,

I consider to use mogilefs for some filesytem storage approx 100TB size (~2.5M files). It will be only used for web users to get files from storage (they cant change the files), so its only for read from filesystem storage in realtime. I wondering about mysql nodes. How can be big mysql db for that number of files? the size?  I need to plan what hw servers I will need for mysql server.

Thank you for your answers.

[dave.l...@gmail.com]

Hi Wheel,

We're running a ~1Pb filestore. I don't have the db in front of me at the moment, but I believe it's of the order of 1Gb in size. Hope this helps.

Dave

--
Sent using from a tiny keypad.

---

From: Wheel <jonas...@gmail.com>

Sender: mog...@googlegroups.com

Date: Thu, 16 May 2013 06:29:15 -0700 (PDT)

To: <mog...@googlegroups.com>

ReplyTo: mog...@googlegroups.com

Subject: mogilefs 2.5M files read only

Hello,

I consider to use mogilefs for some filesytem storage approx 100TB size (~2.5M files). It will be only used for web users to get files from storage (they cant change the files), so its only for read from filesystem storage in realtime. I wondering about mysql nodes. How can be big mysql db for that number of files? the size?  I need to plan what hw servers I will need for mysql server.

Thank you for your answers.

--
 
---
You received this message because you are subscribed to the Google Groups "mogile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mogile+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Wheel]

What is amount of files in this 1Pb files storage? mysql db size depends of number of files in your storage.

[Jochen Billen]

It also depends on the composition of your keys. 

Longer keys (varchar) means more storage, shorter keys (integers) means less storage. 

[Wheel]

Could someone say how big mysql db will be on my case (~2.5M files) ?

[Dave Lambley]

We have 80 million files, and the database on disc is 74Gb.  I would thus estimate 2.3Gb for your 2.5 million files, provided your dkeys are not excessively long.  Hope this helps.

Dave

[Jed Danner]

To add a second data point for you, we have about 250 million files and our database is about 125Gb. Most of our key paths are about 60 characters long.

[Robin H. Johnson]

On Fri, May 17, 2013 at 12:08:24PM -0400, Jed Danner wrote:
> To add a second data point for you, we have about 250 million files and our
> database is about 125Gb. Most of our key paths are about 60 characters long.

And a third datapoint, using the Postgres store.
(Including index data).
Average dkey length is 50 characters.
File table is 363MB per 1M rows.
File_on table is 177MB per 1M rows.

Plus if you have a lot of activity, you should include enough space for
the file_to* tables (mine add another 1GB at the moment).

--
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail : rob...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85

[Jochen Billen]

Our MogileFS system has 45 million files and a database of 25 GB. Average dkey length is about 25 chars long.

Jochen