BeeGFS Metadata Calculation Based On BeeGFS Storage Pool Size (v6)
James Montantes
Hello BeeGFS Users,
Exxact recently published a blog article you may find helpful for BeeGFS Metadata calculations. This is based on Version 6.0. See the following excerpt.
BeeGFS Metadata Calculation Based On BeeGFS Storage Pool Size
If you have a large file store that may be the same complexion as the planned BeeGFS storage pool, you can calculate the average filesize of the files in that storage area and use that to reverse calculate how many files would fit in a specific storage pool and therefore how much metadata that would take, if it were fully utilized. There are some awk/gawk one liners that can be used to get this, but the best way is most likely to use a command to get the total amount of files:
- find /directory/ -type f | wc -l
Then another command to get the total size of the directory and sub-directories
- du -h /directory
Then divide them:
- directory-size / file-count
For example:
- 27000MB / 990983 = 0.02724 (27.2KB)
If you have a 27KB average file size, and you are planning on a roughly 400GB BeeGFS Storage Cluster, and want to have a rough idea of how much metadata will be needed if the BeeGFS cluster was fully utilized, reference below.
- Storage-In-KB / Average-File-Size = Num-Files-If-Filled
- 429496729600 / 27 = 15907286281
Since we know with BeeGFS (v6) – these files take ~4KB of Metadata per file.
- 15907286281 * 4Kb = 63629145126 (63.63Tb)
Hope you've found this information useful, if you have any questions or comments, please let us know.
James Montantes
Exxact Corporation
https://www.exxactcorp.com/BeeGFS-Parallel-Storage