Is 32GB per volume is an upper limit or lower limit?

Mahafuzur Rahman

unread,

Apr 18, 2014, 12:56:04 PM4/18/14

to weed-fil...@googlegroups.com

"Each data volume is size 32GB", what does this exactly mean? If I have a host server with 20GB disk space, will it be capable of running a volume server in it? What if I have a 40GB server, will the rest 8GB left unused? Also can I configure the data volume size to fit my needs? I'm new to weed-fs and is excited about it to use it in a production cluster having several nodes.

Btw, thanks a lot for the efforts to write this exciting file system.

Chris Lu

unread,

Apr 18, 2014, 1:48:01 PM4/18/14

to weed-fil...@googlegroups.com

Each "volume server" can have multiple "volume data files". Each "volume data file" currently has this 32GB limit.

For small servers as you suggested, I would recommend set the volumeSizeLimt to 10GB, and run 2 volumes on 20GB disk, run 4 volumes on 40GB disk. (of course, maybe leave some space for other purposes)

I think weed-fs currently is robust enough for many use cases. Let me know or file a bug if you see anything abnormal.

Chris

On Fri, Apr 18, 2014 at 9:56 AM, Mahafuzur Rahman <masu...@gmail.com> wrote:

"Each data volume is size 32GB", what does this exactly mean? If I have a host server with 20GB disk space, will it be capable of running a volume server in it? What if I have a 40GB server, will the rest 8GB left unused? Also can I configure the data volume size to fit my needs? I'm new to weed-fs and is excited about it to use it in a production cluster having several nodes.

Btw, thanks a lot for the efforts to write this exciting file system.

--
You received this message because you are subscribed to the Google Groups "Weed File System" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weed-file-syst...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

yss s

unread,

Nov 2, 2015, 8:54:54 AM11/2/15

to Seaweed File System, weed-fil...@googlegroups.com

I this limit still valid ??

Chris Lu

unread,

Nov 2, 2015, 11:02:27 AM11/2/15

to seaw...@googlegroups.com, weed-fil...@googlegroups.com

Still valid.

You received this message because you are subscribed to the Google Groups "Seaweed File System" group.
To unsubscribe from this group and stop receiving emails from it, send an email to seaweedfs+...@googlegroups.com.

yss s

unread,

Nov 2, 2015, 1:12:51 PM11/2/15

to Seaweed File System, weed-fil...@googlegroups.com

Which means if i have around 1TB of data, need to create more than 30 Volume files ??

Or can you pls help me understand how this shall work, i mean number of volume servers , no of master servers, replication strategy ??

Still valid.

To unsubscribe from this group and stop receiving emails from it, send an email to seaweedfs+unsubscribe@googlegroups.com.

Chris Lu

unread,

Nov 2, 2015, 3:12:51 PM11/2/15

to seaw...@googlegroups.com

Yes. 1TB would need 30 volume files.

Not sure about your requirements.

Chris

Still valid.

To unsubscribe from this group and stop receiving emails from it, send an email to seaweedfs+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Seaweed File System" group.

To unsubscribe from this group and stop receiving emails from it, send an email to seaweedfs+...@googlegroups.com.

yss s

unread,

Nov 2, 2015, 3:27:15 PM11/2/15

to Seaweed File System

Thanks for the answer.

Am developing an image cache, which shall host more that 10 million images. Estimated size is around 1 ~1.5TB
Need to serve these images to 2000 users, each user making 12 parallel requests.

So have a spring boot application that would redirect the request to seaweed based on our internal key and seaweed key mapping.

So need the optimal configuration of seaweed for the above objective.
Have tested it with 1 million files, without replication.

Thanks,
Yugant Shah.

Still valid.

To unsubscribe from this group and stop receiving emails from it, send an email to seaweedfs+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Chris Lu

unread,

Nov 2, 2015, 4:02:30 PM11/2/15

to seaw...@googlegroups.com

It seems you just need one volume server of 2TB disk with about 500MB memory reserved for SeaweedFS.

I would say just starts small and scales up when you have problem. If no problem, just let it run and spend time on more important stuff.

It should be quite easy to scale up for SeaweedFS: just add more volume servers.

Chris

Still valid.

To unsubscribe from this group and stop receiving emails from it, send an email to seaweedfs+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Seaweed File System" group.
To unsubscribe from this group and stop receiving emails from it, send an email to seaweedfs+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

yss s

unread,

Nov 2, 2015, 4:12:29 PM11/2/15

to Seaweed File System

Appreciate your quick response.

One more thing to confirm, shall only one volume server be able to handle 2000 users(considering 12 parallel requests for each user)

When you mentioned 500Mb for seaweed, does it mean for seaweed metadata ?

Thanks,
Yugant Shah

Still valid.

To unsubscribe from this group and stop receiving emails from it, send an email to seaweedfs+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Seaweed File System" group.
To unsubscribe from this group and stop receiving emails from it, send an email to seaweedfs+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Chris Lu

unread,

Nov 2, 2015, 6:00:09 PM11/2/15

to seaw...@googlegroups.com

Better test with real usage pattern.

500MB memory should be enough for the metadata for 10M files.

Chris

Still valid.

To unsubscribe from this group and stop receiving emails from it, send an email to seaweedfs+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Seaweed File System" group.
To unsubscribe from this group and stop receiving emails from it, send an email to seaweedfs+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

yss s

unread,

Nov 16, 2015, 10:49:40 AM11/16/15

to Seaweed File System

On test env, i have started a volume server with max value as 30 and started pushing data to seaweed, however i see that only 7 volumes are created.
When will seaweed create more volumes?
Is there any other config that needs to be looked into ?

Still valid.

To unsubscribe from this group and stop receiving emails from it, send an email to seaweedfs+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Seaweed File System" group.
To unsubscribe from this group and stop receiving emails from it, send an email to seaweedfs+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

yss s

unread,

Nov 16, 2015, 10:54:56 AM11/16/15

to Seaweed File System

Also is there a way to figure out if a volume file is approaching the max limit of 32GB ??

Chris Lu

unread,

Nov 16, 2015, 4:26:13 PM11/16/15

to seaw...@googlegroups.com

To increase the number of volumes for higher concurrency:

https://github.com/chrislusf/seaweedfs/wiki/API#pre-allocate-volumes

Still valid.

To unsubscribe from this group and stop receiving emails from it, send an email to seaweedfs+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Seaweed File System" group.
To unsubscribe from this group and stop receiving emails from it, send an email to seaweedfs+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Chris Lu

unread,

Nov 16, 2015, 4:26:52 PM11/16/15

to seaw...@googlegroups.com

Check volume status:

https://github.com/chrislusf/seaweedfs/wiki/API#check-volume-server-status

Still valid.

To unsubscribe from this group and stop receiving emails from it, send an email to seaweedfs+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Seaweed File System" group.
To unsubscribe from this group and stop receiving emails from it, send an email to seaweedfs+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply all

Reply to author

Forward