Documentation

Mohammad Ghasemi

unread,

Apr 14, 2020, 3:05:10 PM4/14/20

to Seaweed File System

Hello. I have some questions.

In the wiki pages, "Memory consumption" section says:

For volume servers, the memory consumption is tightly related to the number of files. For example, one 32G volume can easily have 1.5 million files if each file is only 20KB. To store the 1.5 million entries of meta data in memory, currently SeaweedFS consumes 36MB memory, about 24bytes per entry in memory. So if you allocate 64 volumes(2TB), you would need about 3GB memory. However, if the average file size is larger, say 200KB, only about 300MB memory is needed.

But in "Large File Handling" it says:

In order to achieve high concurrency, SeaweedFS tries to read and write the whole file into memory. But this would not work for large files.

What If the system has many reads and writes?

What is the Memory requirement for 1TB storage of 500KB files is needed?

How can I find the full list of master and volume APIs and possible errors? (.e.g. There is no space left, Master can not grow volume..., IO/TimeOut, etc errors)

Thank you.

Chris Lu

unread,

Apr 14, 2020, 6:14:12 PM4/14/20

to Seaweed File System

What If the system has many reads and writes?

The files are in chunks. Each chunk is read into the memory and released.

What is the Memory requirement for 1TB storage of 500KB files is needed?

2 million files? Not much. About 24MB for the in memory index file if all in one volume, and some other memory for a volume server to run.

Very likely this would be split into multiple volume servers.

How can I find the full list of master and volume APIs and possible errors?

There is a weed shell command, where you can see all the details.

There is also web UI to see similar things, and Promethus integration.

--
You received this message because you are subscribed to the Google Groups "Seaweed File System" group.
To unsubscribe from this group and stop receiving emails from it, send an email to seaweedfs+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/seaweedfs/e7ba3319-e55b-4e25-a525-12e41744e984%40googlegroups.com.

Mohammad Ghasemi

unread,

Apr 15, 2020, 4:35:16 AM4/15/20

to Seaweed File System

Thank you for your answer.

I could not find the list of rest APIs and possible responses in the shell and UI. (I did not check Promethus)

I'm writing a python client and need to know for example what are the possible responses (and status codes) in result of calling a master to assign a file id?

As an example, I was calling Mater2 which was not the leader in a loop and I shut down the leader and got a lot of 502 status codes (with no tex response) until the next leader was chosen. But the last 502 response took more than 1 minute while the leader was chosen and Master2 was responding to the new requests.

call number | request time               | status code | response body
...
257         | 2020-04-15T12:52:34.671863 | 200         | {"fid":"6,01023f7083d2","url":"v1:8080","publicUrl":"v1:8080","count":1}
258         | 2020-04-15T12:52:34.677276 | 200         | {"fid":"7,0103ff768f73","url":"v1:8080","publicUrl":"v1:8080","count":1}
259         | 2020-04-15T12:52:34.726519 | 502         |
...
320         | 2020-04-15T12:52:35.029461 | 502         |
321         | 2020-04-15T12:54:46.055645 | 502         |
322         | 2020-04-15T12:54:46.066174 | 200         | {"fid":"5,01814dc5c55b","url":"v2:8081","publicUrl":"v2:8081","count":1}
323         | 2020-04-15T12:54:46.079558 | 200         | {"fid":"7,01829b8e2bef","url":"v1:8080","publicUrl":"v1:8080","count":1}
...

Is there a document about Master and Volume REST APIs?

Thank you.

Chris Lu

unread,

Apr 15, 2020, 1:03:36 PM4/15/20

to Seaweed File System

No documentation on that. The master API to assign file id is here: https://github.com/chrislusf/seaweedfs/blob/5a8c3e4cf93a36bdc35cc3a9c9bfcbbe9857fefe/weed/server/master_server_handlers.go#L96

There are no http.StatusBadGateway thrown in the source code. I am not sure how you get that.

Most internal APIs have moved to protobuf. For example, https://github.com/chrislusf/seaweedfs/blob/master/weed/pb/master.proto

Chris

--

You received this message because you are subscribed to the Google Groups "Seaweed File System" group.
To unsubscribe from this group and stop receiving emails from it, send an email to seaweedfs+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/seaweedfs/ba0837b7-f0dd-470e-9772-67da4f3d9678%40googlegroups.com.

Mohammad

unread,

Apr 19, 2020, 3:45:12 AM4/19/20

to seaw...@googlegroups.com

The reason for StatusBadGateway might be the master redirection to the leader which is down.

For APIs with no redirection (like /cluster/status), StatusBadGateway won't happen. (although the returned leader is down, until choosing a new one).

I started a volume server (with docker) and tested it for uploading 250MB large files in sequence without any concurrency. Uploading the first file required about twice the size of the file, memory. After uploading 3 or 4 files the memory usage increased to about 1.5 GB and remained constant and while all uploads were finished the memory was not released.
Although it worked fine for small files with reasonable memory usage.

I used docker stats to monitor memory usage, and I'm not sure if I'm missing something.

Thank you.

To view this discussion on the web visit https://groups.google.com/d/msgid/seaweedfs/CACJc9-1-CM5A56K4uhQGrphMEQLfOm8GnD7xcvJg9%3Df-u3L6%3Dw%40mail.gmail.com.