Documentation

42 views
Skip to first unread message

Mohammad Ghasemi

unread,
Apr 14, 2020, 3:05:10 PM4/14/20
to Seaweed File System
Hello. I have some questions.

In the wiki pages, "Memory consumption" section says:

But in "Large File Handling" it says:
What If the system has many reads and writes?

What is the Memory requirement for 1TB storage of 500KB files is needed?

How can I find the full list of master and volume APIs and possible errors? (.e.g. There is no space left, Master can not grow volume..., IO/TimeOut, etc errors)

Thank you.

Chris Lu

unread,
Apr 14, 2020, 6:14:12 PM4/14/20
to Seaweed File System
What If the system has many reads and writes?
The files are in chunks. Each chunk is read into the memory and released.

What is the Memory requirement for 1TB storage of 500KB files is needed?
2 million files? Not much. About 24MB for the in memory index file if all in one volume, and some other memory for a volume server to run.
Very likely this would be split into multiple volume servers.

How can I find the full list of master and volume APIs and possible errors?
There is a weed shell command, where you can see all the details.
There is also web UI to see similar things, and Promethus integration.

--
You received this message because you are subscribed to the Google Groups "Seaweed File System" group.
To unsubscribe from this group and stop receiving emails from it, send an email to seaweedfs+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/seaweedfs/e7ba3319-e55b-4e25-a525-12e41744e984%40googlegroups.com.

Mohammad Ghasemi

unread,
Apr 15, 2020, 4:35:16 AM4/15/20
to Seaweed File System
Thank you for your answer.

I could not find the list of rest APIs and possible responses in the shell and UI. (I did not check Promethus)
I'm writing a python client and need to know for example what are the possible responses (and status codes) in result of calling a master to assign a file id?

As an example, I was calling Mater2 which was not the leader in a loop and I shut down the leader and got a lot of 502 status codes (with no tex response) until the next leader was chosen. But the last 502 response took more than 1 minute while the leader was chosen and Master2 was responding to the new requests.

call number | request time               | status code | response body
...
257 | 2020-04-15T12:52:34.671863 | 200 | {"fid":"6,01023f7083d2","url":"v1:8080","publicUrl":"v1:8080","count":1}
258 | 2020-04-15T12:52:34.677276 | 200 | {"fid":"7,0103ff768f73","url":"v1:8080","publicUrl":"v1:8080","count":1}
259 | 2020-04-15T12:52:34.726519 | 502 |
...
320 | 2020-04-15T12:52:35.029461 | 502 |
321 | 2020-04-15T12:54:46.055645 | 502 |
322 | 2020-04-15T12:54:46.066174 | 200 | {"fid":"5,01814dc5c55b","url":"v2:8081","publicUrl":"v2:8081","count":1}
323 | 2020-04-15T12:54:46.079558 | 200 | {"fid":"7,01829b8e2bef","url":"v1:8080","publicUrl":"v1:8080","count":1}
...

Is there a document about Master and Volume REST APIs?

Thank you.

Chris Lu

unread,
Apr 15, 2020, 1:03:36 PM4/15/20
to Seaweed File System
No documentation on that. The master API to assign file id is here: https://github.com/chrislusf/seaweedfs/blob/5a8c3e4cf93a36bdc35cc3a9c9bfcbbe9857fefe/weed/server/master_server_handlers.go#L96

There are no http.StatusBadGateway thrown in the source code. I am not sure how you get that.

Most internal APIs have moved to protobuf. For example, https://github.com/chrislusf/seaweedfs/blob/master/weed/pb/master.proto

Chris

--
You received this message because you are subscribed to the Google Groups "Seaweed File System" group.
To unsubscribe from this group and stop receiving emails from it, send an email to seaweedfs+...@googlegroups.com.

Mohammad

unread,
Apr 19, 2020, 3:45:12 AM4/19/20
to seaw...@googlegroups.com
The reason for StatusBadGateway might be the master redirection to the leader which is down.
For APIs with no redirection (like /cluster/status), StatusBadGateway won't happen. (although the returned leader is down, until choosing a new one).
I started a volume server (with docker) and tested it for uploading 250MB large files in sequence without any concurrency. Uploading the first file required about twice the size of the file, memory. After uploading 3 or 4 files the memory usage increased to about 1.5 GB and remained constant and while all uploads were finished the memory was not released.
Although it worked fine for small files with reasonable memory usage.

I used docker stats to monitor memory usage, and I'm not sure if I'm missing something.

Thank you.



--
با سپاس
محمد صادق قاسمی
Reply all
Reply to author
Forward
0 new messages