ddfs performance

46 views
Skip to first unread message

Krzysztof Kaczmarski

unread,
Sep 4, 2015, 6:35:25 AM9/4/15
to Disco-development, Paweł Tobiś
Hi,

We run ddfs on 4 machines. We insert about 15000 tags per day and remove the same number of tags from the past.
We need to keep a few days back but then old data can be deleted.
Each tag contains data divided into about 3 chunks of 6MB each (compressed). Everything is replicated 3 times.

We observe issues with ddfs performance after some time. If the storage is empty all operations are very quick. However after a few weeks it
slows down. ddfs ls may take about a minute and getting single data file several seconds.
We try to play with various options (mentioned here: https://groups.google.com/forum/#!searchin/disco-dev/slow/disco-dev/AmwN72XUp4A/m7iJUN4rnjsJ)
but it doesn't help much.

GC also takes a few hours to perform single run.

What can we do to solve data accessing time problem?

Regards,
Krzysztof Kaczmarski
Reply all
Reply to author
Forward
0 new messages