Backend run out of space

[Alexey Timofeyev]

Hello

If on one of backend from group has situation with run out of free space, does elliptic will continue genereate keys on write for this backend even if it has flag disable_write?

[Evgeniy Polyakov]

Hi Alexey

05.05.2015, 17:06, "Alexey Timofeyev" <timo...@playform.com>:

> If on one of backend from group has situation with run out of free space, does elliptic will continue genereate keys on write for this backend even if it has flag disable_write?

Elliptics is a storage, it doesn't generate keys, it stores data according to keys provided by its clients.
Backrunner, for example, is such a client - some other user writes data into backrunner proxy, which
either selects bucket (and replica groups) itself, or uses what user has provided: /upload/bucket_name/ vs /nobucket_upload/ urls.

If you use /upload/bucket_name/ url, then backrunner is obligated to write data into groups specific for given 'bucket_name',
even if there is no space in one or more groups.

If you use /nobucket_upload/ backrunner will select bucket (and thus set of associated groups) according
to its metrics. There is a fair number of things it takes into account when selecting particular bucket (space, speed, errors, data stats),
and it is possible that it will select a bucket where one of the groups has no space.
This is not the best solution, and there will be an update soon which tunes this.

You haven't provided detailed info about your write case, so it is rather hard to guess your problem.

[Alexey Timofeyev]

Thanks Evgeniy for you detailed answer.

We have setup when one group consist many backends (20), one backend - one disk. Now we have situation when some disk is full so when key need to write to backen with such disk we have error -28. Disabling write to this backend with 

dnet_ioclient -r 10.10.10.10:1021:2 -b 14 -B disable_write

just change error code to -30 and client try to place key on this backend again.

Does elliptic have some mechanism inside group to avoid write to full backends? I check all flags for backend but I don't find which can help.

We use for now groups with no buckets and upload/get data with rift. As I understand rift does't know about size situation on backend and will try to place keys on full backend?

вторник, 5 мая 2015 г., 17:47:46 UTC+3 пользователь Evgeniy Polyakov написал:

[Evgeniy Polyakov]

05.05.2015, 18:21, "Alexey Timofeyev" <timo...@playform.com>:

> We have setup when one group consist many backends (20), one backend - one disk. Now we have situation when some disk is full so when key need to write to backen with such disk we have error -28. Disabling write to this backend with
>
> dnet_ioclient -r 10.10.10.10:1021:2 -b 14 -B disable_write
>
> just change error code to -30 and client try to place key on this backend again.
>
> Does elliptic have some mechanism inside group to avoid write to full backends? I check all flags for backend but I don't find which can help.

If you tell your application (elliptics client) to write into group X, elliptics client will try to write into that group.
Client may not know in advance about its space situation - server could run defragmentation and suddenly
new space became available in the group which was full just a few moments ago.

It is a task of the client to monitor this and select appropriate groups for write operations.
In particular in our stack this task is performed by backrunner proxy - it monitors all nodes, all groups,
and selects a bucket (or basically set of groups) for write operation according to its realtime stats.

> We use for now groups with no buckets and upload/get data with rift. As I understand rift does't know about size situation on backend and will try to place keys on full backend?

Rift is a simple http proxy which will write data where it was told to,
it doesn't have any sophisticated logic to select groups by itself.

[Alexey Timofeyev]

Ok, I understand, my fault when I design setup. As you write before https://groups.google.com/forum/#!topic/reverbrain/bu68k4MO5g0 backrunner can't handle groups with no buckets. 

So I can't change rift to backrunner for this setup. 

Before backrunner when you use rift, how are you handle this situations? Using groups with one disk? 

Why backens in group fill so irregularity?


вторник, 5 мая 2015 г., 19:15:00 UTC+3 пользователь Evgeniy Polyakov написал:

[Evgeniy Polyakov]

05.05.2015, 19:42, "Alexey Timofeyev" <timo...@playform.com>:

> Ok, I understand, my fault when I design setup. As you write before https://groups.google.com/forum/#!topic/reverbrain/bu68k4MO5g0 backrunner can't handle groups with no buckets.
> So I can't change rift to backrunner for this setup.

If you do not use backrunner, but use different group per disk (and thus new servers add new groups) - you are reinventing
backrunner and its buckets. Bucket is a logical entity holding set of groups and a bit of metadata, that's it.

> Before backrunner when you use rift, how are you handle this situations? Using groups with one disk?
>
> Why backens in group fill so irregularity?

If you have groups with the same size they are naturally filled with the same rate.
It is a bug in your client logic if it writes data into group 1, but not into group 2, when it is supposed
to write data into both replicas. Or one group was offline (alternatively, client was not connected to that group)
for a long enough period of time and you didn't run DC recovery when appropriate backend went back online.

[Alexey Timofeyev]

вторник, 5 мая 2015 г., 21:06:23 UTC+3 пользователь Evgeniy Polyakov написал:

If you have groups with the same size they are naturally filled with the same rate.
It is a bug in your client logic if it writes data into group 1, but not into group 2, when it is supposed
to write data into both replicas. Or one group was offline (alternatively, client was not connected to that group)
for a long enough period of time and you didn't run DC recovery when appropriate backend went back online.

We use groups with same size, but backends into groups filled very irregularity. For example two backends in one group, 
one have - 40710532 records
another - 24525939 records

So we have situation when some backends are full and others have only half size in use. Is a result have errors when key cannot write into group if it come to full backend.

Statistically, for such numbers of records, keys distribution must be more regular.

>It is a task of the client to monitor this and select appropriate groups for write operations. 
>In particular in our stack this task is performed by backrunner proxy - it monitors all nodes, all groups, 
>and selects a bucket (or basically set of groups) for write operation according to its realtime stats. 

If I understand docs, backrunner can check free space only in buckets when using nobuckets_upload uri and cannot manage backends into groups. 

http://doc.reverbrain.com/backrunner:uri?s[]=backrunner

[Евгений Поляков]

Hi Alexey

20.05.2015, 18:08, "Alexey Timofeyev" <timo...@playform.com>:

> We use groups with same size, but backends into groups filled very irregularity. For example two backends in one group,
> one have - 40710532 records
> another - 24525939 records

2 backends in one group is quite different from what we discussed.
Replica *groups* are naturally filled with the same rate, but it does not mean backends will be filled with the same rate.

Backend filling rate depends on the ranges it takes from the group's DHT ring.
Probably your backends are quite uneven in that regard.

dnet_balancer tool will show it and fix if needed. It works for one group.
I recommend just copying ids files from that 'fixed' group to other groups and run 'merge' recovery everywhere.

Here is detailed description of the tool: http://doc.reverbrain.com/elliptics:tools
Also I recommend checking out route table docs: http://doc.reverbrain.com/elliptics:route

> So we have situation when some backends are full and others have only half size in use. Is a result have errors when key cannot write into group if it come to full backend.
> Statistically, for such numbers of records, keys distribution must be more regular.
>
>>It is a task of the client to monitor this and select appropriate groups for write operations.
>>In particular in our stack this task is performed by backrunner proxy - it monitors all nodes, all groups,
>>and selects a bucket (or basically set of groups) for write operation according to its realtime stats.
>
> If I understand docs, backrunner can check free space only in buckets when using nobuckets_upload uri and cannot manage backends into groups.

I do not understand, what does it mean "cannot manage backends into groups"?