Broke ids for nodes

[Alexey Timofeyev]

Hello,

By mistake was deleted file with ids for all group in replica with a lot of keys. dnet_balancer create new ids, but some keys cannot find in elliptics, after that. I have old routing tables in logs. How it can be possible to generate ids files based on old routes, or set routes into elliptics node?

[Evgeniy Polyakov]

Hi

02.12.2015, 16:03, "Alexey Timofeyev" <timo...@playform.com>:

> By mistake was deleted file with ids for all group in replica with a lot of keys. dnet_balancer create new ids, but some keys cannot find in elliptics, after that. I have old routing tables in logs. How it can be possible to generate ids files based on old routes, or set routes into elliptics node?

Before trying to fix it, check whether you may have ids stored in elliptics:
http://doc.reverbrain.com/elliptics:route#ids_file

If you do not, there are 2 ways to fix this problem:
* run 'merge' recovery, which will move keys among nodes according to new route table
* generate old route table based on ids you have in logs

The former method is rather straightforward. 'merge' recovery runs rather fast.

The latter is trickier and may not work 100%, but most likely should be fine.
Elliptics uses ids file in the following format:
[64-byte-id][64-byte-id]...[64-byte-id]

I.e. just array of 64-byte-long keys. Please note, that these ids are not strings, but binary keys.
You can generate it manually by writing ids as binary data using information from logs.
Logs contain not full 64-byte keys (128-byte-long strings), but only some start bytes.
Generally it should be ok to fill the rest of the key with zeroes, but there is a tiny chance that
some old ids were different at those bytes which you will replace with zeroes. In this case
the only way is to run 'merge' recovery and move data to the new places.

[Alexey Timofeyev]

Hello, 

Yes, we use some functions from dnet_balancer and fix it with

def convert_to_list(key):

    id = []

    while key > 0:

        id = [key % 256] + id

        key /= 256

    if len(id) < 64:

        id = [0] * (64 - len(id)) + id

    return id[:64]

old_data = open(old_files)

for r in old_data.readlines():

    route = r.split(',')

    rstr = route[0].split(':')

    bckns = route[1].split('/')

    bcnd = bckns[1].split(')')

    filename = 'ids.' + str(bcnd)

    with open(filename, 'a+') as f:

        print rstr[1]

        print bcnd

        data = pack('B' * 64, *convert_to_list(int(rstr[1], 16)))

        f.write(data)

Most errors have been go away, so your help was very usefull. Many thanx.

среда, 2 декабря 2015 г., 16:03:16 UTC+3 пользователь Alexey Timofeyev написал: