move meta to another group

[Рустам Нарманов]

Hello

There is a problem - the metadata is stored in the group, which also has 50 million other keys, all these 50 million keys are not needed and they need to be removed. Brute force removal of keys piece by piece will take a month. Bulk delete hangs the backrunner. And can I just add to config backrunner to a new group, re-create all metadata using utilities bmeta and completely remove the old group, including to remove them from config backrunner ? Will all metadata be recreated, or is there a need for old groups? Through /stat/ you can get information about groups and buckets and create all the necessary json for bmeta, as I understand it, right?

Maybe there is some other way to transfer metadata?

--

Rustam

[Evgeniy Polyakov]

Hi Rustam

18.02.2019, 17:39, "'Рустам Нарманов' via reverbrain" <rever...@googlegroups.com>:

> There is a problem - the metadata is stored in the group, which also has 50 million other keys, all these 50 million keys are not needed and they need to be removed. Brute force removal of keys piece by piece will take a month. Bulk delete hangs the backrunner. And can I just add to config backrunner to a new group, re-create all metadata using utilities bmeta and completely remove the old group, including to remove them from config backrunner ? Will all metadata be recreated, or is there a need for old groups? Through /stat/ you can get information about groups and buckets and create all the necessary json for bmeta, as I understand it, right?
>
> Maybe there is some other way to transfer metadata?

By metadata do you mean bucket metadata?
If so, did you store it in several groups or just in this one you are about to delete?

If you have another copy, you can safely remove one group and metadata will be recovered from another location, you can even just specify several new groups and one (or more) old groups for this,
and metadata will be recovered into these new groups. Although all data will also be recovered there, which is probably not what you want.

And of course you can recreate metadata in any group using bmeta tool, old copies in different groups do not matter.

[Рустам Нарманов]

Hello

Yes, by metadata I mean the metadata of the buckets. There are 6 groups with this metadata and all of them are filled with unnecessary data to be deleted. Well, then I just create new empty groups and point them to the "metadata-groups" section and restart the backrunner ? The metadata will be transferred by backrunner?

Thank you.

вторник, 19 февраля 2019 г., 17:17:24 UTC+3 пользователь Zbr написал:

[Evgeniy Polyakov]

19.02.2019, 17:39, "'Рустам Нарманов' via reverbrain" <rever...@googlegroups.com>:

> Yes, by metadata I mean the metadata of the buckets. There are 6 groups with this metadata and all of them are filled with unnecessary data to be deleted. Well, then I just create new empty groups and point them to the "metadata-groups" section and restart the backrunner ? The metadata will be transferred by backrunner?

Metadata will be transferred by read-recovery mechanism, so it should be enabled (it is by default).

And also backrunner should start reading from 'empty' group to find out that it has to recover metatada,
iirc, it reads groups either in order they are specified in config or in random order,
so it is better to check logs that metadata groups have been read and written.