Load backup to a table with a S3 storage

[kilic.a...@gmail.com]

Hi,

I was a small regarding the following use case:

- Let's say we have 2 CH cluster: source with a disk storage and destination with a S3 storage.

- We backup the data by parts from the source cluster

- We would like to load them in the destination but having a different storage.

I have 2 questions regarding that:

1. Is it possible ?

2. If yes, what is the best way to handle it?

Kind regards,

Ali

[Alexey Milovidov]

Yes, restore of the backup will work regardless of the storage configuration.
You can backup a cluster with local fs and restore on a cluster with
s3 and vice versa.

PS. This is about the native BACKUP/RESTORE commands,
(there is also a third-party clickhouse-backup tool).

> --
> You received this message because you are subscribed to the Google Groups "ClickHouse" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to clickhouse+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/clickhouse/db211c66-1702-49b9-af6f-74fd4577bb38n%40googlegroups.com.

[Kilic Ali-Firat]

Hi Alexey,

Thank you for your reply but I give it a try and well it's not really working. 

What I've did:

- backup a table to S3 having his data stored on local disk from a shard1

- try to restore from S3 the backup to a table having his storage in S3 on the shard2

But I get this error: "DB::Exception: The table has a different definition" and indeed, if you look at the data stored in S3, we can see a folder called metadata that contains the SQL schema of the table that we just backup.

Maybe I've missed something?

--

Alifirat KILIC

ContentSquare | Data Engineer

+ 33 6 43 13 73 22 

ContentSquare 

5 Boulevard de la Madeleine, 75001 Paris

https://www.contentsquare.com/

[Alexey Milovidov]

You can prepare the table with the same definition, including the name
of the storage_policy in the settings.
So, the table definitions will look the same, but the storage policy
can be configured differently in the server configuration.

[Kilic Ali-Firat]

Hey Alexey,

I was to successfully test the RESTORE / BACKUP commands.

In case of a Clickhouse cluster with multiple shards, what is the best practice for the backups ? Having a backup by shard?

It does not seem to scale so much to me if the cluster size is growing with the time but if you have any tips, I will appreciate it.

[Alexey Milovidov]

Added Vitaly - he can consult about the available options for cluster backups.