How to make schema change on source reflected in the destination ?

[adityak...@exadatum.com]

Hi, 

The batch job does a good job in syncing up the data between the source and the destination. However if there is a change of schema at the source, re-air doesn't takes up the job of appropriately updating the schema at the destination to keep the destination table a replica of the source. Is there a way to fine tune things to take care of this ?

Reply asap, on the strategies which can be adopted to sync the schema changes between the source and the destination as well. 

[Paul Yang]

If you re-run the batch job, it should sync the metadata between the source and destination. Are you not seeing this behavior?

--
You received this message because you are subscribed to the Google Groups "reair" group.
To unsubscribe from this group and stop receiving emails from it, send an email to airbnb-reair+unsubscribe@googlegroups.com.
To post to this group, send email to airbnb...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/1a2f0f18-ae59-438f-b65e-3b6b3a7ead6a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[adityak...@exadatum.com]

Hi Paul, Thanks for reply. It sync meta data but still I need to have an utility to be able to take care of addition of columns or change of data type of present columns being reflected at the destination. Btw for achieving that I have already written my own utilities. Also can you share some documentation which makes it more easy to understand every class being used in batch or incremental job. Being a very recent University fresh out, the project structure if not the code looks daunting. Is there some documentation to make life easier ? Thanks in advance for the help. 

[Paul Yang]

Can you elaborate on what you mean by "It sync meta data but still I need to have an utility to be able to take care of addition of columns or change of data type of present columns being reflected at the destination." ?

When the batch job runs and syncs the metadata, it should already replicate column additions and data type changes to the destination. 

Unfortunately, there's not much additional documentation available. My recommendation would be start with the classes that kick off the job and trace up from there.

To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/1815ce38-0b46-46a0-83ca-2a2889b38add%40googlegroups.com.

[Zheng Shao]

Hi Aditya,

Paul is right.  Starting from the main class is the way to go.

For the batch job, here is the main class and documentation: https://github.com/airbnb/reair/blob/master/main/src/main/java/com/airbnb/reair/batch/hive/MetastoreReplicationJob.java#L53

To view this discussion on the web visit https://groups.google.com/d/msgid/airbnb-reair/CALm6gcfzDdoj6K0NxqR9m9cpXZ6DgRmdmdx39cwOn055GbZOgw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Zheng