Daily backup of bigtable

[Smita Vijayakumar]

Hi Everyone,

I'd like to setup a cron to take a daily dump of bigtable and stash the dump into a bucket.

What is the easiest way to achieve this objective?

I have explored the following options -

1. Setting up a cron service which hits a URL that runs this script with parameters. Disadvantage is that shell scripts can be flaky and non-dependable.

2. Setting up a dataflow pipeline to take the dump. The pipeline is constructed similar to ExportJob.java. However, I seem to be hitting compilation failure for an ambiguous call of MapElements.via(new ResultToKV()). The dependencies are -

beam-sdks-java-core - 2.1.0

bigtable-hbase-beam - 1.0.0

beam-runners-google-cloud-dataflow-java - 2.1.0

beam-sdks-java-io-google-cloud-platform - 2.1.0

Smita

[sm...@indihood.com]

Any input on this? Please help, this is rather urgent production data.

[Solomon Duskis]

Can you please elaborate on the "Compilation failure for an ambiguous call of MapElements.via(new ResultToKV())" issues?  It difficult to debug this classpath problem with the information you've provided.

Solomon Duskis |

Google Cloud Bigtable Tech Lead |

sdu...@google.com |

914-462-0531

--
You received this message because you are subscribed to the Google Groups "Google Cloud Bigtable Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-bigtabl...@googlegroups.com.
To post to this group, send email to google-cloud-b...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-bigtable-discuss/682a71e8-4390-4837-a5f7-96fa18f3ea71%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Smita Vijayakumar]

Sorry about that. Here is the exact error -

<snip>

pipeline
.apply("Read table", Read.from(CloudBigtableIO.read(config)))
.apply("Format results", MapElements.via(new ResultToKV()))
.apply("Write", WriteFiles.to(sink));

</snip>

new ResultToKV() fails to compile with the following reason -
"

Ambiguous method call. Both 

via(SimpleFunction<Result,KV<ImmutableBytesWritable, Result> >) in MapElements and

via(SerializableFunction)                                       in MapElements match
"

Smita

On Tue, Feb 13, 2018 at 9:03 PM, Solomon Duskis <sdu...@google.com> wrote:

Can you please elaborate on the "Compilation failure for an ambiguous call of MapElements.via(new ResultToKV())" issues?  It difficult to debug this classpath problem with the information you've provided.

Solomon Duskis |

Google Cloud Bigtable Tech Lead |

sdu...@google.com |

914-462-0531

On Tue, Feb 13, 2018 at 4:41 AM <sm...@indihood.com> wrote:

Any input on this? Please help, this is rather urgent production data.

On Monday, February 12, 2018 at 4:55:52 PM UTC+5:30, Smita Vijayakumar wrote:

Hi Everyone,

I'd like to setup a cron to take a daily dump of bigtable and stash the dump into a bucket.

What is the easiest way to achieve this objective?

I have explored the following options -

1. Setting up a cron service which hits a URL that runs this script with parameters. Disadvantage is that shell scripts can be flaky and non-dependable.

2. Setting up a dataflow pipeline to take the dump. The pipeline is constructed similar to ExportJob.java. However, I seem to be hitting compilation failure for an ambiguous call of MapElements.via(new ResultToKV()). The dependencies are -

beam-sdks-java-core - 2.1.0

bigtable-hbase-beam - 1.0.0

beam-runners-google-cloud-dataflow-java - 2.1.0

beam-sdks-java-io-google-cloud-platform - 2.1.0

Smita

--
You received this message because you are subscribed to the Google Groups "Google Cloud Bigtable Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-bigtable-discuss+unsub...@googlegroups.com.
To post to this group, send email to google-cloud-bigtable-discuss@googlegroups.com.

[Solomon Duskis]

Since this is a compilation problem, you probably need to do an explicit cast to make sure that you use one of the two methods.  Try both?

Solomon Duskis |

Google Cloud Bigtable Tech Lead |

sdu...@google.com |

914-462-0531

To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-bigtabl...@googlegroups.com.
To post to this group, send email to google-cloud-b...@googlegroups.com.