Setting up Reaper for multi DC Cassandra cluster

389 views
Skip to first unread message

aiman parvaiz

unread,
Nov 2, 2017, 3:40:45 PM11/2/17
to TLP Apache Cassandra Reaper users
Hi all
I seek advice in setting up Reaper 0.7 for a Cassandra 2.2.8 cluster spanning over 2 DCs. I am aware that with 0.7 we can run Reaper across multiple DCs. In my understanding, I can install and run reaper on any one node in each DC (ending up with one reaper instance per DC) and using production C* as backend.
Am I correct in my understanding? Would this be the best setup for managing repairs for a multi DC set up?

I would appreciate any advice.

Thanks

Alexander Dejanovski

unread,
Nov 2, 2017, 5:29:12 PM11/2/17
to aiman parvaiz, TLP Apache Cassandra Reaper users
Hi Aiman,

yes, you can run multiple Reapers, one or more in each DC in 0.7 when using the Cassandra backend and setting datacenterAvailability to LOCAL. Setting it to EACH won't work very well in that version of Reaper.
Both Reaper instances will sync up using the backend and share the work.

The upcoming release (in the coming days) will make EACH a valid (and safer) choice for multi DC clusters.
I'm writing a doc page on cassandra-reaper.io that will explain how to set up Reaper with multi DC, which should be merged in tomorrow.

Cheers,

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/ca7a3d25-6930-4af4-b471-4b4dcbaa7134%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

aiman parvaiz

unread,
Nov 2, 2017, 5:41:31 PM11/2/17
to TLP Apache Cassandra Reaper users
Thanks Alexander for your response. Yeah, I couldn't find any documentation around multi DC set up, looking forward to it. 


On Thursday, November 2, 2017 at 2:29:12 PM UTC-7, Alexander Dejanovski wrote:
Hi Aiman,

yes, you can run multiple Reapers, one or more in each DC in 0.7 when using the Cassandra backend and setting datacenterAvailability to LOCAL. Setting it to EACH won't work very well in that version of Reaper.
Both Reaper instances will sync up using the backend and share the work.

The upcoming release (in the coming days) will make EACH a valid (and safer) choice for multi DC clusters.
I'm writing a doc page on cassandra-reaper.io that will explain how to set up Reaper with multi DC, which should be merged in tomorrow.

Cheers,

On Thu, Nov 2, 2017 at 8:40 PM aiman parvaiz <aimanp...@gmail.com> wrote:
Hi all
I seek advice in setting up Reaper 0.7 for a Cassandra 2.2.8 cluster spanning over 2 DCs. I am aware that with 0.7 we can run Reaper across multiple DCs. In my understanding, I can install and run reaper on any one node in each DC (ending up with one reaper instance per DC) and using production C* as backend.
Am I correct in my understanding? Would this be the best setup for managing repairs for a multi DC set up?

I would appreciate any advice.

Thanks

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.

Alexander Dejanovski

unread,
Nov 3, 2017, 3:05:23 AM11/3/17
to aiman parvaiz, TLP Apache Cassandra Reaper users
Hi Aiman,

here's the doc I was referring to : http://cassandra-reaper.io/docs/usage/multi_dc/

With 0.7.x, you should be using LOCAL instead of EACH for the third case pictured there.

Cheers,

On Thu, Nov 2, 2017 at 10:41 PM aiman parvaiz <aimanp...@gmail.com> wrote:
Thanks Alexander for your response. Yeah, I couldn't find any documentation around multi DC set up, looking forward to it. 


On Thursday, November 2, 2017 at 2:29:12 PM UTC-7, Alexander Dejanovski wrote:
Hi Aiman,

yes, you can run multiple Reapers, one or more in each DC in 0.7 when using the Cassandra backend and setting datacenterAvailability to LOCAL. Setting it to EACH won't work very well in that version of Reaper.
Both Reaper instances will sync up using the backend and share the work.

The upcoming release (in the coming days) will make EACH a valid (and safer) choice for multi DC clusters.
I'm writing a doc page on cassandra-reaper.io that will explain how to set up Reaper with multi DC, which should be merged in tomorrow.

Cheers,

On Thu, Nov 2, 2017 at 8:40 PM aiman parvaiz <aimanp...@gmail.com> wrote:
Hi all
I seek advice in setting up Reaper 0.7 for a Cassandra 2.2.8 cluster spanning over 2 DCs. I am aware that with 0.7 we can run Reaper across multiple DCs. In my understanding, I can install and run reaper on any one node in each DC (ending up with one reaper instance per DC) and using production C* as backend.
Am I correct in my understanding? Would this be the best setup for managing repairs for a multi DC set up?

I would appreciate any advice.

Thanks

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/72ff0e62-884e-4426-adbc-159c32e4c274%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

aiman parvaiz

unread,
Nov 6, 2017, 12:02:54 PM11/6/17
to TLP Apache Cassandra Reaper users
Hi Alexander
Thanks for the documentation link. I have setup reaper and wanted to run the setup by you, also had a question around schedules.

I have 3 DCs; A, B, C. In each DC, I am running reaper on 2 nodes using production C* as the backend where I have created reaper_db with NetworkTopologyStrategy spread over all 3 DCs. I have specified datacenterAvailability: EACH and contactPoints to DC specific seed nodes.

I am able to see the webui on all of these servers which are running reaper and I was able to add the cluster and setup a repair schedule. I can see the repair running under repair section. Is this setup correct for repairing keyspaces spread over multi dc?

My question around schedule is the Interval in days* field, does this mean that a new repair would start after the specified interval in days(irrespective if the existing repair run is complete or not) or does it mean that once the running repair complete the next repair would start after this specified interval?

Thanks a lot for all your help and support.


On Friday, November 3, 2017 at 12:05:23 AM UTC-7, Alexander Dejanovski wrote:
Hi Aiman,

here's the doc I was referring to : http://cassandra-reaper.io/docs/usage/multi_dc/

With 0.7.x, you should be using LOCAL instead of EACH for the third case pictured there.

Cheers,

On Thu, Nov 2, 2017 at 10:41 PM aiman parvaiz <aimanp...@gmail.com> wrote:
Thanks Alexander for your response. Yeah, I couldn't find any documentation around multi DC set up, looking forward to it. 


On Thursday, November 2, 2017 at 2:29:12 PM UTC-7, Alexander Dejanovski wrote:
Hi Aiman,

yes, you can run multiple Reapers, one or more in each DC in 0.7 when using the Cassandra backend and setting datacenterAvailability to LOCAL. Setting it to EACH won't work very well in that version of Reaper.
Both Reaper instances will sync up using the backend and share the work.

The upcoming release (in the coming days) will make EACH a valid (and safer) choice for multi DC clusters.
I'm writing a doc page on cassandra-reaper.io that will explain how to set up Reaper with multi DC, which should be merged in tomorrow.

Cheers,

On Thu, Nov 2, 2017 at 8:40 PM aiman parvaiz <aimanp...@gmail.com> wrote:
Hi all
I seek advice in setting up Reaper 0.7 for a Cassandra 2.2.8 cluster spanning over 2 DCs. I am aware that with 0.7 we can run Reaper across multiple DCs. In my understanding, I can install and run reaper on any one node in each DC (ending up with one reaper instance per DC) and using production C* as backend.
Am I correct in my understanding? Would this be the best setup for managing repairs for a multi DC set up?

I would appreciate any advice.

Thanks

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.

Alexander Dejanovski

unread,
Nov 7, 2017, 8:47:51 AM11/7/17
to aiman parvaiz, TLP Apache Cassandra Reaper users
Hi Aiman,

I have 3 DCs; A, B, C. In each DC, I am running reaper on 2 nodes using production C* as the backend where I have created reaper_db with NetworkTopologyStrategy spread over all 3 DCs. I have specified datacenterAvailability: EACH and contactPoints to DC specific seed nodes. I am able to see the webui on all of these servers which are running reaper and I was able to add the cluster and setup a repair schedule. I can see the repair running under repair section. Is this setup correct for repairing keyspaces spread over multi dc?

yes, that's the right setup.
If you're using Reaper 0.7.x, I would suggest to use LOCAL as datacenterAvailability, and EACH if you're using Reaper 1.0.0-RC1.

My question around schedule is the Interval in days* field, does this mean that a new repair would start after the specified interval in days(irrespective if the existing repair run is complete or not) or does it mean that once the running repair complete the next repair would start after this specified interval?

The new repair would start at : [previous repair start time] + [interval in days]
It won't start a new repair if the previous job is still running though, so you cannot end up with multiple repairs if it takes more time to complete the repair than the interval.

Cheers,


To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/1c056d83-d4ed-4fdf-b735-ea32318bcb6c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

aiman parvaiz

unread,
Nov 7, 2017, 10:09:02 AM11/7/17
to TLP Apache Cassandra Reaper users
Thanks Alexander for all your help. I am running reaper 0.7 and have changed datacenterAvailability to LOCAL. Couple of last questions:
1. Now since reaper_db is also a KS I believe we need to repair it too and if thats the case can we use reaper to repair reaper_db itself?
2. I noticed that I am getting the following errors in cassandra-reaper.err file when I restart my reaper process, I am running reaper as user 'reaper':

ls: cannot access server/target/cassandra-reaper-*.jar: Permission denied
ls: cannot access server/target/cassandra-reaper-*.jar: No such file or directory
14:59:55,106 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
14:59:55,107 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
14:59:55,107 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.xml]
14:59:55,111 |-INFO in ch.qos.logback.classic.BasicConfigurator@7a8fa663 - Setting up default configuration.
14:59:56,735 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5ce33a58 - Propagating INFO level on Logger[ROOT] onto the JUL framework
14:59:56,756 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5ce33a58 - Propagating DEBUG level on Logger[com.datastax.driver.core.QueryLogger.NORMAL] onto the JUL framework
14:59:56,764 |-INFO in c.q.l.core.rolling.TimeBasedRollingPolicy@2023917549 - Will use gz compression
14:59:56,765 |-INFO in c.q.l.core.rolling.TimeBasedRollingPolicy@2023917549 - Will use the pattern query-logger-%d.log for the active file
14:59:56,772 |-INFO in c.q.l.core.rolling.DefaultTimeBasedFileNamingAndTriggeringPolicy - The date pattern is 'yyyy-MM-dd' from file name pattern 'query-logger-%d.log.gz'.
14:59:56,772 |-INFO in c.q.l.core.rolling.DefaultTimeBasedFileNamingAndTriggeringPolicy - Roll-over at midnight.
14:59:56,777 |-INFO in c.q.l.core.rolling.DefaultTimeBasedFileNamingAndTriggeringPolicy - Setting initial period to Tue Nov 07 14:58:42 UTC 2017
14:59:56,783 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[file-appender] - Active log file name: query-logger.log
14:59:56,783 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[file-appender] - File property is set to [query-logger.log]
14:59:56,784 |-ERROR in ch.qos.logback.core.rolling.RollingFileAppender[file-appender] - openFile(query-logger.log,true) call failed. java.io.FileNotFoundException: query-logger.log (Permission denied)
at java.io.FileNotFoundException: query-logger.log (Permission denied)
at at java.io.FileOutputStream.open0(Native Method)
at at java.io.FileOutputStream.open(FileOutputStream.java:270)
at at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
at at ch.qos.logback.core.recovery.ResilientFileOutputStream.<init>(ResilientFileOutputStream.java:26)
at at ch.qos.logback.core.FileAppender.openFile(FileAppender.java:204)
at at ch.qos.logback.core.FileAppender.start(FileAppender.java:127)
at at ch.qos.logback.core.rolling.RollingFileAppender.start(RollingFileAppender.java:100)
at at io.dropwizard.logging.FileAppenderFactory.build(FileAppenderFactory.java:211)
at at io.dropwizard.logging.DefaultLoggingFactory.configureLoggers(DefaultLoggingFactory.java:215)
at at io.dropwizard.logging.DefaultLoggingFactory.configure(DefaultLoggingFactory.java:124)
at at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:81)
at at io.dropwizard.cli.Cli.run(Cli.java:75)
at at io.dropwizard.Application.run(Application.java:79)
at at com.spotify.reaper.ReaperApplication.main(ReaperApplication.java:91)
14:59:56,785 |-INFO in ch.qos.logback.classic.AsyncAppender[async-file-appender] - Attaching appender named [file-appender] to AsyncAppender.
14:59:56,786 |-INFO in ch.qos.logback.classic.AsyncAppender[async-file-appender] - Setting discardingThreshold to 51
14:59:56,786 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5ce33a58 - Propagating WARN level on Logger[io.dropwizard] onto the JUL framework
14:59:56,786 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@5ce33a58 - Propagating WARN level on Logger[org.eclipse.jetty] onto the JUL framework
14:59:56,788 |-INFO in ch.qos.logback.classic.AsyncAppender[async-console-appender] - Attaching appender named [console-appender] to AsyncAppender.
14:59:56,788 |-INFO in ch.qos.logback.classic.AsyncAppender[async-console-appender] - Setting discardingThreshold to 51

Seems to be a permission denied apparently with some jar not found but I used the RPM linked in the documentation on the website for installation. I am running CenOS 7 btw.Let me know if I am missing something here.

Thanks
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.

Alexander Dejanovski

unread,
Nov 7, 2017, 10:15:45 AM11/7/17
to aiman parvaiz, TLP Apache Cassandra Reaper users
Reaper operates on a small dataset and we don't perform much deletions, so you can of course repair reaper_db, but it's not critical.
The query logger should be disabled by default (and it's the case in the latest releases) so you should not mind about the exception and deactivate the query logger in the reaper yaml instead.

Cheers,


To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/f50fe8cb-987c-4e51-9a03-7a19dfe5dbf9%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages