Basic Questions of a New Reaper User

484 views
Skip to first unread message

Shalom

unread,
Sep 6, 2017, 12:43:08 PM9/6/17
to TLP Apache Cassandra Reaper users
Hi Everyone,

I'm new to Reaper and I'm starting to get to know it. This looks like an awesome tool so I'd like to thank you guys (and Spotify) for making such a great tool.

I have a few questions I couldn't find answers to.

  • If I add a wrong parameter to the yaml file, I get an error as follows:

  * Unrecognized field at: cassandra.queryOptions.authProvider
    Did you mean?:
      - fetchSize
      - metadataEnabled
      - reprepareOnUp
      - defaultIdempotence
      - consistencyLevel
        [8 more]

cassandra.queryOptions.authProvider is just an example. If I add a wrong parameter to any other field, I get a short list of possible parameters, related to that field.
I noticed that some parameters aren't shown in the docs. Is there a full list of these parameters and explanations on each one?



  • I added a cluster via the UI and it found and added all of the relevant nodes. However, it looks like it can't pull data on them, although I see repairs are running on them. Am I doing something wrong?

  • How can I tell what should be the segments count? Is there a reason for setting the default value to 200?
  • I'm using the auto scheduler and I try to understand What scheduleSpreadPeriod means?


Sorry for all the questions.... and I'll probably have a lot more when I get to know it better :-)



Thanks a lot!



Auto Generated Inline Image 1

jon

unread,
Sep 6, 2017, 2:49:54 PM9/6/17
to TLP Apache Cassandra Reaper users
The source for the configuration is found in com.spotify.reaper.ReaperApplicationConfiguration, you can see all the config options that could be listed there.  It's possible some of those config values are no longer used but were not removed, so you'll want to keep that in mind.  Personally I prefer to keep my config limited to base backend configuration and do the rest of the setup through the UI.

  • I added a cluster via the UI and it found and added all of the relevant nodes. However, it looks like it can't pull data on them, although I see repairs are running on them. Am I doing something wrong?

Tough to say.  What version of C* are you using?  Did you see any errors in the reaper logs?

  • How can I tell what should be the segments count? Is there a reason for setting the default value to 200?
I'd leave it blank, it'll calculate it for you.  (This is the most confusing part of reaper imo)

shalom sagges

unread,
Sep 7, 2017, 2:49:04 AM9/7/17
to jon, TLP Apache Cassandra Reaper users
Hi Jon,

Thanks for your comments. I'll check it out.

 What version of C* are you using?  Did you see any errors in the reaper logs?
You're right. The cluster of version 2.0.14 didn't show any data, but when I added another cluster of 3.0.12, everything was fine.

Thanks again!


--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/2f5dd9df-91d0-4583-b20f-91f59c3eb3d7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

shalom sagges

unread,
Sep 7, 2017, 7:01:55 AM9/7/17
to jon, TLP Apache Cassandra Reaper users
Hi Everyone,

I have a few more questions I couldn't find answers to.

I created an auto scheduler for repairs and added 2 clusters.
The first cluster (v2.0.14) I added is doing repairs, but the repairs are extremely slow for some reason (it's a QA cluster with barely any data in it).
Inline image 2

Is there any reason why it's repairing 2560 segments? I took Jon's advice and left it blank, but no matter the value I set, I always see 2560.
In addition, I see the scheduler sets a schedule that its time had already passed, but no repairs are running on it and never did:
Inline image 3

Regarding the 2nd cluster (v3.0.12)I added, I didn't see any repairs running on it at all. Here as well I see a schedule that its time had already passed(first row in the image below):
Inline image 4
When I tried to delete the scheduler or delete the cluster, I get a message that it's running repairs and cannot be deleted before I stop the repairs but no repairs are running there at all:

Inline image 1

Inline image 5



I'm using reaper 0.6.2 and this is the yaml I'm using:

# Cassandra Reaper Configuration Example.
# See a bit more complete example in:
# src/test/resources/cassandra-reaper.yaml
segmentCount:
repairParallelism: PARALLEL
repairIntensity: 0.99
scheduleDaysBetween: 1
repairRunThreadCount: 15
hangingRepairTimeoutMins: 600
storageType: cassandra
enableCrossOrigin: true
incrementalRepair: false
allowUnreachableNodes: false
enableDynamicSeedList: true
repairManagerSchedulingIntervalSeconds: 30

#jmxPorts:
#  127.0.0.1: 7100
#  127.0.0.2: 7200
#  127.0.0.3: 7300
#  127.0.0.4: 7400
#  127.0.0.5: 7500
#  127.0.0.6: 7600
#  127.0.0.7: 7700
#  127.0.0.8: 7800

#jmxAuth:
#  username: myUsername
#  password: myPassword

logging:
  level: INFO
  loggers:
    com.datastax.driver.core.QueryLogger.NORMAL:
      level: DEBUG
      additive: false
      appenders:
        - type: file
          currentLogFilename: query-logger.log
          archivedLogFilenamePattern: query-logger-%d.log.gz
          archivedFileCount: 2
    io.dropwizard: WARN
    org.eclipse.jetty: WARN
  appenders:
    - type: console
      logFormat: "%-6level [%d] [%t] %logger{5} - %msg %n"

server:
  type: default
  applicationConnectors:
    - type: http
      port: 8080
      bindHost: 0.0.0.0
  adminConnectors:
    - type: http
      port: 8081
      bindHost: 0.0.0.0
  requestLog:
    appenders: []

# database section will be ignored if storageType is set to "memory"
database:
  # H2 JDBC settings
  driverClass: org.h2.Driver
  url: jdbc:h2:mem:reaper-db;MODE=PostgreSQL
  user: user
  password: password

cassandra:
  clusterName: "Reaper-Cluster"
  contactPoints: ["127.0.0.1"]
  keyspace: reaper_db
  authProvider:
    type: plainText
    username: user
    password: password
  queryOptions:
    consistencyLevel: LOCAL_QUORUM
    serialConsistencyLevel: SERIAL

autoScheduling:
  enabled: true
  initialDelayPeriod: PT15S
  periodBetweenPolls: PT10M
  timeBeforeFirstSchedule: PT5M
  scheduleSpreadPeriod: PT6H
#  excludedKeyspaces:
#    - keyspace1
#    - keyspace2



Sorry for all the questions, I'm still new to this 🙂

To post to this group, send email to tlp-apache-cassandra-reaper-use...@googlegroups.com.

Alexander Dejanovski

unread,
Sep 7, 2017, 7:21:44 AM9/7/17
to shalom sagges, jon, TLP Apache Cassandra Reaper users
Hi Shalom,

about your issue with the cluster status screen, we have an issue opened and the fix is ready. I should merge that into master today.
It affects versions of Cassandra prior to 2.2 (I think it works in 2.1.16/17 though).

For you 2.0.14 cluster and slow repairs : You cannot have less segments than the number of vnodes in your cluster. This is why vnodes are mostly hated when the time comes to run repair (or bootstrap nodes). 
Reaper has a cycle of 30s before trying to start a new segment, so in your case, even if segments are repaired very fast, and assuming you have 10 nodes with 256 vnodes and a replication factor of 3, the total repair time should be around :  2560*30 / 3 = 25600s = ~7h
With RF=3 and 10 nodes, you should be able to have 3 segments running at the same time, which explains why I've divided by 3. This does not account for the time taken to build the merkle trees and the streaming that can occur.
For that kind of case though, you can speed things up by lowering the pause between cycles and you can set the following in your yaml file : 
repairManagerSchedulingIntervalSeconds: 10  

You could even try a lower value (like 1) but it could create additional load on the cluster (up to you to test).

For the 3.x cluster, the error message doesn't state that repairs are running but that schedules are active/running. You can delete them by first pressing the "Stop" button, and then "Delete".

I'd be interested in finding out what's wrong first though : do you spot any interesting message in the logs that could explain why schedules fail to trigger repairs ?
The interval of 1 day you've set between each repair looks a bit low to me as you're running full repairs, not incremental ones. Since this is QA I guess the interval will be different in production and this is just for testing.

Cheers,

To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/CAM3v5Fs%3DEEnRA81Bk1zdT5hKUjV6Za%2BOoEkA69jSUK1NPqFZEA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

shalom sagges

unread,
Sep 7, 2017, 7:55:23 AM9/7/17
to Alexander Dejanovski, jon, TLP Apache Cassandra Reaper users
Hi Alexander,

I see only one error in the logs:

ERROR  [2017-09-07 09:50:23,670] [SchedulingManagerTimer] c.s.r.s.SchedulingManager - failed managing schedule for run with id: 1042e120-9398-11e7-acf4-79d6586be660
ERROR  [2017-09-07 09:50:23,677] [SchedulingManagerTimer] c.s.r.s.SchedulingManager - catch exception
java.lang.ArithmeticException: BigInteger divide by zero
        at java.math.MutableBigInteger.divideKnuth(MutableBigInteger.java:1179)
        at java.math.BigInteger.divideKnuth(BigInteger.java:2108)
        at java.math.BigInteger.divide(BigInteger.java:2089)
        at com.spotify.reaper.service.SegmentGenerator.generateSegments(SegmentGenerator.java:138)
        at com.spotify.reaper.resources.CommonTools.generateSegments(CommonTools.java:123)
        at com.spotify.reaper.resources.CommonTools.registerRepairRun(CommonTools.java:68)
        at com.spotify.reaper.service.SchedulingManager.startNewRunForUnit(SchedulingManager.java:173)
        at com.spotify.reaper.service.SchedulingManager.manageSchedule(SchedulingManager.java:139)
        at com.spotify.reaper.service.SchedulingManager.run(SchedulingManager.java:83)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)

This is also the run ID that its next run is shown as "5 hours ago". I did find that one of the nodes in the cluster was in DN state. I started it back up, deleted and added the cluster again to the scheduler.
I thought that might have been the issue here, but it happened again. The schedule was set to start on the first ID in 5 minutes. Once it started, the "Next run" column started to count the next run backwards.

The above error still persists on the new started repair, this time with a different ID.

Inline image 1



Also, although it said it started, there are still no running repairs on that cluster:

Inline image 2


Could it be something with the autoscheduler configuration? (I think I kept the default values)

autoScheduling:
  enabled: true
  initialDelayPeriod: PT15S
  periodBetweenPolls: PT10M
  timeBeforeFirstSchedule: PT5M
  scheduleSpreadPeriod: PT6H


Thanks!


To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/CAM3v5Fs%3DEEnRA81Bk1zdT5hKUjV6Za%2BOoEkA69jSUK1NPqFZEA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Alexander Dejanovski

unread,
Sep 7, 2017, 9:45:42 AM9/7/17
to shalom sagges, jon, TLP Apache Cassandra Reaper users
OK I see the problem,

your config file has no default value for the segment count, which is interpreted as 0 and generates the error.
Change the "segmentCount" value to something like 200 and recreate your schedules. 

You should then be good to go ;)


To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/CAM3v5Fs%3DEEnRA81Bk1zdT5hKUjV6Za%2BOoEkA69jSUK1NPqFZEA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

shalom sagges

unread,
Sep 7, 2017, 11:53:46 AM9/7/17
to Alexander Dejanovski, jon, TLP Apache Cassandra Reaper users
That's what I did at first.
But on previous replies of this thread, I asked what should be the segments count and is there a reason for setting the default value to 200 and Jon mentioned that I can leave it blank and Reaper will calculate it for me.
Perhaps I should have removed the entire property?

To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

jon

unread,
Sep 7, 2017, 1:23:53 PM9/7/17
to TLP Apache Cassandra Reaper users
Sorry for the confusion, I was referring to leaving the field blank in the UI.  

shalom sagges

unread,
Sep 7, 2017, 1:50:04 PM9/7/17
to jon, TLP Apache Cassandra Reaper users
Got it.. Thanks!



To post to this group, send email to tlp-apache-cassandra-reaper-use...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-use...@googlegroups.com.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/d1a9adc6-cc14-466f-8022-144ea313c906%40googlegroups.com.

shalom sagges

unread,
Sep 13, 2017, 7:17:42 AM9/13/17
to jon, TLP Apache Cassandra Reaper users
Hey Guys,

Me again.... :-)

I'm still seeing next runs that are scheduled to "x hours ago". I've added below the yaml file I'm using. Is there something I'm doing wrong here?


Inline image 1



# Cassandra Reaper Configuration Example.
# See a bit more complete example in:
# src/test/resources/cassandra-reaper.yaml
segmentCount: 200
repairParallelism: PARALLEL
repairIntensity: 0.99
scheduleDaysBetween: 1
repairRunThreadCount: 15
hangingRepairTimeoutMins: 600
storageType: cassandra
enableCrossOrigin: true
incrementalRepair: false
allowUnreachableNodes: false
enableDynamicSeedList: true
repairManagerSchedulingIntervalSeconds: 1
cassandra:
  clusterName: "Reaper-Cluster"
  contactPoints: ["127.0.0.1"]
  keyspace: reaper_db
  authProvider:
    type: plainText
    username: user
    password: password
  queryOptions:
    consistencyLevel: LOCAL_QUORUM
    serialConsistencyLevel: SERIAL

autoScheduling:
  enabled: true
  initialDelayPeriod: PT15S
  periodBetweenPolls: PT10M
  timeBeforeFirstSchedule: PT5M
  scheduleSpreadPeriod: PT6H
#  excludedKeyspaces:
#    - keyspace1
#    - keyspace2


Thanks!


Got it.. Thanks!



Alexander Dejanovski

unread,
Sep 13, 2017, 7:51:08 AM9/13/17
to shalom sagges, jon, TLP Apache Cassandra Reaper users
Hi Shalom,

were the schedules recreated after you changed you settings ?
Do you have any exception in the logs that could help us debug the issue ?

Thanks,

Got it.. Thanks!



To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/CAM3v5FtCr56CMF-u54B0H5hXYD27Mvda2HX%3D-8XAKXwojazUkw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

shalom sagges

unread,
Sep 13, 2017, 10:46:04 AM9/13/17
to Alexander Dejanovski, jon, TLP Apache Cassandra Reaper users
Hi Alexander,

Yes, I've recreated the schedules after changing the settings.

In the logs I see the following error, right after the repair was successfully completed:


INFO   [2017-09-13 17:35:26,887] [qa-cluster:69250020-988e-11e7-bb1a-0b6d8c0d2134:6937c4f7-988e-11e7-bb1a-0b6d8c0d2134] c.s.r.s.SegmentRunner - Repair command 9787 on segment 6937c4f7-988e-11e7-bb1a-0b6d8c0d2134 returned with state DONE
ERROR  [2017-09-13 17:35:26,898] [SchedulingManagerTimer] c.s.r.s.SchedulingManager - failed managing schedule for run with id: d0175a80-95ff-11e7-bb1a-0b6d8c0d2134
ERROR  [2017-09-13 17:35:26,898] [SchedulingManagerTimer] c.s.r.s.SchedulingManager - catch exception
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
        at java.util.ArrayList.rangeCheck(ArrayList.java:653)
        at java.util.ArrayList.get(ArrayList.java:429)
        at com.spotify.reaper.resources.CommonTools.getClusterNodes(CommonTools.java:211)
        at com.spotify.reaper.resources.CommonTools.registerRepairRun(CommonTools.java:71)

        at com.spotify.reaper.service.SchedulingManager.startNewRunForUnit(SchedulingManager.java:173)
        at com.spotify.reaper.service.SchedulingManager.manageSchedule(SchedulingManager.java:139)
        at com.spotify.reaper.service.SchedulingManager.run(SchedulingManager.java:83)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)


I will happily change the logging to DEBUG if need be.

Thanks for your help :-)


Got it.. Thanks!



To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/CAM3v5FtCr56CMF-u54B0H5hXYD27Mvda2HX%3D-8XAKXwojazUkw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Alexander Dejanovski

unread,
Sep 13, 2017, 12:01:45 PM9/13/17
to shalom sagges, jon, TLP Apache Cassandra Reaper users
Hi Shalom,


This means that some token ranges in your cluster have no replica for at least one keyspace (or at least we fail to determine them through JMX).

Can you spot in the logs which specific schedule is failing ?
If so, could you run jconsole or jmxterm on one of the nodes in that schedule's cluster and invoke the following operation on the schedule's keyspace ? 
o.a.c.db.StorageService.getRangeToEndpointMap(keyspace) 

I would need the output of that command because there's something weird here.

Also, what is your definition for that specific keyspace ? (the "CREATE KEYSPACE..." statement).

Thanks



Got it.. Thanks!



To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/CAM3v5FtCr56CMF-u54B0H5hXYD27Mvda2HX%3D-8XAKXwojazUkw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

shalom sagges

unread,
Sep 17, 2017, 4:30:00 AM9/17/17
to Alexander Dejanovski, jon, TLP Apache Cassandra Reaper users
Hi Alexander,

You were right. One of my test keyspaces had a typo in the DC name (I checked all other erroneous run IDs found in the logs, but I didn't see them in the scheduler... Weird. 
So I deleted the bad keyspace, will reschedule all and check again. I'll update on my findings.

I still don't understand why other clusters are affected though (have the same "next run in X days ago").
One cluster has a keyspace that its replication strategy is 3 for its local DC with no replication to the remote DC. That shouldn't be a problem, right?

Thanks!




Got it.. Thanks!



To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/CAHkQdMgGM027b0uN-53uDVzG5e7ExPEPRXqi%3DEphujq9mYwRxg%40mail.gmail.com.

Alexander Dejanovski

unread,
Sep 17, 2017, 2:01:22 PM9/17/17
to shalom sagges, jon, TLP Apache Cassandra Reaper users
I think Reaper was trying to process the schedules sequentially, but failed on the very first one. The exception made it skip all the other schedules.

Let us know how it works now.

Got it.. Thanks!



To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/CAHkQdMgGM027b0uN-53uDVzG5e7ExPEPRXqi%3DEphujq9mYwRxg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

shalom sagges

unread,
Sep 19, 2017, 3:40:14 AM9/19/17
to Alexander Dejanovski, jon, TLP Apache Cassandra Reaper users
Hi Guys,

A small update...
Once I removed the problematic keyspace a couple of days ago, everything is running flawlessly.
Thanks a lot for the guidance!

As a side note, do you think it'd be possible for the scheduler to mark and skip problematic keyspaces without affecting the other schedules?
In this specific case, even other clusters that had no relation with that keyspace were affected.
Just a thought....

Thanks again for the help  :-)





Got it.. Thanks!



To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.

Alexander Dejanovski

unread,
Sep 19, 2017, 3:44:39 AM9/19/17
to shalom sagges, jon, TLP Apache Cassandra Reaper users
Hi Shalom, 

thanks for the update !

You suggestion of skipping erroneous schedules makes a lot of sense. Could you file an issue on the GitHub project page so that we can keep track of your request ? 

thanks 

Got it.. Thanks!



To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

shalom sagges

unread,
Sep 19, 2017, 8:03:54 AM9/19/17
to Alexander Dejanovski, jon, TLP Apache Cassandra Reaper users
This is the first time I open an issue on GitHub so please let me know if I did something wrong and need to edit/fix it.

Thanks!

Got it.. Thanks!



To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
To post to this group, send email to tlp-apache-cassandra-reaper-us...@googlegroups.com.
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
Reply all
Reply to author
Forward
0 new messages