Having trouble getting Reaper setup

[Rhys Campbell]

Hello,

I'm having some issues getting Reaper setup. I basically can't seem to get past the db setup stage. The reaper service runs for a bit but dies after a few minutes. The web side of the application never get started. I've extracted some info from my test env. Can anyone advise?

Cheers,

Rhys

reaper.log

INFO   [main] o.c.c.m.MigrationRepository - Found 19 migration scripts 
WARN   [main] i.c.s.CassandraStorage - Starting db migration from 0 to 19… 
INFO   [main] o.c.c.m.MigrationRepository - Found 19 migration scripts 
WARN   [Reaper-worker-0] c.d.d.c.Cluster - Re-preparing already prepared query is generally an anti-pattern and will likely affect performance. Consider preparing the statement only once. Query='insert into schema_migration(applied_successful, version, script_name, script, executed_at) values(?, ?, ?, ?, ?)' 
WARN   [Reaper-nio-worker-0] c.d.d.c.RequestHandler - /192.168.43.202:9042 replied with server error (java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 38f90860-a4a3-11e9-a908-39bfd52bcef3; expected 38f81e00-a4a3-11e9-a908-39bfd52bcef3)), defuncting connection. 
WARN   [Reaper-nio-worker-0] c.d.d.c.RequestHandler - /192.168.43.201:9042 replied with server error (java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 38f90860-a4a3-11e9-a908-39bfd52bcef3; expected 38f81e00-a4a3-11e9-a908-39bfd52bcef3)), defuncting connection. 
INFO   [main] o.a.s.c.ReflectionBuilder - An instance with name 'authc' already exists.  Redefining this object as a new instance of type org.apache.shiro.web.filter.authc.PassThruAuthenticationFilter 
INFO   [main] i.c.ReaperApplication - initializing runner thread pool with 15 threads 
INFO   [main] i.c.ReaperApplication - initializing storage of type: cassandra 
INFO   [main] c.d.d.core - DataStax Java driver 3.5.0 for Apache Cassandra 
INFO   [main] c.d.d.c.GuavaCompatibility - Detected Guava >= 19 in the classpath, using modern compatibility layer 
INFO   [main] c.d.d.c.ClockFactory - Using native clock to generate timestamps. 
INFO   [main] c.d.d.c.NettyUtil - Found Netty's native epoll transport in the classpath, using it 

/var/log/messages shows show timeouts...

 Jul 12 13:43:59 localhost cassandra-reaper: org.cognitor.cassandra.migration.MigrationException: Error during migration of script 001_Initialize_db.cql while executing 'CREATE TABLE IF NOT EXISTS repair_schedule_by_cluster_and_keyspace(  cluster_name text,  keyspace_name text,  repair_schedule_id bigint,  PRIMARY KEY((cluster_name, keyspace_name), repair_schedule_id));'
Jul 12 13:43:59 localhost cassandra-reaper: at org.cognitor.cassandra.migration.Database.execute(Database.java:178)
Jul 12 13:43:59 localhost cassandra-reaper: at java.util.Collections$SingletonList.forEach(Collections.java:4822)
Jul 12 13:43:59 localhost cassandra-reaper: at org.cognitor.cassandra.migration.MigrationTask.migrate(MigrationTask.java:52)
Jul 12 13:43:59 localhost cassandra-reaper: at io.cassandrareaper.storage.CassandraStorage.migrate(CassandraStorage.java:262)
Jul 12 13:43:59 localhost cassandra-reaper: at io.cassandrareaper.storage.CassandraStorage.initializeAndUpgradeSchema(CassandraStorage.java:232)
Jul 12 13:43:59 localhost cassandra-reaper: at io.cassandrareaper.storage.CassandraStorage.<init>(CassandraStorage.java:190)
Jul 12 13:43:59 localhost cassandra-reaper: at io.cassandrareaper.ReaperApplication.initializeStorage(ReaperApplication.java:330)
Jul 12 13:43:59 localhost cassandra-reaper: at io.cassandrareaper.ReaperApplication.run(ReaperApplication.java:175)
Jul 12 13:43:59 localhost cassandra-reaper: at io.cassandrareaper.ReaperApplication.run(ReaperApplication.java:81)
Jul 12 13:43:59 localhost cassandra-reaper: at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:43)
Jul 12 13:43:59 localhost cassandra-reaper: at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:87)
Jul 12 13:43:59 localhost cassandra-reaper: at io.dropwizard.cli.Cli.run(Cli.java:78)
Jul 12 13:43:59 localhost cassandra-reaper: at io.dropwizard.Application.run(Application.java:93)
Jul 12 13:43:59 localhost cassandra-reaper: at io.cassandrareaper.ReaperApplication.main(ReaperApplication.java:100)
Jul 12 13:43:59 localhost cassandra-reaper: Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /192.168.43.201:9042 (com.datastax.driver.core.exceptions.ServerError: An unexpected error occurred server side on /192.168.43.201:9042: java.lang.RuntimeException: java.util.concurrent.ExecutionExceptio

 

Nodetool reports that I have the same schema on both nodes...

reaper1 ~]# nodetool describecluster
Cluster Information:
Name: Reaper
Snitch: org.apache.cassandra.locator.SimpleSnitch
DynamicEndPointSnitch: enabled
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
c0fdbe22-f760-3699-be56-8d70d7b78c57: [192.168.43.201, 192.168.43.202]

nodetool status looks good...

Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address         Load       Tokens       Owns (effective)  Host ID                               Rack
UN  192.168.43.201  335.05 KiB  256          100.0%            ead85278-99ba-4b07-90e5-79c68c5ba6ac  rack1
UN  192.168.43.202  314.37 KiB  256          100.0%            7746b9ed-c07d-491f-b1be-c2b24577228f  rack1

I have the following tables in my reaper_db

admin@cqlsh> use reaper_db ;
admin@cqlsh:reaper_db> desc tables;
repair_unit            repair_segment                         
repair_run             repair_schedule                        
repair_run_by_cluster  repair_schedule_by_cluster_and_keyspace
schema_migration       repair_run_by_unit                     
cluster                repair_segment_by_run_id  

Reaper is starting up quite a while after Cassandra so it should be responsive...

admin@cqlsh> use reaper_db ;
admin@cqlsh:reaper_db> desc tables;
repair_unit            repair_segment                         
repair_run             repair_schedule                        
repair_run_by_cluster  repair_schedule_by_cluster_and_keyspace
schema_migration       repair_run_by_unit                     
cluster                repair_segment_by_run_id  

[Philip Ó Condúin]

Can you share your config file settings and also the reaper startup command?

[Rhys Campbell]

Hello Philip,

I have used the rpm to install reaper. I don't make many modifications. Mainly at the bottom with a cassandra data store. I've also modified the service file to start cassandra-reaper after cassandra. This seems to have helped. When I browser to http://localhost/8080 I get a bared bones html page looking like this...

Operational Menu

• Metrics
• Ping
• Threads
• Healthcheck
• CPU Profile
• CPU Contention

But I get nothing at all at the webui url.

Rhys 

On Sat, Jul 13, 2019 at 5:57 AM Philip Ó Condúin <philip...@gmail.com> wrote:

Can you share your config file settings and also the reaper startup command?

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To post to this group, send email to tlp-apache-cassa...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/e4ca79a3-45eb-4995-8e83-1b9596894601%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alexander Dejanovski]

Hi,

the schema couldn't be created by the migration lib because you already have a mismatch in the cluster it seems :

WARN   [Reaper-nio-worker-0] c.d.d.c.RequestHandler - /192.168.43.202:9042 replied with server error (java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 38f90860-a4a3-11e9-a908-39bfd52bcef3; expected 38f81e00-a4a3-11e9-a908-39bfd52bcef3)), defuncting connection. 

I advise to roll restart the cluster in order to fix this and try again running Reaper. Upon startup, it will go again through schema migration.

If that fails again, run reaper on a test/local/ccm node or cluster, dump the reaper_db schema to a script and dump the content of the reaper_db.schema_migration table in order to initialize the schema manually (create all the tables and fill the schema_migration table with your dump).

Cheers,

-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting

http://www.thelastpickle.com

To view this discussion on the web visit https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/CAN0uOTS04Oanc_06h6CbfVYpLN0akWNkdNqY%2BYvNf5orcTN%3DCA%40mail.gmail.com.

[Rhys Campbell]

It appeared to be balking on records 5 and 6. Both these rows were flagged applied_successful = False even though the changes appear to have been applied. I deleted these records and ran the below statements...

INSERT INTO schema_migration (applied_successful,version,executed_at,script,script_name) VALUES (True,5,'2019-07-15 10:09:53','ALTER TABLE repair_unit_v1 ADD nodes set<text>;ALTER TABLE repair_unit_v1 ADD "datacenters" set<text>;','005_repair_specific_nodes.cql');
INSERT INTO schema_migration (applied_successful,version,executed_at,script,script_name) VALUES (True,6,'2019-07-15 11:16:32','ALTER TABLE repair_unit_v1 ADD blacklisted_tables set<text>;','006_table_blacklist.cql');

After a restart the migration proceeded up to version 19 with issue. I think I'll do as you suggested and dump the schema and create this manually.

Cheers,

Rhys

[Philip Ó Condúin]

Is this a single node reaper setup or are you installing on 2 reaper nodes?

--

Regards,

Phil

[Rhys Campbell]

2 nodes. But I had similar issues with a one node setup. I was aware a migration ran on startup so I only run one instance of reaper at first.

To view this discussion on the web visit https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/CAM8_W0Y%2BmJZsVM3t5Oc-09sCuGmkdhH_ktRgE-3_zEPowicm2A%40mail.gmail.com.

[Philip Ó Condúin]

Tell me which one of these you are trying to set up, I'm struggling to understand what your architecture is

http://cassandra-reaper.io/docs/usage/multi_dc/

You received this message because you are subscribed to a topic in the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tlp-apache-cassandra-reaper-users/VlXcPuPVf8I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to tlp-apache-cassandra-r...@googlegroups.com.

To post to this group, send email to tlp-apache-cassa...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/CAN0uOTTwb%3DXsHky_TqJSP8E3krQxCk2UARqsPaVBWunXNRXcfw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Regards,

Phil

[Rhys Campbell]

At the moment this is a 2 node vagrant test setup but the plan would be datacenterAvailability: EACH