Reaper UI unable to connect to reaper server

420 views
Skip to first unread message

Philip Henriksen

unread,
Nov 13, 2019, 3:14:02 AM11/13/19
to TLP Apache Cassandra Reaper users
I have deployed the full set of cassandra-applications on a kubernetes cluster:
- casskop
- casskop-managed cassandra cluster with 3 nodes
- cassandra-reaper

I used these variables in the helm-chart to get reaper up and running, and the startup itself seems to have gone fine:

env:
  REAPER_STORAGE_TYPE: cassandra
  REAPER_CASS_CLUSTER_NAME: "cassandra"
  REAPER_CASS_AUTH_ENABLED: true
  REAPER_CASS_AUTH_USERNAME: "reaper_user"
  REAPER_CASS_AUTH_PASSWORD: "****"
  REAPER_CASS_KEYSPACE: "reaper_db"
  REAPER_CASS_PORT: 9042
  REAPER_CASS_CONTACT_POINTS: ["cassandra-cluster"]

Looking at the logs from reaper, it looks like all the nodes in my cluster were found and successfully configured.
However, when entering the webui, I get the error "Server not available! Could not contact reaper server."
Additionally, the "Clusters" entry says "No clusters found", and when entering any of the nodes that reaper previously found in the startup into "Seed node", I get a 404.

What am I doing wrong?
Is there some other way of making use of the Reaper without using webui?
cassandra_reaper.log

Reid Pinchback

unread,
Nov 13, 2019, 10:51:45 AM11/13/19
to Philip Henriksen, TLP Apache Cassandra Reaper users

This sounds like something where the reaper service was up initially but is now down, or for some other reason inaccessible, and you still have the web UI because it was in your browser cache. 

 

I’d start with seeing if the service itself is up.  The reaper service will do a System.exit if it detects something bad going on.  Unfortunately it’s idea of bad is ‘database unavailable’ so a sporadic connectivity issue will cause a service shutdown, unless you’ve twiddled the system unit file to restart on failure.  I don’t use the Cassandra storage type though, so your mileage definitely can vary on that as a cause.  Whatever may be going on, I’d start by looking at the reaper logs.

 

The seed node 404 thing sounds like a red herring, if you can’t get to the reaper http port, anything else you do obviously has to fail.

 

 

From: <tlp-apache-cassa...@googlegroups.com> on behalf of Philip Henriksen <philip...@hotmail.com>
Date: Wednesday, November 13, 2019 at 3:14 AM
To: TLP Apache Cassandra Reaper users <tlp-apache-cassa...@googlegroups.com>
Subject: Reaper UI unable to connect to reaper server

 

Message from External Sender

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/0c7a5851-e04d-471b-8872-d2a983fe8e5b%40googlegroups.com.

Reid Pinchback

unread,
Nov 13, 2019, 1:04:15 PM11/13/19
to Philip Henriksen, TLP Apache Cassandra Reaper users

(ignore the bit about twiddling the system unit file, as your in k8s)

Philip Henriksen

unread,
Nov 13, 2019, 1:18:20 PM11/13/19
to TLP Apache Cassandra Reaper users
The thing that amazes me is that the service is somehow both running and not connected at the same time.

When the service fails, the ui doesn't work at all. That is, I get some generic error message, as K8S recognizes that it's down.
At the moment, Reaper is cleaning the logs (which in principle means nothing more than "Creates an entry in the Reaper log", as there are no logs to clean) every hour or so.

Each of the new log entries looks like this:
INFO [2019-11-13 17:33:53,437] [ReaperApplication-scheduler] i.c.ReaperApplication - Purged 0 repair runs from history

Meanwhile, at the exact same time, the WebUI is telling me that it can't connect to the reaper server.

Reaper seems to have been running without error for a couple hours now, but the UI still doesn't work

To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.

Alexander Dejanovski

unread,
Nov 13, 2019, 1:29:03 PM11/13/19
to Philip Henriksen, TLP Apache Cassandra Reaper users
Hi, 

Could you check your JavaScript console for error messages? You'll see there how the UI tries to communicate with the backend, and how it fails. 
If you get purge related messages in the logs, then Reaper is probably working fine indeed, and you probably have a networking problem.

You can interact with reaper using the rest API or the spreaper command line tool that ships with reaper.
Check the docs at cassandra-reaper.io for more informations.

Cheers,


To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/ffd26538-8d47-4a8a-8b84-2a42cbc9d617%40googlegroups.com.

Philip Henriksen

unread,
Nov 14, 2019, 1:12:35 AM11/14/19
to TLP Apache Cassandra Reaper users
Hello,

Thank you.
Never thought about looking at the network log, seeing what went wrong in the requests.

My problem was that I had specifically told K8S that the path "/webui" was handled, nothing more.
This meant that "reaper.example.com/webui" directed to Reaper, and it was able to download the JavaScript for the UI itself.
However, all requests the UI tried to send, like "/cluster" and "/ping" were all directed to the 404 backend on the cluster.

By generalizing the ingress a bit more, saying "reaper.example.com" instead of specifically "/webui", the requests went through and everything works as expected.

Again, thanks for all your help.

To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-reaper-users+unsubscribe@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages