MBean type=StorageService/getRangeToEndpointMap not implemented?

[Denis Zhdanov]

Hello!

I'm trying to port Yelp's Cassandra monitoring script from https://engineeringblog.yelp.com/2016/06/monitoring-cassandra-at-scale.html to ScyllaDB, but it fails on `type=StorageService/getRangeToEndpointMap` Mbean. I implemented Jolokia on Scylla-JMX, it works for some Mbeans but not for all:

root@mt-scylla001:~# curl http://localhost:8089/jolokia/exec/org.apache.cassandra.db:type=StorageService/getRangeToEndpointMap/my_keyspace | jq .
{
  "request": {
    "mbean": "org.apache.cassandra.db:type=StorageService",
    "arguments": [
      "my_keyspace"
    ],
    "type": "exec",
    "operation": "getRangeToEndpointMap"
  },
  "stacktrace": "java.lang.IllegalStateException: Scylla API server HTTP GET to URL '/storage_service/range/my_keyspace' failed: Not found\n\tat com.scylladb.jmx.api.APIClient.getException(APIClient.java:135)\n\tat com.scylladb.jmx.api.APIClient.getRawValue(APIClient.java:178)\n\tat com.scylladb.jmx.api.APIClient.getRawValue(APIClient.java:192)\n\tat com.scylladb.jmx.api.APIClient.getReader(APIClient.java:216)\n\tat com.scylladb.jmx.api.APIClient.getMapListStrValue(APIClient.java:372)\n\tat com.scylladb.jmx.api.APIClient.getMapListStrValue(APIClient.java:386)\n\tat org.apache.cassandra.service.StorageService.getRangeToEndpointMap(StorageService.java:272)\n\tat sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)\n\tat sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)\n\tat com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)\n\tat com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)\n\tat com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)\n\tat com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)\n\tat com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)\n\tat com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)\n\tat com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)\n\tat com.scylladb.jmx.utils.APIMBeanServer.invoke(APIMBeanServer.java:167)\n\tat org.jolokia.handler.ExecHandler.doHandleRequest(ExecHandler.java:98)\n\tat org.jolokia.handler.ExecHandler.doHandleRequest(ExecHandler.java:40)\n\tat org.jolokia.handler.JsonRequestHandler.handleRequest(JsonRequestHandler.java:89)\n\tat org.jolokia.backend.MBeanServerExecutorLocal.handleRequest(MBeanServerExecutorLocal.java:109)\n\tat org.jolokia.backend.MBeanServerHandler.dispatchRequest(MBeanServerHandler.java:161)\n\tat org.jolokia.backend.LocalRequestDispatcher.dispatchRequest(LocalRequestDispatcher.java:99)\n\tat org.jolokia.backend.BackendManager.callRequestDispatcher(BackendManager.java:428)\n\tat org.jolokia.backend.BackendManager.handleRequest(BackendManager.java:158)\n\tat org.jolokia.http.HttpRequestHandler.executeRequest(HttpRequestHandler.java:197)\n\tat org.jolokia.http.HttpRequestHandler.handleGetRequest(HttpRequestHandler.java:86)\n\tat org.jolokia.jvmagent.handler.JolokiaHttpHandler.executeGetRequest(JolokiaHttpHandler.java:276)\n\tat org.jolokia.jvmagent.handler.JolokiaHttpHandler.doHandle(JolokiaHttpHandler.java:233)\n\tat org.jolokia.jvmagent.handler.JolokiaHttpHandler.handle(JolokiaHttpHandler.java:178)\n\tat com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)\n\tat sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:83)\n\tat com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:82)\n\tat sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:675)\n\tat com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:79)\n\tat sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:647)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\n",
  "error_type": "java.lang.IllegalStateException",
  "error": "java.lang.IllegalStateException : Scylla API server HTTP GET to URL '/storage_service/range/my_keyspace' failed: Not found",
  "status": 500
}

I see that "getRangeToEndpointMap" is present in Scylla code, but maybe not implemented in API?

WBR,

   Denys

[Pekka Enberg]

It is indeed unimplemented in the Scylla server itself.

You can find the JMX proxy parts here:

https://github.com/scylladb/scylla-jmx/blob/master/src/main/java/org/apache/cassandra/service/StorageService.java#L271

The REST API is defined:

https://github.com/scylladb/scylla/blob/master/api/api-doc/storage_service.json#L283 

but the it's unimplemented:

https://github.com/scylladb/scylla/blob/master/api/storage_service.cc#L163

It's worth opening a ticket on GitHub about this and describe your use case.

- Pekka

[Denis Zhdanov]

Well... My usecase is mostly same as Yelp's - monitor availability of separate keyspaces. You can do that with clever parsing of `nodetool status` output probably, but, m y initial idea was to just try working solution from Cassandra.

Will open a issue, thanks!

понедельник, 11 ноября 2019 г., 18:39:50 UTC+1 пользователь Pekka Enberg написал:

[Dan Yasny]

On Mon, Nov 11, 2019 at 1:15 PM Denis Zhdanov <denis....@gmail.com> wrote:

Well... My usecase is mostly same as Yelp's - monitor availability of separate keyspaces.

This can be a nice feature to add to the monitoring stack once it is implemented. Scylla generally focuses on exporting the Prometheus native metrics instead of JMX, for monitoring purposes

 

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/2af9c1af-1962-4a7c-be1c-ebf5b720b11d%40googlegroups.com.

[Denis Zhdanov]

I'm OK with Prometheus too, but then this feature need to be directly implemented in metrics code then.

понедельник, 11 ноября 2019 г., 19:45:29 UTC+1 пользователь Dan Yasny написал:

To unsubscribe from this group and stop receiving emails from it, send an email to scyllad...@googlegroups.com.