Cassandra Metrics

[Steven McCord]

I am installing a Cassandra 3.7 instance but it appears that sending events with the 3.0 reporter with the metrics-core isn't working and the Blue Mountain library doesn't seem to work with this version either, has anyone sent Cassandra metrics to Riemann recently?  Thanks!

[James Turnbull]

Have you tried it with this PR merged?

https://github.com/BlueMountainCapital/riemann-cassandra/pull/2

Regards

James


--
* The Art of Monitoring (https://artofmonitoring.com)
* The Docker Book (https://dockerbook.com)
* The LogStash Book (https://logstashbook.com)
* Pro Puppet (http://tinyurl.com/ppuppet2 )
* Pro Linux System Administration (http://tinyurl.com/linuxadmin)
* Hardening Linux (http://tinyurl.com/hardeninglinux)

[Steven McCord]

James:

Thanks for the response, yeah I actually compiled that jar and tried that as well.  I get a different error though when I try that since it is doing something a little different.

success start riemann-cassandra-client............@Thu Oct 06 19:00:06 UTC 2016

Unable to connect to Cassandra JMX (172.17.0.7:7199) will continue to try silently....

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/share/cassandra/lib/logback-classic-1.1.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/share/cassandra/lib/riemann-cassandra-0.0.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

INFO  19:00:07 Hostname: macho-man-master-3884713917-136u2

INFO  19:00:07 Loading settings from file:/etc/cassandra/cassandra.yaml

INFO  19:00:07 Node configuration:[authenticator=AllowAllAuthenticator; authorizer=AllowAllAuthorizer; auto_bootstrap=false; auto_snapshot=true; batch_size_fail_threshold_in_kb=50; batch_size_warn_threshold_in_kb=5; batchlog_replay_throttle_in_kb=1024; broadcast_address=172.17.0.7; broadcast_rpc_address=172.17.0.7; cas_contention_timeout_in_ms=1000; client_encryption_options=<REDACTED>; cluster_name=ICX; column_index_size_in_kb=64; commit_failure_policy=stop; commitlog_directory=/cassandra_data/commitlog; commitlog_segment_size_in_mb=32; commitlog_sync=periodic; commitlog_sync_period_in_ms=10000; compaction_large_partition_warning_threshold_mb=100; compaction_throughput_mb_per_sec=16; concurrent_counter_writes=32; concurrent_materialized_view_writes=32; concurrent_reads=32; concurrent_writes=32; counter_cache_save_period=7200; counter_cache_size_in_mb=null; counter_write_request_timeout_in_ms=5000; credentials_validity_in_ms=2000; cross_node_timeout=false; data_file_directories=[/cassandra_data/data]; disk_failure_policy=stop; disk_optimization_strategy=ssd; dynamic_snitch_badness_threshold=0.1; dynamic_snitch_reset_interval_in_ms=600000; dynamic_snitch_update_interval_in_ms=100; enable_scripted_user_defined_functions=false; enable_user_defined_functions=false; endpoint_snitch=GossipingPropertyFileSnitch; gc_warn_threshold_in_ms=1000; hinted_handoff_enabled=true; hinted_handoff_throttle_in_kb=1024; hints_flush_period_in_ms=10000; incremental_backups=false; index_summary_capacity_in_mb=null; index_summary_resize_interval_in_minutes=60; inter_dc_tcp_nodelay=false; internode_compression=all; key_cache_save_period=14400; key_cache_size_in_mb=null; listen_address=172.17.0.7; max_hint_window_in_ms=10800000; max_hints_delivery_threads=2; max_hints_file_size_in_mb=128; memtable_allocation_type=heap_buffers; native_transport_port=9042; num_tokens=32; partitioner=org.apache.cassandra.dht.Murmur3Partitioner; permissions_validity_in_ms=2000; range_request_timeout_in_ms=10000; read_request_timeout_in_ms=5000; request_scheduler=org.apache.cassandra.scheduler.NoScheduler; request_timeout_in_ms=10000; role_manager=CassandraRoleManager; roles_validity_in_ms=2000; row_cache_save_period=0; row_cache_size_in_mb=0; rpc_address=0.0.0.0; rpc_keepalive=true; rpc_port=9160; rpc_server_type=sync; saved_caches_directory=/cassandra_data/saved_caches; seed_provider=[{class_name=org.apache.cassandra.locator.SimpleSeedProvider, parameters=[{seeds=macho-man-master}]}]; server_encryption_options=<REDACTED>; snapshot_before_compaction=false; ssl_storage_port=7001; sstable_preemptive_open_interval_in_mb=50; start_native_transport=true; start_rpc=true; storage_port=7000; thrift_framed_transport_size_in_mb=15; tombstone_failure_threshold=100000; tombstone_warn_threshold=1000; tracetype_query_ttl=86400; tracetype_repair_ttl=604800; transparent_data_encryption_options={enabled=false, chunk_length_kb=64, cipher=AES/CBC/PKCS5Padding, key_alias=testing:1, key_provider=[{class_name=org.apache.cassandra.security.JKSKeyProvider, parameters=[{keystore=conf/.keystore, keystore_password=cassandra, store_type=JCEKS, key_password=cassandra}]}]}; trickle_fsync=false; trickle_fsync_interval_in_kb=10240; truncate_request_timeout_in_ms=60000; windows_timer_interval=1; write_request_timeout_in_ms=2000]

ERROR 19:00:07 Fatal configuration error

org.apache.cassandra.exceptions.ConfigurationException: Invalid yaml. Please remove properties [tracetype_query_ttl, credentials_validity_in_ms, windows_timer_interval, enable_scripted_user_defined_functions, tracetype_repair_ttl, roles_validity_in_ms, disk_optimization_strategy, enable_user_defined_functions, concurrent_materialized_view_writes, transparent_data_encryption_options, hints_flush_period_in_ms, role_manager, max_hints_file_size_in_mb, batch_size_fail_threshold_in_kb] from your cassandra.yaml

at org.apache.cassandra.config.YamlConfigurationLoader$MissingPropertiesChecker.check(YamlConfigurationLoader.java:162) ~[riemann-cassandra-0.0.3.jar:na]

at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:115) ~[riemann-cassandra-0.0.3.jar:na]

at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:84) ~[riemann-cassandra-0.0.3.jar:na]

at org.apache.cassandra.config.DatabaseDescriptor.loadConfig(DatabaseDescriptor.java:161) ~[riemann-cassandra-0.0.3.jar:na]

at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:136) ~[riemann-cassandra-0.0.3.jar:na]

at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:168) [riemann-cassandra-0.0.3.jar:na]

at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:564) [riemann-cassandra-0.0.3.jar:na]

at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:653) [riemann-cassandra-0.0.3.jar:na]

Invalid yaml. Please remove properties [tracetype_query_ttl, credentials_validity_in_ms, windows_timer_interval, enable_scripted_user_defined_functions, tracetype_repair_ttl, roles_validity_in_ms, disk_optimization_strategy, enable_user_defined_functions, concurrent_materialized_view_writes, transparent_data_encryption_options, hints_flush_period_in_ms, role_manager, max_hints_file_size_in_mb, batch_size_fail_threshold_in_kb] from your cassandra.yaml

Still working through it myself.  I have removed those settings in the yaml file (although, I don't feel all that comfortable removing the configs) and it does startup and metrics are coming back.  I am a little worried about removing those configs though, anyone else played with that PR?

[Steven McCord]

Just as a follow up on this, I decided to not go with the BlueMountain library, as the removal of configs worried me.  I went with the standard metrics-core approach and utilize the PR mentioned here:

https://github.com/addthis/metrics-reporter-config/pull/27#issuecomment-252359415

Just in case anyone stumbles across this.