cassandra problem

[Marios-Evaggelos Kogias]

Hello everyone,

I have been experimenting with Zipkin for some time but only for testing
purposes using SQLite. So, I decided to use it with Cassandra as well.
However, I face the following problem.

I set up a remote cassandra node. I started the collector and query
services as described in the guide with

bin/sbt 'project zipkin-collector-service' 'run -f
zipkin-collector-service/config/collector-cassandra.scala
bin/sbt 'project zipkin-query-service' 'run -f
zipkin-query-service/config/query-cassandra.scala

In the collector-cassandra.scala and query-cassandra.scala files I changed the
following line

val keyspaceBuilder = cassandra.Keyspace.static(nodes = Set(<cassandra
host ip>))

Is there anything else I have to configure before starting zipkin?

Everything worked well and I managed to send data to the collector which
end up to cassandra. However, when I try to browse these traces from the
UI it is not possible. Although the name of the services is fetched and
appears correctly in the UI, when I press Find Traces the request is
never completed and the browser keeps waiting for the response. I
checked with tcpdump that there is communication between the zipkin and
the cassandra node but the results never appear in the UI.

Any ideas about?

Thanks in advance,
Marios

[Radu Brumariu]

You can either add -log.level='DEBUG' or start query/collector in foreground and see if it logs any errors.

I believe that you can start the collector with -zipkin.store.cassie.dest=<host1:port1,host2:port2>, although I am using a span store, rather than the builder, so this option may not be available. You can try passing a -help flag and check if you have that in your build. 

For the query, I had to do the same trick you did in the config file.

Since it gets the service name, it means that it talks to the database, so there may be something else going on. Enabling the debug mode might help.

Radu

--

---
You received this message because you are subscribed to the Google Groups "zipkin-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zipkin-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Marios-Evaggelos Kogias]

Thanks for your help. I tried what you said and these are the logs I
get when I press Find Traces

For the query-service:

[error] DEB [20140821-10:36:15.230] query: getTraceIds:
QueryRequest(service a,None,None,None,1408606299297000,100,None)
[error] DEB [20140821-10:36:15.232] query: List()
[error] DEB [20140821-10:36:15.244] query: getSpanNames
[error] DEB [20140821-10:36:15.247] cassie: multiget_slice(Zipkin,
[java.nio.HeapByteBuffer[pos=0 lim=13 cap=13], java.nio.HeapByte
Buffer[pos=0 lim=13 cap=13], java.nio.HeapByteBuffer[pos=0 lim=13
cap=13], java.nio.HeapByteBuffer[pos=0 lim=13 cap=13], java.nio.H
eapByteBuffer[pos=0 lim=13 cap=13], java.nio.HeapByteBuffer[pos=0
lim=13 cap=13], java.nio.HeapByteBuffer[pos=0 lim=13 cap=13], jav
a.nio.HeapByteBuffer[pos=0 lim=13 cap=13],
java.nio.HeapByteBuffer[pos=0 lim=13 cap=13],
java.nio.HeapByteBuffer[pos=0 lim=13 cap=1
3]], ColumnParent(column_family:SpanNames),
SlicePredicate(slice_range:SliceRange(start:, finish:, reversed:false,
count:2147483647
)), ONE)
[error] DEB [20140821-10:36:15.251] query: getServiceNames
[error] DEB [20140821-10:36:15.235] cassie: multiget_slice(Zipkin,
[java.nio.HeapByteBuffer[pos=0 lim=13 cap=13], java.nio.HeapByte
Buffer[pos=0 lim=13 cap=13], java.nio.HeapByteBuffer[pos=0 lim=13
cap=13], java.nio.HeapByteBuffer[pos=0 lim=13 cap=13], java.nio.H
eapByteBuffer[pos=0 lim=13 cap=13], java.nio.HeapByteBuffer[pos=0
lim=13 cap=13], java.nio.HeapByteBuffer[pos=0 lim=13 cap=13], jav
a.nio.HeapByteBuffer[pos=0 lim=13 cap=13],
java.nio.HeapByteBuffer[pos=0 lim=13 cap=13],
java.nio.HeapByteBuffer[pos=0 lim=13 cap=1
3]], ColumnParent(column_family:ServiceNameIndex),
SlicePredicate(slice_range:SliceRange(start:00 05 01 1E B6 19 14 E8,
finish:, re
versed:true, count:100)), ONE)
[error] DEB [20140821-10:36:15.261] cassie: multiget_slice(Zipkin,
[java.nio.HeapByteBuffer[pos=0 lim=16 cap=16], java.nio.HeapByte
Buffer[pos=0 lim=16 cap=16], java.nio.HeapByteBuffer[pos=0 lim=16
cap=16], java.nio.HeapByteBuffer[pos=0 lim=16 cap=16], java.nio.H
eapByteBuffer[pos=0 lim=16 cap=16], java.nio.HeapByteBuffer[pos=0
lim=16 cap=16], java.nio.HeapByteBuffer[pos=0 lim=16 cap=16], jav
a.nio.HeapByteBuffer[pos=0 lim=16 cap=16],
java.nio.HeapByteBuffer[pos=0 lim=16 cap=16],
java.nio.HeapByteBuffer[pos=0 lim=16 cap=1
6]], ColumnParent(column_family:ServiceNames),
SlicePredicate(slice_range:SliceRange(start:, finish:, reversed:false,
count:2147483
647)), ONE)
[error] DEB [20140821-10:36:15.331] query: getTraceSummariesByIds.
traceIds: ArrayBuffer(1239964648555775874, 1566321669474179839,
4429831061516081586, 3466932057606665876, 2502538111986814945,
4307922318454050743, 3412541244362807976, 2147812527324170628, 60798
79533378658888, 2045999241244202895) adjust ArrayBuffer(TimeSkew)
[error] DEB [20140821-10:36:15.336] cassie: multiget_slice(Zipkin,
[1566321669474179839, 4307922318454050743, 2147812527324170628,
3412541244362807976, 2045999241244202895, 3466932057606665876,
1239964648555775874, 6079879533378658888, 4429831061516081586, 25025
38111986814945], ColumnParent(column_family:Traces),
SlicePredicate(slice_range:SliceRange(start:, finish:, reversed:false,
count:1
00000)), ONE)

Also asynchronously for both the query-service and the collector
service I get the following:

error] INF [20140821-10:37:47.732] cassie: Mapping cluster...
[error] FAT [20140821-10:37:47.771] cassie: error mapping ring
[error] FAT [20140821-10:37:47.771] cassie:
com.twitter.finagle.FailedFastException
[error] FAT [20140821-10:37:47.771] cassie: at
com.twitter.finagle.NoStacktrace(Unknown Source)
[error] INF [20140821-10:47:47.732] cassie: Mapping cluster...
[error] FAT [20140821-10:47:47.750] cassie: error mapping ring
[error] FAT [20140821-10:47:47.750] cassie:
com.twitter.finagle.FailedFastException
[error] FAT [20140821-10:47:47.750] cassie: at
com.twitter.finagle.NoStacktrace(Unknown Source)

What is the last error?

Thanks in advance

Marios

[Radu Brumariu]

Quick check on the cassie repo, looks like a cassandra failure : 

Here is the pertinent line : https://github.com/twitter/cassie/blob/f6da4e044a8f44f73bfc3376c05660f732a1e01e/cassie-core/src/main/scala/com/twitter/cassie/ClusterRemapper.scala#L76

Check if your cluster is properly configured ( if you have more than 1 box in your cassandra setup ).