Secondary indices don't work with IntegerSerializer due to padded zeros

[tcn]

https://github.com/rantav/hector/issues#issue/126

[Ran Tavory]

What do you guys think?

@tcn you make a good point that this should be fixed on the server, by any chance did you bring it up on users@cassandra?

On Tue, Jan 18, 2011 at 8:47 PM, tcn <timo.n...@gmail.com> wrote:

https://github.com/rantav/hector/issues#issue/126

--

/Ran

[tcn]

On Jan 18, 9:53 pm, Ran Tavory <ran...@gmail.com> wrote:
> @tcn you make a good point that this should be fixed on the server, by any
> chance did you bring it up on users@cassandra?

Sure, but no response so far...

[tcn]

Err, seems to be a big/little endian issue...somewhere.

@Override
public ByteBuffer toByteBuffer(final Integer obj) {
if (obj == null) {
return null;
}
final int l = obj;
return ByteBuffer.wrap(new byte[] {
(byte)l,
(byte)(l >>> 8),
(byte)(l >>> 16),
(byte)(l >>> 24)
});
}

[default@tracking] get crawler where rc<98 and
user_agent=foo;
-------------------
RowKey: 1295426032931
=> (column=rc, value=1627389952, timestamp=1295426033110000)
=> (column=url, value=http://www/0, timestamp=1295426033109000)
=> (column=user_agent, value=foo, timestamp=1295426033093000)

1 Row Returned.

[default@tracking] get crawler where rc=97 and user_agent=foo;

0 Row Returned.
[default@tracking] get crawler where rc>97 and user_agent=foo;
-------------------
RowKey: 1295426032931
=> (column=rc, value=1627389952, timestamp=1295426033110000)
=> (column=url, value=http://www/0, timestamp=1295426033109000)
=> (column=user_agent, value=foo, timestamp=1295426033093000)

1 Row Returned.
[default@tracking] get crawler where rc>98 and user_agent=foo;

0 Row Returned.
[default@tracking] get crawler where rc=1627389952 and user_agent=foo;
-------------------
RowKey: 1295426032931
=> (column=rc, value=1627389952, timestamp=1295426033110000)
=> (column=url, value=http://www/0, timestamp=1295426033109000)
=> (column=user_agent, value=foo, timestamp=1295426033093000)

1 Row Returned.


OMG...

[Nate McCall]

Can you turn on debug logging on cassandra? (You can poke this through
StorageServiceMBean#setLogger)

[tcn]

This works as expected:

.addInsertion(now, cf, createColumn("rc", new String(new
BigInteger("97").toByteArray()), SS, SS)).execute();

[default@tracking] get crawler where user_agent=foo and rc=97;
-------------------
RowKey: 1295451825452
=> (column=rc, value=97, timestamp=1295451825634000)
=> (column=url, value=http://www/, timestamp=1295451825633000)
=> (column=user_agent, value=foo, timestamp=1295451825620000)

1 Row Returned.
[default@tracking] get crawler where user_agent=foo and rc>=97;
-------------------
RowKey: 1295451825452
=> (column=rc, value=97, timestamp=1295451825634000)
=> (column=url, value=http://www/, timestamp=1295451825633000)
=> (column=user_agent, value=foo, timestamp=1295451825620000)

1 Row Returned.
[default@tracking] get crawler where user_agent=foo and rc>=96;
-------------------
RowKey: 1295451825452
=> (column=rc, value=97, timestamp=1295451825634000)
=> (column=url, value=http://www/, timestamp=1295451825633000)
=> (column=user_agent, value=foo, timestamp=1295451825620000)

1 Row Returned.
[default@tracking] get crawler where user_agent=foo and rc<=96;

0 Row Returned.

[tcn]

On Jan 19, 4:10 pm, Nate McCall <n...@riptano.com> wrote:
> Can you turn on debug logging on cassandra? (You can poke this through
> StorageServiceMBean#setLogger)

Which class(es)?

[Nate McCall]

org.apache.cassandra unfortunately. The relevant output could come
from any of db, io, net, service and locator packages.

[tcn]

On Jan 19, 5:32 pm, Nate McCall <n...@riptano.com> wrote:
> org.apache.cassandra unfortunately. The relevant output could come
> from any of db, io, net, service and locator packages.

Maybe watching my example with a debugger makes more sense then :)

[Nate McCall]

If it is local, that is the best way.

[Nate McCall]

[Looping in Jake Luciani]
Jake,
Timo discovered an issue with number serialization to which you might
have some insight:
https://github.com/rantav/hector/issues#issue/126

and the following thrift issue seems related:
https://issues.apache.org/jira/browse/THRIFT-773

I guess we could force number types to BigInteger, but I'm afraid
there might be deeper issues. Thoughts?

On Wed, Jan 19, 2011 at 12:05 PM, tcn <timo.n...@gmail.com> wrote:
> On Jan 19, 6:07 pm, Nate McCall <n...@riptano.com> wrote:
>> If it is local, that is the best way.
>

> I meant you guys :)
>
> So, this one behaves weird:
>
>  .addInsertion(now, cf, createColumn("rc", 256, SS, new
> IntegerSerializer())).execute();
>
> which serializes to [0,0,1,0], i.e little endian.
>
> This one works perfectly:

>
>  .addInsertion(now, cf, createColumn("rc", new String(new

> BigInteger("256").toByteArray()), SS, SS)).execute();
>
> which serializes to [1,0], i.e. big endian and no padding of zeros.
>
> This is where my knowledge about thrift/cassandra/hector ends. I saw
> that the IntegerType of cassandra uses BigInteger.toByteBuffer which
> serializes big endian. When googling for thrift endian you find
> THRIFT-773.
>
> For somebody who knows cassandra and its code base better than me it
> should be much easier to figure out what's going on.
>

[Jake Luciani]

773 only relates to C++

But from http://download.oracle.com/javase/6/docs/api/java/math/BigInteger.html I see toByteArray() always serializes to big-endian

-Jake

[Ed Anuff]

Hey, just saw this thread.  I just finished modifying these to get rid of our hand-rolled interger and long serializing and deseriailzing in favor of using the methods in ByteBuffer.

ed

[tcn]

Probably rather a question to the cassandra guys but anyway. I'm
wondering how it could possibly be that

where indexed_col1=cond1 and indexed_col2=cond2

returns results while flipping the conditions doesn't. Could it be
that cassandra internally doesn't use both secondary indices but only
and always the first one and resolves the second condition by a nested
loop?