Composite Support Pull Request

[Robbie Strickland]

Hello all,

A couple months ago I submitted a pull request that adds support for composite columns, which we use heavily.  I'm wondering if someone could take a look at it, since I'm sure we're not the only ones who benefit from using composites--and because we'd like not to have to maintain our own fork indefinitely... :)  

Thanks!

Robbie

[Ryan King]

I just took a look at this and added some comments.

--
@rk / theryanking.com

[Robbie Strickland]

Thanks!  I added my own comments and made the requested changes.

[Rishi Sharma]

Hi, so has this been merged? do we have composites support in 0.23.0?

Thanks,

Rishi.

[Ryan King]

It has not been merged.

--
@rk / theryanking.com

[Robbie Strickland]

I am working on a version of this that handles the encoding (and cursing Scala's type system in the process).  Unfortunately I get build errors when trying to compile with either Maven or sbt.  I got around this in the past because my codec was a bolt-on and required no changes to the rest of the code.  This change requires the A type parameter on Column to be covariant (i.e. Column[+A, B] instead of Column[A, B]), so I need to be able to build.

My build errors...

SBT:

[warn] No .svnrepo file; no svn repo will be configured.

[info] Standard project rules 1.0.4 loaded (2012-03-22).

[warn] No .svnrepo file; no svn repo will be configured.

[info] Standard project rules 1.0.4 loaded (2012-03-22).

[warn] No .svnrepo file; no svn repo will be configured.

[info] Standard project rules 1.0.4 loaded (2012-03-22).

[warn] No .svnrepo file; no svn repo will be configured.

[info] Standard project rules 1.0.4 loaded (2012-03-22).

[warn] No .svnrepo file; no svn repo will be configured.

[info] Building project cassie 0.20.2-SNAPSHOT against Scala 2.8.1

[info]    using Cassie with sbt 0.7.4 and Scala 2.7.7

[info] 

[info] == cassie-core / check-deps-exist ==

[info] == cassie-core / check-deps-exist ==

[info] 

[info] == cassie-core / auto-compile-thrift-java ==

[info] == cassie-core / auto-compile-thrift-java ==

[info] 

[info] == cassie-stress / check-deps-exist ==

[info] == cassie-stress / check-deps-exist ==

[info] 

[info] == cassie-hadoop / check-deps-exist ==

[info] == cassie-hadoop / check-deps-exist ==

[info] 

[info] == cassie-serversets / check-deps-exist ==

[info] == cassie-serversets / check-deps-exist ==

[info] 

[info] == cassie-core / compile ==

[info]   Source analysis: 83 new/modified, 0 indirectly invalidated, 0 removed.

[info] Compiling main sources...

[error] Note: Some input files use unchecked or unsafe operations.

[error] Note: Recompile with -Xlint:unchecked for details.

[error] /Users/rstrickland/workspace/cassie/cassie-core/src/main/scala/com/twitter/cassie/BaseColumnFamily.scala:48: value setTerminalId is not a member of object com.twitter.finagle.tracing.Trace

[error]         Trace.setTerminalId(Trace.nextId)

[error]               ^

[error] /Users/rstrickland/workspace/cassie/cassie-core/src/main/scala/com/twitter/cassie/BatchMutationBuilder.scala:62: value Void is not a member of object com.twitter.util.Future

[error]       Future.Void

[error]              ^

[error] /Users/rstrickland/workspace/cassie/cassie-core/src/main/scala/com/twitter/cassie/Cluster.scala:56: not found: value collectionAsScalaIterable

[error]     this(collectionAsScalaIterable(seedHosts).toSet, 9160, NullStatsReceiver)

[error]          ^

[error] /Users/rstrickland/workspace/cassie/cassie-core/src/main/scala/com/twitter/cassie/ClusterRemapper.scala:58: not found: value collectionAsScalaIterable

[error]           collectionAsScalaIterable(h.endpoints).map {

[error]           ^

[error] /Users/rstrickland/workspace/cassie/cassie-core/src/main/scala/com/twitter/cassie/ColumnFamily.scala:184: not found: value collectionAsScalaIterable

[error]       for (rowEntry <- collectionAsScalaIterable(rows.entrySet))

[error]                        ^

[error] /Users/rstrickland/workspace/cassie/cassie-core/src/main/scala/com/twitter/cassie/ColumnFamily.scala:231: not found: value collectionAsScalaIterable

[error]         for (rowEntry <- collectionAsScalaIterable(result.entrySet)) {

[error]                          ^

[error] /Users/rstrickland/workspace/cassie/cassie-core/src/main/scala/com/twitter/cassie/ColumnFamily.scala:284: not found: value collectionAsScalaIterable

[error]       for (key <- collectionAsScalaIterable(result.keySet)) {

[error]                   ^

[error] /Users/rstrickland/workspace/cassie/cassie-core/src/main/scala/com/twitter/cassie/CounterBatchMutationBuilder.scala:49: value Void is not a member of object com.twitter.util.Future

[error]       Future.Void

[error]              ^

[error] /Users/rstrickland/workspace/cassie/cassie-core/src/main/scala/com/twitter/cassie/CounterColumnFamily.scala:166: not found: value collectionAsScalaIterable

[error]       for (rowEntry <- collectionAsScalaIterable(rows.entrySet))

[error]                        ^

[error] /Users/rstrickland/workspace/cassie/cassie-core/src/main/scala/com/twitter/cassie/CounterColumnFamily.scala:198: not found: value collectionAsScalaIterable

[error]       for (rowEntry <- collectionAsScalaIterable(result.entrySet)) {

[error]                        ^

[error] /Users/rstrickland/workspace/cassie/cassie-core/src/main/scala/com/twitter/cassie/Keyspace.scala:76: value Void is not a member of object com.twitter.util.Future

[error]     if (batches.size == 0) return Future.Void

[error]                                          ^

[error] /Users/rstrickland/workspace/cassie/cassie-core/src/main/scala/com/twitter/cassie/SuperCounterBatchMutationBuilder.scala:33: value Void is not a member of object com.twitter.util.Future

[error]       Future.Void

[error]              ^

[error] /Users/rstrickland/workspace/cassie/cassie-core/src/main/scala/com/twitter/cassie/SuperCounterColumnFamily.scala:81: not found: value collectionAsScalaIterable

[error]       for (rowEntry <- collectionAsScalaIterable(result.entrySet)) {

[error]                        ^

[error] /Users/rstrickland/workspace/cassie/cassie-core/src/main/scala/com/twitter/cassie/codecs/Codec.scala:18: collectionAsScalaIterable is not a member of scala.collection.JavaConversions

[error] import scala.collection.JavaConversions.collectionAsScalaIterable

[error]        ^

[error] /Users/rstrickland/workspace/cassie/cassie-core/src/main/scala/com/twitter/cassie/codecs/Codec.scala:41: not found: value collectionAsScalaIterable

[error]     for (value <- collectionAsScalaIterable(values))

[error]                   ^

[error] /Users/rstrickland/workspace/cassie/cassie-core/src/main/scala/com/twitter/cassie/codecs/LegacyUtf8Codec.scala:24: overloaded method constructor deprecated with alternatives:

[error]   ()deprecated <and>

[error]   (message: String)deprecated

[error]  cannot be applied to (java.lang.String, java.lang.String)

[error] @deprecated("""Use the new Utf8Codec if you can. You may need to use this for backwards

[error]  ^

[error] /Users/rstrickland/workspace/cassie/cassie-core/src/main/scala/com/twitter/cassie/codecs/LegacyUtf8Codec.scala:28: overloaded method constructor deprecated with alternatives:

[error]   ()deprecated <and>

[error]   (message: String)deprecated

[error]  cannot be applied to (java.lang.String, java.lang.String)

[error]   @deprecated("""Use the new Utf8Codec if you can. You may need to use this for backwards

[error]    ^

[error] /Users/rstrickland/workspace/cassie/cassie-core/src/main/scala/com/twitter/cassie/codecs/LegacyUtf8Codec.scala:32: overloaded method constructor deprecated with alternatives:

[error]   ()deprecated <and>

[error]   (message: String)deprecated

[error]  cannot be applied to (java.lang.String, java.lang.String)

[error]   @deprecated("""Use the new Utf8Codec if you can. You may need to use this for backwards

[error]    ^

[error] 23 errors found

[info] == cassie-core / compile ==

[error] Error running compile: Compilation failed

[info] 

[info] Total time: 30 s, completed Nov 29, 2012 4:42:14 PM

[info] 

[info] Total session time: 35 s, completed Nov 29, 2012 4:42:14 PM

[error] Error during build.

Maven:

[ERROR] Plugin com.twitter:maven-build-properties-plugin:0.0.2 or one of its dependencies could not be resolved: Failed to read artifact descriptor for com.twitter:maven-build-properties-plugin:jar:0.0.2: Could not transfer artifact com.twitter:maven-build-properties-plugin:pom:0.0.2 from/to artifactory (http://artifactory.local.twitter.com/repo): Error transferring file: artifactory.local.twitter.com: Unknown host artifactory.local.twitter.com -> [Help 1]

I have messed around with it a bit, but only succeeded in creating different problems.  Any help is appreciated!

Robbie

[Robbie Strickland]

After messing with this a bit more, I resolved the conversion issues by changing the JavaConversions usages to JavaConverters, which appears to be more accepted Twitter style anyway.  I'll include those changes in the update to the pull request.  I also dug into the SBT compile errors related to missing Future.Void and call to Trace.setTerminalId(), which seem to be stemming from these dependency conflicts:

Warning: org.apache.ivy.plugins.conflict.StrictConflictException: com.twitter#finagle-core;2.0.0 (needed by [com.twitter.common#service-thrift;0.0.22]) conflicts with com.twitter#finagle-core;3.0.0 (needed by [com.twitter#finagle-serversets;3.0.0])

Warning: org.apache.ivy.plugins.conflict.StrictConflictException: com.twitter#util-core;3.0.0 (needed by [com.twitter#finagle-core;3.0.0, com.twitter#util-hashing;3.0.0]) conflicts with com.twitter#util-core;1.8.18 (needed by [com.twitter#util-collection;1.8.18, com.twitter#util-thrift;1.8.18, com.twitter#util-eval;1.8.18, com.twitter#util-hashing;1.8.18, com.twitter#util-reflect;1.8.18, com.twitter#util;1.8.18, com.twitter#util-logging;1.8.18, com.twitter.common#service-thrift;0.0.22])

I'm trying to decipher the SBT build maze, but there's a lot going on there that I'm unfamiliar with.  I'd appreciate any pointers to help resolve these conflicts, like how is this resolved in the official Twitter build?

Thanks,

Robbie

[Ryan King]

Sorry about the sbt problems. We're actually not using sbt anymore.
This thread made me realize that we hadn't pushed to github in awhile.
I just pushed a bunch of updates. Sorry.

-ryan
--
@rk / theryanking.com

[Robbie Strickland]

Is there a public version of artifactory.local.twitter.com?

[Ryan King]

On Mon, Dec 3, 2012 at 7:00 AM, Robbie Strickland
<rostri...@gmail.com> wrote:
> Is there a public version of artifactory.local.twitter.com?

All of our public code should be on maven.twttr.com, but it appears
that we still have references to our internal artifactory repository
on github. I'll try and get that fixed.

-ryan

[Robbie Strickland]

Any update on this?  I plan to wait until you resolve the build issues before moving forward on this.  Thanks!

[Ryan King]

On Wed, Dec 12, 2012 at 12:34 PM, Robbie Strickland
<rostri...@gmail.com> wrote:
> Any update on this? I plan to wait until you resolve the build issues
> before moving forward on this. Thanks!

Sorry. Solving this is requiring some internal yak shaving against a
moving target. So, it might take a bit to get a solid solution for
building cassie in the open source world.

In the the meantime, you should be able to find a workaround by doing
these things:

1. use mvn, not sbt
2. edit the pom files to replace references to artifactory with maven.twttr.com
3. in cases where we depend on SNAPSHOT versions, either build those
and install them locally or edit the version to be the latest stable
version

Let us know if that doesn't work for you and I'll see what else we can
do in the short term.

-ryan

[Robbie Strickland]

Ryan,

I've updated the pull request with a version that I think will address your concerns, and I've explained everything there.  Have a look when you get a moment...

Thanks,

Robbie