Call for testers: Rogue v2.0.0 release imminent

[Jorge Ortiz]

Hey everyone,

We're getting ready to release Rogue v2.0.0 and we'd love it if people took it for a spin before we mark it final.

There have been quite a few breaking changes from Rogue v1.1.x, but we think the vast majority of the breaking changes are done. With your help, we'd like to use this testing period to smooth out any rough edges and put together a migration document before the final release.

The code is available on the "v2" branch on Github:

https://github.com/foursquare/rogue/tree/v2

Binary artifacts are available on Sonatype and Maven Central. 

group: com.foursquare

artifacts: rogue-field, rogue-core, and rogue-lift

latest version: 2.0.0-beta14

If you have any questions, feedback, or comments about the migration, please email the list. We'd like to gather as much info as possible for the migration document.

Thanks!

--j

Below are some notes on some of the major changes:

Sub-projects

Previously, we released a single "rogue" artifact. This has been split up into three sub-projects: rogue-field, rogue-core, and rogue-lift. Rogue v1.x relied on Lift's Record library as it's object-document mapper (ODM). One of the major goals for Rogue v2 was to make it agnostic to the underlying ODM framework.

• rogue-field - A very minimal project (one file, three traits, ten lines of code) that describes the minimal field interface Rogue needs. If you want to use Rogue with a non-Lift ODM, you'll need your document fields to implement the traits in this project (or you'll provide your own set of implicits, as we do with Lift).
• rogue-core - The core of Rogue, which defined Query and operations on queries, but is agnostic to the underlying ODM framework and execution engine.
• rogue-lift - An implementation of Rogue for Lift's Record library, including execution engine.

Foursquare is working on our ODM to replace Lift Record, which will include integration with Thrift and code generation. We hope to open source this project (along with the necessary Rogue integration) in the future. For now though, rogue-lift is the only version of Rogue usable out-of-the-box. If you're porting from an existing Rogue 1.x project, you'll probably want to depend on rogue-lift (and transitively, the other two projects). If you're looking to use Rogue with an alternative ODM, you should only need to depend on rogue-field and rogue-core.

QueryExecutors vs executable Queries

In Rogue v1.x, queries could be executed by calling .fetch(), .count(), etc on the Query. In Rogue v2, we're taking an approach that is slightly friendlier to testing and dependency injection, by separating the definition of a query from the execution of a query.

Methods that touch the database are no longer available by default in the Query class itself. Instead, an instance of QueryExecutor has these methods and can take a Query as an argument.

Before:

val query = Venue.where(_.venuename eqs "Starbucks")
query.count()
query.fetch()

After:

val db: QueryExecutor[...] = ...
val query = Venue.where(_.venuename eqs "Starbucks")
db.count(query)
db.fetch(query)

This lets us get rid of several singletons and static dependencies in Rogue itself, and allows our application code to be easier to test.

For those who prefer the previous syntax, we've included an ExecutableQuery class that will take a Query and a QueryExecutor and make the previous form available again. An implicit conversion from Query to ExecutableQuery can make the previous syntax available seamlessly on Query. One such implicit (which relies on a singleton LiftQueryExecutor) is included by default in LiftRogue, but may be removed in the future.

Single Phantom Type Parameter

Previously, Query had one type parameter for each of it's phantom types: Ordered, Selected, Limited, Skipped, HasOrClause, etc. This was rather unwieldily, both when declaring Query types and when trying to add a new phantom type.

In Rogue v2, Query has a single type parameter for all the phantom types. So instead of having a Query[Venue, Venue, Selected, Limited, Skipped, ...] you have a Query[Venue, Venue, Selected with Limited with Skipped with ...]. This makes some things easier and some things harder, but overall we think it's a readability, maintainability, and flexibility win.

Required parentheses

Because of how we are doing the type checking for the new single phantom type parameter, some methods now require parentheses that didn't use to. These are all methods with side-effects that you should be calling with parens as a matter of good style anyway. Some examples: .exists(), .fetch(), .get(), .bulkDelete_!!!(), .findAndDeleteOne().

Rogue vs LiftRogue

Previously, the com.foursquare.rogue.Rogue singleton contained the implicits and type aliases necessary to use Rogue. Now, the type aliases have all been moved to the com.foursquare.rogue package object, and the implicits for dealing with LIft Rogue models are in com.foursquare.rogue.LiftRogue. These are the two places you probably want to import from.

Shard Key Awareness

One new feature we've added is the ability to make rogue aware of the shard key you have configured in your collections. It's an opt-in feature, so if your collections aren't sharded, you can ignore all of this.

Basically you add "with Sharded" to your record class and "with ShardKey[T]" to the field that your collection is sharded on. Then, rogue ensures at compile time that for all queries on that collection, either (a) you specify the shard key in the query (using `withShardKey` exactly like you'd use `where`) or (b) you add .allShards to your query indicating that you expect it to hit all shards. Additionally, it enforces that a query that does not specify a shard key cannot do an updateOne or upsertOne, but rather must do an updateMulti.

This is useful to help prevent accidental all-shards queries, or at least make the developer aware of the run-time performance of their queries wrt sharding.

[Jorge Ortiz]

Hey all,

Has anyone had a chance to upgrade to the v2 beta?

If you're doing this (or planning on doing this soon) please let us know. If we don't hear from anyone we'll go ahead and push a v2 release.

--j

[Otto]

Hi,

we'll adopt it as soon as possible. Unfortunately, we can only start looking at it in 2-3 weeks.

Thanks for your great work,

Otto

[Otto]

Btw, any news about the model generation library you announced earlier?

[Mateusz Paprocki]

Hi,

On Wednesday, August 22, 2012 8:36:03 PM UTC+2, Jorge Ortiz wrote:

Hey all,

Has anyone had a chance to upgrade to the v2 beta?

If you're doing this (or planning on doing this soon) please let us know. If we don't hear from anyone we'll go ahead and push a v2 release.

I upgraded to Rogue 2.0.0-beta14 today and got this error after compilation phase in sbt (I did issue clean before compilation): 

[error] class file needed by Query is missing.

[error] reference type ReadPreference of package com.mongodb refers to nonexisting symbol.

[error] one error found

I use Scala 2.9.1 and Lift 2.4 (final) which pulls in mongo-java-driver 2.6.5. Adding explicit dependency on mongo-java-driver 2.7.3 (the same as in rogue-core/build.sbt) fixed the problem. I'm not sure if adding explicit dependency is the way to go or there is something wrong with Rogue's build.

I didn't see any other issues for now.

Mateusz 

[Jorge Ortiz]

The model generation library is still in development. Open sourcing it is a few months out at best.

--j

[Jorge Ortiz]

Hey Mateusz,

Thanks for taking the time to try the upgrade!

This is a bit of an unfortunate conflict. Lift is quite a bit behind in upgrading the Mongo library (2.6.5 was released over a year ago).

What we do is declare our dependencies on rogue-*, lift-mongodb, and lift-mongodb-record as "intransitive". Then you can declare explicit dependencies on whichever version of lift and mongo-java-driver you want.

From our sbt build (modified a bit to remove some foursquare-specific stuff):

libraryDependencies ++= Seq(

  "net.liftweb" %% "lift-mongodb" % "2.4" % "compile" withSources() intransitive(),

  "net.liftweb" %% "lift-mongodb-record" % "2.4" % "compile" withSources() intransitive(),

  "com.foursquare" %% "rogue-field" % "2.0.0-beta14" withSources() intransitive(),

  "com.foursquare" %% "rogue-core" % "2.0.0-beta14" withSources() intransitive(),

  "com.foursquare" %% "rogue-lift" % "2.0.0-beta14" withSources() intransitive(),

  "com.mongodb" % "mongo-java-driver"  % "2.7.3" withSources())

(If you mark the lift bits as intransitive, you'll probably have to explicitly declare some other lift-* dependencies if you don't already. We depend on all of them anyway so it's not an issue for us.)

I'll work with Lift to get their mongo-java-driver version upgraded, and probably start requiring a newer version of Lift at some point.

--j

[Mateusz Paprocki]

Hi,

On 23 August 2012 18:48, Jorge Ortiz <jo...@foursquare.com> wrote:

Hey Mateusz,

Thanks for taking the time to try the upgrade!

This is a bit of an unfortunate conflict. Lift is quite a bit behind in upgrading the Mongo library (2.6.5 was released over a year ago).

What we do is declare our dependencies on rogue-*, lift-mongodb, and lift-mongodb-record as "intransitive". Then you can declare explicit dependencies on whichever version of lift and mongo-java-driver you want.

From our sbt build (modified a bit to remove some foursquare-specific stuff):

libraryDependencies ++= Seq(

  "net.liftweb" %% "lift-mongodb" % "2.4" % "compile" withSources() intransitive(),

  "net.liftweb" %% "lift-mongodb-record" % "2.4" % "compile" withSources() intransitive(),

  "com.foursquare" %% "rogue-field" % "2.0.0-beta14" withSources() intransitive(),

  "com.foursquare" %% "rogue-core" % "2.0.0-beta14" withSources() intransitive(),

  "com.foursquare" %% "rogue-lift" % "2.0.0-beta14" withSources() intransitive(),

  "com.mongodb" % "mongo-java-driver"  % "2.7.3" withSources())

(If you mark the lift bits as intransitive, you'll probably have to explicitly declare some other lift-* dependencies if you don't already. We depend on all of them anyway so it's not an issue for us.)

I'll work with Lift to get their mongo-java-driver version upgraded, and probably start requiring a newer version of Lift at some point.

OK. Thanks for explanation.

Mateusz

[Alexandre Richonnier]

Hi,

Le jeudi 23 août 2012 18:48:33 UTC+2, Jorge Ortiz a écrit :

Hey Mateusz,

Thanks for taking the time to try the upgrade!

This is a bit of an unfortunate conflict. Lift is quite a bit behind in upgrading the Mongo library (2.6.5 was released over a year ago).

What we do is declare our dependencies on rogue-*, lift-mongodb, and lift-mongodb-record as "intransitive". Then you can declare explicit dependencies on whichever version of lift and mongo-java-driver you want.

From our sbt build (modified a bit to remove some foursquare-specific stuff):

libraryDependencies ++= Seq(

  "net.liftweb" %% "lift-mongodb" % "2.4" % "compile" withSources() intransitive(),

  "net.liftweb" %% "lift-mongodb-record" % "2.4" % "compile" withSources() intransitive(),

  "com.foursquare" %% "rogue-field" % "2.0.0-beta14" withSources() intransitive(),

  "com.foursquare" %% "rogue-core" % "2.0.0-beta14" withSources() intransitive(),

  "com.foursquare" %% "rogue-lift" % "2.0.0-beta14" withSources() intransitive(),

  "com.mongodb" % "mongo-java-driver"  % "2.7.3" withSources())

work well with  "org.mongodb" % "mongo-java-driver"  % "2.7.3" withSources())

Alexandre

[Scott Abernethy]

On Friday, 24 August 2012 04:48:33 UTC+12, Jorge Ortiz wrote:

What we do is declare our dependencies on rogue-*, lift-mongodb, and lift-mongodb-record as "intransitive". Then you can declare explicit dependencies on whichever version of lift and mongo-java-driver you want.

Just in case someone else misreads the above and marks lift-* as "intransitive" (not much fun)... here's my full lift 2.5 dependencies

libraryDependencies ++= {

  val liftVersion = "2.5-SNAPSHOT"

  Seq(

    "net.liftweb" %% "lift-webkit" % liftVersion % "compile",

    "net.liftweb" %% "lift-record" % liftVersion % "compile",

    "net.liftweb" %% "lift-mongodb" % liftVersion % "compile" intransitive(),

    "net.liftweb" %% "lift-mongodb-record" % liftVersion % "compile" intransitive(),

    "org.mongodb" % "mongo-java-driver" % "2.7.3",

    "net.liftmodules" %% "lift-jquery-module" % (liftVersion + "-1.0-SNAPSHOT"),

    "com.foursquare" %% "rogue-core" % "2.0.0-beta14" withSources() intransitive(),

    "com.foursquare" %% "rogue-field" % "2.0.0-beta14" withSources() intransitive(),

    "com.foursquare" %% "rogue-lift" % "2.0.0-beta14" withSources() intransitive(),

    "org.eclipse.jetty" % "jetty-webapp" % "7.5.4.v20111024"  % "container; test",

    "ch.qos.logback" % "logback-classic" % "1.0.6",

    "org.specs2" %% "specs2" % "1.11" % "test"

  )

}

It's working well.

[Viktor Hedefalk]

I'm trying to migrate just now but I'm having a lot of compile errors.
I'm trying to untangle them, but is there any migration guide?

After updating to use LiftRogue._ instead of Rogue._ I have this:

def findBySingular(name: String)(implicit meta: Meta) = meta where
(_.name.subfield(_.singular) eqs name)

def findByName(name: String)(implicit meta: Meta) =
meta.or(
findBySingular(name)(_),
_ where (_.name.subfield(_.plural) eqs name))

private def findByMisspell(name: String)(implicit meta: Meta) =
meta.where(_.name.subfield(_.misspells) contains name)

def findByNameOrMisspell(name: String) =
Food.or(findByName(name)(_), findByMisspell(name)(_))



[error] /Users/viktor/dev/projects/kostbevakningen/src/main/scala/se/kostbevakningen/model/record/Food.scala:109:
value eqs is not a member of
com.foursquare.rogue.SelectableDummyField[List[String],se.kostbevakningen.model.record.Food]
[error] def findBySingular(name: String)(implicit meta: Meta) = meta
where (_.name.subfield(_.singular) eqs name)
[error]
^
[error] /Users/viktor/dev/projects/kostbevakningen/src/main/scala/se/kostbevakningen/model/record/Food.scala:113:
type mismatch;
[error] found : x$16.type (with underlying type
com.foursquare.rogue.Query[se.kostbevakningen.model.record.Food,se.kostbevakningen.model.record.Food,com.foursquare.rogue.Ordered
with com.foursquare.rogue.Selected with com.foursquare.rogue.Limited
with com.foursquare.rogue.Skipped with
com.foursquare.rogue.HasNoOrClause])
[error] required: se.kostbevakningen.model.record.Food.Meta
[error] findBySingular(name)(_),
[error] ^
[error] /Users/viktor/dev/projects/kostbevakningen/src/main/scala/se/kostbevakningen/model/record/Food.scala:114:
value eqs is not a member of
com.foursquare.rogue.SelectableDummyField[List[String],se.kostbevakningen.model.record.Food]
[error] _ where (_.name.subfield(_.plural) eqs name))
[error] ^
[error] /Users/viktor/dev/projects/kostbevakningen/src/main/scala/se/kostbevakningen/model/record/Food.scala:119:
type mismatch;
[error] found : x$22.type (with underlying type
com.foursquare.rogue.Query[se.kostbevakningen.model.record.Food,se.kostbevakningen.model.record.Food,com.foursquare.rogue.Ordered
with com.foursquare.rogue.Selected with com.foursquare.rogue.Limited
with com.foursquare.rogue.Skipped with
com.foursquare.rogue.HasNoOrClause])
[error] required: se.kostbevakningen.model.record.Food.Meta
[error] def findByNameOrMisspell(name: String) =
Food.or(findByName(name)(_), findByMisspell(name)(_))
[error] ^
[error] /Users/viktor/dev/projects/kostbevakningen/src/main/scala/se/kostbevakningen/model/record/Food.scala:119:
type mismatch;
[error] found : x$23.type (with underlying type
com.foursquare.rogue.Query[se.kostbevakningen.model.record.Food,se.kostbevakningen.model.record.Food,com.foursquare.rogue.Ordered
with com.foursquare.rogue.Selected with com.foursquare.rogue.Limited
with com.foursquare.rogue.Skipped with
com.foursquare.rogue.HasNoOrClause])
[error] required: se.kostbevakningen.model.record.Food.Meta
[error] def findByNameOrMisspell(name: String) =
Food.or(findByName(name)(_), findByMisspell(name)(_))



Any help highly appreciated!

Thanks,
Viktor

[Viktor Hedefalk]

Ok, I found out that my problem is that the subfield query doesn't
seem to work on BsonRecordListField in 2.0. Below, the first line
compiles but not the second:

def findById(id: Int)(implicit meta: Meta) =
meta.where(_.ID.subfield(_.livsmedelsverketId) eqs id)

def findBySingular(name: String)(implicit meta: Meta) =

meta.where(_.name.subfield(_.singular) eqs name)

The difference between them being that "name" is a BsonRecordListField
while "livsmedelsverketId" is a BsonRecordField.

I have no idea how to solve it though so any help is still highly appreciated :)

Thanks,
Viktor

[Jason Liszka]

Just use contains instead of eqs on List fields.

Jason

[Viktor Hedefalk]

I'll report my own progress. I expanded all the implicits and this is
what I get for the first query:

def findById(id: Int)(implicit meta: Meta) =

metaRecordToQueryBuilder(meta).where(x =>
rintFieldtoNumericQueryField(bsonRecordFieldToBsonRecordQueryField(x.ID).subfield(y
=> liftField2Recordv2Field(y.livsmedelsverketId))) eqs id)


which still compiles.

Trying to use something similar explicitly for my BsonRecordListField
I tried this:

def findBySingular(name: String)(implicit meta: Meta) =

metaRecordToQueryBuilder(meta).where(x =>
rstringFieldToStringQueryField(bsonRecordListFieldToBsonRecordListQueryField(x.name).subfield(y
=> liftField2Recordv2Field(y.singular))) eqs name)


Here I got another compile error that might be easier to understand:

[error] /Users/viktor/dev/projects/matdagboken/src/main/scala/se/kostbevakningen/model/record/Food.scala:112:
polymorphic expression cannot be instantiated to expected type;
[error] found :
[V1]com.foursquare.rogue.SelectableDummyField[List[V1],se.kostbevakningen.model.record.Food]
[error] required: com.foursquare.field.Field[String,?]


Thanks,
Viktor

[Viktor Hedefalk]

Excellent! Thanks alot!

I kind of figured the semantics of eqs seemed a bit off on a list but
I tried to put contains outside, something like:

meta.where(_.name.contains(_.subfield(_.singular) eqs name))
or
meta.where(_.name.subfield(_.singular) eqs name)

instead of

meta.where(_.name.subfield(_.singular) contains name)


Thanks again!
/Viktor

[Jason Liszka]

meta.where(_.name.subfield(y) contains name) should compile.

[Jason Liszka]

sorry, I meant: meta.where(_.name.subfield(_.singular) contains name)

[Viktor Hedefalk]

Yep, it compiles. Thanks again!
/Viktor

[Viktor Hedefalk]

So my next issue is with a construct I have to compose queries:

type Meta = Food with MongoMetaRecord[Food]

def findBySingular(name: String)(implicit meta: Meta) =

meta.where(_.name.subfield(_.singular) contains name)

def findByName(name: String)(implicit meta: Meta) =
meta.or(
findBySingular(name)(_),

_ where (_.name.subfield(_.plural) contains name))

[error] /Users/viktor/dev/projects/kostbevakningen/src/main/scala/se/kostbevakningen/model/record/Food.scala:113:
type mismatch;
[error] found : x$16.type (with underlying type
com.foursquare.rogue.Query[se.kostbevakningen.model.record.Food,se.kostbevakningen.model.record.Food,com.foursquare.rogue.Ordered
with com.foursquare.rogue.Selected with com.foursquare.rogue.Limited
with com.foursquare.rogue.Skipped with
com.foursquare.rogue.HasNoOrClause])
[error] required: se.kostbevakningen.model.record.Food.Meta
[error] findBySingular(name)(_),

So it seems some type-info on the meta passed to the or-closures are
lost. Or its not the meta anymore but a Query, right?

I guess I could re-type my findBySingular() to take something this
QueryType instead:


type Meta = Query[Food,Food,Ordered with Selected with Limited with
Skipped with HasNoOrClause]

but then I should probably rename my type to something else. Is there
any shorter version of this? Or some other more idiomatic way to
compose?

Thanks again,

[Roch Delsalle]

Hi,

I'm currently trying an upgrade to Rogue v.2.0 on my Lift App.

I would like to know where is the best place to define my QueryExecutor , do I need to define one before each queries or can I put it in my Boot file ?

Best,

[Jason Liszka]

The easiest thing to do is to import com.foursquare.rogue.LiftRogue._. It supplies some implicit conversions that inject com.foursquare.rogue.LiftQueryExecutor for you, so you can write queries the same way as in rogue 1.x without having to worry about QueryExecutors.

If you want a custom QueryExecutor, you can define it once in your Boot file or in your service layer. Then instead of writing queries like

Model.where(_.foo eqs bar).fetch()

you would have to write

val db = MyQueryExecutor // somewhere, once, for convenience

// elsewhere

db.fetch(Model.where(_.foo eqs bar))

and the same would go for .count(), .fetchBatch(), .updateOne(), etc.

Hope that helps!

Jason

[stephanos]

I was just migrating my app from 1.x to 2.x and noticed that the useIndex method seems to be gone.

Any chance it will come back in a later release?

On Friday, August 10, 2012 8:38:00 PM UTC+2, Jorge Ortiz wrote:

[Jason Liszka]

It's just been renamed to hint(), I believe.

[stephanos]

I think hint is just a flag that gives MongoDB some heads-up, whereby useIndex was a Rogue-specific method that made sure, you used an index correctly.

Related: https://github.com/foursquare/rogue/blob/v2.0.0-beta22/rogue-lift/src/test/scala/com/foursquare/rogue/QueryTest.scala#L687

Furthermore, it seems as the Lift's UUIDField is not supported anymore:

   .modify(_.auth.subfield(_.key) setTo key)

   value setTo is not a member of com.foursquare.rogue.SelectableDummyField[java.util.UUID,MyDoc]

Maybe you could add a test case for UUID ?

[alexmnyc]

Guys, this works with "contains" on Strings as noted earlier.

 

But what is the approach to make it work with ObjectId's as in:

 

 

.....images.subfield(_.id eqs new ObjectId("......"))

 

Thank you.

[alexmnyc]

Never mind. Just answered my own question by trying it against ObjectID with contains and it works. Thank you.