Re: TenGen JSON format ever since upgrade to play-salat 1.4.0 (still on salat 0.0.8-SNAPSHOT)

119 views
Skip to first unread message

rktoomey

unread,
Jun 20, 2012, 8:37:02 AM6/20/12
to scala-salat
Hi Juan,

I apologise but your post is extremely hard for me to read. When you
want to provide code samples - which it's so excellent that you did! -
could you please create a gist instead so there's syntax highlighting
and I can get to the raw version if I want to investigate myself?

Salat does not create" TenGen JSON". That's just what the results
look like when you query in the mongodb shell (http://www.mongodb.org/
display/DOCS/Mongo+Extended+JSON).

I think your real problem is that your docs had "_id" in the first
example and now has "id", which is probably a simple annotation import
issue with @Key wherever you defined MegaPoll, Poll and ResponseUser.

Make sure that your @Key("_id") annotation is being properly applied
to the getter by being imported as:

import com.novus.salat.annotations._

instead of something else? Sometimes your IDE tries to be "helpful"
by importing the raw annotation instead. See https://github.com/novus/salat/wiki/Annotations

Thanks,
Rose

On Jun 20, 5:11 am, opyate <opy...@gmail.com> wrote:
> Hi guys,
>
> I think the issue is more a Salat one than a Play! one, so I post here.
>
> 3 weeks ago, my code worked and wrote entries like this into MongoDB:
>
> {
>     "_id": {
>         "$oid": "4fe0eec8e4b03ed92509d976"
>     },
>     "poll": {
>         "_id": 8,
>         "name": "",
>         "created": 1340141256707,
>         "question": "Poll from b...@example.org",
>         "option_a": "Yup",
>         "option_b": "Nope",
>         "expires": 1340227656476,
>         "short_url": "http://plrz.me/KpAGvU",
>         "user": {
>             "_id": 17,
>             "name": "Bob",
>             "mobile": "",
>             "created": 1340141139094,
>             "avatarURL":
> "https://secure.gravatar.com/avatar/10ac39056a4b6f1f6804d724518ff2dc?s..."
>         }
>     },
>     "votes": []
>
> }
>
> Since I use SNAPSHOT versions of Salat (implicitly via play-salat plugin),
> the code must've changed in the interim, because I started getting a
> compile error:
>
>   1 package util
>   2
>   3 import play.api.Play
>   4 import com.mongodb.casbah._
>   5 import play.api.Play.current
>   6 import com.novus.salat._
>   7 import com.novus.salat.dao._
>   8 import com.mongodb.casbah.Imports._
>   9 import se.radley.plugin.salat._
>  10 import controllers.ResponsePoll
>  11 import controllers.ResponseUser
>  12 import controllers.ResponseVote
>  13 import models.PollResults
>  14 import models.Poll
>  15
>  16 import com.mongodb.casbah.Imports._
>  17
>  18 object MegaPoll extends ModelCompanion[MegaPoll, ObjectId] {
>  19
>  20   val collection = mongoCollection("polls")
>  21   val dao = new SalatDAO[MegaPoll, ObjectId](collection = collection) {}
>  22
>  23   def findOneByPollId(pollId: Long): Option[MegaPoll] =
> dao.findOne(MongoDBObject("poll._id" -> pollId))
>  24
>  25   /**
>  26    * Update the MegaPoll by Mongo Object ID.
>  27    *
>  28    * If the latter wasn't provided, we get it from the database.
>  29    */
>  30   def update(updated: MegaPoll, mongoObjectId: Option[String]) =
> mongoObjectId match {
>  31
>  32     case Some(id) => {
>  33       import com.mongodb.casbah.commons.conversions.scala._
>  34       RegisterJodaTimeConversionHelpers()
>  35       dao.update(MongoDBObject("_id" -> new ObjectId(id)), updated,
> false, false, new WriteConcern)
>  36     }
>  37     case None => {
>  38       Poll.findById(updated.poll.id) match {
>  39         case Some(poll) if poll.mongo_object_id.isDefined => {
>  40           dao.update(MongoDBObject("_id" -> new
> ObjectId(poll.mongo_object_id.get)), updated, false, false, new
> WriteConcern)
>  41         }
>  42         case None => play.Logger.error("The object ID should have been
> saved when the poll=%s was originally created." format updated.poll.id)
>  43       }
>  44     }
>  45   }
>  46 }
>  47
>  48 case class MegaPoll(poll: ResponsePoll, results: Option[PollResults],
> votes: List[ResponseVote])
>
> And the errors were:
>
> [info] Compiling 24 Scala sources and 1 Java source to
> /target/scala-2.9.1/classes...
> [error] /app/util/Mongo.scala:35: ambiguous reference to overloaded
> definition,
> [error] both method update in class SalatDAO of type [A, B](q: A, o: B,
> upsert: Boolean, multi: Boolean, wc: com.mongodb.WriteConcern)(implicit
> evidence$6: A => com.mongodb.DBObject, implicit evidence$7: B =>
> com.mongodb.DBObject)Unit
> [error] and  method update in trait BaseDAOMethods of type [A](q: A, t:
> util.MegaPoll, upsert: Boolean, multi: Boolean, wc:
> com.mongodb.WriteConcern)(implicit evidence$8: A =>
> com.mongodb.DBObject)Unit
> [error] match argument types
> (com.mongodb.casbah.commons.Imports.DBObject,util.MegaPoll,Boolean,Boolean, com.mongodb.WriteConcern)
> [error]       dao.update(MongoDBObject("_id" -> new ObjectId(id)), updated,
> false, false, new WriteConcern)
> [error]           ^
> [error] /app/util/Mongo.scala:40: ambiguous reference to overloaded
> definition,
> [error] both method update in class SalatDAO of type [A, B](q: A, o: B,
> upsert: Boolean, multi: Boolean, wc: com.mongodb.WriteConcern)(implicit
> evidence$6: A => com.mongodb.DBObject, implicit evidence$7: B =>
> com.mongodb.DBObject)Unit
> [error] and  method update in trait BaseDAOMethods of type [A](q: A, t:
> util.MegaPoll, upsert: Boolean, multi: Boolean, wc:
> com.mongodb.WriteConcern)(implicit evidence$8: A =>
> com.mongodb.DBObject)Unit
> [error] match argument types
> (com.mongodb.casbah.commons.Imports.DBObject,util.MegaPoll,Boolean,Boolean, com.mongodb.WriteConcern)
> [error]           dao.update(MongoDBObject("_id" -> new
> ObjectId(poll.mongo_object_id.get)), updated, false, false, new
> WriteConcern)
> [error]               ^
> [error] two errors found
>
> Fast forward to today. I've since upgraded to play-salat 1.4.0 which still
> uses salat 0.0.8-SNAPSHOT.
>
> My model now looks like this (and also compiles without errors):
>
>  1 package util.mongo
>   2
>   3 import play.api.Play
>   4 import com.mongodb.casbah._
>   5 import play.api.Play.current
>   6 import com.novus.salat._
>   7 import com.novus.salat.dao.ModelCompanion
>   8 import com.novus.salat.dao.SalatDAO
>   9 import com.mongodb.casbah.Imports._
>  10 import se.radley.plugin.salat._
>  11 import controllers.ResponsePoll
>  12 import controllers.ResponseUser
>  13 import controllers.ResponseVote
>  14 import models.PollResults
>  15 import models.Poll
>  16
>  17 import com.mongodb.casbah.Imports._
>  18
>  19 import globals.ctx
>  20
>  21 object MegaPoll extends ModelCompanion[MegaPoll, ObjectId] {
>  22
>  23   val collection = mongoCollection("polls")
>  24   val dao = new SalatDAO[MegaPoll, ObjectId](collection = collection) {}
>  25
>  26   def findOneByPollId(pollId: Long): Option[MegaPoll] =
> dao.findOne(MongoDBObject("poll.id" -> pollId))
>  27
>  28   /**
>  29    * Update the MegaPoll by Mongo Object ID.
>  30    *
>  31    * If the latter wasn't provided, we get it from the database.
>  32    */
>  33   def update(updated: MegaPoll, mongoObjectId: Option[String]): Unit =
> mongoObjectId match {
>  34
>  35     case Some(id) => {
>  36       import com.mongodb.casbah.commons.conversions.scala._
>  37       RegisterJodaTimeConversionHelpers()
>  38       val o = MongoDBObject("$set" -> MongoDBObject("poll" -> updated))
>  39       dao.update(MongoDBObject("_id" -> new ObjectId(id)), o, false,
> false, new WriteConcern)
>  40     }
>  41     case None => {
>  42       Poll.findById(updated.poll.id) match {
>  43         case Some(poll) if poll.mongo_object_id.isDefined => {
>  44           val o = MongoDBObject("$set" -> MongoDBObject("poll" ->
> updated))
>  45           dao.update(MongoDBObject("_id" -> new
> ObjectId(poll.mongo_object_id.get)), o, false, false, new WriteConcern)
>  46         }
>  47         case None => play.Logger.error("The object ID should have been
> saved when the poll=%s was originally created." format updated.poll.id)
>  48       }
>  49     }
>  50   }
>  51 }
>  52
>  53 case class MegaPoll(poll: ResponsePoll, results: Option[PollResults],
> votes: List[ResponseVote])
>
> Note the changes I made to overcome the compile errors from before.
> However, I don't know if this is entirely correct, because it now writes
> the following TenGen-formatted JSON to the MongoDB instance, which is NOT
> what I want:
>
> {
>   "_id":ObjectId("4fe18997286f6aca172f95ba"),
>   "_typeHint":"util.mongo.MegaPoll",
>   "poll":{
>     "_typeHint":"controllers.ResponsePoll",
>     "id":NumberLong(1),
>     "name":"Posted via API",
>     "created":NumberLong("1340180885486"),
>     "question":"Was this created in scenario 0?",
>     "option_a":"Sure was!",
>     "option_b":"I don't know.",
>     "expires":NumberLong("1338220191555"),
>     "short_url":"http://localhost:9000/alpha/#polls/1",
>     "user":{
>       "_typeHint":"controllers.ResponseUser",
>       "id":NumberLong(2),
>       "name":"Juan M Uys",
>       "mobile":"+447702783956",
>       "created":NumberLong("1340180884399"),
>       "avatarURL":"https://secure.gravatar.com/avatar/c22bbf2aac2a3841d80f5363d73e3ebe?s..."
>     }
>   },
>   "votes":[
>
>   ]}
>
> Note how $oid is now ObjectId, and longs are NumberLong.
>
> How do I tell Salat to use the old format? Because I have loads of data in the old format, and the new code won't read it.
>
> Thank you,
>
> Juan

rktoomey

unread,
Jun 20, 2012, 2:17:49 PM6/20/12
to scala-salat
Glad to hear it!

If that ClassCastException happens again and you can repro, I have
been trying to track it down just... forever. I think it's fixed, and
then someone else reports it but we can't get a working repro for me
to troubleshoot.

Best,
Rose

On Jun 20, 1:36 pm, opyate <opy...@gmail.com> wrote:
> OK, the issue went away. I don't know if an older cached SNAPSHOT JAR was
> the culprit or what.
> I did follow the 4 point troubleshooting plan on the wiki many times over:
>
> Troubleshooting
>
>    - first, restart
>    - second, clean and then restart
>    - third, try using ModelCompanion
>    - as a last resort, you can brute force the issue by adding a call to
>    Context#clearAllGraters() in GlobalSettings#onStart
>
> Thanks for your help, Rose!

rktoomey

unread,
Jun 20, 2012, 2:48:05 PM6/20/12
to scala-salat
Thanks :)

A note on _id vs id -

You don't have to care about "_id" vs "id", although you may need to
add an index for "id" to improve performance  as "_id" is always
indexed but "id" is not.

If you really truly never care about an auto-generated ObjectId and
only care about your own id, then it is more space efficient to use
@Key("_id") on your id field - but whatever works for you.

On Jun 20, 9:01 am, opyate <opy...@gmail.com> wrote:
> On Wednesday, June 20, 2012 1:37:02 PM UTC+1, rktoomey wrote:
>
> > Hi Juan,
>
> Hi Rose, congrats with the wee one, BTW :-)
>
>
>
> > I apologise but your post is extremely hard for me to read.  When you
> > want to provide code samples - which it's so excellent that you did! -
> > could you please create a gist instead so there's syntax highlighting
> > and I can get to the raw version if I want to investigate myself?
>
> I added both revisions of my model here:https://gist.github.com/2959764
>
>
>
> > Salat does not create" TenGen JSON".  That's just what the results
> > look like when you query in the mongodb shell (http://www.mongodb.org/
> > display/DOCS/Mongo+Extended+JSON<http://www.mongodb.org/display/DOCS/Mongo+Extended+JSON>).
>
> Thanks, this is good to know. It makes sense for the data to be in one
> format in the DB.
>
>
>
> > I think your real problem is that your docs had "_id" in the first
> > example and now has "id", which is probably a simple annotation import
> > issue with @Key wherever you defined MegaPoll, Poll and ResponseUser.
>
> I don't care about Mongo IDs in this instance. My polls' IDs are generated
> elsewhere, and those are the IDs I want to query on (i.e. "poll.id"). The
> fact that the poll's ID was previously persisted as "_id" wasn't something
> I did explicitly. I also don't use any annotations.
>
> Please let me know if this is gross misuse of Mongo, but it helps me keep
> polls and related votes in one blob.
>
>
>
> > Make sure that your @Key("_id") annotation is being properly applied
> > to the getter by being imported as:
>
> >     import com.novus.salat.annotations._
>
> Is this still relevant to me if I only care for my own IDs?

rktoomey

unread,
Jun 21, 2012, 10:38:06 AM6/21/12
to scala-salat
Hmm, sounds like you had a custom context in your imports previously?

Type hinting is controlled by the context type hinting strategy. The
global context in com.novus.salat.global always type hints, which is
not space efficient but is a good default for people starting out with
Salat so they can make sure their data is being correctly serialized.
A better type hinting strategy for production is "when necessary",
which only type hints when it is required to successfully deserialize
(class hierarchies, collections typed to a trait or abstract
superclass). See https://github.com/novus/salat/wiki/TypeHints

As far as "_id" vs "id", although you can individually annotate fields
with @Key, you can also apply the alias at the context level. See the
section on Global key remapping in https://github.com/novus/salat/wiki/CustomContext

Probably you used to import your custom context and now you are
importing com.novus.salat.global._.

Best,
Rose

On Jun 21, 4:02 am, opyate <opy...@gmail.com> wrote:
> On Wednesday, June 20, 2012 7:48:05 PM UTC+1, rktoomey wrote:
>
> > Thanks :)
>
> > A note on _id vs id -
>
> > You don't have to care about "_id" vs "id", although you may need to
> > add an index for "id" to improve performance  as "_id" is always
> > indexed but "id" is not.
>
> > If you really truly never care about an auto-generated ObjectId and
> > only care about your own id, then it is more space efficient to use
> > @Key("_id") on your id field - but whatever works for you.
>
> Ah, thanks for the valuable tips. One other thing I've noticed ever since
> the issues previously discussed was that the IDs for my old data seemed to
> have the @Key annotation automatically applied (i.e. all IDs are saved as
> "_id"), but the new data just uses "id" (but now also with a "_typeHint"
> field). It's almost as if a whole lot of defaults got changed. (You can see
> this from the two blobs of JSON I pasted in the first post.)
>
> How can I have @Key automatically applied, or how can I always have my case
> class' "id" converted to an "_id" using Salat?
Reply all
Reply to author
Forward
0 new messages