Cost of a Record migration

[Stefan Schmidt]

Hello,

to my understanding the switch between different relational databases
when using Mapper is just a matter of changing a few lines. First adding
the dependency to the project description and then changing the database
connection settings in the properties file. For example if i wanted to
switch from MySQL to PostgreSQL it would require these five lines:

val lift_postgresql = "postgresql" % "postgresql" % "8.4-701.jdbc4"

db.driver=org.postgresql.Driver
db.url=jdbc:postgresql:DATABASENAME
db.user=USERNAME
db.password=PASSWORD

What is the cost if I am using Record and I want to migrate from MongoDB
to CouchDB? How many lines do I have to change? If the cost is
significantly higher, can the migration be automated?

Cheers, Stefan

[Ross Mellgren]

Is the use case a one-time migration, or being able to actively switch? Or maintaining two stores with same/similar information?

-Ross

> --
> You received this message because you are subscribed to the Google Groups "Lift" group.
> To post to this group, send email to lif...@googlegroups.com.
> To unsubscribe from this group, send email to liftweb+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/liftweb?hl=en.
>

[Stefan Schmidt]

I was thinking more of a one-time migration. Say you have developed your
application on MongoDB and after a few months you realize that CouchDB
would have been a better choice.

Stefan

Am 31.08.10 00:33, schrieb Ross Mellgren:

[Timothy Perrett]

It shouldn't be too bad right, as your fields will likely remain the same so it should just be a case of altering your implementation. Of course, the main issue would be restructuring your data extraction (queries) as those are likley to be application and backend specific. Perfectly acceptable for a 1 time migration though.

Cheers, Tim

[Stefan Schmidt]

Can you please give concrete examples of what I would have to change in
my implementation? I don't mean developer designed explicit queries, but
the parts where the framework automatically provides persistency mapping
for the models.

Stefan

Am 31.08.10 10:20, schrieb Timothy Perrett:

[David Pollak]

On Tue, Aug 31, 2010 at 4:08 AM, Stefan Schmidt <stefan....@gmx.net> wrote:

Can you please give concrete examples of what I would have to change in my implementation? I don't mean developer designed explicit queries, but the parts where the framework automatically provides persistency mapping for the models.

I think the short answer is "No."

SQL is a standard way of talking to databases and object abstractions on top of the minor differences can (and in the case of Mapper do) mask the minor differences.

NoSQL is a much more amorphous beast and I'd expect that you'll be re-writing 10% - 25% of your persistence logic if you switch between MongoDB (which allows complex documents and complex queries) and CouchDB (which is more of a key-value store, but has much better offline to online merging).
 

--
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Blog: http://goodstuff.im
Surf the harmonics

[Stefan Schmidt]

That surprises me because of two reasons:

1. In "The Definitive Guide to Lift" the authors state: "Record is a new refactorization of Mapper that is backing-store agnostic at its core, so it doesn’t matter whether you want to save your data to JDBC, Java Persistence API (JPA), or even something like XML. With Record, selecting the proper driver will be as simple as hooking the proper traits into your class."

2. MongoDB classfies itself as a "Document-oriented storage" with "JSON-style documents" and CouchDB classifies itself as "A document database server, accessible via a RESTful JSON API." which corresponds to Wikipedia classifying MongoDB and CouchDB as "Document-oriented database" with "JSON(-like) documents". Effectively MongoDB uses BSON which is a binary version of JSON, so to my understanding the document format should be the same.

I'm not sure quite sure what you mean by complex queries, but concerning CouchDB this looks quite promising to me:

http://guide.couchdb.org/draft/cookbook.html

Stefan

Am 31.08.10 14:42, schrieb David Pollak:

[David Pollak]

On Tue, Aug 31, 2010 at 6:28 PM, Stefan Schmidt <stefan....@gmx.net> wrote:

That surprises me because of two reasons:

1. In "The Definitive Guide to Lift" the authors state: "Record is a new refactorization of Mapper that is backing-store agnostic at its core, so it doesn’t matter whether you want to save your data to JDBC, Java Persistence API (JPA), or even something like XML. With Record, selecting the proper driver will be as simple as hooking the proper traits into your class."

DGL was written 18 months ago.  It was at that time, our time to unify all persistence under Record.  It turns out that Mapper is a better solution for SQL and we will continue to maintain and enhance it.  Record is being used where the query mechanics is not normalized... this is MongoDB, CouchDB, and Squeryl.  But each of the backing stores has its own query mechanism and they are not normalized.

 

2. MongoDB classfies itself as a "Document-oriented storage" with "JSON-style documents" and CouchDB classifies itself as "A document database server, accessible via a RESTful JSON API." which corresponds to Wikipedia classifying MongoDB and CouchDB as "Document-oriented database" with "JSON(-like) documents". Effectively MongoDB uses BSON which is a binary version of JSON, so to my understanding the document format should be the same.

Even if the document format is the same, the query semantics are different and we make no attempt to normalize them.

[Ross Mellgren]

On Aug 31, 2010, at 9:28 PM, Stefan Schmidt wrote:

That surprises me because of two reasons:

1. In "The Definitive Guide to Lift" the authors state: "Record is a new refactorization of Mapper that is backing-store agnostic at its core, so it doesn’t matter whether you want to save your data to JDBC, Java Persistence API (JPA), or even something like XML. With Record, selecting the proper driver will be as simple as hooking the proper traits into your class."

This is true, mostly, but you should be aware as to what Record really provides:

  1) the ability to generate web forms and

  2) perform validation on fields

  3) treat different records generically

But what it does not provide is a homogeneous query interface. The reason for this is twofold -- first, the different storage mechanisms have wildly different query semantics, and the second half is that nobody has nailed down a consistent set of useful query semantics that somehow unify those disparate backends.

That said, you can build Records that target multiple backends, something like:

trait Author[MyType <: Author[MyType]] extends Record[MyType] {

    val firstName = new StringField(this, 100)

    val lastName = new StringField(this, 100)

}

trait MetaAuthor[BaseType <: MetaAuthor[BaseType]] extends MetaRecord[BaseType] {

    def findByName(first: String, last: String): Box[Author]

}

class CouchAuthor private () extends Author[CouchAuthor] with CouchRecord {

    def meta = CouchAuthor

}

object CouchAuthor extends CouchAuthor with MetaAuthor[CouchAuthor] with MetaCouchRecord[CouchAuthor] {

    def findByName(first: String, last: String): Box[Author] = queryView("authors", "by_name", _.key(JArray(first::last::Nil))).flatMap(_.headOption)

}

class SquerylAuthor private () extends Author[SquerylAuthor] with KeyedRecord[SquerylAuthor] {

    def meta = SquerylAuthor

}

object SquerylAuthor extends SquerylAuthor with MetaAuthor[SquerylAuthor] {

    import Schema.authors

    def findByName(first: String, last: String): Box[Author] = tryo(from(authors) { a => where(a.firstName === first and a.lastName === last) select(a) }).flatMap(_.headOption)

}

Clearly this is not super-simple -- so for a one-time migration I wouldn't recommend building up this way. Instead, I'd recommend creating a one-time tool that has both a Couch and Mongo version of the same record, and just load up each from one database, copy values from one record to the other (use asJValue / fromJValue, if you like), and then store in the different database.

It's not "plug and play" though. Each database has its own requirements on fields (e.g. Couch needs _id and _rev, Squeryl needs a primary key field, and so on) and different extended fields (e.g. foreign references).

 

2. MongoDB classfies itself as a "Document-oriented storage" with "JSON-style documents" and CouchDB classifies itself as "A document database server, accessible via a RESTful JSON API." which corresponds to Wikipedia classifying MongoDB and CouchDB as "Document-oriented database" with "JSON(-like) documents". Effectively MongoDB uses BSON which is a binary version of JSON, so to my understanding the document format should be the same.

I'm not sure quite sure what you mean by complex queries, but concerning CouchDB this looks quite promising to me:

http://guide.couchdb.org/draft/cookbook.html

Couch supports complex queries, but the way it supports it is quite different from Mongo. With Couch, you define your "queries" ahead of time -- they're called views and you specify map/reduce constructs which populate the views. Views are stored in the database. You can run "ad-hoc" views, like one-off queries in SQL, but that's really not recommended. Conversely, Mongo is more about being able to run SQL-like queries on documents. (Anybody with more experience please feel free to step in and correct me)

Because of this difference, utilizing the two databases is fairly different. With Couch you really need to map out your queries ahead of time. With Mongo to some degree you have to as well to generate indexes and make things fast, but I believe it's less required.

Hope that clarifies,

-Ross

[Stefan Schmidt]

> But what it does not provide is a homogeneous query interface. The
> reason for this is twofold -- first, the different storage mechanisms
> have wildly different query semantics, and the second half is that
> nobody has nailed down a consistent set of useful query semantics that
> somehow unify those disparate backends.

If the nosql hype reaches the plateau of productivity and standard query
languages for document databases and key-value stores evolve will there
be a chance of homogeneous query interfaces for the different nosql
database types?

> Hope that clarifies

Yes, absolutely, I think I have a rough understanding of Lift
persistency now. Thanks for your helpful advice!

Cheers, Stefan

[Ross Mellgren]

On Sep 2, 2010, at 11:28 AM, Stefan Schmidt wrote:
>> But what it does not provide is a homogeneous query interface. The reason for this is twofold -- first, the different storage mechanisms have wildly different query semantics, and the second half is that nobody has nailed down a consistent set of useful query semantics that somehow unify those disparate backends.
>
> If the nosql hype reaches the plateau of productivity and standard query languages for document databases and key-value stores evolve will there be a chance of homogeneous query interfaces for the different nosql database types?

Maybe! In some respects, the variety is nice, because it allows the various folks to pursue different innovative query strategies that won't wedge into SQL or something similar. I suppose after a time we'll implement a very light abstraction for standard CRUD ops and "select all / select one by id" so that we can build things like CRUDify/CRUDops on top of Record.

-Ross

>
>> Hope that clarifies
>
> Yes, absolutely, I think I have a rough understanding of Lift persistency now. Thanks for your helpful advice!
>
> Cheers, Stefan
>

[Timothy Perrett]

You mean, like, perhaps a structured query language? ;-) Cant think where i've heard of that before! Personally I doubt this will happen, as they are solving different problems, with different approaches. Its VHS vs Betamax. Someone will just plain win out in the end and the others will disappear into oblivion.

Cheers, Tim

[Jim Barrows]

Do we really need a language to sit on top of the different query
mechanisms?

Aren't all searches (from a certain point of view) really just partial
functions waiting to be implemented? Or query objects built along the
lines of the command pattern?

Don't we really just need a way to hand off that functionality without
caring what is, or does, to the implementing mechanism?

Maybe I haven't thought through though.

[Andreas S.]

Hi Stefan,
I don't know if that's what you are looking for, but I have the
feeling that it is somehow related.
http://scala-programming-language.1934581.n4.nabble.com/Datalog-meets-NoSql-tp2376588p2376588.html
HTH
regards Andreas

On Aug 31, 12:26 am, Stefan Schmidt <stefan.schm...@gmx.net> wrote:
> Hello,
>
> to my understanding the switch between different relational databases
> when using Mapper is just a matter of changing a few lines. First adding
> the dependency to the project description and then changing the database
> connection settings in the properties file. For example if i wanted to
> switch from MySQL to PostgreSQL it would require these five lines:
>
> val lift_postgresql = "postgresql" % "postgresql" % "8.4-701.jdbc4"
>
> db.driver=org.postgresql.Driver
> db.url=jdbc:postgresql:DATABASENAME
> db.user=USERNAME
> db.password=PASSWORD
>

> What is the cost if I am usingRecordand I want to migrate from MongoDB