Introducing ReactiveMongo, a non-blocking Scala driver for unleashing MongoDB streaming capabilities

[Stéphane Godbillon]

I am very excited to introduce ReactiveMongo, a brand new Scala driver
for MongoDB. More than just non-blocking, it's a reactive driver that
allows you to design very scalable applications unleashing MongoDB
capabilities like streaming infinite live collections and files for
modern real-time Web applications.

With ReactiveMongo, you can build highly scalable web applications,
making heavy usage of streaming capabilities like streaming files from
and into MongoDB (GridFS), or tail a collection through a websocket.
Possibilities are endless!

You can read the complete announce here:
http://stephane.godbillon.com/2012/08/30/reactivemongo-for-scala-unleashing-mongodb-streaming-capabilities-for-realtime-web
. Checkout the driver repository on
https://github.com/zenexity/ReactiveMongo and the plugin repository on
https://github.com/zenexity/Play-ReactiveMongo .

Cheers!

- Stephane Godbillon

Blog: http://stephane.godbillon.com

Twitter: https://twitter.com/sgodbillon

Github: https://github.com/sgodbillon

[Jorge Aliss]

Congrats on the release!.  Looks impressive.

Jorge

[Pascal Voitot Dev]

You're welcome!
This is still draft driver so some bugs may be found and we are going to improve things step by step but it's already quite good to test and validate the concepts ;)

Great Thanks to SGodbillon for his great and harassing job !!!!

Regards
Pascal

--
 
 

[Stéphane Godbillon]

Thanks!

Like Pascal said, it's a developer preview - many things to implement (particularly commands) and a lot of tests to do - but the core is already functional and perfectly useable. Actually we use it already at zenexity for some projects :)

If you try it, let me know, I'd be very interested in your feedback.

[Ike]

Very impressive and exciting work! I'm looking forward to having a real-world need for this.

If only all other drivers worked this way...we could just start with Redis and Cassandra :)

[Stéphane Godbillon]

Thank you :)

Actually you don't need to have a very special use case to use ReactiveMongo. It is as good for "classic" projects as it is for real-time applications. In fact we use it at Zenexity on some projects that don't require special capabilities like streaming data.

The goal is to make a good async/non-blocking Scala driver for MongoDB, with a well-designed API, that you can use for any project built with MongoDB. The streaming/reactive capabilities are its most advanced features, but it is also suitable for all the common use cases.

2012/9/5 Ike <ike...@gmail.com>

--
 
 

[Danny]

Hi 

Great :)

Is it only for Scala 2.10 ?

Thanks

Danny

[Pascal Voitot Dev]

Right now, it depends on a few weeks old Play-2.1SNAPSHOT (not all Play, just the Iteratee part) which was still depending on Scala-2.9.2
Play-2.1 is currently evolving quite a lot to integrate Scala-2.10 so we are waiting for a stable version before migrating to Scala-2.10.

Pascal

--
 
 

[Stephane Godbillon]

Hi,

It targets 2.10 (because it needs the Scala 2.10 futures). Though you can test it with Scala 2.9.2 because the snapshots of Play 2.1 contained them before the 2.10 switch.

-- Stephane Godbillon

Blog: http://stephane.godbillon.com

Twitter: https://twitter.com/sgodbillon

Github: https://github.com/sgodbillon

2012/10/24 Danny <dbms...@gmail.com>

--
 
 

[Pascal Voitot Dev]

There is echo in this group ;)

--
 
 

[Stephane Godbillon]

Great minds think alike ^^

-- Stephane Godbillon

Blog: http://stephane.godbillon.com

Twitter: https://twitter.com/sgodbillon

Github: https://github.com/sgodbillon

2012/10/24 Pascal Voitot Dev <pascal.v...@gmail.com>

--
 
 

[Pascal Voitot Dev]

I decided to rewrite my mind in Scala2.10 recently but my Scala Promise has not yet been redeemed ;)

--
 
 

[Nelson Pereira Junior]

Hi Stéphane, I'm interesting in your async mongo driver. I starting a new project, and need a true mongodb async (non blocking) driver.

Can I create my project in C, C ++, Java or Scala. The choice of language depends on which one has async (non blocking) mongodb driver.

I initially thought of using the C driver, my preference, but it is not async.

I see too the java driver mongodb-async-driver (http://www.allanbank.com/mongodb-async-driver/index.html), it's async, but it is not "Non Blocking", internally use threads and connection pool to real connections with mongodb.

ReactiveMongo is really non blocking, without internals threads, or it is like mongodb-async-driver?

[Stephane Godbillon]

Hi Nelson,

ReactiveMongo is completely asynchronous and non-blocking.  It does have internal threads though, which is normal; the thing is it does not assign a thread per connection, and does not block any thread while waiting for IOs.  You won't ever see a thread blocked on any kind of network operation in ReactiveMongo.

Cheers,

-- Stephane Godbillon

Blog: http://stephane.godbillon.com

Twitter: https://twitter.com/sgodbillon

Github: https://github.com/sgodbillon

--
You received this message because you are subscribed to the Google Groups "ReactiveMongo - http://reactivemongo.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reactivemong...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nelson Pereira Junior]

Hi Stephane, you explained very clearly. Thanks!

--
You received this message because you are subscribed to a topic in the Google Groups "ReactiveMongo - http://reactivemongo.org" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/reactivemongo/agmPCHE8kOE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to reactivemong...@googlegroups.com.

[alex reviyou]

Hello Everybody, Stephane,

we are using play with reactive mongo and so far very happy with it.

However we have a "best practises" question.

Since reactivemongo is async and non blocking- how does it changes things while working with connections pool in mongo? Do we still need to try to limit amount of request we may to the database.

We hear some people say that since it's non blocking and cached - we don't care about connection pool anymore and no need to stick to the 1000\100\10 concept anymore(1000 connections to web server, 100 to app server, 10 to db)

Here is a simple hypothetical  example. I retrieve 50 records from document "Car". For each card I need to retrieve 100 CarDetails. And render it on UI once completed.

Let's say I loop through the Cars(50 Futures) and for each I get all details(100 Futures). I convert List of Futures to a Future of a list and pass to UI. 

Here is my question. With a blocking driver this is very bad. 1 request takes 50*100=5000 blocking connections so even large connection pool doesn't help much.

Is it different with reactivemongo and non blocking concept? Or since it's non blocking it would not slow us down even if we have 100 concurrent requests like that? Or on the contrary it'll waste all available connections very fast.

Again this is a fundamental example only to understand the concept of reactivemongo+connection pool + cache. Do we still care about large number of db connections or driver somehow takes care of it now?

Regards,

Alex

[Nelson Pereira Junior]

If using ajax to get cars and details: (best option)

Ajax send request to app server to get car list.

App server get one connection and make find, and to each car of result list, write to client. 
When client receive all cars (full json)...

Get the first car, and request detail to app server.

App server get one connection, make find, and to each detail write to client.

Client render each detail when receive.

When client receive last detail, make other request to get details of second car...

Only one connection parallels.

If using render on server:

Client request the page cars with details.

App server get one connection. Make cars find. Get all cars into list. After fill list. Make other list with cars ids. Make only one find with details of all cars. To each result, fill details to car list.

App server render page. 

Only one connection used.

Sent from my iPad

You received this message because you are subscribed to a topic in the Google Groups "ReactiveMongo - http://reactivemongo.org" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/reactivemongo/agmPCHE8kOE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to reactivemong...@googlegroups.com.

[alex reviyou]

Thanks Nelson,

it's 2 great examples and I am sure both work. We can surely use them as a best practices

Of course - each solution has some drawback:

for ajax: we have 100 ajax requests i.e network\latency increase for each Car. But I agree that it's more asyn way to render results as we receive them

However the original question is different. What if. What happens in the example we described in terms of connections.

-get 100 Cars

-for each car get 100 Car Details

-after we retrieved everything - pass to the ui

Question is how many connections will be used and how can it affect mongo connections pool. Would it scale 100 concurrent users or it would completely waste all connections from the pool and becomes a bottle neck. Or the opposite, since reactive is non-blocking connections - it would still be fine and scalable.

Do we have to try to aim to have a minimum db connections like with blocking drivers and relational db's? If 100 concurrent users return the same results - would mongo cache help to eliminate some of the connections. Basically we want to know what BAD things can happen for the scalable architecture in the example we can above and why so that we can better understand how it works.

Thanks,

Alex

[alex ander]

Anybody? It scares me a little bit that nobody from the team created this driver answered such fundamental question yet...

[Reid Spencer]

It is a bit concerning for me too. I’ve made several improvements to ReactiveMongo in my own fork with no comment from the owners about them. 

If this keeps up, I’ll just be releasing my own fork. Cooperation would be preferred, but that’s hard to accomplish in silence.

Reid Spencer

re...@viritude.com

415.799.8474

[Stephane Godbillon]

Hi,

The number of connections to each database node in a replica set is fixed (by default, it's 10). So if you have 3 mongod instances in a replica set, the driver will open 30 connections – 10 connections per instance.

The fact that ReactiveMongo is non blocking is that it does block no thread while waiting for a response from the database. In Alex's example, the application needs to make 5000 requests to the database. Let's say for the sake of simplicity that these requests need to be run on the primary.

Each time a request is given to the driver, ReactiveMongo chooses a connection, transforms the request message and writes it on that connection (it uses Netty channels for that). Connections are not implemented as state machines; they don't have a state like 'waiting for response' for example. ReactiveMongo simply writes as fast and as much as it can, and never ever blocks threads neither connections.

The way it works is really simple: every request has an identifier (the requestId). Each mongod response has a responseTo field, which value is the same as the requestId of the original request. ReactiveMongo stores these ids in a map containing Promise instances (among others), and completes those Promise instance with the response, which updates the state of the associated Future.

In this model, it is fundamental to understand that mongod instances basically uses one thread per client connection – meaning that if a mongo client (ReactiveMongo or another driver) opens a connection, the mongodb server will spawn a thread associated with that connection.

Let's get back to Alex's example. The 5000 futures will be run concurrently, so ReactiveMongo will create 5000 mongo requests and write them as fast as it can using the 10 connections it has on the primary. When a response arrives, it looks for the matching Promise and completes it, so that the associated Future's state is updated. The number of threads on the mongod side will not change (since the number of open connections is fixed), and the number of threads on the ReactiveMongo side will not raise either.

Anyway, it is _always_ better to keep in mind that you should cache results that are frequently computed, if you can. It is useless to make 5000 requests to the database each time if you can cache the result, and it will be a lot faster. Also, if you end up making a lot of this kind of requests, then the current data model may not be a good fit for your usage and may be reworked for better efficiency.

Hope it make things clearer. And sorry for the delay of this response :)

-- Stephane Godbillon

Blog: http://stephane.godbillon.com

Twitter: https://twitter.com/sgodbillon

Github: https://github.com/sgodbillon

[Stephane Godbillon]

Hi Reid,

Sorry about that. I didn't get a lot of time to work on the project recently – anyway it is still active and I really want to focus on the next release. By the way, I'm looking for maintainers who'd help me on the day-to-day maintenance of the driver; if you are interested let me know :)

In the meantime, could you show me the improvements you made?

Thanks,

-- Stephane Godbillon

Blog: http://stephane.godbillon.com

Twitter: https://twitter.com/sgodbillon

Github: https://github.com/sgodbillon

2014-12-20 21:50 GMT+01:00 Reid Spencer <re...@viritude.com>:

[alex ander]

thank you Stephane, this is a well written answer and I am sure it will be helpful for many people out there trying this functionality.

"Last question regarding your reply - The number of connections to each database node in a replica set is fixed (by default, it's 10)"

are you referring to the nbChannelsPerNode Number of channels to open per node. Defaults to 10. ?

If so -my question is about tweaking this number for better performance - can we set it to 1000 for example?

What are the limits of specifying this number higher for better performance?

Thanks,

Alex

[re...@reactific.com]

Hi Stephane,

Thanks for showing up. We were getting a little worried. :)  Yes, I would like to help as I have time and interest. I've already made a number of improvements and created some offshoots for my own project. Have a look at the 6 commits of mine in this list and let me know if you're interested in any of them or all (I can squash). In particular, the most recent one fixes some outright bugs in the equals method of some underlying types that prevent one BSONDocument from equalling its identical twin. The others are things I needed in my project to deal with repeated startup/shutdown in unit tests (I have a lot of them!) so as not to get OOMs every 30 seconds.

Best, 

Reid.

[Stephane Godbillon]

I've had a glance to your commits and your changes seem to be quite interesting! :)  It would be great to issue a PR for them.  Then I will review it closely.  If you're willing to do so, please do not squash – it's better to keep the commits meaningful.

Cheers,

-- Stephane Godbillon

Blog: http://stephane.godbillon.com

Twitter: https://twitter.com/sgodbillon

Github: https://github.com/sgodbillon

[Stephane Godbillon]

Yes, `nbChannelsPerNode` allows to change the default number of open connections to each node in the replica set.

You can definitely play with that parameter and tune it to get better performance – basically, since an open connections in mongod is backed by a thread, you are altering the concurrency on the database side.  The best number is completely dependent of your application though, there is no good value for all cases.  The best thing to do is to make some benchmarks for your application.

Cheers,

-- Stephane Godbillon

Blog: http://stephane.godbillon.com

Twitter: https://twitter.com/sgodbillon

Github: https://github.com/sgodbillon

[Miguel Osorio]

Hi,

I´m very new with ReactiveMongo, i´m using it in a project and i´m facing a problem because i need to do bulk updates.
I googled this topic and found the methods bulk insert and bulk insert iteratee, but what i don´t know is if these methods can be used for doing bulk update too, theres few docs about this methods. can you help me plz?

Thanks in advance and regards.


Mike!

[Alok Vats]

Hello,

How to use mapreduce in reactivemongo,please specify me how to write javascript code in reduce and map.

[James L. Cain IV]

Here's an example we use 


  protected def collectionName: String = "client"

  val mapStdDev =
    """
      |function map() {
      |    emit(1, // Or put a GROUP BY key here
      |         {sum: NumberInt(this.metaData.activityTotal), // the field you want stats for
      |          min: NumberInt(this.metaData.activityTotal),
      |          max: NumberInt(this.metaData.activityTotal),
      |          count:1,
      |          diff: 0, // M2,n:  sum((val-mean)^2)
      |    });
      |}
    """.stripMargin

  def mapStdDevApi(api: String) =
    s"""
      |function map() {
      |    emit(1, // Or put a GROUP BY key here
      |         {sum: NumberInt(this.metaData.${api}Activity), // the field you want stats for
      |          min: NumberInt(this.metaData.${api}Activity),
      |          max: NumberInt(this.metaData.${api}Activity),
      |          count:1,
      |          diff: 0, // M2,n:  sum((val-mean)^2)
      |    });
      |}
    """.stripMargin

  val reduceStdDev =
    """
      |function reduce(key, values) {
      |    var a = values[0]; // will reduce into here
      |    for (var i=1/*!*/; i < values.length; i++){
      |        var b = values[i]; // will merge 'b' into 'a'
      | 
      | 
      |        // temp helpers
      |        var delta = a.sum/a.count - b.sum/b.count; // a.mean - b.mean
      |        var weight = (a.count * b.count)/(a.count + b.count);
      |
      |        // do the reducing
      |        a.diff += b.diff + delta*delta*weight;
      |        a.sum += b.sum;
      |        a.count += b.count;
      |        a.min = Math.min(a.min, b.min);
      |        a.max = Math.max(a.max, b.max);
      |    }
      | 
      |    return a;
      |}
      |""".stripMargin


 def standardDeviationActivityByApi(clientId: String, api: String): Future[(Double, Double)] = {
    val mapReduceCommand = BSONDocument(
      "mapreduce" -> BSONString("locations"),
      "map" -> BSONJavaScript(mapStdDevApi(api)),
      "reduce" -> BSONJavaScript(reduceStdDev),
      "finalize" -> BSONJavaScript(finalizeStdDev),
      "query" -> BSONDocument("clientRefId" -> clientId),
      "out" -> BSONDocument("inline" -> "1"))
    db.command(RawCommand(mapReduceCommand)).map(doc => {
      val d = BSONDocumentIdentity.write(doc)
      val value = d.getAs[BSONArray]("results").get.getAs[BSONDocument](0).getOrElse(BSONDocument()).getAs[BSONDocument]("value").getOrElse(BSONDocument())
      (value.getAs[Double]("avg").getOrElse(0.0), value.getAs[Double]("stddev").getOrElse(0.0))
    }
    )
  }

Jay Cain

Sent from iPhone

--

[Alok Vats]

Thanks

...

[Alok Vats]

How to find a day from epoch time in mapreduce and aggregation in reactive mongo.

On Thursday, August 30, 2012 at 8:14:42 PM UTC+5:30, Stéphane Godbillon wrote:

I am very excited to introduce ReactiveMongo, a brand new Scala driver
for MongoDB. More than just non-blocking, it's a reactive driver that
allows you to design very scalable applications unleashing MongoDB
capabilities like streaming infinite live collections and files for
modern real-time Web applications.

With ReactiveMongo, you can build highly scalable web applications,
making heavy usage of streaming capabilities like streaming files from
and into MongoDB (GridFS), or tail a collection through a websocket.
Possibilities are endless!

You can read the complete announce here:
http://stephane.godbillon.com/2012/08/30/reactivemongo-for-scala-unleashing-mongodb-streaming-capabilities-for-realtime-web
. Checkout the driver repository on

[cchantep]

Hi, please ask distinct questions in separate topics. Easier to follow discussions :)

[scotv]

Hi I am using and will use ReactiveMongo and ReacitveMongo-Play, thanks for your work.

Only a suggestion or question for feedback.

Why are we move the discussion GROUP (i mean a official GROUP, gathering ...) to stackoverflow, or github issue. cause Google Group doesn't support Markdown.

and i also noticed when someone rise a question on github issue, you may ask them come to post question on Google Group, but Google Group doesn't support Markdown.

is there any place for GROUP discussion on programming, cause Google Group doesn't support Markdown.

Thanks

[Edem Nsefik]

Hello Stéphane ,

What is the advantage to integrating akka into the reactivemongo driver as opposed to using it alongside?

[Cédric Chantepie]

I don't see your point. Akka is how ReactiveMongo is implemented, but it's an implementation details (you can just ignore it).

[Cédric Chantepie]

Hi,

As the release of the next major version is expected before end of summer, the release candidate 1.0.0-rc.2 is available.

The documentation has been updated, and the future release notes are available to prepare migration.

A new Scalafix module is provided to ease the upgrade process.

A feature and API freeze is planned from this point until the final release of 1.0.0 (according the cases, bugfixes would be backported to release 0.20.x).

Best regards,

P.S. For those wanting to celebrate that with style (or stickers), it's possible to visit redbubble.