Clarification On Clone vs Copy

[William Kennedy]

I am using revel and want to make sure I am doing this right.

For every request I plan on making a clone of an existing session I create at startup.  I am hoping clone reuses the socket pool instead of creating a new one like copy does?

I am assuming this is the correct way to use the pool, per web session?

[Gustavo Niemeyer]

Every single session that is spawned from the same Dial operation will
reuse the same pool of connections. The difference between Copy and
Clone is that Clone will reuse even the socket that is currently bound
to the session, so you may have multiple sessions backed by the same
socket.

Have you had a look at the documentation? It seems to cover that pretty well:

http://labix.org/v2/mgo#Session.Clone
http://labix.org/v2/mgo#Session.Copy

> --
> You received this message because you are subscribed to the Google Groups
> "mgo-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mgo-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.



--

gustavo @ http://niemeyer.net

[William Kennedy]

> Every single session that is spawned from the same Dial operation will
> reuse the same pool of connections. The difference between Copy and
> Clone is that Clone will reuse even the socket that is currently bound
> to the session, so you may have multiple sessions backed by the same
> socket.
>
> Have you had a look at the documentation? It seems to cover that pretty well:
>
> http://labix.org/v2/mgo#Session.Clone
> http://labix.org/v2/mgo#Session.Copy

I did look at that before sending in my question. When I use Copy I see a lot of authenticate calls in the MongoDB server logs. When I use Clone I
was seeing a lot of connections being created and ended. I am not a guru so it is hard to validate everything I am seeing. I just want to make sure I
am doing things right and minimizing a burden on the server.

>
>
> On Fri, Nov 1, 2013 at 5:34 PM, William Kennedy <bi...@thekennedyclan.net> wrote:
>> I am using revel and want to make sure I am doing this right.
>>
>> For every request I plan on making a clone of an existing session I create
>> at startup. I am hoping clone reuses the socket pool instead of creating a
>> new one like copy does?
>>
>> I am assuming this is the correct way to use the pool, per web session?
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mgo-users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to mgo-users+...@googlegroups.com.
>> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
> --
>
> gustavo @ http://niemeyer.net
>
> --

> You received this message because you are subscribed to a topic in the Google Groups "mgo-users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/mgo-users/pk9IsU2bb8o/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to mgo-users+...@googlegroups.com.

[Gustavo Niemeyer]

No problem. I just wanted to make sure you knew that the documentation
actually covers these aspects as well.

It's expected that you might see additional authentication requests if
you're using several sockets rather than a single one.

[William Kennedy]

>
> No problem. I just wanted to make sure you knew that the documentation
> actually covers these aspects as well.

The documentation really doesn't explain how to use sessions:

Why copy a session?
Why clone a session?

If I have multiple routines running operations at the same time, like in my web service:
Should I create a master session and then clone the session for each routine/web request, perform the operations and close the session?
Is it safe to create one session and use it for all routines/web requests?

You see I am very confused about sessions and when to copy or clone them in a multi routine program like a web service.

>
> It's expected that you might see additional authentication requests if
> you're using several sockets rather than a single one.

I was wrong, I saw all the authenticate calls when I was using refresh. I think I see connections being accepts and closed with session copies.

[Gustavo Niemeyer]

On Fri, Nov 1, 2013 at 8:22 PM, William Kennedy <bi...@thekennedyclan.net> wrote:
>>
>> No problem. I just wanted to make sure you knew that the documentation
>> actually covers these aspects as well.
>
> The documentation really doesn't explain how to use sessions:

Sorry, but the documentation does answer the original question you
had, pretty well in fact.

Question:

"""
For every request I plan on making a clone of an existing session I
create at startup. I am hoping clone reuses the socket pool instead
of creating a new one like copy does?

I am assuming this is the correct way to use the pool, per web session?
"""

Excerpt of documentation for mgo.Dial (http://labix.org/v2/mgo#Dial):

"""
This method is generally called just once for a given cluster. Further
sessions to the same cluster are then established using the New or
Copy methods on the obtained session. This will make them share the
underlying cluster, and manage the pool of connections appropriately.
"""

Documentation for Session.New (http://labix.org/v2/mgo#Session.New):

"""
New creates a new session with the same parameters as the original
session, including consistency, batch size, prefetching, safety mode,
etc. The returned session will use sockets from the pool, so there's a
chance that writes just performed in another session may not yet be
visible.
Login information from the original session will not be copied over
into the new session unless it was provided through the initial URL
for the Dial function.
See the Copy and Clone methods.
"""

Documentation for Session.Copy (http://labix.org/v2/mgo#Session.Copy):

"""
Copy works just like New, but preserves the exact authentication
information from the original session.
"""

Documentation for Session.Clone (http://labix.org/v2/mgo#Session.Clone):

"""
Clone works just like Copy, but also reuses the same socket as the
original session, in case it had already reserved one due to its
consistency guarantees. This behavior ensures that writes performed in
the old session are necessarily observed when using the new session,
as long as it was a strong or monotonic session. That said, it also
means that long operations may cause other goroutines using the
original session to wait.

"""

> If I have multiple routines running operations at the same time, like in my web service:
> Should I create a master session and then clone the session for each routine/web request, perform the operations and close the session?
> Is it safe to create one session and use it for all routines/web requests?

Yes, it's safe. If you are using a session in Strong or Monotonic
modes, this will mean all the goroutines will share a single socket,
per the documentation (http://labix.org/v2/mgo#Session.SetMode).

> You see I am very confused about sessions and when to copy or clone them in a multi routine program like a web service.

Sure, I can see that you are missing a prescribed way to do it. Often
this kind of application will have a master session and then use a
Copy of it for each request and then Close at the end. Feel free to
change that prescription to meet your intention, though.

>> It's expected that you might see additional authentication requests if
>> you're using several sockets rather than a single one.
>
> I was wrong, I saw all the authenticate calls when I was using refresh. I think I see connections being accepts and closed with session copies.

I cannot tell what you're actually doing there or referring to, but if
you're suggesting that copying sessions means not using a connection
pool, that is false.


gustavo @ http://niemeyer.net

[William Kennedy]

I have read all of that before but let me see if I have it straight. If this is wrong then I will just find time to study the code and break it down myself. I am sorry for not getting it.


The dial call will establish a pool of socket connections to the underlying cluster

Session New and Copy will be given independent connections from that pool to use.

The operations within a given session have consistency guarantees

New and Copy create sessions that would not have consistency guarantees between each other

Clone will have consistency guarantees with the other session it cloned

You will mange the number of socket connections as need for all sessions

Closing a session returns those sockets back into the pool for future use by other sessions

You manage socket connection issues for sockets not in use

Refresh can be called to handle issues on sockets in established sessions

[Gustavo Niemeyer]

All spot on, William.

You received this message because you are subscribed to the Google Groups "mgo-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mgo-users+...@googlegroups.com.

[Tristan Maindron]

Hello there,

I'm also a Revel user and working on a neat binding between Revel and mgo. At the moment I do a single mgo.Dial and then store the session in a singleton :

var session *mgo.Session

var dial sync.Once

func GetSession() *mgo.Session {

    host, _ := revel.Config.String("mongo.host")

    dial.Do(func() {

        var err error

        session, err = mgo.Dial(host)

        if err != nil {

            panic(err)

        }

    })

    return session

}

And then I always use this session, without cloning of copying it. Is that safe ? I see a lot of people talking about copying or cloning the Session but what is the goal of that ? 

[William Kennedy]

For the long term it is not the best way to go. Every session should get a copy so they can process their request independently. Each request is given its own session in revel and the same should happen with mgo. 

Use the before and after interceptor to copy a session and close it. I have code for mgo and revel that implements all of this if you would like it. Send me an email. 

bi...@ardanstudios.com

[Chris Foresman]

Trying to get a bit of a handle on how the connection pool is managed behind the scenes, because we have been seeing some weird behaviors where we seemed to be creating thousands of connections to our MongoDB. In the process, we ended up changing every call to Copy() to Clone(). Regardless of the effects on performance—we haven't seen any just yet—it seemed to reduce memory overhead quite a bit. (unfortunately, we aren't 100% sure what specific effects are of relying on Clone vs Copy just yet, but we are monitoring it)

When Dial is called to open a connection to the DB and create the first session, it creates some number of sockets ≥ 1, correct? But how many sockets Dial ends up opening, how is that calculated? Does the number ever increase or decrease? I get the feeling maybe it does, since the docs refer to managing a connection pool for you. But sometimes we find it helps to understand how the magic happens—even if we can't manually manage it—just so we have less unexpected things happening.

Thanks,

Chris

[Gustavo Niemeyer]

Hi Chris,

On Fri, Jan 31, 2014 at 6:05 PM, Chris Foresman
<chris.f...@vokalinteractive.com> wrote:
> Trying to get a bit of a handle on how the connection pool is managed behind
> the scenes, because we have been seeing some weird behaviors where we seemed
> to be creating thousands of connections to our MongoDB.

As a general note, please make sure every single session created
(through whatever means) is closed. Session leaks are a quick way to
end up with a large number of connections.

> In the process, we
> ended up changing every call to Copy() to Clone(). Regardless of the effects
> on performance—we haven't seen any just yet—it seemed to reduce memory
> overhead quite a bit. (unfortunately, we aren't 100% sure what specific
> effects are of relying on Clone vs Copy just yet, but we are monitoring it)

The only difference between Copy and Clone, as the documentation
indicates [1], is that Clone shares a previously allocated socket with
the original session. This has resource consumption advantages, as
you've figured, but may also cause contention issues depending on your
application. Most obviously, while a query that takes a long time
runs, it blocks results from arriving on all sessions that are sharing
the same socket.

[1] http://labix.org/v2/mgo#Session.Clone

> When Dial is called to open a connection to the DB and create the first
> session, it creates some number of sockets ≥ 1, correct? But how many
> sockets Dial ends up opening, how is that calculated? Does the number ever
> increase or decrease?

This is an implementation detail, but it will generally match or
approach the number of servers being connected to. If a replica set
has 3 servers, it'll likely create 3 connections, for example.

> I get the feeling maybe it does, since the docs refer
> to managing a connection pool for you. But sometimes we find it helps to
> understand how the magic happens—even if we can't manually manage it—just so
> we have less unexpected things happening.

Makes sense.


gustavo @ http://niemeyer.net

[wensheng tle]

I have a online API service use beego. One session(request) will clone a mgo session(a package will init it) for this request And close it when this request finish. 

But now I found this service run after for a time have increased opened many mongo connection? Why so many? 

[root@ ~]# lsof -i :27607 | wc -l

1379

[root@ ~]# netstat -anop | grep 27607 | wc -l

1378