How to handle disconnections to mongodb server? (aka. autoconnect)

[Fatih Arslan]

Hi Everyone,

I have struct like:

type DB struct {

Session    *mgo.Session

        // other fields

}

This get populated ONCE whenever I start the program with the New function below:

func NewDB() (*DB, error) {

session, err := mgo.Dial("logincredentials")

if err != nil {

return nil, err

}

        

       return &DB{Session: session}

}

After that I'm using this session pointer in concurrent goroutines (10k per second). Like

db := NewDB()

db.Session.DB("").C("test") ...

The problem I have is, when the connection to the mongodb server fails (the server drops the connection sometimes or mongodb server fails), then my pointer to the session variable doesn't work anymore. Even if the internet connection comes back, mgo driver doesn't reconnect anymore, instead of this I get the error (when I do Find().One() method call):

"EOF"

Now how should I handle this behavior  Should I reconnect (Dial again and storing a new session pointer) whenever I get an EOF error? Or is there something else that I should fix? (like making a Dial for every query, instead of storing a session pointer like above.

Thanks in advance

Regards

Fatih Arslan

[Gustavo Niemeyer]

On Mon, Jul 29, 2013 at 8:19 AM, Fatih Arslan <ftha...@gmail.com> wrote:
> The problem I have is, when the connection to the mongodb server fails (the
> server drops the connection sometimes or mongodb server fails), then my
> pointer to the session variable doesn't work anymore. Even if the internet
> connection comes back, mgo driver doesn't reconnect anymore, instead of this
> I get the error (when I do Find().One() method call):

It actually reconnects. What it doesn't do is to reset the session
you're holding, because that might hide real issues in your
application.

There are two easy ways to get rid of the error:

1) Call Refresh on the session, which makes it discard (or put back in
the pool, if the connection is good) the connection it's holding, and
pick a new one when necessary.

2) Instead of using a single session, use many by calling session.Copy
when you need a new session, and then call session.Close when you're
done with it. This will also mean you're using multiple connections to
the database, when necessary.

Please let us know if you need further details.


gustavo @ http://niemeyer.net

[Fatih Arslan]

Hi Gustavo,

Thanks for the reply. I've implemented your first solution. For that I'm looking to the Error returned (unfortunately there is no error type, like mgo.ErrConnectionClosed). 

if err.Error() == "EOF" {

sessioinPointer.Refresh()

}

That means when the the error contains EOF I'll refresh the session until the network has been established. I've tried this locally and it seems to work without any problem.

Is that a correct approach?

And for the second step (using session.Copy), would this affect the performance when I call simultaneously concurrent at least 1000 session.Copy() ? I'm using the first approach (pointer to a session) that it uses only one connection which should be better in terms of performance.

Regards

[Gustavo Niemeyer]

On Thu, Aug 1, 2013 at 7:33 PM, Fatih Arslan <ftha...@gmail.com> wrote:
> Hi Gustavo,
>
> Thanks for the reply. I've implemented your first solution. For that I'm
> looking to the Error returned (unfortunately there is no error type, like
> mgo.ErrConnectionClosed).
>
> if err.Error() == "EOF" {
> sessioinPointer.Refresh()
> }

That EOF should be io.EOF, so you can just compare with the value itself.

That said, why is EOF any different from any other error that you do
not recognize? What if that is a timeout error, or a socket error,
etc? In theory you can call Refresh when you're starting your loop /
request handling / etc, assuming you have a single session across the
whole application, and handle all situations.

(...)

> And for the second step (using session.Copy), would this affect the
> performance when I call simultaneously concurrent at least 1000
> session.Copy() ? I'm using the first approach (pointer to a session) that it
> uses only one connection which should be better in terms of performance.

Session.Copy is cheap. You can call it as necessary.


gustavo @ http://niemeyer.net

[Fatih Arslan]

You mean I should call refresh before checking for io.EOF error ? That make sense and I think it's better.
I assume that calling Refresh() is also thread safe, that means I can call it in concurrent goroutines?

And for Session.Copy, I didn't know there were cheap to use. Good to know.

Regards

--
Fatih Arslan


On 2 August 2013 Friday at 01:41, Gustavo Niemeyer wrote:

> --
> You received this message because you are subscribed to a topic in the Google Groups "mgo-users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/mgo-users/XM0rc6p-V-8/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to mgo-users+...@googlegroups.com (mailto:mgo-users+...@googlegroups.com).
> For more options, visit https://groups.google.com/groups/opt_out.

[Gustavo Niemeyer]

On Thu, Aug 1, 2013 at 7:47 PM, Fatih Arslan <ftha...@gmail.com> wrote:
> You mean I should call refresh before checking for io.EOF error ? That make sense and I think it's better.

Yes, but I mean at the very start of your loop, or the equivalent in
your application.

> I assume that calling Refresh() is also thread safe, that means I can call it in concurrent goroutines?

That's not a good idea, as it will interfere with the other
goroutines. In fact, that's the exact point of the error not being
automatically reset: you don't want goroutines that are running to see
a completely different server and state without any notice. They
should all fail and go back to the start of your processing routine
(loop, request handling, whatever) so you can move on again.

Or, just copy the session and closing it when done, including on errors.


gustavo @ http://niemeyer.net

[Fatih Arslan]

I'm making several thousands concurrent lookups with a single session pointer. That's why I asked you if it was threadsafe.
I have one single ServeHTTP, but this is called also concurrently via Go's ServeAndListen. And every HTTP request makes a lookup.

Then what left is to do a session.Copy and deferring the Close(). This will error-prone and threadsafe too.


--
Fatih Arslan


On 2 August 2013 Friday at 01:56, Gustavo Niemeyer wrote:

[İnanç Gümüş]

Actually, session.Copy and Close does not work in the stressing apps. See this: https://github.com/go-mgo/mgo/issues/473

2 Ağustos 2013 Cuma 03:00:39 UTC+4 tarihinde Fatih Arslan yazdı:

[Gustavo Niemeyer]

If you're getting timeouts due to load, the problem is not just copy/clone. These are trivial data copying operations that offer a way to organize your application in terms of the desired resource usage.

--
You received this message because you are subscribed to the Google Groups "mgo-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mgo-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

gustavo @ http://niemeyer.net

[İnanç Gümüş]

I posted a simple code in the link I posted. One can comment on that code about what's wrong with it.

Inanc Gumus

İnanç Gümüş <in...@pixenka.com> şunları yazdı (1 Ağu 2017 17:57):

To unsubscribe from this group and all its topics, send an email to mgo-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.