Massive performance drop by adding j=True (journaling)

[Joshua Kehn]

Here's a gist of what I'm using: https://gist.github.com/joshkehn/4759398

This is a simplified version of what I'm using on a production site right now.[^1]

The three scary lines are:

(safe mode)

     3000    0.032    0.000  107.296    0.036 collection.py:204(save)

     1000    0.022    0.000   35.887    0.036 collection.py:274(insert)

     2000    0.039    0.000   71.372    0.036 collection.py:364(update)

(unsafe mode)

     2000    0.006    0.000    0.367    0.000 collection.py:204(save)

     1000    0.008    0.000    0.188    0.000 collection.py:274(insert)

     1000    0.007    0.000    0.172    0.000 collection.py:364(update)

Script run time went from 0.428 seconds to 107.451 seconds (~214x slower). I realize that j=True forces it to wait for the journal to be written, but I didn't realize it would be _this_ slow.

1. What can I do to ensure write consistency without this performance penalty?

2. Is there a write-ahead log or something I can change in how the journal is written to improve the performance here?

Thanks,

-Josh

[^1]: Why do I have mongo.py? Centralizes the connection point. Why are there insert/save/find methods? Because often I'm operating on a collection specific to another object and this allows me to catch requests and add additional arguments to the query before committing the result. If there's a better way to do this *let me know*.

[Eliot Horowitz]

Use j=True will definitely have a large latency impact.
There are 2 ways to mitigate:
- use multiple threads. j=True shouldn't impact total throughput,
just per socket latency
- call j=True every N operations. i.e. do 100 inserts, call j=True.
that guarantees all 100 are on disk, but should be 100x faster.

> --
> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com
> See also the IRC channel -- freenode.net#mongodb
>
> ---
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mongodb-user...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Joshua Kehn]

Most of the time I'm not doing bulk inserts / updates though. It's an update or two on this request, an insert on that one, and a half dozen new records during a background task. The second solution doesn't work here.

The first solution (multiple threads) will prevent throughput problems, but it means that thread is going to block and screw up my response times. The application shouldn't have to wait 70-100ms for each database write it makes.

What about deferring save operations with something like Celery? I'd hate to pass data through more potential methods of failure, but maybe writing out my own write ahead log would work. 

    @task(ignore_result=True)

    def write_save (col, op, *args, **kwargs):

        write_ahead.delay(col, *args, **kwargs)

        getattr(col, op)(j=True, *args, **kwargs)

Here's a second question, if I write something (either with j=True or without) does it get cached on the client before it receives the response from the server?

For example:

    collection.save({"name" : "josh"})

    collection.find({"name" : "josh"})             # Does this find the object I just sent down for saving?

Best,

–Josh
____________________________________
Joshua Kehn | @joshkehn
http://joshuakehn.com

[Eliot Horowitz]

What guarantees do you want on the write?
You might want to switch w=2 (or majority) which guarantees the write
is on 2 nodes rather than on disk.
This is typically a lot faster (and survives any single node failure
in the w=2 case).

For your 2nd question, there is no client side caching.

[Joshua Kehn]

The minimum guarantee I would like is that the write reached the server (default write concern). Not big deal.

What threw me was the latency impact by requiring a write to be journaled before returning. Hence wondering if I was doing something wrong or there was a way to obtain the same write functionality without the performance penalty. It doesn't look like there is, so I'm fine with leaving it set at the default.

Thanks,

–Josh
____________________________________
Joshua Kehn | @joshkehn
http://joshuakehn.com