[Request] Transactional write-batch support

104 views
Skip to first unread message

Andrew Armstrong

unread,
Mar 20, 2011, 10:44:46 PM3/20/11
to mongodb-user
Hi,

I wanted to bring up a feature request regarding 'write transactions'
or 'atomic/eventually consistent write batch' support.

A big concern I have with a lack of 'transactions' with MongoDB is
that there can be a chance of data inconsistency.

If the primary server dies mid way through updating/inserting multiple
documents (perhaps across multiple shards), then you can get 'corrupt'
data (according to your application) by just missing a few vital
documents in your domain model.

Mongo's internal storage of the saved documents is fine; the only
problem is that some document's didn't get saved before the crash, so
the ones that were saved are not valid in a domain model sense because
they are missing child documents etc.

I understand transactions are not very desirable for performance
reasons (especially due to locking).

With that in mind, what about the concept of 'write batching', a write-
only transaction, where all writes (across any number of shards) occur
at once (eventually consistent) or not at all? Read locks are never
taken; and you can't lock rows.

For example; picking a simple example for demo purposes; I'd want to
write:

--
mongoDriver.StartBatch()
mongoDriver.db.users.insert({username: andrew});
mongoDriver.db.messages.insert({message: "Welcome!", username:
andrew});
mongoDriver.db.stats.update({$inc: { totalUsers: 1 });
mongoDriver.CommitBatch({atomic: bool})
--

I'd like CommitBatch() to make sure that all of my writes in that
batch are either committed; or none at all.

The 'atomic' flag in commit batch would decide whether a 2pc is used
to ensure all commits occur at once across multiple shards. With
atomic = False; commits happen without a 2PC in the sense that no co-
ordination takes place; and an 'eventually consistent' approach is
taken to the commit (normally, these multi-shard changes would all
appear in a few milliseconds anyway).

This feature would let me know for certain that my data model will
remain consistent (either I insert a new user, create a welcome
message, and update my statistics -- or not at all).

This feature CANNOT be used to perform read locking or 'bank balance'
transfers because you can't block readers trying to read a document
mid 'write batch' to evaluate the response - there is no read locking,
your just applying a range of writes all at once or not at all.

I don't see this impacting performance at all (especially with atomic:
false); there are no locks taking place at any time, except for
atomic: true which would introduce a slight delay when a 2pc is
occurring to coordinate a write batch when requested in the rare case
its needed.

--

Conceptually I'd imagine 'begin batch' would mean each shard just logs
any future write() queries to a local temporary collection (such as
local.writebatches.<connection id>).

A request to 'commit batch' asks each shard whether they have finished
writing to the local writebatch collection (or perhaps just always
issue each writebatch insert with safe: true); if all is well, then a
'commit writebatch' command is sent to each shard (without a 2pc,
unless atomic: true was requested) to persist each write by looping
over local.writebatches.<connection id> collection and really
performing the original request.

Some thought needs to be put into failure handling (such as inserting
a 'prepared' flag to the local writebatch collection in the event of
server failure to ensure its "committed" on recovery), but I think
thats not too difficult.

Thoughts? Anyone else have any similar feelings/use cases?

This would be a nice feature to have, it would prevent data
inconsistency issues when you don't want your application to suffer;
and avoids the locking associated with real transactions (that support
read locks, isolation, etc).

Regards
- Andrew

Andreas Jung

unread,
Mar 21, 2011, 1:28:43 AM3/21/11
to mongod...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

You can have only atomicity for single operations. In order to ensure
durability: use the new journaling feature for standalone MongoDB
servers or use a replica set and ensure that the operation has been
carried out on at least N members of the replica set. Those are your
options.

- -aj


- --
ZOPYX Limited | zopyx group
Charlottenstr. 37/1 | The full-service network for Zope & Plone
D-72070 T�bingen | Produce & Publish
www.zopyx.com | www.produce-and-publish.com
- ------------------------------------------------------------------------
E-Publishing, Python, Zope & Plone development, Consulting


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQGUBAEBAgAGBQJNhuIKAAoJEADcfz7u4AZj+XILvjPR7ZajvpaFs2dRDYZ+nZhm
GEpElS1vGf5qj49mfRF5KbP7Wy0qb7mB7zEhT4tjSnj8kviszlXiRBHxynjprV5Y
xlxGdOHjt6329pBvu5H+VdIA4U03cTbZR6NOtMpskNpyYrvptFrCZ1no3t4/s2S5
jGvs5Fhl0GAJwJkvUlTQdySzCKzEcT74o14GYPHGZ6vUAgwlHfceGJDYr9JCbwsS
NTvnsrHW13YkMLZbZYI5IDgimbDTvTPxeTIWtcVnH6d+uGXSXkIRo11/qRL2xm79
CF97Vi0j7zNi+QXayhSGv06tOaHnnsAG2ws7xeKxbz20cnI4V7dAhPadLiSUVwZV
H6gb09t9A4h8ieK6bV6VzS4p2CfVYCwAJ+Sl2moMt7Uo0/THF05QksrF7aILF3Gq
lsbfnegcjtEf8aHuxDWYNT6gKI8qhOTjnPc09jtM62cRAV5ZaISxy9o4B3u+ZUZJ
/sPtLQlKz3nuCJMgT3yW1NkDzLfDGHA=
=mIFY
-----END PGP SIGNATURE-----

lists.vcf

Andrew Armstrong

unread,
Mar 21, 2011, 3:35:19 AM3/21/11
to mongodb-user
I'm suggesting a new feature that doesn't exist yet, along with
reasons why.
> D-72070 T�bingen        | Produce & Publishwww.zopyx.com          |www.produce-and-publish.com
> - ------------------------------------------------------------------------
> E-Publishing, Python, Zope & Plone development, Consulting
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.11 (Darwin)
> Comment: Using GnuPG with Mozilla -http://enigmail.mozdev.org/
>
> iQGUBAEBAgAGBQJNhuIKAAoJEADcfz7u4AZj+XILvjPR7ZajvpaFs2dRDYZ+nZhm
> GEpElS1vGf5qj49mfRF5KbP7Wy0qb7mB7zEhT4tjSnj8kviszlXiRBHxynjprV5Y
> xlxGdOHjt6329pBvu5H+VdIA4U03cTbZR6NOtMpskNpyYrvptFrCZ1no3t4/s2S5
> jGvs5Fhl0GAJwJkvUlTQdySzCKzEcT74o14GYPHGZ6vUAgwlHfceGJDYr9JCbwsS
> NTvnsrHW13YkMLZbZYI5IDgimbDTvTPxeTIWtcVnH6d+uGXSXkIRo11/qRL2xm79
> CF97Vi0j7zNi+QXayhSGv06tOaHnnsAG2ws7xeKxbz20cnI4V7dAhPadLiSUVwZV
> H6gb09t9A4h8ieK6bV6VzS4p2CfVYCwAJ+Sl2moMt7Uo0/THF05QksrF7aILF3Gq
> lsbfnegcjtEf8aHuxDWYNT6gKI8qhOTjnPc09jtM62cRAV5ZaISxy9o4B3u+ZUZJ
> /sPtLQlKz3nuCJMgT3yW1NkDzLfDGHA=
> =mIFY
> -----END PGP SIGNATURE-----
>
>  lists.vcf
> < 1KViewDownload

Andreas Jung

unread,
Mar 21, 2011, 4:04:16 AM3/21/11
to mongod...@googlegroups.com
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Use Jira to file feature requests.

- -aj

Andrew Armstrong wrote:
> I'm suggesting a new feature that doesn't exist yet, along with
> reasons why.
>
> On Mar 21, 4:28 pm, Andreas Jung <li...@zopyx.com> wrote:
> You can have only atomicity for single operations. In order to ensure
> durability: use the new journaling feature for standalone MongoDB
> servers or use a replica set and ensure that the operation has been
> carried out on at least N members of the replica set. Those are your
> options.
>

lists.vcf
< 1KViewDownload

- --
ZOPYX Limited | zopyx group
Charlottenstr. 37/1 | The full-service network for Zope & Plone

D-72070 T�bingen | Produce & Publish

www.zopyx.com | www.produce-and-publish.com


- ------------------------------------------------------------------------
E-Publishing, Python, Zope & Plone development, Consulting


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQGUBAEBAgAGBQJNhwaAAAoJEADcfz7u4AZju/oLvR98N9KXlq5GQYSXsQUqNS2p
PQXinLK71tpkG/CpZcR32pDlHErV5Q07PnI6QNJjIhRMmp0QVo0FQoKATzicu2D9
g30EAiA38NLHrLQ2HcZF/E4mAwhW3rbz1/a4DKyx0EEVFHwpTf02ZNfo7ekr2mnq
gaC9tuAkjrgkGlMvozTKNmMYsXIjtOkuw7g4Slf1IITO6CJPH2qZY9V2d5G0di76
J+l42b57STI1bG49zOAe7KgfWAp9iZT79l5MMMv7oQaCUluRy6DMZVx0IL+kR4AK
3MwHCaDz366kBOg02ePaNr4ajnaLdgIHpNuRq+9UTm9dLqopjt/Fdrua5eF9tUIw
dERMAAQM/6E6wd5rQN9yPgXqOU1n8VCA7rib+7TuKW0yUUN5BA1xN8lmw0iYjPQa
OmevsX6L2o5XthKqNbAO5mslh7InaKHYbQJBeG1XdETW3w6eiZQjSZYT4OHRAU5e
qJOuCrxaL/SAzcJOKLX4PLl48pTSeto=
=WFRv
-----END PGP SIGNATURE-----

lists.vcf

Andrew Armstrong

unread,
Mar 21, 2011, 5:56:50 AM3/21/11
to mongodb-user
Thanks, I've made http://jira.mongodb.org/browse/SERVER-2804 which
sounds better.
> D-72070 T�bingen        | Produce & Publishwww.zopyx.com          |www.produce-and-publish.com
> - ------------------------------------------------------------------------
> E-Publishing, Python, Zope & Plone development, Consulting
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.11 (Darwin)
> Comment: Using GnuPG with Mozilla -http://enigmail.mozdev.org/
>
> iQGUBAEBAgAGBQJNhwaAAAoJEADcfz7u4AZju/oLvR98N9KXlq5GQYSXsQUqNS2p
> PQXinLK71tpkG/CpZcR32pDlHErV5Q07PnI6QNJjIhRMmp0QVo0FQoKATzicu2D9
> g30EAiA38NLHrLQ2HcZF/E4mAwhW3rbz1/a4DKyx0EEVFHwpTf02ZNfo7ekr2mnq
> gaC9tuAkjrgkGlMvozTKNmMYsXIjtOkuw7g4Slf1IITO6CJPH2qZY9V2d5G0di76
> J+l42b57STI1bG49zOAe7KgfWAp9iZT79l5MMMv7oQaCUluRy6DMZVx0IL+kR4AK
> 3MwHCaDz366kBOg02ePaNr4ajnaLdgIHpNuRq+9UTm9dLqopjt/Fdrua5eF9tUIw
> dERMAAQM/6E6wd5rQN9yPgXqOU1n8VCA7rib+7TuKW0yUUN5BA1xN8lmw0iYjPQa
> OmevsX6L2o5XthKqNbAO5mslh7InaKHYbQJBeG1XdETW3w6eiZQjSZYT4OHRAU5e
> qJOuCrxaL/SAzcJOKLX4PLl48pTSeto=
> =WFRv
> -----END PGP SIGNATURE-----
>
>  lists.vcf
> < 1KViewDownload
Reply all
Reply to author
Forward
0 new messages