[Request] Transactional write-batch support
Andrew Armstrong
I wanted to bring up a feature request regarding 'write transactions'
or 'atomic/eventually consistent write batch' support.
A big concern I have with a lack of 'transactions' with MongoDB is
that there can be a chance of data inconsistency.
If the primary server dies mid way through updating/inserting multiple
documents (perhaps across multiple shards), then you can get 'corrupt'
data (according to your application) by just missing a few vital
documents in your domain model.
Mongo's internal storage of the saved documents is fine; the only
problem is that some document's didn't get saved before the crash, so
the ones that were saved are not valid in a domain model sense because
they are missing child documents etc.
I understand transactions are not very desirable for performance
reasons (especially due to locking).
With that in mind, what about the concept of 'write batching', a write-
only transaction, where all writes (across any number of shards) occur
at once (eventually consistent) or not at all? Read locks are never
taken; and you can't lock rows.
For example; picking a simple example for demo purposes; I'd want to
write:
--
mongoDriver.StartBatch()
mongoDriver.db.users.insert({username: andrew});
mongoDriver.db.messages.insert({message: "Welcome!", username:
andrew});
mongoDriver.db.stats.update({$inc: { totalUsers: 1 });
mongoDriver.CommitBatch({atomic: bool})
--
I'd like CommitBatch() to make sure that all of my writes in that
batch are either committed; or none at all.
The 'atomic' flag in commit batch would decide whether a 2pc is used
to ensure all commits occur at once across multiple shards. With
atomic = False; commits happen without a 2PC in the sense that no co-
ordination takes place; and an 'eventually consistent' approach is
taken to the commit (normally, these multi-shard changes would all
appear in a few milliseconds anyway).
This feature would let me know for certain that my data model will
remain consistent (either I insert a new user, create a welcome
message, and update my statistics -- or not at all).
This feature CANNOT be used to perform read locking or 'bank balance'
transfers because you can't block readers trying to read a document
mid 'write batch' to evaluate the response - there is no read locking,
your just applying a range of writes all at once or not at all.
I don't see this impacting performance at all (especially with atomic:
false); there are no locks taking place at any time, except for
atomic: true which would introduce a slight delay when a 2pc is
occurring to coordinate a write batch when requested in the rare case
its needed.
--
Conceptually I'd imagine 'begin batch' would mean each shard just logs
any future write() queries to a local temporary collection (such as
local.writebatches.<connection id>).
A request to 'commit batch' asks each shard whether they have finished
writing to the local writebatch collection (or perhaps just always
issue each writebatch insert with safe: true); if all is well, then a
'commit writebatch' command is sent to each shard (without a 2pc,
unless atomic: true was requested) to persist each write by looping
over local.writebatches.<connection id> collection and really
performing the original request.
Some thought needs to be put into failure handling (such as inserting
a 'prepared' flag to the local writebatch collection in the event of
server failure to ensure its "committed" on recovery), but I think
thats not too difficult.
Thoughts? Anyone else have any similar feelings/use cases?
This would be a nice feature to have, it would prevent data
inconsistency issues when you don't want your application to suffer;
and avoids the locking associated with real transactions (that support
read locks, isolation, etc).
Regards
- Andrew
Andreas Jung
Hash: SHA1
You can have only atomicity for single operations. In order to ensure
durability: use the new journaling feature for standalone MongoDB
servers or use a replica set and ensure that the operation has been
carried out on at least N members of the replica set. Those are your
options.
- -aj
- --
ZOPYX Limited | zopyx group
Charlottenstr. 37/1 | The full-service network for Zope & Plone
D-72070 T�bingen | Produce & Publish
www.zopyx.com | www.produce-and-publish.com
- ------------------------------------------------------------------------
E-Publishing, Python, Zope & Plone development, Consulting
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQGUBAEBAgAGBQJNhuIKAAoJEADcfz7u4AZj+XILvjPR7ZajvpaFs2dRDYZ+nZhm
GEpElS1vGf5qj49mfRF5KbP7Wy0qb7mB7zEhT4tjSnj8kviszlXiRBHxynjprV5Y
xlxGdOHjt6329pBvu5H+VdIA4U03cTbZR6NOtMpskNpyYrvptFrCZ1no3t4/s2S5
jGvs5Fhl0GAJwJkvUlTQdySzCKzEcT74o14GYPHGZ6vUAgwlHfceGJDYr9JCbwsS
NTvnsrHW13YkMLZbZYI5IDgimbDTvTPxeTIWtcVnH6d+uGXSXkIRo11/qRL2xm79
CF97Vi0j7zNi+QXayhSGv06tOaHnnsAG2ws7xeKxbz20cnI4V7dAhPadLiSUVwZV
H6gb09t9A4h8ieK6bV6VzS4p2CfVYCwAJ+Sl2moMt7Uo0/THF05QksrF7aILF3Gq
lsbfnegcjtEf8aHuxDWYNT6gKI8qhOTjnPc09jtM62cRAV5ZaISxy9o4B3u+ZUZJ
/sPtLQlKz3nuCJMgT3yW1NkDzLfDGHA=
=mIFY
-----END PGP SIGNATURE-----
Andrew Armstrong
reasons why.
> E-Publishing, Python, Zope & Plone development, Consulting
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.11 (Darwin)
> iQGUBAEBAgAGBQJNhuIKAAoJEADcfz7u4AZj+XILvjPR7ZajvpaFs2dRDYZ+nZhm
> GEpElS1vGf5qj49mfRF5KbP7Wy0qb7mB7zEhT4tjSnj8kviszlXiRBHxynjprV5Y
> xlxGdOHjt6329pBvu5H+VdIA4U03cTbZR6NOtMpskNpyYrvptFrCZ1no3t4/s2S5
> jGvs5Fhl0GAJwJkvUlTQdySzCKzEcT74o14GYPHGZ6vUAgwlHfceGJDYr9JCbwsS
> NTvnsrHW13YkMLZbZYI5IDgimbDTvTPxeTIWtcVnH6d+uGXSXkIRo11/qRL2xm79
> CF97Vi0j7zNi+QXayhSGv06tOaHnnsAG2ws7xeKxbz20cnI4V7dAhPadLiSUVwZV
> H6gb09t9A4h8ieK6bV6VzS4p2CfVYCwAJ+Sl2moMt7Uo0/THF05QksrF7aILF3Gq
> lsbfnegcjtEf8aHuxDWYNT6gKI8qhOTjnPc09jtM62cRAV5ZaISxy9o4B3u+ZUZJ
> /sPtLQlKz3nuCJMgT3yW1NkDzLfDGHA=
> =mIFY
> -----END PGP SIGNATURE-----
>
> < 1KViewDownload
Andreas Jung
Hash: SHA1
Use Jira to file feature requests.
- -aj
Andrew Armstrong wrote:
> I'm suggesting a new feature that doesn't exist yet, along with
> reasons why.
>
> On Mar 21, 4:28 pm, Andreas Jung <li...@zopyx.com> wrote:
> You can have only atomicity for single operations. In order to ensure
> durability: use the new journaling feature for standalone MongoDB
> servers or use a replica set and ensure that the operation has been
> carried out on at least N members of the replica set. Those are your
> options.
>
lists.vcf
< 1KViewDownload
- --
ZOPYX Limited | zopyx group
Charlottenstr. 37/1 | The full-service network for Zope & Plone
D-72070 T�bingen | Produce & Publish
www.zopyx.com | www.produce-and-publish.com
- ------------------------------------------------------------------------
E-Publishing, Python, Zope & Plone development, Consulting
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQGUBAEBAgAGBQJNhwaAAAoJEADcfz7u4AZju/oLvR98N9KXlq5GQYSXsQUqNS2p
PQXinLK71tpkG/CpZcR32pDlHErV5Q07PnI6QNJjIhRMmp0QVo0FQoKATzicu2D9
g30EAiA38NLHrLQ2HcZF/E4mAwhW3rbz1/a4DKyx0EEVFHwpTf02ZNfo7ekr2mnq
gaC9tuAkjrgkGlMvozTKNmMYsXIjtOkuw7g4Slf1IITO6CJPH2qZY9V2d5G0di76
J+l42b57STI1bG49zOAe7KgfWAp9iZT79l5MMMv7oQaCUluRy6DMZVx0IL+kR4AK
3MwHCaDz366kBOg02ePaNr4ajnaLdgIHpNuRq+9UTm9dLqopjt/Fdrua5eF9tUIw
dERMAAQM/6E6wd5rQN9yPgXqOU1n8VCA7rib+7TuKW0yUUN5BA1xN8lmw0iYjPQa
OmevsX6L2o5XthKqNbAO5mslh7InaKHYbQJBeG1XdETW3w6eiZQjSZYT4OHRAU5e
qJOuCrxaL/SAzcJOKLX4PLl48pTSeto=
=WFRv
-----END PGP SIGNATURE-----
Andrew Armstrong
sounds better.
> E-Publishing, Python, Zope & Plone development, Consulting
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.11 (Darwin)
> iQGUBAEBAgAGBQJNhwaAAAoJEADcfz7u4AZju/oLvR98N9KXlq5GQYSXsQUqNS2p
> PQXinLK71tpkG/CpZcR32pDlHErV5Q07PnI6QNJjIhRMmp0QVo0FQoKATzicu2D9
> g30EAiA38NLHrLQ2HcZF/E4mAwhW3rbz1/a4DKyx0EEVFHwpTf02ZNfo7ekr2mnq
> gaC9tuAkjrgkGlMvozTKNmMYsXIjtOkuw7g4Slf1IITO6CJPH2qZY9V2d5G0di76
> J+l42b57STI1bG49zOAe7KgfWAp9iZT79l5MMMv7oQaCUluRy6DMZVx0IL+kR4AK
> 3MwHCaDz366kBOg02ePaNr4ajnaLdgIHpNuRq+9UTm9dLqopjt/Fdrua5eF9tUIw
> dERMAAQM/6E6wd5rQN9yPgXqOU1n8VCA7rib+7TuKW0yUUN5BA1xN8lmw0iYjPQa
> OmevsX6L2o5XthKqNbAO5mslh7InaKHYbQJBeG1XdETW3w6eiZQjSZYT4OHRAU5e
> qJOuCrxaL/SAzcJOKLX4PLl48pTSeto=
> =WFRv
> -----END PGP SIGNATURE-----
>
> < 1KViewDownload