Newbie, Manual/Doc Question, Perform Two Phase Commits

[sgon]

Hi, I can not really follow the official manual/doc at the url: http://docs.mongodb.org/manual/tutorial/perform-two-phase-commits/

In section "Recovering from Failure Scenarios", 

all failures that occur after the first step (i.e. setting the transaction set to initial) but before the third step (i.e. applying the transaction to both accounts.)

To recover, applications should get a list of transactions in the pending state and resume from the second step (i.e. switching the transaction state to pending.)

This doesn't make sense to me. The final operation before the third step is to set the state to pending. That means if you get a list of transactions in pending state, you are successful before the third step. But this paragraph is talking about failure before the third step. This is a contradiction.

In "Rollback" section,

 

After you create the transaction (i.e. the first step), but before you apply the transaction (i.e the third step), use the following process:

...

...

Undo the Transaction

Use the following sequence of operations to undo the transaction operation from both accounts:

db.accounts.update({name: t.source, pendingTransactions: t._id}, {$inc: {balance: t.value}, $pull: {pendingTransactions: t._id}})

db.accounts.update({name: t.destination, pendingTransactions: t._id}, {$inc: {balance: -t.value}, $pull: {pendingTransactions: t._id}})

db.accounts.find()

This does't make sense to me either. Before you apply the transaction (the third step), the transaction operation from both accounts never happens yet. That transaction operation happens in the third step. Thus before the third step, transaction operation from both accounts should not be undone because it's never done yet. 

Can anyone tell me if I am wrong or the manual is wrong? 

Thank you very much. 

[Asya Kamsky]

Here is the key. The even steps (the ones on the accounts
collection) are idempotent. That means that you can apply them more
than once and they will have the same effect as if you apply them
once.

The rollback step has the same attribute. Whether it's done once or
several times, it will have the same effect. And whether the
transactions were applies just up to step three or not, the rollback
will have the same effect.

Notice that the query condition includes check of pending
transactions. Since we can't know _when_ before step three we failed,
we can safely try to roll back any steps 2 action - if there were no
pending transactions pushed, then we won't have anything to pull. So
the pending transaction array in accounts document ensures we only do
the $inc once and same for reverse of it.

I encourage you to actually try it - go through the steps and try the
scenario that you feel would not work correctly and see what result
you get - I believe you will get correct result, whether the
transaction was applied or not. That's the key to the correctness of
the algorithm, it does not depend on knowing exactly which statement
failed to execute.

Asya

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user"
> group.
>
> For other MongoDB technical support options, see:
> http://www.mongodb.org/about/support/.
> ---
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mongodb-user...@googlegroups.com.
> To post to this group, send email to mongod...@googlegroups.com.
> Visit this group at http://groups.google.com/group/mongodb-user.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mongodb-user/2c53e8c0-e4e5-4ca3-b085-8f870622e216%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[sgon]

Dear Asya, 

Again, Thank you very much for answering my two posts. But you are not answering my question.

I don't think I need to repost my question here. 

My first question is that the manual is a contradiction. The manual is saying I should query pending state document and set those documents to pending state. what is that? if the state is pending, why should I set to pending again? 

My second question (rollback): the manual ask me to undo the transaction before you apply the transaction. This is another contradiction. why should you undo a transaction which never happens? Rollback something that never happens? what is that? if something never happens, why should you rollback? Look at the mango operation codes, they are actually increasing and decreasing account balance. The manual thought those balance changed should rollback, but the above condition is saying that the balance never changes yet. 

I am not a native English speaker, hopefully my words make sense. 

Thank you very much. 

[Asya Kamsky]

I did try to answer your question, and I don't think I made myself clear enough.

There is *no* contradiction. Here is why:

> My first question is that the manual is a contradiction.
> The manual is saying I should query pending state document and set those documents to pending state. what is that?
> if the state is pending, why should I set to pending again?

I think you're misreading the query part:

db.accounts.update(
{ name: t.source, pendingTransactions: { $ne: t._id } },
{ $inc: { balance: -t.value }, $push: { pendingTransactions: t._id } }
)
db.accounts.update(
{ name: t.destination, pendingTransactions: { $ne: t._id } },
{ $inc: { balance: t.value }, $push: { pendingTransactions: t._id } }
)

This says update the account if pendingTransactions NOT EQUALS
transaction id ($ne:
http://docs.mongodb.org/manual/reference/operator/query/ne/)

> My second question (rollback): the manual ask me to undo the transaction before you apply the transaction.
> This is another contradiction. why should you undo a transaction which never happens?

This is not a contradiction because of the query condition for the update:

db.accounts.update({name: t.source, pendingTransactions: t._id},
{$inc: {balance: t.value}, $pull: {pendingTransactions: t._id}})
db.accounts.update({name: t.destination, pendingTransactions: t._id},
{$inc: {balance: -t.value}, $pull: {pendingTransactions: t._id}})

Note that the query includes the account name *AND* pendingTransaction
id. If the transaction never happened, this update will find ZERO
documents to update.
If the transaction did happen, and needs to be rolled back, then the
first time you run this update, it will find the transaction id and it
will update and balance *and* pull the transaction id from the pending
array *ATOMICALLY* so that if you run the same update again, it will
not find a matching document so there is no risk of doing the rollback
more than once.

The reason I said you should just try it (follow along with the steps
in the manual cutting and pasting every operation) is that you can
then *see* what happens.

Another way of thinking about those update statements is like they are
saying to the DB to do this:


For applying transaction:
IF the transaction has not yet been applied, then { adjust balance and
add transaction id indicating that it happened }
Otherwise (if transaction already happened) do nothing.

For applying the rollback of the transaction:
IF the transaction was applied to this account, then { adjust the
balance and remove indication that transaction happened }
IF the transaction was not applied then do nothing.

Asya

> To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/28366b51-81c3-4d5c-a4b8-7700b54accdf%40googlegroups.com.

[sgon]

First, thank you very much for your reply and your time. 

Second, I don't know if I should reply you or not. Both we completely don't understand each other. 

Can I doubt about the doc? Will a doc be 100% correct?

Please check the following inline reply if you want to go detail again with me or you can ignore my reply if you are getting crazy with me :). 

I am very sorry that we couldn't understand each other.

On Monday, June 2, 2014 3:04:39 AM UTC+8, Asya Kamsky wrote:

I did try to answer your question, and I don't think I made myself clear enough.
There is *no* contradiction.  Here is why:
> My first question is that the manual is a contradiction.
> The manual is saying I should query pending state document and set those documents to pending state. what is that?
> if the state is pending, why should I set to pending again?
I think you're misreading the query part:
db.accounts.update(
   { name: t.source, pendingTransactions: { $ne: t._id } },
   { $inc: { balance: -t.value }, $push: { pendingTransactions: t._id } }
)
db.accounts.update(
   { name: t.destination, pendingTransactions: { $ne: t._id } },
   { $inc: { balance: t.value }, $push: { pendingTransactions: t._id } }
)
This says update the account if  pendingTransactions NOT EQUALS
transaction id 

I have no idea why you show me this update query. I was talking about something before this update query. 

The update query is at third step. BUT what I was talking about it's before third step. 

The official doc says before third step, you should query pending state documents and then update those documents to pending state. The following quote is from the official doc:

To recover, applications should get a list of transactions in the pending state and resume from the second step (i.e. switching the transaction state to pending.) 

 

At the second step, the code "db.transactions.update({_id: t._id}, {$set: {state: "pending"}})".

The doc is very weird. It is asking to get a list of transactions in the pending state and then resume from the second step and set those state to pending state. Please explain this quote/sentence instead of giving me some code examples in later steps which it's not my question at all. If a document is at pending state, why do you want to set the state to pending state again? this does not make any sense. 

> My second question (rollback): the manual ask me to undo the transaction before you apply the transaction.
> This is another contradiction. why should you undo a transaction which never happens?
This is not a contradiction because of the query condition for the update:
db.accounts.update({name: t.source, pendingTransactions: t._id},
{$inc: {balance: t.value}, $pull: {pendingTransactions: t._id}})
db.accounts.update({name: t.destination, pendingTransactions: t._id},
{$inc: {balance: -t.value}, $pull: {pendingTransactions: t._id}})
Note that the query includes the account name *AND* pendingTransaction
id.  If the transaction never happened, this update will find ZERO
documents to update.

The following quote is from the official doc:

After you create the transaction (i.e. the first step), but before you apply the transaction (i.e the third step), use the following process: 

Set Transaction State to Canceling
...
Undo the Transaction
... 

 

You're right with one thing "If the transaction never happened, this update will find ZERO documents to update.". This is true. BUT from the documentation, before the third step, the transaction 100% never happened. If this transaction never happens, why to undo? This does not make sense. The transaction happens at third step. Before the third step, the transaction never happens. I am not asking if that query will update the wrong documents or not. I am talking about the doc is wrong. Why does the doc say "before the third step (before apply the transaction), undo the transaction" ?

[Asya Kamsky]

Okay, now I understand why you are misunderstanding me - the doc
screwed up the numbering of steps.

The numbering referred to in the "handling failure" section is not
consistent with the number of steps.
I opened a DOCS bug to have this fixed (which turned out to be a
duplicate of DOCS-3464).

Each "phase" in the transaction consists of *multiple steps*.

Here are the steps and *MY* numbers on the left, and original numbers
on the right in parens.

1. create initial transaction. (called "first step")
2. set transaction document to "pending" (called "second step")
3a. update balance in first account with pending transaction id
(called "third step")
3b. update balance in second account with pending transaction id
(called "third step")
4. update transaction document to "committed"
5a. removing pending state from first account (called "fourth step")
5b. remove pending state from second account (called "fourth step")
6. update transaction document to indicate it's done and the timestamp
when it was done. (called "fifth step")

Here are the corrections to that page (hopefully they will be
incorporated soon):

2. should be db.transactions({_id:t._id, state:"initial"},
{$set:{state:"pending"}})

Step numbering basically to refer to steps that change states of
transaction only.

So you can recover between 2 and 4, and between 4 and 6 and for
rollback it's the same - between steps four and six.

Asya

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user"
> group.
>
> For other MongoDB technical support options, see:
> http://www.mongodb.org/about/support/.
> ---
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mongodb-user...@googlegroups.com.
> To post to this group, send email to mongod...@googlegroups.com.
> Visit this group at http://groups.google.com/group/mongodb-user.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/mongodb-user/8c5653ef-14d1-414a-a231-0bb9826eb053%40googlegroups.com.