Do I have to keep txn collection?

[AllenDang]

Hi, 

I use txn a lot to simulate transaction, but about a month later, I found the txn collection grows rapidly, so I wonder do I have to keep it or I could just drop it anytime?

Thanks

[Gustavo Niemeyer]

Hey Allen,

Historical transactions that have absolutely no chance of being in use
or pending may be removed from the collection, so pretty much all of
it can go.

I suggest following the following procedure to clean it up:

- Run runner.ResumeAll(), wait until it returns, and ensure there are
no errors. This means all transactions up to that point were fully
applied.

- Ensure there are no misbehaving processes locked up for a long time
for whatever reason. Such a process might expect to see its own
transaction data before being done.

- Remove all transactions older than a day or so, using something like
bson.M{"_id": bson.M{"$lt": bson.NewObjectIdWithTime(minus24h)}} to
find them.

Glad to hear you've been making good use of txn.


gustavo @ http://niemeyer.net

[AllenDang]

Thanks for your helpful reply! The cleanup procedure is absolutely what I'm looking for!

[kondal boreddy]

Hi Gustavo,

Even after running runner.ResumeAll(), affected documents still refer to last applied transactions. If I delete the old applied transactions, subsequent transactions on these documents failing with "cannot find transaction" error. Is there any step that I am missing here?

Thanks,

Kondal.

[Gustavo Niemeyer]

Heya,

A transaction being applied means that it got committed, but indeed other documents can still refer to that transaction precisely to be able to identify whether there's still something to be done about them or not. These references will be automatically cleaned up by the system once further changes are done in the affected documents.

We could clean these references up after the transaction is applied, but that would mean touching every single document that was part of the transaction once more, which is unnecessary churn. As it is, the cleaning up comes for free on follow ups.

--
You received this message because you are subscribed to the Google Groups "mgo-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mgo-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

gustavo @ http://niemeyer.net

[kondal boreddy]

Thanks. Then how do we identify the applied transactions that can be safely deleted? 

Heya,

To unsubscribe from this group and stop receiving emails from it, send an email to mgo-users+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Gustavo Niemeyer]

In terms of safety, I'd recommend only removing transactions that are in a final state for a good amount of time. The time window protects the system from having lingering runners that were blocked for whatever reason and then wake up and try to continue from where they left off. The exact amount of time will vary from deployment to deployment. For example, if you deploy and restart all processes once a week, it'd be fine to prune transactions in a final state for more than a week that are not being referenced by any existing documents.

Coincidently, I've had that exact conversation just last Friday with the developers of juju (https://www.ubuntu.com/cloud/juju), which uses the txn package heavily. They are pruning transactions, and were doing that in a slightly unsafe way, but will be improving their algorithm soon. It'd be nice if we could get them to collaborate back an implementation which could be merged.

To unsubscribe from this group and stop receiving emails from it, send an email to mgo-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.