Multi-threaded transactions

[Pradeep Fernando]

Hi all,

I followed this article, published in 2015 - https://pmem.io/2015/09/16/mt-tx.html.

I have couple of questions related to the $subject.

- Does pmem support thread-local undo log as mentioned in the article?

- The current pmem multi-threaded transaction handling design is consistent with the above article? Has there been any major design changes?

thanks in advance,

--Pradeep

--

Pradeep Fernando.

[Piotr Balcer]

I followed this article, published in 2015 - https://pmem.io/2015/09/16/mt-tx.html.

I have couple of questions related to the $subject.

- Does pmem support thread-local undo log as mentioned in the article?

- The current pmem multi-threaded transaction handling design is consistent with the above article? Has there been any major design changes?

Hi,

That article wasn't meant as a description of libpmemobj's design but rather an idea how to extend it.

Current API does not allow multiple threads contributing to a single undo-log based transaction. It does

employ a thread-local undo log.

Let me answer your second question with with a famous quote from Jurassic Park:

"your scientists were so preoccupied with whether or not they could that they didn't stop to think if they should."

(not that I call myself a scientist...)

I wrote that blog post wondering about how to implement a cool feature, without stopping to think how such feature could ever be used in practice.

And in hindsight, I don't think undo logs are well suited for parallel processing of data because they require exclusive write access to the logged data

until the transaction is finished, otherwise it would not be possible to perform a runtime abort. This means that all threads that collaborate on a single

transaction would have to wait (or at least not drop the locks) for the transaction to be finished, resulting in questionable scaling.

And so, we've focused on optimizing our thread exclusive undo log transactions, and we supplemented it with a redo log API that enables application

to prepare work in many threads, and then publish that work in a single transaction.

http://pmem.io/pmdk/manpages/linux/master/libpmemobj/pmemobj_action.3

This API lends itself nicely to the modern asynchronous programming (async/await).

I hope this makes sense! If not, I'd like to hear your perspective on the topic.

Piotr

[Pradeep Fernando]

Thanks Piotr!. Very helpful explanation...

--Pradeep

On Mon, Jul 8, 2019 at 8:39 AM Piotr Balcer <pi...@balcer.eu> wrote:

I'm afraid we don't have a public documentation of the redo-log transactions implementation. The man page I linked to is the description of the API that uses it.

The mechanism is very simple: each thread creates its own redo log of changes it wants to perform (this includes allocations or deallocations) and then it must send it to the thread that will apply it in a single transaction. It's up to the application to avoid conflicts - libpmemobj does not implement STM, all it does it is that it guarantees that all published operations will be fail-safe atomic. Here's pseudo-code that illustrates what I mean:

void *worker(void *) {

  struct pobj_action *act = malloc(10 * sizeof(struct pobj_action));

  pmemobj_reserve(..., &act[0], ...);

  ...

  return act;

}

TX_BEGIN(...) {

  struct pobj_action act[50];

  for (int i = 0; i < 5; ++i) th[i] = pthread_create(worker);

  for (int i = 0; i < 5; ++i) {

     struct pobj_action *act;

     pthread_join(th[i], &act);

     memcpy(act[i * 10], act, 10 * sizeof(struct pobj_action));

  }

 pmemobj_tx_publish(act, 50);

} TX_END;

This example first creates 5 threads that each create 10 redo log actions. Once all threads are done, all 50 actions are published atomically inside of a single transaction.  All potentially conflicting operations should be performed in the thread that runs the transaction - this avoids synchronization.

I wrote this in C, but this becomes much more managable in C++ and other modern programming languages that natively implement async/await programming model.

pt., 5 lip 2019 o 16:16 Pradeep Fernando <prad...@gmail.com> napisał(a):

Piotr,

Thanks for the explanation.!. 

Spare my ignorance, but is there a pointer (article) that describe this redo-log optimization? Or is it the code itself.

- Is the optimization related to some kind of versioning scheme, where multiple threads resolve conflicts at the commit time? (multi-versioning)

- Or is it some sort of update buffering using redo log, before finally applied to existing undo log based transaction mechanism? Even so, how do you detect conflicts (e.g two thread planning/buffering reading/writing to same memory range in parallel) during writes?

thanks

--Pradeep

--
You received this message because you are subscribed to the Google Groups "pmem" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pmem+uns...@googlegroups.com.
To post to this group, send email to pm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pmem/ff3def75-9c76-4030-b5d4-bcccd3a0dde3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Pradeep Fernando.

--

Pradeep Fernando.