The initial flag value of zero (=BOOST_ONCE_INIT)
is set at load time via static initialization.

It should be safe to do a non-interlocked read of the flag like this:
----------------------------------------------
if(flag!=function_complete_flag_value &&
  ::boost::detail::interlocked_read_acquire(&flag)!=function_complete_flag_va lue)
----------------------------------------------

If our non-interlocked read would see something different than
"function_complete_flag_value", we still do the interlocked read, too.
For normal operation inside a thread safe singleton,
this saves us from always issuing a memory barrier.

Any flaws with this approach?

Also one more (silly?) question: "flag" is not a volatile variable.
Does boost::detail::interlocked_read_acquire() make sure the value
doesn't get cached inside a register? IMHO we lock the mutex
and we might still hold a cached, old value of "flag" in a register.
-> Do we need an interlocked read here, too?
   Or mark the flag type "volatile"?

(http://msdn.microsoft.com/en-us/magazine/cc163405.aspx#S3
 look at figure 7 and the text below)

Your thread will see flag correctly, but the "important data" being
protected/initted by the once, might not bee seen correctly.

Typically the code essentially boils down to (if you imagine the
call_once being "inlned", since to the CPU, it is all "inline"):

First Thread in ("wins", thus does the init)
   important_data = new ...;  // the init call, protected by call_once()
   flag = function_complete_flag_value;

Second Thread:
   if (flag == function_complete_flag_value)  // inside call_once()
       use_important_data(important_data);    // outside call_once()

The CPU (not just the compiler, but the CPU, memory subsystem, etc)
can reorder that as:

First Thread:
   flag = function_complete_flag_value;
   important_data = new ...;

Second Thread:
   register temp = important_data;  // start memory read early, so
that we don't wait for reads to complete
   if (flag == function_complete_flag_value)
       use_important_data(temp);

See the problem?  important_data may be read before it is ready.

You need a memory barrier *in each thread* to ensure neither
reordering happens.  In particular, the memory barrier in the
interlocked_read_acquire(&flag) prevents the reordering in the Second
Thread.

volatile only helps with the compiler - it doesn't control what the
CPU might then do to your instructions (like reorder them, do
speculative execution, etc), so it doesn't help much with threads
running on separate CPUs.  And MS's version of volatile (which _does_
enforce memory barriers) is non-standard.  So don't use it in portable
code.  ie don't use it at all.

On Thursday, 18. November 2010 05:13:03 Gottlob Frege wrote:

One more small thing: IMHO the write reordering in the first
thread can't happen because of the memory barrier created
by interlockedExchange(). The first thread translates to this:

    important_data = new ...;  // the init call

    lock                       // create memory barrier:
                               // - Don't allow reordering
                               //   from below the barrier
                                                           // - Finish all outstanding writes

    flag = function_complete_flag_value;

- register temp = interlocked_read(&flag)     // fetch from mem location xyz
  -> Init flag not set, so execute init code:

  - create mutex (also creates memory barrier)

  - Another thread 2 already entered the mutex,
    executes the init code and does an interlocked
    write of "flag". Then it leaves the mutex
    so thread 1 can continue.

  - Thread 1 re-reads the flag without interlocked read or volatile:
    The compiler recognizes it's the same memory location xyz
    and uses the cached value from "register temp".
    So we would redo the initialization.

    Don't we need an interlocked read here, too?

    Or does the mutex/memory barrier ensure the compiler
    isn't allowed to do register caching?

int main(void)
{
  __sync_synchronize();
  a = *foobar;
  __sync_synchronize();
  return *foobar;

gcc -O4 -S test.c:
main:
.LFB0:
        .cfi_startproc
        mfence
        movq    foobar(%rip), %rax
        movl    (%rax), %eax
        movl    %eax, a(%rip)
        mfence
        movq    foobar(%rip), %rax
        movl    (%rax), %eax
        ret
        .cfi_endproc

-> The memory barrier invalidates the register cache.

Without the second memory barrier in the C source,
the compiler uses the register cached value of *foobar.

Thanks for your help on this again, Tony.

Another way to say it is that you are essentially trying to do the
classic "double-checked locking" (DCL) pattern, which is well-known
not to work and you can find plenty of explanations for.   In fact,
the BOOST_THREAD_ONCE* stuff is there to solve exactly that problem.

There are some papers out there such as "Threads cannot be done in a
library" by Boehm
http://www.stanford.edu/class/cs240/readings/p261-boehm.pdf going into
more detail about this. In particular, exotic cases such as:

if (some_condition)
    lock(mutex);

for (i = 0; i < N; i++)
{
   do_stuff();
   if (some_condition)
       x++;

So the programmer's intent is that x is only updated sometimes, and
that happens to be the only time that the mutex is needed.
This is a case that, without being careful, compilers can optimize
incorrectly.  C++0x will take care of these cases.

But beyond some exotic cases with aggresive optimizations, normal
mutexes with non-volatiles just works.  And I don't think volatiles
fixes the cases listed in the paper.