Atomic int increment/decrement

[Olivier Guilyardi]

Hi,

I need to have multiple threads increment/decrement an int atomically, and I
would like to avoid locking for performance reasons.

I thought about using memory barriers, but I'm not used to that. How should I
proceed?

Or should/can I rely on gcc atomic builtins?

In the end, I need it to be as fast as possible.

--
Olivier

[mic _]

ARM doesn't support arithmetic operations like ADD/SUB on memory locations. The value needs to be read from memory, modified, and written back in separate instructions. I'm not an expert on threads but it seems to me like the scheduler could switch to another thread at any of those three steps if you leave the variable without any kind of lock.

/Michael

--
You received this message because you are subscribed to the Google Groups "android-ndk" group.
To post to this group, send email to andro...@googlegroups.com.
To unsubscribe from this group, send email to android-ndk...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/android-ndk?hl=en.

[Olivier Guilyardi]

What about memory barriers then? Something like this:
http://www.keil.com/support/man/docs/armasm/armasm_cihjfgfe.htm

On 06/18/2010 03:55 PM, mic _ wrote:
> ARM doesn't support arithmetic operations like ADD/SUB on memory
> locations. The value needs to be read from memory, modified, and written
> back in separate instructions. I'm not an expert on threads but it seems
> to me like the scheduler could switch to another thread at any of those
> three steps if you leave the variable without any kind of lock.
>
> /Michael
>
> On Fri, Jun 18, 2010 at 3:05 PM, Olivier Guilyardi <li...@samalyse.com
> <mailto:li...@samalyse.com>> wrote:
>
> Hi,
>
> I need to have multiple threads increment/decrement an int
> atomically, and I
> would like to avoid locking for performance reasons.
>
> I thought about using memory barriers, but I'm not used to that. How
> should I
> proceed?
>
> Or should/can I rely on gcc atomic builtins?
>
> In the end, I need it to be as fast as possible.
>
> --
> Olivier
>
>
>
>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "android-ndk" group.
> To post to this group, send email to andro...@googlegroups.com

> <mailto:andro...@googlegroups.com>.

> To unsubscribe from this group, send email to
> android-ndk...@googlegroups.com

> <mailto:android-ndk%2Bunsu...@googlegroups.com>.

[Chris]

Hi,

maybee the atomic operations atomic_add(), atomic_inc() can halp
you... see more information at http://www.ibm.com/developerworks/linux/library/l-linux-synchronization.html

cheers,
Chris

[fadden]

On Jun 18, 7:16 am, Olivier Guilyardi <l...@samalyse.com> wrote:
> What about memory barriers then? Something like this:http://www.keil.com/support/man/docs/armasm/armasm_cihjfgfe.htm

Memory barriers don't help here, and aren't needed in uniprocessor
ARM.

Atomic operations on ARMv6 and later can be implemented with LDREX/
STREX; referencing a different page on that same site:

http://www.keil.com/support/man/docs/armasm/armasm_cihbghef.htm

Be really sure you want to do this. Memory barriers *do* matter on
SMP hardware, and the penalties for getting it wrong can be subtle.
The pthread calls will always be correct.

I'm not sure offhand what the state of the gcc builtins is.

[Olivier Guilyardi]

On 06/22/2010 12:48 AM, fadden wrote:
> On Jun 18, 7:16 am, Olivier Guilyardi <l...@samalyse.com> wrote:
>> What about memory barriers then? Something like this:http://www.keil.com/support/man/docs/armasm/armasm_cihjfgfe.htm
>
> Memory barriers don't help here, and aren't needed in uniprocessor
> ARM.

Right, that was irrelevant.

>
> Atomic operations on ARMv6 and later can be implemented with LDREX/
> STREX; referencing a different page on that same site:
>
> http://www.keil.com/support/man/docs/armasm/armasm_cihbghef.htm
>
> Be really sure you want to do this. Memory barriers *do* matter on
> SMP hardware, and the penalties for getting it wrong can be subtle.
> The pthread calls will always be correct.
>
> I'm not sure offhand what the state of the gcc builtins is.

I tried to use __sync_fetch_and_add(), compiling was ok, but it wasn't found
when linking.

The trouble with LDREX/STREX is that this isn't supported on ARMv5.

So... I made quite a lot of tests, using:
- a custom atomic_add(): http://lwn.net/Articles/314561/
- Bionic's __atomic_inc() and __atomic_dec() from NDK's <sys/atomics.h>
- and using a semaphore as lock, for performance comparison

The code is here:
http://svn.samalyse.com/android/atomics/jni/atomicstest.c

It starts two threads one which increment an int N times, and another N-1 times.
A result of 1 is /likely/ to indicate success.

You can actually run it with:
$ http://svn.samalyse.com/android/atomics
$ cd atomics
$ ./bootstrap.sh
$ make

This assumes that ndk-build, ant and adb are on your PATH, and that you have a
single device plugged in.

Here are the results on a HTC Magic 32B, running Android 1.6:

=============== Starting Atomics Tests ===============
ADD thread starting for 10000000 iterations using simple incrementation
SUB thread starting for 9999999 iterations using simple decrementation
ADD thread done in 2183ms
SUB thread done in 2190ms
Threads joined, result: 755919 (FAILURE)
ADD thread starting for 10000000 iterations using custom atomic_add()
SUB thread starting for 9999999 iterations using custom atomic_add()
ADD thread done in 3240ms
SUB thread done in 3226ms
Threads joined, result: 1 (SUCCESS)
ADD thread starting for 10000000 iterations using Bionic's __atomic_int()
SUB thread starting for 9999999 iterations using Bionic's __atomic_dec()
ADD thread done in 4563ms
SUB thread done in 4553ms
Threads joined, result: 1 (SUCCESS)
ADD thread starting for 10000000 iterations using locks
SUB thread starting for 9999999 iterations using locks
SUB thread done in 69140ms
ADD thread done in 69236ms
Threads joined, result: 1 (SUCCESS)
=============== Finished Atomics Tests ===============

First thing is that without using atomics or locks, the test fails, which was
expected.

Then, using locks is about 20 times slower than atomics. Also, the custom
atomic_add() seems to perform a little better than Bionic calls, but are these
later public anyway?

I suppose it might worth trying with a pthread mutex too, to see how fast that is.

--
Olivier

[Olivier Guilyardi]

On 06/22/2010 01:59 PM, Olivier Guilyardi wrote:

Errata:

> It starts two threads one which increment an int N times, and another N-1 times.
> A result of 1 is /likely/ to indicate success.

I meant: ... and another which decrement it N-1 times.

> You can actually run it with:
> $ http://svn.samalyse.com/android/atomics
> $ cd atomics
> $ ./bootstrap.sh
> $ make

First line should be: svn co http://svn.samalyse.com/android/atomics

--
Olivier

[fadden]

On Jun 22, 4:59 am, Olivier Guilyardi <l...@samalyse.com> wrote:
> The trouble with LDREX/STREX is that this isn't supported on ARMv5.

Yup. The alternative is to branch to the magic kernel handler and let
it decide what to do (as in your atomic_add). You pay the cost of the
branch, and on ARMv5 it has to do a syscall, but on ARMv6 and later it
just does the appropriate LDREX/STREX for you. (My micro-benchmark
put it at about 5x the cost of just using LDREX/STREX directly.)

> Then, using locks is about 20 times slower than atomics. Also, the custom
> atomic_add() seems to perform a little better than Bionic calls, but are these
> later public anyway?

The bionic mutex and sem calls are simple atomic operations in the
absence of contention. When locks start colliding they'll sleep on
futexes which is going to slow you down some. 20x seems a bit
extreme. Are the numbers closer if you only execute one thread,
avoiding contention?

The bionic atomic call is going to take longer than a local atomic
call because it's in a different .so; I found that executing the same
atomic op across .so boundaries took 56ns, vs. 25ns for a local
function, vs. 8ns for an inline. (We're in the process of making
substantial changes to the atomic ops right now -- DO NOT use the
private bionic functions, it's highly likely they'll change.)

> I suppose it might worth trying with a pthread mutex too, to see how fast that is.

I expect it'll be similar to the sem results; possibly a bit worse
since there's slightly more mechanism involved in a mutex than a
semaphore.

[Olivier Guilyardi]

On 06/22/2010 09:53 PM, fadden wrote:
> On Jun 22, 4:59 am, Olivier Guilyardi <l...@samalyse.com> wrote:
>> The trouble with LDREX/STREX is that this isn't supported on ARMv5.
>
> Yup. The alternative is to branch to the magic kernel handler and let
> it decide what to do (as in your atomic_add). You pay the cost of the
> branch, and on ARMv5 it has to do a syscall, but on ARMv6 and later it
> just does the appropriate LDREX/STREX for you. (My micro-benchmark
> put it at about 5x the cost of just using LDREX/STREX directly.)

If I understand you correctly, the kernel will issue LDREX/STREX from "my"
atomic_add() if it's running on ARMv6, but using LDREX/STREX directly should
still be 5x faster. That's quite a lot.

I see that the NDK targets -march=armv5te, but are there really devices which
are limited to the ARMv5 instruction set? Or are they all/mostly ARMv6?

>> Then, using locks is about 20 times slower than atomics. Also, the custom
>> atomic_add() seems to perform a little better than Bionic calls, but are these
>> later public anyway?
>
> The bionic mutex and sem calls are simple atomic operations in the
> absence of contention. When locks start colliding they'll sleep on
> futexes which is going to slow you down some. 20x seems a bit
> extreme. Are the numbers closer if you only execute one thread,
> avoiding contention?

Starting only the incrementing thread, the runtime is 34s instead of 69s, which
is half runtime for locking 1M times instead of 2M. So it really seems to come
from the locks themselves, because with one thread they don't collide at all.

I think it would be interesting to see if such a thing happen on a standard
Linux system. I may try that..

> The bionic atomic call is going to take longer than a local atomic
> call because it's in a different .so; I found that executing the same
> atomic op across .so boundaries took 56ns, vs. 25ns for a local
> function, vs. 8ns for an inline. (We're in the process of making
> substantial changes to the atomic ops right now -- DO NOT use the
> private bionic functions, it's highly likely they'll change.)

Okay, private, got it. Can you recommend the custom atomic_add() in my tests
then? Is this safe?

Actually, I would quite have like to hear about someone running those tests on a
device with an ARMv7 cpu, since I don't own one. The tests are very easy to run.
It's not only a matter of speed. I'd like to know if those things are reliable.

>> I suppose it might worth trying with a pthread mutex too, to see how fast that is.
>
> I expect it'll be similar to the sem results; possibly a bit worse
> since there's slightly more mechanism involved in a mutex than a
> semaphore.

Are you sure? David recently told that mutexes should be faster in Froyo,
because they rely on private futexes instead of standard ones. But I don't have
a device running Froyo (and didn't check yet the latest ROMs out there ;).

--
Olivier

[Olivier Guilyardi]

On 06/22/2010 10:40 PM, Olivier Guilyardi wrote:

> Starting only the incrementing thread, the runtime is 34s instead of 69s, which
> is half runtime for locking 1M times instead of 2M. So it really seems to come
> from the locks themselves, because with one thread they don't collide at all.

I meant: ".. for locking 10M times instead of 20M.".

--
Olivier

[Angus Lees]

On Wed, Jun 23, 2010 at 05:53, fadden <fad...@android.com> wrote:
> We're in the process of making
> substantial changes to the atomic ops right now -- DO NOT use the
> private bionic functions, it's highly likely they'll change.

Aside: is there a stable futex() syscall that will survive this
reorganisation, or should I embed my own syscall stub? (will even the
kernel ABI be preserved?)

- Gus

[David Turner]

mutexes in Froyo will be faster than mutexes in Eclair.

However, semaphores are slightly faster than mutexes due to the complexity associated to their pthread interface.

In Froyo, mutexes are using private futexes by default, but semaphores are not, so they may appear faster than

semaphores on this specific release. However, this is not the case in Eclair and older, and will not be the case in

Gingerbread.

--
 Olivier

--
You received this message because you are subscribed to the Google Groups "android-ndk" group.

[David Turner]

Gus, you should *not* use futex calls directly. They need to be associated to memory barrier operations for proper SMP support.

I make no claims at binary stability for anyone using the private futex calls in Bionic.

[fadden]

On Jun 22, 1:40 pm, Olivier Guilyardi <l...@samalyse.com> wrote:
> I see that the NDK targets -march=armv5te, but are there really devices which
> are limited to the ARMv5 instruction set? Or are they all/mostly ARMv6?

You can't assume it's anything but ARMv5. You'd have to do runtime
detection to use non-v5 instructions in an armv5-labeled binary.

Results from a Nexus One:

D/AtomicsTest( 662): =============== Starting Atomics Tests
===============
D/AtomicsTest( 662): SUB thread starting for 9999999 iterations using
simple decrementation
D/AtomicsTest( 662): SUB thread done in 0ms
D/AtomicsTest( 662): ADD thread starting for 10000000 iterations
using simple incrementation
D/AtomicsTest( 662): ADD thread done in 0ms
D/AtomicsTest( 662): Threads joined, result: 1 (SUCCESS)
D/AtomicsTest( 662): SUB thread starting for 9999999 iterations using
custom atomic_add()
D/AtomicsTest( 662): ADD thread starting for 10000000 iterations
using custom atomic_add()
D/AtomicsTest( 662): ADD thread done in 987ms
D/AtomicsTest( 662): SUB thread done in 998ms
D/AtomicsTest( 662): Threads joined, result: 1 (SUCCESS)
D/AtomicsTest( 662): SUB thread starting for 9999999 iterations using
Bionic's __atomic_dec()
D/AtomicsTest( 662): ADD thread starting for 10000000 iterations
using Bionic's __atomic_int()
D/AtomicsTest( 662): ADD thread done in 821ms
D/AtomicsTest( 662): SUB thread done in 827ms
D/AtomicsTest( 662): Threads joined, result: 1 (SUCCESS)
D/AtomicsTest( 662): SUB thread starting for 9999999 iterations using
locks
D/AtomicsTest( 662): ADD thread starting for 10000000 iterations
using locks
D/AtomicsTest( 662): SUB thread done in 14927ms
D/AtomicsTest( 662): ADD thread done in 14935ms
D/AtomicsTest( 662): Threads joined, result: 1 (SUCCESS)
D/AtomicsTest( 662): =============== Finished Atomics Tests
===============

Clearly gcc is turning the trivial add-loop into a single store.
Introducing a new function:

static void nonatomic_add(uint32_t* ptr, int val)
__attribute__((noinline));
static void nonatomic_add(uint32_t* ptr, int val)
{
*ptr += val;
}

and using that instead results in:

D/AtomicsTest( 701): ADD thread done in 226ms
D/AtomicsTest( 701): SUB thread done in 236ms
D/AtomicsTest( 701): Threads joined, result: 439194 (FAILURE)

I added a new test for inline LDREX/STREX, and got:

D/AtomicsTest( 729): SUB thread starting for 9999999 iterations using
LDREX
D/AtomicsTest( 729): ADD thread starting for 10000000 iterations
using LDREX
D/AtomicsTest( 729): SUB thread done in 238ms
D/AtomicsTest( 729): ADD thread done in 237ms
D/AtomicsTest( 729): Threads joined, result: 1 (SUCCESS)

If I make it a function instead of coding it inline, it slows down a
bit:

D/AtomicsTest( 744): SUB thread done in 244ms
D/AtomicsTest( 744): ADD thread done in 251ms

I'm a little baffled by the performance of the sem_* functions.

[fadden]

On Jun 22, 4:34 pm, fadden <fad...@android.com> wrote:
> I'm a little baffled by the performance of the sem_* functions.

I added some counters to the semaphore implementation and printed
their value at the end of the run. sem_post() is calling
__futex_wake() every single time.

D/AtomicsTest( 806): SUB thread starting for 9999999 iterations using
mutexes
D/AtomicsTest( 806): ADD thread starting for 10000000 iterations
using mutexes
D/AtomicsTest( 806): SUB thread done in 2772ms
D/AtomicsTest( 806): ADD thread done in 2771ms
D/AtomicsTest( 806): Threads joined, result: 1 (SUCCESS)

So that's 3x the cost of your custom atomic definition, which is about
what you'd expect since it has to do two atomic read-modify-write ops
to manage the lock, plus the additional overhead of calling a function
in a foreign .so and dealing with some mutex overhead.

I'll file a bug about the semaphore performance.

I'll reiterate my recommendation to use mutexes over custom lock
code. It *will* break on SMP if you don't also get the memory
barriers in, and my experiments have shown that using barrier
instructions on non-SMP hardware is a very bad idea.

[Olivier Guilyardi]

On 06/23/2010 01:34 AM, fadden wrote:
> On Jun 22, 1:40 pm, Olivier Guilyardi <l...@samalyse.com> wrote:
>> I see that the NDK targets -march=armv5te, but are there really devices which
>> are limited to the ARMv5 instruction set? Or are they all/mostly ARMv6?
>
> You can't assume it's anything but ARMv5. You'd have to do runtime
> detection to use non-v5 instructions in an armv5-labeled binary.

Okay, let's forget it, ARMv5 is the target, and I don't want to have a different
implementation for that kind of thing if I ever target the new ARMv7 support.

> Results from a Nexus One:

Good cpu I see :)

>
> Clearly gcc is turning the trivial add-loop into a single store.
> Introducing a new function:

That's weird, it doesn't do that here with APP_OPTIM:=debug. I suppose you're
not using the Application.mk which is provided in the test suite.

> I added a new test for inline LDREX/STREX, and got:

Would you mind doing an svn diff and posting it here so that I patch the tests?

> I'm a little baffled by the performance of the sem_* functions.

Actually, I have noticed for a long time that they were slow, and I now try to
avoid locking in my app. Whenever it's not critical I don't lock anymore,
although it often looks non-academic and could cause subtle (harmless) issues.

--
Olivier

[Olivier Guilyardi]

On 06/23/2010 01:47 AM, fadden wrote:
> On Jun 22, 4:34 pm, fadden <fad...@android.com> wrote:
>> I'm a little baffled by the performance of the sem_* functions.
>
> I added some counters to the semaphore implementation and printed
> their value at the end of the run. sem_post() is calling
> __futex_wake() every single time.
>
> D/AtomicsTest( 806): SUB thread starting for 9999999 iterations using
> mutexes
> D/AtomicsTest( 806): ADD thread starting for 10000000 iterations
> using mutexes
> D/AtomicsTest( 806): SUB thread done in 2772ms
> D/AtomicsTest( 806): ADD thread done in 2771ms
> D/AtomicsTest( 806): Threads joined, result: 1 (SUCCESS)
>
> So that's 3x the cost of your custom atomic definition, which is about
> what you'd expect since it has to do two atomic read-modify-write ops
> to manage the lock, plus the additional overhead of calling a function
> in a foreign .so and dealing with some mutex overhead.

svn diff?

> I'll file a bug about the semaphore performance.

Cool, my tests sound useful then :)

> I'll reiterate my recommendation to use mutexes over custom lock
> code. It *will* break on SMP if you don't also get the memory
> barriers in, and my experiments have shown that using barrier
> instructions on non-SMP hardware is a very bad idea.

There's one thing that you didn't answer: is the custom atomic_add() ok? Can I
use that? It doesn't qualify as "custom lock code" to me. Or would it be better
to reuse the __atomic_inc() implementation from arch-arm/bionic/atomics_arm.S?
(which looks similar actually)

--
Olivier

[David Turner]

On Wed, Jun 23, 2010 at 1:06 PM, Olivier Guilyardi <li...@samalyse.com> wrote:

> I'll reiterate my recommendation to use mutexes over custom lock
> code.  It *will* break on SMP if you don't also get the memory
> barriers in, and my experiments have shown that using barrier
> instructions on non-SMP hardware is a very bad idea.

There's one thing that you didn't answer: is the custom atomic_add() ok? Can I
use that? It doesn't qualify as "custom lock code" to me. Or would it be better
to reuse the __atomic_inc() implementation from arch-arm/bionic/atomics_arm.S?
(which looks similar actually)

he did answer that by telling you to use the pthread primitives instead :-)

do not reuse assembly fragment from the C library, they will break on SMP

(and they are also conditionally compiled depending on the device's target CPU,

something you cannot afford in an application).

moreover, the __atomic_inc is private to the C library, do not use them. They could

be removed, and they must be used with memory barriers on SMP hardware.

 

--
 Olivier

--
You received this message because you are subscribed to the Google Groups "android-ndk" group.

[Olivier Guilyardi]

On 06/23/2010 05:54 PM, David Turner wrote:
>
> On Wed, Jun 23, 2010 at 1:06 PM, Olivier Guilyardi <li...@samalyse.com
> <mailto:li...@samalyse.com>> wrote:
>
>
> > I'll reiterate my recommendation to use mutexes over custom lock
> > code. It *will* break on SMP if you don't also get the memory
> > barriers in, and my experiments have shown that using barrier
> > instructions on non-SMP hardware is a very bad idea.
>
> There's one thing that you didn't answer: is the custom atomic_add()
> ok? Can I
> use that? It doesn't qualify as "custom lock code" to me. Or would
> it be better
> to reuse the __atomic_inc() implementation from
> arch-arm/bionic/atomics_arm.S?
> (which looks similar actually)
>
>
> he did answer that by telling you to use the pthread primitives instead :-)

Okay, I thought Fadden was maybe answering Angus about futexes, since an atomic
operation isn't "lock code" to me.

But I have now tried with pthread mutexes (svn updated) and they do provide 6x
speedup over semaphores on Android 1.6. That's *very* interesting to know.
And this speedup doesn't come from Froyo's private futexes apparently (I thought
that was the case in Fadden's tests).

> do not reuse assembly fragment from the C library, they will break on SMP
> (and they are also conditionally compiled depending on the device's
> target CPU,
> something you cannot afford in an application).
> moreover, the __atomic_inc is private to the C library, do not use them.
> They could
> be removed, and they must be used with memory barriers on SMP hardware.

Okay, understood, anyway I found a workaround, and atomic operations are not
crucial anymore right now.

I think that one conclusion of this thread simply is that atomic operations are
not available to user space code in Anrdoid currently.

But please optimize the semaphores ;-)

Thanks

--
Olivier

[Tim in Boulder]

On Jun 22, 1:53 pm, fadden <fad...@android.com> wrote:
> The bionic atomic call is going to take longer than a local atomic
> call because it's in a different .so; I found that executing the same
> atomic op across .so boundaries took 56ns, vs. 25ns for a local
> function, vs. 8ns for an inline.  (We're in the process of making
> substantial changes to the atomic ops right now -- DO NOT use the
> private bionic functions, it's highly likely they'll change.)

I just came across this message, and this is very, very distressing.

As I asked in this thread:

https://groups.google.com/group/android-ndk/browse_thread/thread/3760ff092d5228c7/d689161e98dc26d3?hl=en&#d689161e98dc26d3

...is there a way to know what APIs I can and can't use. STABLE-
APIS.TXT says not to use linux/ and asm/ files, but the atomics are in
sys/, and there are a number of includes in sys/ that are explicitly
OK (e.g., pthreads).

And as I also said in that thread, I'm already using the atomic ops in
sys/atomics.h.

I can change to use something else in this case, but it's really,
really problematic that there's no way to know what functions are safe
to use. I don't want to be constantly rewriting things because I've
come to rely on certain functionality only to find that a future
Android release invalidates it.

I know writing documentation is no fun, but it's really, really
important that someone at Google at least put together a whitelist of
stable APIS. At present the STABLE-APIS.TXT file is currently wrong if
you can't use sys/ include files safely.

Tim

[Tim in Boulder]

On Jul 9, 2:46 pm, Tim in Boulder <tim.men...@gmail.com> wrote:
> ...is there a way to know what APIs I can and can't use. STABLE-
> APIS.TXT says not to use linux/ and asm/ files, but the atomics are in
> sys/, and there are a number of includes in sys/ that are explicitly
> OK (e.g., pthreads).

Sorry, misremembered where pthreads.h lives. It's the BSD socket
library I was thinking of, which has been officially blessed on this
list (I believe) where <sys/select.h> and <sys/socket.h> are the
standard includes.

My point stands; how can we know what is and isn't safe to use?

Tim

[David Turner]

Everything that is under sys/ should be ok, it's the content of linux/ and asm/ which is not.

Tim

[fadden]

On Jul 9, 2:33 pm, David Turner <di...@android.com> wrote:
> Everything that is under sys/ should be ok, it's the content of linux/ and
> asm/ which is not.

So that means these are available to use in the NDK (bionic/libc/
include/sys/atomics.h)?

extern int __atomic_cmpxchg(int old, int _new, volatile int *ptr);
extern int __atomic_swap(int _new, volatile int *ptr);
extern int __atomic_dec(volatile int *ptr);
extern int __atomic_inc(volatile int *ptr);

Note these are currently defined to be barrier-free, and hence
unsuitable for most applications on SMP hardware.