This is code I've posted before.  The fetch and op functions are simulated
for the purposes of illustration and you use an interlocked implementation
to make them atomic.

#define EXCL 1
#define SHRD 0x10000
#define fa(n, i)        ((*(n)+=(i))-(i))
#define fi(n)           ((*(n))++)

        volatile int    n;              /* next available ticket */
        volatile int    c;              /* current request */

        volatile int    x;              /* exclusive */
        volatile int    s;              /* shared */

int main() {
        int     w;                      /* wait ticket */

        /*-------------------------------------------------------------------*/
        /* shared/exclusive access using fetch-and-add                       */
        /*-------------------------------------------------------------------*/
        n=0;c=0;                        /* initialize */

        w=fa(&n, EXCL);                     /* request exclusive access */
        while(w!=c);                    /* wait for it */
        /* critical region */
        fa(&c, EXCL);                       /* release exclusive access */

        w=fa(&n, SHRD);                     /* request shared access */
        while((w%SHRD)!=(c%SHRD));      /* wait for it */
        /* critical region */
        fa(&c, SHRD);                       /* release shared access */

        /*-------------------------------------------------------------------*/
        /* shared/exclusive access using fetch-and-increment                 */
        /*-------------------------------------------------------------------*/
        n=0;x=0;s=0;                    /* initialize */

        w=fi(&n);                   /* request (shared) access */
        while(w!=s);                    /* wait for it */
        fi(&s);                             /* release shared */
        /* critical region */
        fi(&x);                             /* release exclusive */

        w=fi(&n);                   /* request (exclusive) access */
        while(w!=x);                    /* wait for it */
        /* critical region */
        fi(&s);                             /* release shared */
        fi(&x);                             /* release exclusive */

        return 0;

They're bakery algorithms with FIFO service order.  In the first exclusive requests wait for
*all* prior accesses to complete.  Shared requests just wait for prior exclusive accesses
to complete.  Number of waiting threads has to be less than the field sizes.  The algorithm
works even with wrap and carry out from the exclusive access field into the shared access field.

The second version is if you just have fetch and increment.  The shared requests have to
unblock other waiting shared requests as they enter the critical section so it's a little
less efficient.

You can optimize this somewhat by using a power of 2 to define the field size and logically
anding to extract a field value, using a store w/ membar to release exclusive access on the
first one.

Bakery spinlocks may be more efficient than normal spinlocks since they acquire their wait
ticket value first and wait rather than waiting and then all waiting threads attempting to
acquire the lock all in the same instant.  The interlocked updates from ticket requests
are more spread out and cause less contention on the cache.

Joe Seigh