NW 6.5 SP6 SMP & lock API

[adva...@extendedsystems.com]

Hello everyone,
I'm having some troubles with the CLib lock() API on NetWare servers
with multiple CPUs. I've attached the source to a test NLM which
demonstrates the behavior (cannot seem to post a zip attachment). My
test NetWare server is v6.5 SP6 with the latest CLib & LibC:

Novell Open Enterprise Server, NetWare 6.5
Support Pack Revision 06
Server Version 5.70.06 October 26, 2006
Novell eDirectory Version 8.7.3.9 SMP
NDS Version 10553.73 October 9, 2006

LIBC.NLM
Loaded from [C:\NWSERVER\] on Jan 22, 2007 2:46:02 pm
(Address Space = OS)
Standard C Runtime Library for NLMs [optimized, 7]
Version 9.00.05 January 3, 2007

CLIB.NLM
Loaded from [C:\NWSERVER\] on Jan 22, 2007 2:46:23 pm
(Address Space = OS)
(Legacy) Standard C Runtime Library for NLMs
Version 5.90.14 December 14, 2006

NSS.NLM
Loaded from [C:\NWSERVER\] on Jan 22, 2007 2:46:15 pm
(Address Space = OS)
NSS (Novell Storage Services) (Build 1046 MP)
Version 3.25 October 6, 2006

The SYS volume where I've been running the test (where the test file &
NLM are located) is NSS. To run the test with MP:

load sys:locktest.nlm sys:test.dat 1

To run it without MP:
load sys:locktest.nlm sys:test.dat 0

Note: the test requires a file on the server to write to; the file's
contents aren't important, but are overritten.

The test creates multiple threads and each thread simulates a record
append operation. The append uses two locks: header and record. The
header lock is basically a method of protecting the global record
count. The record lock simply protects the new record while it is
updated, although in this test there should be no record lock
contention as each thread appends the new record at a different file
offset.

To enable each thread to use multiple CPUs I call NWSMPThreadToMP()
early in the thread proc (I've verified via Remote Manager that the
test uses both CPUs when this API is called). However, the lock() API
then fails later in the test. If I don't call NWSMPThreadToMP(), the
test runs file (and only one CPU is utilized).

Should the CLib lock() APIs be working with multiple CPUs in this way?
We've had no such problems in the past with MP NetWare servers...

Regards,

Peter Funk
iAnywhere Solutions Inc.

#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <IO.h>
#include <fcntl.h>
#include <nwthread.h>
#include <nwmpk.h>
#include <nwsmp.h>

#define NUM_THREADS 2
#define APPEND_COUNT 100000
#define HEADER_LOCK_OFFSET 0x7FFFFFFEUL
#define RECORD_LOCK_OFFSET 0x7FFFFFFEUL

LONG glDone = 0; // When each worker thread has
incremented glDone once, the main thread will exit
unsigned int guiRecordCount = 0; // Global record count to ensure each
thread gets different record lock offsets
char gbSuspend = FALSE; // If an error occurs, this flag tells
each thread to quit
char gbUseSMP = FALSE; // If TRUE, call NWSMPThreadToMP on
worker threads to use multiple CPUs

// This thread procedure simulates a record append operation:
// 1. Obtain the header lock
// 2. Seek to the new record position
// 3. Lock the new record
// 4. Update the record count
// 5. Unlock the header
// 6. Write the new record
// 7. Unlock the new record
void ThreadProc( char *pcArg )
{
int iFile, iRet;
unsigned int uiRecCount = 0, uiNewRecord;

if ( gbUseSMP )
NWSMPThreadToMP(); // Calling this API causes problems with the
lock() API...

// Open any ol' file for writing to
iFile = sopen( pcArg, O_RDWR, 0x00000040 /* SH_DENYNO */ );
if ( iFile == -1 )
{
printf( "sopen %s failed, err: %d\n", pcArg, errno );
gbSuspend = TRUE;
return;
}

// Append some new records to the file
while (( uiRecCount < APPEND_COUNT ) && ( gbSuspend == FALSE ))
{

// Lock the header byte (spin until we get the lock)
do
{
iRet = lock( iFile, HEADER_LOCK_OFFSET, 1 );
ThreadSwitchWithDelay();
}
while ( iRet );

// Append a new record
if ( lseek( iFile, guiRecordCount, SEEK_SET ) != guiRecordCount )
{
gbSuspend = TRUE;
printf( "Error seeking to %u, err: %d\n", guiRecordCount,
errno );
break;
}

// Lock the new record
if ( lock( iFile, RECORD_LOCK_OFFSET - (guiRecordCount + 1), 1 )
!= 0 )
{
gbSuspend = TRUE;
printf( "Error locking new record at %u, err: %d\n",
RECORD_LOCK_OFFSET - (guiRecordCount + 1), errno );
unlock( iFile, HEADER_LOCK_OFFSET, 1 );
break;
}

// Update the total record count
guiRecordCount++;
uiNewRecord = guiRecordCount;

// Unlock the header byte
iRet = unlock( iFile, HEADER_LOCK_OFFSET, 1 );
if ( iRet )
{
gbSuspend = TRUE;
unlock( iFile, RECORD_LOCK_OFFSET - uiNewRecord, 1 );
printf( "Error unlocking header byte, err: %d\n", errno );
break;
}

// Write the new record
if ( write( iFile, "X", 1 ) != 1 )
{
gbSuspend = TRUE;
printf( "Error writing new record, err: %d\n", errno );
unlock( iFile, RECORD_LOCK_OFFSET - uiNewRecord, 1 );
break;
}

// Unlock the new record
if ( unlock( iFile, RECORD_LOCK_OFFSET - uiNewRecord, 1 ) != 0 )
{
gbSuspend = TRUE;
printf( "Error unlocking new record at %u, err: %d\n",
RECORD_LOCK_OFFSET - uiNewRecord, errno );
break;
}

uiRecCount++;

} // while ( uiRecCount < APPEND_COUNT )

close( iFile );
atomic_inc( &glDone );
}

int main( int argc, char *argv[] )
{
int aiThreadIDs[ NUM_THREADS ], i;

if ( argc < 2 )
{
printf( "usage: lock.nlm <filename> [useSMP flag]\n" );
return 0;
}
else if ( argc == 3 )
{
// if a second parameter is non zero, enable SMP
if ( argv[2][0] != '0' )
gbUseSMP = TRUE;
}

if ( NWSMPIsAvailable())
{
if ( gbUseSMP )
printf( "SMP kernel is available and in use.\n" );
else
printf( "SMP kernel is available but not used.\n" );
}
else
{
printf( "SMP kernel is NOT available.\n" );
gbUseSMP = FALSE;
}

printf( "Starting %d threads...\n", NUM_THREADS );
for ( i = 0; i < NUM_THREADS; i++ )
{
aiThreadIDs[i] = BeginThread( ThreadProc, NULL, 8192, argv[1] );
if ( aiThreadIDs[i] == EFAILURE )
{
printf( "Error creating thread %d, err: %d\n", i, errno );
break;
}
}

// Wait for all threads to finish
while (( glDone < NUM_THREADS ) && ( gbSuspend == FALSE ))
ThreadSwitchWithDelay();

if ( gbSuspend )
printf( "All threads suspended.\n" );
else
printf( "All threads done.\n" );

return 0;
}

[Peter Funk]

Hello everyone,
I'm having some troubles with the CLib lock() API on NetWare servers with
multiple CPUs. I've attached the source to a test NLM which demonstrates

the behavior. My test NetWare server is v6.5 SP6 with the latest CLib &

[Jeff Lawson]

Peter,

Can you tell me what you mean by "We've had no such problems in the past
with MP NetWare servers..."? Does this mean that you have run this test on
MP system using older support packs without problem, or does it mean that
you haven't run into any other MP issues?

I am not certain whether these interface were intended to be used from
multiple processors or not. So if you found them to work before and now they
are broken then I will ask the NSS to see if they changed something that
could have broken this. If this is new (using lock from multiple processors)
then I will simply ask them if they support such use.

Thanks
Jeff Lawson
NetWare Core OS
LibC Engineering

>>> On 1/23/2007 at 8:46 AM, in message
<864d0bcba5df8...@developer-forums.novell.com>, Peter

[Peter Funk]

Jeff,
We've been using the Clib lock() API for many years without problems, certainly
on servers with multiple processors (I know we have customers running our
product on NetWare with multiple CPUs and they haven't reported seeing this
issue; but that doesn't necessarily mean the issue is new). I was unable
to get an older version of NetWare running on our VMWare server with dual
CPUs, so I couldn't verify how far back the issue goes. To answer your question,
I did run the test against NW 6.5 SP5 which reproduced the issue, but I haven't
been able to run it against any older versions.

Regards,
Peter

[Jeff Lawson]

Peter,

Can you tell me what errno and NetWareErrno are when lock() fails? Also can
you send me your binaries?

Thanks,
Jeff

>>> On 1/23/2007 at 12:04 PM, in message
<864d0bcba5f08...@developer-forums.novell.com>, Peter

[Jeff Lawson]

Peter,

The error you are getting indicates that there is another thread currently
in the middle of the locking operation on the same connection (connection 0
in this case). This interface was made way before MP was any kind of
concern. When MP came along the simplest thing to do to allow MP use was to
simply return an error. No actual MP synchronization was done, but the error
return made it so that data integrity was maintained. In that way the
interface is MP safe, but you must code around this error if you want to use
it that way. This was largely not a problem as most access to the locking
mechanism is done from NCP clients and the problem only happens when there
are multiple simultaneous accesses on the same connection (NCP protocol
won't allow that, so only API user have the problem).

So here are your choices:
- Stay UP
- watch for this error errno == 12 NetWareErrno == 1 and then simply retry,
similar to how you handle the not locked case in the loop above).

Jeff

>>> On 1/23/2007 at 3:55 PM, in message
<anwth.6382$Sz4....@prv-forum2.provo.novell.com>, Peter
Funk<noe...@nospam.com> wrote:
> Jeff,
> When lock() returns -1, errno is set to 12 and NetWareErrno is 1. I've
> attached a zip file containing the binaries and source of the test NLM.
>
> Peter

[Robert Charles Mahar]

I have a couple SMP test boxes at home, I'll see what happens too. I
think I can boot these as OES SP2, NW 6.0 SP5, NW 5.1 SP8. So I'll
report back if there are any differences.

Perhaps I missed this in reading the posts, but WHICH lock() fails? The
one for the record header or the one for the new record? Or both at
different times? Does it fail after running a number of iterations or
immediately?

errno == 12, could be mistaken, but I think I've see that when there are
too many simultaneous locks or when the server runs out of memory.
Perhaps the locks are not getting released or aged out of the tables
quick enough.

-- Bob

- - - - - - - - - - - - - - - - -
Robert Charles Mahar
Traffic Shaping Engine for NetWare
http://www.TrafficShaper.com
- - - - - - - - - - - - - - - - -

[Jeff Lawson]

It is the second lock, although the first may fail similarly the loop in the
code will handle it without it ever being known.

>>> On 1/25/2007 at 10:46 AM, in message
<%16uh.7692$Sz4....@prv-forum2.provo.novell.com>, Robert Charles

[Peter Funk]

Thanks Jeff. I understand what you've said, but I have a few clarification
questions:

So any thread of my NLM, on connection 0, in the lock API can cause another
thread calling lock to fail? Doesn't matter what file handle or offset is
passed in?

Please verify that a regular lock failure (ie the byte(s) are already locked)
will return different & consistent errno and NetWareErrno values.. The only
documented error for lock() I've found is 4 (EBADF).

Does this apply to all NW versions & service packs? ie this isn't anything
new?

Do the NXFileRangeLock/NXFileRangeUnLock LibC APIs behave the same way?

How expensive are the NWSMPThreadToNetWare() and NWSMPThreadToMP() APIs if
I were to call them around the lock() API in order to marshall the threads
to CPU 0? I'm showing a 7% performance degredation in an extreme test, though
a real world scenario wouldn't be that bad. Do you have better suggestion
to keep my NLM MP but perform locking on CPU 0?

Regards,
Peter

[Jeff Lawson]

>>> On 1/25/2007 at 1:26 PM, in message
<864d0bcba6708...@developer-forums.novell.com>, Peter

Funk<pf...@nospam.com> wrote:
> Thanks Jeff. I understand what you've said, but I have a few
> clarification
> questions:
>
> So any thread of my NLM, on connection 0, in the lock API can cause
> another
> thread calling lock to fail? Doesn't matter what file handle or offset
> is
> passed in?

yes

>
> Please verify that a regular lock failure (ie the byte(s) are already
> locked)
> will return different & consistent errno and NetWareErrno values.. The
> only
> documented error for lock() I've found is 4 (EBADF).

A lock failure will return a -1 with errno not set and NetWareErrno set to
253

>
> Does this apply to all NW versions & service packs? ie this isn't
> anything
> new?

nothing new here, this is really old code.

>
> Do the NXFileRangeLock/NXFileRangeUnLock LibC APIs behave the same way?
>

No. For the legacy file system the code is very similar, but for NSS it is
completely different.

> How expensive are the NWSMPThreadToNetWare() and NWSMPThreadToMP() APIs
> if
> I were to call them around the lock() API in order to marshall the
> threads
> to CPU 0? I'm showing a 7% performance degredation in an extreme test,
> though
> a real world scenario wouldn't be that bad. Do you have better
> suggestion
> to keep my NLM MP but perform locking on CPU 0?

All methods of funnelling your threads to P0 will have about that same
performance. You could serialize your calls to lock() with a mutex so that
you don't have to funnel. You would just have to try it to see if that
yields better performance. Internally they are being serialized already with
collisions resulting in the error that you are getting so this may be a good
solution.

>
> Regards,
> Peter

Jeff

[Peter Funk]

Great. Thanks for all your help Jeff!

Regards,
Peter