qthreads behavior alongside pthreads

[John Leidel]

All, we have an interesting situation when executing qthreads-enabled apps/libraries against our architectural simulator infrastructure.  We have a library that contains basic qthreads initialization and threading capabilities [qthread_initialize() + qthread_fork() + etc].  We also have an architecture simulation infrastructure that mimics our coprocessor architecture and a given target instruction set.  The Convey hardware runtime interprets an environment variable to determine whether to execute code on the arch sim or "real" hardware.  The arch sim is spawned in a separate pthread for execution.

The simple pseudo code is as follows: 

    

---------------------

init_routine()

--> qthread_initialize()

--> qthread_fork( ... )

--> qthread_fork( ... )

init_routine() complete

enter coprocessor region

--> spawn pthread for sim

--> execute coproc code

exit coprocessor region 

return && exit

---------------------

When I test this, the arch sim will execute but the qthreads will not.  However, if I insert a simple `usleep(1)` after the qthread_fork calls, the simulator will not execute and the qthreads will begin their execution.  

EG: 

---------------------

init_routine()

--> qthread_initialize()

--> qthread_fork( ... )

--> qthread_fork( ... )

--> usleep(1)

init_routine() complete

enter coprocessor region

--> spawn pthread for sim

--> execute coproc code

exit coprocessor region 

return && exit

---------------------

I was curious if anyone has tested/tried mixing a qthreads environment with a standard pthreads environment?  I will attempt to synthesize this in a test.  

cheers

john 

--
John D. Leidel
Software Architect
Convey Computer
jle...@conveycomputer.com
(g) 972.836.7901
(m) 214.578.8510

[Wheeler, Kyle Bruce]

Hi John,

I know what's going on. You're actually falling afoul of a performance optimization we added in 1.7.1 (this is the first time I've seen it bite someone). The feature that's involved here is what we call "spawn-cache" (as in --disable-spawn-cache). Fundamentally, forks ("spawns") are accumulated in a thread-specific buffer and are only made visible to the scheduler when a scheduling operation occurs (e.g. a sleep or a yield or a blocking operation). This provides a rather significant speedup in some circumstances by reducing the contention for the scheduler's queue (we can talk in greater detail about this, if you're interested), but in your case, it's preventing those threads from *ever* being visible to the scheduler. The reason that usleep(1) changes things is, most likely, because it's being intercepted by qthreads and turned into a scheduler event (and thus flushing the spawn cache). Another way of doing it is to call qthread_yield(), which is a scheduling operation. Alternatively, you can simply disable the spawn cache in qthreads. At some point (relatively soon, I think), we'll add a function to manually flush the spawn cache. I'm open to other ideas too, if you have any.

--
Kyle B. Wheeler
Dept. 1423: Scalable System Software
Sandia National Laboratories
505-844-0394

[John Leidel]

Kyle, thanks for the reply.  I rebuilt the library and disabled the spawn cache.  I use default settings everywhere else.  

./configure --prefix=`foo` --disable-spawn-cache

This doesn't appear to change the behavior.  I will attempt to craft a test that exhibits this idiosyncrasy.