NOTE: posted to general Felix list.
Just so you can see, here is some of the problematic new code:
sync_sched::fstate_t sync_sched::frun()
{
dispatch:
if (ft == 0) {
spinguard dummy(active->lockneeded,&(active->active_lock));
ft = active->pop_front();
}
if (ft == 0) {
return blocked;
}
request = ft->run(); // run fthread to get request
if(request == 0) // euthenasia request
{
spinguard dummy(active->lockneeded,&(active->active_lock));
ft = 0;
goto dispatch;
}
You can see, there is a conditionally executed custom spinlock used to access
the scheduler’s active list. The lock is needed if multiple pthreads are
accessing the same list. However the lock isn’t used if only one thread
is operating on the list.
I’m not sure why I did that, maybe for performance, and maybe just so the operation
is the same when not using the concurrent coroutine feature.
You can start concurrency by svc_spawn_process. If you do that in Felix code
by spawn_process procedure, the svc is intercepted by sync_sched, which doesn’t
know what to do with it. So it returns a “delegated” state. The caller, usually
async_sched, then handles it, and calls do_spawn_process().
================
If you look in async_sched, there’s lots of locking and crap happening.
The problem is not just coding, but design issues.
What do you do if there are async events pending?
With one pthread, you just hang on a semaphore, trying to dequeue
the fthread from the wait list. Actually it should spin checking for user
cancellation or something.
With multiple threads, all should be trying for the async event queue.
But wait! What happens when we’re out of sync to process, but some
thread still run? We don’t want to actually cancel the idle threads because
new work can be spawned by a running fibre. And any fibre can make
async requests. So until ALL the fibres have stopped AND there is no
event pending, we should keep all the pthreads active.
getting that logic right is tricky! Its even harder, when it is time to return
control to the user, to terminate all the concurrent threads except the
one that actually called the run function, that one has to return.
It has to be that one!
=============
Note that you can have a STACK of schedulers!!!!!
Any Felix procedure can call “run” which is a subroutine NOT a coroutine.
That’s the secret that took me 20 years to figure out!!!!
Now here’s the FUN part. WHen you call run from inside a *concurrent* process
running on a scheduler .. what happens to the OTHER threads????
You just spawned a new scheduler which itself can run concurrent threads,
but now, there are still threads running in the parent.
Now it’s not clear what happens with async I/O. There’s only one async I/O
queue .. one demux. I think. I’m confused!
Also, *synchronous* coms can now occur because threads in the parent
are still running. Now it seems, a fibre on the child can MIGRATE to the
parent! After all, if there is a shared channel, the child does a write,
then a parent thread can do a read on that same channel.
in this case my spinlock, which is PER SCHEDULER is going to fail
to protect the accesses because there are two schedulers involved.
Note any running fibre, in any thread, on any scheduler, can do
schannel I/O on the same channel.
Its worse! What happens if you start another pthread directly
with spawn_pthread? You’re not supposed to use schannel
I/O between pthreads because originally it wasn’t synchronised.
But wait! What if we use a global spinlock??
That is, put it on the GC object.
now any pthreads can safely do I/O and in fact we can get rid
of pchannels altogether!
The PROBLEM is that spinlocks should never protect complex
operations. You should use a standard mutex lock for that.
By “complex” operation I mean anything that can do a memory
allocation or wait for async I/O.
In particular, anything that can cause a context switch in the OS
***INCLUDING PREEMPTIONS*** should not use a spinlock.
In principle spinlocks should mask interrupts to prevent OS pre-emptions.
Otherwise the lock holder might be pre-empted, and the other threads
waiting on the lock will be logically stalled, but physically eating
CPU up spinning.
====
So there is still some heavy design work to do. The core problem
is that we really want to mask interrupts when a spinlock is held,
but the OS rarely provides a way to do that. Windows used to have
BEGIN/END_CRITICAL_SECTION which is what we want,
but what normally happens is you use an OS mutex, and that tells
the OS to try to avoid pre-emptions. But it doesn’t force interrupts
off, because the OS should pre-empt the lock holder if the lock
holder is taking too long. Or ends up calling malloc() or doing something
else. In this case, all the threads waiting on the lock should be
pre-empted as well, since none can run. This means the OS
will run threads that will allow the lock holder to continue instead
of hanging.
So we have a catch-22 situation. We should be using OS locks,
but they’re too slow. But spinlocks can never work correctly,
because the OS can pre-empt the lock holder, no matter what.
Spinlocks work based on statistics: if the actions of the lock
holder are fast .. such as just doing a couple of memory accesses,
the chances of being pre-empted are low. EXCEPT if the memory
is paged out to disk. Heck, if it isn’t in the L1 cache, we have problems.
And if it is is, but another process is using the same memory,
which is ALWAYS the case because that’s why we’re protecting it,
then the hardware has to do cache-coherence messaging
to synchronise the caches.
Apple apparently and Linux for sure I think, makes mutex locks
spin, with exponentially increasing delays, until it gives up
and calls the OS lock. This is actually the correct method.
Just use “accelerated” system mutex.
But tests on my Mac show, this doesn’t work in that, it
kills performance.
By far the best solution is to use atomic I/O, but that only
works with a single read or write or special instuction.
—
John Skaller
ska...@internode.on.net