Synchronous / codegen region

[Mehdi Amini]

Hi,

It was mentioned from time to time the need to gain "synchronous region" of execution (I think the motivation was performance / reducing the executor overhead). So far I have always seen the codegen from a compiler as emitting mostly "async islands" where each of these islands execute synchronously.

Eugene's recent prototype about how to expose a "parallel for" primitive to the runtime exposed an interesting question: how can the code from a "synchronous" region be able to interact with the thread pool? Has someone thought about this?

The TFRT execution model is mostly focused on task parallelism, and that constraint seems engrained in the underlying queuing APIs that control the thread pool.

As an example, in a single sequential island, we could have something that looks like (pseudo-code):

```

strict_kernel(input1, input2) {

  // do some alloc for temp, etc.

  ...

  // compute some sequential loop

  for(...) { 

    ...

  }

  // compute a data-parallel loop

  parallel_for(...) { 

     ...

  }

  // do some other things.

}

```

The way I see the parallel_for above is more of an opportunistic one, the thread executing this "strict kernel" can process it all itself, but if there are idle threads that can join the gang execution of the loop.

One way I could formulate the desired behavior is one of work-stealing.

The parallel loop iteration space is cut into chunks, and these chunks can be queued in their own execution queue. The calling thread exclusively pulls work from this queue, but other threads from the pool are allowed to "steal" chunks from the queue if they don't have anything else running in the system. 

Another way to phrase it is "work sharing": the thread executing the parallel_for is opportunistically exposing some chunks for sharing with other workers, but it won't do anything else itself than processing these chunks.

-- 

Mehdi