Questions about P0876: fibers

[Nicol Bolas]

I have a few questions about the library fibers proposal, P0876.

* Fibers are bound to the thread that initially created them; they cannot be resumed in any other thread. Is this restriction necessary? What are the performance costs for allowing them to be used in any thread? Is that perhaps an aspect that can be decided on at `fiber` creation time?

* Is it possible to add a way to transfer ownership of a suspended fiber to another thread? Obvious that depends on exactly why fibers are bound to their creating thread.

* Is it possible to have a way to ask if a fiber is associated with a given thread?

* Would it be possible to create a thread with a fiber? I'm not saying that you should be able to create a thread and pass it an existing `std::fiber`. I'm asking if it is reasonable for `std::thread` to have some of the fiber constructors, so that we could pass stack allocators that can control the size of stack they get.

* Would it be reasonable to have an API that can take a terminated fiber and give it a new entry-function? The purpose of this would be to avoid having to deallocate a stack and then immediately reallocate it. You would be able to keep a number of fibers lying around and pull one out when you want to execute a new task. I know that `resume_with` can work like this to some extent, but invoking an unwind operation on the stack will unwind past where `resume_with` started. This way, there is nothing "past" the entry function; a refurbished fiber is functionally identical to a fiber that was previously used.

* It's not clear to me exactly how `resume_with` works, with regard to the underlying parts of the stack. That is, is there a way to use `resume_with` to execute the given function, then immediately go back and resume the execution of whatever was below it on the fiber's stack? Or is `resume_with` completely incapable of interacting with the previous stuff on the stack?

[Oliver Kowalke]

2018-03-10 21:55 GMT+01:00 Nicol Bolas <jmck...@gmail.com>:

I have a few questions about the library fibers proposal, P0876.

* Fibers are bound to the thread that initially created them; they cannot be resumed in any other thread. Is this restriction necessary?

This restriction was requested by SG1 at the last C++ meeting (because of the TLS problem, which is not fiber specific).

 

What are the performance costs for allowing them to be used in any thread?

none, fibers are unaware of the thread
 

Is that perhaps an aspect that can be decided on at `fiber` creation time?

yes, infact fibers are stacks. you can create a fiber in thread A and start/resume it at thread B.

 

* Is it possible to add a way to transfer ownership of a suspended fiber to another thread? Obvious that depends on exactly why fibers are bound to their creating thread.

fibers are not bound to a thread ... suspended fibers can be migrated between threads (but see TLS problem above)

 

* Is it possible to have a way to ask if a fiber is associated with a given thread?

fibers are not 'associated' with a thread but run in a context of a thread (scheduled inside the thread).
fibers are suspend- and resumable stacks. a thread starts with a stack too (the stack that was assigned by the OS) and can be suspended and resumed.

in other words, a thread is a container running one (e.g. the stack created by the OS) or multiple fibers

 

* Would it be possible to create a thread with a fiber?

a thread already runs at least one fiber (== one stack)

 

I'm not saying that you should be able to create a thread and pass it an existing `std::fiber`. I'm asking if it is reasonable for `std::thread` to have some of the fiber constructors, so that we could pass stack allocators that can control the size of stack they get.

p0320r0 suggestes std::thread::attributes controling the stack size and some other stuff
 

* Would it be reasonable to have an API that can take a terminated fiber and give it a new entry-function? The purpose of this would be to avoid having to deallocate a stack and then immediately reallocate it.

allocating and reallocating is done by the stack allocator ... if you want to resuse stacks ismply create a stack allocator that caches stacks

 

You would be able to keep a number of fibers lying around and pull one out when you want to execute a new task.

what about taking a std::function from a queue

 

* It's not clear to me exactly how `resume_with` works, with regard to the underlying parts of the stack. That is, is there a way to use `resume_with` to execute the given function, then immediately go back and resume the execution of whatever was below it on the fiber's stack?

yes, the proposal contains an example describing how resume_with() works

[Nicol Bolas]

On Saturday, March 10, 2018 at 4:37:54 PM UTC-5, Oliver Kowalke wrote:

2018-03-10 21:55 GMT+01:00 Nicol Bolas <jmck...@gmail.com>:

I have a few questions about the library fibers proposal, P0876.

* Fibers are bound to the thread that initially created them; they cannot be resumed in any other thread. Is this restriction necessary?

This restriction was requested by SG1 at the last C++ meeting (because of the TLS problem, which is not fiber specific).

That answer renders all of my subsequent thread-limited-fiber questions moot, since they were all written under the assumption that this was a technical limitation.

But that brings up the question: why impose this limitation?

To my knowledge, there is nothing in the Coroutines TS that prevents a coroutine function from being executed on a different thread at different stages of its execution. Any thread-local storage issues would be just as apparent for the Coroutines TS as it would be for fibers.

Indeed, this limitation makes it effectively impossible to do something as simple as pass a fiber (that is, a lambda which resumes a `std::fiber`) to a networking callback, or any `future.then` or anything of the sort. Oh, the callback function could create a fiber in the callback, but the callback itself could not resume an existing fiber. Any kind of suspend-up style system would be incompatible with fibers that cannot be executed from arbitrary threads.

So why are fibers limited in this way when coroutine functions are not?

* Would it be possible to create a thread with a fiber?

a thread already runs at least one fiber (== one stack)

 

I'm not saying that you should be able to create a thread and pass it an existing `std::fiber`. I'm asking if it is reasonable for `std::thread` to have some of the fiber constructors, so that we could pass stack allocators that can control the size of stack they get.

p0320r0 suggestes std::thread::attributes controling the stack size and some other stuff

Well, my hope would be that we wouldn't have two distinct APIs for this kind of thing. If threads have fibers, then it seems reasonable to be able to construct a thread using the fiber system's techniques. If I have a stack allocator, it'd be nice to be able to use it with `std::thread` just as well as `std::fiber`.

* Would it be reasonable to have an API that can take a terminated fiber and give it a new entry-function? The purpose of this would be to avoid having to deallocate a stack and then immediately reallocate it.

allocating and reallocating is done by the stack allocator ... if you want to resuse stacks ismply create a stack allocator that caches stacks

Outside of allocating stacks, is the cost of constructing a new fiber pretty marginal?

[Nicol Bolas]

On Saturday, March 10, 2018 at 4:37:54 PM UTC-5, Oliver Kowalke wrote:

One question about this. I see that the return value from an injected function is passed back to the previously executing call to `resume/resume_with` on the underlying call stack. But... what happens if there is no underlying call stack yet. That is, you have a fresh `std::fiber` which has never been `resume`d, and you call `resume_with` on it.

My guess is that the return value will be passed as the parameter to the fiber's entry function. But I don't think your current proposal says that. Or at least, the definition of `resume_with` doesn't seem to acknowledge that possibility.

Oh, and one other thing. The proposal says this:

When calling `resume()`, it is conventional to replace the newly-invalidated instance – the instance on which `resume()` was called – with the new instance returned by that `resume()` call. This helps to avoid inadvertent calls to `resume()` on the old, invalidated instance.

That sounds like a good convention. So maybe there should be a couple of global helper functions to help people adhere to it:

void resume(std::fiber &&f) {f = std::move(f).resume();}

template<typename Fn>
void resume_with(std::fiber &&f, Fn &&fn) {f = std::move(f).resume_with(std::forward<Fn>(fn));}

These could be member functions (with the names `resume_inplace` and `resume_with_inplace`) instead of free functions.

[Lee Howes]

> To my knowledge, there is nothing in the Coroutines TS that prevents a coroutine function from being executed on a different thread at different stages of its execution. 

One reason to argue for the difference would be that the coroutines TS is just a low-level primitive. It tells you nothing beyond a set of function calls and a state machine transformation. We would very much want to enforce this requirement for most coroutine code in practice, but in the library infrastructure, and we have the opportunity to enforce this requirement in the library we provide in the standard. You can pass an arbitrary coroutine that completed on an arbitrary thread to another coroutine, and the type system can catch that fact and ensure that awaiting on it does not cause the current coroutine to transition between threads.

Coroutines as defined do not work without the library infrastructure that you have to write. The risk with this proposal is that you really can just resume it from anywhere and suspend that caller. The fiber alone does magical things to your current stack.

There is no safety around the basic primitive. As a result, the feeling in the room was that it is better to start from the conservative position and then relax it later once we fully understand any issues that may come up. It's still an open question whether we want to specify library infrastructure on top of this primitive, or only to specify fiber.

[Nicol Bolas]

On Saturday, March 10, 2018 at 9:37:10 PM UTC-5, Lee Howes wrote:

> To my knowledge, there is nothing in the Coroutines TS that prevents a coroutine function from being executed on a different thread at different stages of its execution. 

One reason to argue for the difference would be that the coroutines TS is just a low-level primitive. It tells you nothing beyond a set of function calls and a state machine transformation. We would very much want to enforce this requirement for most coroutine code in practice,

I may be wrong here, but as I recall, the Networking TS makes no promises that callbacks will be executed on the same thread that registered the callback. The current `experimental::future::then` makes no promises about which thread the continuation gets executed on. And so forth. Which means that if "we" wanted to enforce this on Coroutines, it would be impossible to use them on these interfaces.

I do not see how suspend-up continuations are useful in an environment where you can only be continued in your current thread. If that's what people wanted, they'd just use a message passing system to let them know when the data was available so that they could finish processing it.

but in the library infrastructure, and we have the opportunity to enforce this requirement in the library we provide in the standard.

Declaring something to be UB is not really what I would call "enforcement". Especially since, because there's no technical restriction in play, pretty much every implementation will allow it to work as expected.

The situation this will create is that people won't know about the restriction. They'll expect to be able to use fibers wherever they want, in any thread they want. And yet there's a piece of paper out there saying that their code is wrong. Which makes this just one more place in the standard where "what the standard says" and "what real implementations actually do" diverge.

That's not something C++ really needs more of.

I don't mean to make light of the thread-local storage issues. I understand the issue that code (hidden from external view) which would normally not be able to become disassociated from a thread now can become disassociated. But if I can't do anything as simple as applying a fiber to a future's `.then` method, then the fear of TLS problems is making the overall feature a lot less useful.

You can pass an arbitrary coroutine that completed on an arbitrary thread to another coroutine, and the type system can catch that fact and ensure that awaiting on it does not cause the current coroutine to transition between threads.

Coroutines as defined do not work without the library infrastructure that you have to write.

Can you point to a proposed or even suggested library infrastructure that enforces restrictions of this sort? P0055 outlines a way to make Coroutines work with Networking, as does P0162. It is my understanding that none of them outline the restriction that a continuation given to their systems will certainly and unquestionably be executed within the thread that provided that continuation.

The risk with this proposal is that you really can just resume it from anywhere and suspend that caller. The fiber alone does magical things to your current stack.

There is no safety around the basic primitive. As a result, the feeling in the room was that it is better to start from the conservative position and then relax it later once we fully understand any issues that may come up.

 

I'm of the opinion that a restriction on a feature, especially an unenforced one like this, should not be imposed unless there is certain knowledge of a genuine problem that cannot be resolved in another way. Because doing this massively limits the usefulness of this feature.

A more realistic position would be to say that the thread-fiber, the fiber created for each thread, cannot be resumed on anything other than its originating thread. Subsidiary fibers are far less likely to have thread-specific code in them. And its far less likely that the user will accidentally resume a thread's fiber in another thread. It can certainly still happen of course; since each halted fiber is uniquely owned by some other fiber, it is possible for a subsidiary fiber to own the thread fiber and carry that ownership with them to some other thread.

[Oliver Kowalke]

2018-03-11 2:55 GMT+01:00 Nicol Bolas <jmckesson@gmail.com>:

On Saturday, March 10, 2018 at 4:37:54 PM UTC-5, Oliver Kowalke wrote:

2018-03-10 21:55 GMT+01:00 Nicol Bolas <jmck...@gmail.com>:

* It's not clear to me exactly how `resume_with` works, with regard to the underlying parts of the stack. That is, is there a way to use `resume_with` to execute the given function, then immediately go back and resume the execution of whatever was below it on the fiber's stack?

yes, the proposal contains an example describing how resume_with() works

One question about this. I see that the return value from an injected function is passed back to the previously executing call to `resume/resume_with` on the underlying call stack. But... what happens if there is no underlying call stack yet. That is, you have a fresh `std::fiber` which has never been `resume`d, and you call `resume_with` on it.

It acts like starting the context the first time, executing the function passed to resume_with() and then enters the function that was given the ctor of fiber (should already be addressed in P0876).

 

My guess is that the return value will be passed as the parameter to the fiber's entry function.

correct

 

But I don't think your current proposal says that. Or at least, the definition of `resume_with` doesn't seem to acknowledge that possibility.

API section: notes for the constructor

 

Oh, and one other thing. The proposal says this:

When calling `resume()`, it is conventional to replace the newly-invalidated instance – the instance on which `resume()` was called – with the new instance returned by that `resume()` call. This helps to avoid inadvertent calls to `resume()` on the old, invalidated instance.

That sounds like a good convention. So maybe there should be a couple of global helper functions to help people adhere to it:

void resume(std::fiber &&f) {f = std::move(f).resume();}

template<typename Fn>
void resume_with(std::fiber &&f, Fn &&fn) {f = std::move(f).resume_with(std::forward<Fn>(fn));}

These could be member functions (with the names `resume_inplace` and `resume_with_inplace`) instead of free functions.

yes, sounds reasonable

[Nicol Bolas]

On Sunday, March 11, 2018 at 10:41:09 AM UTC-4, Oliver Kowalke wrote:

2018-03-11 2:55 GMT+01:00 Nicol Bolas <jmckesson@gmail.com>:

On Saturday, March 10, 2018 at 4:37:54 PM UTC-5, Oliver Kowalke wrote:

2018-03-10 21:55 GMT+01:00 Nicol Bolas <jmck...@gmail.com>:

* It's not clear to me exactly how `resume_with` works, with regard to the underlying parts of the stack. That is, is there a way to use `resume_with` to execute the given function, then immediately go back and resume the execution of whatever was below it on the fiber's stack?

yes, the proposal contains an example describing how resume_with() works

One question about this. I see that the return value from an injected function is passed back to the previously executing call to `resume/resume_with` on the underlying call stack. But... what happens if there is no underlying call stack yet. That is, you have a fresh `std::fiber` which has never been `resume`d, and you call `resume_with` on it.

It acts like starting the context the first time, executing the function passed to resume_with() and then enters the function that was given the ctor of fiber (should already be addressed in P0876).

 

My guess is that the return value will be passed as the parameter to the fiber's entry function.

correct

 

But I don't think your current proposal says that. Or at least, the definition of `resume_with` doesn't seem to acknowledge that possibility.

API section: notes for the constructor

Unless you've updated P0876, I don't see anything about this in the constructor notes. Those notes are:

The entry-function fn is not immediately entered. The stack and any other necessary resources are created on construction, but fn is not entered until resume() or resume_with() is called.

The entry-function fn passed to std::fiber will be passed a synthesized std::fiber instance representing the suspended caller of resume().

The function fn passed to resume_with() will be passed a synthesized std::fiber instance representing the suspended caller of resume_with().

This says nothing about where the return value from the `fn` passed to `resume_with` will go. And the second bullet point explicitly states that the entry-function will only get the fiber representing the caller of `resume`; what fiber it gets from `resume_with` is left unstated.

The second bullet point should probably read:

The entry-function fn passed to std::fiber will be passed a synthesized std::fiber instance representing the suspended caller of resume() or the return value of an injected fn from a call to resume_with().

Then, there's the definition in the notes section of `resume_with`, which says nothing of this sort:

An injected function fn() must accept std::fiber&& and return std::fiber. The fiber instance returned by fn() is, in turn, used as the return value for the suspended function: resume() or resume_with().

That paragraph needs to be changed too. I'd say it should look like the following:

An injected function fn() must accept std::fiber&& and return std::fiber. The fiber instance returned by fn() is, in turn, used as the return value for the previously executed suspended function in the fiber: resume() or resume_with(). If there was no previously executed suspend function, then the return value will be passed as the parameter of the entry-function.

[Victor Dyachenko]

On Sunday, March 11, 2018 at 5:41:09 PM UTC+3, Oliver Kowalke wrote:

Oh, and one other thing. The proposal says this:

When calling `resume()`, it is conventional to replace the newly-invalidated instance – the instance on which `resume()` was called – with the new instance returned by that `resume()` call. This helps to avoid inadvertent calls to `resume()` on the old, invalidated instance.

That sounds like a good convention. So maybe there should be a couple of global helper functions to help people adhere to it:

void resume(std::fiber &&f) {f = std::move(f).resume();}

template<typename Fn>
void resume_with(std::fiber &&f, Fn &&fn) {f = std::move(f).resume_with(std::forward<Fn>(fn));}

These could be member functions (with the names `resume_inplace` and `resume_with_inplace`) instead of free functions.

yes, sounds reasonable

Why && ? Shouldn't it be old good lvalue reference in this case?

void resume(std::fiber &f) {f = std::move(f).resume();}



[Oliver Kowalke]

resume() invalidates the instance of f ... resume() with the signature `fiber resume() &&` (== can only be invoked at rvalue references) -> see P0876

[Victor Dyachenko]

But this function doesn't invalidate f! It is an input/output param here

[Oliver Kowalke]

2018-03-12 10:40 GMT+01:00 Victor Dyachenko <victor.d...@gmail.com>:

But this function doesn't invalidate f! It is an input/output param here

resume() invalidates instance `f`, restricting to rvalue reference makes this more explicit to the user

[Nicol Bolas]

On the one hand, I understand what you're getting at. But on the other hand, the function also assigns to `f`, which means that `f` will likely remain valid after the call.

Normally, if you do `some_function(std::move(val))`, you expect that `val` is no longer valid. We're taught as C++ programmers to assume this when reading that code and to either not use `val` anymore or to reinitialize `val` before using it again.

`resume(std::move(f))` will leave `f` in a meaningful state. Even if `f` is not in a valid state, the fact that it's not valid is meaningful to the user. It means something that the resumed fiber didn't return a valid fiber to suspend to. The fiber likely terminated successfully or something.

`std::move(val)` is supposed to be read as "I'm done with this now". But in this case, you're not done with `f` now. So if the intent of the interface is to assign to the given reference, such that it retains meaning relative to the operation, then the interface should not require `std::move`.

Also, using a `&&` parameter means that you can do this:

resume(std::fiber(...));

Which is probably not what you wanted. That code will have the same effect as

std::fiber(...).resume();

But that's precisely why the first code shouldn't compile. The non-member function is for doing an in-place operation. And if you're doing an in-place operation, you really want it to work on lvalues only, since the point of the in-place operation is to resume a variable and assign back to it.

As such, it should take an lvalue reference.

[Nicol Bolas]

Ah, in addition to this, I would suggest that the members `resume` and `resume_with` are prime candidates for the [[nodiscard]] attribute.

[Gor Nishanov]

* Fibers are bound to the thread that initially created them; they cannot be resumed in any other thread. Is this restriction necessary?

TLS limitation is due to how TLS codegen is done on RISC CPUs today:

https://godbolt.org/g/ZyrTTc

TLS access is expensive, registers are cheap, so, on the first TLS access, compiler caches the address of the TLS page in some callee saved register used throughout the function. 

Compiler has no idea that a call to a different function might switch stacks / TLS underneath.

If a fiber is suspended and resumed on a different thread, register used as the beginning of the TLS page may be pointing at the  memory that is gone or have been reused for something else.

Note that it is not the user written code. It is how compiler makes TLS access efficient within a function. Allowing fiber resumption on different thread, would require pessimizing TLS access for every function by reloading TLS after every function call.

If we cannot safely access TLS, it means that it is likely that:

* magic statics do not work

* error_code category does not work

* tc_malloc (thread caching allocator) does not work

* networking TS / asio does not work (it uses a thread caching recycling allocator by default)

* more stuff is broken

Stackless coroutines do not have this problem, since the points where suspension and resumption can happen are marked explicitly and compiler does not caches TLS across suspend points.

[Oliver Kowalke]

2018-03-21 17:01 GMT+01:00 Gor Nishanov <gorni...@gmail.com>:

<snip>

 

Stackless coroutines do not have this problem, since the points where suspension and resumption can happen are marked explicitly and compiler does not caches TLS across suspend points.

the TLS problem is not P0876 specific ... it applies to any C++ code that uses TLS (thread_local vars) and moves functors between threads

migrating fibers between threads is possible if not TLS is used

[Gor Nishanov]

Hmm... My reply got deleted. Let me repost it again:

<snip>

On Wednesday, March 21, 2018 at 10:11:50 AM UTC-7, Oliver Kowalke wrote:

the TLS problem is not P0876 specific ... it applies to any C++ code that uses TLS (thread_local vars) and moves functors between threads

migrating fibers between threads is possible if not TLS is used

Functor running in different threads is perfectly legal and does not produce hidden undefined behavior.

You can use all of the language, all of the libraries, you will always correctly get the TLS of the thread which is executing the code.

There is no UB. Since the operator() of the function object always completely executes in a thread and you will always get a TLS of a thread which is executing 

Of course, a user can write a UB, by say, taking an address of a thread local variable and stashing it somewhere and then referencing it later. I am not talking about user writing a buggy code, 

I am talking about the instructions that compiler generates for TLS access. Compilers assume that entire function executes completely in one thread, therefore they are free to cache TLS page load in non-scratch registers. std::fiber breaks that, since now, in a function that looks normal to the compiler, at any function call there could be a fiber switch and subsequent TLS access in that function will become UB.

This is important point, I'll stay engaged in this thread until this is resolved. You being a primary author on fibers paper should understand that SG1 did not demand changes because of their ignorance. To my knowledge, nobody knows at the moment how to make fibers work across threads without introducing TLS related UB.

Note that some standard library and language features under the covers use TLS. Even if you do not use TLS in your code and none of the 3rd party libraries use TLS, you would still have TLS access from the facilities I mentioned in my earlier post.

Cheers,

Gor

 

[Nicol Bolas]

On Wednesday, March 21, 2018 at 1:11:50 PM UTC-4, Oliver Kowalke wrote:

2018-03-21 17:01 GMT+01:00 Gor Nishanov <gorni...@gmail.com>:

<snip>

 

Stackless coroutines do not have this problem, since the points where suspension and resumption can happen are marked explicitly and compiler does not caches TLS across suspend points.

the TLS problem is not P0876 specific ... it applies to any C++ code that uses TLS (thread_local vars) and moves functors between threads

Gor doesn't seem to be talking about general logic issues behind TLS. He's talking about a specific implementation of TLS that conflicts with arbitrary suspend/resume of unprepared functions.

In the case Gor's talking about, there is an actual register/stack value that functions use to access TLS values. So a fiber suspend/resume will preserve the TLS values. Which means if you resume it on another thread, it will be using the values from the previous thread. But this will only be the case for such implementations.

Simply passing a functor to a different thread won't cause that problem. It only happens if you start the function on one thread and complete it on another.

"Stackless coroutines" don't have that problem because the compiler knows that this specific function can be suspended/resumed on different threads, and at those points it can take steps to adjust the appropriate registers/stack values as needed.

And as Gor points out, there's a lot of hidden code out there that implicitly uses TLS. So simply saying "fibers can be transferred as long as you don't use TLS" doesn't really solve the problem.

[Oliver Kowalke]

the point is: don't use TLS if you want to migrate fibers to other threads

[Nicol Bolas]

On Wednesday, March 21, 2018 at 2:16:15 PM UTC-4, Oliver Kowalke wrote:

the point is: don't use TLS if you want to migrate fibers to other threads

How do you define "use TLS"? As pointed out, a lot of things "use TLS". Some implementations may use TLS for something that other implementations do not.

The standard cannot define that some things don't "use TLS". So how will you know what you can and cannot do with transferable fibers?

[Gor Nishanov]

> the point is: don't use TLS if you want to migrate fibers to other threads

Which means, do not use <filesystem>, do not define static locals within a function, do not use <system_error>, do not use errno, do not use init_once, do not use networking TS, do not use boost::asio and these are just the ones on top of my head of the things which are popular and using thread_local in their implementation.

It is difficult to provide an exhaustive list of all facilities that you should not use if you want to migrate fibers to other threads.

Though, as you probably saw, there is a provision of adding special "expert only" members to std::fiber API that can resume the fiber in a different thread. The burden is of course, is those who use those would have to have super detail code reviews that nothing in what they use can ever touch TLS today or when maintaining this code and upgrading to later revisions of libraries or compiler features.

[Dilip Ranganathan]

On Wed, Mar 21, 2018 at 12:01 PM, Gor Nishanov <gorni...@gmail.com> wrote:

* Fibers are bound to the thread that initially created them; they cannot be resumed in any other thread. Is this restriction necessary?

TLS limitation is due to how TLS codegen is done on RISC CPUs today:

https://godbolt.org/g/ZyrTTc

I am just an interested observer trying to understand this thread (sic). May I 

ask what the fiber suspension/resumption activity has to do with TLS? I am 

struggling to make this connection. Are we saying that code that uses TLS 

and ends up being executed by a Fiber cannot possibly work correctly if it 

resumes in another thread?

Raymond Chen over here[1] seems to imply that Fibers can quite happily get 

themselves suspended by the thread that created them and simply resume on 

another thread.

[1] https://blogs.msdn.microsoft.com/oldnewthing/20100225-00/?p=14813/

[Nicol Bolas]

On Wednesday, March 21, 2018 at 2:42:37 PM UTC-4, Dilip R wrote:

On Wed, Mar 21, 2018 at 12:01 PM, Gor Nishanov <gorni...@gmail.com> wrote:

* Fibers are bound to the thread that initially created them; they cannot be resumed in any other thread. Is this restriction necessary?

TLS limitation is due to how TLS codegen is done on RISC CPUs today:

https://godbolt.org/g/ZyrTTc

I am just an interested observer trying to understand this thread (sic). May I 

ask what the fiber suspension/resumption activity has to do with TLS? I am 

struggling to make this connection. Are we saying that code that uses TLS 

and ends up being executed by a Fiber cannot possibly work correctly if it 

resumes in another thread?

Yes, that's what he's saying. In the implementation he's referring to, getting TLS data requires going though a pointer in a register or stack value. As such, those registers/stack will be preserved across threads. So TLS access will access the wrong thread's local storage.
 

Raymond Chen over here[1] seems to imply that Fibers can quite happily get 

themselves suspended by the thread that created them and simply resume on 

another thread.

[1] https://blogs.msdn.microsoft.com/oldnewthing/20100225-00/?p=14813/

He's talking about a very specific implementation of fibers on a very specific CPU and compiler setup with a very specific way of handling TLS. Gor is talking about different implementations of TLS which are incompatible with fibers being executed on different threads.

[Bengt Gustafsson]

Den onsdag 21 mars 2018 kl. 19:24:09 UTC+1 skrev Gor Nishanov:

> the point is: don't use TLS if you want to migrate fibers to other threads

Which means, do not use <filesystem>, do not define static locals within a function, do not use <system_error>, do not use errno, do not use init_once, do not use networking TS, do not use boost::asio and these are just the ones on top of my head of the things which are popular and using thread_local in their implementation.

I don't think it is this bad: As long as a function inside for instance std::filesystem does not yield it should be ok to call it from a fiber even if said fiber jumps between threads. If on the other hand you have a callback from a function using thread_local and that function yields you are in trouble, if I understand the register usage correctly.

With both thread_local and fiber being part of the language that the compiler vendor is to implement there should be possibilities to improve the situation further. For instance, if a specific register could be reserved for the TLS page pointer use then the only thing needed would be for the fiber switch function to not preserve this register. Thus the TLS page pointer would always be consistent with the running thread.

It is still unclear to me if there is just one TLS "page" per thread or if each of the thread_local variables have their own "page" which may require reserving multiple registers. Maybe you can explain this system in more detail?

In some cases this would not solve the thread jumping of fibers anyway, for instance in the case of errno: Some asynchronous IO function may set errno then call the user provided callback which supposedly checks for errors using errno. But if this callback yields before checking errno and is subsequently resumed in another thread the errno will be wrong. (I'm not at all advocating for this errno usage, its just an example). Even without multithreading I see the potential error source with errno and similar as fibers are typically used for asynchronous IO making it very easy to make mistakes so that multiple fibers can execute code setting errno overwriting each other's values... Do we need fiber_local storage?  

Bengt

[Giovanni Piero Deretta]

On Wednesday, March 21, 2018 at 10:41:09 PM UTC, Bengt Gustafsson wrote:

Den onsdag 21 mars 2018 kl. 19:24:09 UTC+1 skrev Gor Nishanov:

> the point is: don't use TLS if you want to migrate fibers to other threads

Which means, do not use <filesystem>, do not define static locals within a function, do not use <system_error>, do not use errno, do not use init_once, do not use networking TS, do not use boost::asio and these are just the ones on top of my head of the things which are popular and using thread_local in their implementation.

I don't think it is this bad: As long as a function inside for instance std::filesystem does not yield it should be ok to call it from a fiber even if said fiber jumps between threads.

That's not the case unfortunately. Because of inlining and interprocedural optimizations. If you call the same function (or two functions that access the same TLS entry) before and after a yield point, the compiler might still unsafely optimize the TLS access.

-- gpd

[Bengt Gustafsson]

On the other hand we are talking about a new language feature here. Even if a fiber is disguised as a class there has to be some codegen magic to make yielding happen. Yes, boost manages to do it with some assembly function but I assume a language feature would be handled by the compiler in such a way that it can detect a yield point and abstain from caching a TLS entry pointer in a register over it.

If the language feature is specified in this way it will gain an edge over boost's library only version that may increase the motivation to actually standardize it. This said it is not obvious that current uses of TLS will continue to work if you slap yield points into the code at random but it should at least be possible to ensure that the TLS entry corresponds to the current thread at all times.

I could be wrong but I think that there are interesting use cases for fibers that can be served by a thread pool. As it may be impossible to know if a library uses TLS it seems scary to not allow the limited usage I proposed (i.e. still no yielding in callbacks).

[Nicol Bolas]

On Monday, March 26, 2018 at 6:32:06 PM UTC-4, Bengt Gustafsson wrote:

On the other hand we are talking about a new language feature here. Even if a fiber is disguised as a class there has to be some codegen magic to make yielding happen.

Just because there's "codegen magic" doesn't make it a "language feature". This is a pure library extension; any changes to the language parts of the standard would only be to clarify the behavior of fibers with respect to mutexes, data races, and other async coding.
 

Yes, boost manages to do it with some assembly function but I assume a language feature would be handled by the compiler in such a way that it can detect a yield point and abstain from caching a TLS entry pointer in a register over it.

I don't know all of the details of these sorts of implementations, so I may be off-base here. But the way I see it, that could only work if the compiler can see all of the code between the function that is initially called in the fiber and every fiber suspension point. That is, if everything is inline (ala the resumable functions proposal).

Because otherwise, the compiler cannot know that a particular function call will eventually invoke a yield operation. And without that knowledge, it won't know that it should not cache the TLS pointer or to refresh that cache at some later date. And if it gets cached, that could be cached on the stack somewhere, where it can become incorrect relative to resumption on a different thread.

This is why the coroutines TS doesn't have this problem; within each coroutine, every possible suspend point is explicitly spelled out in the text of that function.

If the language feature is specified in this way it will gain an edge over boost's library only version that may increase the motivation to actually standardize it. This said it is not obvious that current uses of TLS will continue to work if you slap yield points into the code at random but it should at least be possible to ensure that the TLS entry corresponds to the current thread at all times.

I could be wrong but I think that there are interesting use cases for fibers that can be served by a thread pool. As it may be impossible to know if a library uses TLS it seems scary to not allow the limited usage I proposed (i.e. still no yielding in callbacks).

I think what would be good is if there is an implementation-defined query that you can set up telling you whether it's OK to share fibers between threads. If we accept that some implementations just can't allow that, it would be nice to at least be able to write code for those that do allow it, with a `static_assert` preventing it from working on other implementations.