Seastar + traditional threads

[Andrew Melo]

Hi all,

I'm searching through the documentation and reading the code and had a question I can't seem to answer myself.

Is there a 'sensible' way to combine an application using seastar with external (possibly blocking) libraries? My thought was to make a separate pthread to run long-running tasks *, with some sort of future/promise on the seastar side to tell the pthread to run something and return the value later. If it helps, my particular use-case involves a one-off configuration/computation step at the beginning of a TCP connection, but nothing else afterwards.

Thanks!

Andrew

[Avi Kivity]

It's pretty hard.  One reason is that seastar uses polling and those threads would have to compete with that.  Another is that seastar substitutes its own allocator for the system allocator (e.g. we provide our own malloc()), and this allocator is not prepared for threads -- all system memory is divided among the shards, leaving nothing for your library.

There are still other options:

1. seastar threads: user-level threading implemented on top of seastar tasks, see http://docs.seastar-project.org/master/group__thread-module.html.

2. pthread API on top of seastar threads: a compatibility wrapper that exposes the pthread API on top of seastar threads, similar to GNU Pth emulation of pthreads.  The main drawback is that it is not implemented.

--
You received this message because you are subscribed to the Google Groups "seastar-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to seastar-dev...@googlegroups.com.
To post to this group, send email to seast...@googlegroups.com.
Visit this group at http://groups.google.com/group/seastar-dev.
To view this discussion on the web visit https://groups.google.com/d/msgid/seastar-dev/ea3b0914-6e00-44fc-a9d5-6dcfeddfc5d5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gleb Natapov]

On Thu, Nov 05, 2015 at 09:19:59PM +0200, Avi Kivity wrote:
> It's pretty hard. One reason is that seastar uses polling and those threads
> would have to compete with that. Another is that seastar substitutes its
> own allocator for the system allocator (e.g. we provide our own malloc()),
> and this allocator is not prepared for threads -- all system memory is
> divided among the shards, leaving nothing for your library.
>
> There are still other options:
>
> 1. seastar threads: user-level threading implemented on top of seastar
> tasks, see http://docs.seastar-project.org/master/group__thread-module.html.
>
> 2. pthread API on top of seastar threads: a compatibility wrapper that
> exposes the pthread API on top of seastar threads, similar to GNU Pth
> emulation of pthreads. The main drawback is that it is not implemented.
>

Those threads will not be able to call blocking function though, so you
will have to implement entire posix api to preempt voluntarily.

--
Gleb.

[Avi Kivity]

Yes, forward read/write calls to seastar APIs. Have to do that anyway
to make use of the network stack.

[Andrew Melo]

Hi Avi,

Thanks for the responses.

On Thu, Nov 5, 2015 at 1:19 PM, Avi Kivity <a...@scylladb.com> wrote:
>
> It's pretty hard. One reason is that seastar uses polling and those threads would have to compete with that.

I'd imagine I'd want to configure seastar to have less threads than
are physically available, leaving the remainder for these other
functions.

> Another is that seastar substitutes its own allocator for the system allocator (e.g. we provide our own malloc()), and this allocator is not prepared for threads -- all system memory is divided among the shards, leaving nothing for your library.

Since the library will be in the same process space, it should use the
same allocator, right? I'd need to be careful
to keep the library's affinity to one core, though...

>
> There are still other options:
>
> 1. seastar threads: user-level threading implemented on top of seastar tasks, see http://docs.seastar-project.org/master/group__thread-module.html.

If I understand the docs right, seastar threads are incompatible with
blocking system calls, which rules this out.

>
> 2. pthread API on top of seastar threads: a compatibility wrapper that exposes the pthread API on top of seastar threads, similar to GNU Pth emulation of pthreads. The main drawback is that it is not implemented.

Same problem as above.

It seems like seastar might not be a good fit. I can't reimplement the
external library from scratch, and using the existing implementation
won't work. No problem though! Glad to know now rather than later :)

Thanks again,
Andrew

>
>
> On 11/05/2015 09:00 PM, Andrew Melo wrote:
>
> Hi all,
>
> I'm searching through the documentation and reading the code and had a question I can't seem to answer myself.
>
> Is there a 'sensible' way to combine an application using seastar with external (possibly blocking) libraries? My thought was to make a separate pthread to run long-running tasks *, with some sort of future/promise on the seastar side to tell the pthread to run something and return the value later. If it helps, my particular use-case involves a one-off configuration/computation step at the beginning of a TCP connection, but nothing else afterwards.
>
> Thanks!
> Andrew
> --
> You received this message because you are subscribed to the Google Groups "seastar-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to seastar-dev...@googlegroups.com.
> To post to this group, send email to seast...@googlegroups.com.
> Visit this group at http://groups.google.com/group/seastar-dev.
> To view this discussion on the web visit https://groups.google.com/d/msgid/seastar-dev/ea3b0914-6e00-44fc-a9d5-6dcfeddfc5d5%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>

--
--
Andrew Melo

[Nadav Har'El]

Beyond everything which Avi and others already wrote, I'd like two propose an additional idea, which depending on what you need might be good enough:

Something you could fairly easily  do is to run your library as a completely separate process (not a thread), which gets commands over, say, a TCP/IP socket, and does its stuff. Then, your Seastar code will communicate with this external process using the normal Seastar future-based TCP network API.

If you do this, when you run the Seastar program and external program side by side, remember to tell Seastar to leave (at least) on CPU unused, and enough memory free, for your additional program to run well (if it uses very little CPU and memory, maybe the defaults will work too).

[Andrew Melo]

Hi Nadav,

On Sun, Nov 8, 2015 at 4:12 AM, Nadav Har'El <n...@scylladb.com> wrote:
> Beyond everything which Avi and others already wrote, I'd like two propose
> an additional idea, which depending on what you need might be good enough:
>
> Something you could fairly easily do is to run your library as a completely
> separate process (not a thread), which gets commands over, say, a TCP/IP
> socket, and does its stuff. Then, your Seastar code will communicate with
> this external process using the normal Seastar future-based TCP network API.
>
> If you do this, when you run the Seastar program and external program side
> by side, remember to tell Seastar to leave (at least) on CPU unused, and
> enough memory free, for your additional program to run well (if it uses very
> little CPU and memory, maybe the defaults will work too).

This would sidestep things nicely ... would such a thing work over a
named pipe as well? Or perhaps with an mmaped() file?

I guess somewhat related -- if I was wanting to encrypt the tcp stream
over SSL, what would be a sensible way to do that? I'd imagine OpenSSL
probably doesn't have the "come back quickly" guarantee seastar is
looking for.

Thanks again for your suggestions,
Andrew

--
--
Andrew Melo

[Avi Kivity]

I'm no openssl expert, but wouldn't you supply an SSL BIO object that
directs openssl to use the seastar APIs instead of the system APIs?

I imagine it can be configured to be non-blocking -- execute as much as
it can, then inform the system why it can't (i.e. whether to wait for
data in or data out).

[Nadav Har'El]

On Sun, Nov 8, 2015 at 7:40 PM, Andrew Melo <andre...@gmail.com> wrote:

Hi Nadav,

On Sun, Nov 8, 2015 at 4:12 AM, Nadav Har'El <n...@scylladb.com> wrote:
> Beyond everything which Avi and others already wrote, I'd like two propose
> an additional idea, which depending on what you need might be good enough:
>
> Something you could fairly easily  do is to run your library as a completely
> separate process (not a thread), which gets commands over, say, a TCP/IP
> socket, and does its stuff. Then, your Seastar code will communicate with
> this external process using the normal Seastar future-based TCP network API.
>
> If you do this, when you run the Seastar program and external program side
> by side, remember to tell Seastar to leave (at least) on CPU unused, and
> enough memory free, for your additional program to run well (if it uses very
> little CPU and memory, maybe the defaults will work too).

This would sidestep things nicely ... would such a thing work over a
named pipe as well?

I don't Seastar supports named pipes (or unix-domain sockets) yet, but adding support for it should be doable with no special difficulties - analogous to the way we support the "Posix" network stack (which calls Linux network system calls, and uses epoll() to poll them).

 

Or perhaps with an mmaped() file?

Yes, you can do this if you use lock-free algorithms (otherwise, you'll also need to also support some sort of synchronization mechanism, e.g., semaphores/mutexes/etc.).

[asomers@gmail.com]

I think that Andrew's problem is pretty common.  There are far too many legacy libraries out there to convert.  I have another idea for how to integrate legacy code, though I have no idea how hard it would be as I'm just getting started with Seastar:

I propose that in addition to Seastar's usual n^2 - n intercore queues, a program could create an extra n queues for communication between Seastar event processor threads and traditional threads.  The traditional part of the program would have one pair of queues to communicate with each event processor thread.  These queues' endpoints could even be shared amongst multiple traditional threads, with appropriate locking.  In this way the high-performance, polling event processor threads could have non-blocking communication with traditional threads (the traditional threads might block, but the event processor threads would not).  The programmer would be free to use any blocking API in the traditional threads, including creating and joining more threads.

The memory allocator would still be a problem, though.  Could Seastar reserve some memory for the traditional threads?

-Alan

[Avi Kivity]

This could work.  One problem would be balancing, how many cores would you allocate to traditional code and how many to to seastar?  Same with memory.

If those traditional libraries are not performance critical, then you can get away with just allocating a few cores for them.

The memory allocator would still be a problem, though.  Could Seastar reserve some memory for the traditional threads?

Sure (see --memory).  Since we intercept the memory allocation calls (malloc/new/etc.) we'd need a little extra glue to call into a traditional allocator when in a traditional context.

[asomers@gmail.com]

Yeah.  This technique would be most useful when the traditional libraries aren't performance critical.  So reserving 1 or 2 cores should be fine.  And after sleep mode gets merged, there won't be any need to reserve cores for traditional threads.
 

The memory allocator would still be a problem, though.  Could Seastar reserve some memory for the traditional threads?

Sure (see --memory).  Since we intercept the memory allocation calls (malloc/new/etc.) we'd need a little extra glue to call into a traditional allocator when in a traditional context.

That's what I thought.  Alas, it's too hard for me to attempt until I get some more practice hacking on Seastar.

-Alan

[Avi Kivity]

On 12/09/2015 05:42 PM, aso...@gmail.com wrote:

On Wednesday, December 9, 2015 at 1:21:14 AM UTC-7, Avi Kivity wrote:

On 12/09/2015 07:00 AM, aso...@gmail.com wrote:

I think that Andrew's problem is pretty common.  There are far too many legacy libraries out there to convert.  I have another idea for how to integrate legacy code, though I have no idea how hard it would be as I'm just getting started with Seastar:

I propose that in addition to Seastar's usual n^2 - n intercore queues, a program could create an extra n queues for communication between Seastar event processor threads and traditional threads.  The traditional part of the program would have one pair of queues to communicate with each event processor thread.  These queues' endpoints could even be shared amongst multiple traditional threads, with appropriate locking.  In this way the high-performance, polling event processor threads could have non-blocking communication with traditional threads (the traditional threads might block, but the event processor threads would not).  The programmer would be free to use any blocking API in the traditional threads, including creating and joining more threads.

This could work.  One problem would be balancing, how many cores would you allocate to traditional code and how many to to seastar?  Same with memory.

If those traditional libraries are not performance critical, then you can get away with just allocating a few cores for them.

Yeah.  This technique would be most useful when the traditional libraries aren't performance critical.  So reserving 1 or 2 cores should be fine.  And after sleep mode gets merged, there won't be any need to reserve cores for traditional threads.

It is merged.

 

The memory allocator would still be a problem, though.  Could Seastar reserve some memory for the traditional threads?

Sure (see --memory).  Since we intercept the memory allocation calls (malloc/new/etc.) we'd need a little extra glue to call into a traditional allocator when in a traditional context.

That's what I thought.  Alas, it's too hard for me to attempt until I get some more practice hacking on Seastar.

Another option is to use an external process, which then gets to use its own allocator.  We can share memory using memfd().

[Michael Xu]

so i've implemented a queue that works for posting. from a seastar thread you can do a safepost() on your local engine(), get a future, and later when your third party thread comes through, you can call the callable and it will schedule back on the seastar thread, setting the promise and performing the next continuation. a simple test shows it work, but im not sure it will really work. i attached a diff in case anyone is interested. definitely its a work in progress as i get used to seastar. 

to get around the memory allocator issue that avi mentioned, i used DEFAULT_ALLOCATOR when compiling seastar. does anyone have any better tips about how to handle this? i like the memory model of seastar, and using the DEFAULT_ALLOCATOR seems like the wrong choice. 

Sure (see --memory).  Since we intercept the memory allocation calls (malloc/new/etc.) we'd need a little extra glue to call into a traditional allocator when in a traditional context.

any thoughts on how to go about doing this? the hardest part right now for me is figuring out how to call a (possibly) overridden memory allocator like jemalloc or tcmalloc, from an overridden memory allocator seastar's memory.cc

regards,

michael

[Avi Kivity]

On 02/05/2016 09:23 AM, Michael Xu wrote:

so i've implemented a queue that works for posting. from a seastar thread you can do a safepost() on your local engine(), get a future, and later when your third party thread comes through, you can call the callable and it will schedule back on the seastar thread, setting the promise and performing the next continuation. a simple test shows it work, but im not sure it will really work. i attached a diff in case anyone is interested. definitely its a work in progress as i get used to seastar. 

to get around the memory allocator issue that avi mentioned, i used DEFAULT_ALLOCATOR when compiling seastar. does anyone have any better tips about how to handle this? i like the memory model of seastar, and using the DEFAULT_ALLOCATOR seems like the wrong choice. 

Sure (see --memory).  Since we intercept the memory allocation calls (malloc/new/etc.) we'd need a little extra glue to call into a traditional allocator when in a traditional context.

any thoughts on how to go about doing this? the hardest part right now for me is figuring out how to call a (possibly) overridden memory allocator like jemalloc or tcmalloc, from an overridden memory allocator seastar's memory.cc

One option is to dlopen() something like jemalloc (with RTLD_LOCAL) and then have malloc() call either the seastar functions or the library functions (obtained with dlsym) depending on whether the thread is a seastar thread or not.  free() would examine the address to check if it's in the seastar allocator's address space and dispatch according to that.

regards,

michael

On Wednesday, December 9, 2015 at 7:50:38 AM UTC-8, Avi Kivity wrote:

On 12/09/2015 05:42 PM, aso...@gmail.com wrote:

On Wednesday, December 9, 2015 at 1:21:14 AM UTC-7, Avi Kivity wrote:

On 12/09/2015 07:00 AM, aso...@gmail.com wrote:

I think that Andrew's problem is pretty common.  There are far too many legacy libraries out there to convert.  I have another idea for how to integrate legacy code, though I have no idea how hard it would be as I'm just getting started with Seastar:

I propose that in addition to Seastar's usual n^2 - n intercore queues, a program could create an extra n queues for communication between Seastar event processor threads and traditional threads.  The traditional part of the program would have one pair of queues to communicate with each event processor thread.  These queues' endpoints could even be shared amongst multiple traditional threads, with appropriate locking.  In this way the high-performance, polling event processor threads could have non-blocking communication with traditional threads (the traditional threads might block, but the event processor threads would not).  The programmer would be free to use any blocking API in the traditional threads, including creating and joining more threads.

This could work.  One problem would be balancing, how many cores would you allocate to traditional code and how many to to seastar?  Same with memory.

If those traditional libraries are not performance critical, then you can get away with just allocating a few cores for them.

Yeah.  This technique would be most useful when the traditional libraries aren't performance critical.  So reserving 1 or 2 cores should be fine.  And after sleep mode gets merged, there won't be any need to reserve cores for traditional threads.

It is merged.

 

The memory allocator would still be a problem, though.  Could Seastar reserve some memory for the traditional threads?

Sure (see --memory).  Since we intercept the memory allocation calls (malloc/new/etc.) we'd need a little extra glue to call into a traditional allocator when in a traditional context.

That's what I thought.  Alas, it's too hard for me to attempt until I get some more practice hacking on Seastar.

Another option is to use an external process, which then gets to use its own allocator.  We can share memory using memfd().

--
You received this message because you are subscribed to the Google Groups "seastar-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to seastar-dev...@googlegroups.com.
To post to this group, send email to seast...@googlegroups.com.

Visit this group at https://groups.google.com/group/seastar-dev.
To view this discussion on the web visit https://groups.google.com/d/msgid/seastar-dev/36668ac8-b338-4850-8c7d-4fb1e6d5cd57%40googlegroups.com.

[Michael Xu]

ok, i gave this a try today... and its not working at all. i attached my latest try as a diff. 

first

i added an if .. then, in allocate() and memory::free in memory.cc, and tried to do it a couple of ways... 

on the first call to allocate()/free() from a non seastar context,

1) 

i tried using dlopen(), and then using dlsym() on that handle. that doesn't appear to work, because being inside of a malloc and having dlopen() internally using malloc causes weird hangs. i'll try it again. 

2) 

i tired using dlsym(RTLD_NEXT) like this guy, http://stackoverflow.com/a/6083624, but this doesnt really appear to be working either. i seem to get hangs on static initialization startup, on cxa_guard_acquire() 

38      ../sysdeps/unix/sysv/linux/x86_64/syscall.S: No such file or directory.

(gdb) bt

#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38

#1  0x00007ffff7076ba3 in __cxa_guard_acquire () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

#2  0x0000000000578a3d in ?? ()

#3  0x000000000057ae02 in ?? ()

#4  0x000000000057b458 in malloc ()

#5  0x000000000057b565 in calloc ()

#6  0x00007ffff4d85690 in _dlerror_run (operate=operate@entry=0x7ffff4d85130 <dlsym_doit>, args=args@entry=0x7fffffffe090) at dlerror.c:141

#7  0x00007ffff4d85198 in __dlsym (handle=<optimized out>, name=<optimized out>) at dlsym.c:70

#8  0x0000000000578a1d in ?? ()

#9  0x0000000000578a52 in ?? ()

#10 0x000000000057ae02 in ?? ()

#11 0x000000000057b458 in malloc ()

#12 0x00007ffff42f244d in __fopen_internal (filename=0x7ffff571886a "/proc/self/status", mode=0x7ffff5718854 "r", is32=1) at iofopen.c:73

#13 0x00007ffff5714108 in ?? () from /usr/lib/x86_64-linux-gnu/libnuma.so.1

#14 0x00007ffff7dea0fd in call_init (l=0x7ffff7fe5000, argc=argc@entry=1, argv=argv@entry=0x7fffffffe2a8, env=env@entry=0x7fffffffe2b8) at dl-init.c:64

#15 0x00007ffff7dea223 in call_init (env=<optimized out>, argv=<optimized out>, argc=<optimized out>, l=<optimized out>) at dl-init.c:36

#16 _dl_init (main_map=0x7ffff7ffe1c8, argc=1, argv=0x7fffffffe2a8, env=0x7fffffffe2b8) at dl-init.c:126

#17 0x00007ffff7ddb30a in _dl_start_user () from /lib64/ld-linux-x86-64.so.2

#18 0x0000000000000001 in ?? ()

#19 0x00007fffffffe5a4 in ?? ()

#20 0x0000000000000000 in ?? ()

and this is with setting jemalloc as the first thing in the libs line in the build.ninja file. 

libs = -ljemalloc -laio -lboost_program_options -lboost_system -lboost_filesystem -lstdc++ -lm -lboost_unit_test_framework -lboost_thread -lcryptopp -lrt -lgnutls -lgnutlsxx -lxenstore -lhwloc -lnuma -lpciaccess -lxml2 -lz -ldl

i'm pretty lost at this point.. if anyone wants to take a look that would be super helpful, or i will continue to go deeper and deeper ;)

regards, 

[Michael Xu]

I think we might be hitting this bug, where the behavior of RTLD_NEXT is still broken after ten years.
https://sourceware.org/bugzilla/show_bug.cgi?id=1319

I shifted around the ordering of libraries and it doesn't seem to make a difference, it definitely wants to find the libc++ Malloc no matter what I do. Will investigate further.

[Avi Kivity]

On 02/06/2016 04:12 AM, Michael Xu wrote:

ok, i gave this a try today... and its not working at all. i attached my latest try as a diff. 

first

i added an if .. then, in allocate() and memory::free in memory.cc, and tried to do it a couple of ways... 

on the first call to allocate()/free() from a non seastar context,

1) 

i tried using dlopen(), and then using dlsym() on that handle. that doesn't appear to work, because being inside of a malloc and having dlopen() internally using malloc causes weird hangs. i'll try it again.

Well, try to dlopen and dlsym during initialization, so you only have to call the function during malloc().

To view this discussion on the web visit https://groups.google.com/d/msgid/seastar-dev/2bf15f2e-2728-47ad-9faa-92a2d86328de%40googlegroups.com.

[Michael Xu]

ok i think i got the third party memory allocator part to work this weekend. in the attached patch, there is a third_party_allocator flag on configure.py now. so you can do configure.py --third_party_allocator=jemalloc if you have jemalloc installed. 

test results look the same before and after this change. 

i'll work on cleaning up the other safepost patch i submitted earlier, and from there i'll try some third party libraries with traditional threads and see what happens. 

regards

michael

[Avi Kivity]

On 02/08/2016 12:10 PM, Michael Xu wrote:

ok i think i got the third party memory allocator part to work this weekend. in the attached patch, there is a third_party_allocator flag on configure.py now. so you can do configure.py --third_party_allocator=jemalloc if you have jemalloc installed. 

Please repost with git-send-email, otherwise people will ignore the patch since it takes more than a click to view it.

To view this discussion on the web visit https://groups.google.com/d/msgid/seastar-dev/aca6e6ff-85c8-44a7-80ae-5b8a9d4b09cd%40googlegroups.com.

[Michael Xu]

after reading the seastar memory allocator code, im not sure i understand why requiring a third party memory allocator is strictly required... 

afaik, seastar memory will still be new'd and free'd correctly even if it is free'd on a thread on a thread which it was not new'd on... 

so is there strictly a requirement that using non-seastar threads actually require their own memory allocator? 

michael

[Andrew Melo]

Hi Michael,

On Tuesday, February 9, 2016, Michael Xu <mx2...@gmail.com> wrote:

after reading the seastar memory allocator code, im not sure i understand why requiring a third party memory allocator is strictly required... 

afaik, seastar memory will still be new'd and free'd correctly even if it is free'd on a thread on a thread which it was not new'd on... 

so is there strictly a requirement that using non-seastar threads actually require their own memory allocator? 

I think the problem comes from seastar's memory allocator requiring memory to be freed from the same core it was allocated from. (At least that was my understanding when I first started this thread.) if you offload operations to external threads who could migrate or share data between cores, it breaks that constraint.

Andrew

--
It's dark in this basement.

[Michael Xu]

hey andrew, thats what i thought too, until i saw these lines of code in core/memory.cc 

line 554 says if the cpu_id of the ptr doesnt match the current cpu_id, put it on the originating cpu's free list. 

 551 void cpu_pages::free(void* ptr) {

 552     auto obj_cpu = object_cpu_id(ptr);

 553     if (obj_cpu != cpu_id) {

 554         return free_cross_cpu(obj_cpu, ptr);

 555     }

 556     page* span = to_page(ptr);

 557     if (span->pool) {

 558         span->pool->deallocate(ptr);

 559     } else {

 560         free_large(ptr);

 561     }

 562 }

--

Michael Xu
(770)634-1738
mic...@michaelxu.co

[Avi Kivity]

On 02/09/2016 09:32 PM, Michael Xu wrote:

hey andrew, thats what i thought too, until i saw these lines of code in core/memory.cc 

line 554 says if the cpu_id of the ptr doesnt match the current cpu_id, put it on the originating cpu's free list. 

 551 void cpu_pages::free(void* ptr) {

 552     auto obj_cpu = object_cpu_id(ptr);

 553     if (obj_cpu != cpu_id) {

 554         return free_cross_cpu(obj_cpu, ptr);

 555     }

 556     page* span = to_page(ptr);

 557     if (span->pool) {

 558         span->pool->deallocate(ptr);

 559     } else {

 560         free_large(ptr);

 561     }

 562 }

Note that this mechanism is very slow.  But that's not the real issue -- the seastar memory allocator doesn't support efficient allocation from non-reactor threads, so if the application is also allocating significant amounts, it will be very slow.

To view this discussion on the web visit https://groups.google.com/d/msgid/seastar-dev/CAHxARCG7XtF_g2Hr%2Bu2EVpu%3DdmVTTWMMh%3DpDh6ZiFcnji0aH-w%40mail.gmail.com.

[Michael Xu]

heres a similar question, but a different situation. 

lets say i've built an api using seastar's futures... so i have functions that return futures. 

future<> doSomething() {

}

what if i have a non-seastar thread, that i want to use to synchronously call this doSomething(). yes, its blocking, but in some situations, such as tests, or third party libraries that expect synchronous execution.. this is something that could happen. 

ideally, a syntax for this would look like 

//in non-seastar thread (maybe a main test thread, maybe a third party library thread that we want to use to call into our seastar API). 

future<int> bar = doSomething();

int result = bar.blocking_wait();

return result; 

i know that boost promise and futures have a blocking get(). is there an analogous call in seastar? i did not see one. the closest thing is that future::get() calls wait(), but that does something with seastar threads, which i dont think applies here.. 

thoughts? 

michael

...

[Avi Kivity]

On 02/16/2016 05:06 AM, Michael Xu wrote:

heres a similar question, but a different situation. 

lets say i've built an api using seastar's futures... so i have functions that return futures. 

future<> doSomething() {

}

what if i have a non-seastar thread, that i want to use to synchronously call this doSomething(). yes, its blocking, but in some situations, such as tests, or third party libraries that expect synchronous execution.. this is something that could happen. 

ideally, a syntax for this would look like 

//in non-seastar thread (maybe a main test thread, maybe a third party library thread that we want to use to call into our seastar API). 

future<int> bar = doSomething();

int result = bar.blocking_wait();

return result; 

i know that boost promise and futures have a blocking get(). is there an analogous call in seastar? i did not see one. the closest thing is that future::get() calls wait(), but that does something with seastar threads, which i dont think applies here.. 

thoughts? 

A very similar problem exists with seastar's smp::submit_to().  If you call

  smp::submit_to(1, doSomething).then(...)

Then the effect is that you run doSomething on shard 1, and shard 0 waits for it.  From shard 1's point of view the calling thread is foreign.

The trick is to do the waiting in the seastar thread in a non-blocking way: call future::then_wrapped().  The result is a future that is ready, then pass the result to the non-seastar thread in a queue.

Perhaps the best API would be to use std::future or boost::future on the foreign side:

   template <typename T, typename Func>
   std::future<T>
   run_on_seastar(unsigned shard, Func func);

where func() returns a seastar future<T>.

It would work something like this:

   1. run_on_seastar converts func to a work item and puts it on a queue.
   2. run_on_seastar creates an std::promise<T>, saves it somewhere, and returns its get_future() to the caller.
   3. a seastar poller picks up the work item from 1, calls the function, and calls then_wrapped() on the result.
   4. 3's continuation extracts the result from the seastar future, and pushes it to a response queue (possibly signalling the other side).
   5. the other side picks up the result from step 4 and writes it the promise from step 2.
  
  

--

You received this message because you are subscribed to the Google Groups "seastar-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to seastar-dev...@googlegroups.com.
To post to this group, send email to seast...@googlegroups.com.
Visit this group at https://groups.google.com/group/seastar-dev.

To view this discussion on the web visit https://groups.google.com/d/msgid/seastar-dev/8d9a20ef-8343-44c7-8681-95065d08ba1f%40googlegroups.com.

[chunmei.liu@intel.com]

was these two patches merged into Seastar? Does the traditional thread really work by these two patches?  I am also interested in this topic.

Thanks!

-Chunmei

[Avi Kivity]

https://github.com/scylladb/seastar/commit/fe824bad88259406b739df282df4708f08e6965d

To view this discussion on the web visit https://groups.google.com/d/msgid/seastar-dev/d4318b29-795b-4ec5-853b-5f046a641aa6%40googlegroups.com.

[Nadav Har'El]

May I hope that one day somebody would document this feature in the Seastar tutorial?

The alien::submit_to() feature seems interesting, but it doesn't quite begin to cover the issue of how to write an application where

some of the threads are not controlled by Seastar. How does one designate these non-Seastar threads? How does memory

allocation work in them? And so on.

Yes, beginners can spend days scouring through the Seastar source code for every feature which they wish to learn.

Or, Seastar features could be clearly explained in the tutorial.

Both approaches work. But guess which approach I think is better, and will improve the adoption of Seastar?

https://github.com/scylladb/seastar/commit/fe824bad88259406b739df282df4708f08e6965d

To unsubscribe from this group and stop receiving emails from it, send an email to seastar-dev+unsubscribe@googlegroups.com.

To post to this group, send email to seast...@googlegroups.com.
Visit this group at https://groups.google.com/group/seastar-dev.
To view this discussion on the web visit https://groups.google.com/d/msgid/seastar-dev/d4318b29-795b-4ec5-853b-5f046a641aa6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "seastar-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to seastar-dev+unsubscribe@googlegroups.com.

To post to this group, send email to seast...@googlegroups.com.
Visit this group at https://groups.google.com/group/seastar-dev.

To view this discussion on the web visit https://groups.google.com/d/msgid/seastar-dev/f4c8db7b-fe36-102c-3869-839e432a6722%40scylladb.com.

[Avi Kivity]

On 2018-05-01 11:52, Nadav Har'El wrote:

May I hope that one day somebody would document this feature in the Seastar tutorial?

The alien::submit_to() feature seems interesting, but it doesn't quite begin to cover the issue of how to write an application where

some of the threads are not controlled by Seastar. How does one designate these non-Seastar threads?

You don't need to designate them.

How does memory

allocation work in them? And so on.

Hybrid applications should use the default allocator, the seastar allocator won't work well with them.

Yes, beginners can spend days scouring through the Seastar source code for every feature which they wish to learn.

Or, Seastar features could be clearly explained in the tutorial.

Both approaches work. But guess which approach I think is better, and will improve the adoption of Seastar?

No idea, a hint?

Not everything needs to be in the tutorial. Some things can be kept in the reference, or in a topic-specific example within the reference.

[Nadav Har'El]

On Tue, May 1, 2018 at 12:09 PM, Avi Kivity <a...@scylladb.com> wrote:

On 2018-05-01 11:52, Nadav Har'El wrote:

May I hope that one day somebody would document this feature in the Seastar tutorial?

The alien::submit_to() feature seems interesting, but it doesn't quite begin to cover the issue of how to write an application where

some of the threads are not controlled by Seastar. How does one designate these non-Seastar threads?

You don't need to designate them.

Maybe I misunderstood you, or you misunderstood me, but I think you *do* need to designate some of the

cores/threads to NOT run Seastar. The threads running the Seastar reactor must never block, so the

"alien" code needs to run on different threads, no?

 

How does memory

allocation work in them? And so on.

Hybrid applications should use the default allocator, the seastar allocator won't work well with them.

Ok, that's an example of a point that needs to be explained somewhere. But more importantly, is
this strictly true? Is it true because this is what we have today, or is there a fundamental reason why
it must be true?  Can't the hybrid application have *both* sharded memory, and, part of the memory shared?

I.e., malloc() on the "alien" code will be done from shared memory, and malloc() on the "seastar" code

be done from per-core memory? (Of course, free() needs to do the right thing, etc.)

 

Yes, beginners can spend days scouring through the Seastar source code for every feature which they wish to learn.

Or, Seastar features could be clearly explained in the tutorial.

Both approaches work. But guess which approach I think is better, and will improve the adoption of Seastar?

No idea, a hint?

Ok, here's a hint :-)

Not everyone appreciates having to spend a few days (!) researching a question - like how to create a hybrid Seastar/regular

application. In this and many other cases which aren't explained in the tutorial, you can find hints all over the mailing list ("default

allocator", "alien threads", etc.) and then you need to sit for hours reading Seastar code, its (very little) comments, and

various commit messages and mailing list messages to collect the information you need.

It's so frustrating when you know that other people know exactly how to do this, but this knowledge is not written down

anywhere.

 

Not everything needs to be in the tutorial. Some things can be kept in the reference, or in a topic-specific example within the reference.

What is this mythical "reference" you are talking about?

Maybe the file name "tutorial.md" is the wrong name (the word "tutorial" doesn't actually appear in the title of the document).

It's not meant to be a "getting started tutorial", if that's what you understood a "tutorial" to be.

It's meant to be both a read-from-cover-to-cover tutorial, and a reference book for more experienced users who already know what feature they are searching for, but now want a clear, step-by-step explanation on how to use this feature.

The "doxygen" as a reference - if that's what you consider "reference" - is not good enough, because it explains each function separately. In this example, you would get an explanation of alien::submit_to(), but without the context, without the bigger picture of how to use it and a dozen other small things to write an application which combines. Yes, you can add various HTML files etc. to the doxygen to explain packages and various "global" stuff, but it's not a good format to find anything in. In this example, which "package" is supposed to contain an explanation on how to use alien::submit_to() together with default malloc() and a bunch of other details from all over the seastar namespace, to create hybrid applications?

[Avi Kivity]

On 2018-05-01 12:51, Nadav Har'El wrote:

On Tue, May 1, 2018 at 12:09 PM, Avi Kivity <a...@scylladb.com> wrote:

On 2018-05-01 11:52, Nadav Har'El wrote:

May I hope that one day somebody would document this feature in the Seastar tutorial?

The alien::submit_to() feature seems interesting, but it doesn't quite begin to cover the issue of how to write an application where

some of the threads are not controlled by Seastar. How does one designate these non-Seastar threads?

You don't need to designate them.

Maybe I misunderstood you, or you misunderstood me, but I think you *do* need to designate some of the

cores/threads to NOT run Seastar. The threads running the Seastar reactor must never block, so the

"alien" code needs to run on different threads, no?

Yes, but you don't need to designate them. Just launch an std::thread, either from within a Seastar thread or outside it.

Seastar doesn't need to know about it.

 

How does memory

allocation work in them? And so on.

Hybrid applications should use the default allocator, the seastar allocator won't work well with them.

Ok, that's an example of a point that needs to be explained somewhere. But more importantly, is
this strictly true? Is it true because this is what we have today, or is there a fundamental reason why
it must be true?  Can't the hybrid application have *both* sharded memory, and, part of the memory shared?

I.e., malloc() on the "alien" code will be done from shared memory, and malloc() on the "seastar" code

be done from per-core memory? (Of course, free() needs to do the right thing, etc.)

I don't like the word "fundamentally" (or fundamentalism in general). Everything must "fundamentally" be so until a workaround is found.

The Seastar allocator will divide all of the memory across shards, and provide very poor service (in both performance and amount of memory available) to non-Seastar threads. So currently you should disable the seastar allocator.

It's possible for a future improvement to provide better service to non-seastar threads, and of course you can reduce the amount of memory managed by Seastar.

 

Yes, beginners can spend days scouring through the Seastar source code for every feature which they wish to learn.

Or, Seastar features could be clearly explained in the tutorial.

Both approaches work. But guess which approach I think is better, and will improve the adoption of Seastar?

No idea, a hint?

Ok, here's a hint :-)

I was sarcastic. Of course having everything documented is better.

Not everyone appreciates having to spend a few days (!) researching a question - like how to create a hybrid Seastar/regular

application. In this and many other cases which aren't explained in the tutorial, you can find hints all over the mailing list ("default

allocator", "alien threads", etc.) and then you need to sit for hours reading Seastar code, its (very little) comments, and

various commit messages and mailing list messages to collect the information you need.

It's so frustrating when you know that other people know exactly how to do this, but this knowledge is not written down

anywhere.

 

Not everything needs to be in the tutorial. Some things can be kept in the reference, or in a topic-specific example within the reference.

What is this mythical "reference" you are talking about?

http://docs.seastar.io.

Maybe the file name "tutorial.md" is the wrong name (the word "tutorial" doesn't actually appear in the title of the document).

It's not meant to be a "getting started tutorial", if that's what you understood a "tutorial" to be.

It's meant to be both a read-from-cover-to-cover tutorial, and a reference book for more experienced users who already know what feature they are searching for, but now want a clear, step-by-step explanation on how to use this feature.

The "doxygen" as a reference - if that's what you consider "reference" - is not good enough, because it explains each function separately. In this example, you would get an explanation of alien::submit_to(), but without the context, without the bigger picture of how to use it and a dozen other small things to write an application which combines. Yes, you can add various HTML files etc. to the doxygen to explain packages and various "global" stuff, but it's not a good format to find anything in. In this example, which "package" is supposed to contain an explanation on how to use alien::submit_to() together with default malloc() and a bunch of other details from all over the seastar namespace, to create hybrid applications?

Of course, more context is needed. That is why we have modules in the reference, for example http://docs.seastar.io/master/group__thread-module.html. These provide (or should provide) a short description of the functionality being provided, followed by a detailed function-by-function description.

I think it is easier (if properly completed) to search for something in a reference, rather than having to read a long tutorial or a short book, once you have the basics down (which the tutorial does a very good job of).

[Nadav Har'El]

On Tue, May 1, 2018 at 1:00 PM, Avi Kivity <a...@scylladb.com> wrote:

On 2018-05-01 12:51, Nadav Har'El wrote:

Not everyone appreciates having to spend a few days (!) researching a question - like how to create a hybrid Seastar/regular

application. In this and many other cases which aren't explained in the tutorial, you can find hints all over the mailing list ("default

allocator", "alien threads", etc.) and then you need to sit for hours reading Seastar code, its (very little) comments, and

various commit messages and mailing list messages to collect the information you need.

It's so frustrating when you know that other people know exactly how to do this, but this knowledge is not written down

anywhere.

 

Not everything needs to be in the tutorial. Some things can be kept in the reference, or in a topic-specific example within the reference.

What is this mythical "reference" you are talking about?

http://docs.seastar.io.

Tzach, where do I put issues for docs.seastar.io?

In particular, I want to see there a page-per-section HTML version of the tutorial (automatic result of "ninja doc/split").
I am hoping that maybe then Avi will no longer see the tutorial as just a book you need to read from cover to cover,

and maybe he'll appreciate why it's a better format than doxygen's "module" feature.

 

Maybe the file name "tutorial.md" is the wrong name (the word "tutorial" doesn't actually appear in the title of the document).

It's not meant to be a "getting started tutorial", if that's what you understood a "tutorial" to be.

It's meant to be both a read-from-cover-to-cover tutorial, and a reference book for more experienced users who already know what feature they are searching for, but now want a clear, step-by-step explanation on how to use this feature.

The "doxygen" as a reference - if that's what you consider "reference" - is not good enough, because it explains each function separately. In this example, you would get an explanation of alien::submit_to(), but without the context, without the bigger picture of how to use it and a dozen other small things to write an application which combines. Yes, you can add various HTML files etc. to the doxygen to explain packages and various "global" stuff, but it's not a good format to find anything in. In this example, which "package" is supposed to contain an explanation on how to use alien::submit_to() together with default malloc() and a bunch of other details from all over the seastar namespace, to create hybrid applications?

Of course, more context is needed. That is why we have modules in the reference, for example http://docs.seastar.io/master/group__thread-module.html. These provide (or should provide) a short description of the functionality being provided, followed by a detailed function-by-function description.

I think it is easier (if properly completed) to search for something in a reference, rather than having to read a long tutorial or a short book,

See above why you don't need to "read a long tutorial". You can just skip to a section on "hybrid applications" if that's what you're looking for,

and read only that section (which should be on its own HTML page - not part of a million-line HTML file). In fact, it's probably easier than

understanding where the heck you can find this information inside a doxygen hierarchy of classes and namespace (the feature we're

discussing now has nothing to do with specific classes or namespaces.).

 

once you have the basics down (which the tutorial does a very good job of).

Proverbially, 90% of the users need 10% of the features - but each needs a different 10% :-) Which is why we can't stop at documenting 10% of the features.

I think I need to rename this effort from "tutorial" to something else because I think it's misleading you.
Perhaps "Users guide".
As I said, the word "tutorial" doesn't even appear in the title of that document.

[Avi Kivity]

I do think we need a tutorial. A one-two page document that can be used to getting started, doesn't explain all the details, but allows someone to get a feel for the framework.

The book reference is also useful for someone looking to dig deeper, but not necessarily when trying to accomplish something right now.

The online reference is useful for someone looking to accomplish something right now. It is the standard documentation - you want to know how to do X, you go to the X documentation, you don't read a chapter in a book that happens to describe X.

[Tzach Livyatan]

On Tue, May 1, 2018 at 1:19 PM, Nadav Har'El <n...@scylladb.com> wrote:

On Tue, May 1, 2018 at 1:00 PM, Avi Kivity <a...@scylladb.com> wrote:

On 2018-05-01 12:51, Nadav Har'El wrote:

Not everyone appreciates having to spend a few days (!) researching a question - like how to create a hybrid Seastar/regular

application. In this and many other cases which aren't explained in the tutorial, you can find hints all over the mailing list ("default

allocator", "alien threads", etc.) and then you need to sit for hours reading Seastar code, its (very little) comments, and

various commit messages and mailing list messages to collect the information you need.

It's so frustrating when you know that other people know exactly how to do this, but this knowledge is not written down

anywhere.

 

Not everything needs to be in the tutorial. Some things can be kept in the reference, or in a topic-specific example within the reference.

What is this mythical "reference" you are talking about?

http://docs.seastar.io.

Tzach, where do I put issues for docs.seastar.io?

In particular, I want to see there a page-per-section HTML version of the tutorial (automatic result of "ninja doc/split").
I am hoping that maybe then Avi will no longer see the tutorial as just a book you need to read from cover to cover,

and maybe he'll appreciate why it's a better format than doxygen's "module" feature.

Seastar.io website source: https://github.com/scylladb/seastar-website

docs.seastar.io is generated automatically from https://github.com/scylladb/seastar/tree/master/doc

so issue to it best fit in seastar project itself.

 

[Nadav Har'El]

That's not a tutorial, that's an advertisement.

After reading a two page document, it's *impossible* to write anything - not even the most basic program - with Seastar.

 

The book reference is also useful for someone looking to dig deeper, but not necessarily when trying to accomplish something right now.

That's what I've been trying to write.

But it's *both* a "book" and "online".

 

The online reference is useful for someone looking to accomplish something right now. It is the standard documentation - you want to know how to do X, you go to the X documentation, you don't read a chapter in a book that happens to describe X.

When you do a Google search for X, you usually end up exactly in a "chapter in a book that happens to describe X".

I just think you're allergic to the word "book". Let's call it a "website" instead. And instead of a "chapter", call it a page.

This is exactly the "split HTML" output format we already have - just not on the seastar.io site (which has the doxygen-generated single-page version, which I think is bad).

 

[chunmei.liu@intel.com]

I still have some questions about seastar thread + traditional thread. 

From the patch and test code, seems traditional thread use  alien::submit_to to talk with seastar thread and seastar thread use alien::smp::poll_queues to get the "message",  how about inverse, if seastar thread need  trigger the talk with traditional thread, which func to call?

if use the default memory allocator, then seastar thread and traditional thread will use the same allocator, right? we will lose the seastar allocator advantage and memory will not be reserved for seastar thread, right? 

Thanks!

[Avi Kivity]

On 2018-05-03 00:55, chunm...@intel.com wrote:

I still have some questions about seastar thread + traditional thread. 

From the patch and test code, seems traditional thread use  alien::submit_to to talk with seastar thread and seastar thread use alien::smp::poll_queues to get the "message",  how about inverse, if seastar thread need  trigger the talk with traditional thread, which func to call?

It can use any method it wants, like std::condition_variable::notify_all() or std::async().

if use the default memory allocator, then seastar thread and traditional thread will use the same allocator, right? we will lose the seastar allocator advantage and memory will not be reserved for seastar thread, right?

Right.

Thanks!

To view this discussion on the web visit https://groups.google.com/d/msgid/seastar-dev/d889eced-78bc-46b7-b7cf-7116902419eb%40googlegroups.com.

[chunmei.liu@intel.com]

On Thursday, May 3, 2018 at 12:55:43 AM UTC-7, Avi Kivity wrote:

On 2018-05-03 00:55, chunm...@intel.com wrote:

I still have some questions about seastar thread + traditional thread. 

From the patch and test code, seems traditional thread use  alien::submit_to to talk with seastar thread and seastar thread use alien::smp::poll_queues to get the "message",  how about inverse, if seastar thread need  trigger the talk with traditional thread, which func to call?

It can use any method it wants, like std::condition_variable::notify_all() or std::async().

Don't understand here, why traditional thread need alien::submit_to to talk with seastar thread, but inverse, seathread thread only need the normal method to talk with traditional thread?  

[Avi Kivity]

On 2018-05-04 02:27, chunm...@intel.com wrote:

On Thursday, May 3, 2018 at 12:55:43 AM UTC-7, Avi Kivity wrote:

On 2018-05-03 00:55, chunm...@intel.com wrote:

I still have some questions about seastar thread + traditional thread. 

From the patch and test code, seems traditional thread use  alien::submit_to to talk with seastar thread and seastar thread use alien::smp::poll_queues to get the "message",  how about inverse, if seastar thread need  trigger the talk with traditional thread, which func to call?

It can use any method it wants, like std::condition_variable::notify_all() or std::async().

Don't understand here, why traditional thread need alien::submit_to to talk with seastar thread, but inverse, seathread thread only need the normal method to talk with traditional thread? 

A seastar thread listens for events in the reactor, so to send it messages, you need to use a method that the reactor understands.

Non-seastar threads use standard ways to wait for events, so you can use the standard ways to wake them.

[nguyenhoangquanptit@gmail.com]

Hi Mr.Avi,

I intend to build a high throughput, low latency async TCP client base on Seastar core. My intend is to run Seastar in some cores and run the user Client task (where user can send requests and receive responses) in other cores.

I would like to use allien::submit_to() to send requests from user thread in to reactor threads.

Can I archive high performance client this way?

Many thanks.

[Kefu Chai]

On Thursday, August 9, 2018 at 9:15:21 AM UTC+8, nguyenhoa...@gmail.com wrote:

Hi Mr.Avi,

I intend to build a high throughput, low latency async TCP client base on Seastar core. My intend is to run Seastar in some cores and run the user Client task (where user can send requests and receive responses) in other cores.

I would like to use allien::submit_to() to send requests from user thread in to reactor threads.

please note, the use case of alien::submit_to() is that one wants to schedule tasks to run on seastar fiber from traditional threads. so in your case, i guess, you are simply using seastar as a messenger. and please note, you might want to figure out how to manage the life cycle of requests/responses, and how to wait for the received request. so, IMHO, whether you will have a high performance client really depends on your use case and app stack.

BTW, if you don't need to wait for the completion of sending requests, probably seastar::alien::run_on() is a better alternative. because alien::submit_to() returns a std::future<>, and we don't get std::future<>/std::promise<> for free, they are implemented using std::condition_variable, which in turn is implemented using pthread_cond_t and pthread_mutex_t.

[nguyenhoangquanptit@gmail.com]

Hi Kefu,

You are right about the case. I have my server side based on Seastar core. By now, I would like to develop a C++ client for it.

We've already had a Java client that uses ExecutorService to execute tasks (make it schedule). In Java, we have Reactors to run as the task scheduler and another Executor pool where received messages from reactors are dispatch to. The executor pool processes user application logic.

I think that C++ client should work in the same mechanism like Java client for high performance.

For benchmarking, I run app_template.run_deprecated([] {}) and put user requests to be scheduled right inside the lambda. The performance is quite good.

In my case, can simple call alien::submit_to() provide a good performance OR do I need to implement my own scheduler to get my need.

Many thanks!

[nguyenhoangquanptit@gmail.com]

By now,

I've changed my mind about the case. All user requests should be processed inside seastar core schedulers.

I'm using alien::run_on() to send message in to it. But it has the problem with the message queues _qs inside alien. It does not implements a Throttler for the queues.

For example, my seastar fiber runs on 4 cores. I run a for loop to send 600 messages using alien::run_on(), it will crash due to the queues' capacity exceeds (4 cores with 4 queues in _qs, 128 items each, total of 512 items).

Do you think that I should implement the Throttler myself or should the Throttler be implemented in alien::_qs?

On Thursday, August 9, 2018 at 7:08:32 PM UTC+7, Kefu Chai wrote:

[luo.mai.cs@gmail.com]

Hello, I am wondering how is your exploration for using alien::run_on()? In addition to Throttler, did you observe a high overhead? I am having a similar use case and considering adopt seastar as a messenger for a traditional application.