Dynamic PTHREAD_POOL_SIZE ?
50 views
Skip to first unread message
Talha Enes Ayrancı
May 8, 2020, 11:31:10 AM5/8/20
to emscripten-discuss
Hi. I have a image processing app with coding C++. When I compile my codes with USE_PTHREAD=1 flag, compiled code is not working. Nothing happening on Chrome. And when I add PTHREAD_POOL_SIZE flag, it's working. But my thread size is different for every file. Big files are using more threads than small sized files. Thread count is increasing to 40-50 and more and more..
50 POOL_SIZE is so big value. What can I do?
50 POOL_SIZE is so big value. What can I do?
Alon Zakai
May 8, 2020, 4:42:35 PM5/8/20
to emscripte...@googlegroups.com
If you want the ability to pick PTHREAD_POOL_SIZE at runtime, you can do that by setting PTHREAD_POOL_SIZE to something like "Module.pthreadPoolSize", and build with MODULARIZE. Then you can pass in { pthreadPoolSize: .. } when you create an instance, and give it the pool size at runtime there.
(Maybe I didn't understand the description, but if your program itself uses more threads depending on the size of the file, you may need to make the program use up to a fixed number of threads.)
On Fri, May 8, 2020 at 8:31 AM Talha Enes Ayrancı <talha...@gmail.com> wrote:
Hi. I have a image processing app with coding C++. When I compile my codes with USE_PTHREAD=1 flag, compiled code is not working. Nothing happening on Chrome. And when I add PTHREAD_POOL_SIZE flag, it's working. But my thread size is different for every file. Big files are using more threads than small sized files. Thread count is increasing to 40-50 and more and more..
50 POOL_SIZE is so big value. What can I do?
--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/e9909928-b9ad-43e3-bb6e-c83935305ee7%40googlegroups.com.
Sam Clegg
May 9, 2020, 12:57:05 AM5/9/20
to emscripte...@googlegroups.com
But doesn't the PTHREAD_POOL_SIZE just determine how many initial threads to create?
IIRC you can still create more threads than that at runtime.. up until you run out of machine resources. i.e. PTHREAD_POOL_SIZE is not an upper limit on the total number of threads.
IIRC you can still create more threads than that at runtime.. up until you run out of machine resources. i.e. PTHREAD_POOL_SIZE is not an upper limit on the total number of threads.
The only caveat is that new threads don't actually start until you return to the event loop (at least when they are started from the main thread).
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CAEX4NpTjA6Bu3BjUB8b_bHtUemniO0XXafOf4K8Kh4urMXom5w%40mail.gmail.com.
Talha Enes Ayrancı
May 9, 2020, 5:26:37 AM5/9/20
to emscripten-discuss
Yes, exactly my program itself uses more threads depending on the size of the file. (Actually not file size, is depending the content of the file but doesn't matter.)
Why we need to pick PTHREAD_POOL_SIZE ? According to this topic, this flag is predefining pool size before main() calling and I don't have a main function. I am calling Embinded functions from Javascript side.
8 Mayıs 2020 Cuma 23:42:35 UTC+3 tarihinde Alon Zakai yazdı:
8 Mayıs 2020 Cuma 23:42:35 UTC+3 tarihinde Alon Zakai yazdı:
If you want the ability to pick PTHREAD_POOL_SIZE at runtime, you can do that by setting PTHREAD_POOL_SIZE to something like "Module.pthreadPoolSize", and build with MODULARIZE. Then you can pass in { pthreadPoolSize: .. } when you create an instance, and give it the pool size at runtime there.(Maybe I didn't understand the description, but if your program itself uses more threads depending on the size of the file, you may need to make the program use up to a fixed number of threads.)
On Fri, May 8, 2020 at 8:31 AM Talha Enes Ayrancı <talha...@gmail.com> wrote:
Hi. I have a image processing app with coding C++. When I compile my codes with USE_PTHREAD=1 flag, compiled code is not working. Nothing happening on Chrome. And when I add PTHREAD_POOL_SIZE flag, it's working. But my thread size is different for every file. Big files are using more threads than small sized files. Thread count is increasing to 40-50 and more and more..--
50 POOL_SIZE is so big value. What can I do?
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsub...@googlegroups.com.
Talha Enes Ayrancı
May 9, 2020, 5:42:34 AM5/9/20
to emscripten-discuss
Probably this is the case. New threads don't start. So I need give higher values to PTHREAD_POOL_SIZE.
But I think this is not a healty solution. Because as the number of threads increases, RAM usage increases. What is your advices?
9 Mayıs 2020 Cumartesi 07:57:05 UTC+3 tarihinde Sam Clegg yazdı:
9 Mayıs 2020 Cumartesi 07:57:05 UTC+3 tarihinde Sam Clegg yazdı:
But doesn't the PTHREAD_POOL_SIZE just determine how many initial threads to create?
IIRC you can still create more threads than that at runtime.. up until you run out of machine resources. i.e. PTHREAD_POOL_SIZE is not an upper limit on the total number of threads.The only caveat is that new threads don't actually start until you return to the event loop (at least when they are started from the main thread).
On Fri, May 8, 2020 at 1:42 PM Alon Zakai <alon...@gmail.com> wrote:
If you want the ability to pick PTHREAD_POOL_SIZE at runtime, you can do that by setting PTHREAD_POOL_SIZE to something like "Module.pthreadPoolSize", and build with MODULARIZE. Then you can pass in { pthreadPoolSize: .. } when you create an instance, and give it the pool size at runtime there.(Maybe I didn't understand the description, but if your program itself uses more threads depending on the size of the file, you may need to make the program use up to a fixed number of threads.)
On Fri, May 8, 2020 at 8:31 AM Talha Enes Ayrancı <talha...@gmail.com> wrote:
Hi. I have a image processing app with coding C++. When I compile my codes with USE_PTHREAD=1 flag, compiled code is not working. Nothing happening on Chrome. And when I add PTHREAD_POOL_SIZE flag, it's working. But my thread size is different for every file. Big files are using more threads than small sized files. Thread count is increasing to 40-50 and more and more..--
50 POOL_SIZE is so big value. What can I do?
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/e9909928-b9ad-43e3-bb6e-c83935305ee7%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsub...@googlegroups.com.
Jukka Jylänki
May 9, 2020, 12:26:23 PM5/9/20
to emscripte...@googlegroups.com
If your threads are not starting, your main browser thread code is
synchronously depending on them after their creation. Refactor the
code to not assume that the threads would be synchronously available
after creation, but periodically yield back to the main thread browser
event loop after the thread creation.
With such structure, you can set -s PTHREAD_POOL_SIZE=0.
There is no possibility to have an infinite/unbounded number of
synchronously available threads on the web. Although by using asyncify
emulation, that might be possible, but not sure if asyncify handles
this atm?
la 9. toukok. 2020 klo 12.42 Talha Enes Ayrancı
(talha...@gmail.com) kirjoitti:
> To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/6c1f75cf-3598-40f6-a534-323737342653%40googlegroups.com.
synchronously depending on them after their creation. Refactor the
code to not assume that the threads would be synchronously available
after creation, but periodically yield back to the main thread browser
event loop after the thread creation.
With such structure, you can set -s PTHREAD_POOL_SIZE=0.
There is no possibility to have an infinite/unbounded number of
synchronously available threads on the web. Although by using asyncify
emulation, that might be possible, but not sure if asyncify handles
this atm?
la 9. toukok. 2020 klo 12.42 Talha Enes Ayrancı
(talha...@gmail.com) kirjoitti:
>
> Probably this is the case. New threads don't start. So I need give higher values to PTHREAD_POOL_SIZE.
>
> But I think this is not a healty solution. Because as the number of threads increases, RAM usage increases. What is your advices?
>
> 9 Mayıs 2020 Cumartesi 07:57:05 UTC+3 tarihinde Sam Clegg yazdı:
>>
>> But doesn't the PTHREAD_POOL_SIZE just determine how many initial threads to create?
>>
>> IIRC you can still create more threads than that at runtime.. up until you run out of machine resources. i.e. PTHREAD_POOL_SIZE is not an upper limit on the total number of threads.
>>
>> The only caveat is that new threads don't actually start until you return to the event loop (at least when they are started from the main thread).
>>
>> On Fri, May 8, 2020 at 1:42 PM Alon Zakai <alon...@gmail.com> wrote:
>>>
>>> If you want the ability to pick PTHREAD_POOL_SIZE at runtime, you can do that by setting PTHREAD_POOL_SIZE to something like "Module.pthreadPoolSize", and build with MODULARIZE. Then you can pass in { pthreadPoolSize: .. } when you create an instance, and give it the pool size at runtime there.
>>>
>>> (Maybe I didn't understand the description, but if your program itself uses more threads depending on the size of the file, you may need to make the program use up to a fixed number of threads.)
>>>
>>>
>>> On Fri, May 8, 2020 at 8:31 AM Talha Enes Ayrancı <talha...@gmail.com> wrote:
>>>>
>>>> Hi. I have a image processing app with coding C++. When I compile my codes with USE_PTHREAD=1 flag, compiled code is not working. Nothing happening on Chrome. And when I add PTHREAD_POOL_SIZE flag, it's working. But my thread size is different for every file. Big files are using more threads than small sized files. Thread count is increasing to 40-50 and more and more..
>>>> 50 POOL_SIZE is so big value. What can I do?
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
> Probably this is the case. New threads don't start. So I need give higher values to PTHREAD_POOL_SIZE.
>
> But I think this is not a healty solution. Because as the number of threads increases, RAM usage increases. What is your advices?
>
> 9 Mayıs 2020 Cumartesi 07:57:05 UTC+3 tarihinde Sam Clegg yazdı:
>>
>> But doesn't the PTHREAD_POOL_SIZE just determine how many initial threads to create?
>>
>> IIRC you can still create more threads than that at runtime.. up until you run out of machine resources. i.e. PTHREAD_POOL_SIZE is not an upper limit on the total number of threads.
>>
>> The only caveat is that new threads don't actually start until you return to the event loop (at least when they are started from the main thread).
>>
>> On Fri, May 8, 2020 at 1:42 PM Alon Zakai <alon...@gmail.com> wrote:
>>>
>>> If you want the ability to pick PTHREAD_POOL_SIZE at runtime, you can do that by setting PTHREAD_POOL_SIZE to something like "Module.pthreadPoolSize", and build with MODULARIZE. Then you can pass in { pthreadPoolSize: .. } when you create an instance, and give it the pool size at runtime there.
>>>
>>> (Maybe I didn't understand the description, but if your program itself uses more threads depending on the size of the file, you may need to make the program use up to a fixed number of threads.)
>>>
>>>
>>> On Fri, May 8, 2020 at 8:31 AM Talha Enes Ayrancı <talha...@gmail.com> wrote:
>>>>
>>>> Hi. I have a image processing app with coding C++. When I compile my codes with USE_PTHREAD=1 flag, compiled code is not working. Nothing happening on Chrome. And when I add PTHREAD_POOL_SIZE flag, it's working. But my thread size is different for every file. Big files are using more threads than small sized files. Thread count is increasing to 40-50 and more and more..
>>>> 50 POOL_SIZE is so big value. What can I do?
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
>>>> To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/e9909928-b9ad-43e3-bb6e-c83935305ee7%40googlegroups.com.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CAEX4NpTjA6Bu3BjUB8b_bHtUemniO0XXafOf4K8Kh4urMXom5w%40mail.gmail.com.
>
>
> --
> You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
> You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
> To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/6c1f75cf-3598-40f6-a534-323737342653%40googlegroups.com.
Alon Zakai
May 9, 2020, 1:30:45 PM5/9/20
to emscripte...@googlegroups.com
Asyncify doesn't have a way to "emulate" threads atm (but it would be possible in theory, with work).
However, using Asyncify and calling emscripten_sleep(0) is an easy way to let the browser event loop run, and allow a Worker thread to start up.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CA%2B6sJ-2htO6s-PpZk1AS6TeP9d2_iegzceWEnsryj%3DBDodjZWQ%40mail.gmail.com.
Talha Enes Ayrancı
May 13, 2020, 11:53:57 AM5/13/20
to emscripten-discuss
Thanks for answers. With some changes at C++ side, I solved this problem. Now, I am setting POOL_SIZE to 3 for every case.
9 Mayıs 2020 Cumartesi 20:30:45 UTC+3 tarihinde Alon Zakai yazdı:
9 Mayıs 2020 Cumartesi 20:30:45 UTC+3 tarihinde Alon Zakai yazdı:
>>>> To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsub...@googlegroups.com.
>>>> To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/e9909928-b9ad-43e3-bb6e-c83935305ee7%40googlegroups.com.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsub...@googlegroups.com.
>>> To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CAEX4NpTjA6Bu3BjUB8b_bHtUemniO0XXafOf4K8Kh4urMXom5w%40mail.gmail.com.
>
> --
> You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsub...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/6c1f75cf-3598-40f6-a534-323737342653%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsub...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages