Doom 3 port / Asyncify test results
249 views
Skip to first unread message
Gabriel CV
Jul 22, 2019, 2:40:13 AM7/22/19
to emscripten-discuss
Hi!
I did some tests with the new Upstream/Asyncify feature (ie. "Bysyncify") on the Doom 3 port.
I am using Chrome 75/Ubuntu 18.04/nVidia binary drivers, and used the "timedemo demo1" command to measure the FPS (not available on the D3 demo though, too bad. I had to do this with the full version of the game).
The good news: UPSTREAM/ASYNCIFY is working well! And easier to use than Emterpreter. However there is a catch on the final wasm size. Here are the raw results:
TARGET FPS SIZE (MB)
O2/FASTCOMP/EMTERPRETER 50 4,55 MB (for reference. NB: I am using whitelisting feature on EMTERPRETER)
O2/UPSTREAM/ASYNCIFY 50 6,81 MB
O2/UPSTREAM (no Asyncify) 50 3,90 MB
O3/UPSTREAM/ASYNCIFY 51 6,96 MB
Os/UPSTREAM/ASYNCIFY 41 5,56 MB
Oz/UPSTREAM/ASYNCIFY 40 5,56 MB
What to read from these numbers:
- Performance
-- FASTCOMP/EMTERPRETER and UPSTREAM/ASYNCIFY have a similar performance profile: with O2 optimization, there 50 FPS on average for both targets.
-- ASYNCIFY have no impact on performance: with O2 optimization, there is 50 FPS on average with and without for both targets (NB: on the D3 port, I really tried to 'yield' as few as possible)
-- There is however an important gap between Os/Oz and O2/O3: using Os lead to a 20% performance hit comparted to O2 (50 FPS with O2/O3 => 40 FPS with Os/Oz)
-- O3 compared to O2 does not bring significant performance improvement
-- Same thing for Oz compared to Os: both are almost the same
- Binary size
-- UPSTREAM/ASYNCIFY do have a big impact on final binary size: this roughly a +50% increase (from 4,55 MB with O2/FASTCOMP/EMTERPRETER => 6,81 MB with O2/UPSTREAM/ASYNCIFY)
-- It is really the ASYNCIFY that cause this binary size increase, as without ASYNCIFY, UPSTREAM produce a binary that is 15% smaller than FASTCOMP (from 4,55 MB with FASTCOMP/EMTERPRETER => 3,90 MB with UPSTREAM)
-- Using Os compared to O2 brings a binary size improvement (from 6,81 MB with O2 => 5,56 MB with Os), but this does not match with FASTCOMP (4,55 MB)
-- Oz compared to Os does not bring significant binary size improvement
So, all in all, my observation is that ASYNCIFY works well, but the binary size increase is not negligible (+50%).
Using Os/Oz instead of O2/O3 allow to reduce that overhead to some extent, but it is at the expense of a 20% performance hit (at least on the D3 port), and not on par with the FASTCOMP binary size.
As it appears it is really the Asyncify transformation that brings the binary size increase, the whitelisting feature could really bring the best of both world:
- By default (that is, without whitelisting):
- Ease of use of ASYNCIFY compared to EMTERPRETER (this works *by default*, without having to do some extra work)
- No performance impact of using ASYNCIFY (at least, when using yield/sleep carefully)
- Cons: +50% binary size
- With whitelisting:
- The binary size issue could be mitigated a lot, as UPSTREAM give smaller binary size than FASTCOMP (-15% on D3)
- Cons: obviously, some work to do with whitelisting, but this is the same as with EMTERPRETER
I did some tests with the new Upstream/Asyncify feature (ie. "Bysyncify") on the Doom 3 port.
I am using Chrome 75/Ubuntu 18.04/nVidia binary drivers, and used the "timedemo demo1" command to measure the FPS (not available on the D3 demo though, too bad. I had to do this with the full version of the game).
The good news: UPSTREAM/ASYNCIFY is working well! And easier to use than Emterpreter. However there is a catch on the final wasm size. Here are the raw results:
TARGET FPS SIZE (MB)
O2/FASTCOMP/EMTERPRETER 50 4,55 MB (for reference. NB: I am using whitelisting feature on EMTERPRETER)
O2/UPSTREAM/ASYNCIFY 50 6,81 MB
O2/UPSTREAM (no Asyncify) 50 3,90 MB
O3/UPSTREAM/ASYNCIFY 51 6,96 MB
Os/UPSTREAM/ASYNCIFY 41 5,56 MB
Oz/UPSTREAM/ASYNCIFY 40 5,56 MB
What to read from these numbers:
- Performance
-- FASTCOMP/EMTERPRETER and UPSTREAM/ASYNCIFY have a similar performance profile: with O2 optimization, there 50 FPS on average for both targets.
-- ASYNCIFY have no impact on performance: with O2 optimization, there is 50 FPS on average with and without for both targets (NB: on the D3 port, I really tried to 'yield' as few as possible)
-- There is however an important gap between Os/Oz and O2/O3: using Os lead to a 20% performance hit comparted to O2 (50 FPS with O2/O3 => 40 FPS with Os/Oz)
-- O3 compared to O2 does not bring significant performance improvement
-- Same thing for Oz compared to Os: both are almost the same
- Binary size
-- UPSTREAM/ASYNCIFY do have a big impact on final binary size: this roughly a +50% increase (from 4,55 MB with O2/FASTCOMP/EMTERPRETER => 6,81 MB with O2/UPSTREAM/ASYNCIFY)
-- It is really the ASYNCIFY that cause this binary size increase, as without ASYNCIFY, UPSTREAM produce a binary that is 15% smaller than FASTCOMP (from 4,55 MB with FASTCOMP/EMTERPRETER => 3,90 MB with UPSTREAM)
-- Using Os compared to O2 brings a binary size improvement (from 6,81 MB with O2 => 5,56 MB with Os), but this does not match with FASTCOMP (4,55 MB)
-- Oz compared to Os does not bring significant binary size improvement
So, all in all, my observation is that ASYNCIFY works well, but the binary size increase is not negligible (+50%).
Using Os/Oz instead of O2/O3 allow to reduce that overhead to some extent, but it is at the expense of a 20% performance hit (at least on the D3 port), and not on par with the FASTCOMP binary size.
As it appears it is really the Asyncify transformation that brings the binary size increase, the whitelisting feature could really bring the best of both world:
- By default (that is, without whitelisting):
- Ease of use of ASYNCIFY compared to EMTERPRETER (this works *by default*, without having to do some extra work)
- No performance impact of using ASYNCIFY (at least, when using yield/sleep carefully)
- Cons: +50% binary size
- With whitelisting:
- The binary size issue could be mitigated a lot, as UPSTREAM give smaller binary size than FASTCOMP (-15% on D3)
- Cons: obviously, some work to do with whitelisting, but this is the same as with EMTERPRETER
Here it is!
Floh
Jul 22, 2019, 7:36:54 AM7/22/19
to emscripten-discuss
Many thanks for the detailed breakdown :)
Is this only using asyncify for an 'infinite game loop' (instead of a frame callback), or is this also used for other synchronous calls like file- and network-I/O? I'm trying to understand the reason behind the 50% size increase. As far as I understand Alon's recent blog post on the topic, there's a control-flow analysis happening, so only functions along call-stacks which need 'asyncification' would need to be instrumented, but that might just be my overly optimistic interpretation ;) (so for instance, if only the main loop would need to be asyncified, the size increase should be very small, but if there are synchronous IO calls all over the place, much more code would need to be instrumented, adding to the code size).
Cheers!
-Floh.
Alon Zakai
Jul 22, 2019, 12:45:22 PM7/22/19
to emscripte...@googlegroups.com
Floh: the analysis works well in many cases, but the main problem is indirect calls - in a big enough codebase with enough of those, any indirect call looks like it can lead to something that sleeps :( We may indeed need a whitelist approach, some thinking is happening here: https://github.com/WebAssembly/binaryen/issues/2218
--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/c9d94058-7dc6-4c3f-9d56-59edbde20955%40googlegroups.com.
Alon Zakai
Jul 22, 2019, 12:50:35 PM7/22/19
to emscripte...@googlegroups.com
Gabriel: very interesting, thanks! Great to see that this works out of the box, and that -O2/-O3 run at full speed.
Probably the Os/Oz differences are due to inlining. Binaryen is very cautious to not increase code size when the flags say to focus on code size, see https://github.com/WebAssembly/binaryen/issues/2144 So I am guessing -O2/-O3 inlining something important there at link time, at the cost of increased code size.
I'll be doing some more optimizations on Asyncify, which will hopefully help with the 50% code size increase. But without a whitelist the benefits may be limited, so we may need do that eventually.
- Alon
--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/167d5725-0a84-4332-90c0-e6be6296dc78%40googlegroups.com.
Floh
Jul 22, 2019, 12:58:23 PM7/22/19
to emscripten-discuss
I see, makes sense! Pretty much the same problem as virtual methods / jump tables which disable dead-code-elimination I guess.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsub...@googlegroups.com.
Gabriel Cuvillier
Jul 23, 2019, 1:35:06 AM7/23/19
to emscripte...@googlegroups.com
@Floh: no, Asyncify is not used to simulate 'infinite game loop' (I am using emscripten_set_main_loop for this), except for one or two very specific cases (there is some 'modal' dialog boxes in D3 UI, and this is implemented as a 'sub-infinite loop' in the 'main infinite loop', so I use Asyncify in that case). Asyncify is mostly used to allow the screen to be 'updated' during long operations such as loading a level (progress bar display). It is also used to wait for a few network I/O operations such as fetching the game data while in the main game menu.
I think the issue is really with the indirect calls: at top
level, D3 mostly have virtual methods. The code base is very
orthodox C++ with classes/abstraction/virtual methods. So, due to
the way analysis works, everything is probably deduced to be
Async! I can't see any other way than Whitelisting in that case.
@Alon: So I have a question: As I already have the list of functions/methods that could be possibly Async thanks to previous work done for the EMTERPRETER whitelist, can I use that emterpreter whitelist 'as is' with the existing ASYNCIFY_WHITELIST option + IGNORE_INDIRECT calls ? Does Upstream change the mangled C++ names compared to Fastcomp ?
Side (silly) question: is there any difference between emscripten_sleep(1) and emscripten_sleep(0) ?
Gabriel
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/7de30e59-da9c-4ad6-af7d-8e23699d5d6a%40googlegroups.com.
Alon Zakai
Jul 23, 2019, 2:33:09 PM7/23/19
to emscripte...@googlegroups.com
Currently Asyncify doesn't have a whitelist. I had hoped it might not be necessary, but as more benchmarks come in it seems like it is. Discussion is in
One of the problems with adding such a whitelist is the name mangling actually - upstream will by default mangle names to a "human-readable" form, so instead of _ZN3 etc. you'll see things like "()" etc. So a previous whitelist wouldn't just work (except for C methods).
To confirm, you were using the emterpretify whitelist, and not the blacklist? (I'd prefer not to add both.)
emscripten_sleep(1) and (0) would probably do the same thing - browsers limit timeout precision, so a few ms either way is going to end up happening at the same time.
- Alon
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/c9d94058-7dc6-4c3f-9d56-59edbde20955%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/7de30e59-da9c-4ad6-af7d-8e23699d5d6a%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/f2b19ecc-90f7-f511-5f9f-bbd03b3aea24%40gmail.com.
Gabriel Cuvillier
Jul 23, 2019, 4:40:34 PM7/23/19
to emscripte...@googlegroups.com
Yes, I am only using the emterpretify whitelist in the D3 case. But it look like from the opened binaryen issue that other use cases such as Julia would prefer a blacklist instead. Oops :)
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CAEX4NpRpwJicy9UYScZgahPmXNTfH0wrKiJFm_2W0FYbL6c09Q%40mail.gmail.com.
Gabriel CV
Aug 11, 2019, 1:48:12 PM8/11/19
to emscripten-discuss
With the ASYNCIFY_WHITELIST, I now have:
O3/UPSTREAM/ASYNCIFY+WHITELIST 55 FPS 4,3 MB
So now, compared to O2/FASTCOMP/EMTERPRETER version, the binary size is 10% smaller while overall performance is 10% better (and very close to the 60 FPS cap of the engine). This is indeed very good!
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/c9d94058-7dc6-4c3f-9d56-59edbde20955%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/7de30e59-da9c-4ad6-af7d-8e23699d5d6a%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/f2b19ecc-90f7-f511-5f9f-bbd03b3aea24%40gmail.com.
--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsub...@googlegroups.com.
Alon Zakai
Aug 11, 2019, 5:50:21 PM8/11/19
to emscripte...@googlegroups.com
Nice!
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/c9d94058-7dc6-4c3f-9d56-59edbde20955%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/7de30e59-da9c-4ad6-af7d-8e23699d5d6a%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/f2b19ecc-90f7-f511-5f9f-bbd03b3aea24%40gmail.com.
--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CAEX4NpRpwJicy9UYScZgahPmXNTfH0wrKiJFm_2W0FYbL6c09Q%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/a2245424-7c7a-4641-8ff1-6ea442c62800%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages