Low installation performance

[Mike Zexter]

I see that only 3-4 of 48 available CPU threads are fully loaded and the unpacking process seems to be much slower than it can be on my 2 * Xeons 24/48. NVMe SSD uses only 2-4% of its performance.

My colleagues with different PC configurations noticed the same problem.

Tested on 3 configurations with the Compression=lzma2/ultra64 flag and ~ 31.5 GB of files:

1) (Mine) 2 * Xeon E5-2678v3 + 64 GB DDR4 RAM (8-channel) + Samsung EVO 970 512 GB.

2) AMD FX 6300 + 16 GB DDR4 RAM + Samsung EVO 970+ 256 GB.

3) (Laptop) AMD Ryzen 5800H + 32 GB DDR4 RAM + AMD NVMe SSD 1 TB.

Please help me to understand how to bypass this bottleneck and say to Inno to use every available CPU thread and unpack my files faster?

I also have posted my questions on SO: https://stackoverflow.com/questions/72894497/inno-setup-v6-2-doesnt-use-the-whole-cpu-performance

Thank you.

[Mike Zexter]

Oh! Forgot to say that our 3 computers have Windows 10/11 x64 of course.

Inno Setup v6.2.1 (the latest stable one).

четверг, 7 июля 2022 г. в 18:41:50 UTC+3, Mike Zexter:

[Christian Blackburn]

Hi Benjamin,

Are you sure your board isn't quad channel?  I could be wrong on that, but my X99 board has 8 slots, but is only quad channel.  Also, your CPU's would have be v4 not v3 to have 12 physical cores each.  

Cheers,

Christian

[Mike Zexter]

Well I have double CPUs motherboard - each Xeon CPU socket supports 4-channel memory, AIDA64 told me that it's 8 channel because of double physical CPUs.

Anyway it's not the main question)

пятница, 8 июля 2022 г. в 00:50:10 UTC+3, cbl...@wgu.edu:

[Eivind Bakkestuen]

For a comparison, how many of your cores activate, and how much SDD % usage do you get, if you use the 7-zip tool directly, and unpack a large file that you previously created?

--
You received this message because you are subscribed to the Google Groups "innosetup" group.
To unsubscribe from this group and stop receiving emails from it, send an email to innosetup+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/innosetup/5a9ef748-0374-4e25-836e-c1b627495d9en%40googlegroups.com.

[Mike Zexter]

Eivind, hello.

I use these parameters in my [Setup]:

LZMANumBlockThreads=6

LZMADictionarySize=1048576

LZMANumFastBytes=273

CompressionThreads=auto // tried 48, 32, 24 etc.. and it doesn't increase the packing speed

InternalCompressLevel=max

SolidCompression=no // 'yes' causes out of memory error due to overloading 16 GB of RAM

Compression=lzma2/ultra64

And here how it goes due to unpacking 31 GB of files:

I am sorry for the installation windows blur, our stakeholder asked me not to publish our product until its release.

As we can see AMD Radeon SSD is busy only for 2-4% and CPU is loaded only by 2 its threads. Core2 took the biggest load and the Core4 the last one. RAM is busy only by 2,7 of 29.9 free GB.

My question is how to say to Inno to use all of the CPU threads due the unpacking process?

Me and my colleagues have much more hardware  performance that Inno is using now.

пятница, 8 июля 2022 г. в 02:00:38 UTC+3, eivind.b...@gmail.com:

[Laurent Combémorel]

Maybe need a little hacking of Windows ?

https://codeinsecurity.wordpress.com/2022/04/07/cpu-socket-and-core-count-limits-in-windows-10-and-how-to-remove-them/

[Eivind Bakkestuen]

I'm not asking for specifics on your installer. I am asking how the 7-zip utility does in comparison, because that is basically what InnoSetup is using in the background.

If the 7-zip utility performs similarly, then complain to the 7-zip developers. If not, then let us know, and perhaps there is some tweaking that can be done in IS.

To view this discussion on the web visit https://groups.google.com/d/msgid/innosetup/a17801ed-609e-4a60-ae6b-2586619b40f8n%40googlegroups.com.

[Martijn Laan]

Actually, it uses the LZMA SDK. It's from the same author as 7-zip but it not the same thing as 7-zip. For more info see https://www.7-zip.org/sdk.html which also lists decompression speeds.

Greetings,
Martijn

[Eivind Bakkestuen]

Martijn,

Test results on the page you link to displays results for up to 144 cores. :) Is there anything that can be done in Inno Setup regarding use of more cores, or is it outside the control that the SDK provides?

--
You received this message because you are subscribed to the Google Groups "innosetup" group.
To unsubscribe from this group and stop receiving emails from it, send an email to innosetup+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/innosetup/88FF9E6E-114D-4019-BBEC-5FD08FC0FE20%40gmail.com.

[Alex Born]

The current version of LZMA SDK is 22.0. And Inno Setup uses LZMA SDK version 9.XX from 2010. Perhaps that is why Inno Setup does not support modern LZMA features.

суббота, 9 июля 2022 г. в 08:57:35 UTC+2, eivind.b...@gmail.com:

[Martijn Laan]

Op 9-7-2022 om 08:57 schreef Eivind Bakkestuen:

Is there anything that can be done in Inno Setup regarding use of more cores, or is it outside the control that the SDK provides?

I don't know, sorry.

Greetings,
Martijn

[Wilenty]

Hello Mike (Benjamin?: https://groups.google.com/g/innosetup/c/WhRgw4JkDjE/m/Bx4iyqQ_AAAJ),
I updated for you the the ZLMA/LZMA2 to the latest version (v22) from 7-ZIP SDK for the InnoSetup.

Before please backup these files of current InnoSetup installation:
islzma.dll
islzma32.exe
islzma64.exe

But, I can't get the 7-ZIP LZMA/LZMA2 working in InnoSetup itself (for now).

Advantages of use 7-ZIP LZMA/ZMA2 SDK v22:
1) Faster compression (in test about 1minute of 900MB of two ISO files)
2) Less memory used for decompression (about 500MB in the new one, compare to the about 1100-1300MB in the old one, in the highest compression methods)

Disadvantages to use 7-ZIP LZMA/ZMA2 SDK v22:
1) a bit bigger result (installer) file

Download link:
https://www35.zippyshare.com/v/Sae92GR6/file.html

P.S.
Above files was compiled using VS2015CE.

Greetings,
Wilenty

[Mike Zexter]

Wilently, thank you very much. Could you explain please why the new islzma.dll file was detected as a VHO:Trojan.Win32.TropicTrooper.gen instead of stock one?

Old library: https://www.virustotal.com/gui/file/0b2e19e473a47e10578b05a2f3b43ad96603f3ee1e397c06a280c3b7458a76e2

Your new library: https://www.virustotal.com/gui/file/9a704b833bda243bf2181ce234b1fa5add16157a2530024bef2934abc39cfcf0

If it's possible could you give me please the sources / recipes to check the code?

воскресенье, 10 июля 2022 г. в 03:13:12 UTC+3, Wilenty:

[Mike Zexter]

If the virus signature was like Adware or Ransomware, I could check it in the virtual machine.

But Tropic Trooper is very dangerous - https://thehackernews.com/2021/12/tropic-trooper-cyber-espionage-hackers.html#:~:text=Earth%20Centaur%2C%20also%20known%20by,dating%20all%20the%20way%20back because it can get the access to the OS NT core with full privilegies and collect encrypted information which is very diffucult to recognize by its behavior.

понедельник, 11 июля 2022 г. в 14:28:23 UTC+3, Mike Zexter:

[Eivind Bakkestuen]

Even if the virus detection would turn out to be a false alarm (and *I'm* certainly not going to download to try to verify), I think we can all agree that an official update to a newer SDK would be a good idea.

--

You received this message because you are subscribed to the Google Groups "innosetup" group.
To unsubscribe from this group and stop receiving emails from it, send an email to innosetup+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/innosetup/e6727b52-417d-4559-869a-bce3a49d3eadn%40googlegroups.com.

[Martijn Laan]

Op 10-7-2022 om 02:13 schreef Wilenty:

islzma.dll
islzma32.exe
islzma64.exe

These files are the compression/encoder part and not decompression/decoder.

The decoder is directly linked via .obj files and does not use .dll's. Its source is in https://github.com/jrsoftware/issrc/tree/main/Projects/Lzma2/Decoder.

There's another decoder in https://github.com/jrsoftware/issrc/tree/main/Projects/LzmaDecode but this is used for something else and should be ignored.

Greetings,
Martijn

[Mike Zexter]

Eivind,

Sure, I didn't mean it's a bad solution. The problem exists in VirusTotal results and specific Trojan detection.

If you have any information about where did Wilently get the sources to build .dll's please provide the link.

понедельник, 11 июля 2022 г. в 17:08:49 UTC+3, Martijn Laan:

[Wilenty]

Mike wrote:
"If you have any information about where did Wilently get the sources to build .dll's please provide the link."

Wilenty wrote:
"I updated for you the the ZLMA/LZMA2 to the latest version (v22) from 7-ZIP SDK for the InnoSetup."

Sorry, but for first my nick-name is not "Wilently".
I don't want to be rude or something, but what do you think from where we all got the 7-ZIP LZMA/LZMA2 SDK?
I made a repack of whole 7-ZIP and people uses it, so, are you thinking that I use self-sabotage and I included a Virus in few files for InnoSetup to destroy my reputation with one move?
https://github.com/Wilenty/7-Zip-X64-X86-with-Extracting-NSIS-script
I've spend half of the day trying to compile this 12 years old source, and next half of the day to checks if everything works fine.

Eivind wrote:
"...(and *I'm* certainly not going to download to try to verify), I think we can all agree that an official update to a newer SDK would be a good idea."
So, wait for the maintainers when they will take an action to update it.

Martijn wrote:
"These files are the compression/encoder part and not decompression/decoder."

Wilenty wrote:
"But, I can't get the 7-ZIP LZMA/LZMA2 working in InnoSetup itself (for now)."

Yes, I updated the compressor only. I made it clear that at that time I did not manage to compile the de-compressor for InnoSetup itself. But, the new one compression methods changes the way of decompression, so...

I thought the updated LZMA/LZMA2 can be useful. I tried to help, but I see that most people here is not interested about it and/or don't want tor read with understanding what I wrote - it's my last post in this thread.

Best regards,
Wilenty

[Christian Blackburn]

Hi Guys,

My experience with VirusTotal has been any time there are only 1-3 tools that flag something as malicious you most likely have an issue with that program's signatures and not with the file itself.  If however 5+ programs list it as malicious it probably is.  I do completely agree that we need to verify the authenticity of the files and upgrade Inno Setup.  There are long-standing CPU-based instructions that will accelerate compression and decompression in addition to better multi-threading.  Inno Setup is the product of the total experience and if the installer size increases 150-300KB, but unzips in 1/3 the amount of time that's a trade off I think most of us are more than willing to make.  The internet gets faster every day making a slight increase in file size irrelevant. 

Thanks,
Christian Blackburn

--
You received this message because you are subscribed to a topic in the Google Groups "innosetup" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/innosetup/WhRgw4JkDjE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to innosetup+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/innosetup/703641dc-8059-43ff-8658-31bb8690179bn%40googlegroups.com.

[Mike Zexter]

Wilenty,

Sorry for the mistake. I never think to blame you for the virus injections, just need to know the sources to check the code and try to understand how can multi-threading work for any PC.

Do you have the recype how to make exactly islzma.dll's from the https://github.com/Wilenty/7-Zip-X64-X86-with-Extracting-NSIS-script sources, not just base lzma libs?

As for me, I'm very interested but sometimes didn't get you because of not enough development experience.

вторник, 12 июля 2022 г. в 00:40:17 UTC+3, cbl...@wgu.edu:

[Mike Zexter]

Guys, how do you think can it be correct to create Discord server for Inno Setup to chat more fluently and rapidly within the categorized channels?

Like:

1. #innosetup-updates
2. #faq
3. #installation-scripting
4. #installation-styles
5. #custom-libraries
6. #dev-problems
7. #performance-problems
8. #discussions
9. #offtopic-channel

...and voice channels to communicate if there's something interesting to discuss.

And who can approve this idea like the stakeholder of the project?

вторник, 12 июля 2022 г. в 14:15:40 UTC+3, Mike Zexter: