Zstd annoyance for new pre-built libraries

[Henri Beauchamp]

Greetings,

Whenever possible, I reuse LL's pre-compiled libraries for the Windows
and macOS builds of my viewer.

It has not posed the slightest issue so far (i.e. for the past 16 years),
but with the newest libraries builds done on github, the compression
format for the packaged tar files changed from bz2 to zst, and the latter
is not recognized by Python's 'tarfile' module used by the build script
to uncompress the packages...

I even tried with the latest official Python v3.12 Windows build, and its
tarfile module still does not have the zst compression format (only gz,
bz2 and xz):
"tarfile.ReadError: file could not be opened successfully:
- method gz: ReadError('not a gzip file')
- method bz2: ReadError('not a bzip2 file')
- method xz: ReadError('not an lzma file')
- method tar: ReadError('invalid header')
"

Since 'xz' is comparable in performances anyway and is recognized by all
Python v3 releases (without any need to install additional modules),
could you guys pretty please change the pre-built libraries packages
format for *.tar.xz ?

Many thanks in advance !

Regards,

Henri.

[Signal Linden (Bennett Goble)]

Hey Henri,

Using zstd support was added to autobuild v3.5.0 last year. Which version are you using?

https://github.com/secondlife/autobuild/releases/tag/v3.5.0

-signal

--
Archives of earlier incarnations of this list are at https://list-archives.secondlife.com
---
You received this message because you are subscribed to the Google Groups "opensource-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opensource-de...@lists.secondlife.com.
To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/20231031194958.b2e7992b945e6431fd03dd46%40free.fr.

[Henri Beauchamp]

On Tue, 31 Oct 2023 11:52:11 -0700, Signal Linden (Bennett Goble) wrote:

> Hey Henri,
>
> Using zstd support was added to autobuild v3.5.0 last year. Which version
> are you using?

I am _not_ using autobuild. Just a modified version of Snowglobe's Python
build scripts, which are part of the viewer source tree.

The idea is that to build my viewer, you just need the compiler, Python,
and then launch a script/batch file provided together with the sources. Lean
and clean, with as little dependencies as possible...:-P

Regards,

Henri.

[Signal Linden (Bennett Goble)]

I'm not opposed to simple build systems, but there are good reasons to keep up with trends in archive distribution. LL developers on viewer and server work through a decent bit of package churn, zstandard is faster to download and decompress. Arch Linux switched their packages from from xz to zstandard back in 2020, vcpkg supports it, conan is likely getting zstd in an upcoming release.

[Henri Beauchamp]

On Tue, 31 Oct 2023 16:35:49 -0700, Signal Linden (Bennett Goble) wrote:

> I'm not opposed to simple build systems, but there are good reasons to keep
> up with trends in archive distribution. LL developers on viewer and server
> work through a decent bit of package churn

Thing is, "autobuild" relies on Python modules that are not part of the
standard Python distribution, meaning it relies on "pip" or similar
automated Python modules installation utilities which are in themselves
wide open doors for hackers, having already been successfully exploited
in the wild for breaching systems security.

Beside the above (big !) problem, it also incurs an increase on external
dependencies when it comes to build the viewer (which, being an Open Source
software, should be straightforward and easy to build by everyone, non-
programmers users included).

While I totally understand that LL needs such a tool, TPV developers and
users alike should not be required to use it.

This is why I reworked my viewer sources and build system to be extremely
easy to build, with very minimalist requirements (and no additional Python
module to install): a compiler, cmake and the standard (bare) Python,
launch linux-build.sh, macos-build.sh or windows-build.bat, and 3 to 10
minutes later (depending on your CPU), you get a working viewer !

> zstandard is faster to download and decompress.

Come on !

xz is *just* as good as zstd (one will win on some packages and loose on
others) and supported by Python's built-in 'tarfile' module, unlike zst,
which requires "pip install pyzstd".

Want an example with LL's pre-built libraries ? Here you go:

wget https://github.com/secondlife/dullahan/releases/download/v118.0.5993.54/dullahan-1.14.0.202310131404_118.4.1_g3dd6078_chromium-118.0.5993.54-windows64-6b02a60.tar.zst
zstd -d dullahan-*
xz dullahan-*.tar
ls -l
-rw-r--r-- 1 root root 97229376 oct. 13 15:21 dullahan-1.14.0.202310131404_118.4.1_g3dd6078_chromium-118.0.5993.54-windows64-6b02a60.tar.xz
-rw-r--r-- 1 root root 98582663 oct. 13 15:21 dullahan-1.14.0.202310131404_118.4.1_g3dd6078_chromium-118.0.5993.54-windows64-6b02a60.tar.zst

> Arch Linux switched their packages from from xz to zstandard back in 2020,
> vcpkg supports it, conan is likely getting zstd in an upcoming release.

And ?... Does it at all matter for the issue at hand ?...

What does it take for you to switch to xz ?... It would still work just
fine with your 'autobuild' system, it will work for everyone else
without any need for pip modules installation, and it will not even make
any measurable differences on speed or storage...

Please, reconsider.

Regards,

Henri.

[Signal Linden (Bennett Goble)]

I'm sympathetic to the idea of switching to xz if it made no difference. Unfortunately, it does: while the compression ratio between xz and zst is similar the decompression and compression times are not. In a concrete example using ReleaseOS packages it takes 2s to decompress with zst compared to 16s with xz. That's an 8x (14 second) difference on an operation which developers frequently perform multiple times a day. Decompression time is the most important metric as once AUTOBUILD_INSTALLABLE_CACHE is populated with downloaded assets they are repeatedly decompressed into the build directory when engineers switch between branches and re-create their build-* directory.

https://github.com/bennettgoble/compare-autobuild-zst-xz

-signal

[Henri Beauchamp]

On Wed, 1 Nov 2023 10:57:02 -0700, Signal Linden (Bennett Goble) wrote:

> I'm sympathetic to the idea of switching to xz if it made no difference.
> Unfortunately, it does: while the compression ratio between xz and zst is
> similar the decompression and compression times are not. In a concrete
> example using ReleaseOS packages it takes 2s to decompress with zst
> compared to 16s with xz. That's an 8x (14 second) difference on an
> operation which developers frequently perform multiple times a day.

14s, a difference ?... Seriously ? LOL !

How much time does it take to build LL's viewer ?... 5 minutes, at the
minimum (LL's viewer sources are *much* larger than mine) ?...
14s is less than 5% of that. You will not convince me that it is such
vital a gain in time.

I, too, perform compilations many times a day (sometimes a couple dozen
of times), and using the old bzip2 packages for now (and not even with
the lbzip2 parallel compressor, because Python does not know how to use
this), but the decompression time never was an issue in the 16 years I
developed the Cool VL Viewer (the compilation time was, in the past, but
with the new many-cores CPUs, it is no more much of an issue either).

But if you do not want to do it, I obviously cannot force you... :-(

Henri.

[Signal Linden (Bennett Goble)]

I've chatted a bit internally about options. LL could host both zst and xz compressed archives on Github releases. This would allow both build systems to exist in beautiful harmony. (See: https://github.com/secondlife/3p-zlib-ng/releases and imagine the Assets section containing xz packages in addition to zst.)

If you or someone can get us PRs that add xz compressed archives as a secondary artifact of the builds we'd be willing to adopt the process and start publishing both zstd and xz. One of the benefits of moving to a public CI system is that LL's build system is a lot more transparent than it used to be. No more black box codeticket, no more secret Teamcity! It also means that our build tools are open to improvements from outside the company.

Here's the repositories that would need changes:

- https://github.com/secondlife/action-autobuild

- https://github.com/secondlife/action-autobuild-release

Suggested approach:

1. action-autobuild `archive-format` input could be changed to accept a CSV of formats, the default could be changed from "tzst" to "tzst,txz"

2. action-autobuild-release will need to be modified so that it only creates installation instructions (autobuild installables edit...) using the zst artifacts

-Signal

[Henri Beauchamp]

On Wed, 1 Nov 2023 13:52:31 -0700, Signal Linden (Bennett Goble) wrote:

> I've chatted a bit internally about options. LL could host both zst and xz
> compressed archives on Github releases. This would allow both build systems
> to exist in beautiful harmony.

That would be great, indeed. :-)

> If you or someone can get us PRs that add xz compressed archives as a
> secondary artifact of the builds we'd be willing to adopt the process and
> start publishing both zstd and xz.

Err... Not me, I'm afraid. I know strictly nothing about how github
actions work, and I'm already over-swamped with coding (and not just a
SL viewer, which I only do as one of my hobbies) to spend time learning
them...

> One of the benefits of moving to a public CI system is that LL's build
> system is a lot more transparent than it used to be. No more black box
> codeticket, no more secret Teamcity! It also means that our build
> tools are open to improvements from outside the company.

And I'm grateful for this move and openness.

My contributions will however stay limited to the viewer code itself and
possibly patches to libraries it uses, as well as Linux support (I could
backport a large bunch of what I wrote for my viewer and restore LL's
viewer code base compatibility with Linux); so far, I could not contribute
much, because I cannot afford spending time, in excess of porting my code
from the Cool VL Viewer to LL's code base (*), to also build LL's viewer
in order to test and verify the validity of my PRs... But as soon as it
will be possible to get github to build test binaries from PRs, I intend
to contribute again/more.

Regards,

Henri.

(*) They diverged a *lot* in the past 16 years, so everything is not
trivial any more when porting from one code base to the other: this
takes time and care, and needs proper testing as well...

[Argent Stonecutter]

Completely off topic but damn.

On 2023-11-01, at 12:57, Signal Linden (Bennett Goble) <sig...@lindenlab.com> wrote:
> That's an 8x (14 second) difference on an operation which developers frequently perform multiple times a day.

Man, I love living in the future.

My first job we got two builds a day, that's it. And we had to submit changes on coding pads and get edited decks back, we were only allowed to make minor changes ourselves on the one punch at the end of the office that always had a line of programmers waiting to use it. Then we'd turn the deck in and get it back with printout of the run.

I'm not making this up.

[Geir Nøklebye]

We were allotted 3 minutes of free CPU time per semester on the University Univac.

Printout was on one of two centrally placed line printers.

Then we got Macs and it changed everything...

> --
> Archives of earlier incarnations of this list are at https://list-archives.secondlife.com
> ---
> You received this message because you are subscribed to the Google Groups "opensource-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to opensource-de...@lists.secondlife.com.

> To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/F1473CB9-B098-4FC3-80E9-8071BCAA041A%40gmail.com.

[Tonya Souther]

You forgot to say "now get off my lawn".

--
Archives of earlier incarnations of this list are at https://list-archives.secondlife.com
---
You received this message because you are subscribed to the Google Groups "opensource-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to opensource-de...@lists.secondlife.com.

To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/F1473CB9-B098-4FC3-80E9-8071BCAA041A%40gmail.com.

[Dahlia Trimble]

I remember doing lab work on an ASR33 teletype. It was connected to a PDP 11/45 thru which we could submit a virtual "deck" to an IBM 360. This was at least a decade before Macs existed. OK it seems I'm revealing my age now so I'll shut up.

To view this discussion on the web visit https://groups.google.com/a/lists.secondlife.com/d/msgid/opensource-dev/CA%2BBE2c6%3DQjr0CaAmfiB_vS92_TL0k5y28j-xgbCwPNOmH%2B7kng%40mail.gmail.com.