7 Zip Zstandard
Kanisha Marchant
A few days ago I started having problems in my Windows 10 machine with installing packages with conda. It was giving me an error message UserWarning: zstandard could not be imported. Running without .conda support.
I did some Googling and tried to reinstall Anaconda. I could not do anaconda--clean because it would not install for me because of the conda issues. But, I used the uninstall program and then deleted the .conda, etc. folders -- everything I could find. I re-installed. Same issue. I then uninstalled again and reinstalled, this time choosing the "add to path" option. It seemed to work while I loaded a few packages, but when I created a new environment and activated it, Jupyter notebook could not recognize the packages. Then it started giving me the error messages in conda again:
I have used Anaconda for a half dozen years and this is the first major problem I've had with it. I've spent about 8 hours so far trying to figure this out and am going a bit nuts so I would really appreciate any suggestions. I have looked at this question and tried the second solution but it didn't work. There does not appear to be a lot on Google so this might be rare.
I think this is a recent bug in conda. I am facing this problem myself. I finished reinstalling and it worked but few updates in and it all breaks again with the same error. The error seems to arise from the libarchive and libgcc_mutex libraries. What I noticed was that, each time a library asks conda to update libgcc_mutex and switch channels to conda-forge, it causes some kind of dependency conflict and then the environment just breaks. Using force-reinstalled broke conda and it stopped working in all for me.
Now, the trouble is that zstandard is used to install packages, in particular to extract .conda files. If you run conda search zstandard you might find that 0.19 is showed to be the latest version. So this is a bit of a catch-22 situation, which you can work around by installing a newer version of zstandard (0.21) directly via pip.
I think I first heard about the Zstandardcompression algorithm at a Mercurial developer sprint in 2015.At one end of a large table a few people were uttering expletives outof sheer excitement. At developer gatherings, that's the universal signalfor something is awesome. Long story short, a Facebook engineer shareda link to theRealTime Data Compression blogoperated by Yann Collet (then known as the author of LZ4 - a compressionalgorithm known for its insane speeds) and people were completelynerding out over the excellent articles and the data within showing thebeginnings of a new general purpose lossless compression algorithm namedZstandard. It promised better-than-deflate/zlib compression ratios andperformance on both compression and decompression. This being a Mercurialmeeting, many of us were intrigued because zlib is used by Mercurialfor various functionality (including on-disk storage and compression overthe wire protocol) and zlib operations frequently appear as performance hotspots.
Before I continue, if you are interested in low-level performance andsoftware optimization, I highly recommend perusing theRealTime Data Compression blog.There are some absolute nuggets of info in there.
Anyway, over the months, the news about Zstandard (zstd) kept gettingbetter and more promising. As the 1.0 release neared, the Facebookengineers I interact with (Yann Collet - Zstandard's author - is nowemployed by Facebook) were absolutely ecstatic about Zstandard and itspotential. I was toying around with pre-release versions and wasabsolutely blown away by the performance and features. I believedthe hype.
Zstandard 1.0 wasreleased on August 31, 2016.A few days later, I started thepython-zstandard project toprovide a fully-featured and Pythonic interface to the underlying zstd CAPI while not sacrificing safety or performance. The ulterior motive wasto leverage those bindings in Mercurial so Zstandard could be a first classcitizen in Mercurial, possibly replacing zlib as the default compressionalgorithm for all operations.
Fast forward six months and I've achieved many of those goals.python-zstandard has a nearly complete interface to the zstd C API.It even exposes some primitives not in the C API, such as batchcompression operations that leverage multiple threads and use minimalmemory allocations to facilitate insanely fast execution. (Expect adedicated post on python-zstandard from me soon.)
The main objective of lossless compression is to spend one resource(CPU) so that you may reduce another (I/O). This trade-off is usuallymade because data - either at rest in storage or in motion over anetwork or even through a machine via software and memory - is alimiting factor for performance. So if compression is needed for youruse case to mitigate I/O being the limiting resource and you can swapin a different compression algorithm that magically reduces both CPUand I/O requirements, that's pretty exciting. At scale, betterand more efficient compression can translate to substantial costsavings in infrastructure. It can also lead to improved applicationperformance, translating to better end-user engagement, sales,productivity, etc. This is why companies like Facebook (Zstandard),Google (brotli, snappy, zopfli), andPied Piper(middle-out) invest in compression.
Today, the most widely used compression algorithm in the world islikely DEFLATE. And, softwaremost often interacts with DEFLATE via what is likely the most widelyused software library in the world, zlib.
Being at least 27 years old, DEFLATE is getting a bit long in thetooth. Computers are completely different today than they were in 1990.The Pentium microprocessor debuted in 1993. If memory serves (punintended), it used PC66 DRAM, which had a transfer rate of 533 MB/s.For comparison, a modern NVMe M.2 SSD (like the Samsung 960 PRO)can read at 3000+ MB/s and write at 2000+ MB/s. In other words,persistent storage today is faster than the RAM from the era whenDEFLATE was invented. And of course CPU and network speeds haveincreased as well. We also have completely different instructionsets on CPUs for well-designed algorithms and software to takeadvantage of. What I'm trying to say is the market is ripe forDEFLATE and zlib to be dethroned by algorithms and software thattake into account the realities of modern computers.
Zstandard initially piqued my attention by promising better-than-zlibcompression and performance in both the compression and decompressiondirections. That's impressive. But it isn't unique. Brotli achievesthe same, for example. But what kept my attention was Zstandard's richfeature set, tuning abilities, and therefore versatility.
Before I do, I need to throw in an obligatory disclaimer about dataand numbers that I use. Benchmarking is hard. Benchmarks should notbe trusted. There are so many variables that can influence performanceand benchmarks. (A recent example that surprised me is theCPU frequency/power ramping properties of Xeon versus non-Xeon Intel CPUs.tl;dr a Xeon won't hit max CPU frequency if only a core or twois busy, meaning that any single or low-threaded benchmark islikely misleading on Xeons unless you change power settings tomitigate its conservative power ramping defaults. And if you changepower settings, does that reflect real-life usage?)
All performance data was obtained on an i7-6700K running Ubuntu 16.10(Linux 4.8.0) with a mostly stock config. Benchmarks were performed inmemory to mitigate storage I/O or filesystem interference. Memory usedis DDR4-2133 with a cycle time of 35 clocks.
While I'm pretty positive about Zstandard, it isn't perfect. There arecorpora for which Zstandard performs worse than other algorithms, evenones I compare it directly to in this post. So, your mileage may vary.Please enlighten me with your counterexamples by leaving a comment.
Compression algorithms typically contain parameters to control howmuch work to do. You can choose to spend more CPU to (hopefully)achieve better compression or you can spend less CPU to sacrificecompression. (OK, fine, there are other factors like memory usage atplay too. I'm simplifying.) This is commonly exposed toend-users as a compression level. (In reality there are oftenmultiple parameters that can be tuned. But I'll just use levelas a stand-in to represent the concept.)
But even with adjustable compression levels, the performance of manycompression algorithms and libraries tend to fall within a relativelynarrow window. In other words, many compression algorithms focus onniche markets. For example, LZ4 is super fast but doesn't yield greatcompression ratios. LZMA yields terrific compression ratios but isextremely slow.
This chart plots the logarithmic compression speed in megabytes persecond against achieved compression ratio. The further right a datapoint is, the better the compression and the smaller the output.The higher up a point is, the faster compression is.
LZ4 is highly vertical, which means its compression ratios arelimited in variance but it is extremely flexible in speed. So forthis data, you might as well stick to a lower compression levelbecause higher values don't buy you much.
Bzip2 is the opposite: a horizontal line. That means it is consistentlythe same speed while yielding different compression ratios. In otherwords, you might as well crank bzip2 up to maximum compression becauseit doesn't have a significant adverse impact on speed.
This small window of flexibility means that you often have to choosea compression algorithm based on the speed versus size trade-off you arewilling to make at that time. That choice often gets baked intosoftware. And as time passes and your software or data gains popularity,changing the software to swap in or support a new compression algorithmbecomes harder because of the cost and disruption it will cause. That'stechnical debt.
What we really want is a single compression algorithm that occupieslots of space in both dimensions of our chart - a curve that hashigh variance in both compression speed and ratio. Such an algorithmwould allow you to make an easy decision choosing a compressionalgorithm without locking you into a narrow behavior profile. It wouldallow you make a completely different size versus speed trade-off inthe future by only adjusting a config knob or two in your application -no swapping of compression algorithms needed!