The latest release of the LZ compression algorithm has been released with support for multithreading, enabling a dramatic improvement in compression speeds of up to eight times.

LZ compression is an algorithm based on LZ dictionary compression. It is used to compress many popular file formats, and is used in GIF, Deflate, Zip, PNG. It was important enough to be named as an IEEE Milestone.

LZ compression is one of a class of dictionary compression methods invented in 1978 by Abraham Lempel and Jacob Ziv. Dictionary compression methods are lossless and work by building up a dictionary of short sequences that can then be used to code the file in question, removing duplicates of the sequences and replacing them with a lookup to the dictionary. The LZW algorithm is also part of the UNIX file compression utility.

The new release, version 1.10 of LZ4, has been significantly improved with the addition of support for multithreading to make use of multi-core processors. This accelerates compression so will improve LZ4's use for high-throughput jobs.

While LZ4 has historically been recognized for its high-speed compression, the demand for even faster throughput has grown, particularly with the advent of nvme (nonvolatile memory express) storage technologies that allow for multi-GB/s throughput.

The developers provided the following benchmark table of performance improvements for compression:

They point out that multithreading is less critical for decompression, as modern nvme drives can still be saturated with a single decompression thread. However, the new version still enhances performance by overlapping I/O operations with decompression processes.

Tested on a x64 Linux platform, decompressing a 5 GB text file locally takes 5 seconds with v1.9.4; this is reduced to 3 seconds in v1.10.0, corresponding to > +60% performance improvement.

Alongside the multithreading support, the new release also adds full support for dictionary compression and decompression. This was experimental in the previous release. The developers say the full support means developers can use dictionary compression in applications that need it, such as for small data, where dictionary initialization can become a bottleneck. The dictionary state can be used by multiple threads concurrently.

This release also adds a new mid-range Level 2 compression that fills the substantial gap between the standard "Fast Level 1" and the more intensive "High Compression Level 3." The developers say it provides a balanced option, optimizing performance and compression, so is ideal for applications requiring better compression than level 1, without the speed trade-offs associated with high compression level 3.

LZ4 1.10 is available now on GitHub.

More Information

LZ4 On GitHub

LZ4 Website

Related Articles

LZ Compression And The One-Bit Catastrophe

New Hutter Prize Milestone For Lossless Compression

Commemorating the Co-Creator of the L-Z Algorithm

Commemorating Jacob Ziv

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info