Facebook has released two technologies as open source at its @Scale conference in San Jose. The Zstandard compression algorithm and the MyRocks storage engine are both being made available for open source use. 

The Zstandard compression is a lossless compression technology that Facebook believes can replace the Deflate compression algorithm used by most existing software. This is the technique used by products such as Zip, gzip, and zlib, and it has remained a market leader because rival newer technologies have either offered better compression or faster compression, but rarely both.

Zstandard 1.0 is a new compression algorithm that combines techniques such as Finite State Entropy with fast performance that is optimized for modern CPUs. Writing about the release on the Facebook Code blog, Yann Collet and Chip Turner of Facebook compared Zstandard to zlib, lz4, and xz, saying:

"Each of these algorithms offers different trade-offs: lz4 aims for speed, xz aims for higher compression ratios, and zlib aims for a good balance of speed and size."

They then showed a comparison of Zstandard compared to those algorithms per lzbench, an in-memory benchmark, in which was substantially faster than xz and zlib, and compressed as well as zlib. The developers say it is also extremely scalable.

Facebook also announced that MyRocks is being made available as open source. Facebook uses MySQL and the InnoDB storage engine for data management, but the developers at Facebook say that InnoDB has inefficiencies on space and write amplification when used with flash storage.

To alleviate this, Facebook developed RocksDB, an embeddable, persistent key-value store for fast storage that is better at using space than InnoDB. However, RocksDB has no support for replication and lacks a SQL layer, so the developers at Facebook have built MyRocks, a new open source project that integrates RocksDB as a new MySQL storage engine. This enables the use of RocksDB as backend storage while supporting all the features in MySQL.

 The developers are still in the process of migrating to MyRocks on the user database (UDB) tier, but say that:

"after deploying MyRocks to this database tier in one of our data center regions, we were able to use 50 percent less storage for the same amount of data compared with compressed InnoDB. Ultimately this will allow us to use half as many UDB servers." 

Both announcements were made at Facebook’s @Scale conference in San Jose. This is aimed at engineers working on large-scale engineering projects in the hope of increasing the use of open-source technology.

More Information

Facebook Code Blog

Zstandard 1.0

MyRocks

Related Articles

Zip Tests Quantum Mechanics

RocksDB on Steroids

RocksDB - Facebook's Database Now Open Source 

To be informed about new articles on I Programmer, sign up for our weekly newsletter,subscribe to the RSS feed and follow us on, Twitter, Facebook, Google+ or Linkedin. 

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info