The Facebook Database Engineering Team has open-sourced RocksDB, an embeddable persistent key-value store for fast storage that’s based on Google’s LevelDB.

RocksDB is aimed at embedded workloads, though the team says it can also be the foundation for a client-server database. RocksDB builds on LevelDB to be scalable to run on servers with many CPU cores. It is designed to use fast storage efficiently, and to support IO-bound, in-memory and write-once workloads.

The Rocksdb project started at Facebook as an experiment to develop an efficient database to use flash storage for server workloads. It is a C++ library and can be used to store keys-and-values where keys and values are arbitrary size byte streams, with support for atomic reads and atomic writes. RocksDB can be tuned to run on data on pure memory, flash, hard disks or on HDFS. It has support for various compression algorithms and good tools for production support and debugging.

The Facebook team began by benchmarking LevelDB, but it was unsuitable for the Facebook server workloads.

According to the RocksDB team: 

“The benchmark results look awesome at first sight, but we quickly realized that those results were for a database whose size was smaller than the size of RAM on the test machine - where the entire database could fit in the OS page cache. When we performed the same benchmarks on a database that was at least 5 times larger than main memory, the performance results were dismal.“

Having developed RocksDB and devised benchmarks to test its for server-workload on Flash, the Facebook team were able to show that RocksDB "solidly outperforms" LevelDB. The team comments:

“We found that LevelDB's single-threaded compaction process was insufficient to drive server workloads. We saw frequent write-stalls with LevelDB that caused 99-percentile latency to be tremendously large. We found that mmap-ing a file into the OS cache introduced performance bottlenecks for reads. We could not make LevelDB consume all the IOs offered by the underlying Flash storage.”

The benchmarks and their results are reported on GitHub. RockDB's code is also available on Github. Facebook Engineer Dhruba Borthakur gave introductory talk on RocksDB at the Data @ Scale 2013 conference, and his slides are available to view.

Suggested uses of RocksDB are applications that need low latency database accesses, such as a user-facing application that stores the viewing history and state of users of a website, or a spam detection application that needs fast access to big data sets. RocksDB can also be used to cache data from Hadoop, thereby allowing applications to query Hadoop data in real time.