RocksDB - Facebook's Database Now Open Source
Written by Kay Ewbank   
Thursday, 21 November 2013

The Facebook Database Engineering Team has open-sourced RocksDB, an embeddable persistent key-value store for fast storage that’s based on Google’s LevelDB.

RocksDB is aimed at embedded workloads, though the team says it can also be the foundation for a client-server database. RocksDB builds on LevelDB to be scalable to run on servers with many CPU cores. It is designed to use fast storage efficiently, and to support IO-bound, in-memory and write-once workloads.




The Rocksdb project started at Facebook as an experiment to develop an efficient database to use flash storage for server workloads. It is a C++ library and can be used to store keys-and-values where keys and values are arbitrary size byte streams, with support for atomic reads and atomic writes. RocksDB can be tuned to run on data on pure memory, flash, hard disks or on HDFS. It has support for various compression algorithms and good tools for production support and debugging.

The Facebook team began by benchmarking LevelDB, but it was unsuitable for the Facebook server workloads.

According to the RocksDB team: 

“The benchmark results look awesome at first sight, but we quickly realized that those results were for a database whose size was smaller than the size of RAM on the test machine - where the entire database could fit in the OS page cache. When we performed the same benchmarks on a database that was at least 5 times larger than main memory, the performance results were dismal.“

Having developed RocksDB and devised benchmarks to test its for server-workload on Flash, the Facebook team were able to show that RocksDB "solidly outperforms" LevelDB. The team comments:

“We found that LevelDB's single-threaded compaction process was insufficient to drive server workloads. We saw frequent write-stalls with LevelDB that caused 99-percentile latency to be tremendously large. We found that mmap-ing a file into the OS cache introduced performance bottlenecks for reads. We could not make LevelDB consume all the IOs offered by the underlying Flash storage.”

The benchmarks and their results are reported on GitHub. RockDB's code is also available on Github. Facebook Engineer Dhruba Borthakur gave introductory talk on RocksDB at the Data @ Scale 2013 conference, and his slides are available to view.



Suggested uses of RocksDB are applications that need low latency database accesses, such as a user-facing application that stores the viewing history and state of users of a website, or a spam detection application that needs fast access to big data sets. RocksDB can also be used to cache data from Hadoop, thereby allowing applications to query Hadoop data in real time.


Hour of Code Aims to Reach 100 Million Worldwide

After its successful debut last year, in which the inaugural Hour of Code reached well over its initial target of 10 Million students, has embarked on a crowdfunding campaign to raise $5 mill [ ... ]

Python Tools for Visual Studio Gets New Focus

PTVS 2.1 has just been released and now can be used for free with Visual Express. The more interesting news is that the PTVS team has become part of the Azure Machine Learning group and it PTVS2. [ ... ]

More News

Last Updated ( Thursday, 21 November 2013 )

RSS feed of news items only
I Programmer News
Copyright © 2014 All Rights Reserved.
Joomla! is Free Software released under the GNU/GPL License.