RocksDB - Facebook's Database Now Open Source
Written by Kay Ewbank   
Thursday, 21 November 2013

The Facebook Database Engineering Team has open-sourced RocksDB, an embeddable persistent key-value store for fast storage that’s based on Google’s LevelDB.

RocksDB is aimed at embedded workloads, though the team says it can also be the foundation for a client-server database. RocksDB builds on LevelDB to be scalable to run on servers with many CPU cores. It is designed to use fast storage efficiently, and to support IO-bound, in-memory and write-once workloads.

 

rocksdbcode

 

The Rocksdb project started at Facebook as an experiment to develop an efficient database to use flash storage for server workloads. It is a C++ library and can be used to store keys-and-values where keys and values are arbitrary size byte streams, with support for atomic reads and atomic writes. RocksDB can be tuned to run on data on pure memory, flash, hard disks or on HDFS. It has support for various compression algorithms and good tools for production support and debugging.

The Facebook team began by benchmarking LevelDB, but it was unsuitable for the Facebook server workloads.

According to the RocksDB team: 

“The benchmark results look awesome at first sight, but we quickly realized that those results were for a database whose size was smaller than the size of RAM on the test machine - where the entire database could fit in the OS page cache. When we performed the same benchmarks on a database that was at least 5 times larger than main memory, the performance results were dismal.“

Having developed RocksDB and devised benchmarks to test its for server-workload on Flash, the Facebook team were able to show that RocksDB "solidly outperforms" LevelDB. The team comments:

“We found that LevelDB's single-threaded compaction process was insufficient to drive server workloads. We saw frequent write-stalls with LevelDB that caused 99-percentile latency to be tremendously large. We found that mmap-ing a file into the OS cache introduced performance bottlenecks for reads. We could not make LevelDB consume all the IOs offered by the underlying Flash storage.”

The benchmarks and their results are reported on GitHub. RockDB's code is also available on Github. Facebook Engineer Dhruba Borthakur gave introductory talk on RocksDB at the Data @ Scale 2013 conference, and his slides are available to view.

 

 

Suggested uses of RocksDB are applications that need low latency database accesses, such as a user-facing application that stores the viewing history and state of users of a website, or a spam detection application that needs fast access to big data sets. RocksDB can also be used to cache data from Hadoop, thereby allowing applications to query Hadoop data in real time.

More Information

RocksDB.org

RocksDB on Github

Related Articles

MariaDB 10.0 Beta Introduces Replication Enhancements

Major Update to Google BigQuery

Google Showcases F1

BlinkDB Alpha of Approximate Query Engine Released 

NuoDB Challenges NoSQL

Google's F1 - Scalable Alternative to MySQL

 

To be informed about new articles on I Programmer, install the I Programmer Toolbar, subscribe to the RSS feed, follow us on, Twitter, Facebook, Google+ or Linkedin,  or sign up for our weekly newsletter.

 

blog comments powered by Disqus

Banner


ShellShock - Yet Another Code Injection Vulnerability
25/09/2014

A vulnerability in the Bash shell has the Internet in full alarmed mode and like all good security holes it even has a catchy name - Shell Shock. How does it work?



TODO - A New Group To Tell Open Source Programmers
17/09/2014

A new group, which currently includes big hitters like Facebook, Google, Twitter and GitHub, aims to tell open sourcers how to do it better.


More News

Last Updated ( Thursday, 21 November 2013 )
 
 

   
RSS feed of news items only
I Programmer News
Copyright © 2014 i-programmer.info. All Rights Reserved.
Joomla! is Free Software released under the GNU/GPL License.