SQLFire - Speeds and Scales
Written by Kay Ewbank   
Friday, 07 September 2012

VMware's in-memory distributed SQL database, vFabric SQLFire, can load 8 million rows in 88 seconds.

SQLFire is designed to work as an in-memory distributed SQL-based cache that can be used alongside a traditional database handling the disk-based side of the process. vFabric SQLFire has JDBC and ADO.NET interfaces that you use to query the data store using SQL.

The keys and indexes are stored in memory to provide high scalability, availability and performance. It can also use a RDBMS where data persistence is required, and there are plans to provide support for major databases including Oracle, MySQL, Sybase, DB2, SQL Server and postGres.

vfabric1

One concern of using data onto a memory based data system with large data sets is the potential time taken to actually load it. Pas Apicella of the vFabric Cloud Application Platform team has posted an interesting blog post about just how well SQLFire achieves this task, loading 8 million rows in 88 seconds. The team estimates that this means SQLFire would be able to load around 40GB of data in an hour. The data was in CSV format, and the team used a multi-threaded load approach.

The team used SQLFire 1.0.3, with two virtual machines each with 18G of memory. The virtual machines shared the same home directory as well as having separate disks for their own disk stores and log files. A separate disk store was used for persistence, and any overflow data would be removed from memory and not written to disk. The in-memory table asynchronously writes its changes to the disk store. This means it only has to do the memory insert prior to moving onto the next change. Only six threads were used, and Apicella says making the thread count too high can have an adverse effect.

Other than ensuring the process loads only for the client exists and preventing the second node from attempting to read the file as well, the rest of the setup looks remarkably straightforward. You can see the scripts and output  here.

 speed

More Information

vFabric SQLFire datasheet (pdf)

VMware vFabric blog

Related Articles

CORFU - Flash Fault Tolerant Distributed Storage

MemSQL - 80,000 queries per second

SQL Fiddle

 

blog comments powered by Disqus

 

To be informed about new articles on I Programmer, install the I Programmer Toolbar, subscribe to the RSS feed, follow us on, Twitter, Facebook, Google+ or Linkedin,  or sign up for our weekly newsletter.

 

Banner


Survey Reveals American Public Suspicious Of Tech
20/04/2014

The American public is split down the middle over whether they would be prepared to ride in a driverless car and whether computers will match humans when it comes to creating works of art.



Microsoft Reveals "Universal" Apps
03/04/2014

Universal apps are part of Microsoft's "One Windows" strategy. A universal app sounds as if it might be what we are all looking for. Write once run anywhere - well as long as it's Windows. Not quite.

 [ ... ]


More News

Last Updated ( Friday, 07 September 2012 )
 
 

   
RSS feed of news items only
I Programmer News
Copyright © 2014 i-programmer.info. All Rights Reserved.
Joomla! is Free Software released under the GNU/GPL License.