Details of the next version of Apache Spark, a free community edition of the Databricks cloud-based big data platform and Dashboards, a reporting front end, have all been announced.

The announcements were made at the recent Spark Summit East by Matei Zaharia, Databricks CTO and Spark creator.

Apache Spark is an open source data processing engine, and Zaharia said in the conference keynote that the next major version, Spark 2.0, is coming in April or May this year. The new version will see speed improvements of five to ten times; a new structured streaming real-time engine on SQL/dataframes; and unification of datasets and dataframes.

The new streaming engine is built on the Spark SQL engine, and also supports interactive and batch queries that aggregate data in a stream, then serve using JDBC. 

Alongside the announcement of Spark 2, Databricks has also announced the beta release of Databricks Community Edition, a free version of the cloud-based big data platform. Databricks was the founder of Apache Spark, and the largest contributor to Spark development. This service will provide users with access to a micro-cluster as well as a cluster manager and notebook environment. The idea is that developers can use the environment to learn Spark without the need to set up and run their own cluster environment.The initial beta rollout is invitation only. Wider access is planned over the next few months, with general availability planned for late Q2 2016.

The third announcement is Databricks Dashboards, a visual reporting application for Apache Spark clusters that can be used to provide reports and interative queries. Dashboards is actually an alternative view of a Databrick notebook aimed at end users who want to see different views of their data. Once a dashboard has been built, it can be shared with other users via its URL. The dashboard can be created with drop-down menus that can be used to choose or input parameters to alter the data being retrieved. The users needn't have any Spark knowledge, not any access to critical code. The dashboards can be updated automatically as underlying data changes. 

More Information

Spark Site

Spark Summit East

Databricks

Related Articles

Apache Releases Spark 1.6

Spark 1.4 Released

MOOC On Apache Spark 

Learning Spark (book review) 

To be informed about new articles on I Programmer, sign up for our weekly newsletter,subscribe to the RSS feed and follow us on, Twitter, Facebook, Google+ or Linkedin. 

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info