Databricks Runtime for Machine Learning
Written by Kay Ewbank   
Monday, 15 April 2019

Databricks Runtime for Machine Learning is now generally available, offering native integration with popular ML/DL frameworks, such as scikit-learn, XGBoost, TensorFlow, PyTorch, Keras, and Horovod.

Databricks Runtime for Machine Learning is a machine learning runtime that contains multiple popular libraries, including TensorFlow, PyTorch, Keras, and XGBoost. It also supports distributed training using Horovod. The benefit of Databricks Runtime ML is that it provides a ready-to-go environment for machine learning.

databrickslogo

Databricks Runtime ML aims to be easy to use. The main libraries supported in it come pre-configured as part of the software, including HorovodRunner, which makes it possible to use the distributed deep learning framework Horovod. Horovod can be trickier to use than some frameworks because it requires you to share code and libraries across nodes, configure SSH, and execute MPI commands. HorovodRunner avoids the needs for these requirements by providing an API to allow you to use Horovod.

The team at Databricks has set up the most popular machine learning libraries as “top-tier” libraries. For these “top-tier” libraries, Databricks plans to make faster updates and provides advanced support. The top-tier libraries are:

  • TensorFlow / TensorBoard / tf.keras
  • spark-tensorflow-connector
  • PyTorch
  • Horovod / HorovodRunner
  • GraphFrames

Performance is another area to have been improved since the beta was released. In this release, improvements have been made to both Apache Spark MLlib logistic regression and tree classifiers. When running in Databricks Runtime for ML, the team at Databricks has observed around a 40% speed-up in Spark Performance Tests compared to Apache Spark 2.4.0.

The GraphFrames library in Databricks Runtime for ML also contains an optimized implementation that runs twice as fast as open-source GraphFrames and supports bigger graphs. In addition, Graph queries will make use of Spark cost-based optimization (CBO) to determine the join orders if the underlying node and edge tables contain column statistics. This can lead to as much as 100 times speed up, according to Databricks.

 

databrickslogo 

 

 

More Information

Databricks Website

Related Articles

Databricks Adds ML Model Export

 Databricks Delta Adds Faster Parquet Import

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Banner


Opaque Systems Introduces Gateway GenAI Solution
14/03/2024

Opaque Systems has announced an early access program for Opaque Gateway, software designed to address data privacy, security, and sovereignty concerns in managing GenAI implementations.



VLOGGER - AI Does Talking Heads
24/03/2024

Developed by Google researchers VLOGGER AI is a system that can create realistic videos of people talking and moving from a single still image and an audio clip as input. 


More News

raspberry pi books

 

Comments




or email your comment to: comments@i-programmer.info

Last Updated ( Monday, 15 April 2019 )