IBM Introduces Hybrid Cloud AI/ML Framework
Thursday, 15 July 2021

IBM is introducing CodeFlare, a serverless framework that aims to reduce the time and effort developers spend training and preparing AI and machine learning models for deployment in hybrid cloud environments.

CodeFlare is a framework to simplify the integration, scaling and acceleration of complex multi-step analytics and machine learning pipelines on the cloud.


CodeFlare can be used for pipeline execution and scaling with tools to define and execute parallel pipelines. The developers say it unifies pipeline workflows across multiple frameworks while providing nearly optimal scale-out parallelism on pipelined computations.

It supports a serverless user experience by integrating Red Hat OpenShift and IBM Cloud Code Engine and providing adapters and connectors for loading data and connecting to data services.

CodeFlare is built on top of Ray, an open-source distributed computing framework for machine learning applications. CodeFlare adds elements to make it easier to scale workflows, so extending the capabilities of Ray.

CodeFlare also has a Python-based interface for creating pipelines that simplifies the task of training and optimizing the model through data cleaning, feature extraction, and model optimization. The researchers say the goal of the new framework is to unify pipeline workflows across multiple platforms without requiring data scientists to learn a new workflow language.

The team says CodeFlare makes the process of analyzing and optimizing pipelines significantly faster, giving the example of one user using the framework for approximately 100,000 pipelines for training machine learning models, who found CodeFlare cut the time it took to execute each pipeline from 4 hours to 15 minutes.

CodeFlare has been made available open source on GitHub, and the developers have written a series of technical blog posts on how it works and what you need to get started using it. They say they've started applying this tech to things they’re building at IBM in AI research.

Future plans include support for more complex pipelines, enhanced fault-tolerance and consistency, along with improved integration and data management for external sources, and adding support for pipeline visualization.


More Information

CodeFlare On GitHub


Related Articles

IBM Releases CodeNet Dataset For AI Coding 

IBM's Elyra AI Toolkit

New MIT–IBM Watson AI Lab

Google Comics Factory Makes ML Easy

Nearly A Third Of Devs Using AI And ML

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.


Inaugural STOC Test Of Time Awards

At the 2021 ACM Symposium on Theory of Computing (STOC), held online in June, sixteen recipients were honored in respect of seven papers presented at past STOC conferences  which have had a long- [ ... ]

Flutter Improves Performance

There's an updated version of Flutter with improvements including performance and feature improvements to help users track down performance problems in their own apps, full screen support for Android, [ ... ]

More News





or email your comment to:

Last Updated ( Thursday, 15 July 2021 )