Netflix has open sourced Metaflow, which the developers describe as a human-centric framework for data science. The framework has been used internally at Netflix for the last two years to build and manage hundreds of data-science projects.

Metaflow is a Python library that provides a unified API to the infrastructure stack that is required to execute data science projects, from prototype to production. It lets data scientists structure their workflow as a Directed Acyclic Graph of steps.

Netflix developed Metaflow to boost the productivity of data scientists working on a wide variety of projects from classical statistics to state-of-the-art deep learning. The aim was to make problems in model prototypes visible earlier in the process. In February Netflix said that Metaflow had cut model deployment times from four months to seven days. 

The steps can be arbitrary Python code, with data and models stored as normal Python instance variables. They work even if the code is executed on a distributed compute platform, which Metaflow supports by default, thanks to its built-in content-addressed artifact store.

The Metaflow team says that using code is important to make the framework useful. They say that while many machine learning infrastructures rely on GUIs, configuration, DSLs, or automation, they tend to fail to scale when the problems become more complex. Instead, Metaflow relies on code. They say:

"We rely on code. General-purpose programming languages are arguably the most effective tool in building functional systems for complex use cases. The main downside of code is a steeper learning curve: for the simplest use cases, almost any other modality would provide a simpler solution. To minimize this cost, we rely on easy to learn, popular languages like Python or R."

Metaflow uses existing infrastructure when feasible. In particular, it is tightly integrated with Amazon Web Services. You can use Metaflow with a variety of machine learning or data science libraries, such as PyTorch, Tensorflow, or SciKit Learn. Metaflow allows you to write your models and business logic as idiomatic Python code with not much new to learn.

There's more information on the Metaflow website and on Github.

More Information

Metaflow Website

Metaflow On GitHub

Related Articles

Netflix Releases Polynote

Sagemaker Studio - An IDE for Machine Learning

Machine Learning With App Inventor

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info