.NET For Apache Spark Updated
Written by Kay Ewbank   
Thursday, 05 November 2020

The .NET bindings for Spark have been updated. The new 1.0 version adds support for .NET apps targeting .NET Standard 2.0 or later, as well as support for Apache Spark DataFrame APIs.

Spark is the general-purpose cluster computing framework that has native support for distributed SQL and enables streaming, graph processing, and machine learning. The .NET bindings for Spark are written on the Spark interop layer, and provide high performance bindings to multiple languages. Until their introduction Spark was accessible by coding using Scala, Java, Python or R but not .NET.

sparklogo

The improvements to the 1.0 release of .NET for Apache Spark start with the support for .NET applications targeting .NET Standard 2.0. In practical terms, it also now lets you use the Spark DataFrame APIs, including the ability to write Spark SQL. You can also write Spark apps using .NET user-defined functions.

netforspark

The library now comes with an API extension framework to add support for additional Spark libraries. Currently this includes Linux foundation Delta Lake, Microsoft OSS Hyperspace, ML.NET, and support for Apache Spark’s MLLib functionality.

The developers have improved the performance when using Apache Arrow to move data between the Spark runtime and .NET UDFs, and have also improved the features available for this. Overall, the team says that apps using .NET for Apache Spark that are not using UDFs show the same speed as Scala and PySpark-based non-UDF Spark applications. If the applications include UDFs, the .apps are at least as fast as PySpark programs, often faster.

Future plans include supporting deployment options, including integration with CI/CD DevOps pipelines and publishing or submitting jobs directly from Visual Studio.

sparklogo

More Information

.NET For Apache Spark On GitHub

.NET For Apache Spark On Microsoft

Related Articles

Have Your Say On .NET For Spark

Visual Spark Studio IDE For Spark Apps

.NET Is One With .NET 5

Spark BI Gets Fine Grain Security

Spark Announcements

Microsoft Asks For Help On The Future Of .NET - Where Do We Start?

Microsoft Open Sources .NET?       

 

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Banner


Why Is Go Good?
30/05/2022

The five co-creators of the Go language, Russ Cox, Robert Griesemer, Rob Pike, Ian Lance Taylor and Ken Thompson, have co-authored a paper in this month's edition of Communications of the ACM in which [ ... ]



Computer Music - Floppotron 3
19/06/2022

Computer-generated music is an art form in its own right. But what about music made by obsolete computer hardware? The third generation PC hardware orchestra, Floppotron 3.0, from Paweł Zadrożn [ ... ]


More News

pythondata

 



 

Comments




or email your comment to: comments@i-programmer.info