Apache Flink has been updated with a new AI Model DDL that can be used to manage AI models through Flink SQL and the Table API.

Apache Flink is an open source platform for distributed stream and batch data processing, with a streaming dataflow engine for data distribution and distributed computations over data streams.

The addition of the new dedicated syntax for AI models in Flink 2.0 means users can define models as easily as creating catalog objects and invoke them like standard functions or table functions in SQL statements. In Flink 2.1 this has been extended to add Model DDLs Table API support, so users can define and manage AI models programmatically via the Table API in both Java and Python. This provides a flexible, code-driven alternative to SQL for model management and integration within Flink applications.

The implementation supports both Flink built-in model providers (OpenAI) and interfaces for users to define custom model providers. The developers say they  plan to introduce more AI functions such as ML_EVALUATE, VECTOR_SEARCH to unlock end-to-end experience for real-time data processing, model training, and inference.

The new version also extends the ML_PREDICT Table-Valued Function (TVF), so you can invoke AI models in real time within Flink SQL. The developers say this lays the foundation for building end-to-end real-time AI workflows.

Real-Time data processing has also been improved with the addition of Process Table Functions (PTFs) to open up the Flink SQL engine for more event-driven application. This gives access to Flink’s managed state, event-time and timer services, and underlying table changelogs. 

There's also a new VARIANT data type for efficient handling of semi-structured data like JSON. Combined with the PARSE_JSON function and lakehouse formats (e.g., Apache Paimon), this enables dynamic schema data analysis.

Finally, streaming joins performance has been improved with the introduction of DeltaJoin and MultiJoin strategies, eliminating state bottlenecks and improving resource utilization and job stability.

More Information

Flink website

Related Articles

Apache Flink ML 2.0 Released

Apache Flink 1.9 Adds New Query Engine

Apache Flink 1.5.0 Adds Support For Broadcast State

Flink Gets Event-time Streaming

FLink Reaches Top Level Status

Related Articles