Microsoft has announced a preview version of Azure Stream Analytics, a data stream and event processing engine that provides real-time analytics on large amounts of data derived from sources such as devices, sensors, cloud infrastructure, and existing data properties in real time.

It is integrated with Azure Event Hubs, and together the two services can be used on data sets representing millions of events, and to produce analytics to find patterns or trigger an action during real-time data streaming. The third service to be announced is Azure Data Factory, which can be used to orchestrate and manage diverse data by letting you work with data pipelines over the Hadoop ecosystem using structured, semi-structured, and unstructured data sources.

The idea behind Stream Analytics is that it provides a way to deal with endpoints that stream high velocity data in the cloud – sources such as stock trading, sensors, web clickstream analytics and alerts from CRM applications. 

Using the service, you can combine streams of data such as clickstreams, logs, metering data or device-generated events with historic records or reference data. Stream Analytics is an alternative to Apache Storm, which Microsoft recently announced has been made available in Azure HDInsight. 

Azure Event Hubs is designed to work alongside Stream Analytics. Event Hubs is a publish-subscribe ingestor that can take in millions of events per second. The data that’s collected can be transformed and stored using suitable analytics services and batching and storage adapters.

Azure Data Factory is another new service, and can be used to process on-premises data like SQL Server, together with cloud data like Azure SQL Database, Blobs, and Tables. These data sources can be composed, processed, and monitored through data pipelines that will process the data with Apache Hive and Pig scripting, or custom C# processing. The Data Factory service can be used to monitor and manage the data, and you can use it to view the data production and data lineage down to the source systems.

In this video Clemens Vasters from the Azure team explains what Azure Event Hubs are: 

Another service to be announced is Azure Batch, a new platform service that you can use to run large-scale parallel and HPC applications in the cloud without needing to set up and manage a cluster or job scheduler. There’s an SDK that you can use to integrate client applications with Azure Batch, stage data to Azure, and build job-execution pipelines so you can deliver self-service applications as a service.

Azure Stream Analytics, Azure Batch, and Azure Data Factory are available in preview and Azure Event Hubs is now generally available.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info