Stream Analytics processing time

Hello World,

Using Stream Analytics, of course you do 🙂

what is the most common problem that you can come across when using it?

  • Data not flowing in
  • Data is processed for too long
  • Data is not getting out

Probably the first and the last problem are easy to process and there are lots of metrics for that Have a look at the list at the end of the article (the list is from Microsoft website).

Input can be analyzed for network problems by OMS or by checking the IoT hub for the information flow.

Output is easy too as its goes to PowerBi or a Databrick and there its been managed by the other systems.

but what about the inner function of the stream? how long it takes to get the data, run ML on it and send to PowerBI ? Now you can know! new counter with the name Watermark Delay

what is it ?

Simple, the time stream got out – time stream got in = Watermark Delay

if you have problems with your inner functions, for example your ML model taking too long, you can fix it right away and most important, you can create an alert of the stream takes too long. just press the bell of the Configure alerts, set your threshold and your done!

List of available counters in the metrics tab of Stream Analytics:

Metric Definition
SU % Utilization The utilization of the Streaming Unit(s) assigned to a job from the Scale tab of the job. Should this indicator reach 80%, or above, there is high probability that event processing may be delayed or stopped making progress.
Input Events Amount of data received by the Stream Analytics job, in number of events. This can be used to validate that events are being sent to the input source.
Output Events Amount of data sent by the Stream Analytics job to the output target, in number of events.
Out-of-Order Events Number of events received out of order that were either dropped or given an adjusted timestamp, based on the Event Ordering Policy. This can be impacted by the configuration of the Out of Order Tolerance Window setting.
Data Conversion Errors Number of data conversion errors incurred by a Stream Analytics job.
Runtime Errors Total number of errors related to query processing (excluding errors found while ingesting events or outputing results)
Late Input Events Number of events arriving late from the source which have either been dropped or their timestamp has been adjusted, based on the Event Ordering Policy configuration of the Late Arrival Tolerance Window setting.
Function Requests Number of calls to the Azure Machine Learning function (if present).
Failed Function Requests Number of failed Azure Machine Learning function calls (if present).
Function Events Number of events sent to the Azure Machine Learning function (if present).
Input Event Bytes Amount of data received by the Stream Analytics job, in bytes. This can be used to validate that events are being sent to the input source.

Click edit button to change this text. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut elit tellus, luctus nec ullamcorper mattis, pulvinar dapibus leo.

Add a Comment

Your email address will not be published. Required fields are marked *