Is your Event Hub ingestion to Azure Data Explorer “throttling”? Do you have trouble increasing the processing of events received at the cluster? Scaling up or scaling out the cluster does not seem to resolve the issue? Then the data management nodes, running the endpoint might be the reason.
In the Azure Portal you have the options of scaling up and scaling out your data explorer cluster. A less known component that is available to scale is the data management nodes. Normally the data management nodes are automatically scaled for you according to the FAQ.
The number and type of data management nodes are auto selected for you depending on your data ingestion needs. Data management nodes are either D3 v2 or D13 v2 VM’s.
Azure Data Explorer pricing information
But what is the data management endpoint you may ask? The data management endpoint is among other things responsible for processing the events coming from Azure Event Hubs and can essentially limit the amount of events being processed when the nodes are scaled incorrectly. This become easily visible when we talk millions of events at large scale.
Ingesting using the blob storage method is of course the best approach when dealing with large amount of data, but what if that is not an option for you and the auto scaling is set at the lower end?
The solution is relatively easy and requires creating a support ticket in the Azure Portal, where Microsoft hopefully will handle the scaling of data management nodes for you. Just remember that scaling the nodes most likely comes at an additional cost.
Is viewing and scaling of the data management nodes a feature that’s important for you? Then I encourage that you vote for it at the Azure Feedback portal:
https://feedback.azure.com/forums/915733-azure-data-explorer/suggestions/37263838-ability-to-view-and-change-data-management-endpoin
