Data Engineer
Is your Event Hub ingestion to Azure Data Explorer “throttling”? Do you have trouble increasing the processing of events received at the cluster? Scaling up or scaling out the cluster does not seem to resolve the issue? Then the data management nodes, running the endpoint might be the reason.
Read more “Scaling the Data Management Nodes for Azure Data Explorer”
Have you heard about Azure Data Explorer, formerly known as project Kusto? The new PaaS from Microsoft that enables fast and highly scalable data exploration? The service that empowers Azure Monitor and Time Series Insights?
Fear not! As of 7 February 2019 it was announced to be General Available, meaning that it’s the new kid on the block. I will therefore dive into the nitty-gritty details in a series of blog posts that hopefully gives you enough information to get started yourself.
Working as a Data Engineer I often see that unit tests are written for the applications that are developed, but often forgotten about when doing database development. We would assume that it’s the de facto standard in 2019 if the target is to be fully DevOps as a development team.
There are many great third party tools out there that simplifies setting up the processes, most of them requires licenses. Therefore I will go back to the basics so that you cannot blame the license cost for not implementing database unit testing.
Read more “Azure SQL unit testing with tSQLt using Azure DevOps”
When I first learned about Azure Data Studio being available September 2018, I was excited for finally getting a cross platform alternative to SQL Server Management Studio that also is built on top of Visual Studio Code. And who doesn’t love Visual Studio Code?
Sadly the excitement dropped a bit when I realized that the only option for logging in to Azure SQL databases was by using SQL logins, but that has changed now!
In many organization a common misconception is that DevOps is about the tools we use, so let’s use a second to read the citation from Microsoft.
DevOps brings together people, processes, and technology, automating software delivery to provide continuous value to your users.
Microsoft
Of course this post will not be about what DevOps is and isn’t, but I think it’s important refresh the citation once in a while. This post will however focus on how the tooling can help you automate the deployment of Data Factory from development to test environment, that can easily be transitioned to any other environments you may have.
