Database migrations using FlyWay

In the world as a developer you have a big toolbox in front of you, constantly evolving. As a result it’s not always easy to navigate between everything and sometimes hard to find out where to start.

You may remember that I previously blogged about testing using the tSQLt framework. At the point I assumed that you were familiar with SQL Server Database Projects, having a state based deployment model. Today, however, I will give you an introduction on how to start with migration based deployments.

Read more

Scaling the Data Management Nodes for Azure Data Explorer

Is your Event Hub ingestion to Azure Data Explorer “throttling”? Do you have trouble increasing the processing of events received at the cluster? Scaling up or scaling out the cluster does not seem to resolve the issue? Then the data management nodes, running the endpoint might be the reason.

Read more “Scaling the Data Management Nodes for Azure Data Explorer”

Azure Data Explorer Series, Part 1

Have you heard about Azure Data Explorer, formerly known as project Kusto? The new PaaS from Microsoft that enables fast and highly scalable data exploration? The service that empowers Azure Monitor and Time Series Insights?

Fear not! As of 7 February 2019 it was announced to be General Available, meaning that it’s the new kid on the block. I will therefore dive into the nitty-gritty details in a series of blog posts that hopefully gives you enough information to get started yourself.

Read more “Azure Data Explorer Series, Part 1”

Azure SQL unit testing with tSQLt using Azure DevOps

Working as a Data Engineer I often see that unit tests are written for the applications that are developed, but often forgotten about when doing database development. We would assume that it’s the de facto standard in 2019 if the target is to be fully DevOps as a development team.

There are many great third party tools out there that simplifies setting up the processes, most of them requires licenses. Therefore I will go back to the basics so that you cannot blame the license cost for not implementing database unit testing.

Read more “Azure SQL unit testing with tSQLt using Azure DevOps”

Azure Data Studio now support MFA

When I first learned about Azure Data Studio being available September 2018, I was excited for finally getting a cross platform alternative to SQL Server Management Studio that also is built on top of Visual Studio Code. And who doesn’t love Visual Studio Code?

Sadly the excitement dropped a bit when I realized that the only option for logging in to Azure SQL databases was by using SQL logins, but that has changed now!

Read more “Azure Data Studio now support MFA”

Azure DevOps with Data Factory

In many organization a common misconception is that DevOps is about the tools we use, so let’s use a second to read the citation from Microsoft.

DevOps brings together people, processes, and technology, automating software delivery to provide continuous value to your users. 

Microsoft

Of course this post will not be about what DevOps is and isn’t, but I think it’s important refresh the citation once in a while. This post will however focus on how the tooling can help you automate the deployment of Data Factory from development to test environment, that can easily be transitioned to any other environments you may have.

Read more “Azure DevOps with Data Factory”