Monthly Archives: June 2023
4. Azure Monitor – Alerts Hi and welcome back. Now in this chapter, I want to go through a special note when it comes to alerts that you can create in Azure Monitor. So if I go on to the monitoring service, so remember, Azure Monitor can be used to gather all of the metrics… Read More »
DP-203 Data Engineering on Microsoft Azure – Monitor and optimize data storage and data processing
1. Best practices for structing files in your data lake Now, in this chapter, I want to give some best practices when it comes to structuring your files. And this is when it comes to building your data lake. So normally when you design a data lake, you might go ahead and create something known… Read More »
DP-203 Data Engineering on Microsoft Azure – Design and Implement Data Security Part 6
17. Lab – Azure Synapse – External Tables Authorization via Azure AD Authentication Now, in this chapter, I want to go through virtual Network service endpoints. Now, what is the purpose of having service endpoints in place? So here in this example, let’s say you have a virtual machine that is part of a virtual… Read More »
DP-203 Data Engineering on Microsoft Azure – Design and Implement Data Security Part 5
14. Lab – Azure Data Lake – Role Based Access Control Now in this chapter I want to explain rolebased access control. Now we know that Azure Active Directory is our identity provider. So here we can create users and the user can actually log into Azure account. Now let’s say that we do define… Read More »
DP-203 Data Engineering on Microsoft Azure – Design and Implement Data Security Part 4
11. Lab – Azure Synapse – Azure AD Authentication – Creating a user Now in the last chapter we had seen how to define a SQL Active Directory admin when it comes to Azure Synapse. Now in this chapter, let’s create a new user in Azure Active Directory and give that user access onto our… Read More »
DP-203 Data Engineering on Microsoft Azure – Design and Implement Data Security Part 3
7. Lab – Azure Synapse – Auditing And welcome back. Now in this chapter I want to go through the auditing feature which is available in Azure Synapse so you can enable the auditing feature for an Aziosql pool in Azure Synapse. This feature can be used to track the database events and write them… Read More »
DP-203 Data Engineering on Microsoft Azure – Design and Implement Data Security Part 2
4. Azure Synapse – Customer Managed Keys Hi, and welcome back. Now, in this chapter, I quickly want to go through the concept of being able to do encryption of your Azure Synapse workspace with the help of your keys defined in your Azure Key vault. So if you want to use Customer Manage keys… Read More »
DP-203 Data Engineering on Microsoft Azure – Design and Implement Data Security
1. Section Introduction Hi and welcome to this section. When it comes to security now, security is a very important aspect and in Azure there are a lot of security aspects that are available for Azure based services. So when it comes to Azure data Lake we’ll be looking at role based access control, we’ll… Read More »
29. Delta Lake Introduction Hi and welcome back. Now in this chapter I just want to go through Delta Lake when it comes to Azure databricks and we’ll see a couple of examples on Delta Lake in data bricks itself. So with the help of a Delta Lake, you get some more features when it… Read More »
24. Autoscaling a cluster Hi and welcome back. Now, in this chapter, I just want to give some notes when it comes to auto scaling. So I’ve shown you that when you create your cluster in Azure databricks, you have this auto scaling concept. So when creating an Azure databricks cluster, you can specify the… Read More »