Demystifying Fabric Deployment Pipelines

Setting the stage

In this video, we'll demystify Microsoft Fabric Deployment Pipelines—explaining what they are, how to create them, and most importantly... how to use them 😉! Through a simplified, yet real-world demo, I'll showcase the deployment process and how to use deployment rules; as described in article Create deployment rules "When you're working in a deployment pipeline, different stages might have different configurations." I'll show you what this mean using two of the most commonly used Fabric items: Semantic models and Python notebooks.

Watch the demo

So, let's go with the video!

Solution

The Concept of Deployment Pipelines

Before jumping into the steps, let’s clarify what a deployment pipeline is. In a typical software development setup, we often have several environments: development, testing, pre-production, and production. Each stage is designed to ensure that your code and data pass through rigorous checks before going live. Deployment pipelines streamline this process by allowing you to automatically move resources, such as data models or reports, from one environment to the next.

Setting Up Your Development Environment

For this example, I’m using a development workspace that contains a number of different Fabric items: a lakehouse, notebooks, a data pipeline, a report, and a semantic model. These are the assets I’ll demonstrate for deployment throughout this article.

• Lakehouse: This contains a parquet file with public holiday data from various countries.
• Notebook: A Python notebook that filters the data to keep only the Nordic countries (Norway, Sweden, Denmark, and Finland).
• Reports: Built on top of this filtered data.

Now, let's walk through the deployment pipeline process.

Creating a Deployment Pipeline

1. Pipeline Creation: To start, create a new deployment pipeline. I’ll call this one "Demo," and for simplicity, it will contain just two stages: development and production.
2. Assigning Workspaces:Once you do this, the pipeline compares the two workspaces to detect differences, enabling seamless updates between them.
   • Assign your development workspace to the development stage.
   • Assign your production workspace to the production stage.
3. Deploying to Production: When you’re ready to move resources from development to production, click Deploy All. The deployment pipeline will copy all resources from the dev workspace to the production workspace.

A Closer Look at Data Handling

One crucial point to understand is that while the deployment pipeline transfers resources like reports, notebooks, and pipelines, it does not transfer data. Data in production might be different from what you use during development, so you need to load production-specific data manually.

In our case:

• The development lakehouse contains a filtered version of the public holiday dataset with only Nordic countries.
• The production lakehouse is initially empty. We’ll load the full dataset, including all countries, using a data pipeline.

Once the data is loaded, the production lakehouse reflects all countries, unlike the development lakehouse, which still contains only the Nordic countries.

Fixing Semantic Model Connections

After deploying resources to production, you might notice that the report in production still points to the development lakehouse. This happens because, by default, the semantic model references the same connections as the dev environment.

Here’s how to fix it:

1. Deployment Rules: Use deployment rules to adjust the data connections when moving between environments. For example, set up a rule that changes the connection of the semantic model to point to the production lakehouse rather than the development lakehouse.
2. Updating the Connection: You’ll need the SQL Analytics Endpoint of the production lakehouse, which can be found in the workspace settings. Replace the dev connection with the production endpoint in the deployment rule.
3. Re-deploy: Once the rule is created, redeploy your pipeline. This time, the semantic model will correctly reference the production lakehouse, and when you refresh your report, it will show data from all countries, not just the Nordic ones.

Handling Notebooks

The same logic applies to notebooks. If a notebook in development is set to work with the development lakehouse, it will still do so even after being deployed to production. You can fix this by creating a deployment rule that switches the notebook’s lakehouse connection to the production lakehouse.

1. Find the Lakehouse IDs: You’ll need the workspace ID and lakehouse ID for both environments. You can get this information from the URLs of your lakehouses.
2. Configure the Rule: Enter these details into a deployment rule to ensure that when the notebook runs in production, it points to the correct lakehouse.

What Else Can You Do with Deployment Pipelines?

Fabric deployment pipelines come with other useful features:

• History: You can track the deployment history, including changes made during each deployment.
• Access Management: At the moment, the only available role is admin, but more granular roles are expected in future updates.
• UI Update: There’s a new user interface for deployment pipelines, but I recommend waiting a bit before using it, as there are still some bugs.

Conclusion

Deployment pipelines are a powerful tool that simplifies moving resources between environments. With deployment rules, you can ensure that all connections (whether in reports or notebooks) point to the right production resources, allowing you to replicate the setup from dev to production without manual intervention.

I hope you found this guide helpful! Stay tuned for more insights in the next Tech Taco Friday, where I’ll dive into data warehouses and how to manage your schemas under version control. Until next time, happy deploying!