This article explains about how the latest feature in Microsoft Fabric Pipelines “Refresh Semantic Model” works and the new possibilities with this feature. This is one of the most requested features for Azure Data Factory and now this is landed into Microsoft Fabric first.
This is a wonderful feature for Microsoft Fabric Data & Analytics project to enable complete end to end use cases – Data Load to target data stores like Lakehouse or Warehouse and refresh the Semantic model – Import or Direct Lake, enables a fully automatic.
Previously, if we want to achieve the semantic model refresh, we need to use the API method – Web activity in Azure Data Factory. There we need to get the token and call the APIs. It was not just one activity to achieve the result.
Note – We have native connector to refresh the semantic model in Power Automate. Some of the organizations are using Power Automate to refresh semantic models.
It is using Enhanced Refresh API. Check the details below.
POST https://api.powerbi.com/v1.0/myorg/groups/{groupId}/datasets/{datasetId}/refreshes
Refer – Datasets – Refresh Dataset In Group – REST API (Power BI Power BI REST APIs) | Microsoft Learn
The application which we use for triggering the API will trigger the semantic model refresh but we need to explicitly call other API to monitor the status of the refresh to understand whether refresh is completed or failed.
This new Activity in Fabric Pipelines simplifies the whole process of refreshing the semantic model easily from Fabric Pipelines. This is also using the same Enhanced API so refreshes will run asynchronously.
It also includes the below interesting features,
Authentication
It supports below authentications.
- Organization Account
- Service Principal
One of the important news is that we can still use different tenant and call the semantic model refresh from different tenant.
Service Principles enables to use one common account to call the refresh for the semantic model.
Note: As of now, some improvements are needed like a dedicated page to manage all the connections. I hope this will be addressed soon future.
Wait on Completion
This is an optional option which helps to wait and see the status of the refresh and following that we can add further actions may be notifications or auditing etc.
If we uncheck this option, then “Semantic model refresh” simply trigger the refresh and complete the pipelines successfully. Example, this activity just took 3 seconds to complete, but the semantic model refresh was taking its own time.
If there are any errors during the semantic model refresh, then pipeline task will get failed.
Pipeline also will fail if the previous refresh is still on.
Ad-hoc Refresh
Lets say if you want to allow users to refresh the semantic model, then we can leverage the Power Automate but let’s assume that you have built a framework in Microsoft Fabric Pipelines which you want to reuse the same for this scenario. You can follow the bellow approach. You also can use this approach to call from any application which you use for ETL which supports API calls.
Fabric Pipelines support APIs, we can leverage the APIs to trigger the pipeline from any application.
Refer – Fabric data pipeline public REST API (Preview) – Microsoft Fabric | Microsoft Learn
This is a POST call,
POST https://api.fabric.microsoft.com/v1/workspaces/<your WS Id>/items/<pipeline id>/jobs/instances?jobType=Pipeline
This is using the Authorization as Bearer Token to call the APIs. As of now, API doesn’t support Service Principal. Watch this space to know the limitations – Fabric data pipeline public REST API (Preview) – Microsoft Fabric | Microsoft Learn
Conclusion
Refreshing a semantic model from Microsoft Fabric Pipelines made easy by the “Semantic Model Refresh” activity. This is the first version from the product team but still it got some interesting features. We probably can expect more sophisticated features like refresh a specific table, partitions etc. in near future. Please share your feedback about this article and let me know your use cases with this new activity.
Happy Learning!!
Leave a Reply