MS Fabric as primary datastore not ingesting data into tables (v0.10)

[MichaelKracker]

I am interested in using MS Fabric as a primary data store. My toolkit deployment and Fabric instance should be configured to the new documentation for v0.10, including the new pre and post template deployment steps for manually configuring the components in Fabric. Data is not ingesting into tables in the Fabric Ingestion database. I don't see any errors in the pipelines. There is also no activity for either ingestion_ExecuteETL or ingestion_ETL_dataExplorer.

These are the triggers in my configuration. Should there be more?

I do see data in the following tables:

• HubSettingsLog
• PricingUnits
• Regions
• ResourceTypes
• Services

As of 4/29/2025 we moved to an MCA. We currently have the new MCA data being exported, but plan to backfill EA data after we see the dashboard and reports working. Our MCA exports are configured at the Billing Profile. I have waited a couple days after the initial export and manually uploaded a manifest.json file to see if it would trigger the ingestion process into Fabric.

From the storage account, I do see .parquet files listed under the Ingestion container. For testing, I have used a Power BI Storage report to view data in the Ingestion container which does allow me to view successfully.

Is there a way to manually trigger ingestion_ExecuteETL? What additional information do you need from my environment to troubleshoot this issue?

We are excited to see more support for Fabric. Thanks to all who helped make that happen.

[flanakin]

@MichaelKracker it looks like you didn't specify the Fabric eventhouse query URI in the FinOps hubs deployment. Can you deploy again with the query URI?

I just validated this earlier this week, so it should work. The ingestion trigger and pipelines only get deployed when an ADX cluster name or Fabric query URI is specified. So if those aren't there, then it won't ingest the data.

I'm at FinOps X next week, but we can hop on a call to investigate if that doesn't work.

[MichaelKracker]

I did have the query URI entered for all of my test deployments. The correct query URI is displaying in the hubDataExplorer item under Linked services in the Data Factory. Also, the config_InitializeHub would stay in a Pending state, or fail, until I added the Managed Identity as an admin to the Eventhouse databases. A Data Explorer URI was not entered during deployment.

I validated multiple times with v0.10, but none of the attempts worked. I noticed that v0.11 was released on June 2nd. PR1622 seems related to my issue, which appears to be included in the latest version. I redeployed v0.11 (over v0.10) and encountered an unusual behavior. Upon reviewing the resources in my Resource Group, I found additional DataFactory and Managed Identity resources. A similarly named resource was created, but the newer one lacks a hyphen in the name for both Data Factory and Managed Identity, differing from my other deployments. I had been using the Recent Resources to open Data Factory from the main Azure Portal page, not realizing a new Data Factory had been deployed.

After adding Admin permissions for the new Data Factory Managed Identity onto the Eventhouse databases, the data is now ingesting. I did stop the Triggers from the other Data Factory and left it in place in case you would like to review my environment. The newly created Data Factory now has the ingestion_ManifestAdded trigger listed and is running successfully.

I can create a separate item, but noticed Savings vary between pages on the Cost Summary Report (v0.11). RI purchases do not appear to be visible on the Inventory page in the Total Savings and the ESR % does not seem correct on the Summary/Services/Subscriptions pages.

[flanakin]

We found and fixed a bug that removed a dash in those names. It was accidentally left out during some refactoring. If you redeploy the latest 0.11 then it should be back. You can safely remove the ones without the dash. The issue only applies to ADF, KeyVault, and supporting resources that shared those names.

[flanakin]

Definitely share details about the savings mismatch so we can investigate. We fixed an issue with the accumulated chart in 0.11 but I'm not aware of any others yet. Same with ESR.

[flanakin]

The Inventory page has a filter to only look at usage, so purchases aren't included.