New installation - msexports_ExecuteETL fails

[butch-albrecht]

I have a new installation of the finops-toolkit .8. I am a MCA customer and have Cost Management & Billing exports setup to go to a storage account. Data Factory connects to the storage account without issues.
I get multiple failures for the msexports_ExecuteETL pipeline, but feel if I fix one the others will get fixed as well. Some msexports_ExecuteETL jobs work - the ones around recommendations are the ones that are failing.
I am new to Data Factory and the toolkit and have tried to go through the troubleshooting guide and common errors, but no luck. I also searched this forum without success.
Problem:
The error I have is:
Operation on target For Each Blob failed: Activity failed because an inner activity failed; Inner activity name: Execute, Error: Operation on target Get Existing Parquet Files failed: ADLS Gen2 operation failed for: Operation returned an invalid status code 'NotFound'. Account: mystorageaccount'. FileSystem: 'ingestion'. Path: 'Recommendations/2025/04/providers/microsoft.billing/billingaccounts/mybillingaccountid/billingprofiles/mybillingprofileID'. ErrorCode: 'PathNotFound'. Message: 'The specified path does not exist.'. RequestId: 'myrequestID#'. TimeStamp: 'Tue, 15 Apr 2025 18:29:29 GMT'.

I can tell you that the storage account does have an ingestion container, but does not have a Recommendations folder.
The msexports_ExecuteETL is configured (I think) to look at the msexports container - but I think that is where I have gone wrong. So this is how msexports_ExectueETL has for settings:

• the source dataset = manifest
• fileName @pipeline().parameters.fileName
• folderPath @pipeline().parameters.folderPath

The source dataset for manifest has the linked service of my storage account which I can do a successful Test Connection.
However, the File Path for this connection is:
File Path [blank box]/@{dataset().folderPath}/@{dataset().fileName}

There is a browse button this page and I can browse to the desired folder. But doing a preview data fails.

Not sure I have explained this sufficiently. But any guidance anyone could provide would be appreciated.

[butch-albrecht]

I see there is .9 will try that

[butch-albrecht]

Upgraded to .9 today and reran an export from Cost Management and Billing. The data factory pipeline msexports_ExecuteETL was triggered and promptly failed with the same error.
Looking at this more closely - it seems that the billingprofileID does not have a '/' after it in the path prior to the next folder name. Will try changing that. (this is between 682 and xxx in the error message below.

Operation on target For Each Blob failed: Activity failed because an inner activity failed; Inner activity name: Execute, Error: Operation on target Get Existing Parquet Files failed: ADLS Gen2 operation failed for: Operation returned an invalid status code 'NotFound'. Account: 'mystorage'. FileSystem: 'ingestion'. Path: 'Recommendations/2025/04/providers/microsoft.billing/billingaccounts/mybilling account/billingprofiles/firstpartofbillingprofileid682bxxx-fin-ops-daily-disk-reservation-recommendations'. ErrorCode: 'PathNotFound'. Message: 'The specified path does not exist.'. RequestId: '67215082-401f-006a-06ea-ae2cd3000000'. TimeStamp: 'Wed, 16 Apr 2025 16:12:32 GMT'.

[butch-albrecht]

More information - in the Ingestion container in the storage account there is no Recommendations folder (and of course any associated sub-folders). Where in this process should the Recommendations folder be generated in the storage account?

[butch-albrecht]

Well the journey continues. Manually created the Recommendations folder and the CosmoDB and VM Recommendations pipelines worked and populated parquet files under Recommendations. But some other pipelines still don't work.
The BlockBlob Reservation Recommendation export is now complaining that it can't do the column mapping for 'SKU' in the .csv. Well this is true, the .csv doesn't have a 'SKU' column it has a "SKUName' and a 'SKU Properties' column.

Hope this is all worth it when it works.

[flanakin]

Sorry or the delays in seeing this...

Get Existing Parquet Files may fail. That's okay. You can ignore that.

It sounds like you got the data from the msexports container into the ingestion container. I'm very surprised to hear blob reservation recommendations are behaving differently. That implies the data is different. The only thing I can guess is to confirm the exports are all configured to go into the msexports container directly from Cost Management. Can you check that? Also, please confirm the pipeline that's giving you this error to be sure we're on the same page.

[butch-albrecht]

Thanks for looking into this Michael @flanakin .

Will try and provide all the information and validations below.

Information format for Cost Management and Billing exports: what is being exported (frequency) -> target container (format/compression)

• Pricesheets (monthly) -> msexports (parquet/snappy)
• Cost (daily) -> msexports (parquet/snappy)
• BlockBlob Reservation Recommendations (daily) -> msexports (csv/none)
• AppService Reservation Recommendations (daily) -> msexports (csv/none)
• Reservation Transactions (daily) -> msexports (csv/none)
• Reservation Details (daily) -> msexports (csv/none)
• SQL Reservation Recommendations (daily) -> msexports (csv/none)
• CosmoDB Reservation Recommendations (daily) -> msexports (csv/none)
• Reservation Recommendations (daily) -> msexports (csv/none)
• Disk Recommendations (daily) -> msexports (csv/none)

Waited for my Data Factory pipelines to run today. I have the following errors related to teh msexports_ExecuteETL pipeline. I am not calling out any of the msexports_ETL_ingestion errors as I feel they are dependent on the msexports_ExecuteETL errors. All other msexports_ExecuteETL run successfully.

First One:

• Operation on target For Each Blob failed: Activity failed because an inner activity failed; Inner activity name: Execute, Error: Operation on target Get Existing Parquet Files failed: ADLS Gen2 operation failed for: Operation returned an invalid status code 'NotFound'. Account: 'mystorageaccount'. FileSystem: 'ingestion'. Path: 'Recommendations/2025/04/providers/microsoft.billing/billingaccounts/myaccountnumber/billingprofiles/mybillingprofileQHR-fin-ops-daily-sql-reservation-recommendations'. ErrorCode: 'PathNotFound'. Message: 'The specified path does not exist.'. RequestId: '1ead554c-b01f-008a-2cfd-b9af4a000000'. TimeStamp: 'Wed, 30 Apr 2025 18:29:41 GMT'.

  • I believe from your note above that this error, and it's subsequent msexports_ETL_ingestion error I can ignore: Get Existing Parquet Files may fail. That's okay. You can ignore that.
  • I can confirm there is no file in the defined folder. There is a SQL recommendation .csv file in the msexports folder - it has column headings but no data

Second

• Operation on target For Each Blob failed: Activity failed because an inner activity failed; Inner activity name: Execute, Error: Operation on target Convert CSV File failed: ErrorCode=MappingColumnNameNotFoundInSourceFile,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Column 'SKU' specified in column mapping cannot be found in 'billingProfiles/mybillingprofile/QHR-fin-ops-daily-disk-reservation-recommendations/1a42971d-95e0-4764-b512-f32658431213/part0.csv' source file.,Source=Microsoft.DataTransfer.ClientLibrary,'

  • I have checked the .csv file that is in the msexports folder and there is no column called SKU. See screenshot
    

    • note there are no entries in the .csv as there are no recommendations - is that the reason for the failue?

Third

• Operation on target For Each Blob failed: Activity failed because an inner activity failed; Inner activity name: Execute, Error: Operation on target Convert CSV File failed: ErrorCode=MappingColumnNameNotFoundInSourceFile,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Column 'SKU' specified in column mapping cannot be found in 'billingProfiles/mybillingprofile/QHR-fin-ops-daily-BlockBlob-reservation-recommendations/ed18b36f-1ee0-4fe4-ab5f-380a8a666a46/part0.csv' source file.,Source=Microsoft.DataTransfer.ClientLibrary,'

• I have checked the .csv file that is in the msexports folder and there is no column called SKU. See screenshot
  

• similar to above, there are no recommendations - is that what is causing the failure in the pipeline?

Upon review I am wondering if these pipelines are failing because there is no data to actually move. If you could validate that I would appreciated it.

[flanakin]

msexports_ExecuteETL is the parent. We need to focus on the msexports_ETL_ingestion errors first. If you re-run msexports_ExecuteETL, you will see not found errors because those files may have been moved or deleted from a previous run. This is expected. (Although we can definitely check to see if the file was already moved and not show an error 🤔 That would be an improvement by making this less noisy.)

What types of exports are the second and third from? I have a hunch that different types of reservation recommendation exports may be presenting different columns. If so, that sounds like a bug given Cost Management docs say they all use the same columns.

[butch-albrecht]

Sorry for not getting back sooner @flanakin - FinOps is not my only gig :)

The three msexport_ETL_ingestion errors are associated with 'The specified path does not exist. for the msexport runs of:

• fin-ops-daily-sql-reservation-recommendations
• fin-ops-daily-disk-reservation-recommendations
• fin-ops-daily-BlockBlob-reservation-recommendations

The activity name that each run fails on is "Get existing Parquet files" -> "Get Metadata". During the creation of these msexports there is no option to select Parquet - they are all .csv

Other reservation exports are successful, like AppService Reservation recommendations, which is in csv format does work.

The other activities that these runs fail on is Convert to Parquet and Convert to .csv

As for re-running an msexports_ExecuteETL on a run that failed - nothing different

• msexports_ExecuteETL fails with the same errors as above for PathNotFound
  • Operation on target For Each Blob failed: Activity failed because an inner activity failed; Inner activity name: Execute, Error: Operation on target Get Existing Parquet Files failed: ADLS Gen2 operation failed for: Operation returned an invalid status code 'NotFound'. Account: 'storageaccountname'. FileSystem: 'ingestion'. Path: 'Recommendations/2025/05/providers/microsoft.billing/billingaccounts/xxxxaccountID/billingprofiles/xxxxxbilliningIDfin-ops-daily-disk-reservation-recommendations'. ErrorCode: 'PathNotFound'. Message: 'The specified path does not exist.'. RequestId: '05ede927-701f-005e-7825-c51f1b000000'. TimeStamp: 'Wed, 14 May 2025 23:13:12 GMT'.
• msexports_ETL_ingestion launches after the msexports_ExecuteETL finishes and also fails
  • Operation on target Get Existing Parquet Files failed: ADLS Gen2 operation failed for: Operation returned an invalid status code 'NotFound'. Account: 'storageaccountname'. FileSystem: 'ingestion'. Path: 'Recommendations/2025/05/providers/microsoft.billing/billingaccounts/xxxxaccountID/billingprofiles/xxxxxbilliningIDfin-ops-daily-disk-reservation-recommendations'. ErrorCode: 'PathNotFound'. Message: 'The specified path does not exist.'. RequestId: '05ede927-701f-005e-7825-c51f1b000000'. TimeStamp: 'Wed, 14 May 2025 23:13:12 GMT'.

I did not rerun the msexport prior to the re-run.

Looking in the msexport container there is data entry for today as a manifest.json and a part0.csv in the expected folder: msexports\billingProfiles\xxxxxid\fin-ops-daily-disk-reservation-recommendations\xxxxotherid.

Looking in the container target for, there is no container called xxxxxbilliningIDfin-ops-daily-disk-reservation-recommendations

[flanakin]

The "Get Existing Parquet Files" activity fails if it's the first export of the month. That's okay and can be ignored. Are there any other errors? If needed we can get on a call.