Use FinOps Hub data for anomalies and cost pattern detection

[josesarsa]

This discussion is to explore how we can more effectively leverage the data within our FinOps Hub to proactively identify cost inefficiency opportunities. I see two opportunities:

1. Cost Pattern Analysis: Identifying recurring patterns in our spending that might indicate sustained inefficiencies. Examples could include:

   • Consistently underutilized resources (e.g., VMs, databases, storage) across specific teams or projects. Some of the infrastructure has a cost for provisioning and a separated one for consumption (app configuraiton, nat gateways, fw, etc) to extract a cost pattern
   • Cyclical workloads that aren't being effectively optimized (e.g., not shutting down non-prod during off-hours, not using auto-scaling appropriately).
   • Suboptimal storage tiering or lifecycle management.

2. Advanced Anomaly Detection & Filtering: Moving beyond basic budget alerts to detect more nuanced or specific anomalies that signal waste. This includes:

   • Sudden spikes in cost for a particular service, resource tag, or account that isn't immediately obvious.
   • Significant deviations from historical spending trends for specific applications or environments.
   • Filter anomalies beyond what native budget anomalies does. False positives reduction

anything to go in this direction?

[flanakin]

Data Explorer has some built-in anomaly detection capabilities. I haven't used it much so I can't shed too much light on it, but that's what we're planning to use when we get to anomaly management. There are a few other things we have ahead of that right now.

But what sort of responses are you looking for right now? I would say we should add separate feature requests for anything you're interested in so we can see how they bubble up. We try to pick one of the top 10 feature requests (based on 👍 votes) each month. We can also work with you if you need guidance on how to implement and contribute something, if you're open to that.

But the bottom line is we want to go this direction for sure. Our latest foray into this space is using GitHub Copilot to identify trends and anomalies. You'll see that in the 0.11 release.

[josesarsa]

This is an example, right?

https://techcommunity.microsoft.com/blog/azuredataexplorer/multivariate-anomaly-detection-in-azure-data-explorer/3689616

I would start with that anomaly feature, for example. Another feature would be to identity the most convenient storage account to apply lifecycle management policies based on cost. Also, it could be interesting identify the lifecycle not well tune (early detection penalties, more cost of operations than savings in store, etc).
I will raise those as a feature request and some queries to start with. But if you have any other relevant optimizations, let me know