feat: Add extra search index fields to Knowledge Agent response

[taylorn-ai]

Purpose

As discussed in #2569, adds an optional “agentic reference hydration” feature to agentic retrieval so the app can fetch the full documents for the agent’s references (all index fields, not just the semantic fields). This enables richer source metadata (e.g., sourcefile, category, scores) and more accurate citations, while keeping the legacy behaviour as default.

Key points:

• New env flag: ENABLE_AGENTIC_REF_HYDRATION (default false).
• When enabled, references returned by the Knowledge Agent are hydrated via a follow-up Azure AI Search query using a composed search.in filter over unique doc_keys.
• Preserves existing behaviour when disabled: builds Document objects from source_data embedded in references.
• Injects the agent’s per-activity search query into each Document (via search_agent_query) for better observability.
• Deterministic ordering:
  • Default: preserves agent order.
  • results_merge_strategy == "interleaved": sorts by reference.id (ascending) to interleave across activities.
• Infra wiring: Bicep + parameters propagate the new flag to App Service env vars.
• Documentation updated to describe the optional hydration behaviour and how to enable it.
• Comprehensive tests added covering:
  • Hydrated vs non-hydrated paths
  • Interleaved sorting
  • Deduplication of duplicate doc_keys
  • Respecting top
  • Missing/empty doc_keys
  • Empty hydration results

Does this introduce a breaking change?

When developers merge from main and run the server, azd up, or azd deploy, will this produce an error?
If you're not sure, try it out on an old environment.

[ ] Yes
[x] No

Why:

• The new behaviour is gated behind ENABLE_AGENTIC_REF_HYDRATION which defaults to false.
• Constructors were extended with a defaulted hydrate_references: bool = False and all call sites updated accordingly.
• Infra adds a new parameter with a default of false; existing environments deploy without change.

Does this require changes to learn.microsoft.com docs?

This repository is referenced by this tutorial
which includes deployment, settings and usage instructions. If text or screenshot need to change in the tutorial,
check the box below and notify the tutorial author. A Microsoft employee can do this for you if you're an external contributor.

[ ] Yes
[x] No

Type of change

[ ] Bugfix
[x] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[x] Documentation content changes
[ ] Other... Please describe:

Code quality checklist

See CONTRIBUTING.md for more details.

• The current tests all pass (python -m pytest).
• I added tests that prove my fix is effective or that my feature works
• I ran python -m pytest --cov to verify 100% coverage of added lines
• I ran python -m mypy to check for type errors
• I either used the pre-commit hooks or ran ruff and black manually on my code.

[taylorn-ai]

@microsoft-github-policy-service agree

[pamelafox]

Thank you @taylorn-ai for the great contribution! I would like @mattgotteiner from the AI Search team to review this, given his expertise with agentic retrieval.

[taylorn-ai]

I ran black on the files I changed, do you want me to re-run on everything? Or do you want to do that?

[taylorn-ai]

My bad, seems I didn't run black on the tests directory.

[taylorn-ai]

Apologies, I thought I had updated the test snapshots. Also @pamelafox, FYI, the pre-commit hooks aren't working. I ran it manually and it changed some other files that I hadn't edited, so I left them as they were.

[pamelafox]

@taylorn-ai Interesting, sometimes that happens when the formatter changes its rules. I do run them locally but I might need to explicitly re-install the pre-commit hooks to see if formatter rules changes/

[taylorn-ai]

Two of the tests failed due to a network issue, is that common? Seems odd. At least it's not my fault 😀