Implementing a resolver that resolves worker configurations from specified probing paths #11258

surgupta-msft · 2025-08-19T02:47:46Z

Issue describing the changes in this PR

resolves #10944

Pull request checklist

IMPORTANT: Currently, changes must be backported to the in-proc branch to be included in Core Tools and non-Flex deployments.

Backporting to the in-proc branch is not required
- Otherwise: Link to backporting PR
My changes do not require documentation changes
- Otherwise: Documentation issue linked to PR
My changes should not be added to the release notes for the next release
- Otherwise: I've added my notes to release_notes.md
My changes do not need to be backported to a previous version
- Otherwise: Backport tracked by issue/PR #issue_or_pr
My changes do not require diagnostic events changes
- Otherwise: I have added/updated all related diagnostic events and their documentation (Documentation issue linked to PR)
I have added all required tests (Unit tests, E2E tests)

Additional information

This pull request introduces changes to decouple language workers from the Host release and enable dynamic worker resolution.

Backlog issues - Link
Design doc - Link

Flows covered in this PR -

Resolving Worker Configurations -
- Introduced DynamicWorkerConfigurationResolver, which dynamically resolves worker configurations based on -
  - Environment settings and probing paths
  - Host-Worker compatibility check
  - Leveraging Release channel concept to point to a previous version of the worker
  - Fallback to the old flow if worker is not found at the probing path level
  - Ignoring versions of a worker specified via Hosting config
RPCWorkerConfigFactory -
- Moved some methods to WorkerConfigurationHelper.cs to enable reusing of workers profile evaluation logic between 2 resolvers.
Environment Updates -
- Added a new feature flag FeatureFlagDisableWorkerProbingPaths to disable dynamic resolution feature by users.
- Added 2 Hosting config settings
  - To get list of workers available for resolution.
  - To get list of workers versions to be ignored.
WorkerConfigurationResolverOptionsSetup -
- Configures WorkerConfigurationResolverOptions by getting required values from IConfiguration and IEnvironment such as worker probing paths, release channel, workers available for resolution via hosting config.

Note: The decoupling workers flow is disabled by default in this PR. The flow can be enabled by setting the Hosting config which will be done after completing other relevant backlog items.

src/WebJobs.Script/Workers/Rpc/Configuration/WorkerConfigurationResolverOptionsSetup.cs

src/WebJobs.Script.WebHost/WebHostServiceCollectionExtensions.cs

src/WebJobs.Script/Config/FunctionsHostingConfigOptions.cs

src/WebJobs.Script/Workers/Rpc/Configuration/WorkerConfigurationResolverOptionsSetup.cs

…Resolver

src/WebJobs.Script/Workers/Rpc/Configuration/WorkerConfigurationResolverOptions.cs

src/WebJobs.Script/Workers/Rpc/Configuration/DynamicWorkerConfigurationResolver.cs

src/WebJobs.Script/Workers/Rpc/Configuration/WorkerConfigurationResolverOptionsSetup.cs

Copilot

Pull Request Overview

This PR implements a dynamic worker configuration resolver that enables language workers to be decoupled from the Host release by resolving worker configurations from specified probing paths. It introduces a new DynamicWorkerConfigurationResolver that can dynamically discover and resolve worker configurations based on environment settings, compatibility checks, and release channel concepts, with fallback to the existing resolution mechanism.

Key changes include:

New dynamic worker configuration resolution system with probing paths support
Enhanced worker configuration options with environment-based settings
Refactored shared worker configuration logic into helper classes

Reviewed Changes

Copilot reviewed 28 out of 28 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`DynamicWorkerConfigurationResolver.cs`	Implements the core dynamic resolution logic with probing paths, version selection, and compatibility checks
`WorkerConfigurationResolverOptionsSetup.cs`	Configures resolver options from environment variables and hosting config
`WorkerConfigurationResolverOptions.cs`	Adds new properties for dynamic resolution settings
`WorkerConfigurationHelper.cs`	Extracts shared worker description logic from RpcWorkerConfigFactory
`RpcWorkerConfigFactory.cs`	Refactored to use shared helper methods
`WebHostServiceCollectionExtensions.cs`	Adds service registration logic to choose between resolvers
Various test files	Comprehensive test coverage for new functionality

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

src/WebJobs.Script/Workers/Rpc/Configuration/WorkerConfigurationHelper.cs

src/WebJobs.Script/Workers/Rpc/Configuration/DynamicWorkerConfigurationResolver.cs

src/WebJobs.Script/ScriptConstants.cs

brettsam · 2025-08-28T17:16:11Z

src/WebJobs.Script/Workers/Rpc/Configuration/WorkerConfigurationResolverOptionsSetup.cs

            _fileSystem = fileSystem ?? throw new ArgumentNullException(nameof(fileSystem));
+            _functionsHostingConfigOptions = functionsHostingConfigOptions ?? throw new ArgumentNullException(nameof(functionsHostingConfigOptions));
+            ArgumentNullException.ThrowIfNull(_functionsHostingConfigOptions.Value);


you shouldn't need this -- i don't think this can ever be null (dotnet will always create one when you call .Value)

src/WebJobs.Script/Workers/Rpc/Configuration/WorkerConfigurationResolverOptionsSetup.cs

brettsam · 2025-08-28T18:23:24Z

src/WebJobs.Script/Workers/Rpc/Configuration/WorkerConfigurationHelper.cs

+
+        private static void GetWorkerDescriptionFromAppSettings(RpcWorkerDescription workerDescription, Dictionary<string, string> languageWorkersSettings)
+        {
+            if (languageWorkersSettings.TryGetValue($"{RpcWorkerConstants.LanguageWorkersSectionName}:{workerDescription.Language}:{WorkerConstants.WorkerDescriptionDefaultExecutablePath}", out string defaultExecutablePathSetting) && defaultExecutablePathSetting is not null)


If we parse LanguageWorkerSettings elsewhere like i suggested, this gets simpler.

brettsam · 2025-08-28T18:29:38Z

src/WebJobs.Script/Workers/Rpc/Configuration/DynamicWorkerConfigurationResolver.cs

+            _fileSystem = fileSystem ?? throw new ArgumentNullException(nameof(fileSystem));
+            _profileManager = workerProfileManager ?? throw new ArgumentNullException(nameof(workerProfileManager));
+            _workerConfigurationResolverOptions = workerConfigResolverOptions ?? throw new ArgumentNullException(nameof(workerConfigResolverOptions));
+            ArgumentNullException.ThrowIfNull(workerConfigResolverOptions.CurrentValue);


don't think this can be null -- likely don't need this

src/WebJobs.Script/Workers/Rpc/Configuration/DynamicWorkerConfigurationResolver.cs

brettsam · 2025-08-28T18:47:30Z

src/WebJobs.Script/Workers/Rpc/Configuration/DynamicWorkerConfigurationResolver.cs

+                return false;
+            }
+
+            JsonElement workerConfigJson = WorkerConfigurationHelper.GetWorkerConfigJsonElement(workerConfigPath);


I see us calling this from at least 2 places, meaning we're loading and parsing this json config at least twice. We should only be doing this once. Can you refactor this so it's only parsed once? Is this the same thing that's available in LanguageWorkerSettings in the Options object?

brettsam · 2025-08-28T18:49:09Z

src/WebJobs.Script/Workers/Rpc/Configuration/DynamicWorkerConfigurationResolver.cs

+        /// </summary>
+        /// <param name="workerConfig"> Worker config: { "hostRequirements": [ "test-capability1", "test-capability2" ] }. </param>
+        /// <returns> HashSet { "test-capability1", "test-capability2" }. </returns>
+        private HashSet<string> GetHostRequirementsFromWorker(JsonElement workerConfig)


I don't like that we're operating on JsonElements in here -- I would think that this would be abstracted away somehow and we'd just be given the WorkerConfig as an object to work with.

liliankasem

leaving initial notes, will continue review this afternoon

src/WebJobs.Script/Workers/Rpc/Configuration/DynamicWorkerConfigurationResolver.cs

liliankasem · 2025-08-28T16:36:44Z

src/WebJobs.Script/Workers/Rpc/Configuration/DynamicWorkerConfigurationResolver.cs

+        private readonly ILogger _logger;
+        private readonly IWorkerProfileManager _profileManager;
+        private readonly IFileSystem _fileSystem;
+        private readonly JsonSerializerOptions _jsonSerializerOptions = new() { PropertyNameCaseInsensitive = true };


Do we have global or host wide JsonSerializerOptions we can use? I know PropertyNameCaseInsensitive = true is used a lot throughout the repo so wondering if there is a single one to be shared instead of implementing this every time

src/WebJobs.Script/Workers/Rpc/Configuration/DynamicWorkerConfigurationResolver.cs

test/WebJobs.Script.Tests/Configuration/LanguageWorkerOptionsSetupTests.cs

liliankasem · 2025-08-28T16:56:23Z

src/WebJobs.Script/Workers/Rpc/Configuration/WorkerConfigurationResolverOptionsSetup.cs

        }

        public void Configure(WorkerConfigurationResolverOptions options)
        {
            var configuration = GetRequiredConfiguration();
            options.WorkersRootDirPath = GetWorkersRootDirPath(configuration);
+            options.WorkerRuntime = _environment.GetFunctionsWorkerRuntime();
+            options.ReleaseChannel = _environment.GetPlatformReleaseChannel();
+            options.IsPlaceholderModeEnabled = _environment.IsPlaceholderModeEnabled();


I thought we were moving away from environment for resolving this, shouldn't this be coming from options? Or am I missing something from the discussion about this change?

src/WebJobs.Script.WebHost/WebHostServiceCollectionExtensions.cs

surgupta-msft added 13 commits August 14, 2025 23:45

Changes to add DynamicWorkerConfigResolver

e18fe94

Adding tests

60d4414

Code cleanup

5b2d5ff

Cleanup code

37ad909

Cleanup RpcWorkerConfigFactoryTests

b4ca3db

Tests cleanup

5bc50ee

Tests update

e2349e9

build fix

c86e2e7

Added logic to ignore worker versions specified via config

1e9301d

Added test in SpecializationE2ETest

56d7918

Updated DynamicWorkerConfigResolver

21cfe06

Updated utility

d6883ab

Fixed test. Added comments in resolver and options setup class

3417d99

surgupta-msft requested a review from a team as a code owner August 19, 2025 02:47

surgupta-msft mentioned this pull request Aug 19, 2025

Implement changes needed in the Host to decouple workers from the Host release #11111

Closed

9 tasks

Fixed FunctionsHostingConfigOptionsTest

c8ec8bf

kshyju reviewed Aug 19, 2025

View reviewed changes

Code improvement, added logs

4b344e4

kshyju reviewed Aug 20, 2025

View reviewed changes

surgupta-msft commented Aug 20, 2025

View reviewed changes

src/WebJobs.Script/Config/FunctionsHostingConfigOptions.cs Outdated Show resolved Hide resolved

surgupta-msft commented Aug 20, 2025

View reviewed changes

src/WebJobs.Script/Workers/Rpc/Configuration/WorkerConfigurationResolverOptionsSetup.cs Outdated Show resolved Hide resolved

surgupta-msft added 5 commits August 20, 2025 18:47

Addressing PR feedback

61db33b

Adding token extensions methods

12c95da

Refactored token extensions class

15bbd31

Updating TokenExtensions

47d86f1

Reverting tokenExtensions changes

7ce1414

surgupta-msft requested review from brettsam, liliankasem, soninaren and kshyju August 21, 2025 20:53

surgupta-msft added 7 commits August 22, 2025 21:52

Merge branch 'dev' into surgupta/dynamic-worker-resolver

17c4209

Addressing PR feedback

1cbc0a7

Added tests

74d77b2

Addressing PR feedback, code cleanup

6f3cb9e

Added tests for RPCWorkerConfigFactory

d389a5e

Cleanup WorkerConfigurationResolverOptionsSetup

a467a6e

Minor renaming, added tests

5186c44

surgupta-msft requested review from kshyju and soninaren August 25, 2025 18:45

surgupta-msft added 5 commits August 25, 2025 16:33

Minor code improvement, comments update in DynamicWorkerConfiguration…

0e2056a

…Resolver

Added tests in DynamicWorkerConfigurationResolverTests

3b5428c

Logging and tests update

e91d86a

Merge branch 'dev' into surgupta/dynamic-worker-resolver

adb8980

Added release notes

8887ee1

surgupta-msft changed the base branch from feature/workers-decoupling to dev August 26, 2025 15:35

soninaren reviewed Aug 26, 2025

View reviewed changes

src/WebJobs.Script/Workers/Rpc/Configuration/WorkerConfigurationResolverOptions.cs Show resolved Hide resolved

soninaren reviewed Aug 26, 2025

View reviewed changes

surgupta-msft added 3 commits August 27, 2025 23:28

Addressing PR feedback, variable renaming

27d80d6

Merge branch 'dev' into surgupta/dynamic-worker-resolver

236ec49

Updating reelase notes

8e91769

Copilot AI review requested due to automatic review settings August 28, 2025 04:30

Copilot AI reviewed Aug 28, 2025

View reviewed changes

src/WebJobs.Script/Workers/Rpc/Configuration/WorkerConfigurationHelper.cs Show resolved Hide resolved

src/WebJobs.Script/Workers/Rpc/Configuration/DynamicWorkerConfigurationResolver.cs Show resolved Hide resolved

Added log and summary comment

3b1cfde

surgupta-msft requested a review from soninaren August 28, 2025 05:06

brettsam reviewed Aug 28, 2025

View reviewed changes

liliankasem reviewed Aug 28, 2025

View reviewed changes

kshyju reviewed Aug 28, 2025

View reviewed changes

src/WebJobs.Script.WebHost/WebHostServiceCollectionExtensions.cs Show resolved Hide resolved

surgupta-msft added 3 commits August 29, 2025 01:14

Addressing PR feedback

b964261

Merge branch 'dev' into surgupta/dynamic-worker-resolver

2afb262

Resolving release notes conflict

a680663

Implementing a resolver that resolves worker configurations from specified probing paths #11258

Are you sure you want to change the base?

Implementing a resolver that resolves worker configurations from specified probing paths #11258

Conversation

surgupta-msft commented Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue describing the changes in this PR

Pull request checklist

Additional information

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

brettsam Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

brettsam Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

brettsam Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

brettsam Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

brettsam Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

liliankasem left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

liliankasem Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

liliankasem Aug 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

surgupta-msft commented Aug 19, 2025 •

edited

Loading