Improved metadata binding parsing and validation. #11101

kshyju · 2025-05-31T00:44:24Z

Improving how metadata binding is being read and used. Handles both worker indexing and host indexing cases.

Pull request checklist

IMPORTANT: Currently, changes must be backported to the in-proc branch to be included in Core Tools and non-Flex deployments.

Backporting to the in-proc branch is not required
- Otherwise: Link to backporting PR
My changes do not require documentation changes
- Otherwise: Documentation issue linked to PR
My changes should not be added to the release notes for the next release
- Otherwise: I've added my notes to release_notes.md
My changes do not need to be backported to a previous version
- Otherwise: Backport tracked by issue/PR #issue_or_pr - will follow
My changes do not require diagnostic events changes
- Otherwise: I have added/updated all related diagnostic events and their documentation (Documentation issue linked to PR)
I have added all required tests (Unit tests, E2E tests)

kshyju · 2025-05-31T00:45:45Z

src/WebJobs.Script.WebHost/Extensions/FunctionMetadataExtensions.cs

@@ -175,7 +176,7 @@ private static async Task<JObject> GetFunctionConfig(FunctionMetadata metadata,

        private static async Task<JObject> GetFunctionConfigFromFile(string path)
        {
-            return JObject.Parse(await FileUtility.ReadAsync(path));
+            return JObject.Parse(Sanitizer.Sanitize(await FileUtility.ReadAsync(path)));


synctrigger endpoint and /functions/{functionname} endpoint code reads the file if it is present. (Valid for custom handlers /non worker indexing use case).

kshyju · 2025-05-31T00:47:24Z

src/WebJobs.Script/Host/WorkerFunctionMetadataProvider.cs

@@ -257,7 +258,7 @@ internal FunctionMetadata ValidateBindings(IEnumerable<string> rawBindings, Func

            foreach (string binding in rawBindings)
            {
-                var deserializedObj = JsonConvert.DeserializeObject<JObject>(binding, _dateTimeSerializerSettings);
+                var deserializedObj = JsonConvert.DeserializeObject<JObject>(Sanitizer.Sanitize(binding), _dateTimeSerializerSettings);


This method gets executed in worker indexing use case.

surgupta-msft · 2025-06-05T06:05:34Z

test/WebJobs.Script.Tests/SanitizerTests.cs

@@ -41,6 +41,9 @@ public class SanitizerTests
        [InlineData("test,aaa://aaa:[email protected]:1111,test", "test,[Hidden Credential],test")]
        [InlineData(@"some text abc://abc:[email protected]:1111 some text abc://abc:[email protected]:1111 text", @"some text [Hidden Credential] some text [Hidden Credential] text")]
        [InlineData(@"some text abc://abc:[email protected]:1111 some text AccountKey=heyyyyyyy text", @"some text [Hidden Credential] some text [Hidden Credential]")]
+        [InlineData("""{"queueName":"my-q-items","connection":"MyConnection","type":"queueTrigger","name":"qTrigger1","direction":"in"}""", "{\"queueName\":\"my-q-items\",\"connection\":\"MyConnection\",\"type\":\"queueTrigger\",\"name\":\"qTrigger1\",\"direction\":\"in\"}")]
+        [InlineData("""{"queueName":"my-q-items","connection":"DefaultEndpointsProtocol=https;AccountName=a;AccountKey=b/c==;EndpointSuffix=core.windows.net","type":"queueTrigger","name":"queueTrigger1","direction":"in"}""", "{\"queueName\":\"my-q-items\",\"connection\":\"[Hidden Credential]\",\"type\":\"queueTrigger\",\"name\":\"queueTrigger1\",\"direction\":\"in\"}")]
+


Could this be a breaking change in some scenarios? Let's say, queue name is "key=myqueue" then we will replace it with "[Hidden Credential]". Will the function stop executing since we changed the queue name or is it only the Sync trigger response that is being sanitized (and it won't break any other component that depends on Sync trigger response)?

The functions are already in a broken state (not executing anything). The extension listener code throws when it is not able to resolve a valid app setting value from the value of these attributes (example code for service bus extension here). So this is not breaking anything.

We are sanitizing the entire binding object for all scenarios and not just the exception cases. If any extension introduces (or already has) a binding property with name "password=" or "key=" (may not be good examples), they will be modified to [Hidden Credential] and the app can break.

In terms of binding values, I checked that the queue name and connection setting name cannot have "=" (but other bindings value may have "="), but they can have other characters like ":", so may be, we need to ensure that the Sanitizer class should never have tokens such as "key:", otherwise we will modify that value to [Hidden Credential]. A unit test should work for this scenario.

I am not sure I follow the last part of the comment. But for inputs like "one:two;three", Sanitizer returns the same value output.

Input Output

one:two;three one:two;three

one:a;two=three one:a;two=three

foo=bar foo=bar

If you meant something else, can you share an example input?

Today, key=myvalue will return [HiddenCredential]. If in future, someone adds a token "key:" in Sanitizer, then
key:myvalue will also return [HiddenCredential]. This key:myvalue is a valid connection setting name. Let's also discuss this offline to make sure we don't have any confusion here.

Right because : is the connection separator, so it won't be unexpected for connection names to include it. Now, if customer's have a section literally called "key" or "secret", or something else covered by our sanitizer that is another question.

Maybe we should reconsider the approach taken here and sanitize only the binding values when we serve the response from the /functions/ endpoints. Let's follow up offline.

kshyju · 2025-06-30T17:40:54Z

Had offline sync with @surgupta-msft and agreed to mask only the "connection" binding value.

surgupta-msft · 2025-07-02T21:36:40Z

src/WebJobs.Script.WebHost/Extensions/FunctionMetadataExtensions.cs

+
+            if (jObject.TryGetValue("bindings", StringComparison.OrdinalIgnoreCase, out JToken bindingsToken) && bindingsToken is JArray bindings)
+            {
+                var bindingObjects = bindings.OfType<JObject>().ToList();


Is it possible to avoid converting JArray to List here? In HostFunctionMetadataProvider, we are able to do similar thing with JArray.

surgupta-msft · 2025-07-02T21:41:07Z

src/WebJobs.Script/Host/MetadataJsonHelper.cs

+            {
+                if (propertyNames.Contains(prop.Name, StringComparer.OrdinalIgnoreCase))
+                {
+                    jsonObject[prop.Name] = Sanitizer.Sanitize(prop.Value.ToString());


Connection value could be null. So, I believe we need a null check for prop.Value.

surgupta-msft · 2025-07-02T21:49:40Z

src/WebJobs.Script/Host/WorkerFunctionMetadataProvider.cs

@@ -257,8 +258,8 @@ internal FunctionMetadata ValidateBindings(IEnumerable<string> rawBindings, Func

            foreach (string binding in rawBindings)
            {
-                var deserializedObj = JsonConvert.DeserializeObject<JObject>(binding, _dateTimeSerializerSettings);


We need to still leverage _dateTimeSerializerSettings. If we don't use it, it will break some datetime stuff.

For example, current serializer setting says to not parse the datetime. If we don't use it, it will start parsing it. I added an example in the test class below. I worked on similar issue before - Azure/azure-functions-dotnet-worker#2442

surgupta-msft · 2025-07-02T21:53:26Z

test/WebJobs.Script.Tests/HostFunctionMetadataProviderTests.cs

+                                      ""type"": ""queue"",
+                                      ""direction"": ""out"",
+                                      ""queueName"": ""test-output-node"",
+                                      ""connection"": ""MyConnection""


Can we check for connection null and empty too?

surgupta-msft · 2025-07-02T22:04:35Z

test/WebJobs.Script.Tests/WorkerFunctionMetadataProviderTests.cs

+            var functionMetadata = new FunctionMetadata();
+            List<string> rawBindings =
+            [
+                """{"type": "queueTrigger","name": "myQueueItem","direction": "in","queueName": "test-input-node","connection": "DefaultEndpointsProtocol=https;AccountName=a;AccountKey=b/c==;EndpointSuffix=core.windows.net"}""",


Add a test with queue name (or any other field) as "2024-04-29T11:35:00+08:00". You will see that the datetime is parsed to some other value. This won't happen if we use

new JsonSerializerSettings { DateParseHandling = DateParseHandling.None }

Improved metadata binding parsing and validation.

bb445bb

kshyju requested a review from a team as a code owner May 31, 2025 00:44

kshyju commented May 31, 2025

View reviewed changes

surgupta-msft reviewed Jun 5, 2025

View reviewed changes

RohitRanjanMS approved these changes Jun 6, 2025

View reviewed changes

kshyju added 2 commits June 10, 2025 06:53

Merge branch 'dev' into shkr/placeholder_start

432e545

Adding another test input to check : in binding value.

25e9506

kshyju requested a review from surgupta-msft July 1, 2025 00:28

kshyju force-pushed the shkr/placeholder_start branch from ed60f35 to c0129c5 Compare July 1, 2025 00:44

Sanitize only subset of bindings.

dff555b

kshyju force-pushed the shkr/placeholder_start branch from b317aee to dff555b Compare July 1, 2025 14:59

surgupta-msft reviewed Jul 2, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Improved metadata binding parsing and validation. #11101

Improved metadata binding parsing and validation. #11101

Uh oh!

kshyju commented May 31, 2025

Uh oh!

kshyju May 31, 2025

Uh oh!

kshyju May 31, 2025

Uh oh!

surgupta-msft Jun 5, 2025 •

edited

Loading

Uh oh!

kshyju Jun 10, 2025

Uh oh!

surgupta-msft Jun 11, 2025 •

edited

Loading

Uh oh!

kshyju Jun 11, 2025

Uh oh!

surgupta-msft Jun 11, 2025 •

edited

Loading

Uh oh!

jviau Jun 18, 2025

Uh oh!

kshyju Jun 18, 2025

Uh oh!

kshyju commented Jun 30, 2025

Uh oh!

surgupta-msft Jul 2, 2025

Uh oh!

surgupta-msft Jul 2, 2025

Uh oh!

surgupta-msft Jul 2, 2025

Uh oh!

surgupta-msft Jul 2, 2025

Uh oh!

surgupta-msft Jul 2, 2025

Uh oh!

Uh oh!

Input	Output
one:two;three	one:two;three
one:a;two=three	one:a;two=three
foo=bar	foo=bar

Improved metadata binding parsing and validation. #11101

Are you sure you want to change the base?

Improved metadata binding parsing and validation. #11101

Uh oh!

Conversation

kshyju commented May 31, 2025

Pull request checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

surgupta-msft Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

surgupta-msft Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

surgupta-msft Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kshyju commented Jun 30, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

surgupta-msft Jun 5, 2025 •

edited

Loading

surgupta-msft Jun 11, 2025 •

edited

Loading

surgupta-msft Jun 11, 2025 •

edited

Loading