Issue with OrchestrationMetadata 'SerializedOutput' for large messages

[ghost]

Description

In our .Net 8 Durable Functions app, we sometimes have large Orchestration output. This was working ok in Microsoft.Azure.Functions.Worker.Extensions.DurableTask 1.1.7, but after updating to 1.2.x, the app crashes with a JSON deserialization error when invoking OrchestrationMetadata.ReadOutputAs<>().

After debugging we discovered that the 'SerializedOutput' property value contains the large message blob URL instead of the actual JSON data - example: http://127.0.0.1:10000/devstoreaccount1/hub-largemessages/FetchJob@y8m81oulAKVO7pwg80XQwUNyNM/history-0000000000000006-ExecutionCompleted-5FE4514C-Result.json.gz

Expected behavior

Expectation is that the 'SerializedOutput' contains the output JSON and not the blob reference URL

Actual behavior

'SerializedOutput' contains a URL pointing to the large message blob

Relevant source code snippets

jobResult = metadata.ReadOutputAs<JsonElementWrapper>();

Known workarounds

Rollback Microsoft.Azure.Functions.Worker.Extensions.DurableTask to version 1.1.7

App Details

• Durable Functions Worker extension version v2.0:
• Azure Functions runtime version 2.0:
• C# - .NET 8:

[cgillum]

The behavior you're seeing is normally controlled by the fetchLargeMessageDataEnabled setting in host.json (though it looks like it's not currently documented).

{
  "extensions": {
    "durableTask": {
      "hubName": "MyTaskHub",
      "storageProvider": {
        "fetchLargeMessageDataEnabled": true,
      }
    }
  }
}

The default value should be true. By chance are you overriding this value in your settings?

As far as how/whether it's related to just the version change, it doesn't look like this default has ever changed. The other possibility is that there is some regression in the DurableTask.AzureStorage dependency that the extension internally uses that's causing a change in behavior.

Adding @bachuv and @nytian in case they may have ideas about this.

[ghost]

Thanks for the feedback. No, we aren't changing that configuration value - this is our current host.json:

{
  "version": "2.0",
  "functionTimeout": "00:30:00",
  "logging": {
    "logLevel": {
      "Host.Triggers": "Warning",
      "Host.Function.Console": "Warning",
      "Function": "Warning"
    },
    "applicationInsights": {      
      "samplingSettings": {
        "isEnabled": false
      }
    }
  },
  "extensions": {
    "durableTask": {
      "hubName": "%TaskHubName%",
      "storageProvider": {
        "maxQueuePollingInterval": "00:00:01",
        "partitionCount": 12
      }
    },
    "http": {
      "maxOutstandingRequests": 500,
      "maxConcurrentRequests": 200
    }
  }
}

We could try explicitly setting it to "true", although if that's already the default I'm guessing it won't make a difference?

[ghost]

We tried explicitly adding that flag to host.json but saw the exact same error as expected.

      "storageProvider": {
        "maxQueuePollingInterval": "00:00:01",
        "partitionCount": 12,
        "fetchLargeMessageDataEnabled": true
      }

[SoleCode]

@cgillum any update regarding this? we are seeing the same problem. I had to roll back to 1.1.7 form 1.2.2 due to same problem. I have done your recommendation, and got the same error.

[cgillum]

Seems like there is some kind of regression that was introduced in this scenario. My guess is that it's related to our migration to DurableTask.AzureStorage v2.x (which is what gets used in the 1.2.x versions of the Durable extension in .NET Isolated). We'll investigate this and work on a fix.

@andystaples let's add this scenario to the list of end-to-end tests we need to automate.