TimeOutException in HttpTrigger Function when DD_TRACE_ENABLED set to true

[ansonpinhero]

I'm encountering a TimeoutException for an HttpTrigger function while running the Function App with Datadog instrumentation in a docker container. Notably, these errors did not occur when the config - DD_TRACE_ENABLED set to false.

I've created a github issue in azure-functions-dotnet-worker project. Please refer to this issue - Azure/azure-functions-dotnet-worker#2910

[kenthellck]

We noticed similar behavior in our application, as mentioned by @ansonpinhero. This issue is occurring in .NET 8 isolated and has resulted in numerous timeouts, causing our application to fail. When we disable Datadog, it works perfectly.
We urgently need a resolution from Datadog, as this issue is preventing us from delivering our applications to production. As a workaround, we are now attempting to implement Azure Application Insights, which functions correctly. And thinking to move away from DataDog.
Datadog needs to improve its APM capabilities to address such issues effectively.

[ashan1992]

Likewise, we are encountering a similar issue with .NET 8 Azure Functions. and there is a huge performance degrade when we enable DD_TRACE. Has there been any update from the Datadog team regarding this?

[bouwkast]

We are actively looking into this at the moment.

We will follow back as we get more information.

[bouwkast]

As an update, we've been able to reproduce the performance issue and will continue to look into the cause and provide a remediation for it.

We don't see the issue if the .NET project uses .ConfigureFunctionsWorkerDefaults() instead of .ConfigureFunctionsWebApplication(), so it may be an issue with our ASP.NET Core integration - however, disabling that integration doesn't appear to address the issue.

We're continuing to investigate this and will follow up again.

[ansonpinhero]

Thank you for the update @bouwkast

[bouwkast]

Thank you all for your patience with this as we've worked through it.

We believe we've identified the root cause of the timeouts affecting the ASP.NET Core integration with the isolated Azure Functions worker model. A fix is in progress and we're adding addtional testing around this.

You can expect a pull request shortly; I'll update this thread as soon as it is ready.

[ashan1992]

Thanks for the update @bouwkast

[bouwkast]

Hi, we've opened a PR for this here: #6567

Will let you know when it is merged and then released.