upstream sent invalid header: "x-correlation-id\x20..." while reading response header from upstream

[sanjay-raj5]

All,
I am using ingress-nginx-controller-v1.12.0 in my cluster.Earlier we were on 1.1.0 version.
Error:
2025/01/21 14:14:15 [error] 3509#3509: *8465327 upstream sent invalid header: "x-correlation-id\x20..." while reading response header from upstream, client: 172.xx.xx.xxx, server: api.test.xyz.com, request: "POST /<my_path> HTTP/1.1", upstream: "http://172.xx.xx.xxx:8081/<my_path>", host: "api.test.xyz.com"

When checking the application, it gives me 200 SUCCESS and even tried to port-forward svc and that is also working, but with ingress, getting this error post upgrade.

Upon reverting back to 1.1.0, the same payload gives me 200 SUCCESS.

So suspecting that ingress brought in strict header validations from the older version to newer version
Please help me on the same

NGINX Ingress controller version (exec into the pod and run /nginx-ingress-controller --version):

NGINX Ingress controller
Release: v1.12.0
Build: ba73b2c
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.25.5

Kubernetes version (use kubectl version): 1.30.8-eks-2d5f260

Environment: Nonprod

• Cloud provider or hardware configuration: AWS

Controller was installed using the manifest files from https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.12.0/deploy/static/provider/aws/deploy.yaml

We changed only the config map and some annotations in svc to create a private nlb
enable-underscores-in-headers: "true"
ignore-invalid-headers: "false"
proxy-connect-timeout: "120"
proxy-read-timeout: "120"
proxy-send-timeout: "120"
ssl-redirect: "false"

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[longwuyuan]

/remove-kind bug
/kind support

Please read the release notes and check if any changes are related to your use case. Maybe snippets or validatons etc.

[sanjay-raj5]

@longwuyuan , there is nothing directly saying about this validations on headers, but from nginx version 1.21.1, there is a strict validation on headers as it rejects invalid headers. But I am not able to find which previous version of controller is equivalent to 1.21.1

Starting with ​version 1.21.1, nginx always rejects invalid headers, such as the one with space in a header name:

*) Change: now nginx always returns an error if spaces or control
   characters are used in a header name.

This applies to both headers received from clients, and headers got from upstream servers.

It looks like your backend server returns responses with an invalid header. Previously this worked, because nginx simply passed such headers to the client, but now such responses are rejected by nginx. To fix this, you have to fix the backend server to return correct responses.

[longwuyuan]

The point was to know how that header was being set (or where in your backend code it was being set), if not in a snippet.

Version 1.1.0 of the controller is just too old and can not be supported so I am sure the version of nginx there is just too old and has been updated to a much recent version in v1.12.0 ( it will be nginx v1.25.5).

So I don't think there is any action item for the project.