Support custom DNS IP --dns-service-ip

[ottodeng]

Version

Karpenter Version: v0.0.0

Kubernetes Version: v1.0.0

Expected Behavior

the pod on new karpenter node, /etc/resolv.conf is 10.0.0.10. not follow az cli create --dns-service-ip 10.97.96.10.

this issue will cause cannot use custom vnet-subnet-id, the pod on karpenter node can not get reponse from default dns nameserver 10.0.0.10.

Actual Behavior

default nodepool

Steps to Reproduce the Problem

each AKS NAP cluster or self-host can reproduce this.

Resource Specs and Logs

(base) [email protected]:/Users/ottodeng/aks/self-host $ kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-deployment-5f7bbf8bb-bkw8z 1/1 Running 0 18m 10.244.4.169 aks-t4-spot-rjz89
nginx-deployment-5f7bbf8bb-hvkvn 1/1 Running 0 18m 10.244.3.132 aks-t4-spot-xg75d
nginx-nodepool-5c64bc7fd8-xl92s 1/1 Running 0 18m 10.244.1.82 aks-nodepool1-32105842-vmss000001
nginx-nodepool-5c64bc7fd8-xs2mr 1/1 Running 0 18m 10.244.2.65 aks-nodepool1-32105842-vmss000000
(base) [email protected]:/Users/ottodeng/aks/self-host $ kubectl exec -it nginx-nodepool-5c64bc7fd8-xl92s cat /etc/resolv.conf
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
search default.svc.cluster.local svc.cluster.local cluster.local kdgjdzanqb1uxn5hrh45i34rib.ix.internal.cloudapp.net
nameserver 10.97.96.10
options ndots:5
(base) [email protected]:/Users/ottodeng/aks/self-host $ kubectl exec -it nginx-deployment-5f7bbf8bb-bkw8z cat /etc/resolv.conf
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
search default.svc.cluster.local svc.cluster.local cluster.local kdgjdzanqb1uxn5hrh45i34rib.ix.internal.cloudapp.net
nameserver 10.0.0.10
options ndots:5
(base) [email protected]:/Users/ottodeng/aks/self-host $

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[Bryce-Soghigian]

im working on a fix for --dns-service-ip here #329. For now you will only be able to use the default values for --dns-service-ip until that change gets in.

[ottodeng]

im working on a fix for --dns-service-ip here #329. For now you will only be able to use the default values for --dns-service-ip until that change gets in.

Thanks Bryce, will it merge into main in these days?

[NMFR]

Hi,

Any update on this or any workaround?

I am also trying to use Karpenter with a custom subnet and the pod DNS is broken.

[gorositopablo]

Thanks @Bryce-Soghigian . Unfortunately, we have run into the same issue.

Does anyone have any workaround for it?

Thanks

[ppodevlabs]

Any update on this? This is blocking us from using Karpenter in AKS

[bedij03]

Hi,

We are also having the same issue.
Any update on when this is likely to be fixed ?

Thanks

[tesharp]

We have setup latest azure karpenter on cilium byocni cluster and got it working using some redirect policy and external secret for bootstrap token rotation. Also using secret reloader to automatically reload karpenter so it uses the latest token, otherwise it just failed to create nodes after a while. Seems to be running ok now with 0.7.1 which fixed problem with deleting vm's that did not exist.

Redirect policy to redirect dns requests on 10.0.0.10 to node-local-dns:

apiVersion: cilium.io/v2
kind: CiliumLocalRedirectPolicy
metadata:
  name: dns-local-redirect
  namespace: kube-system
spec:
  redirectFrontend:
    addressMatcher:
      ip: "10.0.0.10"
      toPorts:
      - name: dns
        port: "53"
        protocol: UDP
      - name: dns-tcp
        port: "53"
        protocol: TCP
  redirectBackend:
    localEndpointSelector:
      matchLabels:
        app.kubernetes.io/instance: node-local-dns
        app.kubernetes.io/name: node-local-dns
    toPorts:
    - name: dns
      port: "53"
      protocol: UDP
    - name: dns-tcp
      port: "53"
      protocol: TCP

External secret to create a secret with bootstrap token

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: karpenter-bootstrap-token
  namespace: kube-system
spec:
  refreshInterval: 5m
  secretStoreRef:
    kind: SecretStore
    name: k8s-store-kube-system
  target:
    name: karpenter-bootstrap-token
    template:
      engineVersion: v2
      data:
        token: |-
          {{- $id := "" }}
          {{- $secret := "" }}
          {{- range $key, $token := . }}
            {{- $json := fromJson $token }}
            {{- $id = index $json "token-id" }}
            {{- $secret = index $json "token-secret" }}
          {{- end -}}
          {{ $id }}.{{ $secret }}
  dataFrom:
  - find:
      name:
        regexp: "bootstrap-token-.*"