Disable EKS auto mode fails

[project0]

Description

Disabling eks auto mode on an existing cluster fails. Apparently there are some related changes and issues that claimed to solve and reverted it again. As all related issues are closed i am have decided to raise a new issues.

• disabling auto mode #3268
• fix: Correct Auto Mode disable #3253
• fix: Ability to desactivate Auto Mod #3252
• Terraform apply fails after importing EKS cluster with Auto Mode disabled #3236

Versions

• Module version [Required]:

• Terraform version:
  Terraform v1.10.2

• Provider version(s):

14:44:13.210 STDOUT terraform: Providers required by configuration:
14:44:13.210 STDOUT terraform: .
14:44:13.210 STDOUT terraform: ├── provider[registry.terraform.io/hashicorp/aws] >= 5.9.0
14:44:13.210 STDOUT terraform: ├── module.irsa_external-dns
14:44:13.210 STDOUT terraform: │   └── provider[registry.terraform.io/hashicorp/aws] >= 4.0.0
14:44:13.210 STDOUT terraform: ├── module.karpenter
14:44:13.210 STDOUT terraform: │   └── provider[registry.terraform.io/hashicorp/aws] >= 5.81.0
14:44:13.210 STDOUT terraform: ├── module.vpc
14:44:13.210 STDOUT terraform: │   └── provider[registry.terraform.io/hashicorp/aws] >= 5.46.0
14:44:13.210 STDOUT terraform: ├── module.eks
14:44:13.210 STDOUT terraform: │   ├── provider[registry.terraform.io/hashicorp/aws] >= 5.81.0
14:44:13.210 STDOUT terraform: │   ├── provider[registry.terraform.io/hashicorp/tls] >= 3.0.0
14:44:13.210 STDOUT terraform: │   ├── provider[registry.terraform.io/hashicorp/time] >= 0.9.0
14:44:13.210 STDOUT terraform: │   ├── module.eks_managed_node_group
14:44:13.210 STDOUT terraform: │       ├── provider[registry.terraform.io/hashicorp/aws] >= 5.81.0
14:44:13.210 STDOUT terraform: │       └── module.user_data
14:44:13.210 STDOUT terraform: │           ├── provider[registry.terraform.io/hashicorp/cloudinit] >= 2.0.0
14:44:13.210 STDOUT terraform: │           └── provider[registry.terraform.io/hashicorp/null] >= 3.0.0
14:44:13.210 STDOUT terraform: │   ├── module.fargate_profile
14:44:13.210 STDOUT terraform: │       └── provider[registry.terraform.io/hashicorp/aws] >= 5.81.0
14:44:13.210 STDOUT terraform: │   ├── module.kms
14:44:13.210 STDOUT terraform: │       └── provider[registry.terraform.io/hashicorp/aws] >= 4.33.0
14:44:13.210 STDOUT terraform: │   └── module.self_managed_node_group
14:44:13.210 STDOUT terraform: │       ├── provider[registry.terraform.io/hashicorp/aws] >= 5.81.0
14:44:13.210 STDOUT terraform: │       └── module.user_data
14:44:13.210 STDOUT terraform: │           ├── provider[registry.terraform.io/hashicorp/cloudinit] >= 2.0.0
14:44:13.210 STDOUT terraform: │           └── provider[registry.terraform.io/hashicorp/null] >= 3.0.0
14:44:13.210 STDOUT terraform: ├── module.irsa_argocd
14:44:13.210 STDOUT terraform: │   └── provider[registry.terraform.io/hashicorp/aws] >= 4.0.0
14:44:13.210 STDOUT terraform: └── module.irsa_aws-load-balancer-controller
14:44:13.210 STDOUT terraform:     └── provider[registry.terraform.io/hashicorp/aws] >= 4.0.0
14:44:13.210 STDOUT terraform: Providers required by state:
14:44:13.210 STDOUT terraform:     provider[registry.terraform.io/hashicorp/aws]
14:44:13.210 STDOUT terraform:     provider[registry.terraform.io/hashicorp/time]
14:44:13.210 STDOUT terraform:     provider[registry.terraform.io/hashicorp/tls]

Reproduction Code [Required]

working with auto mode

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.31"

  cluster_name    = local.name
  cluster_version = "1.31"

  # auto mode
  cluster_compute_config = {
    enabled = true
    # see custom node pools in manifests/modules/cluster-aws-eks/
    node_pools = ["system"]
  }

  cluster_endpoint_public_access = true

  vpc_id            = module.vpc.vpc_id
  subnet_ids        = module.vpc.private_subnets
  cluster_ip_family = "ipv6"

  create_cni_ipv6_iam_policy = true
  iam_role_additional_policies = {
    "policy-eks-cluster" = aws_iam_policy.iam_cluster_policy.arn
  }
}

changing to:

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.31"

  cluster_name    = local.name
  cluster_version = "1.31"

  # auto mode
  #cluster_compute_config = {
  #  # disable auto mode
  #  enabled = false
  #  # see custom node pools in manifests/modules/cluster-aws-eks/
  ##  node_pools = ["system"]
  #}


  cluster_endpoint_public_access = true

  vpc_id            = module.vpc.vpc_id
  subnet_ids        = module.vpc.private_subnets
  cluster_ip_family = "ipv6"

  create_cni_ipv6_iam_policy = true
  iam_role_additional_policies = {
    "policy-eks-cluster" = aws_iam_policy.iam_cluster_policy.arn
  }


  eks_managed_node_group_defaults = {
    iam_role_additional_policies = {
      AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
    }
  }

  eks_managed_node_groups = {
    system = {
      # https://docs.aws.amazon.com/eks/latest/APIReference/API_Nodegroup.html
      ami_type       = "BOTTLEROCKET_ARM_64"
      instance_types = ["t4g.large"]
      capacity_type  = "ON_DEMAND"
      min_size       = 1
      max_size       = 2
      desired_size   = 1

      labels = {
        # Used to ensure Karpenter runs on nodes that it does not manage
        "karpenter.sh/controller" = "true"
        "CriticalAddonsOnly"      = "true"
      }

      taints = {
        # The pods that do not tolerate this taint should run on nodes
        CriticalAddonsOnly = {
          key    = "CriticalAddonsOnly"
          value  = "true"
          effect = "NO_SCHEDULE"
        }
      }
    }
  }

}

Steps to reproduce the behavior:

Expected behavior

no error

Actual behavior

Error: compute_config.enabled, kubernetes_networking_config.elastic_load_balancing.enabled, and storage_config.block_storage.enabled must all be set to either true or false

Terminal Output Screenshot(s)

  │ Error: compute_config.enabled, kubernetes_networking_config.elastic_load_balancing.enabled, and storage_config.block_storage.enabled must all be set to either true or false
  │ 
  │   with module.eks.aws_eks_cluster.this[0],
  │   on .terraform/modules/eks/main.tf line 35, in resource "aws_eks_cluster" "this":
  │   35: resource "aws_eks_cluster" "this" {

Additional context

[colinjohnsonbeyond]

Just hit this issue myself

[mbmblbelt]

Same here

There are three structures in the cluster config that are created/modifed as a result of enabling auto mode.

• kubernetesNetworkConfig
• computeConfig
• storageConfig

Prior to enabling auto mode, kubernetesNetworkConfig does not have an elasticLoadBalancing setting and the computeConfig and storageConfig structures do not exist in the cluster config at all.

❯ aws eks describe-cluster --name <cluster name> | jq '.cluster'
{
  "name": <redacted>,
  "arn": <redacted>,
  "createdAt": "2022-11-30T09:31:35.889000-05:00",
  "version": "1.30",
  "endpoint": <redacted>,
  "roleArn": <redacted>,
  "resourcesVpcConfig": {
    "subnetIds": [
      <redacted>
    ],
    "securityGroupIds": [
      <redacted>
    ],
    "clusterSecurityGroupId": <redacted>,
    "vpcId": <redacted>,
    "endpointPublicAccess": true,
    "endpointPrivateAccess": true,
    "publicAccessCidrs": [
      <redacted>
    ]
  },
  "kubernetesNetworkConfig": {
    "serviceIpv4Cidr": <redacted>,
    "ipFamily": "ipv4"
  },
  "logging": {
    "clusterLogging": [
      {
        "types": [
          "api",
          "audit",
          "authenticator"
        ],
        "enabled": true
      },
      {
        "types": [
          "controllerManager",
          "scheduler"
        ],
        "enabled": false
      }
    ]
  },
  "identity": {
    "oidc": {
      "issuer": <redacted>
    }
  },
  "status": "ACTIVE",
  "certificateAuthority": {
    "data": <redacted>
  },
  "platformVersion": "eks.23",
  "tags": {
   <redacted>
  },
  "health": {
    "issues": []
  },
  "accessConfig": {
    "authenticationMode": "API_AND_CONFIG_MAP"
  },
  "upgradePolicy": {
    "supportType": "EXTENDED"
  }
}

After enabling auto mode, the elasticLoadBalancing setting is added to the kubernetesNetworkConfig structure and the computeConfig and storageConfig structures are created/added.

❯ aws eks describe-cluster --name <cluster name> | jq '.cluster'
{
  "name": <redacted>,
  "arn": <redacted>,
  "createdAt": "2022-11-30T09:31:35.889000-05:00",
  "version": "1.30",
  "endpoint": <redacted>,
  "roleArn": <redacted>,
  "resourcesVpcConfig": {
    "subnetIds": [
      <redacted>
    ],
    "securityGroupIds": [
      <redacted>
    ],
    "clusterSecurityGroupId": <redacted>,
    "vpcId": <redacted>,
    "endpointPublicAccess": true,
    "endpointPrivateAccess": true,
    "publicAccessCidrs": [
      <redacted>
    ]
  },
  "kubernetesNetworkConfig": {
    "serviceIpv4Cidr": <redacted>,
    "ipFamily": "ipv4",
    "elasticLoadBalancing": {
      "enabled": true
    }
  },
  "logging": {
    "clusterLogging": [
      {
        "types": [
          "api",
          "audit",
          "authenticator"
        ],
        "enabled": true
      },
      {
        "types": [
          "controllerManager",
          "scheduler"
        ],
        "enabled": false
      }
    ]
  },
  "identity": {
    "oidc": {
      "issuer": <redacted>
    }
  },
  "status": "ACTIVE",
  "certificateAuthority": {
    "data": <redacted>
  },
  "platformVersion": "eks.23",
  "tags": {
   <redacted>
  },
  "health": {
    "issues": []
  },
  "accessConfig": {
    "authenticationMode": "API_AND_CONFIG_MAP"
  },
  "upgradePolicy": {
    "supportType": "EXTENDED"
  },
  "computeConfig": {
    "enabled": true,
    "nodePools": ["system", "general-purpose"]
  },
  "storageConfig": {
    "blockStorage": {
      "enabled": true
    }
  }
}

These structures can not be removed once they have been added so simply commenting out the cluster_compute_config

# cluster_compute_config = {
#     enabled    = true
#     node_pools = ["system", "general-purpose"]
# }

or disabling it

cluster_compute_config = {
    enabled    = false
    node_pools = []
}

results in the module trying to set the storageConfig to null, which is not allowed now that the storageConfig structure exists in the cluster config

- storage_config {
          - block_storage {
              - enabled = false -> null
            }
        }