Release 18+ Upgrade Guide Breaks Existing Deployments

Description
Attempted to follow the upgrade guide to get to 18+.  Our Terraform deployments generally run from a Jenkins worker pod, that exists ON the same cluster that we are upgrading.  The pod has a service account on it, using the IRSA setup which grants it access to the cluster.
This all works/worked before the upgrade.
Reproduction
Attempt to follow the upgrade guide for 18.
Code Snippet to Reproduce
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "18.0.1"

  cluster_name    = format("eks-%s-%s-%s", var.layer, var.vpc_id_tag, var.platform_env)
  cluster_version = var.cluster_version

  subnet_ids = data.aws_subnet_ids.private.ids
  vpc_id     = data.terraform_remote_state.vpc.outputs.vpc_id

  cluster_endpoint_private_access = true
  cluster_endpoint_public_access  = false

  cluster_security_group_additional_rules = {
    admin_access = {
      description = "Admin ingress to Kubernetes API"
      cidr_blocks = [data.terraform_remote_state.vpc.outputs.vpc_cidr_block]
      protocol    = "tcp"
      from_port   = 443
      to_port     = 443
      type        = "ingress"
    }
  }

  eks_managed_node_group_defaults = {
    ami_type                   = "AL2_x86_64"
    disk_size                  = var.node_group_default_disk_size
    enable_bootstrap_user_data = true
    pre_bootstrap_user_data    = templatefile("${path.module}/templates/userdata.tpl", {})
    desired_size               = lower(var.platform_env) == "prod" ? 3 : 2
    max_size                   = lower(var.platform_env) == "prod" ? 6 : 3
    min_size                   = lower(var.platform_env) == "prod" ? 3 : 1
    instance_types             = lower(var.platform_env) == "prod" ? var.prod_instance_types : var.dev_instance_types
    capacity_type              = "ON_DEMAND"
    additional_tags = {
      Name = format("%s-%s-%s", var.layer, var.vpc_id_tag, var.platform_env)
    }
    update_config = {
      max_unavailable_percentage = 50
    }
    update_launch_template_default_version = true
    create_launch_template                 = true
    create_iam_role                        = true
    iam_role_name                          = format("iam-%s-%s-%s", var.layer, var.vpc_id_tag, var.platform_env)
    iam_role_use_name_prefix               = false
    iam_role_description                   = "EKS managed node group Role"
    iam_role_tags                          = local.tags
    iam_role_additional_policies = [
      "arn:aws-us-gov:iam::aws:policy/AmazonSSMManagedInstanceCore"
    ]
  }
  eks_managed_node_groups = {
    private1 = {
      subnet_ids = [tolist(data.aws_subnet_ids.private.ids)[0]]
    }
    private2 = {
      subnet_ids = [tolist(data.aws_subnet_ids.private.ids)[1]]
    }
    private3 = {
      subnet_ids = [tolist(data.aws_subnet_ids.private.ids)[2]]
    }
  }

  cluster_enabled_log_types              = ["api", "audit", "authenticator", "controllerManager", "scheduler"]
  cloudwatch_log_group_retention_in_days = 7

  enable_irsa = true

  cluster_encryption_config = [
    {
      provider_key_arn = aws_kms_key.eks.arn
      resources        = ["secrets"]
    }
  ]

  tags = merge(
    local.tags,
    {
      "Name"        = format("eks-%s-%s-%s", var.layer, var.vpc_id_tag, var.platform_env),
      "EKS_VERSION" = var.cluster_version
    }
  )

}

resource "null_resource" "patch" {
  triggers = {
    kubeconfig = base64encode(local.kubeconfig)
    cmd_patch  = "kubectl patch configmap/aws-auth --patch \"${local.aws_auth_configmap_yaml}\" -n kube-system --kubeconfig <(echo $KUBECONFIG | base64 --decode)"
  }

  provisioner "local-exec" {
    interpreter = ["/bin/bash", "-c"]
    environment = {
      KUBECONFIG = self.triggers.kubeconfig
    }
    command = self.triggers.cmd_patch
  }
}

locals
locals {

  kubeconfig = yamlencode({
    apiVersion      = "v1"
    kind            = "Config"
    current-context = "terraform"
    clusters = [{
      name = module.eks.cluster_id
      cluster = {
        certificate-authority-data = module.eks.cluster_certificate_authority_data
        server                     = module.eks.cluster_endpoint
      }
    }]
    contexts = [{
      name = "terraform"
      context = {
        cluster = module.eks.cluster_id
        user    = "terraform"
      }
    }]
    users = [{
      name = "terraform"
      user = {
        token = data.aws_eks_cluster_auth.eks.token
      }
    }]
  })

  aws_auth_configmap_yaml = <<-EOT
  ${chomp(module.eks.aws_auth_configmap_yaml)}
      - rolearn: arn:${var.iam_partition}:iam::${data.aws_caller_identity.current.account_id}:role/role-gitlab-runner-eks-${var.platform_env}
        username: gitlab:{{SessionName}}
        groups:
          - system:masters
      - rolearn: arn:${var.iam_partition}:iam::${data.aws_caller_identity.current.account_id}:role/role-jenkins-worker-eks-${var.platform_env}
        username: jenkins:{{SessionName}}
        groups:
          - system:masters
      - rolearn: arn:${var.iam_partition}:iam::${data.aws_caller_identity.current.account_id}:role/AWSReservedSSO_AdministratorAccess_f50fcd43baf05a89
        username: AWSAdministratorAccess:{{SessionName}}
        groups:
          - system:masters
  EOT
}

Expected behavior
Module will run to completion
Actual behavior
Current aws-auth
sh-4.2$ kubectl get configmap aws-auth -n kube-system -o yaml
apiVersion: v1
data:
  mapAccounts: |
    []
  mapRoles: |
    - "groups":
      - "system:bootstrappers"
      - "system:nodes"
      "rolearn": "arn:aws-us-gov:iam:::role/eks-ops-eks-dev20211104211936784200000009"
      "username": "system:node:{{EC2PrivateDNSName}}"
    - "groups":
      - "system:masters"
      "rolearn": "arn:aws-us-gov:iam:::role/role-gitlab-runner-eks-dev"
      "username": "gitlab-runner-dev"
    - "groups":
      - "system:masters"
      "rolearn": "arn:aws-us-gov:iam:::role/role-jenkins-worker-eks-dev"
      "username": "jenkins-dev"
  mapUsers: |
    - "groups":
      - "system:masters"
      "userarn": "arn:aws-us-gov:iam:::user/justin.seiser"
      "username": "jseiser"
kind: ConfigMap

The SA on the pod, that terraform is running from.
sh-4.2$ kubectl get sa jenkins-worker -n jenkins -o yaml
apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws-us-gov:iam:::role/role-jenkins-worker-eks-dev

The error terraform returns
module.eks.kubernetes_config_map.aws_auth[0]: Refreshing state... [id=kube-system/aws-auth]

Error: configmaps "aws-auth" is forbidden: User "system:serviceaccount:jenkins:jenkins-worker" cannot get resource "configmaps" in API group "" in the namespace "kube-system"

Additional context
I do not doubt that im missing something, but that something does not appear to be covered in the documentation that I can find.

Doesnt work if I run from a bastion host, which is also granted access to the cluster.
Error
╷
│ Error: Get "http://localhost/api/v1/namespaces/kube-system/configmaps/aws-auth": dial tcp <IP_ADDRESS>:80: connect: connection refused
│
│
╵

Same server showing access
sh-4.2$ kubectl get configmap aws-auth -n kube-system
NAME       DATA   AGE
aws-auth   3      62d

Ran the first few times with the Terraform kubernetes provider configured as it has been when it was working, and have re-ran it with that provider completely removed.

Are there any requirements to upgrade an existing cluster from 17 to 18?
Im not getting warnings 'variable not expected here' so I think I have everything renamed, but not making any progress on the aws-auth configmap. Does it need removed from the state?

Also, not sure if this matters, but in the current working deployment we have
  kubeconfig_aws_authenticator_command = "aws"
  kubeconfig_aws_authenticator_command_args = [
    "eks",
    "get-token",
    "--cluster-name",
    format("eks-%s-%s-%s", var.layer, var.vpc_id_tag, var.platform_env),
    "--region",
    data.aws_region.current.name
  ]

Same Error: Get "http://localhost/api/v1/namespaces/kube-system/configmaps/aws-auth" error on my side. You are not alone man ;-)

I believe your error message states the issue The connection to the server localhost:8080 was refused - did you specify the right host or port?
By default, this port is not open on the security groups created by this module - https://github.com/terraform-aws-modules/terraform-aws-eks#security-groups

I believe your error message states the issue The connection to the server localhost:8080 was refused - did you specify the right host or port?
By default, this port is not open on the security groups created by this module - https://github.com/terraform-aws-modules/terraform-aws-eks#security-groups

It's more like if it's a wrong url error more than a security group issue, no ? I expect this call being done from my laptop, not from a remote server. But I may be wrong.

I believe your error message states the issue The connection to the server localhost:8080 was refused - did you specify the right host or port?
By default, this port is not open on the security groups created by this module - terraform-aws-modules/terraform-aws-eks#security-groups

It's more like if it's a wrong url error more than a security group issue, no ? I expect this call being done from my laptop, not from a remote server. But I may be wrong.

The error message I was referring to was the one provided by @jseiser further up.
The error message you have provided does not provide enough detail - the module does not construct any URLs so I would suspect its also a security group access issue you are facing as well, but thats just a hunch based on whats provided

Hello, I'm having the exact same issue when upgrading from v17 to v18. It happens during the state refreshing on the terraform plan command.
The problem disappears if I manually remove the module.eks-cluster.local_file.kubeconfig[0] and module.eks-cluster.kubernetes_config_map.aws_auth[0] from the v17 state file, but I'm not exactly sure if there are any consequences in doing this.

I believe your error message states the issue The connection to the server localhost:8080 was refused - did you specify the right host or port?
By default, this port is not open on the security groups created by this module - https://github.com/terraform-aws-modules/terraform-aws-eks#security-groups

@bryantbiggs
I guess i dont follow.  This is a cluster/TF deployment created using the <18.0 module version. We are not wanting/trying to upgrade this enviromen to the latest module version.  Im not able to run a plan because of the error.  The security groups have not changed since i have not been able to run a plan let alone a apply with this new module version.

Hello, I'm having the exact same issue when upgrading from v17 to v18. It happens during the state refresh of the terraform plan command.
The problem disappears if I manually remove the module.eks-cluster.local_file.kubeconfig[0] and module.eks-cluster.kubernetes_config_map.aws_auth[0] from the v17 state file, but I'm not exactly sure if there are any consequences in doing this.

I believe that would be the appropriate change (remove those from your state) - v18.x removes native support for kubeconfig and aws-auth configmap https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/UPGRADE-18.0.md#list-of-backwards-incompatible-changes

How should we handle config-map roles now?
i had a role we used to connect to the cluster but since v18.x drop the support to aws-auth configmap, how can we address that?
For example i had somethig like this
  mapRoles: |
.
.
.
    - "groups":
      - "system:masters"
      "rolearn": "arn:aws:iam::000000000000:role/assumedRole
      "username": "administrator"

and that role was set in the map_roles variable something like this
`module "eks" {
source                      = "terraform-aws-modules/eks/aws"
version                     = "17.24.0"
map_roles = [
{
rolearn = aws_iam_role.eks_administrator.arn
username = "administrator"
groups = [ "system:masters" ]
}
]
}`
Moving forward what would be the best way to adding that types of roles into the aws-auth?

How should we handle config-map roles now?
i had a role we used to connect to the cluster but since v18.x drop the support to aws-auth configmap, how can we address that?
For example i had somethig like this
  mapRoles: |
.
.
.
    - "groups":
      - "system:masters"
      "rolearn": "arn:aws:iam::000000000000:role/assumedRole
      "username": "administrator"
and that role was set in the map_roles variable something like this

module "eks" {
  source                      = "terraform-aws-modules/eks/aws"
  version                     = "17.24.0"

  map_roles = [
    {
      rolearn = aws_iam_role.eks_administrator.arn
      username = "administrator"
      groups = [ "system:masters" ]
    }
  ]

}

Moving forward what would be the best way to adding that types of roles into the aws-auth?

@PadillaBraulio this is left up to users to decide what suites them best (Terraform, Helm, some flavor of GitOps, bash scripts, etc.)

these errors come if your the provider config depends on output of the eks module
i got the same when i used coalesce(module.eks.cluster_id, l"cluster1") for name name filter in the datablock for the provider
after changing that to the static name the error is gone

Since I only had a test cluster, I went ahead and approved the plan to replace the cluster.
This is probably a provider issue, but the execution failed because it tried to create a new cluster before tearing down the old one. But it couldn't create the new cluster because an existing cluster, the old one, had the same name.

These errors also occur on certain changes when your the provider config depends on output of the eks module i got the same when i used coalesce(module.eks.cluster_id, "cluster1") for name name filter in the datablock for the provider after changing that to the static name the error is gone
example:
data "aws_eks_cluster" "cluster1" {
  // name= coalesce(module.cluster1.cluster_id, "cluster1")  will generate error on certain changes
  name  = "cluster1"// works
}

data "aws_eks_cluster_auth" "cluster1" {
  name  = "cluster1"
}

provider "kubernetes" {
  host                   = element(concat(data.aws_eks_cluster.cluster1.*.endpoint, [""]), 0)
  cluster_ca_certificate = base64decode(element(concat(data.aws_eks_cluster.cluster1.*.certificate_authority.0.data, [""]), 0))
  token                  = element(concat(data.aws_eks_cluster_auth.cluster1.*.token, [""]), 0)
}

I get the error even when removing the k8s provider

I get the error even when removing the k8s provider
they also have the data in there example..
https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/examples/complete/main.tf#L235

yes i normaly have it like in the example but if i encounter these kind of errors :  dial tcp <IP_ADDRESS>:80: connect: connection refused
i change it to the static cluster name , to be able to run the plan

have you checked the security groups, as already suggested?   You mentioned the jenkins pod runs in the same cluster or at least inside the same vpc as I understood.  If you have the private api-server endpoint enabled, it could be the jenkins pod will try to connect to the private endpoint, and require a security group rule that is not provided by the eks module by default.   i actually ran into this same problem.

have you checked the security groups, as already suggested? You mentioned the jenkins pod runs in the same cluster or at least inside the same vpc as I understood. If you have the private api-server endpoint enabled, it could be the jenkins pod will try to connect to the private endpoint, and require a security group rule that is not provided by the eks module by default. i actually ran into this same problem.

Its def. not that.  Nothing has changed yet, since this is a working 17.x deployment and only fails when trying to run the plan on the 18.x deployment.
I also showed that it errors out on an external server above as well.
Thanks.

if i adjust the config of the 18.0.4 module so it keeps using the same iam cluster role and cluster security group

@martijnvdp
Ill have to give this a try on monday.

Hoping that this would save time for others dealing with aws-auth configmap management change. This worked for us, modifying the example code from here.
locals {
  kubeconfig = ...

  current_auth_configmap = yamldecode(module.eks.aws_auth_configmap_yaml)

  updated_auth_configmap_data = {
    data = {
      mapRoles = yamlencode(
        distinct(concat(
        yamldecode(local.current_auth_configmap.data.mapRoles), local.map_roles, )
      ))
      mapUsers = yamlencode(local.map_users)
    }
  }
}

resource "null_resource" "patch_aws_auth_configmap" {
  triggers = {
    cmd_patch = "kubectl patch configmap/aws-auth -n kube-system --type merge -p '${chomp(jsonencode(local.updated_auth_configmap_data))}' --kubeconfig <(echo $KUBECONFIG | base64 --decode)"
  }

  provisioner "local-exec" {
    interpreter = ["/bin/bash", "-c"]
    command     = self.triggers.cmd_patch

    environment = {
      KUBECONFIG = base64encode(local.kubeconfig)
    }
  }
}

Users and roles following the syntax in 17.x version.

i'm also still looking for a good solution for the aws-auth config map, have copied the aws-auth.tf from 17.24 for now which works but it still needs that forked http provider.
i cant use the kubectl null resource workaround as we are using terraform cloud agents without kubectl.

i'm also still looking for a good solution for the aws-auth config map, have copied the aws-auth.tf from 17.24 for now which still works but it needs that forked http provider. using it without the http wait will error with resource already exists, only alternative i found is to use kubectl_manifest which seems to be able to patch as well but its a bit hacky and i cant use the kubectl null resource workaround as we are using terraform cloud agents without kubectl. i would rather not add kubectl to tthe tfc cloud container image

kubernetes_patch would have been useful to solve the aws-auth configmap problem, unfortunately it looks like it won't happen. :disappointed:

@bryantbiggs
When you guys were testing upgrades pre-merge, how were you all handling this situation?  Im going to spin up a test env and try and walk through some of the suggestions above, but wanted to know your guys experience.  If I get something working I have no issue creating a pull to update the documentation.

https://github.com/terraform-aws-modules/terraform-aws-eks/pull/1680#issuecomment-989197948
again, it is a BREAKING change - if there was a clean and straightforward path to upgrade without change/disruption then it would not be a breaking change. this module had grown quite quickly and was carrying a lot of pre-0.12 syntax which was severely holding it back (extensive list of lists and index lookup, etc.) and the changes added (most notably due to the numerous changes of EKS itself) led to a patchwork of changes that built up over the years. I can't stress enough that this change was EXTENSIVE and I am sorry we cannot provide the copious amounts of details and upgrade steps to make the process smooth and seamless - the module is complex, EKS is complex, and the changes were substantial.
that said, this is how we generally test modules in this org:

Checkout master
terraform init ; terraform apply
Checkout feature/branch
terraform init ; terraform plan
At this point you have to start interpreting the plan and deducing what Terraform is trying to do and the impact of those changes, etc. With the change of adding the variable prefix_separator, that should have simplified a lot of the "breaking" aspects of the Terraform changes from v17.x to v18.x but that was after the PR and I did not test v17.x to current with the addition of  that change

@PadillaBraulio this is left up to users to decide what suites them best (Terraform, Helm, some flavor of GitOps, bash scripts, etc.)

I understand the motive to introduce breaking changes in order to refactor this module to rid it of historic cruft, but I too am hesitant to upgrade our existing clusters from the latest 17.x version as we are making use of the managed aws-auth functionality in such a way that we cannot re-implement that functionality with 18.x in a straightforward manner.
IMO it would be greatly appreciated if instead of telling users that it's now up to them to figure out how to re-implement functionality that was removed in the interest of tidying up, explicit examples be provided that satisfy the same set of design constraints satisfied by the previous version, i.e., the ability to provision an accessible cluster exclusively with Terraform. Several users have already noted that it is not feasible to rely on local-exec calling kubectl in their deployment environments, us included.
Perhaps I missed the part of the discussion that led up to the removal of managed aws-auth, but I'm struggling to see the justification for removal of that functionality entirely.
Is it safe to rely on the forked HTTP provider and the pure Terraform implementation used in 17.x if users choose to do so?

@PadillaBraulio this is left up to users to decide what suites them best (Terraform, Helm, some flavor of GitOps, bash scripts, etc.)

I understand the motive to introduce breaking changes in order to refactor this module to rid it of historic cruft, but I too am hesitant to upgrade our existing clusters from the latest 17.x version as we are making use of the managed aws-auth functionality in such a way that we cannot re-implement that functionality with 18.x in a straightforward manner.
IMO it would be greatly appreciated if instead of telling users that it's now up to them to figure out how to re-implement functionality that was removed in the interest of tidying up, explicit examples be provided that satisfy the same set of design constraints satisfied by the previous version, i.e., the ability to provision an accessible cluster exclusively with Terraform. Several users have already noted that it is not feasible to rely on local-exec calling kubectl in their deployment environments, us included.
Perhaps I missed the part of the discussion that led up to the removal of managed aws-auth, but I'm struggling to see the justification for removal of that functionality entirely.
Is it safe to rely on the forked HTTP provider and the pure Terraform implementation used in 17.x if users choose to do so?

There is no need for me to go into the depths of aws-auth issues when we can just look at history https://github.com/terraform-aws-modules/terraform-aws-eks/issues?q=is%3Aissue+sort%3Aupdated-desc+aws-auth+is%3Aclosed
Again, a clear boundary line was created with this change and I understand its very controversial - that line is that the module provisions AWS infrastructure resources and via the AWS API (via the Terraform AWS provider) and internal cluster provisioning and management is left up to users
As for the forked http provider, I do not know what its fate is. Most likely what will end up happening is that it gets archived in its current state so users can continue to utilize it - if we're lucky, Hashicorp incorporates the change upstream and the fork can still be archived but users can move off the fork and onto the official provider

There is no need for me to go into the depths of aws-auth issues when we can just look at history https://github.com/terraform-aws-modules/terraform-aws-eks/issues?q=is%3Aissue+sort%3Aupdated-desc+aws-auth+is%3Aclosed

Fair enough. This is a complex problem due to the automatic creation of the aws-auth configmap by EKS in certain configurations.

Again, a clear boundary line was created with this change and I understand its very controversial - this module provisions AWS infrastructure resources via the AWS API (via the Terraform AWS provider) and any internal cluster provisioning and management is left up to users

Point taken, but I would venture to suggest that part of provisioning stateful resources such as Kubernetes clusters, EC2 instances, etc. includes ensuring access control is properly configured. I don't expect this module to install a monitoring and logging workload, for example, but I do expect it to provision my resources in such a way that I can connect to them. I respect your decision to remove this functionality but I'm just trying to determine the best course of action to avoid headaches going forward with the upgrade.
I appreciate the work that went in to refactoring and the local-exec examples provided. AFAICT there isn't really a good way to handle this case without using some external tooling. We'll have to figure out how to work that in to our existing CI infrastructure.

As for the forked http provider, I do not know what its fate is. Most likely what will end up happening is that it gets archived in its current state so users can continue to utilize it - if we're lucky, Hashicorp incorporates the change upstream and the fork can still be archived but users can move off the fork and onto the official provider

Re: your second point, I'm not holding my breath. https://www.hashicorp.com/blog/terraform-community-contributions

I have not tried this myself, but I am in the camp that favors GitOps - but what if users take the route of using GitOps for handling the aws-auth configmap? You can take the output from aws_auth_configmap_yaml and append any additional roles/permissions and write that file back to your manifest repo with github_repository_file which will cause your GitOps operator (ArgoCD, FluxCD, etc.) to pickup the change and apply it on the cluster
Pros:

Works very well for private clusters
No additional tooling required on a deploy server (kubectl, awscli, aws-iam-authenticator, etc.)
Still Terraform controlled

Cons:

As usual with Terraform, the cold start problem. This is only really best solved if using EKS managed node groups because AWS EKS managed node group service (and Fargate profiles) automatically creates an aws-auth configmap. Self-managed node groups require users to provision the configmap for the first time. Again, there are changes around this but not without caveats - null resource with an ignore changes lifecycle block could be used with kubectl to provision the initial configmap

Anywho, that was just an idea that came up when discussing with a colleague. Like I said, there are numerous ways to solve this issue and its highly dependent on what the end user is looking for and the tools/processes they employ.

I have not tried this myself, but I am in the camp that favors GitOps - but what if users take the route of using GitOps for handling the aws-auth configmap? You can take the output from aws_auth_configmap_yaml and append any additional roles/permissions and write that file back to your manifest repo with github_repository_file which will cause your GitOps operator (ArgoCD, FluxCD, etc.) to pickup the change and apply it on the cluster

We also have argocd but i dont think ArgoCD can patch an existing resource: https://github.com/argoproj/argo-cd/issues/2437

Damn, why is everyone avoiding the support of patch?!

These errors also occur on certain changes when your the provider config depends on output of the eks module i got the same when i used coalesce(module.eks.cluster_id, "cluster1") for name name filter in the datablock for the provider after changing that to the static name the error is gone
example:
data "aws_eks_cluster" "cluster1" {
  // name= coalesce(module.cluster1.cluster_id, "cluster1")  will generate error on certain changes
  name  = "cluster1"// works
}

data "aws_eks_cluster_auth" "cluster1" {
  name  = "cluster1"
}

provider "kubernetes" {
  host                   = element(concat(data.aws_eks_cluster.cluster1.*.endpoint, [""]), 0)
  cluster_ca_certificate = base64decode(element(concat(data.aws_eks_cluster.cluster1.*.certificate_authority.0.data, [""]), 0))
  token                  = element(concat(data.aws_eks_cluster_auth.cluster1.*.token, [""]), 0)
}

@martijnvdp Thanks for this. It cleared my "connection refused" issues, but I still have this error.
No resource schema found for local_file.

Have you ran into this error before?

Have you ran into this error before? This worked fine before 18.0 and setting the version to 17.24.0 explicitly also works. Just trying to convert our existing stack to 18.0 without destroying it first as it is production. I don't get this error when I rebuild the entire stack from scratch. The only issue is updating the existing stack (17.24) that was previously deployed.

I have upgraded a test cluster from 17.24 to 18.0.5, but that error doesnt sound familiar to me.
The connection refused indicated that the cluster will be replaced (at least in my case) due the changes to the cluster security group or cluster iam role. Even a change in the description of the cluster security group will cause the security group and the cluster to be replaced.
To prevend this i had to add some vars like the cluster description variable to set it to the default of v17.24 (including a dot)
like i mentioned here: https://github.com/terraform-aws-modules/terraform-aws-eks/issues/1744#issuecomment-1007917721
to match the current cluster security group and iam role. with the end result that only the eks node group(s) are getting replaced.

I have not tried this myself, but I am in the camp that favors GitOps - but what if users take the route of using GitOps for handling the aws-auth configmap? You can take the output from aws_auth_configmap_yaml and append any additional roles/permissions and write that file back to your manifest repo with github_repository_file which will cause your GitOps operator (ArgoCD, FluxCD, etc.) to pickup the change and apply it on the cluster

We also have argocd but i dont think ArgoCD can patch an existing resource: argoproj/argo-cd#2437

Yes it can. ArgoCD does not actually care whether the resource exist or not.

We upgraded to 18+ by creating an EKS module with this module being run as a submodule. The patching has been a great work-around for us as it is contained within our base EKS module. Only issue is the config map is continuously replaced each terraform apply.
I gave ubuntu a dumbed down copy of what we are doing that can be found here: https://launchpadlibrarian.net/579011843/terraform-ubuntu-ami-error.tf
This example was to help them troubleshoot an error with their bootstrap script on their latest AMI. The example might help you. Just thought I would comment here.

@martijnvdp
I can not seem to get this to not want to replace the cluster, because of the role_arn changing.  How exactly did you prevent this?
  cluster_name    = format("eks-%s-%s-%s", var.layer, var.vpc_id_tag, var.platform_env)
  cluster_security_group_description = "EKS cluster security group."
  cluster_security_group_name        = format("eks-%s-%s-%s", var.layer, var.vpc_id_tag, var.platform_env)
  prefix_separator                   = ""
  iam_role_arn                       = "arn:aws-us-gov:iam::xxxx:role/eks-tc-eks-test20220113173038869800000003"

 ~ role_arn                  = "arn:aws-us-gov:iam::xxxx:role/eks-tc-eks-test20220113173038869800000003" -> (known after apply) # forces replacement

I was able to get an upgrade to go.  But terminated node groups now hang, and the cluster is throwing a bunch of IAM errors.

@jseiser ,yes looks about what i have
module "eks" {
  source                                  = "terraform-aws-modules/eks/aws"
  version                                 = "18.0.6"

  cluster_name                            = "clusterName"
  cluster_endpoint_private_access         = true
  cluster_endpoint_public_access_cidrs    = ["<IP_ADDRESS>/0"]
  cluster_security_group_additional_rules = local.cluster_security_group_additional_rules <- additional api access rules
  cluster_security_group_description      = "EKS cluster security group."
  cluster_security_group_name             = "clusterName"
  cluster_version                         = 1.21
  create_iam_role                         = false
  enable_irsa                             = true
  iam_role_name                           = var.cluster_iam_role_name
  iam_role_arn                            = var.cluster_iam_role_arn
  node_security_group_additional_rules    = local.node_security_group_additional_rules <- aditional node rules
  prefix_separator                        = ""
  subnet_ids                              = var.private_subnet_ids
  vpc_id                                  = var.vpc_id
  tags                                    = var.tags

  eks_managed_node_groups = {
    default = {

      min_size     = 2
      max_size     = 3
      desired_size = 2

      instance_types = ["t3.large"]

      block_device_mappings = {
        xvda = {
          device_name = "/dev/xvda"
          ebs = {
            volume_size           = 50
            encrypted             = true
            kms_key_id            = var.kms_key_arn
            delete_on_termination = true
          }
        }
      }
  }

and i copied the code from aws-auth.tf from 17.24 for the config map and do a state mv
terraform state mv aws_iam_role_policy_attachment.workers_AmazonSSMManagedInstanceCore[0] aws_iam_role_policy_attachment.nodes_AmazonSSMManagedInstanceCore[0]

terraform state mv module.eks.kubernetes_config_map.aws_auth[0] kubernetes_config_map.aws_auth[0]

Looks like its even possible to keep the existing legacy node group using something like this instead for the node group :
 eks_managed_node_groups = {
    default = {

      disk_size              = current root disk size
      launch_template_name   = ""
      name                   = "terraform"
      node_group_name_prefix = "terraform-"

      min_size     = 2
      max_size     = 3
      desired_size = 2

      instance_types = ["t3.large"]

  }

than you also need to move the node group in the state

I must have did something dumb, ill re-deploy this test env as 17 and upgrade again.  Thanks for the information.

Do you also have the 'iam_role_name" set besides the 'iam_role_arn'?

iam_role_name

I did not.

@martijnvdp I trying to upgrade my 17.x existing cluster, I can plan but looks like terraform is trying to recreate EKS cluster. How to avoid that? I don't want install many application again :(

@martijnvdp I trying to upgrade my 17.x existing cluster, I can plan but looks like terraform is trying to recreate EKS cluster. How to avoid that? Too many applications is running and i don't want reinstall them.

@thuandt it depends on your situation i think , if you using eks managed nodegroups you could look at my example : here. Or look at your plan and look for the reason the cluster gets replaced, you can find this in the plan " changed... forces replacement" and adjust your params accordingly.

Thanks @martijnvdp. I tried many times but too painful. I think I will recreate EKS cluster instead :(

@martijnvdp
@bryantbiggs
Any ideas on how to get this upgrade to work?
The cluster role is getting removed.\
module.eks.aws_iam_role.cluster[0]: Destruction complete after 1s

Pre upgrade it shows this
Cluster IAM Role ARN
arn:aws-us-gov:iam::xxxx:role/eks-tc-eks-test20220114143109451600000004

Post upgrade, the EKS UI still shows that, but when you click on the role in the UI, you get sent to the IAM space and it says No Entity Found
How does one tell the 18.x  module to just keep using the already working, currently in place, IAM role?
I have this in place
  prefix_separator = ""
  create_iam_role  = false
  iam_role_name    = "eks-tc-eks-test20220114143109451600000004"
  iam_role_arn     = "arn:aws-us-gov:iam::xxxx:role/eks-tc-eks-test20220114143109451600000004"

to be more concise.
The cluster is not getting re-created.
The IAM role is not getting replaced
I just end up with a cluster and a removed IAM role.

Strange why is the iam arn changing to null, maybe to do with this ? https://github.com/kubernetes-sigs/aws-iam-authenticator/issues/153

Strange why is the iam arn changing to null, maybe to do with this ? kubernetes-sigs/aws-iam-authenticator#153

@martijnvdp
No idea.
We have never used aws-iam-authenticator either.
Our deployment looked like..
  kubeconfig_aws_authenticator_command = "aws"
  kubeconfig_aws_authenticator_command_args = [
    "eks",
    "get-token",
    "--cluster-name",
    format("eks-%s-%s-%s", var.layer, var.vpc_id_tag, var.platform_env),
    "--region",
    data.aws_region.current.name
  ]

ah nvm you created the iam role using this module so then you have to have to setv ar.create_iam_role  on true and move the cluster iam role in the state to the new name
from  aws_iam_role.cluster to the new name so it doesnt get replaced.
i created the iam roles myself outside the old module

@martijnvdp
Trying that now.

OK, that gets me farther.
It errors here
Error: error deleting IAM policy arn:aws-us-gov:iam::xxxx:policy/eks-tc-eks-test-deny-log-group20220118161852867300000002: DeleteConflict: Cannot delete a policy attached to entities.
	status code: 409, request id: 5121877e-f428-42b7-9a58-b2e96366d4bb

If i re-run the apply, it does not error out the second time.
I do notice, my IAM role actually loses policies.
These are all stripped
arn:aws-us-gov:iam::aws:policy/AmazonEKSServicePolicy
<cluster-name>-deny-log-group20211202210012245200000001
<cluster-name>-elb-sl-role-creation20211104210851070700000003

yes if you look at the code you see that the cluster iam role in 18.02 has less policies, but you could use var.iam_role_additional_policies to add additional policies

Should this issue really be closed? I feel like the upgrade guide is missing quite a bit of information to successfully upgrade:

map_roles being gone means we have to do this sort of shenanigans: https://github.com/terraform-aws-modules/terraform-aws-eks/issues/1744#issuecomment-1008529086
cluster_security_group_description, cluster_security_group_name, iam_role_arn and iam_role_name need to be manually set https://github.com/terraform-aws-modules/terraform-aws-eks/issues/1744#issuecomment-1007917721
some other manual terraform state manipulations are required

it was closed by the individual who created it, so that is their choice.
if you feel like details are missing, please submit a PR

@jseiser would you consider reopening this?
We really need a migration guide for these changes and until that's done (or the request is rejected) we should have an issue open for it, and this one has a ton of context already in place.

The dev's have zero desire to participate in this, so I dont see

A migration guide for this would be great, I have converted all of my worker_group_launch_templates into self_managed_node_groups but Terraform wants to delete all existing launch templates and rebuild them.
I understand that this is a breaking change but I feel like there is a difference between breaking my terraform configuration and breaking my EKS cluster.

>- Upgrade
# module.eks.module.eks.aws_iam_role.cluster[0] will be destroyed
# module.eks.module.eks.aws_iam_role.this[0] will be created

# module.eks.module.eks.aws_eks_cluster.this[0] must be replaced
+/- resource "aws_eks_cluster" "this" {
      ~ role_arn                  = "arn:aws:iam::1234567890:role/........." -> (known after apply) # forces replacement

I feel the exact same way @Reinkaos
Seems like there's zero chance you'll be able to migrate from 17 to 18 without downtime, and that's a harsh requirement for an upgrade that only takes place on the terraform level. There's no way we'll be able to upgrade, and I think that's quite a shame. It's sad to be left out of new versions for an indefinite time. Not sure if this will ever get addressed if it hasn't been already

I feel the exact same way @Reinkaos
Seems like there's zero chance you'll be able to migrate from 17 to 18 without downtime, and that's a harsh requirement for an upgrade that only takes place on the terraform level. There's no way we'll be able to upgrade, and I think that's quite a shame. It's sad to be left out of new versions for an indefinite time. Not sure if this will ever get addressed if it hasn't been already

This is completely untrue - numerous folks/teams have migrated without downtime. Copying+pasting code off the internet and then expecting those who maintain it to explain to you how to use it in each users specific use case quite absurd. (And provide all of this for free while taking away from their time so that others can profit from that - come on). Just testing, validating, re-validating, etc. took me over a month for the v18.x changeset. Nobody, NOBODY - paid me a dime for this but yet this is the thanks received from the community.
Terraform tells you exactly what its going to do - if you cannot interpret and understand what the plans are telling you, I suggest sitting down and investing in learning more about the tool. This is the gist of what you will see in this thread above (and a huge thank you to those who have been chiming in on whats helped them migrate - I don't have access to various clusters configured in all the various ways where I can document each possible different migration path - nor do I have the time to do so, so having the community share in their learnings is a huge help)
We are not closing this issue because folks have stated there are bits and pieces that have helped them. However, seeing this turn into the vengeful and negative thread is not conducive in any form. Seeing people who have finally figured out how to manage their resources and upgrade to v18.x without downtime only to come back and pile on negativity - its not going to be tolerated.

In my case, well, I don't mind replacing the cluster.
But what I do need is a means to accomplish what user_map does that doesn't make me want to start a fire.
As well, while it isn't as important, I was really appreciating managing a local kubeconfig. That though at least has a clear "replacement" listed in UPGRADE-18.0.md unlike the aforementioned user_map note, which is more "we removed this go figure it out yourself"

In my case, well, I don't mind replacing the cluster.
But what I do need is a means to accomplish what user_map does that doesn't make me want to start a fire.
As well, while it isn't as important, I was really appreciating managing a local kubeconfig. That though at least has a clear "replacement" listed in UPGRADE-18.0.md unlike the aforementioned user_map note, which is more "we removed this go figure it out yourself"

it was simply not feasible to continue to manage aws-auth/kubeconfig in this module - https://github.com/terraform-aws-modules/terraform-aws-eks/issues?q=is%3Aissue+sort%3Aupdated-desc+aws-auth+is%3Aclosed

numerous issues and spurious changes due to windows based users versus *nix based users
there is a shift where K8s is more managed - we see this with EKS managed node groups and Fargate profiles; they both modify/update the aws-auth configmap for their respective IAM roles
we have had to fork and take on maintaining the http provider to provide the wait functionality just to support aws-auth; Hashicorp has not provided any response to the upstream PR

I can go on and on - but just look at the history of issues related to aws-auth in this modules history. The module will be more robust and easier to maintain, quickly provide updates (see all the releases and how quickly we could add them in since v18.x - IPV6 support being one of them which was next day support after provider was updated), and its just generally sound engineering - if we let this module be concerned with too much, its a nightmare to maintain and manage

Oh I understand that. My problem isn't that they were removed, but more of that we were left to our own devices to figure out what to do without it?

Oh I understand that. My problem isn't that they were removed, but more of that we were left to our own devices to figure out what to do without it.
The kubeconfig removal was handled to my satisfaction with the note in the upgrade doc, but the user/role mapping wasn't - or at least the note assumes a lot more understanding of everything than I am currently equipped with.

I think thats down to:

We don't want to be overly-prescriptive only to have that come back on us with "Well you said this would work...", etc.
Since the release of v18.x, I don't know if there is a good recommendation that I can give for a replacement. The Kubernetes Terraform provider does not support kubectl patch, otherwise I would say that would have been the recommendation

However, we did provide these examples to demonstrate A solution (not THE solution) to handling this:

https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/examples/complete/main.tf#L264
https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/examples/self_managed_node_group/main.tf#L261

I'm starting to try if I can migrate all my v17 clusters (more than 20 clusters) to v18 without downtime. For me it is a requirement, as some of them run production services. I'll try as hard as I can to do it, and if I manage to do it I'll explain with details.
Right now I am trying to migrate managed node groups. Do you know if they can be migrated without recreation?

Right now I am trying to migrate managed node groups. Do you know if they can be migrated without recreation?

I've used a self managed nodes module out of scope of eks module, that created a nodes for my cluster. But did not upgrade the eks module itself just yet. In short separated eks resources and nodes, I think guys exactly intended this with the modules their introduced with v18.
ref: https://github.com/terraform-aws-modules/terraform-aws-eks/tree/master/modules

This is the output from terraform plan for a managed group "infrastructure"
module.eks.module.eks_managed_node_group["infrastructure"].aws_eks_node_group.this[0] will be created
module.eks.module.eks_managed_node_group["infrastructure"].aws_iam_role.this[0] will be created
module.eks.module.eks_managed_node_group["infrastructure"].aws_iam_role_policy_attachment.this["arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"] will be created
module.eks.module.eks_managed_node_group["infrastructure"].aws_iam_role_policy_attachment.this["arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"] will be created
module.eks.module.eks_managed_node_group["infrastructure"].aws_iam_role_policy_attachment.this["arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"] will be created
module.eks.module.eks_managed_node_group["infrastructure"].aws_launch_template.this[0] will be created
module.eks.module.eks_managed_node_group["infrastructure"].aws_security_group.this[0] will be created

module.eks.module.node_groups.aws_eks_node_group.workers["infrastructure"] will be destroyed
module.eks.module.node_groups.aws_launch_template.workers["infrastructure"] will be destroyed

This is the output of terraform state list before updating
aws_autoscaling_attachment.node_groups--private_elb["infrastructure"]
aws_autoscaling_attachment.node_groups--public_elb["infrastructure"]
module.eks.module.node_groups.data.cloudinit_config.workers_userdata["infrastructure"]
module.eks.module.node_groups.aws_eks_node_group.workers["infrastructure"]
module.eks.module.node_groups.aws_launch_template.workers["infrastructure"]

A little bit messy, not sure how to start. Also will need to figure out how to fix the load balancing attachments in this new version.
Checking one piece at a time,
module.eks.module.node_groups.aws_eks_node_group.workers["infrastructure"]
becomes:
module.eks.module.eks_managed_node_group["infrastructure"].aws_eks_node_group.this
so it's like we're moving a lit of nodes into a list of "nodes modules" with a single node on each

@jaimehrubiks
I managed to migrate 3 managed node group clusters without downtime, but I allowed my node groups to get re-created.  The only hang up, was you cant enable core-dns addon during the upgrade.
I though @martijnvdp  mentioned somewhere, importing the existing node groups but I cant find it.

was you cant enable core-dns addon during the upgrade.

I decided to just configure add-ons out of scope of the module.

However, we did provide these examples to demonstrate A solution (not THE solution) to handling this:
* https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/examples/complete/main.tf#L264

* https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/examples/self_managed_node_group/main.tf#L261

I saw that, but it means almost nothing to me without explanation. That's the trouble, here.
Also I want to point out that directly pointing at bash in a specific location is fragile.

Hoping that this would save time for others dealing with aws-auth configmap management change. This worked for us, modifying the example code from here.
code snipped

Users and roles following the syntax in 17.x version.

hey @sepehrmavedati - how can this be used when one is not using map_roles (just map_users)? I've tried to define an empty structure but terraform just will not have it, and I'm unsure how (if possible) to modify your code there to omit references to it.

@martijnvdp did you manage to migrate existing managed node groups to v18? Let me know if so, thank you. I'm doing my own research but it doesn't seem easy.

for what its worth, I share my experience with upgrading from v17.x based EKS cluster to v18.x of the module:
Scenario: Bit more complex setup, including Managed-Node-Group, LaunchTemplate with CustomAMI, extra node Security Groups/Rules, extra Cluster SG/Rules, Encryption configs...
after like 1-2h of reading internals of v18 and adapting our v17 module accordingly, I was able to match the MNG + LT.
however that were only 2 of over 30! resources this module managed in background.
So I figured this would be a really time-consuming journey and might even turn out impossible to do in the end.
Advice: if you can afford recreating the cluster, dont invest time in trying to upgrade module in place. I did the same in the end

@philicious I have same experience like you :)
After recreation, to restore cluster, you should have "everything as code". In my case, after manually deploy Argo CD, it restored all our applications using git repository (GitOps).

Hi, dropping this here. I managed to migrate to v17 to v18 (I only had managed node groups). I choose to discard the existing node groups from the state and create new ones and then migrate workload on new node groups. This allowed me to keep existing control plane.
State migration / deletion

terragrunt state rm 'module.node_groups'
terragrunt state rm 'kubernetes_config_map.aws_auth[0]'
terragrunt state mv 'aws_iam_role.cluster[0]' 'aws_iam_role.this[0]'
terragrunt state rm 'aws_eks_addon.kube_proxy' 'aws_eks_addon.vpc_cni' 'aws_eks_addon.coredns'
terragrunt state rm 'aws_iam_role_policy_attachment.workers_AmazonEC2ContainerRegistryReadOnly[0]' 'aws_iam_role_policy_attachment.workers_AmazonEKSWorkerNodePolicy[0]' 'aws_iam_role_policy_attachment.workers_AmazonEKS_CNI_Policy[0]'
terragrunt state rm 'aws_security_group.workers[0]'
terragrunt state rm $(terragrunt state list | grep aws_security_group_rule.workers)
terragrunt state rm $(terragrunt state list | grep aws_security_group_rule.cluster)
terragrunt state rm 'aws_iam_role.workers[0]'
terragrunt import 'aws_eks_addon.this["coredns"]' cluster:coredns
terragrunt import 'aws_eks_addon.this["vpc-cni"]' cluster:vpc-cni
terragrunt import 'aws_eks_addon.this["kube-proxy"]' cluster:kube-proxy

Variable to set in new config to avoid cluster deletion and recreation:
prefix_separator                   = ""
iam_role_name                      = $CLUSTER_NAME
cluster_security_group_name        = $CLUSTER_NAME
cluster_security_group_description = "EKS cluster security group."

I also added the old security group to new managed node groups so that communication between the old node groups and the new ones is ok. I had some minor issue with aws-load-balancer-controller :
⚠ security group tag kubernetes.io/cluster/$NAME should be removed from the old worker security group to avoid load balancer target registration issue.
This will allow new node groups to be created and to join existing cluster while keeping the existing node groups safe and outside of terraform. Once the new nodes are up and running, old node group should be manually deleted and pods will restart on new pool. Once all the workload are migrated, leftover resources should be manually deleted.
Resources to delete on the AWS console one old node groups are deleted:

references to old security group worker in security group rules
old worker security group

Of course every config is different, I had 8 cluster with similar config that I migrated with this without needing to destroy and recreate the clusters.

I planned to move from 15.0.0 to 18.5.1, one version at a time. Cluster is Kubernetes version 1.18. Now I think is to create a new cluster altogether and move all workload there. following @philicious advice will create all three clusters from scratch.

@ArchiFleKs thanks for those steps, there is one thing i am unsure about if you have time to explain.
terragrunt state mv 'aws_iam_role.cluster[0]' 'aws_iam_role.this[0]'

Im assuming that to reuse the currnet role you have to rename that state to aws_iam_role.this[0] which is what v18 is looking for?
There is one thing i had to change to my config different from yours and that was to explicitly set that no new iam role should be created and to use the existing arn value. Simply setting iam_role_name did not work for me and only by adding these 2, the cluster will not be replaced.
  create_iam_role = false
  iam_role_arn = "<ARN_HERE>"

However before this state manipulation it wants to destroy it
# module.eks.aws_iam_role.cluster[0] will be destroyed
  # (because aws_iam_role.cluster is not in configuration)

and after mv the state it also wants to destroy it.
# aws_iam_role.this[0] will be destroyed
  # (because aws_iam_role.this is not in configuration)

Which leaves me a little bit confused...

@kaykhancheckpoint if you specify an iam_role_arn then the module treats it as an externally managed resource (i.e. - the module doesn't manage it) and therefore it shouldn't be in the state. if you want to manage it with the module then remove the create_iam_role = false and iam_role_arn = "<ARN_HERE>" and do the state mv you referenced. it has to be one route or the other, can't be a mix of both

Is it IAM role name or IAM role ARN ? You have both in your example, I think name is enough, you should put the short role name not the full arn

@bryantbiggs thanks for that clarification
@ArchiFleKs yes i did,
For some reason following your steps
  prefix_separator                   = ""
  iam_role_name                      = local.name
  cluster_security_group_name        = local.name
  cluster_security_group_description = "EKS cluster security group."

terraform state mv 'module.eks.aws_iam_role.cluster[0]' 'aws_iam_role.this[0]'

terraform is asking to recreate the cluster.
 # module.eks.aws_eks_cluster.this[0] must be replaced
+/- resource "aws_eks_cluster" "this" {
  ...
      name                      = "<redacted>"
    ~ platform_version          = "eks.7" -> (known after apply)
    ~ role_arn                  = "<redacted> -> (known after apply) # forces replacement
   ...
  }

What does the terraform plan says about the aws_iam_role.this[0] resource ? No changes ?

I can see the end iam_role_name has some difference in naming convention between v15 (version of my original EKS) and v18.
v15
{local.name}20220214102518927900000002

v18
{local.name}-cluster-20220214143603650500000001

Perhaps this naming convention plays into this issue.

only if you don't supply a role name https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/main.tf#L173

I see and setting  prefix_separator ("") should take care of that extra -.
Then at this point im stuck, i think i followed those steps a few times and it is still looking to destroy the role and replace the cluster :/

and without seeing the plan diff, we're stuck being able to provide any further help 🤷🏽♂

@bryantbiggs Sure, youre welcome to take a look at the plan diff :+1:  https://pastebin.com/Zr0kJTtE
  prefix_separator                   = ""
  iam_role_name                      = local.name
  cluster_security_group_name        = local.name
  cluster_security_group_description = "EKS cluster security group."

state changed up until now.
terraform state rm 'kubernetes_secret.aws-secret'
terraform state rm 'module.eks.kubernetes_config_map.aws_auth[0]'
terraform state rm $(terraform state list | grep autoscaling_group.workers)
terraform state rm $(terraform state list | grep aws_launch_configuration)
terraform state rm $(terraform state list | grep module.eks.random_pet)
terraform state rm $(terraform state list | grep aws_iam_instance_profile)
terraform state rm $(terraform state list | grep aws_security_group_rule.workers)
terraform state rm $(terraform state list | grep aws_security_group_rule.cluster)
terraform state mv 'module.eks.aws_iam_role.cluster[0]' 'aws_iam_role.this[0]'
terraform state rm 'module.eks.aws_iam_role_policy_attachment.workers_AmazonEC2ContainerRegistryReadOnly[0]'
terraform state rm 'module.eks.aws_iam_role_policy_attachment.workers_AmazonEKSWorkerNodePolicy[0]'
terraform state rm 'module.eks.aws_iam_role_policy_attachment.workers_AmazonEKS_CNI_Policy[0]'

remaining
terraform state list
data.aws_eks_cluster.cluster
data.aws_eks_cluster_auth.cluster
data.terraform_remote_state.networking
aws_iam_role.this[0]
module.eks.data.aws_ami.eks_worker
module.eks.data.aws_ami.eks_worker_windows
module.eks.data.aws_caller_identity.current
module.eks.data.aws_iam_policy_document.cluster_assume_role_policy
module.eks.data.aws_iam_policy_document.cluster_elb_sl_role_creation[0]
module.eks.data.aws_iam_policy_document.workers_assume_role_policy
module.eks.data.aws_partition.current
module.eks.data.template_file.userdata[0]
module.eks.data.template_file.userdata[1]
module.eks.data.template_file.userdata[2]
module.eks.aws_eks_cluster.this[0]
module.eks.aws_iam_openid_connect_provider.oidc_provider[0]
module.eks.aws_iam_policy.cluster_elb_sl_role_creation[0]
module.eks.aws_iam_role.workers[0]
module.eks.aws_iam_role_policy_attachment.cluster_AmazonEKSClusterPolicy[0]
module.eks.aws_iam_role_policy_attachment.cluster_AmazonEKSServicePolicy[0]
module.eks.aws_iam_role_policy_attachment.cluster_AmazonEKSVPCResourceControllerPolicy[0]
module.eks.aws_iam_role_policy_attachment.cluster_elb_sl_role_creation[0]
module.eks.aws_security_group.cluster[0]
module.eks.aws_security_group.workers[0]
module.eks.local_file.kubeconfig[0]
module.eks.null_resource.wait_for_cluster[0]

Have you tried disabling the iam_role_use_name_prefix which is on by default, and putting the full role name as it was created in v15 ?

Have you tried disabling the iam_role_use_name_prefix which is on by default, and putting the full role name as it was created in v15 ?

thanks for the suggestion, no difference

For those of you experiencing the localhost/<IP_ADDRESS> issue, you might be facing this issue https://github.com/hashicorp/terraform-provider-kubernetes/issues/1028 which have a few workarounds that sadly all come with downsides.
We fixed our setup by locking our Kubernetes provider version to 1.13.4. This version however don't have newer resources like the kubernetes_manifest resource.

I'm using kubernetes_manifest to patch the configmap.
Firstly, we have to tf import the configmap:
$ terraform import \
        kubernetes_manifest.apply "apiVersion=v1,kind=ConfigMap,namespace=kube-system,name=aws-auth"

If the above command fails, try running plan/refresh, and run the import command again
locals {
  map_users = [
    {
      groups   = ["system:xxxx"]
      userarn  = data.aws_iam_user.xxxx.arn
      username = data.aws_iam_user.xxxx.user_name
    },
  ]

  aws_auth = yamldecode(module.eks.aws_auth_configmap_yaml)
  auth_data = {
    mapAccounts = yamlencode([])
    mapRoles    = local.aws_auth["data"].mapRoles,
    mapUsers    = yamlencode(distinct(local.map_users))
  }

  updated_aws_auth = merge(
    local.aws_auth,
    { data = local.auth_data },
    { metadata = {
      namespace = "kube-system"
      labels = {
        "updatedBy"           = "Terraform"
      }
      }
    }
  )
}

resource "kubernetes_manifest" "apply" {
  manifest = local.updated_aws_auth

  field_manager {
    force_conflicts = true
  }
}

@ArchiFleKs I wanted to thank you for taking the time to post your terraform state commands and configuration parameters for upgrading your cluster. I was able to take the relevant ones and use them when upgrading my cluster, which worked mostly fine.
We were managing our MNGs separately from the module and I ran into trouble once I replaced the with MNGs managed by the module (for tagging purposes). I thought I would pay it forward if anyone else runs into this problem.
The security group and rules created for the general MNGs do not account for communication from the control plane to the nodes on ephemeral ports. The default rules account for port 443 and 10250, and at first we thought that was enough. We ran into an error from the api attempting to validate ingresses:
Failed calling webhook, failing closed validate.nginx.ingress.kubernetes.io: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post "[https://ingress-private-ingress-nginx-controller-admission.ingress.svc:443/networking/v1beta1/ingresses?timeout=10s](https://ingress-private-ingress-nginx-controller-admission.ingress.svc/networking/v1beta1/ingresses?timeout=10s)": context deadline exceeded

What was very confusing is that it shows port 443 there, and that port is accounted for. I think, to avoid collisions though, there must be other ports involved behind the scenes. We had to add the following to the module to clear it up:
  node_security_group_additional_rules = {
    ingress_control_plane = {
      description = "Control plane to node ephemeral ports"
      protocol    = "-1"
      from_port   = 1024
      to_port     = 65535
      type        = "ingress"
      source_cluster_security_group = true
    }
}

(On a side note, the from_port and to_port did not for some reason translate and the rule became all ports, but that's fine too)
We also ended up needing to add a rule to allow inter-node communication on all ports:
    ingress_self_all = {
      description = "Node to node all ports/protocols"
      protocol    = "-1"
      from_port   = 0
      to_port     = 0
      type        = "ingress"
      self        = true
    }

Along with egress from the nodes:
    egress_all = {
      description      = "Node all egress"
      protocol         = "-1"
      from_port        = 0
      to_port          = 0
      type             = "egress"
      cidr_blocks      = ["<IP_ADDRESS>/0"]
      ipv6_cidr_blocks = ["::/0"]
    }

And if you don't want to lose access via kubectl from your given subnets, you'll need to also add a rule on the additional control plane security group:
  cluster_security_group_additional_rules = {
    external_ingress = {
      description              = "Allow ingress from permitted cidrs"
      protocol                   = "tcp"
      from_port                = 443
      to_port                    = 443
      type                         = "ingress"
      cidr_blocks = [
        var.vpn-subnet,
        var.ci-a-cidr,
        var.ci-b-cidr,
        var.ci-c-cidr
      ]
    }
  }

@jslusher that port is the k8s internal service port, did you check the associated nodeport with the ingress service?

I must admit it was an inspiring experience to upgrade from 17 to 18 ;) and long weekend too ;) The discussion here helped a lot.
So in order to upgrade without killing the cluster I needed to set:
  prefix_separator                   = ""
  cluster_security_group_description = "EKS cluster security group."
  cluster_security_group_name        = local.cluster_name
  iam_role_name                      = local.cluster_name

and before running I just moved cluster iam role like in https://github.com/terraform-aws-modules/terraform-aws-eks/issues/1744#issuecomment-1045815882 comment.
My current (at that moment) node groups were destroyed but new ones were created in parallel and my whole workload was redeployed automatically to new hosts (most of it without any big disruption).
For aws-auth config map this did a trick for me - so my custom map is merged well with existing configmap - and I can merge roles and users too if needed:
data "aws_eks_cluster" "cluster" {
  name = local.cluster_name
}

data "aws_eks_cluster_auth" "cluster" {
  name = local.cluster_name
}

data "aws_availability_zones" "available" {
}

provider "kubernetes" {
  host                   = element(concat(data.aws_eks_cluster.cluster.*.endpoint, [""]), 0)
  cluster_ca_certificate = base64decode(element(concat(data.aws_eks_cluster.cluster.*.certificate_authority.0.data, [""]), 0))
  token                  = element(concat(data.aws_eks_cluster_auth.cluster.*.token, [""]), 0)
}

data "kubernetes_config_map" "aws-auth" {
  metadata {
    name      = "aws-auth"
    namespace = "kube-system"
  }
}

locals {
  kubeconfig = yamlencode(
    {
      apiVersion      = "v1"
      kind            = "Config"
      current-context = "terraform"
      clusters = [{
        name = module.eks.cluster_id
        cluster = {
          certificate-authority-data = module.eks.cluster_certificate_authority_data
          server                     = module.eks.cluster_endpoint
        }
      }]
      contexts = [{
        name = "terraform"
        context = {
          cluster = module.eks.cluster_id
          user    = "terraform"
        }
      }]
      users = [{
        name = "terraform"
        user = {
          token = data.aws_eks_cluster_auth.cluster.token
        }
      }]
    }
  )

  current_auth_configmap = data.kubernetes_config_map.aws-auth

  updated_auth_configmap_data = {
    data = {
      mapRoles = replace(yamlencode(
        distinct(concat(
          yamldecode(local.current_auth_configmap.data.mapRoles), local.map_roles)
      )), "\"", "")
      mapUsers = yamlencode(local.map_users)
    }
  }
}

resource "null_resource" "patch_aws_auth_configmap" {
  triggers = {
    cmd_patch = "kubectl patch configmap/aws-auth -n kube-system --type merge -p '${chomp(jsonencode(local.updated_auth_configmap_data))}' --kubeconfig <(echo $KUBECONFIG | base64 --decode)"
  }

  provisioner "local-exec" {
    interpreter = ["/bin/bash", "-c"]
    command     = self.triggers.cmd_patch

    environment = {
      KUBECONFIG = base64encode(local.kubeconfig)
    }
  }
}

and regarding managed node groups (using submodule) in order to have pre_bootstrap_user_data and bootstrap_extra_args  and post_bootstrap_user_data working well I needed to provide ami_id and set flags create_launch_template and enable_bootstrap_user_data to true.  So you need to set up all these attributes in group node submodule spec:
  ami_id   = data.aws_ami.amazon-linux-2.id

  create_launch_template     = true
  enable_bootstrap_user_data = true
  force_update_version       = true

  name                      = "node-group-name"
  cluster_name              = local.cluster_name
  cluster_version           = local.k8s_version
  cluster_auth_base64       = module.eks.cluster_certificate_authority_data
  cluster_endpoint          = module.eks.cluster_endpoint
  cluster_security_group_id = module.eks.cluster_security_group_id
  cluster_service_ipv4_cidr = local.eks_service_cidr

  # to let nodes to join control plane properly
  vpc_security_group_ids = [
    module.eks.node_security_group_id,
  ]

to look for ami_id you can use:
data "aws_ami" "amazon-linux-2" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "architecture"
    values = ["x86_64"]
  }

  filter {
    name   = "name"
    values = ["amazon-eks-node-${local.k8s_version}-*"]
  }
}

@dejwsz
Im quite new to v18 and terraform
but where does this data come from?
yamldecode(local.current_auth_configmap.data.mapRoles), local.map_roles)

Just to explain one of the lines from my above comment's code:
yamldecode(local.current_auth_configmap.data.mapRoles), local.map_roles)

You can see above it another local var in locals section:
current_auth_configmap = data.kubernetes_config_map.aws-auth

which refers to the data source block which gets aws-auth configmap value from kube-system namespaces in your cluster:
data "kubernetes_config_map" "aws-auth" {
  metadata {
    name      = "aws-auth"
    namespace = "kube-system"
  }
}

while local.map_roles is my local map variable with additional roles to add inside this configmap, like:
  map_roles = {
      rolearn  = "arn:aws:iam::${var.target_account_id}:role/Administrator"
      username = "admin"
      groups   = ["system:masters"]
    }

where var.target_account_id is target AWS account id holding the referenced role.

I recently upgraded our test, dev and prod clusters to v18 and want to share the experience with the community.
We are using self-managed node groups. Also, we were using launch configs based ASGs, so migrated that to launch templates before the upgrade since v18 doesn't support launch configs.
It’s very important to use a versioned terraform state since any accidental update/deletion would be irreversible otherwise.

Create an RBAC for a service account based kubeconfig that allows read/write access to aws-auth configmap, in case the configmap gets corrupted or deleted accidentally during the process. Only, the IAM user/role that has created the cluster will have access to the cluster when aws-auth is corrupted, so having such a kubeconfig (or an SA based admin kubeconfig) is extremely important.
Update the terraform state
$ terraform state rm 'module.eks.data.http.wait_for_cluster[0]'
$ terraform state rm 'module.eks.kubernetes_config_map.aws_auth[0]'
$ terraform state mv 'module.eks.aws_iam_role.cluster[0]' 'module.eks.aws_iam_role.this[0]'
$ terraform state rm $(terraform state list | grep "module.eks.aws_iam_role_policy_attachment.workers_Ama")
$ terraform state rm $(terraform state list | grep "module.eks.aws_autoscaling_group.workers_launch_template")
$ terraform state rm 'module.eks.aws_security_group.workers[0]'
$ terraform state rm $(terraform state list | grep "module.eks.aws_security_group_rule.workers")
$ terraform state rm $(terraform state list | grep "module.eks.aws_security_group_rule.cluster")
$ terraform state rm 'module.eks.aws_iam_role.workers[0]'
$ terraform state rm $(terraform state list | grep aws_iam_instance_profile)
$ terraform state rm $(terraform state list | grep "module.eks.aws_launch_template.workers_launch_template")

NOTE: the critical state updates are in bold
My eks module looks like below:module "eks" {
  source          = "terraform-aws-modules/eks/aws"
  version         = "~> 18.0"
  cluster_name    = local.cluster_name
  ...

  # Added as part of upgrading this module from v17 to v18
  prefix_separator                   = ""
  iam_role_name                      = local.cluster_name
  cluster_security_group_name        = local.cluster_name
  cluster_security_group_description = "EKS cluster security group."

  ....
}

Carefully read the plan output, add any additional security groups and/or rules that must be attached to workers
When running terraform plan/apply, ensure the EKS cluster resource isn't being replaced (no pun intended), or aws-auth resource isn't deleted
To migrate the workloads without any downtime, the old ASG role should remain in the aws-auth configmaplocals {
   aws_auth_data = yamldecode(module.eks.aws_auth_configmap_yaml)["data"]
   aws_auth_cm = {
     apiVersion = "v1"
     kind       = "ConfigMap"
     data       = {
       mapAccounts = ...
       mapRoles    = yamlencode(concat(
       yamldecode(local.aws_auth_data.mapRoles),
         [{
           groups   = ["system:bootstrappers", "system:nodes"]
           rolearn  = "arn:aws:iam::xxxxxxx:role/xxxxxxxxxxxxx"
           username = "system:node:{{EC2PrivateDNSName}}"
         }]
       ))
       mapUsers    = ...
     }
     metadata = {
       name      = "aws-auth"
       namespace = "kube-system"
       labels = {
         "app.kubernetes.io/managed-by" = "Terraform"
          ...
       }
     }
   }
}

When the upgraded process is successfully completed, ensure new nodes are registered (by running kubectl get nodes, in case the new nodes don't come up you have to access the nodes and check the kubectl service logs) and able to reach the cluster endpoint and to CoreDNS.
Delete the

old ASGs
old ASGs role, policies, launch templates
old ASG role from aws-auth configmap
kubeconfig, and rbac that was created for aws-auth

I think at this point this seems to be a well known consequence of the breaking changes. Should the above workaround(s) be added to the upgrade notes as a mitigation?

anyone tracking here for aws-auth related questions - please see https://github.com/terraform-aws-modules/terraform-aws-eks/issues/1901#issuecomment-1055993663

I followed steps described by @junaid-ali and upgrade went more-less smooth.
Additional thing, 18.x version seems to have much more tight security group rules. That caused ALB ingress controller not working properly in my case. I followed setup mentioned here:
https://github.com/terraform-aws-modules/terraform-aws-eks/issues/1748#issuecomment-1050231374
I think that resembles more less something we had in 17.x version.

I followed steps described by @junaid-ali and upgrade went more-less smooth. Additional thing, 18.x version seems to have much more tight security group rules. That caused ALB ingress controller not working properly in my case. I followed setup mentioned here: #1748 (comment)
I think that resembles more less something we had in 17.x version.

thanks @michalziobro you save my life

I wrote a small module called eks-auth to bridge the gap. Here is a code snippet from the complete example:
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = ">= 18.0.0"

  cluster_name = var.name
  vpc_id       = module.vpc.vpc_id
  subnet_ids   = module.vpc.private_subnets

  eks_managed_node_groups = {
    foo = {}
  }

  fargate_profiles = {
    bar = {}
  }
}

module "eks_auth" {
  source = "aidanmelen/eks-auth/aws"

  eks_aws_auth_configmap_yaml = module.eks.aws_auth_configmap_yaml

  map_roles = [
    {
      rolearn  = "arn:aws:iam::66666666666:role/role1"
      username = "role1"
      groups   = ["system:masters"]
    },
  ]

  map_users = [
    {
      userarn  = "arn:aws:iam::66666666666:user/user1"
      username = "user1"
      groups   = ["system:masters"]
    },
    {
      userarn  = "arn:aws:iam::66666666666:user/user2"
      username = "user2"
      groups   = ["system:masters"]
    },
  ]

  map_accounts = [
    "777777777777",
    "888888888888",
  ]
}

The aws-auth roles created by eks_managed_node_groups and fargate_profiles will be used to initialize a new aws-auth ConfigMap managed by Terraform.

@junaid-ali  I am following this in order to upgrade my EKS module from version 17.0.1 to 18.x.
Since I saw a lot of states needs to be moved around and removed.
Out of curiosity, will it impact live environment?
If so, what are the impacts?
How can I reduce the impact downtime?

@pen-pal I was able to upgrade without any downtime when:

old ASGs aren't removed during the upgrade (that's why I removed those from the state)
aws-auth configmap has old ASGs role attached otherwise your old ASG workers will cause authentication errors. You can remove the role once all the workloads are migrated to new ASG worker nodes.
Security groups allow traffic to the EKS control plane, and all the traffic between old ASG nodes and new ASGs are allowed (probably not needed to allow all traffic, but I allowed that to be on the safe side). If you require additional security group rules e.g for cross-region communication via vpc peering or to connect to a service outside of EKS, you should ensure that those rules are added to newly created ASGs.
Create a new pod on any of the new worker nodes, and test the connection to other pods, services and DNS resolution is working.
Once all the new ASGs are ready, I executed kubectl cordon <all old ASG nodes> + kubectl drain  to migrate the workloads. Ensure you have multiple replicas of critical workloads and pod disruption budgets are set up.

Hi @junaid-ali thank you.
few more queries that I had,
On step 1 you mentioned to create an rbac. Could you clarify it a littlle more.
Since, the module itself with integration of aws_auth as per example is supposed to create the aws_auth. How is this step to be executed?

@pen-pal This module no longer handles the aws-auth configmap: https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/UPGRADE-18.0.md#:~:text=Support for managing,provided by users). The authors have elaborated further in this issue as well. In my example above, I've a local var local.aws_auth_cm which leverages the module's rendered output var aws_auth_configmap_yaml to add users and additional roles to the configmap. We can then use the updated local var and apply it using kubectl in a null_resource

We went through an upgrade in one of our test environments, and we had some issues with the aws-auth configmap changes due to using Terraform Cloud. We have a wrapper around this upstream module, and we were able to manage the configmap by using:
resource "kubernetes_config_map" "aws_auth" {
  metadata {
    name      = "aws-auth"
    namespace = "kube-system"
    labels = {
      "app.kubernetes.io/managed-by" : "terraform"
    }
  }

  data = {
    mapRoles : yamlencode(concat(var.map_roles, [for group in module.eks.eks_managed_node_groups : {
      groups : ["system:bootstrappers", "system:nodes"]
      rolearn : group.iam_role_arn,
      username : "system:node:{{EC2PrivateDNSName}}"
    }]))
    mapAccounts : yamlencode(var.map_accounts)
    mapUsers : yamlencode(var.map_users)
  }
}

rather than trying to concat with the yaml output and manage it through exec commands. You'd have to modify the above if you're using fargate or self managed nodes. This also wouldn't support auth changes from inside the aws-auth-cm.tpl template in the future, but it works for us for now

We had to apply, wait for failure, remove the configmap, and then re-apply.

The aidanmelen/eks-auth/aws module solves this problem by using a Kubernetes job to replace the original aws-auth configmap with new merged aws-auth configmap managed by Terraform.

waves. i don't really have any solutions here, but i just spent the last 5 hours trying to rebuild one of my clusters after following the upgrade "guide". this is great!

Hi @junaid-ali thank you. few more queries that I had, On step 1 you mentioned to create an rbac. Could you clarify it a littlle more. Since, the module itself with integration of aws_auth as per example is supposed to create the aws_auth. How is this step to be executed?

What you want to do is create a separate backup Service Account that has the ability to edit the aws-auth configmap in case of any issues.
What I did was:

Create a new ServiceAccount called "backup-access".

apiVersion: v1
kind: ServiceAccount
metadata:
  name: backup-access

Create a ClusterRoleBinding, binding "backup-access" ServiceAccount to the "cluster-admin" ClusterRole.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: backup-access-cluster-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- namespace: default
  kind: ServiceAccount
  name: backup-user

Create a kubeconfig for this new Service Account. Check this StackOverflow answer for one way to create this. I use a GUI that did this for me.

Once you are finished with the migration, you should remove the ClusterRoleBinding and ServiceAccount

I keep getting this error while trying to carry out the upgrade process.
the module I am using also is used to create ingress resources at the same time once the eks cluster is up and running along with externalsecrets manager as well in my case
│ Error: Kubernetes cluster unreachable: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
│ Error: Failed to get RESTMapper client
│ cannot create discovery client: no client config
│ Error: Failed to get RESTMapper client
│ cannot create discovery client: no client config
│ Error: Failed to get RESTMapper client
│ cannot create discovery client: no client config
│ Error: Kubernetes cluster unreachable: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

You may need to set up a kubernetes config, the old cluster created one, the new does not.
something like this:
data "aws_eks_cluster" "cluster" {
  name = module.eks.cluster_id
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_id
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  token                  = data.aws_eks_cluster_auth.cluster.token
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
}

@bryantbiggs Sure, youre welcome to take a look at the plan diff 👍 https://pastebin.com/Zr0kJTtE
  prefix_separator                   = ""
  iam_role_name                      = local.name
  cluster_security_group_name        = local.name
  cluster_security_group_description = "EKS cluster security group."

state changed up until now.
terraform state rm 'kubernetes_secret.aws-secret'
terraform state rm 'module.eks.kubernetes_config_map.aws_auth[0]'
terraform state rm $(terraform state list | grep autoscaling_group.workers)
terraform state rm $(terraform state list | grep aws_launch_configuration)
terraform state rm $(terraform state list | grep module.eks.random_pet)
terraform state rm $(terraform state list | grep aws_iam_instance_profile)
terraform state rm $(terraform state list | grep aws_security_group_rule.workers)
terraform state rm $(terraform state list | grep aws_security_group_rule.cluster)
terraform state mv 'module.eks.aws_iam_role.cluster[0]' 'aws_iam_role.this[0]'
terraform state rm 'module.eks.aws_iam_role_policy_attachment.workers_AmazonEC2ContainerRegistryReadOnly[0]'
terraform state rm 'module.eks.aws_iam_role_policy_attachment.workers_AmazonEKSWorkerNodePolicy[0]'
terraform state rm 'module.eks.aws_iam_role_policy_attachment.workers_AmazonEKS_CNI_Policy[0]'

remaining
terraform state list
data.aws_eks_cluster.cluster
data.aws_eks_cluster_auth.cluster
data.terraform_remote_state.networking
aws_iam_role.this[0]
module.eks.data.aws_ami.eks_worker
module.eks.data.aws_ami.eks_worker_windows
module.eks.data.aws_caller_identity.current
module.eks.data.aws_iam_policy_document.cluster_assume_role_policy
module.eks.data.aws_iam_policy_document.cluster_elb_sl_role_creation[0]
module.eks.data.aws_iam_policy_document.workers_assume_role_policy
module.eks.data.aws_partition.current
module.eks.data.template_file.userdata[0]
module.eks.data.template_file.userdata[1]
module.eks.data.template_file.userdata[2]
module.eks.aws_eks_cluster.this[0]
module.eks.aws_iam_openid_connect_provider.oidc_provider[0]
module.eks.aws_iam_policy.cluster_elb_sl_role_creation[0]
module.eks.aws_iam_role.workers[0]
module.eks.aws_iam_role_policy_attachment.cluster_AmazonEKSClusterPolicy[0]
module.eks.aws_iam_role_policy_attachment.cluster_AmazonEKSServicePolicy[0]
module.eks.aws_iam_role_policy_attachment.cluster_AmazonEKSVPCResourceControllerPolicy[0]
module.eks.aws_iam_role_policy_attachment.cluster_elb_sl_role_creation[0]
module.eks.aws_security_group.cluster[0]
module.eks.aws_security_group.workers[0]
module.eks.local_file.kubeconfig[0]
module.eks.null_resource.wait_for_cluster[0]

@kaykhancheckpoint  could you share the plan diff again if you have it handy.

You may need to set up a kubernetes config, the old module created one, the new does not. something like this:
data "aws_eks_cluster" "cluster" {
  name = module.eks.cluster_id
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_id
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  token                  = data.aws_eks_cluster_auth.cluster.token
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
}

@jcam  I have this on my setup yet it fails.

@junaid-ali still getting the same error regarding connection refused even though I have followed all the steps as mentioned.
module.eks.helm_release.alb-controller: Refreshing state... [id=aws-load-balancer-controller]
module.eks.kubernetes_service_account.alb-serviceaccount: Refreshing state... [id=kube-system/aws-load-balancer-controller]
╷
│ Warning: Resource targeting is in effect
│ You are creating a plan with the -target option, which means that the
│ result of this plan may not represent all of the changes requested by the
│ current configuration.
│ The -target option is not for routine use, and is provided only for
│ exceptional situations such as recovering from errors or mistakes, or when
│ Terraform specifically suggests to use it as part of an error message.
│ Error: Get "http://localhost/api/v1/namespaces/kube-system/serviceaccounts/aws-load-balancer-controller": dial tcp <IP_ADDRESS>:80: connect: connection refused
│ Error: Kubernetes cluster unreachable: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

In my case I also have alb-load-balancer-controller just besides eks cluster module.
So basically during the eks cluster setup, at the same time alb-controller is also being setup on this side.
folder structure is somewhat like this
main.tf (calls eks.tf which eventually calls eks module in the cloud)
| -> modules
       |-> eks.tf (calls eks module)
       |-> ingress.tf (create ingress alb using helm)

@pen-pal are you able to run kubectl against your cluster? If yes, you can use a null_resource to check if you are getting values for data.aws_eks_cluster_auth.cluster.token, data.aws_eks_cluster.cluster.endpoint and base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data) which are being used by the kubernetes provider

My solution when I had problems like you're running into (this was back on eks module v13) was to split things up... have one terraform run build and deploy the cluster itself, and a separate terraform run in a separate directory build and deploy all the helm charts and kubernetes resource entries onto it...

I would separate it first, and upgrade second. That way there's no chance the EKS cluster upgrade terraform run could impact all your deployed applications.
I don't use terraform cloud, but with my backend I simply did a terraform state pull, made the new folder for all the app components and did a terraform init there, did a terraform state push, then did terraform state rm for all the app-components in the cluster folder, and a terraform state rm for all the cluster components in the app folder

I would separate it first, and upgrade second. That way there's no chance the EKS cluster upgrade terraform run could impact all your deployed applications.
I don't use terraform cloud, but with my backend I simply did a terraform state pull, made the new folder for all the app components and did a terraform init there, did a terraform state push, then did terraform state rm for all the app-components in the cluster folder, and a terraform state rm for all the cluster components in the app folder

were there no conflicts with the resource names? for example for logging of alb-load-balancer is done in  a s3 bucket and that bucket is also created alongside the eks cluster and is inside ingress.tf

I just needed to split things so they were in one place or the other and not both. in your case, I would put the logging bucket in the app deploy stage, or I would keep it in the cluster stage and use a data object in the app stage instead of a resource object

I would separate it first, and upgrade second. That way there's no chance the EKS cluster upgrade terraform run could impact all your deployed applications.

I don't use terraform cloud, but with my backend I simply did a terraform state pull, made the new folder for all the app components and did a terraform init there, did a terraform state push, then did terraform state rm for all the app-components in the cluster folder, and a terraform state rm for all the cluster components in the app folder

do you have any link or directions I can follow for the steps you mentioned below. I am kind of confused to be honest for the steps you mentioned

@pen-pal are you able to run kubectl against your cluster? If yes, you can use a null_resource to check if you are getting values for data.aws_eks_cluster_auth.cluster.token, data.aws_eks_cluster.cluster.endpoint and base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data) which are being used by the kubernetes provider

@junaid-ali no it still does not get the config for some reason even though the steps are followed.

This issue has been resolved in version 18.19.0 :tada:

@junaid-ali folllowed the exact steps you mentioned, however in my case asg are being deleted

BTW if someone needs to solve this via a PR without direct access to state commands you need to "fork" the module temporarily in order to be able to use the moved block:

To reduce coupling between separately-packaged modules, Terraform only allows declarations of moves between modules in the same package. In other words, Terraform would not have allowed moving into module.x above if the source address of that call had not been a local path.

With v18 I am unable to configure a cluster so that pods have network access.
Here's a simple cluster in v17:
provider "aws" {
  region = "eu-central-1"
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_id
}

provider "kubernetes" {
  host                   = module.eks.cluster_endpoint
  token                  = data.aws_eks_cluster_auth.cluster.token
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
}

data "aws_availability_zones" "available" {}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "3.2.0"

  name                 = "test-cluster-vpc"
  cidr                 = "<IP_ADDRESS>/16"
  azs                  = data.aws_availability_zones.available.names
  private_subnets      = ["<IP_ADDRESS>/24", "<IP_ADDRESS>/24", "<IP_ADDRESS>/24"]
  public_subnets       = ["<IP_ADDRESS>/24", "<IP_ADDRESS>/24", "<IP_ADDRESS>/24"]
  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true
}

module "eks" {
  source          = "terraform-aws-modules/eks/aws"
  version         = "17.24.0"
  cluster_name    = "test-cluster"
  cluster_version = "1.22"
  subnets         = module.vpc.private_subnets

  vpc_id = module.vpc.vpc_id

  node_groups = [
    {
      instance_type = "t2.small"
      capacity_type = "SPOT"
    }
  ]
}

If I apply this, I can then run a pod on it and ping an internet host:
$ kubectl run -it --attach --rm --image=alpine/k8s:1.22.9 andre-temp
If you don't see a command prompt, try pressing enter.
/apps # ping <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>): 56 data bytes
64 bytes from <IP_ADDRESS>: seq=0 ttl=53 time=2.157 ms

If I adapt the terraform file to v18:
[...]

module "eks" {
  source          = "terraform-aws-modules/eks/aws"
  version         = "~> 18"
  cluster_name    = "test-cluster"
  cluster_version = "1.22"
  subnet_ids         = module.vpc.private_subnets

  vpc_id = module.vpc.vpc_id

  eks_managed_node_groups = [
    main = {
      instance_type = "t2.small"
      capacity_type = "SPOT"
    }
  ]
}

This doesn't work anymore:
$ kubectl run -it --attach --rm --image=alpine/k8s:1.22.9 andre-temp
If you don't see a command prompt, try pressing enter.
/apps # ping <IP_ADDRESS>
PING <IP_ADDRESS> (<IP_ADDRESS>): 56 data bytes
^C
--- <IP_ADDRESS> ping statistics ---
55 packets transmitted, 0 packets received, 100% packet loss

@AndreKR https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/network_connectivity.md#security-groups

Ah, there it is. The link in the migration guide is broken.

So, I think I finally worked out what security groups there are for a simple cluster with an eks_managed_node_group in version 18.23.0 of this module. Leaving an overview here for anyone interested:

Name
Created by
Attached to
Default rules
Opt out

default
AWS EKS
Nothing
Allow all traffic
-

CLUSTER-cluster
Terraform module,main.tf
Nothing
Allow some traffic
create_cluster_security_group

eks-cluster-sg-CLUSTER-NNN
AWS EKS
Nothing
Allow all traffic
-

NODEGROUP-eks-node-group
Terraform module,eks-managed-node-group/main.tf
Nodes
None
eks_managed_node_groups -><nodegroup> ->create_security_group

CLUSTER-node
Terraform module,node_groups.tf
Nodes
Allow some traffic
create_node_security_group

Hi
I have tried to update from 17.24.0 to 18.x however terraform wants to destroy my cluster and recreate a new one. I have added all these variable as mentioned above but without any sucess.
  cluster_security_group_description = "EKS cluster security group."
  cluster_security_group_name = var.cluster_name
  prefix_separator = ""
  iam_role_arn = "$IAM_ROLE_ARN"

For testing I did not include workers
17.24.0 Config
module "eks" {
  source           = "terraform-aws-modules/eks/aws"
  version          = "17.24.0"

  cluster_name     = var.cluster_name
  cluster_version  = var.cluster_version
  vpc_id           = var.vpc_id
  subnets          = var.subnet
  tags = {
    GithubRepo  = "terraform-aws-eks"
    GithubOrg   = "terraform-aws-modules"
    environment = var.environment
  }

}
data "aws_eks_cluster" "cluster" {
  name = module.eks.cluster_id
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_id
}

18.24.1 config
module "eks" {
  source           = "terraform-aws-modules/eks/aws"
  version          = "18.24.1"

  cluster_name     = var.cluster_name
  cluster_version  = var.cluster_version
  vpc_id           = var.vpc_id
  subnet_ids          = var.subnet
  cluster_security_group_description = "EKS cluster security group."
  cluster_security_group_name = var.cluster_name
  prefix_separator = ""
  iam_role_arn = "$IAM_ROLE_ARN"
  tags = {
    GithubRepo  = "terraform-aws-eks"
    GithubOrg   = "terraform-aws-modules"
    environment = var.environment
  }

}
data "aws_eks_cluster" "cluster" {
  name = module.eks.cluster_id
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_id
}

This forces the replacement
        name                      = "sandbox"
      ~ platform_version          = "eks.7" -> (known after apply)
      ~ role_arn                  = "$IAM_ROLE_ARN" -> (known after apply) # forces replacement
      ~ status                    = "ACTIVE" -> (known after apply)

@isatfg see https://github.com/clowdhaus/eks-v17-v18-migrate#control-plane-changes

Thanks, @ArchiFleKs for steps! One additional step I had to define because I used:
manage_aws_auth_configmap = true

  aws_auth_roles = [
    {
      rolearn  = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/OLD_NODE_GROUP_IAM_ROLE_NAME"
      username = "system:node:{{EC2PrivateDNSName}}"
      groups   = ["system:bootstrappers", "system:nodes"]
    },
  ]

If this is not defined, nodes will become unreachable and all deployments on it.

If this is not defined, nodes will become unreachable and all deployments on it.

This is very important! I did this for one environment and it worked well, I was able to gradually drain and terminate old nodes. I forgot this step for another environment and right after terraform apply all the old nodes were lost instantly, major downtime 😱

@qlikcoe @dusansusic it was mentioned later by couple of others as well, github has collapsed majority of that discussion. For example, this was my experience: https://github.com/terraform-aws-modules/terraform-aws-eks/issues/1744#issuecomment-1051694698


Request for Scroll Event Notification

Feature request: I needed an event that triggers whenever there a change in the status of carousel scrolling (started/stopped). in order to disable/enable gesture interactions with another control (jamesmontemagno: PullToRefresh to be specific). I ended up solving the problem by subclassing CarouselViewRenderer and accessing viewPager through reflection, which contained scrolling information which I then used to call on a custom event.
It took me a while to figure out how to solve the issue. I think that a built in "ScrollStatusChanged" event would be beneficial.

Have you checked PositionSelected event? It happens exactly after scrolling finish. I think o can add Scrolling started but finished will be same as PositionSelected.

Yes I tried that, but PositionSelected only triggers when a page in the CarouselView snaps to position. What I needed was an event that triggers even with tiny scrolling.
My case is very specific, which is a CarouselViewControl inside the PullToRefreshLayout. What happens is if I scroll the CarouselViewControl horizontally, and then if there is even slight gesture down, then the PullToRefreshLayout takes over, and the CarouselViewControl is reset, which is annoying from the user experience perspective.

Then you want to know when the control is being scrolled

yes

Also consider that knowing the exact scrolling would allow linking with other controls to create a nice scroll parallax effect


python310Packages.angr: 9.2.2 -> 9.2.3

Description of changes
Update angr stack.

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 172457 run on aarch64-darwin 1

2 packages failed to build:

python310Packages.angrcli
python39Packages.angrcli

14 packages built:

python310Packages.ailment
python310Packages.angr
python310Packages.angrop
python310Packages.archinfo
python310Packages.claripy
python310Packages.cle
python310Packages.pyvex
python39Packages.ailment
python39Packages.angr
python39Packages.angrop
python39Packages.archinfo
python39Packages.claripy
python39Packages.cle
python39Packages.pyvex

The aarch64-darwin failures are not introduced by this update.