fix(eks): missing support for "InstanceTypes" attribute assignment for AL2023 AMIs #29505

guessi · 2024-03-15T10:12:33Z

Issue # (if applicable)

Reason for this change

After #29335, @aws-eks should receive AL2023 support, despites the GPU types are not yet supported it should at least allow user to customize instance type. However, missing support for NodegroupAmiType[] causing validation error emit, so that user can only create default instance types ( t3.medium or t4g.medium ).

$ cat ./lib/cluster.ts
...
    cluster.addNodegroupCapacity("mng-al2023", {
      amiType: eks.NodegroupAmiType.AL2023_X86_64_STANDARD,
      instanceTypes: [new ec2.InstanceType("t3.medium")],
      ...
    });
...

$ npx cdk version
2.133.0 (build dcc1e75)

$  npx cdk synth

...
                                                                                                                                                                                ^
Error: The specified AMI does not match the instance types architecture, either specify one of AL2_X86_64, BOTTLEROCKET_X86_64, WINDOWS_CORE_2019_X86_64, WINDOWS_CORE_2022_X86_64, WINDOWS_FULL_2019_X86_64, WINDOWS_FULL_2022_X86_64 or don't specify any
    at new Nodegroup (.../node_modules/aws-cdk-lib/aws-eks/lib/managed-nodegroup.js:1:3921)
    at Cluster.addNodegroupCapacity (.../node_modules/aws-cdk-lib/aws-eks/lib/cluster.js:1:19807)
    at EksCluster.createManagedNodeGroups (.../lib/cluster.ts:85:13)
    at new EksCluster (.../lib/cluster.ts:30:10)
    at Object.<anonymous> (.../bin/eks-basic.ts:10:1)
    at Module._compile (node:internal/modules/cjs/loader:1376:14)
    at Module.m._compile (.../node_modules/ts-node/src/index.ts:1618:23)
    at Module._extensions..js (node:internal/modules/cjs/loader:1435:10)
    at Object.require.extensions.<computed> [as .ts] (.../node_modules/ts-node/src/index.ts:1621:12)
    at Module.load (node:internal/modules/cjs/loader:1207:32)

Subprocess exited with error 1

Description of changes

Add eks.NodegroupAmiType.AL2023_X86_64_STANDARD and eks.NodegroupAmiType.AL2023_ARM_64_STANDARD support for node group module.

Description of how you validated changes

$ npx jest aws-eks/test/cluster.test.ts
 PASS  aws-eks/test/cluster.test.ts (37.298 s)
...

=============================== Coverage summary ===============================
Statements   : 48.86% ( 10077/20621 )
Branches     : 27.88% ( 2388/8565 )
Functions    : 33.03% ( 1509/4568 )
Lines        : 49.71% ( 9905/19923 )
================================================================================
Jest: "global" coverage threshold for statements (55%) not met: 48.86%
Jest: "global" coverage threshold for branches (35%) not met: 27.88%
Test Suites: 1 passed, 1 total
Tests:       119 passed, 119 total
Snapshots:   0 total
Time:        41.064 s

$ npx jest aws-eks/test/nodegroup.test.ts
 RUNS  aws-eks/test/nodegroup.test.ts
...
=============================== Coverage summary ===============================
Statements   : 43.94% ( 9062/20621 )
Branches     : 22.08% ( 1892/8565 )
Functions    : 27.16% ( 1241/4568 )
Lines        : 44.8% ( 8927/19923 )
================================================================================
Jest: "global" coverage threshold for statements (55%) not met: 43.94%
Jest: "global" coverage threshold for branches (35%) not met: 22.08%
Test Suites: 1 passed, 1 total
Tests:       59 passed, 59 total
Snapshots:   0 total
Time:        23.334 s

Checklist

My code adheres to the CONTRIBUTING GUIDE and DESIGN GUIDELINES

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

aws-cdk-automation

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

A comment requesting an exemption should contain the text Exemption Request. Additionally, if clarification is needed add Clarification Request to a comment.

guessi · 2024-03-15T14:15:37Z

packages/aws-cdk-lib/aws-eks/lib/managed-nodegroup.ts

-const arm64AmiTypes: NodegroupAmiType[] = [NodegroupAmiType.AL2_ARM_64, NodegroupAmiType.BOTTLEROCKET_ARM_64];
-const x8664AmiTypes: NodegroupAmiType[] = [NodegroupAmiType.AL2_X86_64, NodegroupAmiType.BOTTLEROCKET_X86_64,
+const arm64AmiTypes: NodegroupAmiType[] = [NodegroupAmiType.AL2_ARM_64, NodegroupAmiType.AL2023_ARM_64_STANDARD, NodegroupAmiType.BOTTLEROCKET_ARM_64];
+const x8664AmiTypes: NodegroupAmiType[] = [NodegroupAmiType.AL2_X86_64, NodegroupAmiType.AL2023_X86_64_STANDARD, NodegroupAmiType.BOTTLEROCKET_X86_64,


It's the key to the issue, missing definitions for AmiType: AL2023_[ARM_64|X86_64]_STANDARD.

✅ Updated pull request passes all PRLinter validations. Dismissing previous PRLinter review.

pahud · 2024-03-21T15:06:47Z

Thank you. This PR LGTM. Are you able to update the integ.eks-cluster.ts?

And you'll need to fix the CI before we can move this PR forward.

packages/aws-cdk-lib/aws-eks/test/cluster.test.ts

guessi · 2024-03-22T04:38:29Z

packages/aws-cdk-lib/aws-eks/test/nodegroup.test.ts

+    // THEN
+    expect(() => cluster.addNodegroupCapacity('ng', {
+      amiType: NodegroupAmiType.AL2023_X86_64_STANDARD,
+      instanceTypes: [
+        new ec2.InstanceType('c6g.large'),
+        new ec2.InstanceType('t4g.large'),
+      ],
+    })).toThrow(/The specified AMI does not match the instance types architecture, either specify one of AL2_ARM_64, AL2023_ARM_64_STANDARD, BOTTLEROCKET_ARM_64 or don't specify any/);
+  });


To ensure MNG with LT could capture invalid instance type exception.

guessi · 2024-03-22T04:38:39Z

packages/aws-cdk-lib/aws-eks/test/nodegroup.test.ts

+    // THEN
+    expect(() => cluster.addNodegroupCapacity('ng', {
+      amiType: NodegroupAmiType.AL2023_ARM_64_STANDARD,
+      instanceTypes: [
+        new ec2.InstanceType('m5.large'),
+        new ec2.InstanceType('c5.large'),
+      ],
+    })).toThrow(/The specified AMI does not match the instance types architecture, either specify one of AL2_X86_64, AL2023_X86_64_STANDARD, BOTTLEROCKET_X86_64, WINDOWS_CORE_2019_X86_64, WINDOWS_CORE_2022_X86_64, WINDOWS_FULL_2019_X86_64, WINDOWS_FULL_2022_X86_64 or don't specify any/);
+  });


To ensure MNG with LT could capture invalid instance type exception.

guessi · 2024-03-22T04:39:53Z

packages/aws-cdk-lib/aws-eks/test/nodegroup.test.ts

+          Ref: 'VPCPrivateSubnet2SubnetCFCDAA7A',
+        },
+      ],
+      AmiType: 'AL2023_x86_64_STANDARD',


To ensure MNG could be created with no instance type specified.

guessi · 2024-03-22T04:40:47Z

packages/aws-cdk-lib/aws-eks/lib/managed-nodegroup.ts

+];
+const x8664AmiTypes: NodegroupAmiType[] = [
+  NodegroupAmiType.AL2_X86_64,
+  NodegroupAmiType.AL2023_X86_64_STANDARD,


It's the core to the issue, missing definition for AL2023.

guessi · 2024-03-22T04:42:55Z

packages/@aws-cdk-testing/framework-integ/test/aws-eks/test/integ-tests-kubernetes-version.ts

+const versionMap: { [key: string]: any } = {
+  1.24: KubectlV24Layer,
+  1.29: KubectlV29Layer,
+};


To make sure it could compatible with existing integ tests.

guessi · 2024-03-22T04:43:53Z

@pahud please have a look while you are free, thanks!

pahud

LGTM. We'll need another review and approval from the maintainers.

GavinZZ

Can we please add an example to README.md with AL2023_ARM_64_STANDARD usage example?

guessi · 2024-03-23T07:14:08Z

packages/aws-cdk-lib/aws-eks/README.md

@@ -187,7 +187,6 @@ cluster.addNodegroupCapacity('custom-node-group', {
  instanceTypes: [new ec2.InstanceType('m5.large')],
  minSize: 4,
  diskSize: 100,
-  amiType: eks.NodegroupAmiType.AL2_X86_64_GPU,


Reason for why removing this line

The section here should focus on addNodegroupCapacity but not amiType.

Instance type here m5.large does not support GPU which is misleading.

guessi · 2024-03-23T07:22:31Z

packages/aws-cdk-lib/aws-eks/README.md

+  instanceTypes: [new ec2.InstanceType('m6g.medium')], // NOTE: if amiType is ARM-based image, the instance types here must be ARM-based.
+  amiType: eks.NodegroupAmiType.AL2023_ARM_64_STANDARD,
+});
+```


Discussion: should we address AL2023 here? or maybe keep it as default AL2_ARM_64.

guessi · 2024-03-23T07:22:52Z

packages/aws-cdk-lib/aws-eks/README.md

+// X86_64 based AMI managed node group
+cluster.addNodegroupCapacity('custom-node-group', {
+  instanceTypes: [new ec2.InstanceType('m5.large')], // NOTE: if amiType is x86_64-based image, the instance types here must be x86_64-based.
+  amiType: eks.NodegroupAmiType.AL2023_X86_64_STANDARD,


Discussion: should we address AL2023 here? or maybe keep it as default AL2_X86_64.

guessi · 2024-03-23T08:18:48Z

@GavinZZ I've updated the README.md as separated commit, please review once again, thanks!

As discussed with @pahud offline, I'm here leaving change for ARM64 Support section untouched as making change to addAutoScalingGroupCapacity() would need to make change to the enum of MachineImageType and the would trigger integration tests for cluster.ts, which is not we are going to see at this PR.

I can see there are many if... else... blocks for MachineImageType detection logic for bootstraping

I think AL2023 support for addAutoScalingGroupCapacity() would better to be done in separated PR, at this PR here we should focus on MNG first... WDYT?

mergify · 2024-03-25T16:40:15Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

GavinZZ · 2024-03-25T20:31:37Z

@guessi can you please fix the merge conflict and this PR should be good to go.

Pull request has been modified.

guessi · 2024-03-26T00:19:14Z

@GavinZZ Just rebased, should have no conflict now 👍

aws-cdk-automation · 2024-03-26T00:45:42Z

AWS CodeBuild CI Report

CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
Commit ID: 86b3169
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mergify · 2024-03-26T18:48:10Z

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

github-actions bot added the repeat-contributor [Pilot] contributed between 3-5 PRs to the CDK label Mar 15, 2024

aws-cdk-automation requested a review from a team March 15, 2024 10:12

github-actions bot added the p2 label Mar 15, 2024

aws-cdk-automation previously requested changes Mar 15, 2024

View reviewed changes

guessi force-pushed the missing-instance-types-support-al2023 branch from aee3332 to 81ab615 Compare March 15, 2024 14:12

guessi commented Mar 15, 2024

View reviewed changes

guessi force-pushed the missing-instance-types-support-al2023 branch 3 times, most recently from d87fba6 to 80f6bbc Compare March 15, 2024 15:37

guessi force-pushed the missing-instance-types-support-al2023 branch from 80f6bbc to a9dd543 Compare March 15, 2024 15:43

guessi requested a review from aws-cdk-automation March 15, 2024 16:14

github-actions bot mentioned this pull request Mar 18, 2024

Weekly PR metrics report #29527

Closed

guessi force-pushed the missing-instance-types-support-al2023 branch from a9dd543 to a181afc Compare March 21, 2024 14:42

guessi mentioned this pull request Mar 21, 2024

fix(eks): support AL2023 #29550

Closed

1 task

github-actions bot added bug This issue is a bug. effort/medium Medium work item – several days of effort p1 and removed p2 labels Mar 21, 2024

pahud reviewed Mar 21, 2024

View reviewed changes

packages/aws-cdk-lib/aws-eks/test/cluster.test.ts Outdated Show resolved Hide resolved

pahud reviewed Mar 21, 2024

View reviewed changes

packages/aws-cdk-lib/aws-eks/test/cluster.test.ts Outdated Show resolved Hide resolved

guessi force-pushed the missing-instance-types-support-al2023 branch from a181afc to c48d46b Compare March 22, 2024 03:38

guessi requested a review from pahud March 22, 2024 03:39

guessi force-pushed the missing-instance-types-support-al2023 branch from c48d46b to 811afb5 Compare March 22, 2024 03:59

aws-cdk-automation added the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Mar 22, 2024

guessi commented Mar 22, 2024

View reviewed changes

pahud reviewed Mar 22, 2024

View reviewed changes

GavinZZ self-assigned this Mar 22, 2024

GavinZZ reviewed Mar 22, 2024

View reviewed changes

guessi commented Mar 23, 2024

View reviewed changes

guessi force-pushed the missing-instance-types-support-al2023 branch from 18de6ee to 5d74fdb Compare March 23, 2024 07:20

guessi had a problem deploying to test-pipeline March 23, 2024 07:21 — with GitHub Actions Failure

guessi commented Mar 23, 2024

View reviewed changes

guessi force-pushed the missing-instance-types-support-al2023 branch from 5d74fdb to 0bee78e Compare March 23, 2024 08:04

guessi requested review from GavinZZ and pahud March 23, 2024 08:11

GavinZZ previously approved these changes Mar 25, 2024

View reviewed changes

guessi added 2 commits March 26, 2024 08:16

fix(eks): missing attribute support for AL2023 nodegroups creation

49815c5

doc(eks): fix(eks): update support for AL2023 nodegroups creation

86b3169

guessi force-pushed the missing-instance-types-support-al2023 branch from 0bee78e to 86b3169 Compare March 26, 2024 00:17

guessi requested a review from GavinZZ March 26, 2024 00:24

GavinZZ approved these changes Mar 26, 2024

View reviewed changes

mergify bot merged commit e77ce26 into aws:main Mar 26, 2024
12 checks passed

aws-cdk-automation removed the pr/needs-maintainer-review This PR needs a review from a Core Team Member label Mar 26, 2024

github-actions bot mentioned this pull request Apr 1, 2024

Monthly PR metrics report #29668

Closed

guessi deleted the missing-instance-types-support-al2023 branch April 2, 2024 08:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(eks): missing support for "InstanceTypes" attribute assignment for AL2023 AMIs #29505

fix(eks): missing support for "InstanceTypes" attribute assignment for AL2023 AMIs #29505

guessi commented Mar 15, 2024 •

edited by pahud

aws-cdk-automation left a comment

guessi Mar 15, 2024

pahud commented Mar 21, 2024

guessi Mar 22, 2024 •

edited

guessi Mar 22, 2024 •

edited

guessi Mar 22, 2024

guessi Mar 22, 2024

guessi Mar 22, 2024

guessi commented Mar 22, 2024

pahud left a comment

GavinZZ left a comment

guessi Mar 23, 2024

guessi Mar 23, 2024

guessi Mar 23, 2024

guessi commented Mar 23, 2024

mergify bot commented Mar 25, 2024

GavinZZ commented Mar 25, 2024

guessi commented Mar 26, 2024

aws-cdk-automation commented Mar 26, 2024

mergify bot commented Mar 26, 2024

fix(eks): missing support for "InstanceTypes" attribute assignment for AL2023 AMIs #29505

fix(eks): missing support for "InstanceTypes" attribute assignment for AL2023 AMIs #29505

Conversation

guessi commented Mar 15, 2024 • edited by pahud

Issue # (if applicable)

Reason for this change

Description of changes

Description of how you validated changes

Checklist

aws-cdk-automation left a comment

Choose a reason for hiding this comment

guessi Mar 15, 2024

Choose a reason for hiding this comment

pahud commented Mar 21, 2024

guessi Mar 22, 2024 • edited

Choose a reason for hiding this comment

guessi Mar 22, 2024 • edited

Choose a reason for hiding this comment

guessi Mar 22, 2024

Choose a reason for hiding this comment

guessi Mar 22, 2024

Choose a reason for hiding this comment

guessi Mar 22, 2024

Choose a reason for hiding this comment

guessi commented Mar 22, 2024

pahud left a comment

Choose a reason for hiding this comment

GavinZZ left a comment

Choose a reason for hiding this comment

guessi Mar 23, 2024

Choose a reason for hiding this comment

guessi Mar 23, 2024

Choose a reason for hiding this comment

guessi Mar 23, 2024

Choose a reason for hiding this comment

guessi commented Mar 23, 2024

mergify bot commented Mar 25, 2024

GavinZZ commented Mar 25, 2024

guessi commented Mar 26, 2024

aws-cdk-automation commented Mar 26, 2024

AWS CodeBuild CI Report

mergify bot commented Mar 26, 2024

guessi commented Mar 15, 2024 •

edited by pahud

guessi Mar 22, 2024 •

edited

guessi Mar 22, 2024 •

edited