Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ec2: InvalidPermission.NotFound: The specified rule does not exist in this security group when creating RestrictDefaultSecurityGroupCustomResource #26390

Closed
alehuo opened this issue Jul 17, 2023 · 5 comments · Fixed by #27039
Labels
@aws-cdk/aws-ec2 Related to Amazon Elastic Compute Cloud bug This issue is a bug. effort/medium Medium work item – several days of effort p2

Comments

@alehuo
Copy link

alehuo commented Jul 17, 2023

Describe the bug

After updating aws-cdk-lib from 2.77.0 to 2.87.0 the following error occurs with the @aws-cdk/aws-ec2:restrictDefaultSecurityGroup feature flag set to true. We had the feature flag set to true prior to the update, and there were no errors regarding the deployment.

Expected Behavior

cdk deploy should complete with no UPDATE_ROLLBACK_COMPLETE errors on the CloudFormation stack.

Current Behavior

The stack deployment fails, because the creation of RestrictDefaultSecurityGroupCustomResource fails. The following error occurs:

1:57:09 PM | CREATE_FAILED        | Custom::VpcRestrictDefaultSG                | Rest...omResource/Default
Received response status [FAILED] from custom resource. Message returned: InvalidPermission.NotFound: The specified rule does not exist in this security group.
at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/services/ec2.js:50:35)
at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:686:14)
at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:688:12)
at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:116:18) (RequestId: 0b911b0b-c81b-439a-9591-83cb123924af)
1:57:10 PM | UPDATE_ROLLBACK_IN_P | AWS::CloudFormation::Stack                  | network-stack
The following resource(s) failed to create: [vpcRestrictDefaultSecurityGroupCustomResource].
1:57:37 PM | UPDATE_ROLLBACK_COMP | AWS::CloudFormation::Stack                  | network-stack
1:57:41 PM | DELETE_IN_PROGRESS   | AWS::Lambda::Function                       | Custom::VpcRestric...ceProvider/Handler
1:57:09 PM | CREATE_FAILED        | Custom::VpcRestrictDefaultSG                | vpcRestric...omResource
Received response status [FAILED] from custom resource. Message returned: InvalidPermission.NotFound: The specified rule does not exist in this security group.
at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/services/ec2.js:50:35)
at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:686:14)
at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:688:12)
at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:116:18) (RequestId: 0b911b0b-c81b-439a-9591-83cb123924af)

Reproduction Steps

Use aws-cdk-lib version 2.77.0, set the feature flag ec2:restrictDefaultSecurityGroup to true. Create a new VPC and deploy it:

 const vpc = new Vpc(scope, id, {
    cidr: "10.4.0.0/21",
    maxAzs: 3,
    subnetConfiguration: [
      {
        cidrMask: 24,
        name: 'public-subnet',
        subnetType: SubnetType.PUBLIC
      },
      {
        cidrMask: 24,
        name: 'private-subnet',
        subnetType: SubnetType.PRIVATE_WITH_EGRESS
      },
      {
        cidrMask: 27,
        name: 'isolated-subnet',
        subnetType: SubnetType.PRIVATE_ISOLATED
      }
    ]
  })

After deployment, upgrade aws-cdk-lib to 2.87.0 and run cdk diff - you should see a RestrictDefaultSecurityGroupCustomResource resource being created. At least in our case aws-cdk-lib version 2.87.0 wanted to create the custom resource. Run cdk deploy and you should get an error about InvalidPermission.NotFound.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.87.0 (build 9fca790)

Framework Version

No response

Node.js Version

v16.18.1

OS

MacOS Ventura 13.4.1, ARM64

Language

Typescript

Language Version

TypeScript (5.1.6)

Other information

No response

@alehuo alehuo added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jul 17, 2023
@github-actions github-actions bot added the @aws-cdk/aws-ec2 Related to Amazon Elastic Compute Cloud label Jul 17, 2023
@pahud pahud changed the title @aws-cdk/aws-ec2: InvalidPermission.NotFound: The specified rule does not exist in this security group when creating RestrictDefaultSecurityGroupCustomResource ec2: InvalidPermission.NotFound: The specified rule does not exist in this security group when creating RestrictDefaultSecurityGroupCustomResource Jul 17, 2023
@khushail khushail self-assigned this Jul 17, 2023
@khushail khushail added investigating This issue is being investigated and/or work is in progress to resolve the issue. response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. and removed investigating This issue is being investigated and/or work is in progress to resolve the issue. needs-triage This issue or PR still needs to be triaged. labels Jul 17, 2023
@khushail
Copy link
Contributor

Hi @alehuo , I am able to reproduce this and got the mentioned resource diff -
image

and deployment is also succeeded -
image

Could you check your account settings to see if its not an issue with the enovironment

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Jul 18, 2023
@khushail khushail added the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Jul 18, 2023
@alehuo
Copy link
Author

alehuo commented Jul 19, 2023

I am now testing if it is environment related. I'm getting the same diff as you.

@alehuo
Copy link
Author

alehuo commented Jul 19, 2023

Apparently, the default security group of the VPC has no inbound or outbound rules defined, and in our case AWS CDK assumes it has and the deployment fails. What kind of rules does the default security group of your created VPC have?

@alehuo
Copy link
Author

alehuo commented Jul 19, 2023

Digging deeper into this issue:

If I create a completely new VPC with restrictDefaultSecurityGroup set to false, the deployment completes successfully. VPCs that are newly created this way have the following inbound & outbound rules:

Inbound: Allow all traffic
Outbound: Allow all IPv4 traffic

The deployment of a new VPC also succeeded with restrictDefaultSecurityGroup set to true. If restrictDefaultSecurityGroup is set to true, the custom lambda attempts to delete these rules.

I suspect that if the VPC's default security group did not have any inbound or outbound rules and later on we switch the restrictDefaultSecurityGroup flag to true, the deployment fails. I tried toggling between true and false on the parameter, but I did not manage to replicate the behaviour that exists in our environment.

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Jul 19, 2023
@khushail khushail added p2 effort/medium Medium work item – several days of effort labels Aug 22, 2023
@khushail khushail removed their assignment Aug 22, 2023
@mergify mergify bot closed this as completed in #27039 Sep 20, 2023
mergify bot pushed a commit that referenced this issue Sep 20, 2023

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
…ot present (#27039)

When using [restrictDefaultSecurityGroup](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_ec2-readme.html#restricting-access-to-the-vpc-default-security-group) to remove default security group rules, an error is thrown and the deploy rolls back if the default rules are not found.
This error usually happens when developers previously removed default rules manually or by other means, and then want to switch to using `restrictDefaultSecurityGroup`. They have to re-add default rules and deploy again to cope with the error.
This PR fixes the custom resource to ignore the error when default rules are not found.

Closes #26390

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@github-actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-ec2 Related to Amazon Elastic Compute Cloud bug This issue is a bug. effort/medium Medium work item – several days of effort p2
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants