aws-ecs-patterns: taskDefinition prop ignored on QueueProcessingFargateService #27360

ryanwalters · 2023-09-29T19:39:58Z

Describe the bug

I am trying to set up a Fargate task that processes items from an SQS queue and stumbled onto the QueueProcessingFargateService construct which appears to do just that. However, after configuring the construct, I am getting the following error when running cdk deploy:

"Invalid request provided: Create TaskDefinition: Container list cannot be empty. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: 00000000-0000-0000-0000-000000000000; Proxy: null)"

Expected Behavior

The CDK stack deploys successfully.

Current Behavior

The CDK stack fails to deploy.

> cdk@0.0.0 cdk
> cdk deploy --require-approval never

✨  Synthesis time: 4.1s

ProjectStack:  start: Building db6c89048d09ecd82cfbaeaa88cd73a91eb853fb29d321a61d90014df27f3740:************-*********
ProjectStack:  success: Built db6c89048d09ecd82cfbaeaa88cd73a91eb853fb29d321a61d90014df27f3740:************-*********
ProjectStack:  start: Publishing db6c89048d09ecd82cfbaeaa88cd73a91eb853fb29d321a61d90014df27f3740:************-*********
ProjectStack:  success: Published db6c89048d09ecd82cfbaeaa88cd73a91eb853fb29d321a61d90014df27f3740:************-*********
ProjectStack: deploying... [1/1]
ProjectStack: creating CloudFormation changeset...
ProjectStack |  0/18 | 2:54:13 PM | UPDATE_IN_PROGRESS   | AWS::CloudFormation::Stack                  | ProjectStack User Initiated
ProjectStack |  0/18 | 2:54:19 PM | UPDATE_IN_PROGRESS   | AWS::IAM::Policy                            | QueueProcessingService/QueueProcessingTaskDef/TaskRole/DefaultPolicy (QueueProcessingServiceQueueProcessingTaskDefTaskRoleDefaultPolicy0AD99632)
ProjectStack |  0/18 | 2:54:19 PM | UPDATE_IN_PROGRESS   | AWS::IAM::Policy                            | QueueProcessingService/QueueProcessingTaskDef/ExecutionRole/DefaultPolicy (QueueProcessingServiceQueueProcessingTaskDefExecutionRoleDefaultPolicyC4BA62EF)
ProjectStack |  0/18 | 2:54:19 PM | UPDATE_IN_PROGRESS   | AWS::ECS::TaskDefinition                    | TaskDefinition (TaskDefinitionB36D86D9) Requested update requires the creation of a new physical resource; hence creating one.
ProjectStack |  0/18 | 2:54:19 PM | UPDATE_IN_PROGRESS   | AWS::ECS::TaskDefinition                    | QueueProcessingService/QueueProcessingTaskDef (QueueProcessingServiceQueueProcessingTaskDefAEA09787) Requested update requires the creation of a new physical resource; hence creating one.
ProjectStack |  0/18 | 2:54:20 PM | UPDATE_FAILED        | AWS::ECS::TaskDefinition                    | TaskDefinition (TaskDefinitionB36D86D9) Resource handler returned message: "Invalid request provided: Create TaskDefinition: Container list cannot be empty. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: 00000000-0000-0000-0000-000000000000; Proxy: null)" (RequestToken: 00000000-0000-0000-0000-000000000000, HandlerErrorCode: InvalidRequest)
ProjectStack |  0/18 | 2:54:21 PM | UPDATE_FAILED        | AWS::IAM::Policy                            | QueueProcessingService/QueueProcessingTaskDef/TaskRole/DefaultPolicy (QueueProcessingServiceQueueProcessingTaskDefTaskRoleDefaultPolicy0AD99632) Resource update cancelled
ProjectStack |  0/18 | 2:54:21 PM | UPDATE_FAILED        | AWS::IAM::Policy                            | QueueProcessingService/QueueProcessingTaskDef/ExecutionRole/DefaultPolicy (QueueProcessingServiceQueueProcessingTaskDefExecutionRoleDefaultPolicyC4BA62EF) Resource update cancelled
ProjectStack |  0/18 | 2:54:21 PM | UPDATE_FAILED        | AWS::ECS::TaskDefinition                    | QueueProcessingService/QueueProcessingTaskDef (QueueProcessingServiceQueueProcessingTaskDefAEA09787) Resource creation cancelled
ProjectStack |  0/18 | 2:54:21 PM | UPDATE_ROLLBACK_IN_P | AWS::CloudFormation::Stack                  | ProjectStack The following resource(s) failed to update: [QueueProcessingServiceQueueProcessingTaskDefExecutionRoleDefaultPolicyC4BA62EF, QueueProcessingServiceQueueProcessingTaskDefTaskRoleDefaultPolicy0AD99632, QueueProcessingServiceQueueProcessingTaskDefAEA09787, TaskDefinitionB36D86D9].
ProjectStack |  1/18 | 2:54:25 PM | UPDATE_COMPLETE      | AWS::ECS::TaskDefinition                    | TaskDefinition (TaskDefinitionB36D86D9)
ProjectStack |  1/18 | 2:54:25 PM | UPDATE_IN_PROGRESS   | AWS::IAM::Policy                            | QueueProcessingService/QueueProcessingTaskDef/ExecutionRole/DefaultPolicy (QueueProcessingServiceQueueProcessingTaskDefExecutionRoleDefaultPolicyC4BA62EF)
ProjectStack |  2/18 | 2:54:25 PM | UPDATE_COMPLETE      | AWS::ECS::TaskDefinition                    | QueueProcessingService/QueueProcessingTaskDef (QueueProcessingServiceQueueProcessingTaskDefAEA09787)
ProjectStack |  2/18 | 2:54:25 PM | UPDATE_IN_PROGRESS   | AWS::IAM::Policy                            | QueueProcessingService/QueueProcessingTaskDef/TaskRole/DefaultPolicy (QueueProcessingServiceQueueProcessingTaskDefTaskRoleDefaultPolicy0AD99632)
ProjectStack |  3/18 | 2:54:41 PM | UPDATE_COMPLETE      | AWS::IAM::Policy                            | QueueProcessingService/QueueProcessingTaskDef/ExecutionRole/DefaultPolicy (QueueProcessingServiceQueueProcessingTaskDefExecutionRoleDefaultPolicyC4BA62EF)
ProjectStack |  4/18 | 2:54:41 PM | UPDATE_COMPLETE      | AWS::IAM::Policy                            | QueueProcessingService/QueueProcessingTaskDef/TaskRole/DefaultPolicy (QueueProcessingServiceQueueProcessingTaskDefTaskRoleDefaultPolicy0AD99632)
ProjectStack |  5/18 | 2:54:44 PM | UPDATE_ROLLBACK_COMP | AWS::CloudFormation::Stack                  | ProjectStack
ProjectStack |  5/18 | 2:54:45 PM | DELETE_IN_PROGRESS   | AWS::ECS::TaskDefinition                    | QueueProcessingService/QueueProcessingTaskDef (QueueProcessingServiceQueueProcessingTaskDefAEA09787)
ProjectStack |  4/18 | 2:54:45 PM | DELETE_COMPLETE      | AWS::ECS::TaskDefinition                    | TaskDefinition (TaskDefinitionB36D86D9)
ProjectStack |  3/18 | 2:54:46 PM | DELETE_COMPLETE      | AWS::ECS::TaskDefinition                    | QueueProcessingService/QueueProcessingTaskDef (QueueProcessingServiceQueueProcessingTaskDefAEA09787)
ProjectStack |  4/18 | 2:54:46 PM | UPDATE_ROLLBACK_COMP | AWS::CloudFormation::Stack                  | ProjectStack

Failed resources:
ProjectStack | 2:54:20 PM | UPDATE_FAILED        | AWS::ECS::TaskDefinition                    | TaskDefinition (TaskDefinitionB36D86D9) Resource handler returned message: "Invalid request provided: Create TaskDefinition: Container list cannot be empty. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: 00000000-0000-0000-0000-000000000000; Proxy: null)" (RequestToken: 00000000-0000-0000-0000-000000000000, HandlerErrorCode: InvalidRequest)

❌  ProjectStack failed: Error: The stack named ProjectStack failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "Invalid request provided: Create TaskDefinition: Container list cannot be empty. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: 00000000-0000-0000-0000-000000000000; Proxy: null)" (RequestToken: 00000000-0000-0000-0000-000000000000, HandlerErrorCode: InvalidRequest)
at FullCloudFormationDeployment.monitorDeployment (/home/circleci/project/cdk/node_modules/aws-cdk/lib/index.js:467:10232)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async Object.deployStack2 [as deployStack] (/home/circleci/project/cdk/node_modules/aws-cdk/lib/index.js:470:179911)
at async /home/circleci/project/cdk/node_modules/aws-cdk/lib/index.js:470:163159

❌ Deployment failed: Error: The stack named ProjectStack failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "Invalid request provided: Create TaskDefinition: Container list cannot be empty. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: 00000000-0000-0000-0000-000000000000; Proxy: null)" (RequestToken: 00000000-0000-0000-0000-000000000000, HandlerErrorCode: InvalidRequest)
at FullCloudFormationDeployment.monitorDeployment (/home/circleci/project/cdk/node_modules/aws-cdk/lib/index.js:467:10232)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async Object.deployStack2 [as deployStack] (/home/circleci/project/cdk/node_modules/aws-cdk/lib/index.js:470:179911)
at async /home/circleci/project/cdk/node_modules/aws-cdk/lib/index.js:470:163159

The stack named ProjectStack failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "Invalid request provided: Create TaskDefinition: Container list cannot be empty. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: 00000000-0000-0000-0000-000000000000; Proxy: null)" (RequestToken: 00000000-0000-0000-0000-000000000000, HandlerErrorCode: InvalidRequest)

Exited with code exit status 1

Reproduction Steps

import { App, Duration, Stack, StackProps } from 'aws-cdk-lib';
import { DockerImageAsset } from 'aws-cdk-lib/aws-ecr-assets';
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import * as ecs from 'aws-cdk-lib/aws-ecs';
import { QueueProcessingFargateService } from 'aws-cdk-lib/aws-ecs-patterns';
import { LogGroup } from 'aws-cdk-lib/aws-logs';
import * as path from 'node:path';
import { config } from '../config';

export class FargateServiceStack extends Stack {
  constructor(scope: App, id: string, props?: StackProps) {
    super(scope, id, props);

    const asset = new DockerImageAsset(this, 'TestDockerImage', {
      directory: path.join(__dirname, '..', '..', 'images', 'test'),
    });

    const vpc = ec2.Vpc.fromLookup(this, 'Vpc', {
      vpcName: config.vpcName,
    });

    const cluster = new ecs.Cluster(this, 'EcsCluster', {
      vpc,
      clusterName: config.clusterName,
    });

    const logGroup = new LogGroup(this, 'LogGroup', {
      logGroupName: config.logGroupName,
    });

    const logDriver = new ecs.AwsLogDriver({
      logGroup,
      streamPrefix: config.streamPrefix,
    });

    const taskDefinition = new ecs.FargateTaskDefinition(this, 'TaskDefinition', {
      cpu: 256,
      memoryLimitMiB: 512,
      runtimePlatform: {
        cpuArchitecture: ecs.CpuArchitecture.ARM64,
      },
    });

    new QueueProcessingFargateService(this, 'QueueProcessingAiService', {
      cluster,
      logDriver,
      taskDefinition,
      image: ecs.ContainerImage.fromDockerImageAsset(asset),
      enableLogging: true,
      maxScalingCapacity: 5,
      minScalingCapacity: 0,
      platformVersion: ecs.FargatePlatformVersion.VERSION1_4,
      serviceName: config.serviceName,
      visibilityTimeout: Duration.minutes(1),
    });
  }
}

Possible Solution

Maybe I just misconfigured the construct? 🤷

Is there some kind of underlying change needed to support ecs.ContainerImage.fromDockerImageAsset(asset) vs ecs.ContainerImage.fromRegistry('test')?

Additional Information/Context

No response

CDK CLI Version

2.99.0 (build 0aa1096)

Framework Version

No response

Node.js Version

18.18.0

OS

ubuntu-2204:2023.02.1

Language

Typescript

Language Version

5.2.2

Other information

No response

The text was updated successfully, but these errors were encountered:

peterwoodworth · 2023-09-29T20:28:58Z

Could you share the section of your CloudFormation template that shows the TaskDefinition that's failing to deploy?

ryanwalters · 2023-09-29T21:08:39Z

Thanks for the quick follow-up. There appear to be two AWS::ECS::TaskDefinition resources.

I believe this is the one that is failing to deploy:

  TaskDefinitionB36D86D9:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Cpu: "256"
      Family: ServiceFargateStackTaskDefinition86E89CC0
      Memory: "512"
      NetworkMode: awsvpc
      RequiresCompatibilities:
        - FARGATE
      RuntimePlatform:
        CpuArchitecture: ARM64
      TaskRoleArn:
        Fn::GetAtt:
          - TaskDefinitionTaskRoleFD40A61D
          - Arn
    Metadata:
      aws:cdk:path: ServiceFargateStack-dev/TaskDefinition/Resource

The other one appears to have the missing ContainerDefinitions property:

  QueueProcessingServiceQueueProcessingTaskDefAEA09787:
    Type: AWS::ECS::TaskDefinition
    Properties:
      ContainerDefinitions:
        - Environment:
            - Name: QUEUE_NAME
              Value:
                Fn::GetAtt:
                  - QueueProcessingServiceEcsProcessingQueue72F6FF5A
                  - QueueName
          Essential: true
          Image:
            Fn::Sub: ************.dkr.ecr.*********.${AWS::URLSuffix}/cdk-hnb659fds-container-assets-************-*********:bda004f18ece5320181a8e43a710afe55db93c142a7bd3804eb41e8d420edb9a
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-group:
                Ref: LogGroupF5B46931
              awslogs-stream-prefix: service-fargate
              awslogs-region: *********
          Name: QueueProcessingContainer
      Cpu: "256"
      ExecutionRoleArn:
        Fn::GetAtt:
          - QueueProcessingServiceQueueProcessingTaskDefExecutionRoleFF08A907
          - Arn
      Family: ServiceFargateStackdevQueueProcessingServiceQueueProcessingTaskDefB41D4DE2
      Memory: "512"
      NetworkMode: awsvpc
      RequiresCompatibilities:
        - FARGATE
      TaskRoleArn:
        Fn::GetAtt:
          - QueueProcessingServiceQueueProcessingTaskDefTaskRoleACE5F3E7
          - Arn
    Metadata:
      aws:cdk:path: ServiceFargateStack-dev/QueueProcessingService/QueueProcessingTaskDef/Resource

I guess I wouldn't have expected there to be two AWS::ECS::TaskDefinition resources defined?

peterwoodworth · 2023-09-29T21:13:23Z

Well, that explains it. The taskDefinition prop doesn't appear to do anything for QueueProcessingFargateService. It just ignores it regardless if one was provided

aws-cdk/packages/aws-cdk-lib/aws-ecs-patterns/lib/fargate/queue-processing-fargate-service.ts

Line 72 in 0aa1096

    
           this.taskDefinition = new FargateTaskDefinition(this, 'QueueProcessingTaskDef', {

If you specify the cpu, memory, runtime platform on the QueueProcessingFargateService it should result what you want if you just remove the TaskDefinition you're explicitly defining

alhridoy · 2023-10-01T08:27:17Z

Hi @peterwoodworth , I'm interested in working on this issue. Can I be assigned to it? If there are any specific guidelines or pointers related to this, please let me know. Thanks!

peterwoodworth · 2023-10-04T22:11:35Z

@alhridoy just be sure to follow the contributing guide and you'll be good to go, thanks!

…rvice (#28220) Fixed a bug in queueProcessingFargateService where the taskDefinition provided as an argument was not being used, and a default taskDefinition was always being generated instead. Closes #27360 ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*

github-actions · 2023-12-19T22:21:36Z

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

ryanwalters added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Sep 29, 2023

github-actions bot added the @aws-cdk/aws-ecs-patterns Related to ecs-patterns library label Sep 29, 2023

peterwoodworth added p2 and removed needs-triage This issue or PR still needs to be triaged. labels Sep 29, 2023

peterwoodworth added the effort/small Small work item – less than a day of effort label Sep 29, 2023

peterwoodworth changed the title ~~aws-ecs-patterns: "Container list cannot be empty." error when using QueueProcessingFargateService construct~~ aws-ecs-patterns: taskDefinition prop ignored on QueueProcessingFargateService Sep 29, 2023

peterwoodworth added the good first issue Related to contributions. See CONTRIBUTING.md label Sep 29, 2023

github-actions bot mentioned this issue Oct 1, 2023

Monthly issue metrics report - Sep 2023 #27367

Closed

alhridoy mentioned this issue Oct 5, 2023

fix(ecs-patterns): respect taskDefinition prop in QueueProcessingFargateService. #27410

Closed

badmintoncryer mentioned this issue Dec 1, 2023

fix(ecs-patterns): taskDefinition ignored by queueProcessingFargateService #28220

Merged

mergify bot closed this as completed in #28220 Dec 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

aws-ecs-patterns: taskDefinition prop ignored on QueueProcessingFargateService #27360

aws-ecs-patterns: taskDefinition prop ignored on QueueProcessingFargateService #27360

ryanwalters commented Sep 29, 2023 •

edited

peterwoodworth commented Sep 29, 2023

ryanwalters commented Sep 29, 2023

peterwoodworth commented Sep 29, 2023 •

edited

alhridoy commented Oct 1, 2023

peterwoodworth commented Oct 4, 2023

github-actions bot commented Dec 19, 2023

aws-ecs-patterns: taskDefinition prop ignored on QueueProcessingFargateService #27360

aws-ecs-patterns: taskDefinition prop ignored on QueueProcessingFargateService #27360

Comments

ryanwalters commented Sep 29, 2023 • edited

Describe the bug

Expected Behavior

Current Behavior

Reproduction Steps

Possible Solution

Additional Information/Context

CDK CLI Version

Framework Version

Node.js Version

OS

Language

Language Version

Other information

peterwoodworth commented Sep 29, 2023

ryanwalters commented Sep 29, 2023

peterwoodworth commented Sep 29, 2023 • edited

alhridoy commented Oct 1, 2023

peterwoodworth commented Oct 4, 2023

github-actions bot commented Dec 19, 2023

⚠️COMMENT VISIBILITY WARNING⚠️

ryanwalters commented Sep 29, 2023 •

edited

peterwoodworth commented Sep 29, 2023 •

edited