-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
aws-step-function-tasks: Allow removing launchType from RunEcsEc2Task #7967
Comments
since |
@mb-dev PR incoming once I merge in #8451 to make As per documentation you've referenced
Although this unlocks some usage, it's still incomplete until users can supply a capacity provider strategy. Since we're limited I'm inclined to agree with your suggested stop-gap. It will still be compatible when edit: added following question: What happens if a cluster does not have a default capacity provider strategy? I believe that's optional too? |
const runTask = new tasks.EcsRunTask(stack, 'Run', {
integrationPattern: sfn.IntegrationPattern.RUN_JOB,
cluster,
taskDefinition,
launchTarget: new tasks.EcsEc2LaunchTarget({
launchType: undefined,
}),
});
|
@mb-dev I was thinking of introducing something like const runTask = new tasks.EcsRunTask(stack, 'Run', {
integrationPattern: sfn.IntegrationPattern.RUN_JOB,
cluster,
taskDefinition,
capacityProviders: [new tasks.EcsEc2CapacityProvider({
...
})],
}); and adding types for what do you think? |
@shivlaks thanks for moving the discussion here. The capacity providers are a property of ECS cluster, while the task level property is choosing one of the providers specified in the cluster definition. Given that users might create ECS cluster or capacity providers outside of CDK, and that there's an option for default capacity provider (as this stop gap suggests) there are actually 3 scenarios for using runTask in step functions:
One idea is to expose |
Capacity Providers were added to Cloudformation: So now it's even more important to be able to omit launchType. Related: #5471 |
The recommended workaround for coverage that isn't quite supported in the We're still discussing options, but are not convinced that the stop-gap needs to be introduced directly into the task or folded into the
|
Ooo, i totally missed the escape hatch in the changelog. I will try that tomorrow and maybe that will be enough until full capacity providers support is added. My pre-refactor EC2 integration is straightforward (in Python):
So I personally won't need |
@mb-dev let me know if I can help with checking out the implementation of the custom state or if it would help for me to whip up an example! |
Hello! Not really sure there is an easy way to avoid the problem (I can wrap my step in a parallel step) but annoying... |
is there any progress here? what does the roadmap look like? |
You can use FARGATE_SPOT with Step Functions, as shown in my tests. You just need to omit the Remove the field {
"Comment": "State machine integrated with ECS",
"StartAt": "Initial",
"States": {
"Initial": {
"Type": "Task",
"Resource": "arn:aws:states:::ecs:runTask.sync",
"Parameters": {
"Cluster": "${ECS_CLUSTER_ARN}",
"PlatformVersion": "LATEST",
"TaskDefinition": "${ECS_TASK_DEFINITION_ARN}",
"NetworkConfiguration": {
"AwsvpcConfiguration": {
"Subnets": ${SUBNETS},
"AssignPublicIp": "ENABLED"
}
},
"Overrides": {
"ContainerOverrides": ${ECS_CONTAINER_OVERRIDES}
}
},
"TimeoutSeconds": 3600,
"End": true
}
}
} |
Just bumped into this as well. My use case is spawning long-running tasks on (expensive) GPU instances that are scaled to zero most of the time. When the StepFunction workflow is at
|
I created a class which implements IEcsLaunchTarget so this is a drop-in workaround: export class EcsFargateSpotLaunchTarget implements sfnTasks.IEcsLaunchTarget {
/**
* Launch the ECS task using Fargate Spot.
*
* This functionality is not built into CDK yet.
*/
constructor(private readonly cluster: ecs.Cluster, private readonly options?: sfnTasks.EcsFargateLaunchTargetOptions) { }
/**
* Called when the Fargate launch type configured on RunTask
*/
public bind(_task: sfnTasks.EcsRunTask, launchTargetOptions: sfnTasks.LaunchTargetBindOptions): sfnTasks.EcsLaunchTargetConfig {
this.cluster.enableFargateCapacityProviders();
if (!launchTargetOptions.taskDefinition.isFargateCompatible) {
throw new Error('Supplied TaskDefinition is not compatible with Fargate');
}
return {
parameters: {
PlatformVersion: this.options?.platformVersion,
"CapacityProviderStrategy": [
{
"CapacityProvider": "FARGATE_SPOT",
"Weight": 1
}
],
},
};
}
} const task = new sfnTasks.EcsRunTask(this, 'RunTask', {
...
launchTarget: new EcsFargateSpotLaunchTarget(ecsCluster)
}); |
I like @danw-mpl solution. I went for a slightly different approach and removed the LaunchType property entirely, allowing the ECS task to take the default capacity provider settings from the cluster. C# example:
|
@Muppets I tried that initially and ran into an error from Step Functions - something like 'no container instances found in your cluster'. I'd assume that's a problem with my cluster config though. Even though I had fargate capacity providers enabled. |
capacity providers allow using ECS with dynamic capacity. https://aws.amazon.com/tw/about-aws/whats-new/2019/12/amazon-ecs-capacity-providers-now-available/ , yet CloudFormation and CDK do not yet support this feature #5471.
I propose as a stop gap solution to allow users to submit EC2 tasks without
launchType
:https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_RunTask.html#ECS-RunTask-request-launchType
if
launchType
is provided, there is no way to get the task to use the capacity providers that are created outside CDK.Use Case
I want to create capacity providers using boto3 and use them when launching EC2 tasks using step functions.
Proposed Solution
Provider a way to make
launchType
optional here:https://github.com/aws/aws-cdk/blob/master/packages/%40aws-cdk/aws-stepfunctions-tasks/lib/ecs/run-ecs-ec2-task.ts#L62
Or to change it after construction.
Other
This is a 🚀 Feature Request
The text was updated successfully, but these errors were encountered: