Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: improve task startup logic for Fargate #2428

Merged
merged 3 commits into from
Jan 18, 2024
Merged

fix: improve task startup logic for Fargate #2428

merged 3 commits into from
Jan 18, 2024

Conversation

hassy
Copy link
Member

@hassy hassy commented Jan 18, 2024

Two changes here:

  1. Explicitly exclude node_modules_stream.zip from being synced into workers, and wait for node_modules.zip to be available as intended instead.
  2. Add a short wait in between launching a lead worker and launching more workers. This gives the lead task some time to finish pre-packing dependencies, but still allows other workers to provision in parallel should pre-packing take a while (which would be the case for tests with large dependency trees).

Fixes #2427

@hassy
fix(fargate): prevent race condition between leader and follower tasks
257dba8
@hassy
feat(fargate): emit prepack_end message from leader task
614115f
@hassy
feat(fargate): wait for prepack_end from leader task before continuing
f3ce7e8 
This allows the leader task to get ready before launching more tasks.

The wait period is 60s so as not to increase the overall startup time
for a test run with many workers.
@hassy hassy merged commit a0b6328 into main Jan 18, 2024
18 checks passed
@hassy hassy deleted the fix/ecs-init branch January 18, 2024 14:30
hassy added a commit that referenced this pull request Feb 1, 2024
@hassy
fix(fargate): start workers in parallel with lead task
57f973a 
This reverts the behavior introduced in [1] where worker tasks
may wait for up to 60s whilst the lead task is preparing npm
dependencies.

The only benefit of waiting is to save up to 60s worth of
Fargate compute costs for tests where the lead task fails
to startup. This is rare.

Every test pays a penalty of having to wait for some time
for the lead task to be ready before the rest of the workers
even begin initializing however.

1. #2428
@hassy hassy mentioned this pull request Feb 1, 2024
Merged
hassy added a commit that referenced this pull request Feb 2, 2024
@hassy
fix(fargate): start workers in parallel with lead task (#2459)
d68482d 
This reverts the behavior introduced in [1] where worker tasks
may wait for up to 60s whilst the lead task is preparing npm
dependencies.

The only benefit of waiting is to save up to 60s worth of
Fargate compute costs for tests where the lead task fails
to startup. This is rare.

Every test pays a penalty of having to wait for some time
for the lead task to be ready before the rest of the workers
even begin initializing however.

1. #2428
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
run-aws-tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Fargate timed out error
1 participant
@hassy