Scaling strategy to limit number of Machine pending or provisioning

[lentzi90]

What would you like to be added (User Story)?

An an operator, I would like to control how fast new Machines are created when I create large clusters to avoid overwhelming controllers and infrastructure.

Detailed Description

I want a way to limit the number of Machines that are pending or provisioning. Currently, when creating large clusters we start out small and scale gradually to avoid issues. However, this could be easily automated and solved for all providers if built in to CAPI.

In the Bare Metal Operator we have a PROVISIONING_LIMIT for exactly this reason. It limits the number of BareMetalHosts that are provisioned simultaneously. Having something similar in CAPI would be very useful.

I'm not sure where it would make sense to add this option though. It could be set on the Cluster, the KCP and/or MachineDeployment for example. to get granular control. Or it could be a flag for the controllers. What do you think would work best?

Anything else you would like to add?

No response

Label(s) to be applied

/kind feature
/area machine

[killianmuldoon]

/triage accepted

This is an interesting idea - similar to what was implemented for upgrades in #8432. Maybe these could be generalized into an overall rollout strategy.

[fabriziopandini]

As per 14th Jun office hour discussion, we should discuss UX/API in a doc

[lentzi90]

Older related issue that was closed due to inactivity: #4022
Linking for reference

[lentzi90]

I have started on a document here: https://docs.google.com/document/d/1FjX5rQGYHCyDqRdANWcAWP4AmoxudoPl8IARjXylGzY/edit?usp=sharing
Please check it and comment/edit 🙂

[fabriziopandini]

cc @vincepri @enxebre

[lentzi90]

We ended up setting limits in the "cloud" provider instead. Granted this does not solve the issue for controllers, but they can be pretty well handled by existing config options (e.g. concurrency, resources and rate limits).
Closing (but feel free to reopen if there is interest to continue this in the community)