config-linux: add CFS bandwidth burst #1120

kailun-qin · 2021-08-11T03:21:10Z

Burstable CFS controller is introduced in Linux 5.14. This helps with
parallel workloads that might be bursty. They can get throttled even
when their average utilization is under quota. And they may be latency
sensitive at the same time so that throttling them is undesired.

This feature borrows time now against the future underrun, at the cost
of increased interference against the other system users, by introducing
cfs_burst_us into CFS bandwidth control to enact the cap on unused
bandwidth accumulation, which will then used additionally for burst.

The patch adds the support/control for CFS bandwidth burst.

Fixes #1119

Signed-off-by: Kailun Qin kailun.qin@intel.com

Burstable CFS controller is introduced in Linux 5.14. This helps with parallel workloads that might be bursty. They can get throttled even when their average utilization is under quota. And they may be latency sensitive at the same time so that throttling them is undesired. This feature borrows time now against the future underrun, at the cost of increased interference against the other system users, by introducing cfs_burst_us into CFS bandwidth control to enact the cap on unused bandwidth accumulation, which will then used additionally for burst. The patch adds the support/control for CFS bandwidth burst. runtime-spec: opencontainers/runtime-spec#1120 Signed-off-by: Kailun Qin <kailun.qin@intel.com>

config-linux.md

kailun-qin · 2022-02-28T08:16:25Z

Dear spec maintainers @crosbymichael @cyphar @dqminh @giuseppe @hqhq @mrunalp @tianon @vbatts,

I received this feature request several times from runtime users of different orgnizations, would you please kindly take a look at this one so that we can move forward (also opencontainers/runc#3205)?

Many thanks!

specs-go/config.go

giuseppe · 2022-03-01T10:29:46Z

config-linux.md

@@ -359,7 +359,12 @@ For more information, see the kernel cgroups documentation about [cpusets][cgrou
 The following parameters can be specified to set up the controller:

 * **`shares`** *(uint64, OPTIONAL)* - specifies a relative share of CPU time available to the tasks in a cgroup
-* **`quota`** *(int64, OPTIONAL)* - specifies the total amount of time in microseconds for which all tasks in a cgroup can run during one period (as defined by **`period`** below)
+* **`quota`** *(int64, OPTIONAL)* - specifies the replenished amount of time in microseconds for which all tasks in a cgroup can run during one period (as defined by **`period`** below)
+    A value of `-1` indicates that the cgroup does not have any bandwidth restriction in place (i.e., an unconstrained bandwidth group).


I am not sure about having -1 here. On cgroup v2 quota and period are written together. Does -1 mean it has the same value as period?

For cgroup v2, I suppose -1 means max, which can be written together w/ period.

According to the the cpu.max section of cgroup v2 doc:

cpu.max A read-write two value file which exists on non-root cgroups. The default is "max 100000". The maximum bandwidth limit. It's in the following format: $MAX $PERIOD which indicates that the group may consume upto $MAX in each $PERIOD duration. "max" for $MAX indicates no limit. If only one number is written, $MAX is updated.

I think this applies to all negative integers actually (i.e., not only -1). Any suggestions? Thanks!

I would guess the question here is really why it's important to burst support that we include support for -1 here too -- I'd say this is a breaking change, since the spec doesn't currently say anything about negative numbers, and instead implies that this optional field be left unspecified for no restriction. Does cgroupv1 support negative values?

To be clear, I agree that we should clarify this, especially that unspecified needs to be max in cgroupv2, but I'm not sure how it relates to burst except that it's adjacent.

To be clear, I agree that we should clarify this, especially that unspecified needs to be max in cgroupv2, but I'm not sure how it relates to burst except that it's adjacent.

Thanks, I agree we might clarify it in a separate thread.

Zheaoli · 2022-03-02T07:32:40Z

~~In my opinion, I think cpu_burst mode is a cgroupv2 only feature. Is there any compatibility issue if we change all schema that is used in both cgroupv1 and v2?~~

giuseppe · 2022-03-02T08:03:48Z

In my opinion, I think cpu_burst mode is a cgroupv2 only feature. Is there any compatibility issue if we change all schema that is used in both cgroupv1 and v2?

I see it is present with cgroup v1 too:

# ls /sys/fs/cgroup/cpu/cpu.cfs_burst_us
/sys/fs/cgroup/cpu/cpu.cfs_burst_us

If it was a cgroup v2 only feature then we could just use the unified map.

kailun-qin · 2022-03-02T08:44:31Z

In my opinion, I think cpu_burst mode is a cgroupv2 only feature. Is there any compatibility issue if we change all schema that is used in both cgroupv1 and v2?

I see it is present with cgroup v1 too:

Yes, it supports both cgroup v1 and v2 IMO.

@Zheaoli Any special reason for considering it as cgroup v2 only?

Zheaoli · 2022-03-02T09:47:16Z

Sorry for the bad information, my memory is wrong

I have confirmed the burst mode support both v1 and v2 since Linux 5.14, FYI https://www.kernel.org/doc/html/latest/scheduler/sched-bwc.html

Zheaoli · 2022-03-02T09:52:47Z

But for this, maybe we should think about the kernel issue? The last LTS kernel is 5.10 which is not supported for burst mode(People may use 510 with the newest version of runc . Maybe we should make burst mode as an experimental feature before the Linux 5.15 has been released

config-linux.md

gregkh · 2022-03-02T10:23:46Z

But for this, maybe we should think about the kernel issue? The last LTS kernel is 5.10 which is not supported for burst mode(People may use 510 with the newest version of runc . Maybe we should make burst mode as an experimental feature before the Linux 5.15 has been released

The 5.15 kernel was released last year, and has already been marked as "LTS" for quite some time now. I do not understand why you are not using it now already :)

Zheaoli · 2022-03-02T10:52:40Z

The 5.15 kernel was released last year, and has already been marked as "LTS" for quite some time now. I do not understand why you are not using it now already :)

Many of our machines are using 510 not 515, We can upgrade the runc version easier but it's a tough way to upgrade the kernel version(And my bad again, 515 has been released in 2021-10-31).

Zheaoli · 2022-03-02T13:43:48Z

I think the spec is LGTM to me now.

For my personal idea, I'm very glad to see this spec will be accepted ASAP, because it's very useful for some performance-needed circumstances.

giuseppe

LGTM

Zheaoli · 2022-03-09T03:54:13Z

Hello all, would you mind telling me if this PR is mergeable?

tianon

Overall, I agree we should add this, and I think the suggested name/format are fine (they seem to match what the kernel uses, from what I can tell), but I think we need to make some adjustments to the description -- I'm getting mixed signals about how this feature works from this PR, and I think we can improve that. 😄

config-linux.md

tianon · 2022-03-09T23:21:04Z

config-linux.md

@@ -359,7 +359,12 @@ For more information, see the kernel cgroups documentation about [cpusets][cgrou
 The following parameters can be specified to set up the controller:

 * **`shares`** *(uint64, OPTIONAL)* - specifies a relative share of CPU time available to the tasks in a cgroup
-* **`quota`** *(int64, OPTIONAL)* - specifies the total amount of time in microseconds for which all tasks in a cgroup can run during one period (as defined by **`period`** below)
+* **`quota`** *(int64, OPTIONAL)* - specifies the replenished amount of time in microseconds for which all tasks in a cgroup can run during one period (as defined by **`period`** below)
+    A value of `-1` indicates that the cgroup does not have any bandwidth restriction in place (i.e., an unconstrained bandwidth group).


I would guess the question here is really why it's important to burst support that we include support for -1 here too -- I'd say this is a breaking change, since the spec doesn't currently say anything about negative numbers, and instead implies that this optional field be left unspecified for no restriction. Does cgroupv1 support negative values?

To be clear, I agree that we should clarify this, especially that unspecified needs to be max in cgroupv2, but I'm not sure how it relates to burst except that it's adjacent.

tianon · 2022-03-09T23:25:07Z

config-linux.md

@@ -359,7 +359,12 @@ For more information, see the kernel cgroups documentation about [cpusets][cgrou
 The following parameters can be specified to set up the controller:

 * **`shares`** *(uint64, OPTIONAL)* - specifies a relative share of CPU time available to the tasks in a cgroup
-* **`quota`** *(int64, OPTIONAL)* - specifies the total amount of time in microseconds for which all tasks in a cgroup can run during one period (as defined by **`period`** below)
+* **`quota`** *(int64, OPTIONAL)* - specifies the replenished amount of time in microseconds for which all tasks in a cgroup can run during one period (as defined by **`period`** below)


IMO, "the total amount ... during one period" already implies that this is "replenished" and changing this word makes it less clear 🙈

Thanks for the suggestion. emm... I re-read this again. I agree w/ you.
(FYI, my intention was to highlight that the the quota is actually the "replenished" amount while the actual usage can be "unreplenished" due to the introduction of the burst concept.)

tianon · 2022-03-09T23:29:12Z

config-linux.md

-* **`quota`** *(int64, OPTIONAL)* - specifies the total amount of time in microseconds for which all tasks in a cgroup can run during one period (as defined by **`period`** below)
+* **`quota`** *(int64, OPTIONAL)* - specifies the replenished amount of time in microseconds for which all tasks in a cgroup can run during one period (as defined by **`period`** below)
+    A value of `-1` indicates that the cgroup does not have any bandwidth restriction in place (i.e., an unconstrained bandwidth group).
+    Any (valid) positive value(s) no smaller than `burst` will enact the specified bandwidth limit.


My understanding so far is that burst is either excess time "earned" from past under-usage or borrowed from future periods where we expect/hope we'll be under-utilizing (I'm not actually clear from this PR which it is). Either way, in this specific case, I think the restriction is really on the value of burst, not a restriction on the value of quota (so doesn't seem helpful to mention here).

My understanding so far is that burst is either excess time "earned" from past under-usage or borrowed from future periods where we expect/hope we'll be under-utilizing (I'm not actually clear from this PR which it is). Either way, in this specific case, I think the restriction is really on the value of burst, not a restriction on the value of quota (so doesn't seem helpful to mention here).

In fact, CPU burst is "earned" from past under-usage . Though in this case it's more related to the value of burst, writing a quota smaller than burst will be considered as an EINVAL by the kerenl. It's mentioned in the kernel doc as well. So IMO it's worth mentioning here which may help guide the right configuration.

tianon · 2022-03-09T23:33:30Z

config-linux.md

+* **`quota`** *(int64, OPTIONAL)* - specifies the replenished amount of time in microseconds for which all tasks in a cgroup can run during one period (as defined by **`period`** below)
+    A value of `-1` indicates that the cgroup does not have any bandwidth restriction in place (i.e., an unconstrained bandwidth group).
+    Any (valid) positive value(s) no smaller than `burst` will enact the specified bandwidth limit.
+* **`burst`** *(uint64, OPTIONAL)* - specifies the maximum amount of accumulated time in microseconds for which all tasks in a cgroup can run additionally for burst during one period (as defined by **`period`** below)


This feature borrows time now against the future underrun ...

Given this, I don't think "accumulated" is the right choice of word here -- this feature is about borrowing from the future, not accumulating from the past:

Suggested change

* **`burst`** *(uint64, OPTIONAL)* - specifies the maximum amount of accumulated time in microseconds for which all tasks in a cgroup can run additionally for burst during one period (as defined by **`period`** below)

* **`burst`** *(uint64, OPTIONAL)* - specifies the maximum amount of burst time in microseconds for which all tasks in a cgroup can borrow from future periods during one period (as defined by **`period`** below)

This feature borrows time now against the future underrun ... is directly from the kernel doc and it's actually accumulating from the past rather than borrowing from future. It intends to say that borrowing time now to prepare for the future underestimation.

tianon · 2022-03-09T23:38:33Z

config-linux.md

+    A value of `0` for indicates that the cgroup can not accumulate any unused bandwidth. It makes the traditional bandwidth control behavior unchanged.
+    Any (valid) positive value(s) no larger than `quota` will enact the cap on unused bandwidth accumulation.


I think this would be more consistent if we assume that not specifying this optional field is the way to get "no limit", and I think it would also make sense to be more clear about values bigger than quota are not just ignored but rather an error, perhaps something like this?

Suggested change

A value of `0` for indicates that the cgroup can not accumulate any unused bandwidth. It makes the traditional bandwidth control behavior unchanged.

Any (valid) positive value(s) no larger than `quota` will enact the cap on unused bandwidth accumulation.

If specified, this value MUST be no larger than `quota` (runtimes MAY generate an error).

Edit: coming back to this after opencontainers/runc#3205 (comment), I'm guessing that's related? (And likely why the "negative values" bits were added to quota?)

Yes, indeed it's related.

tianon · 2022-03-09T23:50:46Z

Perhaps it would help me (and other spec reviewers) if we level-set on how the feature works. Here's my current understanding -- is this accurate?

there's at most quota+burst in any one given period, and any use of burst in a period counts against a future period's available quota

Burstable CFS controller is introduced in Linux 5.14. This helps with parallel workloads that might be bursty. They can get throttled even when their average utilization is under quota. And they may be latency sensitive at the same time so that throttling them is undesired. This feature borrows time now against the future underrun, at the cost of increased interference against the other system users, by introducing cfs_burst_us into CFS bandwidth control to enact the cap on unused bandwidth accumulation, which will then used additionally for burst. The patch adds the support/control for CFS bandwidth burst. runtime-spec: opencontainers/runtime-spec#1120 Signed-off-by: Kailun Qin <kailun.qin@intel.com>

Burstable CFS controller is introduced in Linux 5.14. This helps with parallel workloads that might be bursty. They can get throttled even when their average utilization is under quota. And they may be latency sensitive at the same time so that throttling them is undesired. This feature borrows time now against the future underrun, at the cost of increased interference against the other system users, by introducing cfs_burst_us into CFS bandwidth control to enact the cap on unused bandwidth accumulation, which will then used additionally for burst. The patch adds the support/control for CFS bandwidth burst. runtime-spec: opencontainers/runtime-spec#1120 Signed-off-by: Kailun Qin <kailun.qin@intel.com> Signed-off-by: Manjusaka <me@manjusaka.me>

Burstable CFS controller is introduced in Linux 5.14. This helps with parallel workloads that might be bursty. They can get throttled even when their average utilization is under quota. And they may be latency sensitive at the same time so that throttling them is undesired. This feature borrows time now against the future underrun, at the cost of increased interference against the other system users, by introducing cfs_burst_us into CFS bandwidth control to enact the cap on unused bandwidth accumulation, which will then used additionally for burst. The patch adds the support/control for CFS bandwidth burst. runtime-spec: opencontainers/runtime-spec#1120 Co-authored-by: Akihiro Suda <suda.kyoto@gmail.com> Co-authored-by: Nadeshiko Manju <lizheao940510@gmail.com> Signed-off-by: Kailun Qin <kailun.qin@intel.com> Update libcontainer/configs/cgroup_linux.go

Burstable CFS controller is introduced in Linux 5.14. This helps with parallel workloads that might be bursty. They can get throttled even when their average utilization is under quota. And they may be latency sensitive at the same time so that throttling them is undesired. This feature borrows time now against the future underrun, at the cost of increased interference against the other system users, by introducing cfs_burst_us into CFS bandwidth control to enact the cap on unused bandwidth accumulation, which will then used additionally for burst. The patch adds the support/control for CFS bandwidth burst. runtime-spec: opencontainers/runtime-spec#1120 Co-authored-by: Akihiro Suda <suda.kyoto@gmail.com> Co-authored-by: Nadeshiko Manju <lizheao940510@gmail.com> Signed-off-by: Kailun Qin <kailun.qin@intel.com>

Burstable CFS controller is introduced in Linux 5.14. This helps with parallel workloads that might be bursty. They can get throttled even when their average utilization is under quota. And they may be latency sensitive at the same time so that throttling them is undesired. This feature borrows time now against the future underrun, at the cost of increased interference against the other system users, by introducing cfs_burst_us into CFS bandwidth control to enact the cap on unused bandwidth accumulation, which will then used additionally for burst. The patch adds the support/control for CFS bandwidth burst. runtime-spec: opencontainers/runtime-spec#1120 Co-authored-by: Akihiro Suda <suda.kyoto@gmail.com> Co-authored-by: Nadeshiko Manju <me@manjusaka.me> Signed-off-by: Kailun Qin <kailun.qin@intel.com>

kailun-qin mentioned this pull request Aug 11, 2021

Support CFS bandwidth burst opencontainers/runc#3145

Closed

rata mentioned this pull request Aug 31, 2021

Travis CI not working? #1121

Closed

kailun-qin mentioned this pull request Sep 9, 2021

libct/cg: add CFS bandwidth burst for CPU opencontainers/runc#3205

Closed

m-yosefpor reviewed Sep 10, 2021

View reviewed changes

config-linux.md Outdated Show resolved Hide resolved

giuseppe reviewed Feb 28, 2022

View reviewed changes

specs-go/config.go Show resolved Hide resolved

kailun-qin force-pushed the add-cfs-burst branch from 8ed2497 to f56b4a3 Compare March 1, 2022 01:34

kailun-qin requested review from crosbymichael, cyphar, dqminh, hqhq, mrunalp, tianon and vbatts as code owners March 1, 2022 01:34

kailun-qin force-pushed the add-cfs-burst branch from f56b4a3 to a4503eb Compare March 1, 2022 01:35

giuseppe reviewed Mar 1, 2022

View reviewed changes

Zheaoli reviewed Mar 2, 2022

View reviewed changes

config-linux.md Outdated Show resolved Hide resolved

config-linux.md Show resolved Hide resolved

giuseppe approved these changes Mar 3, 2022

View reviewed changes

tianon reviewed Mar 9, 2022

View reviewed changes

AkihiroSuda mentioned this pull request Jun 26, 2023

Release v1.1.0 #1213

Merged

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

config-linux: add CFS bandwidth burst #1120

config-linux: add CFS bandwidth burst #1120

kailun-qin commented Aug 11, 2021

kailun-qin commented Feb 28, 2022 •

edited

giuseppe Mar 1, 2022

kailun-qin Mar 1, 2022

kailun-qin Mar 9, 2022

tianon Mar 9, 2022

kailun-qin Jun 20, 2022

Zheaoli commented Mar 2, 2022 •

edited

giuseppe commented Mar 2, 2022

kailun-qin commented Mar 2, 2022

Zheaoli commented Mar 2, 2022

Zheaoli commented Mar 2, 2022 •

edited

gregkh commented Mar 2, 2022

Zheaoli commented Mar 2, 2022 •

edited

Zheaoli commented Mar 2, 2022

giuseppe left a comment

Zheaoli commented Mar 9, 2022

tianon left a comment

tianon Mar 9, 2022

tianon Mar 9, 2022

kailun-qin Jun 20, 2022

tianon Mar 9, 2022

kailun-qin Jun 20, 2022

tianon Mar 9, 2022

kailun-qin Jun 20, 2022

tianon Mar 9, 2022

kailun-qin Jun 20, 2022

tianon commented Mar 9, 2022

	* `burst` (uint64, OPTIONAL) - specifies the maximum amount of accumulated time in microseconds for which all tasks in a cgroup can run additionally for burst during one period (as defined by `period` below)
	* `burst` (uint64, OPTIONAL) - specifies the maximum amount of burst time in microseconds for which all tasks in a cgroup can borrow from future periods during one period (as defined by `period` below)

		A value of `0` for indicates that the cgroup can not accumulate any unused bandwidth. It makes the traditional bandwidth control behavior unchanged.
		Any (valid) positive value(s) no larger than `quota` will enact the cap on unused bandwidth accumulation.

	A value of `0` for indicates that the cgroup can not accumulate any unused bandwidth. It makes the traditional bandwidth control behavior unchanged.
	Any (valid) positive value(s) no larger than `quota` will enact the cap on unused bandwidth accumulation.
	If specified, this value MUST be no larger than `quota` (runtimes MAY generate an error).

config-linux: add CFS bandwidth burst #1120

config-linux: add CFS bandwidth burst #1120

Conversation

kailun-qin commented Aug 11, 2021

kailun-qin commented Feb 28, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Zheaoli commented Mar 2, 2022 • edited

giuseppe commented Mar 2, 2022

kailun-qin commented Mar 2, 2022

Zheaoli commented Mar 2, 2022

Zheaoli commented Mar 2, 2022 • edited

gregkh commented Mar 2, 2022

Zheaoli commented Mar 2, 2022 • edited

Zheaoli commented Mar 2, 2022

giuseppe left a comment

Choose a reason for hiding this comment

Zheaoli commented Mar 9, 2022

tianon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tianon commented Mar 9, 2022

kailun-qin commented Feb 28, 2022 •

edited

Zheaoli commented Mar 2, 2022 •

edited

Zheaoli commented Mar 2, 2022 •

edited

Zheaoli commented Mar 2, 2022 •

edited