-
Notifications
You must be signed in to change notification settings - Fork 175
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
High CPU usage #1954
Comments
Yes, resources are set. CPU usage is constantly close to whatever limit I set, up to 4 CPUs. Memory does not seem a problem. I don't mind if the app needs to use CPU (usually we don't set CPU limits at all) but from the logs, to me it is not clear what the Operator is doing that it needs so much. |
I do observe all |
No, none of those. It applies to literrally all ClusterRBACAssessmentReports. As mentioned above, also the logs (in DEBUG mode) are flooded with messages related to this:
|
@Pionerd the check for report is a part of processing, do you see report get generated e every few minutes ? |
Yes |
@Pionerd few of question:
|
Yes, they are exactly the same. The reports are deleted and then it takes up to a minute for them to reappear. |
|
If I scale down the deployment of the operator to 0 replicas, the This happens to all ClusterRoles, including all system ones, so I can't imagine how they could be temporarily deleted without causing all other kinds of symptoms in the system. For your info: this is the same cluster as #1970 and even though InfraAssessments are enabled, they are not generated. So a few weird things seem to be going on. |
@Pionerd Thanks for this update. I'll have a look regarding replicaset >0. |
@chen-keinan I think I found the culprit. Based on #1742 I wrote a pretty extensive exception set. Is it possible that the operator does not handle this in a very cpu friendly way? Any possibilities to improve this (or the exception definitions in general)? I'd rather not share the whole exception definition here (it shows our full stack), but I can share it with you if it helps. It contains 26 |
sure, but you'll have to put here some real example so I could try to reproduce it |
Please see your gmail. |
ok , got it now, I will take a look at it later |
Unfortunately the issue reappeared in an environment without those exclusions. Would be really nice if we could make some progress on this issue, please let me know if and how I may assist. |
@Pionerd can you try disable scanner by scanner to isolate the problem and see which one could cause performance issue ? |
Based on the above I did an additional test with enabling the configAudit, but disabling the exceptions sent to you earlier. Although the CPU usage is also high for the first few minutes, after that the usage seems to return to normal. Maybe you can look if you can reproduce the issue using those exceptions. |
@Pionerd thanks for input, I'll have a look at it. |
What steps did you take and what happened:
Trivy Operator is using near 100% of CPU all the time. In debug mode the logs are flooded with messages like this (not sure if the operator is supposed to check this often) but for the rest I do not see anything significant happening
The average workload is recurring every ~3/4 seconds
What did you expect to happen:
No high CPU usage in an idle situation
Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Environment:
trivy-operator version
): 0.19.1kubectl version
): 1.28 AKSThe text was updated successfully, but these errors were encountered: