Some reports are missing

[KevinDW-Fluxys]

Hi,

I am running trivy-operator 0.19.1 deployed with Helm.

I have managed to get it working and I'm getting some reports, but there are some reports which are not existing, and i cant figure out why.
For the following CRDs i DO have reports:

• ClusterComplianceReport
• ClusterSbomReport
• ClusterVulnerabilityReport
• ExposedSecretReport
• SbomReport
• VulnerabilityReport

For the following CRDs i DO NOT have reports:

• ClusterConfigAuditReport
• ClusterInfraAssessmentReport
• ClusterRbacAssessmentReport
• ConfigAuditReport
• InfraAssessmentReport
• RbacAssessmentReport

In the logs i find a lot of these messages: ("evaluating resource: failed to run policy checks on resources")

{"level":"error","ts":"2024-03-25T09:22:59Z","msg":"Reconciler error","controller":"replicaset","controllerGroup":"apps","controllerKind":"ReplicaSet","ReplicaSet":{"name":"kube-state-metrics-69466f9b5","namespace":"kube-state"},"namespace":"kube-state","name":"kube-state-metrics-69466f9b5","reconcileID":"783e928b-4dc2-4e4e-9fdf-92b33e5a2b64","error":"evaluating resource: failed to run policy checks on resources","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.2/pkg/internal/controller/controller.go:227"}

I am running the default values of the helm chart. I have a feeling it could have something to do with the node-collector. I find only 1 node-collector job, and in the logs i find this: (which seems like an empty result to me)

{
	"apiVersion": "v1",
	"kind": "NodeInfo",
	"metadata": {
		"creationTimestamp": "2024-03-21T10:41:33Z"
	},
	"type": "worker",
	"info": {
		"certificateAuthoritiesFileOwnership": {
			"values": []
		},
		"certificateAuthoritiesFilePermissions": {
			"values": []
		},
		"kubeconfigFileExistsOwnership": {
			"values": []
		},
		"kubeconfigFileExistsPermissions": {
			"values": []
		},
		"kubeletAnonymousAuthArgumentSet": {
			"values": [
				"false"
			]
		},
		"kubeletAuthorizationModeArgumentSet": {
			"values": [
				"Webhook"
			]
		},
		"kubeletClientCaFileArgumentSet": {
			"values": [
				"/etc/kubernetes/certs/ca.crt"
			]
		},
		"kubeletConfFileOwnership": {
			"values": []
		},
		"kubeletConfFilePermissions": {
			"values": []
		},
		"kubeletConfigYamlConfigurationFileOwnership": {
			"values": []
		},
		"kubeletConfigYamlConfigurationFilePermission": {
			"values": []
		},
		"kubeletEventQpsArgumentSet": {
			"values": [
				0
			]
		},
		"kubeletHostnameOverrideArgumentSet": {
			"values": []
		},
		"kubeletMakeIptablesUtilChainsArgumentSet": {
			"values": [
				"true"
			]
		},
		"kubeletOnlyUseStrongCryptographic": {
			"values": []
		},
		"kubeletProtectKernelDefaultsArgumentSet": {
			"values": []
		},
		"kubeletReadOnlyPortArgumentSet": {
			"values": []
		},
		"kubeletRotateCertificatesArgumentSet": {
			"values": []
		},
		"kubeletRotateKubeletServerCertificateArgumentSet": {
			"values": []
		},
		"kubeletServiceFileOwnership": {
			"values": []
		},
		"kubeletServiceFilePermissions": {
			"values": []
		},
		"kubeletStreamingConnectionIdleTimeoutArgumentSet": {
			"values": [
				"4h0m0s"
			]
		},
		"kubeletTlsCertFileTlsArgumentSet": {
			"values": []
		},
		"kubeletTlsPrivateKeyFileArgumentSet": {
			"values": []
		}
	}
}

Am I missing something?

[chen-keinan]

@KevinDW-Fluxys can you please share you trivy-operator configmaps

[KevinDW-Fluxys]

@chen-keinan Certainly.
trivy-operator

apiVersion: v1
kind: ConfigMap
metadata:
  name: trivy-operator
  namespace: trivy-operator
data:
  compliance.failEntriesLimit: '10'
  configAuditReports.scanner: Trivy
  node.collector.imageRef: ****.azurecr.io/aquasecurity/node-collector:0.1.2
  node.collector.nodeSelector: 'true'
  nodeCollector.volumeMounts: >-
    [{"mountPath":"/var/lib/etcd","name":"var-lib-etcd","readOnly":true},{"mountPath":"/var/lib/kubelet","name":"var-lib-kubelet","readOnly":true},{"mountPath":"/var/lib/kube-scheduler","name":"var-lib-kube-scheduler","readOnly":true},{"mountPath":"/var/lib/kube-controller-manager","name":"var-lib-kube-controller-manager","readOnly":true},{"mountPath":"/etc/systemd","name":"etc-systemd","readOnly":true},{"mountPath":"/lib/systemd/","name":"lib-systemd","readOnly":true},{"mountPath":"/etc/kubernetes","name":"etc-kubernetes","readOnly":true},{"mountPath":"/etc/cni/net.d/","name":"etc-cni-netd","readOnly":true}]
  nodeCollector.volumes: >-
    [{"hostPath":{"path":"/var/lib/etcd"},"name":"var-lib-etcd"},{"hostPath":{"path":"/var/lib/kubelet"},"name":"var-lib-kubelet"},{"hostPath":{"path":"/var/lib/kube-scheduler"},"name":"var-lib-kube-scheduler"},{"hostPath":{"path":"/var/lib/kube-controller-manager"},"name":"var-lib-kube-controller-manager"},{"hostPath":{"path":"/etc/systemd"},"name":"etc-systemd"},{"hostPath":{"path":"/lib/systemd"},"name":"lib-systemd"},{"hostPath":{"path":"/etc/kubernetes"},"name":"etc-kubernetes"},{"hostPath":{"path":"/etc/cni/net.d/"},"name":"etc-cni-netd"}]
  policies.bundle.oci.ref: ghcr.io/aquasecurity/trivy-policies:0
  report.recordFailedChecksOnly: 'true'
  scanJob.compressLogs: 'true'
  scanJob.podTemplateContainerSecurityContext: >-
    {"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"privileged":false,"readOnlyRootFilesystem":true,"runAsUser":0}
  scanJob.podTemplateLabels: azure.workload.identity/use=true
  scanJob.tolerations: >-
    [{"effect":"NoExecute","key":"agentpool","operator":"Equal","value":"default"}]
  vulnerabilityReports.scanner: Trivy

trivy-operator-config

apiVersion: v1
kind: ConfigMap
metadata:
  name: trivy-operator-config
  namespace: trivy-operator
data:
  CONTROLLER_CACHE_SYNC_TIMEOUT: 5m
  OPERATOR_ACCESS_GLOBAL_SECRETS_SERVICE_ACCOUNTS: 'true'
  OPERATOR_BATCH_DELETE_DELAY: 10s
  OPERATOR_BATCH_DELETE_LIMIT: '10'
  OPERATOR_BUILT_IN_TRIVY_SERVER: 'false'
  OPERATOR_CACHE_REPORT_TTL: 120h
  OPERATOR_CLUSTER_COMPLIANCE_ENABLED: 'true'
  OPERATOR_CLUSTER_SBOM_CACHE_ENABLED: 'false'
  OPERATOR_CONCURRENT_NODE_COLLECTOR_LIMIT: '1'
  OPERATOR_CONCURRENT_SCAN_JOBS_LIMIT: '10'
  OPERATOR_CONFIG_AUDIT_SCANNER_ENABLED: 'true'
  OPERATOR_CONFIG_AUDIT_SCANNER_SCAN_ONLY_CURRENT_REVISIONS: 'true'
  OPERATOR_EXPOSED_SECRET_SCANNER_ENABLED: 'true'
  OPERATOR_HEALTH_PROBE_BIND_ADDRESS: ':9090'
  OPERATOR_INFRA_ASSESSMENT_SCANNER_ENABLED: 'true'
  OPERATOR_LOG_DEV_MODE: 'false'
  OPERATOR_MERGE_RBAC_FINDING_WITH_CONFIG_AUDIT: 'false'
  OPERATOR_METRICS_BIND_ADDRESS: ':8080'
  OPERATOR_METRICS_CLUSTER_COMPLIANCE_INFO_ENABLED: 'true'
  OPERATOR_METRICS_CONFIG_AUDIT_INFO_ENABLED: 'true'
  OPERATOR_METRICS_EXPOSED_SECRET_INFO_ENABLED: 'true'
  OPERATOR_METRICS_FINDINGS_ENABLED: 'true'
  OPERATOR_METRICS_IMAGE_INFO_ENABLED: 'false'
  OPERATOR_METRICS_INFRA_ASSESSMENT_INFO_ENABLED: 'true'
  OPERATOR_METRICS_RBAC_ASSESSMENT_INFO_ENABLED: 'true'
  OPERATOR_METRICS_VULN_ID_ENABLED: 'true'
  OPERATOR_PRIVATE_REGISTRY_SCAN_SECRETS_NAMES: '{}'
  OPERATOR_RBAC_ASSESSMENT_SCANNER_ENABLED: 'true'
  OPERATOR_SBOM_GENERATION_ENABLED: 'true'
  OPERATOR_SCANNER_REPORT_TTL: 24h
  OPERATOR_SCAN_JOB_RETRY_AFTER: 30s
  OPERATOR_SCAN_JOB_TIMEOUT: 5m
  OPERATOR_SCAN_JOB_TTL: ''
  OPERATOR_SEND_DELETED_REPORTS: 'false'
  OPERATOR_VULNERABILITY_SCANNER_ENABLED: 'true'
  OPERATOR_VULNERABILITY_SCANNER_SCAN_ONLY_CURRENT_REVISIONS: 'true'
  OPERATOR_WEBHOOK_BROADCAST_TIMEOUT: 30s
  OPERATOR_WEBHOOK_BROADCAST_URL: ''
  TRIVY_SERVER_HEALTH_CHECK_CACHE_EXPIRATION: 10h

trivy-operator-trivy-config

apiVersion: v1
kind: ConfigMap
metadata:
  name: trivy-operator-trivy-config
  namespace: trivy-operator
data:
  trivy.additionalVulnerabilityReportFields: ''
  trivy.command: image
  trivy.dbRepository: ***.azurecr.io/aquasecurity/trivy-db
  trivy.dbRepositoryInsecure: 'true'
  trivy.filesystemScanCacheDir: /var/trivyoperator/trivy-db
  trivy.imagePullPolicy: IfNotPresent
  trivy.imageScanCacheDir: /tmp/trivy/.cache
  trivy.includeDevDeps: 'false'
  trivy.javaDbRepository: ghcr.io/aquasecurity/trivy-java-db
  trivy.mode: Standalone
  trivy.repository: ***.azurecr.io/aquasecurity/trivy
  trivy.resources.limits.cpu: 500m
  trivy.resources.limits.memory: 500M
  trivy.resources.requests.cpu: 100m
  trivy.resources.requests.memory: 100M
  trivy.sbomSources: ''
  trivy.severity: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL
  trivy.skipJavaDBUpdate: 'false'
  trivy.slow: 'true'
  trivy.supportedConfigAuditKinds: >-
    Workload,Service,Role,ClusterRole,NetworkPolicy,Ingress,LimitRange,ResourceQuota
  trivy.tag: 0.49.1
  trivy.timeout: 5m0s
  trivy.useBuiltinRegoPolicies: 'true'

While pasting the configmaps i made the observation that there are 2 ghcr repo references in there. We are in an air gapped environment, and need to pull from our own azurecr repo. Could the issue be that we cant pull them even though its not visible in the logs?

[KevinDW-Fluxys]

@chen-keinan it was set to that, but now its set to ***.azurecr.io/aquasecurity/trivy-policies:0
We are indeed running on an air-gapped environment. We have set up a caching rule on our repo so that should essentially mirror the ghcr repo.

[chen-keinan]

does this repo ***.azurecr.io/aquasecurity/trivy-policies:0 require authentications ?

[KevinDW-Fluxys]

Yes it does. For the vulnerability database we are doing the same thing and the system is using Azure Workload Identity to authenticate. I dont know if this pull is made in the same way or if we can have other means of authenticating against it? If this is the cause, it would also be nice to see it logged somewhere.

[chen-keinan]

have you tried setting bundle credentials

[KevinDW-Fluxys]

I haven't yet, but is there no other way of setting credentials? (by external secret or by leveraging azure workload identity like the vulnerability scanner)
We cant include credentials in our helm files.

[chen-keinan]

the credential eventually goes to trivy-operator secret , you can deploy or patch trivy-operator secret directly

[KevinDW-Fluxys]

Ok, great. Is there a way we can let helm know that we are manually providing this secret just like with the trivy-operator-trivy-config and its trivy.existingSecret? We are using argocd, and as long as the helm chart is generating this secret it will "fight" the manually created secret

[chen-keinan]

currently no, but can be added easily, can you create github issue. can you also confirm that adding bundle setting will solve your problem ?