Trivy DB Download from Private Registry errors with Authentication Required #1885
Labels
kind/bug
Categorizes issue or PR as related to a bug.
Comments
|
Adding some more context here for the benefit of the maintainers. We dealt with an initial obstacle while hosting the TrivyDB outside Kubernetes using the Trivy CLI server mode. We facilitated the operator's communication with it by integrating the following line in the trivy-operator-trivy-config configMap: data:
trivy.serverURL: <serverURL>But still the issue of not being able to reach artifactory persists. The We attempted to provide the imagePullSecret via different means:
But to no avail. |
|
It is also worth mentioning that in the and this by the end when it crashes and restarts: |
|
@chheda-deshaw please uninstall operator delete all crds : install latest operator |
|
@phsys please share your config maps |
|
trivy-operator.txt As you can see we have made certain alterations to the configmaps to help the operator go through artifactory.
So we could delete the operator and reinstall it but we would need these changes again so wouldn't we be back to square one ? |
|
@chen-keinan We uninstalled the operator and deleted all the crds, followed by installing the latest version (0.19.4) |
have you setup insecure-registries ? I can't find it in your config also make sure db-Repository is also set to insecure |
|
@chen-keinan Thanks for the suggestion. I made the above two edits and now from the tls cert issue we are running into 'Authentication is required' issue.. |
|
@hore-deshaw are you using image pull secrets in pods or service account? is so, how do you create the secrets ? it should be in for of user/ password , like this: |
|
@chen-keinan |
|
@hore-deshaw can please put here example of how to pod/deployment descriptor with image pull secrets? And also could you please create a secret with fake credential but same registry (same way as you created it) and also put it here, I want to see how it look like |
|
The pod descriptor: apiVersion: v1
kind: Pod
metadata:
annotations:
cni.projectcalico.org/containerID: 681c9e3a031da616ab3e6f407a3b2e4563fe398815f29770d7f3d5d6f0ae14ac
cni.projectcalico.org/podIP: 192.168.239.147/32
cni.projectcalico.org/podIPs: 192.168.239.147/32
creationTimestamp: "2024-04-16T13:42:36Z"
generateName: trivy-operator-668c66fb6-
labels:
app.kubernetes.io/instance: trivy-operator
app.kubernetes.io/name: trivy-operator
pod-template-hash: 668c66fb6
name: trivy-operator-668c66fb6-f9xxq
namespace: trivy-system
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: trivy-operator-668c66fb6
uid: 92b44468-7c74-4677-a60c-c0bb51482cd4
resourceVersion: "638276169"
uid: 5129a832-33cd-4a23-8ef0-bf7e0e2ee868
spec:
automountServiceAccountToken: true
containers:
- env:
- name: OPERATOR_NAMESPACE
value: trivy-system
- name: OPERATOR_TARGET_NAMESPACES
- name: OPERATOR_EXCLUDE_NAMESPACES
- name: OPERATOR_TARGET_WORKLOADS
value: pod,replicaset,replicationcontroller,statefulset,daemonset,cronjob,job
- name: OPERATOR_SERVICE_ACCOUNT
value: trivy-operator
envFrom:
- configMapRef:
name: trivy-operator-config
image: artifactory.deshaw.com/k8s/aquasecurity/trivy-operator:0.19.4
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 10
httpGet:
path: /healthz/
port: probes
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: trivy-operator
ports:
- containerPort: 8080
name: metrics
protocol: TCP
- containerPort: 9090
name: probes
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /readyz/
port: probes
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources: {}
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
privileged: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /tmp
name: cache-policies
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-5thmj
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
nodeName: hdc1webwrkqa4.k8s.des.co
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: trivy-operator
serviceAccountName: trivy-operator
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- emptyDir: {}
name: cache-policies
- name: kube-api-access-5thmj
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2024-04-16T13:42:36Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2024-04-16T13:43:27Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2024-04-16T13:43:27Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2024-04-16T13:42:36Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: cri-o://c2700613cff7a185ffe994f4b96f23a117cf8429fc36d0231d4d3a8b389ef14b
image: artifactory.deshaw.com/k8s/aquasecurity/trivy-operator:0.19.4
imageID: artifactory.deshaw.com/k8s/aquasecurity/trivy-operator@sha256:305ef05858765ecd0ba1a6ad7d2519c878bb0b94152b1fcf8470b2b6df896d46
lastState: {}
name: trivy-operator
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2024-04-16T13:43:18Z"
hostIP: 10.240.183.134
phase: Running
podIP: 192.168.239.147
podIPs:
- ip: 192.168.239.147
qosClass: BestEffort
startTime: "2024-04-16T13:42:36Z"The imagePullSecret has been defined in The secret looks like: apiVersion: v1
data:
.dockerconfigjson: <secret>
kind: Secret
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","data":{".dockerconfigjson":"<secret>"},"kind":"Secret","metadata":{"annotations":{},"name":"artifactory-secret","namespace":"trivy-system"},"type":"kubernetes.io/dockerconfigjson"}
creationTimestamp: "2024-02-26T18:11:43Z"
name: artifactory-secret
namespace: trivy-system
resourceVersion: "553969501"
uid: 7a54369e-326e-4283-90c5-88d42c428909
type: kubernetes.io/dockerconfigjson |
|
@chheda-deshaw I see here two things which I would like to challenge:
|
|
Hi @chen-keinan,
This is the image that trivy is scanning, it's not a pod but might be one of the images used in a pod. I can't provide that cause it's proprietary information. |
|
@chheda-deshaw I was trying to understand which method of private registry authentication are using in you cluster so understand if something is misconfigure, if you could share that info. |
|
We are using Artifactory as a private registry. ....<Redacted>....
Containers:
scoreboard:
Container ID: cri-o://5d0bbedd0a28fb11ce511ce2b7db835ba02c040173023c1980385e8fe4c0bacd
Image: artifactory.deshaw.com/k8s/scoreboard
Image ID: artifactory.deshaw.com/k8s/scoreboard@sha256:8653a0b99555c244038946af08ffb14e7139e2253931ce15df87359d3276ce3f
Port: 8000/TCP
Host Port: 0/TCP
SeccompProfile: RuntimeDefault
State: Running
Started: Sat, 13 Apr 2024 22:07:20 -0400
Ready: True
Restart Count: 0
Limits:
cpu: 4
ephemeral-storage: 128M
memory: 512M
Requests:
cpu: 25m
ephemeral-storage: 128M
memory: 512M
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-j92q8 (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
kube-api-access-j92q8:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: topology.kubernetes.io/region=nyc
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events: <none> |
|
is the pod with |
|
Yes. labels:
app.kubernetes.io/managed-by: k8s-selfserve.deshaw.com
k8s-selfserve.deshaw.com/namespace: trivy-systemto the imagePullSecret. Now, the vuln scans have begin, but sometimes they still error out with : But they go through sometimes, for eg this is the vuln report from the namespace where trivy-operator was installed: So I'm not sure why its still inconsistent. |
|
@chheda-deshaw the three main things needed to be checked:
|
Yes it is
It exists with the same name
It is not in a username/password format. But the current format works for pods to talk to artifactory. |
|
trivy support user/password. |
|
We use the image pull secret of type What exactly do you mean by user/password format? |
|
if
if it is created this way then its ok: |
|
Yes that is how it is created. |
|
@chheda-deshaw can you please summarized the current status, some images are scanned and some has authentication issues ? |
|
Yes precisely. I guess the base64 bits mean successful jobs. Here is the logs of trivy-operator: |
|
@chheda-deshaw do you see vulnerabilities reports? |
|
Yes |
|
@chheda-deshaw can you take a look at one of the auth failing pods (containers) see if it match the three conditions I mention above ? |
|
It does. I was watching the operator for the past 1 hour and it has been trying to scan certain pods that are failing again and again. I had set the |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What steps did you take and what happened:
I was installing
trivy-operatoron Kubernetes, and using Artifactory as a private registry.I have tried all of second, third and fourth options from https://aquasecurity.github.io/trivy-operator/v0.5.0/tutorials/private-registries/ to configure an image pull secret for this. The other images download all right. But it is just the trivy database download which fails with the below error
What did you expect to happen:
I would expect this to be able to authenticate as the image pull secret is already provided.
Anything else you would like to add:
I have gone through a number of past/closed issues on trivy for similar kind of occurrences, but none have helped so far.
Environment:
trivy-operator version): v0.18.5kubectl version): 1.25.15The text was updated successfully, but these errors were encountered: