Failed to get new registration token while installing ARC org wide for enterprise GITHUB

[itsmekarthikreddy]

Checks

• I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
• I'm not using a custom entrypoint in my runner image

Controller Version

0.27.6

Helm Chart Version

0.23.7

CertManager Version

1.8.0

Deployment Method

Helm

cert-manager installation

Yes Installed as per ARC Doc

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
• I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

---
apiVersion: actions.summerwind.dev/v1alpha1
kind: HorizontalRunnerAutoscaler
metadata:
  name: k8s-runners-autoscaler
  namespace: XXXX-actions-runners
spec:
  scaleTargetRef:
    name: k8s-runners
  scaleDownDelaySecondsAfterScaleOut: 300
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: TotalNumberOfQueuedAndInProgressWorkflowRuns
    repositoryNames:
    - XXX/XXX
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: Runner
metadata:
  name: k8s-single-runner
  namespace: XXXX-actions-runners
spec:
  repository: XXXX/XXXX
  env: []
---
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: k8s-runners
  namespace: XXXX-actions-runners
spec:
  template:
    spec:
      repository: XXXX/XXXX
      labels:
        - XXX-core-stack
      env: []

To Reproduce

Create GIT HUB app as per https://github.com/actions/actions-runner-controller/blob/master/docs/authenticating-to-the-github-api.md
Create secret controller-manager in actions-runners-system with above details and pem key

Describe the bug

Failed to get new registration token

Describe the expected behavior

Runner should be created successfully.

Whole Controller Logs

https://gist.github.com/itsmekarthikreddy/18c5112333acaee2c54f0945b0f87222

Whole Runner Pod Logs

https://gist.github.com/itsmekarthikreddy/18c5112333acaee2c54f0945b0f87222

Additional Context

No response

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[itsmekarthikreddy]

FYI , I have created a standard cluster not auto pilot in gcp

actions-actions-runner-controller manger pod logs :

github.com/actions-runner-controller/actions-runner-controller/controllers.(*RunnerReconciler).Reconcile
	/workspace/controllers/runner_controller.go:189
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:298
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214
2024-05-02T08:44:52.580Z	ERROR	controller-runtime.manager.controller.runner-controller	Reconciler error	{"reconciler group": "actions.summerwind.dev", "reconciler kind": "Runner", "name": "k8s-runners-4xf79-lhnrk", "namespace": "actions", "error": "failed to create registration token: POST https://api.github.com/repos/XXXX-XXX/XXXX/actions/runners/registration-token: 401 Bad credentials []"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.9.0/pkg/internal/controller/controller.go:214
2024-05-02T08:44:52.580Z	DEBUG	controller-runtime.manager.events	Warning	{"object": {"kind":"Runner","namespace":"actions","name":"k8s-runners-4xf79-lhnrk","uid":"5a59b3b4-8186-4450-b05d-94c0108400e3","apiVersion":"actions.summerwind.dev/v1alpha1","resourceVersion":"8139"}, "reason": "FailedUpdateRegistrationToken", "message": "Updating registration token failed"}
2024-05-02T08:44:52.774Z	ERROR	actions-runner-controller.runner	Failed to get new registration token	{"runner": "k8s-single-runner", "error": "failed to create registration token: POST https://api.github.com/repos/XXXX-XXX/XXXX/actions/runners/registration-token: 401 Bad credentials []"}
github.com/actions-runner-controller/actions-runner-controller/controllers.(*RunnerReconciler).Reconcile```
	

[cdwbrad]

@itsmekarthikreddy I'm currently experiencing a similar issue. Were you able to resolve? And if so, how?