Error when connecting Ubuntu 22.04 Pro with fips-updates enabled

[mikedalton]

Describe the bug

When configuring an Ubuntu 22.04 Pro server with the fips-updates service enabled, the configuration fails with the error The signing token has no private key and cannot be used for signing.

To Reproduce
Steps to reproduce the behavior:

1. Create Ubuntu 22.04 server
2. Connect server to an Ubuntu Pro subscription
3. Enable FIPS security updates with the command sudo pro enable fips-updates
4. Reboot system if necessary
5. Try to configure the Actions Runner client, or if already configured, try to start the client
6. Observe the error The signing token has no private key and cannot be used for signing.

Expected behavior
The runner is able to configure and/or start normally.

Runner Version and Platform

Version of your runner?

2.314.1

OS of the machine running the runner? OSX/Windows/Linux/...

Ubuntu Pro 22.04 LTS

Linux ubuntu-vm 5.15.0-100-fips #110+fips1-Ubuntu SMP Mon Feb 26 18:37:06 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

What's not working?

$ ./run.sh

√ Connected to GitHub

2024-03-28 09:57:06Z: Runner connect error: The signing token has no private key and cannot be used for signing.. Retrying until reconnected.

√ Connected to GitHub


√ Connected to GitHub


√ Connected to GitHub

^CExiting...
Runner listener exit with 0 return code, stop the service, no retry needed.
Exiting runner...

Job Log Output

N/A

Runner and Worker's Diagnostic Logs

[2024-03-28 09:57:00Z INFO MessageListener] Loading Credentials
[2024-03-28 09:57:00Z INFO ConfigurationStore] HasCredentials()
[2024-03-28 09:57:00Z INFO ConfigurationStore] stored True
[2024-03-28 09:57:00Z INFO CredentialManager] GetCredentialProvider
[2024-03-28 09:57:00Z INFO CredentialManager] Creating type OAuth
[2024-03-28 09:57:00Z INFO CredentialManager] Creating credential type: OAuth
[2024-03-28 09:57:00Z INFO HostContext] Well known directory 'Bin': '/home/mike/actions-runner/bin'
[2024-03-28 09:57:00Z INFO HostContext] Well known directory 'Root': '/home/mike/actions-runner'
[2024-03-28 09:57:00Z INFO HostContext] Well known config file 'RSACredentials': '/home/mike/actions-runner/.credentials_rsaparams'
[2024-03-28 09:57:00Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/mike/actions-runner/.credentials_rsaparams
[2024-03-28 09:57:00Z INFO MessageListener] Attempt to create session.
[2024-03-28 09:57:00Z INFO MessageListener] Connecting to the Runner Server...
[2024-03-28 09:57:00Z INFO RunnerServer] EstablishVssConnection
[2024-03-28 09:57:00Z INFO RunnerServer] Establish connection with 100 seconds timeout.
[2024-03-28 09:57:00Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2024-03-28 09:57:01Z INFO RunnerServer] EstablishVssConnection
[2024-03-28 09:57:01Z INFO RunnerServer] Establish connection with 60 seconds timeout.
[2024-03-28 09:57:01Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2024-03-28 09:57:01Z INFO RunnerServer] EstablishVssConnection
[2024-03-28 09:57:01Z INFO RunnerServer] Establish connection with 60 seconds timeout.
[2024-03-28 09:57:01Z INFO GitHubActionsService] Starting operation Location.GetConnectionData
[2024-03-28 09:57:04Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2024-03-28 09:57:04Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2024-03-28 09:57:04Z INFO GitHubActionsService] Finished operation Location.GetConnectionData
[2024-03-28 09:57:05Z INFO MessageListener] VssConnection created
[2024-03-28 09:57:05Z INFO Terminal] WRITE LINE:
[2024-03-28 09:57:05Z INFO Terminal] WRITE LINE:
[2024-03-28 09:57:05Z INFO RSAFileKeyManager] Loading RSA key parameters from file /home/mike/actions-runner/.credentials_rsaparams
[2024-03-28 09:57:06Z ERR  MessageListener] Catch exception during create session.
[2024-03-28 09:57:06Z ERR  MessageListener] GitHub.Services.WebApi.Jwt.InvalidCredentialsException: The signing token has no private key and cannot be used for
signing.
   at GitHub.Services.WebApi.Jwt.JsonWebTokenUtilities.ValidateSigningCredentials(VssSigningCredentials credentials, Boolean allowExpiredToken)
   at GitHub.Services.WebApi.Jwt.JsonWebToken.GetHeader(VssSigningCredentials credentials, Boolean allowExpired)
   at GitHub.Services.WebApi.Jwt.JsonWebToken.Create(String issuer, String audience, DateTime validFrom, DateTime validTo, DateTime issuedAt, IEnumerable`1 addi
tionalClaims, JsonWebToken actor, String actorToken, VssSigningCredentials credentials, Boolean allowExpiredCertificate)
   at GitHub.Services.WebApi.Jwt.JsonWebToken.Create(String issuer, String audience, DateTime validFrom, DateTime validTo, IEnumerable`1 additionalClaims, VssSi
gningCredentials credentials)
   at GitHub.Services.OAuth.VssOAuthJwtBearerAssertion.GetBearerToken()
   at GitHub.Services.OAuth.VssOAuthJwtBearerClientCredential.SetParameters(IDictionary`2 parameters)
   at GitHub.Services.OAuth.VssOAuthClientCredential.GitHub.Services.OAuth.IVssOAuthTokenParameterProvider.SetParameters(IDictionary`2 parameters)
   at GitHub.Services.OAuth.VssOAuthTokenHttpClient.CreateRequestContent(IVssOAuthTokenParameterProvider[] parameterProviders)
   at GitHub.Services.OAuth.VssOAuthTokenHttpClient.GetTokenAsync(VssOAuthGrant grant, VssOAuthClientCredential credential, VssOAuthTokenParameters tokenParameters, CancellationToken cancellationToken)
   at GitHub.Services.OAuth.VssOAuthTokenProvider.OnGetTokenAsync(IssuedToken failedToken, CancellationToken cancellationToken)
   at GitHub.Services.Common.IssuedTokenProvider.GetTokenOperation.GetTokenAsync(VssTraceActivity traceActivity)
   at GitHub.Services.Common.IssuedTokenProvider.GetTokenAsync(IssuedToken failedToken, CancellationToken cancellationToken)
   at GitHub.Services.Common.VssHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at GitHub.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean
 disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at GitHub.Services.WebApi.VssHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken
cancellationToken)
   at GitHub.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpRequestMessage message, Object userState, CancellationToken cancellationToken)
   at GitHub.Services.WebApi.VssHttpClientBase.SendAsync[T](HttpMethod method, IEnumerable`1 additionalHeaders, Guid locationId, Object routeValues, ApiResource
Version version, HttpContent content, IEnumerable`1 queryParameters, Object userState, CancellationToken cancellationToken)
   at GitHub.Runner.Listener.MessageListener.CreateSessionAsync(CancellationToken token)
[2024-03-28 09:57:06Z INFO MessageListener] Retriable exception: The signing token has no private key and cannot be used for signing.
[2024-03-28 09:57:06Z ERR  Terminal] WRITE ERROR: 2024-03-28 09:57:06Z: Runner connect error: The signing token has no private key and cannot be used for signin
g.. Retrying until reconnected.
[2024-03-28 09:57:06Z INFO MessageListener] Sleeping for 30 seconds before retrying.
[2024-03-28 09:57:36Z INFO MessageListener] Attempt to create session.

Investigation

Following clues from the stack trace and error message, the failure appears to be happening in this section of the runner code, where it is attempted to determine whether or not there is a private key by attempting a signing operation. Failure of the signing operation (regardless of actual reason) is resulting in the assumption that there is no private key.

[stimko68]

I think the target for fixes should be fips-preview instead of fips-updates because the former contains the modules which are currently undergoing certification for Ubuntu 22.04 against FIPS 140-3.

Thanks for opening this issue. I've tried different configurations; none of them have worked and they produce the same error seen above.

[nemonik]

I got bit by this one too.

[gitdmb]

HPE has a question on FIPS controls for Ubuntu 22.04 for running their FedRAMP environment and are asking for any insight into a timeline for resolution.