Azure DevOps windows-2022 job canceled: "The hosted runner encountered an error while running your job. (Error Type: Disconnect)"

[dd-eg]

Description

We have an Azure DevOps build that was canceled early this morning (Eastern Time) with this shown in the job log:

##[error]The hosted runner encountered an error while running your job. (Error Type: Disconnect).
,##[warning]Received request to deprovision: The request was cancelled by the remote provider.
Agent: Hosted Agent
Started: Today at 12:01 AM
Duration: 22m 10s

After completing a few steps successfully, one appears to have hung for about 20m before cancellation:

Nothing to show. Final logs are missing. This can happen when the job is cancelled or times out.

The job succeeded when we manually restarted it.

I might have written it off as a side-effect of the step that hung (it was attempting an install, maybe there was some catastrophic failure?), but then I saw a similar issue was logged this morning: #9807

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• Windows Server 2019
• Windows Server 2022

Image version and build link

windows-2022 20240421.1.0

Failed attempt:
https://dev.azure.com/EscherGroup/CPGC/_build/results?buildId=109283&view=logs&j=a6596b93-0b2c-508f-9b86-f5ba206b41f1&s=b8b17376-ff6a-545d-1f5c-adfaeaadce2c

Successful attempt:
https://dev.azure.com/EscherGroup/CPGC/_build/results?buildId=109283&view=logs&j=d3a99131-01cb-512c-0397-73ea34defab2

Is it regression?

No

Expected behavior

The hosted runner should not disconnect.

Actual behavior

The hosted runner disconnected.

Repro steps

Cannot reproduce. Appears intermittent -- we have had several builds since without issue.

[erik-bershel]

Hey @dd-eg!
This error almost always indicates that the runner was overloaded with your task and communication with it was lost. Unfortunately, we do not investigate cases related to the lack of resources. You can independently optimise your workflow so that it consumes fewer resources or take advantage of the community advice presented in the corresponding discussion (provided that you find something similar to your config, of course): #7188.

[dd-eg]

Thanks @erik-bershel, I appreciate the quick response! I'll look through the the provided link and see what improvements we can make.

[erik-bershel]

We can try to find a specific resource that you have exhausted (RAM/CPU/IOPS), but for this we will need a link to one unsuccessful and one successful launch (it does not matter whether they belong to a private repository - we will look at other logs and we do not need access to the repo/launch itself).

[dd-eg]

Oh, that's neat -- I have updated the description to include links to failed/successful attempts (they are attempts #1 and #2 in the same build, if that matters).

[erik-bershel]

Hey @dd-eg!
I checked resources for those runs. Seems like the failed job had random memory leak - second attempt job shows normal resource level. I may recommend to ignore this failure - green job consume reasonable amount of CPU/RAM/IOPS.

[dd-eg]

Thanks for the investigation, much appreciated!