Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed USB connection via port 54050, error 61, in tvOS arm64 Release AllSubsets_Mono #82637

Open
Tracked by #84254
carlossanlop opened this issue Feb 24, 2023 · 10 comments
Labels
arch-arm64 area-VM-meta-mono Known Build Error Use this to report build issues in the .NET Helix tab os-tvos Apple tvOS runtime-mono specific to the Mono runtime
Milestone

Comments

@carlossanlop
Copy link
Member

carlossanlop commented Feb 24, 2023

[13:58:26] dbug: [TCP tunnel] Xamarin.Hosting: Failed to connect to port 54050 (8915) on device (error: 61)
[13:58:26] dbug: [TCP tunnel] Xamarin.Hosting: Attempting USB tunnel between the port 54050 on the device and the port 54050 (8915) on the mac: 61

It's a bit confusing, since there's a net.dot.System.Runtime.Tests.log output file that shows some succeeded System.Runtime test executions.

Anyway, the USB failure seems to be unique, so I'm adding the "known issue" json text to track it.

{
  "ErrorMessage": "Xamarin.Hosting: Failed to connect to port",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Report

Build Definition Test Pull Request
693519 dotnet/runtime System.Runtime.Tests.WorkItemExecution #102907
692884 dotnet/runtime System.Runtime.Tests.WorkItemExecution #102874
692791 dotnet/runtime System.Runtime.Tests.WorkItemExecution #102636
689458 dotnet/runtime System.Threading.Tasks.Parallel.Tests.WorkItemExecution #102748
691915 dotnet/runtime System.Runtime.Tests.WorkItemExecution #102636
691468 dotnet/runtime System.Runtime.Tests.WorkItemExecution #102636
686902 dotnet/runtime System.Threading.Tasks.Parallel.Tests.WorkItemExecution
682021 dotnet/runtime System.Threading.Tasks.Parallel.Tests.WorkItemExecution
681751 dotnet/runtime System.Threading.Tasks.Parallel.Tests.WorkItemExecution
681987 dotnet/runtime System.Threading.Tasks.Parallel.Tests.WorkItemExecution
681837 dotnet/runtime System.Xml.XPath.XmlDocument.Tests.WorkItemExecution
679952 dotnet/runtime System.Runtime.Tests.WorkItemExecution #102392
679632 dotnet/runtime System.Runtime.Tests.WorkItemExecution #102392
679420 dotnet/runtime System.Resources.Writer.Tests.WorkItemExecution
678653 dotnet/runtime System.Threading.Tasks.Parallel.Tests.WorkItemExecution
678355 dotnet/runtime System.Xml.Schema.Extensions.Tests.WorkItemExecution
676930 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution
673265 dotnet/runtime System.Runtime.Tests.WorkItemExecution #102144
672783 dotnet/runtime System.Runtime.Tests.WorkItemExecution #102059
672730 dotnet/runtime System.Runtime.Tests.WorkItemExecution
672709 dotnet/runtime System.Runtime.Tests.WorkItemExecution #102084
672696 dotnet/runtime System.Runtime.Tests.WorkItemExecution #102098
672676 dotnet/runtime System.Runtime.Tests.WorkItemExecution #100823
672669 dotnet/runtime System.Runtime.Tests.WorkItemExecution #101461
670895 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution
665970 dotnet/runtime System.Formats.Tar.Tests.WorkItemExecution

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
1 6 26

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323893
Error message validated: Xamarin.Hosting: Failed to connect to port
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:28:49 PM UTC

@carlossanlop carlossanlop added arch-arm64 blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' area-VM-meta-mono runtime-mono specific to the Mono runtime os-tvos Apple tvOS Known Build Error Use this to report build issues in the .NET Helix tab labels Feb 24, 2023
@carlossanlop carlossanlop added this to the 7.0.x milestone Feb 24, 2023
@ghost
Copy link

ghost commented Feb 24, 2023

Tagging subscribers to 'os-tvos': @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

Issue Details
[13:58:26] dbug: [TCP tunnel] Xamarin.Hosting: Failed to connect to port 54050 (8915) on device (error: 61)
[13:58:26] dbug: [TCP tunnel] Xamarin.Hosting: Attempting USB tunnel between the port 54050 on the device and the port 54050 (8915) on the mac: 61

It's a bit confusing, since there's a net.dot.System.Runtime.Tests.log output file that shows some succeeded System.Runtime test executions.

Anyway, the USB failure seems to be unique, so I'm adding the "known issue" json text to track it.

{
  "ErrorMessage": "Xamarin.Hosting: Failed to connect to port 54050 (8915) on device (error: 61)",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}
Author: carlossanlop
Assignees: -
Labels:

arch-arm64, blocking-clean-ci, area-VM-meta-mono, runtime-mono, os-tvos, Known Build Error

Milestone: 7.0.x

@lewing
Copy link
Member

lewing commented Feb 24, 2023

dotnet/arcade#11683

@steveisok
Copy link
Member

Might be better to keep this open since it's tagged with a known build error label. I don't think we need to have blocking clean ci attached to it.

@carlossanlop
Copy link
Member Author

I originally logged this issue for tvOS, but I'm also seeing it in iOS arm64. Here is a fresh example:

[22:47:19] dbug: [TCP tunnel] Xamarin.Hosting: Attempting USB tunnel between the port 52498 on the device and the port 52498 (4813) on the mac: 61
[22:47:19] dbug: [TCP tunnel] Xamarin.Hosting: Failed to connect to port 52498 (4813) on device (error: 61)

@kotlarmilos
Copy link
Member

https://helix.dot.net/api/jobs/4cbf9a6d-6389-43f9-9e2c-16540105525e/workitems?api-version=2019-06-17

Both tests pass based on the logs, but one of them is unable to establish a connection via TCP, while the other establishes a connection every time. This failure seems to be permanent and could be a good starting point for identifying the issue.

When I attempted to reproduce the error locally, I encountered the same issue. However, I have not opened any ports, which might be a possible cause. I assume that the CI machines have ports 49152-65535 (source https://support.apple.com/en-us/HT202944).

I wonder if certain ports are busy or if there might be a firewall misconfiguration.

@premun Any ideas why the error persists when running the functional test?

/cc @akoeplinger @steveisok

@premun
Copy link
Member

premun commented Jun 30, 2023

@kotlarmilos if this was firewall misconfiguration, I think we should see the same set of machines failing. I don't think this is the case: https://dataexplorer.azure.com/clusters/engsrvprod/databases/engineeringdata?query=H4sIAAAAAAAAAx3MuwoCMRAF0N6vGLbabQI+ECy2EgsLBSvrMbmQoJmESaKr+PEu9odzTXo/VsSy+NLLQ0GHKdR9cqBxpN2KWBxdGhrOHEFBqO9SmcxybTi67cZwzg/Up0kZ0g3zUlqMrOEDsqlJ7Qe6venE1gf5HzNRiINSDrCetf4ANsm6/4QAAAA=

Seems like this is not tied to specific machines?

@premun Any ideas why the error persists when running the functional test?

Do I understand it correctly that you have 100% repro with a specific app?

@kotlarmilos
Copy link
Member

kotlarmilos commented Jun 30, 2023

Host: osx-arm64
Device: iPhone iOS 16.5.1
Xcode: 14.3.1 (14E300c)

  1. Install xharness locally (I don't believe it has anything with the repro, that is just my local setup):
# Using bash on Linux/MacOS
curl -L https://aka.ms/get-xharness | bash -
  1. Download the bundle and place it in a test directory.
    tracing_eventpipe_tcp_issue.zip

  2. Run the following script from the local directory:

./xharness apple test --app "<LOCAL_DIRECTORY>/tracing_eventpipe_tcp_issue.app" --target "ios-device" --output-directory "<LOCAL_DIRECTORY>/output" --set-env=MONO_APPLE_APP_ENTRY_POINT_LIB_NAME=testdir-buffersize/buffersize.dll --set-env=MONO_APPLE_APP_ASSEMBLY_LOAD_PREFIX=testdir-buffersize --xcode /Applications/Xcode.app -v --launch-timeout 00:05:00

The test should start executing and may fail (due to other reasons), but the TCP connection is not established. The app fails locally with

fail: Application test run crashed
      TCP Tunnel Connection Failed to connect to TCP port

@kotlarmilos
Copy link
Member

For the apple test command, XHarness expects the application to contain a TestRunner which is a library you can find in this repository. This library executes unit tests similarly how you would execute them on other platforms. However, the TestRunner from this repository contains more mechanisms that help to work around some issues (mostly in Apple platforms).

The runtime test did not utilize the TestRunner, and it was expected to fail. After further investigation and monitoring of the CI, it appears that the TCP failures might be related to the app. If such a failure occurs, ensure that the app functions correctly or consider using apple run commands.

@ericstj ericstj changed the title [7.0] Failed USB connection via port 54050, error 61, in tvOS arm64 Release AllSubsets_Mono Failed USB connection via port 54050, error 61, in tvOS arm64 Release AllSubsets_Mono Sep 13, 2023
@ericstj
Copy link
Member

ericstj commented Sep 13, 2023

Removed [7.0] from the title since this appears to be happening across branches.

@kotlarmilos
Copy link
Member

@ivanpovazan If you try to run the library tests using the apple run command, please share your results. I suspect the issue is either app-specific or related to the AppleTestRunner. We haven't observed this problem in runtime tests that utilize the apple run command.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arch-arm64 area-VM-meta-mono Known Build Error Use this to report build issues in the .NET Helix tab os-tvos Apple tvOS runtime-mono specific to the Mono runtime
Projects
None yet
Development

No branches or pull requests

7 participants