Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[8.0] [outerloop] jit-format: Format job found errors, please apply the format patch #95159

Closed
carlossanlop opened this issue Nov 23, 2023 · 7 comments
Closed

[8.0] [outerloop] jit-format: Format job found errors, please apply the format patch #95159

carlossanlop opened this issue Nov 23, 2023 · 7 comments
Labels
arch-x64 area-Infrastructure blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' blocking-outerloop Blocking the 'runtime-coreclr outerloop' and 'runtime-libraries-coreclr outerloop' runs Known Build Error Use this to report build issues in the .NET Helix tab os-linux Linux OS (any supported distro)

Comments

@carlossanlop
Copy link
Member

carlossanlop commented Nov 23, 2023 •
edited by build-analysis bot

Error Blob

Is this owned by dnceng? Asking in case I need to transfer it to their repo.

{
  "ErrorMessage": "Format job found errors, please apply the format patch.",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Reproduction Steps

MSBuild version 17.3.2+561848881 for .NET
  Determining projects to restore...
  Restored /tmp/tmpgfcnoyaq/jitutils/src/instructions-retired-explorer/instructions-retired-explorer.csproj (in 716 ms).
  instructions-retired-explorer -> /tmp/tmpgfcnoyaq/jitutils/src/instructions-retired-explorer/bin/Release/net6.0/linux-x64/instructions-retired-explorer.dll
  instructions-retired-explorer -> /tmp/tmpgfcnoyaq/jitutils/bin/
Adding /tmp/tmpgfcnoyaq/jitutils/bin to PATH
Done setting up!
[22:49:48] Running: /tmp/tmpgfcnoyaq/jitutils/bin/jit-format -a x64 -b Checked -o linux -c /__w/1/s/src/coreclr --verbose --projects dll
error: invalid option -c
error: invalid option -c
[22:49:48] Running: /tmp/tmpgfcnoyaq/jitutils/bin/jit-format -a x64 -b Checked -o linux -c /__w/1/s/src/coreclr --verbose --projects standalone
error: invalid option -c
error: invalid option -c
[22:49:48] Running: /tmp/tmpgfcnoyaq/jitutils/bin/jit-format -a x64 -b Checked -o linux -c /__w/1/s/src/coreclr --verbose --projects crossgen
error: invalid option -c
error: invalid option -c
[22:49:48] Running: /tmp/tmpgfcnoyaq/jitutils/bin/jit-format -a x64 -b Debug -o linux -c /__w/1/s/src/coreclr --verbose --projects dll
error: invalid option -c
error: invalid option -c
[22:49:48] Running: /tmp/tmpgfcnoyaq/jitutils/bin/jit-format -a x64 -b Debug -o linux -c /__w/1/s/src/coreclr --verbose --projects standalone
error: invalid option -c
error: invalid option -c
[22:49:49] Running: /tmp/tmpgfcnoyaq/jitutils/bin/jit-format -a x64 -b Debug -o linux -c /__w/1/s/src/coreclr --verbose --projects crossgen
error: invalid option -c
error: invalid option -c
[22:49:49] Running: /tmp/tmpgfcnoyaq/jitutils/bin/jit-format -a x64 -b Release -o linux -c /__w/1/s/src/coreclr --verbose --projects dll
error: invalid option -c
error: invalid option -c
[22:49:49] Running: /tmp/tmpgfcnoyaq/jitutils/bin/jit-format -a x64 -b Release -o linux -c /__w/1/s/src/coreclr --verbose --projects standalone
error: invalid option -c
error: invalid option -c
[22:49:49] Running: /tmp/tmpgfcnoyaq/jitutils/bin/jit-format -a x64 -b Release -o linux -c /__w/1/s/src/coreclr --verbose --projects crossgen
error: invalid option -c
error: invalid option -c
[22:49:49] Creating patch file /__w/1/s/src/coreclr/format.patch
[22:49:49] Deleting /tmp/tmpgfcnoyaq/bootstrap.sh
[22:49:49] There were errors in formatting. Please run jit-format locally with: 

[22:49:49] 	jit-format -a x64 -b Checked -o linux -c <absolute-path-to-coreclr> --verbose --fix --projects dll
	jit-format -a x64 -b Checked -o linux -c <absolute-path-to-coreclr> --verbose --fix --projects standalone
	jit-format -a x64 -b Checked -o linux -c <absolute-path-to-coreclr> --verbose --fix --projects crossgen
	jit-format -a x64 -b Debug -o linux -c <absolute-path-to-coreclr> --verbose --fix --projects dll
	jit-format -a x64 -b Debug -o linux -c <absolute-path-to-coreclr> --verbose --fix --projects standalone
	jit-format -a x64 -b Debug -o linux -c <absolute-path-to-coreclr> --verbose --fix --projects crossgen
	jit-format -a x64 -b Release -o linux -c <absolute-path-to-coreclr> --verbose --fix --projects dll
	jit-format -a x64 -b Release -o linux -c <absolute-path-to-coreclr> --verbose --fix --projects standalone
	jit-format -a x64 -b Release -o linux -c <absolute-path-to-coreclr> --verbose --fix --projects crossgen

[22:49:49] 
Or download and apply generated patch:
[22:49:49] 1. From the GitHub 'Checks' page on the Pull Request, with the failing Formatting
[22:49:49]    job selected (e.g., 'Formatting Linux x64'), click the 'View more details on
[22:49:49]    Azure Pipelines' link.
[22:49:49] 2. Select the '1 artifact produced' at the end of the log.
[22:49:49] 3. Artifacts are located in alphabetical order, target artifact name is
[22:49:49]    'format.<OS>.<architecture>.patch.'. Find appropriate format patch artifact.
[22:49:49] 4. On the right side of the artifact there is a 'More actions' menu shown by a
[22:49:49]    vertical three-dot symbol. Click on it and select 'Download artifacts' option.
[22:49:49] 5. Unzip the patch file.
[22:49:49] 6. git apply format.patch

##[error](NETCORE_ENGINEERING_TELEMETRY=Build) Format job found errors, please apply the format patch.
##[error]Bash exited with code '1'.
Finishing: Run jitformat.py

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=478601
Error message validated: Format job found errors, please apply the format patch.
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 11/23/2023 2:31:07 AM UTC

Report

Build Definition Step Name Console log Pull Request
491504 dotnet/runtime Run jitformat.py Log #89472
490745 dotnet/runtime Run jitformat.py Log #95700
489989 dotnet/runtime Run jitformat.py Log #95665
489637 dotnet/runtime Run jitformat.py Log #94684
488765 dotnet/runtime Run jitformat.py Log #95618
488349 dotnet/runtime Run jitformat.py Log #95565
487515 dotnet/runtime Run jitformat.py Log #95565
484641 dotnet/runtime Run jitformat.py Log
484007 dotnet/runtime Run jitformat.py Log #94684
482667 dotnet/runtime Run jitformat.py Log #94684
482611 dotnet/runtime Run jitformat.py Log #95337
481552 dotnet/runtime Run jitformat.py Log
481483 dotnet/runtime Run jitformat.py Log #94684
480903 dotnet/runtime Run jitformat.py Log #95127
479832 dotnet/runtime Run jitformat.py Log #95127
479340 dotnet/runtime Run jitformat.py Log #95127
479210 dotnet/runtime Run jitformat.py Log #95127
479006 dotnet/runtime Run jitformat.py Log #95127
478984 dotnet/runtime Run jitformat.py Log #95128
478764 dotnet/runtime Run jitformat.py Log
478601 dotnet/runtime Run jitformat.py Log
478317 dotnet/runtime Run jitformat.py Log #95127
478299 dotnet/runtime Run jitformat.py Log
478239 dotnet/runtime Run jitformat.py Log #95128
477803 dotnet/runtime Run jitformat.py Log
477736 dotnet/runtime Run jitformat.py Log
477638 dotnet/runtime Run jitformat.py Log
477552 dotnet/runtime Run jitformat.py Log

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
1 7 28
@carlossanlop carlossanlop added os-linux Linux OS (any supported distro) arch-x64 blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' area-Infrastructure blocking-outerloop Blocking the 'runtime-coreclr outerloop' and 'runtime-libraries-coreclr outerloop' runs Known Build Error Use this to report build issues in the .NET Helix tab labels Nov 23, 2023
@ghost
Copy link

ghost commented Nov 23, 2023

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

Error Blob

Is this owned by dnceng? Asking in case I need to transfer it to their repo.

{
  "ErrorMessage": "Format job found errors, please apply the format patch.",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Reproduction Steps

MSBuild version 17.3.2+561848881 for .NET
  Determining projects to restore...
  Restored /tmp/tmpgfcnoyaq/jitutils/src/instructions-retired-explorer/instructions-retired-explorer.csproj (in 716 ms).
  instructions-retired-explorer -> /tmp/tmpgfcnoyaq/jitutils/src/instructions-retired-explorer/bin/Release/net6.0/linux-x64/instructions-retired-explorer.dll
  instructions-retired-explorer -> /tmp/tmpgfcnoyaq/jitutils/bin/
Adding /tmp/tmpgfcnoyaq/jitutils/bin to PATH
Done setting up!
[22:49:48] Running: /tmp/tmpgfcnoyaq/jitutils/bin/jit-format -a x64 -b Checked -o linux -c /__w/1/s/src/coreclr --verbose --projects dll
error: invalid option -c
error: invalid option -c
[22:49:48] Running: /tmp/tmpgfcnoyaq/jitutils/bin/jit-format -a x64 -b Checked -o linux -c /__w/1/s/src/coreclr --verbose --projects standalone
error: invalid option -c
error: invalid option -c
[22:49:48] Running: /tmp/tmpgfcnoyaq/jitutils/bin/jit-format -a x64 -b Checked -o linux -c /__w/1/s/src/coreclr --verbose --projects crossgen
error: invalid option -c
error: invalid option -c
[22:49:48] Running: /tmp/tmpgfcnoyaq/jitutils/bin/jit-format -a x64 -b Debug -o linux -c /__w/1/s/src/coreclr --verbose --projects dll
error: invalid option -c
error: invalid option -c
[22:49:48] Running: /tmp/tmpgfcnoyaq/jitutils/bin/jit-format -a x64 -b Debug -o linux -c /__w/1/s/src/coreclr --verbose --projects standalone
error: invalid option -c
error: invalid option -c
[22:49:49] Running: /tmp/tmpgfcnoyaq/jitutils/bin/jit-format -a x64 -b Debug -o linux -c /__w/1/s/src/coreclr --verbose --projects crossgen
error: invalid option -c
error: invalid option -c
[22:49:49] Running: /tmp/tmpgfcnoyaq/jitutils/bin/jit-format -a x64 -b Release -o linux -c /__w/1/s/src/coreclr --verbose --projects dll
error: invalid option -c
error: invalid option -c
[22:49:49] Running: /tmp/tmpgfcnoyaq/jitutils/bin/jit-format -a x64 -b Release -o linux -c /__w/1/s/src/coreclr --verbose --projects standalone
error: invalid option -c
error: invalid option -c
[22:49:49] Running: /tmp/tmpgfcnoyaq/jitutils/bin/jit-format -a x64 -b Release -o linux -c /__w/1/s/src/coreclr --verbose --projects crossgen
error: invalid option -c
error: invalid option -c
[22:49:49] Creating patch file /__w/1/s/src/coreclr/format.patch
[22:49:49] Deleting /tmp/tmpgfcnoyaq/bootstrap.sh
[22:49:49] There were errors in formatting. Please run jit-format locally with: 

[22:49:49] 	jit-format -a x64 -b Checked -o linux -c <absolute-path-to-coreclr> --verbose --fix --projects dll
	jit-format -a x64 -b Checked -o linux -c <absolute-path-to-coreclr> --verbose --fix --projects standalone
	jit-format -a x64 -b Checked -o linux -c <absolute-path-to-coreclr> --verbose --fix --projects crossgen
	jit-format -a x64 -b Debug -o linux -c <absolute-path-to-coreclr> --verbose --fix --projects dll
	jit-format -a x64 -b Debug -o linux -c <absolute-path-to-coreclr> --verbose --fix --projects standalone
	jit-format -a x64 -b Debug -o linux -c <absolute-path-to-coreclr> --verbose --fix --projects crossgen
	jit-format -a x64 -b Release -o linux -c <absolute-path-to-coreclr> --verbose --fix --projects dll
	jit-format -a x64 -b Release -o linux -c <absolute-path-to-coreclr> --verbose --fix --projects standalone
	jit-format -a x64 -b Release -o linux -c <absolute-path-to-coreclr> --verbose --fix --projects crossgen

[22:49:49] 
Or download and apply generated patch:
[22:49:49] 1. From the GitHub 'Checks' page on the Pull Request, with the failing Formatting
[22:49:49]    job selected (e.g., 'Formatting Linux x64'), click the 'View more details on
[22:49:49]    Azure Pipelines' link.
[22:49:49] 2. Select the '1 artifact produced' at the end of the log.
[22:49:49] 3. Artifacts are located in alphabetical order, target artifact name is
[22:49:49]    'format.<OS>.<architecture>.patch.'. Find appropriate format patch artifact.
[22:49:49] 4. On the right side of the artifact there is a 'More actions' menu shown by a
[22:49:49]    vertical three-dot symbol. Click on it and select 'Download artifacts' option.
[22:49:49] 5. Unzip the patch file.
[22:49:49] 6. git apply format.patch

##[error](NETCORE_ENGINEERING_TELEMETRY=Build) Format job found errors, please apply the format patch.
##[error]Bash exited with code '1'.
Finishing: Run jitformat.py
Author: carlossanlop
Assignees: -
Labels:

os-linux, arch-x64, blocking-clean-ci, area-Infrastructure, blocking-outerloop, Known Build Error
Milestone: -

@ghost ghost added the untriaged New issue has not been triaged by the area owner label Nov 23, 2023
@akoeplinger
Copy link
Member

Is this owned by dnceng?

No this is owned by the JIT team, /cc @dotnet/jit-contrib

We should probably only run the format job on main and disable it on release branches.

@BruceForstall
Copy link
Member

@carlossanlop Which branch or branches should I create a PR for to fix this? Namely, to disable the formatting job for servicing branches.

I did this for release/6.0 with #61485

We probably need to do this for both 7.0 and 8.0.

(Maybe we could use a condition like

runtime/eng/pipelines/runtime.yml

Lines 185 to 193 in 882c60c

jobParameters:
condition: >-
and(
or(
eq(variables['Build.SourceBranchName'], 'main'),
eq(variables['System.PullRequest.TargetBranch'], 'main')),
or(
eq(dependencies.evaluate_paths.outputs['SetPathVars_coreclr_jit.containsChange'], true),
eq(variables['isRollingBuild'], true)))
to avoid needing to do this every product cycle, but the outerloop pipeline doesn't run the evaluate-default-paths.yml stuff)

@BruceForstall
Copy link
Member

@carlossanlop ping

@carlossanlop
Copy link
Member Author

I missed the first notification, sorry.

So I found this issue in 8.0, as stated by the title. But looking at the autogenerated KnownBuildError table containing all the detected failures, this problem is affecting mostly main. Since you already fixed this in 6.0, then it means the issue could also be in 7.0 (I don't have proof though).

So: main, 8.0, 7.0.

@BruceForstall
Copy link
Member

@carlossanlop 'main' is ok (in any case, if there are problems there, they should be fixed, not have the job disabled).

As for 8.0/7.0, the question is: what is the exact branch name I should use? I get lost as to which branch names we use for ongoing fixes.

@carlossanlop
Copy link
Member Author

carlossanlop commented Nov 29, 2023 •
edited

Ah! For servicing changes, always target the staging branches: release/8.0-staging, release/7.0-staging, release/6.0-staging. Those branches are always open and you have permission to merge the PR when it's ready.

The base branches (release/X.0) are restricted to only admins.

This was referenced Nov 29, 2023
Merged
Merged
@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Dec 7, 2023
@BruceForstall BruceForstall mentioned this issue Dec 7, 2023
Merged
@github-actions github-actions bot locked and limited conversation to collaborators Jan 7, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
arch-x64 area-Infrastructure blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' blocking-outerloop Blocking the 'runtime-coreclr outerloop' and 'runtime-libraries-coreclr outerloop' runs Known Build Error Use this to report build issues in the .NET Helix tab os-linux Linux OS (any supported distro)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@carlossanlop @akoeplinger @BruceForstall