Refactor pipelines and add Installer jobs

[MiYanni]

Note

The commits for this PR will be squashed when merged.

Important

The publishing stage in the dotnet/installer repo MUST be disabled prior to merging this PR.

Summary

In the effort to move the dotnet/installer repo into this repo, we needed to move the different pipeline jobs into this repo. I realized when looking into this that there are a lot of different jobs in the Installer repo versus this repo. So, it was necessary to consolidate and refactor the pipelines in this repo to handle all the new variations.

I had some goals going into this:

• Simply the existing pipeline definition(s)
• Allow shared build logic between CI and PR pipelines
• Add the job variations from the dotnet/installer repo (based on different os/platform/container configuration)
• Remove unused information

Design

Due to the recent changes required for the 1ES PT, this design took several iterations. I attempted to use strategy matrix at first, but this didn't work as much of the information passed to the 1ES PT needs to be compile-time parameters, such as pool and container information. Strategy matrix only provides run-time variables, so I hit a limitation with it when trying to do the Linux container builds.

The main design uses the each expression to scaffold the different jobs and information that pertains to those jobs, which I've called parameter sets. There is one parameter set array per platform (windows, linux, macOS) so that we can have a set of default parameter sets which pertain to the jobs that run tests (generally, PR builds). Since there are also internal PRs, these default parameter set arrays are also used in the CI yaml in that particular situation.

The hierarchy is as such:

• The pipeline yaml (.vsts-ci.yml or .vsts-pr.yml) calls the sdk-job-matrix.yml template per platform (windows, linux, macOS).
  • For official builds, the parameter sets are defined in the .vsts-ci.yml, while the default parameter sets (for testing) are defined in sdk-job-matrix.yml.
• The sdk-job-matrix.yml template uses an each-expression to create separate instances of sdk-build.yml per parameter set array entry.
  • The parameters from both the pipeline yaml and the parameter set are send and flattened as parameters to the sdk-build.yml template.
• The sdk-build.yml template calls the appropriate job.yml template in Arcade depending on if the pipeline using 1ES PT or not.
  • The oneESCompat parameter provides the strings necessary to switch the appropriate paths and tasks for the 1ES PT-based pipeline (.vsts-ci.yml).
  • The steps contained in this template define both a build and test section. The test section is only ran if runTests is set to true, which it is by default, but set to false for regular CI builds.
  • The steps have separate sections for windows versus non-windows since the Arcade scripts used are different.
• Also created a sdk-defaults.yml which has some shared variables between the pipeline yamls (.vsts-ci.yml and .vsts-pr.yml).

Worthy Mentions

• Added a "Run A Test Build" setting when manually queuing a CI build if you want to check the tests without creating a PR (for example, only pushed to an internal branch).
• Made publishing the build-configuration.json a toggleable step as part of the main job, instead of needing a separate job that has to allocate an agent to publish 1 json file to the artifacts.
• Made the OneLocBuild branch a simple parameter in the Windows section, which currently is: locBranch: release/8.0.4xx
• Used displayName for the jobs so that the job names have an easy and human-readable structure to them.
• Fixed the dotnet/format tests so they actually run properly (they were incorrectly specified in the .vsts-ci.yml pipeline instead of the .vsts-pr.yml)
• All jobs run the build configuration as Release, which was decided upon for simplicity.

Test Runs

• CI
  • https://dnceng.visualstudio.com/internal/_build/results?buildId=2454301
• CI (in test build mode)
  • https://dnceng.visualstudio.com/internal/_build/results?buildId=2454302
• PR
  • https://dnceng-public.visualstudio.com/public/_build/results?buildId=679666

[ViktorHofer]

@MiYanni I pushed a couple commits into your branch as the official build was failing because of duplicate assets. I also updated with latest main and fixed a missing DotNetReleaseShipping metadata on the publish items.

[ViktorHofer]

With the latest commits the build manifest diff looks good. VS specific packages were missing which got fixed with the last commit. Wixpacks were also missing from signing/publishing which I fixed with the second last commit.

Official validation build: https://dev.azure.com/dnceng/internal/_build/results?buildId=2457090&view=results
Here's the merged asset manifest diff (left = LKG sdk + installer, right = this PR): https://www.diffchecker.com/GIZaX7pu/

The DotNetReleaseShipping difference is expected as I noticed that it was wrong and fixed it with f256ddd (third last commit)

[ViktorHofer]

The dotnet-sdk-public-ci (Build Dotnet-Format Integration Tests: sdk) leg is consistently failing. Should we disable it and file a tracking issue to unblock this PR? This was probably already the case before this PR and we just didn't notice it as it was running in non-PR builds.

[MiYanni]

@ViktorHofer That job ran just fine on my test runs for this PR: https://dnceng-public.visualstudio.com/public/_build/results?buildId=679666&view=logs&j=57a330b1-3866-5e48-ad49-546a977eda14

[ViktorHofer]

@ViktorHofer That job ran just fine on my test runs for this PR: https://dnceng-public.visualstudio.com/public/_build/results?buildId=679666&view=logs&j=57a330b1-3866-5e48-ad49-546a977eda14

My theory is that this regressed with a recent dependency flow.

[ViktorHofer]

Looking at the VMR windows failure