Skip to content

Commit

Permalink
[release/3.0] Update dependencies from dotnet/arcade (#6938)
Browse files Browse the repository at this point in the history
* Update dependencies from https://github.com/dotnet/arcade build 20231220.2

Microsoft.DotNet.Arcade.Sdk , Microsoft.DotNet.Build.Tasks.Feed , Microsoft.DotNet.Helix.Sdk , Microsoft.DotNet.SignTool , Microsoft.DotNet.SwaggerGenerator.MSBuild , Microsoft.DotNet.XUnitExtensions
 From Version 8.0.0-beta.23265.1 -> To Version 8.0.0-beta.23620.2

* Fixed version update breaks.

* Update XUnitVersion

* Update MicrosoftMLOnnxRuntimeVersion to 1.16.3

* Rollback OnnxRuntime and suppress warning

* Update to Xunit with fix for xunit/xunit#2821

* Update Centos docker containers

* Fix packaging step

* Try including stdint.h to fix missing uint8_t on centos

* Update Centos test queue

* Attempt to use runtime centos-stream8-helix container for tests

* Use centos-stream8-mlnet-helix container for testing

* Undo changes to test data

* Make NETFRAMEWORK ifdef versionless

* Only use semi-colons for NoWarn

* Fix assert by only accessing idx (#6924)

Asserting on `_rowCount <  Utils.Size(_valueBoundaries)` was catching a
case where `_rowCount`'s update was reordered before `_valueBoundaries`

This was unnecessary, since this method doesn't need to use `_rowCount`.

Instead, make the asserts use only `idx` which will be maintained
consistent with the waiter logic in this cache.
�Ensure we only ever use `_rowCount` from the caching thread, so write
reordering won't matter.

* Don't include the SDK in our helix payload (#6918)

* Don't include the SDK in our helix payload

I noticed that the tests included the latest SDK - including the host -
in our helix payloads.

This is a large amount of unnecessary downloads and it also makes it so
we use the latest host on the older frameworks which can fail when the
latest host drops support for distros.

Since our tests shouldn't need the full CLI, remove this from our helix
payloads.

We'll instead get just the runtime we need through `AdditionalDotNetPackage`

* Place Helix downloaded runtime on the PATH

Helix only sets the path when the CLI is included, however we don't
need the CLI.

* Make double assertions compare with tolerance instead of precision (#6923)

Precision might cause small differences to round to a different number.
Instead compare with a tolerance which is not sensitive to rounding.

---------

Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Michael Sharp <misharp@microsoft.com>
Co-authored-by: Eric StJohn <ericstj@microsoft.com>
  • Loading branch information
4 people committed Jan 9, 2024
1 parent 4f268c7 commit f602581
Show file tree
Hide file tree
Showing 83 changed files with 700 additions and 366 deletions.
4 changes: 2 additions & 2 deletions .vsts-dotnet-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ trigger:
resources:
containers:
- container: CentosContainer
image: mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7-mlnet
image: mcr.microsoft.com/dotnet-buildtools/prereqs:centos-stream8-mlnet

- container: UbuntuContainer
image: mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-mlnet
Expand Down Expand Up @@ -86,7 +86,7 @@ jobs:
pool:
name: NetCore-Public
demands: ImageOverride -equals build.ubuntu.1804.amd64.open
helixQueue: Ubuntu.1804.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7-mlnet-helix
helixQueue: Ubuntu.1804.Amd64.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:centos-stream8-mlnet-helix

- template: /build/ci/job-template.yml
parameters:
Expand Down
6 changes: 4 additions & 2 deletions Directory.Build.props
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@
<TargetArchitecture>$(TargetArchitecture.ToLower())</TargetArchitecture>
<PlatformConfig>$(Platform).$(Configuration)</PlatformConfig>
<StrongNameKeyId>Open</StrongNameKeyId>

<!-- temporarily suppress NETSDK 1206 until we can update OnnxRuntime to fix https://github.com/dotnet/machinelearning/issues/6916 -->
<NoWarn>$(NoWarn);NETSDK1206</NoWarn>
</PropertyGroup>

<PropertyGroup>
Expand Down Expand Up @@ -54,8 +57,7 @@
<!-- Optional: Embed source files that are not tracked by the source control manager in the PDB -->
<EmbedUntrackedSources>true</EmbedUntrackedSources>

<!-- Optional: Build symbol package (.snupkg) to distribute the PDB containing Source Link -->
<!-- Optional: Build symbol packages (.symbols.nupkg) to distribute the PDB containing Source Link -->
<IncludeSymbols>true</IncludeSymbols>
<SymbolPackageFormat>snupkg</SymbolPackageFormat>
</PropertyGroup>
</Project>
2 changes: 1 addition & 1 deletion NuGet.config
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<?xml version="1.0" encoding="utf-8"?>
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<solution>
<add key="disableSourceControlIntegration" value="true" />
Expand Down
2 changes: 1 addition & 1 deletion build/.night-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ schedules:
resources:
containers:
- container: CentosContainer
image: mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7-mlnet
image: mcr.microsoft.com/dotnet-buildtools/prereqs:centos-stream8-mlnet

- container: UbuntuContainer
image: mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-mlnet
Expand Down
2 changes: 1 addition & 1 deletion build/.outer-loop-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ schedules:
resources:
containers:
- container: CentosContainer
image: mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7-mlnet
image: mcr.microsoft.com/dotnet-buildtools/prereqs:centos-stream8-mlnet

- container: UbuntuContainer
image: mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-mlnet
Expand Down
2 changes: 0 additions & 2 deletions build/ci/send-to-helix.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@ parameters:
WarnAsError: ''
TestTargetFramework: ''
HelixConfiguration: '' # optional -- additional property attached to a job
IncludeDotNetCli: true # optional -- true will download a version of the .NET CLI onto the Helix machine as a correlation payload; requires DotNetCliPackageType and DotNetCliVersion
EnableXUnitReporter: true # optional -- true enables XUnit result reporting to Mission Control
WaitForWorkItemCompletion: true # optional -- true will make the task wait until work items have been completed and fail the build if work items fail. False is "fire and forget."
HelixBaseUri: 'https://helix.dot.net' # optional -- sets the Helix API base URI (allows targeting int)
Expand All @@ -34,7 +33,6 @@ steps:
/p:HelixBuild=${{ parameters.HelixBuild }}
/p:HelixConfiguration="${{ parameters.HelixConfiguration }}"
/p:HelixAccessToken="${{ parameters.HelixAccessToken }}"
/p:IncludeDotNetCli=${{ parameters.IncludeDotNetCli }}
/p:EnableXUnitReporter=${{ parameters.EnableXUnitReporter }}
/p:WaitForWorkItemCompletion=${{ parameters.WaitForWorkItemCompletion }}
/p:HelixBaseUri=${{ parameters.HelixBaseUri }}
Expand Down
2 changes: 1 addition & 1 deletion build/vsts-ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
resources:
containers:
- container: CentosContainer
image: mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7-mlnet
image: mcr.microsoft.com/dotnet-buildtools/prereqs:centos-stream8-mlnet

- container: UbuntuCrossArmContainer
image: mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-mlnet-cross-arm
Expand Down
24 changes: 12 additions & 12 deletions eng/Version.Details.xml
Original file line number Diff line number Diff line change
Expand Up @@ -7,25 +7,25 @@
</Dependency>
</ProductDependencies>
<ToolsetDependencies>
<Dependency Name="Microsoft.DotNet.Arcade.Sdk" Version="8.0.0-beta.23265.1">
<Dependency Name="Microsoft.DotNet.Arcade.Sdk" Version="8.0.0-beta.23620.2">
<Uri>https://github.com/dotnet/arcade</Uri>
<Sha>2d8d59065b5e090584a8e90c4371fc06ed60bdc5</Sha>
<Sha>84129325171e65373edef24019e1171feeb19cbc</Sha>
</Dependency>
<Dependency Name="Microsoft.DotNet.Build.Tasks.Feed" Version="8.0.0-beta.23265.1">
<Dependency Name="Microsoft.DotNet.Build.Tasks.Feed" Version="8.0.0-beta.23620.2">
<Uri>https://github.com/dotnet/arcade</Uri>
<Sha>2d8d59065b5e090584a8e90c4371fc06ed60bdc5</Sha>
<Sha>84129325171e65373edef24019e1171feeb19cbc</Sha>
</Dependency>
<Dependency Name="Microsoft.DotNet.SignTool" Version="8.0.0-beta.23265.1">
<Dependency Name="Microsoft.DotNet.SignTool" Version="8.0.0-beta.23620.2">
<Uri>https://github.com/dotnet/arcade</Uri>
<Sha>2d8d59065b5e090584a8e90c4371fc06ed60bdc5</Sha>
<Sha>84129325171e65373edef24019e1171feeb19cbc</Sha>
</Dependency>
<Dependency Name="Microsoft.DotNet.Helix.Sdk" Version="8.0.0-beta.23265.1">
<Dependency Name="Microsoft.DotNet.Helix.Sdk" Version="8.0.0-beta.23620.2">
<Uri>https://github.com/dotnet/arcade</Uri>
<Sha>2d8d59065b5e090584a8e90c4371fc06ed60bdc5</Sha>
<Sha>84129325171e65373edef24019e1171feeb19cbc</Sha>
</Dependency>
<Dependency Name="Microsoft.DotNet.SwaggerGenerator.MSBuild" Version="8.0.0-beta.23265.1">
<Dependency Name="Microsoft.DotNet.SwaggerGenerator.MSBuild" Version="8.0.0-beta.23620.2">
<Uri>https://github.com/dotnet/arcade</Uri>
<Sha>2d8d59065b5e090584a8e90c4371fc06ed60bdc5</Sha>
<Sha>84129325171e65373edef24019e1171feeb19cbc</Sha>
</Dependency>
<Dependency Name="Microsoft.DotNet.Maestro.Client" Version="1.1.0-beta.20074.1">
<Uri>https://github.com/dotnet/arcade-services</Uri>
Expand All @@ -39,9 +39,9 @@
<Uri>https://github.com/dotnet/xharness</Uri>
<Sha>89cb4b1d368e0f15b4df8e02a176dd1f1c33958b</Sha>
</Dependency>
<Dependency Name="Microsoft.DotNet.XUnitExtensions" Version="8.0.0-beta.23265.1">
<Dependency Name="Microsoft.DotNet.XUnitExtensions" Version="8.0.0-beta.23620.2">
<Uri>https://github.com/dotnet/arcade</Uri>
<Sha>2d8d59065b5e090584a8e90c4371fc06ed60bdc5</Sha>
<Sha>84129325171e65373edef24019e1171feeb19cbc</Sha>
</Dependency>
<Dependency Name="Microsoft.Net.Compilers.Toolset" Version="3.8.0-3.20460.2">
<Uri>https://github.com/dotnet/roslyn</Uri>
Expand Down
4 changes: 2 additions & 2 deletions eng/Versions.props
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@
<DotNetRuntime80Version>8.0.0</DotNetRuntime80Version>
<FluentAssertionVersion>5.10.2</FluentAssertionVersion>
<MicrosoftCodeAnalysisTestingVersion>1.1.2-beta1.23431.1</MicrosoftCodeAnalysisTestingVersion>
<MicrosoftDotNetXUnitExtensionsVersion>8.0.0-beta.23265.1</MicrosoftDotNetXUnitExtensionsVersion>
<MicrosoftDotNetXUnitExtensionsVersion>8.0.0-beta.23620.2</MicrosoftDotNetXUnitExtensionsVersion>
<MicrosoftExtensionsDependencyModelVersion>2.1.0</MicrosoftExtensionsDependencyModelVersion>
<MicrosoftExtensionsTestVersion>3.0.1</MicrosoftExtensionsTestVersion>
<MicrosoftMLOnnxTestModelsVersion>0.0.6-test</MicrosoftMLOnnxTestModelsVersion>
Expand All @@ -91,7 +91,7 @@
<SystemDataSqlClientVersion>4.8.5</SystemDataSqlClientVersion>
<SystemDataSQLiteCoreVersion>1.0.118</SystemDataSQLiteCoreVersion>
<XunitCombinatorialVersion>1.2.7</XunitCombinatorialVersion>
<XUnitVersion>2.4.2</XUnitVersion>
<XUnitVersion>2.6.3</XUnitVersion>
<!-- Opt-out repo features -->
<UsingToolXliff>false</UsingToolXliff>
</PropertyGroup>
Expand Down
2 changes: 1 addition & 1 deletion eng/common/SetupNugetSources.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ if ($dotnet31Source -ne $null) {
AddPackageSource -Sources $sources -SourceName "dotnet3.1-internal-transport" -SourceEndPoint "https://pkgs.dev.azure.com/dnceng/_packaging/dotnet3.1-internal-transport/nuget/v2" -Creds $creds -Username $userName -Password $Password
}

$dotnetVersions = @('5','6','7')
$dotnetVersions = @('5','6','7','8')

foreach ($dotnetVersion in $dotnetVersions) {
$feedPrefix = "dotnet" + $dotnetVersion;
Expand Down
2 changes: 1 addition & 1 deletion eng/common/SetupNugetSources.sh
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ if [ "$?" == "0" ]; then
PackageSources+=('dotnet3.1-internal-transport')
fi

DotNetVersions=('5' '6' '7')
DotNetVersions=('5' '6' '7' '8')

for DotNetVersion in ${DotNetVersions[@]} ; do
FeedPrefix="dotnet${DotNetVersion}";
Expand Down
2 changes: 1 addition & 1 deletion eng/common/cross/toolchain.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -207,6 +207,7 @@ elseif(ILLUMOS)
set(CMAKE_CXX_STANDARD_LIBRARIES "${CMAKE_CXX_STANDARD_LIBRARIES} -lssp")
elseif(HAIKU)
set(CMAKE_SYSROOT "${CROSS_ROOTFS}")
set(CMAKE_PROGRAM_PATH "${CMAKE_PROGRAM_PATH};${CROSS_ROOTFS}/cross-tools-x86_64/bin")

set(TOOLSET_PREFIX ${TOOLCHAIN}-)
function(locate_toolchain_exec exec var)
Expand All @@ -217,7 +218,6 @@ elseif(HAIKU)
endif()

find_program(EXEC_LOCATION_${exec}
PATHS "${CROSS_ROOTFS}/cross-tools-x86_64/bin"
NAMES
"${TOOLSET_PREFIX}${exec}${CLR_CMAKE_COMPILER_FILE_NAME_VERSION}"
"${TOOLSET_PREFIX}${exec}")
Expand Down
2 changes: 1 addition & 1 deletion eng/common/darc-init.ps1
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
param (
$darcVersion = $null,
$versionEndpoint = 'https://maestro-prod.westus2.cloudapp.azure.com/api/assets/darc-version?api-version=2019-01-16',
$versionEndpoint = 'https://maestro.dot.net/api/assets/darc-version?api-version=2019-01-16',
$verbosity = 'minimal',
$toolpath = $null
)
Expand Down
2 changes: 1 addition & 1 deletion eng/common/darc-init.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

source="${BASH_SOURCE[0]}"
darcVersion=''
versionEndpoint='https://maestro-prod.westus2.cloudapp.azure.com/api/assets/darc-version?api-version=2019-01-16'
versionEndpoint='https://maestro.dot.net/api/assets/darc-version?api-version=2019-01-16'
verbosity='minimal'

while [[ $# > 0 ]]; do
Expand Down
4 changes: 4 additions & 0 deletions eng/common/dotnet-install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,10 @@ cpuname=$(uname -m)
case $cpuname in
arm64|aarch64)
buildarch=arm64
if [ "$(getconf LONG_BIT)" -lt 64 ]; then
# This is 32-bit OS running on 64-bit CPU (for example Raspberry Pi OS)
buildarch=arm
fi
;;
loongarch64)
buildarch=loongarch64
Expand Down
Binary file modified eng/common/loc/P22DotNetHtmlLocalization.lss
Binary file not shown.
2 changes: 1 addition & 1 deletion eng/common/native/init-compiler.sh
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ if [ -z "$CLR_CC" ]; then
# Set default versions
if [ -z "$majorVersion" ]; then
# note: gcc (all versions) and clang versions higher than 6 do not have minor version in file name, if it is zero.
if [ "$compiler" = "clang" ]; then versions="16 15 14 13 12 11 10 9 8 7 6.0 5.0 4.0 3.9 3.8 3.7 3.6 3.5"
if [ "$compiler" = "clang" ]; then versions="17 16 15 14 13 12 11 10 9 8 7 6.0 5.0 4.0 3.9 3.8 3.7 3.6 3.5"
elif [ "$compiler" = "gcc" ]; then versions="13 12 11 10 9 8 7 6 5 4.9"; fi

for version in $versions; do
Expand Down
130 changes: 130 additions & 0 deletions eng/common/native/init-distro-rid.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
#!/usr/bin/env bash

# getNonPortableDistroRid
#
# Input:
# targetOs: (str)
# targetArch: (str)
# rootfsDir: (str)
#
# Return:
# non-portable rid
getNonPortableDistroRid()
{
local targetOs="$1"
local targetArch="$2"
local rootfsDir="$3"
local nonPortableRid=""

if [ "$targetOs" = "linux" ]; then
if [ -e "${rootfsDir}/etc/os-release" ]; then
source "${rootfsDir}/etc/os-release"

if [[ "${ID}" == "rhel" || "${ID}" == "rocky" || "${ID}" == "alpine" ]]; then
# remove the last version digit
VERSION_ID="${VERSION_ID%.*}"
fi

if [[ "${VERSION_ID:-}" =~ ^([[:digit:]]|\.)+$ ]]; then
nonPortableRid="${ID}.${VERSION_ID}-${targetArch}"
else
# Rolling release distros either do not set VERSION_ID, set it as blank or
# set it to non-version looking string (such as TEMPLATE_VERSION_ID on ArchLinux);
# so omit it here to be consistent with everything else.
nonPortableRid="${ID}-${targetArch}"
fi

elif [ -e "${rootfsDir}/android_platform" ]; then
source "$rootfsDir"/android_platform
nonPortableRid="$RID"
fi
fi

if [ "$targetOs" = "freebsd" ]; then
# $rootfsDir can be empty. freebsd-version is shell script and it should always work.
__freebsd_major_version=$($rootfsDir/bin/freebsd-version | { read v; echo "${v%%.*}"; })
nonPortableRid="freebsd.$__freebsd_major_version-${targetArch}"
elif command -v getprop && getprop ro.product.system.model 2>&1 | grep -qi android; then
__android_sdk_version=$(getprop ro.build.version.sdk)
nonPortableRid="android.$__android_sdk_version-${targetArch}"
elif [ "$targetOs" = "illumos" ]; then
__uname_version=$(uname -v)
case "$__uname_version" in
omnios-*)
__omnios_major_version=$(echo "${__uname_version:8:2}")
nonPortableRid=omnios."$__omnios_major_version"-"$targetArch"
;;
joyent_*)
__smartos_major_version=$(echo "${__uname_version:7:4}")
nonPortableRid=smartos."$__smartos_major_version"-"$targetArch"
;;
illumos_*)
nonPortableRid=openindiana-"$targetArch"
;;
esac
elif [ "$targetOs" = "solaris" ]; then
__uname_version=$(uname -v)
__solaris_major_version=$(echo "${__uname_version%.*}")
nonPortableRid=solaris."$__solaris_major_version"-"$targetArch"
elif [ "$targetOs" = "haiku" ]; then
__uname_release=$(uname -r)
nonPortableRid=haiku.r"$__uname_release"-"$targetArch"
fi

echo "$(echo $nonPortableRid | tr '[:upper:]' '[:lower:]')"
}

# initDistroRidGlobal
#
# Input:
# os: (str)
# arch: (str)
# rootfsDir?: (nullable:string)
#
# Return:
# None
#
# Notes:
#
# It is important to note that the function does not return anything, but it
# exports the following variables on success:
#
# __DistroRid : Non-portable rid of the target platform.
# __PortableTargetOS : OS-part of the portable rid that corresponds to the target platform.
#
initDistroRidGlobal()
{
local targetOs="$1"
local targetArch="$2"
local rootfsDir=""
if [ "$#" -ge 3 ]; then
rootfsDir="$3"
fi

if [ -n "${rootfsDir}" ]; then
# We may have a cross build. Check for the existence of the rootfsDir
if [ ! -e "${rootfsDir}" ]; then
echo "Error rootfsDir has been passed, but the location is not valid."
exit 1
fi
fi

__DistroRid=$(getNonPortableDistroRid "${targetOs}" "${targetArch}" "${rootfsDir}")

if [ -z "${__PortableTargetOS:-}" ]; then
__PortableTargetOS="$targetOs"

STRINGS="$(command -v strings || true)"
if [ -z "$STRINGS" ]; then
STRINGS="$(command -v llvm-strings || true)"
fi

# Check for musl-based distros (e.g Alpine Linux, Void Linux).
if "${rootfsDir}/usr/bin/ldd" --version 2>&1 | grep -q musl ||
( [ -n "$STRINGS" ] && "$STRINGS" "${rootfsDir}/usr/bin/ldd" 2>&1 | grep -q musl ); then
__PortableTargetOS="linux-musl"
fi
fi

export __DistroRid __PortableTargetOS
}

0 comments on commit f602581

Please sign in to comment.