Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrade to arrow-14.0.2 #15108

Merged
merged 4 commits into from
Feb 22, 2024
Merged

Upgrade to arrow-14.0.2 #15108

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ec1b081
Upgrade to arrow 15
galipremsagar Feb 21, 2024
defb7aa
upgrade to 14.0.2
galipremsagar Feb 21, 2024
ed05ac7
Apply suggestions from code review
galipremsagar Feb 21, 2024
c395fe4
Apply suggestions from code review
galipremsagar Feb 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
10 changes: 5 additions & 5 deletions conda/environments/all_cuda-118_arch-x86_64.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,15 +40,15 @@ dependencies:
- hypothesis
- identify>=2.5.20
- ipython
- libarrow-acero==14.0.1.*
- libarrow-dataset==14.0.1.*
- libarrow==14.0.1.*
- libarrow-acero==14.0.2.*
- libarrow-dataset==14.0.2.*
- libarrow==14.0.2.*
- libcufile-dev=1.4.0.31
- libcufile=1.4.0.31
- libcurand-dev=10.3.0.86
- libcurand=10.3.0.86
- libkvikio==24.4.*
- libparquet==14.0.1.*
- libparquet==14.0.2.*
- librdkafka>=1.9.0,<1.10.0a0
- librmm==24.4.*
- make
Expand All @@ -71,7 +71,7 @@ dependencies:
- pre-commit
- protobuf>=4.21,<5
- ptxcompiler
- pyarrow==14.0.1.*
- pyarrow==14.0.2.*
- pydata-sphinx-theme!=0.14.2
- pytest-benchmark
- pytest-cases>=3.8.2
Expand Down
10 changes: 5 additions & 5 deletions conda/environments/all_cuda-122_arch-x86_64.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,13 +41,13 @@ dependencies:
- hypothesis
- identify>=2.5.20
- ipython
- libarrow-acero==14.0.1.*
- libarrow-dataset==14.0.1.*
- libarrow==14.0.1.*
- libarrow-acero==14.0.2.*
- libarrow-dataset==14.0.2.*
- libarrow==14.0.2.*
- libcufile-dev
- libcurand-dev
- libkvikio==24.4.*
- libparquet==14.0.1.*
- libparquet==14.0.2.*
- librdkafka>=1.9.0,<1.10.0a0
- librmm==24.4.*
- make
Expand All @@ -68,7 +68,7 @@ dependencies:
- pip
- pre-commit
- protobuf>=4.21,<5
- pyarrow==14.0.1.*
- pyarrow==14.0.2.*
- pydata-sphinx-theme!=0.14.2
- pynvjitlink
- pytest-benchmark
Expand Down
2 changes: 1 addition & 1 deletion conda/recipes/cudf/meta.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ requirements:
- scikit-build-core >=0.7.0
- setuptools
- dlpack >=0.5,<0.6.0a0
- pyarrow ==14.0.1.*
- pyarrow ==14.0.2.*
- libcudf ={{ version }}
- rmm ={{ minor_version }}
{% if cuda_major == "11" %}
Expand Down
2 changes: 1 addition & 1 deletion conda/recipes/libcudf/conda_build_config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ gtest_version:
- ">=1.13.0"

libarrow_version:
- "==14.0.1"
- "==14.0.2"

dlpack_version:
- ">=0.5,<0.6.0a0"
Expand Down
4 changes: 2 additions & 2 deletions cpp/cmake/thirdparty/get_arrow.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# =============================================================================
# Copyright (c) 2020-2023, NVIDIA CORPORATION.
# Copyright (c) 2020-2024, NVIDIA CORPORATION.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
# in compliance with the License. You may obtain a copy of the License at
Expand Down Expand Up @@ -441,7 +441,7 @@ if(NOT DEFINED CUDF_VERSION_Arrow)
set(CUDF_VERSION_Arrow
# This version must be kept in sync with the libarrow version pinned for builds in
# dependencies.yaml.
14.0.1
14.0.2
CACHE STRING "The version of Arrow to find (or build)"
)
endif()
Expand Down
22 changes: 10 additions & 12 deletions dependencies.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -261,7 +261,7 @@ dependencies:
- &numpy numpy>=1.21
# Hard pin the patch version used during the build. This must be kept
# in sync with the version pinned in get_arrow.cmake.
- pyarrow==14.0.1.*
- pyarrow==14.0.2.*
- output_types: conda
packages:
- scikit-build-core>=0.7.0
Expand Down Expand Up @@ -306,27 +306,25 @@ dependencies:
packages:
# Hard pin the Arrow patch version used during the build. This must
# be kept in sync with the version pinned in get_arrow.cmake.
- libarrow-acero==14.0.1.*
- libarrow-dataset==14.0.1.*
- libarrow==14.0.1.*
- libparquet==14.0.1.*
- libarrow-acero==14.0.2.*
- libarrow-dataset==14.0.2.*
- libarrow==14.0.2.*
- libparquet==14.0.2.*
libarrow_run:
common:
- output_types: conda
packages:
# Allow runtime version to float up to minor version
# Disallow libarrow 14.0.0 due to a CVE
- libarrow-acero>=14.0.1,<15.0.0a0
- libarrow-dataset>=14.0.1,<15.0.0a0
- libarrow>=14.0.1,<15.0.0a0
- libparquet>=14.0.1,<15.0.0a0
- libarrow-acero>=14.0.2,<15.0.0a0
- libarrow-dataset>=14.0.2,<15.0.0a0
- libarrow>=14.0.2,<15.0.0a0
- libparquet>=14.0.2,<15.0.0a0
galipremsagar marked this conversation as resolved.
Show resolved Hide resolved
pyarrow_run:
common:
- output_types: [conda, requirements, pyproject]
packages:
# Allow runtime version to float up to minor version
# Disallow pyarrow 14.0.0 due to a CVE
- pyarrow>=14.0.1,<15.0.0a0
- pyarrow>=14.0.2,<15.0.0a0
Copy link
Contributor

@bdice bdice Feb 21, 2024 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want the lower bound to be 14.0.2? I thought we would build with 14.0.2 and allow >=14.0.1,<15.0.0a0 at runtime to pin the major version for compatibility and continue excluding 14.0.0 due to the CVE.

Same in libarrow_run.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason for avoiding 14.0.1 here is to skip the issue where we are not able to downgrade the env from 15 to 14.0.1

Copy link
Contributor

@bdice bdice Feb 21, 2024 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed offline - we want to keep the prior pinnings here for runtime compatibility with any Arrow major version 14 (except 14.0.0 due to the CVE).

galipremsagar marked this conversation as resolved.
Show resolved Hide resolved
cuda_version:
specific:
- output_types: conda
Expand Down
4 changes: 2 additions & 2 deletions python/cudf/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ requires = [
"ninja",
"numpy>=1.21",
"protoc-wheel",
"pyarrow==14.0.1.*",
"pyarrow==14.0.2.*",
"rmm==24.4.*",
"scikit-build-core[pyproject]>=0.7.0",
] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.
Expand Down Expand Up @@ -36,7 +36,7 @@ dependencies = [
"pandas>=2.0,<2.1.5dev0",
"protobuf>=4.21,<5",
"ptxcompiler",
"pyarrow>=14.0.1,<15.0.0a0",
"pyarrow>=14.0.2,<15.0.0a0",
galipremsagar marked this conversation as resolved.
Show resolved Hide resolved
"rich",
"rmm==24.4.*",
"typing_extensions>=4.0.0",
Expand Down
2 changes: 1 addition & 1 deletion python/cudf_kafka/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ requires = [
"cython>=3.0.3",
"ninja",
"numpy>=1.21",
"pyarrow==14.0.1.*",
"pyarrow==14.0.2.*",
"scikit-build-core[pyproject]>=0.7.0",
] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.

Expand Down