[FEA] Add pandas-2.0 support for cudf

[galipremsagar]

Is your feature request related to a problem? Please describe.
Pandas-2.0 is slated for release soon. This is a feature request to enable cudf to work with pandas-2.0

Tasks

Beta Give feedback

1. [REVIEW] Deprecate na_sentinel in factorize #12817
   5 - Ready to Merge Python breaking deprecation improvement +5
   
2. [REVIEW] Deprecate datetime_is_numeric from describe #12818
   5 - Ready to Merge Python breaking deprecation improvement +5
   
3. [REVIEW] Deprecate Index.is_* methods #12820
   5 - Ready to Merge Python breaking deprecation improvement +5
   
4. [REVIEW] Deprecate inplace parameters in categorical methods #12824
   5 - Ready to Merge Python improvement non-breaking +4
   
5. [REVIEW] Deprecate names & dtype in Index.copy #12825
   5 - Ready to Merge Python breaking deprecation improvement +5
   
6. [REVIEW] Adapt value_counts behavior to match pandas-2.x #12835
   5 - Ready to Merge Python breaking improvement +4
   
7. [REVIEW] Drop DataFrame.append and Series.append #12839
   5 - Ready to Merge Python breaking improvement +4
   
8. [REVIEW] Raise error when numeric_only=True for non-numeric Series #12843
   5 - Ready to Merge Python breaking improvement python +5
   
9. [REVIEW] Drop inplace parameter in categorical methods #12846
   5 - Ready to Merge Python breaking improvement +4
   
10. [REVIEW] Update numeric_only behavior in reduction APIs #12847
    5 - Ready to Merge Python breaking python +4
    
11. [REVIEW] Drop is_monotonic from Series and Index #12853
    5 - Ready to Merge Python breaking improvement +4
    
12. [REVIEW] Drop kind parameter from Index.get_slice_bound #12856
    5 - Ready to Merge Python breaking improvement +4
    
13. [REVIEW] Drop names, dtype in Index.copy and dtype, levels, codes in MultiIndex.copy #12898
    5 - Ready to Merge Python breaking improvement python +5
    
14. Deprecate line_terminator in favor of lineterminator in to_csv #12896
    5 - Ready to Merge Python deprecation improvement non-breaking +5
15. [REVIEW] Drop datetime_is_numeric parameter from describe #12890
    5 - Ready to Merge Python breaking improvement python +5
    
16. [REVIEW] Add information about Index.is_* method deprecation #12909
    5 - Ready to Merge Python breaking improvement +4
    
17. [REVIEW] Drop na_sentinel from factorize #12924
    5 - Ready to Merge Python breaking improvement python +5
    
18. [REVIEW] Fix pytests to add coverage for index ufunc support introduced in pandas-2.0 #13083
    5 - Ready to Merge Python +2
    
19. [REVIEW] datetime and timedelta improvements #12934
    5 - Ready to Merge Python breaking bug +4
    
20. [REVIEW] Enable numeric_only for row-wise ops #13090
    5 - Ready to Merge Python breaking improvement python +5
    
21. [REVIEW] Fix MultiIndex as column construction with pandas-2.0 #13092
    5 - Ready to Merge Python bug improvement +4
    
22. [REVIEW] Raise error when there is a binary operation between certain DataFrame and Series objects #13138
    5 - Ready to Merge Python breaking +3
    
23. [REVIEW] Fix DataFrame.__getitem__ to work with pandas-2.0 #13139
    5 - Ready to Merge Python bug non-breaking python +5
    
24. [REVIEW] Add no_default and adapt Series.reset_index to differentiate None for name parameter #13152
    5 - Ready to Merge Python breaking bug +4
    
25. [REVIEW] Fix is_string_dtype to adapt to pandas-2.0 changes #13141
    5 - Ready to Merge Python breaking bug python +5
    
26. [REVIEW] Drop support for pad and backfill in GroupBy #13156
    5 - Ready to Merge Python breaking improvement +4
    
27. [REVIEW] Fix datetime pytests & raise errors for timezone un-aware typecasting #13164
    5 - Ready to Merge Python breaking bug +4
    
28. [REVIEW] Fix pytests where empty column indexes are compared #13166
    5 - Ready to Merge Python bug non-breaking +4
    
29. [REVIEW] Handle pandas warnings for pad and backfill #13168
    5 - Ready to Merge Python bug improvement +4
    
30. [REVIEW] Change default dtype for get_dummies to bool #13174
    5 - Ready to Merge Python breaking improvement +4
    
31. [REVIEW] Fix datetime64 related inconsistencies in pytests #13175
    5 - Ready to Merge Python bug non-breaking +4
    
32. [REVIEW] Update parameter ordering in DataFrame.pivot #13190
    5 - Ready to Merge Python breaking bug +4
    
33. [REVIEW] Fix DataFrame.describe pytests #13191
    5 - Ready to Merge Python bug non-breaking python +5
    
34. Index class deprecation enforcements #13204
    Python breaking improvement +3
    
35. Fix parquet pytests errors with pandas-2.0 #13216
    5 - Ready to Merge Python bug non-breaking +4
    
36. Fix kurtosis pytests to support numeric_only parameter #13217
    5 - Ready to Merge Python bug non-breaking +4
    
37. Fix ceil, floor & round pytests to support pandas-2.0+ #13218
    5 - Ready to Merge Python bug non-breaking +4
    
38. Deprecate StringIndex and use Index instead #13361
    5 - Ready to Merge Python improvement non-breaking +4
    
39. Add warning for default dtype parameter in get_dummies #13365
    5 - Ready to Merge Python improvement non-breaking +4
    
40. Fix csv reader pytest to pass date_format='mixed' #13417
    3 - Ready for Review 4 - Needs Review Python bug non-breaking +5
    
41. Enforce Groupby.__iter__ deprecation and miscellaneous pytest fixes #13423
    5 - Ready to Merge Python bug non-breaking +4
    
42. Preserve Index and grouped columns in Groupby.nth #13442
    5 - Ready to Merge Python improvement non-breaking +4
    
43. Deprecate Groupby.dtypes #13453
    5 - Ready to Merge improvement non-breaking python +4
    
44. Add get_indexer to Index and enforce get_loc deprecations #13234
    5 - Ready to Merge Python breaking improvement +4
    
45. Fix parquet pytest failures with pandas-2.0 #13474
    5 - Ready to Merge Python bug non-breaking +4
    
46. Drop special handling of min_periods for Rolling.count #13483
    5 - Ready to Merge Python breaking bug +4
    
47. Fix JSON reader pytests #13476
    5 - Ready to Merge Python bug non-breaking +4
    
48. Fix a groupby pytest related to numeric_only #13496
    5 - Ready to Merge Python bug non-breaking +4
    
49. Enable sort=True for Index.union, Index.difference and Index.intersection #13497
    5 - Ready to Merge Python improvement non-breaking +4
    
50. Enforce merge validation deprecation for unequal levels #13499
    5 - Ready to Merge Python breaking improvement +4
    
51. Fix DataFrame.mode pytest #13500
    5 - Ready to Merge Python improvement non-breaking +4
    
52. Enable writing column names with mixed dtype in parquet writer when mode.pandas_compatible=True #13505
    5 - Ready to Merge Python improvement non-breaking +4
    
53. List all notable breaking changes #13535
    improvement non-breaking +2
    
54. Fix dask_cudf pytest failures for pandas-2.0 upgrade #13548
    5 - Ready to Merge Python breaking bug +4
    
55. Upgrade pandas pinnings & make style fixes #13599
    Python conda improvement non-breaking +4
56. Deprecate is_categorical_dtype #14274
    Python bug non-breaking +3
    
57. Deprecate is_datetime64tz_dtype and is_interval_dtype #14275
    Python bug non-breaking +3
    
58. Deprecate fill_method and limit in pct_change APIs #14277
    5 - Ready to Merge Python improvement non-breaking +4
    
59. Replace PandasArray with NumpyExtensionArray #14549
    5 - Ready to Merge Python bug non-breaking +4
    
60. Fix typo in Series.value_counts #14550
    5 - Ready to Merge Python bug non-breaking +4
    
61. Fix copy creation of a ColumnAccessor #14551
    5 - Ready to Merge Python bug non-breaking +4
    
62. Fix to_pandas calls in pytests #14552
    5 - Ready to Merge Python bug non-breaking +4
    
63. Enforce Index.to_frame deprecations #14553
    5 - Ready to Merge Python improvement non-breaking +4
    
64. Filter deprecation warning in ffill and bfill APIs #14554
    5 - Ready to Merge Python non-breaking +3
    
65. Add missing is_categorical_dtype to cudf.api.types namespace #14555
    5 - Ready to Merge Python bug non-breaking +4
    
66. Fix result name in Index.difference #14556
    5 - Ready to Merge Python bug non-breaking +4
    
67. Deprecate DataFrame.applymap and use map instead #14579
    5 - Ready to Merge Python improvement non-breaking +4
    
68. Deprecate first and last #14583
    5 - Ready to Merge Python improvement non-breaking +4
    
69. Change is_.._dtype deprecations to DeprecationWarning instead of FutureWarning #14617
    5 - Ready to Merge Python improvement non-breaking +4
    
70. Deprecate reading literal string in cudf.read_json #14619
    5 - Ready to Merge Python improvement non-breaking +4
    
71. Fix DataFrame.sort_index when a index is a MultiIndex #14621
    5 - Ready to Merge Python bug non-breaking +4
    
72. Fix docstring in CategoricalDtype #14622
    5 - Ready to Merge Python bug non-breaking +4
    
73. Preserve column ordering in DataFrame.stack #14626
    5 - Ready to Merge Python bug non-breaking +4
    
74. Version dataframe.mode pytest #14650
    5 - Ready to Merge Python improvement non-breaking +4
    
75. Filter ufunc related warnings in pytests #14652
    5 - Ready to Merge Python improvement non-breaking +4
    
76. Deprecate positional access for label based indexes in Series.__getitem__ #14654
    5 - Ready to Merge Python improvement non-breaking +4
    
77. Fix column parameter handling in read_orc #14666
    5 - Ready to Merge Python bug non-breaking +4
    
78. Deprecate method in interpolate and calculation on object dtype #14667
    5 - Ready to Merge Python improvement non-breaking +4
    
79. Deprecate setting of incompatible dtypes to an existing column #14668
    5 - Ready to Merge Python improvement non-breaking +4
    
80. Add more validation to MultiIndex.to_frame #14671
    5 - Ready to Merge Python bug non-breaking +4
    
81. Deprecate ignoring empty objects in concat #14672
    5 - Ready to Merge improvement non-breaking +3
    
82. Fix datetime related assertions and warnings in pytests #14673
    5 - Ready to Merge improvement non-breaking +3
    
83. Fix pytest condition to include more warning scenarios #14680
    5 - Ready to Merge bug non-breaking +3
    
84. Sort Index.difference & union results for early exit scenarios #14681
    5 - Ready to Merge Python bug non-breaking +4
    
85. Handle missing warning assertions for concat pytests #14682
    5 - Ready to Merge Python improvement non-breaking +4
    
86. Fix a typo error in where pytest #14683
    5 - Ready to Merge Python bug non-breaking +4
    
87. Change empty column dtype to string from float64 #14691
    5 - Ready to Merge Python breaking improvement +4
    
88. Preserve empty index types in parquet reader #14818
    5 - Ready to Merge Python bug non-breaking +4
    
89. Fix Dataframe.agg to not return incorrect dtypes #14851
    5 - Ready to Merge Python breaking bug +4
    
90. Catch warnings in reduction pytest #14852
    5 - Ready to Merge Python bug improvement +4
    
91. Catch groupby jit apply warnings #14858
    5 - Ready to Merge Python bug non-breaking +4
    
92. Fix all reduction pytest failures #14869
    5 - Ready to Merge Python improvement non-breaking +4
    
93. Fix empty groupby return types #14871
    Python +1
94. Support kurt/skew(axis=None) for multi columns/low row count #14874
    Python bug non-breaking +3
95. Preserve columns dtype in dataframe constructor #14878
    5 - Ready to Merge Python bug non-breaking +4
    
96. Fix some pytests #14894
    5 - Ready to Merge Python improvement non-breaking +4
97. Allow any and all only for all-NA and empty string columns #14898
    5 - Ready to Merge Python bug non-breaking +4
    
98. Parquet Writer: Write non-string columns pandas-compatibility mode only #14899
    3 - Ready for Review 4 - Needs Review Python bug +4
    
99. Deprecate method in replace and warn for a specific combination of value and to_replace #14900
    3 - Ready for Review 4 - Needs Review Python bug non-breaking +5
    
100. Prevent converting strings to arrow strings in dask_cudf pytests #14914
     5 - Ready to Merge Python improvement non-breaking +4
     
101. Fix style issues in 2.0 feature branch #14918
     3 - Ready for Review 5 - Ready to Merge Python improvement non-breaking +5
     
102. Remove strict=False from xfail marks #14905
     Python improvement non-breaking +3
     
103. Change pandas version range #14919
     5 - Ready to Merge Python conda improvement non-breaking +5
     
104. Fix custreamz pytests to test on float64 types #14925
     5 - Ready to Merge Python bug non-breaking +4
     
105. Allow hash_array to be findable in pandas 2.0; add workaround for test_resample for cudf.pandas #14908
     Python bug non-breaking +3
106. Remove pandas 1.5 checks #14928
     Python improvement non-breaking +3

[mroeschke]

Side note, would it be worth for cudf to start testing pandas nightly packages?

pip install --pre --extra-index https://pypi.anaconda.org/scipy-wheels-nightly/simple pandas

[galipremsagar]

Pandas 2.1.1 recently released that is causing various pytest failures:

= 24297 failed, 5115 passed, 1613 skipped, 288 xfailed, 480 errors in 1980.46s (0:33:00) =

2.0.2 for comparison:

= 181 failed, 99989 passed, 2069 skipped, 784 xfailed, 308 xpassed in 538.13s (0:08:58) =

cudf_pytest_log.zip

Most of the failures are around(list is WIP):

• is_datetime64tz_dtype is deprecated
• is_categorical_dtype is deprecated
• Looks like setitem with mismatching dtype will now throw/warn
• ...