Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix an issue with multiple short list rowgroups using the Parquet chunked reader. #15342

Merged
merged 7 commits into from
Mar 20, 2024

Commits on Mar 19, 2024

  1. Fixed two list chunked reading related issues:

    First, row groups of lists being loaded together were not getting their end row counts computed correctly on a per-rowgroup basis.
    The main consequence there was that we could potentially have generated splits that were larger than we would have liked. But downstream
    from this was a second bug where we were generating incorrect page indices to be decoded, causing corruption in the decode kernels.
    nvdbaranec committed Mar 19, 2024
    Configuration menu
    Copy the full SHA
    739238d View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d0cb768 View commit details
    Browse the repository at this point in the history

Commits on Mar 20, 2024

  1. Configuration menu
    Copy the full SHA
    7d7c9d5 View commit details
    Browse the repository at this point in the history
  2. Fix indexing issue.

    nvdbaranec committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    23934c1 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    d9fbba1 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    ec63720 View commit details
    Browse the repository at this point in the history
  5. Comment clarification. Simplified some logic in find_next_split to av…

    …oid unnecessarily looping in some cases.
    nvdbaranec committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    9a8a9ba View commit details
    Browse the repository at this point in the history