Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Python] Improve PyArrow support for Substrait ExtendedExpressions #41692

Open
amol- opened this issue May 16, 2024 · 0 comments
Open

[Python] Improve PyArrow support for Substrait ExtendedExpressions #41692

amol- opened this issue May 16, 2024 · 0 comments

Comments

@amol-
Copy link
Contributor

amol- commented May 16, 2024

Describe the enhancement requested

Substrait ExtendedExpressions can be used with PyArrow by converting them to compute.Expression.
That way they can be used in the context of projections and filters when interacting with a Dataset or Table.

Some pieces are still missing for those to be used in practice and the user experience is generally too complex to be convenient. This issue is meant to track work that can be done to improve the Substrait experience in PyArrow

  • Allow accepting Substrait Message objects directly instead of bytes(currently pc.Expression.from_substrait(projection.SerializeToString()) dance is required which is not very convenient)
  • Allow accepting substrait messages directly where a pc.Expression is accepted instead of having to build the expression from the message
  • Have a way to encode PyArrow schemas to Substrait NamedStruct
  • Accept projections as a single substrait ExtendedExpression instead of having to build multiple different expressions for each projected column.

Component(s)

Python

@amol- amol- changed the title Improve PyArrow support for Substrait ExtendedExpressions [Python] Improve PyArrow support for Substrait ExtendedExpressions May 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant