-
Notifications
You must be signed in to change notification settings - Fork 478
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SNOW-920892: v3.3.0 nanoarrow based connector #1750
Conversation
…ssue with the c object life cycle
…, but there is memory leak
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
test/**/*.py
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
setup looks good :)
Can we please add an environment variable that would prevent building the extension even if the required dependencies are available? |
hey @ofek, can you elaborate on your requirements here, is this related to this specific PR or to the general connector build steps |
This is a general request not specific to this PR but it would be easy to add since we are already modifying the file. We have a need to skip compiling under certain circumstances. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
converters
src/snowflake/connector/nanoarrow_cpp/ArrowIterator/CArrowChunkIterator.cpp
Show resolved
Hide resolved
src/snowflake/connector/nanoarrow_cpp/ArrowIterator/CArrowChunkIterator.cpp
Show resolved
Hide resolved
src/snowflake/connector/nanoarrow_cpp/ArrowIterator/CArrowTableIterator.cpp
Show resolved
Hide resolved
src/snowflake/connector/nanoarrow_cpp/ArrowIterator/DecimalConverter.cpp
Show resolved
Hide resolved
@ofek thanks for the feedback, I think this is a good one. however, this PR is dedicated only for the nanoarrow changes. can you open a separate github issue for the request? we'd like to address your issue separately |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Check pointing my review.
@sfc-gh-bwarsaw , your understanding is correct. I think this is a good feedback, I will make the change. |
I put up a draft PR for feedback improvement: https://github.com/snowflakedb/snowflake-connector-python/pull/1758/files |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a welcome addition and will make packaging simpler once you can drop the bundled arrow libs. I will give this a test run in conda-forge in the next days and report back whether I see any build issues.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed all .py and .pyx files
|
||
ext.sources += [ | ||
os.path.join( | ||
NANOARROW_ARROW_ITERATOR_SRC_DIR, "CArrowIterator.cpp" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I feel like we could introduce a helper for all this repeating code, maybe in a follow-up PR
plt.title("per iteration execution time") | ||
plt.show() | ||
plt.plot( | ||
[item[0] for item in memory_records], [item[1] for item in memory_records] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could use map and operator.itemgetter here instead
src/snowflake/connector/nanoarrow_cpp/ArrowIterator/CArrowChunkIterator.cpp
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🚢 🚢
Please answer these questions before submitting your pull requests. Thanks!
SNOW-920892: GA release nanoarow based connector 3.3.0
the recommended way to review the PR:
src/snowflake/connector
src/snowflake/connector/nanoarrow_cpp/ArrowIterator/nanoarrow_arrow_iterator.pyx
src/snowflake/connector/nanoarrow_cpp/flatcc.cc
src/snowflake/connector/nanoarrow_cpp/flatcc
src/snowflake/connector/nanoarrow_cpp/nanoarrow*.*
, except forsrc/snowflake/connector/nanoarrow_cpp/ArrowIterator/nanoarrow_arrow_iterator.pyx
Fill out the following pre-review checklist:
Please describe how your code solves the related issue.
Please write a short description of how your code change solves the related issue.