-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(parquet): add byte buffer when disable buffered stream #302
Conversation
parquet/file/page_reader.go
Outdated
rd := utils.NewBufferedReader( | ||
io.NewSectionReader(p.r.Outer(), p.dictOffset-p.baseOffset, p.dataOffset-p.baseOffset), | ||
p.r.BufferSize()) | ||
int(parquet.DefaultBufSize)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This would remove our respecting of the BufferSize property in the reader properties, we shouldn't be removing the BufferSize()
method
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So how about change it to:
readBufSize := min(p.dataOffset-p.baseOffset, p.r.BufferSize())
rd := utils.NewBufferedReader(
io.NewSectionReader(p.r.Outer(), p.dictOffset-p.baseOffset, p.dataOffset-p.baseOffset),
readBufSize)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
that works
Can I manually rerun the CI? |
Done! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
Rationale for this change
When we create a reader with
BufferedStreamEnabled = false
, we have to first create adata
slice.arrow-go/parquet/reader_properties.go
Lines 76 to 91 in efecae3
And we will create another slice with same size in the
NewBufferedReader
again, which is redundant.arrow-go/internal/utils/buf_reader.go
Lines 45 to 51 in efecae3
What changes are included in this PR?
byteReader
, which is a wrapper ofbytes.NewReader
and implementBufferedReader
interface.BufferSize()
method fromBufferedReader
interfaceAre these changes tested?
BufferedReader
created frombytes.NewReader
in tests are all replaced by new functionNewByteReader
.Are there any user-facing changes?
No.