feat: new text embedding for sparse vector #466
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The failed CI is due to an upstream uncompatiablity:
|
cutecutecat
force-pushed
the
svec-new-text-rep
branch
3 times, most recently
from
2089054
to
0e0c58b
Compare
|
The reason why CI fails is that |
usamoi
reviewed
Apr 18, 2024
src/datatype/text_svecf32.rs
Outdated
| if *x != F32::zero() { | ||
| match need_splitter { | ||
| true => { | ||
| buffer.push_str(format!("{}:{}", i + 1, x).as_str()); |
There was a problem hiding this comment.
I feel not good about indexing from 1. It's not consistent with subscripting.
cutecutecat
force-pushed
the
svec-new-text-rep
branch
4 times, most recently
from
69e527e
to
2823e00
Compare
|
Let's hold this PR for now, due to the conflict between 1-based array and 0-based array |
Signed-off-by: cutecutecat <junyuchen@tensorchord.ai>
Signed-off-by: usamoi <usamoi@outlook.com>
VoVAllen
approved these changes
May 29, 2024
|
This is used to support bm25 extension. It can produce string instead of depending on pgvecto.rs/pgvector. cc @cutecutecat |
usamoi
reviewed
May 30, 2024
| return Err(ParseVectorError::BadParentheses { character: '{' }); | ||
| }; | ||
| let mut token: ArrayVec<u8, 48> = ArrayVec::new(); | ||
| let mut capacity = reserve; |
There was a problem hiding this comment.
It reserves too large since vector is sparse.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Part of #459
proc_macro_byte_characterfrom upstreamReminder
The index is from 1 instead of 0 at pgvector