community: SQLStrStore/SQLDocStore provide an easy SQL alternative to InMemoryStore to persist data remotely in a SQL storage

[gcheron]

Description:

• Implement SQLStrStore and SQLDocStore classes that inherits from BaseStore to allow to persist data remotely on a SQL server.
• SQL is widely used and sometimes we do not want to install a caching solution like Redis.
• Multiple issues/comments complain that there is no easy remote and persistent solution that are not in memory (users want to replace InMemoryStore), e.g., DOC: ParentDocumentRetriever without InMemoryStore #14267, Issue: What docstore to use in ChromaDB that isn't in memory? #15633, ParentDocumentRetriever.get_relevant_documents() got empty result #14643, https://stackoverflow.com/questions/77385587/persist-parentdocumentretriever-of-langchain
• This is particularly painful when wanting to use ParentDocumentRetriever
• This implementation is particularly useful when:
  • it's expensive to construct an InMemoryDocstore/dict
  • you want to retrieve documents from remote sources
  • you just want to reuse existing objects
• This implementation integrates well with PGVector, indeed, when using PGVector, you already have a SQL instance running. SQLDocStore is a convenient way of using this instance to store documents associated to vectors. An integration example with ParentDocumentRetriever and PGVector is provided in docs/docs/integrations/stores/sql.ipynb or here.
• It persists str and Document objects but can be easily extended.

Issue:

Provide an easy SQL alternative to InMemoryStore.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
langchain	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 22, 2024 4:09pm

[hwchase17]

thanks! this looks great. lets not change anything in langchain (even imports) - other than that this looks fantastic!!

[gcheron]

thanks! this looks great. lets not change anything in langchain (even imports) - other than that this looks fantastic!!

Thanks @hwchase17 ! My last commit fixes what you underlined by removing changes in libs/langchain :)

[edmolten]

Was this tested in a ParentDocumentRetriever? I wanted to test your code:

embeddings = HuggingFaceEmbeddings(model_name=MODEL)


chunk_size = math.floor(max_seq_lenght * 0.9)
chunk_overlap = math.floor(max_seq_lenght * 0.3)

parent_splitter = RecursiveCharacterTextSplitter(chunk_size=1000)
child_splitter = RecursiveCharacterTextSplitter(
    chunk_size=chunk_size, chunk_overlap=chunk_overlap
)    

vectorstore = Pinecone.from_existing_index(index_name, embeddings)

store = SQLDocStore("postgresql://postgres:mysecretpassword@localhost:5432/postgres", "manual")

retriever = ParentDocumentRetriever(
    vectorstore=vectorstore,
    docstore=store,
    child_splitter=child_splitter,
    parent_splitter=parent_splitter,

but I'm getting this

ValidationError: 1 validation error for ParentDocumentRetriever
docstore
  instance of BaseStore expected (type=type_error.arbitrary_type; expected_arbitrary_type=BaseStore)

I'm struggling to solve this. I don't see any difference between the way you handle the generic types and what's currently implemented in InMemoryStore

[gcheron]

@edmolten

The SQLDocStore was initially developed as a replacement for the InMemoryStore to address the document persistence issue encountered with the ParentDocumentRetriever. Therefore, it is indeed definitely possible to utilize the SQLDocStore in conjunction with the ParentDocumentRetriever.

Please find a working example:

import os

from langchain.retrievers import ParentDocumentRetriever
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import Chroma
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.document_loaders import TextLoader

from storage.sql import SQLDocStore


def connection_string_from_db_params() -> str:
    """Return connection string from database parameters."""
    dbdriver = os.environ.get("TEST_SQL_DBDRIVER", "postgresql+psycopg2")
    host = os.environ.get("TEST_SQL_HOST", "localhost")
    port = int(os.environ.get("TEST_SQL_PORT", "5432"))
    database = os.environ.get("TEST_SQL_DATABASE", "postgres")
    user = os.environ.get("TEST_SQL_USER", "postgres")
    password = os.environ.get("TEST_SQL_PASSWORD", "postgres")
    return f"{dbdriver}://{user}:{password}@{host}:{port}/{database}"


CONNECTION_STRING = connection_string_from_db_params()

embeddings = HuggingFaceEmbeddings()

parent_splitter = RecursiveCharacterTextSplitter(chunk_size=1000)
child_splitter = RecursiveCharacterTextSplitter(chunk_size=200)

vectorstore = Chroma(
    collection_name="full_documents", embedding_function=HuggingFaceEmbeddings()
)
store = SQLDocStore(CONNECTION_STRING, "manual")

retriever = ParentDocumentRetriever(
    vectorstore=vectorstore,
    docstore=store,
    child_splitter=child_splitter,
    parent_splitter=parent_splitter,
)

loader = TextLoader("./state_of_the_union.txt")
documents = loader.load()

retriever.add_documents(documents)

query = "What did the president say about Ketanji Brown Jackson"
docs = retriever.get_relevant_documents(query)

print(docs[0])

Output:

page_content='In state after state, new laws have been passed, not only to suppress the vote, but to subvert entire elections. \n\nWe cannot let this happen. \n\nTonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. \n\nTonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \n\nOne of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \n\nAnd I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.' metadata={'source': './state_of_the_union.txt'}

This is running with:

Package                                  Version
---------------------------------------- ----------
aiohttp                                  3.9.1
aiosignal                                1.3.1
annotated-types                          0.6.0
anyio                                    4.2.0
asgiref                                  3.7.2
async-timeout                            4.0.3
attrs                                    23.2.0
backoff                                  2.2.1
bcrypt                                   4.1.2
build                                    1.0.3
cachetools                               5.3.2
certifi                                  2023.11.17
charset-normalizer                       3.3.2
chroma-hnswlib                           0.7.3
chromadb                                 0.4.22
click                                    8.1.7
coloredlogs                              15.0.1
dataclasses-json                         0.6.3
Deprecated                               1.2.14
exceptiongroup                           1.2.0
fastapi                                  0.109.0
filelock                                 3.13.1
flatbuffers                              23.5.26
frozenlist                               1.4.1
fsspec                                   2023.12.2
google-auth                              2.26.2
googleapis-common-protos                 1.62.0
grpcio                                   1.60.0
h11                                      0.14.0
httptools                                0.6.1
huggingface-hub                          0.20.2
humanfriendly                            10.0
idna                                     3.6
importlib-metadata                       6.11.0
importlib-resources                      6.1.1
Jinja2                                   3.1.3
joblib                                   1.3.2
jsonpatch                                1.33
jsonpointer                              2.4
kubernetes                               29.0.0
langchain                                0.1.0
langchain-community                      0.0.12
langchain-core                           0.1.10
langsmith                                0.0.80
MarkupSafe                               2.1.3
marshmallow                              3.20.2
mmh3                                     4.1.0
monotonic                                1.6
mpmath                                   1.3.0
multidict                                6.0.4
mypy-extensions                          1.0.0
networkx                                 3.2.1
nltk                                     3.8.1
numpy                                    1.26.3
oauthlib                                 3.2.2
onnxruntime                              1.16.3
opentelemetry-api                        1.22.0
opentelemetry-exporter-otlp-proto-common 1.22.0
opentelemetry-exporter-otlp-proto-grpc   1.22.0
opentelemetry-instrumentation            0.43b0
opentelemetry-instrumentation-asgi       0.43b0
opentelemetry-instrumentation-fastapi    0.43b0
opentelemetry-proto                      1.22.0
opentelemetry-sdk                        1.22.0
opentelemetry-semantic-conventions       0.43b0
opentelemetry-util-http                  0.43b0
overrides                                7.4.0
packaging                                23.2
pillow                                   10.2.0
pip                                      22.3.1
posthog                                  3.3.1
protobuf                                 4.25.2
psycopg2-binary                          2.9.9
pulsar-client                            3.4.0
pyasn1                                   0.5.1
pyasn1-modules                           0.3.0
pydantic                                 2.5.3
pydantic_core                            2.14.6
PyPika                                   0.48.9
pyproject_hooks                          1.0.0
python-dateutil                          2.8.2
python-dotenv                            1.0.0
PyYAML                                   6.0.1
regex                                    2023.12.25
requests                                 2.31.0
requests-oauthlib                        1.3.1
rsa                                      4.9
safetensors                              0.4.1
scikit-learn                             1.3.2
scipy                                    1.11.4
sentence-transformers                    2.2.2
sentencepiece                            0.1.99
setuptools                               65.5.0
six                                      1.16.0
sniffio                                  1.3.0
SQLAlchemy                               2.0.25
starlette                                0.35.1
sympy                                    1.12
tenacity                                 8.2.3
threadpoolctl                            3.2.0
tokenizers                               0.15.0
tomli                                    2.0.1
torch                                    2.1.2
torchvision                              0.16.2
tqdm                                     4.66.1
transformers                             4.36.2
typer                                    0.9.0
typing_extensions                        4.9.0
typing-inspect                           0.9.0
urllib3                                  2.1.0
uvicorn                                  0.25.0
uvloop                                   0.19.0
watchfiles                               0.21.0
websocket-client                         1.7.0
websockets                               12.0
wrapt                                    1.16.0
yarl                                     1.9.4
zipp                                     3.17.0

[edmolten]

@gcheron it was a dumb importing issue in the notebook. Works like charm!

[gcheron]

Happy to help anytime! :)

[kallebl0mquist]

Thanks for your work! I am trying to replace InMemoryStore for a multivectorretriever.

i have postgres running. But I am wondering where SQLDocStore is actually located for import. What package is storage.sql from? Are all of the packages listed in your post necessary for SQLDocStore?

from storage.sql import SQLDocStore

[gcheron]

@kallebl0mquist storage.sql is not in LangChain lib for now, we are waiting for @hwchase17 @baskaryan to merge this PR :)
But you can take the sql.py file of this PR from here for now until the PR is merged.

[kallebl0mquist]

It works! Thank You.

[gcheron]

@baskaryan @hwchase17 could you please approve again, I fixed the one error that only occurred with python3.8 TypeError: 'type' object is not subscriptable in my last commit.

[gcheron]

@baskaryan @hwchase17 I think we are good to go, what do you think?

[rob3r7]

I would suggest to use Uuid (Generic “CamelCase” Types / https://docs.sqlalchemy.org/en/20/core/type_basics.html) instead of UUID. Therefore sqlite could be used as a backend.

[pprados]

Hello @gcheron,
cc: @cbornet

II proposed a similar approach. Your implementation is better.
But I think it's important to be able to have a global transaction, to use ParentDocumentRetriever, with pgvector for example, in a single transaction!

To do this, I suggest adding a customizable make_session (for sync and async ?).

    @contextlib.contextmanager
    def _make_session(self) -> Generator[Session, None, None]:
        """Create a session and close it after use."""

        if isinstance(self.session_factory, async_sessionmaker):
            raise AssertionError("This method is not supported for async engines.")

        session = scoped_session(self.session_factory)()
        try:
            yield session
        finally:
            session.commit()

I find ParentDocumentRetriever inconsistent (adding documents to a retriever is astonishing). I suggest an alternative, rag-vectorstore, which would allow you to use the index() api with your implementation, to manage all the tables in one transaction! Currently, there's no solution with langchain to guarantee data import in all situations (no two-phase commit with vectorstores).

A PR is proposed, but others need to be validated first.

[pprados]

@gcheron
The usage with langchain is to propose an engine parameter to manipulate SQL.
Is the case for :

• langchain_community.cache.SQLAlchemyCache
• langchain_community.cache.SQLAlchemyMd5Cache
• langchain_community.utilities.SQLDatabase

Perhaps you can add this parameter so that you can share an engine instance.

[efriis]

Hey team - sorry about this! Actually going to revert this PR before too many people rely on this implementation. There are a few issues with it.

• breaks imports from langchain.storage for anyone using sqlalchemy v1 (which we support) -> didn't fail linting because lint assumes sqlalchemy v2
• doesn't have unit tests
• relies on a specific schema without facilities for migrating to/off it in the future

For anyone wanting to continue using this, I would recommend copy-pasting the implementation from this PR into your code and maintaining yourself.

Happy to discuss a future PR that brings this back in some form. Maybe in experimental?

[kallebl0mquist]

Oh, no. But it works so well. I had no success using LocalFileStorage. What would you suggest as the best alternative for docstorage?

[gcheron]

Hi @efriis 👋
It has some unit tests, what do you mean by "doesn't have unit tests"?
I did not know about the need for supporting SQL alchemy v1 🤔
Happy to chat for adding modifications (maybe @pprados as well?)

[pprados]

I have another proposition here, the PR16278

[gcheron]

@pprados I do not think it better addresses @efriis concerns.
@efriis how could we discuss in order you tell us what you expect:

• which additional unit tests do you expect?
• how to address migrations? for example, does PGvector (which is also SQL based) have some migration protocol?

[andreshernandez]

Hi! Is there a plan to address the concerns that resulted in this pr getting reverted? I liked this option as I do indexing and q&a in separate processes that don't share memory (I believe many people do). Hope it gets picked up :)

[gcheron]

Oh, no. But it works so well. I had no success using LocalFileStorage. What would you suggest as the best alternative for docstorage?

Hi! Is there a plan to address the concerns that resulted in this pr getting reverted? I liked this option as I do indexing and q&a in separate processes that don't share memory (I believe many people do). Hope it gets picked up :)
...

Users seem to want such solution back to the lib, @efriis @baskaryan @hwchase17 any suggestion for moving forward?

I was asking for this:

@pprados I do not think it better addresses @efriis concerns. @efriis how could we discuss in order you tell us what you expect:

• which additional unit tests do you expect?
• how to address migrations? for example, does PGvector (which is also SQL based) have some migration protocol?

[thompsonp17]

I was really glad trying to make a from langchain_community.storage import SQLStrStore after I got a in disk docstore to crash irreversibly. And then I found it is not available yet... This SQL docstore is really important to become available ASAP. Thanks for the effort, @gcheron!