Merge branch 'langchain-ai:master' into GraphSparqlQAChainGraphDBFix

.github/CONTRIBUTING.md

@@ -3,43 +3,4 @@
 Hi there! Thank you for even being interested in contributing to LangChain.
 As an open-source project in a rapidly developing field, we are extremely open to contributions, whether they involve new features, improved infrastructure, better documentation, or bug fixes.
 
-To learn about how to contribute, please follow the [guides here](https://python.langchain.com/docs/contributing/)
-
-## 🗺️ Guidelines
-
-### 👩‍💻 Ways to contribute
-
-There are many ways to contribute to LangChain. Here are some common ways people contribute:
-
-- [**Documentation**](https://python.langchain.com/docs/contributing/documentation): Help improve our docs, including this one!
-- [**Code**](https://python.langchain.com/docs/contributing/code): Help us write code, fix bugs, or improve our infrastructure.
-- [**Integrations**](https://python.langchain.com/docs/contributing/integrations): Help us integrate with your favorite vendors and tools.
-
-### 🚩GitHub Issues
-
-Our [issues](https://github.com/langchain-ai/langchain/issues) page is kept up to date with bugs, improvements, and feature requests.
-
-There is a taxonomy of labels to help with sorting and discovery of issues of interest. Please use these to help organize issues.
-
-If you start working on an issue, please assign it to yourself.
-
-If you are adding an issue, please try to keep it focused on a single, modular bug/improvement/feature.
-If two issues are related, or blocking, please link them rather than combining them.
-
-We will try to keep these issues as up-to-date as possible, though
-with the rapid rate of development in this field some may get out of date.
-If you notice this happening, please let us know.
-
-### 🙋Getting Help
-
-Our goal is to have the simplest developer setup possible. Should you experience any difficulty getting setup, please
-contact a maintainer! Not only do we want to help get you unblocked, but we also want to make sure that the process is
-smooth for future contributors.
-
-In a similar vein, we do enforce certain linting, formatting, and documentation standards in the codebase.
-If you are finding these difficult (or even just annoying) to work with, feel free to contact a maintainer for help -
-we do not want these to get in the way of getting good code into the codebase.
-
-### Contributor Documentation
-
-To learn about how to contribute, please follow the [guides here](https://python.langchain.com/docs/contributing/)
+To learn how to contribute to LangChain, please follow the [contribution guide here](https://python.langchain.com/docs/contributing/).

.github/ISSUE_TEMPLATE/documentation.yml

@@ -4,13 +4,45 @@ title: "DOC: <Please write a comprehensive title after the 'DOC: ' prefix>"
 labels: [03 - Documentation]
 
 body:
+- type: markdown
+  attributes:
+    value: >
+      Thank you for taking the time to report an issue in the documentation.
+      
+      Only report issues with documentation here, explain if there are
+      any missing topics or if you found a mistake in the documentation.
+      
+      Do **NOT** use this to ask usage questions or reporting issues with your code.
+      
+      If you have usage questions or need help solving some problem, 
+      please use [GitHub Discussions](https://github.com/langchain-ai/langchain/discussions).
+      
+      If you're in the wrong place, here are some helpful links to find a better
+      place to ask your question:
+      
+      [LangChain documentation with the integrated search](https://python.langchain.com/docs/get_started/introduction),
+      [API Reference](https://api.python.langchain.com/en/stable/),
+      [GitHub search](https://github.com/langchain-ai/langchain),
+      [LangChain Github Discussions](https://github.com/langchain-ai/langchain/discussions),
+      [LangChain Github Issues](https://github.com/langchain-ai/langchain/issues?q=is%3Aissue),
+      [LangChain ChatBot](https://chat.langchain.com/)
+- type: checkboxes
+  id: checks
+  attributes:
+    label: Checklist
+    description: Please confirm and check all the following options.
+    options:
+      - label: I added a very descriptive title to this issue.
+        required: true
+      - label: I included a link to the documentation page I am referring to (if applicable).
+        required: true
 - type: textarea
   attributes: 
     label: "Issue with current documentation:"
     description: >
       Please make sure to leave a reference to the document/code you're
-      referring to.
-
+      referring to. Feel free to include names of classes, functions, methods
+      or concepts you'd like to see documented more.
 - type: textarea
   attributes:
     label: "Idea or request for content:"

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,20 +1,24 @@
-<!-- Thank you for contributing to LangChain!
+Thank you for contributing to LangChain!
 
-Please title your PR "<package>: <description>", where <package> is whichever of langchain, community, core, experimental, etc. is being modified.
+Checklist:
 
-Replace this entire comment with:
-  - **Description:** a description of the change, 
-  - **Issue:** the issue # it fixes if applicable,
-  - **Dependencies:** any dependencies required for this change,
-  - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out!
-
-Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` from the root of the package you've modified to check this locally.
-
-See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/
-
-If you're adding a new integration, please include:
+- [ ] PR title: Please title your PR "package: description", where "package" is whichever of langchain, community, core, experimental, etc. is being modified. Use "docs: ..." for purely docs changes, "templates: ..." for template changes, "infra: ..." for CI changes.
+  - Example: "community: add foobar LLM"
+- [ ] PR message: **Delete this entire template message** and replace it with the following bulleted list
+    - **Description:** a description of the change
+    - **Issue:** the issue # it fixes, if applicable
+    - **Dependencies:** any dependencies required for this change
+    - **Twitter handle:** if your PR gets announced, and you'd like a mention, we'll gladly shout you out!
+- [ ] Pass lint and test: Run `make format`, `make lint` and `make test` from the root of the package(s) you've modified to check that you're passing lint and testing. See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/
+- [ ] Add tests and docs: If you're adding a new integration, please include
   1. a test for the integration, preferably unit tests that do not rely on network access,
   2. an example notebook showing its use. It lives in `docs/docs/integrations` directory.
 
-If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17.
- -->
+Additional guidelines:
+- Make sure optional dependencies are imported within a function.
+- Please do not add dependencies to pyproject.toml files (even optional ones) unless they are required for unit tests.
+- Most PRs should not touch more than one package.
+- Changes should be backwards compatible.
+- If you are adding something to community, do not re-import it in langchain.
+
+If no one reviews your PR within a few days, please @-mention one of baskaryan, efriis, eyurtsev, hwchase17.

.github/scripts/check_diff.py

@@ -47,4 +47,4 @@
         else:
             pass
     json_output = json.dumps(list(dirs_to_run))
-    print(f"dirs-to-run={json_output}")
+    print(f"dirs-to-run={json_output}")  # noqa: T201

.github/scripts/get_min_versions.py

@@ -62,4 +62,6 @@ def get_min_version_from_toml(toml_path: str):
 # Call the function to get the minimum versions
 min_versions = get_min_version_from_toml(toml_file)
 
-print(" ".join([f"{lib}=={version}" for lib, version in min_versions.items()]))
+print(
+    " ".join([f"{lib}=={version}" for lib, version in min_versions.items()])
+)  # noqa: T201

.github/workflows/_integration_test.yml

@@ -52,6 +52,7 @@ jobs:
       - name: Run integration tests
         shell: bash
         env:
+          AI21_API_KEY: ${{ secrets.AI21_API_KEY }}
           GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
           ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
           MISTRAL_API_KEY: ${{ secrets.MISTRAL_API_KEY }}
@@ -62,8 +63,13 @@ jobs:
           GOOGLE_CSE_ID: ${{ secrets.GOOGLE_CSE_ID }}
           EXA_API_KEY: ${{ secrets.EXA_API_KEY }}
           NOMIC_API_KEY: ${{ secrets.NOMIC_API_KEY }}
+          WATSONX_APIKEY: ${{ secrets.WATSONX_APIKEY }}
+          WATSONX_PROJECT_ID: ${{ secrets.WATSONX_PROJECT_ID }}
           PINECONE_API_KEY: ${{ secrets.PINECONE_API_KEY }}
           PINECONE_ENVIRONMENT: ${{ secrets.PINECONE_ENVIRONMENT }}
+          ASTRA_DB_API_ENDPOINT: ${{ secrets.ASTRA_DB_API_ENDPOINT }}
+          ASTRA_DB_APPLICATION_TOKEN: ${{ secrets.ASTRA_DB_APPLICATION_TOKEN }}
+          ASTRA_DB_KEYSPACE: ${{ secrets.ASTRA_DB_KEYSPACE }}
         run: |
           make integration_tests
 

.github/workflows/_release.yml

@@ -166,18 +166,30 @@ jobs:
       - name: Run integration tests
         if: ${{ startsWith(inputs.working-directory, 'libs/partners/') }}
         env:
+          AI21_API_KEY: ${{ secrets.AI21_API_KEY }}
           GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
           ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
           MISTRAL_API_KEY: ${{ secrets.MISTRAL_API_KEY }}
           TOGETHER_API_KEY: ${{ secrets.TOGETHER_API_KEY }}
           OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+          AZURE_OPENAI_API_VERSION: ${{ secrets.AZURE_OPENAI_API_VERSION }}
+          AZURE_OPENAI_API_BASE: ${{ secrets.AZURE_OPENAI_API_BASE }}
+          AZURE_OPENAI_API_KEY: ${{ secrets.AZURE_OPENAI_API_KEY }}
+          AZURE_OPENAI_CHAT_DEPLOYMENT_NAME: ${{ secrets.AZURE_OPENAI_CHAT_DEPLOYMENT_NAME }}
+          AZURE_OPENAI_LLM_DEPLOYMENT_NAME: ${{ secrets.AZURE_OPENAI_LLM_DEPLOYMENT_NAME }}
+          AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT_NAME: ${{ secrets.AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT_NAME }}
           NVIDIA_API_KEY: ${{ secrets.NVIDIA_API_KEY }}
           GOOGLE_SEARCH_API_KEY: ${{ secrets.GOOGLE_SEARCH_API_KEY }}
           GOOGLE_CSE_ID: ${{ secrets.GOOGLE_CSE_ID }}
           EXA_API_KEY: ${{ secrets.EXA_API_KEY }}
           NOMIC_API_KEY: ${{ secrets.NOMIC_API_KEY }}
+          WATSONX_APIKEY: ${{ secrets.WATSONX_APIKEY }}
+          WATSONX_PROJECT_ID: ${{ secrets.WATSONX_PROJECT_ID }}
           PINECONE_API_KEY: ${{ secrets.PINECONE_API_KEY }}
           PINECONE_ENVIRONMENT: ${{ secrets.PINECONE_ENVIRONMENT }}
+          ASTRA_DB_API_ENDPOINT: ${{ secrets.ASTRA_DB_API_ENDPOINT }}
+          ASTRA_DB_APPLICATION_TOKEN: ${{ secrets.ASTRA_DB_APPLICATION_TOKEN }}
+          ASTRA_DB_KEYSPACE: ${{ secrets.ASTRA_DB_KEYSPACE }}
         run: make integration_tests
         working-directory: ${{ inputs.working-directory }}
 

.github/workflows/api_doc_build.yml

@@ -0,0 +1,52 @@
+name: API docs build
+
+on:
+  workflow_dispatch:
+  schedule:
+    - cron:  '0 13 * * *'
+env:
+  POETRY_VERSION: "1.7.1"
+  PYTHON_VERSION: "3.10"
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: bagatur/api_docs_build
+
+      - name: Set Git config
+        run: |
+          git config --local user.email "actions@github.com"
+          git config --local user.name "Github Actions"
+
+      - name: Merge master
+        run: | 
+          git fetch origin master
+          git merge origin/master -m "Merge master" --allow-unrelated-histories -X theirs
+
+      - name: Set up Python ${{ env.PYTHON_VERSION }} + Poetry ${{ env.POETRY_VERSION }}
+        uses: "./.github/actions/poetry_setup"
+        with:
+          python-version: ${{ env.PYTHON_VERSION }}
+          poetry-version: ${{ env.POETRY_VERSION }}
+          cache-key: api-docs
+
+      - name: Install dependencies
+        run: |
+          poetry run python -m pip install --upgrade --no-cache-dir pip setuptools
+          poetry run python -m pip install --upgrade --no-cache-dir sphinx readthedocs-sphinx-ext
+          poetry run python -m pip install ./libs/partners/*
+          poetry run python -m pip install --exists-action=w --no-cache-dir -r docs/api_reference/requirements.txt
+
+      - name: Build docs
+        run: |
+          poetry run python -m pip install --upgrade --no-cache-dir pip setuptools
+          poetry run python docs/api_reference/create_api_rst.py
+          poetry run python -m sphinx -T -E -b html -d _build/doctrees -c docs/api_reference docs/api_reference api_reference_build/html -j auto
+
+      # https://github.com/marketplace/actions/add-commit
+      - uses: EndBug/add-and-commit@v9
+        with:
+          message: 'Update API docs build'

.github/workflows/codespell.yml

@@ -34,3 +34,4 @@ jobs:
         with:
           skip: guide_imports.json
           ignore_words_list: ${{ steps.extract_ignore_words.outputs.ignore_words_list }}
+          exclude_file: libs/community/langchain_community/llms/yuan2.py

.github/workflows/extract_ignored_words_list.py

@@ -7,4 +7,4 @@
     pyproject_toml.get("tool", {}).get("codespell", {}).get("ignore-words-list")
 )
 
-print(f"::set-output name=ignore_words_list::{ignore_words_list}")
+print(f"::set-output name=ignore_words_list::{ignore_words_list}")  # noqa: T201

.readthedocs.yaml

@@ -13,15 +13,8 @@ build:
   tools:
     python: "3.11"
   commands:
-      - python -m virtualenv $READTHEDOCS_VIRTUALENV_PATH
-      - python -m pip install --upgrade --no-cache-dir pip setuptools
-      - python -m pip install --upgrade --no-cache-dir sphinx readthedocs-sphinx-ext
-      - python -m pip install ./libs/partners/*
-      - python -m pip install --exists-action=w --no-cache-dir -r docs/api_reference/requirements.txt
-      - python docs/api_reference/create_api_rst.py
-      - cat docs/api_reference/conf.py
-      - python -m sphinx -T -E -b html -d _build/doctrees -c docs/api_reference docs/api_reference $READTHEDOCS_OUTPUT/html -j auto
-
+    - mkdir -p $READTHEDOCS_OUTPUT
+    - cp -r api_reference_build/* $READTHEDOCS_OUTPUT
 # Build documentation in the docs/ directory with Sphinx
 sphinx:
    configuration: docs/api_reference/conf.py

Makefile

@@ -15,7 +15,12 @@ docs_build:
 	docs/.local_build.sh
 
 docs_clean:
-	rm -r _dist
+	@if [ -d _dist ]; then \
+			rm -r _dist; \
+			echo "Directory _dist has been cleaned."; \
+	else \
+			echo "Nothing to clean."; \
+	fi
 
 docs_linkcheck:
 	poetry run linkchecker _dist/docs/ --ignore-url node_modules

README.md

@@ -18,7 +18,7 @@ Looking for the JS/TS library? Check out [LangChain.js](https://github.com/langc
 
 To help you ship LangChain apps to production faster, check out [LangSmith](https://smith.langchain.com). 
 [LangSmith](https://smith.langchain.com) is a unified developer platform for building, testing, and monitoring LLM applications. 
-Fill out [this form](https://airtable.com/appwQzlErAS2qiP0L/shrGtGaVBVAz7NcV2) to get off the waitlist or speak with our sales team.
+Fill out [this form](https://www.langchain.com/contact-sales) to speak with our sales team.
 
 ## Quick Install