-
Notifications
You must be signed in to change notification settings - Fork 13.5k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
text-splitters[minor]: Adding a new section aware splitter to langcha…
…in (#16526) - **Description:** the layout of html pages can be variant based on the bootstrap framework or the styles of the pages. So we need to have a splitter to transform the html tags to a proper layout and then split the html content based on the provided list of tags to determine its html sections. We are using BS4 library along with xslt structure to split the html content using an section aware approach. - **Dependencies:** No new dependencies - **Twitter handle:** @m_setayesh Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` from the root of the package you've modified to check this locally. See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/ If you're adding a new integration, please include: 1. a test for the integration, preferably unit tests that do not rely on network access, 2. an example notebook showing its use. It lives in `docs/docs/integrations` directory. If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17. --> --------- Co-authored-by: Bagatur <baskaryan@gmail.com>
- Loading branch information
Showing
8 changed files
with
531 additions
and
9 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
173 changes: 173 additions & 0 deletions
173
docs/docs/modules/data_connection/document_transformers/HTML_section_aware_splitter.ipynb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,173 @@ | ||
{ | ||
"cells": [ | ||
{ | ||
"cell_type": "markdown", | ||
"id": "c95fcd15cd52c944", | ||
"metadata": { | ||
"collapsed": false, | ||
"jupyter": { | ||
"outputs_hidden": false | ||
} | ||
}, | ||
"source": [ | ||
"# Split by HTML section\n", | ||
"## Description and motivation\n", | ||
"Similar in concept to the [HTMLHeaderTextSplitter](/docs/modules/data_connection/document_transformers/HTML_header_metadata), the `HTMLSectionSplitter` is a \"structure-aware\" chunker that splits text at the element level and adds metadata for each header \"relevant\" to any given chunk. It can return chunks element by element or combine elements with the same metadata, with the objectives of (a) keeping related text grouped (more or less) semantically and (b) preserving context-rich information encoded in document structures. It can be used with other text splitters as part of a chunking pipeline. Internally, it uses the `RecursiveCharacterTextSplitter` when the section size is larger than the chunk size. It also considers the font size of the text to determine whether it is a section or not based on the determined font size threshold. Use `xslt_path` to provide an absolute path to transform the HTML so that it can detect sections based on provided tags. The default is to use the `converting_to_header.xslt` file in the `data_connection/document_transformers` directory. This is for converting the html to a format/layout that is easier to detect sections. For example, `span` based on their font size can be converted to header tags to be detected as a section.\n", | ||
"\n", | ||
"## Usage examples\n", | ||
"#### 1) With an HTML string:" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"id": "initial_id", | ||
"metadata": { | ||
"ExecuteTime": { | ||
"end_time": "2023-10-02T18:57:49.208965400Z", | ||
"start_time": "2023-10-02T18:57:48.899756Z" | ||
}, | ||
"collapsed": false, | ||
"jupyter": { | ||
"outputs_hidden": false | ||
} | ||
}, | ||
"outputs": [], | ||
"source": [ | ||
"from langchain_text_splitters import HTMLSectionSplitter\n", | ||
"\n", | ||
"html_string = \"\"\"\n", | ||
" <!DOCTYPE html>\n", | ||
" <html>\n", | ||
" <body>\n", | ||
" <div>\n", | ||
" <h1>Foo</h1>\n", | ||
" <p>Some intro text about Foo.</p>\n", | ||
" <div>\n", | ||
" <h2>Bar main section</h2>\n", | ||
" <p>Some intro text about Bar.</p>\n", | ||
" <h3>Bar subsection 1</h3>\n", | ||
" <p>Some text about the first subtopic of Bar.</p>\n", | ||
" <h3>Bar subsection 2</h3>\n", | ||
" <p>Some text about the second subtopic of Bar.</p>\n", | ||
" </div>\n", | ||
" <div>\n", | ||
" <h2>Baz</h2>\n", | ||
" <p>Some text about Baz</p>\n", | ||
" </div>\n", | ||
" <br>\n", | ||
" <p>Some concluding text about Foo</p>\n", | ||
" </div>\n", | ||
" </body>\n", | ||
" </html>\n", | ||
"\"\"\"\n", | ||
"\n", | ||
"headers_to_split_on = [(\"h1\", \"Header 1\"), (\"h2\", \"Header 2\")]\n", | ||
"\n", | ||
"html_splitter = HTMLSectionSplitter(headers_to_split_on=headers_to_split_on)\n", | ||
"html_header_splits = html_splitter.split_text(html_string)\n", | ||
"html_header_splits" | ||
] | ||
}, | ||
{ | ||
"cell_type": "markdown", | ||
"id": "e29b4aade2a0070c", | ||
"metadata": { | ||
"collapsed": false, | ||
"jupyter": { | ||
"outputs_hidden": false | ||
} | ||
}, | ||
"source": [ | ||
"#### 2) Pipelined to another splitter, with html loaded from a html string content:" | ||
] | ||
}, | ||
{ | ||
"cell_type": "code", | ||
"execution_count": null, | ||
"id": "6ada8ea093ea0475", | ||
"metadata": { | ||
"ExecuteTime": { | ||
"end_time": "2023-10-02T18:57:51.016141300Z", | ||
"start_time": "2023-10-02T18:57:50.647495400Z" | ||
}, | ||
"collapsed": false, | ||
"jupyter": { | ||
"outputs_hidden": false | ||
} | ||
}, | ||
"outputs": [], | ||
"source": [ | ||
"from langchain.text_splitter import RecursiveCharacterTextSplitter\n", | ||
"\n", | ||
"html_string = \"\"\"\n", | ||
" <!DOCTYPE html>\n", | ||
" <html>\n", | ||
" <body>\n", | ||
" <div>\n", | ||
" <h1>Foo</h1>\n", | ||
" <p>Some intro text about Foo.</p>\n", | ||
" <div>\n", | ||
" <h2>Bar main section</h2>\n", | ||
" <p>Some intro text about Bar.</p>\n", | ||
" <h3>Bar subsection 1</h3>\n", | ||
" <p>Some text about the first subtopic of Bar.</p>\n", | ||
" <h3>Bar subsection 2</h3>\n", | ||
" <p>Some text about the second subtopic of Bar.</p>\n", | ||
" </div>\n", | ||
" <div>\n", | ||
" <h2>Baz</h2>\n", | ||
" <p>Some text about Baz</p>\n", | ||
" </div>\n", | ||
" <br>\n", | ||
" <p>Some concluding text about Foo</p>\n", | ||
" </div>\n", | ||
" </body>\n", | ||
" </html>\n", | ||
"\"\"\"\n", | ||
"\n", | ||
"headers_to_split_on = [\n", | ||
" (\"h1\", \"Header 1\"),\n", | ||
" (\"h2\", \"Header 2\"),\n", | ||
" (\"h3\", \"Header 3\"),\n", | ||
" (\"h4\", \"Header 4\"),\n", | ||
"]\n", | ||
"\n", | ||
"html_splitter = HTMLSectionSplitter(headers_to_split_on=headers_to_split_on)\n", | ||
"\n", | ||
"html_header_splits = html_splitter.split_text(html_string)\n", | ||
"\n", | ||
"chunk_size = 500\n", | ||
"chunk_overlap = 30\n", | ||
"text_splitter = RecursiveCharacterTextSplitter(\n", | ||
" chunk_size=chunk_size, chunk_overlap=chunk_overlap\n", | ||
")\n", | ||
"\n", | ||
"# Split\n", | ||
"splits = text_splitter.split_documents(html_header_splits)\n", | ||
"splits" | ||
] | ||
} | ||
], | ||
"metadata": { | ||
"kernelspec": { | ||
"display_name": "Python 3 (ipykernel)", | ||
"language": "python", | ||
"name": "python3" | ||
}, | ||
"language_info": { | ||
"codemirror_mode": { | ||
"name": "ipython", | ||
"version": 3 | ||
}, | ||
"file_extension": ".py", | ||
"mimetype": "text/x-python", | ||
"name": "python", | ||
"nbconvert_exporter": "python", | ||
"pygments_lexer": "ipython3", | ||
"version": "3.9.1" | ||
} | ||
}, | ||
"nbformat": 4, | ||
"nbformat_minor": 5 | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.