# Chunking Techniques with Langchain and LlamaIndex

> The aim is to get the data in a format where it can be used for anticipated tasks, and retrieved for value later. Rather than asking “How should I chunk my data?”, the actual question should be “What is the optimal way for me to pass data to my language model that it needs for its task?”

Let’s begin with Langchain first!

## Chunking Techniques in Langchain

Before jumping into chunking, make sure to first install **Langchain-text-splitters**

Shell

```shell
pip install langchain-text-splitters
```

These snippets cover the relevant sections of code. To follow along with the working code, use this [Colab](https://colab.research.google.com/github/lancedb/vectordb-recipes/blob/main/tutorials/Langchain-LlamaIndex-Chunking/Langchain_Llamaindex_chunking.ipynb).

### Text Character Splitting

This is the simplest method. This splits based on characters and measures chunk length by number of characters.

Python

```python
from langchain_text_splitters import CharacterTextSplitter

text_splitter = CharacterTextSplitter(
    separator="\n\n",
    chunk_size=1000,
    chunk_overlap=200,
    length_function=len,
    is_separator_regex=False,
)

texts = text_splitter.create_documents([state_of_the_union])
print(texts[0].page_content)
```

### Recursive Character Splitting

This text splitter is the recommended one for generic text. It is parameterized by a list of characters. It tries to split them in order until the chunks are small enough. The default list is `["\n\n", "\n", " ", ""]`. It includes overlapping text which helps build context between text splits.

Python

```python
# Recursive Split Character

# This is a long document we can split up.
with open("state_of_the_union.txt") as f:
    state_of_the_union = f.read()

from langchain_text_splitters import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=100,
    length_function=len,
    is_separator_regex=False,
)

texts = text_splitter.create_documents([state_of_the_union])
print("Chunk 2: ", texts[1].page_content)
print("Chunk 3: ", texts[2].page_content)
```

### HTML Section Splitter

HTML Section Splitter is a “structure-aware” chunker that splits text at the element level and adds metadata for each header “relevant” to any given chunk.

Python

```python
url = "https://www.utoronto.ca/"

# Send a GET request to the URL
response = requests.get(url)
if response.status_code == 200:
    html_doc = response.text

headers_to_split_on = [
    ("h1", "Header 1"),
    ("p", "paragraph")
]

html_splitter = HTMLHeaderTextSplitter(headers_to_split_on=headers_to_split_on)
html_header_splits = html_splitter.split_text(html_doc)
html_header_splits[0].page_content
```

### Code Splitter

CodeTextSplitter allows you to split your code with multiple languages supported. Supported Languages are Python, JS, TS, C, Markdown, Latex, HTML, and Solidity.

Python

```python
    from langchain_text_splitters import Language, RecursiveCharacterTextSplitter

with open("code.py") as f:
        code = f.read()

python_splitter = RecursiveCharacterTextSplitter.from_language(
        language=Language.PYTHON, chunk_size=100, chunk_overlap=0
    )
    python_docs = python_splitter.create_documents([code])
    python_docs[0].page_content
```

### Recursive Json Splitting

This JSON splitter goes through JSON data from the deepest levels first and creates smaller JSON pieces. It tries to keep nested JSON objects intact but may divide them if necessary to ensure that the chunks fall within a specified range from the minimum to the maximum chunk size.

Python

```python
    from langchain_text_splitters import RecursiveJsonSplitter
    import json
    import requests

json_data = requests.get("https://api.smith.langchain.com/openapi.json").json()

splitter = RecursiveJsonSplitter(max_chunk_size=300)
    json_chunks = splitter.split_json(json_data=json_data)
    json_chunks[0]
```

### Semantic Splitting

This method splits the text based on semantic similarity. Here we’ll use OpenAI Embeddings for semantic similarity.

Python

```python
    #!pip install --quiet langchain_experimental langchain_openai

import os
    from langchain_experimental.text_splitter import SemanticChunker
    from langchain_openai.embeddings import OpenAIEmbeddings

# Add OpenAI API key as an environment variable
    os.environ["OPENAI_API_KEY"] = "sk-****"

with open("state_of_the_union.txt") as f:
        state_of_the_union = f.read()

text_splitter = SemanticChunker(OpenAIEmbeddings())

docs = text_splitter.create_documents([state_of_the_union])
    print(docs[0].page_content)
```

### Split by Tokens

Language models come with a token limit, which you can not surpass.

To prevent issues, make sure to track the token count when dividing your text into chunks.

For now, we’ll use Tiktoken as a tokenizer

Python

```python
    # ! pip install --upgrade --quiet tiktoken
    from langchain_text_splitters import CharacterTextSplitter

with open("state_of_the_union.txt") as f:
        state_of_the_union = f.read()

text_splitter = CharacterTextSplitter.from_tiktoken_encoder(chunk_size=100, chunk_overlap=0)
    texts = text_splitter.split_text(state_of_the_union)

print(texts[0])
```

## LlamaIndex Chunking Techniques with Implementation

Make sure to install `llama_index` package.

Shell

```shell
    ! pip install llama_index tree_sitter tree_sitter_languages -q
```

In LlamaIndex, Node parsers terminology is used which breaks down a list of documents into Node objects where each node represents a distinct chunk of the parent document, inheriting all attributes from the Parent document to the Children nodes.

### Node Parser — Simple File

To make it easier to read nodes, there are different file-based parsers you can use.

Python

```python
    from llama_index.core.node_parser import SimpleFileNodeParser
    from llama_index.readers.file import FlatReader
    from pathlib import Path

# Download for running any text file
    !wget https://raw.githubusercontent.com/lancedb/vectordb-recipes/main/README.md

md_docs = FlatReader().load_data(Path("README.md"))

parser = SimpleFileNodeParser()

md_nodes = parser.get_nodes_from_documents(md_docs)
    md_nodes[0].text
```

### Node Parser — HTML

This node parser uses Beautiful Soup to understand raw HTML content. It’s set up to read certain HTML tags automatically.

Python

```python
    from llama_index.core import Document
    from llama_index.core.node_parser import HTMLNodeParser

# URL of the website to fetch HTML from
    url = "https://www.utoronto.ca/"

# Send a GET request to the URL
    response = requests.get(url)
    print(response)

# Check if the request was successful (status code 200)
    if response.status_code == 200:
        # Extract the HTML content from the response
        html_doc = response.text
        document = Document(id_=url, text=html_doc)

parser = HTMLNodeParser(tags=["p", "h1"])
        nodes = parser.get_nodes_from_documents([document])
        print(nodes)
    else:
        print("Failed to fetch HTML content:", response.status_code)
```

### Node Parser — JSON

To handle JSON documents, we’ll use a JSON parser.

Python

```python
    from llama_index.core.node_parser import JSONNodeParser

url = "https://housesigma.com/bkv2/api/search/address_v2/suggest"

payload = {"lang": "en_US", "province": "ON", "search_term": "Mississauga, ontario"}

headers = {
        'Authorization': 'Bearer 20240127frk5hls1ba07nsb8idfdg577qa'
    }

response = requests.post(url, headers=headers, data=payload)

if response.status_code == 200:
        document = Document(id_=url, text=response.text)
        parser = JSONNodeParser()

nodes = parser.get_nodes_from_documents([document])
        print(nodes[0])
    else:
        print("Failed to fetch JSON content:", response.status_code)
```

### Node Parser — Markdown

To handle Markdown files, we’ll use a Markdown parser.

Python

```python
    # Markdown
    from llama_index.core.node_parser import MarkdownNodeParser

md_docs = FlatReader().load_data(Path("README.md"))
    parser = MarkdownNodeParser()

nodes = parser.get_nodes_from_documents(md_docs)
    nodes[0].text
```

### Conclusion

Langchain and Llama Index are popular tools, and one of the key things they do is “chunking.” This means breaking down data into smaller pieces, which is important for making the tools work well. These platforms provide a variety of ways to do chunking, creating a unified solution for processing data efficiently. This article will guide you through all the chunking techniques you can find in Langchain and Llama Index.