Text splitters

Text splitters divide documents into retrievable DocumentChunk objects. The choice of splitter affects retrieval quality.

Available text splitters

Splitter	Best for
Character Text Splitter	Simple text, quick prototyping
Token Text Splitter	Token-aware chunking, LLM context management
Markdown Text Splitter	Markdown documents, structured content with headers
Separator Text Splitter	Custom splitting logic, hierarchical text
Custom Splitters	Any splitting strategy

Information

Each sub-chunk inherits the type and metadata of the chunk it comes from, such as page_number and any custom keys. Only its locator, start_offset, and end_offset change. Non-content chunks, such as summaries and image annotations, pass through unsplit.

Chunk size guidance

Chunk size affects retrieval quality and LLM context usage. Choose it based on your use case:

LLM Context	Chunk Size	Overlap	Use case
4k tokens	300-500 chars	50-100 chars	Memory-constrained, fast retrieval
8k tokens	500-1000 chars	100-200 chars	Balanced retrieval quality
32k+ tokens	1000-2000 chars	200-500 chars	Rich context, complex retrieval
Code/technical	500-1000 chars	100-200 chars	Preserve logical units
Legal/financial	1000-2000 chars	200-500 chars	Full context for interpretation

Rule of thumb: ~400-600 characters ≈ 100-150 tokens. Adjust based on domain complexity and retrieval precision needs.

Chunk overlap benefits:

Prevents semantic boundaries from splitting related concepts
Improves context when chunks are used individually
Increases index size and query latency proportionally

Character text splitter

CharacterTextSplitter splits text by character count, with no overlap. Use it for simple text and quick prototyping.

Example:

from mistralai.search.toolkit.ingestion.text_splitters import CharacterTextSplitter

splitter = CharacterTextSplitter(
    chunk_size=500,     # Characters per chunk
)
chunks = (await splitter.process(document)).chunks

from mistralai.search.toolkit.ingestion.text_splitters import CharacterTextSplitter

splitter = CharacterTextSplitter(
    chunk_size=500,     # Characters per chunk
)
chunks = (await splitter.process(document)).chunks

Parameters:

Parameter	Type	Default	Description
`chunk_size`	`int`	`1000`	Maximum number of characters per chunk

Token text splitter

TokenTextSplitter counts tokens with a Mistral tokenizer, for precise control over chunk size in tokens.

Example:

from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
from mistralai.search.toolkit.ingestion.text_splitters import TokenTextSplitter

splitter = TokenTextSplitter(
    chunk_size=1000,                  # Tokens per chunk
    chunk_overlap=200,                # Token overlap
    tokenizer=MistralTokenizer.v1(),  # Embedding tokenizer
)

chunks = (await splitter.process(document)).chunks

from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
from mistralai.search.toolkit.ingestion.text_splitters import TokenTextSplitter

splitter = TokenTextSplitter(
    chunk_size=1000,                  # Tokens per chunk
    chunk_overlap=200,                # Token overlap
    tokenizer=MistralTokenizer.v1(),  # Embedding tokenizer
)

chunks = (await splitter.process(document)).chunks

Configuration options:

Option	Type	Default	Purpose
`chunk_size`	int	512	Target tokens per chunk
`chunk_overlap`	int	50	Tokens of overlap between chunks
`model_name`	str	"mistral-embed"	Model whose tokenizer is resolved when `tokenizer` is unset
`tokenizer`	`MistralTokenizer \| None`	None	Tokenizer instance; takes precedence over `model_name`

When to use:

Precise token budgeting for LLM context windows
Consistent chunks across different text densities
When working with specific token limits

Markdown text splitter

MarkdownTextSplitter splits markdown documents at header boundaries. It splits on headers first, then runs a SeparatorTextSplitter on each section. Its configuration extends SeparatorTextSplitterConfig with configurable header levels.

Requirements:

Install the text-splitter-langchain extra:

uv add "mistralai-search-toolkit[text-splitter-langchain]"

uv add "mistralai-search-toolkit[text-splitter-langchain]"

Example:

from mistralai.search.toolkit.ingestion.text_splitters import MarkdownTextSplitter, MarkdownTextSplitterConfig

config = MarkdownTextSplitterConfig(
    headers_to_split_on=[
        ("#", "h1"),
        ("##", "h2"),
        ("###", "h3"),
    ],
    strip_headers=False,    # Keep headers in chunks
    chunk_size=1000,        # Inherited from SeparatorTextSplitterConfig
    chunk_overlap=200,      # Inherited from SeparatorTextSplitterConfig
)

splitter = MarkdownTextSplitter(config=config)
chunks = (await splitter.process(document)).chunks

from mistralai.search.toolkit.ingestion.text_splitters import MarkdownTextSplitter, MarkdownTextSplitterConfig

config = MarkdownTextSplitterConfig(
    headers_to_split_on=[
        ("#", "h1"),
        ("##", "h2"),
        ("###", "h3"),
    ],
    strip_headers=False,    # Keep headers in chunks
    chunk_size=1000,        # Inherited from SeparatorTextSplitterConfig
    chunk_overlap=200,      # Inherited from SeparatorTextSplitterConfig
)

splitter = MarkdownTextSplitter(config=config)
chunks = (await splitter.process(document)).chunks

Parameters (MarkdownTextSplitterConfig):

MarkdownTextSplitterConfig extends SeparatorTextSplitterConfig, adding:

Parameter	Type	Default	Description
`headers_to_split_on`	`list[tuple[str, str]]`	`[("#", "Header 1"), ("##", "Header 2"), ("###", "Header 3")]`	Markdown headers to split on. Each tuple is `(header_prefix, label)`.
`strip_headers`	`bool`	`False`	Remove header lines from chunk content

All parameters from SeparatorTextSplitterConfig are also available (chunk_size, chunk_max_size, chunk_overlap, chunk_separators, keep_separator, strip_whitespace).

Separator text splitter

SeparatorTextSplitter splits text recursively on a list of separators. Use it for custom document structures where you need control over split points.

Requirements:

Install the text-splitter-langchain extra:

uv add "mistralai-search-toolkit[text-splitter-langchain]"

uv add "mistralai-search-toolkit[text-splitter-langchain]"

Example:

from mistralai.search.toolkit.ingestion.text_splitters import SeparatorTextSplitter, SeparatorTextSplitterConfig

config = SeparatorTextSplitterConfig(
    chunk_size=1000,          # Target size (characters)
    chunk_max_size=1500,      # Maximum size (characters)
    chunk_overlap=200,        # Overlap (characters)
    chunk_separators=[        # Tried in order
        "\n\n",    # Paragraphs first
        "\n",      # Then lines
        ". ",      # Then sentences
        " ",       # Then words
        "",        # Finally characters
    ],
)

splitter = SeparatorTextSplitter(config=config)
chunks = (await splitter.process(document)).chunks

from mistralai.search.toolkit.ingestion.text_splitters import SeparatorTextSplitter, SeparatorTextSplitterConfig

config = SeparatorTextSplitterConfig(
    chunk_size=1000,          # Target size (characters)
    chunk_max_size=1500,      # Maximum size (characters)
    chunk_overlap=200,        # Overlap (characters)
    chunk_separators=[        # Tried in order
        "\n\n",    # Paragraphs first
        "\n",      # Then lines
        ". ",      # Then sentences
        " ",       # Then words
        "",        # Finally characters
    ],
)

splitter = SeparatorTextSplitter(config=config)
chunks = (await splitter.process(document)).chunks

Parameters (SeparatorTextSplitterConfig):

Parameter	Type	Default	Description
`chunk_size`	`int`	`2048`	Target chunk size in characters
`chunk_max_size`	`int \| None`	`None`	Maximum chunk size, to avoid cutting inside a word. `None` uses `chunk_size`.
`chunk_overlap`	`int`	`0`	Overlap between chunks in characters
`chunk_separators`	`list[str]`	`["\n\n", "\n", ".", ", ", ",", " ", "\t"]`	Separators, tried from left to right
`keep_separator`	`bool \| "start" \| "end"`	`"end"`	Whether and where to keep separators in chunks
`strip_whitespace`	`bool`	`False`	Strip leading and trailing whitespace from each chunk

Features:

Tries separators in order
Respects max size even if splitting fails
Merges small chunks when possible
Returns start/end offsets

Creating custom splitters

Implement the TextSplitter protocol:

from mistralai.search.toolkit.context import IngestContext
from mistralai.search.toolkit.ingestion.text_splitters import TextSplitter
from mistralai.search.toolkit.ingestion.text_splitters.models import TextFragment

class ParagraphSplitter(TextSplitter):
    """Split text into chunks by paragraphs."""

    def __init__(self, max_paragraphs_per_chunk: int = 3):
        self.max_paragraphs_per_chunk = max_paragraphs_per_chunk

    def split_text(self, text: str, context: IngestContext = IngestContext()) -> list[TextFragment]:
        paragraphs = text.split("\n\n")
        fragments = []
        offset = 0

        for i in range(0, len(paragraphs), self.max_paragraphs_per_chunk):
            chunk_paragraphs = paragraphs[i:i + self.max_paragraphs_per_chunk]
            content = "\n\n".join(chunk_paragraphs)

            fragments.append(TextFragment(
                content=content,
                start_offset=offset,
                end_offset=offset + len(content),
            ))
            offset += len(content) + 2

        return fragments

splitter = ParagraphSplitter(max_paragraphs_per_chunk=3)
fragments = splitter.split_text("First paragraph.\n\nSecond paragraph.\n\nThird paragraph.")

from mistralai.search.toolkit.context import IngestContext
from mistralai.search.toolkit.ingestion.text_splitters import TextSplitter
from mistralai.search.toolkit.ingestion.text_splitters.models import TextFragment

class ParagraphSplitter(TextSplitter):
    """Split text into chunks by paragraphs."""

    def __init__(self, max_paragraphs_per_chunk: int = 3):
        self.max_paragraphs_per_chunk = max_paragraphs_per_chunk

    def split_text(self, text: str, context: IngestContext = IngestContext()) -> list[TextFragment]:
        paragraphs = text.split("\n\n")
        fragments = []
        offset = 0

        for i in range(0, len(paragraphs), self.max_paragraphs_per_chunk):
            chunk_paragraphs = paragraphs[i:i + self.max_paragraphs_per_chunk]
            content = "\n\n".join(chunk_paragraphs)

            fragments.append(TextFragment(
                content=content,
                start_offset=offset,
                end_offset=offset + len(content),
            ))
            offset += len(content) + 2

        return fragments

splitter = ParagraphSplitter(max_paragraphs_per_chunk=3)
fragments = splitter.split_text("First paragraph.\n\nSecond paragraph.\n\nThird paragraph.")