Changelog

All notable changes to Search Toolkit are documented here.

Breaking changes

Embedding model on the schema

• create_schema() and create_default_schema() now take embedding_model (an EmbeddingModel or MistralEmbeddingPreset) instead of embedding_dimensions: int. embedding_dimensions is deprecated (emits a DeprecationWarning and builds a CustomEmbeddingModel(name="unknown") under the hood); passing both raises ValueError.
• VespaSchemaDefinition.embedding_dimensions is replaced by embedding_model: EmbeddingModel, threading the full model config (name, dimensions, dtype, distance metric) through the schema layer. VespaSearchIndex now exposes an embedding_model property.

Improvements

• Added a Postgres search backend (mistralai-search-toolkit-plugins-postgres) as a second index alongside Vespa. PostgresStoreIndex implements VectorStoreIndex (dense and hybrid search), NavigableIndex (navigate/read/grep/get_chunk), and PatchableIndex (partial updates). A VectorSearchQuery that carries both query and embedding runs BM25 (pg_textsearch) and HNSW (pgvector) retrieval fused by weighted reciprocal rank fusion; the new PostgresSearchQuery tunes vector_weight / text_weight / rrf_k. The PostgresApp mirrors the Vespa app's get_search_index(config, collection) shape, and collections are declared with PostgresCollectionSchema(collection_name, document_type, embedding_model). Requires the vector and pg_textsearch extensions provisioned in the target database.
• Added an EmbeddingModel abstraction (embedding/models.py) as the single source of truth for vector config: MistralEmbeddingModel and CustomEmbeddingModel (discriminated union via EmbeddingModel), the MistralEmbeddingPreset enum (MISTRAL_EMBED_DIM_1024 / MISTRAL_EMBED_DIM_256 / MISTRAL_EMBED_DIM_128) with build_embedding_model(), plus VectorDType (FLOAT32 / FLOAT16), DistanceMetric (COSINE / INNER_PRODUCT / L2), and a VectorConfig protocol.
• SmartPDFExtractor and LegacyOfficeExtractor now accept any DocumentExtractor as an OCR delegate instead of requiring MistralOCRExtractor, so a deployment with no reachable Mistral API can supply its own engine.
• Added opt-in text-first PDF extraction via pdf_min_chars_per_page (PdfTextFirstExtractor); off by default. PDFs dispatched to the heuristic leave .docx/.pptx/.odt/.epub and image attachments on the OCR route untouched.
• Threaded exclude_ids through the retrieval layer: QueryEngine.search(..., exclude_ids=...) forwards to retrievers and onto the search query. CachedQueryEngine.search forwards it and bypasses the cache when exclusions are set.

Bugfixes

• Updated markdownify from >=0.14,<1 to >=1.0.0,<2 and adapted the custom converter to the 1.x API.

Breaking changes

Checkpoint serialization

• A new document/serialization.py module introduces a DocumentTypeRegistry and stable type ids so that document classes can be round-tripped through an external service safely.

Improvements

• Added MistralClientConfig and build_mistral_client() to centralize Mistral SDK client creation for whole pipelines.
• Added SmartPDFExtractor, which chooses between PyMuPDF and Mistral OCR automatically for PDFs that already have a text layer.
• Added EmailWithAttachmentsExtractor, which adds attachment text to the parent email's Markdown under an ## Attachments section.
• Improved error reporting for ingestion problems.

Bugfixes

• Deterministic ingestion failures now fail faster.

Chores

• Bumped mistralai to >=2.9.1,<2.10.
• Bumped dynaconf from >=3.2.11 to >=3.3.4.
• Renamed the mistralai.search.toolkit.embedders module to mistralai.search.toolkit.embedding. A backward-compatible embedders shim re-exports public symbols and emits a DeprecationWarning. See the Embedders page.

Improvements

• For ingestion, made extraction checkpoints (saving snapshots between stages, for faster retrying) a backend-agnostic capability. Pipeline(checkpoint_store=...) accepts any ObjectStorage backend and saves checkpoints between the ingestion stages; run_file(..., checkpoint_key=...) loads and saves the extraction result so retries and re-ingestion skip re-extraction.
• Introduced a DocumentProcessor protocol for ingestion, unifying TextSplitter, Embedder, and ChunkEnricher.
• Added map-reduce summarization for long documents: SummarizeRequestConfig with overflow_strategy: OverflowStrategy.MAP_REDUCE can split a document, summarize the different parts, and merge them into a final summary. See the Summary Enricher.
• Added OCR v4 features: inline markdown table embedding in page chunks, per-chunk confidence scores, and basic bounding-box support.
• Added a PatchableIndex capability with patch_chunk and patch_document for partial updates on DOCUMENT_PER_CHUNK indices, avoiding a full re-index when only metadata changes.
• Added get_chunk(chunk_id) to the NavigableIndex protocol.
• Vespa: added set_default_ranking_weights() to bake default ranking-feature weights into a schema and its default query profile. See Manage ranking.
• Vespa: added a phase-1 vector closeness ranking function for embedding fields, putting vector relevance into candidate selection. See Ranking profiles.
• Vespa: BoolField gained a fast_search option.

Bugfixes

• Sanitize illegal Unicode code points (lone surrogates) from extracted content.

Chores

• Updated mistralai bound from <2.7 to <2.9.
• Bumped pypdf from 6.13.3 to 6.14.2.
• Bumped azure-storage-blob to 12.30.0.

Improvements

• Added embeddings batch API support for ingestion: MistralEmbedder.run_batch_embedding_job() can be used to lower embedding costs for large ingestion jobs.
• Added header-aware token-budgeted chunking for ingestion: new ParagraphTokenTextSplitter and MarkdownTokenTextSplitter splitters. See the Text splitters page.
• Added a NavigableIndex protocol with navigate, read, and grep methods for positional chunk navigation on DOCUMENT_PER_CHUNK indices.
• Vespa: added YQL search on DocumentPerChunkSearchIndex via VespaSearchQuery, with ranking tuning and chunk-id exclusion (exclude_ids). See Manage ranking.

Bugfixes

• Fixed TokenTextSplitter reporting wrong offsets for repeated content.

Chores

• Relaxed the toolkit mistralai constraint to >=2.5.0,<2.7.

Breaking changes

Document model

Search Toolkit now uses a unified document model built around Document and DocumentChunk, with a deterministic identity derived from a source_id and a locator. See the Document model page for full details.

• Extractors now produce DocumentChunk objects directly; the separate page representation has been removed.
• Document.id and DocumentChunk.id are now computed deterministically from source_id (plus locator for chunks), making indexing idempotent. The explicit id field on File and document_id on DocumentChunk have been removed.
• Added source_id, locator, parent_ref, and chunk_type as first-class fields, along with typed, extensible metadata models. The same identity contract is mirrored on SearchResultChunk.

Vespa indexing model

Vespa now indexes one chunk per document via the new DOCUMENT_PER_CHUNK indexing mode, which becomes the recommended model. The previous single-document model is deprecated.

• Added the DOCUMENT_PER_CHUNK indexing mode, including default fields, ranking profiles, and the full write, delete, and search paths. See Anatomy of a Vespa application.
• Added an IndexingMode to the schema definition with deprecation hooks for migrating existing schemas.
• The index API is split into a base VespaSearchIndex and a dedicated SingleDocumentSearchIndex; the single-document model is deprecated in favor of DOCUMENT_PER_CHUNK.
• The schema id_field is deprecated and is no longer allowed for DOCUMENT_PER_CHUNK indexes.

Other

• Renamed the indices module to search. Update imports accordingly.

Improvements

• Added blob-storage FileLoader implementations for S3, Azure, and GCS, plus a storage-s3 extra. See the File loaders page.
• Added OCR model literals and constants.
• Vespa: extracted a dedicated VespaClient with improved error handling.
• Vespa: added a backend-agnostic services definition, topology v2, and a translator, with automatic v2 topology generation for single-node Docker deployments.
• Vespa: added Vespa-to-Vespa copy and index-to-streaming migration workflows.
• Vespa: emit a metrics consumer in services.xml.
• Vespa: warn when rank2 features are configured without rank1, and when ranking weights default to 0.

Security

• Updated langchain-core to ~=1.4.

Bugfixes

• Vespa: rank by cosine similarity instead of euclidean distance.
• Vespa: fix retrieval of the document count.
• Vespa: thread distribute_across_groups into load_topology_file.
• Vespa CLI: lazy-import index registration SDK models.
• Treat truncated LLM responses as retryable and enrich LLMException / SummaryGenerationError for structured logging.
• OCR extractor: use mimetype metadata for filetype detection.
• Added text/x-file and text/x-script.python MIME types to the registry.

Initial release of Search Toolkit as a tech preview.