# scinr Full Documentation This file contains the complete documentation for the scinr library for LLM agent context. --- ## File: api/config.md # Configuration API Programmatic configuration of ``scinr.newton``. ## Core Functions ::: scinr.newton.config.configure ::: scinr.newton.config.get_config ::: scinr.newton.config.get_available_themes ## Configuration Classes ::: scinr.newton.config.ScinrConfig ::: scinr.newton.config.ThemePath ::: scinr.newton.config.PromptFamily --- ## File: api/converters.md # Converters API Document format converters used in Stage 0 (Preprocess). ## Converter Registry ::: scinr.newton.converters.registry ## Base Classes ::: scinr.newton.converters.base ## Format Converters ::: scinr.newton.converters.pdf ::: scinr.newton.converters.docx ::: scinr.newton.converters.pptx ::: scinr.newton.converters.xlsx ::: scinr.newton.converters.csv ::: scinr.newton.converters.html ::: scinr.newton.converters.api_json ::: scinr.newton.converters.api_xml ::: scinr.newton.converters.text --- ## File: api/deletion.md # Deletion API ::: scinr.newton.ingest.deletion.delete_document --- ## File: api/exceptions.md # Exceptions API ::: scinr.newton.exceptions.ScinrError options: show_submodules: true members: - ConfigurationError - PreconditionError - ExtractionError - IngestionError - ModelError - StorageError - ConversionError --- ## File: api/index.md # API Reference Overview Welcome to the `scinr.newton` API reference. All API documentation is auto-generated from Python docstrings using mkdocstrings. ## Core Modules - [Pipeline](pipeline.md): ``run_pipeline()`` orchestrator. - [Configuration](config.md): ``configure()``, ``get_config()``, ``ScinrConfig``. - [Stages](stages.md): Individual stage runner functions (Stages 0-5). - [Normalization](normalization.md): ``NormalizationEngine`` and normalization utilities. - [Results](results.md): ``PipelineResult``, ``StageResult``, ``DocumentResult``, ``DeletionResult``. - [Exceptions](exceptions.md): ``ScinrError`` hierarchy. - [Deletion](deletion.md): ``delete_document()`` — permanent document removal with cascade and garbage collection. - [Converters](converters.md): Document format converters. - [Storage](storage.md): Storage backends. - [Utilities](utilities.md): Theme registry, LLM factory, and utilities. ## User Guides For tutorials and how-to guides, see the [User Guides](../user-guides/quick-start.md) section. --- ## File: api/normalization.md # Normalization API LLM-based normalization engine for tabular data. ## NormalizationEngine ::: scinr.newton.tabular.normalization.engine.NormalizationEngine ## Utility Functions ::: scinr.newton.tabular.normalization.run_normalization_hook ::: scinr.newton.tabular.normalization.get_normalization_specs ::: scinr.newton.tabular.normalization.instance_has_normalizable_fields ::: scinr.newton.tabular.normalization.extract_source_values --- ## File: api/pipeline.md # Pipeline API ::: scinr.newton.pipeline.run_pipeline --- ## File: api/results.md # Results API ::: scinr.newton.results.DocumentResult ::: scinr.newton.results.StageResult ::: scinr.newton.results.PipelineResult ::: scinr.newton.results.DeletionResult --- ## File: api/stages.md # Stages API Individual stage runner functions. Each can be called independently or via ``run_pipeline()``. ## Stage 0: Preprocess ::: scinr.newton.stages.run_preprocess ## Stage 1: Extraction ::: scinr.newton.stages.run_extraction ## Stage 2: Ingestion ::: scinr.newton.stages.run_ingestion ## Stage 3: Annotation ::: scinr.newton.stages.run_annotation ## Stage 4: Entity Extraction ::: scinr.newton.stages.run_entity_extraction ## Stage 5: Tabular Pipeline ::: scinr.newton.stages.run_tabular_pipeline --- ## File: api/storage.md # Storage API Document storage backends for raw file and page content archival. ## Backend Factory ::: scinr.newton.storage.factory ## Base Classes ::: scinr.newton.storage.base ## MongoDB Backend ### Client & GridFS ::: scinr.newton.storage.mongodb.client ### Page Repository ::: scinr.newton.storage.mongodb.pages ### Raw File Repository ::: scinr.newton.storage.mongodb.raw_files ## Null Backend ::: scinr.newton.storage.null --- ## File: api/utilities.md # Utilities API Supporting utilities for theme discovery, LLM management, and graph operations. ## Theme Registry ::: scinr.newton.utils.theme_registry.ThemeRegistry ## LLM Utilities ::: scinr.newton.utils.llm_factory ## Retry Utilities ::: scinr.newton.utils.llm_retry ::: scinr.newton.utils.neo4j_retry ## Concurrency ::: scinr.newton.utils.neo4j_concurrency ## Logging ::: scinr.newton.utils.logging_config --- ## File: architecture.md # scinr.newton Architecture `scinr.newton` is an **async-first Python library** that processes life sciences documents through a **6-stage modular pipeline**, producing a structured knowledge graph stored in Neo4j with optional binary/auxiliary storage in MongoDB. The pipeline operates via **two parallel tracks**: - **Unstructured pipeline** (Stages 0–4): handles PDF, DOCX, PPTX, HTML, XML, TXT files through format conversion, LLM-powered structural extraction, graph ingestion, annotation, and entity extraction. - **Tabular pipeline** (Stage 5): handles CSV, XLSX, XLS files through a dedicated path that bypasses Stages 0–4 entirely, using LLM-driven column mapping and direct graph writes. Both tracks converge into the **same Neo4j knowledge graph**, sharing node labels, relationship types, and schema constraints. --- ## Table of Contents 1. [Pipeline Architecture Diagram](#1-pipeline-architecture-diagram) 2. [Stage Details](#2-stage-details) - [Stage 0: Preprocess](#stage-0-preprocess-run_preprocess) - [Stage 1: Extraction](#stage-1-extraction-run_extraction) - [Stage 2: Ingestion](#stage-2-ingestion-run_ingestion) - [Stage 3: Annotation](#stage-3-annotation-run_annotation) - [Stage 4: Entity Extraction](#stage-4-entity-extraction-run_entity_extraction) - [Tabular Pipeline](#tabular-pipeline-run_tabular_pipeline) 3. [Async Architecture](#3-async-architecture) 4. [Configuration System](#4-configuration-system) 5. [Module Structure](#5-module-structure) 6. [Data Flow](#6-data-flow) 7. [Neo4j Schema](#7-neo4j-schema) 8. [Storage Backends](#8-storage-backends) 9. [Prompt System](#9-prompt-system) 10. [Error Handling](#10-error-handling) 11. [Result Types](#11-result-types) --- ## 1. Pipeline Architecture Diagram ``` ┌──────────────────────────────────────────────────┐ │ Raw Documents │ │ .pdf .docx .pptx .xlsx .csv .json .html .xml │ └───────────────┬──────────────────────────────────┘ │ ┌─────────────────────────┼─────────────────────────┐ ▼ │ ▼ ┌──────────────────┐ │ ┌──────────────────┐ │ Stage 0: │ │ │ Tabular Files │ │ Preprocess │ │ │ (.csv .xlsx .xls)│ │ Converters │ │ │ │ └───────┬──────────┘ │ └───────┬──────────┘ ▼ │ ▼ ┌──────────────────┐ │ ┌──────────────────┐ │ Stage 1: │ │ │ Tabular Pipeline │ │ Extraction │ │ │ Stage 5 │ │ LLM Chunking │ │ │ │ └───────┬──────────┘ │ └───────┬──────────┘ ▼ │ │ ┌──────────────────┐ │ ┌───────┴──────────┐ │ Stage 2: │ │ │ Document + │ │ Ingestion │ │ │ Table + Row │ │ Neo4j write │ │ │ nodes (direct) │ └───────┬──────────┘ │ └──────────────────┘ ▼ │ ┌──────────────────┐ │ │ Stage 3: │ │ │ Annotation │ │ │ LLM classify │ │ └───────┬──────────┘ │ ▼ │ ┌──────────────────┐ │ │ Stage 4: │ │ │ Entity Extract │ │ │ Pydantic + Neo4j │ │ └────────┬─────────┘ │ │ │ └──────────┬───────────────┘ ▼ ┌────────────────────────┐ │ Neo4j Knowledge │ │ Graph │ └────────────────────────┘ ``` ### Pipeline Entry Points | Function | Module | Description | |---|---|---| | `run_pipeline()` | `pipeline.py` | Main orchestrator; chains stages 0–4 sequentially with tabular auto-detection | | `run_preprocess()` | `stages/preprocess.py` | Standalone Stage 0 | | `run_extraction()` | `stages/extraction.py` | Standalone Stage 1 | | `run_ingestion()` | `stages/ingestion.py` | Standalone Stage 2 | | `run_annotation()` | `stages/annotation.py` | Standalone Stage 3 | | `run_entity_extraction()` | `stages/entity_extraction.py` | Standalone Stage 4 | | `run_tabular_pipeline()` | `stages/tabular.py` | Standalone Stage 5 (tabular only) | All stage functions are `async def` and return typed `StageResult` dataclasses. --- ## 2. Stage Details ### Stage 0: Preprocess (`run_preprocess`) **Purpose:** Convert raw source files into a standardized intermediate JSON format. **Input:** Directory of raw files (`.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.json`, `.html`, `.xml`, `.txt`). **Output:** - `StageResult` with per-file success/failure counts - List of `IntermediateDocument` objects (in-memory) - Optional JSON files on disk (when `output_dir` is provided) **Architecture:** Each file format has a dedicated converter inheriting from `BaseConverter` (abstract class in `converters/base.py`): | Converter | File | Supported Extensions | Dependencies | |---|---|---|---| | `PdfConverter` | `converters/pdf.py` | `.pdf` | `pdfplumber`, Mistral OCR API | | `DocxConverter` | `converters/docx.py` | `.docx` | `python-docx` | | `PptxConverter` | `converters/pptx.py` | `.pptx` | `python-pptx` | | `XlsxConverter` | `converters/xlsx.py` | `.xlsx`, `.xls` | `openpyxl`, `pandas` | | `CsvConverter` | `converters/csv.py` | `.csv` | `pandas` | | `HtmlConverter` | `converters/html.py` | `.html`, `.htm` | `BeautifulSoup` | | `TextConverter` | `converters/text.py` | `.txt`, `.md`, `.rst` | stdlib | | `ApiJsonConverter` | `converters/api_json.py` | `.json` | stdlib | | `ApiXmlConverter` | `converters/api_xml.py` | `.xml` | stdlib | Converters are registered in `converters/registry.py` via a lazy-loaded extension-to-class map. Custom converters can be injected at runtime via `configure(extra_converters={...})` — the `apply_converter_overrides()` function handles both new extensions and built-in overrides. **PDF Conversion Strategy:** The `PdfConverter` uses a two-tier approach: 1. **pdfplumber** for native (text-extractable) PDFs — extracts text, tables, images, and page dimensions. 2. **Mistral OCR API** for scanned PDFs — chunks large PDFs by page count (`mistral_ocr_safe_max_pages`, default 900) and file size (`mistral_ocr_safe_max_bytes`, default 45 MiB), sends each chunk to the Mistral OCR endpoint, and reassembles the result. The `pdf_splitter.py` module handles structural PDF partitioning. Error strategy for Mistral OCR is configurable: `fail_fast` (default, aborts the entire document on any chunk failure) or `best_effort` (skips failed chunks and continues). **Intermediate Document Format:** Every converter produces an `IntermediateDocument` (Pydantic model) with: - `pages`: list of `IntermediatePage` objects, each containing `index`, `markdown` (text content), `images` (base64-encoded with MIME type), `dimensions`, `tables`, `hyperlinks`, `header`, `footer`, and `page_id`. - `folder_path`: relative path of the source file's parent directory from the input root. - `raw_file_id`: MongoDB ObjectId of the stored raw file (when storage backend is configured). - `context_instructions`: free-text user context injected via CLI `--context`. - `document_name`: stem of the original source file. **Concurrency:** Documents are processed with bounded parallelism via `parallel_docs` parameter (default: 1 for sequential processing, matching pre-existing behavior). **Storage Integration:** When `storage_backend` is configured (not `"none"`), Stage 0 stores: - Raw file binary via `RawFileRepository.store()` (GridFS for MongoDB) - Converted pages via `PageRepository.store()` (MongoDB collection) The storage layer is abstracted behind `storage/factory.py` and supports three backends: `"none"`, `"mongodb"`, and `"custom"`. --- ### Stage 1: Extraction (`run_extraction`) **Purpose:** Use an LLM to parse intermediate document pages into a hierarchical tree of `StructureNode` objects, producing `Document` objects. **Input:** Either `IntermediateDocument` objects (from Stage 0 in-memory) or JSON files on disk (from `extraction_input_dir`). **Output:** - `StageResult` with per-document success/failure counts - List of `Document` objects (in-memory) - Optional `extract-*.json` files on disk (when `output_folder` is provided) **Architecture:** The extraction engine (`extraction/` module) processes documents in **sliding-window chunks** of configurable size (`extraction_batch_size`, default: 3 pages per chunk). Each chunk: 1. Builds the **active hierarchy** — the current tree of `StructureNode` objects accumulated so far for this document. 2. Sends the previous page (for context), current pages, and active hierarchy to the LLM via `extract_chunk()`. 3. The LLM returns a `DocumentStructure` (Pydantic model) containing a list of `StructureNode` objects for the new content in the chunk. 4. `compact_extraction()` merges the new nodes into the existing document tree, handling: - Continuation of nodes that started on a previous chunk - Insertion of new top-level and nested nodes - Preservation of parent-child relationships via `parent_id` references - Deduplication of nodes that appear across chunk boundaries **StructureNode Model:** Each `StructureNode` has: - `node_id`: unique identifier (derived from heading number or appearance order + slug) - `title`: exact heading text from the source document (never paraphrased) - `role`: one of `section`, `subsection`, `table`, `appendix`, `field_group`, `freeform_block`, `row` - `appearance_order`: 1-based position among siblings - `parent_id`: reference to parent node's `node_id` (null for top-level) - `theme`: theme path (default: `"default"`) - `source_page_ids`: list of MongoDB page IDs (set by pipeline, not LLM) - `info_units`: list of `InfoUnit` objects — the semantic content extracted from this node - `children`: nested `StructureNode` objects **InfoUnit Model:** The smallest semantic unit, containing: - `title`: short label (3–8 words) - `order`: 0-based position within parent node - `description`: self-contained technical note preserving all quantitative values, named entities, conditions, and qualifiers InfoUnits are the **sole content representation** available to downstream agents (Stages 3 and 4). The LLM is instructed to make each description independently interpretable. **LLM Bounded Concurrency:** Each `extract_chunk()` call is bounded by the global `get_llm_semaphore()` (size: `llm_concurrency`, default: 4). This ensures that Stage 1 never exceeds the configured LLM provider rate limits, regardless of how many documents are processed concurrently. **Output Persistence:** When `output_folder` is provided, the extracted `Document` is written as `extract-{doc_name}.json` after each chunk (crash-safe incremental writes) and after final completion (final write). The subdirectory structure mirrors the input folder hierarchy. --- ### Stage 2: Ingestion (`run_ingestion`) **Purpose:** Write `Document` objects and their `StructureNode` hierarchies into Neo4j. **Input:** Either `Document` objects (from Stage 1 in-memory), `extract-*.json` files on disk (`output_folder` or explicit `files` list), or `extract-*.json` files from `ingestion_input_dir`. **Output:** `StageResult` with per-document success/failure counts. Neo4j graph state updated. **Architecture:** The ingestion module (`ingest/`) provides: - **`setup_schema(driver)`** — Creates all Neo4j constraints and indexes idempotently (using `IF NOT EXISTS`). Includes: - 10 unique constraints (Document path+version, StructureNode id, InfoUnit uid, etc.) - 9 regular indexes (Document name/latest/path, StructureNode role, etc.) - 2 fulltext indexes (InfoUnit description and title for semantic search) - Best-effort Neo4j version check (requires >= 4.4) - **`load_documents(documents, driver, update_mode)`** — In-memory ingestion of `Document` objects. - **`load_files(files, driver, update_mode)`** — Ingestion from explicit file paths. - **`load_folder(folder, driver, update_mode)`** — Ingestion from a directory of `extract-*.json` files. **Neo4j Graph Structure (unstructured pipeline):** ``` (:Document) -[:HAS_STRUCTURE]-> (:StructureNode) (:StructureNode) -[:HAS_CHILD]-> (:StructureNode) (:StructureNode) -[:HAS_INFO_UNIT]-> (:InfoUnit) (:Document) -[:IS_COMPOSED_OF]-> (:Document) [folder hierarchy] (:Document) -[:HAS_NEWER_VERSION]-> (:Document) [versioning] ``` **Node Properties:** - `:Document`: `name`, `path`, `version`, `latest`, `raw_file_id`, `context_instructions`, `ingestion_timestamp` - `:StructureNode`: `id` (composite key), `title`, `role`, `appearance_order`, `theme`, `source_page_ids`, `row_index` (tabular only) - `:InfoUnit`: `uid`, `title`, `order`, `description` **Versioning:** The ingestion loader resolves version numbers by querying existing `:Document` nodes with the same `path`. If `update_mode=True`, the existing version is reused (in-place update without creating a new version). Otherwise, a new version is created (`max_version + 1`). The `replaces` parameter links a newly ingested document to an existing one via `HAS_NEWER_VERSION` relationships. **Update Mode:** When `update_mode=True`, the ingestion process: 1. Finds the latest version of the document by `path` 2. Deletes all existing `StructureNode` and `InfoUnit` descendants 3. Re-inserts the new structure with the same version number 4. Does not create a new version entry **Concurrency:** Stage 2 uses `get_neo4j_sync_semaphore()` (size: `neo4j_sync_concurrency`, default: 8) to bound concurrent dispatches to `asyncio.to_thread()` for the synchronous Neo4j driver operations. The sync driver is used for Stage 2 because the Neo4j Python driver's synchronous API is used for document ingestion (the async driver is reserved for Stages 3 and 4). --- ### Stage 3: Annotation (`run_annotation`) **Purpose:** Classify each `StructureNode` against registered extraction models using an LLM, writing annotation decisions to Neo4j. **Input:** Document name (must already exist in Neo4j from Stage 2). **Output:** `StageResult` with per-node annotation counts and errors. Neo4j graph updated with annotation subgraph. **Architecture:** The annotation module (`annotation/`) provides a two-step LLM pipeline per node: **Step 1 — Model Decision (`decide_model`):** The LLM receives: - The `StructureNode`'s `title` and `role` - All `InfoUnit` descriptions within the node - The **catalog of available extraction models** (loaded from themes) - **Theme descriptions** (via `THEME_DESCRIPTION` in each theme's `catalog.py`) - Optional user-provided `context_instructions` The LLM returns a `ModelDecision` with: - `matched_model_class`: CamelCase name of the best-matching Pydantic model (or `NULL` for no match) - `confidence`: qualitative confidence level - `reasoning`: brief justification **Step 2 — Decision Formatting (`format_decision`):** A second LLM call validates and formats the decision, ensuring the model class name exactly matches a registered class in the model catalog. **Neo4j Annotation Subgraph:** ``` (:StructureNode) -[:HAS_MODEL_DECISION]-> (:ModelDecision) (:ModelDecision) -[:MATCHES_MODEL]-> (:CatalogModel) (:ModelDecision) -[:BELONGS_TO_THEME]-> (:Theme) (:CatalogModel) -[:HAS_FIELD]-> (:ModelField) (:ModelField) -[:HAS_ENTITY_LABEL]-> (:EntityLabel) ``` **Theme System:** Themes organize extraction models into domain-specific groups. Each theme has: - A path (e.g., `"pharma_operations/batch_manufacturing"`) - A `catalog.py` file listing all model classes in the theme - A `THEME_DESCRIPTION` string used by the annotation LLM for theme selection - Model files defining Pydantic schemas with `json_schema_extra` annotations for entity labels, field relationships, and instance keys Built-in themes are registered in `utils/theme_registry.py`. User themes are loaded from `extra_models_paths` at configure time. The `enabled_base_themes` and `enabled_user_themes` configuration parameters act as whitelists. **Catalog and Theme Neo4j Setup:** Before annotation begins, `ensure_catalog_models_once()` and `ensure_theme_structure_once()` are called (idempotent, memoized) to: 1. Create `:CatalogModel` and `:ModelField` nodes for all registered models 2. Create `:Theme` nodes and link them to their models 3. Create `:EntityLabel` nodes for all entity labels defined in model fields These operations run once per pipeline run and are memoized to avoid redundant Neo4j queries. **Modes:** - **LLM Agent Mode** (default): Uses the two-step LLM pipeline for each node. - **Manual Mode** (`manual=True`): Assigns a fixed `model_class` to all qualifying nodes without LLM calls. Requires `model_class` parameter. Useful for bulk annotation of homogeneous document sets. **Resume Flags:** - `only_unannotated=True`: Skips nodes that already have a `:HAS_MODEL_DECISION` relationship, enabling partial re-runs. **Folder Documents:** When `document_name` refers to a folder (a document with `IS_COMPOSED_OF` children), all leaf descendants are resolved and annotated. Up to `parallel_docs` leaves are processed concurrently. Failures on individual leaves are logged but do not stop processing of other leaves. --- ### Stage 4: Entity Extraction (`run_entity_extraction`) **Purpose:** Extract typed domain entities from annotated `StructureNode` objects using Pydantic structured output, writing entity subgraphs to Neo4j. **Input:** Document name (must have annotated nodes from Stage 3). **Output:** `StageResult` with per-node extraction counts and errors. Neo4j graph updated with entity subgraph. **Architecture:** The entity extraction module (`entity_extraction/`) operates per annotated `StructureNode`: **Step 1 — Schema Composition (`schema_composer.py`):** For each target node, the system: 1. Reads the `ModelDecision.matched_model_class` from Neo4j 2. Resolves the Pydantic model class via `model_resolver.py` (consults the theme registry) 3. Builds a **composite schema** — a single Pydantic model that combines the primary model and any complementary models (optional nested models declared in the primary model's fields) 4. The composite schema is used for structured output via `llm.with_structured_output()` **Step 2 — LLM Extraction:** The LLM receives: - The composite Pydantic schema as structured output target - The `StructureNode`'s `InfoUnit` descriptions (the sole content representation) - The node's `title` and `role` for context - Model field descriptions from the Pydantic schema The LLM returns a populated instance of the composite schema. **Step 3 — Graph Write (`graph_mapper.py`):** The populated Pydantic instance is converted into a Neo4j subgraph with three levels of entity representation: **Level 1 — Entity Labeling:** - Fields with `json_schema_extra={"entity_label": "X"}` become `MERGE`d `:LabeledEntity {label, value, normalized_value}` nodes - Same label + same normalized value always resolves to the same node across all extractions (global deduplication) - Connected via `[:REFERENCES]` relationships **Level 2 — Field Relationships:** - Fields with `json_schema_extra={"field_relationships": [{"to_field": "...", "rel_type": "..."}]}` trigger `MERGE` relationships between source and target `:LabeledEntity` nodes **Level 3 — Instance Key Relationships:** - Fields with `json_schema_extra={"instance_key": True}` define a composite key for `:ModelInstance` deduplication - Fields with `json_schema_extra={"instance_relationships": [...]}` trigger `MERGE` of target `:ModelInstance` shells and typed relationships between instances - Enables forward references across `StructureNode` boundaries **Neo4j Entity Subgraph:** ``` (:StructureNode) -[:HAS_EXTRACTION]-> (:ExtractionResult) (:ExtractionResult) -[:USES_PRIMARY_MODEL]-> (:CatalogModel) (:ExtractionResult) -[:USES_COMPLEMENTARY_MODEL]-> (:CatalogModel) [0..*] (:ExtractionResult) -[:HAS_]-> (:ModelInstance) [nested models] (:ModelInstance | :ExtractionResult) -[:REFERENCES]-> (:LabeledEntity) (:LabeledEntity) -[:REL_TYPE]-> (:LabeledEntity) [field_relationships] (:ModelInstance) -[:REL_TYPE]-> (:ModelInstance) [instance_relationships] ``` **Triple (Fallback) Extraction:** For nodes where `ModelDecision.matched_model_class` is `NULL` (no specific domain model matched), a fallback `Triple` model extracts subject-predicate-object statements: ``` (:StructureNode) -[:HAS_EXTRACTION]-> (:ExtractionResult {model_class: "Triple"}) (:ExtractionResult) -[:HAS_ENTITY {role}]-> (:Entity) (:Entity) -[:NORMALIZED_PREDICATE {predicate_raw}]-> (:Entity) ``` Entity nodes are global singletons (MERGE by normalized value), shared across all extractions. **JSON Repair Loop:** When the LLM returns malformed JSON, the `utils/llm_repair.py` module attempts repair via a secondary LLM call (using `repair_llm`, which defaults to the main `llm` if not configured separately). This is transparent to the calling code. **Resume Flags:** - `only_unextracted=True`: Skips nodes that already have a `:HAS_EXTRACTION` relationship, enabling partial re-runs. --- ### Tabular Pipeline (`run_tabular_pipeline`) **Purpose:** Ingest CSV, XLSX, and XLS files directly into Neo4j, bypassing Stages 0–4 entirely. **Input:** Directory of raw tabular files (`.csv`, `.xlsx`, `.xls`). **Output:** `StageResult` with per-file success/failure counts. Neo4j graph updated with Document, Table, and Row nodes. **Architecture:** The tabular pipeline (`tabular/`) uses a **LangGraph**-based state machine (`tabular/graph.py`) with the following nodes: 1. **`load_sheets`** — Reads the file (CSV via pandas, XLSX/XLS via openpyxl), extracts headers and a 5-row preview, and stores per-sheet data in the `TabularState`. 2. **`decide_model`** — Makes one LLM call per sheet to decide which extraction model to use. The LLM receives the sheet headers, preview rows, and the catalog of available models. Returns the `matched_model_class`. 3. **`map_columns`** — Makes one LLM call per sheet to map column names to model fields. The LLM receives the sheet headers, the selected model's field definitions, and returns a column-to-field mapping. 4. **`write_tabular`** — Writes the `Table` and `Row` `StructureNode` subgraph directly to Neo4j. Each row becomes a `:StructureNode {role: "row"}` with `InfoUnit` children for each mapped cell value. **Per-File Process:** For each tabular file: 1. Create `:Document` node and folder hierarchy in Neo4j (single transaction) 2. Run the LangGraph state machine for each sheet 3. Store raw file binary in MongoDB (if storage backend is configured) **NormalizationEngine:** When `normalization_enabled=True` (default: `False`), the `NormalizationEngine` (in `tabular/`) performs post-extraction normalization for nested model fields. It batches entries (configurable via `normalization_batch_size`, default: 5) and uses a dedicated LLM (`normalization_llm`, falls back to main `llm`) to normalize values into consistent formats. **Tabular Neo4j Structure:** ``` (:Document) -[:HAS_STRUCTURE]-> (:StructureNode {role: "table"}) (:StructureNode {role: "table"}) -[:HAS_CHILD]-> (:StructureNode {role: "row"}) (:StructureNode {role: "row"}) -[:HAS_INFO_UNIT]-> (:InfoUnit) ``` Each row node has a `row_index` property (0-based position within the table). **Auto-Detection in Mixed Folders:** When `run_pipeline()` detects tabular files in `input_raw` alongside non-tabular files, it: 1. Runs the tabular pipeline first for all tabular files 2. Proceeds with the unstructured pipeline (Stages 0–4) for non-tabular files 3. Both tracks share the same Neo4j graph and versioning system **Tabular-Only Mode:** When `stages=["tabular"]` is specified, only the tabular pipeline runs. This stage cannot be combined with other stages in the `stages` parameter. --- ## 3. Async Architecture All pipeline stages are `async def`. The async architecture is built around three layers of concurrency control: ### Document-Level Concurrency `run_pipeline()` uses a per-document `asyncio.Semaphore(parallel_docs)` to bound how many documents are processed concurrently across all stages. Each document is dispatched as an independent task via `asyncio.gather()` and runs through its applicable stages sequentially. ```python document_semaphore = asyncio.Semaphore(parallel_docs) unit_results = await asyncio.gather( *[ _process_document_unit(u, document_semaphore=document_semaphore, ...) for u in units ], return_exceptions=True, ) ``` ### LLM-Level Concurrency The global `get_llm_semaphore()` (size: `llm_concurrency`, default: 4) bounds all LLM calls across all stages. Every `extract_chunk()`, annotation decision, entity extraction, and tabular mapping call acquires this semaphore before invoking the LLM. This prevents exceeding provider rate limits. ### Neo4j-Level Concurrency Two separate semaphores control Neo4j access: - **`get_neo4j_semaphore()`** (size: `neo4j_concurrency`, default: 10): Bounds concurrent Neo4j async sessions during annotation (Stage 3) and entity extraction (Stage 4). - **`get_neo4j_sync_semaphore()`** (size: `neo4j_sync_concurrency`, default: 8): Bounds concurrent dispatches to `asyncio.to_thread()` for Stage 2 (synchronous ingestion). Must be acquired/released on the event loop, never inside the worker thread. ### Per-Document Unit Processing `_process_document_unit()` processes a single `DocumentUnit` through all applicable stages. It: 1. Acquires the document semaphore for its entire duration 2. Runs stages sequentially within the unit 3. Implements soft-abort semantics: - **Stage 0/1/2 failure**: Always stops the unit (no valid artifact for subsequent stages) - **Stage 3/4 partial failure** (`nodes_failed > 0`): Stops the unit only when `on_partial_failure="abort"` (default). With `"continue"` or `"warn"`, the unit advances to its next stage despite partial failures. 4. Never propagates exceptions — returns a `UnitResult` with `fatal_error` set for uncaught exceptions ### Pre-Warming Before dispatching document units, `run_pipeline()` pre-warms shared resources: 1. Opens sync Neo4j driver and sets up schema 2. Resolves batch version for all documents 3. Ensures catalog models and theme structure exist in Neo4j 4. Initializes storage backends This reduces per-document startup latency. --- ## 4. Configuration System ### Singleton Pattern Configuration is managed by the `ScinrConfig` dataclass in `config.py` via a module-level singleton (`_config`). The public API is: - **`configure(...)`** — Sets global configuration. Must be called before any pipeline function. - **`get_config()`** — Returns the current configuration. Raises `ConfigurationError` if not configured. ### Triple Resolution All parameters follow a three-tier resolution order: 1. **Explicit argument** passed to `configure()` 2. **Environment variable** (via `os.getenv`) 3. **Hard-coded default** value Example: ```python resolved_neo4j_uri = neo4j_uri or os.getenv("NEO4J_URI", "bolt://localhost:7687") ``` ### Key Configuration Parameters | Parameter | Env Var | Default | Description | |---|---|---|---| | `llm` | `MODEL_ID` (Bedrock) | None | LangChain `BaseChatModel` instance | | `repair_llm` | — | Falls back to `llm` | Secondary LLM for JSON repair | | `neo4j_uri` | `NEO4J_URI` | `bolt://localhost:7687` | Neo4j connection URI | | `neo4j_user` | `NEO4J_USER` / `NEO4J_AUTH` | — | Neo4j username (required) | | `neo4j_password` | `NEO4J_PASSWORD` / `NEO4J_AUTH` | — | Neo4j password (required) | | `storage_backend` | `STORAGE_BACKEND` | `"none"` | `"none"`, `"mongodb"`, or `"custom"` | | `llm_concurrency` | `LLM_CONCURRENCY` | 4 | Max concurrent LLM calls | | `neo4j_concurrency` | `NEO4J_CONCURRENCY` | 10 | Max concurrent Neo4j async sessions | | `neo4j_sync_concurrency` | `NEO4J_SYNC_CONCURRENCY` | 8 | Max concurrent sync ingestion dispatches | | `extraction_batch_size` | `EXTRACTION_BATCH_SIZE` | 3 | Pages per extraction chunk | | `prompt_family` | `PROMPT_FAMILY` | `"generic"` | Prompt variant family | | `prompt_caching_enabled` | `PROMPT_CACHING_ENABLED` | `True` | Bedrock prompt caching | | `normalization_enabled` | `NORMALIZATION_ENABLED` | `False` | Enable tabular normalization | | `normalization_batch_size` | `NORMALIZATION_BATCH_SIZE` | 5 | Max entries per normalization batch | ### LLM Configuration The library supports any LangChain `BaseChatModel` that implements `with_structured_output()`: - `ChatOpenAI` (OpenAI) - `ChatBedrockConverse` (AWS Bedrock) - `ChatAnthropic` (Anthropic/Claude) - `ChatOllama` (Ollama) - Any other LangChain chat model with structured output support When `MODEL_ID` is set (and no explicit `llm` is passed), `configure()` automatically creates a `ChatBedrockConverse` instance with connection pool sizing relative to `llm_concurrency`. ### Prompt Family Three prompt families are supported via `PromptFamily` enum: | Family | Description | Best For | |---|---|---| | `GENERIC` | Simplified, model-agnostic prompts | All LLM families (default) | | `CLAUDE` | XML-structured instructions, multi-step protocols, internal checklists | Claude/Sonnet models | | `GPT_REASONING` | Markdown section headers, goal-based language, no CoT elicitation | OpenAI reasoning models (GPT-5.5, o3, o4-mini) | Each family has dedicated prompt files in `annotation/` and `entity_extraction/` modules (e.g., `prompts_claude.py`, `prompts_gpt_reasoning.py`, `prompts_generic.py`). ### Post-Configure Reset `configure()` resets dependent lazy singletons after setting the config: - Theme registry - Async Neo4j driver singleton - MongoDB client - Catalog memoization - LLM/Neo4j semaphores (manual reset required via `reset_*_semaphore()`) --- ## 5. Module Structure ``` scinr.newton/ ├── __init__.py # Package exports ├── config.py # ScinrConfig, configure(), get_config(), semaphore helpers ├── pipeline.py # run_pipeline() orchestrator + _process_document_unit() ├── pipeline_units.py # DocumentUnit discovery (raw_file, extraction_json, ingestion_json, pre_ingested) ├── results.py # PipelineResult, StageResult, DocumentResult dataclasses ├── exceptions.py # ScinrError hierarchy ├── cli.py # CLI entry point (Typer-based) │ ├── annotation/ # Stage 3: LLM classification │ ├── agent.py # run_annotation_agent(), run_manual_annotation() │ ├── models.py # AnnotationDecision, ModelDecision Pydantic models │ ├── neo4j_ops.py # fetch_nodes_to_annotate(), write_annotation(), catalog/theme setup │ ├── nodes.py # process_single_annotation_node() — per-node LLM pipeline │ ├── prompts.py # Prompt family dispatcher │ ├── prompts_generic.py # GENERIC family prompts │ ├── prompts_claude.py # CLAUDE family prompts │ ├── prompts_gpt_reasoning.py # GPT_REASONING family prompts │ └── state.py # AnnotationState dataclass │ ├── converters/ # File format converters │ ├── base.py # BaseConverter ABC, IntermediateDocument, IntermediatePage │ ├── registry.py # Extension-to-converter map, apply_converter_overrides() │ ├── main.py # convert_one(), convert_folder() — parallel conversion │ ├── pdf.py # PdfConverter (pdfplumber + Mistral OCR) │ ├── pdf_splitter.py # Structural PDF partitioning for OCR chunking │ ├── docx.py # DocxConverter (python-docx) │ ├── pptx.py # PptxConverter (python-pptx) │ ├── xlsx.py # XlsxConverter (openpyxl + pandas) │ ├── csv.py # CsvConverter (pandas) │ ├── html.py # HtmlConverter (BeautifulSoup) │ ├── text.py # TextConverter (stdlib) │ ├── api_json.py # ApiJsonConverter (stdlib) │ ├── api_xml.py # ApiXmlConverter (stdlib) │ └── config.py # Converter-specific configuration │ ├── entity_extraction/ # Stage 4: entity extraction │ ├── agent.py # run_entity_extraction_agent() │ ├── graph_mapper.py # write_extraction_subgraph(), write_triple_subgraph() │ ├── model_resolver.py # resolve_model_class() — theme registry lookup │ ├── neo4j_ops.py # fetch_extraction_targets() │ ├── nodes.py # process_single_extraction_target() — per-node LLM pipeline │ ├── prompts.py # Prompt family dispatcher │ ├── prompts_generic.py # GENERIC family prompts │ ├── prompts_claude.py # CLAUDE family prompts │ ├── prompts_gpt_reasoning.py # GPT_REASONING family prompts │ ├── schema_composer.py # Composite schema construction from primary + complementary models │ └── state.py # EntityExtractionState dataclass │ ├── extraction/ # Stage 1: chunking │ ├── extraction.py # extract_chunk() — LLM call for one chunk │ ├── compact_extraction.py # compact_extraction() — merge chunk results into document tree │ └── prompts/ # Extraction-specific prompt templates │ ├── ingest/ # Stage 2: Neo4j ingestion │ ├── config.py # get_driver(), get_async_driver() — Neo4j driver singletons │ ├── loader.py # load_documents(), load_files(), load_folder(), version resolution │ ├── nodes.py # insert_document(), insert_structure_node(), insert_info_unit() │ └── schema.py # setup_schema() — constraints and indexes │ ├── models/ # Core Pydantic models │ ├── document_structure.py # Document, StructureNode, InfoUnit, DocumentStructure, NodeRole │ └── base.py # StrictModel base class │ ├── prompts/ # System prompt templates │ ├── system_prompt.py # Prompt family dispatcher │ ├── system_prompt_generic.py # GENERIC system prompts │ ├── system_prompt_claude.py # CLAUDE system prompts │ └── system_prompt_gpt_reasoning.py # GPT_REASONING system prompts │ ├── stages/ # Stage orchestrators │ ├── __init__.py # Re-exports all public stage functions │ ├── preprocess.py # run_preprocess() │ ├── extraction.py # run_extraction() │ ├── ingestion.py # run_ingestion(), apply_replacement(), preflight_check_replaces() │ ├── annotation.py # run_annotation() │ ├── entity_extraction.py # run_entity_extraction() │ └── tabular.py # run_tabular_pipeline() │ ├── storage/ # Storage backends │ ├── base.py # RawFileRepository, PageRepository ABCs │ ├── factory.py # get_storage() — backend factory │ ├── null.py # NullRawFileRepository, NullPageRepository (no-op) │ ├── config.py # Storage configuration │ ├── models.py # RawFileRecord, PageRecord Pydantic models │ └── mongodb/ # MongoDB implementation │ ├── client.py # Motor async client singleton │ ├── raw_files.py # MongoDBRawFileRepository (GridFS) │ └── pages.py # MongoDBPageRepository (collection) │ ├── tabular/ # Tabular pipeline (Stage 5) │ ├── agent.py # run_tabular_agent(), run_tabular_agent_sync() │ ├── graph.py # LangGraph state machine (load_sheets → decide_model → map_columns → write) │ ├── state.py # TabularState — LangGraph state schema │ ├── reader.py # CSV/XLSX/XLS file reading │ ├── neo4j_ops.py # Tabular Neo4j writes │ ├── nodes.py # Per-sheet tabular node processing │ ├── prompts.py # Prompt family dispatcher │ ├── prompts_generic.py # GENERIC family prompts │ ├── prompts_claude.py # CLAUDE family prompts │ ├── prompts_gpt_reasoning.py # GPT_REASONING family prompts │ ├── models.py # Tabular-specific Pydantic models │ └── ... │ └── utils/ # Utilities ├── theme_registry.py # Theme discovery, model loading, get_theme_registry() ├── llm_factory.py # LLM instance creation helpers ├── llm_retry.py # LLM call retry with exponential backoff ├── llm_repair.py # JSON repair loop via secondary LLM ├── neo4j_retry.py # Neo4j operation retry with exponential backoff ├── neo4j_concurrency.py # Neo4j concurrency utilities ├── document_resolver.py # resolve_leaf_document_names() — folder → leaf resolution ├── file_archiver.py # File archiving utilities ├── logging_config.py # Structured logging setup └── uid.py # Deterministic UID generation (make_uid, make_instance_uid) ``` --- ## 6. Data Flow ### Between Stages Data flows between stages through three mechanisms: | Mechanism | Direction | Description | |---|---|---| | **In-memory objects** | Stage N → Stage N+1 | `IntermediateDocument` (0→1), `Document` (1→2), document names (2→3→4) | | **Intermediate JSON files** | Stage N → disk → Stage N+1 | `*.json` (0→1), `extract-*.json` (1→2) | | **Neo4j graph** | Stage N → graph → Stage N+1 | `:Document`/`:StructureNode` (2→3→4), annotation subgraph (3→4) | ### Pipeline Data Flow (Full Run) ``` input_raw/ converter_output_dir/ extraction_output_dir/ Neo4j ├── doc1.pdf ──Stage 0──► doc1.json ──Stage 1──► extract-doc1.json ──Stage 2──► (:Document) ├── doc2.docx ──Stage 0──► doc2.json ──Stage 1──► extract-doc2.json ──Stage 2──► (:Document) └── data.csv ──tabular──► (bypassed) (bypassed) (bypassed) ──Stage 5──► (:Document) ``` ### Intermediate Directory Structure When intermediate directories are used: ``` converter_output_dir/ ├── doc1.json # Stage 0 output (IntermediateDocument) ├── doc2.json └── subfolder/ └── doc3.json extraction_output_dir/ ├── extract-doc1.json # Stage 1 output (Document) ├── extract-doc2.json └── subfolder/ └── extract-doc3.json ``` The subdirectory structure mirrors the input folder hierarchy, preserving the `folder_path` metadata. ### Stage Skipping Stages can be skipped by providing input from a later stage: | Skip To | Parameter | Effect | |---|---|---| | Stage 1 | `extraction_input_dir` | Skips Stage 0; reads JSON from disk | | Stage 2 | `ingestion_input_dir` | Skips Stages 0 and 1; reads `extract-*.json` from disk | | Stage 3 | `document_names` | Skips Stages 0–2; uses already-ingested documents | | Stage 3 | `document_names_dir` | Skips Stages 0–2; reads names from `extract-*.json` files | The `stages` parameter controls which stages execute. When `stages=["annotation", "entity_extraction"]`, only Stages 3 and 4 run, and `document_names` or `document_names_dir` must be provided. ### Independent Stage Execution Each stage function can be called independently: ```python # Stage 0 only result, docs = await run_preprocess(input_raw="files/", output_dir="data/json/") # Stage 1 only (from disk) result, docs = await run_extraction(input_folder="data/json/", output_folder="data/extract/") # Stage 2 only (from disk) result = await run_ingestion(output_folder="data/extract/") # Stage 3 only (from Neo4j) result = await run_annotation(document_name="MyDocument") # Stage 4 only (from Neo4j) result = await run_entity_extraction(document_name="MyDocument") # Tabular only result = await run_tabular_pipeline(input_raw="files/") ``` --- ## 7. Neo4j Schema ### Node Labels | Label | Description | Primary Key | |---|---|---| | `:Document` | Ingested document (versioned) | `(path, version)` composite | | `:StructureNode` | Structural division (section, table, row, etc.) | `id` | | `:InfoUnit` | Semantic information unit | `uid` | | `:ModelDecision` | Annotation decision for a node | — | | `:CatalogModel` | Registered Pydantic extraction model | `name` | | `:ModelField` | Field of a CatalogModel | `(name, model)` composite | | `:EntityLabel` | Schema-level entity label singleton | `label` | | `:Theme` | Extraction model theme | — | | `:ExtractionResult` | Entity extraction result | `uid` | | `:ModelInstance` | Extracted model instance (nested) | `uid` | | `:LabeledEntity` | Globally deduplicated entity | `(label, normalized_value)` | | `:Entity` | Triple extraction entity (fallback) | `uid` | ### Relationship Types | Type | Source → Target | Description | |---|---|---| | `HAS_STRUCTURE` | Document → StructureNode | Root structural nodes | | `HAS_CHILD` | StructureNode → StructureNode | Hierarchical nesting | | `HAS_INFO_UNIT` | StructureNode → InfoUnit | Semantic content | | `IS_COMPOSED_OF` | Document → Document | Folder hierarchy | | `HAS_NEWER_VERSION` | Document → Document | Version succession | | `HAS_MODEL_DECISION` | StructureNode → ModelDecision | Annotation result | | `MATCHES_MODEL` | ModelDecision → CatalogModel | Selected model | | `BELONGS_TO_THEME` | ModelDecision → Theme | Theme assignment | | `HAS_FIELD` | CatalogModel → ModelField | Model schema | | `HAS_ENTITY_LABEL` | ModelField → EntityLabel | Entity label declaration | | `HAS_EXTRACTION` | StructureNode → ExtractionResult | Extraction output | | `USES_PRIMARY_MODEL` | ExtractionResult → CatalogModel | Primary model used | | `USES_COMPLEMENTARY_MODEL` | ExtractionResult → CatalogModel | Complementary model | | `HAS_` | ExtractionResult → ModelInstance | Nested model instance | | `REFERENCES` | ModelInstance → LabeledEntity | Entity reference | | `HAS_ENTITY` | ExtractionResult → Entity | Triple extraction entity | | `NORMALIZED_PREDICATE` | Entity → Entity | Triple relationship | ### Constraints and Indexes See `ingest/schema.py` for the complete DDL. Key constraints: - **10 unique constraints** ensuring node identity and preventing duplicates - **9 regular indexes** for query performance - **2 fulltext indexes** for semantic search on InfoUnit content --- ## 8. Storage Backends The storage layer abstracts raw file and converted page persistence behind repository interfaces: ### Backend Types | Backend | Description | Configuration | |---|---|---| | `"none"` | No persistence; null repositories | Default | | `"mongodb"` | MongoDB with GridFS for binaries | `mongodb_uri`, `mongodb_database`, etc. | | `"custom"` | User-provided repository pair | `custom_storage=(raw_repo, page_repo)` | ### MongoDB Structure When `storage_backend="mongodb"`: - **GridFS bucket** (`mongodb_gridfs_bucket`, default: `"raw_binaries"`): Stores raw file binaries with metadata (filename, content_type, folder_path) - **`raw_files` collection** (`mongodb_raw_files_collection`): Raw file metadata records - **`converted_pages` collection** (`mongodb_pages_collection`): Converted page records with markdown content, images, and dimensions ### Repository Interfaces - **`RawFileRepository`** (ABC): `store(filename, content, content_type, folder_path)` → ObjectId - **`PageRepository`** (ABC): `store(document_name, pages, folder_path)` → list of ObjectIds Null implementations (`NullRawFileRepository`, `NullPageRepository`) are used when `storage_backend="none"`, returning `None` for all operations. --- ## 9. Prompt System The prompt system supports three families, each with dedicated prompt files per stage: ### Prompt Resolution 1. `get_prompt_family()` returns the configured `PromptFamily` enum value 2. Each stage's `prompts.py` dispatcher selects the appropriate prompt file based on the family 3. System prompts are resolved from `prompts/system_prompt_*.py` 4. Stage-specific prompts are resolved from their respective module's `prompts_*.py` files ### Prompt File Convention ``` / ├── prompts.py # Dispatcher: selects file based on PromptFamily ├── prompts_generic.py # GENERIC prompts (default, model-agnostic) ├── prompts_claude.py # CLAUDE prompts (XML-structured, extended reasoning) └── prompts_gpt_reasoning.py # GPT_REASONING prompts (Markdown, goal-based) ``` ### Bedrock Prompt Caching When using `ChatBedrockConverse` with `prompt_caching_enabled=True`, the `make_system_message()` function appends a `cachePoint` block to the system message, reducing token costs by ~90% on repeated calls with the same prompt. --- ## 10. Error Handling ### Exception Hierarchy All exceptions inherit from `ScinrError`: ``` ScinrError (base) ├── ConfigurationError # Misconfiguration (missing LLM, Neo4j credentials, etc.) ├── PreconditionError # Pipeline called out of order ├── ExtractionError # LLM extraction failed after retries ├── IngestionError # Neo4j write failed ├── ModelError # Pydantic model resolution failure ├── StorageError # MongoDB unavailable/misconfigured └── ConversionError # File converter failure ``` ### Retry Mechanisms - **LLM Retry** (`utils/llm_retry.py`): Exponential backoff retry for LLM calls - **Neo4j Retry** (`utils/neo4j_retry.py`): Exponential backoff retry for Neo4j operations - **JSON Repair** (`utils/llm_repair.py`): Secondary LLM call to repair malformed JSON output - **Bedrock Retry** (`utils/bedrock_retry.py`): Bedrock-specific retry with service-aware backoff ### Partial Failure Handling The `on_partial_failure` parameter controls pipeline behavior when a stage reports failures: | Value | Behavior | |---|---| | `"abort"` | Stops the failing document's remaining stages (default) | | `"continue"` | Document advances to next stage silently | | `"warn"` | Document advances with per-document warning logged | Note: `on_partial_failure` only affects annotation (Stage 3) and entity extraction (Stage 4) partial failures. Stage 0/1/2 failures always stop the document (no valid artifact to continue with). --- ## 11. Result Types ### Result Hierarchy ``` PipelineResult ├── success: bool ├── total_duration_seconds: float ├── stages_executed: list[str] ├── preprocess: StageResult | None ├── extraction: StageResult | None ├── ingestion: StageResult | None ├── annotation: StageResult | None ├── entity_extraction: StageResult | None └── tabular: StageResult | None StageResult ├── stage: str ├── success: bool ├── documents: list[DocumentResult] ├── total_processed: int ├── total_failed: int ├── duration_seconds: float └── errors: list[str] DocumentResult ├── document_name: str ├── nodes_processed: int ├── nodes_failed: int └── errors: list[str] DeletionResult ├── path: str ├── version: int | None ├── found: bool ├── versions_deleted: list[int] ├── documents_deleted: int ├── structure_nodes_deleted: int ├── info_units_deleted: int ├── model_decisions_deleted: int ├── proposed_models_deleted: int ├── proposed_fields_deleted: int ├── extraction_results_deleted: int ├── gc_entity_model_instance_deleted: int ├── gc_entity_model_instance_passes: int ├── gc_labeled_entity_deleted: int └── gc_labeled_entity_passes: int ``` ### Result Semantics - **Stages 0–2**: `nodes_processed` = 1 for success, 0 for failure (per document) - **Stages 3–4**: `nodes_processed` = number of StructureNodes processed (per document) - **Tabular**: `nodes_processed` = 1 for success, 0 for failure (per file) All result dataclasses are defined in `results.py` and provide structured, type-safe access to pipeline outcomes. --- ## File: configuration.md # Configuration Configure `scinr.newton` the ingestion pipeline using the `configure()` function, environment variables, or a combination of both. --- ## Configuration Resolution scinr uses a **triple-resolution** system. For every setting, the effective value is determined by the following priority (highest to lowest): 1. **Explicit argument** passed to `configure()` 2. **Environment variable** set in the process environment or loaded from a `.env` file 3. **Hard-coded default** built into the library This means you can set sensible defaults via environment variables and override individual values at runtime with `configure()`, or vice versa. ```python # Example: env var sets concurrency to 4, but configure() overrides to 8 # $ export LLM_CONCURRENCY=4 configure(llm_concurrency=8) # final value: 8 ``` --- ## Environment Variables All environment variables are optional unless otherwise noted. They are read at configuration time (when `configure()` is first called or when the config is first accessed). ### LLM / Model | Variable | Default | Description | | :--- | :--- | :--- | | `MODEL_ID` | *(required if no `llm` arg)* | Model ID for the primary LLM. For AWS Bedrock, use ARNs such as `us.anthropic.claude-sonnet-4-6`. | | `REPAIR_MODEL_ID` | *(falls back to `MODEL_ID`)* | Model ID used for repair and retry LLM calls. Can be a cheaper/faster model (e.g. `us.anthropic.claude-haiku-3`). | | `AWS_DEFAULT_REGION` | `us-east-1` | AWS region for Bedrock calls. | | `MAX_TOKENS` | `65536` | Maximum tokens for Bedrock LLM calls. | | `LLM_CONCURRENCY` | `4` | Maximum number of concurrent LLM calls. | ### Neo4j | Variable | Default | Description | | :--- | :--- | :--- | | `NEO4J_URI` | `bolt://localhost:7687` | Bolt URI for the Neo4j instance. | | `NEO4J_USER` | *(required)* | Neo4j database username. **Note:** previous versions used `NEO4J_USERNAME` — this was renamed to `NEO4J_USER`. | | `NEO4J_PASSWORD` | *(required)* | Neo4j user password. | | `NEO4J_AUTH` | *(fallback "user/password")* | Alternative authentication format as a single `user/password` string. Used if `NEO4J_USER` and `NEO4J_PASSWORD` are not both set. | | `NEO4J_CONCURRENCY` | `10` | Maximum async Neo4j concurrency. | | `NEO4J_SYNC_CONCURRENCY` | `8` | Maximum sync Neo4j concurrency. | ### Storage (MongoDB) | Variable | Default | Description | | :--- | :--- | :--- | | `STORAGE_BACKEND` | `none` | Storage backend: `none` (no persistence), `mongodb`, or `custom`. | | `MONGODB_URI` | `mongodb://localhost:27017` | MongoDB connection string. | | `MONGODB_DATABASE` | `scinr` | MongoDB database name. | | `MONGODB_RAW_FILES_COLLECTION` | `raw_files` | Collection name for raw file metadata. | | `MONGODB_PAGES_COLLECTION` | `converted_pages` | Collection name for converted document pages. | | `MONGODB_GRIDFS_BUCKET` | `raw_binaries` | GridFS bucket name for binary file storage. | ### PDF / Mistral OCR | Variable | Default | Description | | :--- | :--- | :--- | | `MISTRAL_API_KEY` | `None` | Mistral API key for PDF OCR extraction. Required to process PDF files. | | `MISTRAL_OCR_SAFE_MAX_PAGES` | `900` | Maximum number of pages before OCR becomes mandatory. | | `MISTRAL_OCR_SAFE_MAX_BYTES` | `47185920` (45 MiB) | Maximum file size in bytes before OCR is required. | | `MISTRAL_OCR_MAX_RETRIES` | `3` | Number of retry attempts for OCR failures. | | `MISTRAL_OCR_RETRY_BACKOFF_SECONDS` | `2.0` | Base backoff in seconds between retries. | | `MISTRAL_OCR_CHUNK_CONCURRENCY` | `1` | Maximum concurrent OCR chunk processing. | | `MISTRAL_OCR_ERROR_STRATEGY` | `fail_fast` | Error handling: `fail_fast` (abort on first error) or `best_effort` (continue and collect what is possible). | ### Pipeline | Variable | Default | Description | | :--- | :--- | :--- | | `PROMPT_CACHING_ENABLED` | `true` | Enable prompt caching. Currently effective for AWS Bedrock; ignored for other providers. | | `EXTRACTION_BATCH_SIZE` | `1` | Number of pages per extraction chunk. | | `PROMPT_FAMILY` | `generic` | Prompt template family: `generic`, `claude`, or `gpt_reasoning`. | | `SCINR_EXTRA_MODELS_PATHS` | `""` (empty) | Colon-separated list of extra model package paths. | ### Normalization | Variable | Default | Description | | :--- | :--- | :--- | | `NORMALIZATION_ENABLED` | `false` | Enable tabular data normalization via LLM. | | `NORMALIZATION_BATCH_SIZE` | `5` | Batch size for normalization LLM calls. | --- ## Programmatic Configuration The `configure()` function is the primary way to set up scinr at runtime. It accepts keyword arguments organized by category. All parameters are optional — omitting a parameter falls back to the environment variable or hard-coded default. ```python from scinr.newton import configure ``` ### LLM Parameters | Parameter | Type | Description | | :--- | :--- | :--- | | `llm` | `Any \| None` | Pre-constructed LLM client instance. When provided, bypasses `MODEL_ID` and AWS Bedrock auto-configuration. | | `repair_llm` | `Any \| None` | Separate LLM client for repair/retry operations. Falls back to `llm` if not provided. | ### Neo4j Parameters | Parameter | Type | Description | | :--- | :--- | :--- | | `neo4j_uri` | `str \| None` | Bolt URI for the Neo4j instance (e.g. `bolt://localhost:7687`). | | `neo4j_user` | `str \| None` | Neo4j username. | | `neo4j_password` | `str \| None` | Neo4j password. | ### Models / Themes Parameters | Parameter | Type | Description | | :--- | :--- | :--- | | `enabled_base_themes` | `list[ThemePath \| str] \| None` | List of base themes to enable for extraction. | | `enabled_user_themes` | `list[str] \| None` | List of user-defined themes to enable. | | `extra_models_paths` | `list[str \| Path] \| None` | Additional paths to model packages. | ### Storage Parameters | Parameter | Type | Description | | :--- | :--- | :--- | | `storage_backend` | `Literal["none", "mongodb", "custom"] \| None` | Storage backend type. `none` = no persistence, `mongodb` = MongoDB, `custom` = user-provided storage. | | `mongodb_uri` | `str \| None` | MongoDB connection string. | | `mongodb_database` | `str \| None` | MongoDB database name. | | `mongodb_raw_files_collection` | `str \| None` | Collection for raw file metadata. | | `mongodb_pages_collection` | `str \| None` | Collection for converted pages. | | `mongodb_gridfs_bucket` | `str \| None` | GridFS bucket for binary storage. | | `custom_storage` | `tuple \| None` | Custom storage backend tuple (driver, connection). | ### Converter Parameters | Parameter | Type | Description | | :--- | :--- | :--- | | `extra_converters` | `dict[str, type] \| None` | Dictionary mapping file extensions to converter classes. | ### PDF / Mistral OCR Parameters | Parameter | Type | Description | | :--- | :--- | :--- | | `mistral_api_key` | `str \| None` | Mistral API key for PDF OCR. | | `mistral_ocr_safe_max_pages` | `int \| None` | Max pages before OCR is required. | | `mistral_ocr_safe_max_bytes` | `int \| None` | Max file size (bytes) before OCR is required. | | `mistral_ocr_max_retries` | `int \| None` | OCR retry count. | | `mistral_ocr_retry_backoff_seconds` | `float \| None` | Retry backoff in seconds. | | `mistral_ocr_chunk_concurrency` | `int \| None` | Concurrent OCR chunk processing. | | `mistral_ocr_error_strategy` | `Literal["fail_fast", "best_effort"] \| None` | OCR error handling strategy. | ### Pipeline Parameters | Parameter | Type | Description | | :--- | :--- | :--- | | `prompt_caching_enabled` | `bool \| None` | Enable prompt caching (Bedrock). | | `full_docstring` | `bool \| None` | Use the full class docstring (vs. only its first line) when building the model catalog description for LLM prompts (annotation stage) and Neo4j `CatalogModel.description`. | | `extraction_batch_size` | `int \| None` | Pages per extraction chunk. | | `llm_concurrency` | `int \| None` | Maximum concurrent LLM calls. | | `neo4j_concurrency` | `int \| None` | Maximum async Neo4j concurrency. | | `neo4j_sync_concurrency` | `int \| None` | Maximum sync Neo4j concurrency. | ### Logging Parameters | Parameter | Type | Description | | :--- | :--- | :--- | | `log_level` | `str` | Python log level. Default: `"INFO"`. Accepts `"DEBUG"`, `"INFO"`, `"WARNING"`, `"ERROR"`, `"CRITICAL"`. | ### Prompt Family Parameters | Parameter | Type | Description | | :--- | :--- | :--- | | `prompt_family` | `PromptFamily \| Literal["generic", "claude", "gpt_reasoning"] \| None` | Prompt template family. See [Prompt Families](#prompt-families) for details. | ### Normalization Parameters | Parameter | Type | Description | | :--- | :--- | :--- | | `normalization_enabled` | `bool \| None` | Enable tabular data normalization. | | `normalization_batch_size` | `int \| None` | Batch size for normalization LLM calls. | | `normalization_llm` | `Any \| None` | Dedicated LLM client for normalization. Falls back to `llm` if not provided. | --- ## Configuration Examples ### Minimal Setup (Environment Variables Only) The simplest approach: set environment variables and call `configure()` to let scinr pick them up automatically. `configure()` always reads `.env` via `python-dotenv`, so you never need to import dotenv manually. ```bash # .env file MODEL_ID=us.anthropic.claude-sonnet-4-6 NEO4J_URI=bolt://localhost:7687 NEO4J_USER=neo4j NEO4J_PASSWORD=your_password MISTRAL_API_KEY=your_mistral_key ``` ```python import asyncio from scinr.newton import configure, run_pipeline async def main(): # configure() reads .env automatically — no arguments needed configure() result = await run_pipeline(input_raw="./raw_docs") print(f"Pipeline: {'success' if result.success else 'failed'}") asyncio.run(main()) ``` > **Note:** `configure()` is always required before calling `run_pipeline()`. Even when all values come from environment variables, you must call `configure()` to resolve and validate the configuration. ### Full AWS Bedrock Setup Complete programmatic configuration for a production Bedrock deployment. ```python from scinr.newton import configure configure( # LLM — AWS Bedrock llm=None, # let scinr auto-create from MODEL_ID env var repair_llm=None, # use same model for repairs # Neo4j neo4j_uri="bolt://neo4j.internal:7687", neo4j_user="scinr_ingest", neo4j_password="secure_password", neo4j_concurrency=10, neo4j_sync_concurrency=8, # PDF / Mistral OCR mistral_api_key="your_mistral_key", mistral_ocr_safe_max_pages=900, mistral_ocr_safe_max_bytes=47185920, mistral_ocr_max_retries=3, mistral_ocr_error_strategy="best_effort", # Pipeline prompt_caching_enabled=True, extraction_batch_size=1, llm_concurrency=4, prompt_family="claude", # Logging log_level="INFO", ) ``` ### With Normalization Enabled Enable tabular data normalization with a dedicated LLM for the normalization step. ```python from scinr.newton import configure configure( llm_concurrency=4, prompt_family="claude", # Normalization normalization_enabled=True, normalization_batch_size=5, # normalization_llm=dedicated_llm_instance, # optional: separate LLM for normalization ) ``` ### With MongoDB Storage Persist raw files and converted pages to MongoDB. ```python from scinr.newton import configure configure( storage_backend="mongodb", mongodb_uri="mongodb://user:pass@mongo.internal:27017", mongodb_database="scinr_production", mongodb_raw_files_collection="raw_files", mongodb_pages_collection="converted_pages", mongodb_gridfs_bucket="raw_binaries", ) ``` ### Custom LLM Client Bring your own LLM client instance (e.g., a custom wrapper or non-Bedrock provider). ```python from scinr.newton import configure # Construct your own LLM client my_llm = build_my_custom_llm() configure( llm=my_llm, repair_llm=my_llm, # reuse same client for repairs neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="password", prompt_family="generic", ) ``` --- ## Using a `.env` File scinr reads standard `.env` files. Copy the provided example and fill in your values: ```bash # Copy the template cp .env.example .env # Edit with your values # $EDITOR .env ``` The `.env.example` file is provided in the project root and contains all available settings with helpful comments. Key notes: - **`NEO4J_USER`** — previous versions used `NEO4J_USERNAME`. The variable was renamed. If you have an old `.env`, update it. - **`LLM_CONCURRENCY`** — previously named `BEDROCK_CONCURRENCY`. The variable was renamed for provider-agnostic naming. - Values in `.env` are overridden by any explicit arguments to `configure()`. --- ## ScinrConfig `configure()` returns a `ScinrConfig` object that holds the resolved configuration. You can also retrieve the active configuration at any time using `get_config()`. ```python from scinr.newton import configure, get_config # Set up configuration configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="secret", llm_concurrency=8, prompt_family="claude", ) # Read back the active configuration config = get_config() print(f"Neo4j URI: {config.neo4j_uri}") print(f"LLM Concurrency: {config.llm_concurrency}") print(f"Prompt Family: {config.prompt_family}") ``` The `ScinrConfig` object is immutable after creation. To change configuration, call `configure()` again with the new values — it will produce a new `ScinrConfig` that replaces the previous one. --- ## Prompt Families The `prompt_family` parameter selects a set of prompt templates optimized for different LLM providers. | Family | Description | | :--- | :--- | | `generic` | Provider-agnostic prompts. Safe default that works with any LLM. | | `claude` | Optimized for Anthropic Claude models. Uses Claude-specific formatting and system prompt conventions. | | `gpt_reasoning` | Optimized for OpenAI reasoning models (o-series). Uses the specific message structure required by reasoning-capable models. | ### Choosing a Prompt Family - **Claude models on Bedrock** — use `"claude"` for best results. - **OpenAI o-series models** — use `"gpt_reasoning"`. - **Other providers or unsure** — use `"generic"` (the default). ```python configure(prompt_family="claude") # for Claude models configure(prompt_family="gpt_reasoning") # for OpenAI o-series configure(prompt_family="generic") # default, works everywhere ``` --- ## Complete Reference: All Settings For quick lookup, here is every configurable setting with its resolution chain: | Setting | `configure()` param | Environment Variable | Default | | :--- | :--- | :--- | :--- | | **LLM Client** | `llm` | *(none)* | `None` | | **Repair LLM Client** | `repair_llm` | *(none)* | `None` (falls back to `llm`) | | **Model ID** | *(via `llm`)* | `MODEL_ID` | *(required if no `llm`)* | | **Repair Model ID** | *(via `repair_llm`)* | `REPAIR_MODEL_ID` | Falls back to `MODEL_ID` | | **AWS Region** | *(via `llm`)* | `AWS_DEFAULT_REGION` | `us-east-1` | | **Max Tokens** | *(via `llm`)* | `MAX_TOKENS` | `65536` | | **Neo4j URI** | `neo4j_uri` | `NEO4J_URI` | `bolt://localhost:7687` | | **Neo4j User** | `neo4j_user` | `NEO4J_USER` | *(required)* | | **Neo4j Password** | `neo4j_password` | `NEO4J_PASSWORD` | *(required)* | | **Neo4j Auth** | *(derived)* | `NEO4J_AUTH` | Fallback format | | **Neo4j Async Concurrency** | `neo4j_concurrency` | `NEO4J_CONCURRENCY` | `10` | | **Neo4j Sync Concurrency** | `neo4j_sync_concurrency` | `NEO4J_SYNC_CONCURRENCY` | `8` | | **LLM Concurrency** | `llm_concurrency` | `LLM_CONCURRENCY` | `4` | | **Storage Backend** | `storage_backend` | `STORAGE_BACKEND` | `none` | | **MongoDB URI** | `mongodb_uri` | `MONGODB_URI` | `mongodb://localhost:27017` | | **MongoDB Database** | `mongodb_database` | `MONGODB_DATABASE` | `scinr` | | **MongoDB Raw Files** | `mongodb_raw_files_collection` | `MONGODB_RAW_FILES_COLLECTION` | `raw_files` | | **MongoDB Pages** | `mongodb_pages_collection` | `MONGODB_PAGES_COLLECTION` | `converted_pages` | | **MongoDB GridFS** | `mongodb_gridfs_bucket` | `MONGODB_GRIDFS_BUCKET` | `raw_binaries` | | **Custom Storage** | `custom_storage` | *(none)* | `None` | | **Mistral API Key** | `mistral_api_key` | `MISTRAL_API_KEY` | `None` | | **OCR Max Pages** | `mistral_ocr_safe_max_pages` | `MISTRAL_OCR_SAFE_MAX_PAGES` | `900` | | **OCR Max Bytes** | `mistral_ocr_safe_max_bytes` | `MISTRAL_OCR_SAFE_MAX_BYTES` | `47185920` | | **OCR Max Retries** | `mistral_ocr_max_retries` | `MISTRAL_OCR_MAX_RETRIES` | `3` | | **OCR Backoff** | `mistral_ocr_retry_backoff_seconds` | `MISTRAL_OCR_RETRY_BACKOFF_SECONDS` | `2.0` | | **OCR Concurrency** | `mistral_ocr_chunk_concurrency` | `MISTRAL_OCR_CHUNK_CONCURRENCY` | `1` | | **OCR Error Strategy** | `mistral_ocr_error_strategy` | `MISTRAL_OCR_ERROR_STRATEGY` | `fail_fast` | | **Prompt Caching** | `prompt_caching_enabled` | `PROMPT_CACHING_ENABLED` | `true` | | **Full Docstring** | `full_docstring` | `FULL_DOCSTRING` | `true` | | **Extraction Batch** | `extraction_batch_size` | `EXTRACTION_BATCH_SIZE` | `1` | | **Prompt Family** | `prompt_family` | `PROMPT_FAMILY` | `generic` | | **Extra Models Paths** | `extra_models_paths` | `SCINR_EXTRA_MODELS_PATHS` | `""` | | **Enabled Base Themes** | `enabled_base_themes` | *(none)* | `None` | | **Enabled User Themes** | `enabled_user_themes` | *(none)* | `None` | | **Extra Converters** | `extra_converters` | *(none)* | `None` | | **Normalization** | `normalization_enabled` | `NORMALIZATION_ENABLED` | `false` | | **Normalization Batch** | `normalization_batch_size` | `NORMALIZATION_BATCH_SIZE` | `5` | | **Normalization LLM** | `normalization_llm` | *(none)* | `None` (falls back to `llm`) | | **Log Level** | `log_level` | *(none)* | `"INFO"` | --- ## File: getting-started.md # Getting Started This guide walks you through installing `scinr`, configuring your environment, and running your first document ingestion pipeline end-to-end. --- ## Installation ### Core Package Install `scinr` with `pip`: ```bash pip install scinr ``` Or with `uv`: ```bash uv add scinr ``` ### Optional Extras `scinr` ships with optional extras for different LLM providers, storage backends, and development tooling. Install only what you need: ```bash # AWS Bedrock (recommended — includes langchain-aws and boto3) pip install "scinr[bedrock]" # OpenAI (includes langchain-openai) pip install "scinr[openai]" # Ollama (includes langchain-ollama) pip install "scinr[ollama]" # MongoDB storage (includes motor and pymongo) pip install "scinr[mongodb]" # Documentation tooling (mkdocs, mkdocstrings, griffe, ruff) pip install "scinr[docs]" # Development tooling (pytest, ruff, mypy) pip install "scinr[dev]" # Multiple extras at once pip install "scinr[bedrock,mongodb,dev]" ``` With `uv`: ```bash uv add "scinr[bedrock]" uv add "scinr[bedrock,mongodb]" ``` --- ## Prerequisites Before running the pipeline, ensure you have the following: ### Required 1. **Python 3.11+** — `scinr` requires Python 3.11 or later. 2. **Neo4j 5.0+** — A running Neo4j instance accessible via the Bolt protocol. You can run Neo4j locally with Docker: ```bash docker run -p 7687:7687 -p 7474:7474 \ -e NEO4J_AUTH=neo4j/your_password \ neo4j:5 ``` 3. **LLM credentials** — depending on your provider: - **AWS Bedrock**: AWS credentials configured via `~/.aws/credentials`, environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`), or an IAM role. The model is selected via the `MODEL_ID` environment variable (e.g., `us.anthropic.claude-sonnet-4-6`). - **OpenAI**: An `OPENAI_API_KEY` environment variable. - **Ollama**: A locally running Ollama instance (`ollama serve`) with the desired model pulled (`ollama pull llama3`). - **Any LangChain-compatible model**: You can pass a `BaseChatModel` instance directly to `configure()`. ### Optional 4. **MongoDB 4.6+** — Required only if you want persistent storage of raw files and converted pages. Run locally with Docker: ```bash docker run -p 27017:27017 mongo:7 ``` Without MongoDB, `scinr` operates in memory-only mode (`storage_backend=none`), which is perfectly fine for most workflows. 5. **Mistral API key** — Required to process PDF files with OCR. Obtain a key from [Mistral AI](https://console.mistral.ai/). Without it, PDFs can still be processed with `pdfplumber` (text-based extraction, no OCR). --- ## Environment Setup `scinr` reads configuration from environment variables. The recommended approach is to create a `.env` file from the provided template. ### Step 1: Create `.env` from the template ```bash cp .env.example .env ``` ### Step 2: Fill in your values Open `.env` and set the values for your environment. Here is what the template looks like and what to fill in: ```ini # ─── LLM (AWS Bedrock) ────────────────────────────────────────────────────── AWS_DEFAULT_REGION=us-east-1 MODEL_ID=us.anthropic.claude-sonnet-4-6 REPAIR_MODEL_ID=us.anthropic.claude-haiku-3 PROMPT_CACHING_ENABLED=true # ─── Neo4j ────────────────────────────────────────────────────────────────── NEO4J_URI=bolt://localhost:7687 NEO4J_USER=neo4j NEO4J_PASSWORD=your_password # ─── PDF Conversion (Mistral OCR) ─────────────────────────────────────────── MISTRAL_API_KEY=your_mistral_api_key # ─── Storage (optional) ───────────────────────────────────────────────────── STORAGE_BACKEND=none MONGODB_URI=mongodb://localhost:27017 MONGODB_DATABASE=scinr # ─── Pipeline ─────────────────────────────────────────────────────────────── EXTRACTION_BATCH_SIZE=1 LLM_CONCURRENCY=4 ``` ### Required fields for a first run | Variable | What to set | | :--- | :--- | | `MODEL_ID` | Your LLM model identifier. For Bedrock: `us.anthropic.claude-sonnet-4-6`. Not used if you pass `llm=` directly to `configure()`. | | `NEO4J_USER` | Your Neo4j username (usually `neo4j`). | | `NEO4J_PASSWORD` | Your Neo4j password. | ### Optional fields | Variable | What to set | | :--- | :--- | | `AWS_DEFAULT_REGION` | AWS region for Bedrock (default: `us-east-1`). | | `REPAIR_MODEL_ID` | A cheaper/faster model for JSON repair (default: falls back to `MODEL_ID`). | | `MISTRAL_API_KEY` | Mistral API key for PDF OCR. | | `STORAGE_BACKEND` | `none` (default) or `mongodb` for persistent storage. | | `PROMPT_FAMILY` | `generic` (default), `claude`, or `gpt_reasoning`. | > **Note:** `python-dotenv` is included as a core dependency. When you call `configure()`, it automatically loads variables from a `.env` file in your current working directory. You do not need to import `dotenv` manually. --- ## First Run ### Step 1: Prepare your documents Create a directory and place some documents in it. `scinr` supports the following formats: | Format | Extension | Notes | | :--- | :--- | :--- | | PDF | `.pdf` | Text-based via `pdfplumber`; OCR via Mistral API | | Word | `.docx` | Full text + structure extraction | | Excel | `.xlsx`, `.xls` | Routed to tabular pipeline automatically | | PowerPoint | `.pptx` | Slide text extraction | | CSV | `.csv` | Routed to tabular pipeline automatically | | HTML | `.html` | Cleaned and parsed | | JSON | `.json` | API responses, structured data | | Text | `.txt`, `.md` | Plain text | ```bash mkdir -p raw_docs # Place your .pdf, .docx, .xlsx, etc. files in raw_docs/ ``` ### Step 2: Run the pipeline The following script configures `scinr` and runs the full 6-stage pipeline on all documents in `raw_docs/`: ```python import asyncio from scinr.newton import configure, run_pipeline async def main(): # configure() reads .env automatically via python-dotenv. # It resolves LLM, Neo4j, and storage settings from: # 1. Explicit arguments (highest priority) # 2. Environment variables / .env file # 3. Hard-coded defaults configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="your_password", ) result = await run_pipeline(input_raw="./raw_docs") # PipelineResult has structured per-stage results print(f"Success: {result.success}") print(f"Stages executed: {result.stages_executed}") print(f"Duration: {result.total_duration_seconds:.2f}s") # Inspect individual stages for stage_name in result.stages_executed: stage = getattr(result, stage_name, None) if stage: print(f" {stage_name}: {stage.total_processed} processed, " f"{stage.total_failed} failed") asyncio.run(main()) ``` Save this as `run_ingestion.py` and execute it: ```bash python run_ingestion.py ``` ### Pipeline stages The full pipeline runs these stages in order: 1. **Preprocess** — Converts raw files to an intermediate JSON/Markdown format. 2. **Extraction** — Uses the LLM to parse document structure and extract hierarchical sections. 3. **Ingestion** — Writes document and structure nodes into Neo4j. 4. **Annotation** — An LLM agent assigns an extraction model to each structural node. 5. **Entity Extraction** — Extracts typed Pydantic entities from annotated nodes and writes them as graph subgraphs. 6. **Tabular** — If `.csv`, `.xlsx`, or `.xls` files are detected in `input_raw`, they are processed through a separate tabular pipeline with LLM-powered normalization. ### Using a specific LLM provider If you prefer to construct the LLM explicitly rather than relying on `MODEL_ID`: #### OpenAI ```python from langchain_openai import ChatOpenAI from scinr.newton import configure, run_pipeline configure( llm=ChatOpenAI(model="gpt-4o"), neo4j_user="neo4j", neo4j_password="your_password", ) ``` #### AWS Bedrock ```python from langchain_aws import ChatBedrockConverse from scinr.newton import configure, run_pipeline configure( llm=ChatBedrockConverse( model="us.anthropic.claude-sonnet-4-6", region_name="us-east-1", max_tokens=65536, temperature=0, ), neo4j_user="neo4j", neo4j_password="your_password", ) ``` #### Ollama ```python from langchain_ollama import ChatOllama from scinr.newton import configure, run_pipeline configure( llm=ChatOllama(model="llama3"), neo4j_user="neo4j", neo4j_password="your_password", ) ``` ### Running individual stages You can run only specific stages by passing the `stages` parameter: ```python # Run only annotation and entity extraction on a known document result = await run_pipeline( stages=["annotation", "entity_extraction"], document_names=["MyDocument"], ) ``` See the [Configuration](configuration.md) documentation for all `run_pipeline()` parameters. --- ## Verifying Results After the pipeline completes, your data is available in Neo4j. Here are ways to verify the results: ### Via Neo4j Browser 1. Open `http://localhost:7474` in your browser. 2. Log in with your Neo4j credentials. 3. Run these queries to inspect the ingested data: ```cypher -- List all ingested documents MATCH (d:Document) RETURN d.document_name AS name, d.version AS version, d.file_path AS path ORDER BY d.document_name; -- Count structure nodes per document MATCH (d:Document)-[:HAS_STRUCTURE_NODE]->(s:StructureNode) RETURN d.document_name AS document, count(s) AS nodes ORDER BY nodes DESC; -- View annotated nodes with their assigned model MATCH (s:StructureNode) WHERE s.model_class IS NOT NULL RETURN s.text AS text, s.model_class AS model, count(s) AS count ORDER BY count DESC; -- Explore extracted entities and their relationships MATCH (e) WHERE head(labels(e)) <> 'Document' AND head(labels(e)) <> 'StructureNode' RETURN head(labels(e)) AS type, count(e) AS count ORDER BY count DESC; ``` ### Via Python ```python from neo4j import AsyncGraphDatabase async def verify(): async with AsyncGraphDatabase.driver( "bolt://localhost:7687", auth=("neo4j", "your_password") ) as driver: async with driver.session() as session: result = await session.run( "MATCH (d:Document) RETURN count(d) AS doc_count" ) record = await result.single() print(f"Documents in Neo4j: {record['doc_count']}") asyncio.run(verify()) ``` ### Inspecting PipelineResult The `PipelineResult` returned by `run_pipeline()` contains detailed per-stage metrics: ```python result = await run_pipeline(input_raw="./raw_docs") # Overall success print(f"Pipeline success: {result.success}") # Per-stage details if result.ingestion: for doc in result.ingestion.documents: print(f" {doc.document_name}: " f"{doc.nodes_processed} processed, " f"{doc.nodes_failed} failed") if doc.errors: for err in doc.errors: print(f" ERROR: {err}") ``` --- ## Troubleshooting ### `ConfigurationError: No LLM configured` You must either: - Set `MODEL_ID` in your `.env` file (for AWS Bedrock), or - Pass an `llm=` argument to `configure()` with a LangChain `BaseChatModel` instance. ### `ConfigurationError: Neo4j username/password is not configured` Set `NEO4J_USER` and `NEO4J_PASSWORD` in your `.env` file, or pass them as arguments to `configure()`. Alternatively, use `NEO4J_AUTH=neo4j/password` as a combined format. ### Neo4j connection refused Make sure your Neo4j instance is running and accessible: ```bash # Test Bolt connectivity python -c "from neo4j import GraphDatabase; d = GraphDatabase.driver('bolt://localhost:7687', auth=('neo4j', 'password')); d.verify_connectivity(); d.close(); print('OK')" ``` ### `ImportError: langchain-aws is not installed` If you set `MODEL_ID` but haven't installed the Bedrock extra: ```bash pip install "scinr[bedrock]" ``` ### PDFs fail to process PDF processing requires either: - A Mistral API key (for OCR) set via `MISTRAL_API_KEY` in your `.env`, or - The PDF must contain extractable text (processed via `pdfplumber` without OCR). If you see OCR-related errors and don't have a Mistral key, try text-based PDFs or set `MISTRAL_API_KEY`. ### `No documents discovered for this run` Check that: - The `input_raw` directory exists and contains supported file types. - File extensions are recognized (`.pdf`, `.docx`, `.xlsx`, `.csv`, `.pptx`, `.html`, `.json`, `.txt`, `.md`). - The path is correct (relative paths are resolved from the current working directory). ### LLM calls are slow or rate-limited Adjust concurrency in your `.env` or via `configure()`: ```python configure( llm=my_llm, neo4j_user="neo4j", neo4j_password="password", llm_concurrency=2, # Reduce concurrent LLM calls neo4j_concurrency=5, # Reduce concurrent Neo4j writes ) ``` --- ## Next Steps Now that you have a working pipeline, explore the rest of the documentation: - **[Configuration](configuration.md)** — Complete reference for `configure()`, all environment variables, prompt families, concurrency tuning, and advanced settings. - **[Architecture](architecture.md)** — Detailed walkthrough of each pipeline stage, data flow between stages, and system design decisions. - **User Guides** — Domain-specific guides for working with extraction models, custom themes, and tabular data processing. --- ## File: index.md # scinr > **AI-Powered Document Knowledge Library for Life Sciences** `scinr` (`scinr.newton`) is a Python library that transforms unstructured and structured life sciences documents into queryable, connected knowledge graphs in **Neo4j** and optional document stores in **MongoDB**. It provides an async 6-stage pipeline that converts raw files into structured, annotated, and graph-connected domain entities — all driven by LLMs and Pydantic extraction models. --- ## Key Features * **6-Stage Async Pipeline** — Preprocess, extract, ingest, annotate, entity-extract, and normalize tabular data in a single orchestrated flow. * **Multi-Format Ingestion** — Supports `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.json`, `.html`, and `.txt`. * **Tabular Pipeline with LLM Normalization** — Auto-detection, structural normalization, and LLM-powered entity extraction for scientific spreadsheets. * **Pydantic Extraction Models** — Define structured schemas for scientific target entities (e.g., compound synthesis, clinical trials, assays) with automatic graph annotations. * **Neo4j Knowledge Graph Output** — Automatically map extracted entities and triples into Neo4j subgraphs with document provenance. * **Optional MongoDB Storage** — Persist raw files, converted pages, and binary assets via GridFS. * **Agent-Ready Documentation** — Native support for [llms.txt](https://github.com/Scinr-AI/scinr/blob/main/llms.txt) and [llms-full.txt](https://github.com/Scinr-AI/scinr/blob/main/llms-full.txt) context windows for AI agents. --- ## Pipeline Overview The ingestion pipeline processes documents through six stages: ``` Raw Documents (.pdf, .docx, .pptx, .xlsx, .csv, .json, .html, .txt) │ ▼ ┌─────────────────────────┐ │ 1. Preprocess │ Format converters → JSON / Markdown └────────────┬────────────┘ ▼ ┌─────────────────────────┐ │ 2. Extraction │ Section chunking & hierarchy parsing └────────────┬────────────┘ ▼ ┌─────────────────────────┐ │ 3. Ingestion │ Document & structure nodes → Neo4j └────────────┬────────────┘ ▼ ┌─────────────────────────┐ │ 4. Annotation │ LLM relevance filtering & schema prep └────────────┬────────────┘ ▼ ┌─────────────────────────┐ │ 5. Entity Extraction │ Pydantic extraction → graph subgraphs └────────────┬────────────┘ ▼ ┌─────────────────────────┐ │ 6. Tabular │ Spreadsheet normalization & extraction └─────────────────────────┘ ``` --- ## Quick Example ```python import asyncio from scinr.newton import configure, run_pipeline async def main(): # 1. Configure backend connections configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="password", ) # 2. Run the end-to-end ingestion pipeline result = await run_pipeline(input_raw="./raw_documents") print(f"Success: {result.success}") print(f"Duration: {result.total_duration_seconds:.2f}s") asyncio.run(main()) ``` --- ## How It Works 1. **Configure** — Set up backends (Neo4j, MongoDB, LLM) via `configure()` or environment variables. 2. **Run the Pipeline** — Call `run_pipeline()` with your input directory. The pipeline handles conversion, extraction, ingestion, annotation, and entity extraction automatically. 3. **Query the Graph** — Explore extracted entities and their relationships in Neo4j. --- ## Documentation * **[Getting Started](getting-started.md)** — Installation, prerequisites, and your first ingestion run. * **[Configuration](configuration.md)** — Complete reference for `configure()`, environment variables, prompt families, and all settings. * **[Architecture](architecture.md)** — Detailed pipeline stages, data flow, and system design. --- ## File: user-guides/custom-models.md # Custom Extraction Models Define domain-specific Pydantic extraction models to extract structured entities from unstructured documents. This is the definitive guide to creating extraction models for `scinr.newton`. Every rule, pattern, and best practice required to build models that work reliably with the annotation agent (Stage 3) and entity extraction engine (Stage 4) is documented here. --- ## 1. Introduction ### What are extraction models? An extraction model is a Pydantic class that defines the schema for structured data extracted from a document section. When you run the `scinr.newton` pipeline, the annotation agent (Stage 3) reads your model definitions and decides which model to apply to each structural node. The entity extraction engine (Stage 4) then uses the LLM to populate the model fields from the document text and writes the resulting data as subgraphs in Neo4j. ### Why `ExtractionModel` over `BaseModel`? The **only requirement** for an extraction model is that it is a valid Pydantic model — it must inherit from `pydantic.BaseModel` or any subclass of it. You can inherit directly from `BaseModel` if you want; the pipeline will still work. However, `ExtractionModel` is the **recommended** base class because it provides a set of helpful defaults that make models more robust in practice: ```python from scinr.newton.models.base import ExtractionModel ``` `ExtractionModel` sets `extra="forbid"` (catching LLM-hallucinated fields), `str_strip_whitespace=True` (auto-trimming string fields), `validate_assignment=True` (re-validating on mutation), and `use_enum_values=True` (clean enum serialization). These defaults prevent common issues without requiring boilerplate in every model. If you inherit directly from `BaseModel` instead, you can manually add the same settings via `model_config = ConfigDict(...)` to get equivalent behavior. ### How models connect to the pipeline Models interact with three pipeline stages. Understanding the full flow — from theme selection in Stage 1 to model annotation in Stage 3 and entity extraction in Stage 4 — is essential for designing models that work correctly. **Stage 1 (Extraction)**: The LLM structures the document into information units AND selects a theme for each structural node based on `THEME_DESCRIPTION`. A **theme** is a collection of extraction models grouped by domain, defined in a `catalog.py` file with `THEME_DESCRIPTION` and `SELECTABLE_MODELS`. During Stage 1, the LLM reads the document content and assigns a theme label to each structural node based on which theme best matches the content. This is why `THEME_DESCRIPTION` is so important — it is the signal the LLM uses to classify nodes. The theme selected in Stage 1 determines which extraction models are available for that node in later stages. **Stage 3 (Annotation)**: The annotation agent considers only the models from the theme selected in Stage 1. It reads each model's class docstring and field descriptions to decide which model best fits a given structural node. **Stage 4 (Entity Extraction)**: The extraction LLM receives the selected model's full schema (fields, descriptions, enums, `json_schema_extra`) and extracts the structured data. The resulting instance is written as a `:ModelInstance` node in Neo4j, with `:LabeledEntity` nodes and relationships created according to the model's graph annotations. | Stage | Name | What themes/models do | | :--- | :--- | :--- | | 1 | **Extraction** | The LLM structures the document into information units AND selects a theme for each structural node based on `THEME_DESCRIPTION`. This determines which extraction models are available for that node. | | 3 | **Annotation** | The annotation LLM reads each model's class docstring and field descriptions to decide which model best fits a given structural node, choosing from the theme selected in Stage 1. | | 4 | **Entity Extraction** | The extraction LLM receives the selected model's full schema and extracts the structured data. The resulting instance is written as a `:ModelInstance` node in Neo4j. | --- ## 2. The `ExtractionModel` Base Class `ExtractionModel` is a thin wrapper around `pydantic.BaseModel` with a strict `ConfigDict`: ```python from pydantic import BaseModel, ConfigDict class ExtractionModel(BaseModel): model_config = ConfigDict( extra="forbid", validate_assignment=True, str_strip_whitespace=True, use_enum_values=True, ) ``` Each setting serves a specific purpose: | Setting | Effect | Why it matters | | :--- | :--- | :--- | | `extra="forbid"` | Rejects any JSON field not declared in the model | Catches LLM hallucinations immediately — the extraction fails loudly instead of silently accepting garbage | | `validate_assignment=True` | Re-validates every field on mutation | Ensures programmatic changes (e.g., post-processing, normalization hooks) still pass validation | | `str_strip_whitespace=True` | Auto-trims leading/trailing whitespace on all string fields | Eliminates OCR artifacts and LLM formatting noise without manual `.strip()` calls | | `use_enum_values=True` | Serializes enum members as their plain string value | Stores `"IA"` instead of `ProcedureType.IA` in Neo4j; guarantees clean JSON output | **Recommendation:** Inherit from `ExtractionModel` (or from a subclass of it) to get its built-in defaults. The only requirement is that the model is a valid Pydantic model — inheriting from `BaseModel` or any of its subclasses works. The `Triple` fallback model inherits directly from `BaseModel` as a historical exception. --- ## 3. Your First Model Here is a complete, production-quality extraction model demonstrating all best practices: ```python """Example extraction models.""" from __future__ import annotations from enum import Enum from pydantic import Field from scinr.newton.models.base import ExtractionModel class StatusEnum(str, Enum): """ Lifecycle status of an item. Values: active — Item is currently in use. inactive — Item has been retired or replaced. unknown — Status not stated in the document. """ ACTIVE = "active" INACTIVE = "inactive" UNKNOWN = "unknown" class ItemModel(ExtractionModel): """A single item entry from a regulatory catalogue document.""" item_code: str = Field( ..., description=( "Unique alphanumeric code as written in the document (e.g. 'A-001', " "'Q.I.a.1'). Never omit the prefix. This field is required." ), json_schema_extra={"entity_label": "ItemCode", "instance_key": True}, ) description: str = Field( ..., description="Verbatim description of the item as stated in the document.", ) status: StatusEnum | None = Field( default=None, description=( "Lifecycle status of the item per StatusEnum. " "'active' if currently valid, 'inactive' if retired. None if not stated." ), ) related_codes: list[str] = Field( default_factory=list, description=( "Other item codes explicitly cross-referenced in this entry. " "Each value is a raw code string. Empty list if none are mentioned." ), ) ``` This model demonstrates: - **`from __future__ import annotations`** — enables PEP 604 `|` union syntax and forward references. - **`str, Enum`** — enum inherits from `str` for safe JSON serialization. - **Enum docstring with Values section** — documents each member for the LLM. - **`ExtractionModel` inheritance** — recommended over raw `BaseModel` for its built-in defaults. - **Class docstring** — ≤ 15 words, starts with entity type ("A single..."). - **Required field** (`...`) — `item_code` must always be present. - **Optional field** (`str | None`, `default=None`) — `status` may be absent. - **List field** with `default_factory=list` — never `default=[]`. - **`entity_label`** — marks `item_code` as a globally deduplicated entity. - **`instance_key`** — marks `item_code` as the unique key for this model instance. - **Field descriptions** — ≥ 15 words, answering what, format, and when None. --- ## 4. Field Design Rules ### 4.1 Scalar fields ```python # Required — always present in this document type code: str = Field(..., description="...") # Optional — field may or may not be present name: str | None = Field(default=None, description="...") # Required but defaults to empty string — prefer for key fields # used in join_via or instance_key context root_code: str = Field(default="", description="...") ``` ### 4.2 List fields ```python # ALWAYS use default_factory — NEVER default=[] items: list[str] = Field(default_factory=list, description="...") sub_models: list[Sub] = Field(default_factory=list, description="...") ``` **Why:** `default=[]` creates a mutable default shared across all instances. Pydantic rejects this pattern or causes subtle data-sharing bugs. `default_factory=list` creates a fresh list for each instance. ### 4.3 Enums ```python class ProcedureType(str, Enum): """ Normalized procedure type code. Use str as the base class — guarantees JSON-serializable values and correct Neo4j storage without extra serialization steps. Values: IA — Type IA notification (immediate effect, notify within 12 months). IB — Type IB notification (implement after 30-day review window). II — Type II prior-approval variation. """ IA = "IA" IB = "IB" II = "II" ``` **Rules:** - Always use `str` as the base class for enums in extraction models. - Include a docstring with a `Values:` section documenting each member. - The `use_enum_values=True` in `ExtractionModel.model_config` stores the plain string value, but `str` base is still required for safe serialization elsewhere. ### 4.4 Field descriptions Every field **must** have a `description=` with **at least 15 words**. The LLM uses the description as its primary extraction signal. A good description answers three questions: 1. **What** is this data point exactly? 2. **What format** is expected (with concrete examples)? 3. **When is `None` (or empty string/empty list) correct?** ```python # ✅ GOOD — answers all three questions variation_code: str = Field( ..., description=( "Full, absolute variation code identifier as written in the document — " "always include the top-level prefix (e.g. 'Q.III.1(b)4', not '(b)4'; " "'Q.I.a.1(a)', not '(a)'; 'B.II.b.1'). " "Never extract a partial or relative sub-code without its parent prefix." ), ) # ❌ BAD — too short, no examples, no None condition variation_code: str = Field(..., description="The variation code.") ``` ### 4.5 Numeric values with units ```python # Preserve original format — do NOT convert to float batch_size: str | None = Field( default=None, description=( "Batch size as stated in the document, including units " "(e.g. '100 kg', '500 L', '1×10⁶ cells'). " "Preserve original formatting and units. None if not stated." ), ) ``` Keep numeric values as strings with their units. Converting to `float` loses precision, units, and formatting context. Store `"100 kg"` not `100.0`. --- ## 5. Docstring Rules ### 5.1 Class docstrings The annotation agent reads **class docstrings** to decide which model to apply to each document section. Poorly written docstrings cause misclassification. **First line rules:** - **≤ 15 words**, in English. - Start with the entity type: `"A single..."`, `"Full definition of..."`, `"Use when..."`. - Must be informative enough to distinguish from similar models. ```python # ✅ GOOD class VariationCodeModel(ExtractionModel): """A single variation code entry from the EU variation guidelines (Official Journal).""" # ✅ GOOD — explicit USE/DO NOT USE class VariationCodeWithDocsAndConditionModelList(ExtractionModel): """Use when TWO OR MORE variation codes and their conditions are defined inline.""" # ❌ BAD — too vague class VariationCodeModel(ExtractionModel): """Variation code model.""" # ❌ BAD — too long for first line class VariationCodeModel(ExtractionModel): """This model captures variation codes from EU regulatory documents including all procedure types and conditions.""" ``` ### 5.2 USE / DO NOT USE conditions For models where misclassification is likely, add explicit conditions in the docstring: ```python class VariationCodeModel(ExtractionModel): """ A single variation code entry from EU variation guidelines. USE THIS MODEL when the section defines exactly ONE variation code, OR when conditions and documentation are listed in separate sibling/child sections. DO NOT use this model when the section defines TWO OR MORE variation codes with their conditions and documentation all inline — use VariationCodeWithDocsAndConditionModelList instead. """ ``` ### 5.3 Complementary model hints When a model is typically used together with other models, declare this explicitly: ```python class DocumentationModel(ExtractionModel): """ A documentation requirement for a variation. They are perfect candidates as ComplementaryModels for VariationCodeModel. """ ``` The annotation agent reads `ComplementaryModels` hints and may suggest them as secondary models to apply alongside the primary model. --- ## 6. List Wrapper Pattern ### 6.1 When to create a `XxxModelList` wrapper Create a list wrapper model alongside the main model when: - A document section **regularly contains a table or list** of the same entity (e.g., a fee schedule table, a variation code table). - A section can contain **zero, one, or many** instances of the entity. - The annotation agent needs a way to extract multiple entities in a single extraction call. **Do NOT create a list wrapper when:** - Every section of this type **always** has exactly one instance. - The entities are already captured as a `list[SubModel]` field inside a parent model. ### 6.2 List wrapper structure ```python class ItemModelList(ExtractionModel): """Use when the section defines TWO OR MORE item entries in a list or table.""" items: list[ItemModel] = Field( default_factory=list, description=( "List of item entries. Each element represents one distinct item. " "Use this model instead of ItemModel when the section is a table or list " "covering two or more items." ), ) ``` **Rules:** - The wrapper inherits from `ExtractionModel`, not from the main model's base class. - The wrapper has **one field**: the list. - Both `XxxModel` AND `XxxModelList` must appear in `SELECTABLE_MODELS`. - The docstring first line should start with `"Use when the section defines TWO OR MORE..."`. --- ## 7. `catalog.py` and Theme Registration ### 7.1 Minimal correct `catalog.py` ```python """Catalog for the example theme.""" from __future__ import annotations from .models import ItemModel, ItemModelList THEME_DESCRIPTION: str = ( "Regulatory catalogue documents containing item entries with codes, " "descriptions, and lifecycle statuses. Covers item cross-references " "and status tracking. Distinct from procedural guidelines." ) SELECTABLE_MODELS: list[type] = [ ItemModelList, # multi-item sections (most specific first) ItemModel, # single-item sections ] ``` ### 7.2 Writing an effective `THEME_DESCRIPTION` The annotation LLM reads `THEME_DESCRIPTION` to decide whether a document section belongs to this theme. A good description: - **Names the document types** it covers (`"EU Official Journal"`, `"EMA Best Practice Guidelines"`). - **Names regulatory standards** when applicable (`"EC Regulation 1234/2008"`, `"ICH CTD Module 3"`). - **Distinguishes** from adjacent themes that could be confused (`"Distinct from BPG..."`). - **Gives examples** of the entities it captures (`"variation codes (e.g. Q.I.a.1)"`). ```python # ✅ GOOD THEME_DESCRIPTION: str = ( "EU pharmaceutical variation guidelines (Official Journal, EC Regulation 1234/2008). " "Covers variation codes (IA, IB, II), conditions, and documentation requirements. " "Distinct from BPG and Q&A documents." ) # ❌ BAD — too vague, LLM will classify everything here THEME_DESCRIPTION: str = "Pharmaceutical regulatory documents." # ❌ BAD — only one sentence, no distinguishing information THEME_DESCRIPTION: str = "Documents about variation codes." ``` ### 7.3 `SELECTABLE_MODELS` ordering Order from most to least specific (the LLM tends to select models appearing earlier when confidence is similar): 1. **List wrapper models** for multi-instance sections (most specific). 2. **Main models** for single-instance sections. 3. **Supporting/complementary models**. ```python SELECTABLE_MODELS: list[type] = [ VariationCodeWithDocsAndConditionModelList, # multi-code sections with inline data VariationCodeModel, # single-code sections or separate docs/conditions DocumentationModelList, # sections listing ≥2 documentation requirements DocumentationModel, # single documentation requirement ConditionModelList, # sections listing ≥2 conditions ConditionModel, # single condition ProcedureTypeModelList, # sections defining ≥2 procedure types ProcedureTypeModel, # single procedure type definition ] ``` ### 7.4 Parent catalog aggregating sub-themes When a parent folder has its own `catalog.py` that aggregates sub-theme models, use explicit relative imports from sub-packages: ```python # pharma_regulatory/catalog.py from .variation_guidelines.models import VariationCodeModel, ProcedureTypeModel from .bpg.models import BPGRecommendationModel from .qa.models import QAEntryModel THEME_DESCRIPTION: str = "..." SELECTABLE_MODELS: list[type] = [ VariationCodeModel, BPGRecommendationModel, QAEntryModel, ] ``` --- ## 8. Directory Structure ### 8.1 The `__init__.py` rule is absolute Every folder that contains Python files **must** be a Python package. This means it needs an `__init__.py` file (which may be empty). **No exceptions.** ``` my_package/ ├── __init__.py ← REQUIRED (may be empty) ├── base.py ← shared ExtractionModel or custom base └── pharma_regulatory/ ← add __init__.py HERE ├── __init__.py ← REQUIRED ├── baseModels.py ← domain-specific base with validators ├── catalog.py ← theme-level catalog ├── variation_guidelines/ ← add __init__.py HERE │ ├── __init__.py ← REQUIRED │ ├── catalog.py ← sub-theme catalog │ └── models.py └── bpg/ ← add __init__.py HERE ├── __init__.py ← REQUIRED ├── catalog.py └── models.py ``` **When creating new sub-themes or helper folders, the first file you create must always be `__init__.py`.** ### 8.2 Theme vs. Sub-theme A **theme** is a top-level extraction domain registered in `ThemeRegistry`. It requires a `catalog.py` with `THEME_DESCRIPTION` and `SELECTABLE_MODELS`. A **sub-theme** is a nested folder with its own `catalog.py`, representing a specialised subset of a parent theme. **Create a sub-theme when:** the domain has clearly differentiated document types with incompatible model sets (e.g., variation guidelines vs. best practice guidelines vs. Q&A documents). **Keep in the same theme when:** models are complementary and often used together on the same document type. ### 8.3 Shared base files Place shared validators, normalization functions, and base classes in a `baseModels.py` (or `base.py`) at the appropriate level of the hierarchy: ``` my_package/ ├── base.py ← ExtractionModel (if needed as a local copy) └── pharma_regulatory/ ├── baseModels.py ← NormalizedBaseModel shared across ALL sub-themes ├── variation_guidelines/ │ └── models.py ← imports from ..baseModels └── bpg/ └── models.py ← imports from ..baseModels ``` --- ## 9. Imports ### 9.1 Always use relative imports inside your package ```python # ✅ CORRECT — import from the same directory from .models import MyModel from .base import MyBaseClass # ✅ CORRECT — import from the parent directory from ..baseModels import NormalizedBaseModel # ✅ CORRECT — import from two levels up from ...base import ExtractionModel # ✅ CORRECT — import from the installed scinr library (absolute is correct here) from scinr.newton.models.base import ExtractionModel # ❌ INCORRECT — bare import (works only if the directory happens to be in sys.path) from models import MyModel # ❌ INCORRECT — absolute path using your own package name from my_package.pharma_regulatory.models import MyModel # ❌ INCORRECT — absolute path using a sibling in your own package from pharma_regulatory.baseModels import NormalizedBaseModel ``` ### 9.2 Counting the dots The number of leading dots equals the number of directory levels to go up, **not including the current file's directory** (which is always `.`): ``` own_models/ ├── base.py ← 3 dots from bpg/models.py: from ...base └── pharma_regulatory/ ├── baseModels.py ← 2 dots from bpg/models.py: from ..baseModels └── bpg/ └── models.py ← I am HERE ``` ```python # In own_models/pharma_regulatory/bpg/models.py: from ...base import ExtractionModel # 3 dots → own_models/base.py from ..baseModels import NormalizedBaseModel # 2 dots → own_models/pharma_regulatory/baseModels.py from .catalog import THEME_DESCRIPTION # 1 dot → own_models/pharma_regulatory/bpg/catalog.py ``` ### 9.3 The one absolute-import exception `scinr` is an installed package. You may (and should) import `ExtractionModel` from it using an absolute path when you do not maintain your own `base.py`: ```python # This is always correct regardless of where your package lives: from scinr.newton.models.base import ExtractionModel ``` --- ## 10. File Structure and Ordering ### 10.1 Canonical order within a `models.py` ``` 1. Module docstring 2. from __future__ import annotations 3. Standard library imports (re, enum, typing) 4. Third-party imports (pydantic) 5. Local relative imports (base classes, shared models) 6. ── ENUMS ──────────────────────────────── (controlled vocabulary) 7. ── BASE / SHARED SUBMODELS ────────────── (reused across main models) 8. ── TARGET MODELS ──────────────────────── (models referenced via instance_relationships) 9. ── MAIN MODELS ─────────────────────────── (one per document section type) 10. ── LIST WRAPPERS ──────────────────────── (XxxModelList for multi-instance sections) ``` Declare models before they are referenced. If model A has a field of type B, declare B first. ### 10.2 Graph Annotations #### `entity_label`: when to use and when NOT to use **Use `entity_label` for:** - Named real-world entities that may recur across documents (codes, substances, facilities, procedure types, country codes). - Values stable enough for normalization (identifiers, proper nouns). - Fields where cross-document deduplication is meaningful. **Do NOT use `entity_label` for:** - Long free-text descriptions unique to each document instance. - Narrative summary paragraphs. - Numeric measurements in context (`"25°C/60% RH"`). - Boolean flags or status strings. - Fields where the value is essentially a sentence or paragraph. ```python root_code: str = Field( ..., json_schema_extra={ "entity_label": "VariationCode", "instance_key": True, }, ) ``` #### `instance_key: True` Mark a field `instance_key: True` when: 1. The model is (or may be) referenced as a `target_model` in another model's `instance_relationships`. 2. The field forms part of the unique key that identifies one instance of this model. When multiple fields together form the key (composite key), mark ALL of them: ```python class Fee(ExtractionModel): country_code: str = Field(..., json_schema_extra={"entity_label": "Country", "instance_key": True}) procedure_type: str = Field(..., json_schema_extra={"entity_label": "ProcedureType", "instance_key": True}) role: str = Field(..., json_schema_extra={"entity_label": "FeeRole", "instance_key": True}) rate: str = Field(..., description="Fee amount without currency symbol.") ``` #### `field_relationships` Connects two `:LabeledEntity` nodes within the same extracted model instance. Both fields must be siblings and both must have `entity_label`. ```python root_code: str | None = Field( default=None, json_schema_extra={ "entity_label": "VariationCode", "field_relationships": [ {"to_field": "child_code", "rel_type": "HAS_CHILD_VARIATION_CODE"}, ], }, ) child_code: str = Field( ..., json_schema_extra={"entity_label": "VariationCode", "instance_key": True}, ) ``` Produces: `(:LabeledEntity:VariationCode {value:"Q.I.a.1"}) -[:HAS_CHILD_VARIATION_CODE]-> (:LabeledEntity:VariationCode {value:"Q.I.a.1(a)"})` **Rules:** - Both source and target fields must have `entity_label`. - `to_field` must be the **name** of a sibling field in the same model. - Relationship is only created when **both** fields are non-null. - `rel_type` must be `UPPER_SNAKE_CASE`. #### `instance_relationships` Connects `:ModelInstance` nodes across different sections or documents. Creates shell nodes for targets that have not yet been extracted. ```python json_schema_extra={ "instance_relationships": [ { "target_model": "TargetModelClassName", # string — PascalCase class name "join_via": { "local_field": "remote_key_field", # scalar sibling → target instance_key field "list_field": "remote_key_field_2", # list field (fan-out) → target instance_key field }, "rel_type": "RELATIONSHIP_TYPE", # UPPER_SNAKE_CASE } ] } ``` **Rules:** - Every `target_model` in `instance_relationships` must mark its key fields with `instance_key: True`. - Fan-out `join_via` field names must exactly match Python field names. - `rel_type` must be `UPPER_SNAKE_CASE`. #### Fan-out pattern (one-to-many via list field) When the source field is a **list**, one target `ModelInstance` is created per list item: ```python condition_ids: list[str] = Field( default_factory=list, description="IDs of associated conditions (e.g. ['1', '2', 'A']).", json_schema_extra={ "instance_relationships": [ { "target_model": "ConditionModel", "join_via": { "root_variation_code": "variation_code", # fixed anchor key "condition_ids": "condition_id", # fan-out: one target per item }, "rel_type": "HAS_CONDITION", } ] }, ) ``` - **Fixed keys** (`root_variation_code → variation_code`): scalar fields that scope the target to the correct parent. - **Fan-out key** (`condition_ids → condition_id`): the list field itself; `join_via` entry maps the list field name to the corresponding `instance_key` field on the target. #### Dual pattern: `entity_label` + `instance_relationships` A field can simultaneously create a `:LabeledEntity` global singleton AND a `:ModelInstance` cross-model edge: ```python procedure_type: str = Field( default="", description="Procedure type code: IA, IB, II, IAIN, A, or BA.", json_schema_extra={ "entity_label": "ProcedureType", # → creates :LabeledEntity node "instance_relationships": [ { "target_model": "ProcedureTypeModel", "join_via": {"procedure_type": "procedure_type"}, "rel_type": "HAS_PROCEDURE_TYPE", } ], # → creates :ModelInstance edge }, ) ``` Use this dual pattern when the value is both a named entity (needs global dedup) AND points to a structured model instance (needs cross-document linking). --- ## 11. Validators and Normalization ### 11.1 The `NormalizedBaseModel` pattern When a domain requires consistent normalization of specific field values across all models (OCR correction, code normalization, case normalization), create a shared base class with `field_validator`: ```python # pharma_regulatory/baseModels.py import re from pydantic import BaseModel, field_validator def normalize_code(v: str) -> str: """Apply domain-specific normalization to a code string.""" v = v.strip().upper() v = re.sub(r"[\s_/-]", "", v) # remove separators return v class NormalizedBaseModel(BaseModel): """Base class that applies field normalization before Pydantic validation.""" @field_validator("procedure_type", "procedure_types_referenced", mode="before", check_fields=False) @classmethod def normalize_procedure_types(cls, v): if isinstance(v, str): return normalize_code(v) if isinstance(v, list): return [normalize_code(item) if isinstance(item, str) else item for item in v] return v ``` **Key details:** - `check_fields=False` makes the validator **optional**: it runs only if the subclass actually has that field. Without this flag, Pydantic raises an error when a subclass inherits the validator but does not declare the field. - `mode="before"` applies normalization before Pydantic's own type validation. - List handling: always check `isinstance(v, list)` and map over items. ### 11.2 When to create a domain-specific base class Create a `NormalizedBaseModel` (or equivalent) when: - Multiple models in the same domain share the same normalization logic. - OCR corrections are needed (e.g., `Q.1.a.1` → `Q.I.a.1` for variation codes). - Code values need to be consistently uppercased and stripped (e.g., `"i a"` → `"IA"` for procedure types). Do NOT create a domain-specific base just for convenience — it adds indirection. Use it only when sharing validation is meaningful. ### 11.3 OCR fix validators A common pattern for regulatory codes that suffer from OCR mis-recognition: ```python def normalize_variation_code(v: str) -> str: # OCR fix: Q.1.a.1 → Q.I.a.1 (digit 1 misread as letter l or numeral) v = re.sub(r"(?<=[A-Za-z])\.(?:1|l)\.", ".I.", v) # OCR fix: Q.I.a.1.a → Q.I.a.1(a) (trailing sub-code format) v = re.sub(r"(?<=\d)\.([a-zA-Z])$", r"(\1)", v) return v ``` Apply this in a `NormalizedBaseModel` validator, not inline in the field description, so it applies consistently without relying on the LLM. ### 11.4 The `normalization_model` mechanism Some nested submodel fields need to be filled in by an LLM **after** a row of structured data (CSV/XLSX/XLS) has already been mapped and instantiated — for example, turning a free-text `"raw_address"` column into a structured `NormalizedAddress` submodel. This is handled by a dedicated `NormalizationEngine` hook in the tabular ingestion pipeline. Trigger it by adding two keys to `json_schema_extra` on the nested field: ```python class NormalizedAddress(ExtractionModel): """Structured, normalized postal address derived from a raw address string.""" street: str | None = Field(default=None, description="...") city: str | None = Field(default=None, description="...") postal_code: str | None = Field(default=None, description="...") country_code: str | None = Field(default=None, description="...") class ContactRecord(ExtractionModel): """A single contact record imported from a CSV/XLSX file.""" raw_name: str = Field(..., description="...") raw_address: str = Field(..., description="Free-text address exactly as it appears in the source column.") raw_phone: str | None = Field(default=None, description="...") normalized_address: NormalizedAddress | None = Field( default=None, description="Structured address derived from raw_address via LLM normalization.", json_schema_extra={ "normalization_model": True, "normalization_source_fields": ["raw_address"], }, ) ``` This mechanism is: - **Opt-in and off by default.** It only runs if the pipeline caller has explicitly called `configure(normalization_enabled=True, normalization_llm=..., normalization_batch_size=...)`. If `normalization_enabled` is `False` (the default), `normalization_model` is completely inert. - **Tabular-only hook, but the keys stay visible everywhere.** The `NormalizationEngine` hook is wired into the tabular ingestion pipeline and nowhere else — it never runs during Stage 3–4 (PDF/DOCX) extraction. During Stage 3–4, the nested field is populated by the extraction LLM call directly, guided by the field's `description=`. - **Additive, not exclusive.** A field marked `normalization_model: True` is otherwise an ordinary nested-model field for every other purpose. Its own nested fields may still carry `entity_label`, `instance_key: True`, `field_relationships`, or `instance_relationships`. ### 11.5 Mandatory clarification: structured, unstructured, or both Whether `normalization_model` / `normalization_source_fields` are **required** or merely **useful** depends entirely on which pipeline(s) the model will be used with: | Model will be used with... | Add `normalization_model` + `normalization_source_fields`? | | :--- | :--- | | (a) Structured data only (tabular) | ✅ **Mandatory** — without these keys the tabular `NormalizationEngine` hook never fires for that field, and the nested submodel is never populated | | (b) Unstructured data only (Stage 3–4) | ⚪ **Optional** — the extraction LLM fills the nested field directly from `description=` with or without these keys; adding them is harmless and can serve as a schema-level hint | | (c) Both | ✅ **Recommended** — mandatory for the tabular half; optional-but-useful for the unstructured half; using the same declaration on both keeps the model consistent across pipelines | **Rule:** Clarify structured vs. unstructured vs. both before writing any normalization key. If the model is used with the tabular pipeline (case a or c), `normalization_model` + explicit `normalization_source_fields` are **mandatory** on every field that needs tabular-time normalization — omitting them silently disables normalization for that field. ### 11.6 `normalization_source_fields`: never rely on the implicit fallback `normalization_source_fields` is a `list[str]` of sibling scalar field names on the SAME parent model whose values are sent to the LLM to populate the normalized submodel. **If you omit it, or leave it empty, the engine silently falls back to using ALL other scalar fields of the parent model as source data.** This implicit fallback is a footgun in any model with more than a couple of fields: it silently vacuums up unrelated columns as "source data" for the normalization LLM call, wasting tokens, leaking irrelevant context into the prompt, and producing normalization results that depend on columns the maintainer never intended to feed in. ```python # ✅ GOOD — explicit, minimal, intentional source fields normalized_address: NormalizedAddress | None = Field( default=None, description="...", json_schema_extra={ "normalization_model": True, "normalization_source_fields": ["raw_address"], # exactly what feeds the LLM — nothing else }, ) # ❌ BAD — omitted normalization_source_fields class ContactRecord(ExtractionModel): raw_name: str = Field(..., description="...") raw_address: str = Field(..., description="...") raw_phone: str | None = Field(default=None, description="...") internal_notes: str | None = Field(default=None, description="...") normalized_address: NormalizedAddress | None = Field( default=None, description="...", json_schema_extra={ "normalization_model": True, # No normalization_source_fields declared. # Implicit fallback silently sends raw_name, raw_address, raw_phone, # AND internal_notes to the LLM — even though only raw_address is relevant. }, ) ``` **Rule:** Always set `normalization_source_fields` explicitly to the exact list of sibling fields the normalization actually needs. Never rely on the implicit "all other scalar fields" fallback. --- ## 12. Anti-Patterns | Anti-pattern | Why it fails | Correct approach | | :--- | :--- | :--- | | `default=[]` on a list field | Mutable default shared across instances — Pydantic rejects it or causes subtle bugs | `default_factory=list` | | Bare import: `from models import X` | Fragile; depends on `sys.path` at runtime | Relative: `from .models import X` | | Absolute own-package import: `from my_pkg.theme.models import X` | Breaks when `sys.path` changes | Relative: `from .models import X` | | Missing `__init__.py` in a subfolder | Python won't treat it as a package; relative imports fail with `ImportError` | Add empty `__init__.py` to every folder with `.py` files | | `entity_label` on a free-text description field | Creates meaningless `:LabeledEntity` singletons; degrades graph quality | Only add `entity_label` to short, stable, identifier-like values | | `field_relationships` pointing to a field without `entity_label` | The target node does not exist; Neo4j write silently ignored | Ensure `to_field` also has `entity_label` | | Missing `instance_key: True` on target model key fields | Shell nodes created by `instance_relationships` never merge with real nodes | Mark ALL key fields on the target model with `instance_key: True` | | Fan-out `join_via` key name mismatch | Zero target nodes created — field name in `join_via` must exactly match Python field name | Double-check that `join_via` keys use the exact Python field names | | `Optional[str]` instead of `str \| None` | Verbose; inconsistent with the codebase style | Use `str \| None` (PEP 604 union syntax) | | Vague `THEME_DESCRIPTION` | LLM misclassifies sections; wrong model applied | Be specific: name document types, regulatory standards, distinguish from adjacent themes | | Not adding `XxxModelList` to `SELECTABLE_MODELS` | Agent can never select it directly | Add both `XxxModel` and `XxxModelList` to `SELECTABLE_MODELS` | | Inheriting from `BaseModel` without `extra="forbid"` | LLM hallucinated fields are silently accepted | Use `ExtractionModel` (recommended) or add `model_config = ConfigDict(extra='forbid')` to your `BaseModel` subclass | | Class docstring longer than 15 words on the first line | Annotation agent truncates; key info may not be read | First line ≤ 15 words; put details on subsequent lines | | Validator without `check_fields=False` on inherited base | Pydantic raises `PydanticUserError` when a subclass does not declare the validated field | Always use `check_fields=False` on validators in shared base classes | | Omitting `normalization_model` on a field of a model used with the TABULAR pipeline | The tabular `NormalizationEngine` hook has nothing to trigger on; the nested submodel field silently stays `None`/unpopulated on every row, with no error raised | Add `normalization_model: True` + explicit `normalization_source_fields` on every field that needs tabular-time normalization | | Omitting `normalization_source_fields` (relying on the implicit fallback) on a wide model | Engine silently sends ALL other scalar fields of the parent model as source data to the normalization LLM — wastes tokens and leaks irrelevant context | Always set `normalization_source_fields` explicitly to the exact sibling fields needed | --- ## 13. Pre-merge Checklist ### Structure - [ ] Every directory with `.py` files contains `__init__.py` (including new sub-theme folders) - [ ] `catalog.py` uses relative imports (`from .models import ...`) - [ ] `models.py` uses relative imports for all internal modules ### Models - [ ] All classes are valid Pydantic models (inherit from `BaseModel` or a subclass) - [ ] Every class has a docstring (first line ≤ 15 words, in English) - [ ] Every field has `description=` with ≥ 15 words (what, format, when None) - [ ] Every optional scalar uses `default=None` - [ ] Every list field uses `default_factory=list` - [ ] Enums use `str` as base class ### Graph annotations - [ ] `entity_label` only on short, stable, identifier-like fields (not free-text) - [ ] `field_relationships` declared where a directed edge between sibling entities is needed - [ ] `instance_relationships` declared where cross-section or cross-document linking is needed - [ ] Every `target_model` in `instance_relationships` has `instance_key: True` on its key fields - [ ] Fan-out `join_via` field names exactly match Python field names ### Normalization (`normalization_model`) - [ ] Structured-vs-unstructured-vs-both usage was clarified before deciding whether to add `normalization_model` keys - [ ] Every field intended to be normalized when the model is used with the tabular pipeline has `normalization_model: True` set — check this explicitly for models shared across both pipelines - [ ] Every field with `normalization_model: True` has an explicit `normalization_source_fields` list — the implicit "all other scalar fields" fallback is never relied upon ### Theme registration - [ ] `THEME_DESCRIPTION` is specific, technical, and distinguishable - [ ] `SELECTABLE_MODELS` lists all top-level models (including list wrappers) - [ ] If list wrapper `XxxModelList` exists, both `XxxModel` and `XxxModelList` are in `SELECTABLE_MODELS` ### Validation - [ ] Auto-discovery verified: `python -c "from scinr.newton.utils.theme_registry import ThemeRegistry; print(ThemeRegistry().list_themes())"` - [ ] No import errors: `python -c "import my_package.my_theme.catalog"` - [ ] At least one real document processed end-to-end (Stage 3 + Stage 4) --- ## See Also - **[Running the Pipeline](running-pipeline.md)** — Full reference for `run_pipeline()`, including manual annotation mode (`manual=True`, `model_class=`). - **[Configuration](../configuration.md)** — All `configure()` parameters, including `normalization_enabled` and `normalization_llm` for the tabular normalization engine. - **[Tabular Pipeline](tabular-pipeline.md)** — Working with CSV, XLSX, and spreadsheet data. - **[Neo4j Graph Storage](neo4j-graph.md)** — Understanding `:ModelInstance`, `:LabeledEntity`, and relationship types in the graph. - **[Architecture](../architecture.md)** — Detailed walkthrough of each pipeline stage, including how annotation (Stage 3) and entity extraction (Stage 4) use extraction models. --- ## File: user-guides/custom-themes.md # Custom Themes Create and register custom extraction themes to extend scinr with domain-specific models. A theme is a collection of extraction models organized under a common domain description. Themes are auto-discovered by `ThemeRegistry` and used by the extraction LLM (Stage 1) to classify document sections. --- ## What is a Theme? A **theme** is a Python package containing two required files: | File | Purpose | | :--- | :--- | | `catalog.py` | Declares `THEME_DESCRIPTION` (for LLM classification) and `SELECTABLE_MODELS` (the models available for this theme). | | `models.py` | Defines the Pydantic extraction model classes. | Themes live in a directory tree. The `ThemeRegistry` scans directories for `catalog.py` files and builds a tree of themes. A folder is a theme if and only if it contains a `catalog.py`. ### How Themes Are Used During Stage 1 (`"extraction"`), the extraction LLM reads every registered theme's `THEME_DESCRIPTION` to decide which thematic domain each structural node belongs to. Each node is classified independently — a single document may have nodes classified to multiple different themes. Once classified, Stage 3 (`"annotation"`) selects a model from that theme's `SELECTABLE_MODELS` to extract structured entities. ``` Stage 1: Document section ──► LLM reads THEME_DESCRIPTIONs ──► picks best theme for this node Stage 3: Classified node ──► LLM reads SELECTABLE_MODELS ──► picks best model Stage 4: Selected model ──► extracts structured entities ``` ### Auto-Discovery Themes are discovered automatically. There is no manual registration step. The `ThemeRegistry` scans: 1. The built-in `scinr.newton.models/` directory at import time. 2. Any additional directories passed via `extra_models_paths` in `configure()`. 3. Packages registered via the `scinr.newton.models` entry-point group. --- ## Theme Structure A minimal theme has the following layout: ``` my_custom_theme/ ├── __init__.py # REQUIRED (may be empty) ├── catalog.py # THEME_DESCRIPTION + SELECTABLE_MODELS └── models.py # Extraction model definitions ``` A theme with sub-themes: ``` my_custom_theme/ ├── __init__.py # REQUIRED (may be empty) ├── catalog.py # Parent catalog (aggregates all sub-themes) ├── models.py # Parent-level models └── sub_theme/ # Optional sub-theme ├── __init__.py # REQUIRED ├── catalog.py # Sub-theme catalog └── models.py # Sub-theme models ``` **Every directory that contains `.py` files must have an `__init__.py`.** This is not optional — without it, Python will not treat the directory as a package and relative imports will fail. --- ## Creating `catalog.py` The `catalog.py` file is the heart of a theme. It declares two module-level variables: | Variable | Type | Required | Description | | :--- | :--- | :--- | :--- | | `THEME_DESCRIPTION` | `str` | Yes | One to three sentences describing the document types this theme covers. Used by the annotation LLM to classify sections. | | `SELECTABLE_MODELS` | `list[type]` | Yes | List of Pydantic model classes the annotation agent can select. Must be subclasses of `ExtractionModel` (or `BaseModel`). | ### Complete Example ```python """Catalog for the clinical trials theme.""" from __future__ import annotations from .models import ( TrialProtocolModel, TrialProtocolModelList, AdverseEventModel, AdverseEventModelList, DosageModel, ) THEME_DESCRIPTION: str = ( "Clinical trial protocol documents and adverse event reports. " "Covers trial phases (I-IV), inclusion/exclusion criteria, " "adverse event grading (CTCAE), dosage regimens, and endpoints. " "Use for sections describing trial design, patient populations, " "safety data, and dosing schedules. " "Distinct from regulatory variation guidelines and manufacturing records." ) SELECTABLE_MODELS: list[type] = [ TrialProtocolModelList, # multi-trial sections (most specific first) TrialProtocolModel, # single trial section AdverseEventModelList, # multi-event sections AdverseEventModel, # single event DosageModel, # dosage regimen ] ``` ### Writing an Effective `THEME_DESCRIPTION` The extraction LLM (Stage 1) reads `THEME_DESCRIPTION` to decide whether a structural node belongs to this theme. A good description: - **Names the document types** it covers (`"Clinical trial protocol documents"`, `"EMA Best Practice Guidelines"`) - **Names regulatory standards** when applicable (`"CTCAE grading"`, `"ICH E6 GCP"`, `"EC Regulation 1234/2008"`) - **Distinguishes** from adjacent themes that could be confused (`"Distinct from regulatory variation guidelines..."`) - **Gives examples** of the entities it captures (`"trial phases (I-IV)"`, `"adverse event grading"`) ```python # GOOD — specific, technical, distinguishable THEME_DESCRIPTION: str = ( "EU pharmaceutical variation guidelines (Official Journal, EC Regulation 1234/2008). " "Covers variation type classification (IA, IAIN, IB, II, A, BA), variation codes " "(e.g. Q.I.a.1, B.II.b.1), conditions, documentation requirements, and procedural rules. " "Distinct from BPG (best practice guidelines) and Q&A documents." ) # BAD — too vague, LLM will misclassify THEME_DESCRIPTION: str = "Pharmaceutical regulatory documents." # BAD — only one sentence, no distinguishing information THEME_DESCRIPTION: str = "Documents about clinical trials." ``` ### `SELECTABLE_MODELS` Ordering Order models from most to least specific. The LLM tends to prefer earlier entries when confidence between models is similar: 1. **List wrapper models** for multi-instance sections (most specific) 2. **Main models** for single-instance sections 3. **Supporting/complementary models** ```python SELECTABLE_MODELS: list[type] = [ VariationCodeWithDocsAndConditionModelList, # 1. multi-code sections with inline data VariationCodeModel, # 2. single-code sections DocumentationModelList, # 3. sections listing >=2 documentation requirements DocumentationModel, # 4. single documentation requirement ConditionModelList, # 5. sections listing >=2 conditions ConditionModel, # 6. single condition ProcedureTypeModelList, # 7. sections defining >=2 procedure types ProcedureTypeModel, # 8. single procedure type ] ``` > **Note:** Both `XxxModel` and `XxxModelList` must appear in `SELECTABLE_MODELS` if you create a list wrapper. If only the list wrapper is present, the agent cannot select the single-instance model directly. --- ## Creating `models.py` Model definition follows the patterns described in [Custom Models](custom-models.md). Extraction models are recommended to inherit from `ExtractionModel` (imported from `scinr.newton.models.base`), which enforces `extra="forbid"` to prevent silent LLM hallucinations. Any valid Pydantic `BaseModel` subclass will work, but `ExtractionModel` is the recommended base. ### Minimal Complete Example ```python """Clinical trial extraction models.""" from __future__ import annotations from enum import Enum from pydantic import Field from scinr.newton.models.base import ExtractionModel class TrialPhase(str, Enum): """Clinical trial phase.""" PHASE_1 = "Phase I" PHASE_2 = "Phase II" PHASE_3 = "Phase III" PHASE_4 = "Phase IV" class TrialProtocolModel(ExtractionModel): """A single clinical trial protocol entry.""" trial_id: str = Field( ..., description=( "Unique trial identifier (e.g. 'NCT01234567', 'Study-2024-001'). " "Include the full prefix as written in the document." ), json_schema_extra={"entity_label": "TrialID", "instance_key": True}, ) phase: TrialPhase | None = Field( default=None, description=( "Clinical trial phase: Phase I through Phase IV. " "None if the phase is not stated in the document." ), ) indication: str = Field( ..., description=( "Disease or condition being studied in this trial. " "Use the exact terminology from the document." ), ) inclusion_criteria: list[str] = Field( default_factory=list, description=( "Patient inclusion criteria as bullet points or numbered items. " "Each element is one criterion. Empty list if not stated." ), ) exclusion_criteria: list[str] = Field( default_factory=list, description=( "Patient exclusion criteria as bullet points or numbered items. " "Each element is one criterion. Empty list if not stated." ), ) class TrialProtocolModelList(ExtractionModel): """Use when the section defines TWO OR MORE trial protocol entries.""" items: list[TrialProtocolModel] = Field( default_factory=list, description=( "List of trial protocol entries. Each element represents one " "distinct trial. Use TrialProtocolModel when there is only one trial. " "Use this model when the section is a table or list with two or more trials." ), ) ``` ### Key Rules | Rule | Why | | :--- | :--- | | Inherit from `ExtractionModel` | Enforces `extra="forbid"` to catch LLM hallucinations | | Every field has `description=` with >= 15 words | The LLM uses the description as its primary extraction signal | | List fields use `default_factory=list` | Never use `default=[]` — mutable default shared across instances | | Optional scalars use `default=None` | Explicit opt-in for None values | | Enums use `str` as base class | Guarantees JSON-serializable values and correct Neo4j storage | | Relative imports for internal modules | `from .models import ...` — never bare `from models import ...` | For detailed model creation patterns (enums, validators, `entity_label`, `instance_relationships`, list wrappers, etc.), see [Custom Models](custom-models.md) and the [Model Creation Guide](https://github.com/scinr-ai/scinr/blob/main/src/scinr/newton/model-creation/AGENTS.md). --- ## Registering Themes There are four ways to make a custom theme available to scinr. ### Method 1: `extra_models_paths` in `configure()` Pass filesystem paths to `configure()`. The `ThemeRegistry` scans each path recursively for `catalog.py` files. ```python from scinr.newton import configure configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="password", extra_models_paths=["/path/to/my_custom_theme"], ) ``` Multiple paths: ```python configure( extra_models_paths=[ "/home/user/projects/clinical_trials", "/opt/scinr/models/device_safety", "./local_models", # relative paths resolve from the working directory ], ) ``` **How it works:** 1. `configure()` stores the paths in `ScinrConfig.extra_models_paths`. 2. On first access, `get_theme_registry()` creates a `ThemeRegistry` with these paths. 3. `ThemeRegistry._scan_extra_roots()` walks each directory recursively. 4. For each folder containing `catalog.py`, the catalog is imported and registered. > **Important:** The path you pass must be the **parent directory** of your theme folders. If your theme is at `/path/to/my_custom_theme/catalog.py`, pass `/path/to/` — not `/path/to/my_custom_theme`. The registry scans the passed directory and discovers `my_custom_theme` as a child folder. > > However, if you pass `/path/to/my_custom_theme` directly and it contains `catalog.py`, the registry will also find it as the root-level theme named `my_custom_theme`. ### Method 2: Environment Variable Set `SCINR_EXTRA_MODELS_PATHS` as a colon-separated list of paths: ```bash # .env file SCINR_EXTRA_MODELS_PATHS=/path/to/theme1:/path/to/theme2 ``` ```python from scinr.newton import configure # No extra_models_paths arg needed — read from environment configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="password", ) ``` The environment variable is read during `configure()` and converted to a list of `Path` objects. Explicit `extra_models_paths` in `configure()` takes precedence and replaces the environment variable entirely. ### Method 3: `enabled_user_themes` Use `enabled_user_themes` to whitelist specific user themes. This is useful when you have many themes in `extra_models_paths` but only want to activate a subset. ```python from scinr.newton import configure configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="password", extra_models_paths=["/path/to/my_models"], enabled_user_themes=["clinical_trials", "device_safety"], ) ``` With `enabled_user_themes`, only the listed themes from user paths are active. All other user themes discovered in `extra_models_paths` are silently excluded. Built-in themes are unaffected (use `enabled_base_themes` for those). **Combining with `enabled_base_themes`:** ```python configure( # Only these built-in themes are active enabled_base_themes=["default", "pharmaceutical_quality"], # Only these user themes are active enabled_user_themes=["clinical_trials"], extra_models_paths=["/path/to/my_models"], ) ``` > **Validation:** If a theme name in `enabled_user_themes` does not exist among the discovered user themes, `configure()` raises a `ConfigurationError`. Similarly, an empty list raises an error — pass `None` to activate all themes. ### Method 4: Entry Points (Packaged Themes) Distribute a theme as a Python package with an entry point. This is the recommended approach for sharing themes across projects or publishing them to PyPI. ```toml # pyproject.toml of your theme package [project] name = "scinr-clinical-trials" version = "0.1.0" [project.entry-points."scinr.newton.models"] clinical_trials = "scinr_clinical_trials.catalog" ``` The entry point value (`"scinr_clinical_trials.catalog"`) is the dotted import path to your `catalog` module. The `ThemeRegistry` discovers entry points from the `scinr.newton.models` group and imports them automatically. **Directory layout for a packaged theme:** ``` scinr-clinical-trials/ ├── pyproject.toml ├── src/ │ └── scinr_clinical_trials/ │ ├── __init__.py │ ├── catalog.py │ ├── models.py │ └── adverse_events/ │ ├── __init__.py │ ├── catalog.py │ └── models.py └── README.md ``` Install the package and the theme is automatically available — no `extra_models_paths` needed: ```bash pip install scinr-clinical-trials ``` ```python from scinr.newton import configure # No extra_models_paths — entry point discovery handles it configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="password", ) ``` > **Precedence:** Built-in themes take precedence over external package themes with the same name. If a conflict is detected, a warning is logged and the built-in theme is used. Rename your external theme to avoid conflicts. --- ## Verifying Theme Registration After setting up your theme, verify it was discovered correctly: ```python from scinr.newton.utils.theme_registry import get_theme_registry registry = get_theme_registry() # List all registered theme paths all_paths = registry.get_all_theme_paths() print("Registered themes:") for path in all_paths: theme = registry._themes[path] print(f" {path}: {len(theme.models)} models") # Inspect a specific theme theme = registry.find_best_theme("clinical_trials") print(f"\nTheme: {theme.path}") print(f"Description: {theme.description}") print(f"Models: {[m.__name__ for m in theme.models]}") # See the catalog block the LLM will receive catalog_block = registry.build_catalog_block(theme) print(f"\nLLM catalog block:\n{catalog_block}") ``` ### The Theme Classification Prompt You can inspect what the annotation LLM sees for theme classification: ```python from scinr.newton.utils.theme_registry import get_theme_registry registry = get_theme_registry() # The full theme list injected into the classification prompt print(registry.get_theme_list_for_prompt()) ``` Example output: ``` - clinical_trials: Clinical trial protocol documents and adverse event reports... - default: Generic fallback for content that does not fit a specific thematic domain - pharmaceutical_quality: Pharmaceutical drug development documents following ICH CTD Module 3... ``` --- ## Sub-Themes A **sub-theme** is a nested folder with its own `catalog.py`, representing a specialized subset of a parent theme. Use sub-themes when the domain has clearly differentiated document types with incompatible model sets. ### Structure ``` my_theme/ ├── __init__.py ├── catalog.py # Parent catalog (aggregates all sub-themes) ├── phase1_trials/ │ ├── __init__.py │ ├── catalog.py # Sub-theme catalog │ └── models.py └── adverse_events/ ├── __init__.py ├── catalog.py └── models.py ``` ### Parent Catalog The parent `catalog.py` aggregates models from sub-themes: ```python """Catalog for the clinical trials parent theme.""" from __future__ import annotations from .phase1_trials.models import Phase1TrialModel, Phase1TrialModelList from .adverse_events.models import AdverseEventModel, AdverseEventModelList THEME_DESCRIPTION: str = ( "Clinical trial documents across all phases and safety reporting. " "Covers Phase I first-in-human trials, adverse event reports (CTCAE grading), " "and related safety documentation. " "Distinct from regulatory variation guidelines and manufacturing records." ) SELECTABLE_MODELS: list[type] = [ Phase1TrialModelList, Phase1TrialModel, AdverseEventModelList, AdverseEventModel, ] ``` ### Sub-Theme Catalog Each sub-theme has its own `catalog.py` with a narrower `THEME_DESCRIPTION`: ```python # phase1_trials/catalog.py """Catalog for Phase I clinical trials.""" from __future__ import annotations from .models import Phase1TrialModel, Phase1TrialModelList THEME_DESCRIPTION: str = ( "Phase I first-in-human clinical trial protocols and reports. " "Covers single-ascending dose (SAD), multiple-ascending dose (MAD), " "food effect, and drug-drug interaction studies. " "Focuses on safety, tolerability, and pharmacokinetics. " "Distinct from Phase II-IV trials and adverse event narratives." ) SELECTABLE_MODELS: list[type] = [ Phase1TrialModelList, Phase1TrialModel, ] ``` ```python # adverse_events/catalog.py """Catalog for adverse event reporting.""" from __future__ import annotations from .models import AdverseEventModel, AdverseEventModelList THEME_DESCRIPTION: str = ( "Adverse event reports and safety narratives from clinical trials. " "Covers CTCAE grading (Grade 1-5), Serious Adverse Events (SAE), " "adverse drug reactions (ADR), and causality assessments. " "Distinct from trial protocol design and pharmacokinetic data." ) SELECTABLE_MODELS: list[type] = [ AdverseEventModelList, AdverseEventModel, ] ``` ### When to Use Sub-Themes | Use sub-themes when... | Keep in same theme when... | | :--- | :--- | | Document types are clearly differentiated | Models are complementary and used together | | Model sets are incompatible (different field structures) | Sections often mix entity types from both domains | | Each sub-domain has its own regulatory standards | The distinction is artificial or marginal | | You want finer-grained theme classification | A single `THEME_DESCRIPTION` covers the domain well | ### How Sub-Themes Are Discovered The `ThemeRegistry` registers both the parent and each sub-theme independently: ``` my_theme/ → registered as "my_theme" ├── catalog.py ├── phase1_trials/ → registered as "my_theme/phase1_trials" │ └── catalog.py └── adverse_events/ → registered as "my_theme/adverse_events" └── catalog.py ``` During classification, the LLM receives all three theme descriptions. If a section matches the narrow sub-theme description, it gets classified to the sub-theme path. If it matches only the parent description, it gets the parent path. The `find_best_theme()` method resolves from most specific to least specific: ```python # If the LLM classifies a section as "my_theme/phase1_trials": registry.find_best_theme("my_theme/phase1_trials") # exact match → sub-theme # If the LLM classifies a section as "my_theme": registry.find_best_theme("my_theme") # exact match → parent theme ``` --- ## Theme Discovery Flow Understanding the complete flow helps with debugging: ``` 1. configure(extra_models_paths=["/path/to/models"]) │ 2. ThemeRegistry.__init__() │ ├── Scans built-in models/ for catalog.py files │ └── Imports via importlib: "scinr.newton.models..catalog" │ ├── Applies enabled_base_themes filter (if set) │ ├── Discovers entry-point packages (scinr.newton.models group) │ ├── Scans extra_models_paths for catalog.py files │ └── Imports via importlib.util (works for any filesystem path) │ ├── Package layout: uses __init__.py chain + importlib.import_module │ └── Standalone layout: uses spec_from_file_location + sys.path manipulation │ └── Applies enabled_user_themes filter (if set) │ 3. Stage 1 (extraction) begins │ ├── get_theme_list_for_prompt() → theme descriptions for classification │ 4. For each structural node (Stage 1): │ ├── LLM reads theme descriptions → picks best theme for this node ├── find_best_theme(detected_path) → resolves to ThemeNode │ 5. Stage 3 (annotation) begins │ ├── build_catalog_block(theme) → model catalog for annotation LLM ├── LLM selects model from SELECTABLE_MODELS │ 6. Stage 4 (extraction) │ └── Selected model → extracts structured entities ``` ### Package Layout vs. Standalone Layout When loading user themes from `extra_models_paths`, the registry supports two layouts: **Package layout** (has `__init__.py` files): ``` my_models/ ├── __init__.py └── clinical_trials/ ├── __init__.py ├── catalog.py ← imported as "my_models.clinical_trials.catalog" └── models.py ``` Relative imports (`from .models import ...`) work correctly because Python's normal package machinery handles them. Pass `/path/to/` (parent of `my_models`) as `extra_models_paths`. **Standalone layout** (no `__init__.py`): ``` my_models/ └── clinical_trials/ ├── catalog.py ← loaded via spec_from_file_location └── models.py ``` The catalog's directory is temporarily added to `sys.path` so bare sibling imports (`from models import ...`) resolve. Pass `/path/to/my_models` as `extra_models_paths`. > **Recommendation:** Always use the package layout with `__init__.py` files. It is more robust and avoids `sys.path` manipulation. --- ## Troubleshooting | Problem | Cause | Fix | | :--- | :--- | :--- | | Theme not discovered | Missing `__init__.py` in a directory | Add empty `__init__.py` to every directory containing `.py` files | | Theme not discovered | Path in `extra_models_paths` is wrong | Pass the parent directory of the theme folder, not the theme folder itself (unless the theme folder has `catalog.py` directly) | | Import errors on startup | Wrong relative import in `catalog.py` | Use dots: `from .models import ...` not `from models import ...` | | Import errors on startup | Wrong relative import in `models.py` | Count dots from the file's directory: `from ..base import ...` for parent, `from ...base import ...` for grandparent | | Model not selectable by annotation agent | Model not in `SELECTABLE_MODELS` | Add the model class to `SELECTABLE_MODELS` in `catalog.py` | | Wrong model applied to sections | Vague `THEME_DESCRIPTION` | Be specific: name document types, regulatory standards, distinguish from adjacent themes | | Wrong model applied to sections | Models in wrong order in `SELECTABLE_MODELS` | Order from most specific (list wrappers) to least specific (single-instance) | | `ConfigurationError: not a subclass of pydantic.BaseModel` | Model in `SELECTABLE_MODELS` doesn't inherit from `BaseModel` | Ensure all models inherit from `ExtractionModel` or any `BaseModel` subclass | | Built-in theme silently overridden | User theme has same name as built-in theme | Rename your theme folder or adjust `extra_models_paths` | | Entry-point theme not loading | Package not installed or entry-point misconfigured | Verify `pip show ` and check `[project.entry-points."scinr.newton.models"]` in `pyproject.toml` | ### Debug Logging Enable debug logging to see the theme discovery process: ```python import logging from scinr.newton import configure logging.basicConfig(level=logging.DEBUG) configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="password", extra_models_paths=["/path/to/my_models"], ) ``` Look for lines like: ``` DEBUG:scinr.newton.utils.theme_registry:ThemeRegistry: registered theme 'clinical_trials' with 5 models DEBUG:scinr.newton.utils.theme_registry:ThemeRegistry: discovered 8 themes: ['clinical_trials', 'default', ...] ``` ### Common Import Errors ```python # catalog.py — CORRECT from .models import TrialProtocolModel # catalog.py — WRONG (bare import, depends on sys.path) from models import TrialProtocolModel # models.py — CORRECT (one level up) from ..baseModels import NormalizedBaseModel # models.py — CORRECT (installed package, absolute is fine) from scinr.newton.models.base import ExtractionModel ``` ### Verifying a Specific Theme ```python from scinr.newton.utils.theme_registry import get_theme_registry registry = get_theme_registry() # Check if a theme exists if "clinical_trials" in registry._themes: theme = registry._themes["clinical_trials"] print(f"Found: {theme.path}") print(f" Description: {theme.description}") print(f" Models: {[m.__name__ for m in theme.models]}") print(f" Children: {list(theme.children.keys())}") else: print("Theme 'clinical_trials' not found.") print("Available themes:", list(registry._themes.keys())) ``` --- ## Complete Example: End-to-End Theme Here is a complete custom theme from scratch, ready to use. ### Directory Layout ``` own_models/ ├── __init__.py └── clinical_trials/ ├── __init__.py ├── catalog.py └── models.py ``` ### `own_models/__init__.py` Empty file. Required to make `own_models` a Python package. ### `own_models/clinical_trials/__init__.py` Empty file. Required to make `clinical_trials` a Python package. ### `own_models/clinical_trials/models.py` ```python """Clinical trial extraction models.""" from __future__ import annotations from enum import Enum from pydantic import Field from scinr.newton.models.base import ExtractionModel class TrialPhase(str, Enum): """Clinical trial phase.""" PHASE_1 = "Phase I" PHASE_2 = "Phase II" PHASE_3 = "Phase III" PHASE_4 = "Phase IV" class AdverseEventGrade(str, Enum): """CTCAE adverse event severity grade.""" GRADE_1 = "Grade 1" GRADE_2 = "Grade 2" GRADE_3 = "Grade 3" GRADE_4 = "Grade 4" GRADE_5 = "Grade 5" class TrialProtocolModel(ExtractionModel): """A single clinical trial protocol entry.""" trial_id: str = Field( ..., description=( "Unique trial identifier (e.g. 'NCT01234567', 'Study-2024-001'). " "Include the full prefix as written in the document." ), json_schema_extra={"entity_label": "TrialID", "instance_key": True}, ) phase: TrialPhase | None = Field( default=None, description=( "Clinical trial phase: Phase I through Phase IV. " "None if the phase is not stated in the document." ), ) indication: str = Field( ..., description=( "Disease or condition being studied in this trial. " "Use the exact terminology as written in the document." ), ) primary_endpoint: str | None = Field( default=None, description=( "Primary efficacy endpoint of the trial. " "None if not explicitly stated in the section." ), ) class TrialProtocolModelList(ExtractionModel): """Use when the section defines TWO OR MORE trial protocol entries.""" items: list[TrialProtocolModel] = Field( default_factory=list, description=( "List of trial protocol entries. Each element represents one " "distinct trial. Use TrialProtocolModel for a single trial. " "Use this model when the section is a table or list with two or more trials." ), ) class AdverseEventModel(ExtractionModel): """A single adverse event entry from a clinical trial report.""" event_term: str = Field( ..., description=( "Preferred term for the adverse event as written in the document " "(e.g. 'headache', 'nausea', 'anaphylactic reaction'). " "Use the exact terminology from the source." ), json_schema_extra={"entity_label": "AdverseEventTerm"}, ) grade: AdverseEventGrade | None = Field( default=None, description=( "CTCAE severity grade: Grade 1 (mild) through Grade 5 (death). " "None if the grade is not stated." ), ) causality: str | None = Field( default=None, description=( "Assessed causality relationship to the study drug " "(e.g. 'related', 'possibly related', 'unrelated'). " "None if causality is not assessed." ), ) outcome: str | None = Field( default=None, description=( "Outcome of the adverse event (e.g. 'resolved', 'not resolved', " "'resolved with sequelae', 'fatal'). None if not stated." ), ) class AdverseEventModelList(ExtractionModel): """Use when the section defines TWO OR MORE adverse event entries.""" items: list[AdverseEventModel] = Field( default_factory=list, description=( "List of adverse event entries. Each element represents one " "distinct event. Use AdverseEventModel for a single event. " "Use this model when the section is a table or list with two or more events." ), ) ``` ### `own_models/clinical_trials/catalog.py` ```python """Catalog for the clinical trials theme.""" from __future__ import annotations from .models import ( TrialProtocolModel, TrialProtocolModelList, AdverseEventModel, AdverseEventModelList, ) THEME_DESCRIPTION: str = ( "Clinical trial protocol documents and adverse event reports. " "Covers trial phases (I-IV), inclusion/exclusion criteria, " "adverse event grading (CTCAE), dosage regimens, and endpoints. " "Use for sections describing trial design, patient populations, " "safety data, and dosing schedules. " "Distinct from regulatory variation guidelines and manufacturing records." ) SELECTABLE_MODELS: list[type] = [ TrialProtocolModelList, # multi-trial sections (most specific first) TrialProtocolModel, # single trial section AdverseEventModelList, # multi-event sections AdverseEventModel, # single event ] ``` ### Usage ```python import asyncio from pathlib import Path from scinr.newton import configure, run_pipeline async def main(): configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="your_password", extra_models_paths=[str(Path(__file__).parent / "own_models")], ) # Verify theme registration from scinr.newton.utils.theme_registry import get_theme_registry registry = get_theme_registry() print("Themes:", registry.get_all_theme_paths()) # Run pipeline with custom theme result = await run_pipeline(input_raw="./raw_docs") print(f"Pipeline result: {result.success}") asyncio.run(main()) ``` --- ## See Also - **[Custom Models](custom-models.md)** — Defining domain-specific Pydantic extraction models. - **[Configuration](../configuration.md)** — Complete reference for `configure()`, including `extra_models_paths`, `enabled_base_themes`, and `enabled_user_themes`. - **[Running the Pipeline](running-pipeline.md)** — Orchestrating the full ingestion pipeline with custom themes. - **[Architecture](../architecture.md)** — Detailed walkthrough of Stage 1 extraction (theme classification) and Stage 3 annotation (model selection). - **[Model Creation Guide](https://github.com/scinr-ai/scinr/blob/main/src/scinr/newton/model-creation/AGENTS.md)** — Comprehensive guide for AI agents creating extraction models. --- ## File: user-guides/document-deletion.md # Document Deletion `delete_document()` permanently removes a `:Document` node and its entire subgraph from Neo4j, then runs garbage collection on orphaned nodes. This is the definitive way to remove a document from your knowledge graph — it is irreversible and cannot be undone. --- ## Introduction The `delete_document()` function is a standalone synchronous operation that: 1. **Locates** the target `:Document` node(s) by `path` (and optionally `version`). 2. **Cascade-deletes** the document and every node reachable from it: - Folder-parent documents and siblings via `IS_COMPOSED_OF*` - All `:StructureNode` descendants via `HAS_STRUCTURE*` / `HAS_CHILD*` - All `:InfoUnit`, `:ModelDecision`, `:ProposedModel`, `:ProposedField`, and `:ExtractionResult` children 3. **Garbage-collects** orphaned `:Entity`, `:ModelInstance`, and `:LabeledEntity` nodes in two independent passes. The function opens and closes its own Neo4j driver — you do not need to manage connections manually. --- ## When to Use Deletion vs. Update `scinr` provides two mechanisms for replacing document content: | Operation | What it does | Use when | |---|---|---| | `delete_document()` | Permanently removes the `:Document` node and all descendants. No undo. | The document should no longer exist in the graph at all. | | `--update` re-ingestion | Keeps the `:Document` node, wipes its content, and re-ingests new data. | You want to refresh the content of an existing document while preserving its identity and version history. | If you simply need to update content, use the `--update` flag with `run_pipeline()`. Use `delete_document()` only when you want complete, permanent removal. --- ## Basic Usage ```python from scinr.newton import delete_document, configure, DeletionResult configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="password", ) result = delete_document("/path/to/document.pdf") print(f"Found: {result.found}") print(f"Documents deleted: {result.documents_deleted}") ``` The `path` parameter matches the `path` property on `:Document` nodes in Neo4j. This is the file path (relative or absolute) as it was recorded at ingestion time. --- ## Version-Targeted Deletion By default, `delete_document()` deletes **all versions** of a document matching the given `path`: ```python # Delete ALL versions of a document result = delete_document("/path/to/document.pdf") ``` To delete a **specific version**, pass the `version` parameter: ```python # Delete only version 2 result = delete_document("/path/to/document.pdf", version=2) ``` When `version` is specified, only that version's `:Document` node and its cascade are removed. Other versions of the same document remain untouched. --- ## Understanding the Cascade When you call `delete_document()`, the following nodes are deleted in a single transaction: ### Target Document(s) The `:Document` node(s) matching the `path` (and `version`, if specified). If the document is part of a folder hierarchy, every `:Document` reachable via `IS_COMPOSED_OF*` is also deleted — this includes folder-parent documents and their sibling documents. ### Structure Tree For each deleted document, all descendants are removed: - `:StructureNode` nodes reached via `HAS_STRUCTURE*` and `HAS_CHILD*` - `:InfoUnit` nodes attached to those structure nodes - `:ModelDecision` nodes (annotation results) - `:ProposedModel` and `:ProposedField` nodes (annotation details) - `:ExtractionResult` nodes (entity extraction results) ### Visual Representation ``` (:Document {path: "/path/to/document.pdf"}) │ ├─[:IS_COMPOSED_OF]→ (:Document) [folder parent — also deleted] │ └─[:HAS_STRUCTURE]→ (:StructureNode) ├─[:HAS_CHILD]→ (:StructureNode) │ ├─[:HAS_INFO_UNIT]→ (:InfoUnit) │ ├─[:HAS_MODEL_DECISION]→ (:ModelDecision) │ │ ├─[:HAS_PROPOSED_MODEL]→ (:ProposedModel) │ │ │ └─[:HAS_PROPOSED_FIELD]→ (:ProposedField) │ └─[:HAS_EXTRACTION]→ (:ExtractionResult) └─[:HAS_CHILD]→ (:StructureNode) ``` All of the above are `DETACH DELETE`d in a single query, meaning all their relationships are severed before the nodes are removed. --- ## Garbage Collection After the cascade delete, two independent garbage-collection passes run to clean up orphaned nodes that were not directly connected to the deleted documents. ### Pass 1: Entity / ModelInstance Finds `:Entity` and `:ModelInstance` nodes that are no longer reachable from any `:ExtractionResult` within 7 hops: ```cypher MATCH (mi:Entity|ModelInstance) WHERE NOT EXISTS { MATCH (e:ExtractionResult)-[*1..7]->(mi) } DETACH DELETE mi ``` ### Pass 2: LabeledEntity Finds `:LabeledEntity` nodes with no incoming relationships at all: ```cypher MATCH (mi:LabeledEntity) WHERE NOT EXISTS { (mi)<--() } DETACH DELETE mi ``` ### Iteration Behavior Each pass runs up to **7 iterations** (`GC_MAX_PASSES = 7`). A pass stops early as soon as an iteration deletes zero nodes. This handles cascading orphans — deleting a batch of `:Entity` nodes might reveal new orphaned `:LabeledEntity` nodes that were only reachable through the deleted entities. --- ## Inspecting DeletionResult `delete_document()` returns a `DeletionResult` dataclass with detailed counters: | Field | Type | Description | |---|---|---| | `path` | `str` | The document `path` that was targeted for deletion. | | `version` | `int \| None` | The specific version requested, or `None` if all versions were targeted. | | `found` | `bool` | `True` if at least one matching `:Document` existed before deletion. When `False`, all counters are `0` and no queries were executed. | | `versions_deleted` | `list[int]` | Sorted list of integer versions that matched and were deleted. Empty when `found` is `False`. | | `documents_deleted` | `int` | Number of `:Document` nodes deleted (matched documents plus any reached via `IS_COMPOSED_OF*`). | | `structure_nodes_deleted` | `int` | Number of `:StructureNode` nodes deleted. | | `info_units_deleted` | `int` | Number of `:InfoUnit` nodes deleted. | | `model_decisions_deleted` | `int` | Number of `:ModelDecision` nodes deleted. | | `proposed_models_deleted` | `int` | Number of `:ProposedModel` nodes deleted. | | `proposed_fields_deleted` | `int` | Number of `:ProposedField` nodes deleted. | | `extraction_results_deleted` | `int` | Number of `:ExtractionResult` nodes deleted. | | `gc_entity_model_instance_deleted` | `int` | Total `:Entity`/`:ModelInstance` nodes deleted across all GC iterations. | | `gc_entity_model_instance_passes` | `int` | Number of GC iterations actually run for the Entity/ModelInstance pass (capped at 7). | | `gc_labeled_entity_deleted` | `int` | Total `:LabeledEntity` nodes deleted across all GC iterations. | | `gc_labeled_entity_passes` | `int` | Number of GC iterations actually run for the LabeledEntity pass (capped at 7). | ### Example Output ```python result = delete_document("/path/to/document.pdf") if result.found: print(f"Deleted {result.documents_deleted} document(s), " f"{result.structure_nodes_deleted} structure node(s)") print(f"GC cleaned up {result.gc_entity_model_instance_deleted} entity/model instance(s) " f"and {result.gc_labeled_entity_deleted} labeled entity(s)") else: print("No document found at that path.") ``` --- ## Important Caveats ### Irreversible Operation `delete_document()` uses `DETACH DELETE` — once nodes are removed, they cannot be recovered. There is no undo mechanism. Always verify the target `path` and `version` before calling. ### No Undo Unlike `--update` re-ingestion (which preserves the `:Document` node and allows you to re-run the pipeline), `delete_document()` removes the document entirely. If you need the document back, you must re-ingest it from the original source file. ### Shared LabeledEntity Deduplication `:LabeledEntity` nodes are globally deduplicated — the same entity value from multiple documents shares a single node. The garbage collection pass only removes `:LabeledEntity` nodes that have **no incoming relationships at all**. If the same labeled entity appears in other documents that remain in the graph, it will **not** be deleted. This is intentional and preserves cross-document entity integrity. ### IS_COMPOSED_OF Cascade Scope If the target document is part of a folder hierarchy (connected via `IS_COMPOSED_OF`), the cascade delete reaches **all** documents connected through that relationship — including folder-parent documents and their siblings. This means deleting a leaf document in a folder hierarchy may also delete the parent folder document and its other children. If you need to delete only a single document without affecting its folder hierarchy, consider using `--update` re-ingestion instead, or manually manage the folder structure before deletion. ### Version Isolation When `version` is specified, only that version's cascade is deleted. However, shared `:LabeledEntity` nodes connected to other versions are preserved by the GC pass (they still have incoming relationships from the remaining versions). ### Driver Management `delete_document()` opens its own Neo4j driver via `get_driver()` and closes it in a `finally` block. You do not need to manage driver lifecycle manually. However, if you are calling `delete_document()` in a tight loop, consider the connection overhead — each call creates and closes a driver. --- ## See Also - **[Neo4j Graph Storage](neo4j-graph.md)** — Understanding the graph model, node types, and relationships affected by deletion. - **[Running the Pipeline](running-pipeline.md)** — Pipeline entry points, including the `--update` flag for in-place document updates. - **[Deletion API](../api/deletion.md)** — Auto-generated reference for `delete_document()`. - **[Results API](../api/results.md)** — `DeletionResult` dataclass reference. - **[Architecture](../architecture.md)** — Pipeline stages and Neo4j schema details. --- ## File: user-guides/graph-relationships.md # Graph Relationships Define edges between extracted entities in your Pydantic extraction models. Relationships declared in `json_schema_extra` become Neo4j graph edges during Stage 4 (entity extraction). --- ## Overview `scinr.newton` builds a Neo4j knowledge graph from extracted entities. Relationships between entities are declared in model field definitions using `json_schema_extra`. Two relationship types exist: | Type | Connects | Scope | Node Type | | :--- | :--- | :--- | :--- | | **`field_relationships`** | Two fields within the same model instance | Intra-model | `:LabeledEntity` — `[:REL_TYPE]`→ `:LabeledEntity` | | **`instance_relationships`** | Model instances across sections or documents | Cross-model | `:ModelInstance` — `[:REL_TYPE]`→ `:ModelInstance` | Both are declared inside `json_schema_extra` on the **source field** of the relationship. ```python from pydantic import Field from scinr.newton.models.base import ExtractionModel class Example(ExtractionModel): """Example model showing both relationship types.""" source_field: str = Field( ..., description="...", json_schema_extra={ "entity_label": "SourceLabel", # Level 2: connects two :LabeledEntity nodes in this model "field_relationships": [ {"to_field": "target_field", "rel_type": "RELATED_TO"}, ], # Level 3: connects this :ModelInstance to another :ModelInstance "instance_relationships": [ { "target_model": "OtherModel", "join_via": {"source_field": "key_field"}, "rel_type": "LINKS_TO", } ], }, ) target_field: str = Field( ..., description="...", json_schema_extra={"entity_label": "TargetLabel"}, ) ``` --- ## Decision Matrix | Question | Answer | Use | | :--- | :--- | :--- | | Both fields from the same structural section? | Yes | `field_relationships` | | Fields from different sections or documents? | Yes | `instance_relationships` | | Relationship between named entities (`:LabeledEntity`)? | Yes | `field_relationships` | | Relationship between model records (`:ModelInstance`)? | Yes | `instance_relationships` | | Need forward reference (target may not exist yet)? | Yes | `instance_relationships` | --- ## Three Levels of Graph Construction Entity extraction builds the graph in three passes: | Level | Mechanism | What it creates | | :--- | :--- | :--- | | **Level 1** | `entity_label` | `:LabeledEntity` nodes (globally deduplicated by label + normalized value) | | **Level 2** | `field_relationships` | Edges between `:LabeledEntity` nodes within the same model instance | | **Level 3** | `instance_relationships` | Edges between `:ModelInstance` nodes across sections/documents | All three levels work together. A single field can participate in all three simultaneously. --- ## `field_relationships` — Intra-Model Entity Edges Connects two `:LabeledEntity` nodes within the same extracted model instance. Both fields must be siblings in the same model and both must have `entity_label`. ### Syntax ```python class VariationLink(ExtractionModel): """A parent-child variation code relationship.""" root_code: str | None = Field( default=None, description=( "Parent variation code (e.g. 'Q.I.a.1'). " "None if this is a top-level code with no parent." ), json_schema_extra={ "entity_label": "VariationCode", "field_relationships": [ {"to_field": "child_code", "rel_type": "HAS_CHILD_VARIATION"}, ], }, ) child_code: str = Field( ..., description="Child variation code (e.g. 'Q.I.a.1(a)').", json_schema_extra={"entity_label": "VariationCode", "instance_key": True}, ) ``` ### Resulting Neo4j Graph ``` (:LabeledEntity:VariationCode {value:"Q.I.a.1"}) -[:HAS_CHILD_VARIATION]-> (:LabeledEntity:VariationCode {value:"Q.I.a.1(a)"}) ``` Both entities share the same label (`VariationCode`) but have different values. The relationship connects them directionally. ### Rules - Both source and target fields **must** have `entity_label` set - `to_field` must be the **exact Python name** of a sibling field in the same model - Relationship is only created when **both** fields are non-null - `rel_type` must be `UPPER_SNAKE_CASE` - Multiple relationships can be declared on the same field (list of dicts) ### Multiple Relationships from One Field A single field can declare relationships to multiple sibling fields: ```python class DrugInteraction(ExtractionModel): """A drug-drug interaction record.""" source_drug: str = Field( ..., description="The initiating drug in the interaction.", json_schema_extra={ "entity_label": "Drug", "field_relationships": [ {"to_field": "target_drug", "rel_type": "INTERACTS_WITH"}, {"to_field": "mechanism", "rel_type": "HAS_MECHANISM"}, ], }, ) target_drug: str = Field( ..., description="The affected drug in the interaction.", json_schema_extra={"entity_label": "Drug"}, ) mechanism: str = Field( ..., description="Biological mechanism of the interaction.", json_schema_extra={"entity_label": "Mechanism"}, ) severity: str | None = Field( default=None, description="Severity level of the interaction.", ) ``` ### Resulting Neo4j Graph ``` (:LabeledEntity:Drug {value:"Warfarin"}) -[:INTERACTS_WITH]-> (:LabeledEntity:Drug {value:"Amiodarone"}) (:LabeledEntity:Drug {value:"Warfarin"}) -[:HAS_MECHANISM]-> (:LabeledEntity:Mechanism {value:"CYP2C9 inhibition"}) ``` Note: `severity` has no `entity_label`, so it becomes a scalar property on the `:ModelInstance` node — no entity node is created for it. --- ## `instance_relationships` — Cross-Model Record Edges Connects `:ModelInstance` nodes across different sections or documents. Creates **shell nodes** for targets that have not yet been extracted, which automatically merge with the real nodes when the target model is later extracted. ### Syntax — Simple 1:1 ```python class DocumentReference(ExtractionModel): """A reference to a supporting document.""" reference_id: str = Field( ..., description="Unique reference identifier for this document.", json_schema_extra={"entity_label": "ReferenceID", "instance_key": True}, ) target_variation_code: str = Field( ..., description="The variation code this document supports.", json_schema_extra={ "entity_label": "VariationCode", "instance_relationships": [ { "target_model": "VariationModel", "join_via": { "target_variation_code": "variation_code", }, "rel_type": "SUPPORTS_VARIATION", } ], }, ) ``` ### Resulting Neo4j Graph ``` (:ModelInstance:DocumentReference {reference_id:"REF-001"}) -[:SUPPORTS_VARIATION]-> (:ModelInstance:VariationModel {variation_code:"Q.I.a.1"}) ``` The source is the `DocumentReference` model instance. The target is a `VariationModel` model instance, identified by its `instance_key` field (`variation_code`). ### Shell Node Mechanics When the source model is extracted before the target model exists, `scinr` creates a **shell node**: a `:ModelInstance` with only the key fields populated. When the target model is later extracted from its own section, the shell node is found via `MERGE` and enriched with the remaining fields. ``` Step 1 (DocumentReference extracted first): (:ModelInstance {uid:"...", model_class:"VariationModel", variation_code:"Q.I.a.1"}) └─ shell node — only key fields set Step 2 (VariationModel extracted from its own section): (:ModelInstance {uid:"...", model_class:"VariationModel", variation_code:"Q.I.a.1", description:"...", procedure_type:"IA"}) └─ shell merged with real data — same uid, all fields populated ``` ### Rules - `target_model` is the **class name** as a string (PascalCase, no module path) - `join_via` maps local field names → target model's `instance_key` field names - All fields referenced in `join_via` on the target model **must** have `instance_key: True` - Creates shell nodes for targets not yet extracted - Shell nodes merge with real nodes when the target model is later extracted - Field names in `join_via` must **exactly** match the Python field names ### Target Model Requirements The target model must mark its key fields with `instance_key: True`: ```python class VariationModel(ExtractionModel): """A single pharmaceutical variation entry.""" variation_code: str = Field( ..., description="The variation code identifier (e.g. 'Q.I.a.1').", json_schema_extra={ "entity_label": "VariationCode", "instance_key": True, # ← REQUIRED for instance_relationships resolution }, ) description: str = Field( ..., description="Full description of the variation as stated in the document.", ) procedure_type: str | None = Field( default=None, description="Procedure type code: IA, IAIN, IB, II, A, or BA.", ) ``` Without `instance_key: True`, shell nodes created by `instance_relationships` will never merge with the real nodes when the target model is extracted. --- ## Fan-Out Pattern (One-to-Many via List) When the source field is a `list[str]`, one target `:ModelInstance` is created per list item. This is the primary pattern for one-to-many relationships. ### Syntax ```python class ConditionGroup(ExtractionModel): """A group of conditions for a variation.""" variation_code: str = Field( ..., description="Parent variation code that scopes these conditions.", json_schema_extra={"entity_label": "VariationCode", "instance_key": True}, ) condition_ids: list[str] = Field( default_factory=list, description="IDs of associated conditions (e.g. ['1', '2', 'A']).", json_schema_extra={ "instance_relationships": [ { "target_model": "ConditionModel", "join_via": { "variation_code": "variation_code", # fixed anchor key "condition_ids": "condition_id", # fan-out key }, "rel_type": "HAS_CONDITION", } ], }, ) ``` ### Resulting Neo4j Graph ``` (:ModelInstance:ConditionGroup {variation_code:"Q.I.a.1"}) -[:HAS_CONDITION]-> (:ModelInstance:ConditionModel {variation_code:"Q.I.a.1", condition_id:"1"}) (:ModelInstance:ConditionGroup {variation_code:"Q.I.a.1"}) -[:HAS_CONDITION]-> (:ModelInstance:ConditionModel {variation_code:"Q.I.a.1", condition_id:"2"}) (:ModelInstance:ConditionGroup {variation_code:"Q.I.a.1"}) -[:HAS_CONDITION]-> (:ModelInstance:ConditionModel {variation_code:"Q.I.a.1", condition_id:"A"}) ``` One `ConditionGroup` produces three `:HAS_CONDITION` edges to three different `ConditionModel` shell nodes. ### Rules - **Fixed keys**: scalar fields in `join_via` that are NOT the annotated field itself. These scope the target to the correct parent context. - **Fan-out key**: the annotated list field itself. One target `:ModelInstance` is created per list item. - Field names in `join_via` must **exactly** match Python field names - If a fixed key field is `None` or empty string, **no relationships** are created for that instance (logged as warning) ### Composite Fan-Out with Multiple Fixed Keys ```python class FeeSchedule(ExtractionModel): """A fee schedule for regulatory procedures.""" country_code: str = Field( ..., description="ISO country code (e.g. 'BE', 'DE', 'FR').", json_schema_extra={"entity_label": "Country", "instance_key": True}, ) procedure_type: str = Field( ..., description="Procedure type: IA, IB, II, IAIN, A, or BA.", json_schema_extra={"entity_label": "ProcedureType", "instance_key": True}, ) fee_roles: list[str] = Field( default_factory=list, description="Fee roles applicable for this country/procedure combination.", json_schema_extra={ "instance_relationships": [ { "target_model": "FeeModel", "join_via": { "country_code": "country_code", # fixed key 1 "procedure_type": "procedure_type", # fixed key 2 "fee_roles": "role", # fan-out key }, "rel_type": "HAS_FEE", } ], }, ) ``` Each `FeeModel` target is identified by a composite key: `(country_code, procedure_type, role)`. The fan-out creates one edge per role. --- ## Dual Pattern: `entity_label` + `instance_relationships` A field can simultaneously create a `:LabeledEntity` global singleton **and** a `:ModelInstance` cross-model edge. Use this when the value is both a named entity (needs global dedup) AND points to a structured model instance (needs cross-document linking). ### Syntax ```python class VariationRecord(ExtractionModel): """A variation with a linked procedure type.""" variation_code: str = Field( ..., description="Variation code identifier.", json_schema_extra={"entity_label": "VariationCode", "instance_key": True}, ) procedure_type: str = Field( default="", description="Procedure type code: IA, IB, II, IAIN, A, or BA.", json_schema_extra={ "entity_label": "ProcedureType", # → creates :LabeledEntity node "instance_relationships": [ { "target_model": "ProcedureTypeModel", "join_via": {"procedure_type": "procedure_type"}, "rel_type": "HAS_PROCEDURE_TYPE", } ], # → creates :ModelInstance edge }, ) ``` ### Resulting Neo4j Graph ``` Level 1 (entity_label): (:ModelInstance:VariationRecord)-[:REFERENCES]->(:LabeledEntity:ProcedureType {value:"IA"}) Level 3 (instance_relationships): (:ModelInstance:VariationRecord)-[:HAS_PROCEDURE_TYPE]->(:ModelInstance:ProcedureTypeModel {procedure_type:"IA"}) ``` The same field value ("IA") produces: 1. A `:LabeledEntity:ProcedureType` node (globally deduplicated — all "IA" values across all documents point to the same node) 2. A `:HAS_PROCEDURE_TYPE` edge from the `VariationRecord` instance to a `ProcedureTypeModel` instance ### When to Use Use the dual pattern when: - The value is a stable identifier that appears across many documents (justifies `entity_label`) - The value also represents a structured concept with its own model (justifies `instance_relationships`) - You want both cross-document entity deduplication AND cross-model structural linking --- ## Relationship Type Naming ### Conventions - Always `UPPER_SNAKE_CASE` - Descriptive direction: the relationship type reads naturally from source to target - Consistent across models: the same relationship type always has the same semantic meaning ### Good Examples | Relationship Type | Reads as | | :--- | :--- | | `HAS_CHILD_VARIATION` | parent HAS_CHILD_VARIATION child | | `SUPPORTS_VARIATION` | document SUPPORTS_VARIATION variation | | `HAS_CONDITION` | variation HAS_CONDITION condition | | `BELONGS_TO` | child BELONGS_TO parent | | `NORMALIZES` | raw value NORMALIZES canonical value | | `INTERACTS_WITH` | drug A INTERACTS_WITH drug B | | `HAS_FEE` | schedule HAS_FEE fee entry | | `HAS_PROCEDURE_TYPE` | record HAS_PROCEDURE_TYPE procedure type | ### Bad Examples | Relationship Type | Problem | | :--- | :--- | | `rel1` | Not descriptive | | `has_child` | Not UPPER_SNAKE_CASE | | `variation_to_condition` | Not UPPER_SNAKE_CASE | | `LINKS` | Too generic — what kind of link? | --- ## Common Patterns — Complete Examples ### Pattern 1: Simple `field_relationships` (Two Entities in Same Model) Two named entities in the same model, connected by a directed edge: ```python class IngredientPair(ExtractionModel): """An active ingredient and its excipient.""" active_ingredient: str = Field( ..., description=( "Name of the active pharmaceutical ingredient. " "Extract the INN or official name without modifications." ), json_schema_extra={ "entity_label": "Substance", "field_relationships": [ {"to_field": "excipient", "rel_type": "HAS_EXCIPIENT"}, ], }, ) excipient: str = Field( ..., description=( "Name of the excipient used with the active ingredient. " "Extract the full chemical or trade name." ), json_schema_extra={"entity_label": "Substance"}, ) concentration: str | None = Field( default=None, description=( "Concentration of the active ingredient in the formulation " "(e.g. '50 mg/g', '10% w/w'). None if not stated." ), ) ``` **Graph:** ``` (:LabeledEntity:Substance {value:"Metformin"}) -[:HAS_EXCIPIENT]-> (:LabeledEntity:Substance {value:"Microcrystalline cellulose"}) ``` --- ### Pattern 2: Simple `instance_relationships` (Cross-Model Join) A document reference linking to a variation defined in another section: ```python class SupportingDocument(ExtractionModel): """A reference to a document supporting a variation.""" doc_title: str = Field( ..., description="Title of the supporting document as stated in the text.", json_schema_extra={"entity_label": "DocumentTitle"}, ) linked_variation: str = Field( ..., description=( "Variation code this document supports (e.g. 'B.II.b.1'). " "This code links to a VariationModel in another section." ), json_schema_extra={ "entity_label": "VariationCode", "instance_relationships": [ { "target_model": "VariationModel", "join_via": { "linked_variation": "variation_code", }, "rel_type": "SUPPORTS_VARIATION", } ], }, ) ``` **Graph:** ``` (:ModelInstance:SupportingDocument {doc_title:"Stability Study"}) -[:SUPPORTS_VARIATION]-> (:ModelInstance:VariationModel {variation_code:"B.II.b.1"}) ``` --- ### Pattern 3: Fan-Out (One-to-Many via List) A variation code with multiple associated conditions: ```python class VariationWithConditions(ExtractionModel): """A variation code with its applicable conditions.""" variation_code: str = Field( ..., description=( "Full variation code (e.g. 'Q.I.a.1'). " "Always include the top-level prefix." ), json_schema_extra={"entity_label": "VariationCode", "instance_key": True}, ) condition_ids: list[str] = Field( default_factory=list, description=( "Condition numbers applicable to this variation " "(e.g. ['1', '2', '3']). Empty list if none." ), json_schema_extra={ "instance_relationships": [ { "target_model": "ConditionModel", "join_via": { "variation_code": "variation_code", "condition_ids": "condition_number", }, "rel_type": "HAS_CONDITION", } ], }, ) ``` **Graph:** ``` (:ModelInstance:VariationWithConditions {variation_code:"Q.I.a.1"}) -[:HAS_CONDITION]-> (:ModelInstance:ConditionModel {variation_code:"Q.I.a.1", condition_number:"1"}) (:ModelInstance:VariationWithConditions {variation_code:"Q.I.a.1"}) -[:HAS_CONDITION]-> (:ModelInstance:ConditionModel {variation_code:"Q.I.a.1", condition_number:"2"}) (:ModelInstance:VariationWithConditions {variation_code:"Q.I.a.1"}) -[:HAS_CONDITION]-> (:ModelInstance:ConditionModel {variation_code:"Q.I.a.1", condition_number:"3"}) ``` --- ### Pattern 4: Dual Pattern (Entity + Instance) A procedure type that is both a named entity and a model instance: ```python class ProcedureRecord(ExtractionModel): """A regulatory procedure with its type classification.""" procedure_name: str = Field( ..., description="Name of the procedure as it appears in the document.", ) procedure_type: str = Field( ..., description="Procedure type code: IA, IAIN, IB, II, A, or BA.", json_schema_extra={ "entity_label": "ProcedureType", "instance_relationships": [ { "target_model": "ProcedureTypeModel", "join_via": {"procedure_type": "procedure_type"}, "rel_type": "HAS_PROCEDURE_TYPE", } ], }, ) ``` **Graph:** ``` Level 1: (:ModelInstance:ProcedureRecord)-[:REFERENCES]->(:LabeledEntity:ProcedureType {value:"IA"}) Level 3: (:ModelInstance:ProcedureRecord)-[:HAS_PROCEDURE_TYPE]->(:ModelInstance:ProcedureTypeModel {procedure_type:"IA"}) ``` --- ### Pattern 5: Multi-Hop Chain (A → B → C) Build multi-hop graph chains by combining `field_relationships` and `instance_relationships` across models: ```python # Step 1: field_relationships creates entity-to-entity edges class SubstanceRoute(ExtractionModel): """A substance and its route of administration.""" substance_name: str = Field( ..., description="Official name of the substance.", json_schema_extra={ "entity_label": "Substance", "field_relationships": [ {"to_field": "route", "rel_type": "ADMINISTERED_VIA"}, ], }, ) route: str = Field( ..., description="Route of administration (e.g. 'oral', 'intravenous', 'topical').", json_schema_extra={"entity_label": "Route"}, ) # Step 2: instance_relationships links SubstanceRoute to a DosageModel class DosageModel(ExtractionModel): """A dosage specification for a substance.""" substance_name: str = Field( ..., description="Substance this dosage applies to.", json_schema_extra={"entity_label": "Substance", "instance_key": True}, ) dosage: str = Field( ..., description="Dosage instruction (e.g. '500 mg twice daily').", ) # Step 3: SubstanceRoute links to DosageModel via instance_relationships class SubstanceRoute(ExtractionModel): """A substance and its route of administration.""" substance_name: str = Field( ..., description="Official name of the substance.", json_schema_extra={ "entity_label": "Substance", "field_relationships": [ {"to_field": "route", "rel_type": "ADMINISTERED_VIA"}, ], "instance_relationships": [ { "target_model": "DosageModel", "join_via": {"substance_name": "substance_name"}, "rel_type": "HAS_DOSAGE", } ], }, ) route: str = Field( ..., description="Route of administration (e.g. 'oral', 'intravenous', 'topical').", json_schema_extra={"entity_label": "Route"}, ) ``` **Graph:** ``` Entity level (field_relationships): (:LabeledEntity:Substance {value:"Metformin"}) -[:ADMINISTERED_VIA]-> (:LabeledEntity:Route {value:"oral"}) Instance level (instance_relationships): (:ModelInstance:SubstanceRoute)-[:HAS_DOSAGE]->(:ModelInstance:DosageModel {substance_name:"Metformin"}) Multi-hop query: (:LabeledEntity:Substance)-[:ADMINISTERED_VIA]->(:LabeledEntity:Route) (:ModelInstance:SubstanceRoute)-[:HAS_DOSAGE]->(:ModelInstance:DosageModel) ``` --- ## Anti-Patterns | Anti-Pattern | Why It Fails | Fix | | :--- | :--- | :--- | | `field_relationships` pointing to a field without `entity_label` | The target `:LabeledEntity` node does not exist; Neo4j write is silently skipped | Add `entity_label` to the target field | | Missing `instance_key: True` on target model key fields | Shell nodes created by `instance_relationships` never merge with real nodes when the target is extracted | Mark ALL key fields on the target model with `instance_key: True` | | `join_via` field name mismatch | Zero target nodes created — the field name in `join_via` must exactly match the Python field name | Double-check that `join_via` keys use the exact Python field names | | `entity_label` on free-text narrative fields | Creates meaningless `:LabeledEntity` singletons; degrades cross-document deduplication quality | Only add `entity_label` to short, stable, identifier-like values | | `target_model` using wrong class name | Shell nodes created with wrong `model_class`; real nodes never merge | Use the exact PascalCase class name as a string | | `field_relationships` on a field with `None` value | Relationship silently skipped — both source and target must be non-null | Make the field required or handle the `None` case in the description | | `instance_relationships` on a scalar field without including it in `join_via` | The engine requires the annotated field itself to be in `join_via` as the fan-out key | Always include the annotated field name in `join_via` | --- ## Verification ### Verify Entity Relationships in Neo4j ```cypher -- List all field_relationships (LabeledEntity → LabeledEntity) MATCH (a:LabeledEntity)-[r]->(b:LabeledEntity) RETURN labels(a) AS source, type(r) AS relationship, labels(b) AS target, count(r) AS count ORDER BY count DESC; ``` ### Verify Instance Relationships in Neo4j ```cypher -- List all instance_relationships (ModelInstance → ModelInstance) MATCH (a:ModelInstance)-[r]->(b:ModelInstance) RETURN a.model_class AS source_model, type(r) AS relationship, b.model_class AS target_model, count(r) AS count ORDER BY count DESC; ``` ### Verify a Specific Model's Relationships ```cypher -- All outgoing relationships from VariationModel instances MATCH (a:ModelInstance)-[r]->(b) WHERE a.model_class = 'VariationModel' RETURN a.model_class AS source, a.variation_code AS code, type(r) AS relationship, labels(b) AS target_labels, b.model_class AS target_model LIMIT 20; ``` ### Verify Shell Nodes (Unresolved Targets) ```cypher -- Find shell nodes that have not been merged with real data MATCH (m:ModelInstance) WHERE m.description IS NULL AND m.model_class IS NOT NULL RETURN m.model_class AS model, count(m) AS shell_count; ``` ### Visualize a Relationship Chain ```cypher -- Visualize the full chain from entity to instance to instance MATCH path = (e:LabeledEntity)-[*1..3]->(m:ModelInstance) WHERE e.label = 'VariationCode' RETURN path LIMIT 20; ``` --- ## See Also - **[Custom Models](custom-models.md)** — Defining Pydantic extraction schemas for domain-specific entities. - **[Neo4j Graph Storage](neo4j-graph.md)** — Understanding the overall graph model and node types. - **[Running the Pipeline](running-pipeline.md)** — Orchestrating the full ingestion pipeline. - **[Architecture](../architecture.md)** — Detailed pipeline stages and data flow. --- ## File: user-guides/model-patterns.md # Advanced Model Design Patterns Production-grade extraction models in `scinr.newton` use a set of interconnected design patterns to produce clean, deduplicated knowledge graphs. This guide covers each pattern in detail, when to use it, and how the patterns compose together. --- ## Table of Contents 1. [Introduction](#1-introduction) 2. [The Dual-Field Pattern](#2-the-dual-field-pattern) 3. [Entity Labels (`entity_label`)](#3-entity-labels-entity_label) 4. [Instance Keys (`instance_key`)](#4-instance-keys-instance_key) 5. [Domain Validators](#5-domain-validators) 6. [OCR Fix Validators](#6-ocr-fix-validators) 7. [The Normalization Model Mechanism](#7-the-normalization-model-mechanism) 8. [The Implicit Fallback (Footgun)](#8-the-implicit-fallback-footgun) 9. [Instance Relationships](#9-instance-relationships) 10. [Field Relationships](#10-field-relationships) 11. [Pattern Summary Table](#11-pattern-summary-table) 12. [Complete Example](#12-complete-example) --- ## 1. Introduction Advanced patterns for robust extraction models that produce clean, deduplicated knowledge graphs. A well-designed extraction model does more than define fields and descriptions. It declares *how* those fields interact with the pipeline: which values should be globally deduplicated, which fields form a unique identity, which raw fields feed a normalization step, and which values need domain-specific cleaning before they reach Neo4j. These patterns are expressed entirely through Pydantic `Field()` annotations and `json_schema_extra` metadata. The pipeline reads this metadata at runtime to wire up entity labeling, instance deduplication, normalization batching, and graph relationships — all without any changes to pipeline code. **The patterns covered in this guide:** | Pattern | Declared Via | Pipeline Effect | |---|---|---| | Dual-Field | Two fields (raw + nested) | Raw preserves source; nested enables structured matching | | `entity_label` | `json_schema_extra` | Creates `:LabeledEntity` nodes for cross-document deduplication | | `instance_key` | `json_schema_extra` | Deterministic UID for `:ModelInstance` deduplication | | `normalization_model` | `json_schema_extra` | Triggers `NormalizationEngine` in tabular pipeline | | `normalization_source_fields` | `json_schema_extra` | Declares which raw fields feed the normalization | | `instance_relationships` | `json_schema_extra` | Cross-instance graph relationships (Level 3) | | `field_relationships` | `json_schema_extra` | Cross-entity graph relationships (Level 2) | | Domain validators | `@field_validator` | Pre-validation normalization and OCR correction | --- ## 2. The Dual-Field Pattern The core pattern: a **raw free-text field** paired with a **normalized nested model**. ```python from pydantic import Field from scinr.newton.models.base import ExtractionModel class NormalizedAddress(ExtractionModel): """Structured, normalized postal address.""" street: str | None = Field( default=None, description="Street address line, without city or postal code.", ) city: str | None = Field( default=None, description="City name.", ) postal_code: str | None = Field( default=None, description="Postal or ZIP code.", ) country_code: str | None = Field( default=None, description="ISO 3166-1 alpha-2 country code (e.g. 'US', 'DE', 'JP').", json_schema_extra={"entity_label": "Country"}, ) class ContactRecord(ExtractionModel): """A single contact record from a regulatory document.""" # ─── Tier 1: Free-text raw field (high tolerance) ─── raw_address: str = Field( ..., description=( "Free-text address exactly as it appears in the source document. " "Preserve original formatting, line breaks, and abbreviations." ), ) # ─── Tier 2: Normalized nested model (structured) ─── normalized_address: NormalizedAddress | None = Field( default=None, description=( "Structured address derived from raw_address: street, city, " "postal code, and country code parsed into separate fields." ), json_schema_extra={ "normalization_model": True, "normalization_source_fields": ["raw_address"], }, ) ``` ### Why dual-field? | Aspect | Raw Field | Normalized Field | |---|---|---| | **Purpose** | Preserves original text verbatim | Enables structured querying and matching | | **LLM tolerance** | High — free-text, no structure required | Lower — must conform to nested schema | | **Neo4j representation** | Scalar property on parent node | Separate `:ModelInstance` child node | | **Cross-document matching** | None (unstructured) | Via `entity_label` fields within | The raw field captures the source text with maximum fidelity. The normalized field breaks that text into queryable, comparable components. Together they give you both provenance and utility. ### How the pipeline uses it - **`normalization_model: True`** — marks the field for the `NormalizationEngine` in the tabular pipeline. The engine scans for this flag and schedules the field for LLM-driven normalization. - **`normalization_source_fields`** — declares which raw fields on the *parent* model feed the normalization. The engine collects values from these fields, builds a dedup hash, and batches them for a single LLM call. ### When mandatory vs. optional | Pipeline | Requirement | Reason | |---|---|---| | **Tabular** (CSV/XLSX/XLS) | **Mandatory** | The `NormalizationEngine` hook requires `normalization_model: True` and `normalization_source_fields` to know which raw columns to use as input. Without them, the normalization step is a no-op. | | **Unstructured** (PDF/DOCX) | **Optional** | The LLM fills the nested field directly from the field description in a single extraction call. No separate normalization step is needed. | | **Both pipelines** | **Recommended** | Keeps the model consistent across pipeline types. A model that works in both pipelines should always declare both keys. | --- ## 3. Entity Labels (`entity_label`) Entity labels create `:LabeledEntity` nodes in Neo4j that are globally deduplicated by `(label, normalized_value)`. This is the primary mechanism for cross-document entity matching. ### Good usage ```python # ✅ GOOD — stable, identifier-like value substance_name: str | None = Field( default=None, description="INN name of the active substance (e.g. 'Metformin').", json_schema_extra={"entity_label": "ActiveSubstance"}, ) # ✅ GOOD — code or standard identifier country_code: str | None = Field( default=None, description="ISO 3166-1 alpha-2 country code (e.g. 'US', 'DE', 'JP').", json_schema_extra={"entity_label": "Country"}, ) # ✅ GOOD — procedure type with stable set of values procedure_type: str | None = Field( default=None, description="Procedure type code: IA, IB, II, IAIN, A, or BA.", json_schema_extra={"entity_label": "ProcedureType"}, ) ``` ### Bad usage ```python # ❌ BAD — free-text narrative (every value is unique, no dedup benefit) full_description: str | None = Field( default=None, description="Detailed paragraph describing the manufacturing process.", json_schema_extra={"entity_label": "ProcessDescription"}, # WRONG ) # ❌ BAD — measurement in context (dedup is meaningless) temperature: float | None = Field( default=None, description="Processing temperature in degrees Celsius.", json_schema_extra={"entity_label": "Temperature"}, # WRONG ) # ❌ BAD — boolean (only two values, no dedup value) is_active: bool | None = Field( default=None, description="Whether the product is currently on the market.", json_schema_extra={"entity_label": "IsActive"}, # WRONG ) ``` ### Rules | Rule | Details | |---|---| | **Use for** | Codes, names, identifiers, stable categorical values | | **Don't use for** | Long descriptions, narratives, free-text paragraphs | | **Don't use for** | Measurements in context, timestamps, booleans | | **Label naming** | Use `CamelCase` (e.g. `ActiveSubstance`, `Country`, `ProcedureType`) | | **Neo4j effect** | Creates `:LabeledEntity {label, value, normalized_value}` nodes merged by `(label, normalized_value)` | ### How deduplication works When the graph mapper encounters a field with `entity_label`, it: 1. Normalizes the value: lowercase, strips accents, collapses whitespace 2. Computes a deterministic UID from the label and normalized value 3. MERGEs a `:LabeledEntity` node with that UID This means the same substance name extracted from ten different documents resolves to a single `:LabeledEntity` node. The `REFERENCES` relationship from each extraction points to that shared node. --- ## 4. Instance Keys (`instance_key`) Instance keys define a composite unique identifier for a `:ModelInstance` node. When a nested model has one or more fields marked with `instance_key: True`, the graph mapper computes a deterministic UID and uses MERGE instead of CREATE, enabling global deduplication of model instances. ### Basic usage ```python class Fee(ExtractionModel): """A single fee entry from a fee schedule table.""" country_code: str = Field( ..., description="ISO 3166-1 alpha-2 country code.", json_schema_extra={"entity_label": "Country", "instance_key": True}, ) procedure_type: str = Field( ..., description="Procedure type code: IA, IB, II, IAIN, A, or BA.", json_schema_extra={"entity_label": "ProcedureType", "instance_key": True}, ) role: str = Field( ..., description="Fee role: applicant, holder, or third party.", json_schema_extra={"entity_label": "FeeRole", "instance_key": True}, ) rate: str = Field( ..., description="Fee amount without currency symbol (e.g. '1234.56').", ) ``` Here, `(country_code, procedure_type, role)` forms the composite key. A `Fee` for `("DE", "IA", "applicant")` extracted from any document will always resolve to the same `:ModelInstance` node. ### Rules | Rule | Details | |---|---| | **Mark ALL key fields** | Every field that forms the unique identity must have `instance_key: True` | | **Composite keys** | Mark ALL constituent fields; the UID is computed from all of them | | **Required for `instance_relationships`** | A model referenced in another model's `instance_relationships` must declare `instance_key` fields | | **UID stability** | The UID is `make_instance_uid(model_class, sorted_key_fields)` — field order does not matter | ### Without instance key A nested model without instance keys gets a random UUID (`uuid.uuid4().hex[:16]`) and is always created as a new node. This is fine for truly unique entities (e.g., a specific manufacturing batch) but prevents deduplication across extractions. ### With instance key A nested model with instance keys gets a deterministic UID and is MERGE'd. If the same instance is extracted from multiple documents, the MERGE ensures a single node with accumulated properties. --- ## 5. Domain Validators Domain validators apply business-specific normalization to field values *before* Pydantic validation. They live in a shared base class and use `check_fields=False` so they are optional per subclass. ### Pattern ```python import re from pydantic import BaseModel, field_validator, Field def normalize_code(v: str) -> str: """Apply domain-specific normalization to a code string.""" v = v.strip().upper() v = re.sub(r"[\s_/-]", "", v) return v class NormalizedBaseModel(BaseModel): """Base class applying field normalization before Pydantic validation.""" @field_validator( "procedure_type", "procedure_types_referenced", mode="before", check_fields=False ) @classmethod def normalize_procedure_types(cls, v): if isinstance(v, str): return normalize_code(v) if isinstance(v, list): return [ normalize_code(item) if isinstance(item, str) else item for item in v ] return v class VariationModel(NormalizedBaseModel): """A variation entry with auto-normalized procedure type.""" procedure_type: str = Field( ..., description="Procedure type code: IA, IB, II, IAIN, A, or BA.", json_schema_extra={"entity_label": "ProcedureType"}, ) procedure_types_referenced: list[str] | None = Field( default=None, description="Other procedure type codes referenced in this variation.", json_schema_extra={"entity_label": "ProcedureType"}, ) ``` ### Key details | Detail | Explanation | |---|---| | **`check_fields=False`** | The validator is registered on the base class but only fires for subclasses that actually declare the named fields. This makes the validator optional per subclass. | | **`mode="before"`** | Applies *before* Pydantic type validation, so normalization happens on the raw LLM output before any type coercion. | | **List handling** | Always check `isinstance(v, list)` and map over items. The LLM may return a single string or a list depending on the field type. | | **Inheritance** | Subclasses of `NormalizedBaseModel` automatically get the validators for any fields they declare that match the validator's field names. | ### Common normalization functions ```python def normalize_country_code(v: str) -> str: """Normalize ISO 3166-1 alpha-2 country code.""" return v.strip().upper()[:2] def normalize_substance_name(v: str) -> str: """Normalize substance name to INN-style format.""" v = v.strip().lower() # Remove common suffixes for suffix in (" hydrochloride", " sulfate", " phosphate", " tablet", " capsule"): v = v.replace(suffix, "") return v.strip() def normalize_date(v: str) -> str: """Normalize date string to YYYY-MM-DD if possible.""" v = v.strip() # Attempt common formats for fmt in ("%d/%m/%Y", "%d-%m-%Y", "%d.%m.%Y", "%Y-%m-%d"): try: from datetime import datetime return datetime.strptime(v, fmt).strftime("%Y-%m-%d") except ValueError: continue return v # Return as-is if no format matched ``` --- ## 6. OCR Fix Validators OCR errors in scanned PDFs produce systematic misrecognitions. OCR fix validators correct these at the validation layer so the rest of the pipeline always sees clean data. ### Pattern ```python import re from pydantic import BaseModel, field_validator, Field def normalize_variation_code(v: str) -> str: """Fix common OCR mis-recognition in variation codes.""" # OCR fix: Q.1.a.1 → Q.I.a.1 (digit 1 → roman numeral I) v = re.sub(r"(?<=[A-Za-z])\.(?:1|l)\.", ".I.", v) # OCR fix: Q.I.a.1.a → Q.I.a.1(a) (trailing letter in parentheses) v = re.sub(r"(?<=\d)\.([a-zA-Z])$", r"(\1)", v) return v class VariationCodeModel(BaseModel): """A variation code entry with OCR-corrected code.""" variation_code: str = Field( ..., description="Variation code (e.g. 'Q.I.a.1', 'II.A.1(a)').", json_schema_extra={"entity_label": "VariationCode"}, ) @field_validator("variation_code", mode="before") @classmethod def fix_ocr_errors(cls, v): if isinstance(v, str): return normalize_variation_code(v) return v ``` ### Common OCR patterns to fix | Pattern | Fix | Regex | |---|---|---| | `O` → `0` (letter O to zero) | Context-dependent | N/A (requires domain logic) | | `l` → `1` (lowercase L to one) | In code positions | `r"(?<=[A-Za-z])\.l\."` → `.1.` | | `1` → `I` (one to roman I) | In variation codes | `r"(?<=[A-Za-z])\.1\."` → `.I.` | | Missing parentheses | `Q.I.a.1.a` → `Q.I.a.1(a)` | `r"(?<=\d)\.([a-zA-Z])$"` → `r"(\1)"` | | Hyphen vs. en-dash | `–` → `-` | `re.sub(r"[\u2013\u2014]", "-", v)` | | Extra spaces | Multiple spaces → single | `re.sub(r"\s+", " ", v)` | --- ## 7. The Normalization Model Mechanism Deep dive into how `normalization_model` and `normalization_source_fields` work together in the tabular pipeline. ### The mechanical flow ``` Tabular file (CSV/XLSX) │ ▼ ┌─────────────┐ │ Column Map │ LLM maps columns → model fields └──────┬──────┘ │ ▼ ┌─────────────┐ │ Instantiate │ Pydantic model created from mapped columns │ Model │ Raw fields populated, nested fields = None └──────┬──────┘ │ ▼ ┌─────────────────────────────────────────────────────────┐ │ NormalizationEngine scans for normalization_model: True │ └──────┬──────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────┐ │ Collects source field values from source_fields list │ └──────┬──────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────┐ │ Dedup hash: MD5 of sorted source values │ │ → Identical source values share one LLM call │ └──────┬──────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────┐ │ Batch entries by target_type (max batch_size per call) │ └──────┬──────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────┐ │ LLM call with structured output → normalized objects │ └──────┬──────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────┐ │ Results written back via setattr (or object.__setattr__) │ │ to the original model instances │ └──────┬──────────────────────────────────────────────────┘ │ ▼ ┌─────────────┐ │ Write to │ Neo4j with populated nested fields │ Neo4j │ └─────────────┘ ``` ### When mandatory **Tabular pipeline (CSV/XLSX/XLS)** — the `NormalizationEngine` hook requires these keys. Without `normalization_model: True`, the engine skips the field entirely. Without `normalization_source_fields`, the engine falls back to *all* scalar fields (the implicit fallback — see Section 8). ### When optional **Unstructured pipeline (PDF/DOCX)** — the LLM fills the nested field directly from the field description during entity extraction. No separate normalization step is needed. The `normalization_model` flag is ignored in this pipeline. ### When recommended **Models used with both pipelines** — always declare both `normalization_model: True` and `normalization_source_fields`. This keeps the model consistent and prevents the implicit fallback in the tabular pipeline. ### Configuration | Parameter | Env Var | Default | Description | |---|---|---|---| | `normalization_enabled` | `NORMALIZATION_ENABLED` | `False` | Enable/disable the normalization engine | | `normalization_batch_size` | `NORMALIZATION_BATCH_SIZE` | `5` | Max entries per LLM batch call | | `normalization_llm` | — | Falls back to main `llm` | Dedicated LLM for normalization calls | --- ## 8. The Implicit Fallback (Footgun) When `normalization_source_fields` is omitted or empty, the `NormalizationEngine` falls back to using **ALL scalar fields** on the model as source data. This is almost never what you want. ### The problem ```python # ❌ BAD — no normalization_source_fields declared class BadContact(ExtractionModel): """Contact record with implicit source fallback.""" raw_address: str = Field(...) phone: str = Field(...) email: str = Field(...) normalized_address: NormalizedAddress | None = Field( default=None, description="Structured address.", json_schema_extra={ "normalization_model": True, # Missing! Falls back to ALL scalar fields: # raw_address, phone, email — all sent to the LLM }, ) ``` In this case, the normalization LLM receives `raw_address`, `phone`, AND `email` as source data for address normalization. The phone and email fields are noise — they dilute the prompt and may confuse the LLM. ### The fix ```python # ✅ GOOD — explicit source fields class GoodContact(ExtractionModel): """Contact record with explicit source fields.""" raw_address: str = Field(...) phone: str = Field(...) email: str = Field(...) normalized_address: NormalizedAddress | None = Field( default=None, description="Structured address.", json_schema_extra={ "normalization_model": True, "normalization_source_fields": ["raw_address"], # exact fields }, ) ``` ### Why it matters | Aspect | Implicit (bad) | Explicit (good) | |---|---|---| | **Source data** | All scalar fields | Only declared fields | | **LLM prompt size** | Larger, noisy | Minimal, focused | | **LLM accuracy** | Lower (distracted) | Higher (focused) | | **Dedup hash** | Includes irrelevant fields | Based on relevant data only | | **Cost** | Higher (more tokens) | Lower | **Always declare `normalization_source_fields` explicitly.** The implicit fallback exists only for backward compatibility and should be treated as a bug if encountered in new code. --- ## 9. Instance Relationships Instance relationships (Level 3 in the graph mapper) create typed relationships between `:ModelInstance` nodes. They enable forward references across `StructureNode` boundaries: a model can reference another model instance that has not yet been extracted. ### Pattern ```python class VariationModel(ExtractionModel): """A variation entry that references conditions.""" variation_code: str = Field( ..., description="Variation code (e.g. 'Q.I.a.1').", json_schema_extra={ "entity_label": "VariationCode", "instance_key": True, }, ) procedure_type: str = Field( ..., description="Procedure type code.", json_schema_extra={ "entity_label": "ProcedureType", "instance_key": True, }, ) condition_ids: list[str] | None = Field( default=None, description="IDs of applicable conditions (e.g. ['1', '2', '3']).", json_schema_extra={ "instance_relationships": [ { "target_model": "ConditionModel", "rel_type": "APPLIES_TO", "join_via": { "condition_ids": "condition_id", "variation_code": "variation_code", }, } ], }, ) ``` ### How it works 1. The `condition_ids` field is a `list[str]` — each item triggers a relationship 2. `join_via` maps local fields to remote fields on the target model 3. The local field itself (`condition_ids`) is the **fan-out** field — it provides the list of values 4. Other fields in `join_via` (`variation_code`) are **fixed** fields — they come from the same instance 5. For each item in `condition_ids`, a `:ModelInstance` shell for `ConditionModel` is MERGE'd with the composite key `(condition_id, variation_code)` 6. A typed relationship `[:APPLIES_TO]` is created from the source to the target ### Rules | Rule | Details | |---|---| | **Target model must have `instance_key`** | The target model class must declare `instance_key: True` on all fields referenced in `join_via` | | **Fan-out field** | The annotated field itself (the one with `instance_relationships`) must be in `join_via` as the fan-out key | | **Fixed fields** | Other fields in `join_via` are read from the same instance and must be non-empty | | **Empty fixed fields** | If a fixed field is `None` or empty string, no relationships are created for this instance (logged as warning) | ### Graph result ``` (:ModelInstance {model_class: "VariationModel"}) -[:APPLIES_TO]-> (:ModelInstance {model_class: "ConditionModel", condition_id: "1", variation_code: "Q.I.a.1"}) -[:APPLIES_TO]-> (:ModelInstance {model_class: "ConditionModel", condition_id: "2", variation_code: "Q.I.a.1"}) ``` The target `ConditionModel` nodes are "shells" — they exist with only the key fields populated. When `ConditionModel` is later extracted in a child section, the MERGE on the same UID populates the remaining fields (`description`, etc.). --- ## 10. Field Relationships Field relationships (Level 2 in the graph mapper) create typed relationships between `:LabeledEntity` nodes. They express domain relationships between entities within the same extraction. ### Pattern ```python class IngredientRelationship(ExtractionModel): """Relationship between an active substance and its excipient.""" active_substance: str = Field( ..., description="INN name of the active substance.", json_schema_extra={ "entity_label": "ActiveSubstance", "field_relationships": [ { "to_field": "excipient", "rel_type": "CONTAINS_EXCIPIENT", } ], }, ) excipient: str = Field( ..., description="Name of the excipient.", json_schema_extra={"entity_label": "Excipient"}, ) ``` ### How it works 1. Both `active_substance` and `excipient` have `entity_label` — they become `:LabeledEntity` nodes 2. `field_relationships` on `active_substance` declares a relationship to the `excipient` field 3. The graph mapper MERGEs a `[:CONTAINS_EXCIPIENT]` relationship between the two entity nodes ### Rules | Rule | Details | |---|---| | **Both fields need `entity_label`** | Source and target fields must both have `entity_label` to create entity nodes | | **Sibling fields** | `to_field` references a field at the same nesting level (or within the same model instance) | | **Target must exist** | If the target field is `None`, the relationship is skipped (logged as debug) | ### Graph result ``` (:LabeledEntity {label: "ActiveSubstance", value: "Metformin"}) -[:CONTAINS_EXCIPIENT]-> (:LabeledEntity {label: "Excipient", value: "Microcrystalline Cellulose"}) ``` --- ## 11. Pattern Summary Table | Pattern | Declared Via | Purpose | When to Use | |---|---|---|---| | **Dual-Field** | Two fields (raw + nested) | Raw preserves source; nested enables structured querying | When you need both original text and structured data | | **`entity_label`** | `json_schema_extra` | Creates `:LabeledEntity` nodes for cross-document deduplication | Stable, identifier-like fields (codes, names, IDs) | | **`instance_key`** | `json_schema_extra` | Deterministic UID for `:ModelInstance` deduplication | Fields forming the unique identity of a model instance | | **`normalization_model`** | `json_schema_extra` | Triggers `NormalizationEngine` in tabular pipeline | CSV/XLSX pipeline with nested models needing LLM normalization | | **`normalization_source_fields`** | `json_schema_extra` | Declares which raw fields feed normalization | Always with `normalization_model: True` — never omit | | **`instance_relationships`** | `json_schema_extra` | Cross-instance graph relationships (Level 3) | When one model references instances of another model | | **`field_relationships`** | `json_schema_extra` | Cross-entity graph relationships (Level 2) | When two entity-labeled fields have a domain relationship | | **Domain validators** | `@field_validator` | Pre-validation normalization and cleaning | Shared normalization across models via base class | | **OCR fix validators** | `@field_validator` | Corrects systematic OCR misrecognitions | Models processing data from scanned PDFs | --- ## 12. Complete Example A complete model file using all patterns together. This example models pharmaceutical variation data — a domain with codes, nested structures, cross-references, and OCR-prone scanned source documents. ```python """ models/variations.py — Pharmaceutical variation extraction models. Demonstrates all advanced model design patterns: - Dual-field pattern (raw + normalized) - Entity labels for cross-document deduplication - Instance keys for model instance deduplication - Domain validators for code normalization - OCR fix validators for scanned document correction - Normalization model hooks for tabular pipeline - Instance relationships for cross-model references - Field relationships for entity-to-entity links """ from __future__ import annotations import re from typing import Any from pydantic import BaseModel, Field, field_validator from scinr.newton.models.base import ExtractionModel # ───────────────────────────────────────────────────────────────────────────── # Normalization helpers # ───────────────────────────────────────────────────────────────────────────── def normalize_code(v: str) -> str: """Apply domain-specific normalization to a code string.""" v = v.strip().upper() v = re.sub(r"[\s_/-]", "", v) return v def normalize_variation_code(v: str) -> str: """Fix common OCR mis-recognition in variation codes.""" # OCR fix: Q.1.a.1 → Q.I.a.1 (digit 1 → roman numeral I in code position) v = re.sub(r"(?<=[A-Za-z])\.(?:1|l)\.", ".I.", v) # OCR fix: Q.I.a.1.a → Q.I.a.1(a) (trailing letter in parentheses) v = re.sub(r"(?<=\d)\.([a-zA-Z])$", r"(\1)", v) return v def normalize_country_code(v: str) -> str: """Normalize ISO 3166-1 alpha-2 country code.""" return v.strip().upper()[:2] # ───────────────────────────────────────────────────────────────────────────── # Base class with shared validators # ───────────────────────────────────────────────────────────────────────────── class VariationBaseModel(BaseModel): """Base class applying field normalization before Pydantic validation.""" @field_validator( "procedure_type", "procedure_types_referenced", mode="before", check_fields=False, ) @classmethod def normalize_procedure_types(cls, v: Any) -> Any: """Normalize procedure type codes: strip whitespace, uppercase, remove separators.""" if isinstance(v, str): return normalize_code(v) if isinstance(v, list): return [ normalize_code(item) if isinstance(item, str) else item for item in v ] return v @field_validator( "variation_code", mode="before", check_fields=False, ) @classmethod def fix_variation_code_ocr(cls, v: Any) -> Any: """Fix OCR errors in variation codes.""" if isinstance(v, str): return normalize_variation_code(v) return v @field_validator( "country_code", mode="before", check_fields=False, ) @classmethod def fix_country_code(cls, v: Any) -> Any: """Normalize country codes to ISO 3166-1 alpha-2.""" if isinstance(v, str): return normalize_country_code(v) return v # ───────────────────────────────────────────────────────────────────────────── # Normalized nested models (Tier 2) # ───────────────────────────────────────────────────────────────────────────── class NormalizedDate(ExtractionModel): """Structured date normalized to ISO 8601.""" year: int | None = Field(default=None, description="Year (e.g. 2024).") month: int | None = Field(default=None, description="Month (1-12).") day: int | None = Field(default=None, description="Day (1-31).") iso_string: str | None = Field( default=None, description="Full ISO 8601 date string (e.g. '2024-03-15').", ) class NormalizedAddress(ExtractionModel): """Structured, normalized postal address.""" street: str | None = Field( default=None, description="Street address line, without city or postal code.", ) city: str | None = Field( default=None, description="City name.", ) postal_code: str | None = Field( default=None, description="Postal or ZIP code.", ) country_code: str | None = Field( default=None, description="ISO 3166-1 alpha-2 country code (e.g. 'US', 'DE', 'JP').", json_schema_extra={"entity_label": "Country"}, ) # ───────────────────────────────────────────────────────────────────────────── # Condition model (referenced by VariationModel via instance_relationships) # ───────────────────────────────────────────────────────────────────────────── class ConditionModel(VariationBaseModel, ExtractionModel): """A regulatory condition applicable to a variation. Has instance_key so that VariationModel can reference this model via instance_relationships and the graph mapper can MERGE the same node across multiple extractions. """ condition_id: str = Field( ..., description="Numeric condition identifier (e.g. '1', '2', '3').", json_schema_extra={ "entity_label": "ConditionId", "instance_key": True, }, ) variation_code: str = Field( ..., description="Parent variation code (e.g. 'Q.I.a.1').", json_schema_extra={ "entity_label": "VariationCode", "instance_key": True, }, ) description: str | None = Field( default=None, description="Free-text description of the condition requirement.", ) is_mandatory: bool | None = Field( default=None, description="Whether this condition is mandatory (True) or optional (False).", ) # ───────────────────────────────────────────────────────────────────────────── # Substance model (dual-field pattern) # ───────────────────────────────────────────────────────────────────────────── class NormalizedSubstance(ExtractionModel): """Structured, normalized substance information.""" inn_name: str | None = Field( default=None, description="International Nonproprietary Name (INN).", json_schema_extra={"entity_label": "ActiveSubstance"}, ) cas_number: str | None = Field( default=None, description="CAS Registry Number (e.g. '1105-50-9').", json_schema_extra={"entity_label": "CasNumber"}, ) strength: str | None = Field( default=None, description="Strength with units (e.g. '500 mg', '10 mg/mL').", ) pharmaceutical_form: str | None = Field( default=None, description="Pharmaceutical form (e.g. 'tablet', 'solution', 'powder').", json_schema_extra={"entity_label": "PharmaceuticalForm"}, ) class SubstanceEntry(VariationBaseModel, ExtractionModel): """A substance entry with dual-field pattern: raw text + normalized structure. The raw_substance field captures the original text with maximum fidelity. The normalized_substance field breaks it into structured, queryable components. In the tabular pipeline, the NormalizationEngine uses raw_substance as input to populate normalized_substance via LLM. In the unstructured pipeline, the LLM fills both fields directly. """ raw_substance: str = Field( ..., description=( "Free-text substance description exactly as it appears in the source. " "Preserve original formatting, trade names, and abbreviations." ), ) normalized_substance: NormalizedSubstance | None = Field( default=None, description=( "Structured substance data derived from raw_substance: INN name, " "CAS number, strength, and pharmaceutical form parsed into separate fields." ), json_schema_extra={ "normalization_model": True, "normalization_source_fields": ["raw_substance"], }, ) # ───────────────────────────────────────────────────────────────────────────── # Fee model (instance_key for deduplication) # ───────────────────────────────────────────────────────────────────────────── class Fee(VariationBaseModel, ExtractionModel): """A fee entry from a fee schedule. Uses a composite instance_key (country_code, procedure_type, role) so that the same fee from different documents resolves to the same ModelInstance node. """ country_code: str = Field( ..., description="ISO 3166-1 alpha-2 country code.", json_schema_extra={ "entity_label": "Country", "instance_key": True, }, ) procedure_type: str = Field( ..., description="Procedure type code: IA, IB, II, IAIN, A, or BA.", json_schema_extra={ "entity_label": "ProcedureType", "instance_key": True, }, ) role: str = Field( ..., description="Fee role: applicant, holder, or third party.", json_schema_extra={ "entity_label": "FeeRole", "instance_key": True, }, ) rate: str = Field( ..., description="Fee amount without currency symbol (e.g. '1234.56').", ) currency: str = Field( ..., description="ISO 4217 currency code (e.g. 'EUR', 'USD').", json_schema_extra={"entity_label": "Currency"}, ) # ───────────────────────────────────────────────────────────────────────────── # Ingredient relationship model (field_relationships) # ───────────────────────────────────────────────────────────────────────────── class IngredientPair(VariationBaseModel, ExtractionModel): """Relationship between an active substance and its excipient. Uses field_relationships to create a typed relationship between the ActiveSubstance and Excipient LabeledEntity nodes. """ active_substance: str = Field( ..., description="INN name of the active substance.", json_schema_extra={ "entity_label": "ActiveSubstance", "field_relationships": [ { "to_field": "excipient", "rel_type": "CONTAINS_EXCIPIENT", } ], }, ) excipient: str = Field( ..., description="Name of the excipient.", json_schema_extra={"entity_label": "Excipient"}, ) ratio: str | None = Field( default=None, description="Mixing ratio or percentage (e.g. '95:5', '10%').", ) # ───────────────────────────────────────────────────────────────────────────── # Main variation model (all patterns combined) # ───────────────────────────────────────────────────────────────────────────── class VariationModel(VariationBaseModel, ExtractionModel): """A pharmaceutical variation entry. Combines all advanced model design patterns: - Dual-field: raw_date + normalized_date - Entity labels: variation_code, procedure_type, country_code - Instance keys: variation_code, procedure_type - Domain validators: procedure_type normalization, variation_code OCR fix - Instance relationships: condition_ids → ConditionModel - Normalization model: normalized_date from raw_date """ # ── Entity labels + instance keys ──────────────────────────────────── variation_code: str = Field( ..., description="Variation code (e.g. 'Q.I.a.1', 'II.A.1(a)').", json_schema_extra={ "entity_label": "VariationCode", "instance_key": True, }, ) procedure_type: str = Field( ..., description="Procedure type code: IA, IB, II, IAIN, A, or BA.", json_schema_extra={ "entity_label": "ProcedureType", "instance_key": True, }, ) country_code: str = Field( ..., description="ISO 3166-1 alpha-2 country code.", json_schema_extra={"entity_label": "Country"}, ) # ── Entity labels (no instance key) ────────────────────────────────── product_name: str | None = Field( default=None, description="Trade name of the medicinal product.", json_schema_extra={"entity_label": "ProductName"}, ) marketing_authorization_holder: str | None = Field( default=None, description="Full legal name of the MAH.", json_schema_extra={"entity_label": "MAH"}, ) # ── Dual-field: raw + normalized ───────────────────────────────────── raw_date: str | None = Field( default=None, description="Free-text date as it appears in the source (e.g. '15 March 2024').", ) normalized_date: NormalizedDate | None = Field( default=None, description="Structured date derived from raw_date.", json_schema_extra={ "normalization_model": True, "normalization_source_fields": ["raw_date"], }, ) # ── Instance relationships (Level 3) ───────────────────────────────── condition_ids: list[str] | None = Field( default=None, description="IDs of applicable conditions (e.g. ['1', '2', '3']).", json_schema_extra={ "instance_relationships": [ { "target_model": "ConditionModel", "rel_type": "APPLIES_TO", "join_via": { "condition_ids": "condition_id", "variation_code": "variation_code", }, } ], }, ) # ── Regular scalar fields ──────────────────────────────────────────── procedure_types_referenced: list[str] | None = Field( default=None, description="Other procedure type codes referenced in this variation.", json_schema_extra={"entity_label": "ProcedureType"}, ) summary: str | None = Field( default=None, description="Brief summary of the variation purpose.", ) regulatory_pathway: str | None = Field( default=None, description="Regulatory pathway: centralized, mutual recognition, or decentralized.", json_schema_extra={"entity_label": "RegulatoryPathway"}, ) # ───────────────────────────────────────────────────────────────────────────── # List wrapper for multi-instance sections # ───────────────────────────────────────────────────────────────────────────── class VariationList(ExtractionModel): """A section containing multiple variation entries. Use this model when a document section contains a table or list of variations. The LLM extracts all entries into the variations list. """ variations: list[VariationModel] = Field( ..., description="All variation entries found in this section.", ) section_title: str | None = Field( default=None, description="Title or heading of the section containing these variations.", ) ``` ### What this example demonstrates | Model | Patterns Used | |---|---| | `VariationBaseModel` | Domain validators (`normalize_procedure_types`, `fix_variation_code_ocr`, `fix_country_code`) with `check_fields=False` | | `NormalizedDate` | Simple nested model for normalization target | | `NormalizedAddress` | Nested model with `entity_label` on `country_code` | | `NormalizedSubstance` | Nested model with multiple `entity_label` fields | | `ConditionModel` | `instance_key` (composite), `entity_label`, inherits validators | | `SubstanceEntry` | **Dual-field** pattern (`raw_substance` + `normalized_substance`), `normalization_model`, `normalization_source_fields` | | `Fee` | Composite `instance_key` (3 fields), `entity_label` on multiple fields | | `IngredientPair` | `field_relationships` creating `[:CONTAINS_EXCIPIENT]` between entities | | `VariationModel` | **All patterns**: dual-field, entity labels, instance keys, validators, instance relationships, normalization model | | `VariationList` | List wrapper for multi-instance sections | ### Neo4j graph produced Processing a document with `VariationModel` produces: ``` (:StructureNode) -[:HAS_EXTRACTION]-> (:ExtractionResult) -[:USES_PRIMARY_MODEL]->(:CatalogModel {name: "VariationModel"}) -[:HAS_CONDITION_IDS {index}]->(:ModelInstance {model_class: "VariationModel"}) -[:REFERENCES]->(:LabeledEntity {label: "VariationCode", value: "Q.I.a.1"}) -[:REFERENCES]->(:LabeledEntity {label: "ProcedureType", value: "II"}) -[:REFERENCES]->(:LabeledEntity {label: "Country", value: "DE"}) -[:REFERENCES]->(:LabeledEntity {label: "ProductName", value: "Metformin 500mg"}) (:ModelInstance {model_class: "VariationModel"}) -[:APPLIES_TO]-> (:ModelInstance {model_class: "ConditionModel", condition_id: "1"}) -[:APPLIES_TO]-> (:ModelInstance {model_class: "ConditionModel", condition_id: "2"}) (:LabeledEntity {label: "VariationCode", value: "Q.I.a.1"}) ← shared across all extractions with this code ``` --- ## Next steps - **[Custom Models](custom-models.md)** — Basic model definition and field descriptions. - **[Tabular Pipeline](tabular-pipeline.md)** — How the tabular pipeline uses normalization models. - **[Neo4j Graph Storage](neo4j-graph.md)** — Understanding the graph model and node types. - **[Architecture](../architecture.md)** — Detailed pipeline stage walkthrough, including entity extraction graph mapping. --- ## File: user-guides/neo4j-graph.md # Neo4j Graph Model `scinr` builds a connected graph representation of input documents and extracted domain concepts inside Neo4j. This guide covers every node type, relationship, and query pattern produced by the `scinr.newton` extraction engine. The graph serves as the primary output store. After a document passes through the Newton pipeline, all structural elements and extracted entities are persisted as interconnected nodes and relationships, enabling complex cross-document queries that would be difficult or impossible with a flat relational schema. Two distinct pipelines produce different graph structures: - **Unstructured pipeline** — processes PDFs, Word documents, and other narrative sources. Produces a hierarchy of headings, paragraphs, tables, and figures, with entities extracted from specific sections. - **Tabular pipeline** — processes CSV, Excel, and other structured tables. Produces one model instance per row, with labeled entities for stable, deduplicated fields. --- ## Node Types ### `:Document` Represents the original input file that was ingested into the system. Every document in the graph is a `:Document` node. ``` (:Document { document_name: "clinical_trial_report.pdf", format: "pdf", ingested_at: 2024-01-15T10:30:00, version: 1 }) ``` | Property | Type | Description | |---|---|---| | `document_name` | String | Original file name as provided at ingestion time. | | `format` | String | File extension or MIME type (e.g., `pdf`, `docx`, `xlsx`, `csv`). | | `ingested_at` | DateTime | Timestamp when the document was first ingested. | | `version` | Integer | Version number. Starts at 1; increments on replacement or update. | --- ### `:StructureNode` Represents a structural element within a document — a heading, paragraph, table, figure, or list item. Structure nodes form a tree rooted at the document. ``` (:StructureNode { uid: "a1b2c3d4-e5f6-7890-abcd-ef1234567890", section_title: "3.2 Adverse Events", node_type: "paragraph", text: "The most common adverse events were headache and nausea...", page_number: 15, hierarchy_level: 2 }) ``` | Property | Type | Description | |---|---|---| | `uid` | String | Unique identifier (UUID) for this structure node. | | `section_title` | String | Heading text, if this node represents a section heading. Null for leaf nodes like paragraphs. | | `node_type` | String | One of: `heading`, `paragraph`, `table`, `figure`, `list`, `row`. | | `text` | String | The raw text content of this structural element. | | `page_number` | Integer | Source page number, when available (PDF, paginated documents). Null for non-paginated sources. | | `hierarchy_level` | Integer | Nesting depth in the document outline. Root headings are level 1, sub-sections increment from there. | --- ### `:LabeledEntity` Represents a stable, deduplicated entity value extracted from a field marked with an `entity_label` in the schema. Labeled entities are created for fields that act as identifiers or cross-references — drug names, product codes, anatomical terms, etc. ``` (:LabeledEntity:ActiveSubstance { value: "Metformin", uid: "a1b2c3d4-e5f6-7890-abcd-ef1234567890" }) ``` | Property | Type | Description | |---|---|---| | `value` | String | The canonical entity value. Used for deduplication across documents. | | `uid` | String | Unique identifier for this specific entity node. | Key characteristics: - **Dynamic label**: In addition to the base `:LabeledEntity` label, the node carries a second label derived from the `entity_label` field metadata (e.g., `:ActiveSubstance`, `:AnatomicalStructure`, `:ProductCode`). - **Deduplication**: If the same `value` is extracted from multiple documents, a single `:LabeledEntity` node is reused. This enables cross-document entity matching and aggregation. - **Stable identity**: The `uid` remains constant for a given value, allowing reliable joins and traces. --- ### `:ModelInstance` Represents a single extracted entity record — one instance of a model class defined in the extraction schema. Each model instance stores all field values as node properties. ``` (:ModelInstance:AdverseEventModel { uid: "a1b2c3d4-e5f6-7890-abcd-ef1234567890", event_type: "Headache", severity: "Mild", occurrence_count: 3 }) ``` | Property | Type | Description | |---|---|---| | `uid` | String | Unique identifier for this model instance. | | *(dynamic)* | Varies | All model field values are stored as additional properties on the node. Property names match the field names defined in the schema. | Key characteristics: - **Dynamic label**: In addition to the base `:ModelInstance` label, the node carries a second label matching the model class name (e.g., `:AdverseEventModel`, `:ProductRecord`, `:LaboratoryResult`). - **One node per record**: Each extracted entity produces exactly one model instance node. In the tabular pipeline, this corresponds to one row per node. - **Field values as properties**: Every field defined in the model schema becomes a property on the node, enabling direct Cypher filtering without additional joins. --- ## Relationship Types ### Document Structure These relationships form the hierarchical tree of a document. | Relationship | From | To | Description | |---|---|---|---| | `[:HAS_STRUCTURE]` | `:Document` | `:StructureNode` | A document contains this top-level structure node. | | `[:HAS_CHILD]` | `:StructureNode` | `:StructureNode` | Parent-child relationship in the document hierarchy. Headings point to their child headings, paragraphs, tables, etc. | ### Extraction These relationships link extracted entities back to the document sections they came from, enabling provenance tracing. | Relationship | From | To | Description | |---|---|---|---| | `[:EXTRACTED_FROM]` | `:ModelInstance` | `:StructureNode` | This entity was extracted from the text in this section. | | `[:EXTRACTED_FROM]` | `:LabeledEntity` | `:StructureNode` | This entity value was extracted from this section. | ### Entity Relationships (from `field_relationships`) When a schema defines `field_relationships` between labeled entity fields, the graph creates domain-specific edges between `:LabeledEntity` nodes. The relationship type is the field relationship name in `UPPER_SNAKE_CASE`. | Relationship | From | To | Description | |---|---|---|---| | Custom `UPPER_SNAKE_CASE` | `:LabeledEntity` | `:LabeledEntity` | Domain-specific semantic edge between two entity values (e.g., `:TREATS`, `:CONTRAINDICATED_WITH`, `:PART_OF`). | ### Instance Relationships (from `instance_relationships`) When a schema defines `instance_relationships` between model instances, the graph creates edges between `:ModelInstance` nodes. The relationship type is the instance relationship name in `UPPER_SNAKE_CASE`. | Relationship | From | To | Description | |---|---|---|---| | Custom `UPPER_SNAKE_CASE` | `:ModelInstance` | `:ModelInstance` | Cross-model record edge (e.g., `:ASSOCIATED_WITH`, `:PRECEDES`, `:BELONGS_TO`). | ### Versioning | Relationship | From | To | Description | |---|---|---|---| | `[:REPLACES]` | `:Document` | `:Document` | The source document is a newer version that replaces the target document. | --- ## Graph Structure Diagrams ### Unstructured Pipeline Output Narrative documents (PDF, Word) produce a heading hierarchy with extracted entities anchored to specific sections: ``` (:Document {document_name: "clinical_trial.pdf"}) └─[:HAS_STRUCTURE]→ (:StructureNode {node_type: "heading", section_title: "3. Results"}) └─[:HAS_CHILD]→ (:StructureNode {node_type: "heading", section_title: "3.2 Adverse Events"}) └─[:HAS_CHILD]→ (:StructureNode {node_type: "paragraph"}) └─[:EXTRACTED_FROM]← (:ModelInstance:AdverseEventModel) └─[:EXTRACTED_FROM]← (:LabeledEntity:ActiveSubstance) ``` In this pattern: - The document contains a top-level heading (`3. Results`). - That heading contains a sub-heading (`3.2 Adverse Events`). - The sub-heading contains a paragraph with text. - Both a model instance (`AdverseEventModel`) and a labeled entity (`ActiveSubstance`) were extracted from that paragraph. - The `:LabeledEntity:ActiveSubstance` node is shared across documents if the same substance appears elsewhere. ### Tabular Pipeline Output Structured documents (CSV, Excel) produce a flat table structure with one model instance per row: ``` (:Document {document_name: "products.csv"}) └─[:HAS_STRUCTURE]→ (:StructureNode {node_type: "table"}) └─[:HAS_CHILD]→ (:StructureNode {node_type: "row"}) └─[:EXTRACTED_FROM]← (:ModelInstance:ProductRecord) └─[:EXTRACTED_FROM]← (:LabeledEntity:ProductCode) └─[:HAS_CHILD]→ (:StructureNode {node_type: "row"}) └─[:EXTRACTED_FROM]← (:ModelInstance:ProductRecord) ``` In this pattern: - The document contains a single table structure node. - Each table row is a child of the table. - Each row produces one model instance (`ProductRecord`). - Labeled entity fields within the model instance share the same extraction provenance, pointing to the same `:StructureNode` via `[:EXTRACTED_FROM]` for deduplication. - `[:HAS_FIELD]` is NOT a real relationship — the labeled entities and model instances are both connected to the same StructureNode via `[:EXTRACTED_FROM]`. The connection between a ModelInstance and its LabeledEntity is implicit through shared field values, not an explicit graph relationship. --- ## Versioning `scinr` supports two document update strategies, both reflected in the graph: ### In-place Update (`update_mode=True`) When `update_mode=True`, the existing document and all its downstream nodes are replaced in-place. The `version` property on the `:Document` node increments, but the `document_name` remains the same. ``` (:Document { document_name: "clinical_trial_report.pdf", version: 2, ingested_at: 2024-06-01T09:00:00 }) ``` ### Replacement (`replaces="old_name"`) When `replaces` is set, a new `:Document` node is created with the new name and version, and a `[:REPLACES]` relationship links it to the old document. ``` (:Document {document_name: "clinical_trial_report_v1.pdf", version: 1}) └─[:REPLACES]← (:Document {document_name: "clinical_trial_report_v2.pdf", version: 2}) ``` This preserves the full history of document versions in the graph, allowing queries to trace which version produced which extraction results. ### Document Deletion When a document needs to be permanently removed from the graph (rather than updated in-place), use `delete_document()`. This function: - Removes the `:Document` node(s) matching the given `path` (and optionally `version`). - Cascade-deletes all connected structure, annotation, and extraction nodes. - Runs garbage collection on orphaned `:Entity`, `:ModelInstance`, and `:LabeledEntity` nodes. Unlike `--update` re-ingestion, deletion is **irreversible** — there is no undo. See the [Document Deletion](document-deletion.md) guide for details. --- ## Query Patterns ### Find All Documents List all ingested documents, ordered by most recent first: ```cypher MATCH (d:Document) RETURN d.document_name, d.format, d.ingested_at, d.version ORDER BY d.ingested_at DESC; ``` ### Find All Entities of a Type Aggregate extracted model instances by their field values: ```cypher MATCH (m:ModelInstance:AdverseEventModel) RETURN m.event_type, m.severity, count(m) AS occurrences ORDER BY occurrences DESC; ``` ### Find Entities Extracted from a Specific Document Trace all model instances back through their source document: ```cypher MATCH (d:Document)-[:HAS_STRUCTURE*0..]->(s:StructureNode)<-[:EXTRACTED_FROM]-(m:ModelInstance) WHERE d.document_name = "clinical_trial_report.pdf" RETURN labels(m) AS entity_type, count(m) AS count ORDER BY count DESC; ``` The `*0..` variable-length relationship allows matching both direct children and deeply nested structure nodes. ### Find Entity Relationships List all domain-specific edges between labeled entities: ```cypher MATCH (a:LabeledEntity)-[r]->(b:LabeledEntity) RETURN labels(a) AS source_type, type(r) AS relationship, labels(b) AS target_type, count(r) AS count ORDER BY count DESC; ``` ### Find Cross-Model Connections List all relationships between model instances: ```cypher MATCH (a:ModelInstance)-[r]->(b:ModelInstance) RETURN labels(a) AS source_type, type(r) AS relationship, labels(b) AS target_type, count(r) AS count ORDER BY count DESC; ``` ### Full Extraction Trace Reconstruct the full provenance chain from document to extracted entity: ```cypher MATCH (d:Document)-[:HAS_STRUCTURE*0..]->(s:StructureNode)<-[:EXTRACTED_FROM]-(m:ModelInstance) RETURN d.document_name, s.section_title, s.node_type, labels(m), m; ``` This query is useful for auditing: it shows exactly which section of which document produced each extracted entity. ### Find Entity Usage Across Documents Identify which documents share a particular entity value: ```cypher MATCH (e:LabeledEntity:ActiveSubstance)-[:EXTRACTED_FROM]->(s:StructureNode)<-[:HAS_STRUCTURE*0..]-(d:Document) WHERE e.value = "Metformin" RETURN d.document_name, s.section_title ORDER BY d.document_name; ``` ### Find Document Version History Trace the lineage of a document through its versions: ```cypher MATCH (old:Document)-[:REPLACES*0..]->(new:Document) WHERE new.document_name = "clinical_trial_report_v2.pdf" RETURN old.document_name AS version_name, old.version, old.ingested_at ORDER BY old.version; ``` --- ## Indexes and Constraints For production workloads, create the following indexes and constraints to ensure performant lookups: ```cypher -- Unique constraint on document name (enforces deduplication at ingestion) CREATE CONSTRAINT document_name_unique FOR (d:Document) REQUIRE d.document_name IS UNIQUE; -- Index on labeled entity value (accelerates deduplication and cross-document lookups) CREATE INDEX labeled_entity_value FOR (n:LabeledEntity) ON (n.value); -- Index on model instance UID (accelerates direct entity lookups) CREATE INDEX model_instance_uid FOR (m:ModelInstance) ON (m.uid); -- Index on structure node UID (accelerates provenance tracing) CREATE INDEX structure_node_uid FOR (s:StructureNode) ON (s.uid); -- Index on document ingestion date (accelerates temporal queries) CREATE INDEX document_ingested_at FOR (d:Document) ON (d.ingested_at); ``` --- ## Graph Size Estimation The following table provides rough estimates for planning database capacity: | Input | Approximate Nodes | Approximate Relationships | |---|---|---| | 1 PDF (50 pages) | ~500 StructureNodes + extracted entities | ~600 (structure + extraction) | | 10 PDFs (50 pages each) | ~5,000 StructureNodes + extracted entities | ~6,000 (structure + extraction) | | 1 CSV (1,000 rows) | ~1,000 ModelInstances + labeled entities | ~1,000 (extraction) + entity/instance relationships | | 1 Excel (5 sheets, 500 rows each) | ~2,500 ModelInstances + labeled entities | ~2,500 (extraction) + entity/instance relationships | | 100 PDFs (mixed size) | ~50,000 StructureNodes + extracted entities | ~60,000 (structure + extraction) | Notes: - Actual counts depend on document complexity, schema definition, and extraction yield. - Labeled entity deduplication reduces total node count when the same entity appears across multiple documents. - Entity and instance relationships add proportionally to the relationship count based on schema configuration. - For large-scale deployments, consider partitioning strategies and Neo4j clustering. --- ## File: user-guides/normalization.md # LLM Normalization System The normalization system turns free-text fields into structured nested Pydantic models via LLM calls. It is wired exclusively into the **tabular pipeline** (CSV/XLSX/XLS), running between column mapping and Neo4j write. The unstructured pipeline (PDF/DOCX) does not use the normalization engine — the LLM fills nested fields directly during entity extraction. --- ## Table of Contents 1. [Introduction](#1-introduction) 2. [How Normalization Works (Mechanical Flow)](#2-how-normalization-works-mechanical-flow) 3. [Declaring Normalization Fields](#3-declaring-normalization-fields) 4. [Configuration](#4-configuration) 5. [Mandatory vs. Optional Decision Matrix](#5-mandatory-vs-optional-decision-matrix) 6. [The Implicit Fallback (Footgun)](#6-the-implicit-fallback-footgun) 7. [NormalizationEngine Architecture](#7-normalizationengine-architecture) 8. [Performance Considerations](#8-performance-considerations) 9. [Multiple Normalization Fields](#9-multiple-normalization-fields) 10. [Troubleshooting](#10-troubleshooting) 11. [Complete Example](#11-complete-example) --- ## 1. Introduction ### What normalization is Normalization is the process of taking a raw, free-text field value and transforming it into a structured nested Pydantic model via an LLM call with structured output. For example, a CSV column containing `"123 Main St, Springfield, IL 62704"` becomes a `NormalizedAddress` instance with separate `street`, `city`, `postal_code`, and `country_code` fields. ### Where it fits in the pipeline ``` CSV Row → Column Mapping → Model Instance │ NormalizationEngine │ LLM structured output │ Apply to Instance (setattr) │ Write to Neo4j ``` The normalization step sits between Pydantic model instantiation and Neo4j graph write. It is a **tabular-only hook**: it never runs during the unstructured pipeline (Stages 3-4). ### Why it matters Tabular data often contains messy, inconsistent, or composite values in a single column. A "Manufacturer Address" column might contain street, city, state, and country all jammed together. Without normalization, you get one unstructured string property on your Neo4j node. With normalization, you get a properly structured `:ModelInstance` node with queryable, comparable, and deduplicatable fields. ### Key characteristics | Characteristic | Detail | |---|---| | **Opt-in** | Only fields with `normalization_model: True` in `json_schema_extra` are processed | | **Off by default** | The engine is disabled unless `normalization_enabled=True` | | **Tabular-only** | Wired into the tabular pipeline; ignored by the unstructured pipeline | | **Additive** | A normalization field is still an ordinary nested model for all other purposes | | **Deduplicated** | Identical source values across rows trigger only one LLM call | | **Batched** | Multiple unique entries of the same target type share a single LLM call | --- ## 2. How Normalization Works (Mechanical Flow) ### End-to-end diagram ``` CSV Row → Column Mapping → Model Instance │ NormalizationEngine │ ┌─────────────┼─────────────┐ ▼ ▼ ▼ Detection Batching LLM Call (scan schema) (group + dedup) (structured output) │ │ │ ▼ ▼ ▼ Source vals Unique keys Normalized model collected cached instances │ │ └─────────┬───────────────┘ ▼ Apply to Instance (setattr + fallback) ▼ Write to Neo4j ``` ### Step-by-step #### Phase 1: Detection (schema inspection) The `NormalizationEngine` calls `get_normalization_specs(model_class)` for each model class in the pipeline. This function inspects the Pydantic schema and extracts all fields that have `json_schema_extra={"normalization_model": True}`. For each such field, it records: - **`field_name`** — the name of the normalization target field on the parent model - **`target_type`** — the Pydantic class of the normalized model (extracted from the field annotation, handling `X | None`, `list[X]`, `Annotated[X, ...]`) - **`source_fields`** — the list of source field names declared in `normalization_source_fields`, or `None` if omitted (implicit fallback) ```python # detector.py — simplified def get_normalization_specs(model_class: type[BaseModel]) -> list[NormalizationSpec]: specs = [] for field_name, field_info in model_class.model_fields.items(): extra = _get_json_schema_extra(field_info) if not extra.get("normalization_model", False): continue target_type = _extract_target_type(field_info.annotation) source_fields = extra.get("normalization_source_fields", []) or [] specs.append(NormalizationSpec( field_name=field_name, target_type=target_type, source_fields=source_fields if source_fields else None, )) return specs ``` #### Phase 2: Entry creation (source value collection) For each model instance, the engine calls `extract_source_values(spec, instance)` to collect the raw values from the declared source fields. If `spec.source_fields` is `None`, it falls back to **all scalar fields** on the model (the implicit fallback — see Section 6). Each entry is wrapped in a `NormalizationEntry` dataclass: ```python @dataclass class NormalizationEntry: instance_id: int # id() of the Pydantic instance model_class_name: str # e.g. "ContactRecord" field_name: str # e.g. "normalized_address" target_type: type[BaseModel] # e.g. NormalizedAddress source_values: dict[str, object] # e.g. {"raw_address": "123 Main St..."} unique_key: str # "{target_type_name}:{md5_hash}" row_indices: list[int] # row indices from pre-scan ``` The **unique key** is constructed as `{target_type.__name__}:{md5_hash}` where the MD5 hash is computed from the sorted, lowercased string representation of the source values. This ensures identical source values across different rows produce the same key. #### Phase 3: Batching (group by target type) Entries are grouped by `target_type.__name__` because the LLM structured output call requires a homogeneous target type. Within each group, entries are batched by `normalization_batch_size` (default: 5). Duplicate entries (same unique key from different rows) are deduplicated — only unique keys proceed to the LLM. ```python # engine.py — simplified unique_entries: dict[str, list[NormalizationEntry]] = {} for entry in entries: if entry.unique_key in seen_keys: continue # dedup seen_keys.add(entry.unique_key) type_key = entry.target_type.__name__ unique_entries.setdefault(type_key, []).append(entry) ``` #### Phase 4: LLM call (structured output) For each batch, the engine: 1. Builds dynamic Pydantic output schemas (`BatchOutput` and `BatchResponse`) wrapping the target type 2. Calls `self.llm.with_structured_output(BatchResponse)` to create a structured-output LLM 3. Builds a prompt with the system message and all entries' source data 4. Calls `ainvoke()` with retry via `with_llm_retry()` 5. Coerces the result to `BatchResponse` (handles dict returns from some providers) The dynamic schemas look like: ```python BatchOutput = type( f"BatchOutput_{target_type.__name__}", (BaseModel,), { "__annotations__": {"key": str, "result": target_type}, "model_config": ConfigDict(extra="forbid"), }, ) BatchResponse = type( f"BatchResponse_{target_type.__name__}", (BaseModel,), { "__annotations__": {"results": list[BatchOutput]}, "model_config": ConfigDict(extra="forbid"), }, ) ``` The system prompt is: ``` You are a data normalization assistant. You receive raw extracted data and must normalize it into a structured format. Fill in all fields you can confidently identify from the source data. Leave uncertain fields as null. ``` #### Phase 5: Result caching and application Results from the LLM are stored in `self.result_cache` keyed by unique key. The engine then applies each result back to the original model instances via `setattr`, with `object.__setattr__` as a fallback for models with `validate_assignment=True` or class identity mismatches: ```python def _apply_to_instance(self, instance_id, field_name, normalized, all_instances): for _, instance in all_instances: if id(instance) == instance_id: try: setattr(instance, field_name, normalized) except Exception: # Fallback: bypass Pydantic validation object.__setattr__(instance, field_name, normalized) return ``` #### Phase 6: Write to Neo4j Once all normalization fields are populated, the instances proceed to the Neo4j write phase. The graph mapper treats the normalized nested models as regular `:ModelInstance` nodes, creating them with their properties and any `entity_label` or `instance_key` relationships they declare. ### Missing results retry If the LLM returns fewer results than entries in the batch (e.g., it skips one entry), the engine automatically retries the missing keys **once**. The retry is done with the same batch mechanism but only for the missing entries. --- ## 3. Declaring Normalization Fields ### The dual-field pattern Normalization uses the **dual-field pattern**: a raw free-text field paired with a normalized nested model. The raw field preserves the original text; the normalized field provides structured, queryable components. ```python from pydantic import Field from scinr.newton.models.base import ExtractionModel class NormalizedAddress(ExtractionModel): """Structured, normalized postal address.""" street: str | None = Field( default=None, description="Street address line, without city or postal code.", ) city: str | None = Field( default=None, description="City name.", ) postal_code: str | None = Field( default=None, description="Postal or ZIP code.", ) country_code: str | None = Field( default=None, description="ISO 3166-1 alpha-2 country code (e.g. 'US', 'DE', 'JP').", json_schema_extra={"entity_label": "Country"}, ) class ContactRecord(ExtractionModel): """A single contact record from a CSV file.""" # ─── Tier 1: Free-text raw field ─── raw_name: str = Field( ..., description="Full name as written in source column.", ) raw_address: str = Field( ..., description="Free-text address from source column.", ) raw_phone: str | None = Field( default=None, description="Phone number if present.", ) # ─── Tier 2: Normalization target ─── normalized_address: NormalizedAddress | None = Field( default=None, description="Structured address derived from raw_address.", json_schema_extra={ "normalization_model": True, "normalization_source_fields": ["raw_address"], }, ) ``` ### Key components | Component | Location | Purpose | |---|---|---| | `normalization_model: True` | `json_schema_extra` on the **target field** | Marks the field for the `NormalizationEngine` | | `normalization_source_fields` | `json_schema_extra` on the **target field** | Declares which parent fields feed the normalization | | Target type | Field annotation (e.g., `NormalizedAddress \| None`) | The Pydantic model to populate via LLM | | Source fields | Regular fields on the **parent model** | The raw data used as LLM input | ### Rules for the target field | Rule | Details | |---|---| | **Must be nullable** | Use `TargetType \| None` with `default=None` — the field starts as `None` until normalization runs | | **Must be a Pydantic model** | The annotation must resolve to a `BaseModel` subclass (through `X`, `X \| None`, `list[X]`, etc.) | | **Must have `normalization_model: True`** | Without this flag, the engine skips the field entirely | | **Should have `normalization_source_fields`** | Without it, the engine falls back to all scalar fields (the implicit footgun — see Section 6) | ### Rules for the target model | Rule | Details | |---|---| | **Inherit from `ExtractionModel`** | Ensures compatibility with the graph mapper | | **Fields should be nullable** | Use `str \| None` with `default=None` — the LLM may not be able to extract all fields | | **Use `entity_label` for dedup fields** | Fields like `country_code` or `substance_name` benefit from cross-document deduplication | | **Use `instance_key` for composite identity** | If the normalized model represents a globally unique entity, declare instance keys | ### Rules for source fields | Rule | Details | |---|---| | **Must exist on the parent model** | The source field names must match actual field names on the parent | | **Should be scalar** | Source fields are typically `str` or `str \| None` — the engine reads their values with `getattr()` | | **Must have data** | If all source fields are `None` or empty string, the entry is skipped entirely | | **Can be multiple** | List multiple source fields if the normalization needs context from several columns | --- ## 4. Configuration ### Via `configure()` ```python from langchain_aws import ChatBedrockConverse from scinr.newton import configure # Option 1: Use the same LLM as the main pipeline configure( normalization_enabled=True, normalization_batch_size=10, ) # Option 2: Use a dedicated (cheaper) LLM for normalization normalize_llm = ChatBedrockConverse( model="us.anthropic.claude-haiku-3", region_name="us-east-1", ) configure( normalization_enabled=True, normalization_batch_size=10, normalization_llm=normalize_llm, ) ``` ### Via environment variables | Parameter | Env Var | Default | Description | |---|---|---|---| | `normalization_enabled` | `NORMALIZATION_ENABLED` | `false` | Enable/disable the normalization engine | | `normalization_batch_size` | `NORMALIZATION_BATCH_SIZE` | `5` | Max entries per LLM batch call | | `normalization_llm` | — | Falls back to main `llm` | Dedicated LLM instance for normalization calls | ### Parameter resolution order For each parameter: **explicit argument** > **environment variable** > **default value**. ```bash # Enable normalization via env var export NORMALIZATION_ENABLED=true export NORMALIZATION_BATCH_SIZE=10 # Run pipeline — env vars are picked up automatically scinr-ingest --input ./data/contacts.csv --theme contacts ``` ### When to use a dedicated normalization LLM | Factor | Main LLM | Dedicated LLM | |---|---|---| | **Cost** | Higher per call | Lower (use a cheaper model like Haiku) | | **Quality** | Best for complex extraction | Sufficient for straightforward normalization | | **Latency** | Shared queue with extraction | Can run in parallel with extraction | | **Recommendation** | Use for small datasets | Use for large tabular datasets (100+ rows) | The normalization task is relatively simple: parse structured text into a known schema. A cheaper, faster model like Claude Haiku or a small GPT variant handles this well. --- ## 5. Mandatory vs. Optional Decision Matrix ### Decision table | Model used with... | Add `normalization_model` keys? | Why | |---|---|---| | **Tabular only** (CSV/XLSX/XLS) | ✅ **Mandatory** | The `NormalizationEngine` requires `normalization_model: True` to trigger. Without it, the field stays `None`. | | **Unstructured only** (PDF/DOCX) | ⚪ **Optional** | The LLM fills the nested field directly from the `description=` during entity extraction. No separate normalization step is needed. | | **Both pipelines** | ✅ **Recommended** | Mandatory for the tabular half; the keys serve as a useful hint for the unstructured LLM. | ### Tabular-only model ```python # This model is used exclusively with CSV files. # Normalization keys are MANDATORY — without them, normalized_address is always None. class SupplierRecord(ExtractionModel): """A supplier record from a CSV spreadsheet.""" supplier_name: str = Field(..., description="Supplier name.") raw_address: str = Field(..., description="Free-text address.") raw_phone: str | None = Field(default=None, description="Phone number.") normalized_address: NormalizedAddress | None = Field( default=None, description="Structured address.", json_schema_extra={ "normalization_model": True, "normalization_source_fields": ["raw_address"], }, ) ``` ### Unstructured-only model ```python # This model is used exclusively with PDF/DOCX documents. # Normalization keys are OPTIONAL — the LLM fills normalized_address directly. class PatentRecord(ExtractionModel): """A patent record extracted from a PDF document.""" patent_number: str = Field(..., description="Patent number.") title: str = Field(..., description="Patent title.") applicant_name: str = Field(..., description="Applicant name.") # No normalization_model needed — the LLM fills this during extraction applicant_address: NormalizedAddress | None = Field( default=None, description="Structured address of the applicant.", ) ``` ### Dual-pipeline model ```python # This model is used with BOTH CSV and PDF. # Normalization keys are RECOMMENDED — mandatory for tabular, helpful for unstructured. class ManufacturerRecord(ExtractionModel): """A manufacturer record usable in both tabular and unstructured pipelines.""" manufacturer_name: str = Field(..., description="Manufacturer name.") raw_address: str = Field(..., description="Free-text address from source.") country: str | None = Field(default=None, description="Country name or code.") normalized_address: NormalizedAddress | None = Field( default=None, description="Structured address derived from raw_address.", json_schema_extra={ "normalization_model": True, "normalization_source_fields": ["raw_address"], }, ) ``` --- ## 6. The Implicit Fallback (Footgun) ### What it is When `normalization_source_fields` is omitted or empty in the `json_schema_extra`, the `NormalizationEngine` falls back to using **ALL scalar fields** on the parent model as source data for the normalization LLM call. This is almost never what you want. ### The problem ```python # ❌ BAD — no normalization_source_fields declared class WideRecord(ExtractionModel): """A wide record with many fields.""" raw_name: str = Field(..., description="Full name.") raw_address: str = Field(..., description="Free-text address.") raw_phone: str | None = Field(default=None, description="Phone number.") internal_notes: str | None = Field(default=None, description="Internal notes.") department: str | None = Field(default=None, description="Department name.") created_at: str | None = Field(default=None, description="Creation timestamp.") normalized_address: NormalizedAddress | None = Field( default=None, description="Structured address.", json_schema_extra={ "normalization_model": True, # Missing! Engine sends ALL scalar fields to LLM: # raw_name, raw_address, raw_phone, internal_notes, department, created_at }, ) ``` In this case, the normalization LLM receives **six fields** as source data for an address normalization task. Only `raw_address` is relevant. The other five fields are noise that: - **Waste tokens** — every irrelevant field adds to the prompt size - **Leak context** — internal notes or department names may confuse the LLM - **Degrade accuracy** — the LLM may try to extract city from a department name - **Break dedup** — the unique key hash includes all fields, so two rows with the same address but different departments get separate LLM calls ### How the fallback works (internally) ```python # detector.py — extract_source_values() if spec.source_fields: # Explicit: only declared fields for src_field in spec.source_fields: val = getattr(instance, src_field, None) if val is not None and val != "": values[src_field] = val else: # Implicit fallback: ALL scalar fields for field_name, field_info in instance.model_fields.items(): if field_name == spec.field_name: continue # skip the target field itself ann_type = _extract_target_type(field_info.annotation) if ann_type is not None: continue # skip nested Pydantic models val = getattr(instance, field_name, None) if val is not None and val != "": values[field_name] = val ``` ### The fix ```python # ✅ GOOD — explicit source fields class NarrowRecord(ExtractionModel): """A record with explicit normalization source.""" raw_name: str = Field(..., description="Full name.") raw_address: str = Field(..., description="Free-text address.") raw_phone: str | None = Field(default=None, description="Phone number.") internal_notes: str | None = Field(default=None, description="Internal notes.") department: str | None = Field(default=None, description="Department name.") normalized_address: NormalizedAddress | None = Field( default=None, description="Structured address.", json_schema_extra={ "normalization_model": True, "normalization_source_fields": ["raw_address"], # Only what's needed }, ) ``` ### Comparison | Aspect | Implicit (bad) | Explicit (good) | |---|---|---| | **Source data** | All scalar fields on parent | Only declared fields | | **LLM prompt size** | Larger, noisy | Minimal, focused | | **LLM accuracy** | Lower (distracted by irrelevant fields) | Higher (focused on relevant data) | | **Dedup hash** | Includes irrelevant fields (fewer cache hits) | Based on relevant data only (more cache hits) | | **Token cost** | Higher | Lower | | **Predictability** | Changes if parent model gains fields | Stable and explicit | **Always declare `normalization_source_fields` explicitly.** The implicit fallback exists only for backward compatibility and should be treated as a bug if encountered in new code. --- ## 7. NormalizationEngine Architecture ### Class overview ```python class NormalizationEngine: def __init__( self, llm: BaseLanguageModel, batch_size: int = 5, concurrency: int = 5, # kept for API compat, not used ) -> None: async def normalize_instances( self, instances: list[tuple[type[BaseModel], BaseModel]], ) -> list[tuple[type[BaseModel], BaseModel]]: async def process_key_batch( self, entries: list[NormalizationEntry], retry_count: int = 0, ) -> dict[str, BaseModel]: def apply_cached_to_instance( self, instance: BaseModel, field_name: str, unique_key: str, ) -> bool: ``` ### `__init__` | Parameter | Type | Default | Description | |---|---|---|---| | `llm` | `BaseLanguageModel` | — | LangChain LLM for normalization calls | | `batch_size` | `int` | `5` | Maximum entries per LLM batch call | | `concurrency` | `int` | `5` | **Deprecated** — kept for API compatibility. Real concurrency is governed by `config.get_llm_semaphore()` | The `concurrency` parameter is no longer used to create a local semaphore. All LLM calls (extraction, entity extraction, annotation, normalization) share a single global semaphore from `config.get_llm_semaphore()` to avoid exceeding the Bedrock botocore connection pool. ### `normalize_instances()` The main entry point. Takes a list of `(model_class, instance)` tuples and returns the same list with normalization fields populated. **Internal phases:** 1. **Collect entries** — scan each instance's model class for normalization specs, extract source values, build unique keys 2. **Build key-to-targets map** — maps each unique key to the list of `(instance_id, field_name)` pairs that need the result 3. **Group by target type** — separate entries by `target_type.__name__` (LLM needs homogeneous batches) 4. **Deduplicate** — skip entries with duplicate unique keys (same source values from different rows) 5. **Batch and dispatch** — split each type group into batches of `batch_size`, create async tasks with `get_llm_semaphore()` 6. **Await all tasks** — `asyncio.gather(*tasks)` runs all batches concurrently (bounded by global semaphore) ### `process_key_batch()` Public method for processing a batch of unique normalization keys. Used by the normalization-first write path in `neo4j_ops.py`. Returns `{unique_key: normalized_result}` for successfully processed keys. Results are cached in `self.result_cache` for reuse. **Internal flow:** 1. Validate all entries share the same `target_type` 2. Build dynamic `BatchOutput` and `BatchResponse` schemas 3. Create structured-output LLM: `self.llm.with_structured_output(BatchResponse)` 4. Build prompt messages via `_build_batch_messages()` 5. Call LLM with retry: `await with_llm_retry(lambda: structured_llm.ainvoke(messages))` 6. Coerce result to `BatchResponse` (handles dict returns) 7. Cache results in `self.result_cache` 8. Retry missing keys once (if `retry_count < 1`) ### `apply_cached_to_instance()` Applies a cached normalization result to a specific instance field. Returns `True` if the key was found and applied, `False` otherwise. Uses `setattr()` with `object.__setattr__()` fallback to handle models with `validate_assignment=True`. ### `_build_batch_messages()` Constructs the prompt for a batch of entries: ```python # System message "You are a data normalization assistant. You receive raw extracted data " "and must normalize it into a structured format. Fill in all fields you can " "confidently identify from the source data. Leave uncertain fields as null." # Human message (simplified) "Normalize the following extracted data entries into structured format. For each entry, return a result with: - key: the exact unique key from the entry (must match exactly) - result: the normalized structured output Entries: --- Entry NormalizedAddress:abc123 --- Source data: raw_address: 123 Main St, Springfield, IL 62704 --- Entry NormalizedAddress:def456 --- Source data: raw_address: 456 Oak Ave, Portland, OR 97201 Return a list of results, one per entry." ``` ### `_hash_source_values()` Generates a deterministic MD5 hash from source values: ```python @staticmethod def _hash_source_values(source_values: dict[str, Any]) -> str: normalized = str(sorted(source_values.items())).lower() return hashlib.md5(normalized.encode()).hexdigest() ``` Sorting ensures the hash is independent of dict ordering. Lowercasing ensures case-insensitive dedup (though the values themselves are not lowercased — only the hash input). --- ## 8. Performance Considerations ### Batch size tuning | Batch size | LLM calls (100 unique entries) | Tokens per call | Total latency | |---|---|---|---| | 1 | 100 | Low | High (100 sequential calls) | | 5 (default) | 20 | Moderate | Balanced | | 10 | 10 | Higher | Lower (fewer calls) | | 20 | 5 | High | Lowest (but risk of context overflow) | **Guidance:** - **Small datasets** (< 50 rows): batch size 5-10 is fine - **Medium datasets** (50-500 rows): batch size 10-15 - **Large datasets** (500+ rows): batch size 15-20, monitor for context overflow - **Very wide source fields** (many source fields per entry): keep batch size lower (5-10) to avoid context overflow ### Caching (deduplication) The engine caches results by unique key (`{target_type}:{md5_hash}`). This means: - **Identical source values** across rows trigger only one LLM call - **Different source values** for the same target type still batch together - **Cache is per-engine-instance** — not persisted across pipeline runs Example: a CSV with 1000 rows where 200 have the same address value: ``` Without caching: 1000 LLM calls (worst case) With caching: 801 unique calls (200 rows share one result) With batching (size 10): ~81 LLM calls ``` ### Dedicated LLM Using a separate, cheaper LLM for normalization: ```python configure( llm=ChatBedrockConverse(model="us.anthropic.claude-sonnet-4-20250514"), # Main LLM normalization_llm=ChatBedrockConverse(model="us.anthropic.claude-haiku-3"), # Normalization LLM ) ``` Benefits: - **Cost reduction** — normalization is a simpler task than full extraction - **Parallel execution** — normalization LLM calls share the global semaphore but use a separate model endpoint - **Quality isolation** — a normalization failure doesn't block the main extraction pipeline ### Concurrency Normalization LLM calls share the global `llm_concurrency` semaphore (default: 4). This means: - Maximum 4 concurrent LLM calls across **all** pipeline stages (extraction, entity extraction, annotation, normalization) - The semaphore is acquired per batch, not per entry - Increasing `llm_concurrency` allows more parallel normalization calls but may exceed Bedrock rate limits ```python configure( llm_concurrency=8, # More parallel LLM calls normalization_batch_size=10, # Larger batches ) ``` ### Pre-scan dedup map (neo4j_ops.py) The normalization-first write path in `neo4j_ops.py` performs a **pre-scan** of all rows before any LLM calls: 1. Scans all rows to build a global dedup map of unique normalization keys 2. Groups unique keys by target type 3. Dispatches all key batches concurrently via `asyncio.gather()` 4. Instantiates composites with cached normalization results 5. Writes to Neo4j in row batches This approach ensures: - **Single-pass dedup** — all rows scanned once before any LLM call - **Maximal batching** — all unique keys of the same type batch together - **Concurrent LLM calls** — all type batches run in parallel - **Atomic writes** — rows are written only after their normalization results are available --- ## 9. Multiple Normalization Fields A single model can declare multiple normalization target fields, each with its own target type and source fields. The engine processes each independently. ```python class ComplexRecord(ExtractionModel): """Record with multiple normalization targets.""" raw_address: str = Field(..., description="Free-text address.") raw_substance: str = Field(..., description="Free-text substance description.") raw_strength: str = Field(..., description="Free-text strength information.") raw_date: str | None = Field(default=None, description="Free-text date.") # ─── Normalization target 1: Address ─── normalized_address: NormalizedAddress | None = Field( default=None, description="Structured address.", json_schema_extra={ "normalization_model": True, "normalization_source_fields": ["raw_address"], }, ) # ─── Normalization target 2: Substance ─── normalized_substance: NormalizedSubstance | None = Field( default=None, description="Structured substance data.", json_schema_extra={ "normalization_model": True, "normalization_source_fields": ["raw_substance"], }, ) # ─── Normalization target 3: Strength ─── normalized_strength: NormalizedStrength | None = Field( default=None, description="Structured strength data.", json_schema_extra={ "normalization_model": True, "normalization_source_fields": ["raw_strength"], }, ) # ─── Normalization target 4: Date ─── normalized_date: NormalizedDate | None = Field( default=None, description="Structured date.", json_schema_extra={ "normalization_model": True, "normalization_source_fields": ["raw_date"], }, ) ``` ### How the engine handles multiple fields 1. **Detection**: `get_normalization_specs()` returns a list of all normalization specs for the model class (one per `normalization_model: True` field) 2. **Entry creation**: Each spec generates its own `NormalizationEntry` with its own unique key 3. **Grouping**: Entries are grouped by `target_type.__name__` — different target types get separate LLM calls 4. **Batching**: Entries of the same target type batch together regardless of which parent field they came from 5. **Application**: Each result is applied to the correct field on the correct instance ### Performance implications | Scenario | Effect | |---|---| | Multiple fields, same target type | Entries batch together — efficient | | Multiple fields, different target types | Separate LLM calls per type — more calls | | Multiple fields, same source fields | Each field still gets its own LLM call — consider if you really need separate models | ### Shared source fields Multiple normalization fields can reference the same source field: ```python class SubstanceRecord(ExtractionModel): raw_description: str = Field(..., description="Full substance description.") normalized_substance: NormalizedSubstance | None = Field( default=None, description="Substance identity.", json_schema_extra={ "normalization_model": True, "normalization_source_fields": ["raw_description"], }, ) normalized_strength: NormalizedStrength | None = Field( default=None, description="Strength and form.", json_schema_extra={ "normalization_model": True, "normalization_source_fields": ["raw_description"], # Same source! }, ) ``` This triggers two separate LLM calls (different target types) with the same source data. The LLM extracts different aspects in each call. This is intentional when you want to decompose a complex field into multiple structured models. --- ## 10. Troubleshooting ### Common problems | Problem | Cause | Fix | |---|---|---| | Normalized field stays `None` | `normalization_enabled=False` | Set to `True` in `configure()` or env var | | Normalized field stays `None` | No `normalization_model: True` on the field | Add the flag to `json_schema_extra` | | Wrong data in normalized field | Implicit fallback (no `normalization_source_fields`) | Add explicit `normalization_source_fields` | | Wrong data in normalized field | Source field name mismatch | Verify source field names match actual model fields | | Slow normalization | Batch size too small | Increase `normalization_batch_size` | | Slow normalization | Too many unique entries | Check for dedup opportunities | | LLM errors | `normalization_llm` not configured and main LLM unavailable | Pass `normalization_llm` or verify main `llm` | | LLM errors | Context overflow (too many entries per batch) | Decrease `normalization_batch_size` | | Missing results | LLM skipped an entry in the batch | Engine retries once; check logs for warnings | | Type errors | Target type not a Pydantic model | Ensure annotation resolves to `BaseModel` subclass | | Validation errors | `validate_assignment=True` rejecting normalized value | Engine uses `object.__setattr__` fallback automatically | ### Debug logging Enable debug logging to see normalization internals: ```python import logging logging.getLogger("scinr.newton.tabular.normalization").setLevel(logging.DEBUG) ``` Key log messages to watch for: ``` # Entry collection "Normalization: collected {N} entries from {M} instances" # Dedup "Normalization: {N} unique keys (from {M} total entries)" # Batching "Normalization: processing batch of {N} entries for {TargetType}" # LLM call "Normalization batch returned {N} results" # Missing results retry "Normalization: {N}/{M} results missing, retrying: {keys}" # Batch failure "Normalization batch failed for {TargetType} ({N} entries): {error}" # Instance not found "Normalization: instance id {id} not found for {field}, skipping" ``` ### Verifying normalization is active ```python from scinr.newton.config import get_config cfg = get_config() print(f"Normalization enabled: {cfg.normalization_enabled}") print(f"Batch size: {cfg.normalization_batch_size}") print(f"Dedicated LLM: {cfg.normalization_llm is not None}") ``` ### Checking model specs ```python from scinr.newton.tabular.normalization.detector import get_normalization_specs specs = get_normalization_specs(ContactRecord) for spec in specs: print(f"Field: {spec.field_name}") print(f" Target: {spec.target_type.__name__}") print(f" Source fields: {spec.source_fields}") ``` --- ## 11. Complete Example ### Scenario A pharmaceutical company has a CSV file of manufacturer contact information. Each row has a free-text address column that needs to be normalized into structured components (street, city, postal code, country). The company also wants to normalize the substance information from a separate column. ### Step 1: Define the models ```python """ models/pharma_contacts.py — Pharmaceutical manufacturer contact models. Demonstrates the complete normalization flow: - Dual-field pattern (raw + normalized) - Multiple normalization targets per model - Entity labels for cross-document deduplication - Instance keys for model instance deduplication """ from __future__ import annotations from pydantic import Field from scinr.newton.models.base import ExtractionModel # ─── Normalization targets ─────────────────────────────────────────────────── class NormalizedAddress(ExtractionModel): """Structured, normalized postal address.""" street: str | None = Field( default=None, description="Street address line, without city or postal code.", ) city: str | None = Field( default=None, description="City name.", ) postal_code: str | None = Field( default=None, description="Postal or ZIP code.", ) country_code: str | None = Field( default=None, description="ISO 3166-1 alpha-2 country code (e.g. 'US', 'DE', 'JP').", json_schema_extra={"entity_label": "Country"}, ) class NormalizedSubstance(ExtractionModel): """Structured, normalized substance information.""" inn_name: str | None = Field( default=None, description="International Nonproprietary Name (INN).", json_schema_extra={"entity_label": "ActiveSubstance"}, ) cas_number: str | None = Field( default=None, description="CAS Registry Number (e.g. '1105-50-9').", json_schema_extra={"entity_label": "CasNumber"}, ) strength: str | None = Field( default=None, description="Strength with units (e.g. '500 mg', '10 mg/mL').", ) pharmaceutical_form: str | None = Field( default=None, description="Pharmaceutical form (e.g. 'tablet', 'solution', 'powder').", json_schema_extra={"entity_label": "PharmaceuticalForm"}, ) # ─── Parent model (tabular record) ─────────────────────────────────────────── class ManufacturerContact(ExtractionModel): """A manufacturer contact record from a CSV file. Uses the dual-field pattern with explicit normalization source fields. The NormalizationEngine processes raw_address and raw_substance columns to populate the normalized nested models. """ # ─── Raw scalar fields ──────────────────────────────────────────────── manufacturer_name: str = Field( ..., description="Legal name of the manufacturing company.", json_schema_extra={"entity_label": "Manufacturer"}, ) raw_address: str = Field( ..., description="Free-text address as it appears in the source column.", ) raw_substance: str | None = Field( default=None, description="Free-text description of the manufactured substance.", ) contact_email: str | None = Field( default=None, description="Contact email address.", ) raw_phone: str | None = Field( default=None, description="Contact phone number.", ) # ─── Normalization targets ──────────────────────────────────────────── normalized_address: NormalizedAddress | None = Field( default=None, description="Structured address derived from raw_address.", json_schema_extra={ "normalization_model": True, "normalization_source_fields": ["raw_address"], }, ) normalized_substance: NormalizedSubstance | None = Field( default=None, description="Structured substance data derived from raw_substance.", json_schema_extra={ "normalization_model": True, "normalization_source_fields": ["raw_substance"], }, ) ``` ### Step 2: Configure the pipeline ```python from langchain_aws import ChatBedrockConverse from scinr.newton import configure # Main LLM for extraction and annotation main_llm = ChatBedrockConverse( model="us.anthropic.claude-sonnet-4-20250514", region_name="us-east-1", ) # Dedicated (cheaper) LLM for normalization normalize_llm = ChatBedrockConverse( model="us.anthropic.claude-haiku-3", region_name="us-east-1", ) configure( llm=main_llm, normalization_enabled=True, normalization_batch_size=10, normalization_llm=normalize_llm, llm_concurrency=8, ) ``` ### Step 3: Sample CSV input ```csv manufacturer_name,raw_address,raw_substance,contact_email,raw_phone PharmaCorp Inc,"123 Main Street, Springfield, IL 62704, USA","Metformin Hydrochloride 500mg tablets, CAS 1105-50-9",info@pharmacorp.com,+1-555-0100 BioMed Labs,"456 Oak Avenue, Portland, OR 97201, USA","Atorvastatin Calcium 20mg tablets, CAS 134523-03-8",contact@biomedlabs.com,+1-555-0200 EuroPharm GmbH,"Hauptstraße 7, 10115 Berlin, Germany","Ibuprofen 400mg capsules, CAS 15357-78-8",info@europharm.de,+49-30-123456 PharmaCorp Inc,"123 Main Street, Springfield, IL 62704, USA","Lisinopril 10mg tablets, CAS 83915-66-8",info@pharmacorp.com,+1-555-0100 ``` Note that row 4 has the same `raw_address` as row 1 — the normalization engine will deduplicate this and use the cached result. ### Step 4: Pipeline execution ```python import asyncio from scinr.newton.pipeline import run_pipeline async def main(): result = await run_pipeline( input_path="./data/manufacturers.csv", theme="pharma_contacts", ) print(f"Processed {result.total_rows} rows") print(f"Normalized {result.normalization_count} fields") asyncio.run(main()) ``` ### Step 5: What happens internally ``` 1. Column Mapping (LLM) CSV columns → ManufacturerContact fields 2. Model Instantiation 4 rows → 4 ManufacturerContact instances normalized_address = None (all 4) normalized_substance = None (all 4) 3. Normalization Detection get_normalization_specs(ManufacturerContact) → [ NormalizationSpec(field_name="normalized_address", target_type=NormalizedAddress, source_fields=["raw_address"]), NormalizationSpec(field_name="normalized_substance", target_type=NormalizedSubstance, source_fields=["raw_substance"]), ] 4. Entry Collection Row 1: NormalizedAddress entry (key: "NormalizedAddress:abc123") Row 2: NormalizedAddress entry (key: "NormalizedAddress:def456") Row 3: NormalizedAddress entry (key: "NormalizedAddress:ghi789") Row 4: NormalizedAddress entry (key: "NormalizedAddress:abc123") ← DUPLICATE Row 1: NormalizedSubstance entry (key: "NormalizedSubstance:jkl012") Row 2: NormalizedSubstance entry (key: "NormalizedSubstance:mno345") Row 3: NormalizedSubstance entry (key: "NormalizedSubstance:pqr678") Row 4: NormalizedSubstance entry (key: "NormalizedSubstance:stu901") 5. Deduplication NormalizedAddress: 3 unique keys (row 4 shares with row 1) NormalizedSubstance: 4 unique keys 6. Batching (batch_size=10) NormalizedAddress batch: [abc123, def456, ghi789] → 1 LLM call NormalizedSubstance batch: [jkl012, mno345, pqr678, stu901] → 1 LLM call 7. LLM Calls (concurrent, bounded by llm_concurrency=8) Call 1: NormalizedAddress batch → 3 results cached Call 2: NormalizedSubstance batch → 4 results cached 8. Result Application Row 1: normalized_address ← cached[abc123], normalized_substance ← cached[jkl012] Row 2: normalized_address ← cached[def456], normalized_substance ← cached[mno345] Row 3: normalized_address ← cached[ghi789], normalized_substance ← cached[pqr678] Row 4: normalized_address ← cached[abc123], normalized_substance ← cached[stu901] 9. Neo4j Write 4 ManufacturerContact :ModelInstance nodes 4 NormalizedAddress :ModelInstance nodes (3 unique, row 4 shares with row 1) 4 NormalizedSubstance :ModelInstance nodes Country :LabeledEntity nodes (deduplicated by entity_label) ActiveSubstance :LabeledEntity nodes (deduplicated by entity_label) ``` ### Step 6: Neo4j graph result ``` (:StructureNode {name: "manufacturers.csv"}) -[:HAS_EXTRACTION]-> (:ExtractionResult) -[:HAS_CONDITION_IDS]-> (:ModelInstance {model_class: "ManufacturerContact"}) -[:HAS_PROPERTIES {manufacturer_name: "PharmaCorp Inc", ...}] -[:REFERENCES]->(:LabeledEntity {label: "Manufacturer", value: "PharmaCorp Inc"}) -[:HAS_NORMALIZED_ADDRESS]-> (:ModelInstance {model_class: "NormalizedAddress"}) -[:HAS_PROPERTIES {street: "123 Main Street", city: "Springfield", ...}] -[:REFERENCES]->(:LabeledEntity {label: "Country", value: "US"}) -[:HAS_NORMALIZED_SUBSTANCE]-> (:ModelInstance {model_class: "NormalizedSubstance"}) -[:HAS_PROPERTIES {inn_name: "Metformin", strength: "500 mg", ...}] -[:REFERENCES]->(:LabeledEntity {label: "ActiveSubstance", value: "Metformin"}) -[:REFERENCES]->(:LabeledEntity {label: "CasNumber", value: "1105-50-9"}) ``` Row 4 (PharmaCorp Inc, same address) shares the same `NormalizedAddress` node as row 1 via the `country_code` entity_label dedup. The `NormalizedSubstance` nodes are different because the substances differ. --- ## Next steps - **[Advanced Model Design Patterns](model-patterns.md)** — The dual-field pattern, entity labels, instance keys, and more. - **[Custom Models](custom-models.md)** — Basic model definition and field descriptions. - **[Tabular Pipeline](tabular-pipeline.md)** — How the tabular pipeline processes CSV/XLSX files. - **[Neo4j Graph Storage](neo4j-graph.md)** — Understanding the graph model and node types. - **[Architecture](../architecture.md)** — Detailed pipeline stage walkthrough. --- ## File: user-guides/performance-tuning.md # Performance Tuning This guide covers every lever available to tune `scinr.newton` pipeline performance for production deployments. It walks through concurrency, batch sizes, prompt optimization, OCR tuning, scenario-based recommendations, and diagnostic techniques. --- ## Introduction Performance tuning is critical for production deployments of `scinr.newton`. The pipeline processes documents through six stages — preprocess, extraction, ingestion, annotation, entity extraction, and tabular normalization — each with different resource profiles and bottlenecks. ### Three Levers There are three primary levers for performance tuning: 1. **Concurrency** — How many operations run in parallel (LLM calls, Neo4j sessions, document processing). 2. **Batch Sizes** — How many items are grouped per LLM call (extraction pages, normalization entries). 3. **Prompt Optimization** — Reducing token waste via prompt caching and model-specific prompt families. ### The Trade-off Triangle Every tuning decision involves a trade-off between: | Dimension | What it means | How to optimize | | :--- | :--- | :--- | | **Speed** | Wall-clock time to process a batch | Increase concurrency, parallel docs | | **Cost** | Token consumption and API charges | Increase batch sizes, use prompt caching | | **Quality** | Accuracy of extraction and annotation | Lower batch sizes, appropriate prompt families | The default configuration (`llm_concurrency=4`, `parallel_docs=5`, `extraction_batch_size=1`) is a conservative starting point that prioritizes quality and cost. Adjust from there based on your workload. --- ## Concurrency Tuning Concurrency is the most impactful performance lever. The pipeline uses three independent semaphores to control parallelism at different layers. ### LLM Concurrency (`llm_concurrency`) **Default:** `4` **Env var:** `LLM_CONCURRENCY` Controls the maximum number of simultaneous LLM calls across **all** pipeline stages. Every extraction chunk, annotation decision, entity extraction, and tabular normalization call acquires this semaphore before invoking the LLM. ```python from scinr.newton import configure configure( llm_concurrency=4, # Default ) ``` #### How it works The global `get_llm_semaphore()` is sized to `llm_concurrency`. All LLM calls across all stages — extraction, annotation, entity extraction, and tabular normalization — share this single semaphore. This prevents exceeding provider rate limits and connection pool limits. #### Recommendations by workload | Workload | Docs | `llm_concurrency` | Notes | | :--- | :--- | :--- | :--- | | Small | < 10 | 2-4 | Conservative; avoids rate limiting | | Medium | 10-50 | 4-8 | Balanced speed and reliability | | Large | 50+ | 8-16 | Requires checking provider limits | | Very large | 100+ | 16-32 | Monitor rate limits closely | #### Provider rate limits | Provider | Default limit | Notes | | :--- | :--- | :--- | | AWS Bedrock | Varies by model | Check your account's TPMS/RPM limits | | OpenAI | Varies by tier | Free tier: very restrictive; paid: higher | | Ollama | Hardware-bound | Limited by your GPU/CPU capacity | > **Warning:** Setting `llm_concurrency` too high will cause rate-limit errors. Start with the default (4) and increase gradually while monitoring for `429 Too Many Requests` responses. ### Neo4j Concurrency (`neo4j_concurrency`, `neo4j_sync_concurrency`) **Default:** `10` / `8` **Env vars:** `NEO4J_CONCURRENCY` / `NEO4J_SYNC_CONCURRENCY` Two separate semaphores control Neo4j access: ```python configure( neo4j_concurrency=10, # Async operations (Stages 3, 4) neo4j_sync_concurrency=8, # Sync operations (Stage 2) ) ``` #### `neo4j_concurrency` — Async Operations Bounds concurrent Neo4j async sessions during annotation (Stage 3) and entity extraction (Stage 4). These stages use the async Neo4j driver for read-heavy operations. #### `neo4j_sync_concurrency` — Sync Operations Bounds concurrent dispatches to `asyncio.to_thread()` for Stage 2 (synchronous ingestion). The sync driver is used for document ingestion because the Neo4j Python driver's synchronous API handles bulk writes efficiently. The semaphore must be acquired and released on the event loop, never inside the worker thread. #### Recommendations by graph size | Graph size | `neo4j_concurrency` | `neo4j_sync_concurrency` | Notes | | :--- | :--- | :--- | :--- | | Small (< 1k nodes) | 5-10 | 5-8 | Default settings are fine | | Medium (1k-10k) | 10-20 | 8-15 | Monitor Neo4j CPU | | Large (10k-100k) | 20-30 | 15-25 | Watch memory and transaction logs | | Very large (100k+) | 30-50 | 25-50 | Consider Neo4j cluster | > **Tip:** If Neo4j CPU is consistently above 80% during ingestion, reduce `neo4j_sync_concurrency`. If the database has ample capacity and ingestion is slow, increase it. ### Parallel Documents (`parallel_docs`) **Default:** `5` **Set via:** `run_pipeline()` argument Controls the number of documents processed concurrently across all stages. Each document goes through all stages independently, bounded by this semaphore for its entire duration. ```python from scinr.newton import run_pipeline # Process 10 documents concurrently result = await run_pipeline( input_raw="./raw_docs", parallel_docs=10, ) # Process one document at a time (sequential) result = await run_pipeline( input_raw="./raw_docs", parallel_docs=1, ) ``` #### How it works The pipeline uses `asyncio.Semaphore(parallel_docs)` at the document level. Each document is dispatched as an independent task via `asyncio.gather()` and runs through its applicable stages sequentially. Within each stage, additional concurrency control is provided by `llm_concurrency` and `neo4j_concurrency`. #### Recommendations by document profile | Profile | `parallel_docs` | Notes | | :--- | :--- | :--- | | Few large documents | 1-3 | Large docs consume more memory per unit | | Many small documents | 5-10 | Parallelism helps significantly | | Mixed sizes | 3-5 | Balanced approach | | Memory-constrained | 1-2 | Reduce to avoid OOM | > **Important:** `parallel_docs` is set per `run_pipeline()` call, not in `configure()`. This allows you to adjust parallelism per batch without changing global configuration. --- ## Batch Size Tuning Batch sizes control how many items are grouped into a single LLM call. Larger batches reduce the number of API calls (lowering cost) but increase context size per call (potentially affecting quality). ### Extraction Batch Size (`extraction_batch_size`) **Default:** `1` **Env var:** `EXTRACTION_BATCH_SIZE` The number of pages grouped per extraction LLM call. Each call sends the grouped pages to the LLM for structure parsing. ```python configure(extraction_batch_size=1) # Default: 1 page per chunk ``` #### Trade-offs | Value | Pros | Cons | | :--- | :--- | :--- | | `1` (default) | Smallest context, highest quality | More LLM calls, higher cost | | `2-3` | Fewer calls, moderate context | Slightly larger prompts | | `4-5` | Fewest calls, lowest cost | Large context may reduce quality | | `6+` | Maximum cost savings | Risk of context overflow, quality drop | #### Recommendations by document type | Document type | `extraction_batch_size` | Reasoning | | :--- | :--- | :--- | | Dense technical documents | 1 | Each page has significant content; keeping context small preserves quality | | Sparse documents (mostly headers) | 2-3 | Pages are lighter; grouping is efficient | | Very long documents (100+ pages) | 2-3 | Reduces call count significantly | | Documents with complex tables | 1 | Tables need focused context; grouping may lose detail | ### Normalization Batch Size (`normalization_batch_size`) **Default:** `5` **Env var:** `NORMALIZATION_BATCH_SIZE` The number of normalization entries grouped per LLM call in the tabular pipeline. Each call sends multiple table entries to the LLM for structural normalization. ```python configure(normalization_batch_size=5) # Default: 5 entries per call ``` #### Trade-offs | Value | Pros | Cons | | :--- | :--- | :--- | | `3-5` (default) | Balanced quality and cost | Moderate number of calls | | `10-15` | Fewer calls, lower cost | Larger context per call | | `15-20` | Maximum cost savings | Risk of context overflow | | `20+` | Fewest calls | May exceed model context window | #### Recommendations by complexity | Normalization complexity | `normalization_batch_size` | Reasoning | | :--- | :--- | :--- | | Simple (few columns, clean data) | 10-20 | Entries are small; batching is efficient | | Complex (many columns, messy data) | 3-5 | Each entry needs more context | | Cost-sensitive | 10-15 | Good balance of cost and quality | | Quality-critical | 3-5 | Smaller batches = more focused normalization | --- ## Prompt Optimization Prompt optimization reduces token waste and improves LLM response quality through caching and model-specific formatting. ### Prompt Caching (Bedrock) **Default:** `True` **Env var:** `PROMPT_CACHING_ENABLED` Caches system prompts across LLM calls. When enabled, the system prompt (which is identical across calls within a stage) is cached on the Bedrock side, reducing both latency and token costs. ```python configure(prompt_caching_enabled=True) # Default: True ``` #### When it helps | Scenario | Impact | | :--- | :--- | | Many small extraction calls | High — system prompt is a large fraction of total tokens | | Few large extraction calls | Low — system prompt is a small fraction | | Annotation stage | Moderate — repeated system prompt across many nodes | | Entity extraction | Moderate — repeated system prompt across many nodes | #### When to disable - **Non-Bedrock providers:** Prompt caching is a Bedrock-specific feature. It is ignored for OpenAI, Ollama, and other providers. - **Very few LLM calls:** If you process only 1-2 documents with few pages, caching overhead may outweigh benefits. - **Frequently changing prompts:** If you use `context_instructions` that change between runs, caching may be less effective. ```python # Disable prompt caching (rarely needed) configure(prompt_caching_enabled=False) ``` ### Prompt Families **Default:** `generic` **Env var:** `PROMPT_FAMILY` The `prompt_family` parameter selects a set of prompt templates optimized for different LLM providers. ```python configure(prompt_family="generic") # Default ``` | Family | Best for | Characteristics | | :--- | :--- | :--- | | `generic` | Any model | Standard prompt format; safe default that works with any LLM | | `claude` | Anthropic Claude models | Optimized for Claude reasoning; uses Claude-specific formatting and system prompt conventions | | `gpt_reasoning` | OpenAI o-series | Optimized for reasoning models; uses the specific message structure required by reasoning-capable models | #### Choosing a prompt family - **Claude models on Bedrock** — use `"claude"` for best results. - **OpenAI o-series models** — use `"gpt_reasoning"`. - **Other providers or unsure** — use `"generic"` (the default). ```python # Claude on Bedrock configure( prompt_family="claude", prompt_caching_enabled=True, ) # OpenAI o-series configure( prompt_family="gpt_reasoning", ) # Ollama or other local model configure( prompt_family="generic", ) ``` > **Tip:** Using the wrong prompt family with a model can degrade extraction quality significantly. Always match the prompt family to your model. --- ## Mistral OCR Tuning PDF processing uses a two-path strategy: small PDFs are processed with `pdfplumber` (fast, no API cost), while large or complex PDFs use Mistral OCR (slower, API cost). The tuning parameters control this boundary and the OCR behavior itself. ```python configure( mistral_ocr_safe_max_pages=900, # Pages threshold mistral_ocr_safe_max_bytes=47185920, # Size threshold (45 MiB) mistral_ocr_max_retries=3, # Retry count mistral_ocr_retry_backoff_seconds=2.0, # Retry backoff mistral_ocr_chunk_concurrency=1, # OCR chunk concurrency mistral_ocr_error_strategy="best_effort", # Error handling ) ``` ### `mistral_ocr_safe_max_pages` **Default:** `900` **Env var:** `MISTRAL_OCR_SAFE_MAX_PAGES` PDFs with fewer pages than this threshold use `pdfplumber` (fast, no API cost) if they contain extractable text. PDFs at or above this threshold use Mistral OCR regardless. | Value | Effect | | :--- | :--- | | Lower (e.g., 100) | More PDFs use OCR; better quality for scanned docs | | Higher (e.g., 2000) | More PDFs use pdfplumber; faster, cheaper | | Default (900) | Balanced; most PDFs use pdfplumber | ### `mistral_ocr_safe_max_bytes` **Default:** `47185920` (45 MiB) **Env var:** `MISTRAL_OCR_SAFE_MAX_BYTES` PDFs larger than this threshold always use Mistral OCR, regardless of page count. Large files are more likely to be scanned images or have complex layouts that benefit from OCR. | Value | Effect | | :--- | :--- | | Lower (e.g., 10 MiB) | More files use OCR | | Higher (e.g., 100 MiB) | Fewer files use OCR | | Default (45 MiB) | Balanced | ### `mistral_ocr_max_retries` **Default:** `3` **Env var:** `MISTRAL_OCR_MAX_RETRIES` Number of retry attempts for OCR failures. Each retry uses exponential backoff based on `mistral_ocr_retry_backoff_seconds`. ### `mistral_ocr_retry_backoff_seconds` **Default:** `2.0` **Env var:** `MISTRAL_OCR_RETRY_BACKOFF_SECONDS` Base backoff in seconds between retries. Actual backoff is exponential: first retry waits 2s, second waits 4s, third waits 8s. ### `mistral_ocr_chunk_concurrency` **Default:** `1` **Env var:** `MISTRAL_OCR_CHUNK_CONCURRENCY` Number of concurrent OCR chunk processing tasks. A large PDF is split into chunks for parallel OCR processing. | Value | Effect | | :--- | :--- | | `1` (default) | Sequential processing; safest, most predictable | | `2-4` | Parallel chunks; faster for very large PDFs | | `4+` | Maximum parallelism; may hit Mistral rate limits | > **Warning:** Increasing `mistral_ocr_chunk_concurrency` increases API calls proportionally. Monitor Mistral rate limits. ### `mistral_ocr_error_strategy` **Default:** `"fail_fast"` **Env var:** `MISTRAL_OCR_ERROR_STRATEGY` Controls error handling during OCR processing. | Value | Behavior | | :--- | :--- | | `"fail_fast"` | Stops processing the document on first OCR error | | `"best_effort"` | Continues processing and collects whatever OCR results are available | For production pipelines processing many documents, `"best_effort"` is recommended to avoid a single OCR failure blocking the entire batch. --- ## Scenario-Based Recommendations ### Small Scale (1-10 documents) Lightweight configuration for development, testing, or small document sets. ```python from scinr.newton import configure, run_pipeline configure( llm_concurrency=2, # Conservative LLM calls neo4j_concurrency=5, # Light Neo4j load neo4j_sync_concurrency=5, # Light sync load extraction_batch_size=1, # Maximum quality normalization_batch_size=5, # Default prompt_caching_enabled=True, # Still useful for small batches ) result = await run_pipeline( input_raw="./docs", parallel_docs=2, # Low document parallelism ) ``` **Expected profile:** - Speed: Moderate (quality-focused) - Cost: Higher per document (small batches) - Quality: Maximum (small context per call) ### Medium Scale (10-100 documents) Balanced configuration for typical production workloads. ```python configure( llm_concurrency=8, # More parallel LLM calls neo4j_concurrency=15, # Moderate Neo4j load neo4j_sync_concurrency=12, # Moderate sync load extraction_batch_size=2, # Slightly larger batches normalization_batch_size=10, # Larger normalization batches prompt_caching_enabled=True, # Significant savings at this scale ) result = await run_pipeline( input_raw="./docs", parallel_docs=5, # Default parallelism ) ``` **Expected profile:** - Speed: Good balance - Cost: Moderate (larger batches reduce calls) - Quality: High (still reasonable context sizes) ### Large Scale (100+ documents) Maximum throughput configuration for large document sets. ```python configure( llm_concurrency=16, # High parallel LLM calls neo4j_concurrency=25, # High Neo4j throughput neo4j_sync_concurrency=20, # High sync throughput extraction_batch_size=2, # Larger extraction batches normalization_batch_size=15, # Large normalization batches prompt_caching_enabled=True, # Critical for cost at this scale ) result = await run_pipeline( input_raw="./docs", parallel_docs=10, # High document parallelism ) ``` **Expected profile:** - Speed: Maximum (high concurrency everywhere) - Cost: Lower per document (large batches, prompt caching) - Quality: Good (slightly larger contexts) > **Warning:** At large scale, monitor Neo4j CPU, memory, and transaction logs. You may need to tune Neo4j database settings independently. ### Cost-Sensitive Minimize API costs while maintaining acceptable quality. ```python configure( llm_concurrency=2, # Fewer parallel calls (no rush) extraction_batch_size=3, # More pages per call normalization_batch_size=15, # Larger normalization batches prompt_caching_enabled=True, # Cache prompts (Bedrock) mistral_ocr_safe_max_pages=2000, # Prefer pdfplumber over OCR mistral_ocr_safe_max_bytes=104857600, # 100 MiB threshold ) result = await run_pipeline( input_raw="./docs", parallel_docs=3, # Moderate parallelism ) ``` **Cost-saving strategies:** - Larger extraction batches = fewer LLM calls per document - Larger normalization batches = fewer LLM calls per table - Higher OCR thresholds = more pdfplumber usage (free) - Prompt caching = reduced token costs on Bedrock - Lower concurrency = no wasted retries from rate limiting ### Speed-Sensitive Minimize wall-clock time while maintaining acceptable cost. ```python configure( llm_concurrency=16, # Max parallel LLM calls neo4j_concurrency=30, # Max Neo4j throughput neo4j_sync_concurrency=25, # Max sync throughput extraction_batch_size=1, # Smaller batches = faster per call normalization_batch_size=5, # Default batch size mistral_ocr_chunk_concurrency=4, # Parallel OCR mistral_ocr_error_strategy="best_effort", # Don't wait on failures ) result = await run_pipeline( input_raw="./docs", parallel_docs=10, # Max document parallelism ) ``` **Speed-optimization strategies:** - Maximum concurrency at every layer - Smaller extraction batches = faster individual LLM calls - Parallel OCR chunks = faster PDF processing - `best_effort` error strategy = no blocking on failures - Higher `parallel_docs` = more documents in flight --- ## Monitoring and Diagnostics ### Pipeline Result Inspection `run_pipeline()` returns a `PipelineResult` with structured per-stage metrics. Use these to identify bottlenecks. ```python result = await run_pipeline(input_raw="./docs") # Overall timing print(f"Total: {result.total_duration_seconds:.1f}s") # Per-stage timing for stage_name in result.stages_executed: stage = getattr(result, stage_name) if stage: print(f"{stage_name}: {stage.duration_seconds:.1f}s, " f"{stage.total_processed} processed, {stage.total_failed} failed") ``` ### Detailed Per-Stage Breakdown ```python result = await run_pipeline(input_raw="./docs") # Calculate stage percentages total = result.total_duration_seconds for stage_name in result.stages_executed: stage = getattr(result, stage_name) if stage and total > 0: pct = (stage.duration_seconds / total) * 100 print(f"{stage_name:>20s}: {stage.duration_seconds:6.1f}s ({pct:5.1f}%)") ``` ### Bottleneck Identification | Symptom | Likely Bottleneck | Tuning Action | | :--- | :--- | :--- | | Stage 0 (preprocess) is the slowest stage | PDF conversion or OCR bottleneck | Increase `mistral_ocr_chunk_concurrency`, check Mistral API | | Stage 1 (extraction) is the slowest stage | LLM concurrency too low | Increase `llm_concurrency` | | Stage 2 (ingestion) is the slowest stage | Neo4j sync concurrency too low | Increase `neo4j_sync_concurrency` | | Stage 3 (annotation) is the slowest stage | LLM concurrency too low | Increase `llm_concurrency` | | Stage 4 (entity extraction) is the slowest stage | LLM concurrency too low | Increase `llm_concurrency` | | High LLM costs per document | Batch sizes too small | Increase `extraction_batch_size` | | OCR timeout errors | Large PDFs, low concurrency | Increase `mistral_ocr_chunk_concurrency` | | Memory errors (OOM) | Too many parallel documents | Decrease `parallel_docs` | | Neo4j connection pool exhausted | Neo4j concurrency too high | Decrease `neo4j_concurrency` | | Rate limit errors (429) | LLM concurrency too high | Decrease `llm_concurrency` | | Inconsistent extraction quality | Batch size too large | Decrease `extraction_batch_size` | ### Finding the Dominant Stage The stage that consumes the most time is your primary bottleneck. Focus tuning efforts there: ```python result = await run_pipeline(input_raw="./docs") # Find the slowest stage slowest = max( (getattr(result, name) for name in result.stages_executed), key=lambda s: s.duration_seconds if s else 0 ) print(f"Bottleneck: {slowest.stage} ({slowest.duration_seconds:.1f}s)") ``` ### Monitoring Concurrency Saturation If a stage is slow but you suspect concurrency is the issue, check if you're hitting semaphore limits: ```python from scinr.newton import get_config config = get_config() print(f"LLM semaphore size: {config.llm_concurrency}") print(f"Neo4j async semaphore: {config.neo4j_concurrency}") print(f"Neo4j sync semaphore: {config.neo4j_sync_concurrency}") ``` If the LLM stage is slow and `llm_concurrency` is at its maximum, try increasing it. If Neo4j stages are slow and the semaphore is maxed, try increasing the relevant Neo4j concurrency. --- ## Provider-Specific Tuning ### AWS Bedrock Bedrock is the primary supported provider with the most tuning options. ```python configure( # Prompt caching — critical for cost at scale prompt_caching_enabled=True, # Use a cheaper model for repair/retry operations # repair_llm=ChatBedrockConverse(model="us.anthropic.claude-haiku-3"), # Set appropriate MAX_TOKENS for your model # (via environment variable: MAX_TOKENS=65536) # Prompt family for Claude models prompt_family="claude", # Concurrency tuned for Bedrock rate limits llm_concurrency=8, ) ``` **Bedrock-specific tips:** - **Prompt caching** is the single biggest cost reducer. Always keep it enabled. - **REPAIR_MODEL_ID** can be set to a cheaper/faster model (e.g., Claude Haiku) for JSON repair operations, saving cost on retries. - **MAX_TOKENS** should be set appropriately for your model. Too high wastes tokens; too low truncates responses. - Check your account's **TPMS** (Tokens Per Minute) and **RPM** (Requests Per Minute) limits in the AWS Console. ### OpenAI OpenAI has different rate limit characteristics and prompt requirements. ```python from langchain_openai import ChatOpenAI from scinr.newton import configure configure( llm=ChatOpenAI(model="gpt-4o"), # Use gpt_reasoning family for o-series models prompt_family="gpt_reasoning", # for o-series # prompt_family="generic", # for gpt-4o, etc. # OpenAI rate limits vary by tier llm_concurrency=4, # Start conservative ) ``` **OpenAI-specific tips:** - **Rate limits** vary significantly by tier. Free tier is very restrictive; paid tiers are more permissive. Check your dashboard. - Use `gpt_reasoning` prompt family for o-series models (o1, o3, etc.). These models require a specific message structure. - Use `generic` prompt family for standard models (gpt-4o, gpt-4o-mini, etc.). - Prompt caching is not available; consider larger batch sizes to compensate. ### Ollama (Local Models) Local models have no rate limits but are hardware-bound. ```python from langchain_ollama import ChatOllama from scinr.newton import configure configure( llm=ChatOllama(model="llama3"), # No rate limits, but hardware-bound llm_concurrency=2, # Start low; increase if GPU has headroom # Generic prompt family for local models prompt_family="generic", # Prompt caching not applicable prompt_caching_enabled=False, ) ``` **Ollama-specific tips:** - **No rate limits** — you can set `llm_concurrency` as high as your hardware allows. - **GPU-bound** — each concurrent LLM call consumes GPU memory. Start with `llm_concurrency=2` and increase if you have headroom. - **CPU-bound** — if running on CPU, keep `llm_concurrency=1` to avoid context-switching overhead. - **Model size matters** — larger models (70B+) may only support 1 concurrent call on consumer hardware. - Prompt caching is not applicable for local models. --- ## Advanced Tuning Patterns ### Two-Tier Configuration Use different concurrency settings for different pipeline phases by reconfiguring between runs: ```python from scinr.newton import configure, run_pipeline # Phase 1: Preprocess + Extraction (LLM-heavy) configure( llm_concurrency=16, # Max LLM calls for extraction neo4j_concurrency=5, # Minimal Neo4j (not used yet) ) result1 = await run_pipeline( input_raw="./docs", extraction_output_dir="./data/extracted/", stages=["preprocess", "extraction"], parallel_docs=10, ) # Phase 2: Ingestion (Neo4j-heavy) configure( llm_concurrency=2, # Minimal LLM (not used) neo4j_concurrency=25, # Max Neo4j for ingestion neo4j_sync_concurrency=20, ) result2 = await run_pipeline( ingestion_input_dir="./data/extracted/", stages=["ingestion"], parallel_docs=10, ) # Phase 3: Annotation + Extraction (LLM-heavy again) configure( llm_concurrency=16, # Max LLM calls neo4j_concurrency=15, # Moderate Neo4j ) result3 = await run_pipeline( stages=["annotation", "entity_extraction"], document_names_dir="./data/extracted/", parallel_docs=10, ) ``` ### Adaptive Concurrency Dynamically adjust concurrency based on pipeline feedback: ```python import asyncio from scinr.newton import configure, run_pipeline async def adaptive_run(input_dir: str, max_docs: int = 100) -> None: # Start conservative configure( llm_concurrency=4, neo4j_concurrency=10, ) # Process a small batch first result = await run_pipeline( input_raw=input_dir, parallel_docs=3, ) # Analyze results for stage_name in result.stages_executed: stage = getattr(result, stage_name) if stage and stage.total_failed == 0: # No failures — safe to increase concurrency pass elif stage and stage.total_failed > 0: # Failures detected — keep conservative settings print(f"Warning: {stage_name} had failures. " f"Keeping conservative concurrency.") return # If first batch was clean, increase concurrency for the main run configure( llm_concurrency=12, neo4j_concurrency=20, ) result = await run_pipeline( input_raw=input_dir, parallel_docs=8, ) print(f"Adaptive run complete: {result.success}") ``` ### Batch Processing with Progress Tracking For very large document sets, process in batches with progress reporting: ```python import asyncio from pathlib import Path from scinr.newton import configure, run_pipeline async def batch_run(input_dir: str, batch_size: int = 20) -> None: configure( llm_concurrency=8, neo4j_concurrency=15, parallel_docs=5, ) files = list(Path(input_dir).rglob("*")) files = [f for f in files if f.is_file()] total = len(files) processed = 0 for i in range(0, total, batch_size): batch = files[i:i + batch_size] batch_dir = f"./temp_batch_{i // batch_size}" Path(batch_dir).mkdir(exist_ok=True) # Copy batch files to temp directory for f in batch: import shutil shutil.copy(f, batch_dir) result = await run_pipeline( input_raw=batch_dir, parallel_docs=5, ) processed += len(batch) print(f"Batch {i // batch_size + 1}: " f"{processed}/{total} files " f"({result.total_duration_seconds:.1f}s)") # Clean up temp directory import shutil shutil.rmtree(batch_dir) ``` --- ## See Also - **[Configuration](../configuration.md)** — Complete reference for `configure()`, environment variables, and all settings. - **[Running the Pipeline](running-pipeline.md)** — Full `run_pipeline()` reference including `parallel_docs` and other parameters. - **[Architecture](../architecture.md)** — Detailed walkthrough of concurrency layers, semaphores, and async design. - **[Tabular Pipeline](tabular-pipeline.md)** — Tabular normalization performance and `normalization_batch_size` tuning. - **[Custom Models](custom-models.md)** — Defining extraction models that affect annotation and extraction stage performance. - **[Pipeline API](../api/pipeline.md)** — Auto-generated docstring for `run_pipeline()`. - **[Results API](../api/results.md)** — Auto-generated documentation for `PipelineResult`, `StageResult`, and `DocumentResult`. --- ## File: user-guides/quick-start.md # Quick Start — Your First Knowledge Graph Get from zero to a working knowledge graph in under 15 minutes. This guide walks you through installing `scinr`, configuring your environment, preparing documents, running the full 6-stage ingestion pipeline, and verifying the extracted knowledge graph in Neo4j. --- ## Prerequisites Before you begin, make sure you have: 1. **Python 3.11+** installed on your system. 2. **Neo4j 5.0+** running and accessible via the Bolt protocol. If you do not have Neo4j installed, start one locally with Docker: ```bash docker run -p 7687:7687 -p 7474:7474 \ -e NEO4J_AUTH=neo4j/your_password \ neo4j:5 ``` 3. **LLM credentials** for at least one provider (AWS Bedrock, OpenAI, or Ollama). --- ## Step 1: Install scinr Install `scinr` with `pip`. Choose the extras that match your LLM provider: ```bash # AWS Bedrock (recommended — includes langchain-aws and boto3) pip install "scinr[bedrock]" # Or with OpenAI (includes langchain-openai) pip install "scinr[openai]" # Or with Ollama (includes langchain-ollama) pip install "scinr[ollama]" # Or with all extras at once pip install "scinr[bedrock,openai,ollama,mongodb]" ``` > **Tip:** If you plan to process PDF files with OCR, you will also need a [Mistral AI](https://console.mistral.ai/) API key. Without it, text-based PDFs are still processed via `pdfplumber`. --- ## Step 2: Set up your environment `scinr` reads configuration from environment variables. The recommended approach is to create a `.env` file in your working directory. ### Create `.env` from the template If you cloned the `scinr` repository, copy the example template: ```bash cp .env.example .env ``` ### Fill in the required values At a minimum, you need to set the LLM model identifier and Neo4j credentials. Here is a minimal `.env` for a first run: ```env # ─── LLM (AWS Bedrock) ────────────────────────────────────────────────────── MODEL_ID=us.anthropic.claude-sonnet-4-6 # ─── Neo4j ────────────────────────────────────────────────────────────────── NEO4J_URI=bolt://localhost:7687 NEO4J_USER=neo4j NEO4J_PASSWORD=your_password ``` ### Required vs. optional fields | Variable | Required? | What to set | | :--- | :--- | :--- | | `MODEL_ID` | Yes (unless you pass `llm=` to `configure()`) | Your LLM model ID. For Bedrock: `us.anthropic.claude-sonnet-4-6`. | | `NEO4J_URI` | No | Neo4j Bolt URI. Default: `bolt://localhost:7687`. | | `NEO4J_USER` | Yes | Neo4j username (usually `neo4j`). | | `NEO4J_PASSWORD` | Yes | Neo4j password. | | `MISTRAL_API_KEY` | No | Mistral API key for PDF OCR. | | `STORAGE_BACKEND` | No | `none` (default, in-memory) or `mongodb`. | > **Note:** `python-dotenv` is included as a core dependency. When you call `configure()`, it automatically loads variables from a `.env` file in your current working directory. You do not need to import `dotenv` manually. --- ## Step 3: Prepare your documents Create a directory and place your source documents in it: ```bash mkdir -p raw_docs # Place your files here ``` ### Supported file formats | Format | Extensions | Notes | | :--- | :--- | :--- | | PDF | `.pdf` | Text-based via `pdfplumber`; OCR via Mistral API | | Word | `.docx` | Full text + structure extraction | | Excel | `.xlsx`, `.xls` | Auto-routed to tabular pipeline | | PowerPoint | `.pptx` | Slide text extraction | | CSV | `.csv` | Auto-routed to tabular pipeline | | HTML | `.html`, `.htm` | Cleaned and parsed | | JSON | `.json` | API responses, structured data | | Text | `.txt`, `.md`, `.rst` | Plain text | > **Warning:** Tabular files (`.csv`, `.xlsx`, `.xls`) are automatically routed to the tabular pipeline and bypass the standard 5-stage extraction flow. This is intentional — spreadsheets have a fundamentally different structure than narrative documents. --- ## Step 4: Run the pipeline Create a Python script to configure `scinr` and run the full pipeline: ```python # quickstart.py import asyncio from scinr.newton import configure, run_pipeline async def main(): # configure() reads .env automatically via python-dotenv. # It resolves LLM, Neo4j, and storage settings from: # 1. Explicit arguments (highest priority) # 2. Environment variables / .env file # 3. Hard-coded defaults configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="your_password", ) # Run the full 6-stage pipeline result = await run_pipeline(input_raw="./raw_docs") # Check overall result print(f"Pipeline success: {result.success}") print(f"Total duration: {result.total_duration_seconds:.1f}s") print(f"Stages executed: {result.stages_executed}") # Inspect individual stages for stage_name in result.stages_executed: stage = getattr(result, stage_name, None) if stage is not None: status = "OK" if stage.success else "FAIL" print( f" [{status}] {stage_name}: " f"{stage.total_processed} processed, " f"{stage.total_failed} failed " f"({stage.duration_seconds:.1f}s)" ) # Show per-document details for doc in stage.documents: if doc.errors: for err in doc.errors: print(f" - {doc.document_name}: {err}") asyncio.run(main()) ``` Run the script: ```bash python quickstart.py ``` ### What happens behind the scenes The pipeline runs these stages in order: | Stage | Name | What it does | | :--- | :--- | :--- | | 0 | **Preprocess** | Converts raw files to an intermediate JSON/Markdown format | | 1 | **Extraction** | Uses the LLM to parse document structure and extract hierarchical sections | | 2 | **Ingestion** | Writes document and structure nodes into Neo4j | | 3 | **Annotation** | An LLM agent assigns an extraction model to each structural node | | 4 | **Entity Extraction** | Extracts typed Pydantic entities from annotated nodes and writes them as graph subgraphs | | 5 | **Tabular** | If `.csv`, `.xlsx`, or `.xls` files are detected in `input_raw`, they are processed through a separate tabular pipeline with LLM-powered normalization | ### Expected output A successful run produces output similar to: ``` Pipeline success: True Total duration: 45.2s Stages executed: ['preprocess', 'extraction', 'ingestion', 'annotation', 'entity_extraction'] [OK] preprocess: 3 processed, 0 failed (2.1s) [OK] extraction: 3 processed, 0 failed (18.5s) [OK] ingestion: 3 processed, 0 failed (4.3s) [OK] annotation: 47 processed, 0 failed (12.8s) [OK] entity_extraction: 47 processed, 2 failed (8.5s) ``` > **Tip:** If you see failures in entity extraction, this is often expected — not every structural node contains extractable domain entities. The pipeline continues gracefully. Check the per-document errors for details. --- ## Step 5: Verify in Neo4j After the pipeline completes, your data is available in Neo4j. Use the Neo4j Browser (`http://localhost:7474`) or any Cypher client to inspect the results. ### List all ingested documents ```cypher MATCH (d:Document) RETURN d.document_name, d.format, d.ingested_at, d.version ORDER BY d.ingested_at DESC; ``` ### View document structure ```cypher MATCH (d:Document)-[:HAS_STRUCTURE]->(s:StructureNode) RETURN d.document_name AS document, s.section_title AS section, s.node_type LIMIT 20; ``` ### View the full hierarchy ```cypher MATCH (d:Document)-[:HAS_STRUCTURE]->(s:StructureNode)-[:HAS_CHILD]->(c:StructureNode) RETURN d.document_name AS document, s.section_title AS parent, c.section_title AS child, c.node_type AS child_type LIMIT 30; ``` ### View extracted entity types ```cypher MATCH (m:ModelInstance) RETURN labels(m) AS type, count(m) AS count ORDER BY count DESC; ``` ### View labeled entities (globally deduplicated) ```cypher MATCH (e:LabeledEntity) RETURN head(labels(e)[1..]) AS entity_type, e.value, count(e) AS occurrences ORDER BY occurrences DESC LIMIT 20; ``` ### View entity relationships ```cypher MATCH (a:ModelInstance)-[r]->(b:ModelInstance) RETURN type(r) AS relationship, count(r) AS count ORDER BY count DESC; ``` ### Explore a specific document's extraction results ```cypher MATCH (d:Document)-[:HAS_STRUCTURE*0..]->(s:StructureNode)<-[:EXTRACTED_FROM]-(m:ModelInstance) WHERE d.document_name = 'YourDocumentName' RETURN s.section_title AS section, head(labels(m)[1..]) AS model_type, count(m) AS entities ORDER BY entities DESC LIMIT 20; ``` ### Visualize the graph In Neo4j Browser, run this query to get a visual overview: ```cypher MATCH path = (d:Document)-[:HAS_STRUCTURE*1..3]->(s:StructureNode) WHERE d.document_name = 'YourDocumentName' RETURN path LIMIT 50; ``` --- ## Step 6: Next steps Now that you have a working pipeline, explore the rest of the documentation: - **[Configuration](../configuration.md)** — Full configuration reference: all environment variables, `configure()` parameters, prompt families, concurrency tuning, and advanced settings. - **[Custom Models](custom-models.md)** — Define your own Pydantic extraction schemas for domain-specific entities. - **[Architecture](../architecture.md)** — Detailed walkthrough of each pipeline stage, data flow between stages, and system design decisions. - **[Tabular Pipeline](tabular-pipeline.md)** — Working with CSV, XLSX, and spreadsheet data. - **[Neo4j Graph Storage](neo4j-graph.md)** — Understanding the graph model and node/relationship types. --- ## Common variations ### Using OpenAI instead of Bedrock ```python from langchain_openai import ChatOpenAI from scinr.newton import configure, run_pipeline configure( llm=ChatOpenAI(model="gpt-4o"), neo4j_user="neo4j", neo4j_password="your_password", ) ``` Set `OPENAI_API_KEY` in your `.env` file. ### Using Ollama (local models) ```python from langchain_ollama import ChatOllama from scinr.newton import configure, run_pipeline configure( llm=ChatOllama(model="llama3"), neo4j_user="neo4j", neo4j_password="your_password", ) ``` Make sure Ollama is running locally (`ollama serve`) and the model is pulled (`ollama pull llama3`). ### Running only specific stages You can skip stages by providing input from a later point in the pipeline: ```python # Run only annotation and entity extraction on an already-ingested document result = await run_pipeline( stages=["annotation", "entity_extraction"], document_names=["MyDocument"], ) ``` ### Processing a single document ```python # Only process files matching a specific name pattern result = await run_pipeline( input_raw="./raw_docs", document_names=["ClinicalTrialReport"], ) ``` ### Enabling MongoDB storage Persist raw files and converted pages to MongoDB: ```python configure( neo4j_user="neo4j", neo4j_password="your_password", storage_backend="mongodb", mongodb_uri="mongodb://localhost:27017", mongodb_database="scinr", ) ``` --- ## Troubleshooting ### `"No LLM configured"` You must either: - Set `MODEL_ID` in your `.env` file (for AWS Bedrock), or - Pass an `llm=` argument to `configure()` with a LangChain `BaseChatModel` instance. ### `"Neo4j username/password is not configured"` Set `NEO4J_USER` and `NEO4J_PASSWORD` in your `.env` file, or pass them as arguments to `configure()`. ### Neo4j connection refused Make sure your Neo4j instance is running and accessible: ```bash # Test Bolt connectivity python -c " from neo4j import GraphDatabase driver = GraphDatabase.driver('bolt://localhost:7687', auth=('neo4j', 'your_password')) driver.verify_connectivity() driver.close() print('Neo4j connection OK') " ``` ### `"No documents discovered for this run"` Check that: - The `input_raw` directory exists and contains supported file types. - File extensions are recognized (`.pdf`, `.docx`, `.xlsx`, `.csv`, `.pptx`, `.html`, `.json`, `.txt`, `.md`). - The path is correct — relative paths are resolved from your current working directory. ### ImportError: `langchain-aws is not installed` If you set `MODEL_ID` but have not installed the Bedrock extra: ```bash pip install "scinr[bedrock]" ``` ### PDFs fail to process PDF processing requires either: - A Mistral API key (for OCR) set via `MISTRAL_API_KEY` in your `.env`, or - The PDF must contain extractable text (processed via `pdfplumber` without OCR). If you see OCR-related errors and do not have a Mistral key, try text-based PDFs or set `MISTRAL_API_KEY`. ### LLM calls are slow or rate-limited Adjust concurrency in your `.env` or via `configure()`: ```python configure( llm=my_llm, neo4j_user="neo4j", neo4j_password="your_password", llm_concurrency=2, # Reduce concurrent LLM calls neo4j_concurrency=5, # Reduce concurrent Neo4j writes ) ``` ### Annotation stage returns no model matches This is expected for documents that do not contain content matching any registered extraction model. The pipeline falls back to generic triple extraction for unmatched nodes. To get specific entity extraction, define custom models matching your domain — see [Custom Models](custom-models.md). --- ## File: user-guides/running-pipeline.md # Running the Pipeline This is the definitive reference for `run_pipeline()` — the single entry point that orchestrates the full `scinr.newton` ingestion pipeline. Every parameter, option, and workflow pattern is documented here. --- ## Quick Start The simplest possible pipeline run — convert raw files, extract structure, ingest to Neo4j, annotate, and extract entities: ```python import asyncio from scinr.newton import configure, run_pipeline async def main(): configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="your_password", ) result = await run_pipeline(input_raw="./raw_docs") print(f"Success: {result.success}") print(f"Duration: {result.total_duration_seconds:.2f}s") print(f"Stages: {result.stages_executed}") asyncio.run(main()) ``` `configure()` automatically reads `.env` via `python-dotenv`, so if your environment variables are set, you can call `configure()` with no arguments and it works. --- ## Pipeline Stages The pipeline consists of six named stages. The default run executes Stages 0-4 in order; Stage 5 (`"tabular"`) is auto-detected from file extensions in `input_raw` and runs alongside the main pipeline when tabular files are present. | Stage | Name | Description | | :--- | :--- | :--- | | 0 | `"preprocess"` | Convert raw files (PDF, DOCX, PPTX, etc.) to intermediate JSON/Markdown | | 1 | `"extraction"` | Parse document structure into hierarchical sections via LLM | | 2 | `"ingestion"` | Write `:Document` and `:StructureNode` nodes into Neo4j | | 3 | `"annotation"` | LLM agent assigns extraction models to each structure node | | 4 | `"entity_extraction"` | Extract typed Pydantic entities and write graph subgraphs | | 5 | `"tabular"` | Process CSV/XLSX/XLS with normalization and table understanding | > **Note:** The `"tabular"` stage cannot be combined with other stages in a `stages=` list. When you set `stages=["tabular"]`, it runs exclusively. When you omit `"tabular"` from `stages` (the default), tabular files in `input_raw` are auto-detected and processed automatically alongside the main pipeline. --- ## Full Signature ```python async def run_pipeline( # ── Raw input (Stage 0 source) ──────────────────────────────────────────── input_raw: str | None = None, # ── Directory params — control data flow and stage skipping ────────────── converter_output_dir: str | None = None, extraction_input_dir: str | None = None, extraction_output_dir: str | None = None, ingestion_input_dir: str | None = None, # ── Stage selection ─────────────────────────────────────────────────────── stages: list[str] | None = None, # ── Document identity for annotation / entity_extraction only runs ──────── document_names: list[str] | None = None, document_names_dir: str | None = None, # ── Annotation options ──────────────────────────────────────────────────── manual: bool = False, model_class: str | None = None, only_unannotated: bool = False, only_unextracted: bool = False, context_instructions: str | None = None, # ── Versioning / replacement ────────────────────────────────────────────── update_mode: bool = False, replaces: str | None = None, # ── Parallelism ─────────────────────────────────────────────────────────── parallel_docs: int = 5, # ── Behaviour on partial failure ───────────────────────────────────────── on_partial_failure: Literal["abort", "continue", "warn"] = "warn", # ── Tabular options (auto-detected from input_raw) ──────────────────────── tabular_extensions: set[str] | None = None, tabular_delimiter: str | None = None, ) -> PipelineResult ``` --- ## Parameter Reference ### `input_raw` — Raw Input Directory **Type:** `str | None` **Default:** `None` Path to a folder containing raw source files (PDF, DOCX, PPTX, XLSX, CSV, HTML, JSON, TXT, MD). This activates Stage 0 (`"preprocess"`), which converts all supported files into an intermediate representation. ```python result = await run_pipeline(input_raw="./raw_docs") ``` Tabular files (`.csv`, `.xlsx`, `.xls`) found in this directory are automatically routed to the tabular pipeline in addition to the standard Stages 0-4. See [Tabular Options](#tabular_extensions-and-tabular_delimiter-tabular-options) for customizing this behavior. > **Mutual exclusion:** `input_raw` cannot be used together with `extraction_input_dir` or `ingestion_input_dir`. These parameters represent different entry points into the pipeline. ### Directory Parameters — Intermediate Data Flow These four parameters control how data flows between stages and allow you to skip stages by providing pre-computed intermediate files. #### `converter_output_dir` **Type:** `str | None` **Default:** `None` (in-memory only) Folder where Stage 0 (`"preprocess"`) writes intermediate JSON files to disk. When `None`, converted documents are kept in memory only and a temporary directory is used internally (cleaned up automatically). Set this when you want to persist the Stage 0 output for reuse in a later run: ```python # First run: convert and persist intermediate JSON result = await run_pipeline( input_raw="./raw_docs", converter_output_dir="./data/converted/", stages=["preprocess"], ) # Later run: skip Stage 0, read from persisted JSON result = await run_pipeline( converter_output_dir="./data/converted/", stages=["extraction", "ingestion"], ) ``` #### `extraction_input_dir` **Type:** `str | None` **Default:** `None` Folder where Stage 1 (`"extraction"`) reads JSON input from disk, **skipping Stage 0 entirely**. When provided, the pipeline starts directly at extraction using the files found in this directory. > **Precedence:** `extraction_input_dir` takes absolute priority over `document_names` and `document_names_dir` for document discovery. If both are provided, `document_names` / `document_names_dir` are silently ignored. This is intentional — the directory contents define the document set. ```python # Skip Stage 0; start extraction from pre-converted JSON result = await run_pipeline( extraction_input_dir="./data/converted/", stages=["extraction", "ingestion", "annotation", "entity_extraction"], ) ``` #### `extraction_output_dir` **Type:** `str | None` **Default:** `None` (in-memory only) Folder where Stage 1 (`"extraction"`) writes `extract-*.json` output files. When `None`, extracted documents are kept in memory only. Useful for persisting Stage 1 output so Stage 2 can be run independently later: ```python # First run: extract and persist result = await run_pipeline( input_raw="./raw_docs", extraction_output_dir="./data/extracted/", stages=["preprocess", "extraction"], ) # Later run: ingest from persisted extraction output result = await run_pipeline( extraction_output_dir="./data/extracted/", stages=["ingestion"], ) ``` #### `ingestion_input_dir` **Type:** `str | None` **Default:** `None` Folder where Stage 2 (`"ingestion"`) reads `extract-*.json` files from disk, **skipping both Stages 0 and 1**. When provided, the pipeline starts directly at ingestion. > **Precedence:** Same absolute-priority rule as `extraction_input_dir`. Takes precedence over `document_names` / `document_names_dir` regardless of which stages are requested. ```python # Skip Stages 0 and 1; start directly at ingestion result = await run_pipeline( ingestion_input_dir="./data/extracted/", stages=["ingestion", "annotation", "entity_extraction"], ) ``` #### Directory Parameter Precedence Summary The directory parameters are mutually exclusive with each other and with `input_raw`: | Parameter | Skips | Input for | | :--- | :--- | :--- | | `input_raw` | *(none)* | Stage 0 | | `extraction_input_dir` | Stage 0 | Stage 1 | | `ingestion_input_dir` | Stages 0, 1 | Stage 2 | You can combine `converter_output_dir` and `extraction_output_dir` with other parameters to control where intermediate data is written: ```python # Full pipeline with all intermediate data persisted to disk result = await run_pipeline( input_raw="./raw_docs", converter_output_dir="./data/converted/", extraction_output_dir="./data/extracted/", ) ``` ### `stages` — Stage Selection **Type:** `list[str] | None` **Default:** `["preprocess", "extraction", "ingestion", "annotation", "entity_extraction"]` Ordered list of stage names to execute. Omitting stages skips them entirely. The default runs Stages 0-4. ```python # Only preprocess + extraction (Stages 0-1) result = await run_pipeline( input_raw="./raw_docs", stages=["preprocess", "extraction"], ) # Only annotation + entity extraction (Stages 3-4) result = await run_pipeline( stages=["annotation", "entity_extraction"], document_names=["my_document.pdf"], ) # Tabular-only (Stage 5) result = await run_pipeline( input_raw="./data", stages=["tabular"], ) ``` > **Important:** `"tabular"` cannot be combined with other stages. Use `stages=["tabular"]` alone, or omit `"tabular"` from `stages` and let the pipeline auto-detect tabular files from `input_raw`. When running annotation or entity extraction without ingestion, you must provide document names via `document_names` or `document_names_dir` (see [Document Selection](#document_names-and-document_names_dir-document-selection)). ### `document_names` and `document_names_dir` — Document Selection **Type:** `list[str] | None` / `str | None` **Default:** `None` These parameters select which documents to process when running annotation (`"annotation"`) or entity extraction (`"entity_extraction"`) without running ingestion first. #### `document_names` An explicit list of Neo4j `document_name` values. The pipeline looks up each document in Neo4j and processes only those documents. ```python # Run annotation on specific documents result = await run_pipeline( stages=["annotation", "entity_extraction"], document_names=["Clinical_Trial_Report_2024", "Safety_Summary_Q3"], ) ``` #### `document_names_dir` A directory containing `extract-*.json` files. The pipeline extracts document names from these files and processes the corresponding documents in Neo4j. ```python # Derive document names from extraction JSON files result = await run_pipeline( stages=["annotation"], document_names_dir="./data/extracted/", ) ``` > **Mutual exclusion:** `document_names` and `document_names_dir` cannot both be provided. Use one or the other. > > **Precedence:** Both are silently ignored if `extraction_input_dir` or `ingestion_input_dir` is also provided — the directory parameters take absolute priority for document discovery. ### `manual` and `model_class` — Manual Annotation Mode **Type:** `bool` / `str | None` **Default:** `False` / `None` When `manual=True`, Stage 3 (`"annotation"`) assigns `model_class` to **all** structure nodes without making any LLM calls. This forces a specific extraction model on every node in the selected documents. ```python # Force a specific model on all structure nodes result = await run_pipeline( stages=["annotation", "entity_extraction"], document_names=["my_document.pdf"], manual=True, model_class="CompoundAssayResult", ) ``` > **Validation rules:** > - `manual=True` requires `model_class` to be set (and vice versa). > - `model_class` requires `manual=True`. > - `manual=True` is only valid when `"annotation"` is in `stages`. Use this when you already know which extraction model applies to a document and want to skip the LLM annotation step entirely. This is significantly faster than the default annotation mode. ### `only_unannotated` — Skip Already-Annotated Nodes **Type:** `bool` **Default:** `False` When `True`, Stage 3 (`"annotation"`) skips structure nodes that already have an annotation decision (a `model_class` property set in Neo4j). This is the primary mechanism for resuming an interrupted annotation run. ```python # First run: annotate all nodes result = await run_pipeline( stages=["annotation"], document_names=["large_document.pdf"], ) # ... run is interrupted after annotating 40 of 120 nodes ... # Resume: only annotate the remaining 80 nodes result = await run_pipeline( stages=["annotation"], document_names=["large_document.pdf"], only_unannotated=True, ) ``` ### `only_unextracted` — Skip Already-Extracted Nodes **Type:** `bool` **Default:** `False` When `True`, Stage 4 (`"entity_extraction"`) skips structure nodes that already have extracted entities connected in the graph. Use this to resume an interrupted extraction run: ```python # Resume extraction after an interruption result = await run_pipeline( stages=["entity_extraction"], document_names=["large_document.pdf"], only_unextracted=True, ) ``` ### `context_instructions` — Custom LLM Instructions **Type:** `str | None` **Default:** `None` Free-text instructions injected into both the converter prompts (Stage 0) and the annotation prompts (Stage 3). Use this to add domain-specific guidance for the LLM. ```python result = await run_pipeline( input_raw="./raw_docs", context_instructions=( "Focus on extracting clinical trial data. " "Pay special attention to adverse events and dosage information. " "When a table contains numerical data, preserve the exact values " "and units of measurement." ), ) ``` This parameter is forwarded to every document unit processed by the pipeline. It is particularly useful when processing documents from a specific domain where the default prompts need additional context. ### `update_mode` — In-Place Document Update **Type:** `bool` **Default:** `False` When `True`, Stage 2 (`"ingestion"`) replaces the latest version of an existing document in Neo4j **without incrementing the version number**. This is designed for single-document correction runs where you want to fix a document in place. ```python # Re-ingest a single document, overwriting the existing version result = await run_pipeline( input_raw="./corrected_docs/", update_mode=True, stages=["preprocess", "extraction", "ingestion"], ) ``` > **Constraints:** > - `update_mode=True` is not allowed when ingesting multiple documents. It is designed for single-document correction. > - `update_mode` and `replaces` are mutually exclusive — they cannot be used together. ### `replaces` — Document Replacement **Type:** `str | None` **Default:** `None` The `document_name` of an existing document that is superseded by the newly ingested document. After ingestion completes, the pipeline creates a replacement relationship in Neo4j linking the new document as the successor of the old one. ```python # Ingest a new version that replaces an old document result = await run_pipeline( input_raw="./new_version/", replaces="Clinical_Trial_Report_2024_v1", ) ``` The pipeline performs a pre-flight check before ingestion to verify that the document named in `replaces` actually exists in Neo4j. If it does not exist, the pipeline raises an error before processing any documents. > **Mutual exclusion:** `replaces` and `update_mode` cannot be used together. `update_mode` fixes the current version in-place; `replaces` creates a new version linked as the successor. ### `parallel_docs` — Document-Level Parallelism **Type:** `int` **Default:** `5` Maximum number of documents processed concurrently across all stages. The pipeline uses an `asyncio.Semaphore` to bound concurrency at this level. ```python # Process 10 documents concurrently result = await run_pipeline( input_raw="./raw_docs", parallel_docs=10, ) # Process one document at a time (sequential) result = await run_pipeline( input_raw="./raw_docs", parallel_docs=1, ) ``` Each document unit is bounded by this semaphore for its entire duration across all stages. Within each stage, additional concurrency control is provided by `llm_concurrency` and `neo4j_concurrency` (configured via `configure()`). > **Default is 5**, not 1. The pipeline processes up to 5 documents concurrently by default. ### `on_partial_failure` — Error Handling Strategy **Type:** `Literal["abort", "continue", "warn"]` **Default:** `"warn"` Controls behavior when a stage reports partial failures (`nodes_failed > 0`). The pipeline **never** stops processing other documents — every document in the batch runs independently. This parameter only affects whether a **single document** continues to its remaining stages after a partial failure. #### `"abort"` Stops the document from advancing to its remaining stages after a partial failure in annotation or entity extraction. This is the strictest mode. ```python # Abort a document's remaining stages on first failure result = await run_pipeline( input_raw="./raw_docs", on_partial_failure="abort", ) ``` #### `"continue"` The document keeps advancing to its next requested stage even if some nodes failed in the previous one. Completely silent — no warnings are logged. ```python # Continue processing despite failures, no warnings result = await run_pipeline( input_raw="./raw_docs", on_partial_failure="continue", ) ``` #### `"warn"` (default) Behaves like `"continue"` (the document keeps advancing) but additionally logs warnings at two levels: 1. **Immediately:** A per-document warning is emitted the moment a specific document decides to keep advancing despite a partial failure — naming the document, the stage, the failed-node count, and the concrete errors. 2. **At the end:** An aggregated per-stage warning fires whenever a stage reports one or more failed documents overall. ```python # Continue with detailed logging (default behavior) result = await run_pipeline( input_raw="./raw_docs", on_partial_failure="warn", ) ``` #### Stage-Specific Behavior The effect of `on_partial_failure` depends on which stage failed: | Failed Stage | Effect of `on_partial_failure` | | :--- | :--- | | `"preprocess"` | **Always** stops the document — no valid artifact exists for subsequent stages. | | `"extraction"` | **Always** stops the document — no valid document object for subsequent stages. | | `"ingestion"` | **Always** stops the document — no valid Neo4j document for subsequent stages. | | `"annotation"` | Partial failure (`nodes_failed > 0`). `"abort"` stops the document; `"continue"` / `"warn"` let it advance. | | `"entity_extraction"` | Partial failure (`nodes_failed > 0`). `"abort"` stops the document; `"continue"` / `"warn"` let it advance. | Stages 0-2 (preprocess, extraction, ingestion) are **total** failures for a document — there is nothing valid to continue with. Only Stages 3-4 (annotation, entity_extraction) are **partial** failures where `on_partial_failure` has an effect. ### `tabular_extensions` and `tabular_delimiter` — Tabular Options **Type:** `set[str] | None` / `str | None` **Default:** `{".csv", ".xlsx", ".xls"}` / `None` #### `tabular_extensions` File extensions to process via the tabular pipeline. Files with these extensions found in `input_raw` are automatically routed to the tabular pipeline alongside the standard Stages 0-4. ```python # Include .dat files as tabular data result = await run_pipeline( input_raw="./data", tabular_extensions={".csv", ".tsv", ".dat", ".xlsx"}, ) ``` #### `tabular_delimiter` Delimiter character for CSV tabular files. When `None`, the pipeline auto-detects the delimiter (`,` , `;`, `\t`, `|`). ```python # Force tab-delimited CSV processing result = await run_pipeline( input_raw="./data", tabular_delimiter="\t", ) ``` Both parameters are also forwarded to the tabular pipeline when it runs as an auto-detected sidecar alongside the main pipeline. --- ## Complete Workflows ### 1. First-Time Full Ingestion The canonical first run — convert raw files, extract structure, ingest to Neo4j, annotate, and extract entities. ```python import asyncio from scinr.newton import configure, run_pipeline async def main(): configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="your_password", ) result = await run_pipeline(input_raw="./raw_docs") # Inspect results print(f"Success: {result.success}") print(f"Duration: {result.total_duration_seconds:.2f}s") print(f"Stages: {result.stages_executed}") for stage_name in result.stages_executed: stage = getattr(result, stage_name) if stage: print(f" {stage_name}: " f"{stage.total_processed} processed, " f"{stage.total_failed} failed") asyncio.run(main()) ``` ### 2. Re-Run Annotation Only Re-run annotation and entity extraction on documents already in Neo4j, without touching Stages 0-2. ```python import asyncio from scinr.newton import configure, run_pipeline async def main(): configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="your_password", ) # Re-annotate and re-extract entities for specific documents result = await run_pipeline( stages=["annotation", "entity_extraction"], document_names=["Clinical_Trial_Report_2024"], only_unannotated=True, only_unextracted=True, ) print(f"Annotation: {result.annotation.total_processed} nodes") print(f"Extraction: {result.entity_extraction.total_processed} nodes") asyncio.run(main()) ``` ### 3. Add New Documents to Existing Graph Ingest new documents into a Neo4j graph that already contains previously ingested documents. ```python import asyncio from scinr.newton import configure, run_pipeline async def main(): configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="your_password", ) # New documents in a separate folder result = await run_pipeline( input_raw="./new_batch/", parallel_docs=3, ) print(f"New batch ingested: {result.success}") if result.ingestion: for doc in result.ingestion.documents: print(f" {doc.document_name}: " f"{doc.nodes_processed} processed") asyncio.run(main()) ``` ### 4. Replace a Document with Updated Version A document has been corrected or updated. Replace it in the graph while maintaining a link to the old version. ```python import asyncio from scinr.newton import configure, run_pipeline async def main(): configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="your_password", ) # Ingest the new version, linking it as the replacement result = await run_pipeline( input_raw="./corrected/", replaces="Clinical_Trial_Report_2024_v1", ) print(f"Replacement ingested: {result.success}") if result.ingestion: for doc in result.ingestion.documents: print(f" New document: {doc.document_name}") asyncio.run(main()) ``` ### 5. Process Tabular Data Only Process only CSV/XLSX/XLS files without running the standard document pipeline. ```python import asyncio from scinr.newton import configure, run_pipeline async def main(): configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="your_password", ) # Tabular-only pipeline result = await run_pipeline( input_raw="./tabular_data/", stages=["tabular"], tabular_extensions={".csv", ".xlsx"}, tabular_delimiter=",", ) print(f"Tabular pipeline: {result.success}") if result.tabular: print(f" {result.tabular.total_processed} files processed") asyncio.run(main()) ``` ### 6. Manual Model Application Force a specific extraction model on all nodes of a document, skipping the LLM annotation step entirely. ```python import asyncio from scinr.newton import configure, run_pipeline async def main(): configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="your_password", ) # Apply a known model to all nodes without LLM annotation result = await run_pipeline( stages=["annotation", "entity_extraction"], document_names=["Assay_Results_Q4.xlsx"], manual=True, model_class="CompoundAssayResult", ) print(f"Manual extraction: {result.success}") if result.entity_extraction: print(f" {result.entity_extraction.total_processed} nodes extracted") asyncio.run(main()) ``` --- ## PipelineResult Inspection `run_pipeline()` returns a `PipelineResult` dataclass with structured access to per-stage metrics. ### Overall Pipeline ```python result = await run_pipeline(input_raw="./raw_docs") # Overall success print(f"Success: {result.success}") # bool # Total wall-clock time print(f"Duration: {result.total_duration_seconds:.2f}s") # float # Ordered list of stages that were actually executed print(f"Stages: {result.stages_executed}") # list[str] ``` ### Per-Stage Access Each stage is accessible as an attribute on the result. If a stage was not executed, its attribute is `None`. ```python result = await run_pipeline(input_raw="./raw_docs") # StageResult attributes (or None if stage was skipped) result.preprocess # Stage 0 result.extraction # Stage 1 result.ingestion # Stage 2 result.annotation # Stage 3 result.entity_extraction # Stage 4 result.tabular # Stage 5 ``` ### StageResult Details Each `StageResult` contains: ```python stage = result.ingestion if stage: print(f"Stage: {stage.stage}") # str — stage name print(f"Success: {stage.success}") # bool print(f"Duration: {stage.duration_seconds:.2f}s") # float print(f"Processed: {stage.total_processed}") # int print(f"Failed: {stage.total_failed}") # int print(f"Errors: {stage.errors}") # list[str] ``` ### Per-Document Details Each `StageResult` has a `documents` list of `DocumentResult` entries: ```python if result.ingestion: for doc in result.ingestion.documents: print(f" {doc.document_name}: " f"{doc.nodes_processed} processed, " f"{doc.nodes_failed} failed") if doc.errors: for err in doc.errors: print(f" ERROR: {err}") ``` ### DocumentResult Fields | Field | Type | Description | | :--- | :--- | :--- | | `document_name` | `str` | The Neo4j `document_name` (or filename stem) of the processed document. | | `nodes_processed` | `int` | Nodes (or files) successfully processed. For Stages 0-2: 1 for success, 0 for failure. For Stages 3-4: number of structure nodes processed. | | `nodes_failed` | `int` | Nodes (or files) that failed processing. | | `errors` | `list[str]` | Error messages for this document. Empty on full success. | --- ## Multi-Step Workflows with Intermediate Directories For production pipelines, you often want to split processing into separate runs with persisted intermediate data. Here is a complete multi-step workflow: ```python import asyncio from scinr.newton import configure, run_pipeline async def main(): configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="your_password", ) # ── Step 1: Convert raw files to intermediate JSON ────────────────── r1 = await run_pipeline( input_raw="./raw_docs/", converter_output_dir="./data/converted/", stages=["preprocess"], ) print(f"Step 1 (preprocess): {r1.success}") # ── Step 2: Extract structure from converted JSON ─────────────────── r2 = await run_pipeline( extraction_input_dir="./data/converted/", extraction_output_dir="./data/extracted/", stages=["extraction"], ) print(f"Step 2 (extraction): {r2.success}") # ── Step 3: Ingest extracted documents into Neo4j ─────────────────── r3 = await run_pipeline( ingestion_input_dir="./data/extracted/", stages=["ingestion"], ) print(f"Step 3 (ingestion): {r3.success}") # ── Step 4: Annotate and extract entities ─────────────────────────── r4 = await run_pipeline( stages=["annotation", "entity_extraction"], document_names_dir="./data/extracted/", context_instructions="Focus on clinical trial data.", ) print(f"Step 4 (annotation + extraction): {r4.success}") asyncio.run(main()) ``` This pattern is useful when: - Different teams own different stages of the pipeline. - You want to re-run a specific stage without re-processing earlier stages. - You need to inspect intermediate data between stages. - You want to distribute work across different machines or time slots. --- ## Error Handling The pipeline can raise several exceptions before or during execution: ```python import asyncio from scinr.newton import ( configure, run_pipeline, ConfigurationError, PreconditionError, ExtractionError, IngestionError, ) async def main(): configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="your_password", ) try: result = await run_pipeline(input_raw="./raw_docs") except ConfigurationError as e: # Missing Neo4j or LLM configuration print(f"Configuration error: {e}") except PreconditionError as e: # Invalid parameter combination print(f"Precondition error: {e}") except ExtractionError as e: # Entity extraction failure print(f"Extraction error: {e}") except IngestionError as e: # Neo4j graph write failure print(f"Ingestion error: {e}") asyncio.run(main()) ``` ### Parameter Validation Errors The pipeline validates parameter combinations before execution. Invalid combinations raise `ValueError`: | Invalid Combination | Error Message | | :--- | :--- | | `input_raw` + `extraction_input_dir` | Mutually exclusive — different entry points | | `input_raw` + `ingestion_input_dir` | Mutually exclusive — different entry points | | `extraction_input_dir` + `ingestion_input_dir` | Mutually exclusive — different entry points | | `update_mode=True` + `replaces` | Mutually exclusive — different versioning strategies | | `manual=True` without `model_class` | `model_class` required when `manual=True` | | `model_class` without `manual=True` | `manual=True` required when `model_class` is set | | `document_names` + `document_names_dir` | Mutually exclusive — use one or the other | | `"tabular"` + other stages | `"tabular"` must be used alone | | `parallel_docs < 1` | Must be >= 1 | | `"preprocess"` without `input_raw` | Requires raw file input | | `"annotation"` without document names | Requires `document_names` or `document_names_dir` (or `ingestion` in stages) | --- ## Common Patterns ### Running the Pipeline from a Script ```python #!/usr/bin/env python """run_ingestion.py — Full pipeline run from command line.""" import asyncio import sys from pathlib import Path from scinr.newton import configure, run_pipeline async def run(input_dir: str, parallel: int = 5) -> None: configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="your_password", ) if not Path(input_dir).is_dir(): print(f"Error: '{input_dir}' is not a directory.", file=sys.stderr) sys.exit(1) result = await run_pipeline( input_raw=input_dir, parallel_docs=parallel, on_partial_failure="warn", ) print(f"\nPipeline {'succeeded' if result.success else 'failed'} " f"in {result.total_duration_seconds:.1f}s") for stage_name in result.stages_executed: stage = getattr(result, stage_name) if stage: status = "OK" if stage.success else f"FAILED ({stage.total_failed})" print(f" {stage_name}: {status}") if __name__ == "__main__": input_dir = sys.argv[1] if len(sys.argv) > 1 else "./raw_docs" parallel = int(sys.argv[2]) if len(sys.argv) > 2 else 5 asyncio.run(run(input_dir, parallel)) ``` ### Conditional Pipeline Based on File Types ```python import asyncio from pathlib import Path from scinr.newton import configure, run_pipeline async def smart_run(input_dir: str) -> None: configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="your_password", ) tabular_ext = {".csv", ".xlsx", ".xls"} files = list(Path(input_dir).rglob("*")) has_tabular = any(f.suffix.lower() in tabular_ext for f in files if f.is_file()) has_docs = any(f.suffix.lower() not in tabular_ext for f in files if f.is_file()) if has_docs: # Run full pipeline for documents (auto-detects tabular too) result = await run_pipeline(input_raw=input_dir) elif has_tabular: # Tabular-only result = await run_pipeline( input_raw=input_dir, stages=["tabular"], ) else: print("No supported files found.") return print(f"Pipeline result: {result.success}") asyncio.run(smart_run("./raw_docs")) ``` ### Resuming a Failed Pipeline ```python import asyncio from scinr.newton import configure, run_pipeline async def resume_pipeline(document_name: str) -> None: configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="your_password", ) # Skip already-annotated and already-extracted nodes result = await run_pipeline( stages=["annotation", "entity_extraction"], document_names=[document_name], only_unannotated=True, only_unextracted=True, on_partial_failure="warn", ) # Report what was skipped vs. what was processed if result.annotation: print(f"Annotation: {result.annotation.total_processed} nodes " f"({result.annotation.total_failed} failed)") if result.entity_extraction: print(f"Extraction: {result.entity_extraction.total_processed} nodes " f"({result.entity_extraction.total_failed} failed)") asyncio.run(resume_pipeline("Large_Document.pdf")) ``` --- ## See Also - **[Configuration](../configuration.md)** — Complete reference for `configure()`, environment variables, and all settings. - **[Architecture](../architecture.md)** — Detailed walkthrough of each pipeline stage and data flow. - **[Custom Models](custom-models.md)** — Defining domain-specific Pydantic extraction models. - **[Tabular Pipeline](tabular-pipeline.md)** — Tabular data normalization and processing. - **[Neo4j Graph Storage](neo4j-graph.md)** — Understanding the graph model and querying results. - **[Pipeline API](../api/pipeline.md)** — Auto-generated docstring for `run_pipeline()`. - **[Results API](../api/results.md)** — Auto-generated documentation for `PipelineResult`, `StageResult`, and `DocumentResult`. --- ## File: user-guides/storage-backends.md # Storage Backends `scinr.newton` provides an optional persistent storage layer that runs alongside the Neo4j graph pipeline. Storage backends archive raw source files and their converted pages, giving you a durable record of every document that passes through the pipeline. Neo4j remains the primary output store. Storage is supplementary — it exists for raw file archival, audit trails, and compliance requirements. You can run the full pipeline with Neo4j alone and never touch storage. Three backends are available: - **`none`** (default) — no persistent storage; all data stays in-memory during pipeline execution. - **`mongodb`** — MongoDB with GridFS for raw files and a document collection for converted pages. - **`custom`** — user-defined repositories implementing the `RawFileRepository` and `PageRepository` interfaces. --- ## Backend Comparison | Feature | `none` | `mongodb` | `custom` | |---|---|---|---| | Raw file storage | No | Yes (GridFS) | User-defined | | Page content | No | Yes (document collection) | User-defined | | Document metadata | No | Yes (raw_files collection) | User-defined | | Dependencies | None | `motor`, `pymongo` | User-defined | | Use case | Dev/testing, Neo4j-only workflows | Production with audit trail | Custom infrastructure needs | --- ## Architecture The storage layer is composed of two abstract repository interfaces: ``` RawFileRepository PageRepository ┌─────────────────┐ ┌──────────────────┐ │ .store() │ │ .store_page() │ │ → raw_file_id │ │ → page_id │ │ │ │ │ │ (binary files) │ │ .get_pages() │ │ │ │ → list[pages] │ │ │ │ │ │ │ │ (markdown pages) │ └─────────────────┘ └──────────────────┘ ``` - **`RawFileRepository`** — stores the original binary file and returns a `raw_file_id`. - **`PageRepository`** — stores converted page content (Markdown) linked to a `raw_file_id`, and supports retrieval. The pipeline calls `get_storage()` to obtain the configured pair of repositories. All downstream code interacts with the abstract interfaces, keeping the pipeline backend-agnostic. --- ## The "none" Backend (Default) When `storage_backend="none"` (the default), scinr uses no-op repository implementations that silently discard all writes. This is the recommended setting for development, testing, or when Neo4j alone is sufficient. ### Configuration ```python from scinr.newton import configure # Explicit — same as omitting the parameter entirely configure( storage_backend="none", ) # Or via environment variable # $ export STORAGE_BACKEND=none configure() # picks up STORAGE_BACKEND=none from environment ``` ### Behavior - Raw files are **not** archived to any persistent store. - Converted pages are **not** persisted. - All data lives in-memory during pipeline execution. - Intermediate JSON files are written to disk only if you set `converter_output_dir` or `extraction_output_dir` on `run_pipeline()`. - No additional dependencies are required. ### When to Use - **Development and testing** — fastest setup, no infrastructure needed. - **Neo4j-only workflows** — when the graph is the sole source of truth. - **CI/CD pipelines** — avoids requiring a MongoDB instance in test environments. - **Quick prototyping** — focus on extraction models without storage concerns. ### Complete Example ```python import asyncio from scinr.newton import configure, run_pipeline async def main(): configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="your_password", storage_backend="none", # explicit, but this is the default ) result = await run_pipeline(input_raw="./raw_docs") print(f"Pipeline: {'success' if result.success else 'failed'}") if result.preprocess: print(f" Converted: {result.preprocess.total_processed} files") asyncio.run(main()) ``` --- ## MongoDB Backend The MongoDB backend stores raw files in GridFS (for arbitrary file sizes) and converted pages in a standard document collection. It provides full durability, queryability, and audit capability. ### Installation ```bash pip install "scinr[mongodb]" ``` This installs `motor` (async MongoDB driver) and `pymongo` (sync driver, used for connection validation). ### Configuration ```python from scinr.newton import configure configure( storage_backend="mongodb", mongodb_uri="mongodb://localhost:27017", mongodb_database="scinr", mongodb_raw_files_collection="raw_files", mongodb_pages_collection="converted_pages", mongodb_gridfs_bucket="raw_binaries", ) ``` Or via environment variables: ```bash # .env STORAGE_BACKEND=mongodb MONGODB_URI=mongodb://user:pass@mongo.internal:27017 MONGODB_DATABASE=scinr_production MONGODB_RAW_FILES_COLLECTION=raw_files MONGODB_PAGES_COLLECTION=converted_pages MONGODB_GRIDFS_BUCKET=raw_binaries ``` ### Collections MongoDB creates three storage areas automatically on first use: #### `raw_files` — Raw File Metadata Lightweight metadata documents for each ingested file. The binary content itself lives in GridFS. ```json { "_id": "ObjectId('67a3b2c1d4e5f6a7b8c9d0e1')", "filename": "clinical_trial_report.pdf", "folder_path": "ModuleA/Section3", "content_type": "application/pdf", "size_bytes": 2458624, "checksum_sha256": "a1b2c3d4e5f6789012345678abcdef01234567890abcdef012345678901234567", "stored_at": "2025-01-15T10:30:00Z", "gridfs_id": "ObjectId('67a3b2c1d4e5f6a7b8c9d0e2')" } ``` | Field | Type | Description | |---|---|---| | `_id` | ObjectId | Unique identifier. Used as `raw_file_id` by pages. | | `filename` | String | Original filename including extension. | | `folder_path` | String or null | Relative path from the ingestion root, or `null` for root-level files. | | `content_type` | String | MIME type of the original file (e.g., `application/pdf`). | | `size_bytes` | Integer | Size of the binary content in bytes. | | `checksum_sha256` | String | SHA-256 hex digest of the original binary content. Used for deduplication and integrity verification. | | `stored_at` | DateTime | UTC timestamp when the record was persisted. | | `gridfs_id` | ObjectId | Reference to the file stored in GridFS. | #### `converted_pages` — Converted Page Content One document per converted page, linked to its parent raw file. ```json { "_id": "ObjectId('67a3b2c1d4e5f6a7b8c9d0e3')", "raw_file_id": "67a3b2c1d4e5f6a7b8c9d0e1", "filename": "clinical_trial_report", "folder_path": "ModuleA/Section3", "page_index": 0, "markdown": "# 3. Clinical Trial Results\n\nThe primary endpoint was...", "converted_at": "2025-01-15T10:30:05Z" } ``` | Field | Type | Description | |---|---|---| | `_id` | ObjectId | Unique identifier for the page record. | | `raw_file_id` | String | Reference to the parent `raw_files._id`. | | `filename` | String | Stem of the source file without extension. | | `folder_path` | String or null | Relative path from the ingestion root, or `null`. | | `page_index` | Integer | Zero-based page index. Matches the converter's page ordering. | | `markdown` | String | Full Markdown text of the page as produced by the converter. | | `converted_at` | DateTime | UTC timestamp when the page was persisted. | #### `raw_binaries` — GridFS Bucket GridFS automatically creates two internal collections: - `raw_binaries.files` — file metadata (filename, length, chunk size, upload date, GridFS metadata). - `raw_binaries.chunks` — binary data chunks (255 kB each by default). GridFS handles files of arbitrary size, removing the 16 MB BSON document limit. The `gridfs_id` in `raw_files` points to the corresponding GridFS file document. ### Indexes The MongoDB backend creates the following indexes on first use (via `ensure_indexes()`): ```python # Indexes created automatically by ensure_indexes(): # # converted_pages: primary lookup by raw_file_id + page ordering db.converted_pages.create_index( [("raw_file_id", 1), ("page_index", 1)], name="pages_by_raw_file_and_index", ) # converted_pages: secondary lookup by filename + folder db.converted_pages.create_index( [("filename", 1), ("folder_path", 1)], name="pages_by_filename_folder", ) # raw_files: deduplication by SHA-256 checksum db.raw_files.create_index( [("checksum_sha256", 1)], name="raw_files_by_checksum", ) ``` ### MongoDB Queries #### List All Stored Documents ```javascript db.raw_files.find().pretty(); ``` #### Get All Pages for a Document ```javascript // Find the raw_file_id first db.raw_files.findOne({ filename: "clinical_trial_report.pdf" }); // Then get all pages, ordered by page index db.converted_pages .find({ raw_file_id: "67a3b2c1d4e5f6a7b8c9d0e1" }) .sort({ page_index: 1 }); ``` #### Get Pages by Filename ```javascript db.converted_pages .find({ filename: "clinical_trial_report" }) .sort({ page_index: 1 }); ``` #### File Size Statistics by Format ```javascript db.raw_files.aggregate([ { $group: { _id: "$content_type", count: { $sum: 1 }, total_size: { $sum: "$size_bytes" }, avg_size: { $avg: "$size_bytes" } } }, { $sort: { total_size: -1 } } ]); ``` #### Find Duplicate Files by Checksum ```javascript db.raw_files.aggregate([ { $group: { _id: "$checksum_sha256", count: { $sum: 1 }, filenames: { $push: "$filename" } } }, { $match: { count: { $gt: 1 } } } ]); ``` #### Storage Usage Over Time ```javascript db.raw_files.aggregate([ { $group: { _id: { year: { $year: "$stored_at" }, month: { $month: "$stored_at" } }, count: { $sum: 1 }, total_bytes: { $sum: "$size_bytes" } } }, { $sort: { "_id.year": 1, "_id.month": 1 } } ]); ``` #### Retrieve Raw File from GridFS ```javascript // Using the gridfs_id from a raw_files document var gridfsId = ObjectId("67a3b2c1d4e5f6a7b8c9d0e2"); var bucket = new GridFSBucket(db, { bucketName: "raw_binaries" }); var stream = bucket.openDownloadStream(gridfsId); stream.on("data", function(chunk) { /* process chunk */ }); ``` ### Connection Validation When `storage_backend="mongodb"`, the factory validates the MongoDB connection at startup using a synchronous ping with a 5-second timeout. If the server is unreachable, a `StorageError` is raised immediately: ```python from scinr.newton import configure, run_pipeline from scinr.newton.exceptions import StorageError try: configure( storage_backend="mongodb", mongodb_uri="mongodb://wrong-host:27017", ) await run_pipeline(input_raw="./raw_docs") except StorageError as e: print(f"Storage unavailable: {e}") ``` ### Complete Example ```python import asyncio from scinr.newton import configure, run_pipeline async def main(): configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="your_password", storage_backend="mongodb", mongodb_uri="mongodb://user:pass@mongo.internal:27017", mongodb_database="scinr_production", ) result = await run_pipeline( input_raw="./raw_docs", converter_output_dir="./data/converted/", ) print(f"Pipeline: {'success' if result.success else 'failed'}") print(f"Raw files and pages stored in MongoDB.") asyncio.run(main()) ``` --- ## Custom Backend The `custom` backend lets you provide your own storage implementation. You implement two abstract base classes — `RawFileRepository` and `PageRepository` — and pass them as a tuple to `configure()`. ### Repository Interfaces ```python from abc import ABC, abstractmethod class RawFileRepository(ABC): @abstractmethod async def store( self, filename: str, content: bytes, content_type: str, folder_path: str | None, ) -> str: """Store a raw binary file and return its ID.""" ... class PageRepository(ABC): @abstractmethod async def store_page( self, raw_file_id: str, filename: str, folder_path: str | None, page_index: int, markdown: str, ) -> str: """Store a converted page and return its ID.""" ... @abstractmethod async def get_pages(self, raw_file_id: str) -> list[ConvertedPageRecord]: """Retrieve all pages for a raw file, ordered by page_index.""" ... ``` ### Implementing a Custom Backend Here is a complete example using S3 for raw files and DynamoDB for pages: ```python import hashlib from datetime import UTC, datetime from scinr.newton.storage.base import PageRepository, RawFileRepository from scinr.newton.storage.models import ConvertedPageRecord class S3RawFileRepository(RawFileRepository): """Stores raw files in Amazon S3.""" def __init__(self, bucket: str, region: str = "us-east-1"): self.bucket = bucket self.region = region # Initialize boto3 client from boto3 import client self.s3 = client("s3", region_name=region) async def store( self, filename: str, content: bytes, content_type: str, folder_path: str | None, ) -> str: # Build S3 key from folder path and filename key = f"{folder_path}/{filename}" if folder_path else filename # Compute checksum for metadata checksum = hashlib.sha256(content).hexdigest() # Upload to S3 self.s3.put_object( Bucket=self.bucket, Key=key, Body=content, ContentType=content_type, Metadata={ "checksum_sha256": checksum, "stored_at": datetime.now(UTC).isoformat(), }, ) # Return an identifier (S3 key as string) return key class DynamoDBPageRepository(PageRepository): """Stores converted pages in Amazon DynamoDB.""" def __init__(self, table_name: str, region: str = "us-east-1"): self.table_name = table_name self.region = region from boto3 import client self.dynamodb = client("dynamodb", region_name=region) async def store_page( self, raw_file_id: str, filename: str, folder_path: str | None, page_index: int, markdown: str, ) -> str: import uuid page_id = str(uuid.uuid4()) self.dynamodb.put_item( TableName=self.table_name, Item={ "page_id": {"S": page_id}, "raw_file_id": {"S": raw_file_id}, "filename": {"S": filename}, "folder_path": {"S": folder_path or ""}, "page_index": {"N": str(page_index)}, "markdown": {"S": markdown}, "converted_at": {"S": datetime.now(UTC).isoformat()}, }, ) return page_id async def get_pages(self, raw_file_id: str) -> list[ConvertedPageRecord]: from boto3.dynamodb.types import TypeDeserializer deserializer = TypeDeserializer() response = self.dynamodb.query( TableName=self.table_name, KeyConditionExpression="raw_file_id = :rfid", ExpressionAttributeValues={":rfid": {"S": raw_file_id}}, ScanIndexForward=True, ) pages = [] for item in response.get("Items", []): pages.append(ConvertedPageRecord( id=deserializer.deserialize(item["page_id"]), raw_file_id=deserializer.deserialize(item["raw_file_id"]), filename=deserializer.deserialize(item["filename"]), folder_path=deserializer.deserialize(item["folder_path"]) or None, page_index=int(deserializer.deserialize(item["page_index"])), markdown=deserializer.deserialize(item["markdown"]), converted_at=datetime.fromisoformat( deserializer.deserialize(item["converted_at"]) ), )) return pages ``` ### Registering the Custom Backend ```python from scinr.newton import configure # Instantiate your custom repositories raw_repo = S3RawFileRepository(bucket="scinr-raw-files", region="us-east-1") page_repo = DynamoDBPageRepository(table_name="scinr-pages", region="us-east-1") # Register them as a tuple configure( storage_backend="custom", custom_storage=(raw_repo, page_repo), ) ``` ### Key Points - **`custom_storage` expects a tuple of instances**, not a class and kwargs. The tuple is `(RawFileRepository, PageRepository)`. - Both repositories must be **async** — all methods use `async def`. - The `store()` and `store_page()` methods return a string identifier. The pipeline uses these IDs to link pages to their parent raw file. - `get_pages()` returns `ConvertedPageRecord` Pydantic models ordered by `page_index` ascending. - If `storage_backend="custom"` but `custom_storage` is not provided, the pipeline raises a `ConfigurationError` at `get_storage()` time. ### Minimal Custom Backend (In-Memory) For testing or lightweight scenarios, an in-memory implementation is straightforward: ```python from scinr.newton.storage.base import PageRepository, RawFileRepository from scinr.newton.storage.models import ConvertedPageRecord class InMemoryRawFileRepository(RawFileRepository): def __init__(self): self._files: dict[str, bytes] = {} async def store(self, filename, content, content_type, folder_path) -> str: import uuid file_id = str(uuid.uuid4()) self._files[file_id] = content return file_id class InMemoryPageRepository(PageRepository): def __init__(self): self._pages: dict[str, list[ConvertedPageRecord]] = {} async def store_page( self, raw_file_id, filename, folder_path, page_index, markdown ) -> str: import uuid from datetime import UTC, datetime page_id = str(uuid.uuid4()) record = ConvertedPageRecord( id=page_id, raw_file_id=raw_file_id, filename=filename, folder_path=folder_path, page_index=page_index, markdown=markdown, converted_at=datetime.now(UTC), ) self._pages.setdefault(raw_file_id, []).append(record) return page_id async def get_pages(self, raw_file_id) -> list[ConvertedPageRecord]: return sorted( self._pages.get(raw_file_id, []), key=lambda p: p.page_index, ) ``` --- ## When to Use Each Backend | Scenario | Recommended Backend | Rationale | |---|---|---| | Development / Testing | `none` | Zero infrastructure, fastest iteration. | | Production with audit trail | `mongodb` | Full durability, queryable, GridFS for large files. | | Production with existing cloud infrastructure | `custom` | Reuse S3, Azure Blob, or other storage you already manage. | | Neo4j-only workflow | `none` | Storage is optional; Neo4j is the primary output. | | Compliance (raw file retention) | `mongodb` or `custom` | Persistent archive of every ingested file. | | CI/CD pipeline | `none` | Avoids external dependencies in test environments. | | Multi-region deployment | `custom` | Route storage to region-appropriate infrastructure. | --- ## Storage and Pipeline Integration ### Where Storage Is Used Storage is called during **Stage 0 (preprocess)** and the **tabular pipeline**: 1. **Raw file storage** — immediately after reading a file from disk, before conversion. The binary content is stored and a `raw_file_id` is returned. 2. **Page storage** — after each page is converted to Markdown, the page content is stored and linked to the `raw_file_id`. ``` Pipeline Flow (with storage enabled): ┌──────────────┐ ┌──────────────────┐ ┌──────────────┐ │ Read File │ ──→ │ Store Raw File │ ──→ │ Convert to │ │ (binary) │ │ (raw_file_id) │ │ Markdown │ └──────────────┘ └──────────────────┘ └──────┬───────┘ │ ┌──────────────┐ ┌──────────────────┐ ┌──────▼───────┐ │ Write JSON │ ←── │ Store Page │ ←── │ Page N │ │ (intermed.) │ │ (page_id) │ │ (markdown) │ └──────────────┘ └──────────────────┘ └──────────────┘ ``` ### Independence from Neo4j Storage operates independently of Neo4j: - You can configure storage independently of the Neo4j pipeline stages — storage is called during Stage 0 (preprocess) and the tabular pipeline, while Neo4j is used in Stages 2-4. Both are optional components that can be tuned independently. - You can have Neo4j without storage (the default `none` backend). - Storage does not affect Stages 1-4 (extraction, ingestion, annotation, entity extraction). - If storage fails, the pipeline continues — storage errors are caught and reported without aborting the pipeline. ### Storage in the Tabular Pipeline The tabular pipeline (Stage 5) also uses storage when available: - Raw tabular files (CSV, XLSX) are stored via `RawFileRepository`. - Converted tabular pages are stored via `PageRepository`. - If no storage backend is configured, the tabular pipeline uses null repositories automatically. --- ## Configuration Resolution Storage settings follow the standard triple-resolution pattern: 1. **Explicit argument** to `configure()` (highest priority) 2. **Environment variable** (medium priority) 3. **Hard-coded default** (lowest priority) ```python # Example: env var sets backend to "mongodb", configure() overrides to "none" # $ export STORAGE_BACKEND=mongodb configure(storage_backend="none") # final value: "none" ``` ### All Storage Settings | Setting | `configure()` param | Environment Variable | Default | |---|---|---|---| | Backend type | `storage_backend` | `STORAGE_BACKEND` | `"none"` | | MongoDB URI | `mongodb_uri` | `MONGODB_URI` | `"mongodb://localhost:27017"` | | MongoDB database | `mongodb_database` | `MONGODB_DATABASE` | `"scinr"` | | Raw files collection | `mongodb_raw_files_collection` | `MONGODB_RAW_FILES_COLLECTION` | `"raw_files"` | | Pages collection | `mongodb_pages_collection` | `MONGODB_PAGES_COLLECTION` | `"converted_pages"` | | GridFS bucket | `mongodb_gridfs_bucket` | `MONGODB_GRIDFS_BUCKET` | `"raw_binaries"` | | Custom storage | `custom_storage` | *(none)* | `None` | --- ## Troubleshooting | Problem | Cause | Fix | |---|---|---| | `StorageError: Cannot connect to MongoDB` | Wrong URI or MongoDB not running | Verify `mongodb_uri`; check MongoDB is accessible. Use `storage_backend="none"` to bypass. | | `ConfigurationError: storage_backend='custom' requires passing custom_storage` | Missing `custom_storage` tuple | Pass `custom_storage=(raw_repo, page_repo)` to `configure()`. | | `ConfigurationError: Unknown storage_backend` | Invalid backend name | Use one of: `"none"`, `"mongodb"`, `"custom"`. | | Pages not found after ingestion | Storage backend was `none` during pipeline run | Re-run with `storage_backend="mongodb"` or `custom`. | | GridFS errors on large files | MongoDB version < 4.6 or missing GridFS support | Upgrade MongoDB to 4.6+ or use a managed MongoDB service. | | `ImportError: No module named 'motor'` | MongoDB extras not installed | Run `pip install "scinr[mongodb]"`. | | Custom backend methods not called | Passed class instead of instance | `custom_storage` expects instantiated objects: `(MyRawRepo(), MyPageRepo())`. | | Duplicate files ingested | No deduplication check | The `checksum_sha256` index on `raw_files` enables dedup queries. Implement pre-ingest checks using this field. | ### Debugging Storage Enable debug logging to see storage operations: ```python import logging from scinr.newton import configure logging.basicConfig(level=logging.DEBUG) configure( storage_backend="mongodb", mongodb_uri="mongodb://localhost:27017", log_level="DEBUG", ) ``` Debug output includes: ``` DEBUG:scinr.newton.storage.mongodb.raw_files:Stored raw file 'report.pdf' → raw_file_id=67a3..., gridfs_id=67a4... (2458624 bytes) DEBUG:scinr.newton.storage.mongodb.pages:Stored page 0 of 'report' → page_id=67a5... DEBUG:scinr.newton.storage.mongodb.pages:Stored page 1 of 'report' → page_id=67a6... DEBUG:scinr.newton.storage.mongodb.client:MongoDB indexes ensured. ``` --- ## See Also - **[Configuration](../configuration.md)** — Complete reference for `configure()`, environment variables, and all settings. - **[Running the Pipeline](running-pipeline.md)** — Pipeline entry points, stage selection, and workflow patterns. - **[Neo4j Graph Storage](neo4j-graph.md)** — Understanding the graph model and querying results. - **[Architecture](../architecture.md)** — Detailed walkthrough of each pipeline stage and data flow. - **[Pipeline API](../api/pipeline.md)** — Auto-generated docstring for `run_pipeline()`. --- ## File: user-guides/tabular-pipeline.md # Tabular Pipeline The tabular pipeline is a **complete alternative path** to the standard Stages 0-4 of `scinr.newton`. It processes `.csv`, `.xlsx`, and `.xls` files directly — reading headers, mapping columns to extraction model fields, instantiating Pydantic models from row data, and writing structured graph subgraphs to Neo4j. This is the definitive reference for the tabular pipeline. Every aspect of file discovery, header normalization, column mapping, model instantiation, normalization integration, and Neo4j output is documented here. --- ## 1. Introduction ### What the tabular pipeline is The tabular pipeline takes structured tabular files (CSV, XLSX) and converts each row into typed Pydantic model instances, which are then written as `:ModelInstance` and `:LabeledEntity` nodes in Neo4j. It bypasses the document-oriented Stages 0-4 entirely, using a dedicated LangGraph workflow instead: ``` load_sheets → prepare_sheet → classify_theme → decide_model → map_columns → write_tabular → (loop for next sheet) ``` ### How it differs from the unstructured pipeline The standard pipeline (Stages 0-4) is designed for unstructured documents (PDF, DOCX, PPTX). It converts files to intermediate representations, extracts document structure via LLM, ingests a hierarchical graph of `:Document` and `:StructureNode` nodes, annotates each section with a model, and extracts entities from free text. The tabular pipeline replaces this entire flow for structured data: | Aspect | Unstructured Pipeline (Stages 0-4) | Tabular Pipeline | | :--- | :--- | :--- | | **Input** | PDF, DOCX, PPTX, HTML, TXT, MD | CSV, XLSX | | **Stages** | 0 → 1 → 2 → 3 → 4 | Direct (bypasses 0-4) | | **Internal flow** | Sequential stages with intermediate files | LangGraph StateGraph per file | | **Document hierarchy** | Full hierarchy (`:Document` → `:StructureNode` tree) | `:Document` + `:Table` + `:Row` (flat per sheet) | | **Model selection** | LLM annotation per structure node (Stage 3) | LLM column mapping per sheet | | **Entity extraction** | LLM extracts from free text per section | Column values mapped to model fields directly | | **LLM calls** | One per section (annotation) + one per section (extraction) | Three per sheet: classify theme, decide model, map columns | | **Normalization** | Optional (LLM hint via `description=`) | **Primary use case** for `normalization_model` fields | | **Neo4j output** | `:Document` + `:StructureNode` tree + entities | `:Document` + `:Table` + `:Row` + `:ModelInstance` + `:LabeledEntity` | ### When to use it Use the tabular pipeline when: - Your source data is **already structured** in tabular format (CSV, XLSX). - You have a known extraction model that can receive column values directly. - You need **LLM-based normalization** of raw column values into structured nested models (the `normalization_model` mechanism). - You want to avoid the overhead of document conversion, structure extraction, and free-text entity extraction. --- ## 2. Pipeline Architecture ### 2.1 LangGraph StateGraph The tabular pipeline uses a LangGraph `StateGraph` that processes one file at a time, iterating over its sheets (CSV files have one sheet; XLSX files can have multiple): ``` ┌─────────────┐ │ load_sheets │ Read file, build previews, store pages └──────┬──────┘ │ ▼ ┌───────────────┐ │ check_done? ──┼──── end ──► END └──────┬────────┘ │ more sheets ▼ ┌───────────────┐ │ prepare_sheet │ Load current sheet data └──────┬────────┘ │ ▼ ┌────────────────┐ │ classify_theme │ LLM Call 0: detect thematic domain └──────┬─────────┘ │ ▼ ┌───────────────┐ │ decide_model │ LLM Call 1: select best extraction model └──────┬────────┘ │ ▼ ┌───────────────┐ │ map_columns │ LLM Call 2: map columns → model fields └──────┬────────┘ │ ▼ ┌───────────────┐ │ write_tabular │ Write Table + Row subgraph + entities to Neo4j └──────┬────────┘ │ ▼ ┌───────────────┐ │ check_done? ──┼──── end ──► END └───────────────┘ │ more sheets └──────► loop back to prepare_sheet ``` ### 2.2 Per-sheet processing Each sheet goes through three LLM calls: 1. **`classify_theme`** — The LLM examines column headers and a data preview to classify the sheet's thematic domain (e.g., `"pharmaceutical_quality"`). This narrows the model catalog to the relevant theme. 2. **`decide_model`** — The LLM receives the sheet preview (headers + up to 5 representative rows as Markdown) and the catalog of models from the classified theme. It selects the best `AnnotationDecision` (primary model class, optional complementary models, supplementary fields). 3. **`map_columns`** — The LLM maps each column header to a field in the selected model, producing a `ColumnMapping` with confidence scores and notes. Unmapped columns are tracked separately. After mapping, `write_tabular` instantiates Pydantic models from each row, runs normalization (if enabled), and writes the complete subgraph to Neo4j. --- ## 3. Running the Tabular Pipeline ### 3.1 Via `run_pipeline()` — Auto-Detection When `run_pipeline()` receives an `input_raw` directory containing tabular files, it automatically routes them to the tabular pipeline alongside the standard Stages 0-4: ```python import asyncio from scinr.newton import configure, run_pipeline async def main(): configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="your_password", ) # Auto-detection: CSV/XLSX files in input_raw are processed by tabular pipeline # PDF/DOCX files are processed by Stages 0-4 result = await run_pipeline(input_raw="./mixed_data") # Inspect tabular results if result.tabular: print(f"Tabular: {result.tabular.total_processed} files, " f"{result.tabular.total_failed} failed") asyncio.run(main()) ``` ### 3.2 Via `run_pipeline()` — Tabular Only To process **only** tabular files (skipping Stages 0-4 entirely): ```python # Tabular-only pipeline result = await run_pipeline( input_raw="./tabular_data", stages=["tabular"], ) ``` > **Important:** `"tabular"` cannot be combined with other stages in the `stages=` list. When you set `stages=["tabular"]`, it runs exclusively. When you omit `"tabular"` from `stages` (the default), tabular files in `input_raw` are auto-detected and processed automatically alongside the main pipeline. ### 3.3 Via `run_tabular_pipeline()` — Direct Call For full control, call the tabular pipeline directly: ```python from scinr.newton import configure, run_tabular_pipeline async def main(): configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="your_password", ) result = await run_tabular_pipeline( input_raw="./tabular_data", parallel_docs=4, tabular_extensions={".csv", ".xlsx"}, tabular_delimiter=",", ) print(f"Success: {result.success}") print(f"Files: {result.total_processed}") asyncio.run(main()) ``` ### 3.4 Full Signature ```python async def run_tabular_pipeline( input_raw: str, update_mode: bool = False, parallel_docs: int = 1, tabular_extensions: set | None = None, tabular_delimiter: str | None = None, ) -> StageResult ``` | Parameter | Type | Default | Description | | :--- | :--- | :--- | :--- | | `input_raw` | `str` | *(required)* | Folder containing raw tabular files (searched recursively). | | `update_mode` | `bool` | `False` | If `True`, wipe existing Table/Row subgraph and re-insert at the same version. | | `parallel_docs` | `int` | `1` | Maximum number of files to process concurrently. Default is 1 (sequential). | | `tabular_extensions` | `set[str] \| None` | `{".csv", ".xlsx", ".xls"}` | File extensions to treat as tabular. | | `tabular_delimiter` | `str \| None` | `None` | Field delimiter for CSV files. When `None`, auto-detected. | --- ## 4. File Discovery ### 4.1 Default Extensions The pipeline searches recursively in `input_raw` for files with these extensions (case-insensitive): | Extension | Format | Support | | :--- | :--- | :--- | | `.csv` | Comma-separated values | Full (auto-delimiter detection) | | `.xlsx` | Excel 2007+ | Full (multi-sheet support) | | `.xls` | Excel 97-2003 | **Not supported** — raises `ConversionError` | > **Note:** `.xls` files (Excel 97-2003 binary format) are not supported by `openpyxl`. If discovered, the pipeline raises a `ConversionError` with instructions to convert the file to `.xlsx` first. ### 4.2 Custom Extensions Extend the set of recognized extensions: ```python # Include .tsv and .dat files as tabular data result = await run_pipeline( input_raw="./data", tabular_extensions={".csv", ".tsv", ".dat", ".xlsx"}, ) ``` Custom extensions are passed through to the tabular agent. If the extension is not `.csv` or `.xlsx`, the reader will raise a `ValueError` for unsupported formats. ### 4.3 Custom Delimiter Force a specific delimiter for CSV files: ```python # Force tab-delimited CSV processing result = await run_pipeline( input_raw="./data", tabular_delimiter="\t", ) ``` When `tabular_delimiter` is `None` (default), the pipeline uses Python's `csv.Sniffer` to auto-detect the delimiter from a 4096-byte sample. Supported delimiters: `,`, `;`, `\t`, `|`. Falls back to `,` if detection fails. --- ## 5. File Reading and Header Normalization ### 5.1 CSV Reading CSV files are read with UTF-8-BOM awareness (`utf-8-sig` encoding). The reader: 1. Reads the entire file content. 2. Auto-detects delimiter via `csv.Sniffer` on first 4096 bytes. 3. Parses all rows, skipping empty rows. 4. Treats row 0 as headers, rows 1+ as data. 5. Converts all cell values to strings (strips whitespace). 6. **Deduplicates headers** — if duplicate column names exist, appends `_2`, `_3`, etc. to subsequent occurrences. 7. Pads or trims each data row to match the header count. ```python # Example: CSV with duplicate headers # Name, Code, Name, Value # A, X, B, Y # # After deduplication: # headers = ["Name", "Code", "Name_2", "Value"] ``` ### 5.2 XLSX Reading XLSX files are read via `openpyxl` in `read_only=True, data_only=True` mode. The reader: 1. Opens the workbook. 2. Iterates over all worksheets. 3. For each worksheet: converts cells to strings, strips whitespace, skips empty rows. 4. Treats row 0 as headers. 5. Deduplicates headers (same as CSV). 6. Pads/trims data rows to header count. 7. Skips empty worksheets entirely. Each worksheet becomes a separate `TabularSheet` entry, processed independently through the LangGraph. ### 5.3 Preview Generation For LLM calls (classify_theme, decide_model, map_columns), the pipeline generates a preview of up to 5 representative rows: - **≤ 5 rows:** all rows included. - **> 5 rows:** rows at indices 0, ~25%, ~50%, ~75%, and last row. The preview is rendered as a GFM Markdown table for LLM context. --- ## 6. Column Mapping ### 6.1 Theme Classification (LLM Call 0) Before model selection, the pipeline classifies the sheet's thematic domain. The LLM receives: - Document name and sheet name. - All column headers. - Preview rows as Markdown table. Output: a `ThemeClassification` with the detected theme path and a justification. On any failure, falls back to `"default"` (never crashes the graph). ### 6.2 Model Decision (LLM Call 1) The LLM receives: - The catalog of models from the classified theme (class docstrings, field descriptions). - Sheet preview as Markdown (headers + up to 5 rows). - Total row count. Output: an `AnnotationDecision` containing: - `matched_model_class` — the primary extraction model class name. - `complementary_models` — optional additional models to extract alongside. - `supplementary_fields` — optional additional fields to include. - `confidence` — the LLM's confidence level. - `justification` — reasoning for the decision. If no model matches, `matched_model_class` is `None` and all columns are mapped to `__extra__` (stored as raw data without model instantiation). ### 6.3 Column-to-Field Mapping (LLM Call 2) The LLM receives: - The selected model class and its full schema. - Sheet preview as Markdown. - All column headers. Output: a `ColumnMapping` containing: - `mappings` — list of `ColumnFieldMapping` entries (column name → model field name, with confidence and notes). - `unmapped_columns` — columns that could not be mapped to any field. Each mapping entry: ```python class ColumnFieldMapping: column_name: str # Original column header model_field_name: str # Target field in the extraction model confidence: str # "high", "medium", "low" notes: str # Explanation of the mapping ``` ### 6.4 Mapping Fallbacks If the LLM mapping fails entirely (parse error + repair loop exhaustion), all columns are mapped to `__extra__` with `confidence="low"`. This ensures the pipeline never crashes — data is still stored, just without model structuring. --- ## 7. Model Instantiation ### 7.1 Row-to-Model Conversion After column mapping, each data row is converted into a Pydantic model instance: 1. The column mapping defines which column value goes to which model field. 2. Column values are assembled into a dictionary matching the model's field names. 3. The Pydantic model is instantiated from the dictionary. 4. Pydantic validation runs (including `extra="forbid"` from `ExtractionModel`). ### 7.2 Validation Behavior Since extraction models inherit from `ExtractionModel` (which sets `extra="forbid"`), any column value that doesn't map to a declared field causes a validation error. The pipeline handles this gracefully: - **Mapped columns:** values are set on the model instance. - **Unmapped columns:** tracked in `ColumnMapping.unmapped_columns` and stored as `__extra__` data on the row. - **Type mismatches:** Pydantic's `str_strip_whitespace=True` auto-trims strings. Other type coercion follows Pydantic's default behavior. ### 7.3 Complementary Models When the `AnnotationDecision` includes complementary models, the pipeline resolves them and composes a composite schema. Each row can produce instances of the primary model and all complementary models simultaneously. ### 7.4 Combining Values When Multiple Columns Map to the Same Field It is common for the LLM column mapping (`map_columns`) to map **two or more columns to the same model field** — for example, when a sheet does not have a clean 1:1 header-to-field correspondence and the LLM reasonably assigns overlapping columns to the same target. When this happens, the pipeline **combines** the values instead of silently letting the last-processed column overwrite the field. **Combination logic — applies only to `str` and `list[str]` fields:** 1. Every non-empty value routed to the same field is collected, in the order the LLM emitted the corresponding entries in `ColumnMapping.mappings` — **not** the column's left-to-right position in the source file. 2. The collected values are deduplicated **by containment**, not by exact string match: if a value is fully contained in — or all of its individual words appear in — another, more complete value in the same group, the shorter/redundant value is dropped and only the more complete value survives. 3. The surviving values are combined: - **`str` fields:** joined with `"; "` as the separator. - **`list[str]` fields:** appended as separate list elements (no string concatenation). 4. Empty values (`""`, whitespace-only) never participate — they are filtered out before deduplication even starts. **Example:** Three columns — `"Drug"`, `"Strength (mg)"`, `"Unit"` — are all mapped to the same field (e.g. `product_description: str`). For a given row: ``` "Drug" → "Amox 500 mg" "Strength (mg)" → "500 mg" "Unit" → "mg" ``` Both `"500 mg"` and `"mg"` are fully contained (as a substring, and word-for-word) inside `"Amox 500 mg"`, so they are discarded as redundant. The final value written to `product_description` is: ``` "Amox 500 mg" ``` If the field were `list[str]` instead of `str`, the same containment-based dedup applies, but the surviving value(s) are appended as list elements rather than joined into a single string. **Any other field type (`int`, `float`, `bool`, `date`, `datetime`, `Enum`, nested submodels, …):** The combine/dedup logic described above applies **only** to `str` and `list[str]` fields. For every other field type, the pipeline does **not** attempt to merge multiple column values — it keeps the **last** value it processes (the pre-existing "last write wins" behavior) and emits a **warning** in the logs. No exception is raised and the pipeline never crashes because of this, but values from earlier-processed columns mapped to that field are silently discarded. > **⚠️ Model design recommendation:** because of this asymmetry, **extraction models intended for the tabular pipeline should declare every mappable field as `str` or `list[str]`.** If a field genuinely needs a non-string type (a parsed `int`, a `date`, a validated `Enum` member, etc.), **do not** try to solve it with a `@model_validator`/`@field_validator` on the model — every row instantiation in the tabular pipeline uses `model_construct(**kwargs)` (never a normal constructor call), and `model_construct()` unconditionally skips **all** Pydantic validators, in every mode (`"before"`, `"after"`, `"wrap"`). A validator added for this purpose will simply never execute in the tabular pipeline; the field will silently stay a raw string. The only mechanism that actually performs real Pydantic validation/coercion in this pipeline today is the existing LLM-based normalization system: wrap the value in a **nested Pydantic submodel** and mark it `json_schema_extra={"normalization_model": True, ...}` (see **[§8.1](#81-the-normalization_model-mechanism)**) — the `NormalizationEngine` fills it via `with_structured_output()` + `TypeAdapter(...).validate_python(...)`, which is real validation, applied to the nested submodel. See **[Custom Models](custom-models.md)** and the model-creation `AGENTS.md` §4.7 for the full explanation and a worked example. **Determinism and configuration:** - Combination order follows the order of entries for that field in `ColumnMapping.mappings` — i.e., the order the LLM emitted the mappings, not the column's position in the source file. - This behavior is **always active**; there is no `configure()` flag to disable it. Given the same row values and the same column mapping, the combined result is always the same. --- ## 8. Normalization Integration ### 8.1 The `normalization_model` Mechanism The `normalization_model` field annotation is the **primary feature** of the tabular pipeline. It enables LLM-based normalization of raw column values into structured nested models: ```python from pydantic import Field from scinr.newton.models.base import ExtractionModel class NormalizedSubstance(ExtractionModel): """Structured, normalized substance data.""" substance_name: str | None = Field( default=None, description="Canonical substance name.", ) substance_type: str | None = Field( default=None, description="Type: API, excipient, preservative, etc.", ) cas_number: str | None = Field( default=None, description="CAS registry number, if present.", ) class ProductRecord(ExtractionModel): """A single product record from a product catalogue CSV.""" product_name: str = Field( ..., description="Product name from the 'Name' column.", ) raw_substance: str = Field( ..., description="Active substance as written in the source column.", ) raw_strength: str = Field( ..., description="Strength as written in the source column.", ) manufacturer: str = Field( ..., description="Manufacturer name.", ) # Normalization: raw → structured (tabular pipeline) normalized_substance: NormalizedSubstance | None = Field( default=None, description="Structured substance data derived from raw_substance.", json_schema_extra={ "normalization_model": True, "normalization_source_fields": ["raw_substance"], }, ) normalized_strength: NormalizedStrength | None = Field( default=None, description="Structured strength data derived from raw_strength.", json_schema_extra={ "normalization_model": True, "normalization_source_fields": ["raw_strength"], }, ) ``` ### 8.2 How Normalization Works The `NormalizationEngine` processes instances in batches: 1. **Detection:** For each model instance, the engine scans fields for `json_schema_extra["normalization_model"] == True`. 2. **Source extraction:** For each normalizable field, it extracts the values of the sibling fields listed in `normalization_source_fields`. 3. **Deduplication:** Instances with identical source values are grouped by a hash key — the LLM is called once per unique source combination. 4. **Batching:** Unique entries are grouped by target type and processed in batches of `normalization_batch_size` (default: 3). 5. **LLM call:** Each batch is sent to the LLM with structured output, requesting normalized instances of the target type. 6. **Application:** Results are applied back to the original instances via `setattr` (with validation bypass fallback). 7. **Caching:** Results are cached by hash key — duplicate source values reuse the cached normalization. ### 8.3 `normalization_model` is Mandatory for Tabular Pipeline Without the `normalization_model: True` + `normalization_source_fields` annotation on a nested field, the tabular `NormalizationEngine` hook has nothing to trigger on. The nested submodel field silently stays `None` for every row, with no error raised. | Pipeline | `normalization_model` required? | | :--- | :--- | | Tabular only | ✅ **Mandatory** — without it, the nested field is never populated | | Unstructured only (Stages 3-4) | ⚪ Optional — the extraction LLM fills it from `description=` | | Both | ✅ **Recommended** — mandatory for tabular, optional-but-useful for unstructured | ### 8.4 `normalization_source_fields`: Always Explicit **Never omit `normalization_source_fields`.** If you omit it or leave it empty, the engine silently uses **all other scalar fields** of the parent model as source data — wasting tokens and leaking irrelevant context. ```python # ✅ GOOD — explicit, minimal source fields normalized_address: NormalizedAddress | None = Field( default=None, description="...", json_schema_extra={ "normalization_model": True, "normalization_source_fields": ["raw_address"], # only what's needed }, ) # ❌ BAD — implicit fallback vacuums ALL scalar fields normalized_address: NormalizedAddress | None = Field( default=None, description="...", json_schema_extra={ "normalization_model": True, # Missing normalization_source_fields — sends raw_name, raw_address, # raw_phone, internal_notes, etc. to the LLM }, ) ``` ### 8.5 Normalization Caching The `NormalizationEngine` caches results by a hash of the source values. If two rows have identical source data (e.g., the same `"Paracetamol 500mg"` in `raw_strength`), the LLM is called only once and the result is reused. This dramatically reduces LLM calls for datasets with repeated values. --- ## 9. Neo4j Output ### 9.1 Graph Structure The tabular pipeline writes the following nodes and relationships to Neo4j: ``` (:Document) └── [:HAS_STRUCTURE] ──► (:StructureNode:Table) ├── [:HAS_MODEL_DECISION] ──► (:ModelDecision) │ ├── [:MATCHES_MODEL] ──► (:Model {class: "ProductRecord"}) │ └── [:MATCHES_THEME] ──► (:Theme) └── [:HAS_STRUCTURE] ──► (:StructureNode:Row) ├── [:HAS_INFO_UNIT] ──► (:InfoUnit) ├── [:HAS_MODEL_DECISION] ──► (:ModelDecision) └── [:HAS_MODEL_INSTANCE] ──► (:ModelInstance:ProductRecord) ├── [:HAS_LABELED_ENTITY] ──► (:LabeledEntity) └── (field properties from model data) ``` ### 9.2 Node Types | Node | Labels | Created by | Description | | :--- | :--- | :--- | :--- | | Document | `:Document` | Tabular agent | Source file tracking node (path, version, raw_file_id). | | Table | `:StructureNode:Table` | `write_tabular` | One per sheet. Contains sheet metadata (column/row count, theme). | | Row | `:StructureNode:Row` | `write_tabular` | One per data row. Contains row data as InfoUnit Markdown. | | ModelDecision | `:ModelDecision` | `write_annotation` | The LLM's model selection decision for the table. | | ModelInstance | `:ModelInstance:{ModelName}` | `write_extraction_subgraph` | One per row. Contains all model field values as properties. | | LabeledEntity | `:LabeledEntity:{Label}` | `write_extraction_subgraph` | One per `entity_label` field value. Globally deduplicated. | | InfoUnit | `:InfoUnit` | `write_tabular` | Markdown table representation of a single row. | ### 9.3 Relationships | Relationship | Source → Target | Description | | :--- | :--- | :--- | | `HAS_STRUCTURE` | Document → Table | Links document to its table sheets. | | `HAS_STRUCTURE` | Table → Row | Links table to its data rows. | | `HAS_MODEL_DECISION` | Table → ModelDecision | The model selection decision for this table. | | `HAS_MODEL_DECISION` | Row → ModelDecision | Links row to the table's model decision. | | `HAS_INFO_UNIT` | Row → InfoUnit | Row's data rendered as Markdown. | | `HAS_MODEL_INSTANCE` | Row → ModelInstance | The Pydantic model instance for this row. | | `HAS_LABELED_ENTITY` | ModelInstance → LabeledEntity | Entity fields (from `entity_label` annotation). | | `MATCHES_MODEL` | ModelDecision → Model | The selected extraction model class. | | `MATCHES_THEME` | ModelDecision → Theme | The classified thematic domain. | ### 9.4 Entity Relationships Fields with `field_relationships` or `instance_relationships` in their `json_schema_extra` create additional graph edges: - **`field_relationships`:** Connects two `:LabeledEntity` nodes within the same model instance (sibling fields with `entity_label`). - **`instance_relationships`:** Connects `:ModelInstance` nodes across rows or documents via `join_via` key matching. These work identically to the unstructured pipeline — the tabular pipeline uses the same entity extraction subgraph writer. --- ## 10. Configuration ### 10.1 Normalization Settings The tabular normalization engine is configured via `configure()`: ```python from scinr.newton import configure configure( # ── Neo4j ────────────────────────────────────────────────────── neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="your_password", # ── LLM (used for theme classification, model decision, column mapping) ── llm=my_llm, # ── Tabular normalization ────────────────────────────────────── normalization_enabled=True, # Enable the NormalizationEngine normalization_batch_size=10, # Max entries per LLM batch (default: 3) normalization_llm=cheaper_llm, # Optional dedicated LLM for normalization ) ``` | Parameter | Type | Default | Description | | :--- | :--- | :--- | :--- | | `normalization_enabled` | `bool` | `True` | Enable/disable the `NormalizationEngine`. When `False`, `normalization_model` fields are inert and stay `None`. | | `normalization_batch_size` | `int` | `3` | Maximum number of unique normalization entries per LLM call. Higher values batch more entries but increase prompt size. | | `normalization_llm` | `BaseChatModel` | `None` | Dedicated LLM for normalization calls. Falls back to the main `llm` when `None`. Use a cheaper/faster model here. | ### 10.2 Concurrency Tabular normalization LLM calls share the global LLM semaphore configured via `llm_concurrency` in `configure()`. This prevents the normalization engine from exceeding the provider's connection pool limits. The `NormalizationEngine.concurrency` parameter is kept for API compatibility but no longer creates a local semaphore. --- ## 11. Complete Example ### 11.1 Full Pipeline Run ```python import asyncio from scinr.newton import configure, run_pipeline async def main(): configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="your_password", # Enable normalization for tabular data normalization_enabled=True, normalization_batch_size=10, ) result = await run_pipeline( input_raw="./product_catalogues", stages=["tabular"], tabular_extensions={".csv", ".xlsx"}, ) print(f"Success: {result.success}") if result.tabular: print(f"Files processed: {result.tabular.total_processed}") print(f"Files failed: {result.tabular.total_failed}") for doc in result.tabular.documents: status = "OK" if doc.nodes_failed == 0 else f"FAILED ({doc.errors})" print(f" {doc.document_name}: {status}") asyncio.run(main()) ``` ### 11.2 Mixed Pipeline (Documents + Tabular) ```python import asyncio from scinr.newton import configure, run_pipeline async def main(): configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="your_password", normalization_enabled=True, ) # input_raw contains both PDFs and CSVs # PDFs → Stages 0-4 (unstructured pipeline) # CSVs → tabular pipeline (auto-detected) result = await run_pipeline(input_raw="./mixed_data") # Inspect both pipeline results if result.ingestion: print(f"Documents ingested: {result.ingestion.total_processed}") if result.tabular: print(f"Tabular files processed: {result.tabular.total_processed}") asyncio.run(main()) ``` ### 11.3 Direct Tabular Pipeline with Custom Settings ```python import asyncio from scinr.newton import configure, run_tabular_pipeline async def main(): configure( neo4j_uri="bolt://localhost:7687", neo4j_user="neo4j", neo4j_password="your_password", normalization_enabled=True, normalization_batch_size=5, ) result = await run_tabular_pipeline( input_raw="./clinical_data", update_mode=False, parallel_docs=4, tabular_extensions={".csv", ".xlsx"}, tabular_delimiter=";", # Force semicolon delimiter ) print(f"Tabular pipeline: {result.success}") print(f"Duration: {result.duration_seconds:.2f}s") asyncio.run(main()) ``` --- ## 12. Troubleshooting ### 12.1 Common Issues | Problem | Cause | Fix | | :--- | :--- | :--- | | Tabular files not processed | Extension not in `tabular_extensions` | Add extension to `tabular_extensions={".csv", ".tsv", ".dat"}` | | Normalized fields are `None` | `normalization_enabled=False` in `configure()` | Set `normalization_enabled=True` | | Normalized fields are `None` | Missing `normalization_model: True` on field | Add `json_schema_extra={"normalization_model": True, ...}` | | Normalized fields are `None` | Missing `normalization_source_fields` | Add explicit `normalization_source_fields` list | | Column mapping wrong | Model fields don't match column semantics | Adjust field descriptions to be more specific | | Column mapping wrong | Theme classification incorrect | Check `THEME_DESCRIPTION` in your `catalog.py` | | CSV delimiter wrong | Auto-detection failed on unusual format | Set `tabular_delimiter=";"` or `tabular_delimiter="\t"` | | `.xls` files fail | Excel 97-2003 format not supported | Convert to `.xlsx` first | | Only one of several mapped columns' data survives on a field | Field type is not `str`/`list[str]` — multi-column combination only applies to those two types; other types keep the last value processed | Change the field to `str` or `list[str]`. Do **not** add a `model_validator` — it never runs in the tabular pipeline (`model_construct()` skips it). If a real type is genuinely needed, wrap it in a nested submodel marked `normalization_model: True` (see [§7.4](#74-combining-values-when-multiple-columns-map-to-the-same-field) and [§8.1](#81-the-normalization_model-mechanism)) | | Validation errors on row | Extra columns not mapped to model fields | Add missing fields to model, or use `default=None` | | Duplicate headers cause issues | Source file has repeated column names | Headers are auto-deduped (`col`, `col_2`, `col_3`) — check mapping | | XLSX sheet skipped | Worksheet is entirely empty | Empty worksheets are silently skipped (expected behavior) | | Model decision is `None` | No model in catalog matches the sheet data | Add appropriate extraction models to your theme catalog | | All columns mapped to `__extra__` | Model decision failed + repair loop exhausted | Check LLM connectivity and model catalog availability | ### 12.2 Debugging Column Mapping To inspect the column mapping for a specific sheet, query Neo4j: ```cypher MATCH (t:Table) WHERE t.title = 'Sheet1' MATCH (t)-[:HAS_MODEL_DECISION]->(md:ModelDecision) RETURN t.title AS table, md.matched_model_class AS model, md.confidence AS confidence, md.justification AS justification ``` ### 12.3 Debugging Normalization To check which normalizations were applied: ```cypher MATCH (mi:ModelInstance) WHERE mi.normalized_substance IS NOT NULL RETURN count(mi) AS normalized_count MATCH (mi:ModelInstance) WHERE mi.normalized_substance IS NULL RETURN count(mi) AS unnormalized_count ``` --- ## 13. Performance Considerations ### 13.1 LLM Call Budget Each sheet requires exactly 3 LLM calls (classify theme, decide model, map columns). Normalization adds additional calls proportional to unique source-value combinations divided by `normalization_batch_size`. For a file with N sheets and M unique normalization entries per type: ``` Total LLM calls = N × 3 + (M / normalization_batch_size) × number_of_normalization_types ``` ### 13.2 Parallelism `parallel_docs` controls how many files are processed concurrently. Default is 1 (sequential). Increase this when: - Processing many small files. - LLM provider has high concurrency limits. - Network latency is the bottleneck. ### 13.3 Normalization Caching The `NormalizationEngine` caches results by source-value hash. Datasets with many repeated values (e.g., a product catalogue with the same manufacturer across hundreds of rows) benefit significantly from this — the LLM is called once per unique value, not once per row. --- ## See Also - **[Running the Pipeline](running-pipeline.md)** — Full reference for `run_pipeline()`, including `stages=["tabular"]` and tabular options. - **[Configuration](../configuration.md)** — All `configure()` parameters, including `normalization_enabled`, `normalization_batch_size`, and `normalization_llm`. - **[Custom Models](custom-models.md)** — Defining extraction models with `normalization_model` fields for tabular use. - **[Neo4j Graph Storage](neo4j-graph.md)** — Understanding `:ModelInstance`, `:LabeledEntity`, and relationship types in the graph. - **[Architecture](../architecture.md)** — Detailed walkthrough of each pipeline stage, including the tabular LangGraph workflow. - **[Pipeline API](../api/pipeline.md)** — Auto-generated docstring for `run_pipeline()`. - **[Tabular API](../api/stages.md)** — Auto-generated docstring for `run_tabular_pipeline()`. ---