Neo4j Graph Model

scinr builds a connected graph representation of input documents and extracted domain concepts inside Neo4j. This guide covers the node types, relationships, and query patterns produced by the scinr.newton extraction engine.

The graph serves as the primary output store. After a document passes through the Newton pipeline, all structural elements and extracted entities are persisted as interconnected nodes and relationships, enabling complex cross-document queries that would be difficult or impossible with a flat relational schema.

Two distinct pipelines produce different graph structures:

Unstructured pipeline — processes PDFs, Word documents, and other narrative sources. Produces a hierarchy of headings, paragraphs, tables, and figures, with entities extracted from specific sections.
Tabular pipeline — processes CSV, Excel, and other structured tables. Produces one model instance per row, with labeled entities for stable, deduplicated fields.

Node Types

`:Document`

Represents the original input file or folder that was ingested into the system. Every document in the graph is a :Document node.

(:Document {
  name: "clinical_trial_report",
  path: "/path/to/clinical_trial_report",
  version: 1,
  load_date: 2024-01-15T10:30:00,
  latest: true,
  is_folder: false,
  tenant_id: "acme-corp",
  created_by_user_id: "user-42",
  job_id: "ingest-2026-09-06-a"
})

Property	Type	Description
`name`	String	Name of the file or folder as provided at ingestion time. Without extension.
`path`	String	Full path identifier for this document within the ingestion hierarchy. Without extension.
`version`	Integer	Version number. Starts at 1; increments on replacement or update. Numbered per tenant: another tenant's document at the same path does not affect it.
`load_date`	DateTime	Timestamp when the document was first ingested.
`latest`	Boolean	Indicates if this is the latest version of the document.
`is_folder`	Boolean	Indicates if this node represents a folder (true) vs a file (false).
`raw_file_id`	String	Storage-backend id of the stored raw file (empty for folders / when no storage backend is configured).
`context_instructions`	String \| null	Free-text ingestion context passed via `run_pipeline(context_instructions=...)`; `null` when not supplied.
`tenant_id`	String	Owner tenant from `run_pipeline(tenant_id=...)`, or the reserved value `"__public__"` for a public document (no tenant supplied) — never `null`. Part of the document identity: a `:Document` is unique on `(tenant_id, path, version)`. Written on leaf and folder-parent nodes. See Multi-tenancy.
`created_by_user_id`	String \| null	Caller-supplied id of the user that launched the ingestion, from `run_pipeline(created_by_user_id=...)`. Always set (`null` when not supplied).
`job_id`	String \| null	Caller-supplied ingestion job/run id from `run_pipeline(job_id=...)`. Always set (`null` when not supplied). Usable as a bulk-delete selector — see Document Deletion.
`frozen`	Boolean	`true` while the document is a frozen stub: its subtree is in a snapshot, not in the graph. Absent until the document is frozen for the first time, `false` after a restore. See Document Freezing.
`frozen_blob_id`, `frozen_at`	String	Id of the snapshot in the freeze backend, and ISO-8601 timestamp of the freeze. Only on a frozen stub; removed by the restore.
`frozen_keep_structure_nodes`, `frozen_keep_annotations`, `frozen_keep_extraction_results`	Boolean	The `keep_*` flags the document was frozen with. Only on a frozen stub.
`frozen_cleanup_pending`	Boolean	`true` from the moment the stub is marked until its subtree has been deleted and garbage-collected. Still there afterwards only if the freeze was interrupted.
`last_backup_blob_id`, `last_backup_at`	String	Snapshot and ISO-8601 timestamp of the last `freeze_document(delete_after_export=False)`.
`deletion_pending`	Boolean	Set by `delete_document()` before it deletes anything. A `:Document` that still carries it belongs to an interrupted deletion: call `delete_document()` again.

Multi-tenancy: the tenant is part of the document identity

Key. A :Document is identified by (tenant_id, path, version) (uniqueness constraint constraint_document_tenant_path_version). Two tenants that ingest the same path get two independent uploads: their own leaf and folder :Document nodes, versions, structure, annotations and extractions, with no relationship between them. Versioning (latest, HAS_NEWER_VERSION), update_mode, replaces, annotation, entity extraction and delete_document() only ever select documents of one tenant.
Public documents. A document ingested without a tenant is public: readable by every tenant, but still its own upload. It is stored with tenant_id: "__public__" (a non-null value, so it is indexed, usable in MERGE, and covered by the uniqueness constraint); on the write side None and "__public__" are the same thing (public), and "" is rejected. Reads expose the stored value ("__public__"), so None stays free to mean "no tenant filter" — see Graph Navigation — Scope.
Everything below inherits it. Every per-document node (:StructureNode, :InfoUnit, :ModelDecision, :ExtractionResult, …) carries the same scalar tenant_id, and the tenant is folded into the uid of the content-deduplicated nodes (:ModelInstance, :LabeledEntity, :Entity), so identical content from two tenants — or from a tenant and a public document — is never merged into one node. A tenant reading tenant_id IN [$tenant, "__public__"] may therefore see two :ModelInstance nodes for the same key (its own and the public one).
No relationship crosses tenants — public included. Every relationship an extraction writes targets a uid computed with its own tenant, so data relationships always stay inside one tenant. This is what lets the navigation API filter the tenant on the starting node only; the price is that a tenant's content is never linked to public content. See Graph Navigation — Why a traversal cannot leave the tenant.
Global catalog. :CatalogModel, :ModelField, :EntityLabel and :Theme are shared by all tenants.

// Tenant acme's latest documents plus the public ones
MATCH (d:Document {latest: true})
WHERE d.tenant_id IN ["acme", "__public__"]
RETURN d.tenant_id, d.path, d.version;

`:StructureNode`

Represents a structural element within a document — a section, subsection, freeform block, table, field group, appendix, or row. Structure nodes form a tree rooted at the document.

(:StructureNode:Section {
  id: "acme-corp::M3- Notice to applicant::1::1-foreword",
  node_id: "1-foreword",
  role: "section",
  title: "Foreword",
  theme: "pharmaceutical_quality",
  source_page_ids: ["..."],
  appearance_order: 1
})

Property	Type	Description
`id`	String	Unique hierarchical identifier for this structure node (the primary key; a uniqueness constraint is enforced): `{tenant_id}::{doc_path}::{version}::{ancestor node_ids/node_id}` — e.g. `__public__::M3/doc::2::5_3/5_3_1` for a nested node of a public document. The tenant prefix keeps two tenants' structure apart when they ingest the same path; the uids of `:InfoUnit`, `:ModelDecision`, `:ExtractionResult` and tabular rows are derived from it. Use this to address a node.
`node_id`	String	Short local identifier derived from the heading number or appearance-order slug. Convenient for display, but not guaranteed unique across the graph — do not use it as a key.
`role`	String	One of: `section`, `subsection`, `freeform_block`, `table`, `field_group`, `appendix`, `row`. Indexed.
`title`	String	Heading text, if this node represents a section heading. Null for leaf blocks like paragraphs.
`theme`	String	Theme path assigned to this node during extraction (default `"default"`).
`source_page_ids`	List	IDs of the converted source pages this node was extracted from.
`appearance_order`	Integer	1-based order of appearance among sibling nodes.

Key characteristics: Unlike :ModelInstance and :LabeledEntity (which use a single label plus a type property), :StructureNode genuinely carries a second dynamic label derived from its role property, applied via SET n:{RoleLabel} at ingestion time (see ingest/nodes.py). The mapping is deterministic PascalCase-of-role:

`role` (property)	Second label
`section`	`:Section`
`subsection`	`:Subsection`
`freeform_block`	`:FreeformBlock`
`table`	`:Table`
`field_group`	`:FieldGroup`
`appendix`	`:Appendix`
`row`	`:Row`

So a real node looks like (:StructureNode:Section {role: "section", ...}) — both the base label and the role-specific label coexist. The role property remains the recommended way to filter in Cypher (WHERE s.role = "table"), since it is indexed (idx_structure_node_role) and avoids needing to know the exact label spelling, but MATCH (s:Table) also works and is sometimes faster for label-scan-heavy queries.

`:LabeledEntity`

Represents a stable, deduplicated entity value extracted from a field marked with an entity_label in the schema. Labeled entities are created for fields that act as identifiers or cross-references — drug names, product codes, anatomical terms, etc.

(:LabeledEntity {
  label: "ActiveSubstance",
  value: "Metformin",
  uid: "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  normalized_value: "metformin"
})

Property	Type	Description
`label`	String	The entity label name (e.g., `ActiveSubstance`, `AnatomicalStructure`, `ProductCode`).
`value`	String	The canonical entity value. Used for deduplication across documents.
`uid`	String	Unique identifier for this specific entity node.
`normalized_value`	String	Normalized version of the value for case-insensitive matching.

Key characteristics:

Single label: The node has a single :LabeledEntity label and uses a label property to store the entity category.
Deduplication: If the same value is extracted from multiple documents of the same tenant, a single :LabeledEntity node is reused. This enables cross-document entity matching and aggregation. Tenants (and public documents) never share one: the tenant is part of the uid.
Stable identity: The uid remains constant for a given value, allowing reliable joins and traces.

`:ModelInstance`

Represents a single extracted entity record — one instance of a model class defined in the extraction schema. Each model instance stores all field values as node properties.

(:ModelInstance {
  model_class: "AdverseEventModel",
  uid: "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  event_type: "Headache",
  severity: "Mild",
  occurrence_count: 3
})

Property	Type	Description
`model_class`	String	The model class name (e.g., `AdverseEventModel`, `ProductRecord`, `LaboratoryResult`).
`uid`	String	Unique identifier for this model instance.
(dynamic)	Varies	All model field values are stored as additional properties on the node. Property names match the field names defined in the schema.

Key characteristics:

Single label: The node has a single :ModelInstance label and uses a model_class property to store the model type.
One node per record: Each extracted entity produces exactly one model instance node. In the tabular pipeline, this corresponds to one row per node.
Field values as properties: Every field defined in the model schema becomes a property on the node, enabling direct Cypher filtering without additional joins.

Other node types

These nodes complete the graph. The extraction, annotation, and catalog layers all use them.

Label	Created by	Description
`:InfoUnit`	Ingestion (Stage 2)	The smallest semantic content unit of a `:StructureNode` — `{uid, title, order, description}`. Attached via `[:HAS_INFO_UNIT]`. The `description` / `title` fields back the fulltext search indexes.
`:ExtractionResult`	Entity extraction (Stage 4)	One per extracted `:StructureNode`. Anchors a section's extraction output; carries `uid` and `model_class`.
`:ModelDecision`	Annotation (Stage 3)	The annotation decision for a `:StructureNode` — `{matched_model_class, confidence, reasoning}`.
`:CatalogModel`	Catalog setup	A registered Pydantic extraction model. Keyed by `name`.
`:ModelField`	Catalog setup	One field of a `:CatalogModel`. Composite key `(name, model)`.
`:EntityLabel`	Catalog setup	Schema-level `entity_label` singleton, keyed by `label`.
`:Theme`	Catalog setup	An extraction-model theme (from a `catalog.py`).
`:ProposedModel` / `:ProposedField`	Annotation	Details of a "no exact match" annotation proposal.
`:SupplementaryField` / `:ComplementaryMatch`	Annotation	Supplementary fields and complementary-model matches on a `:ModelDecision`.
`:Entity`	Triple fallback (Stage 4)	A subject/object node from the generic subject–predicate–object fallback used when no domain model matches. Global singleton, MERGE'd by normalized value.
Role labels (`:Section`, `:Subsection`, `:FreeformBlock`, `:Table`, `:FieldGroup`, `:Appendix`, `:Row`)	Ingestion	The second, role-derived label on every `:StructureNode` (see the `:StructureNode` section above).

Relationship Types

Document Structure

These relationships form the hierarchical tree of a document.

Relationship	From	To	Description
`[:HAS_STRUCTURE]`	`:Document`	`:StructureNode`	A document contains this top-level structure node. Connects the `:Document` only to its root structure node — never repeats between `:StructureNode`s.
`[:HAS_CHILD]`	`:StructureNode`	`:StructureNode`	Parent-child relationship in the document hierarchy. Headings point to their child headings, paragraphs, tables, etc.
`[:HAS_INFO_UNIT]`	`:StructureNode`	`:InfoUnit`	The semantic content units of a structure node.
`[:IS_COMPOSED_OF]`	`:Document`	`:Document`	Folder hierarchy — a folder document contains child documents.

Extraction

These relationships link extracted entities back to the document sections they came from, enabling provenance tracing.

Relationship	From	To	Description
`[:HAS_EXTRACTION]`	`:StructureNode`	`:ExtractionResult`	This section contains an extraction result.
`[:HAS_<FIELD>]`	`:ExtractionResult`	`:ModelInstance`	The extraction result contains this model instance (field name determines relationship type).
`[:REFERENCES]`	`:ModelInstance`	`:LabeledEntity`	A model instance references a globally deduplicated entity value (from an `entity_label` field). Always originates from `:ModelInstance`, never the reverse.
`[:USES_PRIMARY_MODEL]`	`:ExtractionResult`	`:CatalogModel`	The primary extraction model used for this section.
`[:USES_COMPLEMENTARY_MODEL]`	`:ExtractionResult`	`:CatalogModel`	A complementary model applied alongside the primary (0..*).

Annotation & Catalog

Relationship	From	To	Description
`[:HAS_MODEL_DECISION]`	`:StructureNode`	`:ModelDecision`	The annotation decision for this section.
`[:MATCHED_MODEL]`	`:ModelDecision`	`:CatalogModel`	The model class the annotation agent selected.
`[:HAS_FIELD]`	`:CatalogModel`	`:ModelField`	The declared fields of a catalog model.
`[:PRODUCES_ENTITY]`	`:CatalogModel`	`:EntityLabel`	An `entity_label` produced by one of the model's fields.
`[:BELONGS_TO_THEME]`	`:CatalogModel`	`:Theme`	The theme a catalog model belongs to.
`[:AGGREGATES]`	`:CatalogModel`	`:CatalogModel`	A parent (list-wrapper / aggregate) model that contains other models.

Triple Fallback

When no domain model matches a section, a generic subject–predicate–object fallback runs instead.

Relationship	From	To	Description
`[:HAS_ENTITY {role}]`	`:ExtractionResult`	`:Entity`	A subject or object entity of a fallback triple (`role` distinguishes them).
Custom predicate (`UPPER_SNAKE_CASE` / normalized)	`:Entity`	`:Entity`	The predicate edge of a triple. Predicate types are open-ended — there can be thousands of distinct ones; the Graph Navigation API filters them out by default.

Entity Relationships (from `field_relationships`)

When a schema defines field_relationships between labeled entity fields, the graph creates domain-specific edges between :LabeledEntity nodes. The relationship type is the field relationship name in UPPER_SNAKE_CASE.

Relationship	From	To	Description
Custom `UPPER_SNAKE_CASE`	`:LabeledEntity`	`:LabeledEntity`	Domain-specific semantic edge between two entity values (e.g., `:TREATS`, `:CONTRAINDICATED_WITH`, `:PART_OF`).

Instance Relationships (from `instance_relationships`)

When a schema defines instance_relationships between model instances, the graph creates edges between :ModelInstance nodes. The relationship type is the instance relationship name in UPPER_SNAKE_CASE.

Relationship	From	To	Description
Custom `UPPER_SNAKE_CASE`	`:ModelInstance`	`:ModelInstance`	Cross-model record edge (e.g., `:ASSOCIATED_WITH`, `:PRECEDES`, `:BELONGS_TO`).

Cross-Section `:ModelInstance` Linking via `instance_key`

The instance_relationships mechanism described above creates a relationship, but the real power of the graph comes from instance_key: a json_schema_extra flag on one or more fields of a model that makes the same logical entity, extracted from different sections or documents, collapse into a single Neo4j node instead of creating a duplicate.

How the UID is computed: when a model declares one or more fields with "instance_key": True, the graph mapper computes a deterministic 16-character UID (make_instance_uid) from the model's class name plus every instance_key field, sorted alphabetically by field name, and each value normalized (Unicode NFKD, accents stripped, lowercased, whitespace collapsed) before hashing with SHA-256. This means:

The UID does not depend on which document or section the value came from.
The UID is effectively case-insensitive and accent-insensitive (because inputs are normalized before hashing), even though the hash function itself is not.
A model without any instance_key field gets a random UID (uuid4) on every extraction — no deduplication is possible, and each extraction always creates a brand-new node.

Real example from this graph — CTDSectionSpec uses ctd_code as its instance_key. The same section code referenced from a HAS_REQUIREMENTS edge (written first, as a shell) and later extracted directly from its own document section (written second, with full detail) resolve to the exact same node:

MATCH (a:ModelInstance {model_class: "CTDSectionSpec"})-[:HAS_REQUIREMENTS]->(:ModelInstance)-[:REQUIREMENT_APPLIES_TO]->(a2:ModelInstance {model_class: "CTDSectionSpec"})
RETURN a.ctd_code, a.uid = a2.uid AS same_node
LIMIT 3;
-- {a.ctd_code: "3.2.S.2.2", same_node: true}
-- {a.ctd_code: "3.2.S.6",   same_node: true}

Shell nodes: when a source model references a target model via instance_relationships before the target has been extracted from its own section, the graph mapper writes a :ModelInstance node containing only its instance_key field(s) plus model_class and uid (a "shell"). The exact same MERGE ... ON CREATE SET ... ON MATCH SET ... Cypher pattern is used whether the node is being created as a shell or enriched later — there is no separate "shell-writing" code path. When the target model is later extracted from its own section, the graph mapper recomputes the same instance_key-derived UID, the MERGE finds the existing shell, and the remaining (non-key) fields are added to that same node via a plain property SET — with no guard against overwriting, but since only the shell's key fields were ever set, nothing is lost.

Real example — a CTDSectionSpec shell (3 properties) next to a fully-extracted one (11 properties), same model_class, same graph:

MATCH (m:ModelInstance {model_class: "CTDSectionSpec"})
RETURN m.ctd_code, size(keys(m)) AS property_count
ORDER BY property_count ASC
LIMIT 1;
-- {m.ctd_code: "3.2.p.6", property_count: 3}   ← shell: only ctd_code + model_class + uid

MATCH (m:ModelInstance {model_class: "CTDSectionSpec"})
RETURN m.ctd_code, size(keys(m)) AS property_count
ORDER BY property_count DESC
LIMIT 1;
-- {m.ctd_code: "3.2.S.2.5", property_count: 11}  ← fully extracted: title, nta_summary, cross_references, ...

Important caveats: - The relationship to a shell target is created unconditionally as soon as the fixed join_via key fields are non-empty — there is no check that the target model will ever actually be extracted. If it never is, the shell simply stays a shell (3 properties) forever. - Orphaned shells are not garbage-collected automatically after ingestion. A shell goes when delete_document() or freeze_document() removes the last instance that referenced it, or when collect_orphans() sweeps the tenant: both delete any ModelInstance/Entity no longer reachable from any ExtractionResult within 7 hops (see Document Deletion — Garbage Collection). - A list field driving instance_relationships (fan-out) creates one shell per list item, each with its own UID derived from the fixed anchor key(s) plus that one item's value.

Query: find shell nodes for any model class (generic property-count heuristic)

-- A genuine shell only has its instance_key fields populated, so it will have
-- far fewer properties than the average fully-extracted instance of the same model_class.
-- This is domain-agnostic: it does not assume any specific field like `description` exists.
MATCH (m:ModelInstance)
WITH m.model_class AS model, avg(size(keys(m))) AS avgProps
MATCH (m2:ModelInstance {model_class: model})
WITH model, avgProps, m2, size(keys(m2)) AS propCount
WHERE propCount < avgProps * 0.5
RETURN model, count(*) AS likely_shell_count
ORDER BY likely_shell_count DESC;

Normalized Models (Tabular Pipeline)

The tabular pipeline (CSV/XLSX/XLS ingestion) supports an optional normalization step for fields whose raw source value needs to be turned into a structured sub-model before it becomes useful for cross-dataset linking. This is triggered by two json_schema_extra keys on a nested-submodel field: normalization_model: True and normalization_source_fields: [...].

Non-normalized vs. normalized fields — the core distinction:

A non-normalized field (e.g. raw_address: str) holds the value exactly as it came from the source dataset — whatever string was in that CSV/Excel column, untouched. It is populated purely by column-mapping, with no LLM call involved.
A normalized field (e.g. normalized_address: NormalizedAddress | None) is a nested sub-model that the NormalizationEngine fills in by sending the listed normalization_source_fields to an LLM (with_structured_output()), validating the result with TypeAdapter.validate_python(), and writing it back onto the instance. This runs in memory, before the row is written to Neo4j — it never touches Cypher directly; the same graph writer (write_extraction_subgraph) used by the unstructured (PDF/DOCX) pipeline is reused afterward.

Why this matters for the graph — linking across datasets: the raw field on its own is just a string property; it does not participate in any deduplication unless it separately carries its own entity_label. The normalized sub-model, by contrast, is written to the graph as an ordinary :ModelInstance node (linked from its parent via [:HAS_<FIELDNAME>]) — and if that sub-model declares its own instance_key and/or entity_label fields (exactly as described in the section above), it deduplicates and links globally just like any other keyed ModelInstance. In other words: normalization does not create graph links by itself — it produces a clean, structured value that is much easier to key or label consistently than free-text, which is what actually enables reliable joins between rows/documents that describe the same real-world entity but wrote it differently (e.g. "123 Main St, Springfield" vs "123 main street springfield").

Example pattern (see docs/user-guides/normalization.md for the full guide):

class NormalizedAddress(ExtractionModel):
    street: str | None = Field(default=None, description="...")
    city: str | None = Field(default=None, description="...")
    postal_code: str | None = Field(default=None, description="...")
    country_code: str | None = Field(default=None, description="...")

class ContactRecord(ExtractionModel):
    raw_address: str = Field(..., description="Free-text address exactly as it appears in the source column.")
    normalized_address: NormalizedAddress | None = Field(
        default=None,
        description="Structured address derived from raw_address via LLM normalization.",
        json_schema_extra={
            "normalization_model": True,
            "normalization_source_fields": ["raw_address"],  # explicit — see caveat below
        },
    )

Resulting graph shape — note normalized_address is a separate node, never an embedded property map (Neo4j cannot store nested maps as a property value):

(:ModelInstance {model_class: "ContactRecord", raw_address: "123 Main St, Springfield"})
  └─[:HAS_NORMALIZED_ADDRESS]→ (:ModelInstance {model_class: "NormalizedAddress", street: "123 Main St", city: "Springfield", ...})

Caveat — normalization_source_fields fallback: if this list is omitted or empty, the engine silently falls back to sending every other scalar field of the parent model to the LLM as context. Always set it explicitly to avoid leaking unrelated columns into the normalization prompt.

Versioning

Relationship	From	To	Description
`[:HAS_NEWER_VERSION]`	`:Document`	`:Document`	The target document is a newer version that replaces the source document.

The latest property on :Document nodes provides a direct indicator of the current version, while the [:HAS_NEWER_VERSION] relationship enables version chain traversal.

Graph Structure Diagrams

Unstructured Pipeline Output

Narrative documents (PDF, Word) produce a heading hierarchy with extracted entities anchored to specific sections:

(:Document {name: "clinical_trial.pdf", latest: true})
  └─[:HAS_STRUCTURE]→ (:StructureNode {role: "section", title: "3. Results", latest: true})
                          └─[:HAS_CHILD]→ (:StructureNode {role: "section", title: "3.2 Adverse Events", latest: true})
                                              └─[:HAS_CHILD]→ (:StructureNode {role: "freeform_block", latest: true})
                                                                  └─[:HAS_EXTRACTION]→ (:ExtractionResult {model_class: "AdverseEventModel"})
                                                                                      └─[:HAS_EVENTS]→ (:ModelInstance {model_class: "AdverseEventModel", uid: "..."})
                                                                                                          └─[:REFERENCES]→ (:LabeledEntity {label: "ActiveSubstance", value: "Metformin"})

In this pattern:

The document contains a top-level section (3. Results).
That section contains a sub-section (3.2 Adverse Events).
The sub-section contains a freeform block with text.
A model instance (AdverseEventModel) was extracted from that block.
The :LabeledEntity (ActiveSubstance) is reached through the :ModelInstance via [:REFERENCES], not directly from :ExtractionResult.
The :LabeledEntity node is shared across documents if the same entity value appears elsewhere.

Tabular Pipeline Output

Structured documents (CSV, Excel) produce a flat table structure with one model instance per row:

(:Document {name: "products.csv", latest: true})
  └─[:HAS_STRUCTURE]→ (:StructureNode {role: "table", latest: true})
                          └─[:HAS_CHILD]→ (:StructureNode {role: "row", latest: true})
                                              └─[:HAS_EXTRACTION]→ (:ExtractionResult {model_class: "ProductRecord"})
                                                                  └─[:HAS_FIELDS]→ (:ModelInstance {model_class: "ProductRecord", uid: "..."})
                                                                                      └─[:REFERENCES]→ (:LabeledEntity {label: "ProductCode", value: "PROD-001"})
                          └─[:HAS_CHILD]→ (:StructureNode {role: "row", latest: true})
                                              └─[:HAS_EXTRACTION]→ (:ExtractionResult {model_class: "ProductRecord"})
                                                                  └─[:HAS_FIELDS]→ (:ModelInstance {model_class: "ProductRecord", uid: "..."})

In this pattern:

The document contains a single table structure node.
Each table row is a child of the table.
Each row produces one model instance (ProductRecord) via an :ExtractionResult.
Labeled entity fields within the model instance are connected via [:REFERENCES] from the :ModelInstance node, not directly from :ExtractionResult.

Versioning

scinr supports two document update strategies, both reflected in the graph:

In-place Update (`update_mode=True`)

When update_mode=True, the existing document and all its downstream nodes are replaced in-place. The version property on the :Document node is reused (it does not increment), and the path remains the same. The ingestion loader finds the latest version by path, deletes its :StructureNode / :InfoUnit descendants, and re-inserts the new structure at the same version number.

The :Entity / :ModelInstance / :LabeledEntity nodes that only the old content reached are not deleted by the re-ingestion. collect_orphans(tenant_id=...) removes them (see Document Deletion — Garbage Collection). The re-ingestion does not check whether the version is frozen either (see Document Freezing — Caveats).

(:Document {
  name: "clinical_trial_report.pdf",
  path: "/path/to/clinical_trial_report.pdf",
  version: 1,
  load_date: 2024-06-01T09:00:00,
  latest: true,
  is_folder: false
})

Replacement (`replaces="old_name"`)

When replaces is set, a new :Document node is created with the same path and an incremented version, and a [:HAS_NEWER_VERSION] relationship links the old document to the new one.

(:Document {name: "clinical_trial_report.pdf", path: "/path/to/clinical_trial_report.pdf", version: 1, latest: false})
  └─[:HAS_NEWER_VERSION]→ (:Document {name: "clinical_trial_report.pdf", path: "/path/to/clinical_trial_report.pdf", version: 2, latest: true})

This preserves the full history of document versions in the graph, allowing queries to trace which version produced which extraction results. The latest property indicates the current version, while [:HAS_NEWER_VERSION] enables version chain traversal.

Document Deletion

When a document needs to be permanently removed from the graph (rather than updated in-place), use delete_document(). This function:

Always acts within one tenant: tenant_id is mandatory (None / "__public__" = public documents).
Removes the tenant's :Document node(s) matching the given path (and optionally version), or every :Document of the tenant carrying a given job_id (delete_document(job_id=..., tenant_id=...)), optionally narrowed further by created_by_user_id (job_id and created_by_user_id accept a list).
Cascade-deletes all connected structure, annotation, and extraction nodes.
Garbage-collects the :Entity, :ModelInstance, and :LabeledEntity nodes of the tenant that the deletion left orphaned.
Works in bounded transactions, so a large folder does not overrun Neo4j's transaction memory; an interrupted deletion is finished by calling it again.

Unlike update_mode=True re-ingestion, deletion is irreversible — there is no undo. See the Document Deletion guide for details.

Document Freezing

freeze_document() is the reversible alternative. It exports the document's subgraph to a snapshot in the freeze backend and reduces the document to a stub:

The :Document node stays, with frozen=true and frozen_blob_id, so the document is still listed and its version chain is intact.
The :StructureNode tree, the :InfoUnit nodes, the annotations and the extraction results are deleted, unless keep_structure_nodes, keep_annotations or keep_extraction_results keep them. InfoUnits always go.
A kept :ModelDecision / :ExtractionResult whose :StructureNode was deleted hangs from the :Document instead, through a temporary (:Document)-[:HAS_MODEL_DECISION|HAS_EXTRACTION]->() relationship. The restore removes it.

restore_document() rebuilds the subgraph from the snapshot, with the same ids and uids and no LLM calls. See the Document Freezing guide.

Query Patterns

Prefer Python?

The Graph Navigation module wraps these patterns in typed, async methods — list root documents, walk the structure tree, pull a document's model instances, filter by class and properties, and more — so you rarely need to write Cypher by hand.

Find All Documents

List all ingested documents, ordered by most recent first:

MATCH (d:Document)
WHERE d.latest = true
RETURN d.name, d.path, d.version, d.load_date
ORDER BY d.load_date DESC;

Find All Entities of a Type

Aggregate extracted model instances by their field values:

MATCH (m:ModelInstance {model_class: "AdverseEventModel"})
RETURN m.event_type, m.severity, count(m) AS occurrences
ORDER BY occurrences DESC;

Find Entities Extracted from a Specific Document

Trace all model instances back through their source document:

MATCH (d:Document {latest: true})-[:HAS_STRUCTURE]->(:StructureNode)-[:HAS_CHILD*0..]->(s:StructureNode)-[:HAS_EXTRACTION]->(er:ExtractionResult)-[r]->(m:ModelInstance)
WHERE d.name = "clinical_trial_report.pdf"
RETURN m.model_class AS entity_type, count(m) AS count
ORDER BY count DESC;

The correct pattern to descend the hierarchy is [:HAS_STRUCTURE] (once, from the :Document to its root structure node) followed by [:HAS_CHILD*0..] (recursive between :StructureNodes) — not [:HAS_STRUCTURE*0..], since HAS_STRUCTURE only connects a :Document to its root structure node and never repeats between :StructureNodes. Also, the relationship between :ExtractionResult and :ModelInstance is [:HAS_<FIELD>] with a dynamic name based on the schema field, so the query uses a generic -[r]-> instead of a fixed relationship type.

Find Entity Relationships

List all domain-specific edges between labeled entities:

MATCH (a:LabeledEntity)-[r]->(b:LabeledEntity)
RETURN a.label AS source_type, type(r) AS relationship, b.label AS target_type, count(r) AS count
ORDER BY count DESC;

Find Cross-Model Connections

List all relationships between model instances:

MATCH (a:ModelInstance)-[r]->(b:ModelInstance)
RETURN a.model_class AS source_type, type(r) AS relationship, b.model_class AS target_type, count(r) AS count
ORDER BY count DESC;

Full Extraction Trace

Reconstruct the full provenance chain from document to extracted entity:

MATCH (d:Document {latest: true})-[:HAS_STRUCTURE]->(:StructureNode)-[:HAS_CHILD*0..]->(s:StructureNode)-[:HAS_EXTRACTION]->(er:ExtractionResult)-[r]->(m:ModelInstance)
RETURN d.name, s.title, s.role, m.model_class, m;

This query is useful for auditing: it shows exactly which section of which document produced each extracted entity.

Find Entity Usage Across Documents

Identify which documents share a particular entity value. Since labeled entities are deduplicated (shared across documents), find where the entity is referenced via model instances:

// Find which sections reference this entity via ModelInstance-[:REFERENCES]->LabeledEntity
MATCH (e:LabeledEntity {label: "ActiveSubstance", value: "Metformin"})
MATCH (m:ModelInstance)-[:REFERENCES]->(e)
MATCH (er:ExtractionResult)-[r]->(m)
MATCH (s:StructureNode)-[:HAS_EXTRACTION]->(er)
MATCH (d:Document)-[:HAS_STRUCTURE]->(:StructureNode)-[:HAS_CHILD*0..]->(s)
WHERE d.latest = true
RETURN DISTINCT d.name, s.title
ORDER BY d.name;

Note: [:HAS_EXTRACTION] connects exclusively :StructureNode→:ExtractionResult. The :ExtractionResult→:ModelInstance segment is always [:HAS_<FIELD>] (dynamic), hence the generic -[r]->.

Find Document Version History

Trace the lineage of a document through its versions:

MATCH (old:Document)-[:HAS_NEWER_VERSION*0..]->(new:Document)
WHERE new.path = "/path/to/clinical_trial_report.pdf" AND new.latest = true
RETURN old.name AS document_name, old.path AS path, old.version AS version, old.load_date AS load_date
ORDER BY old.version;

Indexes and Constraints

setup_schema() (run by run_ingestion, the pipeline, the tabular pipeline and the loader) creates every constraint and index below with IF NOT EXISTS; there is nothing to create by hand. The authoritative list is ingest/schema.py.

Unique constraints

Label	Key	Notes
`:Document`	`(tenant_id, path, version)`	The document identity. `setup_schema()` drops the legacy `constraint_document_path_version`.
`:StructureNode`	`id`	The id is tenant-prefixed.
`:InfoUnit`, `:ExtractionResult`, `:ModelInstance`, `:LabeledEntity`, `:Entity`	`uid`	The uid hashes the tenant.
`:ModelField`	`(name, model)`	Catalog.
`:EntityLabel`	`label`	Catalog.

A lookup by id / uid is a seek on these constraints, as long as the pattern carries the label.

Single-property indexes

tenant_id on Document, StructureNode, InfoUnit, ExtractionResult, ModelInstance, ModelDecision, LabeledEntity and Entity; created_by_user_id / job_id on Document, StructureNode and ExtractionResult; plus Document(name), Document(latest), Document(path), StructureNode(role), StructureNode(row_index), StructureNode(source_page_ids), LabeledEntity(label), ExtractionResult(node_full_id) and ModelInstance(model_class).

uid on ModelDecision, ProposedModel, ProposedField, ComplementaryMatch and SupplementaryField, and CatalogModel(name). These are plain indexes, not constraints, because older graphs may hold duplicates. restore_document() looks nodes up through them, and creates them itself when they are missing (a graph ingested before they existed).

Composite tenant indexes

Index	Properties	Serves
`idx_model_instance_tenant_model_class`	`ModelInstance(tenant_id, model_class)`	`get_model_instances_by_class` / `count_model_instances_by_class`, properties by class
`idx_document_tenant_latest`	`Document(tenant_id, latest)`	`get_documents`, `list_root_documents` / `count_root_documents` with `latest_only`, graph summary
`idx_document_tenant_name`	`Document(tenant_id, name)`	lookups by name within a tenant (e.g. the `replaces=` resolution)
`idx_structure_node_tenant_role`	`StructureNode(tenant_id, role)`	`find_structure_nodes(role=...)`
`idx_labeled_entity_tenant_label`	`LabeledEntity(tenant_id, label)`	`get_labeled_entities(label=...)`

The single tenant_id indexes are kept: a query over all tenants (tenant_id=None) has no tenant predicate and cannot use a composite index.

`instance_key` indexes

For every distinct field name marked instance_key: True in the registered models, ensure_catalog_models() creates

CREATE INDEX idx_mi_key_<field> IF NOT EXISTS
FOR (mi:ModelInstance) ON (mi.tenant_id, mi.`<field>`)

The label is always :ModelInstance, so models sharing a key field name share the index. Names that are not plain identifiers are sanitized to [A-Za-z0-9_] with a short hash suffix. New indexes are built in the background (POPULATING) and do not block ingestion.

Full key: use nav.get_model_instance_by_key(...) — it rebuilds the uid and seeks on the unique constraint. Do not filter with toLower(mi.field) = ...: a function on the property prevents any index from being used. Key values are stored normalized (normalize_key).
Part of the key: where={"field": normalize_key(value)} with the tenant uses idx_mi_key_<field>.
If a field stops being a key its index is not dropped (it is harmless). To clean it up: SHOW INDEXES YIELD name WHERE name STARTS WITH 'idx_mi_key_', then DROP INDEX <name>.

How the planner uses them

Checked with EXPLAIN ... USING INDEX on Neo4j 2026.05 Community (a hint the planner cannot honour is an error):

tenant_id IN [t, '__public__'] can seek on the tenant_id prefix of a composite index, so include_public=True still uses it.
A composite index is only used when the query constrains every property in it (IS NOT NULL counts). path + latest cannot use (tenant_id, path, version); "latest version of this path" uses idx_document_path instead, and path is very selective.
One index per node. With single indexes only, "instances of class X for tenant T" seeks on one of them and filters every node for the other — hence the composite indexes.
No label, no index. MATCH (n {uid: $uid}) scans every node of every tenant. Every pattern that looks a node up by a property must carry its label; tests/unit/test_cypher_label_guard.py fails on any unlabelled (var {prop: ...}) pattern in the library.

Known limitation: get_document_stats' per-role counts traverse from the document and use no role index.

A predicate placed after a WITH (WITH n, x WHERE n.tenant_id = ...) is not a condition of the preceding MATCH and cannot drive its index seek: keep tenant and class filters in the MATCH's own WHERE.

Community edition

Neo4j Community does not support property existence constraints (IS NOT NULL), node key constraints or property type constraints; setup_schema() does not create them. Range indexes (single and composite), unique constraints (also composite) and fulltext indexes are all available, and are the only kinds scinr uses.

Verifying index usage

tests/integration/test_index_usage.py checks that the representative queries can use their index — it does not measure time and needs no data volume:

Neo4j: runs setup_schema(), creates one instance_key index, and runs EXPLAIN <query> USING INDEX <expected index> for each navigation query (captured from the real navigator), the instance_key lookup and the graph_mapper writes. Nothing is executed.
MongoDB: in a throwaway scinr_it_<uuid> database (dropped afterwards), checks that get_storage() created the indexes, inserts a few documents and asserts IXSCAN on the expected index and no COLLSCAN for the page, raw-file inventory and delete queries.

NEO4J_URI=bolt://localhost:7687 NEO4J_USER=neo4j NEO4J_PASSWORD=... NEO4J_DATABASE=neo4j \
MONGODB_URI=mongodb://user:pass@localhost:27017/?authSource=admin \
pytest -m integration tests/integration/test_index_usage.py

Each backend is skipped when its variables are missing or the server is unreachable. The Neo4j user needs schema privileges (it runs setup_schema()); the MongoDB user needs to create and drop a database.

Graph Size Estimation

The following table provides rough estimates for planning database capacity:

Input	Approximate Nodes	Approximate Relationships
1 PDF (50 pages)	~500 StructureNodes + extracted entities	~600 (structure + extraction)
10 PDFs (50 pages each)	~5,000 StructureNodes + extracted entities	~6,000 (structure + extraction)
1 CSV (1,000 rows)	~1,000 ModelInstances + labeled entities	~1,000 (extraction) + entity/instance relationships
1 Excel (5 sheets, 500 rows each)	~2,500 ModelInstances + labeled entities	~2,500 (extraction) + entity/instance relationships
100 PDFs (mixed size)	~50,000 StructureNodes + extracted entities	~60,000 (structure + extraction)

Notes:

Actual counts depend on document complexity, schema definition, and extraction yield.
Labeled entity deduplication reduces total node count when the same entity appears across multiple documents.
Entity and instance relationships add proportionally to the relationship count based on schema configuration.
For large-scale deployments, consider partitioning strategies and Neo4j clustering.

Shell Nodes — Two Distinct Concepts

"Shell node" can refer to two unrelated things in this graph — do not confuse them:

:ModelInstance shells (the mechanism documented in detail above, under "Cross-Section :ModelInstance Linking via instance_key"): a ModelInstance node created ahead of time by an instance_relationships reference, containing only its instance_key fields until the target model is extracted from its own section. Detect these with the property-count heuristic query shown above.
:Document folder nodes (is_folder: true): these are not a deduplication artifact at all — they are legitimate, permanent nodes representing a directory in the ingestion hierarchy (as opposed to a leaf file). They are not "incomplete" or waiting to be enriched; is_folder: true is simply their normal, final state. Do not apply the ModelInstance shell-detection heuristic to :Document nodes — check is_folder directly instead:

MATCH (d:Document {latest: true, is_folder: true})
RETURN d.name, d.path;

Neo4j Graph Model

Node Types

:Document

Multi-tenancy: the tenant is part of the document identity

:StructureNode

:LabeledEntity

:ModelInstance

Other node types

Relationship Types

Document Structure

Extraction

Annotation & Catalog

Triple Fallback

Entity Relationships (from field_relationships)

Instance Relationships (from instance_relationships)

Cross-Section :ModelInstance Linking via instance_key

Normalized Models (Tabular Pipeline)

Versioning

Graph Structure Diagrams

Unstructured Pipeline Output

Tabular Pipeline Output

Versioning

In-place Update (update_mode=True)

Replacement (replaces="old_name")

Document Deletion

Document Freezing

Query Patterns

Find All Documents

Find All Entities of a Type

Find Entities Extracted from a Specific Document

Find Entity Relationships

Find Cross-Model Connections

Full Extraction Trace

Find Entity Usage Across Documents

Find Document Version History

Indexes and Constraints

Unique constraints

Single-property indexes

Composite tenant indexes

instance_key indexes

How the planner uses them

Community edition

Verifying index usage

Graph Size Estimation

Shell Nodes — Two Distinct Concepts

`:Document`

`:StructureNode`

`:LabeledEntity`

`:ModelInstance`

Entity Relationships (from `field_relationships`)

Instance Relationships (from `instance_relationships`)

Cross-Section `:ModelInstance` Linking via `instance_key`

In-place Update (`update_mode=True`)

Replacement (`replaces="old_name"`)

`instance_key` indexes