Skip to content

Freezing API

scinr.newton.ingest.freeze.freeze_document async 

freeze_document(
    path: str | None = None,
    version: int | None = None,
    *,
    tenant_id: str | None,
    created_by_user_id: str | Sequence[str] | None = None,
    job_id: str | Sequence[str] | None = None,
    keep_structure_nodes: bool = False,
    keep_annotations: bool = False,
    keep_extraction_results: bool = False,
    delete_after_export: bool = True,
) -> FreezeResult

Archive the subtree of the selected document(s) and reduce them to stubs.

Target selection is identical to delete_document(): tenant_id mandatory (keyword-only, no default; None / "__public__" = public documents), exactly one of path (all versions, or version) or job_id (one value or several), created_by_user_id as an extra AND filter, and the IS_COMPOSED_OF* cascade downwards from the matched documents (freezing a folder freezes its whole subtree; freezing a leaf by path leaves its folder and siblings alone).

Steps:

  1. Resolve the document set; found=False (nothing touched) if empty, :class:FreezeError if any of them is already frozen (except the stubs of an interrupted freeze — see below).
  2. Export the complete snapshot (always the whole subtree, whatever the keep_* flags) to the freeze backend. Fail-fast: if it fails, the graph is not touched.
  3. delete_after_export=False (backup mode): only set last_backup_blob_id / last_backup_at on the documents and stop.
  4. Otherwise mark every Document frozen=true with frozen_blob_id, frozen_at, the frozen_keep_* flags and frozen_cleanup_pending=true (one transaction), and then, in bounded transactions (ingest/_cascade.py): re-link kept ModelDecision / ExtractionResult nodes to their Document when their StructureNode goes, and delete the subtree (InfoUnits always; StructureNodes, annotations and extraction results unless kept). The Document nodes themselves stay.
  5. Delete the :Entity / :ModelInstance / :LabeledEntity nodes of the tenant this freeze leaves orphaned (ingest/_gc.py), and remove frozen_cleanup_pending. Only what hung from the deleted :ExtractionResult nodes is checked, not the whole tenant (that is collect_orphans()); a call that finishes an interrupted freeze has lost those candidates and sweeps the whole tenant instead.

The duration of each phase is logged (export = graph → snapshot file, upload, mark, delete, gc, clear_pending).

The subtree is not deleted atomically (one transaction holding the subtree of hundreds of documents overruns the server's transaction memory). If step 4 or 5 fails, the documents are already frozen stubs pointing at the complete snapshot, with part of their subtree still in the graph and frozen_cleanup_pending=true. Calling freeze_document() again on them finishes the deletion (no new export; the keep_* flags of the first call apply, the ones passed are ignored for those documents), and restore_document() brings them back whole.

Documental storage (raw files, converted pages) is not touched: a frozen document keeps its raw_file_id.

Parameters

keep_structure_nodes: Keep the whole :StructureNode tree (only :InfoUnit nodes go). keep_annotations: Keep :ModelDecision nodes and their children. keep_extraction_results: Keep :ExtractionResult nodes (and so what hangs from them within the GC hop bound). delete_after_export: False = backup only (no keep_* flag may be set).

Raises

TypeError If tenant_id is not passed. ValueError Invalid selector, or delete_after_export=False with a keep_* flag. FreezeError A selected document is already frozen, or the subtree violates the tenant invariant. ConfigurationError If the freeze backend resolves to "none".

scinr.newton.ingest.restore.restore_document async 

restore_document(
    path: str | None = None,
    version: int | None = None,
    *,
    tenant_id: str | None,
    created_by_user_id: str | Sequence[str] | None = None,
    job_id: str | Sequence[str] | None = None,
    frozen_blob_id: str | None = None,
    batch_size: int = DEFAULT_BATCH_SIZE,
    concurrency: int = DEFAULT_CONCURRENCY,
) -> RestoreResult

Rebuild document(s) from a snapshot: frozen stubs, deleted documents or backups.

The selector is the one of freeze_document() / delete_document() (tenant_id mandatory, exactly one of path / job_id, created_by_user_id as an AND filter). Where the snapshot comes from:

  • Frozen stubs (no frozen_blob_id, the selector matches Documents in the graph): the frozen ones are restored from their own snapshot, together with their IS_COMPOSED_OF* descendants frozen in the same snapshot (restoring a folder restores what freezing it froze). If none of them is frozen, :class:FreezeError — to restore a backup of a document that still exists, delete it first.
  • No Document in the graph (deleted, or never frozen and only backed up): the newest snapshot of the tenant holding each matched document is looked up in the freeze backend (:meth:FreezeRepository.find_snapshots), and the Documents are recreated from it — with their links to other Documents (folder, version chain) where those still exist, and latest recomputed.
  • An explicit frozen_blob_id: that snapshot, whatever the graph holds; its entries matching the selector (plus their descendants in the snapshot) are restored.

Either way, a selected Document that exists but is not frozen (or is frozen with another snapshot) raises :class:FreezeError before any write. A snapshot of a freeze is deleted from the backend once no Document of the tenant references it; backups and exports are kept.

Steps: ensure the lookup indexes the rebuild needs → download each snapshot (tenant-scoped) → validate it without writing (tenant of header, entries, nodes and Document links) → rebuild in batches → drop the temporary Document links, unmark the stubs → sweep the tenant's orphans if stale ExtractionResults had to be deleted (ingest/_gc.py) → delete snapshots no longer referenced. The duration of each phase is logged. Raw files are not part of snapshots: a recreated Document whose raw file was deleted keeps its raw_file_id and a warning is logged.

Parameters

frozen_blob_id: Restore from this snapshot instead of the stubs' / the newest one. batch_size: Rows per write transaction of the rebuild. Larger batches mean fewer round trips but longer transactions: more Neo4j heap per transaction and locks held longer (other writers — e.g. a running ingestion — wait on the shared catalog / entity nodes meanwhile). concurrency: Rebuild transactions in flight at once (each on its own pooled connection). Lower it (e.g. 1) while an ingestion is writing to the same Neo4j; Neo4j heap in use by the rebuild grows roughly with batch_size × concurrency.

Returns

RestoreResult found=False (nothing written) when neither a Document nor a snapshot matches.

Raises

TypeError If tenant_id is not passed. ValueError Invalid selector, or batch_size / concurrency below 1. FreezeError Nothing restorable (matched Documents not frozen), a selected Document exists unfrozen, a snapshot is missing / belongs to another tenant / is malformed, or the backend cannot look snapshots up. ConfigurationError If the freeze backend resolves to "none".

scinr.newton.ingest.freeze.export_document_snapshot async 

export_document_snapshot(
    path: str | None = None,
    version: int | None = None,
    *,
    tenant_id: str | None,
    created_by_user_id: str | Sequence[str] | None = None,
    job_id: str | Sequence[str] | None = None,
    destination: Literal[
        "dict", "file", "storage"
    ] = "dict",
    file_path: Path | str | None = None,
) -> dict | Path | str

Serialise the subtree of the selected document(s) to a snapshot.

Read-only: the graph is never modified. The selector is the one of :func:freeze_document (and delete_document()): tenant_id mandatory, exactly one of path / job_id, the IS_COMPOSED_OF* descendants of the matched documents included.

Parameters

destination: - "dict" (default): return the snapshot as a dict, built in memory — for small documents and debugging. - "file": stream it to file_path and return that Path. - "storage": stream it to a temporary file, upload it to the freeze backend (get_freeze_storage()) and return the frozen_blob_id. Only this destination needs a freeze backend. file_path: Destination file; required with destination="file" only.

Raises

FreezeError If nothing matches, a selected document is frozen (its subtree is no longer in the graph — restore it first), or a node of the subtree belongs to another tenant. ConfigurationError With destination="storage" when the freeze backend resolves to "none". ValueError Invalid selector, unknown destination, or file_path missing / given without destination="file".

scinr.newton.freeze.base.FreezeRepository

Bases: ABC

Stores the JSON snapshots of frozen (or backed-up) documents.

delete_snapshot abstractmethod async 

delete_snapshot(
    frozen_blob_id: str, *, tenant_id: str
) -> None

Delete the snapshot frozen_blob_id (binary and metadata).

Must be idempotent: when the snapshot does not exist (already deleted, invalid id, repeated call) or belongs to another tenant, it logs and returns without raising — same contract as :meth:~scinr.newton.storage.base.RawFileRepository.delete.

Parameters

frozen_blob_id: Id returned by :meth:store_snapshot. tenant_id: Stored tenant key of the frozen document.

find_snapshots async 

find_snapshots(
    *,
    tenant_id: str,
    path: str | None = None,
    version: int | None = None,
    job_id: Sequence[str] | None = None,
    created_by_user_id: Sequence[str] | None = None,
) -> list[SnapshotRecord]

List the snapshots of tenant_id holding a matching document, newest first.

Used by restore_document() when the selected document has no frozen stub in the graph (it was deleted, or the snapshot is a backup). A document matches when one entry of the snapshot's documents metadata satisfies every filter given (path / version, or job_id; created_by_user_id on top).

Optional for custom backends: the default raises :class:~scinr.newton.exceptions.FreezeError, and callers must then name the snapshot explicitly (restore_document(frozen_blob_id=...)).

Parameters

tenant_id: Stored tenant key. job_id, created_by_user_id: Lists of values (matched with IN), as delete_document() takes them.

read_snapshot_to_file abstractmethod async 

read_snapshot_to_file(
    frozen_blob_id: str, dest_path: Path, *, tenant_id: str
) -> bool

Download the snapshot frozen_blob_id to dest_path in streaming.

Parameters

frozen_blob_id: Id returned by :meth:store_snapshot. dest_path: Local file to write (created or truncated). tenant_id: Stored tenant key of the frozen document.

Returns

bool True when the snapshot was written. False — and nothing written — when it does not exist, the id is invalid for the backend, or it belongs to another tenant (indistinguishable).

store_snapshot abstractmethod async 

store_snapshot(
    path: Path,
    *,
    tenant_id: str | None,
    created_by_user_id: str | Sequence[str] | None = None,
    job_id: str | Sequence[str] | None = None,
    metadata: dict[str, Any],
) -> str

Upload the snapshot file at path and return its frozen_blob_id.

The file is already fully written; backends must upload it in streaming (never read it whole into memory) — see MongoDBRawFileRepository.store_file for the reference pattern.

Parameters

path: Local JSON snapshot file. tenant_id: Owning tenant (None or "__public__" = public), stored as :func:~scinr.newton.utils.tenancy.tenant_key. Mandatory: a snapshot belongs to a single tenant. created_by_user_id, job_id: The freeze selector's provenance filters (one value or several), kept for audit / listing. metadata: Descriptive fields about the snapshot content (documents, schema_version, frozen_at, keep_flags…), stored as is.

Returns

str The frozen_blob_id assigned by the backend.

scinr.newton.freeze.base.SnapshotRecord dataclass

A stored snapshot, as :meth:FreezeRepository.find_snapshots lists it.

Attributes:

Name Type Description
frozen_blob_id str

Id to read / delete the snapshot with.
mode str | None

"freeze", "backup" or "export" (the call that stored it).
frozen_at str | None

ISO timestamp of the export.
documents list[dict[str, Any]]

One {"path", "version", "job_id", "created_by_user_id"} dict per document entry of the snapshot (job_id / created_by_user_id only in snapshots stored since they were recorded).

scinr.newton.freeze.factory.get_freeze_storage 

get_freeze_storage() -> FreezeRepository

Return the freeze repository for the configured backend.

Raises

ConfigurationError If the backend resolved to 'none', is unknown, or is 'custom' without custom_freeze_storage. StorageError If backend='mongodb' but the MongoDB server is unreachable.

Notes

With backend='mongodb' the first call per process (and per MongoDB target) pings the server and creates the indexes — the same readiness check as :func:~scinr.newton.storage.factory.get_storage, shared with it.