Storage API

Document storage backends for raw file and page content archival.

Backend Factory

scinr.newton.storage.factory

storage/factory.py — Repository factory.

Call get_storage() to obtain the configured pair of repository implementations. The backend is determined by ScinrConfig (from scinr_config) which reads STORAGE_BACKEND from the environment or from configure().

get_storage

get_storage() -> tuple[RawFileRepository, PageRepository]

Return the repository implementations for the configured backend.

Returns

tuple[RawFileRepository, PageRepository] A (raw_file_repo, page_repo) pair ready to use.

Raises

ConfigurationError If the storage backend is unknown or misconfigured. If backend='custom' but custom_storage was not provided. StorageError If backend='mongodb' but the MongoDB server is unreachable.

Notes

With backend='mongodb' the first call per process (and per MongoDB target) pings the server and, unless mongodb_ensure_indexes=False, creates the required indexes. Later calls skip both.

reset_mongodb_readiness

reset_mongodb_readiness() -> None

Forget which MongoDB targets were already verified in this process.

Called by :func:~scinr.newton.storage.mongodb.client.reset_client (hence by configure()), so a configuration change re-checks the connection and the indexes.

Base Classes

scinr.newton.storage.base

storage/base.py — Abstract repository interfaces.

All storage backends must implement these ABCs so that the rest of the pipeline can remain backend-agnostic.

Multi-tenancy

Every record carries the same tenant / provenance fields as the graph:

tenant_id — stored value (:func:~scinr.newton.utils.tenancy.tenant_key): the tenant, or "__public__" for a public upload — never null.
created_by_user_id / job_id — provenance of the upload (scalars).

Writes take tenant_id in its public-API form: None and "__public__" both mean public and are stored as "__public__".

Reads and deletes take the same four keyword-only filters as the graph navigation API (see :mod:scinr.newton.utils.scope)::

tenant_id: str | None = None                            # None = all tenants
include_public: bool = False                            # add the public records
created_by_user_id: str | Sequence[str] | None = None   # IN
job_id: str | Sequence[str] | None = None               # IN

The library does not impose a restriction — tenant_id=None reads across every tenant (legacy records without a tenant included). A multi-tenant API layer should always pass its tenant. A record outside the requested scope behaves exactly as a record that does not exist.

PageRepository

Bases: ABC

Almacena las páginas convertidas (markdown) de un documento.

delete_pages `abstractmethod` `async`

delete_pages(
    raw_file_id: str,
    *,
    tenant_id: str | None = None,
    include_public: bool = False,
    created_by_user_id: str | Sequence[str] | None = None,
    job_id: str | Sequence[str] | None = None,
) -> int

Borra las páginas asociadas a raw_file_id dentro del scope.

No es un error que no existan páginas para ese raw_file_id (devuelve 0 en ese caso, sin lanzar excepción).

Parameters

raw_file_id: ID del :class:~storage.models.RawFileRecord cuyas páginas se quieren borrar. tenant_id, include_public, created_by_user_id, job_id: Filtros de scope (ver el docstring del módulo).

Returns

int Número de páginas borradas (0 si no había ninguna).

Raises

ScopeError Si los filtros son inválidos.

get_pages `abstractmethod` `async`

get_pages(
    raw_file_id: str,
    *,
    tenant_id: str | None = None,
    include_public: bool = False,
    created_by_user_id: str | Sequence[str] | None = None,
    job_id: str | Sequence[str] | None = None,
) -> list[ConvertedPageRecord]

Recupera las páginas de un fichero por su raw_file_id, dentro del scope.

Parameters

raw_file_id: ID del :class:~storage.models.RawFileRecord cuyas páginas se quieren recuperar. tenant_id, include_public, created_by_user_id, job_id: Filtros de scope (ver el docstring del módulo).

Returns

list[ConvertedPageRecord] Lista ordenada por page_index ascendente. Puede estar vacía si aún no se han almacenado páginas o si quedan fuera del scope.

Raises

ScopeError Si los filtros son inválidos.

get_pages_by_ids `abstractmethod` `async`

get_pages_by_ids(
    page_ids: Sequence[str],
    *,
    tenant_id: str | None = None,
    include_public: bool = False,
    created_by_user_id: str | Sequence[str] | None = None,
    job_id: str | Sequence[str] | None = None,
) -> list[ConvertedPageRecord]

Recupera las páginas cuyos page_id están en page_ids, dentro del scope.

Es la lectura directa del texto de un nodo de estructura (sus source_page_ids), sin cargar el resto de páginas del documento.

Parameters

page_ids: page_id devueltos por :meth:store_page. Los que no existen, quedan fuera del scope o no son válidos para el backend se ignoran. Con una lista vacía devuelve []. tenant_id, include_public, created_by_user_id, job_id: Filtros de scope (ver el docstring del módulo).

Returns

list[ConvertedPageRecord] Las páginas encontradas, ordenadas por page_index ascendente.

Raises

ScopeError Si los filtros son inválidos.

store_page `abstractmethod` `async`

store_page(
    raw_file_id: str,
    filename: str,
    folder_path: str | None,
    page_index: int,
    markdown: str,
    *,
    tenant_id: str | None = None,
    created_by_user_id: str | None = None,
    job_id: str | None = None,
) -> str

Persiste una página y devuelve su page_id (str).

Parameters

raw_file_id: ID del :class:~storage.models.RawFileRecord al que pertenece esta página. filename: Stem del fichero sin extensión (p.ej. "3.2.P.1"). folder_path: Ruta relativa de la carpeta contenedora, o None. page_index: Índice 0-based de la página, idéntico a :attr:~converters.base.IntermediatePage.index. markdown: Texto completo de la página en formato Markdown. tenant_id, created_by_user_id, job_id: Igual que en :meth:RawFileRepository.store (deben coincidir con los del fichero original).

Returns

str El page_id: identificador único de la página persistida.

RawFileRepository

Bases: ABC

Almacena el fichero original tal cual fue recibido (binario).

delete `abstractmethod` `async`

delete(
    raw_file_id: str,
    *,
    tenant_id: str | None = None,
    include_public: bool = False,
    created_by_user_id: str | Sequence[str] | None = None,
    job_id: str | Sequence[str] | None = None,
) -> None

Borra el fichero binario (y su metadata) identificado por raw_file_id.

Solo borra si el registro está dentro del scope pedido; fuera de él se comporta como si no existiera.

Debe ser idempotente: si raw_file_id no existe (ya borrado, ID inválido, fuera de scope, o llamada repetida), no debe lanzar excepción — solo loggear y no hacer nada.

Parameters

raw_file_id: ID del :class:~storage.models.RawFileRecord a borrar. tenant_id, include_public, created_by_user_id, job_id: Filtros de scope (ver el docstring del módulo).

Raises

ScopeError Si los filtros son inválidos.

get `abstractmethod` `async`

get(
    raw_file_id: str,
    *,
    tenant_id: str | None = None,
    include_public: bool = False,
    created_by_user_id: str | Sequence[str] | None = None,
    job_id: str | Sequence[str] | None = None,
) -> RawFileRecord | None

Devuelve la metadata de raw_file_id, o None si no existe o queda fuera del scope pedido (ambos casos son indistinguibles).

Un raw_file_id inválido para el backend también devuelve None.

Raises

ScopeError Si los filtros son inválidos (tenant_id="", lista vacía).

list_raw_files `abstractmethod` `async`

list_raw_files(
    *,
    tenant_id: str | None = None,
    include_public: bool = False,
    created_by_user_id: str | Sequence[str] | None = None,
    job_id: str | Sequence[str] | None = None,
    folder_path: str | None = None,
    filename: str | None = None,
) -> list[RawFileRecord]

Inventario de ficheros originales dentro del scope, ordenado por stored_at ascendente.

folder_path / filename filtran por igualdad exacta (None = sin filtro).

Raises

ScopeError Si los filtros son inválidos.

open `abstractmethod` `async`

open(
    raw_file_id: str,
    *,
    tenant_id: str | None = None,
    include_public: bool = False,
    created_by_user_id: str | Sequence[str] | None = None,
    job_id: str | Sequence[str] | None = None,
) -> AsyncIterator[bytes] | None

Abre el binario original de raw_file_id para leerlo en streaming.

Devuelve un iterador asíncrono de bloques de bytes (async for chunk in stream), o None si el registro no existe o queda fuera del scope (mismas reglas que :meth:get). El backend debe localizar el binario a partir del registro ya filtrado, nunca a partir de un id recibido del llamador.

Raises

ScopeError Si los filtros son inválidos. StorageError Si el backend no guarda binarios.

open_with_record `abstractmethod` `async`

open_with_record(
    raw_file_id: str,
    *,
    tenant_id: str | None = None,
    include_public: bool = False,
    created_by_user_id: str | Sequence[str] | None = None,
    job_id: str | Sequence[str] | None = None,
) -> tuple[RawFileRecord, AsyncIterator[bytes]] | None

Como :meth:open, pero devuelve también la metadata del registro: (record, stream) con una sola lectura del registro, o None si no existe o queda fuera del scope.

Raises

ScopeError Si los filtros son inválidos. StorageError Si el backend no guarda binarios, o el binario del registro falta.

store `abstractmethod` `async`

store(
    filename: str,
    content: bytes,
    content_type: str,
    folder_path: str | None,
    *,
    tenant_id: str | None = None,
    created_by_user_id: str | None = None,
    job_id: str | None = None,
) -> str

Persiste el fichero y devuelve su raw_file_id (str).

Parameters

filename: Nombre del fichero original (p.ej. "3.2.P.1.pdf"). content: Contenido binario del fichero. content_type: MIME type del fichero (p.ej. "application/pdf"). folder_path: Ruta relativa de la carpeta contenedora desde la raíz de ingesta, o None si el fichero está en la raíz. tenant_id: Tenant propietario (None o "__public__" = público). Se guarda como :func:~scinr.newton.utils.tenancy.tenant_key. created_by_user_id, job_id: Procedencia de la subida (se guardan tal cual; None = sin dato).

Returns

str El raw_file_id: representación en cadena del identificador único asignado por el backend de almacenamiento.

Notes

Los backends propios pueden sobrescribir además :meth:store_file para subir el fichero en streaming sin materializarlo en memoria.

store_file `async`

store_file(
    path: Path,
    filename: str,
    content_type: str,
    folder_path: str | None,
    *,
    tenant_id: str | None = None,
    created_by_user_id: str | None = None,
    job_id: str | None = None,
) -> str

Persiste el fichero situado en path y devuelve su raw_file_id.

Implementación por defecto: lee el fichero completo y delega en :meth:store. Los backends capaces de hacer streaming (p.ej. MongoDB/GridFS) la sobrescriben para no mantener el fichero entero en memoria. No es abstracto para no romper backends propios que solo implementen :meth:store.

Parameters

path: Ruta del fichero a persistir. filename, content_type, folder_path, tenant_id, created_by_user_id, job_id: Igual que en :meth:store.

Returns

str El raw_file_id asignado por el backend.

Scope Filters

The read / delete scope is shared with the navigation API (scinr.newton.utils.scope); this module renders it for MongoDB.

scinr.newton.storage.filters

storage/filters.py — Render the shared read scope as a MongoDB filter.

The scope (tenant_id / include_public / created_by_user_id / job_id) is validated and resolved by :func:scinr.newton.utils.scope.make_scope — the very same function the graph navigation API uses — so storage reads and deletes filter exactly like the navigation (None = no filter, a list = IN).

mongo_scope_filter

mongo_scope_filter(
    tenant_id: str | None = None,
    include_public: bool = False,
    created_by_user_id: str | Sequence[str] | None = None,
    job_id: str | Sequence[str] | None = None,
) -> dict[str, Any]

Validate the four public filters and render them for MongoDB.

Raises:

Type	Description
`ScopeError`	tenant_id is empty, or a list filter is empty.

scope_to_mongo

scope_to_mongo(scope: Scope) -> dict[str, Any]

Return the MongoDB filter fragment for scope ({} when unfiltered).

Records written before multi-tenancy have no tenant_id: any tenant filter excludes them, so they are only reachable with tenant_id=None.

scinr.newton.utils.scope

utils/scope.py — The read-side scope: tenant, user and job filters.

Shared by the graph navigation API (navigation.scope) and the storage repositories (storage.base), so both layers accept the same four keyword-only arguments with the same semantics and the same errors::

tenant_id: str | None = None                            # None = all tenants
include_public: bool = False                            # add the public documents
created_by_user_id: str | Sequence[str] | None = None   # IN
job_id: str | Sequence[str] | None = None               # IN

They are turned into a :class:Scope by :func:make_scope; each backend then renders it as plain, parametrized predicates on the denormalized properties written at ingestion (see utils/tenancy.py) — Cypher via :meth:Scope.clauses, MongoDB via storage.filters.scope_to_mongo.

=============== =============== =========================================== tenant_id include_public Predicate on the node / record =============== =============== =========================================== None any none (all tenants, legacy records included) "__public__" any tenant_id = '__public__' "acme" False tenant_id = 'acme' "acme" True tenant_id IN ['acme', '__public__'] "" — :class:~scinr.newton.exceptions.ScopeError =============== =============== ===========================================

created_by_user_id / job_id accept one value or several (IN): OR inside each filter, AND between filters and with the tenant. An empty list is an error (it would be ambiguous between "no filter" and "match nothing").

Scope `dataclass`

Resolved read scope. None in any field means "no filter".

clauses

clauses(
    alias: str,
    kind: ScopeKind = "scalar",
    *,
    tenant: bool = True,
) -> list[str]

WHERE fragments for alias; parameters come from :meth:params.

params

params(
    *, tenant: bool = True, provenance: bool = True
) -> dict[str, Any]

Query parameters for :meth:clauses (only the ones in use).

with_tenant

with_tenant(tenant: str) -> Scope

Same scope pinned to a single (already stored) tenant.

make_scope

make_scope(
    tenant_id: str | None = None,
    include_public: bool = False,
    created_by_user_id: str | Sequence[str] | None = None,
    job_id: str | Sequence[str] | None = None,
) -> Scope

Validate the four public filters and resolve them into a :class:Scope.

Raises:

Type	Description
`ScopeError`	tenant_id is empty / not a string, or a list filter is empty.

MongoDB Backend

Client & GridFS

scinr.newton.storage.mongodb.client

storage/mongodb/client.py — Motor async client singleton + GridFS access.

A single :class:~motor.motor_asyncio.AsyncIOMotorClient instance is kept per process. All repositories in this backend share it to avoid exhausting connection-pool resources.

Public API

get_client() Return the singleton Motor client (creates it on first call). get_db() Return the configured Motor database object. get_gridfs_bucket(bucket_name=None) Return an :class:~motor.motor_asyncio.AsyncIOMotorGridFSBucket for binary file storage (raw files by default; the freeze backend passes its own snapshot bucket). ensure_indexes() Coroutine — create all required indexes (idempotent). ensure_indexes_sync() Same indexes with a synchronous pymongo client. Called automatically by :func:~scinr.newton.storage.factory.get_storage once per process.

ensure_indexes `async`

ensure_indexes() -> None

Create all required MongoDB indexes (idempotent).

Uses Motor's create_index which maps to a createIndex command that is a no-op if the index already exists with the same key pattern and name.

:func:~scinr.newton.storage.factory.get_storage already creates the indexes (synchronously, once per process) unless mongodb_ensure_indexes=False; call this only when managing them explicitly from async code.

ensure_indexes_sync

ensure_indexes_sync(cfg, client=None) -> None

Create all required MongoDB indexes with a synchronous pymongo client.

Same indexes as :func:ensure_indexes. pymongo is used instead of Motor because this runs from :func:~scinr.newton.storage.factory.get_storage, which is called inside running event loops, and a Motor client binds to the first loop it is used on.

Parameters

cfg : ScinrConfig Configuration providing the URI, database and collection names. client : pymongo.MongoClient, optional Client to reuse. When omitted a temporary one is created and closed.

Raises

pymongo.errors.PyMongoError If an index cannot be created (e.g. OperationFailure when the user lacks the createIndex privilege).

get_client

get_client() -> AsyncIOMotorClient

Return the Motor singleton client, creating it on first call.

The client is module-level so that it survives across coroutine calls within the same process and reuses the underlying connection pool.

Returns

AsyncIOMotorClient The shared Motor client instance.

get_db

get_db()

Return the Motor database object for the configured database.

Returns

AsyncIOMotorDatabase The Motor database identified by cfg.mongodb_database.

get_gridfs_bucket

get_gridfs_bucket(
    bucket_name: str | None = None,
) -> AsyncIOMotorGridFSBucket

Return a GridFS bucket for binary file storage.

Parameters

bucket_name: Bucket to open. None (default) opens the raw-files bucket, cfg.mongodb_gridfs_bucket (default "raw_binaries").

Returns

AsyncIOMotorGridFSBucket GridFS bucket ready for upload/download operations.

reset_client

reset_client() -> None

Reset the singleton Motor client.

Forces :func:get_client to create a new client on the next call. Must be called whenever the configuration changes (e.g. after :func:~scinr.newton.config.configure) so that the new URI and database settings are picked up. Also clears the per-process cache of verified connections, so the next :func:get_storage call pings the server and ensures the indexes again.

Page Repository

scinr.newton.storage.mongodb.pages

storage/mongodb/pages.py — MongoDB implementation of PageRepository.

Converted pages (Markdown text) are stored as plain documents in the converted_pages collection. Each document maps 1-to-1 to an :class:~storage.models.ConvertedPageRecord and references its parent raw file via raw_file_id.

Every page carries the stored tenant_id ("__public__" for public uploads), created_by_user_id and job_id of its upload; reads and deletes filter by the requested scope (see :mod:scinr.newton.storage.base).

MongoDBPageRepository

Bases: PageRepository

Stores and retrieves converted pages in the converted_pages collection.

Each document in the collection represents a single page of a converted document. Pages are ordered by page_index which mirrors :attr:~converters.base.IntermediatePage.index.

delete_pages `async`

delete_pages(
    raw_file_id: str,
    *,
    tenant_id: str | None = None,
    include_public: bool = False,
    created_by_user_id: str | Sequence[str] | None = None,
    job_id: str | Sequence[str] | None = None,
) -> int

Delete the converted pages of raw_file_id inside the scope.

Not an error if no pages exist for this raw_file_id — returns 0 in that case rather than raising.

Parameters

raw_file_id: ID of the :class:~storage.models.RawFileRecord whose pages should be deleted. tenant_id, include_public, created_by_user_id, job_id: Scope filters (see :mod:scinr.newton.storage.base).

Returns

int Number of pages deleted (0 if none matched).

get_pages `async`

get_pages(
    raw_file_id: str,
    *,
    tenant_id: str | None = None,
    include_public: bool = False,
    created_by_user_id: str | Sequence[str] | None = None,
    job_id: str | Sequence[str] | None = None,
) -> list[ConvertedPageRecord]

Retrieve the pages of a raw file inside the scope, ordered by page index.

Parameters

raw_file_id: ID of the :class:~storage.models.RawFileRecord whose pages should be retrieved. tenant_id, include_public, created_by_user_id, job_id: Scope filters (see :mod:scinr.newton.storage.base).

Returns

list[ConvertedPageRecord] Pages sorted by page_index ascending. Returns an empty list if no pages have been stored for this raw_file_id or they lie outside the scope.

get_pages_by_ids `async`

get_pages_by_ids(
    page_ids: Sequence[str],
    *,
    tenant_id: str | None = None,
    include_public: bool = False,
    created_by_user_id: str | Sequence[str] | None = None,
    job_id: str | Sequence[str] | None = None,
) -> list[ConvertedPageRecord]

Retrieve the pages whose page_id is in page_ids, inside the scope, ordered by page index.

A single _id $in lookup (default _id index). Ids that are not valid ObjectIds, do not exist or lie outside the scope are skipped.

store_page `async`

store_page(
    raw_file_id: str,
    filename: str,
    folder_path: str | None,
    page_index: int,
    markdown: str,
    *,
    tenant_id: str | None = None,
    created_by_user_id: str | None = None,
    job_id: str | None = None,
) -> str

Persist a single converted page in MongoDB.

Parameters

raw_file_id: ID of the parent :class:~storage.models.RawFileRecord. filename: Stem of the source file without extension, e.g. "3.2.P.1". folder_path: Relative path of the containing folder, or None. page_index: Zero-based page index. markdown: Full Markdown text of this page. tenant_id: Owning tenant (None / "__public__" = public), stored as :func:~scinr.newton.utils.tenancy.tenant_key. created_by_user_id, job_id: Provenance of the upload, stored verbatim.

Returns

str The page_id: str(ObjectId) of the newly inserted document.

Raw File Repository

scinr.newton.storage.mongodb.raw_files

storage/mongodb/raw_files.py — MongoDB/GridFS implementation of RawFileRepository.

Binary content is stored in GridFS to support files of arbitrary size (including PDFs larger than the 16 MB BSON document limit). A lightweight metadata document is inserted into the raw_files collection so that records can be queried by checksum, filename, or folder path without fetching the full binary from GridFS.

Both the raw_files document and the GridFS metadata carry the stored tenant_id ("__public__" for public uploads), created_by_user_id and job_id. Every read / delete filters the raw_files document by the requested scope first and only then touches GridFS, through the gridfs_id of that already-filtered record.

MongoDBRawFileRepository

Bases: RawFileRepository

Stores binary files in GridFS with metadata in raw_files.

GridFS splits large files into 255 kB chunks and stores them across two internal collections (<bucket>.files and <bucket>.chunks), removing the 16 MB BSON size constraint.

The raw_files collection holds only metadata plus a gridfs_id reference so callers can retrieve the binary when needed.

delete `async`

delete(
    raw_file_id: str,
    *,
    tenant_id: str | None = None,
    include_public: bool = False,
    created_by_user_id: str | Sequence[str] | None = None,
    job_id: str | Sequence[str] | None = None,
) -> None

Delete a raw file's binary (GridFS) and metadata (raw_files).

Only a record inside the requested scope is deleted; outside it the record is treated as missing.

Idempotent: if raw_file_id is not a valid ObjectId, or no matching metadata document is found (already deleted, out of scope, or a repeated call), this logs a warning and returns without raising. If the metadata document exists but its GridFS binary is already gone (gridfs.errors.NoFile), that is also logged and swallowed — the metadata document is still deleted.

Parameters

raw_file_id: The raw_file_id (str(ObjectId)) to delete. tenant_id, include_public, created_by_user_id, job_id: Scope filters (see :mod:scinr.newton.storage.base).

get `async`

get(
    raw_file_id: str,
    *,
    tenant_id: str | None = None,
    include_public: bool = False,
    created_by_user_id: str | Sequence[str] | None = None,
    job_id: str | Sequence[str] | None = None,
) -> RawFileRecord | None

Return the metadata of raw_file_id, or None if it does not exist or lies outside the requested scope.

list_raw_files `async`

list_raw_files(
    *,
    tenant_id: str | None = None,
    include_public: bool = False,
    created_by_user_id: str | Sequence[str] | None = None,
    job_id: str | Sequence[str] | None = None,
    folder_path: str | None = None,
    filename: str | None = None,
) -> list[RawFileRecord]

Inventory of the raw files inside the scope, oldest first.

open `async`

open(
    raw_file_id: str,
    *,
    tenant_id: str | None = None,
    include_public: bool = False,
    created_by_user_id: str | Sequence[str] | None = None,
    job_id: str | Sequence[str] | None = None,
) -> AsyncIterator[bytes] | None

Stream the original binary of raw_file_id in GridFS chunks.

Returns None when the record does not exist or lies outside the scope. The GridFS stream is opened through the gridfs_id of the already-filtered record; a record whose binary is gone raises :class:~scinr.newton.exceptions.StorageError.

open_with_record `async`

open_with_record(
    raw_file_id: str,
    *,
    tenant_id: str | None = None,
    include_public: bool = False,
    created_by_user_id: str | Sequence[str] | None = None,
    job_id: str | Sequence[str] | None = None,
) -> tuple[RawFileRecord, AsyncIterator[bytes]] | None

Metadata and stream of raw_file_id from a single raw_files lookup (same rules as :meth:get and :meth:open).

store `async`

store(
    filename: str,
    content: bytes,
    content_type: str,
    folder_path: str | None,
    *,
    tenant_id: str | None = None,
    created_by_user_id: str | None = None,
    job_id: str | None = None,
) -> str

Upload a binary file to GridFS and persist its metadata.

Parameters

filename: Original filename including extension, e.g. "3.2.P.1.pdf". content: Raw binary content of the file. content_type: MIME type, e.g. "application/pdf". folder_path: Relative path of the containing folder from the ingestion root, or None for files at the root. tenant_id: Owning tenant (None / "__public__" = public), stored as :func:~scinr.newton.utils.tenancy.tenant_key. created_by_user_id, job_id: Provenance of the upload, stored verbatim.

Returns

str The raw_file_id: str(ObjectId) of the newly inserted document in the raw_files collection.

store_file `async`

store_file(
    path: Path,
    filename: str,
    content_type: str,
    folder_path: str | None,
    *,
    tenant_id: str | None = None,
    created_by_user_id: str | None = None,
    job_id: str | None = None,
) -> str

Stream the file at path to GridFS without loading it into memory.

Motor runs upload_from_stream in an executor thread, so the reads from the file handle happen off the event loop and in chunk_size blocks. SHA-256 and size are computed on the fly by :class:_HashingReader.

Parameters

path: Path of the file to upload. filename, content_type, folder_path, tenant_id, created_by_user_id, job_id: Same meaning as in :meth:store.

Returns

str The raw_file_id: str(ObjectId) of the metadata document.

Null Backend

scinr.newton.storage.null

storage/null.py — No-op storage repositories for when storage_backend='none'.

These implementations satisfy the RawFileRepository and PageRepository interfaces without performing any I/O. They are used as the default when no storage backend is configured, eliminating the need for None checks throughout the codebase.

Arguments are still validated like in a real backend (an empty tenant_id on write, an invalid read scope) so that a misuse does not go unnoticed only because storage is disabled.

NullPageRepository

Bases: PageRepository

No-op implementation. All writes are silently discarded.

NullRawFileRepository

Bases: RawFileRepository

No-op implementation. All writes are silently discarded.

Storage API

Backend Factory

scinr.newton.storage.factory

get_storage

Returns

Raises

Notes

reset_mongodb_readiness

Base Classes

scinr.newton.storage.base

Multi-tenancy

PageRepository

delete_pages abstractmethod async

Parameters

Returns

Raises

get_pages abstractmethod async

Parameters

Returns

Raises

get_pages_by_ids abstractmethod async

Parameters

Returns

Raises

store_page abstractmethod async

Parameters

Returns

RawFileRepository

delete abstractmethod async

Parameters

Raises

get abstractmethod async

Raises

list_raw_files abstractmethod async

Raises

open abstractmethod async

Raises

open_with_record abstractmethod async

Raises

store abstractmethod async

Parameters

Returns

Notes

store_file async

Parameters

Returns

Scope Filters

scinr.newton.storage.filters

mongo_scope_filter

scope_to_mongo

scinr.newton.utils.scope

Scope dataclass

clauses

params

with_tenant

make_scope

MongoDB Backend

Client & GridFS

scinr.newton.storage.mongodb.client

Public API

ensure_indexes async

ensure_indexes_sync

Parameters

Raises

get_client

Returns

get_db

Returns

get_gridfs_bucket

Parameters

Returns

reset_client

Page Repository

scinr.newton.storage.mongodb.pages

MongoDBPageRepository

delete_pages async

Parameters

Returns

get_pages async

Parameters

delete_pages `abstractmethod` `async`

get_pages `abstractmethod` `async`

get_pages_by_ids `abstractmethod` `async`

store_page `abstractmethod` `async`

delete `abstractmethod` `async`

get `abstractmethod` `async`

list_raw_files `abstractmethod` `async`

open `abstractmethod` `async`

open_with_record `abstractmethod` `async`

store `abstractmethod` `async`

store_file `async`

Scope `dataclass`

ensure_indexes `async`

delete_pages `async`

get_pages `async`

get_pages_by_ids `async`

store_page `async`

delete `async`

get `async`

list_raw_files `async`

open `async`

open_with_record `async`

store `async`

store_file `async`