Skip to content

Storage API

Document storage backends for raw file and page content archival.

Backend Factory

scinr.newton.storage.factory

storage/factory.py — Repository factory.

Call get_storage() to obtain the configured pair of repository implementations. The backend is determined by ScinrConfig (from scinr_config) which reads STORAGE_BACKEND from the environment or from configure().

get_storage 

get_storage() -> tuple[RawFileRepository, PageRepository]

Return the repository implementations for the configured backend.

Returns

tuple[RawFileRepository, PageRepository] A (raw_file_repo, page_repo) pair ready to use.

Raises

ConfigurationError If the storage backend is unknown or misconfigured. If backend='custom' but custom_storage was not provided. StorageError If backend='mongodb' but the MongoDB server is unreachable.

Base Classes

scinr.newton.storage.base

storage/base.py — Abstract repository interfaces.

All storage backends must implement these ABCs so that the rest of the pipeline can remain backend-agnostic.

PageRepository

Bases: ABC

Almacena las páginas convertidas (markdown) de un documento.

delete_pages abstractmethod async 

delete_pages(raw_file_id: str) -> int

Borra todas las páginas asociadas a raw_file_id.

No es un error que no existan páginas para ese raw_file_id (devuelve 0 en ese caso, sin lanzar excepción).

Parameters

raw_file_id: ID del :class:~storage.models.RawFileRecord cuyas páginas se quieren borrar.

Returns

int Número de páginas borradas (0 si no había ninguna).

get_pages abstractmethod async 

get_pages(raw_file_id: str) -> list[ConvertedPageRecord]

Recupera todas las páginas de un fichero por su raw_file_id.

Parameters

raw_file_id: ID del :class:~storage.models.RawFileRecord cuyas páginas se quieren recuperar.

Returns

list[ConvertedPageRecord] Lista ordenada por page_index ascendente. Puede estar vacía si aún no se han almacenado páginas.

store_page abstractmethod async 

store_page(
    raw_file_id: str,
    filename: str,
    folder_path: str | None,
    page_index: int,
    markdown: str,
) -> str

Persiste una página y devuelve su page_id (str).

Parameters

raw_file_id: ID del :class:~storage.models.RawFileRecord al que pertenece esta página. filename: Stem del fichero sin extensión (p.ej. "3.2.P.1"). folder_path: Ruta relativa de la carpeta contenedora, o None. page_index: Índice 0-based de la página, idéntico a :attr:~converters.base.IntermediatePage.index. markdown: Texto completo de la página en formato Markdown.

Returns

str El page_id: identificador único de la página persistida.

RawFileRepository

Bases: ABC

Almacena el fichero original tal cual fue recibido (binario).

delete abstractmethod async 

delete(raw_file_id: str) -> None

Borra el fichero binario (y su metadata) identificado por raw_file_id.

Debe ser idempotente: si raw_file_id no existe (ya borrado, ID inválido, o llamada repetida), no debe lanzar excepción — solo loggear y no hacer nada.

Parameters

raw_file_id: ID del :class:~storage.models.RawFileRecord a borrar.

store abstractmethod async 

store(
    filename: str,
    content: bytes,
    content_type: str,
    folder_path: str | None,
) -> str

Persiste el fichero y devuelve su raw_file_id (str).

Parameters

filename: Nombre del fichero original (p.ej. "3.2.P.1.pdf"). content: Contenido binario del fichero. content_type: MIME type del fichero (p.ej. "application/pdf"). folder_path: Ruta relativa de la carpeta contenedora desde la raíz de ingesta, o None si el fichero está en la raíz.

Returns

str El raw_file_id: representación en cadena del identificador único asignado por el backend de almacenamiento.

MongoDB Backend

Client & GridFS

scinr.newton.storage.mongodb.client

storage/mongodb/client.py — Motor async client singleton + GridFS access.

A single :class:~motor.motor_asyncio.AsyncIOMotorClient instance is kept per process. All repositories in this backend share it to avoid exhausting connection-pool resources.

Public API

get_client() Return the singleton Motor client (creates it on first call). get_db() Return the configured Motor database object. get_gridfs_bucket() Return an :class:~motor.motor_asyncio.AsyncIOMotorGridFSBucket for binary file storage. ensure_indexes() Coroutine — create all required indexes (idempotent). Call once at application startup.

ensure_indexes async 

ensure_indexes() -> None

Create all required MongoDB indexes (idempotent).

Uses Motor's create_index which maps to a createIndex command that is a no-op if the index already exists with the same key pattern and name.

Should be called once at application startup to guarantee optimal query performance from the first request.

get_client 

get_client() -> AsyncIOMotorClient

Return the Motor singleton client, creating it on first call.

The client is module-level so that it survives across coroutine calls within the same process and reuses the underlying connection pool.

Returns

AsyncIOMotorClient The shared Motor client instance.

get_db 

get_db()

Return the Motor database object for the configured database.

Returns

AsyncIOMotorDatabase The Motor database identified by cfg.mongodb_database.

get_gridfs_bucket 

get_gridfs_bucket() -> AsyncIOMotorGridFSBucket

Return the GridFS bucket for binary file storage.

The bucket name is read from cfg.mongodb_gridfs_bucket (default "raw_binaries").

Returns

AsyncIOMotorGridFSBucket GridFS bucket ready for upload/download operations.

reset_client 

reset_client() -> None

Reset the singleton Motor client.

Forces :func:get_client to create a new client on the next call. Must be called whenever the configuration changes (e.g. after :func:~scinr.newton.config.configure) so that the new URI and database settings are picked up.

Page Repository

scinr.newton.storage.mongodb.pages

storage/mongodb/pages.py — MongoDB implementation of PageRepository.

Converted pages (Markdown text) are stored as plain documents in the converted_pages collection. Each document maps 1-to-1 to an :class:~storage.models.ConvertedPageRecord and references its parent raw file via raw_file_id.

MongoDBPageRepository

Bases: PageRepository

Stores and retrieves converted pages in the converted_pages collection.

Each document in the collection represents a single page of a converted document. Pages are ordered by page_index which mirrors :attr:~converters.base.IntermediatePage.index.

delete_pages async 

delete_pages(raw_file_id: str) -> int

Delete every converted page belonging to raw_file_id.

Not an error if no pages exist for this raw_file_id — returns 0 in that case rather than raising.

Parameters

raw_file_id: ID of the :class:~storage.models.RawFileRecord whose pages should be deleted.

Returns

int Number of pages deleted (0 if none matched).

get_pages async 

get_pages(raw_file_id: str) -> list[ConvertedPageRecord]

Retrieve all pages for a given raw file, ordered by page index.

Parameters

raw_file_id: ID of the :class:~storage.models.RawFileRecord whose pages should be retrieved.

Returns

list[ConvertedPageRecord] Pages sorted by page_index ascending. Returns an empty list if no pages have been stored for this raw_file_id.

store_page async 

store_page(
    raw_file_id: str,
    filename: str,
    folder_path: str | None,
    page_index: int,
    markdown: str,
) -> str

Persist a single converted page in MongoDB.

Parameters

raw_file_id: ID of the parent :class:~storage.models.RawFileRecord. filename: Stem of the source file without extension, e.g. "3.2.P.1". folder_path: Relative path of the containing folder, or None. page_index: Zero-based page index. markdown: Full Markdown text of this page.

Returns

str The page_id: str(ObjectId) of the newly inserted document.

Raw File Repository

scinr.newton.storage.mongodb.raw_files

storage/mongodb/raw_files.py — MongoDB/GridFS implementation of RawFileRepository.

Binary content is stored in GridFS to support files of arbitrary size (including PDFs larger than the 16 MB BSON document limit). A lightweight metadata document is inserted into the raw_files collection so that records can be queried by checksum, filename, or folder path without fetching the full binary from GridFS.

MongoDBRawFileRepository

Bases: RawFileRepository

Stores binary files in GridFS with metadata in raw_files.

GridFS splits large files into 255 kB chunks and stores them across two internal collections (<bucket>.files and <bucket>.chunks), removing the 16 MB BSON size constraint.

The raw_files collection holds only metadata plus a gridfs_id reference so callers can retrieve the binary when needed.

delete async 

delete(raw_file_id: str) -> None

Delete a raw file's binary (GridFS) and metadata (raw_files).

Idempotent: if raw_file_id is not a valid ObjectId, or no matching metadata document is found (already deleted, or a repeated call), this logs a warning and returns without raising. If the metadata document exists but its GridFS binary is already gone (gridfs.errors.NoFile), that is also logged and swallowed — the metadata document is still deleted.

Parameters

raw_file_id: The raw_file_id (str(ObjectId)) to delete.

store async 

store(
    filename: str,
    content: bytes,
    content_type: str,
    folder_path: str | None,
) -> str

Upload a binary file to GridFS and persist its metadata.

Parameters

filename: Original filename including extension, e.g. "3.2.P.1.pdf". content: Raw binary content of the file. content_type: MIME type, e.g. "application/pdf". folder_path: Relative path of the containing folder from the ingestion root, or None for files at the root.

Returns

str The raw_file_id: str(ObjectId) of the newly inserted document in the raw_files collection.

Null Backend

scinr.newton.storage.null

storage/null.py — No-op storage repositories for when storage_backend='none'.

These implementations satisfy the RawFileRepository and PageRepository interfaces without performing any I/O. They are used as the default when no storage backend is configured, eliminating the need for None checks throughout the codebase.

NullPageRepository

Bases: PageRepository

No-op implementation. All writes are silently discarded.

NullRawFileRepository

Bases: RawFileRepository

No-op implementation. All writes are silently discarded.