Configuration

Configure scinr.newton the ingestion pipeline using the configure() function, environment variables, or a combination of both.

Configuration Resolution

scinr uses a triple-resolution system. For every setting, the effective value is determined by the following priority (highest to lowest):

Explicit argument passed to configure()
Environment variable set in the process environment or loaded from a .env file
Hard-coded default built into the library

This means you can set sensible defaults via environment variables and override individual values at runtime with configure(), or vice versa.

# Example: env var sets concurrency to 4, but configure() overrides to 8
# $ export LLM_CONCURRENCY=4
configure(llm_concurrency=8)  # final value: 8

Environment Variables

All environment variables are optional unless otherwise noted. They are read at configuration time (when configure() is first called or when the config is first accessed).

LLM / Model

The LLM has no environment variable — build a LangChain chat model in code and pass it as configure(llm=..., repair_llm=...) (not required at configure() time; needed for LLM stages). Only concurrency is env-configurable:

Variable	Default	Description
`LLM_CONCURRENCY`	`4`	Maximum number of concurrent LLM calls.

Neo4j

Variable	Default	Description
`NEO4J_URI`	`bolt://localhost:7687`	Bolt URI for the Neo4j instance.
`NEO4J_USER`	(required)	Neo4j database username. Note: previous versions used `NEO4J_USERNAME` — this was renamed to `NEO4J_USER`.
`NEO4J_PASSWORD`	(required)	Neo4j user password.
`NEO4J_DATABASE`	(required)	Neo4j database name
`NEO4J_CONCURRENCY`	`10`	Maximum async Neo4j concurrency.
`NEO4J_SYNC_CONCURRENCY`	`8`	Maximum sync Neo4j concurrency.
`GRAPH_BACKEND`	`neo4j`	Backend for the read-only graph-navigation API (`scinr.newton.navigation`). Validated like `STORAGE_BACKEND`; reserved for future engines.

Storage (MongoDB)

Variable	Default	Description
`STORAGE_BACKEND`	`none`	Storage backend: `none` (no persistence), `mongodb`, or `custom`.
`MONGODB_URI`	`mongodb://localhost:27017`	MongoDB connection string.
`MONGODB_DATABASE`	`scinr`	MongoDB database name.
`MONGODB_RAW_FILES_COLLECTION`	`raw_files`	Collection name for raw file metadata.
`MONGODB_PAGES_COLLECTION`	`converted_pages`	Collection name for converted document pages.
`MONGODB_GRIDFS_BUCKET`	`raw_binaries`	GridFS bucket name for binary file storage.
`MONGODB_ENSURE_INDEXES`	`true`	Create the MongoDB indexes automatically on the first `get_storage()` call of the process. Set to `false` when the database user lacks the `createIndex` privilege and operations manage the indexes.
`FREEZE_BACKEND`	(inherits `STORAGE_BACKEND`)	Backend for document snapshots (`freeze_document()`, `restore_document()`): `none`, `mongodb` or `custom`. See Document Freezing.
`MONGODB_FROZEN_COLLECTION`	`frozen_documents`	Collection for snapshot metadata.
`MONGODB_FROZEN_GRIDFS_BUCKET`	`frozen_snapshots`	GridFS bucket for snapshot files.

PDF / Mistral OCR

Variable	Default	Description
`MISTRAL_API_KEY`	`None`	Mistral OCR API key. Required to ingest any PDF — PDF conversion is Mistral OCR only, there is no fallback.
`MISTRAL_OCR_SAFE_MAX_PAGES`	`900`	Page count above which a PDF is split into chunks before each chunk is sent to the OCR API.
`MISTRAL_OCR_SAFE_MAX_BYTES`	`47185920` (45 MiB)	Byte size above which a PDF is split into chunks before each chunk is sent to the OCR API.
`MISTRAL_OCR_MAX_RETRIES`	`15`	Number of retry attempts for OCR failures. Retry uses exponential backoff capped at 5 minutes between retries.
`MISTRAL_OCR_RETRY_BACKOFF_SECONDS`	`2.0`	Base backoff in seconds between retries.
`MISTRAL_OCR_CHUNK_CONCURRENCY`	`1`	Maximum concurrent OCR chunk processing.
`MISTRAL_OCR_ERROR_STRATEGY`	`fail_fast`	Error handling: `fail_fast` (abort on first error) or `best_effort` (continue and collect what is possible).

Pipeline

Variable	Default	Description
`PROMPT_CACHING_ENABLED`	`true`	Enable prompt caching. Currently effective for AWS Bedrock; ignored for other providers.
`EXTRACTION_BATCH_SIZE`	`1`	Number of pages per extraction chunk.
`PROMPT_FAMILY`	`generic`	Prompt template family: `generic`, `claude`, or `gpt_reasoning`.
`SCINR_EXTRA_MODELS_PATHS`	`""` (empty)	Colon-separated list of extra model package paths.
`FULL_DOCSTRING`	`true`	Saves the full docstring from the models. On False, it only saves the first line before a jumpline.

Normalization

Variable	Default	Description
`NORMALIZATION_ENABLED`	`true`	Enable tabular data normalization via LLM.
`NORMALIZATION_BATCH_SIZE`	`5`	Batch size for normalization LLM calls.

Programmatic Configuration

The configure() function is the primary way to set up scinr at runtime. It accepts keyword arguments organized by category. Every parameter is optional at configure() time: most fall back to an environment variable or hard-coded default, while llm= (which has neither) is only needed once the pipeline runs an LLM stage.

from scinr.newton import configure

LLM Parameters

Parameter	Type	Description
`llm`	`BaseChatModel \\| None`	A ready-built LangChain chat model for all LLM calls (`ChatBedrockConverse`, `ChatOpenAI`, `ChatOllama`, …). Optional at `configure()` time; needed for the extraction / annotation / entity-extraction stages and tabular normalization.
`repair_llm`	`BaseChatModel \\| None`	Separate model for the JSON repair loop. Falls back to `llm` if not provided.

Neo4j Parameters

The same settings as the Neo4j environment variables, passed as lowercase keyword arguments.

Parameter	Type	Description
`neo4j_uri`	`str \\| None`	Bolt URI. Default `bolt://localhost:7687`.
`neo4j_user`	`str \\| None`	Username. Required (via arg or `NEO4J_USER`).
`neo4j_password`	`str \\| None`	Password. Required (via arg or `NEO4J_PASSWORD`).
`neo4j_database`	`str \\| None`	Database name. Required (via arg or `NEO4J_DATABASE`).
`neo4j_concurrency`	`int \\| None`	Max concurrent async Neo4j sessions. Default `10`.
`neo4j_sync_concurrency`	`int \\| None`	Max concurrent sync ingestion dispatches. Default `8`.
`graph_backend`	`str \\| None`	Read-only navigation backend. Default `"neo4j"`.

Models / Themes Parameters

Parameter	Type	Description
`enabled_base_themes`	`list[ThemePath \\| str] \\| None`	List of base themes to enable for extraction.
`enabled_user_themes`	`list[str] \\| None`	List of user-defined themes to enable.
`extra_models_paths`	`list[str \\| Path] \\| None`	Additional paths to model packages.

Storage Parameters

Parameter	Type	Description
`mongodb_ensure_indexes`	`bool \\| None`	Create the MongoDB indexes automatically (default `True`). `False` skips it, for users without the `createIndex` privilege.
`custom_storage`	`tuple \\| None`	Custom storage backend tuple (driver, connection).
`freeze_backend`	`str \\| None`	Snapshot backend: `none`, `mongodb` or `custom`. Unset → `FREEZE_BACKEND`, else the resolved `storage_backend`.
`mongodb_frozen_collection`	`str \\| None`	Collection for snapshot metadata (default `frozen_documents`).
`mongodb_frozen_gridfs_bucket`	`str \\| None`	GridFS bucket for snapshot files (default `frozen_snapshots`).
`custom_freeze_storage`	`FreezeRepository \\| None`	Snapshot repository instance when `freeze_backend='custom'`.

Converter Parameters

Parameter	Type	Description
`extra_converters`	`dict[str, type] \\| None`	Dictionary mapping file extensions to converter classes.

PDF / Mistral OCR Parameters

The same settings as the PDF / Mistral OCR environment variables, passed as lowercase keyword arguments.

Parameter	Type	Description
`mistral_api_key`	`str \\| None`	Mistral API key for PDF OCR. Default `None`.
`mistral_ocr_safe_max_pages`	`int \\| None`	Page threshold above which OCR is forced. Default `900`.
`mistral_ocr_safe_max_bytes`	`int \\| None`	Byte threshold above which OCR is forced. Default `47185920` (45 MiB).
`mistral_ocr_max_retries`	`int \\| None`	Retry attempts per chunk on retryable errors. Default `15`.
`mistral_ocr_retry_backoff_seconds`	`float \\| None`	Base backoff between retries (exponential, capped at 5 min). Default `2.0`.
`mistral_ocr_chunk_concurrency`	`int \\| None`	Concurrent OCR chunk tasks. Default `1`.
`mistral_ocr_error_strategy`	`Literal["fail_fast", "best_effort"] \\| None`	OCR error handling. Default `"fail_fast"`.

Pipeline Parameters

The same settings as the Pipeline environment variables, passed as lowercase keyword arguments.

Parameter	Type	Description
`llm_concurrency`	`int \\| None`	Max concurrent LLM calls across all stages. Default `4`.
`extraction_batch_size`	`int \\| None`	Pages per extraction chunk. Default `1`.
`prompt_caching_enabled`	`bool \\| None`	Bedrock system-prompt caching. Default `True`.
`full_docstring`	`bool \\| None`	Store the full model docstring (`True`) or only its first line (`False`). Default `True`.
`consolidation_token_safety_margin`	`float \\| None`	Fraction of `max_tokens` reserved for output during `fast_extraction=True` consolidation. Default `0.75`.
`consolidation_max_output_tokens`	`int \\| None`	Explicit output-token ceiling for consolidation; derived from the margin when unset. Default `None`.
`consolidation_max_input_tokens`	`int \\| None`	Explicit input-size ceiling for consolidation; no check when unset. Default `None`.

Note: fast_extraction itself is not a configure() setting — it is a per-call run_pipeline(fast_extraction=...) argument. See Running the Pipeline.

Logging Parameters

Parameter	Type	Description
`log_level`	`str`	Python log level. Default: `"INFO"`. Accepts `"DEBUG"`, `"INFO"`, `"WARNING"`, `"ERROR"`, `"CRITICAL"`.

Prompt Family Parameters

Parameter	Type	Description
`prompt_family`	`PromptFamily \\| Literal["generic", "claude", "gpt_reasoning"] \\| None`	Prompt template family. See Prompt Families for details.

Normalization Parameters

Parameter	Type	Description
`normalization_enabled`	`bool \\| None`	Enable tabular data normalization.
`normalization_batch_size`	`int \\| None`	Batch size for normalization LLM calls.
`normalization_llm`	`Any \\| None`	Dedicated LLM client for normalization. Falls back to `llm` if not provided.

Configuration Examples

Graph navigation (scinr.newton.navigation) is read-only and does not need any LLM. Configure only the Neo4j connection:

# Navigation only — no LLM required
configure(
    neo4j_user="neo4j",
    neo4j_password="your_password",
    neo4j_database="neo4j",
)  # storage_backend defaults to "none"; graph_backend to "neo4j"

Minimal Setup

Put Neo4j / OCR settings in .env; build the LLM in code and pass it to configure(). configure() always reads .env via python-dotenv, so you never need to import dotenv manually.

# .env file
NEO4J_URI=bolt://localhost:7687
NEO4J_USER=neo4j
NEO4J_PASSWORD=your_password
NEO4J_DATABASE=neo4j

MISTRAL_API_KEY=your_mistral_key

import asyncio
from scinr.newton import configure, run_pipeline
from langchain_aws import ChatBedrockConverse
async def main():
    # The model is passed via llm=; everything else can come from .env.
    llm = ChatBedrockConverse(model="us.anthropic.claude-sonnet-4-6", region_name="us-east-1")
    configure(llm=llm)

    result = await run_pipeline(input_raw="./raw_docs")
    print(f"Pipeline: {'success' if result.success else 'failed'}")

asyncio.run(main())

Note: configure() must be called before run_pipeline(). Supply llm= whenever the run includes an LLM stage (extraction, annotation, entity extraction, tabular normalization).

Full AWS Bedrock Setup

Complete programmatic configuration for a production Bedrock deployment.

from scinr.newton import configure
from langchain_aws import ChatBedrockConverse

llm = ChatBedrockConverse(model="us.anthropic.claude-sonnet-4-6", region_name="us-east-1")
configure(
    # LLM — needed for LLM stages
    llm=llm,
    repair_llm=None,  # falls back to `llm`

    # Neo4j
    neo4j_uri="bolt://neo4j.internal:7687",
    neo4j_user="scinr_ingest",
    neo4j_password="secure_password",
    neo4j_concurrency=10,
    neo4j_sync_concurrency=8,

    # PDF / Mistral OCR
    mistral_api_key="your_mistral_key",
    mistral_ocr_safe_max_pages=900,
    mistral_ocr_safe_max_bytes=47185920,
    mistral_ocr_max_retries=15,
    mistral_ocr_error_strategy="best_effort",

    # Pipeline
    prompt_caching_enabled=True,
    extraction_batch_size=1,
    llm_concurrency=4,
    prompt_family="claude", # If using a Claude Model (Best tested performance). Generic for any other model (Kimi K2.5 tested good performance)

    # Logging
    log_level="INFO",
)

With Normalization Enabled

Enable tabular data normalization with a dedicated LLM for the normalization step.

from scinr.newton import configure

configure(
    llm_concurrency=4,
    prompt_family="claude",

    # Normalization
    normalization_enabled=True,
    normalization_batch_size=5,
    # normalization_llm=dedicated_llm_instance,  # optional: separate LLM for normalization
)

With MongoDB Storage

Persist raw files and converted pages to MongoDB.

from scinr.newton import configure

configure(
    storage_backend="mongodb",
    mongodb_uri="mongodb://user:pass@mongo.internal:27017",
    mongodb_database="scinr_production",
    mongodb_raw_files_collection="raw_files",
    mongodb_pages_collection="converted_pages",
    mongodb_gridfs_bucket="raw_binaries",
)

Using a `.env` File

scinr reads standard .env files. You can copy the provided example from the repository and fill in your values.

cp .env.example .env

The .env.example file in the project root contains every available setting with inline comments. Key notes:

configure() loads the .env file from the current working directory automatically (via python-dotenv). You never import dotenv yourself.
NEO4J_USER, NEO4J_PASSWORD, and NEO4J_DATABASE are strictly required. The LLM has no env var — pass it as configure(llm=...) when a run includes an LLM stage.
Environment values are overridden by explicit configure(...) arguments (see Configuration Resolution).
Leaving an optional variable unset falls back to the hard-coded default in the Complete Reference table.

ScinrConfig

configure() returns a ScinrConfig object that holds the resolved configuration. You can also retrieve the active configuration at any time using get_config().

from scinr.newton import configure, get_config

# Set up configuration
configure(
    neo4j_uri="bolt://localhost:7687",
    neo4j_user="neo4j",
    neo4j_password="secret",
    llm_concurrency=8,
    prompt_family="claude",
)

# Read back the active configuration
config = get_config()
print(f"Neo4j URI: {config.neo4j_uri}")
print(f"LLM Concurrency: {config.llm_concurrency}")
print(f"Prompt Family: {config.prompt_family}")

The ScinrConfig object is immutable after creation. To change configuration, call configure() again with the new values — it will produce a new ScinrConfig that replaces the previous one.

Prompt Families

The prompt_family parameter selects a set of prompt templates optimized for different LLM providers.

Family	Description
`generic`	Provider-agnostic prompts. Safe default that works with any LLM.
`claude`	Optimized for Anthropic Claude models. Uses Claude-specific formatting and system prompt conventions.
`gpt_reasoning`	Optimized for OpenAI reasoning models (o-series). Uses the specific message structure required by reasoning-capable models.

Choosing a Prompt Family

Claude models on Bedrock — use "claude" for best results.
OpenAI o-series models — use "gpt_reasoning".
Other providers or unsure — use "generic" (the default).

configure(prompt_family="claude")  # for Claude models
configure(prompt_family="gpt_reasoning")  # for OpenAI o-series
configure(prompt_family="generic")  # default, works everywhere

Complete Reference: All Settings

For quick lookup, here is every configurable setting with its resolution chain:

Setting	`configure()` param	Environment Variable	Default
LLM Client	`llm`	(none)	`None`
Repair LLM Client	`repair_llm`	(none)	`None` (falls back to `llm`)
Neo4j URI	`neo4j_uri`	`NEO4J_URI`	`bolt://localhost:7687`
Neo4j User	`neo4j_user`	`NEO4J_USER`	(required)
Neo4j Password	`neo4j_password`	`NEO4J_PASSWORD`	(required)
Neo4j Database	`neo4j_database`	`NEO4J_DATABASE`	(required)
Neo4j Auth	(derived)	`NEO4J_AUTH`	Fallback format
Neo4j Async Concurrency	`neo4j_concurrency`	`NEO4J_CONCURRENCY`	`10`
Neo4j Sync Concurrency	`neo4j_sync_concurrency`	`NEO4J_SYNC_CONCURRENCY`	`8`
Graph Navigation Backend	`graph_backend`	`GRAPH_BACKEND`	`neo4j`
LLM Concurrency	`llm_concurrency`	`LLM_CONCURRENCY`	`4`
Storage Backend	`storage_backend`	`STORAGE_BACKEND`	`none`
MongoDB URI	`mongodb_uri`	`MONGODB_URI`	`mongodb://localhost:27017`
MongoDB Database	`mongodb_database`	`MONGODB_DATABASE`	`scinr`
MongoDB Raw Files	`mongodb_raw_files_collection`	`MONGODB_RAW_FILES_COLLECTION`	`raw_files`
MongoDB Pages	`mongodb_pages_collection`	`MONGODB_PAGES_COLLECTION`	`converted_pages`
MongoDB GridFS	`mongodb_gridfs_bucket`	`MONGODB_GRIDFS_BUCKET`	`raw_binaries`
MongoDB Ensure Indexes	`mongodb_ensure_indexes`	`MONGODB_ENSURE_INDEXES`	`True`
Custom Storage	`custom_storage`	(none)	`None`
Freeze Backend	`freeze_backend`	`FREEZE_BACKEND`	resolved `storage_backend`
MongoDB Frozen Collection	`mongodb_frozen_collection`	`MONGODB_FROZEN_COLLECTION`	`frozen_documents`
MongoDB Frozen GridFS	`mongodb_frozen_gridfs_bucket`	`MONGODB_FROZEN_GRIDFS_BUCKET`	`frozen_snapshots`
Custom Freeze Storage	`custom_freeze_storage`	(none)	`None`
Mistral API Key	`mistral_api_key`	`MISTRAL_API_KEY`	`None`
OCR Max Pages	`mistral_ocr_safe_max_pages`	`MISTRAL_OCR_SAFE_MAX_PAGES`	`900`
OCR Max Bytes	`mistral_ocr_safe_max_bytes`	`MISTRAL_OCR_SAFE_MAX_BYTES`	`47185920`
OCR Max Retries	`mistral_ocr_max_retries`	`MISTRAL_OCR_MAX_RETRIES`	`15`
OCR Backoff	`mistral_ocr_retry_backoff_seconds`	`MISTRAL_OCR_RETRY_BACKOFF_SECONDS`	`2.0`
OCR Concurrency	`mistral_ocr_chunk_concurrency`	`MISTRAL_OCR_CHUNK_CONCURRENCY`	`1`
OCR Error Strategy	`mistral_ocr_error_strategy`	`MISTRAL_OCR_ERROR_STRATEGY`	`fail_fast`
Prompt Caching	`prompt_caching_enabled`	`PROMPT_CACHING_ENABLED`	`true`
Full Docstring	`full_docstring`	`FULL_DOCSTRING`	`true`
Extraction Batch	`extraction_batch_size`	`EXTRACTION_BATCH_SIZE`	`1`
Prompt Family	`prompt_family`	`PROMPT_FAMILY`	`generic`
Extra Models Paths	`extra_models_paths`	`SCINR_EXTRA_MODELS_PATHS`	`""`
Enabled Base Themes	`enabled_base_themes`	(none)	`None`
Enabled User Themes	`enabled_user_themes`	(none)	`None`
Extra Converters	`extra_converters`	(none)	`None`
Normalization	`normalization_enabled`	`NORMALIZATION_ENABLED`	`true`
Normalization Batch	`normalization_batch_size`	`NORMALIZATION_BATCH_SIZE`	`5`
Normalization LLM	`normalization_llm`	(none)	`None` (falls back to `llm`)
Consolidation output margin	`consolidation_token_safety_margin`	(none)	`0.75` (used only when `fast_extraction=True`)
Consolidation max output tokens	`consolidation_max_output_tokens`	(none)	`None` (derived from the margin)
Consolidation max input tokens	`consolidation_max_input_tokens`	(none)	`None` (no check)
Log Level	`log_level`	(none)	`"INFO"`