Configuration

Configure scinr.newton the ingestion pipeline using the configure() function, environment variables, or a combination of both.

Configuration Resolution

scinr uses a triple-resolution system. For every setting, the effective value is determined by the following priority (highest to lowest):

Explicit argument passed to configure()
Environment variable set in the process environment or loaded from a .env file
Hard-coded default built into the library

This means you can set sensible defaults via environment variables and override individual values at runtime with configure(), or vice versa.

# Example: env var sets concurrency to 4, but configure() overrides to 8
# $ export LLM_CONCURRENCY=4
configure(llm_concurrency=8)  # final value: 8

Environment Variables

All environment variables are optional unless otherwise noted. They are read at configuration time (when configure() is first called or when the config is first accessed).

LLM / Model

Variable	Default	Description
`LLM_CONCURRENCY`	`4`	Maximum number of concurrent LLM calls.

Neo4j

Variable	Default	Description
`NEO4J_URI`	`bolt://localhost:7687`	Bolt URI for the Neo4j instance.
`NEO4J_USER`	(required)	Neo4j database username. Note: previous versions used `NEO4J_USERNAME` — this was renamed to `NEO4J_USER`.
`NEO4J_PASSWORD`	(required)	Neo4j user password.
`NEO4J_DATABASE`	(required)	Neo4j database name
`NEO4J_CONCURRENCY`	`10`	Maximum async Neo4j concurrency.
`NEO4J_SYNC_CONCURRENCY`	`8`	Maximum sync Neo4j concurrency.
`GRAPH_BACKEND`	`neo4j`	Backend for the read-only graph-navigation API (`scinr.newton.navigation`). Validated like `STORAGE_BACKEND`; reserved for future engines.

Storage (MongoDB)

Variable	Default	Description
`STORAGE_BACKEND`	`none`	Storage backend: `none` (no persistence), `mongodb`, or `custom`.
`MONGODB_URI`	`mongodb://localhost:27017`	MongoDB connection string.
`MONGODB_DATABASE`	`scinr`	MongoDB database name.
`MONGODB_RAW_FILES_COLLECTION`	`raw_files`	Collection name for raw file metadata.
`MONGODB_PAGES_COLLECTION`	`converted_pages`	Collection name for converted document pages.
`MONGODB_GRIDFS_BUCKET`	`raw_binaries`	GridFS bucket name for binary file storage.

PDF / Mistral OCR

Variable	Default	Description
`MISTRAL_API_KEY`	`None`	Mistral API key for PDF OCR extraction. Required to process PDF files.
`MISTRAL_OCR_SAFE_MAX_PAGES`	`900`	Maximum number of pages before OCR becomes mandatory.
`MISTRAL_OCR_SAFE_MAX_BYTES`	`47185920` (45 MiB)	Maximum file size in bytes before OCR is required.
`MISTRAL_OCR_MAX_RETRIES`	`15`	Number of retry attempts for OCR failures. Retry uses exponential backoff capped at 5 minutes between retries.
`MISTRAL_OCR_RETRY_BACKOFF_SECONDS`	`2.0`	Base backoff in seconds between retries.
`MISTRAL_OCR_CHUNK_CONCURRENCY`	`1`	Maximum concurrent OCR chunk processing.
`MISTRAL_OCR_ERROR_STRATEGY`	`fail_fast`	Error handling: `fail_fast` (abort on first error) or `best_effort` (continue and collect what is possible).

Pipeline

Variable	Default	Description
`PROMPT_CACHING_ENABLED`	`true`	Enable prompt caching. Currently effective for AWS Bedrock; ignored for other providers.
`EXTRACTION_BATCH_SIZE`	`1`	Number of pages per extraction chunk.
`PROMPT_FAMILY`	`generic`	Prompt template family: `generic`, `claude`, or `gpt_reasoning`.
`SCINR_EXTRA_MODELS_PATHS`	`""` (empty)	Colon-separated list of extra model package paths.
`FULL_DOCSTRING`	`true`	Saves the full docstring from the models. On False, it only saves the first line before a jumpline.

Normalization

Variable	Default	Description
`NORMALIZATION_ENABLED`	`true`	Enable tabular data normalization via LLM.
`NORMALIZATION_BATCH_SIZE`	`5`	Batch size for normalization LLM calls.

Programmatic Configuration

The configure() function is the primary way to set up scinr at runtime. It accepts keyword arguments organized by category. All parameters are optional — omitting a parameter falls back to the environment variable or hard-coded default. It includes all the parameters previously mentioned.

from scinr.newton import configure

LLM Parameters

Parameter	Type	Description
`llm`	`Any \\| None`	Pre-constructed LLM client instance. When provided, bypasses `MODEL_ID` and AWS Bedrock auto-configuration.
`repair_llm`	`Any \\| None`	Separate LLM client for repair/retry operations. Falls back to `llm` if not provided.

Neo4j Parameters

The same settings as the Neo4j environment variables, passed as lowercase keyword arguments.

Parameter	Type	Description
`neo4j_uri`	`str \\| None`	Bolt URI. Default `bolt://localhost:7687`.
`neo4j_user`	`str \\| None`	Username. Required (via arg or `NEO4J_USER`).
`neo4j_password`	`str \\| None`	Password. Required (via arg or `NEO4J_PASSWORD`).
`neo4j_database`	`str \\| None`	Database name. Required (via arg or `NEO4J_DATABASE`).
`neo4j_concurrency`	`int \\| None`	Max concurrent async Neo4j sessions. Default `10`.
`neo4j_sync_concurrency`	`int \\| None`	Max concurrent sync ingestion dispatches. Default `8`.
`graph_backend`	`str \\| None`	Read-only navigation backend. Default `"neo4j"`.

Models / Themes Parameters

Parameter	Type	Description
`enabled_base_themes`	`list[ThemePath \\| str] \\| None`	List of base themes to enable for extraction.
`enabled_user_themes`	`list[str] \\| None`	List of user-defined themes to enable.
`extra_models_paths`	`list[str \\| Path] \\| None`	Additional paths to model packages.

Storage Parameters

Parameter	Type	Description
`custom_storage`	`tuple \\| None`	Custom storage backend tuple (driver, connection).

Converter Parameters

Parameter	Type	Description
`extra_converters`	`dict[str, type] \\| None`	Dictionary mapping file extensions to converter classes.

PDF / Mistral OCR Parameters

The same settings as the PDF / Mistral OCR environment variables, passed as lowercase keyword arguments.

Parameter	Type	Description
`mistral_api_key`	`str \\| None`	Mistral API key for PDF OCR. Default `None`.
`mistral_ocr_safe_max_pages`	`int \\| None`	Page threshold above which OCR is forced. Default `900`.
`mistral_ocr_safe_max_bytes`	`int \\| None`	Byte threshold above which OCR is forced. Default `47185920` (45 MiB).
`mistral_ocr_max_retries`	`int \\| None`	Retry attempts per chunk on retryable errors. Default `15`.
`mistral_ocr_retry_backoff_seconds`	`float \\| None`	Base backoff between retries (exponential, capped at 5 min). Default `2.0`.
`mistral_ocr_chunk_concurrency`	`int \\| None`	Concurrent OCR chunk tasks. Default `1`.
`mistral_ocr_error_strategy`	`Literal["fail_fast", "best_effort"] \\| None`	OCR error handling. Default `"fail_fast"`.

Pipeline Parameters

The same settings as the Pipeline environment variables, passed as lowercase keyword arguments.

Parameter	Type	Description
`llm_concurrency`	`int \\| None`	Max concurrent LLM calls across all stages. Default `4`.
`extraction_batch_size`	`int \\| None`	Pages per extraction chunk. Default `1`.
`prompt_caching_enabled`	`bool \\| None`	Bedrock system-prompt caching. Default `True`.
`full_docstring`	`bool \\| None`	Store the full model docstring (`True`) or only its first line (`False`). Default `True`.
`consolidation_token_safety_margin`	`float \\| None`	Fraction of `max_tokens` reserved for output during `fast_extraction=True` consolidation. Default `0.75`.
`consolidation_max_output_tokens`	`int \\| None`	Explicit output-token ceiling for consolidation; derived from the margin when unset. Default `None`.
`consolidation_max_input_tokens`	`int \\| None`	Explicit input-size ceiling for consolidation; no check when unset. Default `None`.

Note: fast_extraction itself is not a configure() setting — it is a per-call run_pipeline(fast_extraction=...) argument. See Running the Pipeline.

Logging Parameters

Parameter	Type	Description
`log_level`	`str`	Python log level. Default: `"INFO"`. Accepts `"DEBUG"`, `"INFO"`, `"WARNING"`, `"ERROR"`, `"CRITICAL"`.

Prompt Family Parameters

Parameter	Type	Description
`prompt_family`	`PromptFamily \\| Literal["generic", "claude", "gpt_reasoning"] \\| None`	Prompt template family. See Prompt Families for details.

Normalization Parameters

Parameter	Type	Description
`normalization_enabled`	`bool \\| None`	Enable tabular data normalization.
`normalization_batch_size`	`int \\| None`	Batch size for normalization LLM calls.
`normalization_llm`	`Any \\| None`	Dedicated LLM client for normalization. Falls back to `llm` if not provided.

Configuration Examples

Minimal Setup (Environment Variables Only)

The simplest approach: set environment variables and call configure() to let scinr pick them up automatically. configure() always reads .env via python-dotenv, so you never need to import dotenv manually.

# .env file
NEO4J_URI=bolt://localhost:7687
NEO4J_USER=neo4j
NEO4J_PASSWORD=your_password
NEO4J_DATABASE=neo4j

MISTRAL_API_KEY=your_mistral_key

import asyncio
from scinr.newton import configure, run_pipeline
from langchain_aws import ChatBedrockConverse
async def main():
    # configure() reads .env automatically — no arguments needed
    llm = ChatBedrockConverse(...)
    configure(llm=llm)

    result = await run_pipeline(input_raw="./raw_docs")
    print(f"Pipeline: {'success' if result.success else 'failed'}")

asyncio.run(main())

Note: configure() is always required before calling run_pipeline(). Even when all values come from environment variables, you must call configure() to resolve and validate the configuration.

Full AWS Bedrock Setup

Complete programmatic configuration for a production Bedrock deployment.

from scinr.newton import configure
from langchain_aws import ChatBedrockConverse
llm = ChatBedrockConverse(...)
configure(
    # LLM — AWS Bedrock
    llm=None,  # let scinr auto-create from MODEL_ID env var
    repair_llm=None,  # use same model for repairs

    # Neo4j
    neo4j_uri="bolt://neo4j.internal:7687",
    neo4j_user="scinr_ingest",
    neo4j_password="secure_password",
    neo4j_concurrency=10,
    neo4j_sync_concurrency=8,

    # PDF / Mistral OCR
    mistral_api_key="your_mistral_key",
    mistral_ocr_safe_max_pages=900,
    mistral_ocr_safe_max_bytes=47185920,
    mistral_ocr_max_retries=15,
    mistral_ocr_error_strategy="best_effort",

    # Pipeline
    prompt_caching_enabled=True,
    extraction_batch_size=1,
    llm_concurrency=4,
    prompt_family="claude", # If using a Claude Model (Best tested performance). Generic for any other model (Kimi K2.5 tested good performance)

    # Logging
    log_level="INFO",
)

With Normalization Enabled

Enable tabular data normalization with a dedicated LLM for the normalization step.

from scinr.newton import configure

configure(
    llm_concurrency=4,
    prompt_family="claude",

    # Normalization
    normalization_enabled=True,
    normalization_batch_size=5,
    # normalization_llm=dedicated_llm_instance,  # optional: separate LLM for normalization
)

With MongoDB Storage

Persist raw files and converted pages to MongoDB.

from scinr.newton import configure

configure(
    storage_backend="mongodb",
    mongodb_uri="mongodb://user:pass@mongo.internal:27017",
    mongodb_database="scinr_production",
    mongodb_raw_files_collection="raw_files",
    mongodb_pages_collection="converted_pages",
    mongodb_gridfs_bucket="raw_binaries",
)

Using a `.env` File

scinr reads standard .env files. You can copy the provided example from the repository and fill in your values.

cp .env.example .env

The .env.example file in the project root contains every available setting with inline comments. Key notes:

configure() loads the .env file from the current working directory automatically (via python-dotenv). You never import dotenv yourself.
Only NEO4J_USER, NEO4J_PASSWORD, and NEO4J_DATABASE are strictly required. MODEL_ID is required too unless you pass a ready-built llm= to configure().
Environment values are overridden by explicit configure(...) arguments (see Configuration Resolution).
Leaving an optional variable unset falls back to the hard-coded default in the Complete Reference table.

ScinrConfig

configure() returns a ScinrConfig object that holds the resolved configuration. You can also retrieve the active configuration at any time using get_config().

from scinr.newton import configure, get_config

# Set up configuration
configure(
    neo4j_uri="bolt://localhost:7687",
    neo4j_user="neo4j",
    neo4j_password="secret",
    llm_concurrency=8,
    prompt_family="claude",
)

# Read back the active configuration
config = get_config()
print(f"Neo4j URI: {config.neo4j_uri}")
print(f"LLM Concurrency: {config.llm_concurrency}")
print(f"Prompt Family: {config.prompt_family}")

The ScinrConfig object is immutable after creation. To change configuration, call configure() again with the new values — it will produce a new ScinrConfig that replaces the previous one.

Prompt Families

The prompt_family parameter selects a set of prompt templates optimized for different LLM providers.

Family	Description
`generic`	Provider-agnostic prompts. Safe default that works with any LLM.
`claude`	Optimized for Anthropic Claude models. Uses Claude-specific formatting and system prompt conventions.
`gpt_reasoning`	Optimized for OpenAI reasoning models (o-series). Uses the specific message structure required by reasoning-capable models.

Choosing a Prompt Family

Claude models on Bedrock — use "claude" for best results.
OpenAI o-series models — use "gpt_reasoning".
Other providers or unsure — use "generic" (the default).

configure(prompt_family="claude")  # for Claude models
configure(prompt_family="gpt_reasoning")  # for OpenAI o-series
configure(prompt_family="generic")  # default, works everywhere

Complete Reference: All Settings

For quick lookup, here is every configurable setting with its resolution chain:

Setting	`configure()` param	Environment Variable	Default
LLM Client	`llm`	(none)	`None`
Repair LLM Client	`repair_llm`	(none)	`None` (falls back to `llm`)
Model ID	(via `llm`)	`MODEL_ID`	(required if no `llm`)
Repair Model ID	(via `repair_llm`)	`REPAIR_MODEL_ID`	Falls back to `MODEL_ID`
AWS Region	(via `llm`)	`AWS_DEFAULT_REGION`	`us-east-1`
Max Tokens	(via `llm`)	`MAX_TOKENS`	`65536`
Neo4j URI	`neo4j_uri`	`NEO4J_URI`	`bolt://localhost:7687`
Neo4j User	`neo4j_user`	`NEO4J_USER`	(required)
Neo4j Password	`neo4j_password`	`NEO4J_PASSWORD`	(required)
Neo4j Database	`neo4j_database`	`NEO4J_DATABASE`	(required)
Neo4j Auth	(derived)	`NEO4J_AUTH`	Fallback format
Neo4j Async Concurrency	`neo4j_concurrency`	`NEO4J_CONCURRENCY`	`10`
Neo4j Sync Concurrency	`neo4j_sync_concurrency`	`NEO4J_SYNC_CONCURRENCY`	`8`
Graph Navigation Backend	`graph_backend`	`GRAPH_BACKEND`	`neo4j`
LLM Concurrency	`llm_concurrency`	`LLM_CONCURRENCY`	`4`
Storage Backend	`storage_backend`	`STORAGE_BACKEND`	`none`
MongoDB URI	`mongodb_uri`	`MONGODB_URI`	`mongodb://localhost:27017`
MongoDB Database	`mongodb_database`	`MONGODB_DATABASE`	`scinr`
MongoDB Raw Files	`mongodb_raw_files_collection`	`MONGODB_RAW_FILES_COLLECTION`	`raw_files`
MongoDB Pages	`mongodb_pages_collection`	`MONGODB_PAGES_COLLECTION`	`converted_pages`
MongoDB GridFS	`mongodb_gridfs_bucket`	`MONGODB_GRIDFS_BUCKET`	`raw_binaries`
Custom Storage	`custom_storage`	(none)	`None`
Mistral API Key	`mistral_api_key`	`MISTRAL_API_KEY`	`None`
OCR Max Pages	`mistral_ocr_safe_max_pages`	`MISTRAL_OCR_SAFE_MAX_PAGES`	`900`
OCR Max Bytes	`mistral_ocr_safe_max_bytes`	`MISTRAL_OCR_SAFE_MAX_BYTES`	`47185920`
OCR Max Retries	`mistral_ocr_max_retries`	`MISTRAL_OCR_MAX_RETRIES`	`15`
OCR Backoff	`mistral_ocr_retry_backoff_seconds`	`MISTRAL_OCR_RETRY_BACKOFF_SECONDS`	`2.0`
OCR Concurrency	`mistral_ocr_chunk_concurrency`	`MISTRAL_OCR_CHUNK_CONCURRENCY`	`1`
OCR Error Strategy	`mistral_ocr_error_strategy`	`MISTRAL_OCR_ERROR_STRATEGY`	`fail_fast`
Prompt Caching	`prompt_caching_enabled`	`PROMPT_CACHING_ENABLED`	`true`
Full Docstring	`full_docstring`	`FULL_DOCSTRING`	`true`
Extraction Batch	`extraction_batch_size`	`EXTRACTION_BATCH_SIZE`	`1`
Prompt Family	`prompt_family`	`PROMPT_FAMILY`	`generic`
Extra Models Paths	`extra_models_paths`	`SCINR_EXTRA_MODELS_PATHS`	`""`
Enabled Base Themes	`enabled_base_themes`	(none)	`None`
Enabled User Themes	`enabled_user_themes`	(none)	`None`
Extra Converters	`extra_converters`	(none)	`None`
Normalization	`normalization_enabled`	`NORMALIZATION_ENABLED`	`true`
Normalization Batch	`normalization_batch_size`	`NORMALIZATION_BATCH_SIZE`	`5`
Normalization LLM	`normalization_llm`	(none)	`None` (falls back to `llm`)
Consolidation output margin	`consolidation_token_safety_margin`	(none)	`0.75` (used only when `fast_extraction=True`)
Consolidation max output tokens	`consolidation_max_output_tokens`	(none)	`None` (derived from the margin)
Consolidation max input tokens	`consolidation_max_input_tokens`	(none)	`None` (no check)
Log Level	`log_level`	(none)	`"INFO"`