# scinr Full Documentation

This file contains the complete documentation for the scinr library for LLM agent context.

---
## File: api/config.md

# Configuration API

::: scinr.newton.config.configure

::: scinr.newton.config.get_config

::: scinr.newton.config.get_available_themes

::: scinr.newton.config.ThemePath


---
## File: api/exceptions.md

# Exceptions API

::: scinr.newton.exceptions.ScinrError
    options:
      show_submodules: true
      members:
        - ConfigurationError
        - PreconditionError
        - ExtractionError
        - IngestionError
        - ModelError
        - StorageError
        - ConversionError


---
## File: api/index.md

# API Reference Overview

Welcome to the `scinr` API reference documentation.

## Core Modules

- [Pipeline API](pipeline.md): Public orchestrator `run_pipeline()`.
- [Configuration API](config.md): Connection setup `configure()` and `get_config()`.
- [Stages API](stages.md): Individual runner functions for Stages 0-4.
- [Exceptions API](exceptions.md): Typed error hierarchy (`ScinrError`).
- [Results API](results.md): Dataclasses (`PipelineResult`, `StageResult`, `DocumentResult`).


---
## File: api/pipeline.md

# Pipeline API

::: scinr.newton.pipeline.run_pipeline


---
## File: api/results.md

# Results API

::: scinr.newton.results.DocumentResult

::: scinr.newton.results.StageResult

::: scinr.newton.results.PipelineResult


---
## File: api/stages.md

# Stages API

::: scinr.newton.stages.run_preprocess

::: scinr.newton.stages.run_extraction

::: scinr.newton.stages.run_ingestion

::: scinr.newton.stages.run_annotation

::: scinr.newton.stages.run_entity_extraction

::: scinr.newton.stages.run_tabular_pipeline


---
## File: architecture.md

# Architecture

`scinr.newton` processes life sciences documents through a modular 5-stage pipeline, storing raw structure and extracted entities in Neo4j and MongoDB.

---

## 5-Stage Ingestion Pipeline

```
Raw Documents (.pdf, .docx, .pptx, .xlsx, .csv)
                    │
                    ▼
       ┌─────────────────────────┐
       │ Stage 0: Preprocess     │  Format Converters -> JSON / Markdown
       └────────────┬────────────┘
                    ▼
       ┌─────────────────────────┐
       │ Stage 1: Extraction     │  Section Chunking & Hierarchy Parsing
       └────────────┬────────────┘
                    ▼
       ┌─────────────────────────┐
       │ Stage 2: Ingestion      │  Document & Structure Nodes written to Neo4j
       └────────────┬────────────┘
                    ▼
       ┌─────────────────────────┐
       │ Stage 3: Annotation     │  LLM Relevance Filtering & Target Schema Prep
       └────────────┬────────────┘
                    ▼
       ┌─────────────────────────┐
       │ Stage 4: Entity Extract │  Structured Pydantic Extraction & Graph Subgraph
       └─────────────────────────┘
```

### Stage Details

* **Stage 0: Preprocess** — Uses custom document converters (`python-docx`, `pdfplumber`, `python-pptx`, `openpyxl`, `pandas`) to convert heterogeneous raw formats into standardized internal representations.
* **Stage 1: Extraction** — Chunks documents into structural blocks (paragraphs, tables, headers, figures) while retaining parent-child document hierarchy.
* **Stage 2: Ingestion** — Writes document metadata and structural nodes into Neo4j (`:Document`, `:StructureNode`).
* **Stage 3: Annotation** — Uses LLMs to evaluate structure nodes against target extraction models and attach domain annotations.
* **Stage 4: Entity Extraction** — Invokes target Pydantic schemas on annotated nodes to extract typed domain entities and write subgraph nodes and triples into Neo4j.



---
## File: cli.md

# CLI Reference

The `newton` command line tool orchestrates `scinr` ingestion pipeline tasks from your terminal.

---

## Commands

### `newton run`

Orchestrates full or partial ingestion pipeline runs.

```bash
newton run [OPTIONS]
```

#### Key Options

* `--input-raw PATH`: Directory containing input document files.
* `--stages LIST`: Comma-separated list of stages to run (e.g. `preprocess,extraction,ingestion`).
* `--model-class NAME`: Registered Pydantic target extraction model class name.
* `--parallel-docs INT`: Number of documents to process in parallel (default: `1`).
* `--neo4j-uri URI`: Neo4j Bolt connection URI.
* `--neo4j-user USER`: Neo4j database username.
* `--neo4j-pass PASS`: Neo4j database password.
* `--llm-provider NAME`: Provider name (`openai`, `bedrock`, `ollama`).
* `--llm-model NAME`: Model identifier (e.g., `gpt-4o`, `claude-3-5-sonnet`).

---

## Usage Examples

Run Stages 0 to 2 only:

```bash
newton run --input-raw ./data --stages preprocess,extraction,ingestion
```


---
## File: configuration.md

# Configuration

Configure `scinr` programmatic execution using the `configure()` function or environment variables.

---

## Environment Variables

| Variable | Description | Default |
| :--- | :--- | :--- |
| `NEO4J_URI` | Bolt URI for Neo4j instance | `bolt://localhost:7687` |
| `NEO4J_USERNAME` | Neo4j database user | `neo4j` |
| `NEO4J_PASSWORD` | Neo4j user password | — |
| `OPENAI_API_KEY` | OpenAI API key for LLM calls | — |
| `MONGODB_URI` | Connection string for MongoDB storage | `mongodb://localhost:27017` |

---

## Programmatic Configuration

```python
from scinr.newton import configure, get_config

configure(
    neo4j_uri="bolt://localhost:7687",
    neo4j_auth=("neo4j", "secret_pass"),
    llm_provider="openai",
    llm_model="gpt-4o",
    parallel_docs=4,
)

config = get_config()
print(f"Active Provider: {config.llm_provider}")
```


---
## File: getting-started.md

# Getting Started

## Installation

Install `scinr` using `pip` or `uv`:

```bash
pip install scinr
```

Or with `uv`:

```bash
uv add scinr
```

### Optional Provider Dependencies

`scinr` provides optional extras for different LLM providers and storage backends:

```bash
# OpenAI support
pip install "scinr[openai]"

# AWS Bedrock support
pip install "scinr[bedrock]"

# Ollama support
pip install "scinr[ollama]"

# MongoDB storage support
pip install "scinr[mongodb]"

# Install all optional dependencies
pip install "scinr[openai,bedrock,ollama,mongodb]"
```

---

## Prerequisites

Before running the full extraction pipeline, ensure you have:

1. **Python 3.11+** installed.
2. **Neo4j 5.0+** instance running (Bolt protocol).
3. **MongoDB 4.6+** (optional, for persistent document storage).
4. API keys for your preferred LLM provider (e.g., `OPENAI_API_KEY`, `AWS_ACCESS_KEY_ID`).

---

## First Ingestion Run

Create a raw data directory with document files:

```bash
mkdir raw_docs
# Place sample PDF, DOCX, or XLSX files in raw_docs/
```

Run via the `newton` CLI:

```bash
newton run \
  --input-raw ./raw_docs \
  --neo4j-uri bolt://localhost:7687 \
  --neo4j-user neo4j \
  --neo4j-pass password \
  --llm-provider openai \
  --llm-model gpt-4o
```


---
## File: index.md

# scinr

> **AI-Powered Document Knowledge Library for Life Sciences**

`scinr` (`scinr.newton`) is a Python library and command-line interface (`newton`) designed to transform unstructured and structured life sciences documents into queryable, connected knowledge graphs in **Neo4j** and document stores in **MongoDB**.

---

## Key Features

* 🚀 **5-Stage Pipeline**: Seamlessly convert, extract, ingest, annotate, and extract entities from raw files.
* 📄 **Multi-Format Ingestion**: Supports `.pdf`, `.docx`, `.pptx`, `.xlsx`, `.csv`, `.json`, `.html`, and `.txt`.
* 📊 **Tabular Pipeline**: Auto-detection, structural normalization, and LLM entity extraction for scientific spreadsheets.
* 🏷️ **Pydantic Extraction Models**: Define structured schemas for scientific target entities (e.g. compound synthesis, clinical trials, assays).
* 🕸️ **Neo4j Graph Integration**: Automatically map extracted entities and triples into Neo4j subgraphs.
* 🤖 **Agent Ready**: Native support for <a href="llms.txt">llms.txt</a> and <a href="llms-full.txt">llms-full.txt</a> context windows for AI agents.

---

## Quick Example

```python
import asyncio
from scinr.newton import configure, run_pipeline

async def main():
    # 1. Configure backend connections and model defaults
    configure(
        neo4j_uri="bolt://localhost:7687",
        neo4j_auth=("neo4j", "password"),
        llm_provider="openai",
        llm_model="gpt-4o",
    )

    # 2. Run the end-to-end ingestion pipeline on a directory of documents
    result = await run_pipeline(
        input_raw="./raw_documents",
        model_class="CompoundAssayModel",
        parallel_docs=4,
    )

    print(f"Pipeline Succeeded: {result.success}")
    print(f"Total Duration: {result.total_duration_seconds:.2f}s")
    print(f"Executed Stages: {result.stages_executed}")

if __name__ == "__main__":
    asyncio.run(main())
```


---
## File: user-guides/custom-models.md

# Custom Extraction Models

Define domain-specific Pydantic extraction models to extract structured entities from unstructured documents.

---

## Defining a Model

Inherit from `pydantic.BaseModel`. Field docstrings and descriptions serve as guidance for the LLM extraction engine.

> **Note**: Do not alter model field docstrings once registered, as they directly format the prompt schema.

```python
from pydantic import BaseModel, Field

class CompoundAssayResult(BaseModel):
    """Structured measurement of compound potency in a biological assay."""

    compound_name: str = Field(..., description="Canonical chemical or code name of tested compound.")
    target_protein: str = Field(..., description="Biological target protein or receptor.")
    ic50_nm: float | None = Field(None, description="Half maximal inhibitory concentration in nanomolar (nM).")
    assay_type: str = Field(..., description="Assay format, e.g., Binding, Enzymatic, Cell-based.")
```


---
## File: user-guides/neo4j-graph.md

# Neo4j Graph Storage

`scinr` builds a connected graph representation of input documents and extracted domain concepts inside Neo4j.

---

## Graph Model

* `:Document` node: Represents the input file (`document_name`, format, ingestion timestamp).
* `:StructureNode`: Paragraphs, headings, tables, or figures linked via `[:HAS_CHILD]` relationships.
* Extracted Entity Nodes: Domain entities extracted during Stage 4, connected to their source `:StructureNode` via `[:EXTRACTED_FROM]`.


---
## File: user-guides/tabular-pipeline.md

# Tabular Pipeline

The tabular pipeline automatically routes `.csv`, `.xlsx`, and `.xls` files to specialized normalization and table-understanding algorithms.

---

## Capabilities

* **Header Normalization**: Detects multi-line headers, merged cells, and empty metadata rows.
* **Delimiter Detection**: Auto-detects `,`, `;`, `\t`, `|` delimiters in CSV files.
* **Schema Inference**: Converts spreadsheet columns into normalized tabular structures prior to entity extraction.


---