Quick Start — Your First Knowledge Graph

Get from zero to a working knowledge graph in under 15 minutes.

This guide walks you through installing scinr, configuring your environment, preparing documents, running the full ingestion pipeline (Stages 0–4, plus an auto-detected tabular path), and verifying the extracted knowledge graph in Neo4j.

---

Prerequisites

Before you begin, make sure you have:

1. Python 3.11+ installed on your system.
2. Neo4j 5.0+ running and accessible via the Bolt protocol. If you do not have Neo4j installed, start one locally with Docker:

docker run -p 7687:7687 -p 7474:7474 \
  -e NEO4J_AUTH=neo4j/your_password \
  neo4j:5

1. LLM credentials for at least one provider (AWS Bedrock, OpenAI, or Ollama).

---

Step 1: Install scinr

Install scinr with pip. Choose the extras that match your LLM provider:

# AWS Bedrock (recommended — includes langchain-aws and boto3)
pip install "scinr[bedrock]"

# Or with OpenAI (includes langchain-openai)
pip install "scinr[openai]"

# Or with Ollama (includes langchain-ollama)
pip install "scinr[ollama]"

# Or with all extras at once
pip install "scinr[bedrock,openai,ollama,mongodb]"

Tip: To process any PDF you need a Mistral AI API key — PDF conversion is done entirely through the Mistral OCR API. There is no fallback path for PDFs without a key.

---

Step 2: Set up your environment

scinr reads Neo4j, storage, and OCR settings from environment variables. The LLM is configured in code — you pass a ready-built LangChain model as llm= to configure() (see Step 4). The recommended approach for everything else is a .env file in your working directory.

Create .env from the template

If you cloned the scinr repository, copy the example template:

cp .env.example .env

Fill in the required values

At a minimum, you need to set the Neo4j credentials. Here is a minimal .env for a first run:

# ─── Neo4j ──────────────────────────────────────────────────────────────────
NEO4J_URI=bolt://localhost:7687
NEO4J_USER=neo4j
NEO4J_PASSWORD=your_password
NEO4J_DATABASE=neo4j

# ─── PDF conversion ────────────────────────────────────────────────────────
MISTRAL_API_KEY=your_mistral_key

Required vs. optional fields

Variable	Required?	What to set
NEO4J_URI	No	Neo4j Bolt URI. Default: bolt://localhost:7687.
NEO4J_USER	Yes	Neo4j username (usually neo4j).
NEO4J_PASSWORD	Yes	Neo4j password.
NEO4J_DATABASE	Yes	Neo4j database name (usually neo4j).
MISTRAL_API_KEY	Yes, to process any PDF	Mistral OCR API key. PDF conversion has no non-Mistral path.
STORAGE_BACKEND	No	none (default, in-memory) or mongodb.

LLM: there is no MODEL_ID environment variable to set. Build a LangChain chat model in your script and pass it as llm= to configure().

Note: python-dotenv is included as a core dependency. When you call configure(), it automatically loads variables from a .env file in your current working directory. You do not need to import dotenv manually.

---

Step 3: Prepare your documents

Create a directory and place your source documents in it:

mkdir -p raw_docs
# Place your files here

Supported file formats

Format	Extensions	Notes
PDF	.pdf	Converted via the Mistral OCR API — requires MISTRAL_API_KEY
Word	.docx	Full text + structure extraction
Excel	.xlsx, .xls	Auto-routed to tabular pipeline
CSV	.csv	Auto-routed to tabular pipeline

Roadmap: .pptx, .html, .json, .xml, and .txt converters exist in the codebase but are not yet officially supported or tested. Stick to the formats above for production use.

Warning: Tabular files (.csv, .xlsx, .xls) are automatically routed to the tabular pipeline and bypass the standard extraction flow (Stages 0–4). This is intentional — spreadsheets have a fundamentally different structure than narrative documents.

---

Step 4: Run the pipeline

Create a Python script to configure scinr and run the full pipeline:

# quickstart.py
import asyncio

from langchain_aws import ChatBedrockConverse  # or any LangChain chat model

from scinr.newton import configure, run_pipeline


async def main():
    # The model is passed via llm= (no MODEL_ID env var). Neo4j and storage
    # settings still come from .env (loaded automatically by configure()).
    configure(
        llm=ChatBedrockConverse(model="us.anthropic.claude-sonnet-4-6", region_name="us-east-1"),
        neo4j_uri="bolt://localhost:7687",
        neo4j_user="neo4j",
        neo4j_password="your_password",
        neo4j_database="neo4j",
    )

    # Run the default pipeline (Stages 0–4; tabular files are auto-detected)
    result = await run_pipeline(input_raw="./raw_docs")

    # Check overall result
    print(f"Pipeline success: {result.success}")
    print(f"Total duration: {result.total_duration_seconds:.1f}s")
    print(f"Stages executed: {result.stages_executed}")

    # Inspect individual stages
    for stage_name in result.stages_executed:
        stage = getattr(result, stage_name, None)
        if stage is not None:
            status = "OK" if stage.success else "FAIL"
            print(
                f"  [{status}] {stage_name}: "
                f"{stage.total_processed} processed, "
                f"{stage.total_failed} failed "
                f"({stage.duration_seconds:.1f}s)"
            )

            # Show per-document details
            for doc in stage.documents:
                if doc.errors:
                    for err in doc.errors:
                        print(f"    - {doc.document_name}: {err}")


asyncio.run(main())

Run the script:

python quickstart.py

What happens behind the scenes

The pipeline runs these stages in order:

Stage	Name	What it does
0	Preprocess	Converts raw files to an intermediate JSON/Markdown format
1	Extraction	Uses the LLM to parse document structure and extract hierarchical sections
2	Ingestion	Writes document and structure nodes into Neo4j
3	Annotation	An LLM agent assigns an extraction model to each structural node
4	Entity Extraction	Extracts typed Pydantic entities from annotated nodes and writes them as graph subgraphs
5	Tabular	If .csv, .xlsx, or .xls files are detected in input_raw, they are processed through a separate tabular pipeline with LLM-powered normalization

Expected output

A successful run produces output similar to:

Pipeline success: True
Total duration: 45.2s
Stages executed: ['preprocess', 'extraction', 'ingestion', 'annotation', 'entity_extraction']
  [OK] preprocess: 3 processed, 0 failed (2.1s)
  [OK] extraction: 3 processed, 0 failed (18.5s)
  [OK] ingestion: 3 processed, 0 failed (4.3s)
  [OK] annotation: 47 processed, 0 failed (12.8s)
  [OK] entity_extraction: 47 processed, 2 failed (8.5s)

Tip: If you see failures in entity extraction, this is often expected — not every structural node contains extractable domain entities. The pipeline continues gracefully. Check the per-document errors for details.

---

Step 5: Verify in Neo4j

After the pipeline completes, your data is available in Neo4j. Use the Neo4j Browser (http://localhost:7474) or any Cypher client to inspect the results.

Schema note: HAS_STRUCTURE connects a :Document only to its root :StructureNode; deeper nesting is HAS_CHILD. Extracted entities hang off (:StructureNode)-[:HAS_EXTRACTION]->(:ExtractionResult)-[:HAS_<field>]->(:ModelInstance)-[:REFERENCES]->(:LabeledEntity). See Neo4j Graph Model for the full reference.

List all ingested documents

MATCH (d:Document)
WHERE d.latest = true
RETURN d.name, d.path, d.version, d.load_date
ORDER BY d.load_date DESC;

View document structure

MATCH (d:Document)-[:HAS_STRUCTURE]->(:StructureNode)-[:HAS_CHILD*0..]->(s:StructureNode)
RETURN d.name AS document, s.title AS section, s.role
LIMIT 20;

View the full hierarchy

MATCH (d:Document)-[:HAS_STRUCTURE]->(:StructureNode)-[:HAS_CHILD*0..]->(parent:StructureNode)-[:HAS_CHILD]->(child:StructureNode)
RETURN d.name AS document, parent.title AS parent, child.title AS child, child.role AS child_role
LIMIT 30;

View extracted entity types

MATCH (m:ModelInstance)
RETURN m.model_class AS type, count(m) AS count
ORDER BY count DESC;

View labeled entities (globally deduplicated)

MATCH (:ModelInstance)-[:REFERENCES]->(e:LabeledEntity)
RETURN e.label AS entity_type, e.value, count(*) AS references
ORDER BY references DESC
LIMIT 20;

View entity relationships

MATCH (a:ModelInstance)-[r]->(b:ModelInstance)
RETURN type(r) AS relationship, count(r) AS count
ORDER BY count DESC;

Explore a specific document's extraction results

MATCH (d:Document {name: 'YourDocumentName'})-[:HAS_STRUCTURE]->(:StructureNode)-[:HAS_CHILD*0..]->(s:StructureNode)
MATCH (s)-[:HAS_EXTRACTION]->(:ExtractionResult)-[r]->(m:ModelInstance)
RETURN s.title AS section,
       m.model_class AS model_type,
       count(m) AS entities
ORDER BY entities DESC
LIMIT 20;

Visualize the graph

In Neo4j Browser, run this query to get a visual overview:

MATCH path = (d:Document {name: 'YourDocumentName'})-[:HAS_STRUCTURE]->(:StructureNode)-[:HAS_CHILD*0..2]->(:StructureNode)
RETURN path
LIMIT 50;

---

Step 6: Next steps

Now that you have a working pipeline, explore the rest of the documentation:

• Configuration — Full configuration reference: all environment variables, configure() parameters, prompt families, concurrency tuning, and advanced settings.
• Custom Models — Define your own Pydantic extraction schemas for domain-specific entities.
• Architecture — Detailed walkthrough of each pipeline stage, data flow between stages, and system design decisions.
• Tabular Pipeline — Working with CSV, XLSX, and spreadsheet data.
• Neo4j Graph Storage — Understanding the graph model and node/relationship types.

---

Common variations

Using OpenAI instead of Bedrock

from langchain_openai import ChatOpenAI
from scinr.newton import configure, run_pipeline

configure(
    llm=ChatOpenAI(model="gpt-4o"),
    neo4j_user="neo4j",
    neo4j_password="your_password",
)

Set OPENAI_API_KEY in your .env file.

Using Ollama (local models)

from langchain_ollama import ChatOllama
from scinr.newton import configure, run_pipeline

configure(
    llm=ChatOllama(model="llama3"),
    neo4j_user="neo4j",
    neo4j_password="your_password",
)

Make sure Ollama is running locally (ollama serve) and the model is pulled (ollama pull llama3).

Running only specific stages

You can skip stages by providing input from a later point in the pipeline:

# Run only annotation and entity extraction on an already-ingested document
result = await run_pipeline(
    stages=["annotation", "entity_extraction"],
    document_names=["MyDocument"],
)

Processing a single document

# Only process files matching a specific name pattern
result = await run_pipeline(
    input_raw="./raw_docs",
    document_names=["ClinicalTrialReport"],
)

Enabling MongoDB storage

Persist raw files and converted pages to MongoDB:

configure(
    llm=ChatOllama(model="llama3"),
    neo4j_user="neo4j",
    neo4j_password="your_password",
    neo4j_database="neo4j",
    storage_backend="mongodb",
    mongodb_uri="mongodb://localhost:27017",
    mongodb_database="scinr",
)

---

Troubleshooting

An LLM stage fails with "no LLM configured"

The extraction, annotation, and entity-extraction stages (and tabular normalization) need a model. Pass an llm= argument to configure() with a LangChain BaseChatModel instance (e.g. ChatBedrockConverse, ChatOpenAI, ChatOllama). There is no environment-variable shortcut for the model. configure() itself does not require it — only these stages do.

"Neo4j username is not configured" / "Neo4j password is not configured"

configure() raises these separately. Set NEO4J_USER and NEO4J_PASSWORD in your .env file, or pass them as arguments to configure(). NEO4J_DATABASE is also required.

Neo4j connection refused

Make sure your Neo4j instance is running and accessible:

# Test Bolt connectivity
python -c "
from neo4j import GraphDatabase
driver = GraphDatabase.driver('bolt://localhost:7687', auth=('neo4j', 'your_password'))
driver.verify_connectivity()
driver.close()
print('Neo4j connection OK')
"

"No documents discovered for this run"

Check that:

• The input_raw directory exists and contains supported file types.
• File extensions are recognized (.pdf, .docx, .xlsx, .xls, .csv).
• The path is correct — relative paths are resolved from your current working directory.

ImportError: langchain-aws is not installed

If you build a ChatBedrockConverse model but have not installed the Bedrock extra:

pip install "scinr[bedrock]"

PDFs fail to process

PDF conversion goes entirely through the Mistral OCR API. Set MISTRAL_API_KEY in your .env (get a key at console.mistral.ai). Without it, no PDF can be processed — there is no local/text-extraction fallback.

LLM calls are slow or rate-limited

Adjust concurrency in your .env or via configure():

configure(
    llm=my_llm,
    neo4j_user="neo4j",
    neo4j_password="your_password",
    llm_concurrency=2,       # Reduce concurrent LLM calls
    neo4j_concurrency=5,     # Reduce concurrent Neo4j writes
)

Annotation stage returns no model matches

This is expected for documents that do not contain content matching any registered extraction model. The pipeline falls back to generic triple extraction for unmatched nodes. To get specific entity extraction, define custom models matching your domain — see Custom Models.