Converters API
Document format converters used in Stage 0 (Preprocess).
Converter Registry
converters/registry.py — Extension-to-converter registry.
Provides get_converter(path) which returns the appropriate
BaseConverter instance for a given file path.
Apply converter overrides/additions to the registry.
Parameters
overrides: Dict mapping file extensions (without leading dot, lowercase) to BaseConverter subclasses. Existing extensions are overridden. New extensions are registered.
Raises
ConfigurationError If any converter class is not a subclass of BaseConverter.
Example
configure(extra_converters={ "pdf": MyCustomPdfConverter, # override built-in PDF converter "epub": EpubConverter, # add new format })
Return the appropriate converter instance for path.
Parameters
path: File to be converted. The extension (case-insensitive, without leading dot) is used to look up the converter.
Returns
BaseConverter A freshly instantiated converter for the file's format.
Raises
UnsupportedFormatError If no converter is registered for the file's extension.
Base Classes
converters/base.py — Core dataclasses and abstract base converter.
Defines the intermediate document format that all converters produce, and the BaseConverter abstract class that all format-specific converters must implement.
Bases: ABC
Abstract base class for all format-specific converters.
Subclasses must:
1. Define supported_extensions (frozenset of lowercase extensions
without leading dot, e.g. frozenset({"pdf"})).
2. Implement convert(source).
Parameters
supported_extensions : frozenset[str] Class attribute listing the file extensions this converter handles.
abstractmethod
Convert source to the intermediate document format.
Parameters
source: Path to the source file to convert.
Returns
IntermediateDocument The converted document with one or more pages.
Raises
ConversionError If conversion fails for any reason. FileNotFoundError If source does not exist.
Convert source and write the result to output_dir.
The output file is named {source.stem}.json. If a file with
that name already exists, a numeric suffix is appended to avoid
silent overwriting (e.g. doc_1.json).
Parameters
source: Path to the source file to convert. output_dir: Directory where the JSON output will be written. Created automatically if it does not exist.
Returns
Path Path to the written output file.
Raises
ConversionError If conversion fails.
Bases: ConverterError, ScinrError
Raised when a converter fails to process a specific file.
Bases: Exception
Base class for converter errors (kept for backward compatibility).
Bases: BaseModel
A complete document in the intermediate format.
This is the root object serialised to JSON and written to
data/input/ or data/input-pruebas/.
Bases: BaseModel
A single page of a document in the intermediate format.
This schema mirrors the Mistral OCR output consumed by Stage 1.
Bases: BaseModel
Physical dimensions of a page.
Bases: BaseModel
An image extracted from a page.
Parameters
index:
Zero-based position of the image within the page.
base64:
Base64-encoded image bytes.
media_type:
MIME type of the image (e.g. "image/png").
description:
Optional textual description. Left empty by converters;
filled by Stage 1 using LLM vision.
Bases: ConverterError
Raised when no converter is registered for the given file extension.
Format Converters
converters/pdf.py — PDF to Markdown converter via Mistral OCR API.
Encodes the PDF as base64 and submits it to the Mistral OCR endpoint
(POST https://api.mistral.ai/v1/ocr). The API response already
matches the intermediate document format, so each page is mapped
directly to an IntermediatePage.
Para PDFs que exceden los límites de la API de Mistral OCR (máx. 1000
páginas / 50 MB por solicitud), el documento se divide automáticamente
en chunks contiguos (ver pdf_splitter.py), cada uno se envía por
separado, y los resultados se reúnen de forma transparente preservando
el índice de página absoluto del documento original. El manejo de
errores por chunk es configurable vía mistral_ocr_error_strategy
("fail_fast" por defecto, o "best_effort").
Bases: BaseConverter
Convert .pdf files to the intermediate format via Mistral OCR.
Sends the PDF to the Mistral OCR API and maps the response to an
:class:~converters.base.IntermediateDocument. Each page returned
by Mistral becomes one :class:~converters.base.IntermediatePage.
Si el PDF excede los límites configurados de páginas o bytes, se
divide en chunks contiguos (ver :mod:pdf_splitter), cada uno se
envía por separado a la API, y las páginas resultantes se reúnen
preservando el índice absoluto del documento original.
Parameters
api_key:
Mistral API key. If None, the value of the environment
variable MISTRAL_API_KEY is used at conversion time.
safe_max_pages:
Override explícito del máximo de páginas por chunk. Si None,
se resuelve vía get_config() o el default del módulo.
safe_max_bytes:
Override explícito del máximo de bytes por chunk. Si None,
se resuelve vía get_config() o el default del módulo.
max_retries:
Override explícito del número máximo de intentos por chunk.
retry_backoff_seconds:
Override explícito de la base del backoff exponencial entre
reintentos.
error_strategy:
Override explícito de la estrategia de manejo de errores por
chunk: "fail_fast" o "best_effort".
Nota: esta estrategia solo aplica a fallos de red/API por chunk
(reintentos agotados, errores HTTP no reintentables) sobre chunks
ya generados por ``split_pdf()``. NO cubre el caso en que la
propia partición inicial falla estructuralmente
(:class:`PdfSplitError`, una página individual que excede
``safe_max_bytes`` incluso aislada) — en ese caso el documento
aborta siempre, independientemente del ``error_strategy``
configurado.
Convert a PDF file to the intermediate format.
Parameters
source:
Path to the .pdf file.
Returns
IntermediateDocument
Document with one :class:~converters.base.IntermediatePage
per PDF page recognised by Mistral OCR. If el documento fue
dividido en chunks y alguno falló en modo best_effort,
missing_page_ranges contendrá los rangos omitidos.
Raises
FileNotFoundError If source does not exist. ConversionError If the Mistral API key is not available, the HTTP request fails, or the response is in an unexpected format.
converters/docx.py — Microsoft Word DOCX to Markdown converter.
The DOCX document is split into pages using a two-layer strategy (tried in order, first match wins):
- Explicit breaks — any
<w:br w:type="page"/>run or<w:sectPr>section break triggers a page flush. A safety cap of_PARAGRAPHS_PER_PAGEelements is also applied so that a single logical section never grows unbounded. - Paragraph count — fallback; a new page every
_PARAGRAPHS_PER_PAGEelements (paragraphs + tables counted together).
No content is ever discarded.
Bases: BaseConverter
Convert .docx files to a paginated Markdown document.
Uses python-docx to iterate over paragraphs and tables in document
order. Headings are detected via paragraph style names and prefixed
with the appropriate number of # characters. Tables are rendered
as GFM Markdown tables.
Pagination strategy (first applicable wins):
- Explicit breaks (
<w:br w:type="page"/>or<w:sectPr>), with a_PARAGRAPHS_PER_PAGEsafety cap per section. - Paragraph/table count — new page every
_PARAGRAPHS_PER_PAGEelements.
Convert a DOCX file to the intermediate format.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
Path
|
Path to the |
required |
Returns:
| Type | Description |
|---|---|
IntermediateDocument
|
Multi-page document. The number of pages depends on which |
IntermediateDocument
|
pagination strategy is applied (see class docstring). |
Raises:
| Type | Description |
|---|---|
ConversionError
|
If |
FileNotFoundError
|
If source does not exist. |
converters/pptx.py — Microsoft PowerPoint PPTX to Markdown converter.
Each slide is rendered as a separate page. Text frames are converted to
Markdown (titles get a # prefix). Images are extracted as base64-encoded
PageImage objects. Tables are rendered as GFM Markdown tables.
Bases: BaseConverter
Convert .pptx files to the intermediate format.
Each slide becomes a separate IntermediatePage. Text is extracted
from all text frames (title shapes get a # prefix). Images are
base64-encoded and stored as PageImage objects. Tables are
converted to GFM Markdown tables.
Requires python-pptx and optionally Pillow for media-type
detection (falls back to image/png if Pillow is unavailable).
Convert a PPTX file to the intermediate format.
Parameters
source:
Path to the .pptx file.
Returns
IntermediateDocument Multi-page document, one page per slide.
Raises
ConversionError
If python-pptx is not installed or the file cannot be parsed.
FileNotFoundError
If source does not exist.
converters/xlsx.py — XLSX/XLS files are handled by the tabular ingestion pipeline.
XLSX/XLS files do not require conversion to an intermediate format.
Use: python main.py --stage tabular --input-raw
Bases: BaseConverter
Redirect handler for .xlsx and .xls files.
XLSX/XLS files are processed directly and efficiently by the tabular ingestion
pipeline. This converter raises a :class:ConversionError with a clear
redirect message so that callers know to use the correct pipeline stage.
converters/csv.py — CSV files are handled by the tabular ingestion pipeline.
CSV files do not require conversion to an intermediate format.
Use: python main.py --stage tabular --input-raw
Bases: BaseConverter
Redirect handler for .csv files.
CSV files are processed directly and efficiently by the tabular ingestion
pipeline. This converter raises a :class:ConversionError with a clear
redirect message so that callers know to use the correct pipeline stage.
converters/html.py — HTML/HTM to Markdown converter.
Extracts structured text from HTML using BeautifulSoup4 + lxml, converts headings/paragraphs/lists/tables to GFM Markdown, and removes noise (scripts, styles, nav, footer).
The document is split into pages at H1 headings (# … in Markdown).
Each H1 and its following content form one page. If the HTML has no H1
headings the entire document is a single page. No content is discarded.
Bases: BaseConverter
Convert .html and .htm files to the intermediate format.
Parses the HTML with BeautifulSoup4 (lxml backend) and converts the
document structure to GFM Markdown. The output is split into one page
per H1 heading (# …). Documents without H1 headings produce a
single page.
Convert an HTML file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
Path
|
Path to the |
required |
Returns:
| Type | Description |
|---|---|
IntermediateDocument
|
Multi-page document split at H1 headings (one page per H1 |
IntermediateDocument
|
section). Produces a single page when no H1 headings are |
IntermediateDocument
|
present. |
Raises:
| Type | Description |
|---|---|
ConversionError
|
If BeautifulSoup4 or lxml is not installed, or the file cannot be parsed. |
converters/api_json.py — REST JSON API to intermediate document converter.
Fetches one or more pages of a JSON API, navigates the response with
JSONPath expressions, and maps each item to an
:class:~converters.base.IntermediatePage. Supports two pagination
strategies: following a next URL field (next_url) and classic
offset/limit pagination (offset_limit).
Config files can be YAML or JSON and are validated with Pydantic v2.
Convert a REST JSON API response to the intermediate document format.
This converter does not inherit from
:class:~converters.base.BaseConverter because its source is a URL,
not a file path. Use :meth:convert_from_url as the main entry
point.
Parameters
config: Mapping configuration describing how to navigate the API response. headers: Optional HTTP headers sent with every request (e.g. auth tokens).
Fetch the API and convert the response to the intermediate format.
Parameters
url: Base URL of the API endpoint.
Returns
IntermediateDocument
One :class:~converters.base.IntermediatePage per item
extracted from the API response.
Raises
ConversionError If the HTTP request fails or the response cannot be parsed.
classmethod
Load configuration from a YAML or JSON file.
Parameters
config_path:
Path to a .yaml, .yml, or .json config file.
headers:
Optional HTTP headers forwarded to the constructor.
Returns
ApiJsonConverter
Raises
ConversionError If the file cannot be read, parsed, or validated.
Bases: BaseModel
Mapping configuration for a JSON REST API.
Parameters
document_name:
Human-readable name used in log messages.
items_path:
JSONPath expression pointing to the array of items in each
API response. Use "$" when the response is directly a list.
page_fields:
Mapping of field roles to JSONPath expressions evaluated on each
item. Recognised keys: "title", "content",
"metadata" (value may be a list of JSONPath strings).
pagination:
Optional pagination configuration.
Bases: BaseModel
Pagination strategy for a JSON API.
Parameters
type:
Pagination strategy: "next_url" follows a URL embedded in the
response; "offset_limit" increments an offset query parameter.
next_url_path:
JSONPath expression pointing to the next-page URL in the response.
Only used when type == "next_url".
offset_param:
Name of the offset query parameter.
Only used when type == "offset_limit".
limit_param:
Name of the limit query parameter.
Only used when type == "offset_limit".
limit:
Page size for offset/limit pagination.
total_path:
JSONPath expression pointing to the total item count in the
response. Only used when type == "offset_limit".
max_pages:
Hard cap on the number of requests made to prevent infinite loops.
converters/api_xml.py — REST XML / SOAP API to intermediate document converter.
Fetches XML responses (REST or SOAP), parses them with lxml, navigates
the document with XPath expressions, and maps each matched element to an
:class:~converters.base.IntermediatePage. Supports two pagination
strategies (next_url and offset_limit) and optional SOAP
envelope wrapping.
Config files can be YAML or JSON and are validated with Pydantic v2.
Convert a REST XML or SOAP API response to the intermediate format.
This converter does not inherit from
:class:~converters.base.BaseConverter because its source is a URL,
not a file path. Use :meth:convert_from_url as the main entry
point.
Parameters
config: Mapping configuration describing how to navigate the API response. headers: Optional HTTP headers merged into every request (auth tokens, custom headers, etc.). SOAP-related headers are added automatically when required.
Fetch the API and convert the response to the intermediate format.
Parameters
url:
URL of the API endpoint.
soap_body:
SOAP body content (without the envelope). If provided and
config.soap_envelope_template is not None (or
config.soap_action is set), the body is wrapped in the
SOAP envelope template before sending.
Returns
IntermediateDocument
One :class:~converters.base.IntermediatePage per XML element
matched by items_xpath.
Raises
ConversionError If the HTTP request fails or the XML cannot be parsed.
classmethod
Load configuration from a YAML or JSON file.
Parameters
config_path:
Path to a .yaml, .yml, or .json config file.
headers:
Optional HTTP headers forwarded to the constructor.
Returns
ApiXmlConverter
Raises
ConversionError If the file cannot be read, parsed, or validated.
Bases: BaseModel
Mapping configuration for an XML REST or SOAP API.
Parameters
document_name:
Human-readable name used in log messages.
items_xpath:
XPath expression selecting the elements to convert to pages
(e.g. "//record").
page_fields:
Mapping of field roles to XPath expressions evaluated on each
element. Recognised keys: "title", "content",
"metadata" (value may be a list of XPath strings).
namespaces:
XML namespace prefix-to-URI mapping passed to all XPath calls.
pagination:
Optional pagination configuration.
soap_action:
Value of the SOAPAction HTTP header. When set, requests are
sent as SOAP POSTs.
soap_envelope_template:
XML template for the SOAP envelope. Must contain a {body}
placeholder. When None and soap_action is set, the
default SOAP 1.1 envelope is used.
Bases: BaseModel
Pagination strategy for an XML API.
Parameters
type:
Pagination strategy: "next_url" follows a URL embedded in the
response; "offset_limit" increments an offset query parameter.
next_url_xpath:
XPath expression pointing to the next-page URL text.
Only used when type == "next_url".
offset_param:
Name of the offset query parameter.
Only used when type == "offset_limit".
limit_param:
Name of the limit query parameter.
Only used when type == "offset_limit".
limit:
Page size for offset/limit pagination.
total_xpath:
XPath expression pointing to the total item count text.
Only used when type == "offset_limit".
max_pages:
Hard cap on the number of requests made to prevent infinite loops.
converters/text.py — Plain text and Markdown converter.
The file is paginated into chunks of _LINES_PER_PAGE lines. Each
chunk becomes a separate IntermediatePage. .md files are passed
through unchanged (already Markdown); .txt files are used directly.
No lines are ever discarded.
Bases: BaseConverter
Convert .txt and .md files to the intermediate format.
The file content is split into pages of _LINES_PER_PAGE lines.
Markdown files are passed through as-is; plain text files are used
directly. An empty file produces a single empty page.
Convert a text or Markdown file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
Path
|
Path to the |
required |
Returns:
| Type | Description |
|---|---|
IntermediateDocument
|
Multi-page document (one page per |
IntermediateDocument
|
An empty file produces a single empty page. |
Raises:
| Type | Description |
|---|---|
ConversionError
|
If the file cannot be read. |