Skip to content

Media API

Speech-to-text and ffmpeg helpers used by the audio and video converters, and the file metadata that reaches the :Document node. For the end-to-end behaviour see the Multimodal Ingestion guide.

Transcriber and TranscriptSegment are also exported from scinr.newton.

Transcriber Interface

scinr.newton.media.transcriber

Speech-to-text interface used by the audio and video converters.

Implement :class:Transcriber to plug any provider in. The contract:

  • :meth:Transcriber.transcribe is called once per segment with a temporary FLAC file, mono, 16 kHz, at most media_segment_seconds long. The converter cuts the segments with ffmpeg; the transcriber never sees the whole medium.
  • The returned times are relative to the start of that file (the converter adds the segment offset).
  • It may return a single segment with all the text when the provider gives no timestamps.
  • language is an ISO 639-1 code, or None to let the provider detect it.
  • Raise any exception on failure: the converter applies media_error_strategy to the segment (best_effort skips it and records it in missing_segments; fail_fast fails the document). Retries are the implementation's job.

Transcriber

Bases: ABC

Turns one audio segment into text. See the module docstring for the contract.

transcribe abstractmethod async

transcribe(
    path: Path, language: str | None = None
) -> list[TranscriptSegment]

Transcribe the audio file path (FLAC, mono, 16 kHz, one segment).

Parameters

path: Temporary file with the segment's audio. Deleted by the caller right after this call returns. language: ISO 639-1 code ("es", "en"…) or None for automatic detection.

Returns

list[TranscriptSegment] Segments in time order, times relative to the start of path. An empty list (or segments with empty text) means silence.

TranscriptSegment

Bases: BaseModel

A piece of transcribed speech.

Attributes

start_ms, end_ms: Time range in milliseconds, relative to the start of the file passed to :meth:Transcriber.transcribe. text: Transcribed text. speaker: Speaker label. Not used in this version (no diarisation).

Ready-made Transcribers

scinr.newton.media.transcribers.openai_compatible

Transcription through an OpenAI-compatible /audio/transcriptions endpoint.

The multipart body is streamed from disk (one block in memory at a time).

Works with OpenAI and with any server that implements the same endpoint (self-hosted Whisper servers, vLLM, other providers): point base_url at it. api_key is optional for servers that do not ask for one.

Timestamps: the request asks for response_format=verbose_json with timestamp_granularities[]=segment. Servers and models that reject those parameters (HTTP 400 / 422) get the same segment again with response_format=json, and the instance remembers it so the next segments go straight to json. Without timestamps the whole segment is one :class:~scinr.newton.media.transcriber.TranscriptSegment.

Silence: Whisper tends to invent text on silent or music-only audio. With verbose_json each segment carries no_speech_prob and avg_logprob; a segment with no_speech_prob > no_speech_threshold and avg_logprob < -1 is dropped (Whisper's own silence rule). Servers that do not report those fields, or answer plain json, are not filtered.

OpenAICompatibleTranscriber

Bases: Transcriber

POST {base_url}/audio/transcriptions (multipart, file streamed from disk).

Parameters

model: Model name on the server ("whisper-1", "gpt-4o-mini-transcribe", the name a self-hosted server expects…). api_key: Bearer token. None sends no Authorization header. base_url: API root, without /audio/transcriptions. timeout: Seconds per request. max_retries: Attempts per segment on network errors and HTTP 429 / 5xx, with exponential backoff. retry_backoff_seconds: Base of the backoff. no_speech_threshold: Drop the segments the model itself marks as silence (see the module docstring). None keeps every segment.

TranscriptionError

Bases: RuntimeError

The transcription request failed (after retries, when retryable).

scinr.newton.media.transcribers.faster_whisper

Local transcription with faster-whisper (pip install "scinr[media]").

The model is loaded once per process and cached by (model_size, device, compute_type). Transcriptions run in a worker thread, one at a time per model (a threading.Lock): the model is not shared by concurrent calls, and its memory is paid once.

Silence: Whisper tends to invent text ("Thanks for watching") on silent or music-only audio. vad_filter=True (the default here, unlike faster-whisper's own default) runs its bundled Silero voice-activity detector first, so the stretches without speech are not transcribed and a silent segment produces no page.

FasterWhisperTranscriber

Bases: Transcriber

Transcribe locally with a Whisper model through faster-whisper.

Parameters

model_size: Model name or path ("tiny", "base", "small", "medium", "large-v3"…). device: "auto", "cpu" or "cuda". compute_type: "default", "int8", "float16"… vad_filter: Skip the stretches without speech before transcribing (see the module docstring). False transcribes everything. vad_parameters: Options of the voice-activity detector (faster-whisper's VadOptions, e.g. {"min_silence_duration_ms": 500}).

ffmpeg Helpers

scinr.newton.media.ffmpeg

ffmpeg / ffprobe helpers for the audio and video converters.

Every call is an asyncio subprocess with an argument list (never a shell) and absolute paths; stdout goes to DEVNULL (ffprobe reads its JSON from stdout) and only the tail of stderr is kept for the error message. The medium is never read in Python: ffmpeg writes one segment (or the frames of one segment) to a temporary file at a time.

ffmpeg and ffprobe are system binaries, not installed with the package: ffmpeg_path / ffprobe_path in :func:~scinr.newton.config.configure, or the PATH.

MediaInfo dataclass

What ffprobe reports about a medium.

extract_audio_segment async

extract_audio_segment(
    ffmpeg: str,
    src: Path,
    start_s: float,
    duration_s: float,
    dst: Path,
) -> None

Write [start_s, start_s + duration_s) of src to dst as FLAC, mono, 16 kHz.

extract_frames async

extract_frames(
    ffmpeg: str,
    src: Path,
    start_s: float,
    duration_s: float,
    count: int,
    max_side: int,
    out_dir: Path,
) -> list[Path]

Write count frames spread uniformly over the segment to out_dir as JPEG.

Frames are scaled down to fit in max_side × max_side (aspect kept). Returns the frame files in time order.

find_binaries

find_binaries(
    ffmpeg_path: str | None, ffprobe_path: str | None
) -> Binaries

Resolve ffmpeg and ffprobe (configured path, else the PATH).

Raises

ConversionError Either binary is missing, with how to install or configure it.

probe async

probe(ffprobe: str, path: Path) -> MediaInfo

Duration, streams and codecs of path (ffprobe -show_format -show_streams).

File Metadata

scinr.newton.models.file_metadata

File-level metadata of a source file (images, audio, video).

Closed model on purpose: the ingestion writes every key of :data:FILE_METADATA_KEYS on the :Document node (file_<key>) on every ingestion, null included, so update_mode never leaves stale properties behind, and no property name ever comes from data.

FileMetadata

Bases: BaseModel

Metadata read from a source file. Every field is optional.

Attributes

captured_at: Capture time in ISO 8601 (EXIF DateTimeOriginal), as written by the device (no time zone conversion). device_make, device_model: Camera or device make and model (EXIF). gps_latitude, gps_longitude: Location in decimal degrees (EXIF GPS). Never set when the strip_location option is True (the default). width, height: Size in pixels (images and video). duration_ms: Duration in milliseconds (audio and video). container, audio_codec, video_codec: Container format and codecs, as reported by ffprobe. frame_rate: Frames per second (video).