Media API
Speech-to-text and ffmpeg helpers used by the audio and video converters, and
the file metadata that reaches the :Document node. For the end-to-end
behaviour see the Multimodal Ingestion guide.
Transcriber and TranscriptSegment are also exported from scinr.newton.
Transcriber Interface
scinr.newton.media.transcriber
Speech-to-text interface used by the audio and video converters.
Implement :class:Transcriber to plug any provider in. The contract:
- :meth:
Transcriber.transcribeis called once per segment with a temporary FLAC file, mono, 16 kHz, at mostmedia_segment_secondslong. The converter cuts the segments withffmpeg; the transcriber never sees the whole medium. - The returned times are relative to the start of that file (the converter adds the segment offset).
- It may return a single segment with all the text when the provider gives no timestamps.
languageis an ISO 639-1 code, orNoneto let the provider detect it.- Raise any exception on failure: the converter applies
media_error_strategyto the segment (best_effortskips it and records it inmissing_segments;fail_fastfails the document). Retries are the implementation's job.
Transcriber
Bases: ABC
Turns one audio segment into text. See the module docstring for the contract.
transcribe
abstractmethod
async
Transcribe the audio file path (FLAC, mono, 16 kHz, one segment).
Parameters
path:
Temporary file with the segment's audio. Deleted by the caller
right after this call returns.
language:
ISO 639-1 code ("es", "en"…) or None for automatic
detection.
Returns
list[TranscriptSegment] Segments in time order, times relative to the start of path. An empty list (or segments with empty text) means silence.
TranscriptSegment
Bases: BaseModel
A piece of transcribed speech.
Attributes
start_ms, end_ms:
Time range in milliseconds, relative to the start of the file passed
to :meth:Transcriber.transcribe.
text:
Transcribed text.
speaker:
Speaker label. Not used in this version (no diarisation).
Ready-made Transcribers
scinr.newton.media.transcribers.openai_compatible
Transcription through an OpenAI-compatible /audio/transcriptions endpoint.
The multipart body is streamed from disk (one block in memory at a time).
Works with OpenAI and with any server that implements the same endpoint
(self-hosted Whisper servers, vLLM, other providers): point base_url at
it. api_key is optional for servers that do not ask for one.
Timestamps: the request asks for response_format=verbose_json with
timestamp_granularities[]=segment. Servers and models that reject those
parameters (HTTP 400 / 422) get the same segment again with
response_format=json, and the instance remembers it so the next segments
go straight to json. Without timestamps the whole segment is one
:class:~scinr.newton.media.transcriber.TranscriptSegment.
Silence: Whisper tends to invent text on silent or music-only audio. With
verbose_json each segment carries no_speech_prob and
avg_logprob; a segment with no_speech_prob > no_speech_threshold and
avg_logprob < -1 is dropped (Whisper's own silence rule). Servers that
do not report those fields, or answer plain json, are not filtered.
OpenAICompatibleTranscriber
Bases: Transcriber
POST {base_url}/audio/transcriptions (multipart, file streamed from disk).
Parameters
model:
Model name on the server ("whisper-1", "gpt-4o-mini-transcribe",
the name a self-hosted server expects…).
api_key:
Bearer token. None sends no Authorization header.
base_url:
API root, without /audio/transcriptions.
timeout:
Seconds per request.
max_retries:
Attempts per segment on network errors and HTTP 429 / 5xx, with
exponential backoff.
retry_backoff_seconds:
Base of the backoff.
no_speech_threshold:
Drop the segments the model itself marks as silence (see the module
docstring). None keeps every segment.
TranscriptionError
Bases: RuntimeError
The transcription request failed (after retries, when retryable).
scinr.newton.media.transcribers.faster_whisper
Local transcription with faster-whisper (pip install "scinr[media]").
The model is loaded once per process and cached by
(model_size, device, compute_type). Transcriptions run in a worker thread,
one at a time per model (a threading.Lock): the model is not shared by
concurrent calls, and its memory is paid once.
Silence: Whisper tends to invent text ("Thanks for watching") on silent or
music-only audio. vad_filter=True (the default here, unlike
faster-whisper's own default) runs its bundled Silero voice-activity
detector first, so the stretches without speech are not transcribed and a
silent segment produces no page.
FasterWhisperTranscriber
Bases: Transcriber
Transcribe locally with a Whisper model through faster-whisper.
Parameters
model_size:
Model name or path ("tiny", "base", "small", "medium",
"large-v3"…).
device:
"auto", "cpu" or "cuda".
compute_type:
"default", "int8", "float16"…
vad_filter:
Skip the stretches without speech before transcribing (see the
module docstring). False transcribes everything.
vad_parameters:
Options of the voice-activity detector (faster-whisper's
VadOptions, e.g. {"min_silence_duration_ms": 500}).
ffmpeg Helpers
scinr.newton.media.ffmpeg
ffmpeg / ffprobe helpers for the audio and video converters.
Every call is an asyncio subprocess with an argument list (never a
shell) and absolute paths; stdout goes to DEVNULL (ffprobe reads
its JSON from stdout) and only the tail of stderr is kept for the
error message. The medium is never read in Python: ffmpeg writes one
segment (or the frames of one segment) to a temporary file at a time.
ffmpeg and ffprobe are system binaries, not installed with the
package: ffmpeg_path / ffprobe_path in :func:~scinr.newton.config.configure,
or the PATH.
MediaInfo
dataclass
What ffprobe reports about a medium.
extract_audio_segment
async
extract_audio_segment(
ffmpeg: str,
src: Path,
start_s: float,
duration_s: float,
dst: Path,
) -> None
Write [start_s, start_s + duration_s) of src to dst as FLAC, mono, 16 kHz.
extract_frames
async
extract_frames(
ffmpeg: str,
src: Path,
start_s: float,
duration_s: float,
count: int,
max_side: int,
out_dir: Path,
) -> list[Path]
Write count frames spread uniformly over the segment to out_dir as JPEG.
Frames are scaled down to fit in max_side × max_side (aspect kept).
Returns the frame files in time order.
find_binaries
Resolve ffmpeg and ffprobe (configured path, else the PATH).
Raises
ConversionError Either binary is missing, with how to install or configure it.
File Metadata
scinr.newton.models.file_metadata
File-level metadata of a source file (images, audio, video).
Closed model on purpose: the ingestion writes every key of
:data:FILE_METADATA_KEYS on the :Document node (file_<key>) on every
ingestion, null included, so update_mode never leaves stale
properties behind, and no property name ever comes from data.
FileMetadata
Bases: BaseModel
Metadata read from a source file. Every field is optional.
Attributes
captured_at:
Capture time in ISO 8601 (EXIF DateTimeOriginal), as written by the
device (no time zone conversion).
device_make, device_model:
Camera or device make and model (EXIF).
gps_latitude, gps_longitude:
Location in decimal degrees (EXIF GPS). Never set when the
strip_location option is True (the default).
width, height:
Size in pixels (images and video).
duration_ms:
Duration in milliseconds (audio and video).
container, audio_codec, video_codec:
Container format and codecs, as reported by ffprobe.
frame_rate:
Frames per second (video).