Frame Extraction

Provides the parallel DeepLabCut k-means and model-outlier frame-extraction pipelines.

class sollertia_video_tracking.frame_extraction.AggregateBar(progress_queue, total_video_count, frame_totals, preparing_label='preparing...', stream=None, *, key_video=None)

Bases: LiveBar

Renders a single progress bar that tracks the total frames read across all extraction or inference workers.

Notes

The renderer consumes ("progress", video_index, count) and ("done", video_index) messages from the shared queue, on top of the warm-up, spinner, interval, and stop-sentinel handling inherited from LiveBar. Each reporter announces its video with a count of zero before its first frame is decoded, so the bar leaves the warm-up line for the active bar as soon as any worker begins. It counts every announced but unfinished video as actively decoding.

_total_video_count

The total number of videos in the run.

_frame_totals

The mapping of work-unit key to the number of frames that unit contributes to the bar.

_grand_frame_total

The sum of all per-unit frame totals, clamped to at least one.

_frames

The mapping of work-unit key to the most recent frame count reported for that unit.

_key_video

The mapping of work-unit key to the video index it belongs to, so chunk completions roll up.

_video_remaining

The mapping of video index to the number of its work units that have not yet finished.

_videos_done

The number of videos that have finished.

class sollertia_video_tracking.frame_extraction.ExtractionAlgorithm(*values)

Bases: StrEnum

Defines the supported algorithms for selecting which flagged candidate frames to extract.

KMEANS = 'kmeans'

Clusters the flagged candidates and keeps one representative frame per cluster.

UNIFORM = 'uniform'

Keeps flagged candidates spread uniformly across the flagged range.

class sollertia_video_tracking.frame_extraction.FrameExtractionSummary(extracted_video_count, cleared_frame_count, total_video_count, worker_count, used_core_count, total_core_count, clustering_frame_count, existing_frame_count=0, target_frame_count=-1, errors=())

Bases: object

Summarizes the outcome of a parallel k-means frame-extraction run.

Notes

The pipeline never aborts on a single bad video, so failures are collected here as (video_path, detail) pairs rather than raised, where the detail is an error:-prefixed traceback or the marker "empty" for a video that produced no frames. Callers inspect failed_video_count (or errors) to decide the process exit status.

cleared_frame_count: int

The number of unlabeled bootstrap frames removed across the selection by overwrite or reset before re-extraction; zero when neither option was set.

clustering_frame_count: int

The total number of frames scheduled to be read and clustered across the videos selected for extraction this pass, estimated from the video headers before extraction. Videos already at the per-video ceiling, and videos already in outlier refinement, are excluded from this count.

errors: tuple[tuple[str, str], ...]

The (video_path, detail) pairs for every video that failed to extract or produced no frames, where the detail is an error:-prefixed traceback or the marker "empty".

existing_frame_count: int

The number of frames already extracted across the selection before this run, reported in sampling mode.

extracted_video_count: int

The number of videos for which frames were freshly extracted.

property failed_video_count: int

Returns the number of videos that failed to extract or produced no frames.

property successful: bool

Returns whether the run completed with no failed videos.

target_frame_count: int

The requested total-frame budget when sampling videos, or -1 when budgeted sampling was disabled.

total_core_count: int

The total number of CPU cores available on the machine.

total_video_count: int

The total number of videos considered in the run.

used_core_count: int

The number of distinct CPU cores the workers were pinned across.

worker_count: int

The number of worker processes that ran concurrently.

class sollertia_video_tracking.frame_extraction.OutlierAlgorithm(*values)

Bases: StrEnum

Defines the supported frame-selection modes.

jump, uncertain, and fitting flag likely-wrong frames from a trained model’s predictions; list extracts an explicit, caller-supplied index list instead.

FITTING = 'fitting'

Flags frames that depart from a fitted per-keypoint motion trajectory.

JUMP = 'jump'

Flags frames in which a bodypart moves further than the pixel threshold from the previous frame.

LIST = 'list'

Extracts an explicit, caller-supplied list of frame indices instead of detecting outliers.

UNCERTAIN = 'uncertain'

Flags frames holding any bodypart predicted below the confidence threshold.

class sollertia_video_tracking.frame_extraction.OutlierExtractionSummary(config_path, outlier_algorithm, extraction_algorithm, total_video_count, extracted_video_count, worker_count, used_core_count, total_core_count, candidate_frame_count, extracted_frame_count, unanalyzed_videos=(), errors=())

Bases: object

Summarizes the outcome of a parallel outlier-frame extraction run.

Notes

The pipeline never aborts on a single bad video, so per-video problems are collected here rather than raised. Videos whose predictions are missing are listed in unanalyzed_videos (they must be analyzed first), and videos that raised during detection or extraction are listed in errors as (video_path, detail) pairs. Callers inspect successful to decide the process exit status.

candidate_frame_count: int

The number of candidate frames submitted for extraction across all videos that had candidates, after any candidate_step sub-sampling of the flagged frames.

config_path: Path

The path to the DeepLabCut project’s config.yaml the run used.

describe()

Builds a one-line human-readable summary of the extraction run for the CLI.

Return type:

str

Returns:

A compact description of how many videos yielded outlier frames and how many frames were written.

errors: tuple[tuple[str, str], ...]

The (video_path, detail) pairs for every video that raised during detection or extraction.

extracted_frame_count: int

The total number of frames freshly written into the project’s labeled-data tree across all videos.

extracted_video_count: int

The number of videos from which outlier frames were extracted.

extraction_algorithm: ExtractionAlgorithm

The frame-selection algorithm that picked the extracted frames from the candidates.

property failed_video_count: int

Returns the number of videos that raised during detection or extraction.

outlier_algorithm: OutlierAlgorithm

The outlier-detection algorithm that flagged the candidate frames.

property successful: bool

Returns whether the run completed with every video analyzed and extracted without error.

total_core_count: int

The total number of CPU cores available on the machine.

total_video_count: int

The total number of videos considered in the run.

unanalyzed_videos: tuple[str, ...]

The videos skipped because no matching predictions were found; they must be analyzed before refinement.

used_core_count: int

The number of distinct CPU cores the extraction workers were pinned across.

worker_count: int

The number of worker processes that ran concurrently during the extraction phase.

class sollertia_video_tracking.frame_extraction.PurgeSummary(config_path, executed, removed_directories, labeled_directories, frame_count, unmatched_videos=())

Bases: object

Summarizes a wholesale labeled-data purge, whether previewed as a dry run or actually executed.

Notes

A purge is deliberately destructive: unlike the frame and outlier re-extraction options, which preserve every human-labeled and machine-labeled frame, it removes each targeted video’s entire labeled-data directory, including its labels. The executed flag distinguishes a dry-run preview from a completed deletion, and labeled_directories names the directories that held human labels so callers can warn before deleting them.

config_path: Path

The path to the DeepLabCut project’s config.yaml the purge targeted.

executed: bool

Indicates whether the directories were actually deleted (True) or only previewed as a dry run (False).

frame_count: int

The total number of extracted frames across the targeted directories, reported to convey the purge’s scope.

labeled_directories: tuple[Path, ...]

The targeted directories that held a human CollectedData label file, whose labels the purge discards.

property labeled_directory_count: int

Returns the number of targeted directories that held human labels.

removed_directories: tuple[Path, ...]

The labeled-data/<stem> directories that were removed, or that a dry run would remove.

property removed_directory_count: int

Returns the number of labeled-data directories removed, or that a dry run would remove.

unmatched_videos: tuple[str, ...]

The requested videos that matched no registered project video and were skipped.

class sollertia_video_tracking.frame_extraction.RefinementDirectoryStatus(directory, unrefined_frame_count)

Bases: object

Summarizes one video directory’s machine-labeled frames still awaiting refinement for the current iteration.

directory: Path

The labeled-data/<stem> directory that still holds unrefined machine frames.

unrefined_frame_count: int

The number of the directory’s current-iteration machine frames the human has not refined yet.

class sollertia_video_tracking.frame_extraction.RefinementStatusSummary(config_path, iteration, pending_directories, unmatched_videos=(), unreadable=())

Bases: object

Summarizes which of a project’s video directories still hold machine-labeled frames awaiting refinement.

Notes

Only directories that still hold unrefined machine frames for the project’s current refinement iteration are recorded in pending_directories; directories that are already fully refined, purely human-labeled, or still bootstrapping are left out because they need no attention this round. This is what extract pending lists.

config_path: Path

The path to the DeepLabCut project’s config.yaml the status was computed from.

describe()

Builds a one-line human-readable summary of the pending refinement work for the CLI.

Return type:

str

Returns:

A compact description of how many directories and frames still need refinement.

iteration: int

The project’s current refinement iteration, whose machine-label tables were inspected.

pending_directories: tuple[RefinementDirectoryStatus, ...]

The directories that still hold current-iteration machine frames the human has not refined.

property pending_directory_count: int

Returns the number of directories that still need refinement.

property pending_frame_count: int

Returns the total number of unrefined machine frames across all pending directories.

property successful: bool

Returns whether every scanned directory’s label tables were read without error.

unmatched_videos: tuple[str, ...]

The requested videos that matched no registered project video and were skipped.

unreadable: tuple[tuple[Path, str], ...]

The (directory, detail) pairs for directories whose label tables could not be read and were skipped.

class sollertia_video_tracking.frame_extraction.TrackingMethod(*values)

Bases: StrEnum

Defines the supported multi-animal trackers that may have produced a video’s predictions.

BOX = 'box'

Identifies the bounding-box tracker.

ELLIPSE = 'ellipse'

Identifies the ellipse tracker.

SKELETON = 'skeleton'

Identifies the skeleton tracker.

sollertia_video_tracking.frame_extraction.extract_frames_kmeans(config_path, *, clustering_stride=1, worker_count=-1, cores_per_worker=-1, reserved_core_count=2, frames_per_video=-1, total_frame_budget=-1, balance_groups=False, group_by_pattern=None, requested_videos=(), exclusive=False, clustering_resize_width=30, cluster_in_color=False, overwrite=False, reset=False, display_progress=True)

Runs DeepLabCut k-means frame extraction across a project’s videos in parallel and reports the outcome.

Reads the run parameters from the project’s config.yaml, plans the CPU-core allocation, and clusters every selected video in its own pinned worker process. numframes2pick is a per-video ceiling: each selected video is topped up to it, so a not-yet-extracted video gains a full set while a partly-extracted one gains only the frames that reach the ceiling, and a video already at the ceiling is skipped. overwrite and reset instead clear a video’s unlabeled bootstrap frames first so it is re-rolled from scratch, always preserving every human-labeled and outlier-extracted frame. Frame extraction is the bootstrap step that precedes outlier refinement, so a video already in refinement is off-limits: it is dropped from the candidate pool, and an explicit refined requested_videos target is refused under overwrite. A single bad video is recorded in the returned summary rather than aborting the run.

When total_frame_budget is set, the run selects just enough videos to reach that project-wide frame total, preferring not-yet-extracted videos and falling back to below-ceiling ones, so coverage grows before existing videos are deepened. The full project’s video set can be added once and repeated passes grow it toward the budget without manual selection. When the existing frames already meet the budget, the run extracts nothing and warns; if even topping every eligible video to the ceiling cannot reach the budget in one pass, the run reports the shortfall and raises rather than extracting a partial set.

Notes

The pipeline uses the spawn multiprocessing start method on every platform for reproducible behavior, so a programmatic caller must guard the call with if __name__ == "__main__":. The installed console-script entry point is already guarded. CPU-affinity pinning is applied on Linux and Windows; macOS exposes no affinity API, so its workers run unpinned but still in parallel.

overwrite and reset remove only a video’s bare bootstrap frames: the extracted images that carry no human label and belong to no outlier-refinement iteration. Frames the human has annotated (a finite CollectedData coordinate) and machine-labeled outlier frames are always kept, so re-rolling the diverse bootstrap set never disturbs labeling or outlier work. overwrite clears the videos this run selects to process (refusing any already in outlier refinement), while reset clears every non-refined project video, so the run re-rolls its whole selection or the whole project respectively. The video subset is drawn fresh each run, and k-means picks the frames within a video, so a re-rolled selection differs each run. To reproduce a specific selection, name the videos explicitly with requested_videos. To instead discard a video’s labels and start its labeled-data directory over from scratch, use purge_labeled_data.

Empty labeled-data directories left by videos that were registered but never extracted are removed after every run, so the labeling GUI shows only the videos that have frames.

Parameters:

• config_path (Path) – The path to the DeepLabCut project’s config.yaml.

• clustering_stride (int, default: 1) – The clustering stride passed to DeepLabCut as cluster_step; every Nth frame is sampled, where N is this stride.

• worker_count (int, default: -1) – The number of videos to decode in parallel. Set to -1 to fill the usable cores automatically.

• cores_per_worker (int, default: -1) – The number of CPU cores pinned to each worker. Set to -1 to spread the usable cores evenly.

• reserved_core_count (int, default: 2) – The number of CPU cores to leave free for other tasks.

• frames_per_video (int, default: -1) – The per-video frame ceiling, overriding numframes2pick in config.yaml. Each selected video is topped up to this many frames. Set to -1 to use the value already stored in the configuration file.

• total_frame_budget (int, default: -1) – The total number of frames the project should hold, reached by topping up videos toward their per-video ceiling, preferring not-yet-extracted videos over below-ceiling ones. Set to -1 to top up every below-ceiling selected video instead of sampling toward a budget.

• balance_groups (bool, default: False) – Determines whether the budgeted sampling is balanced across groups rather than drawn uniformly, so every group is represented and coverage evens out across repeated passes. The group of each video is inferred from its file name, with videos that share their non-date name components grouped together. Only affects the run when total_frame_budget is set.

• group_by_pattern (str | None, default: None) – A regular expression whose first capturing group names the group for each video’s file-name stem, overriding the built-in inference for naming schemes it does not cover. Setting it implies balance_groups.

• requested_videos (tuple[str | Path, ...], default: ()) – The specific project video files this run targets, matched against the project’s registered videos by resolved path. In budgeted mode they are the always-included pins, selected before the remaining budget is filled from the project’s other videos. With exclusive they are the whole set to top up. Ignored when neither a budget, exclusive, nor overwrite applies.

• exclusive (bool, default: False) – Determines whether to restrict the run to exactly the requested_videos, topping each up to frames_per_video frames and bypassing the total-frame budget and group balancing. Requires requested_videos to be non-empty. A requested video already at the ceiling is skipped unless overwrite clears it first.

• clustering_resize_width (int, default: 30) – The downsample width applied before clustering, passed to DeepLabCut as cluster_resizewidth.

• cluster_in_color (bool, default: False) – Determines whether to cluster on color channels instead of grayscale.

• overwrite (bool, default: False) – Determines whether to clear the selected videos’ unlabeled bootstrap frames before re-extracting, so the run re-rolls the diverse selection for every video it processes rather than topping it up. Any requested_videos already in outlier refinement are refused. Human labels and outlier-extracted frames are preserved. Mutually exclusive with reset.

• reset (bool, default: False) – Determines whether to clear every registered project video of its unlabeled bootstrap frames before re-extracting, re-rolling the diverse selection project-wide. Videos already in outlier refinement are left untouched. Human labels and outlier-extracted frames are preserved. Mutually exclusive with overwrite.

• display_progress (bool, default: True) – Determines whether to render the run header and the aggregate progress bar to the standard error stream.

Return type:

FrameExtractionSummary

Returns:

A FrameExtractionSummary describing how many videos were extracted and failed and how many bootstrap frames were cleared, alongside the resolved core-allocation plan and, in sampling mode, the existing and target frame counts.

Raises:

• FileNotFoundError – If config_path does not point to an existing file.

• ValueError – Raised when the options conflict: overwrite and reset are both set, exclusive and reset are both set, or exclusive is set without any requested_videos. Raised when overwrite targets a video already in outlier refinement. Raised when a value is out of range: clustering_stride is below one, or frames_per_video or total_frame_budget (other than the -1 sentinel) is below one. Raised when the project’s numframes2pick is not a positive integer, when config.yaml defines no video_sets, or when an exclusive run’s requested videos match no eligible registered project video. Raised when a budgeted run cannot reach total_frame_budget in one pass even after topping every eligible video to its ceiling. Raised when two selected videos share a file-name stem and would collide in the labeled-data tree.

sollertia_video_tracking.frame_extraction.extract_outlier_frames_parallel(config_path, videos, *, shuffle_index=1, training_set_index=0, outlier_algorithm=OutlierAlgorithm.JUMP, explicit_frame_indices=(), comparison_bodyparts=(), pixel_distance_threshold=20.0, minimum_confidence=0.01, autoregressive_degree=3, moving_average_degree=1, extraction_algorithm=ExtractionAlgorithm.KMEANS, candidate_step=1, frames_per_video=-1, clustering_resize_width=30, cluster_in_color=False, save_labeled_frames=False, tracking_method=None, pose_snapshot_index=None, detector_snapshot_index=None, worker_count=-1, cores_per_worker=-1, reserved_core_count=2, fitting_worker_count=-1, overwrite=False, reset=False, display_progress=True)

Flags and extracts a trained model’s likely-wrong frames across many analyzed videos in parallel.

Reads the predictions a trained model already wrote for each video (which must therefore be analyzed first), flags putative outlier frames with the chosen algorithm, and pulls a numframes2pick budget of them into each video’s labeled-data directory for correction. The run has two phases. Detection loads every video’s predictions and computes its outlier candidates, fanning the fitting algorithm’s per-keypoint SARIMAX fits out across a process pool that spans the whole run. Extraction then decodes and selects the frames one video per pinned worker. Only videos already registered in the project’s config.yaml are refined, matched by resolved path like the k-means extractor, so the workers only ever read the configuration file and never race on writing it. A single bad video is recorded in the returned summary rather than aborting the run.

Notes

The pipeline uses the spawn multiprocessing start method on every platform, so a programmatic caller must guard the call with if __name__ == "__main__":. The installed console-script entry point is already guarded. Outlier extraction is additive by default: re-running a video appends further frames rather than replacing the existing ones, so coverage grows across repeated passes. Setting overwrite first discards the current refinement iteration’s already-extracted outlier frames for the selected videos so they are replaced instead of added to, and reset discards the iteration’s outlier frames across every project video for a clean slate. Both clear only this iteration’s freshly extracted outlier frames (those recorded in the iteration’s machine labels) and preserve every frame already carried in the human CollectedData labels.

Empty labeled-data directories left by videos that were registered but never extracted are removed after every run, so the labeling GUI shows only the videos that have frames.

Parameters:

• config_path (Path) – The path to the DeepLabCut project’s config.yaml.

• videos (list[str | Path]) – The project video files to refine on. Each must already be registered in the project’s config.yaml video_sets and analyzed. Requested paths that match no registered video are skipped with a warning. Leave empty to refine every registered video the current model has already analyzed.

• shuffle_index (int, default: 1) – The shuffle index whose trained model wrote the predictions.

• training_set_index (int, default: 0) – The training-set fraction index.

• outlier_algorithm (OutlierAlgorithm, default: <OutlierAlgorithm.JUMP: 'jump'>) – The detection algorithm: "jump", "uncertain", "fitting", or "list".

• explicit_frame_indices (tuple[int, ...], default: ()) – The explicit frame indices to extract when outlier_algorithm is "list".

• comparison_bodyparts (tuple[str, ...], default: ()) – The bodyparts the detectors consider; an empty tuple considers every bodypart.

• pixel_distance_threshold (float, default: 20.0) – The pixel bound for the jump and fitting algorithms.

• minimum_confidence (float, default: 0.01) – The likelihood bound for the uncertain algorithm and the fitting model’s missing-data mask.

• autoregressive_degree (int, default: 3) – The autoregressive degree of the fitting algorithm’s SARIMAX model.

• moving_average_degree (int, default: 1) – The moving-average degree of the fitting algorithm’s SARIMAX model.

• extraction_algorithm (ExtractionAlgorithm, default: <ExtractionAlgorithm.KMEANS: 'kmeans'>) – The frame-selection algorithm applied to the candidates: "kmeans" or "uniform".

• candidate_step (int, default: 1) – The stride at which the flagged candidates are sub-sampled before selection. A value above one keeps every candidate_step-th flagged frame, trading coverage for a smaller decode and, when it thins the candidates enough, a switch to seeking that avoids decoding the whole frame range.

• frames_per_video (int, default: -1) – The number of frames to extract per video, overriding numframes2pick in config.yaml. Set to -1 to use the value already stored in the configuration file.

• clustering_resize_width (int, default: 30) – The downsample width applied before clustering when selecting with "kmeans".

• cluster_in_color (bool, default: False) – Determines whether to cluster on color channels instead of grayscale.

• save_labeled_frames (bool, default: False) – Determines whether to also save each extracted frame with the model’s predictions drawn on it.

• tracking_method (TrackingMethod | None, default: None) – The multi-animal tracker that produced the predictions, or None to read it from the project configuration.

• pose_snapshot_index (int | None, default: None) – The pose snapshot index whose scorer named the prediction files, or None for the default.

• detector_snapshot_index (int | None, default: None) – The detector snapshot index, for top-down models, or None for the default.

• worker_count (int, default: -1) – The number of videos to extract in parallel. Set to -1 to fill the usable cores automatically.

• cores_per_worker (int, default: -1) – The number of CPU cores pinned to each extraction worker. Set to -1 to spread them evenly.

• reserved_core_count (int, default: 2) – The number of CPU cores to leave free for other tasks.

• fitting_worker_count (int, default: -1) – The number of processes fitting SARIMAX models during fitting detection. Set to -1 to use every usable core.

• overwrite (bool, default: False) – Determines whether to re-extract the videos this run refines, first discarding this refinement iteration’s already-extracted outlier frames for those videos, along with their machine labels and any manual refinement of them, so the frames are replaced rather than added to. Other videos’ outlier frames are left intact, and every frame already carried in the human CollectedData labels is preserved. Mutually exclusive with reset.

• reset (bool, default: False) – Determines whether to discard this refinement iteration’s extracted outlier frames across every project video, along with their machine labels and any manual refinement, before re-extracting the selected videos, giving the whole iteration a clean slate. Every frame already carried in the human CollectedData labels is preserved. Mutually exclusive with overwrite.

• display_progress (bool, default: True) – Determines whether to render the run header and the aggregate progress bar.

Return type:

OutlierExtractionSummary

Returns:

An OutlierExtractionSummary describing how many videos yielded frames, how many frames were written, and which videos were unanalyzed or failed.

Raises:

• FileNotFoundError – If config_path does not point to an existing file.

• ValueError – Raised when the options conflict: overwrite and reset are both set. Raised when an argument is invalid: outlier_algorithm or extraction_algorithm is unknown, frames_per_video (other than the -1 sentinel) or candidate_step is below one, or outlier_algorithm is "list" without explicit_frame_indices. Raised when the comparison bodyparts resolve to none. Raised when no videos can be refined: the project lists none in video_sets, no requested video matches a registered one, or no videos are named and the current model has analyzed none. Raised when two selected videos share a file-name stem and would collide in the labeled-data tree. Raised when the fitting algorithm is selected but the project’s numframes2pick is missing or not a positive integer and no frames_per_video override is supplied.

sollertia_video_tracking.frame_extraction.make_progress_reporter(progress_queue, video_index, frame_total)

Builds a drop-in replacement for tqdm that streams frame counts from a worker to the parent process.

DeepLabCut wraps its frame-reading loop in tqdm(enumerate(index)). The returned callable wraps that same iterable, forwards every item unchanged, and reports progress (throttled to a bounded number of messages per video) so the parent can render a single aggregate bar.

Parameters:

• progress_queue (Any) – The shared queue used to forward progress messages to the parent renderer.

• video_index (int) – The index identifying which video this reporter tracks.

• frame_total (int) – The total number of frames this video contributes to the aggregate bar.

Return type:

Callable[..., Iterable[Any]]

Returns:

A callable that mirrors the tqdm interface and emits throttled progress messages while iterating.

sollertia_video_tracking.frame_extraction.plan_core_allocation(video_count, total_core_count, worker_count, cores_per_worker, reserved_core_count)

Determines how many videos run at once and which cores each worker is pinned to.

Partitions the usable cores (total minus reserved) into one disjoint block per worker, so the running workers occupy the usable cores without oversubscribing them. When both the worker count and cores-per-worker are left automatic, each worker is given a roughly saturating core block and only as many workers run as the selected videos need. The full core budget is therefore deployed only when there are enough videos to fill it, and any remaining videos run in later waves. When more videos than full blocks remain and the budget does not divide evenly, one extra worker runs on the leftover cores as a smaller block so the whole budget is used. An explicit cores-per-worker instead gives each worker exactly that many cores, sizing an automatic worker count from the usable cores, while an explicit worker count splits the usable cores evenly across those workers. In both cases the worker count is capped at the number of videos, so a request for more workers than videos runs one worker per video. The pinned blocks stay within the usable band, leaving the reserved cores free regardless. An explicit request that cannot fit (more workers than usable cores, or a worker count times cores-per-worker that exceeds them) raises ValueError instead of overlapping the blocks and thrashing.

Notes

DeepLabCut reads a single video’s frames in one serial Python loop that cannot be sped up, but each frame’s HEVC / H264 decode is itself multithreaded and keeps several cores busy. Throughput therefore comes from decoding many videos at once rather than from accelerating any single video.

Parameters:

• video_count (int) – The number of videos that will be processed.

• total_core_count (int) – The total number of CPU cores available on the machine.

• worker_count (int) – The requested number of concurrent workers, or -1 to resolve the count automatically.

• cores_per_worker (int) – The requested number of cores per worker. Set to -1 to give each worker a saturating core block when the worker count is automatic, or to split the usable cores evenly across an explicit worker count.

• reserved_core_count (int) – The number of cores to leave free for other tasks.

Return type:

tuple[int, list[set[int]]]

Returns:

A tuple of the resolved worker count and a list of core-id sets, one per worker.

Raises:

ValueError – If an explicit worker count or cores-per-worker (or their combination) needs more cores than are usable, which would force two or more workers to share cores. Lower the request so the workers’ core blocks fit within the usable cores.

sollertia_video_tracking.frame_extraction.purge_labeled_data(config_path, *, videos=(), execute=False)

Removes targeted videos’ entire labeled-data directories, previewing the deletion unless execute is set.

This is the wholesale counterpart to the frame and outlier re-extraction options. Where those clear only a video’s unlabeled bootstrap frames or a single iteration’s outlier frames and always keep the human labels, a purge deletes each targeted directory outright, labels included. It exists for the rare start-completely-over case, such as changing the project crop, that the label-preserving options cannot serve. It defaults to a dry run so the caller sees the scope before committing.

Parameters:

• config_path (Path) – The path to the DeepLabCut project’s config.yaml, whose parent holds the labeled-data tree.

• videos (tuple[str | Path, ...], default: ()) – The specific project video files whose directories to purge, matched to the project’s registered videos by resolved path. Leave empty to purge every video directory in the project’s labeled-data tree.

• execute (bool, default: False) – Determines whether to actually delete the directories. When False, the directories are only reported for a dry-run preview and nothing is removed.

Return type:

PurgeSummary

Returns:

A PurgeSummary naming the directories removed (or that a dry run would remove), which of them held human labels, the total frame count, and any requested videos that matched no registered project video.

Raises:

FileNotFoundError – If config_path does not point to an existing file.

sollertia_video_tracking.frame_extraction.summarize_refinement_status(config_path, *, videos=())

Reports which video directories still hold machine-labeled outlier frames the human has not refined.

Outlier extraction writes a trained model’s likely-wrong frames into each video’s machinelabels-iter<N>.h5 table for the project’s current iteration, and the human refines them in the labeling GUI, which saves the corrected coordinates into the directory’s human CollectedData labels. A machine frame therefore counts as refined once it carries a finite human coordinate; an all-NaN placeholder row the GUI writes for an opened-but-untouched frame does not count. This scans the project’s labeled-data tree and reports, per directory, how many of the current iteration’s machine frames remain unrefined, so the human knows which directories to open next. It only reads the project, mutating nothing.

Parameters:

• config_path (Path) – The path to the DeepLabCut project’s config.yaml, whose parent holds the labeled-data tree.

• videos (tuple[str | Path, ...], default: ()) – The specific project video files to inspect, matched to the project’s registered videos by resolved path. Leave empty to inspect every directory in the project’s labeled-data tree.

Return type:

RefinementStatusSummary

Returns:

A RefinementStatusSummary listing the directories that still hold unrefined current-iteration machine frames, any requested videos that matched no registered project video, and any directories whose label tables could not be read.

Raises:

FileNotFoundError – If config_path does not point to an existing file.

Model Training

Provides hardware-optimized DeepLabCut model training, including device profiling and mixed-precision DDP runners.

class sollertia_video_tracking.training.MultiGpuStrategy(*values)

Bases: StrEnum

The multi-GPU execution strategy: automatic, DDP, DataParallel, or a single device.

AUTO = 'auto'

Selects DistributedDataParallel when two or more GPUs are chosen, otherwise a single device. Request-only.

DDP = 'ddp'

Trains with DistributedDataParallel, one process per GPU.

DP = 'dp'

Trains with DataParallel in one process, which is slower and cannot combine with mixed precision.

SINGLE = 'single'

Trains on a single device. This is the resolved outcome of selecting one GPU, not a user-selectable request.

class sollertia_video_tracking.training.OptimizationProfile(device, gpus, multi_gpu_strategy, amp_dtype, use_gradient_scaler, tf32, cudnn_benchmark, torch_compile, dataloader_workers, pin_memory, cpu_threads)

Bases: object

Captures the fully resolved set of hardware optimizations to apply to a single training run.

Notes

Every field is a concrete decision: the tri-state request flags and hardware capabilities have already been reconciled by resolve_optimization_profile, so consumers apply these values directly without any further capability checks. The profile is device-aware and holds cores back for other work rather than saturating the machine, favoring smoother operation.

property amp_device_type: str

Returns the torch.autocast device-type string for this run’s base device.

amp_dtype: dtype | None

The autocast compute dtype for mixed precision, or None to train in full float32 precision.

cpu_threads: int | None

The intra-op thread count to restore for CPU training, or None to leave the process default untouched.

cudnn_benchmark: bool

Determines whether the cuDNN convolution autotuner is enabled, which trades reproducibility for speed on fixed input sizes (CUDA only).

dataloader_workers: int

The number of worker processes each training process uses to load and augment data.

describe()

Builds a one-line human-readable summary of the active optimizations for logging.

Return type:

str

Returns:

A compact description of the device, parallelism, precision, and dataloader settings.

device: str

"cuda", "cpu", or "mps" (per-rank cuda:N is derived from gpus inside the runner).

Type:

The base torch device type training runs on

gpus: tuple[int, ...]

The CUDA device indices in use, empty for CPU or MPS runs.

multi_gpu_strategy: MultiGpuStrategy

The resolved multi-GPU execution strategy.

pin_memory: bool

Determines whether dataloaders pin host memory to speed up host-to-device transfers (meaningful for CUDA only).

tf32: bool

Determines whether TF32 acceleration is enabled for float32 matmuls and convolutions (CUDA only).

torch_compile: bool

Determines whether the model is wrapped with torch.compile before training.

property use_amp: bool

Returns whether mixed precision is enabled for this run.

property use_ddp: bool

Returns whether the run trains with DistributedDataParallel across multiple processes.

use_gradient_scaler: bool

Determines whether a gradient scaler is required, which is the case only for float16 mixed precision.

property world_size: int

Returns the number of training processes, which is the GPU count under DDP and one otherwise.

class sollertia_video_tracking.training.TrainingDatasetSummary(config, shuffle, net_type, detector_type, weight_init, from_shuffle)

Bases: object

Captures the outcome of creating a training-dataset shuffle for reporting to the caller.

config: Path

The path of the DeepLabCut project configuration file the shuffle was created for.

describe()

Builds a one-line human-readable summary of the created shuffle for the CLI.

Return type:

str

Returns:

A compact description of the model, weights, and split source of the created shuffle.

detector_type: str | None

The object detector the shuffle was created for, or None for bottom-up and conditional top-down models.

from_shuffle: int | None

The shuffle whose train/test split was reused, or None when a fresh split was drawn.

net_type: str | None

The pose-model architecture the shuffle was created for, or None when the project default was used.

shuffle: int

The shuffle index that was created.

weight_init: str

A description of the weight initialization used (imagenet or <super_animal> (transfer|fine-tune)).

class sollertia_video_tracking.training.TrainingSummary(config, shuffle, model_folder, tasks_trained, device, strategy, world_size, precision, epochs, evaluation=None)

Bases: object

Captures the outcome of a completed training run for reporting to the caller.

Notes

The summary is constructed once, after training returns, from the resolved run configuration and optimization profile. It reports what was trained and how, not per-epoch metrics, which are streamed to the monitor and written to the model directory’s per-model learning-statistics CSV (learning_stats.csv for the pose model, learning_stats_detector.csv for the detector).

config: Path

The path of the DeepLabCut project configuration file training ran for.

describe()

Builds a human-readable summary of the training run, and the evaluation when one ran, for the CLI.

Return type:

str

Returns:

A compact description of what was trained and the hardware configuration used, with the evaluation summary appended on a second line when a post-training evaluation ran.

device: str

The base device type training ran on ("cuda", "cpu", or "mps").

epochs: int

The number of epochs the pose model was trained for.

evaluation: EvaluationSummary | None

The post-training evaluation summary, or None when evaluation was skipped or failed.

model_folder: Path

The directory containing the trained snapshots and training statistics.

precision: str

The compute precision used ("bfloat16", "float16", or "fp32").

shuffle: int

The shuffle index that was trained.

strategy: str

The multi-GPU execution strategy used ("ddp", "dp", or "single").

tasks_trained: tuple[str, ...]

The models that were trained, in order (e.g. ("detector", "pose") for a top-down model).

world_size: int

The number of training processes used.

class sollertia_video_tracking.training.WeightInitializationMethod(*values)

Bases: StrEnum

Defines how a shuffle’s model weights are initialized before training.

imagenet transfers from a stock ImageNet backbone and needs no external model, while transfer and fine-tune both start from a SuperAnimal model, so they require a SuperAnimal dataset and a pose-model architecture to be named. Fine-tuning additionally loads the SuperAnimal decoder head and is the only method that can enable memory replay.

FINE_TUNE = 'fine-tune'

Initializes from a SuperAnimal model including its decoder head (fine-tuning), the only memory-replay method.

IMAGENET = 'imagenet'

Initializes the backbone from ImageNet transfer learning, the default that needs no SuperAnimal model.

TRANSFER = 'transfer'

Initializes from a SuperAnimal model with a fresh decoder head (transfer learning).

sollertia_video_tracking.training.build_conditional_top_down_conditions(conditions_path)

Converts a conditioning-file path into the format create_training_dataset expects for CTD models.

Parameters:

conditions_path (str | Path) – The path to a predictions file (.h5/.json) or a conditioning snapshot (.pt).

Return type:

Path | tuple[int, str]

Returns:

The predictions path unchanged, or a (shuffle, snapshot_name) pair parsed from the snapshot path.

Raises:

ValueError – When the file type is unsupported or a shuffle index cannot be parsed from a snapshot path.

sollertia_video_tracking.training.build_superanimal_weight_init(config, super_animal, network_type, detector_type, *, fine_tune=False, memory_replay=False, customized_pose_checkpoint=None, customized_detector_checkpoint=None)

Builds a SuperAnimal weight initialization for transfer learning or fine-tuning, mirroring the GUI selector.

Parameters:

• config (str | Path) – The path of the DeepLabCut project configuration file.

• super_animal (str) – The SuperAnimal dataset to initialize from, one of get_available_super_animals.

• network_type (str) – The project’s pose-model architecture; a top_down_ prefix is stripped to name the SuperAnimal pose model.

• detector_type (str | None) – The project’s detector architecture for top-down models, or None.

• fine_tune (bool, default: False) – Determines whether to fine-tune, loading the SuperAnimal decoder head (requires a conversion table), rather than transfer learning with a fresh head.

• memory_replay (bool, default: False) – Determines whether to enable memory replay, which is only valid when fine-tuning.

• customized_pose_checkpoint (str | Path | None, default: None) – A custom SuperAnimal pose checkpoint to use instead of the downloaded one.

• customized_detector_checkpoint (str | Path | None, default: None) – A custom SuperAnimal detector checkpoint to use instead of the downloaded one.

Return type:

WeightInitialization

Returns:

The weight initialization to pass to create_training_dataset.

sollertia_video_tracking.training.create_training_dataset(config, *, shuffle=1, network_type=None, detector_type=None, weight_initialization=None, conditional_top_down_conditions=None, from_shuffle=None, from_training_set_index=0, overwrite=False)

Creates a training-dataset shuffle for a project at parity with the DeepLabCut GUI’s create-training-dataset tab.

Wraps DeepLabCut’s training-dataset creation for the PyTorch engine, which bakes the model architecture, weight initialization, and train/test split into the shuffle (training is run afterward with slvt train). Multi-animal projects are handled automatically. When from_shuffle is given, the new shuffle reuses that shuffle’s train/test split instead of drawing a fresh one.

Parameters:

• config (str | Path) – The path of the DeepLabCut project configuration file.

• shuffle (int, default: 1) – The shuffle index to create.

• network_type (str | None, default: None) – The pose-model architecture, one of get_available_pose_models. None uses the project default. A top_down_ prefix creates a top-down model that also trains a detector.

• detector_type (str | None, default: None) – The object detector for top-down models, one of get_available_object_detectors.

• weight_initialization (WeightInitialization | None, default: None) – A SuperAnimal weight initialization from build_superanimal_weight_init, or None for ImageNet transfer learning.

• conditional_top_down_conditions (Path | tuple[int, str] | None, default: None) – The conditioning source for conditional top-down models, from build_conditional_top_down_conditions, or None.

• from_shuffle (int | None, default: None) – The existing shuffle whose train/test split to reuse, or None to draw a fresh split.

• from_training_set_index (int, default: 0) – The training-set fraction index of from_shuffle when reusing a split.

• overwrite (bool, default: False) – Determines whether to overwrite the shuffle if the index already exists.

Return type:

TrainingDatasetSummary

Returns:

A summary of the created shuffle.

Raises:

• ValueError – When network_type or detector_type is not in the installed engine’s catalog.

• ValueError – When DeepLabCut creates no shuffle (for example, no labeled frames exist, or every labeled frame was annotated by a scorer other than the one named in the project configuration).

sollertia_video_tracking.training.detect_fixed_input_size(config, *, shuffle=1, training_set_index=0)

Determines whether a shuffle’s training transform feeds the network a single fixed input resolution.

The cuDNN autotuner only pays off, and only preserves deterministic training safely, when the convolution input shapes stay constant across steps, so this reports whether that precondition holds instead of asking the operator to assert it. A run is fixed-size when its pose data pipeline crops or resizes every image to one resolution and no variable-size detector is trained alongside it. Any inability to read the shuffle’s configuration is treated conservatively as not fixed, since a wrong assertion of fixed size makes the autotuner harmful.

Parameters:

• config (str | Path) – The path of the DeepLabCut project configuration file.

• shuffle (int, default: 1) – The shuffle index that will be trained.

• training_set_index (int, default: 0) – The training-set fraction index the shuffle was created with.

Return type:

bool

Returns:

True when the training transform’s spatial input size is constant across the whole run, False otherwise.

sollertia_video_tracking.training.get_available_object_detectors()

Returns the sorted catalog of object detectors the PyTorch engine supports for top-down models.

Return type:

tuple[str, ...]

Returns:

Every detector_type value the installed DeepLabCut PyTorch engine can train, sorted alphabetically.

sollertia_video_tracking.training.get_available_pose_models()

Returns the sorted catalog of pose-model architectures the PyTorch engine supports.

Return type:

tuple[str, ...]

Returns:

Every net_type value the installed DeepLabCut PyTorch engine can train, sorted alphabetically.

sollertia_video_tracking.training.get_available_super_animals()

Returns the SuperAnimal datasets available for transfer learning and fine-tuning.

Return type:

tuple[str, ...]

Returns:

Every SuperAnimal dataset name that can initialize a project’s weights, matching the GUI’s options.

sollertia_video_tracking.training.resolve_optimization_profile(*, device=None, gpus=None, multi_gpu=MultiGpuStrategy.AUTO, amp=AmpMode.AUTO, tf32=Toggle.AUTO, cudnn_benchmark=Toggle.AUTO, torch_compile=Toggle.AUTO, dataloader_workers=-1, pin_memory=Toggle.AUTO, fixed_input_size=False)

Reconciles the requested optimization flags with the available hardware into a concrete profile.

Every optimization is exposed as an explicit request so an operator who knows their silicon can override the automatic defaults. "auto" selects a capability-detected default suited to the chosen device. An explicit request is honored where it applies to the chosen device. A forced AMP dtype the device cannot support (bfloat16 on MPS, float16 off CUDA) is disabled with a warning, while the CUDA-only toggles (tf32, cudnn_benchmark, pin_memory) are forced off on non-CUDA devices. Mixed precision is additionally disabled with a warning when the DataParallel ("dp") strategy is selected, because autocast does not reach DataParallel’s per-GPU replica threads; use DDP to combine mixed precision with multi-GPU training.

Parameters:

• device (DeviceType | None, default: None) – The requested base device ("auto", "cpu", "mps", or "cuda"), or None to select automatically.

• gpus (tuple[int, ...] | None, default: None) – The explicitly requested CUDA device indices, or None to train on GPU 0. List two or more indices to train across multiple GPUs.

• multi_gpu (MultiGpuStrategy, default: <MultiGpuStrategy.AUTO: 'auto'>) – The requested multi-GPU strategy applied when two or more GPUs are selected. Resolves to a single device when fewer than two GPUs are used.

• amp (AmpMode, default: <AmpMode.AUTO: 'auto'>) – The requested mixed-precision mode.

• tf32 (Toggle, default: <Toggle.AUTO: 'auto'>) – The requested TF32 setting (CUDA only; a no-op on other devices).

• cudnn_benchmark (Toggle, default: <Toggle.AUTO: 'auto'>) – The requested cuDNN autotuner setting; its "auto" default follows fixed_input_size, since the autotuner only pays off, and only stays deterministic-safe, when the input spatial size is fixed.

• torch_compile (Toggle, default: <Toggle.AUTO: 'auto'>) – The requested torch.compile setting; disabled by default because of its warm-up cost.

• dataloader_workers (int, default: -1) – The number of dataloader workers per process, or -1 to choose automatically.

• pin_memory (Toggle, default: <Toggle.AUTO: 'auto'>) – The requested host-memory pinning setting (meaningful for CUDA only).

• fixed_input_size (bool, default: False) – Determines whether the training transform produces one fixed input resolution, normally supplied by detect_fixed_input_size rather than the operator. The cuDNN autotuner’s "auto" default enables it only when this holds, since a varying input size makes the autotuner harmful and non-deterministic.

Return type:

OptimizationProfile

Returns:

The resolved OptimizationProfile describing exactly what to apply to the run.

sollertia_video_tracking.training.train_model(config, profile, *, shuffle=1, training_set_index=0, epochs=None, batch_size=None, save_epochs=None, display_iterations=None, snapshot_path=None, detector_path=None, detector_batch_size=None, detector_epochs=None, detector_save_epochs=None, maximum_snapshots_to_keep=None, load_head_weights=True, evaluate=True, evaluation_batch_size=1, evaluation_confidence_cutoff=None, display_progress=True)

Trains a DeepLabCut shuffle with the resolved hardware optimizations and a clean progress monitor.

The training-dataset options (model architecture, split) are fixed when the shuffle is created; this function fits the already-created shuffle. It applies the requested overrides and the optimization profile to the configuration once, then launches training as either a DistributedDataParallel process group or a single process. The single-process path covers one GPU, the CPU, MPS, and DataParallel across multiple GPUs. For top-down shuffles the detector is trained before the pose model.

Parameters:

• config (str | Path) – The path of the DeepLabCut project configuration file.

• profile (OptimizationProfile) – The resolved optimization profile describing the device, precision, and parallelism to use.

• shuffle (int, default: 1) – The shuffle index to train.

• training_set_index (int, default: 0) – The training-set fraction index.

• epochs (int | None, default: None) – The maximum number of pose-model epochs, or None to use the configured value.

• batch_size (int | None, default: None) – The pose-model batch size, or None to use the configured value.

• save_epochs (int | None, default: None) – The number of epochs between pose-model snapshots, or None to use the configured value.

• display_iterations (int | None, default: None) – The number of iterations between intra-epoch loss logs, or None for the configured value.

• snapshot_path (str | Path | None, default: None) – The pose snapshot to resume from, if any.

• detector_path (str | Path | None, default: None) – The detector snapshot to resume from, if any.

• detector_batch_size (int | None, default: None) – The detector batch size (top-down only), or None to use the configured value.

• detector_epochs (int | None, default: None) – The maximum number of detector epochs (top-down only); zero skips detector training.

• detector_save_epochs (int | None, default: None) – The epochs between detector snapshots (top-down only), or None for the configured value.

• maximum_snapshots_to_keep (int | None, default: None) – The maximum number of snapshots to retain, or None to use the configured value.

• load_head_weights (bool, default: True) – Determines whether to load head weights when resuming a pose model from a snapshot.

• evaluate (bool, default: True) – Determines whether to score the trained snapshot against the labeled frames as a final step and write the evaluation feather and provenance sidecar.

• evaluation_batch_size (int, default: 1) – The number of frames scored per forward pass during the post-training evaluation.

• evaluation_confidence_cutoff (float | None, default: None) – The confidence cutoff for the evaluation’s cutoff-filtered metrics, or None to fall back to the project configuration’s pcutoff (0.6 when unset).

• display_progress (bool, default: True) – Determines whether to render the live progress monitor and route DeepLabCut’s logs off the console.

Return type:

TrainingSummary

Returns:

A summary of what was trained and the hardware configuration used.

Raises:

ValueError – When the shuffle requests SuperAnimal memory-replay fine-tuning, which this trainer does not support.

Video Inference

Provides optimized multi-device DeepLabCut video inference over one or many videos.

class sollertia_video_tracking.inference.InferenceProfile(device, gpus, gpu_processes, chunks, cpu_workers, cpu_threads_per_worker, amp_dtype, tf32, cudnn_benchmark, channels_last, torch_compile)

Bases: object

Captures the fully resolved set of hardware optimizations to apply to a video-inference run.

Notes

Every field is a concrete decision: the tri-state request flags and hardware capabilities have already been reconciled by resolve_inference_profile, so consumers apply these values directly without any further capability checks. Parallelism is expressed as independent worker processes that each pull whole videos from a shared queue: gpu_processes per CUDA device (raising it oversubscribes a device to fill the GPU’s idle gaps between videos) or cpu_workers core-block-pinned processes on CPU. The profile holds cores back rather than saturating the machine.

property amp_device_type: str

Returns the device type string to pass to torch.autocast for this run.

amp_dtype: dtype | None

The autocast compute dtype for mixed precision, or None to run in full float32 precision.

channels_last: bool

Determines whether the model and its inputs use the channels-last memory format, which accelerates convolutions on tensor-core GPUs and oneDNN CPU backends.

chunks: int

The number of contiguous frame-range pieces each concurrently analyzed video is split into, every piece run in parallel by its own worker. One analyzes each video as a single unbroken range, matching the whole-video path. A value above one multiplies per-device concurrency, so the run spawns gpu_processes * chunks workers per CUDA device.

cpu_threads_per_worker: int | None

The intra-op thread count each CPU worker restores, or None when not running on CPU.

cpu_workers: int

The number of CPU inference worker processes, or 0 when not running on CPU.

cudnn_benchmark: bool

Determines whether the cuDNN convolution autotuner is enabled, which trades a warm-up for speed on fixed input sizes (CUDA only).

describe()

Builds a one-line human-readable summary of the active optimizations for logging.

Return type:

str

Returns:

A compact description of the device, parallelism, precision, and enabled accelerators.

device: str

"cuda", "cpu", or "mps" (per-worker cuda:N is derived from gpus inside the pipeline).

Type:

The base torch device type inference runs on

gpu_processes: int

The number of inference worker processes to run per CUDA device, or 0 when not running on CUDA.

gpus: tuple[int, ...]

The CUDA device indices in use, empty for CPU or MPS runs.

property on_cuda: bool

Returns whether inference runs on CUDA devices.

tf32: bool

Determines whether TF32 acceleration is enabled for float32 matmuls and convolutions (CUDA only).

torch_compile: bool

Determines whether the model is wrapped with torch.compile before inference.

property total_workers: int

Returns the configured maximum number of worker processes; the run spawns fewer when videos are scarce.

property use_amp: bool

Returns whether mixed precision is enabled for this run.

class sollertia_video_tracking.inference.InferenceSummary(config, video_count, destinations, device, workers, precision, outputs, failures)

Bases: object

Captures the outcome of a completed multi-video inference run for reporting to the caller.

Notes

The summary is built after every worker has finished. outputs holds the produced DeepLabCut .h5 prediction files in video order, and failures pairs each failed video with its error message so a partial run is reported honestly rather than silently.

config: Path

The path of the DeepLabCut project configuration file inference ran for.

describe()

Builds a one-line human-readable summary of the inference run for the CLI.

Return type:

str

Returns:

A compact description of how many videos were processed, on what hardware, and where results were written.

destinations: tuple[Path, ...] | None

The distinct directories the prediction files were written to, or None when each video’s predictions were written beside the video itself.

device: str

The base device type inference ran on ("cuda", "cpu", or "mps").

failures: tuple[tuple[str, str], ...]

The videos that failed, each paired with its error message.

outputs: tuple[Path, ...]

The produced DeepLabCut .h5 prediction files, one per successfully processed video.

precision: str

The compute precision used ("bfloat16", "float16", or "fp32").

video_count: int

The number of videos submitted for inference.

workers: int

The number of worker processes used.

sollertia_video_tracking.inference.detect_fixed_input_size(config, videos, crop_override=None)

Determines whether every video would feed the pose network a single fixed input resolution.

The cuDNN autotuner only pays off when the convolution input shapes stay constant across the run, so this reports whether that precondition holds instead of asking the operator to assert it. When per-video crop rectangles are provided (the --crop override), each video is reduced to its rectangle, so the run is fixed-size exactly when all rectangles share one size. Otherwise, when the project is configured to crop, every analyzed video is reduced to its configured crop rectangle, so the run is fixed-size exactly when all videos share one crop size. When the project is not configured to crop, the network sees each video’s native resolution, so the run is fixed-size exactly when all videos share one resolution. A single video is therefore always fixed-size. Any inability to read the configuration or a video’s dimensions is treated conservatively as not fixed, since a wrong assertion of fixed size makes the autotuner harmful.

Parameters:

• config (str | Path) – The path of the DeepLabCut project configuration file.

• videos (list[str | Path]) – The video files the run will analyze.

• crop_override (Sequence[tuple[int, int, int, int]] | None, default: None) – The per-video crop rectangles that override the project configuration, parallel to videos, or None to derive each video’s input size from the project configuration.

Return type:

bool

Returns:

True when the network’s spatial input size is provably constant across the whole run, False otherwise.

sollertia_video_tracking.inference.resolve_inference_profile(*, device=None, gpus=None, amp=AmpMode.AUTO, tf32=Toggle.AUTO, cudnn_benchmark=Toggle.AUTO, channels_last=Toggle.AUTO, torch_compile=Toggle.AUTO, gpu_processes=-1, chunks=1, cpu_workers=-1, cpu_threads_per_worker=-1, fixed_input_size=False)

Reconciles the requested inference optimization flags with the available hardware into a concrete profile.

Every optimization is exposed as an explicit request so an operator who knows their silicon can override the automatic defaults. "auto" selects a capability-detected default suited to the chosen device. An explicit "on"/"off" toggle is honored wherever it applies, though the CUDA-only tf32 and cuDNN-benchmark toggles are forced off on non-CUDA devices. A forced AMP dtype is honored wherever the device can run it and is otherwise disabled with a warning rather than a silent refusal (bfloat16 on MPS and float16 on any non-CUDA device fall back to float32). The device selection cascades cuda -> cpu when no CUDA device is visible so the same call works unchanged on a GPU server or a CPU-only server.

Parameters:

• device (DeviceType | None, default: None) – The requested base device ("auto", "cpu", "mps", or "cuda"), or None to select automatically.

• gpus (tuple[int, ...] | None, default: None) – The explicitly requested CUDA device indices, or None to use every visible device.

• amp (AmpMode, default: <AmpMode.AUTO: 'auto'>) – The requested mixed-precision mode; "auto" enables bfloat16 only where it is natively fast.

• tf32 (Toggle, default: <Toggle.AUTO: 'auto'>) – The requested TF32 setting (CUDA only; a no-op on other devices).

• cudnn_benchmark (Toggle, default: <Toggle.AUTO: 'auto'>) – The requested cuDNN autotuner setting; its "auto" default follows fixed_input_size, since the autotuner only pays off when the input spatial size is fixed.

• channels_last (Toggle, default: <Toggle.AUTO: 'auto'>) – The requested channels-last memory-format setting.

• torch_compile (Toggle, default: <Toggle.AUTO: 'auto'>) – The requested torch.compile setting; disabled by default because of its warm-up cost, which may not amortize over short videos.

• gpu_processes (int, default: -1) – The number of worker processes per CUDA device, or -1 to use the default of one video per GPU.

• chunks (int, default: 1) – The number of contiguous frame-range pieces to split each concurrently analyzed video into, each run in parallel by its own worker. One disables intra-video chunking, and values above one raise per-video concurrency to gpu_processes * chunks workers per CUDA device.

• cpu_workers (int, default: -1) – The number of CPU worker processes, or -1 to choose automatically from the physical core count.

• cpu_threads_per_worker (int, default: -1) – The intra-op thread count per CPU worker, or -1 to choose automatically.

• fixed_input_size (bool, default: False) – Determines whether every video feeds the network one fixed input resolution, normally supplied by detect_fixed_input_size rather than the operator. The cuDNN autotuner’s "auto" default enables it only when this holds, since a varying input size makes the autotuner harmful rather than beneficial.

Return type:

InferenceProfile

Returns:

The resolved InferenceProfile describing exactly what to apply to the run.

sollertia_video_tracking.inference.resolve_project_videos(config)

Returns the existing video files registered in the project configuration’s video_sets.

Reads the paths the DeepLabCut project configuration registers and keeps the ones that still exist on disk, so inference can analyze the whole project without re-listing every video on the command line.

Parameters:

config (str | Path) – The path of the DeepLabCut project configuration file.

Return type:

list[Path]

Returns:

The registered video paths that currently exist on disk, in the order the configuration lists them.

sollertia_video_tracking.inference.run_inference(config, videos, profile, *, destination_override=None, shuffle=1, snapshot_index=None, detector_snapshot_index=None, batch_size=None, detector_batch_size=None, crop_override=None, display_progress=True)

Runs DeepLabCut inference over many videos, distributing whole videos across GPU or CPU worker slots.

Each worker pulls whole videos from a shared queue and analyzes them with DeepLabCut, so the work is balanced across slots without splitting any video. On CUDA a slot is a device (gpu_processes of them per device); on CPU a slot is a disjoint, thread-bounded block of physical cores. Every worker’s forward pass is wrapped with the profile’s mixed precision and channels-last format, and each video’s predictions are written as DeepLabCut’s native .h5 prediction file, beside the video or into its chosen output directory.

Parameters:

• config (str | Path) – The path of the DeepLabCut project configuration file.

• videos (list[str | Path]) – The video files to analyze.

• profile (InferenceProfile) – The resolved optimization profile describing the device, precision, and parallelism to use.

• destination_override (Sequence[str | Path] | None, default: None) – The per-video output directories prediction files are written to, parallel to videos, or None to write each video’s predictions beside the video itself, matching DeepLabCut’s own default and the location the outlier-extraction step reads. Pass one directory per video, so a single chosen directory can collect every video’s predictions or each video’s predictions can be bundled with its own directory.

• shuffle (int, default: 1) – The shuffle index whose trained model is used.

• snapshot_index (int | None, default: None) – The pose snapshot index to use, or None for the configured default.

• detector_snapshot_index (int | None, default: None) – The detector snapshot index to use, or None for the configured default.

• batch_size (int | None, default: None) – The pose-model inference batch size, or None to use the configured value.

• detector_batch_size (int | None, default: None) – The detector inference batch size, or None to use the configured value.

• crop_override (Sequence[tuple[int, int, int, int]] | None, default: None) – The per-video crop rectangles to analyze, parallel to videos, each an (x1, x2, y1, y2) tuple, or None to resolve each video’s crop from the project configuration. Overrides the project’s cropping so de-novo videos that are not registered in the configuration can be analyzed at a caller-chosen crop.

• display_progress (bool, default: True) – Determines whether to render the live aggregate progress bar.

Return type:

InferenceSummary

Returns:

A summary of what was analyzed and the hardware configuration used.

Raises:

ValueError – Raised when no videos are provided, or when crop_override or destination_override is provided but its length does not match the number of videos. Raised when the profile selects CUDA but resolves no GPU indices to build worker slots from. Raised when an explicit CPU worker/thread configuration cannot be pinned to disjoint core blocks. Raised when profile.chunks exceeds one and the project is multi-animal or the model is not bottom-up.

Command-Line Interface

slvt

Designs and deploys DeepLabCut video-tracking pipelines for the Sollertia platform.

Usage

slvt [OPTIONS] COMMAND [ARGS]...

extract

Selects or clears frames in a project’s videos to bootstrap a model, refine a trained one, or start over.

--config-path names the DeepLabCut project’s config.yaml every subcommand operates on. Use frames to bootstrap a project’s training frames by clustering raw video, outliers to extract a trained model’s likely-wrong frames for refinement, or purge to delete a video’s entire labeled-data directory for a clean start. Use pending to list the directories whose machine-labeled frames still await your refinement. The parallelism and clustering options apply to frames and outliers, while --videos, --overwrite, and --reset are shared by the subcommands that accept them. All of these shared options must be given before the subcommand name.

Usage

slvt extract [OPTIONS] COMMAND [ARGS]...

Options

-cfg, --config-path <config_path>

The path to the DeepLabCut project’s config.yaml for which to extract new or outlier frames.

-w, --workers <workers>

How many videos to process at the same time. Set to -1 to launch about one worker per four usable cores, capped at the number of videos.

Default:

-1

-c, --cores <cores>

How many CPU cores to devote to each video being processed. Set to -1 to give each worker a saturating block (about four cores) when –workers is automatic, or split the usable cores evenly across an explicit –workers count. Each video keeps about four cores busy while it decodes, so larger values mostly idle cores and smaller values slow each video worker down.

Default:

-1

-fpv, --frames-per-video <frames_per_video>

How many frames to keep from each processed video. Set to -1 to use the project’s configured amount.

Default:

-1

-cs, --clustering-stride <N>

How far apart, in frames, to sample before clustering. The default of 1 uses every frame. For ‘frames’ this strides the whole video; for ‘outliers’ it strides the flagged candidate frames. Raise it to cluster fewer frames trading coverage for processing speed.

Default:

1

-crw, --clustering-resize-width <clustering_resize_width>

The width, in pixels, that frames are shrunk to before they are compared for similarity. Smaller values are faster but compare more coarsely.

Default:

30

-co, --color, --grayscale

Determines whether frames are compared in color instead of grayscale when selecting them.

Default:

False

-p, --progress, --no-progress

Determines whether the aggregate progress bar is shown during extraction.

Default:

True

-v, --videos <PATH>

A project video, registered in config.yaml, that the subcommand targets. For ‘frames’ it is included in the sample first, or is one of the only videos processed with –exclusive. For ‘outliers’ it is a video to refine. For ‘purge’ its labeled-data directory is removed, and for ‘pending’ its directory is inspected. Provide the option several times for several videos. Omit –videos to target the whole project: every video for ‘frames’, ‘purge’, and ‘pending’, and every video the current iteration model has analyzed for ‘outliers’.

-o, --overwrite

Re-roll the videos this run processes instead of topping them up, clearing each one’s removable frames first. For ‘frames’ it clears each selected video’s unlabeled bootstrap frames, refused for any already in outlier refinement. For ‘outliers’ it clears the current refinement iteration’s outlier frames for the videos being refined. Mutually exclusive with –reset.

-r, --reset

Re-roll across the whole project instead of topping up, clearing the relevant frames first. For ‘frames’ this clears every not-yet-refined video’s unlabeled bootstrap frames and leaves videos already in outlier refinement untouched. For ‘outliers’ it clears the current iteration’s outlier frames for every video. Mutually exclusive with –overwrite.

frames

Selects initial training frames from a subset of the project’s videos by clustering them in parallel.

Each video is clustered in its own worker process pinned to a disjoint block of CPU cores. --frames-per-video is a per-video ceiling: each selected video is topped up to it, so a not-yet-extracted video gains a full set and a partly-extracted one gains only the frames that reach the ceiling. Passing --total-frames selects just enough videos to reach that project-wide total, preferring not-yet-extracted videos and falling back to below-ceiling ones. Any videos named with --videos are included first. If even topping every eligible video to the ceiling cannot reach the total in one pass, the run reports the shortfall and stops, so raise --frames-per-video, lower --total-frames, or register more videos. Passing --exclusive with --videos instead restricts the run to exactly those files, topping each up to --frames-per-video and ignoring the budget and group balancing. Passing --overwrite clears the selected videos’ unlabeled frames first so they are re-rolled from scratch (refused for any already in outlier refinement), and --reset does the same across every not-yet-refined project video; both keep already-labeled and outlier frames.

Usage

slvt extract frames [OPTIONS]

Options

-tf, --total-frames <total_frames>

The total number of extracted frames you want across the whole project. When set, videos are selected until this many frames are reached, preferring not-yet-extracted videos and falling back to below-ceiling ones; each is topped up to –frames-per-video frames (a fresh video by a full set, a partly-extracted one by its remainder). Set to -1 to top up every below-ceiling selected video instead.

Default:

200

-bg, --balance-groups

Spread the sampled frames evenly across groups of related videos instead of drawing purely at random, so every group is represented. Groups are inferred from the parts of the file names the videos share. Only applies together with –total-frames.

-gr, --group-regex <REGEX>

A regular expression used to derive group names for each video, for naming schemes the automatic grouping does not recognize (for example ‘(Grpd+)’ for names like D1_Grp2). Implies –balance-groups.

-e, --exclusive

Restrict the run to exactly the –videos files, topping each up to –frames-per-video frames and ignoring the –total-frames budget and group balancing. A video already at that count is skipped unless –overwrite re-rolls it. Requires –videos.

outliers

Extracts a trained model’s likely-wrong frames from the project’s analyzed videos to refine the model.

Refines on the project videos given with --videos, extracting --frames-per-video outlier frames from each. Omit --videos to refine every registered video the current model has already analyzed. Each named video must be registered in the project’s config.yaml and analyzed, since the detectors read the model’s predictions rather than re-running the model. Requested paths that are not registered project videos are skipped with a warning. The flagged outlier frames are clustered in parallel, one video per worker pinned to a disjoint block of CPU cores, and added to each video’s labeled-data directory alongside the model’s predictions as machine pre-labels. Outlier extraction is additive by default, so repeated passes grow the refinement set. Pass --overwrite to replace the refined videos’ outlier frames for the current iteration, or --reset to clear the whole iteration’s outlier frames before re-extracting. Both preserve every already-labeled training frame.

Usage

slvt extract outliers [OPTIONS]

Options

-oa, --outlier-algorithm <outlier_algorithm>

How likely-wrong frames are identified. ‘jump’ flags large frame-to-frame jumps (motion), ‘uncertain’ flags low-confidence frames, ‘fitting’ flags departures from a fitted motion trajectory, and ‘list’ takes an explicit frame list.

Default:

'uncertain'

Options:

jump | uncertain | fitting | list

-ea, --extraction-algorithm <extraction_algorithm>

How the frames to keep are chosen from the identified candidates.

Default:

'kmeans'

Options:

kmeans | uniform

-s, --shuffle <shuffle>

The shuffle index whose trained model produced the predictions. It is the only model-selection option; everything else is taken from the project configuration.

Default:

1

-pdt, --pixel-distance-threshold <pixel_distance_threshold>

How far, in pixels, a bodypart may move (for ‘jump’) or depart from its fitted trajectory (for ‘fitting’) before its frame is flagged.

Default:

20.0

-mc, --minimum-confidence <minimum_confidence>

The confidence below which a prediction is treated as unreliable: ‘uncertain’ flags a frame holding one, and ‘fitting’ excludes it from driving the fitted trajectory.

Default:

0.01

-cb, --comparison-bodyparts <BODYPART>

A bodypart the detectors consider. Provide the option several times to restrict to several; omit to consider every bodypart.

-fi, --frame-index <FRAME>

An explicit frame index to extract when –outlier-algorithm list is used. Provide the option several times to extract multiple frames.

-ad, --autoregressive-degree <autoregressive_degree>

How many past frames the ‘fitting’ detector’s motion model uses to predict the next position.

Default:

3

-mad, --moving-average-degree <moving_average_degree>

How many past prediction errors the ‘fitting’ detector’s motion model smooths over.

Default:

1

-sl, --save-labeled

Determines whether to also save a copy of each extracted frame with the model’s predictions drawn on it.

-tm, --tracking-method <tracking_method>

The multi-animal tracker that produced the data. Omit to use the project’s setting.

Options:

box | skeleton | ellipse

-fw, --fit-workers <fit_workers>

How many SARIMAX keypoint-trajectory fits to run in parallel during ‘fitting’ detection, the most expensive step. Set to -1 to use every usable core, which leaves a small reserve free.

Default:

-1

pending

Lists the video directories that still hold machine-labeled frames you have not refined for the current iteration.

After outliers extracts a trained model’s likely-wrong frames, each is saved as a machine pre-label that you refine in the labeling GUI. This reports each video directory that still has unrefined machine frames, and how many, so you know which directories to open next. It only reads the project, changing nothing. Target specific videos with --videos, or omit --videos to scan the whole project.

Usage

slvt extract pending [OPTIONS]

purge

Deletes targeted videos’ entire labeled-data directories, including their labels, after a dry-run preview.

This is the wholesale reset the frame and outlier re-extraction options deliberately avoid: where --overwrite and --reset clear only unlabeled or single-iteration frames and always keep the human labels, purge removes each targeted labeled-data directory outright. It exists for the rare start-completely-over case, such as changing the project crop, that the label-preserving options cannot serve. Target specific videos with --videos, or omit --videos to purge the whole project. The command previews what it would delete and removes nothing until --yes is given.

Usage

slvt extract purge [OPTIONS]

Options

-y, --yes

Actually delete the directories. Without this flag the command only previews what it would remove, deleting nothing.

gui

Launches the standard DeepLabCut GUI, used to create a project and manually label its frames.

This opens the same fully functional DeepLabCut application reached elsewhere by running python -m deeplabcut. From its project-management window, a project’s config.yaml is created or opened to label the frames that slvt extract selects, correct the outlier frames it flags, and merge the refined data into the next iteration. The frame labeler itself opens in napari, reached from that window. Manual labeling is the only part of the refinement loop this library does not implement, so it is the reason to open the GUI: the extraction, training, evaluation, and analysis tabs it also offers run stock DeepLabCut and are slower than the equivalent slvt commands. The GUI needs a graphical session, so it is run on a workstation rather than a headless training or inference server, and the command blocks until the window is closed.

Usage

slvt gui [OPTIONS]

infer

Analyzes videos with a trained DeepLabCut model, distributing whole videos across GPU or CPU worker slots.

--config-path names the DeepLabCut project’s config.yaml whose trained model runs. Provide the videos to analyze with --videos (given several times for several files), or omit --videos to analyze every existing video registered in the project configuration. Each worker pulls work from a shared queue, so the work is balanced across slots. By default a worker analyzes a whole video, and --chunks instead splits each running video into that many contiguous frame ranges analyzed concurrently, with the parent stitching each video’s ranges back into one prediction file. Each forward pass runs with the mixed precision and memory format chosen for the detected hardware, except conditional-top-down models, which run at stock precision. Each video’s predictions are written as DeepLabCut’s native .h5 prediction file, beside the video or into an --output directory (one shared directory, or one per video). Pass --crop to analyze a chosen region rather than the project’s configured crop, which lets de-novo videos that are not registered in the project be analyzed. The same command runs on multiple GPUs, one GPU, or a CPU-only machine.

Usage

slvt infer [OPTIONS]

Options

-cfg, --config-path <config_path>

Required The path to the DeepLabCut project’s config.yaml whose trained model analyzes the videos.

-v, --videos <PATH>

A video file to analyze. Provide the option several times for several videos. These need not be registered in the project, so de-novo videos can be analyzed, optionally at a chosen region with –crop. Omit –videos to analyze every video registered in the project’s config.yaml.

-o, --output <DIRECTORY>

A directory the prediction files are written to. Pass once to collect every video’s predictions in one directory, or once per –videos file (in order, matching the video count) to bundle each video’s predictions with its own directory. Per-video outputs require explicit –videos. Omit to write each video’s predictions beside the video itself, where ‘slvt extract outliers’ reads them.

-s, --shuffle <shuffle>

The shuffle index whose trained model analyzes the videos.

Default:

1

-si, --snapshot-index <snapshot_index>

The index of the trained pose-model snapshot to run. Omit to use the project’s configured default.

-dsi, --detector-snapshot-index <detector_snapshot_index>

The index of the detector snapshot to run, for top-down models. Omit to use the project’s configured default.

-b, --batch-size <batch_size>

The number of frames the pose model processes per forward pass. Larger batches use more GPU memory and can speed up analysis. Omit to use the model’s default value.

-dbs, --detector-batch-size <detector_batch_size>

The number of frames the object detector processes per step, for top-down models. Omit to use the model’s default value.

-cr, --crop <X1,X2,Y1,Y2>

A crop rectangle ‘x1,x2,y1,y2’ to analyze instead of the project’s configured crop, so de-novo videos can be analyzed at a chosen region. Pass once to apply one rectangle to every video, or once per –videos file (in order, matching the video count) for per-video crops. Per-video crops require explicit –videos.

-d, --device <device>

The base device to run on. ‘auto’ uses every visible CUDA GPU when present and otherwise the CPU. ‘cuda’ targets every visible GPU but warns before falling back to the CPU when none is present. ‘cpu’ and ‘mps’ (Apple Metal) force those devices. Choose specific GPUs with –gpus.

Default:

'auto'

Options:

auto | cpu | mps | cuda

-g, --gpus <INDICES>

The comma-separated CUDA device indices to run on (e.g. ‘0,1’). Omit to use every visible GPU. Applies only when the device is a CUDA GPU.

-gp, --gpu-processes <gpu_processes>

The number of inference worker processes per GPU, each analyzing one video at a time. Raise it to oversubscribe a GPU and fill decode gaps. Most GPUs fully saturate with 1 or 2 workers. Set to -1 for the default of one process (one video) per GPU.

Default:

-1

-ch, --chunks <chunks>

The number of contiguous frame-range pieces each running video is split into, all analyzed concurrently. Raise it to run several frame ranges of one video at once, filling decode gaps within a single video, so total per-GPU concurrency becomes gpu-processes x chunks. Set to 1 to analyze each video as a single unbroken frame range.

Default:

1

-cw, --cpu-workers <cpu_workers>

The number of CPU inference worker processes, each pinned to a disjoint block of CPU cores. Set to -1 to choose automatically from the core count.

Default:

-1

-ctpw, --cpu-threads-per-worker <cpu_threads_per_worker>

The number of CPU threads (and cores) each CPU worker uses. Set to -1 to choose automatically.

Default:

-1

-a, --amp <amp>

The mixed-precision compute mode, which trades some numerical precision for speed and lower memory use. ‘auto’ enables bfloat16 on Ampere or newer GPUs and stays in float32 elsewhere. ‘off’ forces float32. ‘bf16’ forces bfloat16 (disabled with a warning on MPS) and ‘fp16’ forces float16 on CUDA (disabled with a warning elsewhere).

Default:

'auto'

Options:

auto | off | bf16 | fp16

-t, --tf32 <tf32>

TF32 acceleration for float32 matmuls and convolutions on CUDA, which speeds up float32 math at slightly reduced precision. ‘auto’ enables it on Ampere or newer GPUs. ‘on’ and ‘off’ force it. It is a no-op off CUDA.

Default:

'auto'

Options:

auto | on | off

-cb, --cudnn-benchmark <cudnn_benchmark>

The cuDNN convolution autotuner on CUDA. ‘auto’ enables it only when the run is detected to feed one fixed input size (a shared –crop rectangle, a shared project crop, or one shared video resolution), where it speeds up convolutions rather than re-tuning per size. ‘on’ and ‘off’ force it.

Default:

'auto'

Options:

auto | on | off

-cl, --channels-last <channels_last>

The channels-last memory format, which speeds up convolutions on tensor-core GPUs. ‘auto’ enables it on CUDA. ‘on’ and ‘off’ force it.

Default:

'auto'

Options:

auto | on | off

-cm, --compile-model <compile_model>

Determines whether the model is compiled with torch.compile for faster inference. ‘auto’ leaves it off because its one-time warm-up cost may not amortize. ‘on’ and ‘off’ force it.

Default:

'auto'

Options:

auto | on | off

-p, --progress, --no-progress

Determines whether the aggregate progress bar is shown during analysis.

Default:

True

prepare

Creates a training-dataset shuffle for a project, selecting the model, weights, and split.

--config-path names the DeepLabCut project’s config.yaml. The shuffle bakes in the model architecture, weight initialization, and a train/test split. Training is run afterward with slvt train. Multi-animal projects are handled automatically. This mirrors the DeepLabCut GUI’s create-training-dataset tab for headless and scripted use.

Usage

slvt prepare [OPTIONS]

Options

-cfg, --config-path <config_path>

Required The path to the DeepLabCut project’s config.yaml to create the training shuffle for.

-s, --shuffle <shuffle>

The shuffle index to create.

Default:

1

-n, --network <network>

The pose-model architecture. Omit to use the project default. A top-down architecture also trains an object detector. Required when a SuperAnimal –weight-initialization is selected.

Options:

animaltokenpose_base | cspnext_m | cspnext_s | cspnext_x | ctd_coam_w32 | ctd_coam_w48 | ctd_coam_w48_human | ctd_prenet_hrnet_w32 | ctd_prenet_hrnet_w48 | ctd_prenet_rtmpose_m | ctd_prenet_rtmpose_s | ctd_prenet_rtmpose_x | ctd_prenet_rtmpose_x_human | dekr_w18 | dekr_w32 | dekr_w48 | dlcrnet_stride16_ms5 | dlcrnet_stride32_ms5 | hrnet_w18 | hrnet_w32 | hrnet_w48 | resnet_101 | resnet_50 | rtmpose_m | rtmpose_s | rtmpose_x | top_down_cspnext_m | top_down_cspnext_s | top_down_cspnext_x | top_down_hrnet_w18 | top_down_hrnet_w32 | top_down_hrnet_w48 | top_down_resnet_101 | top_down_resnet_50

-d, --detector <detector>

The object detector for top-down models. Omit to use the default detector.

Options:

fasterrcnn_mobilenet_v3_large_fpn | fasterrcnn_resnet50_fpn_v2 | ssdlite

-wi, --weight-initialization <weight_initialization>

How the model’s starting weights are set. ‘imagenet’ uses ImageNet transfer learning and needs nothing further. ‘transfer’ and ‘fine-tune’ are for SuperAnimal models only: both require a –super-animal dataset and a –network, doing SuperAnimal transfer learning (a fresh head) or fine-tuning (the SuperAnimal head).

Default:

'imagenet'

Options:

imagenet | transfer | fine-tune

-sa, --super-animal <super_animal>

The SuperAnimal dataset to initialize from. Required when –weight-initialization is ‘transfer’ or ‘fine-tune’.

Options:

superanimal_bird | superanimal_quadruped | superanimal_topviewmouse

-mr, --memory-replay

Determines whether to enable SuperAnimal memory replay (only valid with –weight-initialization ‘fine-tune’). Note: ‘slvt train’ cannot train memory-replay shuffles.

-ctdc, --conditional-top-down-conditions <conditional_top_down_conditions>

The conditioning file for conditional top-down models: a predictions file (.h5 or .json) or a model snapshot (.pt). Required when –network is a conditional top-down (‘ctd_*’) architecture.

-fs, --from-shuffle <from_shuffle>

An existing shuffle whose train/test split to reuse instead of drawing a fresh one.

-o, --overwrite

Determines whether to overwrite the shuffle if its index already exists. WARNING: this replaces the existing shuffle’s training-dataset files.

train

Trains a DeepLabCut shuffle with hardware optimizations and a clean progress monitor.

--config-path names the DeepLabCut project’s config.yaml. The shuffle’s model architecture and train/test split are fixed when the shuffle is created (see slvt prepare) and this command fits that shuffle. Every optimization is exposed as a flag: automatic defaults are chosen for the detected hardware and never run slower than stock DeepLabCut, while explicit flags allow further tuning for known hardware. Training runs on a single GPU (index 0) by default, since multi-GPU training is often slower for DeepLabCut workloads. Select two or more GPUs with --gpus to train across them, as a DistributedDataParallel process group (--multi-gpu ddp, the default for multiple GPUs) or the slower DataParallel (--multi-gpu dp). Single-process training also covers the CPU and Apple MPS.

Usage

slvt train [OPTIONS]

Options

-cfg, --config-path <config_path>

Required The path to the DeepLabCut project’s config.yaml that owns the shuffle to train.

-s, --shuffle <shuffle>

The index of the shuffle to train. A shuffle pairs a train/test split with a model architecture, both fixed when it was created by ‘slvt prepare’. Change it to train a different prepared shuffle.

Default:

1

-e, --epochs <epochs>

The maximum number of passes over the training set for the pose model. Higher values train longer, but may be necessary to achieve model convergence on complex datasets. Omit to use the model’s default value.

-b, --batch-size <batch_size>

The number of frames the pose model processes per optimization step. Larger batches use more GPU memory and can speed up training. Omit to use the model’s default value.

-se, --save-epochs <save_epochs>

The number of epochs between saved pose-model snapshots. Smaller values checkpoint more often at the cost of disk space. Omit to use the model’s default value.

-di, --display-iterations <display_iterations>

The number of training iterations between loss readouts within each epoch. Omit to use the model’s default value.

-ms, --maximum-snapshots <maximum_snapshots>

The maximum number of recent snapshots to retain per model. Older snapshots beyond this count are deleted. Omit to use the model’s default value.

-sp, --snapshot-path <snapshot_path>

The path to a pose-model snapshot to resume training from, instead of starting from the shuffle’s initial weights.

-dp, --detector-path <detector_path>

The path to a detector snapshot to resume training from. Applies to top-down models only.

-dbs, --detector-batch-size <detector_batch_size>

The number of frames the object detector processes per step, for top-down models. Omit to use the model’s default value.

-de, --detector-epochs <detector_epochs>

The maximum number of training passes for the object detector, for top-down models. Set to 0 to skip detector training and fit only the pose model.

-dse, --detector-save-epochs <detector_save_epochs>

The number of epochs between saved detector snapshots, for top-down models. Omit to use the model’s default value.

-lhw, --load-head-weights, --no-load-head-weights

Determines whether the pose model’s head weights are restored when resuming from a snapshot. Disable this when the bodypart set has changed, since the snapshot’s head no longer matches.

Default:

True

-d, --device <device>

The base device to train on. ‘auto’ selects a CUDA GPU when one is visible and otherwise the CPU. ‘cpu’ and ‘mps’ (Apple Metal) force those devices. Choose specific GPUs with –gpus.

Default:

'auto'

Options:

auto | cpu | mps | cuda

-g, --gpus <INDICES>

The comma-separated CUDA device indices to train on. Omit to train on GPU 0. List two or more indices (e.g. ‘0,1’) to train across them, with the strategy chosen by –multi-gpu. Only applicable when training device is a CUDA-compatible GPU.

-mg, --multi-gpu <multi_gpu>

The strategy for training across the GPUs selected with –gpus. ‘auto’ uses DistributedDataParallel when two or more GPUs are selected. ‘ddp’ forces it. ‘dp’ uses the slower DataParallel. Ignored when a single GPU is used.

Default:

'auto'

Options:

auto | ddp | dp

-a, --amp <amp>

The mixed-precision compute mode, which trades some numerical precision for speed and lower memory use. ‘auto’ enables bfloat16 on Ampere or newer GPUs and stays in float32 elsewhere. ‘off’ forces float32. ‘bf16’ forces bfloat16 (disabled with a warning on MPS) and ‘fp16’ forces float16 on CUDA (disabled with a warning elsewhere).

Default:

'auto'

Options:

auto | off | bf16 | fp16

-t, --tf32 <tf32>

TF32 acceleration for float32 matmuls and convolutions on CUDA, which speeds up float32 math at slightly reduced precision. ‘auto’ enables it on Ampere or newer GPUs. ‘on’ and ‘off’ force it. It is a no-op off CUDA.

Default:

'auto'

Options:

auto | on | off

-cb, --cudnn-benchmark <cudnn_benchmark>

The cuDNN convolution autotuner on CUDA. ‘auto’ enables it only when the shuffle’s training transform is detected to feed one fixed input size, where it speeds up convolutions. It disables deterministic training and can slow variable-size augmentation, so it stays off otherwise. ‘on’ and ‘off’ force it.

Default:

'auto'

Options:

auto | on | off

-cm, --compile-model <compile_model>

Determines whether the model is compiled with torch.compile for faster steps. ‘auto’ leaves it off because its one-time warm-up cost may not amortize. ‘on’ and ‘off’ force it.

Default:

'auto'

Options:

auto | on | off

-dw, --dataloader-workers <dataloader_workers>

The number of worker processes each training process uses to load and augment data. More workers feed the GPU faster until the CPU saturates. Set to -1 to derive it from the CPU count, capped at 8 per process.

Default:

-1

-pm, --pin-memory <pin_memory>

Determines whether dataloaders pin host memory to speed up host-to-device transfers on CUDA. ‘auto’ enables it on CUDA. ‘on’ and ‘off’ force it. It has no effect off CUDA.

Default:

'auto'

Options:

auto | on | off

-ev, --evaluate, --no-evaluate

Determines whether the trained snapshot is scored against the labeled frames as a final step, writing the evaluation feather and its provenance sidecar. Disable to finish at the last snapshot without evaluating.

Default:

True

-ebs, --evaluation-batch-size <evaluation_batch_size>

The number of frames scored per forward pass during the post-training evaluation, which runs in float32 on a single device. Larger values evaluate faster but use more GPU memory. Raise it on a capable GPU.

Default:

1

-ecc, --evaluation-confidence-cutoff <evaluation_confidence_cutoff>

The confidence cutoff for the evaluation’s cutoff-filtered metrics. Predictions the model makes below this confidence are excluded from the cutoff-filtered error, so it reflects accuracy on only the keypoints the model is confident about. Omit to fall back to the project config’s p-cutoff (0.6 when unset).

-p, --progress, --no-progress

Determines whether the aggregate progress bar is shown during training.

Default:

True

Hardware Detection

Provides shared device, capability, and mixed-precision detection for the training and inference optimizers.

class sollertia_video_tracking.hardware.AmpMode(*values)

Bases: StrEnum

The automatic-mixed-precision selection: the capability default, disabled, or a forced compute dtype.

AUTO = 'auto'

Enables bfloat16 only where it is natively fast, staying at full float32 otherwise.

BF16 = 'bf16'

Forces bfloat16 autocast, falling back with a warning on a device that cannot run it.

FP16 = 'fp16'

Forces float16 autocast (CUDA only), falling back with a warning elsewhere.

OFF = 'off'

Disables mixed precision and runs in full float32.

class sollertia_video_tracking.hardware.DeviceType(*values)

Bases: StrEnum

The base compute device to run on: automatic selection, the CPU, Apple MPS, or a CUDA GPU.

AUTO = 'auto'

Selects CUDA when a GPU is visible, otherwise the CPU.

CPU = 'cpu'

Runs on the CPU.

CUDA = 'cuda'

Runs on CUDA GPUs. Choose which indices with the GPU-indices option.

MPS = 'mps'

Runs on the Apple Metal (MPS) backend.

class sollertia_video_tracking.hardware.Toggle(*values)

Bases: StrEnum

The tri-state control for one optimization: the capability-detected default, forced on, or forced off.

AUTO = 'auto'

Uses the capability-detected default chosen for the selected device.

OFF = 'off'

Forces the optimization off.

ON = 'on'

Forces the optimization on wherever the device can run it.

sollertia_video_tracking.hardware.apply_backend_flags(*, device, tf32, cudnn_benchmark)

Applies the process-global CUDA backend flags shared by training and inference workers.

TF32 and the cuDNN autotuner are process-global CUDA backend flags, so each worker process must apply them itself before its first forward pass. This is a no-op on non-CUDA devices.

Parameters:

• device (str) – The resolved base device type the worker runs on.

• tf32 (bool) – Whether TF32 acceleration is enabled for float32 matmuls and convolutions.

• cudnn_benchmark (bool) – Whether the cuDNN convolution autotuner is enabled.

Return type:

None

sollertia_video_tracking.hardware.precision_label(amp_dtype)

Returns the human-readable precision label for an autocast dtype.

Parameters:

amp_dtype (dtype | None) – The autocast compute dtype, or None for full float32 precision.

Return type:

str

Returns:

The precision label ("bfloat16", "float16", or "fp32").

sollertia_video_tracking.hardware.resolve_amp_dtype(amp, device, gpus)

Reconciles the requested mixed-precision mode with the device and its capabilities into an autocast dtype.

A forced dtype the device cannot support (bfloat16 on MPS, float16 off CUDA) is disabled with a warning rather than a silent refusal. The "auto" default enables bfloat16 only where it is natively fast, so it stays close to stock float32 behavior; on CPU the benefit is chip-dependent and left as an explicit opt-in.

Parameters:

• amp (AmpMode) – The requested mixed-precision mode.

• device (str) – The resolved base device type.

• gpus (tuple[int, ...]) – The resolved CUDA device indices.

Return type:

dtype | None

Returns:

The autocast dtype to use, or None when mixed precision is disabled. Callers that train derive their own gradient-scaler requirement from a torch.float16 result.

sollertia_video_tracking.hardware.resolve_target_device(device, gpus, *, role, default_all_gpus=True)

Reconciles the requested device and GPU indices with the available hardware.

The device selection cascades cuda -> cpu when no CUDA device is visible, so the same call works unchanged on a GPU server or a CPU-only server.

Parameters:

• device (str | None) – The requested device ("auto", "cpu", "mps", "cuda", or "cuda:N"), or None for automatic selection.

• gpus (tuple[int, ...] | None) – The explicitly requested CUDA device indices, or None to select them automatically.

• role (str) – The run role named in error messages, for example "training" or "inference".

• default_all_gpus (bool, default: True) – When gpus is None on the CUDA path, determines whether every visible GPU is selected (True) or only the first (False). Inference defaults to every GPU to spread videos across them, while training selects only the first so that multi-GPU stays an explicit opt-in.

Return type:

tuple[str, tuple[int, ...]]

Returns:

A tuple of the resolved base device type and the tuple of CUDA indices to use.

Raises:

ValueError – When an explicitly requested CUDA index is not present on the machine, or when the requested device does not begin with "cuda" and is not one of "auto", "cpu", or "mps".

sollertia_video_tracking.hardware.resolve_toggle(value, *, auto)

Resolves a tri-state toggle to a boolean, using the given capability-detected default for "auto".

Parameters:

• value (Toggle) – The requested tri-state value.

• auto (bool) – The capability-detected default applied when the value is "auto".

Return type:

bool

Returns:

The resolved boolean decision.

sollertia_video_tracking.hardware.supports_ampere(gpus)

Determines whether every listed CUDA device reaches Ampere compute capability or newer.

Ampere is the threshold for both TF32 matmul/convolution acceleration and native bfloat16 tensor cores, so the same check gates both optimizations.

Parameters:

gpus (tuple[int, ...]) – The CUDA device indices to check.

Return type:

bool

Returns:

True when at least one device is listed and every listed device reports at least Ampere compute capability, False otherwise (including when the device list is empty).

sollertia_video_tracking.hardware.warn(message)

Writes a non-fatal warning to the standard error stream.

Parameters:

message (str) – The warning text to emit, without the WARNING: prefix or trailing newline.

Return type:

None

Progress Reporting

Provides the shared live progress-bar assets the training, inference, and frame-extraction bars build on.

class sollertia_video_tracking.reporting.LiveBar(progress_queue, *, preparing_label='preparing...', stream=None, width=30)

Bases: Thread

Renders a single in-place live progress bar from messages streamed over a shared queue.

Notes

This base owns the rendering scaffolding every progress bar in the library shares: interactive-terminal detection, the message-consuming run loop, and the per-mode minimum render interval. It also owns the liveness spinner advanced on every drawn line, the elapsed clock, the warm-up preparing line shown before the first unit of work is reported, and the in-place carriage-return versus one-line-per-update rendering. It breaks the run loop on the shared stop sentinel, so a subclass never handles termination itself.

Subclasses supply only their own behavior: _ingest merges one message and reports whether to force an immediate redraw, _is_preparing reports whether work has started, and _compose_active builds the line body once it has. The shared _bar and _eta helpers keep the bar glyphs and the honest ETA consistent across every subclass, and _compose_preparing may be overridden to add warm-up context such as a count.

_progress_queue

The shared queue the producer streams progress messages to.

_preparing_label

The text shown during warm-up, before the first unit of work is reported.

_width

The width, in characters, of the rendered bar.

_stream

The output stream the bar renders to.

_is_tty

True when the output stream is an interactive terminal.

_start_time

The monotonic timestamp captured when the renderer was constructed.

_last_render_time

The monotonic timestamp of the most recent render.

_spinner_index

The number of lines drawn so far, used to advance the liveness spinner on each drawn line.

run()

Consumes queue messages and re-renders the bar until the shared stop sentinel arrives.

Return type:

None

stop()

Signals the renderer to draw a final frame and exit.

Return type:

None

sollertia_video_tracking.reporting.format_duration(seconds)

Formats a duration as MM:SS, or as H:MM:SS when it spans an hour or more.

Parameters:

seconds (float) – The duration to format, in seconds.

Return type:

str

Returns:

The duration rendered as MM:SS, or H:MM:SS past an hour.