uval.stages package

Submodules

uval.stages.combine_files module

A StageData object acts like a data container that manages results generated by stages and read by subsequent stages.

uval.stages.combine_files.create_supported_dataset_from_intersection(gt_files: uval.stages.stage_data.Hdf5FilesData, det_files: uval.stages.stage_data.Hdf5FilesData)[source]

Given a list of ground truth and detection files, try to join them given their volume ids.

uval.stages.combine_files.support_dataset_with_file_paths(gt_files: uval.stages.stage_data.Hdf5FilesData, det_files: uval.stages.stage_data.Hdf5FilesData, soc_files: uval.stages.stage_data.Hdf5FilesData)[source]

Given a dataset and sets of Hdf5 files found on disk, try to combine everything to create a supported dataset.

uval.stages.dataset_specification module

A StageData object acts like a data container that manages results generated by stages and read by subsequent stages.

uval.stages.dataset_specification.load_datasplit(file_path: str, subsets=None, output=None) → uval.stages.stage_data.DatasetSpecificationData[source]

This stage will load a data split file in YAML format and keep the id lists for train, val and test set.

Parameters

  • file_path (str) – path to the yaml file.

  • subsets ([str], optional) – which subsets of the yaml files are to be loaded,

  • None (if set to) –

  • None. (all subsets are used. Defaults to) –

  • output – The output file to be generated, if needed

Returns

returns the dataset.

Return type

DatasetSpecificationData

uval.stages.hdf5 module

This module include stages that are used to find and load HDF5 files from disk. The result will be returned as a Hdf5FilesData object

uval.stages.hdf5.load_detections(folder_path: str, recursive: bool = False, dataset=None, ignore_missing_files=False, max_workers=None) → uval.stages.stage_data.Hdf5FilesData[source]

Detections in non negative images are loaded based on the YAML file.

Parameters

  • folder_path (str) – path to the data.

  • recursive (bool, optional) – set to True if subdirectories should also be searched. Defaults to False.

  • dataset ([DatasetSpecificationData], optional) – The dataset that was loaded from YAML files. Defaults to None.

  • ignore_missing_files – If a HDF5 file is missing, no error is raised

Raises

Exception – if a file includes in the dataset but not in the directory (sanity check).

Returns

Detections in positive images.

Return type

Hdf5FilesData

uval.stages.hdf5.load_evaulation_files(folder_path: str, recursive=False, dataset=None, ignore_missing_files=False, max_workers=None) → Tuple[uval.stages.stage_data.Hdf5FilesData, uval.stages.stage_data.Hdf5FilesData, uval.stages.stage_data.Hdf5FilesData][source]

This is a wrapper that calls underlying functions and loads positive detections, GTs and negative detections.

Parameters

  • folder_path (str) – path to the data folder.

  • recursive (bool, optional) – set to True if subdirectories should also be searched. Defaults to False.

  • dataset ([DatasetSpecificationData], optional) – The input dataset

  • None. (loaded from YAML file. Defaults to) –

  • ignore_missing_files – If a HDF5 file is missing, no error is raised

Returns

Tuple including all three parts of the dataset.

Return type

Hdf5FilesData

uval.stages.hdf5.load_gt(folder_path: str, recursive: bool = False, dataset=None, ignore_missing_files=False, max_workers=None) → uval.stages.stage_data.Hdf5FilesData[source]

Ground truth in non negative images are loaded based on the YAML file.

Parameters

  • folder_path (str) – path to the data.

  • recursive (bool, optional) – set to True if subdirectories should also be searched. Defaults to False.

  • dataset ([DatasetSpecificationData], optional) – The dataset that was loaded from YAML files. Defaults to None.

  • ignore_missing_files – If a HDF5 file is missing, no error is raised

Raises

Exception – if a file includes in the dataset but not in the directory (sanity check).

Returns

Ground truths in positive images.

Return type

Hdf5FilesData

uval.stages.hdf5.load_hdf5_files(folder_path: str, recursive: bool = False, file_filter: str = '*.h5', list_of_files=None, ignore_missing_files=False, max_workers=None) → uval.stages.stage_data.Hdf5FilesData[source]

This function searches for uval compatible HDF5 files in the given folder. It will load their meta data to make sure we can match and access them quickly by volume id.

uval.stages.hdf5.load_negatives(folder_path: str, recursive: bool = False, dataset: uval.stages.stage_data.DatasetSpecificationData = None, ignore_missing_files: bool = False, max_workers=None) → uval.stages.stage_data.Hdf5FilesData[source]

Detections in negative images are loaded based on the YAML file.

Parameters

  • folder_path (str) – path to the data.

  • recursive (bool, optional) – set to True if subdirectories should also be searched. Defaults to False.

  • dataset ([DatasetSpecificationData], optional) – The dataset that was loaded from YAML files. Defaults to None.

  • ignore_missing_files – If a HDF5 file is missing, no error is raised

Raises

Exception – if a file includes in the dataset but not in the directory (sanity check).

Returns

Detections in Negative images.

Return type

Hdf5FilesData

uval.stages.metrics module

This module provides stages that can compute metrics like overlaps between groundtruth and detections, or simply count detections per bag.

class uval.stages.metrics.DetectionEntry(volume_id: str, class_name: str, confidence_score: float, roi_start: Tuple[int], roi_shape: Tuple[int])[source]

Bases: object

Any detection entry wether it is from GT or detections will be converted in this form.

class uval.stages.metrics.Metrics(dataset: uval.stages.stage_data.SupportedDatasetSpecificationData, metrics_settings, output_settings)[source]

Bases: object

This class implements all the evaluation metrics.

basic_metric(iou_threshold: float = None, confidence_threshold: float = None) → List[dict][source]

Get the TP, FP, Precision and recall.

Parameters

  • iou_threshold (float, optional) – Threshold for IOU. Defaults to None.

  • confidence_threshold (float, optional) – Threshold for confidence. Defaults to None.

Returns

A list of dictionaries. Each dictionary contains information and metrics of each class.

Return type

List[dict]

static calculate_average_precision(rec: List[float], prec: List[float]) → List[Any][source]
property confidence_threshold
data_preparations()[source]
static eleven_point_interpolated_ap(rec: List[float], prec: List[float]) → List[Any][source]
evaluate()[source]
evaluate_range(iou_range: List[float], confidence_threshold: Optional[float] = None)[source]
generate_report(results_cluttered: dict) → None[source]
get_average_precision(basic_metrics: List[dict], method: str = None) → List[dict][source]

Get the average precision. This will be used in multiple other metrics such as COCO or pascal voc.

Parameters

  • basic_metrics (List[dict]) – [description]

  • method (str, optional) – choice between precise (EveryPointInterpolation or None)

  • estimation (or) –

Returns

adds ap to the each class of the output dictionaries.

Return type

List[dict]

get_average_recall(recalls: Dict[float, list], iou_range: List) → List[source]
get_fscore(basic_metrics: List[dict]) → List[dict][source]

Get the f score metrics.

Parameters

  • basic_metrics (List[dict]) – output of basic_metric method.

  • method. (needs to be called before this) –

Returns

adds dict[‘F score’] and dict[‘F score soft’] to the inputs.

Return type

List[dict]

get_pascal_voc2007_metric(confidence_threshold=None) → list[source]
get_pascal_voc2012_metric(confidence_threshold=None) → list[source]
static iou(start_a: List[float], shape_a: List[float], start_b: List[float], shape_b: List[float]) → float[source]

Calculates the intersection over union of the two cubes A and B.

Parameters

  • start_a (List[float]) – bottom left corner of the cube A.

  • shape_a (List[float]) – size of each dimension in the cube A.

  • start_b (List[float]) – bottom left corner of the cube B.

  • shape_b (List[float]) – size of each dimension in the cube B.

Returns

3D IOU of these cubes.

Return type

float

property iou_threshold
property output_path
plot_precision_recall_curve(pascal_voc_metrics: List[dict], show_ap: bool = True, show_interpolated_precision: bool = True, show_graphic: bool = False) → None[source]

Plot the Precision x Recall curve for a given class.

Parameters

  • pascal_voc_metrics (List[dict]) – Output of some pascal voc metric. needs to be

  • method. (called before this) –

  • show_ap (bool, optional) – if True, the average precision value will be shown

  • False. (the interpolated precision. Defaults to) –

  • show_interpolated_precision (bool, optional) – if True, it will show in the plot

  • False. –

  • show_graphic (bool, optional) – if True, the plot will be shown. Defaults to False.

Raises

IOError – [description]

plot_recall_iou_curve(recalls: Dict[float, List[float]], iou_thresholds: List[float], classes: List[str], show_ar: bool = True) → None[source]

Plot the Recall x IOU curve for a given class.

Parameters

  • (Dict[float (recalls) – list]): keys are iou thresholds. value is a list of recall for each class.

  • iou_thresholds – list of iou thresholds.

  • classes – list containing names of all classes.

plot_roc_curves(roc_metrics: List[dict], show_graphic: bool = False) → None[source]

Plot the ROC curve for every class.

Parameters

  • roc_metrics (List[dict]) – Output of some basic_metric. needs to be

  • method. (called before this) –

  • show_graphic (bool, optional) – if True, the plot will be shown. Defaults to False.

Raises

IOError – [description]

property templates_path
worker(iou_threshold)[source]

uval.stages.stage module

This module defines the abstract base class for a stage. A stage is one step of an evaluation pipeline to be executed, for example converting detections from one format to another.

uval.stages.stage.uval_stage(func: Callable)[source]

This decorator has to be applied to every stage that is defined. We can use this to log the start and end of stages, handle caching etc.

uval.stages.stage_data module

A StageData object acts like a data container that manages results generated by stages and read by subsequent stages.

Depending on the stage, each class that inherits from StageData will contain different things, documented below. Many of them will make use of pandas DataFrames to keep the data in a table-like shape.

class uval.stages.stage_data.BagLevelAlarmsData[source]

Bases: uval.stages.stage_data.TableStageData

For every bag id, stores the number of alarms on this bag

frame_columns: List[str] = ['volume_id', 'num_alarms']
class uval.stages.stage_data.DatasetSpecificationData[source]

Bases: uval.stages.stage_data.TableStageData

Contains a list of targets to be detected. Every entry / row refers to a single ground truth target. So multiple rows can refer to the same volume_id if there are multiple targets in one volume.

for more information please see the The Data-split YAML format and HDF5 format in the uval docs.

frame_columns: List[str] = ['volume_id', 'label_id', 'is_negative', 'subset', 'class_name']
class uval.stages.stage_data.DetectionMetricsData[source]

Bases: uval.stages.stage_data.TableStageData

For every bag id, for every GT object, stores the matched detection alarm (index) and its overlap. In addition, for every bag, stores the detection alarm indices that do not have a matching ground truth

class uval.stages.stage_data.Hdf5FilesData[source]

Bases: uval.stages.stage_data.TableStageData

Keeps a list of files that are available on disk. The files can be identified by bag id. For each bag id, contains the file path and the file_meta and volume_meta information. Also stores detections and groundtruth and their scores and box data etc. but not the masks or projections.

frame_columns: List[str] = ['volume_id', 'file_path', 'file_stat', 'hdf5_meta']
class uval.stages.stage_data.StageData[source]

Bases: abc.ABC

The abstract base class for all data that is passed around between stages. For now, we don’t need anything here, but we will sure need this later, for example to store and restore the data when using caching.

class uval.stages.stage_data.SupportedDatasetSpecificationData[source]

Bases: uval.stages.stage_data.DatasetSpecificationData

Similar to DatasetSpecificationData, but in addition, every entry is supported by Hdf5 files for ground truth and detections

frame_columns: List[str] = ['volume_id', 'label_id', 'is_negative', 'subset', 'class_name', 'hdf5_detection', 'hdf5_groundtruth', 'target_id']
class uval.stages.stage_data.TableStageData[source]

Bases: uval.stages.stage_data.StageData

A subclass for all data that is table-like and can be handled using pandas

frame_columns: List[str] = []
from_dict_as_rows(data_dict: dict, check_data_validity: bool = False) → bool[source]
to_csv(file_path: str)[source]
to_html(file_path: str)[source]

Module contents