Module reference¶
Benchmark class¶
A class that manages the database entries for the measured metrics which are logged into the database.
Parameters¶
- db_filestr
The path of the database file
- descriptionstr, optional
The description of the whole pipeline use case. Even though the description is optional, it should be set so the database entries are distinguishable without evaluating the uuid’s. This parameter is ignored for Benchmark objects initialized in mode ‘r’.
- modestr, default=’a’
One of [‘w’, ‘a’, ‘r’]. The mode corresponds to conventional python file handling modes. Modes ‘a’ and ‘w’ are used for storing metrics in a database during a pipeline run and ‘r’ is used for querying metrics from the database.
Attributes¶
- db_filestr
path to the database file
- modestr
mode of the Benchmark object. One of [‘w’, ‘a’, ‘r’].
- descriptionstr
description of the pipeline run. Not relevant if mode is ‘r’.
- sessionsqlalchemy.orm.session.Session
SQLalchemy session
Metrics¶
Supervised metrics¶
- class umlaut.BenchmarkSupervisor(metrics, benchmark)¶
A supervisor object managing all supervised metrics
This object should be used as a decorator.
- metrics: list of Metric
A list of metrics to be collected while running the decorated function
- benchmark: Benchmark
The central benchmark object used in the pipeline
- class umlaut.CPUMetric(description, interval=1)¶
The metric object to measure CPU usage of the running Python instance in percent
- description: str
The description of this metric and function which is added to the database
- interval: int, default=1
The number of seconds between CPU usage measurements
- class umlaut.ThroughputMetric(description)¶
The metric object to measure throughput of the pipeline function
To pass the number of data points processed, use method track()
- description: str
The description of this metric and function which is added to the database
- class umlaut.LatencyMetric(description)¶
The metric object to measure latency of the pipeline function
To pass the number of data points processed, use method track()
- description: str
The description of this metric and function which is added to the database
- class umlaut.PowerMetric(description, interval=1)¶
The metric object to measure power used in the execution
- description: str
The description of this metric and function which is added to the database
- interval: int, default=1
The number of seconds between memory measurements
- class umlaut.EnergyMetric(description)¶
The metric object to measure energy used in the execution
- description: str
The description of this metric and function which is added to the database
- class umlaut.MemoryMetric(description, interval=1)¶
The metric object to measure memory used in the execution
- description: str
The description of this metric and function which is added to the database
- interval: int, default=1
The number of seconds between memory measurements
- class umlaut.TimeMetric(description)¶
The metric object to measure the time taken for the execution
- description: str
The description of this metric and function which is added to the database
Tracked metrics¶
- class umlaut.LossTracker(benchmark)¶
- serialize(loss_values)¶
- loss_valueslist of ints
List of tracked loss values of the run.
- pickle object
Serialized data.
- track(loss_values, description)¶
- loss_valueslist of ints
List of tracked loss values of the run.
- descriptionstr
Description of tracked loss.
- class umlaut.TTATracker(benchmark)¶
- serialize(accuracies)¶
- accuracieslist of ints
List of tracked accuracies of the run.
- pickle object
Serialized data.
- track(accuracies, description)¶
- accuracieslist of ints
List of tracked accuracies of the run.
- descriptionstr
Description of tracked TTA.
- class umlaut.HyperparameterTracker(benchmark, description, hyperparameters, target, low_means_good=True)¶
- serialize()¶
- pickle object
Serialized data.
- track(measurement)¶
- measurementlist of str
List of hyperparameters
- class umlaut.ConfusionMatrixTracker(benchmark)¶
- serialize(matrix, labels)¶
- matrixlist of list of ints
Tracker matrix values.
- labelslist of str
Class labels of confustion matrix.
- pickle object
Serialized data.
- track(matrix, labels, description)¶
Pass an ndarray where axis 0 is predicted and axis 1 is actual.
- matrixlist of list of ints
Tracker matrix values.
- labelslist of str
Class labels of confusion matrix.
- description:
Description of tracked confusion matrix.