Module Handbook

Archive

class pbmohpo.archive.Archive(space: ConfigurationSpace)

Bases: object

property incumbents: List[Evaluation]

Get incumbents.

Returns list of incumbents, i.e. configurations with highest utility values.

Returns:

Evaluation with highest utility

Return type:

list[Evaluation]

property max_utility: float

Get current best utility value.

Returns:

highest utility value

Return type:

float

to_numpy(on_search_space: bool = True) → Tuple

Convert evaluated configurations and utility values to numpy arrays

Parameters:

on_search_space (bool) – Whether configurations are on the search space, i.e. respecting log transformations

Returns:

feature values x and utility values y

Return type:

tuple(x, y)

to_torch(on_search_space: bool = True) → Tuple

Convert evaluated configurations and targets to torch arrays

Parameters:

on_search_space (bool) – Whether configurations are on the search space, i.e. respecting log transformations

Returns:

feature values x and utility values y

Return type:

tuple(x, y)

class pbmohpo.archive.Evaluation(config: Configuration, objectives: dict, utility: float)

Bases: object

Result of one Tuning step.

Contains the data from one tuning iteration

Parameters:

  • config (CS.Configuration) – Evaluated configuration

  • objectives (Dict) – Dict of resulting objective values

  • utility (float) – Utility assigned by DM

config: Configuration
objectives: dict
utility: float

Benchmark

class pbmohpo.benchmark.Benchmark(problem: Problem, optimizer: Optimizer, dm: DecisionMaker, eval_budget: int, dm_budget: int, eval_batch_size: int = 1, dm_batch_size: int = 1)

Bases: object

Conduct a benchmark.

Run an optimizer with a decision maker and problem for a given budget.

Parameters:

  • problem (Problem) – Problem to be optimized

  • optimizer (Optimizer) – Used optimizer

  • dm (DecisionMaker) – Decision maker that evaluates objective values from problems

  • eval_budget (int) – Number of configurations that can be evaluated

  • dm_budget (int) – Number of comparisons the DM can make

  • eval_batch_size (int) – How many configurations to propose in one step

  • dm_batch_size (int) – How many comparisons does the DM in one step

run() → None

Run the benchmark.

Run the benchmark by conducting as many steps as given by the budget and populate the archive with the results

Problems

class pbmohpo.problems.problem.Problem(seed: RandomState | int | None = 42)

Bases: ABC

Definition of an Optimization Problem.

Implements the abstract interface to problems used as benchmarks.

Parameters:

seed (int, np.random.RandomState) – Seed passed to the problem

abstract get_config_space() → ConfigurationSpace

Defines the configuration space of the problem

Returns:

The configuration space of the problem

Return type:

ConfigSpace.ConfigurationSpace

abstract get_objective_names() → List

Get the names of the objectives.

Returns:

Names of objectives

Return type:

List

property n_objectives: int

Get number of objectives.

Returns:

Number of objectives

Return type:

int

class pbmohpo.problems.lgboml.LgbOpenML(task_id: int, seed: RandomState | int | None = 42)

Bases: Problem

LightGBM Tuning Problem.

Tune LightGBM for a multiclass classification Problem.

One call evaluates a cross-validation as defined in the OpenML task and computes the average accuracy per class.

Objectives are the accuracies of each class (representing different missclassification costs of a DM)

Parameters:

  • task_id (int) – OpenML task ID

  • seed (int, np.random.RandomState) – Seed passed to the problem

get_config_space() → ConfigurationSpace

Defines the tuning space for LightGBM

Number of iterations (1, 100, log), Learning rate (0.001, 0.3, log), Bagging fraction (0, 1), Feature fraction (0, 1)

Returns:

The configuration space of the problem

Return type:

ConfigSpace.ConfigurationSpace

get_objective_names() → List

Get the names of the objectives, i.e. names of classes

Returns:

Names of objectives

Return type:

List

class pbmohpo.problems.zdt1.ZDT1(seed: RandomState | int | None = 42, dimension: int = 2)

Bases: Problem

Synthetic Test Function ZDT1

Can be scaled to an arbitrary number of dimensions and has two objectives.

Since we are always maximizing we’re taking the negative function values.

Parameters:

  • seed (int, np.random.RandomState) – Seed passed to the problem

  • dimension (int) – Number of dimensions, needs to be at least 2.

get_config_space() → ConfigurationSpace

Defines the configuration space of the problem

Returns:

The configuration space of the problem

Return type:

ConfigSpace.ConfigurationSpace

get_objective_names() → List

Get the names of the objectives.

Returns:

Names of objectives

Return type:

List

objective_names = ['y0', 'y1']
class pbmohpo.problems.yahpo.YAHPO(id: str, instance: str, objective_names: List, fix_hps: dict | None = None, objective_scaling_factors: dict | None = None, seed: RandomState | int | None = 42)

Bases: Problem

YAHPO Gym Problem.

This class wraps YAHPO Gym (https://github.com/slds-lmu/yahpo_gym/).

Parameters:

  • id (str) – Benchmark class from YAHPO

  • instance (str) – Instance of benchmark

  • objective_names (List[str]) – Objectives to optimize

  • fix_hps (Dict) – Dictionary of fixed HPs that should not be optimized

  • objective_scaling_factors (Dict) – Dictionary with objective names as keys and factors to divide output. If not provided, no scaling is done.

  • seed (int, np.random.RandomState) – Seed passed to the problem

get_config_space() → ConfigurationSpace

Defines the configuration space of the problem

Returns:

The configuration space of the problem

Return type:

ConfigSpace.ConfigurationSpace

get_objective_names() → List

Get the names of the objectives.

Returns:

Names of objectives

Return type:

List

Optimizers

class pbmohpo.optimizers.optimizer.BayesianOptimization(config_space: ConfigurationSpace)

Bases: Optimizer

propose_config(archive: Archive, n: int = 1) → List[Configuration]

Propose a new configuration to evaluate.

Takes an archive of previous evaluations and duels and proposes n new configurations.

Parameters:

  • archive (Archive) – Archive containing previous evaluations

  • n (int) – Number of configurations to propose in one batch

Returns:

Proposed Configuration

Return type:

CS.Configuration

class pbmohpo.optimizers.optimizer.Optimizer(config_space: ConfigurationSpace)

Bases: ABC

Definition of an Optimizer

Implements the abstract interface to an Optimizer class.

Parameters:

config_space (CS.ConfigurationSpace) – The config space the optimizer searches over

abstract property dueling: bool
abstract propose_config(archive: Archive, n: int = 1) → List[Configuration]

Propose a new configuration to evaluate.

Takes an archive of previous evaluations and duels and proposes n new configurations.

Parameters:

  • archive (Archive) – Archive containing previous evaluations

  • n (int) – Number of configurations to propose in one batch

Returns:

Proposed Configuration

Return type:

CS.Configuration

abstract propose_duel(archive: Archive, n: int = 1) → List[Tuple[int, int]]

Propose a duel between two Evaluations

Takes an archive of previous evaluations and duels to propose n new duels of two configurations each.

Parameters:

  • archive (Archive) – Archive containing previous evaluations

  • n (int) – Number of duels to propose in one batch

Returns:

List of tuples of two indicies of Archive evaluations to compare

Return type:

List(Tuple(int, int))

class pbmohpo.optimizers.utility_bayesian_optimization.UtilityBayesianOptimization(config_space: ConfigurationSpace, initial_design_size: int | None = None)

Bases: BayesianOptimization

Single objective Bayesian optimization of utility scores.

Implements a simple BO loop to optimize the utility scores provided by the decision maker. Uses a GP surrogate and UCB acquisition function with beta=0.1.

Parameters:

  • config_space (CS.ConfigurationSpace) – The config space the optimizer searches over

  • initial_design_size (int, None) – Size of the initial design, if not specified, two times the number of HPs is used

property dueling: bool
propose_duel(archive: Archive, n: int = 1) → List[Tuple[int, int]]

Propose a duel between two Evaluations

Takes an archive of previous evaluations and duels to propose n new duels of two configurations each.

Parameters:

  • archive (Archive) – Archive containing previous evaluations

  • n (int) – Number of duels to propose in one batch

Returns:

List of tuples of two indicies of Archive evaluations to compare

Return type:

List(Tuple(int, int))

class pbmohpo.optimizers.eubo.EUBO(config_space: ConfigurationSpace, initial_design_size: int | None = None)

Bases: BayesianOptimization

Bayesian Optimization for Pairwise Comparison Data.

Implements the Analytic Expected Utility Of Best Option algrithm. For details see: https://botorch.org/tutorials/preference_bo

Parameters:

  • config_space (CS.ConfigurationSpace) – The config space the optimizer searches over

  • initial_design_size (int, None) – Size of the initial design, if not specified, two times the number of HPs is used

property dueling: bool
propose_duel(archive: Archive, n: int = 1) → List[Tuple[int, int]]

Propose a duel between two Evaluations

Takes an archive of previous evaluations and duels to propose n new duels of two configurations each.

Parameters:

  • archive (Archive) – Archive containing previous evaluations

  • n (int) – Number of duels to propose in one batch

Returns:

List of tuples of two indices of Archive evaluations to compare

Return type:

List(Tuple(int, int))

class pbmohpo.optimizers.eubo.qEUBO(config_space: ConfigurationSpace, initial_design_size: int | None = None)

Bases: EUBO

Bayesian Optimization for Pairwise Comparison Data.

Implements the expected utility of the best option algorithm.

For details see: https://arxiv.org/pdf/2303.15746.pdf

Their implementation can be found here: https://github.com/facebookresearch/qEUBO

Large parts of that code have been used in this implementation.

Parameters:

  • config_space (CS.ConfigurationSpace) – The config space the optimizer searches over

  • initial_design_size (int, None) – Size of the initial design, if not specified, two times the number of HPs is used

Decision Makers

class pbmohpo.decision_makers.decision_maker.DecisionMaker(preferences: Dict | None = None, objective_names: List | None = None, seed: RandomState | int | None = 42)

Bases: object

Simple Decision Maker.

This class constructs a decision maker that has a weight for each objective. Preference weights need to sum to 1.

Parameters:

  • preferences (Dict, None) – Dict containing a weight for each objective

  • objective_names (List, None) – List of objective names, if preferences are not given, preference values are sampled randomly

  • seed (int, np.random.RandomState) – Seed used to generate the preferences if not given

compare(objectives1: Dict, objectives2: Dict) → bool

Check if the DM prefers objectives 1 over objectives 2.

Parameters:

  • objectives1 (Dict) – Dict of objectives with associated values that should be checked for preference

  • objectives2 (Dict) – Dict of objectives with associated values, that should be checked if objectives1 is preferred by DM

Returns:

Does DM prefer objectives1 over objectives2

Return type:

bool

Utils

pbmohpo.utils.color_generator()

Generates colors for archives visualization to match YAHPO colors.

pbmohpo.utils.get_botorch_bounds(space: ConfigurationSpace, on_search_space: bool = True) → List

Get bounds of hyperparameters in the format botorch needs, If on_search_space is True, the bounds are returned as on the search space, i.e. respecting log transformations.

Parameters:

  • space (CS.configuration_space) – Configuration space of the problem

  • on_search_space (bool) – Whether candidates are on the search space, i.e. respecting log transformations

Returns:

list of [lower, upper] bounds of each hyperparameter

Return type:

list

pbmohpo.utils.get_config_values(config: Configuration, space: ConfigurationSpace, on_search_space: bool = True) → List

Get the values of a configuration. If on_search_space is True, the values are returned as on the search space, i.e. respecting log transformations. If on_search_space is False, the values are returned as on the original space.

Parameters:

  • config (CS.Configuration) – Configuration to be evaluated

  • space (CS.ConfigurationSpace) – Search space of the problem

  • on_search_space (bool) – Whether candidates are on the search space, i.e. respecting log transformations

Returns:

List of values of the configuration

Return type:

List

pbmohpo.utils.remove_hp_from_cs(old_cs: ConfigurationSpace, remove_hp_dict: Dict) → ConfigurationSpace

Remove parameters from the search space.

Parameters:

  • old_cs (CS.ConfigurationSpace) –

  • remove_hp_dict (Dict) – Dict of hyperparameters with hyperparameters as keys and defaults as values

Return type:

CS.ConfigurationSpace

pbmohpo.utils.visualize_archives(archive_list: List[Archive], plot_elements: List[str] = ['incumbent'], legend_elements: List[str] = None)

Visualize archive utility and incumbent utility over iterations.

Parameters:

  • archive_list (list[Archive]) – List of archives to be visualized

  • plot_elements (list[str]) – List of elements that should be plotted. Currently, supports “incumbent”, which plots the incumbent utility over iteration and “utilities”, which plots the utility for each iteration over iteration.

  • legend_elements (list[str]) – List of elements that should be included in the legend. Must be of the same length as archive_list.

Return type:

matplotlib item