Hyperopt.jl

Hyperopt.jl is a package for hyperparameter optimization in Julia. It provides the following methods:

• Random search, RandomSampler.
• Latin hypercube sampling with continuous hyperparameters, LHSampler, and with continuous and categorical hyperparameters, CLHSampler.
• Hyperband, Hyperband, which evaluates many configurations with a small budget and evaluates the most promising configurations again with larger budgets.
• BOHB, BOHB, which replaces the random sampling of Hyperband by sampling from kernel density estimates fitted to the evaluated configurations.

The package is designed to add hyperparameter optimization to existing code. The macro @hyperopt is wrapped around the code that evaluates a configuration of the hyperparameters, and the user specifies the names of the hyperparameters and their candidate values.

Installation

using Pkg
Pkg.add("Hyperopt")

Quick start

Consider code that trains a model with the manually selected hyperparameters a and b:

a = manually_selected_value
b = other_value
cost = train_model(a, b)

To optimize a and b, the code is wrapped in @hyperopt, and the manually selected values are replaced by vectors of candidate values:

ho = @hyperopt for i = number_of_samples,
                   a = candidate_values,
                   b = other_candidate_values
    cost = train_model(a, b)
end

The following example evaluates 50 configurations drawn at random:

using Hyperopt
train_model(a, b) = (a - 3)^2 + (b ? 0.0 : 1.0)

ho = @hyperopt for i = 50,
                   a = LinRange(1, 5, 100),
                   b = [true, false]
    cost = train_model(a, b)
end

Hyperoptimizer with
  a length: 100
  b candidates: [true, false]
  minimum / maximum: (0.0004081216202428376, 4.379655137230895)
  minimizer:
        a         b 
     3.02      true 

The returned Hyperoptimizer records the configurations and the results, and provides the best configuration:

ho.minimizer, ho.minimum

(Real[3.0202020202020203, true], 0.0004081216202428376)

printmin(ho)

a = 3.0202020202020203
b = true

Which sampler to use?

• RandomSampler is the default sampler and a robust baseline. It supports candidate vectors of any length and any element type.
• LHSampler and CLHSampler distribute the configurations evenly over the candidate values. They are preferable to random search if the number of iterations is small, but they require candidate vectors whose lengths are related to the number of iterations, see Samplers.
• Hyperband is preferable if the cost of an evaluation is determined by a budget, e.g., the number of training epochs or a time limit, and the result obtained with a small budget indicates the result obtained with a large budget.
• BOHB is used as the inner sampler of Hyperband. It proposes configurations based on the results of the configurations evaluated before.

Contents

• The @hyperopt macro
• Samplers
• Hyperband
• BOHB
• Function interface
• Optim
• Plotting
• API

Citing

If you use Hyperopt.jl in academic work, please cite it using the entry in CITATION.bib:

@misc{bagge2018hyperopt,
  title        = {Hyperopt.jl: Hyperparameter optimization in {J}ulia},
  author       = {Bagge Carlson, Fredrik},
  year         = {2018},
  month        = aug,
  howpublished = {\url{https://github.com/baggepinnen/Hyperopt.jl}},
  url          = {https://github.com/baggepinnen/Hyperopt.jl},
  note         = {Lund University Publications, \url{https://lup.lub.lu.se/record/6ec19989-9b30-448c-be5e-bae4c4257c7b}}
}