The `@hyperopt` macro

Syntax

ho = @hyperopt for i = n,
                   sampler = RandomSampler(),
                   ho = existing_ho,
                   a = candidates_a,
                   b = candidates_b
    loop body
end

The first argument of the for loop, i = n, determines the number of iterations n, i.e., the number of configurations that are drawn and evaluated. In the loop body, i holds the index of the current iteration. With a Hyperband sampler, n is instead the maximum budget, and i holds the budget of the current evaluation, see Hyperband.
The optional argument sampler selects the sampler that draws the configurations. The default is RandomSampler(), see Samplers.
The optional argument ho continues the optimization of an existing Hyperoptimizer, see Continuing an optimization.
The remaining arguments name the hyperparameters and give their candidate vectors. In the loop body, the variables a and b hold the values of the hyperparameters of the current configuration. A logarithmically spaced candidate vector, e.g., exp10.(LinRange(-3, 1, 100)), results in log-uniform sampling.

The value of the loop body is the result of the evaluation, typically a real-valued cost that is to be minimized. The loop body is the body of a function, and the variables of the enclosing scope are accessible as in a closure. The variable state is defined in the loop body; it is nothing unless the sampler is a Hyperband sampler. See the docstring of @hyperopt for details.

Example

The function f is minimized with respect to the hyperparameters a, b and c. The candidate vector of c is logarithmically spaced.

f(x, a, b = true; c = 10) = sum(@. x + (a - 3)^2 + (b ? 10 : 20) + (c - 100)^2)

ho = @hyperopt for i = 50,
                   sampler = RandomSampler(),
                   a = LinRange(1, 5, 1000),
                   b = [true, false],
                   c = exp10.(LinRange(-1, 3, 1000))
    x = 100
    f(x, a, b; c)
end

Hyperoptimizer with
  a length: 1000
  b candidates: [true, false]
  c length: 1000
  minimum / maximum: (703.0463883220827, 701520.0901880633)
  minimizer:
        a         b         c 
    2.153      true     75.66

Results

The Hyperoptimizer ho records the evaluated configurations in ho.history and the results in ho.results; ho.results[k] is the result of the configuration ho.history[k]:

ho.history[1], ho.results[1]

(Real[3.6346346346346348, false, 8.588828559546247], 8476.405025235506)

The properties minimum, minimizer, maximum and maximizer provide the optimal result and configuration:

ho.minimum, ho.minimizer

(703.0463883220827, Real[2.153153153153153, true, 75.66218500481055])

printmin and printmax print the optimal configuration together with the names of the hyperparameters:

printmin(ho)

a = 2.153153153153153
b = true
c = 75.66218500481055

A NaN result, e.g., of an evaluation that failed, is treated as Inf by minimum and minimizer and as -Inf by maximum and maximizer. It is therefore not selected unless all results are NaN:

ho_nan = @hyperopt for i = 20, a = LinRange(-1, 1, 21)
    a < 0 ? NaN : a^2
end
ho_nan.minimum, ho_nan.maximum

(0.0, 1.0)

If no result has been recorded, ho.minimum and ho.maximum are NaN.

warn_on_boundary emits a warning for each hyperparameter whose optimal value is the smallest or the largest sampled value, which indicates that the candidate range is to be extended. In the following example, the cost is minimized by a = 3, which is outside of the candidate range:

ho_boundary = @hyperopt for i = 30, a = LinRange(0, 2, 100)
    (a - 3)^2
end
boundary_parameters = warn_on_boundary(ho_boundary)

┌ Warning: Parameter a obtained its optimum on an extremum of the sampled region: 1.97979797979798
└ @ Hyperopt ~/work/Hyperopt.jl/Hyperopt.jl/src/Hyperopt.jl:497

The names of these hyperparameters are returned:

boundary_parameters

1-element Vector{Symbol}:
 :a

Continuing an optimization

The argument ho = ho continues the optimization of an existing Hyperoptimizer. The configurations and results of the new run are appended to ho.history and ho.results, and the sampler of ho is continued, including the state of its random number generator:

ho = @hyperopt for i = 20, ho = ho,
                   a = LinRange(1, 5, 1000),
                   b = [true, false],
                   c = exp10.(LinRange(-1, 3, 1000))
    f(100, a, b; c)
end
length(ho.history), length(ho.results)

(70, 70)

The returned Hyperoptimizer shares history and results with the existing one. A sampler given in the continued run is not used; a warning is emitted if it is of another type than the sampler of the existing Hyperoptimizer.

Categorical hyperparameters

The candidates of a hyperparameter can be values of any type, e.g., functions or symbols. RandomSampler draws them like numerical values:

ho_cat = @hyperopt for i = 30, fun = [tanh, sin, cos], x = LinRange(0, 1, 50)
    fun(x) + (x - 0.5)^2
end
printmin(ho_cat)

fun = tanh
x = 0.02040816326530612

CLHSampler and BOHB require the categorical hyperparameters to be declared with Categorical or UnorderedCategorical, see Samplers and BOHB.

Interruption and errors

If an evaluation throws an InterruptException, e.g., after Ctrl-C, the optimization is stopped and the Hyperoptimizer with the results obtained so far is returned. Other exceptions are rethrown. If the optimization of an existing Hyperoptimizer was continued, its history then contains only the configurations whose evaluation completed, so that ho.history and ho.results have equal lengths. An evaluation that is expected to fail for some configurations can return NaN instead, e.g., by

ho = @hyperopt for i = 100, a = LinRange(1, 5, 100)
    try
        train_model(a)
    catch
        NaN
    end
end

Progress meters

@hyperopt, hyperband and hyperoptim display a progress meter. The progress meters are disabled by

Hyperopt.SHOW_PROGRESS[] = false

Parallel evaluation

The macros @phyperopt and @thyperopt accept the same syntax as @hyperopt and evaluate the loop body in parallel, on the worker processes of the Distributed standard library and on threads, respectively. See their docstrings for details.