Function interface

The macro @hyperopt generates a function from the loop body and evaluates it. This page describes two interfaces without the macro: the manual iteration over a Hyperoptimizer, and the function hyperband.

Manual iteration

A Hyperoptimizer is constructed with the number of iterations, optionally a sampler, and one keyword argument per hyperparameter, which gives its candidate vector. Iterating over the Hyperoptimizer yields one configuration per iteration, as a NamedTuple that contains the index of the iteration i and the values of the hyperparameters:

ho = Hyperoptimizer(10, a = LinRange(1, 2, 50), b = [true, false], c = randn(100))
for (i, a, b, c) in ho
    println(i, "\t", a, "\t", b, "\t", c)
end

1	1.7959183673469388	true	-1.3819748518823003
2	1.0408163265306123	false	-1.2601642953877346
3	1.5306122448979593	true	-0.29415075749027164
4	1.5714285714285714	false	0.27261968870626063
5	2.0	false	-1.0484721702319963
6	1.4489795918367347	true	-1.2601642953877346
7	1.306122448979592	false	-0.9639451905647899
8	1.2653061224489797	false	0.5583656057835097
9	1.0612244897959184	true	-0.5424504046441639
10	1.6122448979591837	false	-0.6705231877296826

Each iteration appends the configuration to ho.history. The results are not recorded by the iteration; to record them, the result of each configuration is appended to ho.results:

cost(a, b, c) = (a - 1.5)^2 + b + c^2

ho = Hyperoptimizer(10, a = LinRange(1, 2, 50), b = [true, false], c = randn(100))
for (i, a, b, c) in ho
    push!(ho.results, cost(a, b, c))
end
ho.minimum, ho.minimizer

(0.12158615644984314, Real[1.306122448979592, false, -0.28982348362438165])

ho.history[k] and ho.results[k] then refer to the same evaluation. When an iteration begins, the configurations at the end of ho.history without a corresponding result are removed, so that repeated iterations do not accumulate configurations that were never evaluated. An iteration that begins after the results of all configurations were recorded appends ho.iterations new configurations, which continues the optimization:

for (i, a, b, c) in ho
    push!(ho.results, cost(a, b, c))
end
length(ho.history), length(ho.results)

(20, 20)

The configurations can also be collected first and evaluated later, e.g., by an external program:

ho = Hyperoptimizer(4, LHSampler(), a = LinRange(1, 2, 4), b = [true, false, true, false])
configurations = collect(ho)

4-element Vector{@NamedTuple{i::Int64, a::Float64, b::Bool}}:
 (i = 1, a = 1.0, b = 0)
 (i = 2, a = 1.6666666666666665, b = 1)
 (i = 3, a = 1.3333333333333335, b = 0)
 (i = 4, a = 2.0, b = 1)

append!(ho.results, [cost(x.a, x.b, 0.0) for x in configurations])
printmin(ho)

a = 1.3333333333333335
b = false

For LHSampler and CLHSampler, an iteration that begins after results were recorded evaluates a new plan.

Hyperband without the macro

hyperband runs Hyperband with an objective function f(budget, x) and returns the Hyperoptimizer. x is the vector of hyperparameter values in the first evaluation of a configuration, and the state returned by the previous evaluation of the same configuration otherwise. The objective returns (loss, state), or only the loss, in which case x is the vector of hyperparameter values in every evaluation. The candidate vectors are given as a vector or as a NamedTuple; a NamedTuple names the hyperparameters, e.g., in plots.

In the following example, the state is a NamedTuple that contains the hyperparameters, the weights of the model and the number of epochs trained so far:

const scales = [1.0, 10.0, 100.0]
loss(w) = sum(scales .* (w .- 1) .^ 2)
gradient(w) = 2 .* scales .* (w .- 1)

function objective(epochs, x)
    s = x isa NamedTuple ? x : (lr = x[1], β = x[2], w = zeros(3), v = zeros(3), trained = 0)
    w, v = s.w, s.v
    for _ in 1:(round(Int, epochs) - s.trained)
        v = s.β * v - s.lr * gradient(w)
        w = w + v
    end
    loss(w), (; s.lr, s.β, w, v, trained = round(Int, epochs))
end

candidates = (lr = exp10.(LinRange(-4, -1.5, 100)), β = LinRange(0, 0.95, 100))
ho = hyperband(objective, candidates; R = 81, η = 3)
ho.minimum, ho.minimizer

(0.00016440226720563258, [0.005857020818056668, 0.8444444444444443])

As with the macro, ho.sampler.minimum contains the state of the best evaluation with the largest budget:

ho.sampler.minimum.state.w

3-element Vector{Float64}:
 0.9992091489592047
 1.001059129452811
 1.001235148856854

The keyword argument inner selects the inner sampler, e.g., inner = LHSampler() or inner = BOHB(dims = [Continuous(), Continuous()]). See hyperoptim and Optim for an application of hyperband to multi-start optimization.