API

Hyperparameter optimization by random search, Latin hypercube sampling, Hyperband and BOHB.

The main interface is the macro @hyperopt, which evaluates the body of a for loop for configurations of the hyperparameters and returns a Hyperoptimizer that records the configurations and the results. The documentation is available at https://baggepinnen.github.io/Hyperopt.jl/stable/.

ho = @hyperopt for i = n, [sampler = RandomSampler(),] [ho = existing_ho,] name1 = candidates1, name2 = candidates2, ...
    loop body
end

Evaluate the loop body for configurations of the hyperparameters name1, name2, ... drawn from the candidate vectors candidates1, candidates2, ..., and return the Hyperoptimizer ho that records the configurations in ho.history and the results in ho.results. The optimal configuration is obtained with ho.minimizer, see Hyperoptimizer.

The arguments of the for loop are:

• i = n: The first argument. Without Hyperband, n is the number of iterations, i.e., the number of configurations that are drawn and evaluated, an integer, and i holds the index of the current iteration. With a Hyperband sampler, n is the maximum budget R, a real number of at least 1, and i holds the budget of the current evaluation. Any variable name can be used instead of i.
• sampler = s (optional): The sampler that draws the configurations: RandomSampler (the default), LHSampler, CLHSampler or Hyperband. BOHB is only valid as the inner sampler of Hyperband.
• ho = existing_ho (optional): A Hyperoptimizer whose optimization is continued. The configurations and results of the new run are appended to existing_ho.history and existing_ho.results, which the returned Hyperoptimizer shares with existing_ho. The sampler of existing_ho is continued, including the state of its random number generator and, for BOHB, its observations. A sampler given with sampler = s is not used, and a warning is emitted if it is of another type than the sampler of existing_ho. With a Hyperband sampler, the value n is not used, and a warning is emitted if it differs from the maximum budget R of the existing sampler.
• name = candidates: One argument per hyperparameter. candidates is a vector of the values that the hyperparameter can take, e.g., LinRange(1, 5, 100), exp10.(LinRange(-3, 1, 100)) for logarithmically spaced values, [true, false] or [tanh, sin]. In the loop body, the variable name holds the value of the hyperparameter.

The names sampler and ho are reserved and cannot be used as names of hyperparameters.

The loop body becomes the body of a function whose arguments are the first loop variable and the hyperparameters; variables of the enclosing scope are accessible as in a closure. The keyword argument state of this function is also accessible in the loop body; it is nothing unless the sampler is a Hyperband sampler.

The value of the loop body is the result of the evaluation:

• Without Hyperband, the result is appended to ho.results. The result is typically a real-valued cost, which ho.minimum and ho.minimizer minimize.
• With Hyperband, the loop body returns a tuple (loss, state), where state is any object from which a later evaluation of the same configuration with a larger budget can resume, or only the real-valued loss. The loss is appended to ho.results, and the state is passed to the next evaluation of the configuration as the keyword argument state. See Hyperband.

If an evaluation throws an InterruptException, e.g., after Ctrl-C, the optimization is stopped and ho is returned with the results obtained so far. Other exceptions are rethrown. A progress meter is displayed unless Hyperopt.SHOW_PROGRESS[] is false. The macros @phyperopt and @thyperopt evaluate the loop body in parallel.

Example

cost(a, b, c) = (a - 3)^2 + (b ? 10 : 20) + (log10(c) - 2)^2
ho = @hyperopt for i = 50, sampler = RandomSampler(), a = LinRange(1, 5, 1000), b = [true, false], c = exp10.(LinRange(-1, 3, 1000))
    cost(a, b, c)
end
ho.minimum, ho.minimizer

ho = @phyperopt for i = n, [sampler = RandomSampler(),] [ho = existing_ho,] name1 = candidates1, ...
    loop body
end [pmap]

Same as @hyperopt, but the evaluations of the loop body are distributed over the worker processes with Distributed.pmap. The functions that the loop body calls must be defined on all workers, e.g., with @everywhere.

All configurations are drawn from the sampler on the calling process before the evaluations start. For every evaluation, a worker receives the function that evaluates the loop body, the iteration index and the configuration, and returns only the result. The amount of data transferred per evaluation is thus independent of the number of evaluations. The evaluations are recorded in ho.history and ho.results in the order of the iterations, with the same types as by @hyperopt. With a seeded sampler, e.g., RandomSampler(Xoshiro(0)), @phyperopt therefore records the same configurations as @hyperopt.

The optional second argument is a function with the interface of pmap, which is called as pmap(f, itr), where itr is an iterator of unknown length. It can, e.g., pass keyword arguments to pmap, as in (f, itr) -> pmap(f, itr; on_error = e -> nothing), or use a CachingPool, as in (f, itr) -> pmap(f, CachingPool(workers()), itr), which sends f to each worker only once. The latter reduces the transferred data if the loop body refers to large local variables, e.g., if @phyperopt is called inside a function.

Errors and interruptions:

• If the loop body throws an error, no further evaluations are started, and the error is rethrown as a RemoteException when the evaluations that have started have completed. Other tasks on the workers are not interrupted. The results obtained so far are recorded in the Hyperoptimizer, which remains accessible if it was given with ho = existing_ho.
• If pmap handles errors, e.g., with the keyword argument on_error, the failed evaluations are not recorded, and the values that on_error returns are ignored.
• If the calling task is interrupted, e.g., by Ctrl-C, pmap receives no further configurations, and the Hyperoptimizer with the results obtained so far is returned. The evaluations of the configurations that pmap has received, which include the running evaluations and at most one configuration that pmap has not yet sent to a worker, are not interrupted; they complete in the background, and their results are discarded. Distributed.interrupt() interrupts the running evaluations, together with any other task that runs on the workers.

With a Hyperband sampler, the configurations of each rung of successive halving are evaluated in parallel by pmap, see hyperband. The value that on_error returns for a failed evaluation is then used as its result, e.g., on_error = e -> Inf, since successive halving requires a loss for every configuration of a rung.

Example

using Distributed
addprocs(4)
@everywhere using Hyperopt
@everywhere cost(a, b) = (a - 3)^2 + (b ? 10 : 20)
ho = @phyperopt for i = 100, a = LinRange(1, 5, 1000), b = [true, false]
    cost(a, b)
end

ho = @thyperopt for i = n, [sampler = RandomSampler(),] [ho = existing_ho,] name1 = candidates1, ...
    loop body
end

Same as @hyperopt, but the loop body is evaluated in Threads.nthreads() tasks spawned with Threads.@spawn, see Hyperopt.tmap. The loop body must therefore be safe to evaluate concurrently, and the memory consumption increases with the number of concurrent evaluations.

The configurations are drawn and the evaluations are recorded as described for @phyperopt. Errors and interruptions are also handled as described for @phyperopt, except that an error in the loop body is rethrown as a TaskFailedException. With a Hyperband sampler, the configurations of each rung of successive halving are evaluated in parallel.

Hyperopt.tmap(f, c...; ntasks = Threads.nthreads())

Multithreaded map: return the values of f(x...) for the elements x of zip(c...), in the order of the elements. The evaluations of f are distributed over ntasks tasks spawned with Threads.@spawn. A task takes the next element when it has completed its previous evaluation, the collections are thus consumed in order and may be iterators of unknown length.

If an evaluation of f throws an error, no further elements are taken, and a TaskFailedException is thrown after the evaluations that have started have completed. If the calling task is interrupted while it waits, no further elements are taken, and the InterruptException is rethrown without waiting for the evaluations that have started.

Hyperoptimizer(iterations, sampler = RandomSampler(); name1 = candidates1, name2 = candidates2, ...)
Hyperoptimizer(; iterations, params, candidates, history = [], results = [], sampler = RandomSampler(), objective = nothing)

Specification and record of a hyperparameter optimization. @hyperopt constructs and returns a Hyperoptimizer.

The first constructor creates a Hyperoptimizer for manual iteration. The keyword arguments name the hyperparameters and give their candidate vectors. Each iteration over the Hyperoptimizer draws a configuration from the sampler, appends it to ho.history and yields it as a NamedTuple (; i, name1, name2, ...), where i is the index of the iteration. The result of each configuration is to be appended to ho.results by the caller:

ho = Hyperoptimizer(10, a = LinRange(1, 2, 50), b = [true, false])
for (i, a, b) in ho
    push!(ho.results, cost(a, b))
end

See iterate for the semantics of repeated iteration. The second constructor sets the fields directly.

Fields

• iterations::Int: The number of configurations that one run draws, i.e., one execution of @hyperopt or one iteration over the Hyperoptimizer. With a Hyperband sampler, hyperband sets it to the number of configurations that one run of Hyperband draws from the inner sampler.
• params: The names of the hyperparameters, as symbols.
• candidates: The candidate vectors, one per hyperparameter, in the order of params.
• history: The evaluated configurations. history[k] contains one value per hyperparameter, in the order of params.
• results: The results of the evaluations. results[k] is the result of the evaluation of the configuration history[k].
• sampler: The sampler, e.g., RandomSampler, LHSampler, CLHSampler or Hyperband.
• objective: The function that @hyperopt generates from the loop body, called as objective(i, values...; state), or nothing for a Hyperoptimizer that is iterated manually.

Properties

• ho.minimum: The smallest result.
• ho.minimizer: The configuration of the smallest result, an element of ho.history.
• ho.maximum: The largest result.
• ho.maximizer: The configuration of the largest result, an element of ho.history.

These properties compare the elements of ho.results. A NaN result, e.g., of an evaluation that failed, is treated as Inf by minimum and minimizer and as -Inf by maximum and maximizer; it is therefore selected only if all compared results are NaN, in which case ho.minimum is Inf and ho.maximum is -Inf. If ho.results is empty, ho.minimum and ho.maximum are NaN, and ho.minimizer and ho.maximizer are empty vectors. With a Hyperband sampler, only the results obtained with the largest budget in ho.sampler.budgets are compared, since results obtained with different budgets are not comparable.

minimum(ho) and maximum(ho) are equivalent to ho.minimum and ho.maximum. printmin and printmax print the minimizer and the maximizer, and warn_on_boundary reports hyperparameters whose optimal value is located on the boundary of the sampled values. If Plots.jl is loaded, plot(ho) plots the results against the value of each hyperparameter.

iterate(ho::Hyperoptimizer, state = 1; update_history = true)

Draw the sample of iteration state from the sampler of ho and return it as a NamedTuple with the iteration index i and one entry per parameter, e.g., (i = 1, a = 0.3, b = true).

If update_history = true, the sample is appended to ho.history; the corresponding result is to be appended to ho.results by the caller:

for (i, a, b) in ho
    push!(ho.results, f(a, b))
end

ho.history[k] and ho.results[k] then refer to the same evaluation. Entries at the end of ho.history without a corresponding entry in ho.results, e.g., the samples of an earlier iteration whose results were not recorded, are removed when the iteration starts (state = 1). Iterating over ho again after the results of all samples were recorded appends ho.iterations new samples, which corresponds to continuing the optimization. For LHSampler and CLHSampler, such an iteration evaluates a new plan.

minimum(ho::Hyperoptimizer)

Return the smallest result in ho.results, equivalent to ho.minimum. NaN results are treated as Inf. If ho contains no results, NaN is returned. With a Hyperband sampler, only the results obtained with the largest budget are compared.

maximum(ho::Hyperoptimizer)

Return the largest result in ho.results, equivalent to ho.maximum. NaN results are treated as -Inf. If ho contains no results, NaN is returned. With a Hyperband sampler, only the results obtained with the largest budget are compared.

printmin([io = stdout,] ho)

Print the name and the value of each hyperparameter of the configuration ho.minimizer, one hyperparameter per line. With a Hyperband sampler, the configuration is the minimizer among the results obtained with the largest budget, see Hyperoptimizer.

printmax([io = stdout,] ho)

Print the name and the value of each hyperparameter of the configuration ho.maximizer, one hyperparameter per line. With a Hyperband sampler, the configuration is the maximizer among the results obtained with the largest budget, see Hyperoptimizer.

warn_on_boundary(ho, sense = :min)

Emit a warning for each hyperparameter whose value in the optimal configuration is the smallest or the largest value of this hyperparameter in ho.history, and return the names of these hyperparameters as a vector of symbols. The optimal configuration is ho.minimizer if sense = :min, and ho.maximizer otherwise.

Example: if the hyperparameter a has the candidates 1:10 and the optimum is obtained with a = 1, the optimum may be located outside of the candidate range, and the range is to be extended. The warning is also emitted if the smallest sampled value of a is larger than 1 and the optimum is obtained with this value.

Only hyperparameters whose candidates are real numbers with more than three distinct values are considered, since categorical candidates have no order.

RandomSampler{T<:AbstractRNG} <: Sampler

Sample a value for each parameter uniformly at random from the candidate vectors. Log-uniform sampling available by providing a log-spaced candidate vector.

Optionally, pass an AbstractRNG to initialize. By default, a Xoshiro generator is seeded with a random number drawn from the global RNG, so that Random.seed! makes the samples reproducible.

RandomSampler is the default sampler of @hyperopt and the default inner sampler of Hyperband. The values are drawn independently and with replacement, a configuration may therefore be drawn more than once. The candidate vectors may have different lengths and may contain values of any type, e.g., functions.

LHSampler(; rng = Xoshiro(rand(UInt)), steps = nothing)

Sample from an optimized Latin hypercube plan. The plan contains one point per iteration, and every element of each candidate vector occurs in exactly one point. All candidate vectors must therefore have the same length as the number of iterations. CLHSampler admits categorical hyperparameters with fewer candidates.

The plan is obtained from a random Latin hypercube by steps proposed exchanges of the entries of two points in one dimension. An exchange is accepted if it reduces the Audze-Eglais potential energy of the plan, i.e., the sum of the inverse squared distances between all pairs of points, in the coordinates given by the candidate indices divided by the number of iterations. The computational cost of a step is proportional to n*d for n iterations and d hyperparameters. The default number of steps, clamp(10^8 ÷ (n*d), n*d, 100*n*d), thus limits the cost to approximately 10^8 evaluations of coordinate differences if n*d ≤ 10^4. With steps = 0, the plan is a random Latin hypercube.

The plan is drawn from rng. By default, a Xoshiro generator is seeded with a random number drawn from the global RNG, so that Random.seed! makes the plans reproducible.

A new plan is generated for every run of @hyperopt, including a run that resumes an existing Hyperoptimizer with ho = ..., and for every iteration over a Hyperoptimizer that begins after results were recorded. The optimization of a new plan reduces the number of its points that coincide with configurations in ho.history with priority over the potential energy: an exchange that reduces this number is accepted, and an exchange that increases it is rejected. Configurations are thus not evaluated again if the number of steps suffices. With a single hyperparameter, however, every plan consists of all candidates.

As the inner sampler of Hyperband, the plan contains one point per element of the candidate vectors instead of one point per iteration, see Hyperband.

Example

ho = @hyperopt for i = 100, sampler = LHSampler(), a = LinRange(1, 5, 100), b = repeat([true, false], 50)
    (a - 3)^2 + b
end

CLHSampler(; dims, rng = Xoshiro(rand(UInt)), steps = nothing)

Sample from an optimized Latin hypercube plan with continuous and categorical dimensions. dims contains one dimension type per hyperparameter, in the order of the candidate vectors, e.g., dims = [Continuous(), Categorical(2), Continuous()]:

• Continuous(): the candidate vector must have the same length as the number of iterations, and every candidate occurs in exactly one point of the plan.
• Categorical(k) or UnorderedCategorical(k): a categorical hyperparameter with k candidates, i.e., the candidate vector must have k elements. In a plan with n points, every candidate occurs in n ÷ k or n ÷ k + 1 points. Both types are treated as unordered: a categorical dimension contributes 1 to the squared distance between two points with different candidates and 0 otherwise. The optimized plan thus distributes the points that share a categorical candidate over the continuous dimensions.

The keyword arguments rng and steps, and the generation of new plans are described in LHSampler.

Example

ho = @hyperopt for i = 100, sampler = CLHSampler(dims = [Continuous(), Categorical(2)]), a = LinRange(1, 5, 100), b = [true, false]
    (a - 3)^2 + b
end

Hyperband(; R = nothing, η = 3, inner = RandomSampler())
Hyperband(R)

Sampler that implements Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization. Hyperband runs several brackets of successive halving. In each bracket, configurations drawn from the sampler inner are evaluated with a budget, i.e., an amount of resources. The fraction 1/η of the configurations with the smallest losses is evaluated again with η times the budget, until the maximum budget R is reached.

With @hyperopt, the first loop argument names the variable that holds the budget of an evaluation, and its value is the maximum budget R, which then does not have to be given to the sampler:

ho = @hyperopt for resources = 81, sampler = Hyperband(η = 3), a = LinRange(1, 5, 100), b = [true, false]
    model = state === nothing ? initial_model(a, b) : state
    train!(model, resources)
    loss(model), model
end

In every evaluation, the hyperparameter variables hold the configuration. The keyword state is nothing in the first evaluation of a configuration, and otherwise the state returned by the previous evaluation of the same configuration. The loop body returns (loss, state), where the state is any object from which a later evaluation can resume, e.g., the weights of a model and the state of its optimizer. If the loop body returns only the loss, state is nothing in every evaluation. If R is also given to the sampler and differs from the value of the first loop argument, the value of the sampler is used and a deprecation warning is emitted.

The budgets of the evaluations are R/η^k for k = 0, 1, ... and are not smaller than 1; R is therefore a real number of at least 1, e.g., R = 4.5 for a time limit of 4.5 seconds with a smallest budget of 1 second. Successive halving requires R ≥ η; with R < η, a run evaluates a single configuration with the budget R, and a warning is emitted.

The evaluations are recorded in ho.history (the configurations), ho.results (the losses) and ho.sampler.budgets (the budgets). ho.minimum, ho.minimizer, ho.maximum and ho.maximizer compare only the results obtained with the largest budget. ho.sampler.minimum is nothing before the first evaluation, and otherwise the named tuple (; loss, budget, state, configuration) of the evaluation that ho.minimum refers to. plot(ho) shows the results of each budget as a separate series. ho.iterations is the number of configurations that one run of Hyperband draws from inner. A Hyperband sampler records the evaluations of one Hyperoptimizer and cannot be used for several.

A run that continues an existing Hyperoptimizer with ho = ... runs all brackets again and continues the sampler of the existing Hyperoptimizer, including R, η, the recorded budgets and minimum. The value of the first loop argument of the continued run is not used, and a warning is emitted if it differs from R.

With inner = LHSampler() or CLHSampler(...), the configurations of all brackets of a run are distinct points of a Latin hypercube plan, which contains one point per element of the candidate vectors. The candidate vectors must therefore contain at least ho.iterations elements.

See also hyperband for an interface without the macro.

BOHB(; dims, ρ = 1/3, q = 0.15, N_s = 64, N_min = nothing, bw_factor = 3, min_bandwidth = 1e-3, random_sampler = RandomSampler())

Inner sampler of Hyperband that implements the model-based sampling of BOHB: Robust and Efficient Hyperparameter Optimization at Scale. A fraction ρ of the configurations is sampled at random. The remaining configurations are proposed by maximizing the ratio l(x)/g(x) of two kernel density estimates, fitted to the best and the worst configurations evaluated with the largest budget for which sufficiently many observations are available. The observations are the evaluated configurations, i.e., the values of the hyperparameters, and their losses; the states returned by the objective are not used. The observations are recorded by Hyperband; BOHB is therefore only valid as the inner sampler of a Hyperband sampler, and @hyperopt throws an ArgumentError for sampler = BOHB(...). A run that continues an existing Hyperoptimizer with ho = ... continues the sampler, including its observations.

dims contains one dimension type per hyperparameter, in the order of the candidate vectors:

• Continuous(): an ordered, real-valued hyperparameter. The densities are estimated in the coordinate given by the index into the sorted candidate vector, scaled to [0, 1]. A log-spaced candidate vector thus corresponds to a logarithmic coordinate. Ordered discrete hyperparameters, such as integers, may also be declared Continuous.
• Categorical(n) or UnorderedCategorical(n): a categorical hyperparameter with n candidates. Both types are treated as unordered and use an Aitchison-Aitken kernel.

Every proposal consists of elements of the candidate vectors.

The remaining keyword arguments refer to the symbols in the paper:

• ρ: Fraction of random samples
• q: Fraction of the best observations used to fit the density of good configurations l(x)
• N_s: Number of samples drawn from l(x), of which the one with the largest ratio l(x)/g(x) is proposed
• N_min: Minimum number of observations used to fit each density, the default is the number of hyperparameters plus one
• bw_factor: Factor by which the bandwidths of the continuous dimensions are multiplied when sampling from l(x)
• min_bandwidth: Lower bound of the bandwidths. This parameter does not occur in the paper but is used in the reference implementation HpBandSter.
• random_sampler: RandomSampler used for the random configurations. Its RNG is used for all random decisions of BOHB, a run is thus reproducible with, e.g., random_sampler = RandomSampler(MersenneTwister(0)).

Example

ho = @hyperopt for resources = 27, sampler = Hyperband(η = 3, inner = BOHB(dims = [Continuous(), Categorical(2)])), a = LinRange(1, 5, 100), b = [true, false]
    (a - 3)^2 + b + 1/resources
end

Continuous()

An ordered hyperparameter, e.g., a real-valued or an integer-valued hyperparameter. CLHSampler requires the candidate vector of a Continuous hyperparameter to have the same length as the number of iterations. BOHB requires real-valued candidates and estimates the densities in the coordinate given by the index into the sorted candidate vector.

Categorical(levels)

A categorical hyperparameter with levels candidates, i.e., its candidate vector has levels elements. CLHSampler and BOHB treat the candidates as unordered, Categorical(levels) is therefore equivalent to UnorderedCategorical(levels).

UnorderedCategorical(levels)

A categorical hyperparameter with levels unordered candidates, e.g., UnorderedCategorical(3) for fun = [tanh, sin, cos]. Equivalent to Categorical(levels).

DimensionType

Abstract supertype of the types Continuous, Categorical and UnorderedCategorical, which describe the hyperparameters for CLHSampler and BOHB.

hyperband(ho::Hyperoptimizer{Hyperband}; threads = false, mapfun = threads ? Hyperopt.tmap : map)

Run Hyperband with the sampler ho.sampler and the objective ho.objective, which is called as ho.objective(budget, configuration...; state) and returns (loss, state) or only the loss, see Hyperband. The evaluations are appended to ho.history, ho.results and ho.sampler.budgets. mapfun evaluates the configurations of a rung, see Hyperopt.successive_halving.

Return ho.sampler.minimum.

hyperband(f, candidates; R, η = 3, inner = RandomSampler(), threads = false, mapfun = threads ? Hyperopt.tmap : map)

Run Hyperband without the macro @hyperopt and return the Hyperoptimizer. candidates is a vector or a NamedTuple of candidate vectors, one per hyperparameter. The objective is called as f(budget, pars), where pars is the vector of hyperparameter values in the first evaluation of a configuration, and the state returned by the previous evaluation of the same configuration otherwise. f returns (loss, state) or only the loss; in the latter case, pars is the vector of hyperparameter values in every evaluation. mapfun evaluates the configurations of a rung, e.g., Distributed.pmap, see Hyperopt.successive_halving. See Hyperband for the remaining arguments and the recorded results.

successive_halving(ho, n, r = 1, s = round(Int, log(ho.sampler.η, n)); threads = false, mapfun = threads ? Hyperopt.tmap : map)
successive_halving(ho, n, budgets::AbstractVector; threads = false, mapfun = threads ? Hyperopt.tmap : map)

Run successive halving with the Hyperband sampler ho.sampler. n configurations are drawn from the inner sampler and evaluated with the budget budgets[1]. The fraction 1/η of the configurations with the smallest losses is evaluated again with the budget budgets[2], and so on. The first method uses the budgets r * η^i for i = 0:s.

mapfun(f, configurations, states) evaluates the configurations of a rung and returns the values of f, e.g., map, Hyperopt.tmap or Distributed.pmap. The function f captures only ho.objective and the budget.

Return ho.sampler.minimum.

hyperoptim(f, candidates, algorithm = Optim.NelderMead(), opts = Optim.Options(); R = 30, η = 3, inner = RandomSampler(), threads = true, verbose = false, Rmin = 0.1, kwargs...)

Multi-start optimization with Optim and Hyperband. The interface is similar to that of Optim.optimize, but the initial guess is replaced by candidate vectors, one per element of the initial guess, from which hyperband draws initial guesses. The budget of an evaluation is the time limit in seconds of a call to Optim.optimize. An evaluation with a larger budget continues the optimization from the minimizer found by the previous evaluation of the same initial guess.

Returns the Hyperoptimizer ho. ho.minimum is the smallest value of f found with the largest budget, ho.minimizer is the initial guess from which this value was found, and ho.sampler.minimum.state is the corresponding minimizer of f.

This function requires Optim.jl to be loaded, i.e., using Optim.

Arguments:

• f: A function from Rⁿ to R, i.e., it takes a vector of parameters.
• candidates: A vector or a NamedTuple of candidate vectors, one per element of the vector of parameters.
• algorithm: One of the Optim algorithms.
• opts: Optim.Options. The field time_limit is replaced by the budget of each call to Optim.optimize.
• R: The maximum budget, i.e., the largest time limit in seconds.
• η: The ratio of the budgets of successive rounds of successive halving, see Hyperband.
• inner: The inner sampler
• threads: Use threading to evaluate the cost function
• verbose: Print the budget and the minimum after each call to Optim.optimize.
• Rmin: The smallest time limit in seconds; smaller budgets are replaced by Rmin.
• kwargs: Are sent to Optim.optimize.

Hyperopt.SHOW_PROGRESS[] = false

Disable the progress meters that @hyperopt, hyperband and hyperoptim display. The progress meters are enabled by default.