Why You Can't Just Start Bayesian Optimization With Any Old Sample Size
You've got a black-box function. Maybe it's a machine learning model's hyperparameters, maybe it's a physical simulation, maybe it's some expensive experiment you can only run a handful of times. That's why you've heard that Bayesian optimization is the elegant way to find the sweet spot without burning through all your budget. So you jump in, grab whatever data you have lying around, and fire up an acquisition function.
Then it falls apart. Still, the surrogate model is garbage. The recommendations are nonsense. You're chasing noise instead of signal.
Here's the thing most guides won't tell you clearly: the minimum sample size for Bayesian optimization isn't a single magic number. It depends on what you're optimizing, how noisy your observations are, and what surrogate model you're using. But there are real principles that govern how little data you can get away with before the whole framework starts making unreliable decisions.
Let's dig into what's actually going on.
What Is Bayesian Optimization, Really
Bayesian optimization is a sequential design strategy for finding the optimum of expensive, unknown functions. Instead of brute-forcing through combinations or relying on random search, it builds a probabilistic model of the objective function — usually a Gaussian process, though tree-structured Parzen estimators and other surrogates are common too — and uses that model to decide where to sample next.
The official docs gloss over this. That's a mistake.
The core loop is straightforward. Practically speaking, you evaluate a few points, fit a surrogate model to those observations, use an acquisition function (like expected improvement or upper confidence bound) to pick the next point to evaluate, and repeat. The idea is that each new evaluation teaches the model something, and the model gets better at directing you toward the optimum.
But here's the catch: the surrogate model needs enough data to form a useful belief about the landscape. Start with too few points, and the model has no idea what's going on. Start with too many, and you've burned through your budget before the optimization even gets interesting.
Why Minimum Sample Size Matters
The minimum sample size is the threshold below which Bayesian optimization stops being Bayesian optimization in any meaningful sense. It's the point where the surrogate model has just enough information to distinguish signal from noise, to identify promising regions, and to make acquisition decisions that are actually better than random guessing.
If you skip below that threshold, you're essentially running a guided random search that happens to look fancy. The acquisition function will point you toward regions that look promising only because the model is overconfident or completely uncertain — neither of which is useful.
Why does this matter in practice? Because most people working with Bayesian optimization are dealing with expensive evaluations. A single run of a hyperparameter tuning experiment might take hours or days. That's why a physical experiment might cost real money or consume scarce materials. Wasting even a handful of evaluations on a poorly initialized optimization is a real cost Took long enough..
How the Minimum Sample Size Emerges
The Role of the Surrogate Model
The surrogate model is the engine of Bayesian optimization, and different models have different appetites for data. Still, a Gaussian process, for instance, places a prior over functions and updates that prior with each observation. Which means with one or two observations, the posterior is almost entirely determined by the prior. The model hasn't learned anything about the actual shape of the function — it's just regressing toward the mean with wide uncertainty bands everywhere The details matter here..
Some disagree here. Fair enough.
A tree-structured Parzen estimator works differently. It models the probability of good and bad observations separately using kernel density estimates. It can sometimes produce reasonable suggestions with fewer points because it's not trying to learn a smooth function — it's just tracking where good outcomes have clustered. But even TPE needs a minimum number of observations to separate the good from the bad distributions in a meaningful way.
In general, the more complex the surrogate model, the more data it needs to stop acting like it's just repeating the prior.
Noise and Variability
If your observations are noisy — and most real-world evaluations are — you need more samples to get a clear picture. Which means a single noisy observation can mislead the surrogate model into thinking a region is promising (or terrible) when it's really just an outlier. More samples let the model average out the noise and converge on the true underlying function Still holds up..
The higher the noise level, the more observations you need before the model can trust its own predictions. This is one reason why there's no universal minimum sample size: a low-noise problem might work fine with 3 or 4 initial points, while a high-noise problem might need 10 or more just to get off the ground No workaround needed..
The Dimensionality of the Search Space
Here's a factor that catches a lot of people off guard. As the number of parameters you're optimizing grows, the search space expands exponentially. Still, a two-dimensional problem is manageable with a handful of points. In real terms, a ten-dimensional problem is a completely different beast. The surrogate model needs more data to cover a high-dimensional space adequately, and the risk of the model fitting to spurious patterns increases with dimensionality Practical, not theoretical..
This is related to the curse of dimensionality, and it directly impacts the minimum sample size. Think about it: in practice, Bayesian optimization tends to work best on problems with a modest number of dimensions — usually somewhere between 2 and 20, depending on the structure of the problem. Beyond that, you may need specialized approaches or dimensionality reduction before the optimization becomes reliable.
Acquisition Function Behavior at Low Sample Sizes
The acquisition function is what translates the surrogate model's beliefs into a decision about where to sample next. So at low sample sizes, the acquisition function can behave erratically. Expected improvement, for example, might become dominated by the uncertainty term rather than the exploitation term, leading the optimizer to explore randomly rather than focusing on genuinely promising regions.
This isn't necessarily a bug — exploration is valuable early on — but it means that the first few evaluations aren't doing much useful optimization. So naturally, they're just seeding the model. The minimum sample size is partly the point at which the acquisition function starts making consistently better-than-random decisions.
Common Mistakes People Make With Initial Sample Sizes
Using a Single Initial Point
This is a surprisingly common mistake, especially among newcomers who treat Bayesian optimization as a black box and don't think about initialization. A single point gives the surrogate model no information about gradients, curvature, or the shape of the landscape. The model will essentially recommend the same region over and over, or it will explore based entirely on the prior, which is rarely useful.
Copying Default Settings Without Thinking
Many Bayesian optimization libraries come with default initial sample sizes — often 3, 5, or 10 points. The problem is that the right number depends on your specific problem. Here's the thing — people grab the default and move on. A default of 5 might be fine for a low-dimensional, low-noise problem and woefully insufficient for a high-dimensional, high-noise one.
Ignoring the Warm-Up Phase
Some practitioners treat the first few evaluations as part of the optimization, when really they're a warm-up or initialization phase. Also, the optimization hasn't really started until the surrogate model has enough data to make informed decisions. Recognizing this distinction helps you set appropriate expectations and avoid prematurely judging the optimization's performance.
Scaling the Problem Without Adjusting the Sample Size
If you add a new parameter to your optimization, you can't just keep the same initial sample size and expect it to work. The search
When additional parameters are introduced, the effective dimensionality of the search space grows, and the surrogate model must now capture a richer set of interactions. In real terms, a fixed handful of points that was sufficient for a two‑ or three‑dimensional problem will almost certainly be inadequate once the dimension climbs into double digits. In higher dimensions, the volume of the space expands exponentially, so a random or space‑filling set of points becomes increasingly unlikely to cover the regions that matter. So naturally, the acquisition function may spend many iterations hopping from one unrelated corner of the space to another, wasting evaluations before it zeroes in on promising areas.
Not the most exciting part, but easily the most useful.
Noise compounds the problem. Consider this: more initial samples provide a better statistical foundation, reducing the variance of the surrogate’s predictions and allowing the acquisition function to separate genuine improvement potential from mere fluctuation. In a noisy objective, each observation is a perturbed version of the true function value, and a small initial dataset cannot reliably estimate the underlying trend. When the objective is especially noisy, a larger warm‑up set—often 10 % to 20 % of the total budget—helps the model converge faster The details matter here..
Parallel or batch acquisition strategies further influence the choice of initial points. If you plan to evaluate several candidates simultaneously, you need enough starting points to populate the first batch without forcing the algorithm to pick points that are overly correlated. Also, a common practice is to draw an initial Latin hypercube sample (LHS) of size proportional to the dimensionality—typically 5 × D to 10 × D points, where D is the number of active variables. This range offers enough diversity to seed the batch while still being computationally tractable That alone is useful..
The choice of surrogate also matters. That said, gaussian processes excel in low‑ to moderate‑dimensional settings with moderate noise, but they scale poorly beyond roughly 15 dimensions. That said, in higher‑dimensional or highly structured problems, tree‑based models (e. That said, g. , random forests, Gradient Boosted Trees) or Bayesian neural networks may be more solid, and they often benefit from a larger initial dataset to learn the underlying patterns before the acquisition function can make informed proposals Worth keeping that in mind..
Practitioners should therefore view the initial sample size as a dynamic parameter rather than a static default. A pragmatic workflow might look like this:
- Assess problem complexity – estimate the number of effective dimensions, the expected level of noise, and whether parallel evaluations are feasible.
- Select a design – use a space‑filling method (LHS, Sobol sequence, or maximin Latin hypercube) and set the initial count based on the factors above.
- Run a brief warm‑up – treat the first N evaluations as initialization; monitor surrogate variance and acquisition variance to gauge whether the model is stabilizing.
- Adapt as needed – if the surrogate’s predictive variance remains high or the acquisition function shows little directional progress, consider adding a few more points before entering the main optimization loop.
By aligning the initial sample budget with the intrinsic difficulty of the problem, you give the Bayesian optimization process a solid statistical footing. This early investment pays off in fewer total evaluations, more reliable convergence, and a higher chance of locating the true optimum without unnecessary wandering.
Conclusion
Choosing an appropriate initial sample size is not a one‑size‑fits‑all decision; it is a balancing act between exploration and exploitation, dimensionality, noise, and computational budget. Too few points leave the surrogate under‑informed, causing the acquisition function to behave erratically and wasting evaluation effort. On top of that, too many points consume resources that could be spent on genuine search progress. A thoughtful, problem‑specific approach—often starting with a modest, space‑filling set and then adjusting based on observed variance and acquisition behavior—provides the best foundation for efficient Bayesian optimization. When practitioners respect these principles, the early warm‑up phase becomes a catalyst rather than a bottleneck, leading to faster, more reliable discovery of the optimum.
