ML optimization & parameter space analysis • 2026 edition
\( \text{Total Combinations} = \prod_{i=1}^{n} |P_i| \)
Where:
For different search strategies:
Example: For a model with 3 hyperparameters: Learning Rate (5 values), Batch Size (3 values), Hidden Units (4 values):
\( \text{Total Combinations} = 5 \times 3 \times 4 = 60 \)
Grid search would evaluate 60 combinations, while random search might evaluate only 10-20 for efficiency.
| Parameter | Type | Cardinality |
|---|---|---|
| Learning Rate | Discrete | 5 |
| Batch Size | Discrete | 4 |
| Hidden Units | Discrete | 4 |
Hyperparameter optimization is the process of finding the optimal hyperparameters for a machine learning model. Unlike model parameters learned during training, hyperparameters are set before training begins and control the model's behavior, architecture, and learning process. Proper optimization can significantly improve model performance.
The total search space size calculation:
Where \(P_i\) represents the possible values for hyperparameter \(i\).
Each strategy has different trade-offs:
Process of finding optimal model hyperparameters.
\( \text{Total Combinations} = \prod_{i=1}^{n} |P_i| \)
Where Pi = possible values for hyperparameter i.
Systematic approach to hyperparameter optimization and model development.
Which hyperparameter optimization strategy guarantees finding the global optimum within the defined search space?
The answer is B) Grid Search. Grid search systematically evaluates all possible combinations within the defined parameter space, guaranteeing that it will examine every point in the grid. This exhaustive approach ensures that if the global optimum exists within the defined search space, grid search will find it. Other methods like random search, Bayesian optimization, and evolutionary algorithms use intelligent sampling but do not guarantee examining every possible combination.
Students must understand the fundamental difference between exhaustive and sampling-based optimization methods. Grid search is deterministic and exhaustive, while other methods are probabilistic and may miss the global optimum. However, grid search becomes computationally prohibitive for large search spaces due to the curse of dimensionality.
Global Optimum: Best possible combination within search space
Exhaustive Search: Examining every possible combination
Curse of Dimensionality: Exponential growth of search space with parameters
• Grid search is exhaustive and guaranteed to find global optimum
• Other methods are more efficient but not guaranteed
• Computational cost increases exponentially with parameters
• Use grid search for small search spaces
• Consider Bayesian optimization for large spaces
• Combine coarse and fine-grained searches
• Thinking all optimization methods guarantee global optimum
• Using grid search for large search spaces
• Not considering computational constraints
Calculate the total number of combinations for a model with 4 hyperparameters: Learning Rate (3 values), Batch Size (4 values), Hidden Units (5 values), Dropout (2 values). Show your work.
Using the formula: \( \text{Total Combinations} = \prod_{i=1}^{n} |P_i| \)
Given parameters:
Step 1: Calculate total combinations
\( \text{Total} = 3 \times 4 \times 5 \times 2 = 120 \)
Step 2: Interpret results
Grid search would evaluate 120 different combinations
With 4 hyperparameters and 120 total combinations, this represents a moderately sized search space that could be computationally intensive for expensive model evaluations.
This calculation demonstrates the exponential growth of search space with the number of hyperparameters. Adding just one more parameter with 3 values would increase the total to 360 combinations (3×4×5×2×3). This exponential growth is why intelligent optimization methods become crucial for larger search spaces.
Search Space Cardinality: Total number of possible combinations
Curse of Dimensionality: Exponential growth with parameters
Cartesian Product: Mathematical operation for total combinations
• Total combinations = product of all parameter cardinalities
• Search space grows exponentially with parameters
• Consider computational feasibility
• Calculate search space before optimization
• Use log scales for learning rates
• Consider parameter interactions
• Forgetting to multiply all parameter values
• Not considering computational implications
• Confusing parameter types with cardinality
A data scientist is optimizing a neural network with 5 hyperparameters: Learning Rate (4 values), Batch Size (3 values), Hidden Units (4 values), Layers (3 values), and Dropout (3 values). Each model evaluation takes 10 minutes. Calculate the total time required for grid search and suggest a more efficient approach if the team has only 24 hours available.
Grid search calculation:
Time calculation:
Recommendation:
With random search, the team can evaluate 144 combinations in 24 hours instead of 432, achieving 33% of the grid search coverage with significantly better computational efficiency.
This example demonstrates the practical constraints of hyperparameter optimization. With 432 combinations requiring 72 hours, grid search becomes infeasible within the 24-hour constraint. This highlights why intelligent optimization methods like random search or Bayesian optimization are essential for large search spaces, trading complete coverage for computational feasibility.
Computational Feasibility: Practical constraints on computation time
Resource Planning: Matching optimization strategy to available time
Efficiency Trade-off: Coverage vs computational cost
• Always consider computational constraints
• Match strategy to available resources
• Calculate total time before starting optimization
• Use coarse-to-fine search strategies
• Consider parallel execution options
• Not considering computational time constraints
• Using grid search for large search spaces
• Failing to plan resource allocation
A machine learning engineer needs to optimize a model with a search space of 10,000 combinations. Each evaluation takes 5 minutes. The team has 40 hours of compute budget. Which optimization strategy should they use? Calculate the number of evaluations possible with each strategy and justify your recommendation.
Available resources:
Strategy analysis:
Recommendation:
With Bayesian optimization, the team can intelligently explore the search space by focusing on promising regions based on previous evaluations, making the most of their limited computational budget.
This problem illustrates the importance of matching optimization strategy to computational constraints. With only 4.8% coverage of the search space, intelligent methods like Bayesian optimization become crucial. These methods learn from each evaluation to guide subsequent trials toward more promising regions, making the limited evaluations more effective than random sampling.
Intelligent Sampling: Methods that learn from previous evaluations
Computational Budget: Limited time/resources for optimization
Exploration vs Exploitation: Balance between searching and refining
• Match strategy to available computational resources
• Large search spaces require intelligent optimization
• Bayesian optimization learns from evaluations
• Always calculate evaluation budget first
• Consider parallel evaluation capabilities
• Use surrogate models for expensive evaluations
• Attempting grid search on large search spaces
• Not considering computational budget constraints
• Using random search when Bayesian is more appropriate
Which of the following hyperparameter types is best suited for logarithmic scaling during optimization?
The answer is B) Learning rate (float: 0.0001-0.1). Learning rates are best optimized on a logarithmic scale because they span several orders of magnitude (e.g., 0.0001, 0.001, 0.01, 0.1). On a linear scale, the difference between 0.0001 and 0.001 seems small, but it's actually a 10x change that significantly impacts training. Logarithmic scaling treats multiplicative changes equally, making optimization more effective for parameters that affect model behavior multiplicatively.
Students must understand when logarithmic scaling is appropriate. Parameters that span multiple orders of magnitude (like learning rates, regularization coefficients, or network widths) benefit from log scaling because it treats percentage changes equally. This is crucial for optimization algorithms that assume uniform importance across the parameter range.
Logarithmic Scaling: Parameter values spaced by multiplication factor
Orders of Magnitude: Powers of 10 in value ranges
Multiplicative Effect: Parameter changes affect model multiplicatively
• Use log scale for parameters spanning multiple orders of magnitude
• Learning rates are typically log-scaled
• Regularization parameters often benefit from log scaling
• Learning rates: log scale [1e-6, 1e-1]
• Regularization: log scale [1e-4, 1e2]
• Count parameters: linear scale [1, 100]
• Using linear scale for learning rates
• Not considering the multiplicative nature of some parameters
• Applying log scaling to count parameters unnecessarily
Q: What's the difference between grid search, random search, and Bayesian optimization?
A: Here's a comparison of the main hyperparameter optimization strategies:
Grid Search:
Random Search:
Bayesian Optimization:
For a search space with 100,000 combinations, grid search would evaluate all 100,000, random search might evaluate 100-1000, and Bayesian optimization would typically find good results in 100-200 evaluations.
Q: How do I decide which hyperparameters to optimize and how to set their ranges?
A: Here's a systematic approach to hyperparameter selection and range setting:
Hyperparameter Prioritization:
Range Setting Strategies:
Guidelines for Ranges:
Example for Neural Networks:
Always validate your choices with a small pilot study before committing to full optimization.