I'm trying to figure out how the Keras implementation of Hyperband calculates the number of configurations to test per round within a bracket. When I run Keras Hyperband with max_epochs=10 and factor=2 and look at the output, this is what I get:
Where s = bracket number, i = round number, n = number of configurations tested in round i, ri = number of epochs each of those configurations get in round i.
However, when we look at the source code (https://github.com/keras-team/keras-tuner/blob/0dd114dd5353c941e8e14e68356a3fb124e3a0d1/keras_tuner/tuners/hyperband.py#L23), it seems to me that we should instead have:
In particular, I'm looking at the functions,
def _get_size(self, bracket_num, round_num):
# Set up so that each bracket takes approx. the same amount of
# resources.
bracket0_end_size = math.ceil(
1 + math.log(self.max_epochs, self.factor)
)
bracket_end_size = bracket0_end_size / (bracket_num + 1)
return math.ceil(
bracket_end_size * self.factor ** (bracket_num - round_num)
)
def _get_epochs(self, bracket_num, round_num):
return math.ceil(
self.max_epochs / self.factor ** (bracket_num - round_num)
)
def _get_num_rounds(self, bracket_num):
# Bracket 0 just runs random search, others do successive halving.
return bracket_num + 1
def _get_num_brackets(self):
epochs = self.max_epochs
brackets = 0
while epochs >= self.min_epochs:
epochs = epochs / self.factor
brackets += 1
return brackets
and assuming that get_size is getting the number of configurations in that bracket and that round.
Am I just making some really silly calculations? Why is my model getting slightly different numbers of configurations, but the same number of epochs? (For reference, I had 5 different hyper parameters, each with 2 options, so 32 possible configurations available to test, and I used an early stopping callback with a patience of 8, though I doubt it came into effect here.)
I've tried running with several different values of max_epochs and factor, and I can only seem to get it the same if there are a very small number of brackets (everything goes kind of wonk once you get up to the s=3 bracket).
source https://stackoverflow.com/questions/76202927/the-number-of-configurations-keras-hyperband-gives-me-per-round-doesnt-match-th
Comments
Post a Comment