in Bounding hyperparameter optimization with Tensorflow bijector chain in GPflow 2.0, I found an excellent explanation of how to set boundaries to my hyperparameters.
Unfortunately, I noticed that using the
tensorflow_probability.bijectors.Sigmoid transform causes numerical instabilities which lead to parameter values outside
[low, high] for me.
My current workaround is to define my own sigmoid transform that uses the alternative implementation in the comments of the
tensorflow_probability source code:
import tensorflow as tf from tensorflow_probability import bijectors as tfb from tensorflow_probability import math as tfm class mySigmoid(tfb.Sigmoid): def _stable_sigmoid(x): """A (more) numerically stable sigmoid than `tf.math.sigmoid`.""" x = tf.convert_to_tensor(x) if x.dtype == tf.float64: cutoff = -20 else: cutoff = -9 return tf.where(x < cutoff, tf.exp(x), tf.math.sigmoid(x)) def _forward(self, x): if self._is_standard_sigmoid: return self._stable_sigmoid(x) lo = tf.convert_to_tensor(self.low) # Concretize only once hi = tf.convert_to_tensor(self.high) ans = hi * tf.sigmoid(x) + lo * tf.sigmoid(-x) return tfb.math.clip_by_value_preserve_gradient(ans, lo, hi)
It is noted in the Tensorflow source that this approach has some drawbacks, however, so I wanted to ask if there are alternative ways to bound hyperparameters in GPFlow?
Another way to constrain a parameter in GPflow is to place a prior on it.
k = gpflow.kernels.Matern32() k.variance.prior = tfp.distributions.Gamma(to_default_float(2), to_default_float(3))
See more here.
Whether a prior is an appropriote solution depends on the details of what you’re trying to accomplish.
Answered By – Jesper Nielsen