Bayesian Neural Network

An implementation of a Bayesian Neural Network that is trained using our SGMCMC sampling methods.

class bayesian_neural_network.BayesianNeuralNetwork(sampling_method=<SamplingMethod.SGHMC: 'SGHMC'>, n_nets=100, learning_rate=0.001, mdecay=0.05, n_iters=50000, batch_size=20, burn_in_steps=1000, sample_steps=100, normalize_input=True, normalize_output=True, seed=None, get_net=<function get_default_net>, session=None)

__init__(sampling_method=<SamplingMethod.SGHMC: 'SGHMC'>, n_nets=100, learning_rate=0.001, mdecay=0.05, n_iters=50000, batch_size=20, burn_in_steps=1000, sample_steps=100, normalize_input=True, normalize_output=True, seed=None, get_net=<function get_default_net>, session=None)

Bayesian Neural Networks use Bayesian methods to estimate the posterior distribution of a neural network’s weights. This allows to also predict uncertainties for test points and thus makes Bayesian Neural Networks suitable for Bayesian optimization.

This module uses stochastic gradient MCMC methods to sample from the posterior distribution.

See [1] for more details.

[1] J. T. Springenberg, A. Klein, S. Falkner, F. Hutter

Bayesian Optimization with Robust Bayesian Neural Networks. In Advances in Neural Information Processing Systems 29 (2016).

Parameters:

sampling_method : SamplingMethod, optional

Method used to sample networks for this BNN. Defaults to SamplingMethod.SGHMC.

n_nets: int, optional

Number of nets to sample during training (and use to predict). Defaults to 100.

learning_rate: float, optional

Learning rate to use during sampling. Defaults to 1e-3.

mdecay: float, optional

Momentum decay per time-step (parameter for SGHMCSampler). Defaults to 0.05.

n_iters: int, optional

Total number of iterations of the sampler to perform. Defaults to 50000

batch_size: int, optional

Number of datapoints to include in each minibatch. Defaults to 20 datapoints per minibatch.

burn_in_steps: int, optional

Number of burn-in steps to perform Defaults to 1000.

sample_steps: int, optional

Number of sample steps to perform. Defaults to 100.

normalize_input: bool, optional

Specifies whether or not input data should be normalized. Defaults to True

normalize_output: bool, optional

Specifies whether or not outputs should be normalized. Defaults to True

seed: int, optional

Random seed to use in this BNN. Defaults to None.

get_net: callable, optional

Callable that returns a network specification. Expected inputs are a tensorflow.Placeholder object that serves as feedable input to the network and an integer random seed. Expected return value is the networks final output. Defaults to get_default_net.

session: tensorflow.Session, optional

A tensorflow.Session object used to delegate computations performed in this network over to tensorflow. Defaults to None which indicates we should start a fresh tensorflow.Session.

__weakref__

list of weak references to the object (if defined)

compute_network_output(params, input_data)

Compute and return the output of the network when

parameterized with params on input_data.

Parameters:

params : list of ndarray objects

List of parameter values (ndarray) for each tensorflow.Variable parameter of our network.

input_data : ndarray (N, D)

Input points to compute the network output for.

Returns:

network_output: ndarray (N,)

Output of the network parameterized with params on the given input_data points.

negative_log_likelihood(X, Y)

Compute the negative log likelihood of the

current network parameters with respect to inputs X with labels Y.

Parameters:

X : tensorflow.Placeholder

Placeholder for input datapoints.

Y : tensorflow.Placeholder

Placeholder for input labels.

Returns:

neg_log_like: tensorflow.Tensor

Negative log likelihood of the current network parameters with respect to inputs X with labels Y.

mse: tensorflow.Tensor

Mean squared error of the current network parameters with respect to inputs X with labels Y.

To discretize possible user choices for the method used to train a Bayesian Neural Network, we maintain an Enum class called SamplingMethod.

The Enum class also provides facilities to obtain a supported sampler directly. To obtain a sampler, it is enough to call SamplingMethod.get_sampler(sampling_method, **sampler_args) with a supported sampling_method and corresponding keyword arguments in sampler_args.

class bayesian_neural_network.SamplingMethod

Enumeration type for all sampling methods we support.

static get_sampler(sampling_method, **sampler_args)

TODO: Docstring for get_sampler.

Parameters:

sampling_method : SamplingMethod

Enum corresponding to sampling method to return a sampler for.

**sampler_args : dict

Keyword arguments that contain all input arguments to the desired the constructor of the sampler for the specified sampling_method.

Returns:

sampler : Subclass of sampling.MCMCSampler

A sampler instance that implements the specified sampling_method and is initialized with inputs sampler_args.

static is_supported(sampling_method)

Static method that returns true if val is a supported sampling method (e.g. there is an entry for it in SamplingMethod enum).

Examples

Supported sampling methods give True:

>>> SamplingMethod.is_supported(SamplingMethod.SGHMC)
True

Other input types give False:

>>> SamplingMethod.is_supported(0)
False
>>> SamplingMethod.is_supported("test")
False

This module contains our implementation of priors for the weights and log variance of our Bayesian Neural Network.

class bnn_priors.LogVariancePrior(mean=0.01, var=2)

Prior on the log predicted variance.

__init__(mean=0.01, var=2)

Initialize prior for a given mean and variance.

Parameters:

mean : float, optional

Actual mean on a linear scale. Default value is `10e-3`.

var : float, optional

Variance on a log scale. Default value is `2`.

__weakref__

list of weak references to the object (if defined)

log_like(log_var)

Compute the log likelihood of this prior for a given input.

Parameters:log_var: tensorflow.Tensor Returns:log_like_output: tensorflow.Tensor

class bnn_priors.WeightPrior

Prior on the weights.

__init__()

Initialize weight prior with weight decay initialized to 1.

__weakref__

list of weak references to the object (if defined)

log_like(params)

Compute the log log likelihood of this prior for a given input.

Parameters:params : list of tensorflow.Variable objects Returns:log_like: tensorflow.Tensor