Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations [62]

Original Abstract

There has been much interest in unsupervised learning of hierarchical generative models such as deep belief networks. Scaling such models to full-sized, high-dimensional images remains a difficult problem. To address this problem, we present the convolutional deep belief network, a hierarchical generative model which scales to realistic image sizes. This model is translation-invariant and supports efficient bottom-up and top-down probabilistic inference. Key to our approach is probabilistic max-pooling, a novel technique which shrinks the representations of higher layers in a probabilistically sound way. Our experiments show that the algorithm learns useful high-level visual features, such as object parts, from unlabeled images of objects and natural scenes. We demonstrate excellent performance on several visual recognition tasks and show that our model can perform hierarchical (bottom-up and top-down) inference over full-sized images.

Main points

• Probabilistic max-pooling
• Scale DBN to real-sized images
  • Computationally intractable
  • Need invariance in representation
• RBM
  • Binary valued: Independent Bernoulli random variables
  • Real valued: Gaussian with diagonal covariance
  • Training:
    • Stochastic gradient ascent on log-likelihood of training data
    • Contrastive divergence approximation
• Convolutional RBM
  • detection layers: convolving feature maps
  • pooling layers: shrink the representation
    • Block: CxC from bottom layer
    • Max-pooling : minimizes energy subject to only one unit can be active.
  • Sparsity regularization: hidden units have a mean activation close to a small constant
• Convolutional Deep belief network
  • Stacking CRBM on top of one another
  • Training:
    • Gibbs sampling
    • Mean-field (5 iterations in this paper)