One long-term goal of machine learning research is to produce methods thatare applicable to highly complex tasks, such as perception (vision, audition), reasoning, intelligent control, and other artificially intelligent behaviors. We arguethat in order to progress toward this goal, the Machine Learning community mustendeavor to discover algorithms that can learn highly complex functions, with minimal need for prior knowledge, and with minimal human intervention. We presentmathematical and empirical evidence suggesting that many popular approachesto non-parametric learning, particularly kernel methods, are fundamentally limited in their ability to learn complex high-dimensional functions. Our analysisfocuses on two problems. First, kernel machines are shallow architectures, inwhich one large layer of simple template matchers is followed by a single layerof trainable coefficients. We argue that shallow architectures can be very inefficient in terms of required number of computational elements and examples. Second, we analyze a limitation of kernel machines with a local kernel, linked to thecurse of dimensionality, that applies to supervised, unsupervised (manifold learning) and semi-supervised kernel machines. Using empirical results on invariantimage recognition tasks, kernel methods are compared with deep architectures, inwhich lower-level features or concepts are progressively combined into more abstract and higher-level representations. We argue that deep architectures have thepotential to generalize in non-local ways, i.e., beyond immediate neighbors, andthat this is crucial in order to make progress on the kind of complex tasks requiredfor artificial intelligence
cited: 290 (01/06/2014)