Learning good representations from a large set of unlabeled data is a particularlychallenging task. Recent work (see Bengio (2009) for a review) shows that training deep architectures is a good way to extract such representations, by extractingand disentangling gradually higher-level factors of variation characterizing the inputdistribution. In this paper, we describe different kinds of layers we trained for learning representations in the setting of the Unsupervised and Transfer Learning Challenge. The strategy of our team won the final phase of the challenge. It combined andstacked different one-layer unsupervised learning algorithms, adapted to each of thefive datasets of the competition. This paper describes that strategy and the particularone-layer learning algorithms feeding a simple linear classifier with a tiny number oflabeled training samples (1 to 64 per class).