A computationally efficient algorithm
for finding sparse codes
A postscript version of the thesis is also available (17.1 Mb)
The activity ratio affects several aspects of information processing such as the architecture and robustness of networks, the number of distinct states that can be represented and stored, generalisation properties, and the speed and rules of learning. Sparse codes combine advantages of local and dense codes while avoiding most of their drawbacks.
In this work, a new efficient algorithm for finding sparse codes is presented. The computational complexity depends linearly on the number of neurons in the network, as opposed to previous algorithms, where the complexity has typically been quadratically dependent on the number of neurons. The ability of the algorithm to find meaningful features has been demonstrated in simulations with artificial and natural data.