Using LVQ and SOM in speech recognition with multiple feature streams

M. Kurimo and P. Somervuo

The main objective of this research has been to study and test some neural network algorithms in speech recognition. Special interests are the selection of suitable variants of the algorithms and the feature extraction methods in order to minimize recognition errors and maintain the real-time demonstration system. Experiments are also made to up-scale the system to utilize increasing computational resources efficiently.

In current speech recognition system the phonemes are modelled by mixture density hidden Markov models (HMM), which assume that phonemes can be divided into successive states producing acoustical features according to some stationary stochastic processes. The probability density function of the short-time observations in each state is estimated using a mixture of Gaussian densities. The building block of the speech recognition system studied here is a short time feature vector. But because of overlapping phoneme classes it is advantageous to add some information about the context of the local feature vector. This can be done by combining several consecutive short time feature vectors by averaging, concatenation and using the difference of features from fixed time delay apart.

Using different types of feature vectors lead to multiple parallel observation streams to be connected by the classifier. One method is to simply concatenate all simultaneous outputs to a single observation vector. However, by processing the different input streams separately, the scaling problems and difficulties associated with very large codebooks could be avoided. This suggests the construction of a multi channel classifier of parallel input streams in which the classification decisions of multiple codebooks are joined by the HMM.

LVQ is applied in the training of the mixture density HMMs to increase the discrimination between different phoneme models. The traditional Viterbi training is enhanced so that instead of just maximizing the likelihood of the correct phoneme sequences, the mixture densities are tuned to also lower the likelihood of the best, but incorrect, rival by a segmental LVQ3 training. This is shown to provide similar recognition error rates than applying an additional LVQ2-based corrective tuning, but in a more efficient and elegant manner.

The semi-continuous HMMs are modified to phoneme-wise tied mixtures, where the states in one HMM use a common codebook of Gaussians. This reduces significantly the excessive amount of mixture weights in conventional semi-continuous models, but it still ties some highly overlapping Gaussians intensifying the use of the training data.

The hierarchical SOMs have been studied by some experiments. In the single phoneme recognition experiments the tree search and normal search have given similar results while remarkable speedup has been gained by using the tree search. This is due to the limited search area in each layer of the map. Because SOMs act a significant part in the training of mixture density codebooks, the application of the hierarchical SOMs could provide a possibility to treat larger codebooks.

New speech database has been collected from 14 Finnish male speakers. Each of them has spoken four times 350 Finnish words. This database has been used in experiments.