It has turned out quite recently that a simpler document encoding method than the one that was used in the Publications might produce even better results.
It is a standard practice in information retrieval (IR) [Salton and McGill, 1983] to encode documents with vectors, in which each component corresponds to a different word, and the value of the component reflects the frequency of occurrence of the word in the document. If word occurs times in document j, is defined to be equal to , and is the unit vector corresponding to the kth vector component, it is possible to code the document j by
A problem with this encoding method is that if the vocabulary is very large the dimensionality of the vector is also high. In the Publications this problem was solved by eliminating some of the most common and some of the rarest words, and by clustering the words into word categories. (Actually an extended version of Equation 11 was used, where probabilistic information about the similarity of use of different words was incorporated into the coefficients .)
An alternative approach for reducing the dimensionality is simply to reduce the dimensionality of the vectors that in effect represent the words. A simple method is to project them randomly to a space of a lower dimensionality [Ritter and Kohonen, 1989]. It has turned out in experiments that using the reduced-dimensional version of Equation 11 without any contextual information results in better separability of different Usenet discussion areas (accuracy of the newsgroup separation was 69.1 %) than using either the current WEBSOM method (accuracy 62.6 %), or a method where vectors estimated based on contextual information are used in the place of the [Gallant et al., 1992, Gallant, 1994] (accuracy 65.4 %). The experimental procedures have been described in more detail in Publication 6. If this reduced-dimensional version of Equation 11 is used, however, the fast processing of documents by table lookups and subsequent convolutions, used in Publications 3, 4, 5, and 6, would become impossible.
These preliminary results need more thorough validation. It may in any case be concluded that the methods for utilizing contextual information can still be improved.