لطفا منتظر باشید ...
Tokenizing a Heuristic Function
The one heuristic aspect of statistical filtering is tokenization. Even though the process of identifying features is dynamic, the way those features are initially established—how they are parsed out of an email—is programmed by a human. Fortunately, languages change slowly, and only a few minor tweaks are necessary to adapt the tokenization process to handle some of the wrenches thrown at it by spammers. Tokenization is the type of heuristic process that is usually defined once at build time and rarely requires further maintenance. In light of its simplicity, many attempts are still being made to establish tokenization through artificial intelligence, to remove all sense of heuristic programming from the equation. Within a few years, filters should be able to efficiently perform their own type of “DNA sequencing” on messages, determining the best possible way to extract data. In fact, this is already being researched as a solution to filtering some foreign languages that don’t use spaces or any other type of word delimiter.
