6 lines
226 B
Plaintext
6 lines
226 B
Plaintext
Tokenizer: A tokenizer for Icelandic text
|
|
|
|
Tokenization is a necessary first step in many natural language processing
|
|
tasks, such as word counting, parsing, spell checking, corpus generation, and
|
|
statistical analysis of text.
|