Text normalization
<templatestyles src="Module:Hatnote/styles.css"></templatestyles>
Text normalization is the process of transforming text into a single canonical form that it might not have had before. Normalizing text before storing or processing it allows for separation of concerns, since input is guaranteed to be consistent before operations are performed on it. Text normalization requires being aware of what type of text is to be normalized and how it is to be processed afterwards; there is no all-purpose normalization procedure.[1]
Contents
Applications
Text normalization is frequently used when converting text to speech. Numbers, dates, acronyms, and abbreviations are non-standard "words" that need to be pronounced differently depending on context.[2] For example:
- "$200" would be pronounced as "two hundred dollars" in English, but as "lua selau tālā" in Samoan.[3]
- "vi" could be pronounced as "vie," "vee," or "the sixth" depending on the surrounding words.[4]
Text can also be normalized for storing and searching in a database. For instance, if a search for "resume" is to match the word "résumé," then the text would be normalized by removing diacritical marks; and if "john" is to match "John", the text would be converted to a single case. To prepare text for searching, it might also be stemmed (e.g. converting "flew" and "flying" both into "fly"), canonicalized (e.g. consistently using American or British English spelling), or have stop words removed.
Techniques
For simple, context-independent normalization, such as removing non-alphanumeric characters or diacritical marks, regular expressions would suffice. For example, the sed script sed -e "s/\s+/ /g" inputfile would normalize runs of whitespace characters into a single space. More complex normalization requires correspondingly complicated algorithms, including domain knowledge of the language and vocabulary being normalized. Among other approaches, text normalization has been modeled as a problem of tokenizing and tagging streams of text[5] and as a special case of machine translation.[6][7]
References
- ↑ Lua error in package.lua at line 80: module 'strict' not found.
- ↑ Sproat, R.; Black, A.; Chen, S.; Kumar, S.; Ostendorfk, M.; Richards, C. (2001). "Normalization of non-standard words." Computer Speech and Language 15; 287–333. doi:10.1006/csla.2001.0169.
- ↑ Lua error in package.lua at line 80: module 'strict' not found.
- ↑ Lua error in package.lua at line 80: module 'strict' not found.
- ↑ Zhu, C.; Tang, J.; Li, H.; Ng , H.; Zhao, T. (2007). "A Unified Tagging Approach to Text Normalization." Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics; 688–695. doi:10.1.1.72.8138.
- ↑ Filip, G.; Krzysztof, J.; Agnieszka, W.; Mikołaj, W. (2006). "Text Normalization as a Special Case of Machine Translation." Proceedings of the International Multiconference on Computer Science and Information Technology 1; 51–56.
- ↑ Mosquera, A.; Lloret, E.; Moreda, P. (2012). "Towards Facilitating the Accessibility of Web 2.0 Texts through Text Normalisation" Proceedings of the LREC workshop: Natural Language Processing for Improving Textual Accessibility (NLP4ITA); 9-14
