개발 일기

CNN (Convolutional Neural Network) Model Khaiii used the CNN Model as an algorithm for machine learning. Syllable-Based model When it comes to morphological analysis of Korean, the analysis result often displays a different form and length than the input due to syllable restoration, irregular eogan (parts of word like ‘빌’ of ‘빌리다’) and eomi (parts of word that may change like ‘리다’, ‘렸다’ of ‘빌리다’..

Analysis Error Patch There can be errors in any machine learning model’s analysis. There cannot be a 100% accurate morpheme analyzer for any input. The analysis error patch is a user dictionary which can correct the model’s analysis errors. Pre- Analysis Dictionary vs Analysis Error Patch The difference between the pre-analysis dictionary and the Analysis Error Patch is written below. Pre- Analy..

CNN Model Training Process Learning Corpus Khaiii used the Sejong Corpus for the model’s learning. The corpus format is as seen below. BTAA0001-00000001 1993/06/08 1993/SN + //SP + 06/SN + //SP + 08/SN BTAA0001-00000002 19 19/SN BTAA0001-00000012 프랑스의 프랑스/NNP + 의/JKG BTAA0001-00000013 세계적인 세계/NNG + 적/XSN + 이/VCP + ㄴ/ETM BTAA0001-00000014 의상 의상/NNG BTAA0001-00000015 디자이너 디자이너/NNG BTAA0001-0000001..

Test for Specialized Spacing Error Model.** Overfitting: If a model is overfitted, it would have a high accuracy to the training data, but it does not work properly on the verification or test data. An overfitted model struggles to evaluate new data. This means the model has been adapted too closely to the training data and even learned its noise. A user often forgets to put a space (especially ..

This post contains key terms that appear frequently in Korean morphological analysis and natural language processing (NLP). It encompasses concepts that are used in Khaiii project and will be helpful for those are not familiar with the Korean structure. I would recommend to review this glossary first before reading the translations. Terms (1) Korean Unit A. Jaso: a Korean Character ( e.g. ㄱ,ㄴ,ㄷ,..

Pre-analyzed Dictionary Pre-analyzed dictionary is used when a word’s analysis shows a consistent result regardless of its context. Type of Dictionary Entry There are two types of pre-analyzed dictionaries. Exact Match: When two words are fully matched Partial Match: When two words have the same beginning in common but are not exact matches Below are the examples of pre-analyzed dictionaries Num..

Khaiii Khaiii (Kakao Hangul Analyzer III) is the third morphological analyzer developed by Kakao. It is also named after DHA2 (Daumkakao Hangul Analyzer 2), the 2nd version of Kakao’s morphological analyzer. A morpheme is the smallest linguistic unit that can have meaning. As it is the smallest meaningful part of a word, a morpheme cannot be further analyzed. A morphological analyzer refers to s..

What is Khaiii? With an interest in machine learning translation systems, I have found an open-source project called the Khaiii project a couple of months ago. Khaiii has built by Jamie Lim and other Kakao developers in 2018. As introduced in its Github, Khaiii (Kakao Hangul Analyzer III) is the third morphological analyzer developed by Kakao. Morphological analysis is the most basic process in ..