Sinhala news analysis using text mining and machine learning

Abstract

Due to the rapid development of information technology, vast amounts of information are generated daily. Unstructured data such as news reports are a significant part of these growing information repositories. This study focuses on analyzing Sinhala news reports published online to extract important features using text mining and machine learning techniques. Then, represent this extracted information in a way that news readers find it easy to read news or do research on past news reports. For a morphologically rich complex language like Sinhala, it makes text mining a difficult task. In our approach, we first pre processed dataset with filtering, stop w ord removal, stemming and then experimented with feature selection methods such as n gram combinations, count vectorizer and TF IDF vectorizer. Text classification methods such as Naive Bayes, Support Vector Machines, Decision Trees, K means and hierarchic al clustering methods were evaluated. Later, we represented the mined knowledge using information visualization methods such as charts, tag clouds and tree structures. Unigram features with TF IDF vectorizer for feature selection, Naïve Bayes for document classification and K means for clustering were the most accurate techniques for Sinhala news. The accuracy of the information visualization methods was measured with human experts. Our results reveal that language specific text pre processing and feature selection increases the efficiency of information retrieval tasks when compared to generally used existing methods and the new representation model saves users’ time and effort to find news reports based on their preferences rather than going through exist ing news websites.