Enshuo's Data Mining Notes

This article is a continuous of the previous post, Reddit text mining and visualization with R Shiny . In this article, we will introduce techniques on exploring topics on Reddit, worldnews board. Take a look at the data First, we load the data and select date range to 2017-01-01 to 2017-05-01, which is the nearest date I collected. We have 8785 posts, with medians of 9 points and 3 comments. The post-over-time is stable. About 500 posts per week. We see how the posts are made by plotting the author-post barchart. There's one guy made over 300 posts while others contributed less than 100. Most people made no more than 10. Find keywords Now we plot the keywords in a barchart base on their tf-idf. There're important keywords like "trump", "china", "korea". But we also see adjectives like "north", "south" that should be connected with a noun.  We plot the bigrams with high frequencies. Combini...

This article is a continuous of the previous post, Reddit text mining and visualization with R Shiny . In this article, we will explore the most popular music styles in 2016 based on Reddit, Music board. Take a look at the data We first load the data. Select Music board, set data range to 2016-01-01 to 2016-12-31 and we get the following outputs. In 2016, there are 23,774 posts in total. Stat. and boxplot shows most posts has 1 point or 0 comments (median). While the most popular quarter (3rd Qu.) has 3 points or 2 comments. The histogram shows a consistency of post number over the year. We draw Author-post barchart to see how those posts are made. The vertical axis is author ID's (too many, not listed), and horizontal axis is post numbers. As circled in red, a few people made a lot of posts. As circled in orange, most only made one post through the year. Identify frequent music types Now let's move on to identifying popular music types. The Term frequency barc...

Nowadays, there's a lot of information on the internet. Yet when they separate among web pages, there's no way analysis can be done. In this article, we will visit Reddit.com, download posts, parse information and store in a database with Python scrapy. The steps are: 1. Visiting URL This includes finding the specific URL that your desired information is in, and interact with the website's server by GET or POST. In practice, it can be sending usernames and password, setting cookies and so on. At the end of this step, you'll get a html file with all the contents. 2. Parsing information In the previous step, we have a html file, we can simply store it as a text file. But this is not too helpful for further analysis as there're irrelevant contents. We want to extract information. To do this, we need some background knowledge on HTML and CSS. Identify in which HTML element the information is contained and parse it out! 3. Store information in da...

Text mining, unlike other fields of data mining, it requires expertise on many disciplines. To make meaningful conclusions, we'll need psychology, sociology, marketing, just to name a few. It can't be and shouldn't be finished by data scientists only. However, text mining is not accessible for most people. Take web data text mining for exam, there're three difficulties: 1. Collecting data is hard Though data are open and free online, there're often too big and messy to be downloaded by hand. It requires some programming and data management skills. 2. Data is dirty and messy Text data is made by people. People can make terrible data with strange spelling, grammar, format, encoding... Data cleaning on text data is a big challenge. 3. Fitting model & visualization is complicated Text data is unstructured. That makes modeling and visualizing complicated. Knowledge on both math and programming is required.  About this project In this project, ...