Enshuo's Data Mining Notes

Image downloaded from https://blog.4tests.com/21-study-habits-reddit-community/ In this article, we will make a text generator with LSTM Recurrent Neural Network with Python Keras. We train the network with post titles from Raddit.com, LifeProTips board. At the end, it generates brilliant life tips in fluent English. My Python code can be found on my Github . What we are going to include: Loading and processing text data Training a naive LSTM neural network Training a modified LSTM neural network Summary: Comparing the two models Loading and Processing Text Data The Text data were crawled from Raddit.com, LifeProTips board. I made a web crawler with Python scrapy .  Check my previous post for details. We first load the data from database with some SQL. Here I use Microsoft Access. Since the data is large, to reduce training time and memory needed, we only use posts in Jan, 2017. Here is the code. Load data import pyodbc conn_str = ( ...

This article is a continuous of the previous post, Reddit text mining and visualization with R Shiny . In this article, we will introduce techniques on exploring topics on Reddit, worldnews board. Take a look at the data First, we load the data and select date range to 2017-01-01 to 2017-05-01, which is the nearest date I collected. We have 8785 posts, with medians of 9 points and 3 comments. The post-over-time is stable. About 500 posts per week. We see how the posts are made by plotting the author-post barchart. There's one guy made over 300 posts while others contributed less than 100. Most people made no more than 10. Find keywords Now we plot the keywords in a barchart base on their tf-idf. There're important keywords like "trump", "china", "korea". But we also see adjectives like "north", "south" that should be connected with a noun.  We plot the bigrams with high frequencies. Combini...

This article is a continuous of the previous post, Reddit text mining and visualization with R Shiny . In this article, we will explore the most popular music styles in 2016 based on Reddit, Music board. Take a look at the data We first load the data. Select Music board, set data range to 2016-01-01 to 2016-12-31 and we get the following outputs. In 2016, there are 23,774 posts in total. Stat. and boxplot shows most posts has 1 point or 0 comments (median). While the most popular quarter (3rd Qu.) has 3 points or 2 comments. The histogram shows a consistency of post number over the year. We draw Author-post barchart to see how those posts are made. The vertical axis is author ID's (too many, not listed), and horizontal axis is post numbers. As circled in red, a few people made a lot of posts. As circled in orange, most only made one post through the year. Identify frequent music types Now let's move on to identifying popular music types. The Term frequency barc...

Nowadays, there's a lot of information on the internet. Yet when they separate among web pages, there's no way analysis can be done. In this article, we will visit Reddit.com, download posts, parse information and store in a database with Python scrapy. The steps are: 1. Visiting URL This includes finding the specific URL that your desired information is in, and interact with the website's server by GET or POST. In practice, it can be sending usernames and password, setting cookies and so on. At the end of this step, you'll get a html file with all the contents. 2. Parsing information In the previous step, we have a html file, we can simply store it as a text file. But this is not too helpful for further analysis as there're irrelevant contents. We want to extract information. To do this, we need some background knowledge on HTML and CSS. Identify in which HTML element the information is contained and parse it out! 3. Store information in da...