Practical AI: Machine Learning and Data Mining with Weka and Java



Practical AI: Machine Learning and Data Mining with Weka and Java

Most people tend to think of Artificial Intelligence with depictions of autonomous bipedal robots speaking and greeting humans in a monotone voice. The computer science field of Artificial Intelligence is far from that science fiction world that we see portrayed in the movies. The most prominent past AI research seems more like it would fit under a branch of mathematics and not computer science. AI research sub-fields include machine learning, data mining, computer vision, string match and indexing, search algorithms and neural networks. There is a new focus on not just faster search algorithms and pattern recognition, but building a relationship between computer science, neuroscience, philosophy, psychology, biology. AI is branching out beyond the algorithms and merging with those other general science and medical fields. Marvin Minsky, a prominent researcher in the AI field, called some of the previous interests like Neural Network, technology fads. Marvin Minsky's most recent book, the Emotion Machine describes some of human's behavior with a simple set of rules. The book is a useful guide for computer scientists that want to model that behavior. Jeff Hawkins of Numenta and previously the creator Palm Inc has shifted his focus from mobile computing to developing a sophisticated AI system. Numenta has patented HTM (Hierarchical Temporal Memory) technology. According to Jeff Hawkins, you cannot mimic brain functions without including a hierarchical system of memory. The lower level has more input/output nodes than the level above it. According to Jeff, you must also take into account "temporal" memory. For example, the human brain has many parts of the brain that handle visual information. The brain may be able to detect a particular object but also factor in the time that the visual event occurred. If you are at the zoo, your brain predicts that you will see animals and animals in cages at the zoo. It is rare that you see a plane take off or land in the middle of your zoo visit. You would have had previous memories or seen pictures of the zoo in the past and parts of your brain activate other things associated the zoo. The context in time is a visit to the zoo. The memory of a zoo visit is probably in a different area of the brain than a trip to an airport. In the case of an airport, you expect to see planes landing and taking off.

Basic Data Mining With Weka

In my post, I present a practical Hello World using WEKA. WEKA (Waikato Environment for Knowledge Analysis) is a suite of machine learning tools and libraries that can be used to mine data.

What is data mining and how could you use data mining techniques? Many enterprise companies are connected to large databases. These databases contain millions of records. Data mining and machine learning techniques are used to find patterns within that massive trove of data. These techniques can also used to filter out noise. Popular email filter software utilizes data mining to remove or categorize spam email.

Hello World Weka for Java Web Server Log Files

I have dozens of Apache web server log files. I wanted to find groups or clusters between some of the log files and the time that a user requested a page on my site. I used WEKA to find clusters and categorize various groups of relevant data.

12_15_10_10_14_03 [HASERR N] log.fileName1[]:Wed 22:14:03 EST 2010 [SIZE 144]
12_15_10_10_14_03 [HASERR N] log.fileName2[]:Wed 22:14:03 EST 2010 [SIZE 121]
12_15_10_10_14_03 [HASERR N] log.fileName2[]:Wed 22:14:03 EST 2010 [SIZE 156]
12_15_10_10_15_33 [HASERR N] log.fileName3[]:Wed 22:15:33 EST 2010 [SIZE 160]
12_15_10_10_15_33 [HASERR N] log.fileName3[]:Wed 22:15:33 EST 2010 [SIZE 146]
...

The timestamp with hour, minute and seconds are on the furthest column to the left. The middle column contains filename information. There are only four or five rows shown in the example. But the log files contains millions of requests. I determined the data that I want to categorize, then I converted the timestamp log file into an ARFF file format that WEKA requires. WEKA has several tools to convert generic CSV files into a ARFF format. ARFF is essentially a text database with column attributes and rows of the data that you to analyze.

Example ARFF File for Weather Data:
@relation weather
@attribute outlook {sunny, overcast, rainy}
@attribute temperature real
@attribute humidity real
@attribute windy {TRUE, FALSE}
@attribute play {yes, no}
@data
sunny,85,85,FALSE,no
sunny,80,90,TRUE,no
overcast,83,86,FALSE,yes
rainy,70,96,FALSE,yes
rainy,68,80,FALSE,yes
rainy,65,70,TRUE,no
overcast,64,65,TRUE,yes
sunny,72,95,FALSE,no
sunny,69,70,FALSE,yes
rainy,75,80,FALSE,yes
sunny,75,70,TRUE,yes
overcast,72,90,TRUE,yes
overcast,81,75,FALSE,yes
rainy,71,91,TRUE,no
Here is my arff file with web server log data. The attributes consist of the log file name and the rows contain information with a count the server wrote a line to the log file at a particular period of time.

Example arff file for server logs:
% Title: Database for application line count
@relation scan_app_data
@attribute LogXX0 real
@attribute LogXX1 real
...
@attribute LogXX36 real
@attribute LogXX37 real
@attribute LogXX38 real
@attribute LogXX39 real
@attribute timeperiod real
@attribute class { 'night_morn', 'earl_morn', 'midday', 'after', 'night', 'late_nigh' }
@data
0,0,0,0,0,0,0,0,0,0,3,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,night_morn
,0,0,0,0,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,night_morn


I used the WEKA GUI tools to visualize the patterns. I simply clicked on ./weka.jar to launch the WEKA explorer.

It was interesting to focus on the groups of requests. I could see a cluster of requests combined with errors displayed to the user. After looking at the visual tools, I could easily determine what time period had the most errors and in which log file.

I just wanted to cover some basics. WEKA is mature Java software but the developer and the researcher must do some work to determine which data to feed to WEKA. In the case of the log files, I wanted to cluster log file data, errors and the time period that these events occurred. In future test, I might change my ARFF file to include the web server execution time vs the application requested to see if there is any correlation between the two.

Resources:

http://www.cs.waikato.ac.nz/ml/weka/

http://www.numenta.com/

Comments

Post a Comment

Popular posts from this blog

On Unit Testing, Java TDD for developers to write

It has been a while, let's kick off 2017 with a blog entry. I have read and I am reading about four or five posts a day about unit testing. It really has been a long time obsession for me. I have moved past the technical and practical considerations on unit testing frameworks and done with the debates with "should you use Junit or Mockito or Karma?" I am more interested in the psychology of unit testing, who does it, likes it, hates it? It really is one of those easy to learn, hard to master concepts. For example, many many may play chess when they are young and can end up being horribly chess players most of their life, I am part of that majority. Unfortunately, I have never played chess and sat down for hours and tried to master it. I never see the common patterns or have a developed end game. I mostly just play with a knowledge of the basic rules. Following good unit testing practices within your software development shop is a lot like playing chess. It is easy to le...
Read more

Is Java the new COBOL? Yes. What does that mean, exactly? (Part 1)

Image
"If you want to know the past, look at your present. If you want to know the future, look at your present." -- Gautama Buddha Java is the new COBOL, whaaaat?     If you are software developer that mainly writes applications for the web; even if you aren't a Java/J2EE developer, you have probably heard the new meme circulating the web and blogosphere. "Java is the new COBOL" . Most blogs and technical writers regurgitate that statement and follow-up by saying that EJBs suck, and some companies are considering Ruby on Rails. I didn't really think much of these posts. I get the impression that a lot of these "managers" don't write any Java or COBOL code. They are really not highlighting the seriousness of the problem. COBOL development has remained stagnant for decades. Java may see the same problem in the future. The Pentagon/DOD has a trillion dollar COBOL problem because it is so difficult to change their ancient infrastructure. Anyon...
Read more

JVM Notebook: Basic Clojure, Java and JVM Language performance

Image
"Measure, don't guess. The primary goal of all performance tuning exercises should be maximize the end user experience given the resource constraints." [1] "It isn't so much a "farewell to the J" as an expansion of the platform opportunities Java provides. Sun's investment to power ongoing development of JRuby and Jython broadens the range and reach of Java, as a whole." -- Rick Ross Overview and JVM Languages    One of the exciting trends to recently emerge from the Java community is the concept of the JVM language. These technologies are all that you would expect them to be. They are implementations of languages that run on the Java Virtual Machine. Some are newly created and some are based on existing, more mature languages. JRuby, Jython are two JVM languages based on CRuby and CPython. Groovy, Scala, Clojure are three completely new JVM languages that were created to add new language features that weren't supported by the core Ja...
Read more