A Programmer’s Guide To Data Mining Compressed
User Manual:
Open the PDF directly: View PDF .
Page Count: 395
Download | ![]() |
Open PDF In Browser | View PDF |
A Programmer’s Guide to Data Mining The Ancient Art of the Numerati Ron Zacharski A Programmer’s Guide to Data Mining: The Ancient Art of the Numerati www.guidetodatamining.com by Ron Zacharski Creative Commons Attribution Noncommercial 3.0 license Attribution information for all photographs is available on the website. Thanks to ... my son Adam my wife Cheryl Roper Roz and Bodhi also a huge thanks to all the photographers who put their work in the Creative Commons ii INTRO Preface If you continue this simple practice every day, you will obtain some wonderful power. Before you attain it, it is something wonderful, but after you attain it, it is nothing special. Shunryu Suzuki Zen Mind, Beginner's Mind. Before you work through this book you might think that systems like Pandora, Amazon's recommendations, and automatic data mining for terrorists, must be very complex and the math behind the algorithms must be extremely complex requiring a PhD to understand. You might think the people who work on developing these systems are like rocket scientists. One goal I have for this book is to pull back this curtain of complexity and show some of the rudimentary methods involved. Granted there are super-smart people at Google, the National Security Agency and elsewhere developing amazingly complex algorithms, but for the most part data mining relies on easy-to-understand principles. Before you start the book you might think data mining is pretty amazing stuff. By the end of the book, I hope you will be able to say nothing special. The Japanese characters above, Shoshin, represent the concept of Beginner's Mind—the idea of having an open mind that is eager to explore possibilities. Most of us have heard some version of the following story (possibly from Bruce Lee's Enter the Dragon). A professor is seeking enlightenment and goes to a wise monk for spiritual direction. The professor dominates the discussion outlining everything he has learned in his life and summarizing papers he has written. The monk asks tea? and begins to pour tea into the professor's cup. And continues to pour, and continues to pour, until the tea over pours the teacup, the table, and spills onto the floor. What are you doing? the professor shouts. Pouring tea the monk says and continues: Your mind is like this teacup. It is so filled with ideas that nothing else will go in. You must empty your mind before we can begin. iii To me, the best programmers are empty cups, who constantly explore new technology (noSQL, node-js, whatever) with open minds. Mediocre programmers have surrounded their minds with cities of delusion—C++ is good, Java is bad, PHP is the only way to do web programming, MySQL is the only database to consider. My hope is that you will find some of the ideas in this book valuable and I ask that you keep a beginner's mind when reading it. As Shunryu Suzuki says: In the beginner's mind there are many possibilities, In the expert's mind there are few. iv Chapter 1 The Intro Intro to data mining & how to use this book Imagine life in a small American town 150 years ago. Everyone knows one another. A crate of fabric arrives at the general store. The clerk notices that the pattern of a particular bolt would highly appeal to Mrs. Clancey because he knows that she likes bright floral patterns and makes a mental note to show it to her next time she comes to the store. Chow Winkler mentions to Mr. Wilson, the saloon keeper, that he is thinking of selling his spare Remington rifle. Mr. Wilson mentions that information to Bud Barclay, who he knows is looking for a quality rifle. Sheriff Valquez and his deputies know that Lee Pye is someone to keep an eye on as he likes to drink, has a short temper, and is strong. Life in a small town 100 years ago was all about connections. People knew your likes and dislikes, your health, the state of your marriage. For better or worse, it was a personalized experience. And this highly personalized life in the community was true throughout most of the world. Let's jump ahead one hundred years to the 1960s. Personalized interactions are less likely but they are still present. A regular coming into a local bookstore might be greeted with "The new James Michener is in"-- the clerk knowing that the regular loves James Michener books. Or the clerk might recommend to the regular The Conscience of a Conservative by Barry Goldwater, because the clerk knows the regular is a staunch conservative. A regular customer comes into a diner and the waitress says "The usual?" Even today there are pockets of personalization. I go to my local coffee shop in Mesilla and the barista says "A venti latte with an extra shot?" knowing that is what I get every morning. I take my standard poodle to the groomers and the groomer doesn't need to ask what style of clip I want. She knows I like the no frills sports clip with the German style ears. But things have changed since the small towns of 100 years ago. Large grocery stores and big box stores replaced neighborhood grocers and other merchants At the start of this change choices were limited. Henry Ford once said "Any customer can have a car painted any color that he wants so long as it is black." The record store carried a limited number of records; the bookstore carried a limited number of books. Want ice cream? The choices were vanilla, chocolate, and maybe strawberry. Want a washing machine? In 1950 you had two choices at the local Sears: the standard model for $55 or the deluxe for $95. Welcome to the 21st century In the 21st century those limited choices are a thing of the past. I want to buy some music? iTunes has some 11 million tracks to choose from. 11 million! They have sold 16 billion tracks as of October 2011. I need more choices? I can go to Spotify which has over 15 million songs. I want to buy a book? Amazon has over 2 million titles to chose from. 1-2 CONTENT BASED FILTERING & CLASSIFICATION I want to watch a video? There are plenty of choices: over 100,00 0 titles nearly 50,000 titles over 100,000 titles I want to buy a laptop? When I type in laptop into the Amazon search box I get 3,811 results I type in rice cooker and get over 1,000 possibilities. In the near future there will be even more choice—billions of music tracks online—a wide variety of video—products that can be customized with 3D printing. 1-3 Finding Relevant Stuff The problem is finding relevant stuff. Amid all those 11 million tracks on iTunes, there are probably quite a number that I will absolutely love, but how do I find them. I want to watch a streaming movie from Netflix tonight, what should I watch. I want to download a movie using P2P, but which movie. And the problem is getting worse. Every minute terabytes of media are added to the net. Every minute 100 new files are available on usenet. Every minute 24 hours of video is uploaded to YouTube. Every hour 180 new books are published. Every day there are more and more options of stuff to buy in the real world. It gets more and more difficult to find the relevant stuff in this ocean of possibilities. If you are a producer of media—say Zee Avi from Malaysia—the danger isn't someone downloading your music illegally—the danger is obscurity. LObscurityN But how to find stuff? Years ago, in that small town, our friends helped us find stuff. That bolt of fabric that would be perfect for us; that new novel at the bookstore; that new 33 1/3 LP at the record store. Even today we rely on friends to help us find some relevant stuff. We used experts to help us find stuff. Years ago Consumer Reports could evaluate all the washing machines sold—all 20 of them—or all the rice cookers sold-- all 10 of them and make recommendations. Today there are hundreds of different rice cookers available on Amazon and it is unlikely that a single expert source can rate all of them. Years ago, Roger Ebert would review virtually all the movies available. Today about 25,000 movies are made each year worldwide. Plus, we now have access to video from a variety of sources. Roger Ebert, or any single expert, cannot review all the movies that are available to us. We also use the thing itself to help us find stuff. For example, I owned a Sears washing machine that lasted 30 years, I am going to buy another Sears washing machine. I liked one album by the Beatles—I will buy another thinking chances are good I will like that too. 1-4 CONTENT BASED FILTERING & CLASSIFICATION These methods of finding relevant stuff— friends, experts, the thing itself—are still present today but we need some computational help to transform them into the 21st century where we have billions of choices. These methods of finding relevant stuff—friends, experts, the thing itself—are still present today but we need some computational help to transform them into the 21st century where we have billions of choices. In this book we will explore methods of aggregating people's likes and dislikes, their purchasing history, and other data—exploiting the power of social net (friends)—to help us mine for relevant stuff. We will examine methods that use attributes of the thing itself. For example, I like the band Phoenix. The system might know attributes of Phoenix—that it uses electric rock instrumentation, has punk influences, has a subtle use of vocal harmony. It might recommend to me a similar band that has similar attributes, for example, The Strokes. It’s just not stuff... Data mining is just not about recommending stuff to us, or having merchants sell more stuff. Consider these examples. The mayor of that small town of 100 years ago, knew everybody. When he ran for re-election he knew how to tailor what he said to each individual. Martha, I know you are interested in schools and I will do everything in my power to bring another teacher to town. John, how is your bakery doing? I promise to get more parking in your area of downtown. 1-5 My father belonged to the United Auto Workers' Union. Around election time I remember the union representative coming to our house to remind my father what candidates to vote for: Hey Syl, how are the wife and kids? … Now let me tell you why you should vote for Frank Zeidler, the Socialist candidate for mayor... This individualized political message changed to the homogenous ads during the rise of television. Everyone got the exact same message. A good example of this is the famous Daisy television ad in support of Lyndon Johnson ( a young girl pulling petals off a daisy while a nuclear bomb goes off in the background). Now, with elections Frank Zeidler was the determined by small margins and the growing use of data mining, Socialist mayor of Milwaukee from individualization has returned. You are interested in a women's 1948 to 1960. right to choose? You might get a robo-call directed at that very issue. The sheriff of that small town knew who the trouble makers were. Now, threats seem to be hidden, terrorists can be anywhere. On October 11, 2001 the US government passed the USA Patriot Act (short for Uniting and Strengthening America by Providing Appropriate Tools Required to Intercept and Obstruct Terrorism). In part this bill enables investigators to obtain records for a variety of sources including libraries (what books we read), hotels (who stayed where and for how long), credit card companies, toll roads registering that we passed by. For the most part the government uses private companies to keep data on us. Companies like Seisint have data on almost all of us, photos of us, where we live, what we drive, our income, our buying behavior, our friends. Seisint owns supercomputers that use data mining techniques to make predictions about people. Their product by the way is called... 1-6 CONTENT BASED FILTERING & CLASSIFICATION The Matrix. Data Mining Extends what we already do! Stephen Baker begins his book The Numerati this way: Imagine you are in a café, perhaps the noisy one I'm sitting in at this moment. A young women at a table to your right is typing on her laptop. You turn your head and look at her screen. She surfs the Internet. You watch. Hours pass. She reads an online paper. You notice that she reads three articles about China. She scouts movies for Friday night and watches the trailer for Kung Fu Panda. She clicks on an ad that promises to connect her to old high school classmates. You sit there taking notes. With each passing minute, you're learning more about her. Now imagine that you could watch 150 million people surfing at the same time. Data mining is focused on finding patterns in data. At the small scale, we are expert at building mental models and finding patterns. I want to watch a movie tonight with my wife. I have a mental model of what she likes. I know she dislikes violent movies (she didn't like District 9 for that reason). She likes movies by Charlie Kaufman. I can use that mental model I have of her movie preferences to predict what movies she may or may not like. 1-7 A friend is visiting from Europe. I know she is a vegetarian and I can use that information to predict she would not like the local rib joint. People are good at making models and making predictions. Data mining expands this ability and enables us to handle large quantities of information—the 150 million people in the Baker quote above. It enables the Pandora Music Service to tailor a music station to your specific musical preferences. It enables Netflix to make specific personalized movie recommendations for you. Tera-mining is not something from Starcraft II At the end of the 20th century a million word data set was considered large. When I was a graduate student in the 1990s (yes, I am that ancient) I worked as a programmer for a year on the Greek New Testament. It's only around 200,000 words but the analysis was too large to fit into the mainframe's memory necessitating spooling results off to magnetic tapes, which I had to request to be mounted. The book resulting from this work is the Analytical Greek New Testament by Timothy and Barbara Friberg (available on Amazon). I was just one of three programmers on this project done at the University of Minnesota. 1-8 CONTENT BASED FILTERING & CLASSIFICATION Today it is not unusual to be doing data mining on terabytes of information. Google has over 5 petabytes (that's 5,000 terabytes) of web data. In 2006 Google released a dataset to the research community based on one trillion words. The National Security Agency has call records for trillions of phone calls. Acxiom, a company that collects information (credit card purchases, telephone records, medical records, car registrations, etc) on 200 million adults in the US, has amassed over 1 petabyte of data. a 1 petabyte server shipping container Robert O'Harrow, Jr., author of No Place to Hide, in an effort to help us grasp how much information is 1 petabyte says it is the equivalent of 50,000 miles of stacked King James Bibles. I frequently drive 2,000 between New Mexico and Virginia. When I try to imagine bibles stacked along the entire way that seems like an unbelievable amount of data. New Mexico to Virginia 1-9 The Library of Congress has around 20 terabytes of text. You could store the entire collection of the Library of Congress on a few thousand dollar's worth of hard drives! In contrast, Walmart has over 570 terabytes of data. All this data just doesn't sit there—it is constantly being mined, new associations made, patterns identified. Tera-mining. Throughout this book we will be dealing with small datasets. It's good thing. We don't want our algorithm to run for a week only to discover we have some error in our logic. The biggest dataset we will use is under 100MB; the smallest just tens of lines of data. The format of the book. This book follows a learn-by-doing approach. Instead of passively reading the book, I encourage you to work through the exercises and experiment with the Python code I provide. Experimenting around, code hacking, and trying out methods with different data sets is the key to really gaining an understanding for the techniques. I try to strike a balance between hands-on, nuts-and-bolts discussion of Python data mining code that you can use and modify, and the theory behind the data mining techniques. To try to prevent the brain freeze associated with reading theory, math, and Python code, I tried to stimulate a different part of your brain by adding drawings and pictures. 1-10 CONTENT BASED FILTERING & CLASSIFICATION Peter Norvig, Director of Research at Google, had this to say in his great Udacity course. Design of a Computer Program: “I’ll show you and discuss my solution. It’s important to note, there is more than one way to approach a problem. And I don’t mean that my solution is the ONLY way or the BEST way. My solutions are there to help you learn a style and some techniques for programming. If you solve problems a different way, that’s fine. Good for you. All the learning that goes on happens inside of your head. Not inside of my head. So what’s important is that you understand the relation between your code and my code, that you get the right answer by writing out the solution yourself and then you can examine my code and maybe pick out some pointers and techniques that you can use later.” I couldn’t agree more! This book is not a comprehensive textbook on data mining techniques. There are textbooks, like Introduction to Data Mining by Pang-Ning Tan, Michael Steinbach, and Vipin Kumar that provide significantly better coverage of data mining methods and provide more in-depth analysis of the mathematic underpinnings of these methods. This book—the one you are holding—is intended more as a quick, gritty, hands-on introduction designed to give you a basic foundation of data mining techniques. Later, you can pick up a more comprehensive book to fill in any gaps that you wish. 1-11 Part of the usefulness of this book is the accompanying Python code and the datasets. I think the inclusion of both these make it easier for the learner to understand key concepts, but at the same time, not shoe-horn the learner into a scripted exploration. What will you be able to do when you finish this book? When you finish this book you will be able to design and implement recommendation systems for websites using Python or any language you know. For example, when you look at a product on Amazon, or a tune on Pandora, you are presented with a list of recommendations (You might also like …). You will learn how to develop such systems. In addition, the book should provide you with the necessary vocabulary to enable you to work in development teams on data mining efforts. As part of this goal, this book should help shed the mystery of recommendation systems, terrorist identification systems, and other data mining systems. You should at least have a rough idea of how they work. Why – why does this matter? Why should you use your time reading (and working through) this book on data mining? At the beginning of this chapter I gave examples related to the importance of data mining. The summary of that section would go as follows. There's lots of stuff out there (movies, music, books, rice cookers). There's going to be a huge growth in the amount of stuff out there. The problem with having all this stuff available is finding the stuff that is relevant to us. Of all the movies out there, what movie should I watch. What's the next book I should read? This problem of identifying relevant stuff is what data mining is about. Most websites will have some component dealing with 'finding stuff'. In addition to the movies, music, books, and rice cookers mentioned above, you might want recommendations about what friends to follow. How about a personalized newspaper showing just the news you are most interested in? If you are a programmer, particularly a web developer, it would be useful to know data mining techniques. Okay, so you can see the reason to devote some of your time to learning data mining, but why this book? There are books that give you a non-technical overview of data mining. They are a quick read, entertaining, inexpensive, and can be read late at night (no hairy technical bits). A great example of this is The Numerati by Stephen Baker. I recommend this book—I listened to the audio version of it while driving between Virginia and New Mexico. It was engrossing. On the other extreme are college textbooks on data mining. They are 1-12 CONTENT BASED FILTERING & CLASSIFICATION comprehensive and provide an in-depth analysis of data mining theory and practice. Again, I recommend books in this category. I wrote this book to fill a gap. It's a book designed for people who love to program—hackers. The book is intended to be read at a computer so the reader can participate and mess with code. Eeeks! The book has math formulas but I try to explain them in a way that is intelligible to average programmers, who may have forgotten a hunk of the math they took in college. s(i, j) = ∑ (R u,i − Ru )(Ru, j − Ru ) u∈U ∑ (R u,i u∈U − R u )2 ∑ (R u, j − R u )2 u∈U If that doesn't convince you, this book is also free (as in no cost) and free as in you can share it. 1-13 What's with the ‘Ancient Art of the Numerati’ part of the title In June of 2010 I was trying to come up with a title for this book. I like clever titles, but unfortunately, I have no talent in the area. I recently published a paper titled Linguistic Dumpster Diving: Geographical Classification of Arabic Text (yep, a data mining paper). I like the title and it is clever because it fits with the content of the paper, but I have to confess my wife came up with the title. I co-wrote a paper Mood and Modality: Out of the theory and into the fray. My co-author Marjorie McShane came up with the title. Anyway, back to June, 2010. All my clever title ideas were so vague that you wouldn't have a clue what the book was about. I finally settled on A Programmer's Guide to Data Mining as part of the title. I believe that bit is a concise description of the content of the book—I intend the book be a guide for the working programmer. You might wonder what is the meaning of the part after the colon: A Programmer's Guide to Data Mining: !e Ancient A" of # Numerati. The Numerati is a term coined by Stephen Baker. Each one of us generates an amazing amount of digital data everyday. credit card purchases, Twitter posts, Gowalla posts, Foursquare check-ins, cell phone calls, email messages, text messages, etc. You get up. The Matrix knows you boarded the subway at the Foggy Bottom Station at 7:10 and departed the Westside Station at 7:32. The Matrix knows you got a venti latte and a blueberry scone at the Starbucks on 5th and Union at 7:45; you used Gowalla to check-in at work at 8:05; you made an Amazon purchase for the P90X Extreme Home Fitness Workout Program 13 DVD set and a chin-up bar at 9:35; you had lunch at the Golden Falafel. Stephen Baker writes: 1-14 CONTENT BASED FILTERING & CLASSIFICATION The only folks who can make sense of the data we create are crack mathematicians, computer scientists, and engineers. What will these Numerati learn about us as they run us into dizzying combinations of numbers? First they need to find us. Say you're a potential SUV shopper in the northern suburbs of New York, or a churchgoing, antiabortion Democrat in Albuquerque. Maybe you're a Java programmer ready to relocate to Hyderabad, or a jazz-loving, Chianti-sipping Sagittarius looking for walks in the country and snuggles by the fireplace in Stockholm, or—heaven help us—maybe you're eager to strap bombs to your waist and climb onto a bus. Whatever you are—and each of us is a lot of things—companies and governments want to identify and locate you. Baker As you can probably guess, I like this term Numerati and Stephen Baker's description of it. 1-15 Chapter 2: Collaborative filtering I like what you like We are going to start our exploration of data mining by looking at recommendation systems. Recommendation systems are everywhere—from Amazon: ! to last.fm recommending music or concerts: In the Amazon example, above, Amazon combines two bits of information to make a recommendation. The first is that I viewed The Lotus Sutra translated by Gene Reeves; the second, that customers who viewed that translation of the Lotus Sutra also viewed several other translations. The recommendation method we are looking at in this chapter is called collaborative filtering. It's called collaborative because it makes recommendations based on other people— in effect, people collaborate to come up with recommendations. It works like this. Suppose the task is to recommend a book to you. I search among other users of the site to find one that is similar to you in the books she enjoys. Once I find that similar person I can see what she likes and recommend those books to you—perhaps Paolo Bacigalupi's The Windup Girl. 2-2 COLLABORATIVE FILTERING How do I find someone who is similar? So the first step is to find someone who is similar. Here's the simple 2D (dimensional) explanation. Suppose users rate books on a 5 star system—zero stars means the book is terrible, 5 stars means the book is great. Because I said we are looking at the simple 2D case, we restrict our ratings to two books: Neal Stephenson's Snow Crash and the Steig Larsson's The Girl with the Dragon Tattoo. First, here's a table showing 3 users who rated these books Amy Bill Jim Snow Crash 5✩ 2✩ 1✩ Girl with the Dragon Tattoo 5✩ 5✩ 4✩ I would like to recommend a book to the mysterious Ms. X who rated Snow Crash 4 stars and The Girl with the Dragon Tattoo 2 stars. The first task is to find the person who is most similar, or closest, to Ms. X. I do this by computing distance. 2-3 Manhattan Distance The easiest distance measure to compute is what is called Manhattan Distance or cab driver distance. In the 2D case, each person is represented by an (x, y) point. I will add a subscript to the x and y to refer to different people. So (x1, y1) might be Amy and (x2, y2) might be the elusive Ms. X. Manhattan Distance is then calculated by | x1 - x2| + | y1 - y2 | (so the absolute value of the difference between the x values plus the absolute value of the difference between the y values). So the Manhattan Distance for Amy and Ms. X is 4: Computing the distance between Ms. X and all three people gives us: Amy Bill Jim 2-4 Distance from Ms. X 4 5 5 COLLABORATIVE FILTERING Amy is the closest match. We can look in her history and see, for example, that she gave five stars to Paolo Bacigalupi's The Windup Girl and we would recommend that book to Ms. X. Euclidean Distance One benefit of Manhattan Distance is that it is fast to compute. If we are Facebook and are trying to find who among one million users is most similar to little Danny from Kalamazoo, fast is good. Pythagorean Theorem You may recall the Pythagorean Theorem from your distant educational past. Here, instead of finding the Manhattan Distance between Amy and Ms. X (which was 4) we are going to figure out the straight line, as-the-crow-flies, distance 2-5 The Pythagorean Theorem tells us how to compute that distance. c= c a p a 2 + b2 b This straight-line, as-the-crow-flies distance we are calling Euclidean Distance. The formula is p (x1 x2 )2 + (y1 y2 ) 2 Recall that x1 is how well person 1 liked Dragon Tattoo and x2 is how well person 2 liked it; y1 is how well person 1 liked Snow Crash and y2 is how well person 2 liked it. Amy rated both Snow Crash and Dragon Tattoo a 5; The elusive Ms. X rated Dragon Tattoo a 2 and Snow Crash a 4. So the Euclidean distance between p (5 2)2 + (5 4)2 = p 32 + 12 = p 10 = 3.16 Computing the rest of the distances we get Amy Bill Jim 2-6 Distance from Ms. X 3.16 3.61 3.61 COLLABORATIVE FILTERING N-dimensional thinking Let's branch out slightly from just looking at rating two books (and hence 2D) to looking at something slightly more complex. Suppose we work for an online streaming music service and we want to make the experience more compelling by recommending bands. Let's say users can rate bands on a star system 1-5 stars and they can give half star ratings (for example, you can give a band 2.5 stars). The following chart shows 8 users and their ratings of eight bands. Blues Traveler Broken Bells Deadmau5 Norah Jones Phoenix Slightly Stoopid The Strokes Vampire Weekend Angelica 3.5 2 4.5 5 1.5 2.5 2 Bill 2 3.5 4 2 3.5 3 Chan 5 1 1 3 5 1 - Dan 3 4 4.5 3 4.5 4 2 Hailey 4 1 4 4 1 Jordyn 4.5 4 5 5 4.5 4 4 Sam 5 2 3 5 4 5 - Veronica 3 5 4 2.5 3 - The hyphens in the table indicate that a user didn't rate that particular band. For now we are going to compute the distance based on the number of bands they both reviewed. So, for example, when computing the distance between Angelica and Bill, we will use the ratings for Blues Traveler, Broken Bells, Phoenix, Slightly Stoopid, and Vampire Weekend. So the Manhattan Distance would be: 2-7 Blues Traveler Broken Bells Deadmau5 Norah Jones Phoenix Slightly Stoopid The Strokes Vampire Weekend Manhattan Distance: Angelica 3.5 2 4.5 5 1.5 2.5 2 Bill 2 3.5 4 2 3.5 3 Difference 1.5 1.5 3 2 1 9 The Manhattan Distance row, the last row of the table, is simply the sum of the differences: (1.5 + 1.5 + 3 + 2 + 1). Computing the Euclidean Distance is similar. We only use the bands they both reviewed: Blues Traveler Broken Bells Deadmau5 Norah Jones Phoenix Slightly Stoopid The Strokes Vampire Weekend Sum of squares Euclidean Distance 2-8 Angelica 3.5 2 4.5 5 1.5 2.5 2 Bill 2 3.5 4 2 3.5 3 Difference 1.5 1.5 Difference2 2.25 2.25 3 2 1 9 4 1 18.5 4.3 COLLABORATIVE FILTERING To parse that out a bit more: Euclidean = (3.5 − 2)2 +(2 − 3.5)2 + (5 − 2)2 + (1.5 − 3.5)2 + (2 − 3)2 = 1.5 2 + (−1.5)2 + 32 + (−2)2 + (−1)2 = 2.25 + 2.25 + 9 + 4 + 1 = 18.5 = 4.3 Got it? Try an example on your own... 2-9 Blues Traveler Broken Bells Deadmau5 Norah Jones Phoenix Slightly Stoopid The Strokes Vampire Weekend s Angelica 3.5 2 4.5 5 1.5 2.5 2 Bill 2 3.5 4 2 3.5 3 Chan 5 1 1 3 5 1 - Dan 3 4 4.5 3 4.5 4 2 Hailey 4 1 4 4 1 Jordyn 4.5 4 5 5 4.5 4 4 Sam 5 2 3 5 4 5 - sharpen your pencil Compute the Euclidean Distance between Hailey and Veronica. Compute the Euclidean Distance between Hailey and Jordyn 2-10 Veronica 3 5 4 2.5 3 - COLLABORATIVE FILTERING s sharpen your pencil - solution Compute the Euclidean Distance between Hailey and Veronica. p = (4 5)2 + (4 3)2 = p 1+1= p 2 = 1.414 Compute the Euclidean Distance between Hailey and Jordyn = = = p p (4 4.5)2 + (1 4)2 + (4 5)2 + (4 4)2 + (1 4)2 ( 0.5)2 + ( 3)2 + ( 1)2 + (0)2 + ( 3)2 p .25 + 9 + 1 + 0 + 9 = p 19.25 = 4.387 A flaw It looks like we discovered a flaw with using these distance measures. When we computed the distance between Hailey and Veronica, we noticed they only rated two bands in common (Norah Jones and The Strokes), whereas when we computed the distance between Hailey and Jordyn, we noticed they rated five bands in common. This seems to skew our distance measurement, since the Hailey-Veronica distance is in 2 dimensions while the Hailey-Jordyn 2-11 distance is in 5 dimensions. Manhattan Distance and Euclidean Distance work best when there are no missing values. Dealing with missing values is an active area of scholarly research. Later in the book we will talk about how to deal with this problem. For now just be aware of the flaw as we continue our first exploration into building a recommendation system. A generalization We can generalize Manhattan Distance and Euclidean Distance to what is called the Minkowski Distance Metric: n 1 r r d(x, y) = (∑ | xk − yk | ) k=1 When • r = 1: The formula is Manhattan Distance. • r = 2: The formula is Euclidean Distance • r = ∞: Supremum Distance h Arghhhh Math! When you see formulas like this in a book you have several options. One option is to see the formula-brain neurons fire that say math formula--and then you quickly skip over it to the next English bit. I have to admit that I was once a skipper. The other option is to see the formula, pause, and dissect it. 2-12 COLLABORATIVE FILTERING Many times you’ll find the formula quite understandable. Let’s dissect it now. When r = 1 the formula reduces to Manhattan Distance: d(x, y) = ∑ k=1 | xk − yk | n So for the music example we have been using throughout the chapter, x and y represent two people and d(x, y) represents the distance between them. n is the number of bands they both rated (both x and y rated that band). We’ve done that calculation a few pages back: Blues Traveler Broken Bells Deadmau5 Norah Jones Phoenix Slightly Stoopid The Strokes Vampire Weekend Manhattan Distance: Angelica 3.5 2 4.5 5 1.5 2.5 2 Bill 2 3.5 4 2 3.5 3 Difference 1.5 1.5 3 2 1 9 That difference column represents the absolute value of the difference and we sum those up to get 9. When r = 2, we get the Euclidean distance: d(x, y) = ∑ n k=1 (xk − yk )2 2-13 Here’s the scoop! The greater the r, the more a large difference in one dimension will influence the total difference. Representing the data in Python (finally some coding) There are several ways of representing the data in the table above using Python. I am going to use Python's dictionary (also called an associative array or hash table): Remember, All the code for the book is available at www.guidetodatamining.com. 2-14 COLLABORATIVE FILTERING users = {"Angelica": {"Blues Traveler": 3.5, "Broken Bells": 2.0, ! ! "Norah Jones": 4.5, "Phoenix": 5.0, ! "Slightly Stoopid": 1.5, "The Strokes": 2.5, "Vampire Weekend": 2.0}, "Bill": {"Blues Traveler": 2.0, "Broken Bells": 3.5, "Deadmau5": 4.0, "Phoenix": 2.0, "Slightly Stoopid": 3.5, "Vampire Weekend": 3.0}, "Chan": {"Blues Traveler": 5.0, "Broken Bells": 1.0, "Deadmau5": 1.0, "Norah Jones": 3.0, "Phoenix": 5, "Slightly Stoopid": 1.0}, "Dan": {"Blues Traveler": 3.0, "Broken Bells": 4.0, "Deadmau5": 4.5, "Phoenix": 3.0, "Slightly Stoopid": 4.5, "The Strokes": 4.0, "Vampire Weekend": 2.0}, "Hailey": {"Broken Bells": 4.0, "Deadmau5": 1.0, "Norah Jones": 4.0, "The Strokes": 4.0, "Vampire Weekend": 1.0}, "Jordyn": {"Broken Bells": 4.5, "Deadmau5": 4.0, "Norah Jones": 5.0, "Phoenix": 5.0, "Slightly Stoopid": 4.5, "The Strokes": 4.0, "Vampire Weekend": 4.0}, "Sam": {"Blues Traveler": 5.0, "Broken Bells": 2.0, "Norah Jones": 3.0, "Phoenix": 5.0, "Slightly Stoopid": 4.0, "The Strokes": 5.0}, "Veronica": {"Blues Traveler": 3.0, "Norah Jones": 5.0, "Phoenix": 4.0, "Slightly Stoopid": 2.5, "The Strokes": 3.0}} We can get the ratings of a particular user as follows: >>> users["Veronica"] {"Blues Traveler": 3.0, "Norah Jones": 5.0, "Phoenix": 4.0, "Slightly Stoopid": 2.5, "The Strokes": 3.0} >>> 2-15 The code to compute Manhattan distance I'd like to write a function that computes the Manhattan distance as follows: def manhattan(rating1, rating2): """Computes the Manhattan distance. Both rating1 and rating2 are dictionaries of the form {'The Strokes': 3.0, 'Slightly Stoopid': 2.5 ...""" distance = 0 for key in rating1: if key in rating2: distance += abs(rating1[key] - rating2[key]) return distance To test the function: >>> manhattan(users['Hailey'], users['Veronica']) 2.0 >>> manhattan(users['Hailey'], users['Jordyn']) 7.5 >>> Now a function to find the closest person (actually this returns a sorted list with the closest person first): def computeNearestNeighbor(username, users): """creates a sorted list of users based on their distance to username""" distances = [] for user in users: if user != username: distance = manhattan(users[user], users[username]) distances.append((distance, user)) # sort based on distance -- closest first distances.sort() return distances 2-16 COLLABORATIVE FILTERING And just a quick test of that function: >>> computeNearestNeighbor("Hailey", users) [(2.0, ''Veronica'), (4.0, 'Chan'),(4.0, 'Sam'), (4.5, 'Dan'), (5.0, 'Angelica'), (5.5, 'Bill'), (7.5, 'Jordyn')] Finally, we are going to put this all together to make recommendations. Let's say I want to make recommendations for Hailey. I find her nearest neighbor—Veronica in this case. I will then find bands that Veronica has rated but Hailey has not. Also, I will assume that Hailey would have rated the bands the same as (or at least very similar to) Veronica. For example, Hailey has not rated the great band Phoenix. Veronica has rated Phoenix a '4' so we will assume Hailey is likely to enjoy the band as well. Here is my function to make recommendations. def recommend(username, users): """Give list of recommendations""" # first find nearest neighbor nearest = computeNearestNeighbor(username, users)[0][1] recommendations = [] # now find bands neighbor rated that user didn't neighborRatings = users[nearest] userRatings = users[username] for artist in neighborRatings: if not artist in userRatings: recommendations.append((artist, neighborRatings[artist])) # using the fn sorted for variety - sort is more efficient return sorted(recommendations, key=lambda artistTuple: artistTuple[1], reverse = True) And now to make recommendations for Hailey: >>> recommend('Hailey', users) [('Phoenix', 4.0), ('Blues Traveler', 3.0), ('Slightly Stoopid', 2.5)] That fits with our expectations. As we saw above, Hailey's nearest neighbor was Veronica and Veronica gave Phoenix a '4'. Let's try a few more: >>> recommend('Chan', users) 2-17 [('The Strokes', 4.0), ('Vampire Weekend', 1.0)] >>> recommend('Sam', users) [('Deadmau5', 1.0)] We think Chan will like The Strokes and also predict that Sam will not like Deadmau5. >>> recommend('Angelica', users) [] Hmm. For Angelica we got back an empty set meaning we have no recommendations for her. Let us see what went wrong: >>> computeNearestNeighbor('Angelica', users) [(3.5, 'Veronica'), (4.5, 'Chan'), (5.0, 'Hailey'), (8.0, 'Sam'), (9.0, 'Bill'), (9.0, 'Dan'), (9.5, 'Jordyn')] Angelica's nearest neighbor is Veronica. When we look at their ratings: Blues Traveler Broken Bells Deadmau5 Norah Jones Phoenix Slightly Stoopid The Strokes Vampire Weekend Angelica 3.5 2 4.5 5 1.5 2.5 2 Bill 2 3.5 4 2 3.5 3 Chan 5 1 1 3 5 1 - Dan 3 4 4.5 3 4.5 4 2 Hailey 4 1 4 4 1 Jordyn 4.5 4 5 5 4.5 4 4 Sam 5 2 3 5 4 5 - Veronica 3 5 4 2.5 3 - We see that Angelica rated every band that Veronica did. We have no new ratings, so no recommendations. Shortly, we will see how to improve the system to avoid these cases. 2-18 COLLABORATIVE FILTERING s exercise 1) Implement the Minkowski Distance function. 2) Alter the computeNearestNeighbor function to use Minkowski Distance. 2-19 s exercise - solution 1) Implement the Minkowski Distance function. def minkowski(rating1, rating2, r): """Computes the Minkowski distance. Both rating1 and rating2 are dictionaries of the form {'The Strokes': 3.0, 'Slightly Stoopid': 2.5}""" distance = 0 commonRatings = False for key in rating1: if key in rating2: distance += pow(abs(rating1[key] - rating2[key]), r) commonRatings = True if commonRatings: return pow(distance, 1/r) else: return 0 #Indicates no ratings in common 2) Alter the computeNearestNeighbor function to use Minkowski Distance. just need to alter the distance = line to distance = minkowski(users[user], users[username], 2) (the 2 as the r argument implements Euclidean) 2-20 COLLABORATIVE FILTERING Blame the users Let's take a look at the user ratings in a bit more detail. We see that users have very different behaviors when it comes to rating bands Jordyn seems to like everthing. Her ratings range from 4 to 5. Bill seems to avoid the extremes. His ratings range from 2 to 4 Blues Traveler Broken Bells Deadmau5 Norah Jones Phoenix Slightly Stoopid The Strokes Vampire Weekend Angelica 3.5 2 4.5 5 1.5 2.5 2 Bill 2 3.5 4 2 3.5 3 Chan 5 1 1 3 5 1 - Dan 3 4 4.5 3 4.5 4 2 Hailey 4 1 4 4 1 Jordyn 4.5 4 5 5 4.5 4 4 Sam 5 2 3 5 4 5 - Veronica 3 5 4 2.5 3 - Hailey is a binary person giving either 1s or 4s to bands. 2-21 So how do we compare, for example, Hailey to Jordan? Does Hailey's '4' mean the same as Jordyn's '4' or Jordyn's '5'? I would guess it is more like Jordyn's '5'. This variability can create problems with a recommendation system. I absolutely love Broken Bells! They’re tight! I give them a 4. 2-22 Broken Bells is ok. I’d give them a ‘4’. COLLABORATIVE FILTERING Pearson Correlation Coefficient One way to fix this problem is to use the Pearson Correlation Coefficient. First, the general idea. Consider the following data (not from the data set above): Blues Traveler Norah Jones Phoenix The Strokes Weird Al Clara 4.75 4.5 5 4.25 4 Robert 4 3 5 2 1 This is an example of what is called 'grade inflation' in the data mining community. Clara's lowest rating is 4—all her rating are between 4 and 5. If we are to graph this chart it would look like Phoenix 5 Blues Traveler Norah Clara 4.5 The Strokes 4 Weird Al 3.5 3 1 2 3 4 5 Robert Straight line = Perfect Agreement!!! 2-23 The fact that this is a straight line indicates a perfect agreement between Clara and Robert. Both rated Phoenix as the best band, Blues Traveler next, Norah Jones after that, and so on. As Clara and Robert agree less, the less the data points reside on a straight line: Pretty Good Agreement: 5 4.5 Phoenix Weird Al Norah Jones Blues Traveler Clara 4 The Strokes 3.5 3 1 2 3 4 5 Robert Not So Good Agreement: 5 4.5 Norah Jones Weird Al Clara 4 The Strokes Phoenix 3.5 3 Blues Traveler 1 2 3 Robert 2-24 4 5 COLLABORATIVE FILTERING So chart-wise, perfect agreement is indicated by a straight line. The Pearson Correlation Coefficient is a measure of correlation between two variables (in this specific case the correlation between Angelica and Bill). It ranges between -1 and 1 inclusive. 1 indicates perfect agreement. -1 indicates perfect disagreement. To give you a general feel for this, the chart above with the straight line has a Pearson of 1, the chart above that I labelled ‘pretty good agreement’ has a Pearson of 0.91, and the ‘not so good agreement’ chart has a Pearson of 0.81 So we can use this to find the individual who is most similar to the person we are interested in. The formula for the Pearson Correlation Coefficient is r= h ∑ ∑ n i=1 n i=1 (xi − x )(yi − y ) (xi − x )2 ∑ n i=1 (yi − y )2 Arghhhh Math Again! Here's a personal confession. I have a Bachelor of Fine Arts degree in music. While I have taken courses in ballet, modern dance, and costume design, I did not have a single math course as an undergrad. Before that, I attended an all boys trade high school where I took courses in plumbing and automobile repair, but no courses in math other than the basics. Either due to this background or some innate wiring in my brain, when I read a book that has formulas like the one above, I tend to skip over the formulas and continue with the text below them. If you are like me I would urge you to fight 2-25 that urge and actually look at the formula. Many formulas that on a quick glimpse look complex are actually understandable by mere mortals. Other than perhaps looking complex, the problem with the formula above is that the algorithm to implement it would require multiple passes through the data. Fortunately for us algorithmic people, there is an alternative formula, which is an approximation of Pearson: ∑ r= n ∑ xy − i=1 i i (∑ i=1 xi )2 x y i=1 i ∑ i=1 i n n n n ∑ n x2 − i=1 i n (∑ i=1 yi )2 n ∑ n y − 2 i=1 i n (Remember what I said two paragraphs above about not skipping over formulas) This formula, in addition to looking initially horribly complex is, more importantly, numerically unstable meaning that what might be a small error is amplified by this reformulation. The big plus is that we can implement it using a single-pass algorithm, which we will get to shortly. First, let’s dissect this formula and work through the example we saw a few pages back: Blues Traveler Norah Jones Phoenix The Strokes Weird Al Clara 4.75 4.5 5 4.25 4 Robert 4 3 5 2 1 To start with, let us compute 2-26 COLLABORATIVE FILTERING ∑ n xy i=1 i i Which is in the first expression in the numerator. Here the x and y represent Clara and Robert. Blues Traveler Norah Jones Phoenix The Strokes Weird Al Clara 4.75 4.5 5 4.25 4 Robert 4 3 5 2 1 For each band we are going to multiple Clara’s and Robert’s rating together and sum the results: (4.75 × 4) + (4.5 × 3) + (5 × 5) + (4.25 × 2) + (4 × 1) = 19 + 13.5 + 25 + 8.5 + 4 = 70 Sweet! Now let’s compute the rest of the numerator: ∑ x y i=1 i ∑ i=1 i n n n 2-27 So the ∑ n x i=1 i is the sum of Clara’s ratings, which is 22.5. The sum of Robert’s is 15 and they rated 5 bands: 22.5 × 15 = 67.5 5 So the numerator in the formula on page 26 is 70 - 67.5 = 2.5 Now let’s dissect the denominator. (∑ i=1 xi )2 n ∑ n x − 2 i=1 i n First, Blues Traveler Norah Jones Phoenix The Strokes Weird Al Clara 4.75 4.5 5 4.25 4 Robert 4 3 5 2 1 ∑ n x = (4.75)2 + (4.5)2 + (5)2 + (4.25)2 + (4)2 = 101.875 2 i=1 i 2-28 COLLABORATIVE FILTERING We’ve already computed the sum of Clara’s ratings, which is 22.5. Square that and we get 506.25. We divide that by the number of co-rated bands (5) and we get 101.25. Putting that together: 101.875 − 101.25 = .625 = .79057 Next we do the same computation for Robert: (∑ i=1 yi )2 n ∑ n y − 2 i=1 i n = 55 − 45 = 3.162277 Putting this altogether we get: r= 2.5 2.5 = = 1.00 .79057(3.162277) 2.5 So 1 means there was perfect agreement between Clara and Robert! Take a break before moving on!! 2-29 s exercise Before going to the next page, implement the algorithm in Python. You should get the following results. >>> pearson(users['Angelica'], users['Bill']) -0.90405349906826993 >>> pearson(users['Angelica'], users['Hailey']) 0.42008402520840293 >>> pearson(users['Angelica'], users['Jordyn']) 0.76397486054754316 >>> For this implementation you will need 2 Python functions sqrt (square root) and power operator ** which raises its left argument to the power of its right argument: >>> from math import sqrt >>> sqrt(9) 3.0 >>> 3**2 9 2-30 COLLABORATIVE FILTERING s exercise - solution Here is my implementation of Pearson def pearson(rating1, rating2): sum_xy = 0 sum_x = 0 sum_y = 0 sum_x2 = 0 sum_y2 = 0 n = 0 for key in rating1: if key in rating2: n += 1 x = rating1[key] y = rating2[key] sum_xy += x * y sum_x += x sum_y += y sum_x2 += x**2 sum_y2 += y**2 # if no ratings in common return 0 if n == 0: return 0 # now compute denominator denominator = sqrt(sum_x2 - (sum_x**2) / n) * sqrt(sum_y2 - (sum_y**2) / n) if denominator == 0: return 0 else: return (sum_xy - (sum_x * sum_y) / n) / denominator 2-31 One last formula – Cosine Similarity I would like to present one last formula, which is very popular in text mining but also used in collaborative filtering—cosine similarity. To see when we might use this formula, let’s say I change my example slightly. We will keep track of the number of times a person played a particular song track and use that information to base our recommendations on. number of plays The Decemberists The King is Dead Radiohead The King of Limbs Katy Perry E.T. Ann 10 5 32 Ben 15 25 1 Sally 12 6 27 Just by eye-balling the above chart (and by using any of the distance formulas mentioned above) we can see that Sally is more similar in listening habits to Ann than Ben is. So what is the problem? I have around four thousand tracks in iTunes. Here is a snapshot of the top few ordered by number of plays: 2-32 COLLABORATIVE FILTERING So my top track is Moonlight Sonata by Marcus Miller with 25 plays. Chances are that you have played that track zero times. In fact, chances are good that you have not played any of my top tracks. In addition, there are over 15 million tracks in iTunes and I have only four thousand. So the data for a single person is sparse since it has relatively few non-zero attributes (plays of a track). When we compare two people by using the number of plays of the 15 million tracks, mostly they will have shared zeros in common. However, we do not want to use these shared zeros when we are computing similarity. A similar case can be made when we are comparing text documents using words. Suppose we liked a certain book, say Tom Corbett Space Cadet: The Space Pioneers by Carey Rockwell and we want to find a similar book. One possible way is to use word frequency. The attributes will be individual words and the values of those attributes will be the frequency of those words in the book. So 6.13% of the words in The Space Pioneers are occurrences of the word the, 0.89% are the word Tom, 0.25% of the words are space. I can compute the similarity of this book to others by using these word frequencies. However, the same problem related to sparseness of data occurs here. There are 6,629 unique words in The Space Pioneers and there are a bit over one million unique words in English. So if our attributes are English words, there will be relatively few non-zero attributes for The Space Pioneers or any other book. Again, any measure of similarity should not depend on the shared-zero values. 2-33 Cosine similarity ignores 0-0 matches. It is defined as x⋅y x × y cos(x, y) = where · indicates the dot product and ||x|| indicates the length of the vector x. The length of a vector is ∑ n 2 i=1 i x Let’s give this a try with the perfect agreement example used above: Blues Traveler Norah Jones Phoenix The Strokes Weird Al Clara 4.75 4.5 5 4.25 4 Robert 4 3 5 2 1 The two vectors are: x = (4.75, 4.5,5, 4.25, 4) y = (4, 3,5,2,1) then x = 4.75 2 + 4.5 2 + 5 2 + 4.25 2 + 4 2 = 101.875 = 10.09 y = 4 2 + 32 + 5 2 + 2 2 + 12 = 55 = 7.416 The dot product is 2-34 COLLABORATIVE FILTERING x ⋅ y = (4.75 × 4) + (4.5 × 3) + (5 × 5) + (4.25 × 2) + (4 × 1) = 70 And the cosine similarity is cos(x, y) = 70 70 = = 0.935 10.093 × 7.416 74.85 The cosine similarity rating ranges from 1 indicated perfect similarity to -1 indicate perfect negative similarity. So 0.935 represents very good agreement. s sharpen your pencil Compute the Cosine Similarity between Angelica and Veronica (from our dataset). (Consider dashes equal to zero) Blues Traveler Broken Bells Deadmau 5 Norah Jones Phoenix Slightly Stoopid The Strokes Vampire Weekend Angelica 3.5 2 - 4.5 5 1.5 2.5 2 Veronica 3 - - 5 4 2.5 3 - 2-35 s sharpen your pencil - solution Compute the Cosine Similarity between Angelica and Veronica (from our dataset). Blues Traveler Broken Bells Deadmau 5 Norah Jones Phoenix Slightly Stoopid The Strokes Vampire Weekend Angelica 3.5 2 - 4.5 5 1.5 2.5 2 Veronica 3 - - 5 4 2.5 3 - x = (3.5,2,0, 4.5,5,1.5,2.5,2) y = (3,0,0,5, 4,2.5, 3,0) x = 3.5 2 + 2 2 + 0 2 + 4.5 2 + 5 2 + 1.5 2 + 2.5 2 + 2 2 = 74 = 8.602 y = 32 + 0 2 + 0 2 + 5 2 + 4 2 + 2.5 2 + 32 + 0 2 = 65.25 = 8.078 The dot product is x⋅y = (3.5 × 3) + (2 × 0) + (0 × 0) + (4.5 × 5) + (5 × 4) + (1.5 × 2.5) + (2.5 × 3) + (2 × 0) = 64.25 Cosine Similarity is cos(x, y) = 2-36 64.25 64.25 = = 0.9246 8.602 × 8.078 69.487 COLLABORATIVE FILTERING Which similarity measure to use? We will be exploring this question throughout the book. For now, here are a few helpful hints: If the data is subject to grade-inflation (different users may be using different scales) use Pearson. Good job, guys, nailed it! If your data is dense (almost all attributes have nonzero values) and the magnitude of the attribute values is important, use distance measures such as Euclidean or Manhattan. If the data is sparse consider using Cosine Similarity. 2-37 So, if the data is dense (nearly all attributes have non-zero values) then Manhattan and Euclidean are reasonable to use. What happens if the data is not dense? Consider an expanded music rating system and three people, all of which have rated 100 songs on our site: Jake: hardcore fan of Co untry Linda and Eric: love, love, love 60s rock! Linda and Eric enjoy the same kind of music. In fact, among their ratings, they have 20 songs in common and the difference in their ratings of those 20 songs (on a scale of 1 to 5) averages only 0.5!! The Manhattan Distance between them would be 20 x .5 = 10. The Euclidean Distance would be: d = (.5)2 × 20 = .25 × 20 = 5 = 2.236 2-38 COLLABORATIVE FILTERING Linda and Jake have rated only one song in common: Chris Cagle’s What a Beautiful Day. Linda thought it was okay and rated it a 3, Jake thought it was awesome and gave it a 5. So the Manhattan Distance between Jake and Linda is 2 and the Euclidean Distance is d = (3 − 5)2 = 4 = 2 So both the Manhattan and Euclidean Distances show that Jake is a closer match to Linda than Eric is. So in this case both distance measures produce poor results. Hey, I have an idea that might fix this problem. Right now, people rate tunes on a scale of 1 to 5. How about for the tunes people don’t rate I will assume the rating is 0. That way we solve the problem of sparse data as every object has a value for every attribute! Good idea, but that doesn’t work either. To see why we need to bring in a few more characters into our little drama: Cooper and Kelsey. Jake, Cooper and Kelsey have amazingly similar musical tastes. Jake has rated 25 songs on our site. 2-39 Cooper has rated 26 songs, and 25 of them are the same songs Jake rated. They love the same kind of music and the average distance in their ratings is only 0.25!! Kelsey loves both music and our site and has rated 150 songs. 25 of those songs are the same as the ones Cooper and Jake rated. Like Cooper, the average distance in her ratings and Jake’s is only 0.25!! Our gut feeling is that Cooper and Kelsey are equally close matches to Jake. Now consider our modified Manhattan and Euclidean distance formulas where we assign a 0 for every song the person didn’t rate. Cooper With this scheme, Cooper is a much closer match to Jake than Kelsey is. Why? Kelsey 2-40 COLLABORATIVE FILTERING To answer why, let us look at a the following simplified example (again, a 0 means that person did not rate that song): Song: 1 2 3 4 5 7 6 8 9 10 Jake 0 0 0 4.5 5 4.5 0 0 0 0 Cooper 0 0 4 5 5 5 0 0 0 0 Kelsey 5 4 4 5 5 5 5 5 4 4 Again, looking at the songs they mutually rated (songs 4, 5, and 6), Cooper and Kelsey seem like equally close matches for Jake. However, Manhattan Distance using those zero values tells a different story: dCooper,Jake = (4 − 0) + (5 − 4.5) + (5 − 5) + 5 − 4.5) = 4 + 0.5 + 0 + 0.5 = 5 dKelsey,Jake = (5 − 0) + (4 − 0) + (4 − 0) + (5 − 4.5) + (5 − 5) + (5 − 4.5) + (5 − 0) +(5 − 0) + (4 − 0) + (4 − 0) = 5 + 4 + 4 + 0.5 + 0 + .5 + 5 + 5 + 4 + 4 = 32 The problem is that these zero values tend to dominate any measure of distance. So the solution of adding zeros is no better than the original distance formulas. One workaround people have used is to compute—in some sense—an ‘average’ distance where one computes the distance by using songs they rated in common divided that by the number of songs they rated in common. Again, Manhattan and Euclidean work spectacularly well on dense data, but if the data is sparse it may be better to use Cosine Similarity. 2-41 Weirdnesses Suppose we are trying to make recommendations for Amy who loves Phoenix, Passion Pit and Vampire Weekend. Our closest match is Bob who also loves Phoenix, Passion Pit, and Vampire Weekend. His father happens to play accordion for the Walter Ostanek Band, this year's Grammy winner in the polka category. Because of familial obligations, Bob gives 5 stars to the Walter Ostanek Band. Based on our current recommendation system, we think Amy will absolutely love the band. But common sense tells us she probably won't. Or think of Professor Billy Bob Olivera who loves to read data mining books and science fiction. His closest match happens to be me, who also likes data mining books and science fiction. However, I like standard poodles and have rated The Secret Lives of Standard Poodles highly. Our current recommendation system would likely recommend that book to the professor. 2-42 COLLABORATIVE FILTERING The problem is that we are relying on a single “most similar” person. Any quirk that person has is passed on as a recommendation. One way of evening out those quirks is to base our recommendations on more than one person who is similar to our user. For this we can use the k-nearest neighbor approach. K-nearest neighbor In the k-nearest neighbor approach to collaborative filtering we use k most similar people to determine recommendations. The best value for k is application specific—you will need to do some experimentation. Here's an example to give you the basic idea. Suppose I would like to make recommendations for Ann and am using k-nearest neighbor with k=3. The three nearest neighbors and their Pearson scores are shown in the following table: 2-43 Person Sally Eric Amanda Pearson 0.8 0.7 0.5 .0 .5 = 2 0 + 7 0. 0.8 + Each of these three people are going to influence the recommendations. The question is how can I determine how much influence each person should have. If there is a Pie of Influence™, how big a slice should I give each person? If I add up the Pearson scores I get 2. Sally's share is 0.8/2 or 40%. Eric's share is 35% (0.7 / 2) and Amanda's share is 25%. Suppose Amanda, Eric, and Sally, rated the band, The Grey Wardens as follows Person Amanda Eric Sally 2-44 Grey Wardens Rating 4.5 5 3.5 COLLABORATIVE FILTERING Person Amanda Eric Sally Grey Wardens Rating 4.5 5 3.5 Influence 25.00% 35.00% 40.00% Projected rating = (4.5 x 0.25) + (5 x 0.35) + (3.5 x 0.4) = 4.275 s sharpen your pencil Suppose I use the same data as above but use a k-nearest neighbor approach with k=2. What is my projected rating for Grey Wardens? Person Sally Eric Amanda Person Amanda Eric Sally Pearson 0.8 0.7 0.5 Grey Wardens Rating 4.5 5 3.5 2-45 s solution Person Sally Eric Amanda Pearson 0.8 0.7 0.5 Person Amanda Eric Sally Grey Wardens Rating 4.5 5 3.5 Projected rating = Sally’s portion + Eric’s portion = (3.5 x (0.8 / 1.5)) + (5 x (0.7 / 1.5)) = (3.5 x .5333) + (5 x 0.4667) = 1.867 + 2.333 = 4.2 2-46 COLLABORATIVE FILTERING A Python Recommendation Class I combined some of what we covered in this chapter in a Python Class. Even though it is slightly long I have included the code here (don't forget you can download the code at http:// www.guidetodatamining.com). import codecs from math import sqrt users = {"Angelica": {"Blues Traveler": 3.5, "Broken Bells": 2.0, "Norah Jones": 4.5, "Phoenix": 5.0, "Slightly Stoopid": 1.5, "The Strokes": 2.5, "Vampire Weekend": 2.0}, "Bill":{"Blues Traveler": 2.0, "Broken Bells": 3.5, "Deadmau5": 4.0, "Phoenix": 2.0, "Slightly Stoopid": 3.5, "Vampire Weekend": 3.0}, "Chan": {"Blues Traveler": 5.0, "Broken Bells": 1.0, "Deadmau5": 1.0, "Norah Jones": 3.0, "Phoenix": 5, "Slightly Stoopid": 1.0}, "Dan": {"Blues Traveler": 3.0, "Broken Bells": 4.0, "Deadmau5": 4.5, "Phoenix": 3.0, "Slightly Stoopid": 4.5, "The Strokes": 4.0, "Vampire Weekend": 2.0}, "Hailey": {"Broken Bells": 4.0, "Deadmau5": 1.0, "Norah Jones": 4.0, "The Strokes": 4.0, "Vampire Weekend": 1.0}, "Jordyn": {"Broken Bells": 4.5, "Deadmau5": 4.0, "Norah Jones": 5.0, "Phoenix": 5.0, "Slightly Stoopid": 4.5, "The Strokes": 4.0, "Vampire Weekend": 4.0}, 2-47 "Sam": {"Blues Traveler": 5.0, "Broken Bells": 2.0, "Norah Jones": 3.0, "Phoenix": 5.0, "Slightly Stoopid": 4.0, "The Strokes": 5.0}, "Veronica": {"Blues Traveler": 3.0, "Norah Jones": 5.0, "Phoenix": 4.0, "Slightly Stoopid": 2.5, "The Strokes": 3.0} } class recommender: def __init__(self, data, k=1, metric='pearson', n=5): """ initialize recommender currently, if data is dictionary the recommender is initialized to it. For all other data types of data, no initialization occurs k is the k value for k nearest neighbor metric is which distance formula to use n is the maximum number of recommendations to make""" self.k = k self.n = n self.username2id = {} self.userid2name = {} self.productid2name = {} # for some reason I want to save the name of the metric self.metric = metric if self.metric == 'pearson': self.fn = self.pearson # # if data is dictionary set recommender data to it # if type(data).__name__ == 'dict': self.data = data 2-48 COLLABORATIVE FILTERING def convertProductID2name(self, id): """Given product id number return product name""" if id in self.productid2name: return self.productid2name[id] else: return id def userRatings(self, id, n): """Return n top ratings for user with id""" print ("Ratings for " + self.userid2name[id]) ratings = self.data[id] print(len(ratings)) ratings = list(ratings.items()) ratings = [(self.convertProductID2name(k), v) for (k, v) in ratings] # finally sort and return ratings.sort(key=lambda artistTuple: artistTuple[1], reverse = True) ratings = ratings[:n] for rating in ratings: print("%s\t%i" % (rating[0], rating[1])) def loadBookDB(self, path=''): """loads the BX book dataset. Path is where the BX files are located""" self.data = {} i = 0 # # First load book ratings into self.data # f = codecs.open(path + "BX-Book-Ratings.csv", 'r', 'utf8') for line in f: i += 1 2-49 # separate line into fields fields = line.split(';') user = fields[0].strip('"') book = fields[1].strip('"') rating = int(fields[2].strip().strip('"')) if user in self.data: currentRatings = self.data[user] else: currentRatings = {} currentRatings[book] = rating self.data[user] = currentRatings f.close() # # Now load books into self.productid2name # Books contains isbn, title, and author among other fields # f = codecs.open(path + "BX-Books.csv", 'r', 'utf8') for line in f: i += 1 # separate line into fields fields = line.split(';') isbn = fields[0].strip('"') title = fields[1].strip('"') author = fields[2].strip().strip('"') title = title + ' by ' + author self.productid2name[isbn] = title f.close() # # Now load user info into both self.userid2name and # self.username2id # f = codecs.open(path + "BX-Users.csv", 'r', 'utf8') for line in f: i += 1 # separate line into fields fields = line.split(';') userid = fields[0].strip('"') 2-50 COLLABORATIVE FILTERING location = fields[1].strip('"') if len(fields) > 3: age = fields[2].strip().strip('"') else: age = 'NULL' if age != 'NULL': value = location + ' (age: ' + age + ')' else: value = location self.userid2name[userid] = value self.username2id[location] = userid f.close() print(i) def pearson(self, rating1, rating2): sum_xy = 0 sum_x = 0 sum_y = 0 sum_x2 = 0 sum_y2 = 0 n = 0 for key in rating1: if key in rating2: n += 1 x = rating1[key] y = rating2[key] sum_xy += x * y sum_x += x sum_y += y sum_x2 += pow(x, 2) sum_y2 += pow(y, 2) if n == 0: return 0 # now compute denominator denominator = (sqrt(sum_x2 - pow(sum_x, 2) / n) * sqrt(sum_y2 - pow(sum_y, 2) / n)) 2-51 if denominator == 0: return 0 else: return (sum_xy - (sum_x * sum_y) / n) / denominator def computeNearestNeighbor(self, username): """creates a sorted list of users based on their distance to username""" distances = [] for instance in self.data: if instance != username: distance = self.fn(self.data[username], self.data[instance]) distances.append((instance, distance)) # sort based on distance -- closest first distances.sort(key=lambda artistTuple: artistTuple[1], reverse=True) return distances def recommend(self, user): """Give list of recommendations""" recommendations = {} # first get list of users ordered by nearness nearest = self.computeNearestNeighbor(user) # # now get the ratings for the user # userRatings = self.data[user] # # determine the total distance totalDistance = 0.0 for i in range(self.k): totalDistance += nearest[i][1] # now iterate through the k nearest neighbors # accumulating their ratings for i in range(self.k): 2-52 COLLABORATIVE FILTERING # compute slice of pie weight = nearest[i][1] / totalDistance # get the name of the person name = nearest[i][0] # get the ratings for this person neighborRatings = self.data[name] # get the name of the person # now find bands neighbor rated that user didn't for artist in neighborRatings: if not artist in userRatings: if artist not in recommendations: recommendations[artist] = (neighborRatings[artist] * weight) else: recommendations[artist] = (recommendations[artist] + neighborRatings[artist] * weight) # now make list from dictionary recommendations = list(recommendations.items()) recommendations = [(self.convertProductID2name(k), v) for (k, v) in recommendations] # finally sort and return recommendations.sort(key=lambda artistTuple: artistTuple[1], reverse = True) # Return the first n items return recommendations[:self.n] 2-53 Example of this program executing First, I will construct an instance of the recommender class with the data we previously used: >>> r = recommender(users) Some simple examples using these band ratings: >>> r.recommend('Jordyn') [('Blues Traveler', 5.0)] >>> r.recommend('Hailey') [('Phoenix', 5.0), ('Slightly Stoopid', 4.5)] A New Dataset Ok, it is time to look at a more realistic dataset. Cai-Nicolas Zeigler collected over one million ratings of books from the Book Crossing website. This ratings are of 278,858 users rating 271,379 books. This anonymized data is available at http://www.informatik.uni-freiburg.de/ ~cziegler/BX/ both as an SQL dump and a text file of comma-separated-values (CSV). I had some problems loading this data into Python due to apparent character encoding problems. My fixed version of the CSV files are available on this book's website. The CSV files represent three tables: • BX-Users, which, as the name suggests, contains information about the users. There is an integer user-id field, as well as the location (i.e., Albuquerque, NM) and age. The names have been removed to anonymize the data. • BX-Books. Books are identified by the ISBN, book title, author, year of publication, and publisher. • BX-Book-Ratings, which includes a user-id, book ISBN, and a rating from 0-10. 2-54 COLLABORATIVE FILTERING The function loadBookDB in the recommender class loads the data from these files. Now I am going to load the book dataset. The argument to the loadBookDB function is the path to the BX book files. >>> r.loadBookDB('/Users/raz/Downloads/BX-Dump/') 1700018 Note: This is a large datase t an d may take a bit of time to loa d on yo ur co mputer. On my Hackintosh (2.8 GHz i7 860 with 8GB RAM ) it takes 24 seconds to loa d the dataset an d 30 se conds to run a quer y. Now I can get recommendations for user 17118, a person from Toronto: >>> r.recommend('171118') [("The Godmother's Web by Elizabeth Ann Scarborough", 10.0), ("The Irrational Season (The Crosswicks Journal, Book 3) by Madeleine L'Engle", 10.0), ("The Godmother's Apprentice by Elizabeth Ann Scarborough", 10.0), ("A Swiftly Tilting Planet by Madeleine L'Engle", 10.0), ('The Girl Who Loved Tom Gordon by Stephen King', 9.0), ('The Godmother by Elizabeth Ann Scarborough', 8.0)] >>> r.userRatings('171118', 5) Ratings for toronto, ontario, canada 2421 The Careful Writer by Theodore M. Bernstein! 10 Wonderful Life: The Burgess Shale and the Nature of History by Stephen Jay Gould! 10 Pride and Prejudice (World's Classics) by Jane Austen! 10 The Wandering Fire (The Fionavar Tapestry, Book 2) by Guy Gavriel Kay! 10 Flowering trees and shrubs: The botanical paintings of Esther Heins by Judith Leet! 10 2-55 Projects You won't really learn this material unless you play around with the code. Here are some suggestions of what you might try. 1. Implement Manhattan distance and Euclidean distance and compare the results of these three methods. 2. The book website has a file containing movie ratings for 25 movies. Create a function that loads the data into your classifier. The recommend method described above should recommend movies for a specific person. 2-56 Chapter 3: Collaborative filtering Implicit ratings and item based filtering In chapter 2 we learned the basics of collaborative filtering and recommendation systems. The algorithms described in that chapter are general purpose and could be used with a variety of data. Users rated different items on a five or ten point scale and the algorithms found other users who had similar ratings. As was mentioned, there is some evidence to suggest users typically do not use this fine-grain distinction and instead tend to either give the top rating or the lowest one. This all-or-nothing rating strategy can sometimes lead to unusable results. In this chapter we will examine ways to fine tune collaborative filtering to produce more accurate recommendations in an efficient manner. Explicit ratings One way of distinguishing types of user preferences is whether they are explicit or implicit. Explicit ratings are when the user herself explicitly rates the item. One example of this is the thumbs up / thumbs down rating on sites such as Pandora and YouTube. And Amazon’s star system: 3-2 COLLABORATIVE FILTERING Implicit Ratings For implicit ratings, we don't ask users to give any ratings—we just observe their behavior. An example of this is keeping track of what a user clicks on in the online New York Times. After observing what a user clicks on for a few weeks you can imagine that we could develop a reasonable profile of that user—she doesn't like sports but seems to like technology news. If the user clicks on the article “Fastest Way to Lose Weight Discovered by Professional Trainers” and the article “Slow and Steady: How to lose weight and keep it off” perhaps she wishes to lose weight. If she clicks on the iPhone ad, she perhaps has an interest in that product. (By the way, the term used when a user clicks on an ad is called 'click through'.) Consider what information we can gain from recording what products a user clicks on in Amazon. On your personalized Amazon front page this information is displayed: 3-3 In this example, Amazon keeps track of what people click on. It knows, for example, that people who viewed the book Jupiter’s Travels: Four years around the world on a Triumph also viewed the DVD Long Way Round, which chronicles the actor Ewan McGregor as he travels with his mate around the world on motorcycles. As can be seen in the Amazon screenshot above, this information is used to display the items in the section “Customers who viewed this also viewed.” Another implicit rating is what the customer actually buys. Amazon also keeps track of this information and uses it for their recommendations “Frequently Bought Together” and “Customers Who Viewed This Item Also Bought”: You would think that “Frequently Bought Together” would lead to some unusual recommendations but this works surprisingly well. 3-4 COLLABORATIVE FILTERING Imagine what information a program can acquire by monitoring your behavior in iTunes. First, there's the fact that I added a song to iTunes. That indicates minimally that I was interested enough in the song to do so. Then there is the Play Count information. In the image above, I've listened to Zee Avi's “Anchor” 52 times. That suggests that I like that song (and in fact I do). If I have a song in my library for awhile and only listened to it once, that might indicate that I don't like the song. k brain calisthenics Do you think having a user explicitly give a rating to an item is more accurate? Or do you think watching what a user buys or does (for example, the play count) is a more accurate judge of what an individual likes? 3-5 Jim Explicit Rating: match.com bio: I am a vegan. I enjoy a fine Cabernet Sauvignon, long walks in the woods, reading Chekov by the fire, French Films, Saturdays at the art museum, and Schumann piano works. Implicit Ratings: what we found in Jim’s pocket Receipts for: 12 pack of Pabst Blue Ribbon beer, Whataburger, Ben and Jerry’s ice cream, pizza & donuts DVD rental receipts: Marvel’s The Avengers, Resident Evil: Retribution, Ong Bak 3 3-6 COLLABORATIVE FILTERING Problems with explicit ratings Problem 1: People are lazy and don't rate items. First, users will typically not bother to rate items. I imagine most of you have bought a substantial amount of stuff on Amazon. I know I have. In the last month I bought a microHelicopter, a 1TB hard drive, a USB-SATA converter, a bunch of vitamins, two Kindle books (Murder City: Ciudad Juarez and the Global Economy's New Killing Fields and Ready Player One) and the physical books No Place to Hide, Dr. Weil's 8 Weeks to Optimum Health, Anticancer: A new way of life, and Rework. That's twelve items. How many have I rated? Zero. I imagine most of you are the same. You don't rate the items you buy. I have a gimp knee. I like hiking in the mountains and as a result own a number of trekking poles including some cheap ones I bought on Amazon that have taken a lot of abuse. Last year I flew to Austin for the 3 day Austin City Limits music festival. I aggravated my knee injury dashing from one flight to another and ended up going to REI to buy a somewhat pricey REI branded trekking pole. It broke in less than a day of walking on flat grass at a city park. Here I own $10 poles that don't break during constant use of hiking around in the Rockies and this pricey model broke on flat ground. At the time of the festival, as I was fuming, I planned to rate and write a review of the pole on the REI site. Did I? No, I am too lazy. So even in this extreme case I didn't rate the item. I think there are a lot of lazy people like me. People in general are too lazy or unmotivated to rate products. my slightly bent REI pole ➭ 3-7 Problem 2: People may lie or give only partial information. Let's say someone gets over that initial laziness and actually rates a product. That person may lie. This is illustrated in the drawing a few pages back. They can lie directly—giving inaccurate ratings or lie via omission—providing only partial information. Ben goes on a first date with Ann to see the 2010 Cannes Film Festival Winner, a Thai film, Uncle Boonmee Who Can Recall His Past Lives. They go with Ben's friend Dan and Dan's friend Clara. Ben thinks it was the worst film he ever saw. All the others absolutely loved it and gushed about it afterwards at the restaurant. It would not be surprising if Ben upped his rating of the film on online rating sites that his friends might see or just not rate the film. Problem 3: People don't update their ratings. Suppose I am motivated by writing this chapter to rate my Amazon purchases. That 1TB hard drive works well—it's very speedy and also very quiet. I rate it five stars. That microHelicopter is great. It is easy to fly and great fun and it survived multiple crashes. I rate it five stars. A month goes by. The hard drive dies and as a result I lose all my downloaded movies and music—a major bummer. The microHelicopter suddenly stops working—it looks like the motor is fried. Now I think both products suck. Chances are pretty good that I will not go to Amazon and update my ratings (laziness again). People still think I would rate both 5 stars. 3-8 COLLABORATIVE FILTERING Consider Mary, a college student. For some reason, she loves giving Amazon ratings. Ten years ago she rated her favorite music albums with five stars: Giggling and Laughing: Silly Songs for Kids, and Sesame Songs: Sing Yourself Silly! Her most recent ratings included 5 stars for Wolfgang Amadeus Phoenix and The Twilight Saga: Eclipse Soundtrack. Based on these recent ratings she ends up being the closest neighbor to another college student Jen. It would be odd to recommend Giggling and Laughing: Silly Songs for Kids to Jen. This is a slightly different type of update problem than the one above, but a problem none-the-less. k brain calisthenics What do you think are the problems with implicit ratings? (hint: think about the purchases you made on Amazon) 3-9 A few pages ago I gave a list of items I bought at Amazon in the last month. It turns out I bought two of those items for other people. I bought the anticancer book for my cousin and the Rework book for my son. To see why this is a problem, let me come up with a more compelling example by going further back in my purchase history. I bought some kettlebells and the book Enter the Kettlebell! Secret of the Soviet Supermen as a gift for my son and a Plush Chase Border Collie stuffed animal for my wife because our 14-year-old border collie died. Using purchase history as an implicit rating of what a person likes, might lead you to believe that people who like kettlebells, like stuffed animals, like microHelicopters, books on anticancer, and the book Ready Player One. Amazon's purchase history can't distinguish between purchases for myself and purchases I make as gifts. Stephen Baker describes a related example: Figuring out that a certain white blouse is business attire for a female baby boomer is merely step one for the computer. The more important task is to build a profile of the shopper who buys that blouse. Let's say it's my wife. She goes to Macy's and buys four or five items for herself. Underwear, pants, a couple of blouses, maybe a belt. All of the items fit that boomer profile. She's Baker 2008.60-61. coming into focus. Then, on the way out she remembers to buy a birthday present for our 16-year-old niece. Last time we saw her, this girl was wearing black clothing with a lot of writing on it, most of it angry. She told us she was a goth. So my wife goes into an “alternative” section and—what the hell—picks up one of those dog collars bristling with sharp spikes. 3-10 COLLABORATIVE FILTERING If we are attempting to build a profile of a person—what a particular person likes—this dog collar purchase is problematic. Finally, consider a couple sharing a Netflix account. He likes action flicks with lots of explosions and helicopters; she likes intellectual movies and romantic comedies. If we just look at rental history, we build an odd profile of someone liking two very different things. Recall that I said my purchase of the book Anticancer: A New Way of Life was as a gift to my cousin. If we mine my purchase history a bit more we would see that I bought this book before. In fact, in the last year I purchased multiple copies of three books. One can imagine that I am making these multiple purchases not because I am losing the books, or that I am losing my mind and forgetting that I read the books. The most rational reason, is that I liked the books so much I am in a sense recommending these books to others by giving them as gifts. So we can gain a substantial amount of information from a person's purchase history. k brain calisthenics What can we use as implicit data when we are observing a person’s behavior at a computer? Before turning the page come up with a list of possibilities 3-11 k Implicit Data: Web pages: clicking on the link to a page time spent looking at a page repeated visits referring a page to others what a person watches on Hulu Music players: what the person plays skipping tunes number of times a tune is played This just scratches the surface! ! ! ! Keep in mind that the algorithms described in chapter 2 can be used regardless of whether the data is explicit or implicit. The problems of success You have a successful streaming music service with a built in recommendation system. What could possibly go wrong? Suppose you have one million users. Every time you want to make a recommendation for someone you need to calculate one million distances (comparing that person to the 999,999 other people). If we are making multiple recommendations per second, the number of calculations get extreme. Unless you throw a lot of iron at the problem the system will get slow. To say this in a more formal way, latency can be a major drawback of neighbor-based 3-12 COLLABORATIVE FILTERING recommendation systems. Fortunately, there is a solution. Lots of iron: a server farm User-based filtering. So far we have been doing user-based collaborative filtering. We are comparing a user with every other user to find the closest matches. There are two main problems with this approach: 1. Scalability. As we have just discussed, the computation increases as the number of users increases. User-based methods work fine for thousands of users, but scalability gets to be a problem when we have a million users. 2. Sparsity. Most recommendation systems have many users and many products but the average user rates a small fraction of the total products. For example, Amazon carries millions of books but the average user rates just a handful of books. Because of this the algorithms we covered in chapter 2 may not find any nearest neighbors. Because of these two issues it might be better to do what is called item-based filtering. 3-13 Item-based filtering. Suppose I have an algorithm that identifies products that are most similar to each other. For example, such an algorithm might find that Phoenix's album Wolfgang Amadeus Phoenix is similar to Passion Pit's album, Manners. If a user rates Wolfgang Amadeus Phoenix highly we could recommend the similar album Manners. Note that this is different than what we did for user-based filtering. In user-based filtering we had a user, found the most similar person (or users) to that user and used the ratings of that similar person to make recommendations. In item-based filtering, ahead of time we find the most similar items, and combine that with a user's rating of items to generate a recommendation. Can you give me an example? Suppose our streaming music site has m users and n bands, where the users rate bands. This is shown in the following table. As before, the rows represent the users and the columns represent bands. Users 1 Tamera Young 2 Jasmine Abbey 3 Arturo Alvarez ... ... u Cecilia De La Cueva ... ... m-1 m 3-14 ... Phoenix ... Passion Pit 5 4 1 2 5 5 Jessica Nguyen 4 5 Jordyn Zamora 4 ... n COLLABORATIVE FILTERING We would like to compute the similarity of Phoenix to Passion Pit. To do this we only use users who rated both bands as indicated by the blue squares. If we were doing user-based filtering we would determine the similarity between rows. For item-based filtering we are determining the similarity between columns—in this case between the Phoenix and Passion Pit columns. ory based User-based filtering is also called mem need to we e aus Bec collaborative filtering. Why? e store all the ratings in order to mak recommendations. el-based Item-based filtering is also called mod we don’t need collaborative filtering. Why? Because el to store all the ratings. We build a mod every other to is item representing how close every item! 3-15 Adjusted Cosine Similarity. To compute the similarity between items we will use Cosine Similarity which was introduced in chapter 2. We also already talked about grade inflation where a user gives higher ratings than expected. To compensate for this grade inflation we will subtract the user's average rating from each rating. This gives us the adjusted cosine similarity formula shown on the following page. I like Phoenix, I’ll give them a ‘5’. I don’t like Passion Pit, I’ll give them a ‘3’! Phoenix is awesome, They’re definitely a ‘4’. Passion Pit sucks. A definite 0! 3-16 COLLABORATIVE FILTERING s(i, j) = ∑ (R u,i − Ru )(Ru, j − Ru ) u∈U ∑ (R u,i − R u )2 u∈U ∑ (R u, j − R u )2 o sers wh t of all u nd j! e s e h t ia U is th items rated bo u∈U This formula is from a seminal article in collaborative filtering: “Item-based collaborative filtering recommendation algorithms” by Badrul Sarwar, George Karypis, Joseph Konstan, and John Reidl (http://www.grouplens.org/papers/pdf/www10_sarwar.pdf) (R u,i − Ru ) means the rating R user u gives to item i minus the average rating that user gave for all items she rated. This gives us the normalized rating. In the formula above for s(i,j) we are finding the similarity between items i and j. The numerator says that for every user who rated both items multiply the normalized rating of those two items and sum the results. In the denominator we sum the squares of all the normalized ratings for item i and then take the square root of that result. We do the same for item j. And then we multiply those two together. To illustrate adjusted cosine similarity we will use the following data where five students rated five musical artists. Users average rating Kacey Musgraves Imagine Dragons Daft Punk Lorde Fall Out Boy David 3 5 4 1 Matt 3 4 4 1 3 1 3 1 Ben 4 3 Chris 4 4 4 Torri 5 4 5 3 The first thing to do is to compute each user’s average rating. That is easy! Go ahead and fill that in. 3-17 Users average rating Kacey Musgraves Imagine Dragons Daft Punk Lorde Fall Out Boy David 3.25 3 5 4 1 Matt 3.0 3 4 4 1 Ben 2.75 4 3 3 1 3.2 4 4 4 3 1 4.25 5 4 5 Chris Tori 3 Now for each pair of musical artists we are going to compute their similarity. Let’s start with Kacey Musgraves and Imagine Dragons. In the above table, I have circled the cases where a user rated both bands. So the adjusted cosine similarity formula is s(Musgraves, Dragons) = ∑ (R u,Musgraves ∑ (R u,Musgraves − R u )2 u∈U Ben’s ratings = = 3-18 − Ru )(Ru,Dragons − Ru ) u∈U ∑ (R u,Dragons − R u )2 u∈U Chris’s ratings Tori’s ratings (4 − 2.75)(3 − 2.75) + (4 − 3.2)(4 − 3.2) + (5 − 4.25)(4 − 4.25) (4 − 2.75)2 + (4 − 3.2)2 + (5 − 4.25)2 (3 − 2.75)2 + (4 − 3.2)2 + (4 − 4.25)2 0.7650 0.7650 0.7650 = = = 0.5260 2.765 0.765 (1.6628)(0.8746) 1.4543 COLLABORATIVE FILTERING So the similarity between Kacey Musgraves and Imagine Dragons is 0.5260. I have computed some of the other similarities and entered them in this table: Fall Out Boy Lorde Daft Punk Imagine Dragons 0.5260 Kacey Musgraves -0.9549 1.0000 Imagine Dragons -0.3378 0.0075 Daft Punk -0.9570 Lorde -0.6934 Fall Out Boy s sharpen your pencil Compute the rest of the values in the table above! 3-19 s sharpen your pencil - solution Fall Out Boy Lorde Daft Punk Imagine Dragons Kacey Musgraves -0.9549 0.3210 1.0000 0.5260 Imagine Dragons -0.3378 -0.2525 0.0075 Daft Punk -0.9570 0.7841 Lorde -0.6934 To compute these values I wrote a small Python script: def computeSimilarity(band1, band2, userRatings): averages = {} for (key, ratings) in userRatings.items(): averages[key] = (float(sum(ratings.values())) / len(ratings.values())) num = 0 # numerator dem1 = 0 # first half of denominator dem2 = 0 for (user, ratings) in userRatings.items(): if band1 in ratings and band2 in ratings: avg = averages[user] num += (ratings[band1] - avg) * (ratings[band2] - avg) dem1 += (ratings[band1] - avg)**2 dem2 += (ratings[band2] - avg)**2 return num / (sqrt(dem1) * sqrt(dem2)) The format for the userRatings is shown on the following page! 3-20 COLLABORATIVE FILTERING s sharpen your pencil - solution cont’d users3 = {"David": {"Imagine Dragons": 3, "Daft Punk": 5, "Lorde": 4, "Fall Out Boy": 1}, "Matt": {"Imagine Dragons": 3, "Daft Punk": 4, "Lorde": 4, "Fall Out Boy": 1}, "Ben": {"Kacey Musgraves": 4, "Imagine Dragons": 3, "Lorde": 3, "Fall Out Boy": 1}, "Chris": {"Kacey Musgraves": 4, "Imagine Dragons": 4, "Daft Punk": 4, "Lorde": 3, "Fall Out Boy": 1}, "Tori": {"Kacey Musgraves": 5, "Imagine Dragons": 4, "Daft Punk": 5, "Fall Out Boy": 3}} Fall Out Boy Lorde Daft Punk Imagine Dragons Kacey Musgraves -0.9549 0.3210 1.0000 0.5260 Imagine Dragons -0.3378 -0.253 0.0075 Daft Punk -0.9570 0.7841 Lorde -0.6934 Now that we have this nice matrix of similarity values, it would be dreamy if we could use it to make predictions.! (I wonder how well David will like Kacey Musgraves?) 3-21 p(u,i) = ∑ N∈similarTo(i ) ∑ (Si,N × Ru,N ) N∈similarTo(i ) ( Si,N ) English, please! Okay! p(u,i) means we are going to predict the rating user u will give item i. so, P(David, Kacey Musgraves) means our prediction for the rating David (the u in the equation) will give Kacey Musgraves (the i in the equation) N is each of the items that person u rated that are similar to item i. By ‘similar’ I mean that there is a similarity score between N and i in our matrix! 3-22 COLLABORATIVE FILTERING Si,N is the similarity between i and N (from the similarity matrix) p(u,i) = Ru,N is the rating user u gave item N ∑ N∈similarTo(i ) ∑ (Si,N × Ru,N ) N∈similarTo(i ) ( Si,N ) Ru,N is We are trying to predict how well user u will like item i (what rating user u will give item i) For this to work best, RN,i should be a value in the range -1 to 1. Our ratings are in the range 1 to 5. So we will need some numeric Kung Fu to convert our ratings to the -1 to 1 scale. 3-23 Our curre nt MaxR be th music ratings rang e fr e MinR be th maximum rating (5 om 1 to 5. Let e minimu m rating (1 in our case) and current ra ting user ). Ru,N is th ug e normalize d rating (t ave item N. NRu,N is he new ra -1 to 1. Th th e ting e equation to normali on the scale of ze the rati ng is NRu ,N = 2( Ru ,N − Min ) − ( Max − M R R inR ) ( Max − M R inR ) The equation to denormalize (go from the normalized rating to one in our original scale of 1-5 is: 1 Ru,N = ((NRu,N + 1) × (Max R − MinR )) + MinR 2 Let’s say someone rated Fall Out Boy a 2. Our normalized rating would be ... NRu,N = 2(Ru,N − MinR ) − (Max R − MinR ) 2(2 − 1) − (5 − 1) −2 = = = −0.5 (Max R − MinR ) (5 − 1) 4 and to go the other way ... 3-24 COLLABORATIVE FILTERING 1 Ru,N = ((NRu,N + 1) × (Max R − MinR )) + MinR 2 1 1 = ((−0.5 + 1) × 4) + 1 = (2) + 1 = 1+ 1 = 2 2 2 Okay. We now have that numeric Kung Fu under our belt! Let’s see how this works with an example! We are trying to predict what rating David would give Kacey Musgraves. The first thing we are going to do is normalize David’s ratings: David’s Ratings Artist R NR Imagine Dragons 3 0 Daft Punk 5 1 Lorde 4 0.5 Fall Out Boy 1 -1 more about We will learn n in the next normalizatio chapter! 3-25 Similarity Matrix Fall Out Boy Lorde Daft Punk Imagine Dragons Kacey Musgraves -0.9549 0.3210 1.0000 0.5260 Imagine Dragons -0.3378 -0.2525 0.0075 Daft Punk -0.9570 0.7841 Lorde -0.6934 David rated Imagine Dragons, Daft Punk, Lorde, and Fall Out Boy so we will use those in our calculations to determine how well he will like Kacey Musgraves. And we will be using the normalized ratings! p(u,i) = ∑ N∈similarTo(i ) ∑ (Si,N × NRu,N ) ( Si,N ) N∈similarTo(i ) Imagine Dragons Daft Punk = Lorde Fall Out Boy (.5260 × 0) + (1.00 × 1) + (.321× 0.5) + (−.955 × −1) 0.5260 + 1.000 + 0.321+ 0.955 3-26 COLLABORATIVE FILTERING = 0 + 1+ 0.1605 + 0.955 2.1105 = = 0.753 2.802 2.802 So we predict that David will rate Kacey Musgraves a 0.753 on a scale of -1 to 1. To get back to our scale of 1 to 5 we need to denormalize: 1 Ru,N = ((NRu,N + 1) × (Max R − MinR )) + MinR 2 1 1 = ((0.753 + 1) × 4) + 1 = (7.012) + 1 = 3.506 + 1 = 4.506 2 2 So we predict that David will rate Kacey Musgraves a 4.506! Colloborative a Model-Based is y it ar ili m Si , one advantage Adjusted Cosine few pages back a ed on ti en m d. As es is that they Filtering Metho emory-based on m to d re pa m ds tend to ds co of these metho el-based metho od m , ts se ta da r large scale better. Fo y. ire less memor qu be fast and re e artists I les differently. I may rat sca ing rat use ple peo en Oft y rate artists I like a ‘4’. You ma am not keen on a ‘3’ and ed artists you like a ‘5’. Adjust artists you dislike a ‘1’ and ng the this problem by subtracti Cosine Similarity handles ing. rat h eac rage rating from corresponding user’s ave 3-27 Slope One Another popular algorithm for item-based collaborative filtering is Slope One. A major advantage of Slope One is that it is simple and hence easy to implement. Slope One was introduced in the paper “Slope One Predictors for Online Rating-Based Collaborative Filtering” by Daniel Lemire and Anna Machlachlan (http://www.daniel-lemire.com/fr/ abstracts/SDM2005.html). This is an awesome paper and well worth the read. Here's the basic idea in a minimalist nutshell. Suppose Amy gave a rating of 3 to PSY and a rating of 4 to Whitney Houston. Ben gave a rating of 4 to PSY. We'd like to predict how Ben would rate Whitney Houston. In table form the problem might look like this: PSY Whitney Houston Amy 3 4 Ben 4 ? To guess what Ben might rate Whitney Houston we could reason as follows. Amy rated Whitney one whole point better than PSY. We can predict then than Ben would rate Whitney one point higher so we will predict that Ben will give her a '5'. There are actually several Slope One algorithms. I will present the Weighted Slope One algorithm. Remember that a major advantage is that the approach is simple. What I present may look complex, but bear with me and things should become clear. You can consider Slope One to be in two parts. First, ahead of time (in batch mode, in the middle of the night or whenever) we are going to compute what is called the deviation between every pair of items. In the simple example above, this step will determine that Whitney is rated 1 better than PSY. Now we have a nice database of item deviations. In the second phase we actually make predictions. A user comes along, Ben for example. He has never heard Whitney Houston and we want to predict how he would rate her. Using all the bands he did rate along with our database of deviations we are going to make a prediction. 3-28 COLLABORATIVE FILTERING The Broad Brush Picture Part 1 (done ahead of time) Compute deviations between every pair of items Part 2 Use deviations to make predictions Part 1: Computing deviation Let's make our previous example way more complex by adding two users and one band: Taylor Swift PSY Whitney Houston Amy 4 3 4 Ben 5 2 ? Clara ? 3.5 4 Daisy 5 ? 3 The first step is to compute the deviations. The average deviation of an item i with respect to item j is: ui − u j devi, j = ∑ u∈Si , j ( X ) card(Si, j (X)) where card(S) is how many elements are in S and X is the entire set of all ratings. So 3-29 card(Sj,i(X)) is the number of people who have rated both j and i. Let's consider the deviation of PSY with respect to Taylor Swift. In this case, card(Sj,i(X)) is 2—there are 2 people (Amy and Ben) who rated both Taylor Swift and PSY. The uj – ui numerator is (that user’s rating for Taylor Swift) minus (that user’s rating for PSY). So the deviation is: devswift , psy = (4 − 3) (5 − 2) 1 3 + = + =2 2 2 2 2 So the deviation from PSY to Taylor Swift is 2 meaning that on average users rated Taylor Swift 2 better than PSY. What is the deviation from Taylor Swift to PSY? dev psy,swift = s (3 − 4) (2 − 5) 1 3 + = − + − = −2 2 2 2 2 sharpen your pencil Compute the rest of the values in this table: Taylor Swift PSY Taylor Swift 0 2 PSY -2 0 Whitney Houston 3-30 Whitney Houston 0 COLLABORATIVE FILTERING s sharpen your pencil - solution Compute the rest of the values in this table: Taylor Swift with respect to Whitney Houston: devswift ,houston = (4 − 4) (5 − 3) 0 2 + = + =1 2 2 2 2 PSY with respect to Whitney Houston: dev psy,houston = (3 − 4) (3.5 − 4) −1 −.5 + = + = −.75 2 2 2 2 Taylor Swift PSY Whitney Houston Taylor Swift 0 2 1 PSY -2 0 -0.75 Whitney Houston -1 0.75 0 3-31 k brain calisthenics Consider our streaming music site with one million users rating 200,000 artists. If we get a new user who rates 10 artists do we need to run the algorithm again to generate the deviations of all 200k x 200k pairs or is there an easier way? (answer on next page) 3-32 COLLABORATIVE FILTERING k brain calisthenics Consider our streaming music site with one million users rating 200,000 artists. If we get a new user who rates 10 artists do we need to run the algorithm again to generate the deviations of all 200k x 200k pairs or is there an easier way? You don't need to run the algorithm on the entire dataset again. That's the beauty of this method. For a given pair of items we only need to keep track of the deviation and the total number of people rating both items. For example, suppose I have a deviation of Taylor Swift with respect to PSY of 2 based on 9 people. I have a new person who rated Taylor Swift 5 and PSY 1 the updated deviation would be ((9 * 2) + 4) / 10 = 2.2 3-33 Part 2: Making predictions with Weighted Slope One Okay, so now we have a big collection of deviations. How can we use that collection to make predictions? As I mentioned, we are using Weighted Slope One or PwS1 --for Weighted Slope One Prediction. The formula is: P wS1 (u) j = ∑ (dev j,i + ui )c j,i i∈S(u )−{ j} ∑ c j,i i∈S(u )−{ j} where c j,i = card(S j,i ( χ )) PwS1(u)j means our prediction using Weighted Slope One of user u’s rating for item j. So, for example PwS1(Ben)Whitney Houston means our prediction for what Ben would rate Whitney Houston. Let's say I am interested in answering that question: How might Ben rate Whitney Houston? Let's dissect the numerator. ∑ i∈S(u )−{ j} means for every musician that Ben has rated (except for Whitney Houston that is the {j} bit). The entire numerator means for every musician i that Ben has rated (except for Whitney Houston) we will look up the deviation of Whitney Houston to that musician and we will add that to Ben's rating for musician i. We multiply that by the cardinality of that pair—the number of people that rated both musicians (Whitney and musician i). 3-34 COLLABORATIVE FILTERING Let's step through this: First, here are Ben’s ratings and our deviations table from before: Taylor Swift PSY Whitney Houston 5 2 ? Ben 1. Taylor Swift PSY Whitney Houston Taylor Swift 0 2 1 PSY -2 0 -0.75 Whitney Houston -1 0.75 0 Ben has rated Taylor Swift and gave her a 5—that is the ui. 2. The deviation of Whitney Houston with respect to Taylor Swift is -1 —this is the devj,i. 3. devj,i + ui then is 4. 4. Looking at page 3-19 we see that there were two people (Amy and Daisy) that rated both Taylor Swift and Whitney Houston so cj,i = 2 5. So (devj,i + ui) cj,i = 4 x 2 = 8 6. Ben has rated PSY and gave him a 2. 7. The deviation of Whitney Houston with respect to PSY is 0.75 8. devj,i + ui then is 2.75 9. Two people rated both Whitney Houston and PSY so (devj,i + ui) cj,i = 2.75 x 2 = 5.5 10. We sum up steps 5 and 9 to get 13.5 for the numerator DENOMINATOR 11. Dissecting the denominator we get something like for every musician that Ben has rated, sum the cardinalities of those musicians (how many people rated both that musician and 3-35 Whitney Houston). So Ben has rated Taylor Swift and the cardinality of Taylor Swift and Whitney Houston (that is, the total number of people that rated both of them) is 2. Ben has rated PSY and his cardinality is also 2. So the denominator is 4. 12. So our prediction of how well Ben will like Whitney Houston is 3-36 13.5 = 3.375 4 COLLABORATIVE FILTERING Putting this into Python I am going to extend the Python class developed in chapter 2. To save space I will not repeat the code for the recommender class here—just refer back to it (and remember that you can download the code at http://guidetodatamining.com). Recall that the data for that class was in the following format: users2 = {"Amy": {"Taylor Swift": 4, "PSY": 3, "Whitney Houston": 4}, "Ben": {"Taylor Swift": 5, "PSY": 2}, "Clara": {"PSY": 3.5, "Whitney Houston": 4}, "Daisy": {"Taylor Swift": 5, "Whitney Houston": 3}} First computing the deviations. Again, the formula for computing deviations is devi, j = ui − u j u∈Si , j ( X ) card(Si, j (X)) ∑ So the input to our computeDeviations function should be data in the format of users2 above. The output should be a representation of the following data: Taylor Swift PSY Whitney Houston Taylor Swift 0 2 (2) 1 (2) PSY -2 (2) 0 -0.75 (2) Whitney Houston -1 (2) 0.75 (2) 0 The number in the parentheses is the frequency (that is, the number of people that rated that pair of musicians). So for each pair of musicians we need to record both the deviation and the frequency. 3-37 The pseudoCode for our function could be def computeDeviations(self): for each i in bands: for each j in bands: if i ≠ j: compute dev(j,i) That pseudocode looks pretty nice but as you can see, there is a disconnect between the data format expected by the pseudocode and the format the data is really in (see users2 above as an example). As code warriors we have two possibilities, either alter the format of the data, or revise the psuedocode. I am going to opt for the second approach. This revised pseudocode looks like def computeDeviations(self): for each person in the data: get their ratings " for each item & rating in that set of ratings: " for each item2 & rating2 in that set of ratings: " add the difference between the ratings to our computation Let's construct the method step-by-step Step 1: def computeDeviations(self): " # for each person in the data: " # get their ratings " for ratings in self.data.values(): Python dictionaries (aka hash tables) are key value pairs. Self.data is a dictionary. The values method extracts just the values from the dictionary. Our data looks like users2 = {"Amy": {"Taylor Swift": 4, "PSY": 3, "Whitney Houston": 4}, "Ben": {"Taylor Swift": 5, "PSY": 2}, "Clara": {"PSY": 3.5, "Whitney Houston": 4}, "Daisy": {"Taylor Swift": 5, "Whitney Houston": 3}} 3-38 COLLABORATIVE FILTERING So the first time through the loop ratings = {"Taylor Swift": 4, "PSY": 3, "Whitney Houston": 4}. Step 2 def computeDeviations(self): # for each person in the data: # get their ratings for ratings in self.data.values(): #for each item & rating in that set of ratings: " for (item, rating) in ratings.items(): " self.frequencies.setdefault(item, {}) self.deviations.setdefault(item, {}) In the recommender class init method I initialized self.frequencies and self.deviations to be dictionaries. def __init__(self, data, k=1, metric='pearson', n=5): ... # # The following two variables are used for Slope One # self.frequencies = {} self.deviations = {} The Python dictionary method setdefault takes 2 arguments: a key and an initialValue. This method does the following. If the key does not exist in the dictionary it is added to the dictionary with the value initialValue. Otherwise it returns the current value of the key. 3-39 Step 3 def computeDeviations(self): # for each person in the data: # get their ratings for ratings in self.data.values(): # for each item & rating in that set of ratings: for (item, rating) in ratings.items(): " self.frequencies.setdefault(item, {}) " " " " " self.deviations.setdefault(item, {}) # for each item2 & rating2 in that set of ratings: for (item2, rating2) in ratings.items(): if item != item2: # add the difference between the ratings # to our computation self.frequencies[item].setdefault(item2, 0) self.deviations[item].setdefault(item2, 0.0) self.frequencies[item][item2] += 1 self.deviations[item][item2] += rating - rating2 The code added in this step computes the difference between two ratings and adds that to the self.deviations running sum. Again, using the data: {"Taylor Swift": 4, "PSY": 3, "Whitney Houston": 4} when we are in the outer loop where item = “Taylor Swift” and rating = 4 and in the inner loop where item2 = “PSY” and rating2 = 3 the last line of the code above adds 1 to self.deviations[“Taylor Swift”][“PSY”]. Step 4: Finally, we need to iterate through self.deviations to divide each deviation by its associated frequency. 3-40 COLLABORATIVE FILTERING def computeDeviations(self): # for each person in the data: # get their ratings for ratings in self.data.values(): # for each item & rating in that set of ratings: for (item, rating) in ratings.items(): self.frequencies.setdefault(item, {}) self.deviations.setdefault(item, {}) # for each item2 & rating2 in that set of ratings: for (item2, rating2) in ratings.items(): if item != item2: # add the difference between the ratings # to our computation self.frequencies[item].setdefault(item2, 0) self.deviations[item].setdefault(item2, 0.0) self.frequencies[item][item2] += 1 self.deviations[item][item2] += rating - rating2 for (item, ratings) in self.deviations.items(): for item2 in ratings: ratings[item2] /= self.frequencies[item][item2] That's it! Even with comments we implemented ui − u j devi, j = ∑ u∈Si , j ( X ) card(Si, j (X)) in only 18 lines of code. Incredible! When I run this method on the data I have been using in this example: users2 = {"Amy": {"Taylor Swift": 4, "PSY": 3, "Whitney Houston": 4}, "Ben": {"Taylor Swift": 5, "PSY": 2}, "Clara": {"PSY": 3.5, "Whitney Houston": 4}, "Daisy": {"Taylor Swift": 5, "Whitney Houston": 3}} 3-41 I get >>> r = recommender(users2) >>> r.computeDeviations() >>> r.deviations {'PSY': {'Taylor Swift': -2.0, 'Whitney Houston': -0.75}, 'Taylor Swift': {'PSY': 2.0, 'Whitney Houston': 1.0}, 'Whitney Houston': {'PSY': 0.75, 'Taylor Swift': -1.0}} which is what we obtained when we computed this example by hand: Taylor Swift PSY Whitney Houston Taylor Swift 0 2 1 PSY -2 0 -0.75 Whitney Houston -1 0.75 0 Shout out to Bryan O’Sullivan and his blog teideal glic deisbhéalach (serpentine.com/blog) which presented a Python implementation of Slope One! The code presented here is based on his work. 3-42 COLLABORATIVE FILTERING Weighted Slope 1: The recommendation component Now it is time to code the recommendation component: P wS1 (u) j = ∑ (dev j,i + ui )c j,i i∈S(u )−{ j} ∑ c j,i i∈S(u )−{ j} The big question I have is can we beat the 18 line implementation of computeDeviations. First, let's parse that formula and put it into English and/or pseudocode. You try: s sharpen your pencil The formula in pseudo English: 3-43 s sharpen your pencil - a solution Here's my version of the formula: I would like to make recommendations for a particular user. I have that user's recommendations in the form {"Taylor Swift": 5, "PSY": 2} For every userItem and userRating in the user's recommendations: For every diffItem that the user didn't rate (item2 ≠ item): add the deviation of diffItem with respect to userItem to the userRating of the userItem. Multiply that by the number of people that rated both userItem and diffItem. Add that to the running sum for diffItem Also keep a running sum for the number of people that rated diffItem. Finally, for every diffItem that is in our results list, divide the total sum of that item by the total frequency of that item and return the results. 3-44 COLLABORATIVE FILTERING And here is my conversion of that to Python: def slopeOneRecommendations(self, userRatings): recommendations = {} frequencies = {} # for every item and rating in the user's recommendations for (userItem, userRating) in userRatings.items(): # for every item in our dataset that the user didn't rate for (diffItem, diffRatings) in self.deviations.items(): if diffItem not in userRatings and \ userItem in self.deviations[diffItem]: freq = self.frequencies[diffItem][userItem] recommendations.setdefault(diffItem, 0.0) frequencies.setdefault(diffItem, 0) # add to the running sum representing the numerator # of the formula recommendations[diffItem] += (diffRatings[userItem] + userRating) * freq # keep a running sum of the frequency of diffitem frequencies[diffItem] += freq recommendations = [(self.convertProductID2name(k), v / frequencies[k]) for (k, v) in recommendations.items()] # finally sort and return recommendations.sort(key=lambda artistTuple: artistTuple[1], reverse = True) return recommendations And here is a simple test of the complete Slope One implementation: >>> r = recommender(users2) >>> r.computeDeviations() >>> g = users2['Ben'] >>> r.slopeOneRecommendations(g) [('Whitney Houston', 3.375)] 3-45 This results matches what we calculated by hand. So the recommendation part of the algorithm weighs in at 18 lines. So in 36 lines of Python code we implemented the Slope One algorithm. With Python you can write pretty compact code. MovieLens data set Let's try out the Slope One recommender on a different dataset. The MovieLens dataset— collected by the GroupLens Research Project at the University of Minnesota—contains user ratings of movies. The data set is available for download at www.grouplens.org. The data set is available in three sizes; for the demo here I am using the smallest one which contains 100,000 Again, you can download ratings (1-5) from 943 users on 1,682 movies. I the code to this chapter at wrote a short function that will import this data www.guidetodatamining.com! into the recommender class. Let's give it a try. First, I will load the data into the Python recommender object: >>> r = recommender(0) >>> r.loadMovieLens('/Users/raz/Downloads/ml-100k/') 102625 I will be using the info from User 1. Just to peruse the data, I will look at the top 50 items the user 1 rated: >>> r.showUserTopItems('1', 50) When Harry Met Sally... (1989)" 5 Jean de Florette (1986)"5 Godfather, The (1972)" 5 Big Night (1996)" 5 Manon of the Spring (Manon des sources) (1986)"5 Sling Blade (1996)" 5 Breaking the Waves (1996)" 5 Terminator 2: Judgment Day (1991)" 5 Searching for Bobby Fischer (1993)" 5 3-46 COLLABORATIVE FILTERING Maya Lin: A Strong Clear Vision (1994)" Mighty Aphrodite (1995)"5 Bound (1996)" 5 Full Monty, The (1997)" 5 Chasing Amy (1997)" 5 Ridicule (1996)" 5 Nightmare Before Christmas, The (1993)" Three Colors: Red (1994)" 5 Professional, The (1994)" 5 Priest (1994)" 5 5 5 ... User 1 rated all these movies a ‘5’! Now I will do the first step of Slope One: computing the deviations: >>> r.computeDeviations() (this takes about 50 seconds to run on my laptop) Finally, let's get recommendations for User 1: >>> r.slopeOneRecommendations(r.data['1']) [('Entertaining Angels: The Dorothy Day Story (1996)', 6.375), ('Aiqing wansui (1994)', 5.849056603773585), ('Boys, Les (1997)', 5.644970414201183), ("Someone Else's America (1995)", 5.391304347826087), ('Santa with Muscles (1996)', 5.380952380952381), ('Great Day in Harlem, A (1994)', 5.275862068965517), ... and user 25: >>> r.slopeOneRecommendations(r.data['25']) [('Aiqing wansui (1994)', 5.674418604651163), ('Boys, Les (1997)', 5.523076923076923), ('Star Kid (1997)', 5.25), ('Santa with Muscles (1996)', 3-47 Projects 1. See how well the Slope One recommender recommends movies for you. Rate 10 movies or so (ones that are in the MovieLens dataset). Does the recommender suggest movies you might like? 2. Implement Adjusted Cosine Similarity. Compare its performance to Slope One. 3. (harder) I run out of memory (I have 8GB on my desktop) when I try to run this on the Book Crossing Dataset. Recall that there are 270,000 books that are rated. So we would need a 270,000 x 270,000 dictionary to store the deviations. That's roughly 73 billion dictionary entries. How sparse is this dictionary for the MovieLens dataset? Alter the code so we can handle larger datasets. Congratulations on finishing chapter 3!! There was some hard work in this chapter--dissecting complex-looking formulas to gain an understanding of them and then implementing them. 3-48 Chapter 4 Content Based Filtering & Classification Classification based on item attributes In the previous chapters we talked about making recommendations by collaborative filtering (also called social filtering). In collaborative filtering we harness the power of a community of people to help us make recommendations. You buy Wolfgang Amadeus Phoenix. We know that many of our customers who bought that album also bought Contra by Vampire Weekend. So we recommend that album to you. I watch an episode of Doctor Who and Netflix recommends Quantum Leap because many people who watched Doctor Who also watched Quantum Leap. In previous chapters we talked about some of the difficulties of collaborative filtering including problems with data sparsity and scalability. Another problem is that recommendation systems based on collaborative filtering tend to recommend already popular items—there is a bias toward popularity. As an extreme case, consider a debut album by a brand new band. Since that band and album have never been rated by anyone (or purchased by anyone since it is brand new), it will never be recommended. “ These reco m men ders can create a rich richer effect -getfor po pular p ro ducts an d versa for un v icepo pular ones” Daniel Fle der & Kartik Hos anagar. 2009. Culture’s Nex “Blockbuster t Rise or Fall s : The Impact Systems on S of Recom men de ales Diversity r ” Managemen t Science vol 55 In this chapter we look at a different approach. Consider the streaming music site, Pandora. In Pandora, as many of you know, you can set up different streaming radio stations. You seed each station with an artist and Pandora will play music that is similar to that artist. I can create a station seeded with the band Phoenix. It then plays bands it thinks are similar to Phoenix—for example, it plays a tune by El Ten Eleven. It doesn't do this with collaborative filtering—because people who listened to Phoenix also listened to the El Ten Eleven. It plays El Ten Eleven because the algorithm believes that El Ten Eleven is musically similar to Phoenix. In fact, we can ask Pandora why it played a tune by the group: It plays El Ten Eleven’s tune My Only Swerving on the Phoenix station because “Based on what you told us so far, we’re playing this track because it features repetitive melodic phrasing, mixed acoustic and electric instrumentation, major key tonality, electric guitar riffs and an instrumental arrangement.” On my Hiromi station it plays a tune by E.S.T. because “it features classic jazz roots, a well-articulated piano solo, light drumming, an interesting song structure and interesting part writing.” 4-2 CONTENT BASED FILTERING & CLASSIFICATION Pandora bases its recommendation on what it calls The Music Genome Project. They hire professional musicians with a solid background in music theory as analysts who determine the features (they call them 'genes') of a song. These analysts are given over 150 hours of training. Once trained they spend an average of 20-30 minutes analyzing a song to determine its genes/features. Many of these genes are technical 4-3 The analyst provides values for over 400 genes. Its a very labor intensive process and approximately 15,000 new songs are added per month. NOTE: The Pandora algorithms are proprietary and I have no knowledge as to how they work. What follows is not a description of how Pandora works but rather an explanation of how to construct a similar system. The importance of selecting appropriate values Consider two genes that Pandora may have used: genre and mood. The values of these might look like this: Mood genre Country 1 Jazz 2 Rock 3 Soul 4 Rap 5 Melancholy 1 joyful 2 passion 3 angry 4 unknown 5 So a genre value of 4 means ‘Soul’ and a mood value of 3 means ‘passion’. Suppose I have a rock song that is melancholy—for example the gag-inducing You’re Beautiful by James Blunt. In 2D space, inked quickly on paper, that would look as follows: 4-4 CONTENT BASED FILTERING & CLASSIFICATION FACT: the tone poll on In a Rolling S , g Songs ever Most Annoyin # ful placed 7! You’re Beauti Let's say Tex just absolutely loves You're Beautiful and we would like to recommend a song to him. That “You’re Beautiful” is so sad and beautiful. I love it! 4-5 Let me populate our dataset with more songs. Song 1 is a jazz song that is melancholy; Song 2 is a soul song that is angry and Song 3 is a jazz song that is angry. Which would you recommend to Tex? closest! Song 1 looks I hope you see that we have a fatal flaw in our scheme. Let's take a look at the possible values for our variables again: Mood genre melancholy 1 Country 1 joyful 2 Jazz 2 passion 3 Rock 3 angry 4 Soul 4 unknown 5 Rap 5 If we are trying to use any distance metrics with this scheme we are saying that jazz is closer to rock than it is to soul (the distance between jazz and rock is 1 and the distance between 4-6 CONTENT BASED FILTERING & CLASSIFICATION jazz and soul is 2). Or melancholy is closer to joyful than it is to angry. Even when we rearrange values the problem remains. Mood genre melancholy 1 Country 1 angry 2 Jazz 2 passion 3 Soul 3 joyful 4 Rap 4 unknown 5 Rock 5 Re-ordering does not solve the problem. No matter how we rearrange the values this won't work. This shows us that we have chosen our features poorly. We want features where the values fall along a meaningful scale. We can easily fix our genre feature by dividing it into 5 separate features—one for country, another for jazz, etc. They all can be on a 1-5 scale—how 'country' is the sound of this track—‘1’ means no hint of country to ‘5’ means this is a solid country sound. Now the scale does mean something. If we are trying to find a song similar to one that rated a country value of ‘5’, a song that rated a country of ‘4’ would be closer than one of a ‘1’. 4-7 This is exactly how Pandora constructs its gene set. The values of most genes are on a scale of 1-5 with ½ integer increments. Genes are arranged into categories. For example, there is a musical qualities category which contains genes for Blues Rock Qualities, Folk Rock Qualities, and Pop Rock Qualities among others. Another category is instruments with genes such as Accordion, Dirty Electric Guitar Riffs and Use of Dirty Sounding Organs. Using these genes, each of which has a well-defined set of values from 1 to 5, Pandora represents each song as a vector of 400 numeric values (each song is a point in a 400 dimensional space). Now Pandora can make recommendations (that is, decide to play a song on a user-defined radio station) based on standard distance functions like those we already have seen. A simple example Let us create a simple dataset so we can explore this approach. Suppose we have seven features each one ranging from 1-5 in ½ integer increments (I admit this isn't a very rational nor complete selection): Amount of piano 1 indicates lack of piano; 5 indicates piano throughout and featured prominently Amount of vocals 1 indicates lack of vocals; 5 indicates prominent vocals throughout song. Driving beat Combination of constant tempo, and how the drums & bass drive the beat. Blues Influence Presence of dirty electric guitar Presence of backup vocals Rap Influence Now, using those features I rate ten tunes: 4-8 CONTENT BASED FILTERING & CLASSIFICATION Piano Vocals Driving beat Blues infl. Dirty elec. Guitar Backup vocals Rap infl. Dr. Dog/ Fate 2.5 4 3.5 3 5 4 1 Phoenix/ Lisztomania 2 5 5 3 2 1 1 Heartless Bastards / Out at Sea 1 5 4 2 4 1 1 Todd Snider/ Don't Tempt Me 4 5 4 4 1 5 1 The Black Keys/ Magic Potion 1 4 5 3.5 5 1 1 Glee Cast/ Jessie's Girl 1 5 3.5 3 4 5 1 Black Eyed Peas/ Rock that Body 2 5 5 1 2 2 4 La Roux/ Bulletproof 5 5 4 2 1 1 1 2.5 4 4 1 1 1 1 1 5 3 2 1 2 1 Mike Posner/ Cooler than me Lady Gaga/ Alejandro Thus, each tune is represented as a list of numbers and we can use any distance function to compute the distance between tunes. For example, The Manhattan Distance between Dr. Dog’s Fate and Phoenix’s Lisztomania is: Dr. Dog/ Fate 2.5 4 3.5 3 5 4 1 Phoenix/ Lisztomania 2 5 5 3 2 1 1 0.5 1 1.5 0 3 3 0 Distance summing those distances gives us a Manhattan Distance of 9. 4-9 s sharpen your pencil I am trying to find out what tune is closest to Glee’s rendition of Jessie’s Girl using Euclidean Distance. Can you finish the following table and determine what group is closest? distance to Glee’s Jessie’s Girl Dr. Dog/ Fate ?? Phoenix/ Lisztomania 4.822 Heartless Bastards / Out at Sea 4.153 Todd Snider/ Don't Tempt Me 4.387 The Black Keys/ Magic Potion 4.528 Glee Cast/ Jessie's Girl 0 Black Eyed Peas/ Rock that Body 5.408 La Roux/ Bulletproof 6.500 Mike Posner/ Cooler than me 5.701 Lady Gaga/ Alejandro 4-10 ?? CONTENT BASED FILTERING & CLASSIFICATION s sharpen your pencil - solution distance to Glee’s Jessie’s Girl Dr. Dog/ Fate 2.291 Lady Gaga/ Alejandro 4.387 Recall that the Euclidean Distance between any two objects, x and y, which have n attributes is: n ∑ (x d(x, y) = k − yk ) 2 k=1 So the Euclidean Distance between Glee and Lady Gaga piano vocals beat blues guitar backup rap Glee 1 5 3.5 3 4 5 1 Lady G 1 5 3 2 1 2 1 (x-y) 0 0 0.5 1 3 3 0 (x-y)2 0 0 0.25 1 9 9 0 SUM SQRT 19.25 4.387 4-11 Doing it Python Style! Recall that our data for social filtering was of the format: users = {"Angelica": {"Blues Traveler": 3.5, "Broken Bells": 2.0, ! ! ! "Norah Jones": 4.5, "Phoenix": 5.0, "Slightly Stoopid": 1.5, "The Strokes": 2.5, "Vampire Weekend": 2.0}, "Bill": {"Blues Traveler": 2.0, "Broken Bells": 3.5, "Deadmau5": 4.0, "Phoenix": 2.0, "Slightly Stoopid": 3.5, "Vampire Weekend": 3.0}} We can represent this current data in a similar way: music = {"Dr Dog/Fate": {"piano": 2.5, "vocals": 4, "beat": 3.5, "blues": 3, "guitar": 5, "backup vocals": 4, "rap": 1}, ! "Phoenix/Lisztomania": {"piano": 2, "vocals": 5, "beat": 5, "blues": 3, "guitar": 2, "backup vocals": 1, "rap": 1}, ! "Heartless Bastards/Out at Sea": {"piano": 1, "vocals": 5, "beat": 4, "blues": 2, ! ! ! ! ! ! "guitar": 4, "backup vocals": 1, "rap": 1}, ! "Todd Snider/Don't Tempt Me": {"piano": 4, "vocals": 5, "beat": 4, "blues": 4, ! ! ! ! ! ! "guitar": 1, "backup vocals": 5, "rap": 1}, ! "The Black Keys/Magic Potion":{"piano": 1, "vocals": 4, "beat": 5, "blues": 3.5, ! ! ! ! ! ! "guitar": 5, "backup vocals": 1, "rap": 1}, ! "Glee Cast/Jessie's Girl": {"piano": 1, "vocals": 5, "beat": 3.5, "blues": 3, ! ! ! ! ! ! "guitar":4, "backup vocals": 5, "rap": 1}, ! "La Roux/Bulletproof": {"piano": 5, "vocals": 5, "beat": 4, 4-12 CONTENT BASED FILTERING & CLASSIFICATION ! ! ! ! "blues": 2, "guitar": 1, "backup vocals": 1, "rap": 1}, "Mike Posner": {"piano": 2.5, "vocals": 4, "beat": 4, "blues": 1, "guitar": 1, "backup vocals": 1, "rap": 1}, "Black Eyed Peas/Rock That Body": {"piano": 2, "vocals": 5, "beat": 5, "blues": 1, ! ! ! ! ! "guitar": 2, "backup vocals": 2, "rap": 4}, ! "Lady Gaga/Alejandro": {"piano": 1, "vocals": 5, "beat": 3, "blues": 2, "guitar": 1, "backup vocals": 2, "rap": 1}} Now suppose I have a friend who says he likes the Black Keys Magic Potion. I can plug that into my handy Manhattan distance function: >>> computeNearestNeighbor('The Black Keys/Magic Potion', music) [(4.5, 'Heartless Bastards/Out at Sea'), (5.5, 'Phoenix/Lisztomania'), (6.5, 'Dr Dog/Fate'), (8.0, "Glee Cast/Jessie's Girl"), (9.0, 'Mike Posner'), (9.5, 'Lady Gaga/Alejandro'), (11.5, 'Black Eyed Peas/Rock That Body'), (11.5, 'La Roux/Bulletproof'), (13.5, "Todd Snider/Don't Tempt Me")] and I can recommend to him Heartless Bastard's Out at Sea. This is actually a pretty good recommendation. NOTE: The code for this example, as well as all examples in this book, is available on the book website http://www.guidetodatamining.com 4-13 Answering the question “Why?” When Pandora recommends something it explains why you might like it: We can do the same. Remember our friend who liked The Black Keys Magic Potion and we recommended Heartless Bastards Out at Sea. What features influenced that recommendation? We can compare the two feature vectors: Piano Vocals Driving beat Blues infl. Dirty elec. Backup Guitar vocals Rap infl. Black Keys Magic Potion 1 5 4 2 4 1 1 Heartless Bastards Out at Sea 1 4 5 3.5 5 1 1 difference 0 1 1 1.5 1 0 0 The features that are closest between the two tunes are piano, presence of backup vocals, and rap influence—they all have a distance of zero. However, all are on the low end of the scale: no piano, no presence of backup vocals, and no rap influence and it probably would not be helpful to say “We think you would like this tune because it lacks backup vocals.” Instead, we will focus on what the tunes have in common on the high end of the scale. 4-14 CONTENT BASED FILTERING & CLASSIFICATION We think you might like Heartless Bastards Out at Sea because it has a driving beat and features vocals and dirty electric guitar. Because our data set has few features, and is not well-balanced, the other recommendations are not as compelling: >>> computeNearestNeighbor("Phoenix/Lisztomania", music) [(5, 'Heartless Bastards/Out at Sea'), (5.5, 'Mike Posner'), (5.5, 'The Black Keys/Magic Potion'), (6, 'Black Eyed Peas/Rock That Body'), (6, 'La Roux/Bulletproof'), (6, 'Lady Gaga/Alejandro'), (8.5, "Glee Cast/ Jessie's Girl"), (9.0, 'Dr Dog/Fate'), (9, "Todd Snider/Don't Tempt Me")] >>> computeNearestNeighbor("Lady Gaga/Alejandro", music) [(5, 'Heartless Bastards/Out at Sea'), (5.5, 'Mike Posner'), (6, 'La Roux/Bulletproof'), (6, 'Phoenix/Lisztomania'), (7.5, "Glee Cast/ Jessie's Girl"), (8, 'Black Eyed Peas/Rock That Body'), (9, "Todd Snider/Don't Tempt Me"), (9.5, 'The Black Keys/Magic Potion'), (10.0, 'Dr Dog/Fate')] That Lady Gaga recommendation is particularly bad. 4-15 A problem of scale Suppose I want to add another feature to my set. This time I will add beats per minute (or bpm). This makes some sense—I might like fast beat songs or slow ballads. Now my data would look like this: Piano Vocals Driving Blues beat infl. Dirty elec. Guitar Backup vocals Rap infl. bpm Dr. Dog/ Fate 2.5 4 3.5 3 5 4 1 140 Phoenix/ Lisztomania 2 5 5 3 2 1 1 110 Heartless Bastards / Out at Sea 1 5 4 2 4 1 1 130 The Black Keys/ Magic Potion 1 4 5 3.5 5 1 1 88 Glee Cast/ Jessie's Girl 1 5 3.5 3 4 5 1 120 Bad Plus/ Smells like Teen Spirit 5 1 2 1 1 1 1 90 Without using beats per minute, the closest match to The Black Keys’ Magic Potion is Heartless Bastards’ Out to Sea and the tune furthest away is Bad Plus’s version of Smells Like Teen Spirit. However, once we add beats per minute, it wrecks havoc with our distance function—bpm dominates the calculation. Now Bad Plus is closest to The Black Keys simply because the bpm of the two tunes are close. 4-16 CONTENT BASED FILTERING & CLASSIFICATION Consider another example. Suppose I have a dating site and I have the weird idea that the best attributes to match people up by are salary and age. gals guys age salary 75,000 name age salary Yun L 35 55,000 Brian A 53 70,000 Allie C 52 45,000 Abdullah K 25 105,000 Daniela C 27 37 David A 35 69,000 Rita A 115,000 Michael W 48 43,000 name Here the scale for age ranges from 25 to 53 for a difference of 28 and the salary scale ranges from 43,000 to 115,000 for a difference of 72,000. Because these scales are so different, salary dominates any distance calculation. If we just tried to eyeball matches we might recommend David to Yun since they are the same age and their salaries are fairly close. However, if we went by any of the distance formulas we have covered, 53-year old Brian would be the person recommended to Yun. This does not look good for my fledgling dating site. In fact, this difference in scale among attributes is a big problem for any recommendation system. Arghhhh. 4-17 Normalization Shhh. I’m normalizing No need to panic. Relax. The solution is normalization! To remove this bias we need to standardize or normalize the data. One common method of normalization involves having the values of each feature range from 0 to 1. For example, consider the salary attribute in our dating example. The minimum salary was 43,000 and the max was 115,000. That makes the range from minimum to maximum 72,000. To convert each value to a value in the range 0 to 1 we subtract the minimum from the value and divide by the range. gals name Yun L 75,000 Allie C 55,000 Daniela C Rita A 4-18 salary So the normalized value for Yun is normalized salary 45,000 115,000 0.444 0.167 0.028 1.0 (75,000 - 43,000) / 72,000 = 0.444 Depending on the dataset this rough method of normalization may work well. CONTENT BASED FILTERING & CLASSIFICATION If you have taken a statistics course you will be familiar with more accurate methods for standardizing data. For example, we can use what is called The Standard Score which can be computed as follows We can standardize a value using the Standard Score (aka z-score) which tells us how many deviations the value is from the mean! (each value) - (mean) (standard deviation) = Standard Score Standard Deviation is sd = ∑ (x − x) 2 i i card(x) card(x) is the cardinality of x—that is, how many values there are. By the way, if you are rusty wit h statistics and like manga be sure to check out the aweso me book “Th e Manga Gui de to Statistics” by Shi n Takahashi. 4-19 Consider the data from the dating site example a few pages back. name salary Yun L 75,000 Allie C 55,000 Daniela C 45,000 Rita A 115,000 Brian A 70,000 Abdullah K 105,000 David A 69,000 Michael W 43,000 Yun’s salary The sum of all the salaries is 577,000. Since there are 8 people, the mean is 72,125. Now let us compute the standard deviation: sd = ∑ (x − x) 2 i i card(x) so that would be Allie’s salary Daniela’s salary etc. (75,000 − 72,125)2 + (55,000 − 72,125)2 + (45,000 − 72,125)2 + ... 8 8,265,625 + 293,265,625 + 735, 765,625 + ... = 602, 395, 375 8 = 24,543.01 = 4-20 CONTENT BASED FILTERING & CLASSIFICATION Again, the standard score is (each value) - (mean) (standard deviation) So the Standard Score for Yun’s salary is 75000 − 72125 2875 = = 0.117 24543.01 24543.01 s sharpen your pencil Can you compute the Standard Scores for the following people? name salary Yun L 75,000 Allie C 55,000 Daniela C 45,000 Rita A 115,000 Standard Score 0.117 4-21 s sharpen your pencil — solution Can you compute the Standard Scores for the following people? name salary Standard Score Yun L 75,000 0.117 Allie C 55,000 -0.698 Daniela C 45,000 -1.105 Rita A 115,000 1.747 Allie: (55,000 - 72,125) / 24,543.01 = -0.698 Daniela: (45,000 - 72,125) / 24,543.01 = -1.105 Rita: (115,000 - 72,125) / 24,543.01 = 1.747 The problem with using Standard Score The problem with the standard score is that it is greatly influenced by outliers. For example, if all the 100 employees of LargeMart make $10/hr but the CEO makes six million a year the mean hourly wage is ( 100 * $10 + 6,000,000 / (40 * 52)) / 101 = (1000 + 2885) / 101 = $38/hr. 4-22 CONTENT BASED FILTERING & CLASSIFICATION Not a bad average wage at LargeMart. As you can see, the mean is greatly influenced by outliers. Because of this problem with the mean, the standard score formula is often modified. Modified Standard Score To compute the Modified Standard Score you replace the mean in the above formula by the median (the middle value) and replace the standard deviation by what is called the absolute standard deviation: asd = where Modified Standard Score: (each value) - (median) (absolute standard deviation) 1 ∑ xi − µ card(x) i is the median. To compute the median you arrange the values from lowest to highest and pick the middle value. If there are an even number of values the median is the average of the two middle values. 4-23 Okay, let’s give this a try. In the table on the right I’ve arranged our salaries from lowest to highest. Since there are an equal number of values, the median is the average of the two middle values: median = (69,000 + 70,000) = 69,500 2 The absolute standard deviation is asd = 1 ∑ xi − µ card(x) i Name Salary Michael W 43,000 Daniela C 45,000 Allie C 55,000 David A 69,000 Brian A 70,000 Yun L 75,000 Abdullah K 105,000 Rita A 115,000 1 asd = ( 43,000 − 69,500 + 45,000 − 69,500 + 55,000 − 69,500) + ...) 8 1 = (26,500 + 24,500 + 14,500 + 500 + ...) 8 1 = (153,000) = 19,125 8 Now let us compute the Modified Standard Score for Yun. Modified Standard Score: (each value) - (median) (absolute standard deviation) 4-24 mss = (75,000 − 69,500) 5,500 = = 0.2876 19,125 19,125 CONTENT BASED FILTERING & CLASSIFICATION s sharpen your pencil The following table shows the play count of various tracks I played. Can you standardize the values using the Modified Standard Score? track play count Power/Marcus Miller 21 I Breathe In, I Breathe Out/ Chris Cagle 15 Blessed / Jill Scott 12 Europa/Santana 3 Santa Fe/ Beirut 7 modified standard score 4-25 s sharpen your pencil — solution The following table shows the play count of various tracks I played. Can you standardize the values using the Modified Standard Score? Step 1. Computing the median. I put the values in order (3, 7, 12, 15, 21) and select the middle value, 12. The median µ is 12. Step 2. Computing the Absolute Standard Deviation. 1 asd = ( 3 − 12 + 7 − 12 + 12 − 12 + 15 − 12 + 21− 12 ) 5 1 1 = (9 + 5 + 0 + 3 + 9) = (26) = 5.2 5 5 Step 3. Computing the Modified Standard Scores. Power / Marcus Miller: (21 - 12) / 5.2 = 9/5.2 = 1.7307692 I Breathe In, I Breathe Out / Chris Cagle: (15 - 12) / 5.2 = 3/5.2 = 0.5769231 Blessed / Jill Scott: (12 - 12) / 5.2 = 0 Europa / Santana: (3 - 12) / 5.2 = -9 / 5.2 = -1.7307692 Santa Fe / Beirut: (7 - 12) / 5.2 = - 5 / 5.2 = -0.961538 4-26 CONTENT BASED FILTERING & CLASSIFICATION To normalize or not. Normalization makes sense when the scale of the features—the scales of the different dimensions—significantly varies. In the music example earlier in the chapter there were a number of features that ranged from one to five and then beats-per-minute that could potentially range from 60 to 180. In the dating example, there was also a mismatch of scale between the features of age and salary. Suppose I am dreaming of being rich and looking at homes in the Santa Fe, New Mexico area. asking price bedrooms bathrooms sq. ft. $1,045,000 2 2.0 1,860 $1,895,000 3 4.0 2,907 $3,300,000 6 7.0 10,180 $6,800,000 5 6.0 8,653 $2,250,000 3 2.0 1,030 The table on the left shows a few recent homes on the market. Here we see the problem again. Because the scale of one feature (in this case asking price) is so much larger than others it will dominate any distance calculation. Having two bedrooms or twenty will not have much of an effect on the total distance between two homes. We should normalize when 1. our data mining method calculates the distance between two entries based on the values of their features. 2. the scale of the different features is different (especially when it is drastically different—for ex., the scale of asking price compared to the scale of the number of bedrooms). Consider a person giving thumbs up and thumbs down ratings to news articles on a news site. Here a list representing a user’s ratings consists of binary values (1 = thumbs up; 0 = thumbs down): 4-27 Bill = {0, 0, 0, 1, 1, 1, 1, 0, 1, 0 … } Obviously there is no need to normalize this data. What about the Pandora case: all variables lie on a scale from 1 to 5 inclusive. Should we normalize or not? It probably wouldn't hurt the accuracy of the algorithm if we normalized, but keep in mind that there is a computational cost involved with normalizing. In this case, we might empirically compare results between using the regular and normalized data and select the best performing approach. Later in this chapter we will see a case where normalization reduces accuracy. Back to Pandora In the Pandora inspired example, we had each song represented by a number of attributes. If a user creates a radio station for Green Day we decide what to play based on a nearest neighbor approach. Pandora allows a user to give a particular tune a thumbs up or thumbs down rating. How do we use the information that a user gives a thumbs up for a particular song.? Suppose I use 2 attributes for songs: the amount of dirty guitar and the presence of a driving beat both rated on a 1-5 scale. A user has given the thumbs up to 5 songs indicating he liked the song (and indicated on the following chart with a 'L'); and a thumbs down to 5 songs indicating he disliked the song (indicated by a 'D'). Do you think the user will like or dislike the song indicated by the ‘?’ in this chart? 4-28 CONTENT BASED FILTERING & CLASSIFICATION I am guessing you said he would like the song. We base this on the fact that the ‘?’ is closer to the Ls in the chart than the Ds. We will spend the rest of this chapter and the next describing computational approaches to this idea. The most obvious approach is to find the nearest neighbor of the “?” and predict that it will share the class of the nearest neighbor. The question mark’s nearest neighbor is an L so we would predict that the ‘? tune’ is something the user would like. The Python nearest neighbor classifier code Let's use the example dataset I used earlier—ten tunes rated on 7 attributes (amount of piano, vocals, driving beat, blues influence, dirty electric guitar, backup vocals, rap influence). Piano Vocals Driving beat Blues infl. Dirty elec. Guitar Backup vocals Rap infl. Dr. Dog/ Fate 2.5 4 3.5 3 5 4 1 Phoenix/ Lisztomania 2 5 5 3 2 1 1 Heartless Bastards / Out at Sea 1 5 4 2 4 1 1 Todd Snider/ Don't Tempt Me 4 5 4 4 1 5 1 The Black Keys/ Magic Potion 1 4 5 3.5 5 1 1 Glee Cast/ Jessie's Girl 1 5 3.5 3 4 5 1 Black Eyed Peas/ Rock that Body 2 5 5 1 2 2 4 La Roux/ Bulletproof 5 5 4 2 1 1 1 2.5 4 4 1 1 1 1 1 5 3 2 1 2 1 Mike Posner/ Cooler than me Lady Gaga/ Alejandro 4-29 Earlier in this chapter we developed a Python representation of this data: music = {"Dr Dog/Fate": {"piano": 2.5, "vocals": 4, "beat": 3.5, "blues": 3, "guitar": 5, "backup vocals": 4, "rap": 1}, ! "Phoenix/Lisztomania": {"piano": 2, "vocals": 5, "beat": 5, "blues": 3, "guitar": 2, "backup vocals": 1, "rap": 1}, ! "Heartless Bastards/Out at Sea": {"piano": 1, "vocals": 5, "beat": 4, "blues": 2, ! ! ! ! ! ! "guitar": 4, "backup vocals": 1, "rap": 1}, ! "Todd Snider/Don't Tempt Me": {"piano": 4, "vocals": 5, "beat": 4, "blues": 4, ! ! ! ! ! ! "guitar": 1, "backup vocals": 5, "rap": 1}, Here the strings piano, vocals, beat, blues, guitar, backup vocals, and rap occur multiple times; if I have a 100,000 tunes those strings are repeated 100,000 times. I'm going to remove those strings from the representation of our data and simply use vectors: # # the item vector represents the attributes: piano, vocals, # beat, blues, guitar, backup vocals, rap # items = {"Dr Dog/Fate": [2.5, 4, 3.5, 3, 5, 4, 1], "Phoenix/Lisztomania": [2, 5, 5, 3, 2, 1, 1], "Heartless Bastards/Out at Sea": [1, 5, 4, 2, 4, 1, 1], "Todd Snider/Don't Tempt Me": [4, 5, 4, 4, 1, 5, 1], "The Black Keys/Magic Potion": [1, 4, 5, 3.5, 5, 1, 1], "Glee Cast/Jessie's Girl": [1, 5, 3.5, 3, 4, 5, 1], "La Roux/Bulletproof": [5, 5, 4, 2, 1, 1, 1], "Mike Posner": [2.5, 4, 4, 1, 1, 1, 1], "Black Eyed Peas/Rock That Body": [2, 5, 5, 1, 2, 2, 4], "Lady Gaga/Alejandro": [1, 5, 3, 2, 1, 2, 1]} 4-30 CONTENT BASED FILTERING & CLASSIFICATION In linear algebra, a vector is a quantity that has magnitude and direction. Various well defined operators can be performed on vectors including adding and subtracting vectors and scalar multiplication. In data mining, a vector is simply a list of numbers that represent the attributes of an object. The example on the previous page represented attributes of a song as a list of numbers. Another example, would be representing a text document as a vector—each position of the vector would represent a particular word and the number at that position would represent how many times that word occurred in the text. Plus, using the word “vector” instead of “list of attributes” is cool! Once we define attributes this way, we can perform vector operations (from linear algebra) on them. 4-31 In addition to representing the attributes of a song as a vector, I need to represent the thumbs up/ thumbs down ratings that users gives to songs. Because each user doesn't rate all songs (sparse data) I will go with the dictionary of dictionaries approach: users = {"Angelica": {"Dr Dog/Fate": "L", "Phoenix/Lisztomania": "L", "Heartless Bastards/Out at Sea": "D", "Todd Snider/Don't Tempt Me": "D", "The Black Keys/Magic Potion": "D", "Glee Cast/Jessie's Girl": "L", "La Roux/Bulletproof": "D", "Mike Posner": "D", "Black Eyed Peas/Rock That Body": "D", "Lady Gaga/Alejandro": "L"}, "Bill": {"Dr Dog/Fate": "L", "Phoenix/Lisztomania": "L", "Heartless Bastards/Out at Sea": "L", "Todd Snider/Don't Tempt Me": "D", "The Black Keys/Magic Potion": "L", "Glee Cast/Jessie's Girl": "D", "La Roux/Bulletproof": "D", "Mike Posner": "D", "Black Eyed Peas/Rock That Body": "D", "Lady Gaga/Alejandro": "D"} } My way of representing ‘thumbs up’ as L for like and ‘thumbs down’ as D is arbitrary. You could use 0 and 1, like and dislike. In order to use the new vector format for songs I need to revise the Manhattan Distance and the computeNearestNeighbor functions. def manhattan(vector1, vector2): """Computes the Manhattan distance.""" distance = 0 total = 0 n = len(vector1) for i in range(n): distance += abs(vector1[i] - vector2[i]) return distance 4-32 CONTENT BASED FILTERING & CLASSIFICATION def computeNearestNeighbor(itemName, itemVector, items): """creates a sorted list of items based on their distance to item""" distances = [] for otherItem in items: if otherItem != itemName: distance = manhattan(itemVector, items[otherItem]) distances.append((distance, otherItem)) # sort based on distance -- closest first distances.sort() return distances Finally, I need to create a classify function. I want to predict how a particular user would rate an item represented by itemName and itemVector. For example: "Chris Cagle/ I Breathe In. I Breathe Out" [1, 5, 2.5, 1, 1, 5, 1] (NOTE: To better format the Python example below, I will use the string Cagle to represent that singer and song pair.) The first thing the function needs to do is find the nearest neighbor of this Chris Cagle tune. Then it needs to see how the user rated that nearest neighbor and predict that the user will rate Chris Cagle the same. Here's my rudimentary classify function: def classify(user, itemName, itemVector): """Classify the itemName based on user ratings Should really have items and users as parameters""" # first find nearest neighbor nearest = computeNearestNeighbor(itemName, itemVector, items)[0][1] rating = users[user][nearest] return rating Ok. Let's give this a try. I wonder if Angelica will like Chris Cagle's I Breathe In, I Breathe Out? classify('Angelica', 'Cagle', [1, 5, 2.5, 1, 1, 5, 1]) "L" We are predicting she will like it! Why are we predicting that? 4-33 computeNearestNeighbor('Angelica', 'Cagle', [1, 5, 2.5, 1, 1, 5, 1]) [(4.5, 'Lady Gaga/Alejandro'), (6.0, "Glee Cast/Jessie's Girl"), (7.5, "Todd Snider/Don't Tempt Me"), (8.0, 'Mike Posner'), (9.5, 'Heartless Bastards/Out at Sea'), (10.5, 'Black Eyed Peas/Rock That Body'), (10.5, 'Dr Dog/Fate'), (10.5, 'La Roux/Bulletproof'), (10.5, 'Phoenix/ Lisztomania'), (14.0, 'The Black Keys/Magic Potion')] We are predicting that Angelica will like Chris Cagle's I Breathe In, I Breathe Out because that tune's nearest neighbor is Lady Gaga’s Alejandro and Angelica liked that tune. What we have done here is build a classifier—in this case, our task was to classify tunes as belonging to one of two groups—the like group and the dislike group. Attention, Attention. We just built a classifier!! 4-34 CONTENT BASED FILTERING & CLASSIFICATION A classifier is a program that uses an object’s attributes to determine what group or class it belongs to! A classifier uses a set of objects that are already labeled with the class they belong to. It uses that set to classify new, unlabeled objects. So in our example, we knew about songs that Angelica liked (labeled ‘liked’) and songs she did not like. We wanted to predict whether Angelica would like a Chris Cagle tune. First we found a song Angelica rated that was m ost similar to the Chris Cagle tu ne. It was Lady Gaga’ s Alejandro I like Phoenix, Lady Gaga and Dr. Dog. I don’t like The Black Keys and Mike Posner! Next, we checked whether Angelica liked or disliked the Alejandro—sh e liked it. So we pr edict that Angelic a will also like the Chris Cagle tune ,I Breathe In, I Brea the Out. Classifiers can be used in a wide range of applications. The following page lists just a few. 4-35 Twitter Sentiment Classification A number of people are working on classifying the sentiment (a positive or negative opinion) in tweets. This can be used in a variety of ways. For example, if Axe releases a new underarm deoderant, they can check whether people like it or not. The attributes are the words in the tweet. Classification for Targeted Political Ads This is called microtargeting. People are classified into such groups as “Barn Raisers”, “Inner Compass”, and “Hearth Keepers.” Hearth Keepers, for example, focus on their family and keep to themselves. Health and the Quantified Self It’s the start of the quanitifed self explosion. We can now buy simple devices like the Fitbit, and the Nike Fuelband. Intel and other companies are working on intelligent homes that have floors that can weigh us, keep track of our movements and alert someone if we deviate from normal. Experts are predicting that in a few years we will be wearing tiny compu-patches that can monitor dozens of factors in real time and make instant classifications. Automatic identification of people in photos. There are apps now that can identify and tag your friends in photographs. (And the same techniques apply to identifying people walking down the street using public video cams.) Techniques vary but some of them use attributes like the relative position and size of a person’s eyes, nose, jaw, etc. Targeted Marketing Similar to political microtargeting. Instead of a broad advertising campaign to sell my expensive Vegas time share luxury condos, can I identify likely buyers and market just to them? Even better if I can identify subgroups of likely buyers and I can really tailor my ads to specific groups. The list is endless • classifying people as terrorist or nonterrorist • automatic classification of email (hey, this email looks pretty important; this is regular email; this looks like spam) • predicting medical clinical outcomes • identifying financial fraud (for ex., credit card fraud) 4-36 CONTENT BASED FILTERING & CLASSIFICATION What sport? To give you a preview of what we will be working on in the next few chapters let us work with an easier example than those given on the previous page—classifying what sport various world-class women athletes play based solely on their height and weight. In the following table I have a small sample dataset drawn from a variety of web sources. Name Sport Age Height Weight Asuka Teramoto Gymnastics 16 54 66 Brittainey Raven Basketball 22 72 162 Chen Nan Basketball 30 78 204 Gabby Douglas Gymnastics 16 49 90 Helalia Johannes Track 32 65 99 Irina Miketenko Track 40 63 106 Jennifer Lacy Basketball 27 75 175 Kara Goucher Track 34 67 123 Linlin Deng Gymnastics 16 54 68 Nakia Sanford Basketball 34 76 200 Nikki Blue Basketball 26 68 163 Qiushuang Huang Gymnastics 20 61 95 Rebecca Tunney Gymnastics 16 58 77 Rene Kalmer Track 32 70 108 Shanna Crossley Basketball 26 70 155 Shavonte Zellous Basketball 24 70 155 Tatyana Petrova Track 29 63 108 Tiki Gelana Track 25 65 106 Valeria Straneo Track 36 66 97 Viktoria Komova Gymnastics 17 61 76 4-37 The gymnastic data lists some of the top participants in the 2012 and 2008 Olympics. The basketball players play for teams in the WNBA. The women track stars were finishers in the 2012 Olympic marathon . Granted this is a trivial example but it will allow us to apply some of the techniques we have learned. As you can see, I've included age in the table. Just scanning the data you can see that age alone is a moderately good predictor. Try to guess the sports of these athletes. McKayla Maroney; Age 16 Candace Parker; Age 26 Lisa Jane Weightman; Age 34 Olivera Jevtić: Age 35 4-38 CONTENT BASED FILTERING & CLASSIFICATION The answers Candace Parker plays basketball for the WNBA’s Los Angeles Sparks and Russia’s UMMC Ekaterinburg. McKayla Maroney was a member of the U.S. Women’s Gymnastic Team and won a Gold and a Silver. Olivera Jevtić is a Serbian long-distance runner who competed in the 2008 and 2012 Olympics. Lisa Jane Weightman is an Australian long-distance runner who also competed in the 2008 and 2012 Olympics. You just performed classification—you predicted the class of objects based on their attributes. (In this case, predicting the sport of athletes based on a single attribute, age.) k brain calisthenics Suppose I want to guess what sport a person plays based on their height and weight. My database is small—only two people. Nakia Sanford, the center for the Women’s National Basketball Association team Phoenix Mercury, is 6’4” and weighs 200 pounds. Sarah Beale, a forward on England’s National Rugby Team, is 5’10” and weighs 190. Based on that database, I want to classify Catherine Spencer as either a basketball player or rugby player. She is 5’10” and weighs 200 pounds. What sport do you think she plays? 4-39 k brain calisthenics - cont’d If you said rugby, you would be correct. Catherine Spencer is a forward on England’s national team. However, if we based our guess on a distance formula like Manhattan Distance we would be wrong. The Manhattan Distance between Catherine and Basketball player Nakia is 6 (they weigh the same and have a six inch difference in height). The distance between Catherine and Rugby player Sarah is 10 (their height is the same and they differ in weight by 10 pounds). So we would pick the closest person, Nakia, and predict Catherine plays the same sport. Is there anything we learned that could help us make more accurate classifications? Hmmm. This rings a bell. I think there was something related to this earlier in the chapter... 4-40 CONTENT BASED FILTERING & CLASSIFICATION k brain calisthenics - cont’d We can use the Modified Standard Score!!! (each value) - (median) (absolute standard deviation) Test Data. Let us remove age from the picture. Here is a group of individuals I would like to classify: Name Height Weight Crystal Langhorne 74 190 Li Shanshan 64 101 Kerri Strug 57 87 60 97 Kelly Miller 70 140 Zhu Xiaolin 67 123 Lindsay Whalen 69 169 Koko Tsurumi 55 75 Paula Radcliffe 68 120 Erin Thorn 69 144 Jaycie Phelps Sport ` Let’s build a classifier! 4-41 Python Coding Instead of hard-coding the data in the Python code, I decided to put the data for this example into two files: athletesTrainingSet.txt and athletesTestSet.txt. I am going to use the data in the athletesTrainingSet.txt file to build the classifier. The data in the athletesTestSet.txt file will be used to evaluate this classifier. In other words, each entry in the test set will be classified by using all the entries in the training set. The data files and the Python code are on the book’s website, guidetodatamining.com. The format of these files looks like this: Asuka Teramoto Gymnastics 54 66 Brittainey Raven Basketball 72 162 Chen Nan Basketball 78 204 Gabby Douglas Gymnastics 49 90 Each line of the text represents an object described as a tab-separated list of values. I want my classifier to use a person’s height and weight to predict what sport that person plays. So the last two columns are the numerical attributes I will use in the classifier and the second column represents the class that object is in. The athlete’s name is not used by the classifier. I don’t try to predict what sport a person plays based on their name and I am not trying to predict the name from some attributes. Hey, you look what... maybe five foot eleven and 150? I bet your name is Clara Coleman. 4-42 CONTENT BASED FILTERING & CLASSIFICATION However, keeping the name might be useful as a means of explaining the classifier’s decision to users: “We think Amelia Pond is a gymnast because she is closest in height and weight to Gabby Douglas who is a gymnast.” As I said, I am going to write my Python code to not be so hard coded to a particular example (for example, to only work for the athlete example). To help meet this goal I am going to add an initial header line to the athlete training set file that will indicate the function of each column. Here are the first few lines of that file: comment class num num Asuka Teramoto Gymnastics 54 66 Brittainey Raven Basketball 72 162 Any column labeled comment will be ignored by the classifier; a column labeled class represents the class of the object, and columns labeled num indicate numerical attributes of that object. k brain calisthenics How do you think we should represent this data in Python? Here are some possibilities (or come up with your own representation). a dictionary of the form: {'Asuka Termoto': ('Gymnastics', [54, 66]), 'Brittainey Raven': ('Basketball', [72, 162]), ... a list of lists of the form: [['Asuka Termoto', 'Gymnastics', 54, 66], ['Brittainey Raven', 'Basketball', 72, 162], ... a list of tuples of the form: [('Gymnastics', [54, 66], ['Asuka Termoto']), ('Basketball', [72, 162], ['Brittainey Raven'],... 4-43 k brain calisthenics - answer a dictionary of the form: {'Asuka Termoto': ('Gymnastics', [54, 66]), 'Brittainey Raven': ('Basketball', [72, 162]), ... This is not a very good representation of our data. The key for the dictionary is the athlete’s name, which we do not even use in the calculations. a list of lists of the form: [['Asuka Termoto', 'Gymnastics', 54, 66], ['Brittainey Raven', 'Basketball', 72, 162], ... This is not a bad representation. It mirrors the input file and since the nearest neighbor algorithm requires us to iterate through the list of objects, a list makes sense. a list of tuples of the form: [('Gymnastics', [54, 66], ['Asuka Termoto']), ('Basketball', [72, 162], ['Brittainey Raven'],... I like this representation better than the above since it separates the attributes into their own list and makes the division between class, attributes, and comments precise. I made the comment (the name in this case) a list since there could be multiple columns that are comments. 4-44 CONTENT BASED FILTERING & CLASSIFICATION My python code that reads in a file and converts it to the format [('Gymnastics', [54, 66], ['Asuka Termoto']), ('Basketball', [72, 162], ['Brittainey Raven'],... looks like this: class Classifier: def __init__(self, filename): self.medianAndDeviation = [] # reading the data in from the file f = open(filename) lines = f.readlines() f.close() self.format = lines[0].strip().split('\t') self.data = [] for line in lines[1:]: fields = line.strip().split('\t') ignore = [] vector = [] for i in range(len(fields)): if self.format[i] == 'num': vector.append(int(fields[i])) elif self.format[i] == 'comment': ignore.append(fields[i]) elif self.format[i] == 'class': classification = fields[i] self.data.append((classification, vector, ignore)) 4-45 s code it Before we can standardize the data using the Modified Standard Score we need methods that will compute the median and absolute standard deviation of numbers in a list: >>> >>> >>> 65 >>> >>> 8.0 heights = [54, 72, 78, 49, 65, 63, 75, 67, 54] median = classifier.getMedian(heights) median asd = classifier.getAbsoluteStandardDeviation(heights, median) asd Can you write these methods? ? nError tio Asser xt See ne page Download the template testMedianAndASD.py to write and test these methods at guidetodatamining.com 4-46 CONTENT BASED FILTERING & CLASSIFICATION Assertion Errors and the Assert statement. It is important that each component of a solution to a problem be turned into a piece of code that implements it and a piece of code that tests it. In fact, it is good practice to write the test code before you write the implementation. The code template I have provided contains a test function called unitTest. A simplified version of that function, showing only one test, is shown here: def unitTest(): list1 = [54, 72, 78, 49, 65, 63, 75, 67, 54] classifier = Classifier('athletesTrainingSet.txt') m1 = classifier.getMedian(list1) assert(round(m1, 3) == 65) print("getMedian and getAbsoluteStandardDeviation work correctly") The getMedian function you are to complete initially looks like this: def getMedian(self, alist): """return median of alist""" """TO BE DONE""" return 0 So initially, getMedian returns 0 as the median for any list. You are to complete getMedian so it returns the correct value. In the unitTest procedure, I call getMedian with the list [54, 72, 78, 49, 65, 63, 75, 67, 54] The assert statement in unitTest says the value returned by getMedian should equal 65. If it does, execution continues to the next line and getMedian and getAbsoluteStandardDeviation work correctly is printed. If they are not equal the program terminates with an error: 4-47 File "testMedianAndASD.py", line 78, in unitTest assert(round(m1, 3) == 65) AssertionError If you download the code from the book’s website and run it without making any changes, you will get this error. Once you have correctly implemented getMedian and getAbsoluteStandardDeviation this error will disappear. This use of assert as a means of testing software components is a common technique among software developers. “it is important that each part of the specification be turned into a piece of code that implements it and a test that tests it. If you don’t have tests like these then you don’t know when you are done, you don’t know if you got it right, and you don’t know that any future changes might be breaking something.” - Peter Norvig 4-48 CONTENT BASED FILTERING & CLASSIFICATION Solution Here is one way of writing these algorithms: def getMedian(self, alist): """return median of alist""" if alist == []: return [] blist = sorted(alist) length = len(alist) if length % 2 == 1: # length of list is odd so return middle element return blist[int(((length + 1) / 2) - 1)] else: # length of list is even so compute midpoint v1 = blist[int(length / 2)] v2 =blist[(int(length / 2) - 1)] return (v1 + v2) / 2.0 def getAbsoluteStandardDeviation(self, alist, median): """given alist and median return absolute standard deviation""" sum = 0 for item in alist: sum += abs(item - median) return sum / len(alist) As you can see my getMedian method first sorts the list before finding the median. Because I am not working with huge data sets I think this is a fine solution. If I wanted to optimize my code, I might replace this with a selection algorithm. Right now, the data is read from the file athletesTrainingSet.txt and stored in the list data in the classifier with the following format: [('Gymnastics', ('Basketball', ('Basketball', ('Gymnastics', [54, [72, [78, [49, 66], ['Asuka Teramoto']), 162], ['Brittainey Raven']), 204], ['Chen Nan']), 90], ['Gabby Douglas']), ... 4-49 Now I would like to normalize the vector so the list data in the classifier contains normalized values. For example, [('Gymnastics', [-1.93277, -1.21842], ['Asuka Teramoto']), ('Basketball', [1.09243, 1.63447], ['Brittainey Raven']), ('Basketball', [2.10084, 2.88261], ['Chen Nan']), ('Gymnastics', [-2.77311, -0.50520], ['Gabby Douglas']), ('Track', [-0.08403, -0.23774], ['Helalia Johannes']), ('Track', [-0.42017, -0.02972], ['Irina Miketenko']), To do this I am going to add the following lines to my init method: # get length of instance vector self.vlen = len(self.data[0][1]) # now normalize the data for i in range(self.vlen): self.normalizeColumn(i) In the for loop we want to normalize the data, column by column. So the first time through the loop we will normalize the height column, and the next time through, the weight column. s code it Can you write the normalizeColumn method? Download the template normalizeColumnTemplate.py to write and test this method at guidetodatamining.com 4-50 CONTENT BASED FILTERING & CLASSIFICATION Solution Here is an implementation of the normalizeColumn method: def normalizeColumn(self, columnNumber): """given a column number, normalize that column in self.data""" # first extract values to list col = [v[1][columnNumber] for v in self.data] median = self.getMedian(col) asd = self.getAbsoluteStandardDeviation(col, median) #print("Median: %f ASD = %f" % (median, asd)) self.medianAndDeviation.append((median, asd)) for v in self.data: v[1][columnNumber] = (v[1][columnNumber] - median) / asd You can see I also store the median and absolute standard deviation of each column in the list medianAndDeviation. I use this information when I want to use the classifier to predict the class of a new instance. For example, suppose I want to predict what sport is played by Kelly Miller, who is 5 feet 10 inches and weighs 170. The first step is to convert her height and weight to Modified Standard Scores. That is, her original attribute vector is [70, 140]. After processing the training data, the value of meanAndDeviation is [(65.5, 5.95), (107.0, 33.65)] meaning the data in the first column of the vector has a median of 65.5 and an absolute standard deviation of 5.95; the second column has a median of 107 and a deviation of 33.65. I use this info to convert the original vector [70,140] to one containing Modified Standard Scores. This computation for the first attribute is mss = xi − x! 70 − 65.5 4.5 = = = 0.7563 asd 5.95 5.95 4-51 and the second: mss = xi − x! 140 − 107 33 = = = 0.98068 asd 33.65 33.65 The python method that does this is: def normalizeVector(self, v): """We have stored the median and asd for each column. We now use them to normalize vector v""" vector = list(v) for i in range(len(vector)): (median, asd) = self.medianAndDeviation[i] vector[i] = (vector[i] - median) / asd return vector The final bit of code to write is the part that predicts the class of a new instance—in our current example, the sport a person plays. To determine the sport played by Kelly Miller, who is 5 feet 10 inches (70 inches) and weighs 170 we would call classifier.classify([70, 170]) In my code, classify is just a wrapper method for nearestNeighbor: def classify(self, itemVector): """Return class we think item Vector is in""" return(self.nearestNeighbor(self.normalizeVector(itemVector))[1][0]) s code it Can you write the nearestNeighbor method? (for my solution, I wrote an additional method, manhattanDistance.) Yet again, download the template classifyTemplate.py to write and test this method at guidetodatamining.com. 4-52 CONTENT BASED FILTERING & CLASSIFICATION Solution The implementation of the nearestNeighbor methods turns out to be very short. def manhattan(self, vector1, vector2): """Computes the Manhattan distance.""" return sum(map(lambda v1, v2: abs(v1 - v2), vector1, vector2)) def nearestNeighbor(self, itemVector): """return nearest neighbor to itemVector""" return min([ (self.manhattan(itemVector, item[1]), item) for item in self.data]) That’s it!!! We have written a nearest neighbor classifier in roughly 200 lines of Python. 4-53 In the complete code which you can download from our website, I have included a function, test, which takes as arguments a training set file and a test set file and prints out how well the classifier performed. Here is how well the classifier did on our athlete data: >>> test("athletesTrainingSet.txt", "athletesTestSet.txt") - Track + + Aly Raisman! Gymnastics! 62! 115 Basketball Crystal Langhorne! Basketball! 74! 190 Basketball Diana Taurasi! Basketball! 72! 163- Track Hannah Whelan! Gymnastics! 63! 117 + Gymnastics Jaycie Phelps! Gymnastics! 60! 97 80.00% correct As you can see, the classifier was 80% accurate. It performed perfectly on predicting basketball players but made four errors between track and gymnastics. Irises Data Set I also tested our simple classifier on the Iris Data Set, arguably the most famous data set used in data mining. It was used by Sir Ronald Fisher back in the 1930s. The Iris Data Set consists of 50 samples for each of three species of Irises (Iris Setosa, Iris Virginica, and Iris Versicolor). The data set includes measurements for two parts of the Iris’s flower: the sepal (the green covering of the flower bud) and the petals. 4-54 Sir Fisher was a remarkable person. He revolutionized statistics and Richard Dawkins called him “the greatest biologist since Darwin.” CONTENT BASED FILTERING & CLASSIFICATION All the data sets described in the book are available on the book’s website: guidetodatamining.com. This allows you to download the data and experiment with the algorithm. Does normalizing the data improve or worsen the accuracy? Does having more data in the training set improve results? What effect does switching to Euclidean Distance have? REMEMBER: Any learning that takes place happens in your brain, not mine. The more you interact with the material in the book, the more you will learn. The Iris data set looks like this (species is what the classifier is trying to predict): Sepal length Sepal width Petal Length Petal Width Species 5.1 3.5 1.4 0.2 l.setosa 4.9 3.0 1.4 0.2 l setosa 4-55 There were 120 instances in the training set and 30 in the test set (none of the test set instances were in the training set). How well did our classifier do on the Iris Data Set? >>> test('irisTrainingSet.data', 'irisTestSet.data') 93.33% correct Again, a fairly impressive result considering how simple our classifier is. Interestingly, without normalizing the data the classifier is 100% accurate. We will explore this normalization problem in more detail in a later chapter. miles per gallon. Finally, I tested our classifier on a modified version of another widely used data set, the Auto Miles Per Gallon data set from Carnegie Mellon University. It was initially used in the 1983 American Statistical Association Exposition. The format of the data looks like this mpg cylinders c.i. HP weight secs. 0-60 make/model 30 4 68 49 1867 19.5 fiat 128 45 4 90 48 2085 21.7 vw rabbit (diesel) 20 8 307 130 3504 12 chevrolet chevelle malibu In the modified version of the data, we are trying to predict mpg, which is a discrete category (with values 10, 15, 20, 25, 30, 35, 40, and 45) using the attributes cylinders, displacement, horsepower, weight, and acceleration. 4-56 CONTENT BASED FILTERING & CLASSIFICATION There are 342 instances of cars in the training set and 50 in the test set. If we just predicted the miles per gallon randomly, our accuracy would be 12.5%. >>> test('mpgTrainingSet.txt', 'mpgTestSet.txt') 56.00% correct Without normalization the accuracy is 32%. How can we improve the accuracy of our predictions? Will improving the classification algorithm help? How about increasing the size of our training set? How about having more attributes. Tune in to the next chapter to find out! 4-57 odds and ends Heads Up on Normalization In this chapter we talked the importance of normalizing data. This is critical when attributes have drastically different scales (for example, income and age). In order to get accurate distance measurements, we should rescale the attributes so they all have the same scale. While most data miners use the term ‘normalization’ to refer to this rescaling, others make a distinction between ‘normaliza-tion’ and ‘standardization.’ For them, normalization means scaling values so they lie on a scale from 0 to 1. Standardization, on the other hand, refers to scaling an attribute so the average (mean or median) is 0, and other values are deviations from this average (standard deviation or absolute standard deviation). So for these data miners, Standard Score and Modified Standard Score are examples of standardization. Recall that one way to normalize an attribute on a scale between 0 and 1 is to find the minimum (min) and maximum (max) values of that attribute. The normalized value of a value is then value − min max− min 4-58 Let’s compare the accuracy of a classifer that uses this formula over one that uses the Modified Standard CONTENT BASED FILTERING & CLASSIFICATION L You say normalize and I say standardize N You say tomato and I say tomato s M code it Can you modify our classifier code so that it normalizes the attributes using the formula on our previous page? You can test its accuracy with our three data sets: classifier built data set using no normalization using the formula on previous page using Modified Standard Score Athletes 80.00% ? 80.00% Iris 100.00% ? 93.33% MPG 32.00% ? 56.00% 4-59 s my results Here are my results: classifier built data set using no normalization using the formula on previous page using Modified Standard Score Athletes 80.00% 60.00% 80.00% Iris 100.00% 83.33% 93.33% MPG 32.00% 36.00% 56.00% Hmm. These are disappointing results compared with using Modified Standard Score. It is fun playing with data sets and trying different methods. I obtained the Iris and MPG data sets from the UCI Machine Learning Repository (archive.ics.uci.edu/ml). I encourage you to go there, download a data set or two, convert the data to match data format, and see how well our classifier does. 4-60 Chapter 5: Further Explorations in Classification Evaluating algorithms and kNN Let us return to the athlete example from the previous chapter. In that example we built a classifier which took the height and weight of an athlete as input and classified that input by sport—gymnastics, track, or basketball. So Marissa Coleman, pictured on the left, is 6 foot 1 and weighs 160 pounds. Our classifier correctly predicts she plays basketball: >>> cl = Classifier('athletesTrainingSet.txt') >>> cl.classify([73, 160]) 'Basketball' and predicts that someone 4 foot 9 and 90 pounds is likely to be a gymnast: >>> cl.classify([59, 90]) 'Gymnastics' Once we build a classifier, we might be interested in answering some questions about it such as: How accurate is the classifier? How good is this classifier? How does this classifier compare with others? How can we answer these questions? Training set and test set. At the end of the previous chapter we worked with three different datasets: the women athlete dataset, the iris dataset, and the auto miles-per-gallon one. We divided each of these datasets in turn into two subsets. One subset we used to construct the classifier. This data set is called the training set. The other set was used to evaluate the classifier. That data is called the test set. Training set and test set are common terms in data mining. 5-2 EVALUATION AND KNN People in data mining never test with the data they used to train the system. You can see why we don't use the training data for testing if we consider the nearest neighbor algorithm. If Marissa Coleman the basketball player from the above example, was in our training data, she at 6 foot 1 and 160 pounds would be the nearest neighbor of herself. So when evaluating a nearest neighbor algorithm, if our test set is a subset of our training data we would always be close to 100% accurate. More generally, in evaluating any data mining algorithm, if our test set is a subset of our training data the results will be optimistic and often overly optimistic. So that doesn’t seem like a great idea. How about the idea we used in the last chapter? We divide our data into two parts. The larger part we use for training and the smaller part we use for evaluation. As it turns out that has its problems too. We could be extremely unlucky in how we divide up our data. For example, all the basketball players in our test set might be short (like Debbie Black who is only 5 foot 3 and weighs 124 pounds) and get classified as marathoners. And all the track people in the test set might be short and lightweight for that sport like Tatyana Petrova (5 foot 3 and 108 pounds) and get classified as gymnasts. With a test set like this, our accuracy will be poor. On the other hand, we could be very lucky in our selection of a test set. Every person in the test set is the prototypical height and weight for their respective sports and our accuracy is near 100%. In either case, the accuracy based on a single test set may not reflect the true accuracy when our classifier is used with new data. A solution to this problem might be to repeat the process a number of times and average the results. For example, we might divide the data in half. Let’s call the parts Part 1 and Part 2: Data set Part 1 Part 2 5-3 We can use the data in Part 1 to train our classifier and the data in Part 2 to test it. Then we will repeat the process, this time training with Part 2 and testing with Part 1. Finally we average the results. One problem with this though, is that we are only using 1/2 the data for training during each iteration. But we can fix this by increasing the number of parts. For example, we can have three parts and for each iteration we will train on 2/3 of the data and test on 1/3. So it might look like this iteration 1 train with parts 1 and 2 test with part 3 iteration 2 train with parts 1 and 3 test with part 2 iteration 3 train with parts 2 and 3 test with part 1 Average the results. In data mining, the most common number of parts is 10, and this method is called ... 10-fold Cross Validation With this method we have one data set which we divide randomly into 10 parts. We use 9 of those parts for training and reserve one tenth for testing. We repeat this procedure 10 times each time reserving a different tenth for testing. Let’s look at an example. Suppose I want to build a classifier that just answers yes or no to the question Is this person a professional basketball player? My data consists of information about 500 basketball players and 500 non-basketball players. 5-4 EVALUATION AND KNN ten-fold cross validation example: Step 1, we equally divide the data into 10 buckets: Data So we will put 50 basketball players in each bucket and 50 non-players. Each bucket holds information on 100 individuals. Step 2, we iterate through the following steps ten times: 2.1. During each iteration hold back one of the buckets. For iteration 1, we will hold back bucket 1, iteration 2, bucket 2, and so on. 2.2 We will train the classifier with data from the other buckets. (during the first iteration we will train with the data in buckets 2 through 10). 2.3 We will test the classifier we just built using data from the bucket we held back and save the results. In our case these results might be: 35 of the basketball players were classified correctly 29 of the non basketball players were classified correctly Step 3, we sum up the results. 5-5 Often we will put the final results in a table that looks like this: classified as a basketball player classified as not a basketball player really a basketball player 372 128 really not a basketball player 220 280 So of the 500 basketball players 372 of them were classified correctly. One thing we could do is add things up and say that of the 1,000 people we classified 652 (372 + 280) of them correctly. So our accuracy is 65.2%. The measures we obtain using ten-fold cross-validation are more likely to be truly representative of the classifiers performance compared with twofold, or three-fold cross-validation. This is so, because each time we train the classifier we are using 90% of our data compared with using only 50% for two-fold cross-validation. Hmmm. I have an idea. If 10-fold cross validation is good because we are training on 90% of the data, how about using n-fold cross validation where n is the number of entries in our data set? For example, if we have 1,000 entries, we will train our classifier on 999 of them and test on 1, and repeat this process 1,000 times. Using the largest possible amount of our data for training should result in a highly accurate classifier. 5-6 EVALUATION AND KNN Leave-One-Out In the machine learning literature, n-fold cross validation (where n is the number of samples in our data set) is called leave-one-out. We already mentioned one benefit of leave-one-out— at every iteration we are using the largest possible amount of our data for training. The other benefit is that it is deterministic. What do we mean by ‘deterministic’? Suppose Lucy spends an intense 80 hour week creating and coding a new classifier. It is Friday and she is exhausted so she asks two of her colleagues (Emily and Li) to evaluate the classifier over the weekend. She gives each of them the classifier and the same dataset and asks them to use 10-fold cross validation. On Monday she asks for the results ... I am happy to report that the classifier was 73.69% accurate!! The classifier was only 71.27% accuate. 5-7 Hmm. They did not get the same results. Did Emily or Li make a mistake? Not necessarily. In 10-fold cross validation we place the data randomly into 10 buckets. Since there is this random element, it is likely that Emily and Li did not divide the data into buckets in exactly the same way. In fact, it is highly unlikely that they did. So when they train the classifier, they are not using exactly the same data and when they test this classifier they are using different test sets. So it is quite logical that they would get different results. This result has nothing to do with the fact that two different people were performing the evaluation. If Lucy herself ran 10-fold cross validation twice, she too would get slightly different results. The reason we get different results is that there is a random component to placing the data into buckets. So 10fold cross validation is called non-deterministic because when we run the test again we are not guaranteed to get the same result. In contrast, the leave-one-out method is deterministic. Every time we use leave-one-out on the same classifier and the same data we will get the same result. That is a good thing! The disadvantages of leave-one-out The main disadvantage of leave-one-out is the computational expense of the method. Consider a modest-sized dataset of 1,000 instances and that it takes one minute to train a classifier. For 10-fold cross validation we will spend 10 minutes in training. In leave-one-out we will spend 16 hours in training. If our dataset contains a million entries the total time spent in training would nearly be two years. Eeeks! I’ll get that report to you in two years! 5-8 EVALUATION AND KNN The other disadvantage of leave-one-out is related to stratification. Stratification. Let us return to an example from the previous chapter—building a classifier that predicts what sport a woman plays (basketball, gymnastics, or track). When training the classifier we want the training data to be representative and contain data from all three classes. Suppose we assign data to the training set in a completely random way. It is possible that no basketball players would be included in the training set and because of this, the resulting classifier would not be very good at classifying basketball players. Or consider creating a data set of 100 athletes. First we go to the Women’s NBA website and write down the info on 33 basketball players; next we go to Wikipedia and get 33 women who competed in gymnastics, at the 2012 Olympics and write that down; finally, we go again to Wikipedia to get information on women who competed in track at the Olympics and record data for 34 people. So our dataset looks like this: comment class Asuka Teramoto Brittainey Raven Chen Nan Basketball Gabby Douglas Gymnastics Helalia Johannes Irina Miketenko Track Jennifer Lacy Basketball Kara Goucher Track Linlin Deng Gymnastics Nakia Sanford Basketball Nikki Blue Basketball Qiushuang Huang Rebecca Tunney Rene Kalmer Track Shanna Crossley Shavonte Zellous Tatyana PetrovaTrack Tiki Gelana Track Valeria Straneo Track Viktoria Komova comment class Asuka Teramoto Brittainey Raven Chen Nan Basketball Gabby Douglas Gymnastics Helalia Johannes Irina Miketenko Track Jennifer Lacy Basketball Kara Goucher Track Linlin Deng Gymnastics Nakia Sanford Basketball Nikki Blue Basketball Qiushuang Huang Rebecca Tunney Rene Kalmer Track Shanna Crossley Shavonte Zellous Tatyana PetrovaTrack Tiki Gelana Track Valeria Straneo Track Viktoria Komova comment class Asuka Teramoto Brittainey Raven Chen Nan Basketball Gabby Douglas Gymnastics Helalia Johannes Irina Miketenko Track Jennifer Lacy Basketball Kara Goucher Track Linlin Deng Gymnastics Nakia Sanford Basketball Nikki Blue Basketball Qiushuang Huang Rebecca Tunney Rene Kalmer Track Shanna Crossley Shavonte Zellous Tatyana PetrovaTrack Tiki Gelana Track Valeria Straneo Track Viktoria Komova num Gymnastics Basketball 78 49 Track 63 75 67 54 76 68 Gymnastics Gymnastics 70 Basketball Basketball 63 65 66 Gymnastics num Gymnastics Basketball 78 49 Track 63 75 67 54 76 68 Gymnastics Gymnastics 70 Basketball Basketball 63 65 66 Gymnastics num Gymnastics Basketball 78 49 Track 63 75 67 54 76 68 Gymnastics Gymnastics 70 Basketball Basketball 63 65 66 Gymnastics num 54 72 204 90 65 106 175 123 68 200 163 61 58 108 70 70 108 106 97 61 num 54 72 204 90 65 106 175 123 68 200 163 61 58 108 70 70 108 106 97 61 num 54 72 204 90 65 106 175 123 68 200 163 61 58 108 70 70 108 106 97 61 66 162 99 33 women baskball players 95 77 155 155 76 66 162 99 33 women gymnasts 95 77 155 155 76 66 162 99 95 77 34 women marathoners 155 155 76 5-9 Let’s say we are doing 10-fold cross validation. We start at the beginning of the list and put every ten people in a different bucket. In this case we have 10 basketball players in both the first and second buckets. The third bucket has both basketball players and gymnasts. The fourth and fifth buckets solely contain gymnasts and so on. None of our buckets are representative of the dataset as a whole and you would be correct in thinking this would skew our results. The preferred method of assigning instances to buckets is to make sure that the classes (basketball players, gymnasts, marathoners) are representing in the same proportions as they are in the complete dataset. Since one-third of the complete dataset consists of basketball players, one-third of the entries in each bucket should also be basketball players. And one-third the entries should be gymnasts and one-third marathoners. This is called stratification and this is a good thing. The problem with the leave-one-out evaluation method is that necessarily all the test sets are non-stratified since they contain only one instance. In sum, while leave-one-out may be appropriate for very small datasets, 10-fold cross validation is by far the most popular choice. Confusion Matrices So far, we have been evaluating our classifier by computing the percent accuracy. That is, number of test cases correctly classified Total number of test cases sometimes we may want a more detailed picture of the performance of our classification algorithm and one such detailed visualization is a table called the confusion matrix. The rows of the confusion matrix represent the actual class of the test cases, the columns represent what our classifier predicted. 5-10 EVALUATION AND KNN The name confusion matrix comes from the observation that it is easy for us to see where our algorithm gets confused. Let’s look at an example using our women athlete domain. Suppose we have a dataset that consists of attributes for 100 women gymnasts, 100 players in the Women’s National Basketball Association, and 100 women marathoners. We evaluate the classifier using 10-fold cross-validation. In 10-fold cross-validation we use each instance of our dataset exactly once for testing. The results of this test might be the following confusion matrix: gymnast basketball player marathoner gymnast 83 0 17 basketball player 0 92 8 marathoner 9 16 75 Again, the real class of each instance is represented by the rows; the class predicted by our classifier is represented by the columns. So, for example, 83 instances of gymnasts were classified correctly as gymnasts but 17 were misclassified as marathoners. 92 basketball players were classified correctly as basketball players but 8 were misclassified as marathoners. 75 marathoners were classified correctly but 9 were misclassified as gymnasts and 16 misclassified as basketball players. The diagonal of the confusion matrix represents instances that were classified correctly. gymnast basketball player marathoner 83 0 17 basketball player 0 92 8 marathoner 9 16 85 gymnast In this case the accuracy of the algorithm is: 83 + 92 + 75 250 = = 83.33% 300 300 5-11 It is easy to inspect the matrix to get an idea of what type of errors our classifier is making. It this example, it seems our algorithm is pretty good at distinguishing between gymnasts and basketball players. Sometimes gymnasts and basketball players get misclassified as marathoners and marathoners occasionally get misclassified as gymnasts or basketball players. Confusion matrices are not that confusing! A programming example Let us go back to a dataset we used in the last chapter, the Auto Miles Per Gallon data set from Carnegie Mellon University. The format of the data looked like: mpg cylinders c.i. HP weight secs. 0-60 make/model 30 4 68 49 1867 19.5 fiat 128 45 4 90 48 2085 21.7 vw rabbit (diesel) 20 8 307 130 3504 12 chevrolet chevelle malibu I am trying to predict the miles per gallon of a vehicle based on number of cylinders, displacement (cubic inches), horsepower, weight, and acceleration. I put all 392 instances in a file named mpgData.txt and wrote the following short Python program that divided the data into ten buckets using a stratified method. (Both the data file and Python code are available on the website guidetodatamining.com.) 5-12 EVALUATION AND KNN import random def buckets(filename, bucketName, separator, classColumn): """the original data is in the file named filename bucketName is the prefix for all the bucket names separator is the character that divides the columns (for ex., a tab or comma) and classColumn is the column that indicates the class""" # put the data in 10 buckets numberOfBuckets = 10 data = {} # first read in the data and divide by category with open(filename) as f: lines = f.readlines() for line in lines: if separator != '\t': line = line.replace(separator, '\t') # first get the category category = line.split()[classColumn] data.setdefault(category, []) data[category].append(line) # initialize the buckets buckets = [] for i in range(numberOfBuckets): buckets.append([]) # now for each category put the data into the buckets for k in data.keys(): #randomize order of instances for each class random.shuffle(data[k]) bNum = 0 # divide into buckets for item in data[k]: buckets[bNum].append(item) bNum = (bNum + 1) % numberOfBuckets # write to file for bNum in range(numberOfBuckets): f = open("%s-%02i" % (bucketName, bNum + 1), 'w') for item in buckets[bNum]: f.write(item) f.close() buckets("mpgData.txt", 'mpgData','\t',0) 5-13 Executing this code will produce ten files labelled mpgData01, mpgData02, etc. s code it Can you revise the nearest neighbor code from the last chapter so the function test performs 10-fold cross validation on the 10 data files we just created (you can download them at guidetodatamining.com)? Your program should output a confusion matrix that looks something like: predicted MPG actual MPG ! 5-14 10 15 20 25 30 35 40 45 10 3 10 0 0 0 0 0 0 15 3 68 14 1 0 0 0 0 20 0 14 66 9 5 1 1 0 25 0 1 14 35 21 6 1 1 30 0 1 3 17 21 14 5 2 35 0 0 2 8 9 14 4 1 40 0 0 1 0 5 5 0 0 45 0 0 0 2 1 1 0 2 53.316% accurate total of 392 instances EVALUATION AND KNN s code it - one solution One solution involves only • Changing the initializer method to read in data from 9 buckets. • Adding a new method to test from data in one bucket • Adding a new procedure that performs 10-fold cross-validation Let us look at these in turn. initializer method __init__ The signature of the init method looks like: def __init__(self, bucketPrefix, testBucketNumber, dataFormat): The filenames of the buckets will be something like mpgData-01, mpgData-02, etc. In this case, bucketPrefix will be “mpgData”. testBucketNumber is the bucket containing the test data. If testBucketNumber is 3, the classifier will be trained on buckets 1, 2, 4, 5, 6, 7, 8, 9, and 10. dataFormat is a string specifying how to interpret the columns in the data. For example, "class! num! num! num! num! num! comment" specifies that the first column represents the class of the instance. The next 5 columns represent numerical attributes of the instance and the final column should be interpreted as a comment. The complete, new initializer method is as follows: 5-15 import copy class Classifier: def __init__(self, bucketPrefix, testBucketNumber, dataFormat): """ a classifier will be built from files with the bucketPrefix excluding the file with textBucketNumber. dataFormat is a string that describes how to interpret each line of the data files. For example, for the mpg data the format is: "class! num! num! num! num! num! comment" """ self.medianAndDeviation = [] # reading the data in from the file self.format = dataFormat.strip().split('\t') self.data = [] # for each of the buckets numbered 1 through 10: for i in range(1, 11): # if it is not the bucket we should ignore, read the data if i != testBucketNumber: filename = "%s-%02i" % (bucketPrefix, i) f = open(filename) lines = f.readlines() f.close() for line in lines: fields = line.strip().split('\t') ignore = [] vector = [] for i in range(len(fields)): if self.format[i] == 'num': vector.append(float(fields[i])) elif self.format[i] == 'comment': ignore.append(fields[i]) elif self.format[i] == 'class': classification = fields[i] self.data.append((classification, vector, ignore)) self.rawData = copy.deepcopy(self.data) # get length of instance vector self.vlen = len(self.data[0][1]) # now normalize the data for i in range(self.vlen): self.normalizeColumn(i) 5-16 EVALUATION AND KNN testBucket method Next, we write a new method that will test the data in one bucket: def testBucket(self, bucketPrefix, bucketNumber): """Evaluate the classifier with data from the file bucketPrefix-bucketNumber""" filename = "%s-%02i" % (bucketPrefix, bucketNumber) f = open(filename) lines = f.readlines() totals = {} f.close() for line in lines: data = line.strip().split('\t') vector = [] classInColumn = -1 for i in range(len(self.format)): if self.format[i] == 'num': vector.append(float(data[i])) elif self.format[i] == 'class': classInColumn = i theRealClass = data[classInColumn] classifiedAs = self.classify(vector) totals.setdefault(theRealClass, {}) totals[theRealClass].setdefault(classifiedAs, 0) totals[theRealClass][classifiedAs] += 1 return totals This takes as input a bucketPrefix and a bucketNumber. If the prefix is "mpgData " and the number is 3, the test data will be read from the file mpgData-03. testBucket will return a dictionary in the following format: {'35':! '40': ! '30': ! '15': ! '10': ! '20': ! '25': ! {'35': {'30': {'35': {'20': {'15': {'15': {'30': 1, '20': 1}, 3, '30': 3, '15': 1}, 2, '20': 5, '25': 1, '30': 1}, 1, '45': 1, '25': 1}, 4, '10': 1}, 4, '30': 2, '25': 1}, 3}} 5-17 The key of this dictionary represents the true class of the instances. For example, the first line represents results for instances whose true classification is 35 mpg. The value for each key is another dictionary that represents how our classifier classified the instances. For example, the line '15': ! {'20': 3, '15': 4, '10': 1}, represents a test where 3 of the instances that were really 15mpg were misclassified as 20mpg, 4 were classified correctly as 15mpg, and 1 was classified incorrectly as 10mpg. procedure to perform 10-fold cross-validation. Finally, we need to write a procedure that will perform 10-fold cross-validation. That is, it builds 10 classifiers. Each classifier is trained on 9 of the buckets and tested on data from the remaining bucket. def tenfold(bucketPrefix, dataFormat): results = {} for i in range(1, 11): c = Classifier(bucketPrefix, i, dataFormat) t = c.testBucket(bucketPrefix, i) for (key, value) in t.items(): results.setdefault(key, {}) for (ckey, cvalue) in value.items(): results[key].setdefault(ckey, 0) results[key][ckey] += cvalue # now print results categories = list(results.keys()) categories.sort() print( "\n Classified as: ") header = " " subheader = " +" for category in categories: header += category + " " subheader += "----+" print (header) print (subheader) total = 0.0 correct = 0.0 5-18 EVALUATION AND KNN for category in categories: row = category + " |" for c2 in categories: if c2 in results[category]: count = results[category][c2] else: count = 0 row += " %2i |" % count total += count if c2 == category: correct += count print(row) print(subheader) print("\n%5.3f percent correct" %((correct * 100) / total)) print("total of %i instances" % total) tenfold("mpgData", "class! num! num! num! num! num! comment") Running the program yields the following results: 10 15 20 25 30 35 40 45 Classified as: 10 15 20 25 30 35 40 45 +----+----+----+----+----+----+----+----+ | 5 | 8 | 0 | 0 | 0 | 0 | 0 | 0 | | 8 | 63 | 14 | 1 | 0 | 0 | 0 | 0 | | 0 | 14 | 67 | 8 | 5 | 1 | 1 | 0 | | 0 | 1 | 13 | 35 | 22 | 6 | 1 | 1 | | 0 | 1 | 3 | 17 | 21 | 14 | 5 | 2 | | 0 | 0 | 2 | 7 | 10 | 13 | 5 | 1 | | 0 | 0 | 1 | 0 | 5 | 5 | 0 | 0 | | 0 | 0 | 0 | 2 | 1 | 1 | 0 | 2 | +----+----+----+----+----+----+----+----+ 52.551 percent correct total of 392 instances 5-19 Kappa Statistic! At the start of this chapter we mentioned some of the questions we might be interested in answering about a classifier including How good is this classifier. We also have been refining our evaluation methods and looked at 10-fold cross-validation and confusion matrices. In the example on the previous pages we determined that our classifier for predicted miles per gallon of selected car models was 53.316% accurate. But does 53.316% mean our classifier is good or not so good? To answer that question we are going to look at one more statistics, the Kappa Statistic. How good is this classifier? Does 53% accuracy mean the classifier is a good one? 5-20 EVALUATION AND KNN The Kappa Statistic compares the performance of a classifier to that of a classifier that makes predictions based solely on chance. To show you how this works I will start with a simpler example than the mpg one and again return to the women athlete domain. Here are the results of a classifier in that domain: gymnast basketball player marathoner TOTALS gymnast 35 5 20 60 basketball player 0 88 12 100 marathoner 5 7 28 40 TOTALS 40 100 60 200 I also show the totals for the rows and columns. To determine the accuracy we sum the numbers on the diagonal (35 + 88 + 28 = 151) and divide by the total number of instances (200) to get 151 / 200 = .755 Now I am going to generate another confusion matrix that will represent the results of a random classifier (a classifier that makes random predictions). First, we are going to make a copy of the above table only containing the totals: gymnast basketball player marathoner TOTALS gymnast 60 basketball player 100 marathoner 40 TOTALS 40 100 60 200 Looking at the bottom row, we see that 50% of the time (100 instances out of 200) our classifier classifies an instance as “Basketball Player”, 20% of the time (40 instances out of 200) it classifies an instance as “gymnast” and 30% as “marathoner.” 5-21 We are going to use these percentages to fill in the rest of the table. There were 60 total real gymnasts. Our random classifier will classify 20% of those as gymnasts. 20% of 60 is 12 so we put a 12 in the table. It will classify 50% as basketball players (or 30 of them) and 30% as marathoners. Classifier: 0% gymnast: 2 player: 50% basketball : 30% marathoner gymnast gymnast basketball player marathoner TOTALS 12 30 18 60 basketball player 100 marathoner 40 TOTALS 40 100 60 200 And we will continue in this way. There are 100 real basketball players. The random classifier will classify 20% of them (or 20) as gymnasts, 50% as basketball players and 30% as marathoners. And so on: gymnast basketball player marathoner TOTALS gymnast 12 30 18 60 basketball player 20 50 30 100 marathoner 8 20 12 40 TOTALS 40 100 60 200 To determine the accuracy of the random method we sum the numbers on the diagonal and divide by the total number of instances: P(r) = 5-22 12 +50 +12 74 = = .37 200 200 EVALUATION AND KNN The Kappa Statistic will tell us how much better the real classifier is compared to this random one. The formula is κ= P(c) − P(r) 1− P(r) where P(c) is the accuracy of the real classifier and P(r) is the accuracy of the random one. In this case the accuracy of the real classifier was .755 and that of the random one was .37 so κ= .755 − .37 .385 = = .61 1− .37 .63 How do we interpret that .61? Does that mean our classifier is poor, good, or great? Here is a chart that will help us interpret that number: A commonly cited* scale on how to inte rpret Kappa < 0: less than chance performance 0.01-0.20 slightly good 0.21-0.40 fair performance 0.41-0.60 moderate performance 0.61-0.80 substantially good performance 0.81-1.00 near perfect performance * Landis, JR, Koch, GG. 1977. The measurement of observer agreement for categorical data. Biomet rics 33:159-74 5-23 s sharpen your pencil Suppose we developed a somewhat silly classifier that predicts the major of current university students based on how well they liked 10 movies. We have a data set of 600 students consisting of computer science (cs) majors, education majors (ed), English majors (eng) and psychology majors (psych). The confusion matrix is shown below. Can you compute the Kappa Statistic and interpret what that statistic means? predicted major cs ed eng psych cs 50 8 15 7 ed 0 75 12 33 eng 5 12 123 30 psych 5 25 30 170 accuracy = 0.697 5-24 Total EVALUATION AND KNN s solution How good is our classifier? Can you compute the Kappa Statistic and interpret what that statistic means? First, we sum all the columns: SUM % cs ed eng psych TOTAL 60 120 180 240 600 10% 20% 30% 40% 100% Next, we construct the confusion matrix for the random classifier predicted major cs ed eng psych Total cs 8 16 24 32 80 ed 12 24 36 48 120 eng 17 34 51 68 170 psych 23 46 69 92 230 Total 60 120 180 240 600 The accuracy of this random classifier is: (8 + 24 + 51 + 92) / 600 = (175 / 600) = 0.292 5-25 s solution continued So the accuracy of our classifier P(c) is 0.697 and that of the random classifier P(r) is 292 The Kappa Statistic is κ= κ= P(c) − P(r) 1− P(r) 0.697 − 0.292 0.405 = = 0.572 1− 0.292 0.708 This suggests our algorithm performs moderately well. 5-26 EVALUATION AND KNN Improvements to the Nearest Neighbor Algorithm! One trivial example of a classifier is the Rote Classifier, which just memorizes the entire training set and only classifies an instance if that instance exactly matches one in the training set. If we only evaluated classifiers on instances in the training data, the Rote Classifier would always be 100% accurate. In real life, the rote classifier is not a good choice because there will be instances we want to classify that are not in the training set. You can view the nearest neighbor algorithm we have been working with as an extension of the rote classifier. Instead of requiring exact matches we are looking at instances that are close matches. PangNing Tan, Michael Steinbach, and Vipin Kumar in their data mining textbook 1 call this the If it walks like a duck, quacks like a duck, and looks like a duck, then it's probably a duck approach. One problem with the nearest neighbor algorithm occurs when we have outliers. Let me explain what I mean by that. And let us return, yet again, to the women athlete domain; this time only looking at gymnasts and marathoners. Suppose we have a particularly short and lightweight marathoner. In diagram form, this data might be represented as on the next page, where m indicates ‘marathoner’ and g, ‘gymnast. 1 Introduction to Data Mining. 2005. Addison-Wesley 5-27 normalized height m g g g m m g x m g m m normalized weight We can see that short lightweight marathoner as the sole m in the group of g’s. Suppose x is an instance we would like to classify. Its nearest neighbor is that outlier m, so it would get classified as a marathoner. If we just eyeballed the diagram we would say that x is most likely a gymnast since it appears to be in the group of gymnasts. kNN normalized height One way to improve our current nearest neighbor approach is instead of looking at one nearest neighbor we look at a number of nearest neighbors—k nearest neighbors or kNN. Each neighbor will get a vote and the algorithm will predict that the instance will belong to the class with the highest number of votes. For example, suppose we are using three nearest neighbors (k = 3). In that case we have 2 votes for gymnast and one for marathoner, so we would predict x is a gymnast: m g g g m m g x g normalized weight 5-28 m m m EVALUATION AND KNN gymnast marathoner! gymnast So when we are trying to predict a discrete class (marathoners, gymnasts, or basketball players, for example) we can use this voting method. The class with the most votes will be the one assigned to the instance. If there is a tie the predicted class will be selected randomly from the classes that are tied. When we are trying to predict a numeric value like how many stars a person will give the band Funky Meters we can apportion influence from the nearest neighbors to compute a distance-weighted value. Let me parse that out a bit more. Suppose we are trying to predict how well Ben will like Funky Meters and Ben’s three closest neighbors are Sally, Tara, and Jade. Here are their distances from Ben and their ratings for Funky Meters. 5-29 User Distance Rating Sally 5 4 Tara 10 5 Jade 15 5 So Sally was closest to Ben and she gave Funky Meters a 4. Because I want the rating of the closest person to be weighed more heavily in the final value than the other neighbors, the first step we will do is to convert the distance measure to make it so that the larger the number the closer that person is. We can do this by computing the inverse of the distance (that is, 1 over the distance). So the inverse of Sally’s distance of 5 is 1 = 0.2 5 User Inverse Distance Rating Sally 0.2 4 Tara 0.1 5 Jade 0.067 5 Now I am going to divide each of those inverse distances by the sum of all the inverse distances. The sum of the inverse distances is 0.2 + 0.1 + 0.067 = 0.367. User Influence Rating Sally 0.545 4 Tara 0.272 5 Jade 0.183 5 We should notice two things. First, that the sum of the influence values totals 1. The second thing to notice is that with the original distance numbers Sally was twice as close to Ben as Tara was, and that is preserved in the final numbers were Sally has twice the influence as 5-30 EVALUATION AND KNN Tara does. Finally we are going to multiple each person’s influence and rating and sum the results: predicted Score for Ben = 0.545 × 4 + 0.272 × 5 + 0.183 × 5 = 2.18 + 1.36 + 0.915 = 4.455 s sharpen your pencil I am wondering how well Sofia will like the jazz pianist Hiromi. What is the predicted value given the following data using the k nearest neighbor algorithm with k = 3.? person Gabriela Ethan Jayden distance from Sofia 4 8 10 rating for Hiromi 3 3 5 5-31 s sharpen your pencil - solution the first thing to do is to compute the inverse ( 1 over the distance) of each distance: Person Inverse Distance Rating Gabriela 1/4 = 0.25 3 Ethan 1/8 = 0.125 3 Jayden 1/10 = 0.1 5 The sum of the inverse distances is 0.475. Next I am going to compute the influence of each person by dividing the inverse distance by the sum of each distance Person Influence Rating Gabriela 0.526 3 Ethan 0.263 3 Jayden 0.211 5 Finally, I multiply the influence by the rating and sum the results: = (0.526 × 3)+ (0.263 × 3) +(0.211× 5) = 1.578 + 0.789 + 1.055 = 3.422 5-32 EVALUATION AND KNN A new dataset and a challenge! It is time to look at a new dataset, the Pima Indians Diabetes Data Set developed by the United States National Institute of Diabetes and Digestive and Kidney Diseases. Astonishingly, over 30% of Pima people develop diabetes. In contrast, the diabetes rate in the United States is 8.3% and in China it is 4.2%. Each instance in the dataset represents information about a Pima woman over the age of 21 and belonged to one of two classes: a person who developed diabetes within five years, or a person that did not. There are eight attributes: 5-33 attributes: 1. Number of times pregnant 2. Plasma glucose concentration 3. Diastolic blood pressure (mm Hg) 4. Triceps skin fold thickness (mm) 5. 2-Hour serum insulin (mu U/ml) 6. Body mass index (weight in kg/(height in m)^2) 7. Diabetes pedigree function 8. Age (years) Here is an example of the data (the last column represents the class—0=no diabetes; 1=diabetes): 2 99 52 15 94 24.6 0.637 21 0 3 83 58 31 18 34.3 0.336 25 0 5 139 80 35 160 31.6 0.361 25 1 3 170 64 37 225 34.5 0.356 30 1 So, for example, the first woman has had 2 children, has a plasma glucose concentration of 99, a diastolic blood pressure of 52 and so on. 5-34 EVALUATION AND KNN s code it - part 1 There are two files on our website. pimaSmall.zip is a zip file containing 100 instances of the data divided into 10 files (buckets). pima.zip is a zip file containing 393 instances. When I used the pimaSmall data with the nearest neighbor classifier we built in the previous chapter using 10-fold crossvalidation I got these results: 0 1 Classified as: 0 1 +----+----+ | 45 | 14 | | 27 | 14 | +----+----+ Hint: The py thon functi on heapq.nsma llest(n, list) w list with th e n smalles ill return a t items. 59.000 percent correct total of 100 instances Here is your task: Download the classifier code from our website and implement the kNN algorithm. Let us change the initializer method of the class to add another argument, k: def __init__(self, bucketPrefix, testBucketNumber, dataFormat, k): The method signature should look like def knn(self, itemVector): It should make use of self.k (remember to set that value in the init method) and return the class (in this Pima Cancer dataset case ‘0’ or ‘1’). You should also modify the procedure tenfold to pass k to the initializer. 5-35 s code it - answer My modification to _init__ was simply: def __init__(self, bucketPrefix, testBucketNumber, dataFormat, k): self.k = k ... My knn method was def knn(self, itemVector): """returns the predicted class of itemVector using k Nearest Neighbors""" # changed from min to heapq.nsmallest to get the # k closest neighbors neighbors = heapq.nsmallest(self.k, [(self.manhattan(itemVector, item[1]), item) for item in self.data]) # each neighbor gets a vote results = {} for neighbor in neighbors: theClass = neighbor[1][0] results.setdefault(theClass, 0) results[theClass] += 1 resultList = sorted([(i[1], i[0]) for i in results.items()], reverse=True) #get all the classes that have the maximum votes maxVotes = resultList[0][0] possibleAnswers = [i[1] for i in resultList if i[0] == maxVotes] # randomly select one of the classes that received the max votes answer = random.choice(possibleAnswers) return( answer) 5-36 EVALUATION AND KNN My slight modification to tenfold was: def tenfold(bucketPrefix, dataFormat, k): results = {} for i in range(1, 11): c = Classifier(bucketPrefix, i, dataFormat, k) ... You can download this code at guidet odatamining.com. Remember, this is ju st one way to implem ent this method, and it is not necess arily the best way. s code it - part 2 Which makes the most difference? Having more data (comparing the results from pimaSmall and pima) or having a better algorithm (comparing k=1 to k=3)? 5-37 s code it - results! Here are my accuracy results (k=1 is the nearest neighbor algorithm from the last chapter): pimaSmall pima k=1 59.00% 71.247% k=3 61.00% 72.519% So it seems that roughly tripling the amount of data increases the accuracy much more than improving the algorithm does. 5-38 EVALUATION AND KNN s sharpen your pencil Hmm. 72.519% seems like pretty good accuracy but is it? Compute the Kappa Statistic to find out: no diabetes diabetes no diabetes 219 44 diabetes 64 66 Performance: ☐ ☐ ☐ ☐ ☐ slightly good fair moderate substantially good near perfect 5-39 s sharpen your pencil — answer no diabetes diabetes TOTAL no diabetes 219 44 263 diabetes 64 66 130 TOTAL 283 110 393 ratio 0.7201 0.2799 random (r) classifier: no diabetes diabetes κ= no diabetes diabetes 189.39 73.61 93.61 36.39 189.39 + 36.39 393 = .5745 p(r)= P(c) − P(r) .72519 − .5745 .15069 = = = .35415 1− P(r) 1− .5745 .4255 Only fair performance 5-40 accuracy EVALUATION AND KNN More data, better algorithms & a broken bus Several years ago I was at a conference in Mexico City. This conference was a bit unusual in that it alternated between a day of presentations and a day of touring (the Monarch Butterflies, Inca ruins, etc). The days of touring involved riding long distances on a bus and the bus had a tendency to break down. As a result, a bunch of us PhD types spend a good deal of time standing at the side of road talking to one another as the bus was being attended to. These roadside exchanges were the highpoint of the conference for me. One of the people I talked to was a person named Eric Brill. Eric Brill is famous for developing what is called the Brill tagger, which does part-ofspeech tagging. Similar to what we have been doing in the last few chapters, the Brill tagger classifies data—in this case, it classifies words by their part of speech (noun, verb, etc.). The algorithm Brill came up with was significantly better than its predecessors (and as a result Brill became famous in natural language processing circles). At the side of that Mexican road, I got to talking with Eric Brill about improving the performance of algorithms. His view is that you get more of an improvement by getting more data for the training set, than you would by improving the algorithm. In fact, he felt that if he kept the original part-of-speech tagging algorithm and just increased the size of the training data, the improvement would exceed that of his famous algorithm. Although, he said, you cannot get a PhD for just collecting more data, but you can for developing an algorithm with marginally improved performance! Here's another example. In various machine translation competitions, Google always places at the top. Granted that Google has a large number of very bright people developing great algorithms, much of Google's dominance is due to its enormous training sets it acquired from web. ➯ Més dades ➯ More data 5-41 This isn't to say that you shouldn't pick the best algorithm for the job. As we have already seen picking a good algorithm makes a significant difference. However, if you are trying to solve a practical problem (rather than publish a research paper) it might not be worth your while to spend a lot of time researching and tweaking algorithms. You will perhaps get more bang for your buck—or a better return on your time—if you concentrate on getting more data. With that nod toward the importance of data, I will continue my path of introducing new algorithms. People have used kNN cla ssifiers for recommending items at Am azon assessing consumer credit risk classifying land cover usi ng image analysis recognizing faces classifying the gender of people in images recommending web pages recommending vacation pa ckages 5-42 Chapter 6: Probability and Naive Bayes Naïve Bayes Let us return yet again to our women athlete example. Suppose I ask you what sport Brittney Griner participates in (gymnastics, marathon running, or basketball) and I tell you she is 6 foot 8 inches and weighs 207 pounds. I imagine you would say basketball and if I ask you how confident you feel about your decision I imagine you would say something along the lines of “pretty darn confident.” Now I ask you what sport Heather Zurich (pictured on the right) plays. She is 6 foot 1 and weighs 176 pounds. Here I am less certain how you will answer. You might say ‘basketball’ and I ask you how confident you are about your prediction. You probably are less confident than you were about your prediction for Brittney Griner. She could be a tall marathon runner. Finally, I ask you about what sport Yumiko Hara participates in; she is 5 foot 4 inches tall and weighs 95 pounds. Let's say you say ‘gymnastics’ and I ask how confident you feel about your decision. You will probably say something along the lines of “not too confident.” A number of marathon runner have similar heights and weights. With the nearest neighbor algorithms, it is difficult to quantify confidence about a classification. With classification methods based on probability— Bayesian methods—we can not only make a classification but we can make probabilistic classifications—this athlete is 80% likely to be a basketball player, this patient has a 40% chance of getting diabetes in the next five years, the probability of rain in Las Cruces in the next 24 hours is 10%. Nearest Neighbor approaches are called lazy learners. They are called this because when we give them a set of training data, they just basically save— or remember—the set. Each time it classifies an instance, it goes through the entire training dataset. If we have a 100,000 music tracks in our training data, it goes through the entire 100,000 tracks each time it classifies an instance. Bayesian methods are called eager learners. When given a training set eager learners immediately analyze the data and build a model. When it wants to classify an instance it uses this internal model. Eager learners tend to classify instances faster than lazy learners. The ability to make probabilistic classifications, and the fact that they are eager learners are two advantages of Bayesian methods. 6-2 PROBABILITY AND NAÏVE BAYES Probability I am assuming you have some basic knowledge of probability. I flip a coin; what is the probably of it beings a 'heads'? I roll a 6 sided fair die, what is the probability that I roll a '1'? that sort of thing. I tell you I picked a random 19 year old and have you tell me the probability of that person being female and without doing any research you say 50%. These are examples of what is called prior probability and is denoted P(h)—the probability of hypothesis h. So for a coin: P(heads) = 0.5 For a six sided dice, the probability of rolling a ‘1’: P(1) = 1/6 If I have an equal number of 19 yr. old male and females → P(female) = .5 Suppose I give you some additional information about that 19 yr. old—the person is a student at the Frank Lloyd Wright School of Architecture in Arizona. You do a quick Google search, see that the student body is 86% female and revise your estimate of the likelihood of the person being female to 86%. This we denote as P(h|D) —the probability of the hypothesis h given some data D. For example: P(female | attends Frank Lloyd Wright School) = 0.86 which we could read as “The probability the person is female given that person attends the Frank Lloyd Wright School is 0.86 6-3 The formula is P(A | B) = P(A ∩ B) P(B) An example. In the following table I list some people and the types of laptops and phones they have: name laptop phone Kate PC Android Tom PC Android Harry PC Android Annika Mac iPhone Naomi Mac Android Joe Mac iPhone Chakotay Mac iPhone Neelix Mac Android Kes PC iPhone B’Elanna Mac iPhone What is the probability that a randomly selected person uses an iPhone? There are 5 iPhone users out of 10 total users so P(iPhone) = 5 = 0.5 10 What is the probability that a randomly selected person uses an iPhone given that person uses a Mac laptop? P(iPhone | mac) = P(mac ∩ iPhone) P(mac) First, there are 4 people who use both a Mac and an iPhone: P(mac ∩ iPhone) = 4 = 0.4 10 and the probability of a random person using a mac is P(mac) = 6-4 6 = 0.6 10 PROBABILITY AND NAÏVE BAYES So the probability of that some person uses an iPhone given that person uses a Mac is P(iPhone | mac) = 0.4 = 0.667 0.6 That is the formal definition of posterior probability. Sometimes when we implement this we just use raw counts: number of people who use a mac and an iPhone P(iPhone|mac) = P(iPhone | mac)= s number of people who use a mac 4 = 0.667 6 sharpen your pencil What’s the probability of a person owning a mac given that they own an iPhone i.e., P(mac|iPhone)? tip If you feel you need practice with basic probabilities please see the links to tutorials at guidetodatamining.com. 6-5 s sharpen your pencil — solution What’s the probability of a person owning a mac given that they own an iphone i.e., P(mac|iPhone)? P(mac | iPhone) = = P(iPhone ∩ mac) P(iPhone) 0.4 = 0.8 0.5 Some terms: P(h), the probability that some hypothesis h is true, is called the prior probability of h. Before we have any evidence, the probability of a person owning a Mac is 0.6 (the evidence might be knowing that the person also owns an iPhone). P(h|d) is called the posterior probability of h. After we observe some data d what is the probability of h? For example, after we observe that a person owns an iPhone, what is the probability of that same person owning a Mac? It is also called conditional probability. In our quest to build a Bayesian Classifier we will need two additional probabilities, P(D) and P(D|h). To explain these consider the following example. 6-6 PROBABILITY AND NAÏVE BAYES Microsoft Shopping Cart Did you know that Microsoft makes smart grocery store shopping carts? Yep, they do. Well, actually, Microsoft has contracted with a company called Chaotic Moon to develop them. Chaotic Moon’s slogan is We are smarter than you. We are more creative than you. You can decide whether they are arrogant, cheeky, or something else. Anyway, the cart combines a shopping cart with a Windows 8 tablet, a Kinect, a Bluetooth speaker (so the cart can talk to you), and a mobile robotics platform (so the cart can follow you around the store). You come in with your grocery store loyalty card. The cart recognizes you. It has recorded all previous purchases (as well as the purchases of everyone else in the store). Suppose the cart software wants to determine whether to show you a targeted ad for Japanese Sensha Green Tea. It only wants to show that ad if you are likely to purchase the tea. The cart system has accumulated the small dataset shown on the next page from other shoppers P(D) is the probability that some training data will be observed. For example, looking on the next page we see that the probability that the zip code will be 88005 is 5/10 or 0.5. P(88005) = 0.5 P(D|h) is the probability that some data value holds given the hypothesis. For example, the probability of the zip code being 88005 given that the person bough Sencha Green Tea or P(88005|Sencha Tea). 6-7 Customer ID Zipcode bought organic produce? bought Sencha green tea? 1 88005 Yes Yes Zipcodes are a set of postal codes 2 88001 No No used in the U.S. 3 88001 Yes Yes 4 88005 No No 5 88003 Yes No 6 88005 No Yes 7 88005 No No 8 88001 No No 9 88005 Yes Yes 10 88003 Yes Yes In this case we are looking at all the instances where the person bought Sensha Tea. There are 5 such instances. Of those, 3 are with the 88005 zip code. P(88005 | SenchaTea) = s 3 = 0.6 5 sharpen your pencil What’s the probability of the zip code being 88005 given that the person did not buy Sencha tea? 6-8 PROBABILITY AND NAÏVE BAYES s sharpen your pencil — solution What’s the probability of the zip code being 88005 given that the person did not buy Sencha tea? There are 5 occurrences of a person not buying Sencha tea. Of those, 2 lived in the 88005 zip code. So P(88005 | ¬SenchaTea) = s 2 = 0.4 5 That ¬ sym bol mea ns ‘not’. sharpen your pencil This is key to understanding the rest of the chapter so let us practice just a bit more. 1. What is the probability of a person being in the 88001 zipcode (without knowing anything else)? 2. What is the probability of a person being in the 88001 zipcode knowing that they bought Sencha tea? 3. What is the probability of a person being in the 88001 zipcode knowing that they did not buy Sencha tea? 6-9 s sharpen your pencil — solution This is key to understanding the rest of the chapter so let us practice just a bit more. 1. What is the probability of a person being in the 88001 zipcode (without knowing anything else)? There are 10 total entries in our database and only 3 of them are from 88001 so P(88001) is 0.3 2. What is the probability of a person being in the 88001 zipcode knowing that they bought Sencha tea? There are 5 instances of buying Sencha tea and only 1 of them is from the 88001 zipcode so P(88001 | SenchaTea) = 1 = 0.2 5 3. What is the probability of a person being in the 88001 zipcode knowing that they did not buy Sencha tea? There are 5 instances of not buying Sencha tea and 2 of them are from the 88001 zipcode: P(88001 | ¬SenchaTea) = 6-10 2 = 0.4 5 PROBABILITY AND NAÏVE BAYES Bayes Theorem Bayes Theorem describes the relationship between P(h), P(h|D), P(D), and P(D|h): P(h | D) = P(D | h)P(h) P(D) This theorem is the cornerstone of all Bayesian methods. Usually in data mining we use this theorem to decide among alternative hypotheses. Given the evidence, is the person a gymnast, marathoner, or basketball player. Given the evidence, will this person buy Sencha tea, or not. To decide among alternatives we compute the probability for each hypothesis. For example, We want to display an ad for Sencha Tea on our smart shopping cart display only if we think that person is likely to buy the tea. We know that person lives in the 88005 zipcode. There are two competing hypotheses: The person will buy Sencha tea. We compute P(buySenchaTea|88005) The person will not buy Sencha tea. We compute P(¬buySenchaTea|88005) We pick the hypothesis with the highest probability! So if P(buySenchaTea|88005) = 0.6 and P(¬buySenchaTea|88005) = 0.4 So it is more likely that the person will buy the tea so we will display the ad. 6-11 Suppose we work for an electronics store and we have three sales flyers in email form. One flyer features a laptop, another features a desktop and the final flyer a tablet. Based on what we know about each customer we will email that customer the flyer that will most likely generate a sale. For example, I may know that a customer lives in the 88005 zipcode, that she has a college age daughter living at home, and that she goes to yoga class. Should I send her the flyer with the laptop, desktop, or tablet? Let D represent all that I know about that customer: • lives in 88005 zipcode • has college age daughter • goes to yoga class My hypotheses are which flyer is the best: laptop, desktop, tablet. So I compute: P(D | laptop)P(laptop) P(D) P(laptop | D) = P(desktop | D) = P(tablet | D) = P(D | desktop)P(desktop) P(D) P(D | tablet)P(tablet) P(D) And pick the hypothesis with the highest probability. More abstractly, in a classification task we have a number of possible hypotheses: h1, h2, ...hn. These hypotheses are the different categories of our task (for example, basketball players, marathoners, gymnasts, or ‘will get diabetes’, ‘will not get diabetes’). 6-12 PROBABILITY AND NAÏVE BAYES P(h1 | D) = P(D | h1 )P(h1 ) P(D) ... , P(hn | D) = P(h2 | D) = P(D | h2 )P(h2 ) P(D) P(D | hn )P(hn ) P(D) Once we compute all these probabilities, we will pick the hypothesis with the highest probability. This is called the maximum a posteriori hypothesis, or hMAP. That’s right! You got it! Ok, I compute the probability of each possible hypothesis and select the hypothesis with the highest probability. That hypothesis is called the maximum a posteriori hypothesis! 6-13 We can translate that English description of calculating the maximum a posteriori hypothesis into the following formula: hMAP = arg max h∈H P(h | D) H is the set of all the hypotheses. So h∈H means “for every hypothesis in the set of hypotheses.” The full formula means something like “for every hypothesis in the set of hypotheses compute P (h|D) and pick the hypothesis with the largest probability.” Using Bayes Theorem we can convert that formula to: hMAP = arg max h∈H P(D | h)P(h) P(D) So for every hypothesis we are going to compute: P(D | h)P(h) P(D) You might notice that for all these calculations, the denominators are identical—P(D). Thus, they are independent of the hypotheses. If a specific hypothesis has the max probability with the formula used above, it will still be the largest if we did not divide all the hypotheses by P(D). If our goal is to find the most likely hypothesis, we can simplify our calculations: hMAP = arg max h∈H P(D | h)P(h) To see how this works, we will use an example from Tom M. Mitchell’s book, Machine Learning. Tom Mitchell is chair of the Machine Learning Department at Carnegie Mellon University. He is a great researcher and an extremely nice guy. On to the example from the book. Consider a medical domain where we want to determine whether a patient has a particular kind of cancer or not. We know that only 0.8% of the people in the U.S. have this form of cancer. There is a simple blood test we can do that will help us determine whether someone has it. The test is a binary one—it comes back either POS or NEG. When the disease is present the test returns a correct POS result 98% of the time; it returns a correct NEG result 97% of the time in cases when the disease is not present. 6-14 PROBABILITY AND NAÏVE BAYES Our hypotheses: • The patient has the particular cancer • The patient does not have that particular cancer. s sharpen your pencil Let’s translate what I wrote above into probability notation. Please match up the English statements below with their associated notations and write in the probabilities. If there is no English statement matching a probability, please write one. We know that only 0.8% of the people in the U.S. have this form of cancer. P(POS|cancer) = _______ P(POS|¬cancer) = _______ When the disease is present the test returns a correct POS result 98% of the time; P(cancer) = _______ P(¬cancer) = _______ P(NEG|cancer) = _______ it returns a correct NEG result 97% of the time in cases when the disease is not present P(NEG|¬cancer) = _______ 6-15 s sharpen your pencil — solution We know that only 0.8% of the people in the U.S. have this form of cancer. P(POS|cancer) = 0.98 99.2% of people don’t have this cancer When the disease is present the test returns a correct POS result 98% of the time; When the disease is present the test returns a incorrect NEG result 2% of P(POS|¬cancer) = 0.03 P(cancer) = 0.008 P(¬cancer) = 0.992 P(NEG|cancer) = 0.02 it returns a correct NEG result 97% of the time in cases when the disease is not present it returns an incorrect POS result 3% of the time in cases when the disease is not present 6-16 P(NEG|¬cancer) = 0.97 PROBABILITY AND NAÏVE BAYES s sharpen your pencil — solution Suppose Ann, comes into the doctor's office A blood test for cancer is given and the test result is POS. This is not looking good for Ann. After all, the test is 98% accurate. Using Bayes Theorem determine whether it is more likely that Ann has cancer or that she does not. P(cancer) = 0.008 P(¬cancer) = 0.992 P(POS|cancer) = 0.98 P(POS|¬cancer) = 0.03 P(NEG|cancer) = 0.02 P(NEG|¬cancer) = 0.97 6-17 s sharpen your pencil — solution Suppose Ann, comes into the doctor's office A blood test for the cancer is given and the test result is POS. This is not looking good for Ann. After all, the test is 98% accurate. Using Bayes Theorem determine whether it is more likely that Ann has cancer or that she does not. We are finding the maximum a posteriori probability: P(POS | cancer)P(cancer) = .98(.008) = .0078 P(POS | ¬ cancer) P(¬ cancer) = .03(.992) = .0298 We select hMAP and classify the patient as not having cancer. If we want to know the exact probability we can normalize these values by having them sum to 1: P(cancer | POS) = 0.0078 = 0.21 0.0078 + 0.0298 Ann has a 21% chance of having cancer. 6-18 PROBABILITY AND NAÏVE BAYES You may think “That just doesn’t make sense. After all, the test is 98% accurate, but yet you re telling me Ann is most likely not to have cancer. “ You are in good company. 85% of medical doctors get the answer wrong as well. I just didn’t make that 85% number up. See, among others, T. ger, A., and Grayboys, Casscells, W., Schoenber physicians of clinical by (1978): "Interpretation gl J Med. 299:999-1001. En N s." ult res y tor labora Gigerenzer, Gerd and Hoffrage, Ulrich (1995): Here is why the results seem so "How to improve Bayesian reasonin g without instruction: Frequency formats." counterintuitive. Most people see the statistic Psychological Review. 102: 684-704. that 98% of the people who have this particular cancer will have a positive test result and also conclude that 98% of the Eddy, David M. (1982): "Probabilistic reasoning people who have a positive test result have in clinical medicine: Problems and this particular cancer. This fails to take into opportunities." In D. Kahneman, P. Slovic, and A. Tversky, eds, Judgement under uncertainty: account that this cancer affects only 0.8% of Heuristics and biases. Cambridge University the population. Let’s say we give the test to Press, Cambridge, UK. everyone in a city of 1 million people. That means that 8,000 people have cancer and 992,000 do not. First, let’s consider giving the test to the 8,000 people with cancer. We know that 98% of the time when we give the test to people with cancer the test correctly returns a positive result. So 7,840 people have a correct positive result and 160 of those people with cancer have an incorrect negative result. Now let’s turn to the 992,000 people without cancer. When we give the test to them, 97% of the time we get a correct negative result so (992,000 * 0.97) or 962,240 of them have a correct negative result and 30,000 have an incorrect positive result. I have summarized these results on the following page. 6-19 positive test result negative test result 7,840 160 30,000 962,240 people with cancer people without cancer Now, consider Ann getting a positive test result and the data in the ‘positive test result’ column. 30,000 of the people with a positive test result had no cancer while only 7,840 of them had cancer. So it seems probable that Ann does not have cancer. Still don’t get it? Don’t worry. Many people don’t. After more practice you will gain a better understanding. Why do we need Bayes Theorem? Yet again, Bayes Theorem is Customer ID Zipcode bought organic produce? bought Sencha green tea? 1 88005 Yes Yes 2 88001 No No 3 88001 Yes Yes 4 88005 No No Say we know a customer lives in the 88005 zipcode and our two competing hypotheses are that they will buy Sencha tea or they will not. So: 5 88003 Yes No 6 88005 No Yes 7 88005 No No P(h1|D) = P(buySenchaTea|88005) 8 88001 No No and 9 88005 Yes Yes P(h2|D) = P(¬ buySenchaTea|88005) 10 88003 Yes Yes P(h | D) = P(D | h)P(h) P(D) Let us return to the shopping cart example presented earlier. In that example, we obtained the information on the right from customers. 6-20 PROBABILITY AND NAÏVE BAYES In this case you may wonder why we need to compute P(88005 | buySenchaTea)P(buySenchaTea) P(88005) when we can just as easily compute P(buySenchaTea|88005) directly from the data in the table. In this simple case you would be correct but for many real world problems it is very difficult to compute P(h|D) directly. Consider the previous medical example where we were interested in determining whether a person had cancer or not given that a certain test returned a positive result. P(cancer | POS) ≈ P(POS | cancer)P(cancer) P(¬cancer | POS) ≈ P(POS | ¬cancer)P(¬cancer) It is relatively easy to compute the items on the right hand side. We can estimate P(POS|cancer) by giving the cancer test to a representative sample of people with cancer and P(POS|¬ cancer) by giving the test to a sample of people without cancer. P(cancer) seems like a statistic that would be available on government websites and P(¬ cancer) is simply 1 - P(cancer) However, computing P(cancer|POS) directly would be significantly more challenging. This is asking us to determine the probability that when we give the test to a random average person in the entire population and the test result is POS then that person has cancer. To do this we want a representative sample of the population but since only 0.8% of people have cancer a sample size of 1,000 people would only have 8 people with cancer—far too few to feel that our counts are representative of the population as a whole. So we would need an extremely large sample size. So Bayes Theorem provides a strategy for computing P(h|D) when it is hard to do so directly. 6-21 Naïve Bayes Most of the time we have more evidence than just a single piece of data. In the Sencha tea example we had two types of evidence: zip code and whether the person purchased organic food. To compute the probability of an hypothesis given all the evidence, we simply multiply the individual probabilities. In this example Code: Customer ID Zipcode bought organic produce? bought Sencha green tea? 1 88005 Yes Yes Sencha tea 2 88001 No No P(88005|tea) = probability that a person lives in the 88005 zipcode given that person bought Sencha tea. 3 88001 Yes Yes 4 88005 No No 5 88003 Yes No 6 88005 No Yes 7 88005 No No 8 88001 No No 9 88005 Yes Yes 10 88003 Yes Yes tea = Person buy Sencha tea ¬ tea = Person does not buy etc. We would like to know whether a person who lives in the 88005 zipcode and bought organic produce will likely buy tea: P(tea|88005 & organic) and for that we simply multiply the probabilities: P(tea|88005 & organic) = P(88005 | tea) P(organic | tea) P(tea) = .6(.8)(.5) = .24 P(¬tea|88005 & organic) = P(88005 |¬tea) P(organic |¬tea) P(¬tea) = .4(.25)(.5) = .05 So a person who lives in the trendy 88005 zip code area and buys organic food is more likely to buy Sencha Green tea than not. So let's display the Green Tea ad on the shopping cart display! 6-22 PROBABILITY AND NAÏVE BAYES Here's how Stephen Baker describes the smart shopping cart technology: … here's what shopping with one of these carts might feel like. You grab a cart on the way in and swipe your loyalty card. The welcome screen pops up with a shopping list. It's based on patterns of your last purchases. Milk, eggs, zucchini, whatever. Smart systems might provide you with the quickest route to each item. Or perhaps they'll allow you to edit the list, to tell it, for example, never to promote cauliflower or salted peanuts again. This is simple stuff. But according to Accenture's studies, shoppers forget an average of 11 percent of the items they intend to buy. If stores can effectively remind us of what we want, it means fewer midnight runs to the convenience store for us and more sales for them. Baker. 2008. P49. The Numerati I've mentioned this book by Stephen Baker several times. I highly encourage you to read this book. The paperback is only $10 and it is a good late night read. 6-23 i100 i500 Let's say we are trying to help iHealth, a company that sells wearable exercise monitors that compete with the Nike Fuel and the Fitbit Flex. iHealth sells two models that increase in functionality: the i100 and the i500: iHealth100: heart rate, GPS (to compute miles per hour, etc), wifi to automatically connect to iHealth website to upload data. iHealth500: i100 features + pulse oximetry (oxygen in blood) + free 3G connection to iHealth website They sell these online and they hired us to come up with a recommendation system for their customers. To get data to build our system when someone buys a monitor, we ask them to fill out the questionnaire. Each question in the questionnaire relates to an attribute. First, we ask them what their main reason is for starting an exercise program and have them select among three options: health, appearance or both. We ask them what their current exercise level is: sedentary, moderate, or active. We ask them how motivated they are: moderate or aggressive. And finally we ask them if they are comfortable with using technological devices. Our results are as follows. 6-24 PROBABILITY AND NAÏVE BAYES Main Interest Current How Motivated Comfortable Exercise Level with tech. Devices? Model # both sedentary moderate yes i100 both sedentary moderate no i100 health sedentary moderate yes i500 appearance active moderate yes i500 appearance moderate aggressive yes i500 appearance moderate aggressive no i100 health moderate aggressive no i500 both active moderate yes i100 both moderate aggressive yes i500 appearance active aggressive yes i500 both active aggressive no i500 health active moderate no i500 health sedentary aggressive yes i500 appearance active moderate no i100 health sedentary moderate no i100 s sharpen your pencil Using the naïve Bayes method, which model would you recommend to a person whose main interest is health current exercise level is moderate is moderately motivated and is comfortable with technological devices Turn the page if you need a hint! 6-25 s sharpen your pencil clue Ok. So we want to compute P(i100 | health, moderateExercise, moderateMotivation, techComfortable) and P(i500 | health, moderateExercise, moderateMotivation, techComfortable) and pick the model with the highest probability. Let me lay out what we need to do for the first one: P(i100 | health, moderateExercise, moderateMotivation, techComfortable) = P(health|i100) P(moderateExercise|i100) P(moderateMotivated|i100) P(techComfortable|i100)P(i100) So here is what we need to first compute P(health|i100) = 1/6 There were 6 occurrences of people buying i100s and only one of those people had a main interest of ‘health’ P(moderateExercise|i100) = P(moderateMotivated|i100) = P(techComfortable|i100) = P(i100) = 6 / 15 That was my clue. Now hopefully you can figure out the example 6-26 PROBABILITY AND NAÏVE BAYES s sharpen your pencil solution First we compute P(i100 | health, moderateExercise, moderateMotivation, techComfortable) which equals the product of all these terms: P(health|i100) P(moderateExercise|i100) P(moderateMotivated|i100) P(techComfortable|i100)P(i100) P(health|i100) = 1/6 P(moderateExercise|i100) = 1/6 P(moderateMotivated|i100) = 5/6 P(techComfortable|i100) = 2/6 P(i100) = 6 / 15 so P(i100| evidence) = .167 * .167 * .833 * .333 * .4 = .00309 Now we compute P(i500 | health, moderateExercise, moderateMotivation, techComfortable) P(health|i500) = 4/9 P(moderateExercise|i500) = 3/9 P(moderateMotivated|i500) = 3/9 P(techComfortable|i500) = 6/9 P(i500) = 9 / 15 P(i500| evidence) = .444 * .333 * .333 * .667 * .6 = .01975 6-27 Doing it in Python Great! Now that we understand how a Naïve Bayes Classifier works let us consider how to implement it in Python. The format of the data files will be the same as that in the previous chapter, a text file where each line consists of tab-separated values. For our iHealth example, the data file would look like the following: comfortable with tech devices? current exercise level main interest how motivated both! ! sedentary! both! ! sedentary! health! ! sedentary! appearance! acti ve! ! appearance! mode rate! appearance! mode rate! health! ! moderate! both! ! active! ! both! ! moderate! appearance! acti ve! ! both! ! active! ! health! ! active! ! health! ! sedentary! appearance! acti ve! ! health! ! sedentary! which model moderate! yes! i100 moderate! no! i100 moderate! yes! i500 moderate! yes! i500 aggressive! yes! i500 aggressive! no! i100 aggressive! no! i500 moderate! yes! i100 aggressive! yes! i500 aggressive! yes! i500 aggressive! no! i500 moderate! no! i500 aggressive! yes! i500 moderate! no! i100 moderate! no! i100 Shortly we will be using an example with substantially more data and I would like to keep the ten-fold cross validation methods we used in code from the previous chapter. Recall that that method involved dividing the data into ten buckets (files). We would train on nine of them and test the classifier on the remaining bucket. And we would repeat this ten times; each time withholding a different bucket for testing. The simple iHealth example, with only 15 instances, was designed so we could work through the Naïve Bayes Classifier method by hand. With only 15 instances it seems silly to divide them into 10 buckets. The ad hoc, not very elegant solution we will use, is to have ten buckets but all the 15 instances will be in one bucket and the rest of the buckets will be empty. 6-28 PROBABILITY AND NAÏVE BAYES The Naïve Bayes Classifier code consists of two components, one for training and one for classifying. Training The output of training needs to be: • a set of prior probabilities—for example, P(i100) = 0.4 • a set of conditional probabilities—for example, P(health|i100) = 0.167 I am going to represent the set of prior probabilities as a Python dictionary (hash table): self.prior = {'i500': 0.6, 'i100': 0.4} The conditional probabilities are a bit more complex. My way of doing this—and there are probably better methods—is to associate a set of conditional probabilities with each class. {'i500': {1: {'appearance': 0.3333333333333, 'health': 0.4444444444444, 'both': 0.2222222222222}, 2: {'sedentary': 0.2222222222222, 'moderate': 0.333333333333, 'active': 0.4444444444444444}, 3: {'moderate': 0.333333333333, 'aggressive': 0.66666666666}, 4: {'no': 0.3333333333333333, 'yes': 0.6666666666666666}}, 'i100': {1: {'appearance': 0.333333333333, 'health': 0.1666666666666, 'both': 0.5}, 2: {'sedentary': 0.5, 'moderate': 0.16666666666666, 'active': 0.3333333333333}, 3: {'moderate': 0.83333333334, 'aggressive': 0.166666666666}, 4: {'no': 0.6666666666666, 'yes': 0.3333333333333}}} The 1, 2, 3, 4 represent column numbers. So the first line of the above is “the probability of the value of the first column being ‘appearance’ given that the device is i500 is 0.333.” 6-29 The first step in computing these probabilities is simply to count things. Here are the first few lines of the input file: both! ! both! ! health! ! appearance! sedentary! sedentary! sedentary! active! ! moderate! moderate! moderate! moderate! yes!i100 no! i100 yes!i500 yes!i500 Yet again I am going to use dictionaries. One, called, classes, which will count the occurrences of each class or category. So, after the first line classes will look like Counting things {'i100': 1} After the second line: {'i100': 2} And after the third: {'i500': 1, 'i100': 2} After I process all the data, the value of classes is Prior probability {'i500': 9, 'i100': 6} To obtain the prior probabilities I simply divide those number by the total number of instances. To determine the conditional probabilities I am going Conditional to count the occurrences of attribute values in the different columns in a dictionary called counts. and I am going to maintain separate counts for each class. So, in processing the string ‘both’ in the first line, counts will be: {'i100': {1: {'both': 1}} and at the end of processing the data, the value of counts will be 6-30 probability PROBABILITY AND NAÏVE BAYES {'i100': {1: 2: 3: 4: 'i500': {1: 2: 3: 4: {'appearance':2, 'health': 1, 'both': 3}, {'active': 2, 'moderate': 1, 'sedentary': 3}, {'moderate': 5, 'aggressive': 1}, {'yes': 2, 'no': 4}}, {'health': 4, 'appearance': 3, 'both': 2}, {'active': 4, 'moderate': 3, 'sedentary': 2}, {'moderate': 3, 'aggressive': 6}, {'yes': 6, 'no': 3}}} So, in the first column of the i100 instances there were 2 occurrences of ‘appearance’, 1 of ‘health’ and 3 of ‘both’. To obtain the conditional probabilities we divide those numbers by the total number of instances of that class. For example, there are 6 instances of i100 and 2 of them had a value of ‘appearance’ for the first column, so P(‘appearance’|i100) = 2/6 = .333 With that background here is the Python code for training the classifier (remember, you can download this code at guidetodatamining.com). # _____________________________________________________________________ class BayesClassifier: def __init__(self, bucketPrefix, testBucketNumber, dataFormat): """ a classifier will be built from files with the bucketPrefix excluding the file with textBucketNumber. dataFormat is a string that describes how to interpret each line of the data files. For example, for the iHealth data the format is: "attr! attr! attr! attr! class" """ total = 0 classes = {} counts = {} # reading the data in from the file self.format = dataFormat.strip().split('\t') self.prior = {} self.conditional = {} 6-31 # for each of the buckets numbered 1 through 10: for i in range(1, 11): #if it is not the bucket we should ignore, read in the data if i != testBucketNumber: filename = "%s-%02i" % (bucketPrefix, i) f = open(filename) lines = f.readlines() f.close() for line in lines: fields = line.strip().split('\t') ignore = [] vector = [] for i in range(len(fields)): if self.format[i] == 'num': vector.append(float(fields[i])) elif self.format[i] == 'attr': vector.append(fields[i]) elif self.format[i] == 'comment': ignore.append(fields[i]) elif self.format[i] == 'class': category = fields[i] # now process this instance total += 1 classes.setdefault(category, 0) counts.setdefault(category, {}) classes[category] += 1 # now process each attribute of the instance col = 0 for columnValue in vector: col += 1 counts[category].setdefault(col, {}) counts[category][col].setdefault(columnValue,0) counts[category][col][columnValue] += 1 6-32 PROBABILITY AND NAÏVE BAYES # # ok done counting. now compute probabilities # # first prior probabilities p(h) # for (category, count) in classes.items(): self.prior[category] = count / total # # now compute conditional probabilities p(h|D) # for (category, columns) in counts.items(): self.conditional.setdefault(category, {}) for (col, valueCounts) in columns.items(): self.conditional[category].setdefault(col, {}) for (attrValue, count) in valueCounts.items(): self.conditional[category][col][attrValue] = ( count / classes[category]) That’s it for training! No Complex math. Just basic counting!!! Classifying Okay, we have trained the classifier. Now we want to classify various instances. For example, which model should we recommend for someone whose primary interest is health, and who is moderately active, moderately motivated, and is comfortable with technology: c.classify(['health', 'moderate', 'moderate', 'yes']) For this we need to compute hMAP = arg max h∈H P(D | h)P(h) 6-33 When we did this by hand we computing the probability of each hypothesis given the evidence and we simply translate that method to code: def classify(self, itemVector): """Return class we think item Vector is in""" results = [] for (category, prior) in self.prior.items(): prob = prior col = 1 for attrValue in itemVector: if not attrValue in self.conditional[category][col]: # we did not find any instances of this attribute value # occurring with this category so prob = 0 prob = 0 else: prob = prob * self.conditional[category][col][attrValue] col += 1 results.append((prob, category)) # return the category with the highest probability return(max(results)[1]) And when I try the code I get the same results we received when we did this by hand: >>c = Classifier("iHealth/i", 10, "attr\tattr\tattr\tattr\tclass") >>print(c.classify(['health', 'moderate', 'moderate', 'yes']) i500 6-34 PROBABILITY AND NAÏVE BAYES Republicans vs. Democrats Let us look at a new data set, the Congressional Voting Records Data Set, available from the Machine Learning Repository (http://archive.ics.uci.edu/ml/index.html). It is available in a form that can be used by our programs at http://guidetodatamining.com. The data consists of the voting record of United States Congressional Representatives. The attributes are how that representative voted on 16 different bills. Attribute Information: 1. Class Name: 2 (democrat, republican) 2. handicapped-infants: 2 (y,n) 3. water-project-cost-sharing: 2 (y,n) 4. adoption-of-the-budget-resolution: 2 (y,n) 5. physician-fee-freeze: 2 (y,n) 6. el-salvador-aid: 2 (y,n) 7. religious-groups-in-schools: 2 (y,n) 8. anti-satellite-test-ban: 2 (y,n) 9. aid-to-nicaraguan-contras: 2 (y,n) 10. mx-missile: 2 (y,n) 11. immigration: 2 (y,n) 12. synfuels-corporation-cutback: 2 (y,n) 13. education-spending: 2 (y,n) 14. superfund-right-to-sue: 2 (y,n) 15. crime: 2 (y,n) 16. duty-free-exports: 2 (y,n) 17. export-administration-act-south-africa: 2 (y,n) The file consists of tab separated values: democrat democrat democrat republican y y y y n y y y y y y y n n n n n n n n n n n y y y y y y y y y y y n y n y n y n n n n n n n n n n n n n n y n y y n n y y y y 6-35 Our Naïve Bayes Classifier works fine with this example (the format string says that the first column is to be interpreted as the class of the instance and the rest of the columns are to be interpreted as attributes): format = "””class\tattr\tattr\tattr\tattr\tattr\tattr\tattr\tattr\tattr \tattr\tattr\tattr\tattr\tattr\tattr\tattr””" tenfold("house-votes/hv", format) Classified as: democrat republican +-------+-------+ democrat | 111 | 13 | |-------+-------| republican | 9 | 99 | +-------+-------+ 90.517 percent correct total of 232 instances That’s great! Wait! There are some problems with this approach. To see one of the problems with this approach consider a different United States House of Representatives example. Out of the 435 voting representatives I have drawn a training sample of 200—100 Democrats and 100 Republicans. The following table indicates what percent voted ‘yes‘ to 4 different bills. 6-36 PROBABILITY AND NAÏVE BAYES CISPA Reader Privacy Act Internet Sales Tax Internet Snooping Bill Republican 0.99 0.01 0.99 0.5 Democrat 0.01 0.99 0.01 1.0 % voting ‘yes’ This table shows that 99% of Republicans in the sample voted for the CISPA (Cyber Intelligence Sharing and Protection Act), only 1% voted for the Reader Privacy Act, 99% voted for Internet Sales Tax and 50% voted for the Internet Snooping Bill. (I made up these numbers and they do not reflect reality.) We pick a U.S. Representative who wasn’t in our sample, Representative X, who we would like to classify as either a Democrat or Republican. I added how that representative voted to our table: CISPA Reader Privacy Act Internet Sales Tax Internet Snooping Bill Republican 0.99 0.01 0.99 0.5 Democrat 0.01 0.99 0.01 1.0 Rep. X N Y N N t ocra erson is a Dem p e th k in th u Do yo or Republican? 6-37 I would guess Democrat. Let us work through the example step-by-step using Naïve Bayes. The prior probabilities of P(Democrat) and P(Republican) are both 0.5 since there are 100 Republicans and 100 Democrats in the sample. We know that Representative X voted ‘no’ to CISPA and we also know P(Republican|C=no) = 0.01 and P(Democrat|C=no) = 0.99 where C = CISPA. And with that bit of evidence our current P(h|D) probabilities are h= p(h) P(C=no|h) P(h|D) Republican 0.5 0.01 0.005 Democrat 0.5 0.99 0.495 Factoring in Representative X’s ‘yes’ vote to the Reader Privacy Act and X’s ‘no’ to the sales tax bill we get: h= p(h) P(C=no|h) P(R=yes|h) P(T=no|h) P(h|D) Republican 0.5 0.01 0.01 0.01 0.0000005 Democrat 0.5 0.99 0.99 0.99 0.485 If we normalize these probabilities: P(Democrat | D)= 0.485 0.485 = = 0.99999 0.485 + 0.0000005 0.4850005 So far we are 99.99% sure Representative X is a Democrat. Finally, we factor in Representative X’s ‘no’ vote on the Internet Snooping Bill. 6-38 PROBABILITY AND NAÏVE BAYES h= p(h) P(C=no|h) P(R=yes|h) P(T=no|h) P(S=no|h) P(h|D) Republican 0.5 0.01 0.01 0.01 0.50 2.5E-07 Democrat 0.5 0.99 0.99 0.99 0.00 0 Whoops. We went from 99.99% likelihood that X was a Democrat to 0%. This is so because we had 0 occurrences of a Democrat voting ‘no’ for the snooping bill. Based on these probabilities we predict that person X is a Republican. This goes against our intuition! Estimating Probabilities The probabilities in Naïve Bayes are really estimates of the true probabilities. True probabilities are those obtained from the entire population. For example, if we could give a cancer test to everyone in the entire population, we could, for example, get the true probability of the test returning a negative result given that the person does not have cancer. However, giving the test to everyone is near impossible. We can estimate that probability by selecting a random representative sample of the population, say 1,000 people, giving the test to them, and computing the probabilities. Most of the time this gives us a very good estimate of the true probabilities, but when the true probabilities are very small, these estimates are likely to be poor. Here is an example. Suppose the true probability of a Democrat voting no to the Internet Snooping Bill is 0.03—P(S=no|Democrat) = 0.03. s Brain Calisthenics Suppose we try to estimate these probabilities by selected a sample of 10 Democrats and 10 Republicans. What is the most probable number of Democrats in the sample that voted no to the snooping bill? ☐0 ☐1 ☐2 ☐3 6-39 s Brain Calisthenics—answer Suppose we try to estimate these probabilities by selected a sample of 10 Democrats and 10 Republicans. What is the most probable number of Democrats in the sample that voted no to the snooping bill? 0 So based on the sample P(S=no|Democrat) = 0. As we just saw in the previous example, when a probability is 0 it dominates the Naïve Bayes calculation—it doesn’t matter what the other values are. Another problem is that probabilities based on a sample produce a biased underestimate of the true probability. Fixing this. If we a trying to calculate something like P(S=no|Democrat) our calculation has been the number that both are Democrats and voted no on the snooping bill. P(S=no|Democrat) = total number of Democrats For expository ease let me simplify this by using shorter variable names: P(x | y) = nc n Here n is the total number of instances of class y in the training set; nc is the total number of instances of class y that have the value x. 6-40 PROBABILITY AND NAÏVE BAYES The problem we have is when nc equals zero. We can eliminate this problem by changing the formula to: P(x | y) = nc + mp n+m This formula is from p179 of the bo ok “Machine Lear ning” by To m Mitchell. m is a constant called the equivalent sample size. The method for determining the value of m varies. For now I will use the number of different values that attribute takes. For example, there are 2 values for how a person voted on the snooping bill, yes, or no. So I will use an m of 2. p is the prior estimate of the probability. Often we assume uniform probability. For example, what is the probability of someone voting no to the snooping bill knowing nothing about that person? 1/2. So p in this case is 1/2. Let’s go through the previous example to see how this works. First, here are tables showing the vote: Republican Vote CISPA Reader Privacy Act Internet Sales Tax Internet Snooping Bill Yes 99 1 99 50 No 1 99 1 50 CISPA Reader Privacy Act Internet Sales Tax Internet Snooping Bill Yes 1 99 1 100 No 99 1 99 0 Democratic Vote 6-41 The person we are trying to classify voted no to CISPA. First we compute the probability that he’s a Republican given that vote. Our new formula is P(x | y) = nc + mp n+m n is the number of Republicans, which is 100 and nc is the number of Republicans who voted no to CISPA, which is 1. m is the number of values for the attribute “how they voted on CISPA”, which is 2 (yes or no). So plugging those number into our formula P(cispa = no | republican) = 1+ 2(.5) 2 = = 0.01961 100 + 2 102 We follow the same procedure for a person voting no to CISPA given they are a Democrat. P(cispa = no | democrat) = 99 + 2(.5) 100 = = 0.9804 100 + 2 102 With that bit of evidence our current P(h|D) probabilities are h= p(h) P(C=no|h) P(h|D) Republican 0.5 0.01961 0.0098 Democrat 0.5 0.9804 0.4902 Factoring in Representative X’s ‘yes’ vote to the Reader Privacy Act and X’s ‘no’ to the sales s sharpen your pencil Finish this problem and classify the individual as either a Republican or Democrat. Recall, he voted no to Cispa, yes to the Reader Privacy act, and no both to the sales tax and snooping bills. 6-42 PROBABILITY AND NAÏVE BAYES s sharpen your pencil -answer Finish this problem and classify the individual as either a Republican or Democrat. Recall, he voted no to CISPA, yes to the Reader Privacy act, and no both to the Internet sales tax and snooping bills. The calculations for the next 2 columns mirror those we did for the CISPA vote. The probability that this person voted no to the snooping bill given that he is a Republican is P(s = no | republican) = 50 + 2(.5) 51 = = 0.5 100 +2 102 and that he voted no given that he is a Democrat: P(s = no | democrat) = 0 + 2(.5) 1 = = 0.0098 100 +2 102 Multiplying those probabilities together gives us h= p(h) P(C=no|h) P(R=yes|h) P(I=no|h) P(S=no|h) P(h|D) Republican 0.5 0.01961 0.01961 0.01961 0.5 0.000002 Democrat 0.5 0.9804 0.9804 0.9804 0.0098 0.004617 So unlike the previous approach we would classify this individual as a Democrat. This matches our intuitions. 6-43 A clarification For this example, the value of m was 2 for all calculations. However, it is not the case that m remains necessarily constant across attributes. Consider the health monitor example discussed earlier in the chapter. The attributes for that example included: survey What is your m ain interest in ge tting a monitor ◦ health ? ● appearance ◦ both What is your cu rrent exercise level? ● sedentary ◦ moderate ◦ active Are you comfort able with tech devices? ● yes ◦ no For this attrib ute, m = 3 sin ce the attribute can take one of 3 values (health, appe arance, both). If we assume unifor m = 1/3 since th probabilities, then p e probability of attribute bein g any one of th the e values is This attribute also has m = 3 and p = 1/3 For this attrib ute, m = 2 sin ce the attribute can take one of 2 values and p = 1/2 since the probabilit y of attribute bein g any one of th the ose is 1/2 Let us say the number of the people surveyed who own the i500 monitor is 100 (this is n). The number of people who own a i500 and are sedentary is 0 (nc). So, the probability of someone being sedentary given they own an i500 is P(sedentary | i500) = 6-44 nc + mp 0 + 3(.333) 1 = = = 0.0097 n+m 100 + 3 103 PROBABILITY AND NAÏVE BAYES Numbers You probably noticed that I changed from numerical data which I used in all the nearest neighbor approaches I discussed to using categorical data for the naïve Bayes formula. By “categorical data” we mean that the data is put into discrete categories. For example, we divide people up in how they voted for a particular bill and the people who voted ‘yes’ go in one category and the people who voted ‘no’ go in another. Or we might categorize musicians by the instrument they play. So all saxophonists go in one bucket, all drummers in another, all pianists in another and so on. And these categories do not form a scale. So, for example, saxophonists are not ‘closer’ to pianists than they are to drummers. Numerical data is on a scale. An annual salary of $105,000 is closer to a salary of $110,000 than it is to one of $40,000. For Bayesian approaches we count things—how many occurrences are there of people who are sedentary—and it may not be initially obvious how to count things that are on a scale—for example, something like grade point average. There are two approaches. Method 1: Making categories One solution is to make categories by discretizing the continuous attribute. You often see this on websites and on survey forms. For example: Age ◦ < 18 ◦ 18-22 ◦ 23-30 ◦ 30-40 ◦ > 40 Once we have this information divided nicely into discrete values, we can use Naïve Bayes exactly as we did before. Annual Salary ◦ > $200,000 ◦ 150,000 - 200,000 ◦ 100,00 - 150,000 ◦ 60,000-100,000 ◦ 40,000-60,000 6-45 Method 2: Gaussian distributions! Harumph! Well I would take that income attribute and discretize it into distinct categories! Then we can use Naïve Bayes! 6-46 That’s sort of old school. I would just use a Gaussian distribution and deal with that attribute using a probability density function. We can still use Bayes. PROBABILITY AND NAÏVE BAYES The terms “Gaussian Distribution” and “Probability Density Function” sound cool, but they are more than good phrases to know so you can impress your friends at dinner parties. So what do they mean and how they can be used with the Naïve Bayes method? Consider the example we have been using with an added attribute of income: Main Interest Current How Motivated Comfortable Income Exercise Level with tech. (in $1,000) Devices? Model # both sedentary moderate yes 60 i100 both sedentary moderate no 75 i100 health sedentary moderate yes 90 i500 appearance active moderate yes 125 i500 appearance moderate aggressive yes 100 i500 appearance moderate aggressive no 90 i100 health moderate aggressive no 150 i500 both active moderate yes 85 i100 both moderate aggressive yes 100 i500 appearance active aggressive yes 120 i500 both active aggressive no 95 i500 health active moderate no 90 i500 health sedentary aggressive yes 85 i500 appearance active moderate no 70 i100 health sedentary moderate no 45 i100 Let’s think of the typical purchaser of an i500, our awesome, premiere device. If I were to ask you to describe this person you might give me the average income: mean = 90 + 125 + 100 + 150 + 100 + 120 + 95 + 90 + 85 955 = = 106.111 9 9 And perhaps after reading chapter 4 you might give the standard deviation: 6-47 sd = ∑ (x − x ) 2 i i card(x) Recall that the standard deviation describes the range of scattering. If all the values are bunched up around the mean, the standard deviation is small; if the values are scattered the standard deviation is large s sharpen your pencil What is the income standard deviation of the people who bought the i500? (those values are shown in the column below) Income (in $1,000) 90 125 100 150 100 120 95 90 85 6-48 PROBABILITY AND NAÏVE BAYES s sharpen your pencil - solution What is the standard deviation of the income of the people who bought the i500? (those values are shown in the column above) Income (in $1,000) (x-106.111) (x-106.111)2 90 -16.111 259.564 125 18.889 356.794 100 -6.111 37.344 150 43.889 1926.244 100 -6.111 37.344 120 13.889 192.904 95 -11.111 123.454 90 -16.111 259.564 85 -21.111 445.674 sd = 3638.889 9 = 404.321 = 20.108 ∑ = 3638.889 6-49 Population standard deviation and sample standard deviation. The formula for standard deviation that we just used is called the population standard deviation. It is called that because we use this formula when we have data on the entire population we are interested in. For example, we might give a test to 500 students and then compute the mean and standard deviation. In this case, we would use the population standard deviation, which is what we have been using. Often, though, we do not have data on the entire population. For example, suppose I am interested in the effects of drought on the deer mice in Northern New Mexico and as part of that study I want the average (mean) and standard deviation of their weights. In this case I am not going to weigh every mouse in Northern New Mexico. Rather I will collect and weigh some representative sample of mice. For this sample, I can use the standard deviation formula I used above, but there is another formula that has been shown to be a better estimate of the entire population standard deviation. This formula is called the sample standard deviation and it is just a slight modification of the previous formula: sd = ∑ (x − x ) 2 i i card(x) − 1 The sample standard deviation of the income example is 6-50 PROBABILITY AND NAÏVE BAYES sd = 3638.889 = 9 −1 3638.889 8 = 454.861 = 21.327 For the rest of this chapter we will be using sample standard deviation. You probably have heard terms such as normal distribution, bell curve, and Gaussian distribution. Gaussian distribution is just a high falutin term for normal distribution. The function that describes this distribution is called the Gaussian function or bell curve. Most of the time the Numerati (aka data miners) assume attributes follow a Gaussian distribution. What is means is that about 68% of the instances in a Gaussian distribution fall within 1 standard deviation of the mean and 95% of the instances fall within 2 standard deviations of the mean: In our case, the mean was 106.111 and the sample standard deviation was 21.327. So 95% of the people who purchase an i500 earn between $42,660 and $149,770. If I asked you if you thought P(100k| i500) —the likelihood that an i500 purchaser earns $100,000—was, you might think that's pretty likely. If I asked you what you thought the likelihood of P(20k| i500)—the likelihood that an i500 purchaser earns $20,000—was , you might think it was pretty unlikely. 6-51 To formalize this, we are going to use the mean and standard deviation to compute this probability as follows: 1 P(xi | y j ) = e 2πσ ij Maybe putting the formula in a bigger font makes it look simpler! −( xi − µij )2 2 σ ij2 Everytime I type a complex looking formula into this book, I feel the need to say something like “don’t panic.” It could be that none of you readers panic and I am just the one panicking. However, let me say this. Data mining has professional terminology and formulas. Before you dive into data mining you might think “those things look difficult.” But after you study, even for a short time, these formulas become nothing special. It is just a matter of working through the formula out step-by-step. Let’s jump right into dissecting this formula so we can see how simple it really is. Let us say we are interested in computing P(100k|i500), the probability that a person earns $100,000 (or 100k) given they purchased an i500. A few pages ago we computed the average income (mean) of people who bought the i500. We also computed the sample standard deviation. These values are shown on the following page. In Numerati speak, we represent the mean with the Greek letter µ (mu) and the standard deviation as σ (sigma). 6-52 PROBABILITY AND NAÏVE BAYES 1 P(xi | y j ) = e 2πσ ij −( xi − µij )2 2 σ ij2 µij = 106.111 σij = 21.327 xi = 100 Let’s plug these values into the formula: 1 e 2π (21.327) P(xi | y j ) = −(100−106.111)2 2(21.327)2 and do some math: −(37.344 ) 1 P(xi | y j ) = e 909.68 6.283(21.327) and more math: P(xi | y j ) = 1 e−0.0411 53.458 The e is a mathematical constant that is the base of the natural logarithm. It’s value is approximately 2.718. P(xi | y j ) = 1 (2.718)−0.0411 = (0.0187)(0.960) = 0.0180 53.458 So the probability that the income of a person who bought the i500 is $100,000 is 0.0180. 6-53 s sharpen your pencil In the table below I have the horsepower ratings for cars that get 35 miles per gallon. I would like to know the probability of a Datsun 280z having 132 horsepower given it gets 35 miles per gallon. car HP Datsun 210 65 Ford Fiesta 66 VW Jetta 74 Nissan Stanza 88 Ford Escort 65 Triumph tr7 coupe 88 Plymouth Horizon 70 Suburu DL 67 6-54 μij = _____ σij = _____ xi = _____ PROBABILITY AND NAÏVE BAYES s sharpen your pencil -solution - part 1 In the table below I have the horsepower ratings for cars that get 35 miles per gallon. I would like to know the probability of a Datsun 280z having 132 horsepower given it gets 35 miles per gallon. μij = 72,875 car HP Datsun 210 65 Ford Fiesta 66 VW Jetta 74 Nissan Stanza 88 Ford Escort 65 Triumph tr7 coupe 88 Plymouth Horizon 70 Suburu DL 67 σij = 9.804 xi =132 σ= (65 − µ)2 +(66 − µ)2 + (74 − µ)2 + (88 − µ)2 + (65 − µ)2 + (88 − µ)2 + (70 − µ)2 + (67 − µ)2 7 σ= 672.875 = 96.126 = 9.804 7 6-55 s sharpen your pencil -solution - part 2 In the table below I have the horsepower ratings for cars that get 35 miles per gallon. I would like to know the probability of a Datsun 280z having 132 horsepower given it gets 35 miles per gallon. μij = 72,875 σij = 9.804 P(xi | y j ) = 1 e 2πσ ij −( xi − µij )2 2 σ ij2 xi =132 1 P(132hp | 35mpg) = e 2π (9.804) −(132−72.875)2 2(9.804 )2 −3495.766 1 1 = e 192.237 = e−18.185 24.575 6.283(9.804) = 0.0407(0.00000001266) = 0.0000000005152 Ok. it is extremely unlikely that a Datsun 280z, given that it gets 35 miles to the gallon has 132 horsepower. (but it does!) 6-56 PROBABILITY AND NAÏVE BAYES A few implementation notes. In the training phase for Naive Bayes, we will compute the mean and sample standard deviation of every numeric attribute. Shortly, we will see how to do this in detail. In the classification phase, the above formula can be implemented with just a few lines of Python: import math def pdf(mean, ssd, x): """Probability Density Function computing P(x|y) input is the mean, sample standard deviation for all the items in y, and x.""" ePart = math.pow(math.e, -(x-mean)**2/(2*ssd**2)) return (1.0 / (math.sqrt(2*math.pi)*ssd)) * ePart Let’s test this with the examples we did above: >>>pdf(106.111, 21.327, 100) 0.017953602706962717 >>>pdf(72.875, 9.804, 132) 5.152283971078022e-10 Whew! Time for a break! 6-57 Python Implementation Training Phase The Naïve Bayes method relies on prior and conditional probabilities. Let’s go back to our Democrat/Republican example. Prior probabilities are the probabilities that hold before we have observed any evidence. For example, if I know there are 233 Republicans and 200 Democrats, then the prior probability of some arbitrary member of the U.S. House of Representatives being a Republican is P(republican) = 233 = 0.54 433 This is denoted P(h). Conditional Probability P(h|D) is the probability that h holds given that we know D, for example, P(democrat|bill1Vote=yes). In Naïve Bayes, we flip that probability and compute P(D|h)—for example, P(bill1Vote=yes|democrat). In our existing Python program we store these conditional probabilities in a dictionary of the following form: {'democrat': {'bill 1': {'yes': 0.333, 'no': 0.667}, 'bill 2': {'yes': 0.778, 'moderate': 0.222}} 'republican': {'bill 1': {'yes': 0.811, 'no': 0.189}, 'bill 2': {'yes': 0.250, 'no': 0.750}}} So the probability that someone voted yes to bill 1 given that they are a Democrat (P(bill 1=yes|democrat)) is 0.333. We will keep this data structure for attributes whose values are discrete values (for example, ‘yes’, ‘no’, ‘sex=male’, ‘sex=female’). However, when attributes are numeric we will be using the probability density function and we need to store the mean and sample standard deviation for that attribute. For these numeric attributes I will use the following structures: 6-58 PROBABILITY AND NAÏVE BAYES mean = {'democrat': {'age': 57, 'years served': 12} 'republican': {'age': 53, 'years served': 7}} ssd = {'democrat': {'age': 7, 'years served': 3} 'republican': {'age': 5, 'years served': 5}} As before, each instance is represented by a line in a data file. The attributes of each instances are separated by tabs. For example, the first few lines of a data file for the Pima Indians Diabetes Data set might be: 3! 78! 50 4! 111! ! 32! 88! 31.0! 72! 47! 0.248! 207! 37.1! 1! 189! 26! 1 60 1.390! 1! 117! ! 23! 846! 30.1! 5 6! 1 88 0.398! 3! 107! ! 24! 145! 34.5! 59! 1 62! 13! 0 .403! 4 48! 22.9! 7! 81! 0! 1 78! 40! 0.678! 48! 46.7! 2! 99! 23! 1 70! 16! 0 . 261! 42! 44! 20.4! 5! 105! 0 72 0.235! 2! 142! ! 29! 325! 36.9! 27! 0 82! 18! 0 . 159! 28! 64! 24.7! 1! 81! 0 72 0.761! 0! 100! ! 18! 40! 26.6! 21! 0 88! 60! 0 . 283! 24! 110! 46.8! 0 0.962! 31! 0 The columns represent, in order, the number of times pregnant, plasma glucose concentration, blood pressure, triceps skin fold thickness, serum insulin level, body mass index, diabetes pedigree function, age, and a ‘1’ in the last column represents that they developed diabetes and a ‘0’ they did not. Also as before, we are going to represent how the program should interpret each column by use of a format string, which uses the terms • attr identifies columns that should be interpreted as non-numeric attributes, and which will use the Bayes methods shown earlier in this chapter. • num identifies columns that should be interpreted as numeric attributes, and which will use the Probability Density Function (so we will need to compute the mean and standard deviation during training). • class identifies the column representing the class of the instance (what we are trying to learn) 6-59 In the Pima Indian Diabetes data set the format string will be "num num num num num num num num class" To compute the mean and sample standard deviation we will need some temporary data structures during the training phase. Again, let us look at a small sample of the Pima data set. 3! 78! 50 4! 111! ! 32! 88! 31.0! 72! 47! 207! 37.1! 1! 189! 60 2! 142! ! 23! 846! 30.1! 82! 18! 64! 24.7! 1! 81! 72 0! 100! ! 18! 40! 26.6! 88! 60! 110! 46.8! 0.248! 1.390! 0.398! 0.761! 0.283! 0.962! 26! 1 56! 1 59! 1 21! 0 24! 0 31! 0 The last column represents the class of each instance. So the first three individuals developed diabetes and that last three did not. All the other columns represent numeric attributes. of which we need to compute the mean and standard deviation for each of the two classes. To compute the mean for each class and attribute I will need to keep track of the running total. In our existing code we already keep track of the total number of instances. I will implement this using a dictionary: totals {'1': {1: 8, 2: 378, 3: 182, 4: 102, 5: 1141, 6: 98.2, 7: 2.036, 8: 141}, {'0': {1: 3, 2: 323, 3: 242, 4: 96, 5: 214, 6: 98.1, 7: 2.006, 8: 76} So for class 1, the column 1 total is 8 (3 + 4 + 1), the column 2 total is 378, etc. For class 0, the column 1 total is 3 (2 + 1 + 0), the column 2 total is 323 and so on. For standard deviation, we will also need to keep the original data, and for that we will use a dictionary in the following format: 6-60 PROBABILITY AND NAÏVE BAYES numericValues {'1': 1: [3, 4, 1], 2: [78, 111, 189], ...}, {'0': {1: [2, 1, 0], 2: [142, 81, 100]} I have added the code to create these temporary data structures to the __init__() method of our Classifier class as shown below: import math class Classifier: def __init__(self, bucketPrefix, testBucketNumber, dataFormat): """ a classifier will be built from files with the bucketPrefix excluding the file with textBucketNumber. dataFormat is a string that describes how to interpret each line of the data files. For example, for the iHealth data the format is: "attr!attr! attr! attr! class" """ total = 0 classes = {} # counts used for attributes that are not numeric counts = {} # totals used for attributes that are numereric # we will use these to compute the mean and sample standard deviation # for each attribute - class pair. totals = {} numericValues = {} # reading the data in from the file self.format = dataFormat.strip().split('\t') # self.prior = {} self.conditional = {} # for each of the buckets numbered 1 through 10: for i in range(1, 11): # if it is not the bucket we should ignore, read in the data if i != testBucketNumber: filename = "%s-%02i" % (bucketPrefix, i) f = open(filename) 6-61 lines = f.readlines() f.close() for line in lines: fields = line.strip().split('\t') ignore = [] vector = [] nums = [] for i in range(len(fields)): if self.format[i] == 'num': nums.append(float(fields[i])) elif self.format[i] == 'attr': vector.append(fields[i]) elif self.format[i] == 'comment': ignore.append(fields[i]) elif self.format[i] == 'class': category = fields[i] # now process this instance total += 1 classes.setdefault(category, 0) counts.setdefault(category, {}) totals.setdefault(category, {}) numericValues.setdefault(category, {}) classes[category] += 1 # now process each non-numeric attribute of the instance col = 0 for columnValue in vector: col += 1 counts[category].setdefault(col, {}) counts[category][col].setdefault(columnValue, 0) counts[category][col][columnValue] += 1 # process numeric attributes col = 0 for columnValue in nums: col += 1 totals[category].setdefault(col, 0) #totals[category][col].setdefault(columnValue, 0) totals[category][col] += columnValue numericValues[category].setdefault(col, []) numericValues[category][col].append(columnValue) 6-62 PROBABILITY AND NAÏVE BAYES # # ok done counting. now compute probabilities # first prior probabilities p(h) # for (category, count) in classes.items(): self.prior[category] = count / total # # now compute conditional probabilities p(h|D) # for (category, columns) in counts.items(): self.conditional.setdefault(category, {}) for (col, valueCounts) in columns.items(): self.conditional[category].setdefault(col, {}) for (attrValue, count) in valueCounts.items(): self.conditional[category][col][attrValue] = ( count / classes[category]) self.tmp = counts # # now compute mean and sample standard deviation # s code it Can you add the code to compute the means and standard deviations? Download the file naiveBayesDensityFunctionTraining.py from guidetodatamining.com. Your program should produce the data structures ssd and means: c = Classifier("pimaSmall/pimaSmall", 1, "num!num! num! num! num! num! num! num! >> c.ssd {'0': {1: 2.54694671925252, 2: 23.454755259159146, ...}, '1': {1: 4.21137914295475, 2: 29.52281872377408,}} >>> c.means {'0': {1: 2.8867924528301887, 2: 111.90566037735849, ...}, '1': {1: 5.25, 2: 146.05555555555554, ...}} class") ! 6-63 s code it solution Here is my solution: # # now compute mean and sample standard deviation # self.means = {} self.ssd = {} self.totals = totals for (category, columns) in totals.items(): self.means.setdefault(category, {}) for (col, cTotal) in columns.items(): self.means[category][col] = cTotal / classes[category] # standard deviation for (category, columns) in numericValues.items(): self.ssd.setdefault(category, {}) for (col, values) in columns.items(): SumOfSquareDifferences = 0 theMean = self.means[category][col] for value in values: SumOfSquareDifferences += (value - theMean)**2 columns[col] = 0 self.ssd[category][col] = math.sqrt(SumOfSquareDifferences / (classes[category] - 1)) The file containing this solution is naiveBayesDensityFunctionTrainingSolution.py at our website. 6-64 PROBABILITY AND NAÏVE BAYES s code it 2 Can you revise the classify method so it uses the probability density function for numeric attributes? The file to modify is naiveBayesDensityFunctionTemplate.py. Here is the original classify method: def classify(self, itemVector, numVector): """Return class we think item Vector is in""" results = [] sqrt2pi = math.sqrt(2 * math.pi) for (category, prior) in self.prior.items(): prob = prior col = 1 for attrValue in itemVector: if not attrValue in self.conditional[category][col]: # we did not find any instances of this attribute value # occurring with this category so prob = 0 prob = 0 else: prob = prob * self.conditional[category][col][attrValue] col += 1 # return the category with the highest probability #print(results) return(max(results)[1]) 6-65 s code it 2 - solution Can you revise the classify method so it uses the probability density function for numeric attributes? The file to modify is naiveBayesDensityFunctionTemplate.py. Solution: def classify(self, itemVector, numVector): """Return class we think item Vector is in""" results = [] sqrt2pi = math.sqrt(2 * math.pi) for (category, prior) in self.prior.items(): prob = prior col = 1 for attrValue in itemVector: if not attrValue in self.conditional[category][col]: # we did not find any instances of this attribute value # occurring with this category so prob = 0 prob = 0 else: prob = prob * self.conditional[category][col] [attrValue] col += 1 col = 1 for x in numVector: mean = self.means[category][col] ssd = self.ssd[category][col] ePart = math.pow(math.e, -(x - mean)**2/(2*ssd**2)) prob = prob * ((1.0 / (sqrt2pi*ssd)) * ePart) col += 1 results.append((prob, category)) # return the category with the highest probability #print(results) return(max(results)[1]) 6-66 PROBABILITY AND NAÏVE BAYES Is this any better than the Nearest Neighbor Algorithm? In Chapter 5 we evaluated how well the k Nearest Neighbor algorithm did with both the total Pima data set and a subset. Here are those results: pimaSmall pima k=1 59.00% 71.247% k=3 61.00% 72.519% Here are the results when we use Naïve Bayes with these two data sets: Bayes pimaSmall pima 72.000% 77.354% Wow! So it looks like Naïve Bayes performs better than kNN! The kappa score for the kNN where k=3 on the large data set was 0.35415, only fair performance. I wonder what kappa is for Naïve Bayes? 6-67 The kappa is 0.4875, moderate agreement! So for this example, Naïve Bayes is better than k Advantages of Bayes • simple to implement (just counting things) • need less training data than many other methods • a good method to use if you want something that performs well and has good performance times. Advantages of kNN • simple to implement. • does not assume the data has any particular structure—a good thing! • large amount of memory needed to store the training set. 6-68 Main disadvantage of Bayes: It cannot learn interactions among features. For example, it cannot learn that I like foods with cheese and I like foods with rice but I do not like foods with both kNN k Nearest Neighbors is a reasonable choice when the training set is large. kNN is extremely versatile and used in a large number of fields including recommendation systems, proteomics (the study of the entire protein set of an organism), the interaction among proteins, and image classification. PROBABILITY AND NAÏVE BAYES What enables us to multiple probabilities together is the fact that the events these probabilities represent are independent. For example, consider a game where we flip a coin and roll a die. These events are independent meaning what we roll on the die does not depend on whether we flip a heads or tails on the coin. And, as I just said, if events are independent we can determine their joint probability (the probability that they both occurred) by multiplying the individual probabilities together. So the probability of getting a heads and rolling a 6 is P(heads ∧ 6) = P(heads)× P(6) = 0.5 × 1 = 0.08333 6 Let's say I alter a deck of cards keeping all the black cards (26 of them) but only retaining the face cards for the red suits (6 of them). That makes a 32 card deck. What is the probability of selecting a face card? P( facecard) = 12 = 0.375 32 6-69 The probability of selecting a red card is P(red) = 6 = 0.1875 32 What is the probability of selecting a single card that is both red and a face card? Here we do not multiply probabilities. We do not do P(red ∧ facecard) = P(red) × P( facecard) = 0.375 × 0.185 = 0.0703 Here is what our common sense tells us. The chance of picking a red card is .1875. But if we pick a red card it is 100% likely it will be a face card. So it seems that the probability of picking a card that is both red and a face card is .1875. Or we can start a different way. The probability of picking a face card is .375. The way the deck is arranged half the face cards are red. So the probability of picking a card that is both red and a face card is .375 * .5 = .1875. Here we cannot multiply probabilities together because the attributes are not independent— if we pick red the probability of a face card changes—and vice versa. In many if not most real world data mining problems there are attributes that are not independent. Consider the athlete data. Here we had 2 attributes weight and height. Weight and height are not independent. The taller you get the more likely you will be heavier. 6-70 Suppose I have attributes zip code, income, and age. These are not indepen dent. Certain zipcodes have big bucks houses others consist of trailer parks. Palo Alto zipcodes may be dominated by 20somethings—Arizona zipcodes may be dominated by retirees. PROBABILITY AND NAÏVE BAYES gs Think about the music attributes—thin (1-5 ar guit like amount of distorted scale), amount of classical violin sound. Here many of these attributes are not d independent. If I have a lot of distorte a ing hav of ility guitar sound, the probab . classical violin sound decreases Suppose I have a dataset consisting of blood test results. Many of these values are not independent. For example, there are multiple thyroid tests including free T4 and TSH. There is an inverse relationship between the values of these two tests. Think about cases yourself. For example, consider attributes of cars. Are they independent? Attributes of a movie? Amazon purchases? So, for Bayes to work we need to use attributes that are independent, but most real-world problems violate that condition. What we are going to do is just to assume that they are independent! We are using the magic wand of sweeping things under the rug™—and ignoreing this problem. We call it naïve Bayes because we are naïvely assuming independence even though we know it is not. It turns out that naïve Bayes works really, really, well even with this naïve assumption. s code it Can you run the naïve Bayes code on our other data sets? For example, our kNN algorithm was 53% accurate on the auto MPG data set. Does a Bayes approach produce better results? tenfold("mpgData/mpgData", "class attr! num num num num! comment") ????? ! 6-71 Chapter 7: Naïve Bayes and Text Classifying unstructured text In previous chapters we've looked at recommendation systems that have people explicitly rate things with star systems (5 stars for Phoenix), thumbs-up/thumbs-down (Inception-thumbs-up!), and numerical scales. We've looked at implicit things like the behavior of people—did they buy the item, did they click on a link. We have also looked at classification systems that use attributes like height, weight, how people voted on a particular bill. In all these cases the information in the datasets can easily be represented in a table. age 26 56 23 glucose level 78 111 81 blood pressure 50 72 78 diabetes? 1 1 0 mpg cylinde 30 45 20 rs HP sec. 0-6 4 68 4 48 8 130 19.5 21.7 12 0 This type of data is called “structured data”—data where instances (rows in the tables above) are described by a set of attributes (for example, a row in a table might describe a car by a set of attributes including miles per gallon, the number of cylinders and so on). Unstructured data includes things like email messages, twitter messages, blog posts, and newspaper articles. These types of things (at least at first glance) do not seem to be neatly represented in a table. For example, suppose we are interested in determining whether various movies are good or not good and we want to analyze Twitter messages: We, as speakers of English can see that Andy Gavin likes Gravity, since he said “puts the thrill back in thriller” and “good acting.” We know that Debra Murphy seems not so excited about the movie since she said “save your $$$.” And if someone writes “I wanna see Gravity sooo bad, we should all go see it!!!” that person probably likes the movie even though they used the word bad. Suppose I am at my local food co-op and see something called Chobani Greek Yogurt. It looks interesting but is it any good? I get out my iPhone, do a google search and find the following from the blog “Woman Does Not Live on Bread Alone”: 7-2 NAIVE BAYES AND TEXT Chobani nonfat greek yogurt. Have you ever had greek yogurt? If not, stop reading, gather your keys (and a coat if you live in New York) and get to your local grocery. Even when nonfat and plain, greek yogurt is so thick and creamy, I feel guilty whenever I eat it. It is definitely what yogurt is MEANT to be. The plain flavor is tart and fantastic. Those who can have it, try the honey version. There's no sugar, but a bit of honey for a taste of sweetness (or add your own local honey-- local honey is good for allergies!). I must admit, even though I'm not technically supposed to have honey, if I've had a bad day, and just desperately need sweetness, I add a teaspoon of honey to my yogurt, and it's SO worth it. The fruit flavors from Chobani all have sugar in them, but fruit is simply unnecessary with this delicious yogurt. If your grocery doesn't carry the Chobani brand, Fage (pronounced Fa-yeh) is a well known, and equally delicious brand. Now, for Greek yogurt, you will pay about 50 cents to a dollar more, and there are about 20 more calories in each serving. But it's worth it, to me, to not feel deprived and saddened over an afternoon snack! http://womandoesnotliveonbreadalone.blogspot.com/2009/03/sugar-free-yogurt-reviews.html Is that a positive or negative review for Chobani? Even based on the second sentence: If not, stop reading, gather your keys … and get to your local grocery store, it seems that this will be a positive review. She describes the flavor as fantastic and calls the yogurt delicious. It seems that I should buy it and check it out. I will be right back... 7- 3 An automatic system for determining positive and negative texts. John, that looks like a positive tweet for Gravity! Let's imagine an automatic system that can read some text and decide whether it is a positive or negative report about a product. Why would we want such a system? Suppose there is a company that sells health monitors, they might want to know about what people are saying about their products. Are what people say mostly positive or negative? They release an ad campaign for a new product. Are people favorable about the product (Man, I sooo want this!) or negative (looks like crap). Apple has a press conference to talk about the iPhone problems. Is the resulting press coverage positive? A Senate candidate delivers a major policy speech— do the political bloggers view it favorably? So an automatic system does sound useful. So how can I create an automatic text classification system? 7-4 NAIVE BAYES AND TEXT Let's say I want to create a system that can tell whether a person likes or dislikes various food products. We might come up with an idea of having a list of words that would provide evidence that a person likes the product and another list of words that provides evidence that the person doesn't like the product. ‘Like’ words: delicious tasty good love smooth ‘Dislike’ words: awful blan d bad hate gritty If we are trying to determine if a particular reviewer likes Chobani yogurt or not, we can just count the number of ‘like’ words and the number of ‘dislike’ words in their text. We will classify the text based on which number is higher. We can do this for other classification tasks. For example, if we want to decide whether someone is pro-choice or pro-life, we can base it on the words and phrases they use. If they use the phrase 'unborn child' then chances are they are pro-life; if they use fetus they are more likely to be pro-choice. It's not surprising that we can use the occurrence of words to classify text. Rather than just using raw counts to classify text, let’s use the naïve Bayes!! hMAP = arg max h∈H P(D | h)P(h) Let’s dissect that formula! 7- 5 I am going to go through all the hypotheses and pick the one with the maximum probability The probability of that hypotheses hMAP = arg max h∈H P(D | h)P(h) For each hypothesis, h, in the set of hypotheses, H... The probability of the data given the hypothesis ( for example, the probability of seeing specific words in the text given the text We will use the naïve Bayes methods that were introduced in the previous chapter. We start with a training data set and, since we are now interested in unstructured text this data set is called the training corpus. Each entry in the corpus we will call a document even if it is a 140 character Twitter post. Each document is labeled with its class. So, for example, we might have a corpus of Twitter posts that rated movies. Each post is labeled in some way as a ‘favorable’ review or ‘unfavorable’ and we are going to train our classifier using this corpus of labeled documents. The P(h) in the formula above is the probability of these labels. If we have 1,000 documents in our training corpus and 500 of them are favorable reviews and 500 unfavorable then P( favorable) = 0.5 7-6 P(unfavorable)= 0.5 NAIVE BAYES AND TEXT When we start with labeled training data it is called ‘supervised learning.’ Text classification is an example of supervised learning. Learning from unlabeled text is called unsupervised learning. One example of unsupervised learning is clustering which we will cover in the next chapter. There is also semi-supervised learning where the system learns from both labeled and unlabeled data. Often the system is bootstrapped using labeled data and then in subsequent learning makes use of unlabeled data. Okay, back to hMAP = arg max h∈H P(D | h)P(h) Now let's examine the P(D|h) part of the formula—the probability of seeing some evidence, some data D given the hypothesis h. The data D we are going to use is the words in the text. One approach would be to start with the first sentence of a document, for example, Puts the Thrill back in Thriller. And compute things like the probability that a 'like' document starts with the word Puts; what's the probability of a 'like' document having a second word of the; and the probability of the third word of a like document being Thrill and so on. And then compute the probability of a dislike document starting with the word Puts, the probability of the second word of a dislike document being the and so on. 7- 7 Google estimates that there are about 1 million words in the English language. If a Twitter message has about 14 words, we would need to compute... 1,000,000 x 1,000,000 x 1,000,000 x , 1,000,000 x 1,000,000 x 1,000,000 x ,1,000,000 x 1,000,000 x 1,000,000 x 1,000,000 x 1,000,000 x 1,000,000 x 1,000,000 x 1,000,000 probabilities That’s a huge number of probabilities to compute! There must be a better approach! Hmm. yeah. That is a huge number of probabilities which makes this approach unworkable. And, fortunately, there is a better approach. We are going to simplify things a bit by treating the documents as bags of unordered words. Instead of asking things like What's the probability that the third word is thrill given it is a 'like' document we will ask What's the probability that the word thrill occurs in a 'like' document. Here is how we are going to compute those probabilities. Training Phase First, we are going to determine the vocabulary—the unique words—of all the documents (both like and dislike documents). So, for example, even though the may occur thousands of times in our training corpus it only occurs once in our vocabulary. Let Vocabulary denote the number of words in the vocabulary. Next, for each word wk in the vocabulary we are going to compute the probability of that word occurring given each hypothesis: P(wk |hi). 7-8 NAIVE BAYES AND TEXT We are going to compute this as follows. For each hypothesis (in this case 'like' and dislike') 1. combine the documents tagged with that hypothesis into one text file. 2. count how many word occurrences there are in the file. This time, if there are 500 occurrences of the we are going to count it 500 times. Let’s call this n. 3. For each word in the vocabulary wk, count how many times that word occurred in the text. Call this nk 4. For each word in the vocabulary wk, compute P(wk | hi ) = nk +1 n + Vocabulary Naïve Bayes Classification Phase Once we have completed the training phase we can classify documents using the formula that was already presented: hMAP = arg max h∈H P(D | h)P(h) That seems simple enough. Let’s work through an example! 7- 9 Let’s say our training corpus consisted of 500 Twitter messages with positive reviews of movies and 500 negative. So P(dislike) = 0.5 P(like)= 0.5 After training the probabilities are as follows: word P(word|like) P(word|dislike) am 0.007 0.009 by 0.012 0.012 good 0.002 0.0005 gravity 0.00001 0.00001 great 0.003 0.0007 hype 0.0007 0.002 I 0.01 0.01 over 0.005 0.0047 stunned 0.0009 0.002 the 0.047 0.0465 How should we classify: I am stunned by the hype over gravity We are going to compute P(like) × P(I | like) × P(am | like) × P(stunned | like) × ... and P(dislike) × P(I | dislike) × P(am | dislike) × P(stunned | dislike) × ... and chose the hypothesis associated with the highest probability. 7-10 NAIVE BAYES AND TEXT word P(word|like) P(word|dislike) P(like) = 0.5 P(dislike) =0.05 I 0.01 0.01 am 0.007 0.009 stunned 0.0009 0.002 by 0.012 0.012 the 0.047 0.0465 hype 0.0007 0.002 over 0.005 0.0047 gravity 0.00001 0.00001 ∏ 6.22E-22 4.72E-21 So the probabilities are Just a reminder: like 0.000000000000000000000622 dislike 0.000000000000000000004720 The probability of dislike is larger than that for like so we classify the tweet as a dislike. That e notation means how many places to move the decimal point. If the number is positive we move the decimal to the right, negative means move it to the left. So 1.23e-1 = 0.123 1.23e-2 = 0.0123 1.23e-3 = 0.00123 and so on 7- 11 Yes. If we multiply the word probabilities for even a short document of 100 words we are going to get a very, very, very small number. wow. those are pretty small numbers! Right. But Python can’t handle very small numbers. They’ll just end up being zero. Exactly. We can fix this using logs. Instead of multiplying the probabilities we will add the logs of the probabilities!! Here’s an illustration of the problem. Let’s say we have a 100 word document and the average probability of each word is 0.001 (words like tell, reported, average, morning, and am have a probability of around 0.001). If I multiply those probabilities in Python we get zero: >>> 0.0001**100 0.0 However, if we add the log of the probabilities we do get a non-zero value: >>> import math >>> p = 0 >>> for i in range(100): ! p += math.log(0.0001) >>> p -921.034037197617 7-12 NAIVE BAYES AND TEXT in case you forgot ... bn = x The logorithm (or log) of a number (the x above) is the exponent (the n above) that you need to raise a base (b) to equal that number. For example, suppose the base is 10, log10(1000) = 3 since 1000 equals 103 The base of the Python log function is the mathematical constant e. We don’t really need to know about e. What is of interest to us is: 1. logs compress the scale of a number ( with logs we can represent smaller numbers in Python) for ex., .0000001 x .000005 = .000000000005 the logs of those numbers are: -16.11809 + -9.90348 = -26.02157 2. instead of multiplying the probabilities we are going to add the logs of the probabilities (as shown above). Newsgroup Corpus We will first investigate how this algorithm works by using a standard reference corpus of usenet newsgroup posts. The data consists of posts from 20 different newsgroups: comp.graphics misc.forsale soc.religion.christian alt.atheism comp.os.ms-windows-misc rec.autos talk.politics.guns sci.space comp.sys.ibm.pc.hardware rec.motorcycles talk.politics.mideast sci.crypt comp.sys.mac.hardware rec.sport.baseball talk.politics.misc sci.electronics comp.windows.x rec.sport.hockey talk.religion.misc sci.med 7- 13 We would like to build a classifier that can correctly determine what group the post came from. For example, we would like to classify this post From: essbaum@rchlan d.vnet.ibm.com (Alexander Essbaum) Subject: Re: Mail order response time Disclaimer: This posti ng represents the poste r's views, not necessarily those of IBM Nntp-Posting-Host: rel va.rchland.ibm.com Organization: IBM Ro chester Lines: 18 > I have ordered many times from Competitio n > accesories and ussua lly get 2-3 day delivery. ordered 2 fork seals an d2 CA for my FZR. two we guide bushings from eks and 1 guide bushing. cal later get 2 fork seals l CA and ask for remaining *guide* bushi ng bushings (explain on the and order 2 *slide* phone which bushings are which; the guy see med to understand). tw o weeks later get 2 guide bushings. as being from rec.motorcycles Notice the misspellings (accesories and ussually). This might be challenging for a classifier! *sigh* how much you wanna bet that once i get ALL the parts and take the fork apart that some parts won't fit? The data is available at http://qwone.com/~jason/20Newsgroups/ (we are using the 20news=bydate dataset) . It is also available on the website for the book, http:// guidetodatamining.com. The data consists of 18,846 documents and is already sorted into training (60% of the data) and test sets. The training and test data are in separate directories. Within each directory are subdirectories representing each newsgroup. Within those are the separate documents representing posts to that newsgroup. 7-14 NAIVE BAYES AND TEXT Throwing things out! Before we start coding, let’s think about this task in more detail. ain lemen. On the m Ladies and Gent ds in the sed on the wor stage ... Just ba to tell ing to attempt text, we are go fr the post is om which newsgroup For example, we would like to build a system that would classify the following post as being from rec.motorcycle: I am looking at buying a Dual Sport type motorcycle. This is my first cycle as well. I am interested in any experiences people have with the following motorcycles, good or bad. Honda XR250L Suzuki DR350S Suzuki DR250ES Yamaha XT350 Most XXX vs. YYY articles I have seen in magazines pit the Honda XR650L against another cycle, and the 650 always comes out shining. Is it safe to assume that the 250 would be of equal quality ? 7- 15 Let’s consider which words might be helpful in the classification task: I... “I’ is not helpful am... not helpful looking... not helpful at... not helpful buying... erm. probably helpful a .... not helpful dual... definitely helpful sport ... definitely type... probably not motorcycle definitely!!!! If we throw out the 200 most frequent words in English our document looks like this: I am looking at buying a Dual Sport type motorcycle. This is my first cycle as well. I am interested in any experiences people have with the following motorcycles, good or bad. Honda XR250L Suzuki DR350S Suzuki DR250ES Yamaha XT350 Most XXX vs. YYY articles I have seen in magazines pit the Honda XR650L against another cycle, and the 650 always comes out shining. Is it safe to assume that the 250 would be of equal quality ? 7-16 NAIVE BAYES AND TEXT Removing these words cuts down the size of our text by about half. Plus, it doesn't look like removing these words will have any impact on our ability to categorize texts. Indeed data miners have called such words words without any content, and fluff words. H.P. Luhn, in his seminal paper 'The automatic creation of literature abstracts' says of these words that they are “too common to have the type of significance being sought and would constitute 'noise' in the system.” That noise argument is interesting as it implies that removing these words will improve performance. These words that we remove are called 'stop words'. We have a list of such words, the 'stop word list', and remove these words from the text in a preprocessing step. We remove these words because 1) it cuts down on the amount of processing we need to do and 2) it does not negatively impact the performance of our system—as the noise argument suggests removing them might improve performance. Common Words vs. Stop Words classification task, While it is true that common words like ‘the’ and ‘a’ may not help us in our on our classification ing depend other common words such as ‘work’, ‘write’, and ‘school’ may help may be helpful. You can task. When we create a stop word list, we often omit common words that lists found on the web. explore these differences by comparing stop word lists and frequent word The counter argument: the hazards of stop word removal You whippersnapper. You shouldn’t be throwing away those common words! While removing stop words may be useful in some situations, you should not just automatically remove them without thinking. For example, it turns out just using the most frequent words and throwing out the rest (the reverse technique of the above) provides 7- 17 sufficient information to identify where Arabic documents were written. (If you are curious about this check out the paper Linguistic Dumpster Diving: Geographical Classification of Arabic Text I co-wrote with some of my colleagues at New Mexico State University. It is available on my website http://zacharski.org). In looking at online chats, sexual predators use words like I, me, and you, much more frequently than non-predators. If your task is to identify sexual predators, removing frequent words would actually hurt your performance. top words. remove s Don’t blindly Think First. Coding it — Python Style Let us first consider coding the training part of the Naïve Bayes Classifier. Recall that the training data is organized as follows: 20news-bydate-train ! alt.atheism ! ! text file ! ! text file ! ! … ! ! text file ! comp.graphics ! ! text file ! ! ... 7-18 1 for alt.atheism 2 n 1 for comp.graphics NAIVE BAYES AND TEXT So I have a directory (in this example called ‘20news-bydate-train’). Underneath this directory are subdirectories representing different classification categories (in this case alt.atheism, comp.graphics, etc). The names of these subdirectories match the category names. The test directory is organized in a similar way. So, in matching this structure, the Python code for training will need to know the training directory (for example, /Users/raz/Downloads/20news-bydate/20news-bydate-train/). The outline for the training code is as follows. class BayesText 1. the init method: a. read in the words from the stoplist b. read the training directory to get the names of the subdirectories (in addition to being the names of the subdirectories, these are the names of the categories). c. For each of those subdirectories, call a method “train” that will count the occurrences of words in all the files of that subdirectory. d. compute the probabilities using P(wk | hi ) = nk +1 n + Vocabulary Yet another reminder that all the code is available at guidetodatamining.com 7- 19 from __future__ import print_function import os, codecs, math class BayesText: def __init__(self, trainingdir, stopwordlist): """This class implements a naive Bayes approach to text classification trainingdir is the training data. Each subdirectory of trainingdir is titled with the name of the classification category -- those subdirectories in turn contain the text files for that category. The stopwordlist is a list of words (one per line) will be removed before any counting takes place. """ self.vocabulary = {} self.prob = {} self.totals = {} self.stopwords = {} f = open(stopwordlist) for line in f: self.stopwords[line.strip()] = 1 f.close() categories = os.listdir(trainingdir) #filter out files that are not directories self.categories = [filename for filename in categories if os.path.isdir(trainingdir + filename)] print("Counting ...") for category in self.categories: print(' ' + category) (self.prob[category], self.totals[category]) = self.train(trainingdir, category) # I am going to eliminate any word in the vocabulary # that doesn't occur at least 3 times toDelete = [] for word in self.vocabulary: if self.vocabulary[word] < 3: # mark word for deletion # can't delete now because you can't delete # from a list you are currently iterating over toDelete.append(word) 7-20 NAIVE BAYES AND TEXT # now delete for word in toDelete: del self.vocabulary[word] # now compute probabilities vocabLength = len(self.vocabulary) print("Computing probabilities:") for category in self.categories: print(' ' + category) denominator = self.totals[category] + vocabLength for word in self.vocabulary: if word in self.prob[category]: count = self.prob[category][word] else: count = 1 self.prob[category][word] = (float(count + 1) / denominator) print ("DONE TRAINING\n\n") def train(self, trainingdir, category): """counts word occurrences for a particular category""" currentdir = trainingdir + category files = os.listdir(currentdir) counts = {} total = 0 for file in files: #print(currentdir + '/' + file) f = codecs.open(currentdir + '/' + file, 'r', 'iso8859-1') for line in f: tokens = line.split() for token in tokens: # get rid of punctuation and lowercase token token = token.strip('\'".,?:-') token = token.lower() if token != '' and not token in self.stopwords: self.vocabulary.setdefault(token, 0) self.vocabulary[token] += 1 counts.setdefault(token, 0) counts[token] += 1 total += 1 f.close() return(counts, total) 7- 21 The results of the training phase are stored in a dictionary (hash table) called prob. The keys of the dictionary are the different classifications (comp.graphics, rec.motorcycles, soc.religion.christian, etc); the values are dictionaries. The keys of these subdictionaries are the words and the values are the probabilities of those words. Here is an example: bT = BayesText(trainingDir, stoplistfile) >>>bT.prob["rec.motorcycles"]["god"] 0.00013035445075435553 >>>bT.prob["soc.religion.christian"]["god"] 0.004258192391884386 >>>bT.prob["rec.motorcycles"]["the"] 0.028422937849264914 >>>bT.prob["soc.religion.christian"]["the"] 0.039953678998362795 So, for example, the probability of the word ‘god’ occurring in a text in the rec.motorcycles newsgroup is 0.00013 (or one occurrence of god in every 10,000 words). The probability of the word ‘god’ occurring in a text in soc.religion.christian is .00424 (one occurrence in every 250 words). Training also results in a list called categories, which, as you might predict, is simply a list of all the categories: ['alt.atheism', 'comp.graphics', 'comp.os.ms-windows.misc', 'comp.sys.ibm.pc.hardware', ...] So that is the training phase. Let us now turn to classifying a document. 7-22 NAIVE BAYES AND TEXT s code it Can you code a method called classify that will predict the classification of a document? For example: >>> bT.classify("20news-bydate-test/rec.motorcycles/104673") 'rec.motorcycles' >>> bT.classify("20news-bydate-test/sci.med/59246") 'sci.med' >>> bT.classify("20news-bydate-test/soc.religion.christian/21424") 'soc.religion.christian' As you can see, the classify method takes a filename as an argument and returns a string denoting the classification. A Python file you can use as a template, bayesText-ClassifyTemplate.py, is available on our website. class BayesText: def __init__(self, trainingdir, stopwordlist): self.vocabulary = {} self.prob = {} self.totals = {} self.stopwords = {} f = open(stopwordlist) for line in f: self.stopwords[line.strip()] = 1 f.close() categories = os.listdir(trainingdir) #filter out files that are not directories self.categories = [filename for filename in categories if os.path.isdir(trainingdir + filename)] print("Counting ...") for category in self.categories: print(' ' + category) (self.prob[category], self.totals[category]) = self.train(trainingdir, category) # I am going to eliminate any word in the vocabulary 7- 23 s code it - one possible solution def classify(self, filename): results = {} for category in self.categories: results[category] = 0 f = codecs.open(filename, 'r', 'iso8859-1') for line in f: tokens = line.split() for token in tokens: token = token.strip('\'".,?:-').lower() if token in self.vocabulary: for category in self.categories: if self.prob[category][token] == 0: print("%s %s" % (category, token)) results[category] += math.log( self.prob[category][token]) f.close() results = list(results.items()) results.sort(key=lambda tuple: tuple[1], reverse = True) # for debugging I can change this to give me the entire list return results[0][0] Finally, let’s have a method that classifies every document in the test directory and prints out the percent accuracy of this method. 7-24 NAIVE BAYES AND TEXT def testCategory(self, directory, category): files = os.listdir(directory) total = 0 correct = 0 for file in files: total += 1 result = self.classify(directory + file) if result == category: correct += 1 return (correct, total) def test(self, testdir): """Test all files in the test directory--that directory is organized into subdirectories--each subdir is a classification category""" categories = os.listdir(testdir) #filter out files that are not directories categories = [filename for filename in categories if os.path.isdir(testdir + filename)] correct = 0 total = 0 for category in categories: (catCorrect, catTotal) = self.testCategory( testdir + correct += catCorrect total += catTotal print("Accuracy is %f%% (%i test instances)" % ((float(correct) / total) * 100, total)) When I run this code using an empty stoplist file I get: DONE TRAINING Running Test ... .................... Accuracy is 77.774827% (7532 test instances) 7- 25 77.77% accuracy is pretty good... I wonder what the accuracy would be if we used a stoplist? Only one way to find out ... s code it Can you run the classifier with a few stop word lists? Does performance improve? Which is most accurate? (You will need to search the web to find these lists) stop list size accuracy 0 77.774827 list 1 list 2 7-26 NAIVE BAYES AND TEXT s code it - some results I found a 25 word stop word list at: http://nlp.stanford.edu/IR-book/html/htmledition/droppingcommon-terms-stop-words-1.html And a 174 word one at http://www.ranks.nl/resources/stopwords.html (these word lists are available on our website) Here are the results: stop list size accuracy 0 77.774827% 25 word list 78.757302% 174 word list 79.938927% So in this case, it looks like having a 174 word stop word list improved performance about 2% over having no stop word list? Does this match your results? 7- 27 Naïve Bayes and Sentiment Analysis The goal of sentiment analysis is to determine the writer’s attitude (or opinion). Katy Perry is awesome! Katy Perry? Bland uninspired pop. Lorde is awesome! Okay, I agree. Lorde IS awesome! One common type of sentiment analysis is to determine the polarity of a review or comment (positive or negative) and we can use a Naïve Bayes Classifier for this task. We can try this out by using the polarity movie review dataset first presented in Pang and Lee 2004 1. Their dataset consists of 1,000 positive and 1,000 negative reviews. Here are some examples: r thriller of the month the second serial-kille rts deceptively okay , is just awful . oh , it sta uing characters and with a handful of intrig rk . ... some solid location wo when i first heard that romeo & juliet had been " updated " i shuddered . i thought that yet another of shakespeare' s classics had been destroyed . fortunately , i was wron g. directed an " in your fac baz luhrman has e " , and visually Pang, Bo and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. Proceedings of ACL. 1 7-28 NAIVE BAYES AND TEXT You can download the original dataset from http://www.cs.cornell.edu/People/pabo/moviereview-data/. I have organized the data into 10 buckets (folds) with the following directory structure: review_polarity_buckets ! txt_sentoken ! ! neg ! ! ! 0 ! ! ! ! files in fold ! ! ! 1 ! ! ! ! files in fold ! ! ! ... ! ! ! 9! ! ! ! ! files in fold ! ! pos ! ! ! 0 ! ! ! ! files in fold ! ! ! ... 0 1 9 0 This re-organized dataset is available on our website. s code it Can you modify the Naive Bayes Classifier code to do 10-fold cross validation of the classifier on this data set. The output should look something like: Classified as: neg pos +-----+-----+ neg | 1 | 2 | pos | 3 | 4 | +-----+-----+ 12.345 percent correct total of 2000 instances Also compute the kappa coefficient. 7- 29 Obvious Disclaimer You won’t become proficient in data m ining by reading this book an ymore than reading a book about piano playing will make you profic ient at piano playing. You need to practice! Brahms Practice makes the heart grow fonder! Woman practicing Woman practicing Naïve Bayes 7-30 NAIVE BAYES AND TEXT s code it — my results Here are the results I got: neg pos Classified as: neg pos +-----+-----+ | 845 | 155 | | 222 | 778 | +-----+-----+ 81.150 percent correct total of 2000 instances Also compute the kappa coefficient. κ= P(c) − P(r) .8115 − 0.5 .3115 = = = 0.623 1− P(r) 1− 0.5 .5 So we have good performance of the algorithm on this data. My code is on the following page! er: Yet another remind e book’s for download on th le The code is availab / todatamining.com website http://guide 7- 31 from __future__ import print_function import os, codecs, math class BayesText: def __init__(self, trainingdir, stopwordlist, ignoreBucket): """This class implements a naive Bayes approach to text classification trainingdir is the training data. Each subdirectory of trainingdir is titled with the name of the classification category -- those subdirectories in turn contain the text files for that category. The stopwordlist is a list of words (one per line) will be removed before any counting takes place. """ self.vocabulary = {} self.prob = {} self.totals = {} self.stopwords = {} f = open(stopwordlist) for line in f: self.stopwords[line.strip()] = 1 f.close() categories = os.listdir(trainingdir) #filter out files that are not directories self.categories = [filename for filename in categories if os.path.isdir(trainingdir + filename)] print("Counting ...") for category in self.categories: #print(' ' + category) (self.prob[category], self.totals[category]) = self.train(trainingdir, category, ignoreBucket) # I am going to eliminate any word in the vocabulary # that doesn't occur at least 3 times toDelete = [] 7-32 NAIVE BAYES AND TEXT for word in self.vocabulary: if self.vocabulary[word] < 3: # mark word for deletion # can't delete now because you can't delete # from a list you are currently iterating over toDelete.append(word) # now delete for word in toDelete: del self.vocabulary[word] # now compute probabilities vocabLength = len(self.vocabulary) #print("Computing probabilities:") for category in self.categories: #print(' ' + category) denominator = self.totals[category] + vocabLength for word in self.vocabulary: if word in self.prob[category]: count = self.prob[category][word] else: count = 1 self.prob[category][word] = (float(count + 1) / denominator) #print ("DONE TRAINING\n\n") def train(self, trainingdir, category, bucketNumberToIgnore): """counts word occurrences for a particular category""" ignore = "%i" % bucketNumberToIgnore currentdir = trainingdir + category directories = os.listdir(currentdir) counts = {} total = 0 for directory in directories: if directory != ignore: currentBucket = trainingdir + category + "/" + \ directory files = os.listdir(currentBucket) #print(" " + currentBucket) for file in files: 7- 33 f = codecs.open(currentBucket + '/' + file, 'r', 'iso8859-1') for line in f: tokens = line.split() for token in tokens: # get rid of punctuation # and lowercase token token = token.strip('\'".,?:-') token = token.lower() if token != '' and not token in \ self.stopwords: self.vocabulary.setdefault(token, 0) self.vocabulary[token] += 1 counts.setdefault(token, 0) counts[token] += 1 total += 1 f.close() return(counts, total) def classify(self, filename): results = {} for category in self.categories: results[category] = 0 f = codecs.open(filename, 'r', 'iso8859-1') for line in f: tokens = line.split() for token in tokens: #print(token) token = token.strip('\'".,?:-').lower() if token in self.vocabulary: for category in self.categories: if self.prob[category][token] == 0: print("%s %s" % (category, token)) results[category] += math.log( self.prob[category][token]) f.close() results = list(results.items()) results.sort(key=lambda tuple: tuple[1], reverse = True) 7-34 NAIVE BAYES AND TEXT # for debugging I can change this to give me the entire list return results[0][0] def testCategory(self, direc, category, bucketNumber): results = {} directory = direc + ("%i/" % bucketNumber) #print("Testing " + directory) files = os.listdir(directory) total = 0 correct = 0 for file in files: total += 1 result = self.classify(directory + file) results.setdefault(result, 0) results[result] += 1 #if result == category: # correct += 1 return results def test(self, testdir, bucketNumber): """Test all files in the test directory--that directory is organized into subdirectories--each subdir is a classification category""" results = {} categories = os.listdir(testdir) #filter out files that are not directories categories = [filename for filename in categories if os.path.isdir(testdir + filename)] correct = 0 total = 0 for category in categories: #print(".", end="") results[category] = self.testCategory( testdir + category + '/', category, bucketNumber) return results def tenfold(dataPrefix, stoplist): results = {} for i in range(0,10): 7- 35 bT = BayesText(dataPrefix, stoplist, i) r = bT.test(theDir, i) for (key, value) in r.items(): results.setdefault(key, {}) for (ckey, cvalue) in value.items(): results[key].setdefault(ckey, 0) results[key][ckey] += cvalue categories = list(results.keys()) categories.sort() print( "\n Classified as: ") header = " " subheader = " +" for category in categories: header += "% 2s " % category subheader += "-----+" print (header) print (subheader) total = 0.0 correct = 0.0 for category in categories: row = " %s |" % category for c2 in categories: if c2 in results[category]: count = results[category][c2] else: count = 0 row += " %3i |" % count total += count if c2 == category: correct += count print(row) print(subheader) print("\n%5.3f percent correct" %((correct * 100) / total)) print("total of %i instances" % total) # change these to match your directory structure theDir = "/Users/raz/Downloads/review_polarity_buckets/txt_sentoken/" stoplistfile = "/Users/raz/Downloads/20news-bydate/stopwords25.txt" tenfold(theDir, stoplistfile) 7-36 NAIVE BAYES AND TEXT 7- 37 Chapter 8: Clustering Discovering Groups In previous chapters we have been developing classification systems. In these systems we train a classifier on a set of labeled examples. the label (class) we are learning to predict sport Height Weight basketball 72 162 gymnastics 54 66 track 63 106 basketball 78 204 plasma glucose diastolic BP BMI diabetes? 99 52 24.6 0 83 58 34.4 0 139 80 31.6 1 After we train the classifier, we can use it to label new examples. ! ! This person looks like a basketball player. That one a gymnast. That person is unlikely to get diabetes in 3 years. and so on. In other words, the classifier selects a label from a set of labels it acquired during the training phase—it knows the possible labels. But what happens if I don’t know the possible labels? Suppose I want a system that discovers the possible groups. For example, I have 1,000 people, each one represented by 20 attributes and I want a system to cluster the people into groups. This task is called clustering. The system divides a set of instances into clusters or groups based on some measure of similarity. There are two main types of clustering algorithms. k-means clustering For one type, we tell the algorithm how many clusters to make. Please cluster these 1,000 people into 5 groups. Please classify these web pages into 15 groups. These methods go by the name of k-means clustering algorithms and we will discuss those a bit later in the chapter. hierarchical clustering For the other approach we don’t specify how many clusters to make. Instead the algorithm starts with each instance in its own cluster. At each iteration of the algorithm it combines the two most similar clusters into one. It repeatedly does this until there is only one cluster. This 8-2 CLUSTERING is called hierarchical clustering and its name makes sense. The running of the algorithm results in one cluster, which consists of two sub-clusters. Each of those two sub-clusters in turn, consist of 2 sub-sub clusters and so on. Initially, each item is in its own cluster We join the 2 closest clusters into one cluster Then we repeat... We join the 2 closest clusters into one cluster Then we repeat... We join the 2 closest clusters into one cluster Then we repeat... We join the 2 closest clusters into one cluster We stop when there is only one cluster! 8-3 Again, at each iteration of the algorithm we join the two closest clusters. To determine the ‘closest clusters’ we use a distance formula. But we have some choices in how we compute the distance between two clusters, which leads to different clustering methods. Consider the three clusters (A, B, and C) illustrated below each containing two members. Which pair of clusters should we join? Cluster A with B, or cluster C with B? B2 A1 B1 C1 C2 A2 Single-linkage clustering In single-linkage clustering we define the distance between two clusters as the shortest distance between any member of one cluster to any member of the other. With this definition, the distance between Cluster A and Cluster B is the distance between A1 and B1, since that is shorter than the distances between A1 and B2, A2 and B1, and A2 and B2. With single-linkage clustering, Cluster A is closer to Cluster B than C is to B, so we would combine A and B into a new cluster. Complete-linkage clustering In complete-linkage clustering we define the distance between two clusters as the greatest distance between any member of one cluster to any member of the other. With this definition, the distance between Cluster A and Cluster B is the distance between A2 and B2. With complete-linkage clustering, Cluster C is closer to Cluster B than A is to B, so we would combine B and C into a new cluster. Average-linkage clustering In average-linkage clustering we define the distance between two clusters as the average distance between any member of one cluster to any member of the other. In the diagram above, it appears that the average distance between Clusters C and B would be less than the average between A and B and we would combine B and C into a new cluster. 8-4 CLUSTERING Hey! Let’s work through an example of single-linkage clustering! Good idea! Let’s practice by clustering dog breeds based on height and weight! height (inches) weight (pounds) Border Collie 20 45 Boston Terrier 16 20 Brittany Spaniel 18 35 Bullmastiff 27 120 Chihuahua 8 8 German Shepherd 25 78 Golden Retriever 23 70 Great Dane 32 160 Portuguese Water Dog 21 50 Standard Poodle 19 65 Yorkshire Terrier 6 7 breed ng. Psst! I think we are forgetting somethi re Isn’t there something we should do befo computing distance? 8-5 d Normalization! Let’s change those numbers to Modified Standard Scores Modified Standard Scores height weight 0 -0.1455 Boston Terrier -0.7213 -0.873 Brittany Spaniel -0.3607 -0.4365 Bullmastiff 1.2623 2.03704 Chihuahua -2.1639 -1.2222 German Shepherd 0.9016 0.81481 Golden Retriever 0.541 0.58201 Great Dane 2.16393 3.20106 Portuguese Water Dog 0.1803 0 Standard Poodle -0.1803 0.43651 Yorkshire Terrier -2.525 -1.25132 breed Border Collie 8-6 Next we are going to compute the Euclidean distance between breeds! CLUSTERING Euclidean Distances (a few of the shortest distances are highlighted): Border Collie BT BS B C GS GR GD PWD SP YT 1.024 0.463 2.521 2.417 1.317 0.907 3.985 0.232 0.609 2.756 0.566 3.522 1.484 2.342 1.926 4.992 1.255 1.417 1.843 2.959 1.967 1.777 1.360 4.428 0.695 0.891 2.312 4.729 1.274 1.624 1.472 2.307 2.155 5.015 3.681 3.251 6.188 2.644 2.586 0.362 0.429 2.700 1.088 1.146 4.001 3.081 0.685 0.736 3,572 3.766 3.625 6.466 0.566 2.980 Boston Terrier Brittany Spaniel Bullmastiff Chihuahua German Shphrd Golden Retriever Great Dane Portuguese WD 2.889 Standard Poodle 4.0 Great Dane Based on this chart, which two breeds do you think are the closest? 3.0 Bullmastiff 2.0 1.0 0 Border Collie Boston Terrier -1.0 Yorkshire -2.0 -3.00 weight German Shepherd St. Poodle -2.25 Golden Retriever Portuguese WD Brittany Spaniel Chihuahua -1.50 -0.75 0 0.75 1.50 2.25 3.00 height 8-7 If you said Border Collie and Portuguese Water Dog, you would be correct! The algorithm. Step 1. Initially, each breed is in its own cluster. We find the two closest clusters and combine them into one cluster. From the table on the preceding page we see that the closest clusters are the Border Collie and the Portuguese Water Dog (distance of 0.232) so we combine them. Border Collie Portuguese WD Step 2. We find the two closest clusters and combine them into one cluster. From the table on the preceding page we see that these are the Chihuahua and the Yorkshire Terrier (distance of 0.362) so we combine them. Chihuahua Yorkshire T. Border Collie Portuguese WD Step 3. We repeat the process again. This time combining the German Shepherd and the Golden Retriever. Chihuahua Yorkshire T. German Shphrd Golden Retriever Border Collie Portuguese WD 8-8 CLUSTERING Step 4. We repeat the process yet again. From the table we see that the next closest pair is the Border Collie and the Brittany Spaniel. The Border Collie is already in a cluster with the Portuguese Water Dog which we created in Step 1. So in this step we are going to combine that cluster with the Brittany Spaniel. Chihuahua Yorkshire T. German Shphrd Golden Retriever Border Collie Portuguese WD called a diagram is f o e p ty This ally a tree It is basic . m a r g o r clusters. dend epresents r t a th m diagra Brittany Spaniel And we continue: Chihuahua Yorkshire T. German Shphrd Golden Retriever Border Collie Portuguese WD Brittany Spaniel Boston Terrier 8-9 s sharpen your pencil Finish the clustering of the dog data! To help you in this task, there is a sorted list of dog breed distances on this chapter’s webpage (https:// raw.githubusercontent.com/zacharski/pg2dm-python/0684ec677a1a1baaecb47bc0f8f21ec121e83339/ data/ch8/dogDistanceSorted.txt). Chihuahua Yorkshire T. German Shphrd Golden Retriever Border Collie Portuguese WD Brittany Spaniel Boston Terrier 8-10 CLUSTERING s sharpen your pencil solution Finish the clustering of the dog data! To help you in this task, there is a sorted list of dog breed distances on this chapter’s webpage (http://guidetodatamining.com/guide/ch8/dogDistanceSorted.txt). Chihuahua Yorkshire T. German Shphrd Golden Retriever Border Collie Portuguese WD Brittany Spaniel Boston Terrier Standard Poodle Bullmastiff Great Dane 8-11 coding a hierarchical clustering algorithm For coding the clusterer we can use a priority queue! Can you remind me what a priority queue is? Sure!! In a regular queue, the order in which you put the items in the queue is the order you get the items out of the queue... Suppose I put tuples representing a person’s age and name into a queue. First the tuple for Moa is put into the queue, then the one for Suzuka and then for Yui. When I get an item from the queue, I first get the tuple for Moa since that was the first one put in the queue; then the one for Suzuka and then Yui! 3rd (13, Yui) 2nd (16, Suzuka) 1st (15, Moa) Queue (13, Yui) 3rd 8-12 (16, Suzuka) 2nd (15, Moa) 1st CLUSTERING In a priority queue each item put into the queue has an associated priority. The order in which items are retrieved from the queue is based on this priority. Items with a higher priority are retrieved before items with a lower one. In our example data, suppose the younger a person is, the higher their priority. We put the tuples into the queue in the as before! (13, Yui) (16, Suzuka) same order Priority Queue (15, Moa) ) (15, Moa (16, Suzuka) (13, Yui) queue will be The first item to be retrieved from the has the highest Yui because she is youngest and thus priority! Priority Queue ) (16, Suzuka) (15, Moa (16, Suzuka) (15, Moa) 3rd 2nd (13, Yui) 1st (13, Yui) Let’s see how this works in Python >>> from queue import PriorityQueue # load the PriorityQueue library >>> singersQueue = PriorityQueue() # create a PriorityQueue called ! # singersQueue ! ! ! ! >>> singersQueue.put((16, 'Suzuka Nakamoto')) # put a few items in the queue >>> singersQueue.put((15, 'Moa Kikuchi')) >>> singersQueue.put((14, 'Yui Mizuno')) 8-13 >>> singersQueue.put((17, 'Ayaka Sasaki')) >>> singersQueue.get() # The first item retrieved (14, 'Yui Mizuno') # will be the youngest, Yui. >>> singersQueue.get() (15, 'Moa Kikuchi') >>> singersQueue.get() (16, 'Suzuka Nakamoto') >>> singersQueue.get() (17, 'Ayaka Sasaki') For our task of building a hierarchical clusterer, we will put the clusters in a priority queue. The priority will be the shortest distance to a cluster’s nearest neighbor. Using our dog breed example, we will put the Border Collie in our queue recording that it’s nearest neighbor is the Portuguese Water Dog at a distance of 0.232. We put similar entries into the queue for the other breeds: Priority Queue cluster: (Border Collie) neighbor: Portuguese Wa ter Dog distance: 0.232 cluster: (Chihuahua) neighbor: Yorkshire Terrier distance: 0.362 cluster: (Portuguese Wa ter Dog) neighbor: Border Collie distance: 0.232 etc. etc. etc. We will get the two entries with the shortest distance, making sure we have a matching pair. In this case we get the entries for Border Collie and Portuguese Water Dog. Next, we join the clusters into one cluster. In this case, we create a Border Collie - Portuguese Water Dog cluster. And put that cluster on the queue: 8-14 CLUSTERING Priority Queue cluster: (Border Collie, Portuguese Water Dog) neighbor: Brittany Spaniel distance: 0.463 cluster: (Chihuahua) neighbor: Yorkshire Terrier distance: 0.362 etc. etc. etc. And repeat until there is only one cluster on the queue. The entries we will put on the queue need to be slightly more complex than those used in this example. So let’s look at this example in more detail. Reading the data from a file The data will be in a CSV (comma separated values) file where the first column is the name of the instance and the rest of the columns are the values of various attributes. The first line of the file will be a header that describes these attributes: ounds) ,weight (p ht (inches) ig he d, ee br ie,20,45 Border Coll r,16,20 ie rr Te Boston 5 aniel,18,3 Brittany Sp 27,120 f, Bullmastif 8 8, Chihuahua, herd,25,78 German Shep ver,23,70 ie tr Golden Re ,32,160 ne Da t ea Gr 21,50 Water Dog, Portuguese 65 9, ,1 odle Standard Po rrier,6,7 Te Yorkshire The data in this file is read into a list called, not surprisingly, data. The list data saves the information by column. Thus, data[0] is a list containing the breed names (data[0][0] is the string ‘Border Collie, data[0][1] is ‘Boston Terrier’ and so on). data[1] is a list 8-15 containing the height values, and data[2] is the weight list. All the data except that in the first column is converted into floats. For example, data[1][0] is the float 20.0 and data[2][0] is the float 45. Once the data is read in, it is normalized. Throughout the description of the algorithm I will use the term index to refer to the row number of the instance (for example, Border Collie is index 0, Boston Terrier is index 1, and Yorkshire Terrier is index 10). Initializing the Priority Queue At the start of the algorithm, we will put in the queue, entries for each breed. Let’s consider the entry for the Border Collie. First, we calculate the distance of the Border Collie to all other breeds and put that information into a Python dictionary: {1: ((0, 1), 1.0244), the distance between the Border Collie (index 0) and the Boston Terrier (index 1), is 1.0244 2: ((0, 2), 0.463), the distance between the Border Collie the Brittany Spaniel is 0.463 ... 10: ((0, 10), 2.756)} the Border Collie -- Yorkshire Terrier distance is 2.756 We will also keep track of the Border Collie’s nearest neighbor and the distance to that nearest neighbor: closest distance: 0.232 nearest pair: (0, 8) to the Border Collie The closest neighbor ese Water Dog (index 0) is the Portugu sa. (index 8) and vice ver The problem of identical distances and what is with all those tuples. You may have noticed that in the table on page 8-7, the distance between the Portuguese Water Dog and the Standard Poodle and the distance between the Boston Terrier and the Brittany Spaniel are the same—0.566. If we retrieve items from the priority queue based on distance there is a possibility that we will retrieve Standard Poodle and Boston Terrier and join them in a cluster, which would be an error. To prevent this error we will use a tuple containing the indices (based on the data list) of the two breeds that the distance represents. For example, Portuguese Water Dog is entry 8 in our data and the Standard 8-16 CLUSTERING Poodle is entry 9, so the tuple will be (8,9). This tuple is added to the nearest neighbor list. The nearest neighbor for the poodle will be: ['Portuguese Water Dog', 0.566, (8,9)] and the nearest neighbor for the Portuguese Water Dog will be: ['Standard Poodle', 0.566, (8,9)] By using this tuple, when we retrieve items from the queue we can see if they are a matching pair. Another thing to consider about identical distances. When I introduced Python Priority Queues a few pages ago, I inserted into the queue, tuples representing the ages and names of Japanese Idol performers. These entries were retrieved based on age. What happens if some of the entries have the same age (the same priority)? Let’s try: >>> singersQueue.put((15,'Suzuka Nakamoto')) >>> singersQueue.put((15,'Moa Kikuchi')) >>> singersQueue.put((15, 'Yui Mizuno')) >>> singersQueue.put((15, 'Avaka Sasaki')) >>> singersQueue.put((12, 'Megumi Okada')) >>> singersQueue.get() (12, 'Megumi Okada') >>> singersQueue.get() (15, 'Avaka Sasaki') >>> singersQueue.get() (15, 'Moa Kikuchi') >>> singersQueue.get() (15, 'Suzuka Nakamoto') >>> singersQueue.get() (15, 'Yui Mizuno') >>> You can see that if the first items in the tuples match, Python uses the next item to break the tie. In the case of all those 15 year olds, the entries are retrieved based on the next item, the person’s name. And since these are strings, they are ordered alphabetically. Thus the entry for Avaka Sasaki is retrieved before Moa Kikuchi and Moa is retrieved before Suzuka, which is retrieved before Yui. 8-17 In our case of hierarchical clustering, We use the distance between breeds as the primary priority. To resolve ties we will use an index number. The first element we put on the queue will have an index of 0, the second element an index of 1, the third , 2, and so on. Our complete entry we add to the queue will be of the form: distance to nearest neighbor index number current cluster information about nearest (0.23170921460558744, 0, neighbor [['Border Collie'], ['Portuguese Water Dog', 0.23170921460558744, (0, 8)], {1: ((0, 1), 1.0244831578726061), distances to all other breeds. 2: ((0, 2), 0.4634184292111748), The tuple (0, 1) indicates that ... this is the distance between 9: ((0, 9), 0.6093065384986165), breed 0 (Border Collie) and 10: ((0, 10), 2.756155583828758)}]) breed 1 (Boston Terrier) We initialize the priority queue by placing on the queue, an entry like this for each breed. Repeat the following until there is only one cluster. We get two items from the queue, merge them into one cluster and put that entry on the queue. In our dog breed example, we get the entry for Border Collie and the entry for Portuguese Water Dog. We create the queue ['Border Collie', 'Portuguese Water Dog'] Next we compute the distance of this new cluster to all the other dog breeds except those in the new cluster. We do this by merging the distance dictionaries of the two initial clusters in the following way. Let’s call the distance dictionary of the first item we get from the queue distanceDict1, the distance dictionary of the second item we get from the queue distanceDict2, and the distance dictionary we are constructing for the new cluster newDistanceDict. 8-18 CLUSTERING Initialize newDistanceDict to an empty dictionary for each key, value pair in distanceDict1: if there is an entry in distanceDict2 with that key: if the distance for that entry in distanceDict1 is shorter than that in distanceDict2: ! ! place the distanceDict1 entry in newDistanceDict ! else: place the distanceDict1 entry in newDistanceDict key value in the Border Collie Distance List value in the Portuguese Water Dog Distance List value in the Distance List for the new cluster 0 - ((0, 8), 0.2317092146055) - 1 ((0, 1), 1.02448315787260) ((1, 8), 1.25503395239308) ((0, 1), 1.02448315787260) 2 ((0, 2), 0.46341842921117) ((2, 8), 0.69512764381676) (0, 2), 0.46341842921117) 3 ((0, 3), 2.52128307411504) ((3, 8), 2.3065500082408) ((3, 8), 2.3065500082408) 4 ((0, 4), 2.41700998092941) ((4, 8), 2.643745991701) ((0, 4), 2.41700998092941) 5 ((0, 5), 1.31725590972761) ((5, 8), 1.088215707936) ((5, 8), 1.088215707936) 6 ((0, 6), 0.90660838225252) ((6, 8), 0.684696194462) ((6, 8), 0.684696194462) 7 ((0, 7), 3.98523295438990) ((7, 8), 3.765829069545) ((7, 8), 3.765829069545) 8 ((0, 8), 0.23170921460558) - - 9 ((0, 9), 0.60930653849861) ((8, 9), 0.566225873458) ((8, 9), 0.566225873458) 10 ((0, 10), 2.7561555838287) ((8, 10), 2.980333906137) ((0, 10), 2.7561555838287) The complete entry that will be placed on the queue as a result of merging the Border Collie and the Portuguese Water Dog will be (0.4634184292111748, 11, [('Border Collie', 'Portuguese Water Dog'), [2, 0.4634184292111748, (0, 2)], {1: ((0, 1), 1.0244831578726061), 2: ((0, 2), 0.4634184292111748), 3: ((3, 8), 2.306550008240866), 4: ((0, 4), 2.4170099809294157), 5: ((5, 8), 1.0882157079364436), 6: ((6, 8), 0.6846961944627522), 7: ((7, 8), 3.7658290695451373), 9: ((8, 9), 0.5662258734585477), 10: ((0, 10), 2.756155583828758)}]) 8-19 s Code It Can you implement the algorithm presented above in Python? To help you in this task, there is a Python file on the book’s website, hierarchicalClustererTemplate.py (http://guidetodatamining.com/guide/pg2dm-python/ch8/hierarchicalClustererTemplate.py) that gives you a starting point. You need to: 1. Finish the init method. For each entry in the data: 1. compute the Euclidean Distance from that entry to all other entries and create a Python Dictionary as described above. 2. Find the nearest neighbor 3. Place the info for this entry on the queue. 2. Write a cluster method. This method should repeatedly: 1. retrieve the top 2 entries on the queue 2. merge them 3. place the new cluster on the queue until there is only one cluster on the queue. 8-20 CLUSTERING s Code It - solution Remember: This is only my solution and not necessarily the best solution. You might have come up with a better one! from queue import PriorityQueue import math """ Example code for hierarchical clustering """ def getMedian(alist): """get median value of list alist""" tmp = list(alist) tmp.sort() alen = len(tmp) if (alen % 2) == 1: return tmp[alen // 2] else: return (tmp[alen // 2] + tmp[(alen // 2) - 1]) / 2 def normalizeColumn(column): """Normalize column using Modified Standard Score""" median = getMedian(column) asd = sum([abs(x - median) for x in column]) / len(column) result = [(x - median) / asd for x in column] return result class hClusterer: """ this clusterer assumes that the first column of the data is a label not used in the clustering. The other columns contain numeric data""" def __init__(self, filename): file = open(filename) self.data = {} self.counter = 0 self.queue = PriorityQueue() lines = file.readlines() 8-21 file.close() header = lines[0].split(',') self.cols = len(header) self.data = [[] for i in range(len(header))] for line in lines[1:]: cells = line.split(',') toggle = 0 for cell in range(self.cols): if toggle == 0: self.data[cell].append(cells[cell]) toggle = 1 else: self.data[cell].append(float(cells[cell])) # now normalize number columns (that is, skip the first column) for i in range(1, self.cols): self.data[i] = normalizeColumn(self.data[i]) ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### I have read in the data and normalized the columns. Now for each element i in the data, I am going to 1. compute the Euclidean Distance from element i to all the other elements. This data will be placed in neighbors, which is a Python dictionary. Let's say i = 1, and I am computing the distance to the neighbor j and let's say j is 2. The neighbors dictionary for i will look like {2: ((1,2), 1.23), 3: ((1, 3), 2.3)... } 2. find the closest neighbor 3. place the element on a priority queue, called simply queue, based on the distance to the nearest neighbor (and a counter used to break ties. # now push distances on queue rows = len(self.data[0]) for i in range(rows): minDistance = 99999 nearestNeighbor = 0 neighbors = {} for j in range(rows): if i != j: dist = self.distance(i, j) if i < j: pair = (i,j) else: pair = (j,i) neighbors[j] = (pair, dist) 8-22 CLUSTERING if dist < minDistance: minDistance = dist nearestNeighbor = j nearestNum = j # create nearest Pair if i < nearestNeighbor: nearestPair = (i, nearestNeighbor) else: nearestPair = (nearestNeighbor, i) # put instance on priority queue self.queue.put((minDistance, self.counter, [[self.data[0][i]], nearestPair, neighbors])) self.counter += 1 def distance(self, i, j): sumSquares = 0 for k in range(1, self.cols): sumSquares += (self.data[k][i] - self.data[k][j])**2 return math.sqrt(sumSquares) def cluster(self): done = False while not done: topOne = self.queue.get() nearestPair = topOne[2][1] if not self.queue.empty(): nextOne = self.queue.get() nearPair = nextOne[2][1] tmp = [] ## ## I have just popped two elements off the queue, ## topOne and nextOne. I need to check whether nextOne ## is topOne's nearest neighbor and vice versa. ## If not, I will pop another element off the queue ## until I find topOne's nearest neighbor. That is what ## this while loop does. ## while nearPair != nearestPair: tmp.append((nextOne[0], self.counter, nextOne[2])) self.counter += 1 nextOne = self.queue.get() nearPair = nextOne[2][1] ## ## this for loop pushes the elements I popped off in the ## above while loop. ## 8-23 for item in tmp: self.queue.put(item) if len(topOne[2][0]) == 1: item1 = topOne[2][0][0] else: item1 = topOne[2][0] if len(nextOne[2][0]) == 1: item2 = nextOne[2][0][0] else: item2 = nextOne[2][0] ## curCluster is, perhaps obviously, the new cluster ## which combines cluster item1 with cluster item2. curCluster = (item1, item2) ## ## ## ## ## ## ## ## Now I am doing two things. First, finding the nearest neighbor to this new cluster. Second, building a new neighbors list by merging the neighbors lists of item1 and item2. If the distance between item1 and element 23 is 2 and the distance betweeen item2 and element 23 is 4 the distance between element 23 and the new cluster will be 2 (i.e., the shortest distance). minDistance = 99999 nearestPair = () nearestNeighbor = '' merged = {} nNeighbors = nextOne[2][2] for (key, value) in topOne[2][2].items(): if key in nNeighbors: if nNeighbors[key][1] < value[1]: dist = nNeighbors[key] else: dist = value if dist[1] < minDistance: minDistance = dist[1] nearestPair = dist[0] nearestNeighbor = key merged[key] = dist if merged == {}: return curCluster else: self.queue.put( (minDistance, self.counter, [curCluster, nearestPair, merged])) self.counter += 1 8-24 CLUSTERING def printDendrogram(T, sep=3): """Print dendrogram of a binary tree. Each tree node is represented by a length-2 tuple. printDendrogram is written and provided by David Eppstein 2002. Accessed on 14 April 2014: http://code.activestate.com/recipes/139422-dendrogram-drawing/ """ ! def isPair(T): return type(T) == tuple and len(T) == 2 def maxHeight(T): if isPair(T): h = max(maxHeight(T[0]), maxHeight(T[1])) else: h = len(str(T)) return h + sep activeLevels = {} def traverse(T, h, isFirst): if isPair(T): traverse(T[0], h-sep, 1) s = [' ']*(h-sep) s.append('|') else: s = list(str(T)) s.append(' ') while len(s) < h: s.append('-') if (isFirst >= 0): s.append('+') if isFirst: activeLevels[h] = 1 else: del activeLevels[h] A = list(activeLevels) A.sort() for L in A: if len(s) < L: while len(s) < L: s.append(' ') s.append('|') print (''.join(s)) if isPair(T): 8-25 traverse(T[1], h-sep, 0) traverse(T, maxHeight(T), -1) filename = '//Users/raz/Dropbox/guide/pg2dm-python/ch8/dogs.csv' n hg = hClusterer(filename) cluster = hg.cluster() printDendrogram(cluster) When I run this code I get the following results: Chihuahua -------------------------------+ |--+ Yorkshire Terrier -----------------------+ | |-Great Dane ------------------------------+ | |--+ Bullmastiff --------------------------+ | |--+ German Shepherd ----------------+ | |--+ | Golden Retriever ---------------+ | | |--+ Standard Poodle ----------------+ | |--+ Boston Terrier --------------+ | |--+ Brittany Spaniel ---------+ | |--+ Border Collie ---------+ | |--+ Portuguese Water Dog --+ which match the results we computed by hand. That’s encouraging. 8-26 CLUSTERING s you try! Breakfast Cere als On the book’s website, there is a file containing nutritional information about 77 breakfast cereals including cereal name calories per serving protein (in grams) fat (in grams) sodium (in mg) fiber (grams) carbohydrates (grams) sugars (grams) potassium (mg) vitamins (% of RDA) Can you perform hierarchical clustering of this data? Which cereal is most similar to Trix? To Muesli Raisins & Almonds? This data set is from Carnegie Mellon Universi http://lib.stat.cm ty: u.edu/DASL/Da tafiles/Cereals. html 8-27 s you try - results To run the clusterer on this dataset we only needed to change the filename from dogs.csv to cereal.csv. Here is an abbreviated version of the results: Mueslix Crispy Blend --------------------------------------------------------------------+ |--+ Muesli Raisins & Almonds -------------------------------------------------------------+ | |--+ Muesli Peaches & Pecans --------------------------------------------------------------+ ... Lucky Charms ----------+ |--+ Fruity Pebbles --+ | |--+ | Trix ------------+ | | |--+ Cocoa Puffs -----+ | |--+ Count Chocula ---+ Trix, is most similar to Fruity Pebbles. (I recommend you confirm this by running out right now and buying a box of each.) Perhaps not surprisingly, Muesli Raisins & Almonds is closest to Muesli Peaches & Pecans. That’s it for hierarchical clustering! 8-28 That was pretty easy! CLUSTERING k Introducing ... -means clustering With k-means clustering we specify how many clusters to make. This is the ‘k’. If we want to make 2 groups k = 2, if we want to make 100, k=100. k-means clustering is The Most Popular clustering algorithm! K-means is cool! The algorithm is over 50 years old! It was first proposed by Dr. Stuart Lloyd of Bell Labs in 1957. Here is what you need to know about k-means 8-29 Here are some instances we want to cluster into 3 groups (k=3). Suppose they are dog breeds as mentioned earlier and the dimensions are height and weight. Because k=3, we pick 3 random points as the initial centroids of each cluster (‘initial centroid’ means the initial center or mean of the cluster). Right then. We’ve indicated these initial centroids as red, green, and blue circles. Okay. Next, we are going to assign each instance to the nearest centroid. The points assigned to each centroid are a cluster. So we have created k initial clusters!! Now, for each cluster, we compute the mean (average) point of that cluster. This will be our updated centroid. And repeat (assign each instance to the centroid & recompute centroids) until the centroids don’t move much or we have reached some maximum number of iterations. 8-30 CLUSTERING The basic k-means algorithm is: 1. select k random instances to be the initial centroids 2. REPEAT 3. assign each instance to the nearest centroid. (forming k clusters) 4. update centroids by computing mean of each cluster 5. UNTIL centroids don’t change (much). Let’s go through an example. Consider the following points (x and y coordinates): (1, (1, (2, (2, (4, (4, (5, (5, 2) 4) 2) 3) 2) 4) 1) 3) Say we want to cluster these into 2 groups. step 1 of above algorithm: select k random instances to be initial centroids. Suppose we randomly select (1, 4) as centroid 1 and (4, 2) as centroid 2. step 3: assign each instance to the nearest centroid To assign each instance to the nearest centroid we can use any of the distance measures we have previously discussed. To keep things simple, for this example let’s use Manhattan Distance. 8-31 point distance from centroid 1 (1, 4) distance from centroid 2 (4, 2) (1, 2) 2 3 (1,4) 0 5 (2, 2) 3 2 (2, 3) 2 3 (4, 2) 5 0 (4, 4) 3 2 (5, 1) 7 2 (5, 3) 5 2 Based on these distances we assign the points to the following clusters: CLUSTER 1 (1, 2) (1, 4) (2, 3) CLUSTER 2 (2, 2) (4, 2) (4, 4) (5, 1) (5, 3) step 4: update centroids We compute the new centroids by computing the mean of each cluster. The mean x coordinate of cluster 1 is: (1 + 1 + 2) / 3 = 4/3 = 1.33 and the mean y is (2 + 4 + 3) / 3 = 9/3 = 3 So the new cluster 1 centroid is (1.33, 3). The new centroid for cluster 2 is (4, 2.4) 8-32 CLUSTERING step 5: until centroids don’t change The old centroids were (1, 4) and (4, 2) and the new ones are (1.33, 3) and (4, 2.4). The centroids changed so we repeat. step 3: assign each instance to the nearest centroid Again we compute Manhattan Distance. point distance from centroid 1 (1.33, 3) distance from centroid 2 (4, 2.4) (1, 2) 1.33 3.4 (1, 4) 1.33 4.6 (2, 2) 1.67 2.4 (2, 3) 0.67 2.6 (4, 2) 3.67 0.4 (4, 4) 3.67 1.6 (5, 1) 5.67 2.4 (5, 3) 3.67 1.6 and based on these distances assign the points to clusters: CLUSTER 1 (1, 2) (1, 4) (2, 2) (2, 3) CLUSTER 2 (4, 2) (4, 4) (5, 1) (5, 3) step 4: update centroids We compute the new centroids by computing the mean of each cluster. Cluster 1 centroid: (1.5, 2.75) Cluster 2 centroid: (4.5, 2.5) 8-33 step 5: until centroids don’t change The centroids changed so we repeat. step 3: assign each instance to the nearest centroid Again we compute Manhattan Distance. point distance from centroid 1 (1.5, 2.75) distance from centroid 2 (4.5, 2.5) (1, 2) 1.25 4.0 (1, 4) 1.75 5.0 (2, 2) 1.25 3.0 (2, 3) 0.75 3.0 (4, 2) 3.25 1.0 (4, 4) 3.75 2.0 (5, 1) 5.25 2.0 (5, 3) 3.75 1.0 and based on these distances assign the points to clusters: CLUSTER 1 (1, 2) (1, 4) (2, 2) (2, 3) CLUSTER 2 (4, 2) (4, 4) (5, 1) (5, 3) step 4: update centroids We compute the new centroids by computing the mean of each cluster. Cluster 1 centroid: (1.5, 2.75) Cluster 2 centroid: (4.5, 2.5) 8-34 CLUSTERING step 5: until centroids don’t change The updated centroids are identical to the previous ones so the algorithm converged on a solution and we can stop. The final clusters are CLUSTER 1 (1, 2) (1, 4) (2, 2) (2, 3) CLUSTER 2 (4, 2) (4, 4) (5, 1) (5, 3) We stop when the centroids don’t change. This is the same condition as saying no point are shifting from one cluster to another. This is what we mean when we say the algorithm ‘converges’. During the execution of the algorithm, the centroids shift from their initial position to some final position. The vast majority of this shift occurs during the first few iterations. Often, the centroids barely move during the final iterations. This means that the k-means algorithm produces good clusters early on and later iterations are likely to produce only minor refinements. 8-35 Because of this behavior of the algorithm, we can dramatically reduce its execution time by relaxing our criteria of “no points are shifting from one cluster to another” to “fewer than 1% of the points are shifting from one cluster to another.” This is a common approach! For you computer science geeks: N 8-36 K-means is simple! K-means is an instance of the Expectation Maximization (EM) Algorithm, which is an iterative method that alternates between two phases. We start with an initial estimate of some parameter. In the Kmeans case we start with an estimate of the centroids. In the expectation (E) phase, we use this estimate to place points into their expected cluster. In the Maximization (M) phase we use these expected values to adjust the estimate of the centroids. If you are interested in learning more about the EM algorithm the wikipedia page http:// en.wikipedia.org/wiki/Expectation %E2%80%93maximization_algorithm is a good place to start. CLUSTERING Hill Climbing I would like to briefly interrupt our discussion of K-means clustering to talk about hill climbing algorithms. Suppose our goal is to reach the peak of some mountain and we come up with the following algorithm: start at some random location on the mountain. REPEAT take a step in the direction that will take you higher. UNTIL there is no direction that will take you higher. This seems like a reasonable algorithm. Consider using it with the mountain shown here ➯ You can see that regardless of where we are plopped down on the mountain, we will reach the peak if we follow the algorithm. And if we think of this as a graph, we will reach the peak value regardless of where we start on the graph. Now let’s consider using the algorithm with the graph on the following page 8-37 Here, things don’t work out as expected. If we start at ‘A’ on the graph... We will reach the peak ‘B’ but not reach the highest peak ‘D’. Or, to put it another way, we reach a local maximum, B, but not the global maximum, D. Sometimes thing Thus, this simple version of the hill-climbing algorithm is not guaranteed to reach the optimal solution. The k-means clustering algorithm is like this. There is no guarantee that it will find the optimal division of the data into clusters. Why? Because at the start of the algorithm we select an initial set of centroids randomly, which is much like picking a random spot like point ‘A’ on the graph above. Then, based on this initial set, we optimize the clusters finding the local optimum (similar to point ‘B’ on the graph). The final clusters are heavily dependent on the selection of the initial centroids. Even so, the k-means algorithm generates decent clusters. 8-38 CLUSTERING How do we know whether one set of clusters (division of the data into clusters) is better than another? SSE or Scatter To determine the quality of a set of clusters we can use the sum of the squared error (SSE). This is also called scatter. Here is how to compute it: for each point we will square the distance from that point to its centroid, then add all those squared distances together. More formally, k SSE = ∑ ∑ dist(ci , x)2 i=1 x∈Ci Let’s dissect that. In the first summation sign we are iterating over the clusters. So initially i equals cluster 1, then i equals cluster 2, up to i equals cluster k. The next summation sign iterates over the points in that cluster—something like, for each point x in cluster i. Dist is whatever distance formula we are using (for example, Manhattan, or Euclidean). So we compute the distance between that point, x, and the centroid for the cluster ci, square that distance and add it to our total. Let’s say we run our k-means algorithm twice on the same data and for each run we pick a different set of random initial centroids. Is the set of clusters that were computed during the first run worse or better than the set computed during the second run? To answer that question we compute the SSE for both sets of clusters. The set with the smaller SSE is the better of the two. 8-39 g! Time to start codin r basic k-means Here’s the code fo import math import random def getMedian(alist): """get median of list""" tmp = list(alist) tmp.sort() alen = len(tmp) if (alen % 2) == 1: return tmp[alen // 2] else: return (tmp[alen // 2] + tmp[(alen // 2) - 1]) / 2 def normalizeColumn(column): """normalize the values of a column using Modified Standard Score that is (each value - median) / (absolute standard deviation)""" median = getMedian(column) asd = sum([abs(x - median) for x in column]) / len(column) result = [(x - median) / asd for x in column] return result class kClusterer: """ Implementation of kMeans Clustering This clusterer assumes that the first column of the data is a label not used in the clustering. The other columns contain numeric data """ def __init__(self, filename, k): """ k is the number of clusters to make This init method: 1. reads the data from the file named filename 2. stores that data by column in self.data 3. normalizes the data using Modified Standard Score 8-40 CLUSTERING 4. randomly selects the initial centroids 5. assigns points to clusters associated with those centroids """ file = open(filename) self.data = {} self.k = k self.counter = 0 self.iterationNumber = 0 # used to keep track of % of points that change cluster membership # in an iteration self.pointsChanged = 0 # Sum of Squared Error self.sse = 0 # # read data from file # lines = file.readlines() file.close() header = lines[0].split(',') self.cols = len(header) self.data = [[] for i in range(len(header))] # we are storing the data by column. # For example, self.data[0] is the data from column 0. # self.data[0][10] is the column 0 value of item 10. for line in lines[1:]: cells = line.split(',') toggle = 0 for cell in range(self.cols): if toggle == 0: self.data[cell].append(cells[cell]) toggle = 1 else: self.data[cell].append(float(cells[cell])) self.datasize = len(self.data[1]) self.memberOf = [-1 for x in range(len(self.data[1]))] # # now normalize number columns # for i in range(1, self.cols): self.data[i] = normalizeColumn(self.data[i]) # select random centroids from existing points random.seed() self.centroids = [[self.data[i][r] for i in range(1, len(self.data))] for r in random.sample(range(len(self.data[0])), self.k)] self.assignPointsToCluster() 8-41 def updateCentroids(self): """Using the points in the clusters, determine the centroid (mean point) of each cluster""" members = [self.memberOf.count(i) in range(len(self.centroids))] self.centroids = [[sum([self.data[k][i] for i in range(len(self.data[0])) if self.memberOf[i] == centroid])/members[centroid] for k in range(1, len(self.data))] for centroid in range(len(self.centroids))] def assignPointToCluster(self, i): """ assign point to cluster based on distance from centroids""" min = 999999 clusterNum = -1 for centroid in range(self.k): dist = self.euclideanDistance(i, centroid) if dist < min: min = dist clusterNum = centroid # here is where I will keep track of changing points if clusterNum != self.memberOf[i]: self.pointsChanged += 1 # add square of distance to running sum of squared error self.sse += min**2 return clusterNum def assignPointsToCluster(self): """ assign each data point to a cluster""" self.pointsChanged = 0 self.sse = 0 self.memberOf = [self.assignPointToCluster(i) for i in range(len(self.data[1]))] def euclideanDistance(self, i, j): """ compute distance of point i from centroid j""" sumSquares = 0 for k in range(1, self.cols): sumSquares += (self.data[k][i] - self.centroids[j][k-1])**2 return math.sqrt(sumSquares) def kCluster(self): """the method that actually performs the clustering As you can see this method repeatedly updates the centroids by computing the mean point of each cluster re-assign the points to clusters based on these new centroids 8-42 CLUSTERING until the number of points that change cluster membership is less than 1%. """ done = False while not done: self.iterationNumber += 1 self.updateCentroids() self.assignPointsToCluster() # # we are done if fewer than 1% of the points change clusters # if float(self.pointsChanged) / len(self.memberOf) < 0.01: done = True print("Final SSE: %f" % self.sse) def showMembers(self): """Display the results""" for centroid in range(len(self.centroids)): print ("\n\nClass %i\n========" % centroid) for name in [self.data[0][i] for i in range(len(self.data[0])) if self.memberOf[i] == centroid]: print (name) ## ## RUN THE K-MEANS CLUSTERER ON THE DOG DATA USING K = 3 ### km = kClusterer('dogs2.csv', 3) km.kCluster() km.showMembers() Let’s dissect that code a bit! 8-43 As with our code for the hierarchical clusterer, we are storing the data by column. Consider our dog breed data. If we represent the data in spreadsheet form, it would likely look like this (the height and weight are normalized): height weight 0 -0.1455 Boston Terrier -0.7213 -0.873 Brittany Spaniel -0.3607 -0.4365 Bullmastiff 1.2623 2.03704 German Shepherd 0.9016 0.81481 ... ... ... breed Border Collie And if we were to transfer this data to Python we would likely make a list that looks like the following: data = [ data for the Border Collie, data for the Boston Terrier, ... ] So to fully specify the data format: data = [ [‘Border Collie’, 0, -0.1455], [‘Boston Terrier’, -0.7213, -0.873], ... ] So we are storing the data by row. This seems like the common sense approach and the one we have been using throughout the book. Alternatively, we can store the data column first: 8-44 CLUSTERING data = [ column 1 data, column 2 data, column 3 data ] So for our dog example: data = [ [‘Border Collie’, ‘Boston Terrier’, ‘Brittany Spaniel’, ...], [ 0, -0.7213, -0.3607, ...], [-0.1455, -0.7213, -0.4365, ...], ... ] This is what we did for the hierarchical clusterer and what we are doing here for k-means. The benefit of this approach is that it makes implementing many of the math functions easier. We can see this in the first two procedures in the code above, getMedian and normalizeColumn. Because we stored the data by column, these procedures take simple lists as arguments. >>> normalizeColumn([8, 6, 4, 2]) [1.5, 0.5, -0.5, -1.5] The constructor method, __init__ takes as arguments, the filename of the data file and k, the number of clusters to construct. It reads the data from the file and stores the data by column. It normalizes the data using the normalizeColumn procedure, which implements the Modified Standard Score method. Finally, it selects k elements from this data as the initial centroids and assigns each point to a cluster depending on that point’s distance to the initial centroids. It does this assignment using the method assignPointsToCluster. The method, kCluster actually performs the clustering by repeatedly calling updateCentroids, which computes the mean of each cluster and assignPointsToCluster until fewer than 1% of the points change clusters. The method showMembers simply displays the results. Running the code on the dog breed data yields the following results: Final SSE: 5.243159 Class 0 ======== Bullmastiff Great Dane 8-45 Class 1 ======== Boston Terrier Chihuahua Yorkshire Terrier Class 2 ======== Border Collie Brittany Spaniel German Shepherd Golden Retriever Portuguese Water Dog Standard Poodle Wow! For this small dataset the clusterer does extremely well. s You try How well does the kmeans clusterer work with the cereal dataset with k = 4 • Do the sweet cereals cluster together (Cap’n’Crunch, Cocoa Puffs, Froot Loops, Lucky Charms? • Do the bran cereals cluster together (100% Bran, All-Bran, All-Bran with Extra Fiber, Bran Chex? • What does Cheerios cluster with? Try the clusterer with the auto mpg dataset with different values for k=8? Does this follow your expectations of how these cars should be grouped? 8-46 CLUSTERING s You try - my results How well does the kmeans clusterer work with the cereal dataset with k = 4. Your results may vary from mine but here is what I found out. • Do the sweet cereals cluster together (Cap’n’Crunch, Cocoa Puffs, Froot Loops, Lucky Charms? Yes, all these sweet cereals (plus Count Chocula, Fruity Pebbles, and others) are in the same sweet cluster. • Do the bran cereals cluster together (100% Bran, All-Bran, All-Bran with Extra Fiber, Bran Chex? Again, yes! Included in this cluster are also Raisin Bran and Fruitful Bran. • What does Cheerios cluster with? Cheerios always seems to be in the same cluster as Special K Try the clusterer with the auto mpg dataset with different values for k=8? Does this follow your expectations of how these cars should be grouped? The clusterer seems to do a reasonable job on this dataset but on rare occasions you will notice one or more of the clusters are empty. OMG! I told the clusterer to make 8 groups but 1 of them is empty. There must be something wrong with the code! 8-47 6 5 3 1 7 4 8 Nothing wrong with the code. Let’s look at an example to see how this happens. 2 Consider clustering these points with k = 3. We randomly pick points 1, 7 & 8 as the initial centroids.1 1. This example from Tolga Can http://www.ceng.metu.edu.tr/~tcan/ ceng465_f1314/Schedule/KMeansEmpty.html 6 5 3 1 7 Here we assign the points to clusters. Point 6 is closer to point 7 than it is to point 1 so we assign it to the pink cluster.1 8 4 2 Next we update the centroids (shown by the ‘+’) 5 3 1 8-48 2 6 7 4 8 1. For those of you who are not looking at this in color, the pink cluster now contains points 6 and 7. CLUSTERING Then we reassign points to clusters based on these new centroids. Point 6 is closer to the blue centroid than it is the pink one so it gets reassigned to blue. Point 7 is closer to the green centroid than the pink one so it also gets reassigned leaving the pink cluster empty. 5 3 1 6 7 4 8 2 In sum, just because we specify how many groups to make does not mean that the k-means clusterer will produce that many non-empty groups. This may be a good thing. Just looking at the data above, it appears to be naturally clustered into two groups and our attempt to cluster the data into three failed. Suppose we have 1,000 instances we would like to cluster into 10 groups and when we run the clusterer two of the groups are empty. This result may indicate something about the underlying structure of the data. Perhaps the data does not naturally divide into ten groups and we can explore other groupings (trying to cluster into eight groups, for example). On the other hand, sometimes when we specify 10 clusters we actually want 10 non-empty clusters. If that is the case, we need to alter the algorithm so it detects an empty cluster. Once one is detected the algorithm changes that cluster’s centroid to a different point. One possibility is to change it to the instance that is furthest from its corresponding centroid. (In the example above, once we detect the pink cluster is empty, we re-assign the pink centroid to point 1, since point 1 is the furthest point to its corresponding centroid. That is, I compute the distances from 1 2 3 4 5 6 7 8 to to to to to to to to its its its its its its its its centroid centroid centroid centroid centroid centroid centroid centroid and pick the point that is furthest from its centroid as the new centroid of the empty cluster. 8-49 (sigh) Wouldn’t it be dreamy if we could make k-means faster and more accurate. With a simple change to k-means we can! The new algorithm is called k-means++ Even the name makes it sound newer, better, faster, and more accurate —a turbocharged k-means! k-means++ In the previous section we examined the k-means algorithm in its original form as it was developed in the late 50s. As we have seen, it is easy to implement and performs well. It is still the most widely used clustering algorithm on the planet. But it is not without its flaws. A major weakness in k-means is in the first step where it randomly picks k of the datapoints to be the initial centroids. As you can probably tell by my bolding and embiggening the word ‘random’, it is the random part that is the problem. Because it is random, sometimes the initial centroids are a great pick and lead to near optimal clustering. Other times the initial centroids are a reasonable pick and lead to good clustering. But sometimes—again, because we pick randomly—sometimes the initial centroids are poor leading to non-optimal clustering. The k-means++ algorithm fixes this defect by changing the way we pick the initial centroids. Everything else about k-means remains the same. embiggen: verb. To make larger, to make the size increase. 8-50 CLUSTERING k-means++ -- selecting the initial set of centroids 1. Initially, the set of initial centroids is empty. 2. Select the first centroid randomly from the data points as before. 3. Until we have k initial centroids: a. Compute the distance, D, between each datapoint (dp) and its closest centroid. This distance is D(dp). b. In a probability proportional to D(dp) select one datapoint at random to be a new centroid and add it to the set of centroids. c. REPEAT Let’s dissect the meaning of “In a probability proportional to D(dp) select one datapoint to be a new centroid.” To do this, I will present a simple example. Suppose we are in the middle of this process. We have already selected two initial centroids and are in the process of selecting another one. So we are on step 3a of the above algorithm. Let’s say we have 5 remaining centroids and their distances to the 2 centroids (c1 and c2) are as follows: Dc1 Dc2 dp1 5 7 dp2 9 8 dp3 2 5 dp4 3 7 dp5 5 2 entroid 1 ance to c t is d “ s n Dc1 mea ance to ans “dist e m 2 c D esents and dp1 repr .” 2 id o r cent t 1. datapoin 8-51 Step 3a says we pick the closest distance so we get: closest dp1 5 dp2 8 dp3 2 dp4 3 dp5 2 Now we are going to convert those numbers to a decimals whose sum equals 1 (I’ll call this the weight). To do that we sum the original numbers. In this case the sum equals 20. Now we divide each number by the sum. The result is shown here weight I like to think of this as a roulette wheel that looks like this: dp1 dp2 10% dp3 dp4 dp1 0.25 dp2 0.40 dp3 0.10 dp4 0.15 dp5 0.10 sum 1.00 dp5 25% 15% 10% 40% We are going to spin a ball on that wheel, see where it lands, and pick that as the new centroid. This is what we mean by “In a probability proportional to D(dp) select one datapoint to be a new centroid.” Let us rough out this idea in Python. Say we have a list tuples containing a datapoint and its weight data = [("dp1", 0.25), ("dp2", 0.4), ("dp3", 0.1), ("dp4", 0.15), ("dp5", 0.1)] 8-52 CLUSTERING The function roulette will now select a datapoint in a probability proportional to its weight: import random random.seed() def roulette(datalist): ! i = 0 ! soFar = datalist[0][1] ! ball = random.random() ! while soFar < ball: ! i += 1 ! soFar += datalist[i][1] ! return datalist[i][0] If the function did pick with this proportion, we would predict that if we picked 100 times, 25 of them would be dp1; 40 of them would be dp2; 10 of them dp3; 15 dp4; and 10, dp5. Let’s see if that is true: import collections results = collections.defaultdict(int) for i in range(100): ! results[roulette(data)] += 1 print results {'dp5': 11, 'dp4': 15, 'dp3': 10, 'dp2': 38, 'dp1': 26} Great! Our function does return datapoints in roughly the correct proportion. The idea in k-means++ clustering is that, while we still pick the initial centroids randomly, we prefer centroids that are far away from one another. Time to do some coding! 8-53 s Code It Can you implement k-means++ in Python? Again, the only difference between our previous implementation of k-means and this code is in how we select the initial centroids. Make a copy of our original k-means code and modify it. Our original code created the initial centroids in this line: self.centroids = [[self.data[i][r] for i in range(1, len(self.data))] for r in random.sample(range(len(self.data[0])), self.k)] Let us replace that line with: self.selectInitialCentroids() Your job is to write that method! Good luck! Throughout the book, the author has been putting pictures of hip people like us using laptops in hopes of influencing you the reader to put down the book and do some coding. If he has been successful, please let him know at ron.zacharski@gmail.com the book put down 8-54 a reader coding (and apparently enjoying it!) CLUSTERING s Code It -solution Here is my version of selectInitialCentroids: def distanceToClosestCentroid(self, point, centroidList): result = self.eDistance(point, centroidList[0]) for centroid in centroidList[1:]: distance = self.eDistance(point, centroid) if distance < result: result = distance return result def selectInitialCentroids(self): """implement the k-means++ method of selecting the set of initial centroids""" centroids = [] total = 0 # first step is to select a random first centroid current = random.choice(range(len(self.data[0]))) centroids.append(current) # loop to select the rest of the centroids, one at a time for i in range(0, self.k - 1): # for every point in the data find its distance to # the closest centroid weights = [self.distanceToClosestCentroid(x, centroids) for x in range(len(self.data[0]))] total = sum(weights) # instead of raw distances, convert so sum of weight = 1 weights = [x / total for x in weights] # # now roll virtual die num = random.random() + k-means+ total = 0 the entire : r fo e d o nc ite x = -1 The Pytho ok’s webs on the bo is r ie m if o s s .c g cla # the roulette wheel simulation minin idetodata http://gu while total < num: x += 1 total += weights[x] centroids.append(x) self.centroids = [[self.data[i][r] for i in range(1, len(self.data))] for r in centroids] 8-55 Summary Clustering is all about discovery. However, the simple examples we have been using in this chapter may obscure this fundamental idea. After all, we know how to cluster breakfast cereals without a computer’s help—sugary cereals, healthy cereals. And we know how to cluster car models—a Ford F150 goes in the truck category, a Mazda Miata in the sports car category, and a Honda Civic in the fuel efficient category. But consider a task where discovery IS important. When we do a web search we are presented with a long list of results. For example, when I just did a Google search on “carbon sequestration” I get over 2.8 million results. A number of researchers have examined the benefits of clustering these results. Instead of that long list of carbon sequestration results we might also see categories like “carbon sequestration in freshwater wetlands” and “carbon sequestration in forests.” Josh Gotbaum’s team conducted extensive interviews with 3,000 people asking them questions about their values. Using these interviews they clustered the people into five groups. When they examined the clusters they gave them the descriptions: 1. extending opportunity to others 2. working within a community 3. achieving independence 4. focusing on family 5. defending righteousness They then crafted targeted campaign ads to each group. from The Numerati by Stephen Baker 8-56 CLUSTERING We just learned two clustering techniques, hierarchical clustering and k-means. When should we use one over the other? Got it! What about hierarchical clustering? Brilliant! Maybe I should practice by trying it out on some new data. Good question! The benefits of K-means is that it is simple and has fast execution time. It is a great choice in general. It is also good choice for your first steps in exploring your data even if you eventually move to another clustering technique. However, it does not handle outliers well. Although, we can remedy this by identifying and removing the outliers. The obvious use of hierarchical clustering is when we want to create a taxonomy or hierarchy from our data. This hierarchy may be more informative about the data than a flat set of clusters. It is also not as efficient in terms of execution speed and memory requirements. 8-57 Enron E N RO EN Perhaps you remember Enron and the Enron Scandal. In its heyday Enron was a mega-huge energy company with revenues over $100 billion and over 20,000 employees (Microsoft’s revenue then was only $22 billion). Due to systemic sleaziness and corruption including creating an artificial energy shortage that resulted in electricity blackouts in California, Enron went bankrupt and a bunch of people went to jail. For a documentary about this see Enron: The Smartest Guys in the Room, which is available for streaming from Netflix and Amazon Prime. Now you might be thinking “Hey, this Enron stuff is sort of interesting but what does it have to do with data mining?” Hmm. This Enron stuff is sort of interesting but what does it have to do with data mining? Well, as part of the investigation, the U.S. Federal Energy Regulatory Commision acquired 600,000 emails from Enron employees. This database is now available to researchers. It may be the largest email database in the world! For more information on the Enron database see the Wikipedia entry: http://en.wikipedia.org/wiki/Enron_Corpus and https://www.cs.cmu.edu/~./enron/ This database is an amazing resource for reearchers in a wide variety of areas. 8-58 CLUSTERING We are going to try to cluster a small part of the Enron corpus. For our simple test corpus, I have extracted the information of who sent email to whom and represented it in table form as shown here: Kay Chris Sara Tana Steven Mark Kay 0 53 37 6 0 12 Chris 53 0 1 0 2 0 Sara 37 1 0 1144 0 962 Tana 6 0 1144 0 0 1201 Steven 0 0 2 0 0 0 Mark 12 0 962 1201 0 0 In the dataset provided on our website, I’ve extracted this information for 90 individuals. Suppose I am interested in clustering people to discover the relationships among these individuals. Link analysis d alysis devote called link an g in in d m an ta ) da of tities tire subfield hips among en en ns o an ti is la e re er g Th valuatin . of problem (e d to this task to this type rithms devote go al ed liz ia ec there are sp s You try Can you perform hierarchical clustering on the Enron email dataset? You can download the data from our website. (http://www.guidetodatamining.com. You may need to alter the code slightly to better match the problem. Good luck! 8-59 s You try - solution In the dataset provided on our website, I’ve extracted this information for 90 individuals. We are clustering the people based on similarity of email correspondence. If most of my email correspondence is with Ann, Ben and Clara, and most of yours is with these people as well, that provides evidence that we are in the same group. The idea is something like this: between -> Ann Ben Clara Dongmei Emily Frank my emails 127 25 119 5 1 6 your emails 172 35 123 7 3 5 Because our rows are similar, we cluster together. A problem arises when we add in our columns: between -> my emails your emails me you Ann Ben Clara Dongmei Emily Frank 2 190 127 25 119 5 1 6 190 3 172 35 123 7 3 5 In looking at the ‘me’ column, you corresponded with me 190 times but I only sent myself email twice. The ‘you’ column is similar. Now when we compare our rows they don’t look so similar. Before I included the ‘me’ and ‘you’ columns the Euclidean distance was 46 and after I included them it was 269! To avoid this problem when I compute the Euclidean distance between two people I eliminate the columns for those two people. This required a slight change to the distance formula: def distance(self, i, j): #enron specific distance formula sumSquares = 0 for k in range(1, self.cols): if (k != i) and (k != j) : sumSquares += (self.data[k][i] - self.data[k][j])**2 return math.sqrt(sumSquares) 8-60 CLUSTERING Here is a subtree of the results: cara.semperger@enron.com ------------------+ |--+ michelle.cash@enron.com ----------------+ | |--+ patrice.mims@enron.com -----------+ | |--+ | soblander@carrfut.com ---------+ | | | |--+ | | pete.davis@enron.com -------+ | | | |--+ | | judy.hernandez@enron.com ---+ | | |--+ mike.carson@enron.com ------------+ | |--+ chris.dorland@enron.com -------+ | |--+ benjamin.rogers@enron.com --+ | |--+ larry.campbell@enron.com ---+ I also performed k-means++ on the data, with k = 8. Here are some of the groups it discovered: Class 5 ======== chris.germany@enron.com scott.neal@enron.com marie.heard@enron.com leslie.hansen@enron.com mike.carson@enron.com Class 6 ======== sara.shackleton@enron.com mark.taylor@enron.com susan.scott@enron.com Class 7 ======== tana.jones@enron.com louise.kitchen@enron.com mike.grigsby@enron.com david.forster@enron.com m.presto@enron.com 8-61 These results are interesting. Class 5 contains a number of traders. Chris Germany and Leslie Hansen are traders. Scott Neal is a vice president of trading. Marie Heard is a lawyer. Mike Carson is a manager of South East trading. The members of Class 7 are also interesting. All I know about Tana Jones is that she is an ‘executive’. Louise Kitchen is President of online trading. Mike Grigsby was Vice President of Natural Gas. David Forster was a Vice President of trading. Kevin Presto (m.presto) was also a Vice President and a senior trader. There are many amazing hidden patterns in this Enron data. Can you find some? Download the complete data set and give it a try! (let me know what you find out) Or try your hand at clustering other datasets. Remember, practice makes the heart grow fonder And, hey, congratulations on getting to the end of this chapter! 8-62
Source Exif Data:
File Type : PDF File Type Extension : pdf MIME Type : application/pdf Linearized : Yes Warning : Bad CreationDate reference Modify Date : 2018:10:05 01:02:32Z Producer : 3-Heights(TM) PDF Optimization Shell 4.8.25.2 (http://www.pdf-tools.com) Page Count : 395 PDF Version : 1.4EXIF Metadata provided by EXIF.tools