27.2: Decision Trees

Last updated
Save as PDF

Page ID: 88803

\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \) \( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)\(\newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\) \( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\) \( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\) \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\) \( \newcommand{\Span}{\mathrm{span}}\) \(\newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\) \( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\) \( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\) \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\) \( \newcommand{\Span}{\mathrm{span}}\)\(\newcommand{\AA}{\unicode[.8,0]{x212B}}\)

First, let’s get our head around what a decision tree is. Our inaugural example is shown in Figure 27.2.1. The first thing you’ll notice is that it has a branching structure that branches...down. I’m not sure why Data Scientists draw trees growing down while the rest of the world (including trees themselves: look outside if you don’t believe me) has them growing up, but this is the convention so we’ll just deal with it. To make it even more comical, the oval at the top of the tree is called the root of the tree. Really.

Continuing full bore with the botany analogy, the lines connecting the various shapes are, as you might suspect, called branches, and the darker rectangles are called leaves. One non-botanic bit of lingo is the name for the other ovals: they’re called nodes.

Figure \(\PageIndex(1)\): A decision tree (not a particularly good one, as it’ll turn out) for the videogame data set.

Okay. Now what does a decision tree “mean?” Geekily-put, it’s the pictorial codification of an algorithm for classification. Not so geekily, it’s a map that tells your classifier what rules to follow as it forms its prediction for an example data point. You simply start at the root, considering feature values at each node, and following the matching branch down the tree. When you reach a leaf, the prediction you give is written on the leaf node. It’s that simple.

First example: suppose we have a 24-year-old male Psychology major. We want to know whether he’s likely to play videogames. The decision tree in Figure 27.2.1 tells us to first consider his Major, since that’s the root. Now because this guy’s major is PSYC, we take the left branch and are immediately done: we’ve already reached a leaf. Our prediction for this guy will be No, he probably doesn’t play videogames.
Second example: we have an 18-year-old Math major who doesn’t identify with either of the binary genders. Starting again at the root, we now follow the middle branch for MATH. Now, we look at the person’s Gender. Since it is O, we follow the right branch, and give a prediction of Yes: we predict they do play videogames.
Third example: we now have a 22-year-old female Computer Science major. Do we think she would play videogames? The root tells us to look at her Major first, which means we go right; then we look at her Age, and since she’s positively ancient we go right again; and finally, her Gender tells us to predict No, she’s probably not a gamer.

Most students find this process very straightforward. In the next chapter, we’ll look at two key questions: first, how to turn a diagram like Figure 27.2.1 into Python code? And second, what’s the best way to make a good tree – i.e., one that makes as many successful predictions as possible?