![\"data-layout\"](\"https://raw.githubusercontent.com/astro-informatics/course_mlbd_images/master/Lecture16_Images/DecisionTree.jpg\")
\n",
"\n",
"[[Image source](https://inside-machinelearning.com/en/decision-tree-and-hyperparameters/)]"
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": "slide"
},
"tags": []
},
"source": [
"## Walk-through of decision tree\n",
"\n",
"Let's consider an illustration using the Iris Data set (introduced in Lecture 3)."
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"source": [
"### Images of different Iris species\n",
"\n",
"#### Iris Setosa\n",
"\n",
"![](\"https://raw.githubusercontent.com/astro-informatics/course_mlbd_images/master/Lecture03_Images/iris_setosa.jpg\")
\n",
"\n",
"#### Iris Versicolor\n",
"\n",
"![](\"https://raw.githubusercontent.com/astro-informatics/course_mlbd_images/master/Lecture03_Images/iris_versicolor.jpg\")
\n",
"\n",
"#### Iris Virginica\n",
"\n",
"![](\"https://raw.githubusercontent.com/astro-informatics/course_mlbd_images/master/Lecture03_Images/iris_virginica.jpg\")
\n",
"\n",
"[[Image source](https://github.com/jakevdp/sklearn_tutorial)]\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"source": [
"Load feature matrix, where each row correpsonds to an observed (*sampled*) flower, with a number of *features*, with corresponding target vector.\n",
"\n",
"Consider two features only for now (petal length and width)."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"editable": true,
"execution": {
"iopub.execute_input": "2025-02-27T23:21:19.532095Z",
"iopub.status.busy": "2025-02-27T23:21:19.531889Z",
"iopub.status.idle": "2025-02-27T23:21:20.377758Z",
"shell.execute_reply": "2025-02-27T23:21:20.377159Z"
},
"slideshow": {
"slide_type": ""
},
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"DecisionTreeClassifier(max_depth=2, random_state=42)In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
DecisionTreeClassifier(max_depth=2, random_state=42)
![\"data-layout\"](\"https://raw.githubusercontent.com/astro-informatics/course_mlbd_images/master/Lecture16_Images/iris_tree.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": ""
},
"tags": []
},
"source": [
"Tree consists of a number of nodes.\n",
"- Top node is the _root node_.\n",
"- Intermediate _split nodes_.\n",
"- Lower nodes are _leaf nodes_.\n",
"\n",
"Decisions based on *features* and *thresholds*.\n",
"\n",
"Navigate tree to make predictions."
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"source": [
"### Interpreting node outputs\n",
"\n",
"![\"data-layout\"](\"https://raw.githubusercontent.com/astro-informatics/course_mlbd_images/master/Lecture16_Images/iris_tree_1node.png\")
\n",
"\n",
"Arguments in the nodes are: \n",
"- Top argument shows the _threshold_ upon which the classification division was made.\n",
"- ```gini``` (see next slides) is a quantitative measure of impurity.\n",
"- ```samples``` denotes the number of training instances that satisfy the criteria.\n",
"- ```values``` denotes the number of training instances per class that satisfy the criteria.\n",
"- ```class``` prediction for the node."
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"source": [
"### Decision boundaries"
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": ""
},
"tags": []
},
"source": [
"![\"data-layout\"](\"https://raw.githubusercontent.com/astro-informatics/course_mlbd_images/master/Lecture16_Images/decision_tree_decision_boundaries_plot.png\")
\n",
"\n",
"We set ```max_depth=2```, so algorithm stopped after two divisions. "
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"source": [
"### Estimating class probabilities"
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": ""
},
"tags": []
},
"source": [
"Also want to know the _probability_ that an instance $i$ belongs to class $k$. \n",
"\n",
"Class probability founds by finding the leaf node for instance $i$, then returns ratio of training instances of class $k$ in this node.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"source": [
"#### Example \n",
"For case where flower has petals=5cm long and 1.5cm wide\n",
"corresponding leaf node is at depth-2 left node. \n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"source": [
"![\"data-layout\"](\"https://raw.githubusercontent.com/astro-informatics/course_mlbd_images/master/Lecture16_Images/iris_tree_1node2.png\")
\n",
"\n",
"\n",
"So probabilities are: 0% (Setosa), 49/54=90.7% (Versicolor), 5/54=9.3% (Virginica)."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"editable": true,
"execution": {
"iopub.execute_input": "2025-02-27T23:21:20.572369Z",
"iopub.status.busy": "2025-02-27T23:21:20.572178Z",
"iopub.status.idle": "2025-02-27T23:21:20.577551Z",
"shell.execute_reply": "2025-02-27T23:21:20.576948Z"
},
"slideshow": {
"slide_type": ""
},
"tags": []
},
"outputs": [
{
"data": {
"text/plain": [
"array([[0. , 0.907, 0.093]])"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"tree_clf.predict_proba([[5, 1.5]]).round(3)"
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": "slide"
},
"tags": []
},
"source": [
"## Quality measures"
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"source": [
"### Gini impurity\n",
"\n",
"Gini impurity is defined by\n",
"$$\n",
"G_i=1-\\sum_{k=1}^{n}p^2_{i,k} ,\n",
"$$\n",
"where $p_{i,k}$ is the ratio of class $k$ instances among training instances in the $i^{\\rm th}$ node. \n",
"\n",
"$G_i=0$ means the sample is 100% _pure_ i.e. all instances are in a single class. "
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"source": [
"#### Gini example calculation\n",
"\n",
"\n",
"\n",
"$$\n",
"G_i = 1 - (0/54)^2 - (49/54)^2 - (5/54)^2= 0.168\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"source": [
"### Entropy \n",
"\n",
"Alternative to Gini is to use entropy as the purity measure \n",
"$$\n",
"H_i=-\\sum_{k=1}^n p_{i,k}\\log_2(p_{i,k}),\n",
"$$\n",
"for $p_{i,k}\\not=0$. \n",
"\n",
"Measures the information content of variable (number of bits required to encode)."
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"source": [
"#### Entropy example calculation\n",
"\n",
"\n",
"\n",
"$$\n",
"H_i = - (49/54) \\log_2(49/54) - (5/54) \\log_2(5/54)= 0.445\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"source": [
"### Does it make a difference what quality measure is used? \n",
"\n",
"Not usually, although Gini tends to isolate the most frequent classes, and entropy leads to more \"balanced\" trees. Entropy is slightly more expensive to compute."
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"source": [
"### Decision tree using entropy criterion"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"editable": true,
"execution": {
"iopub.execute_input": "2025-02-27T23:21:20.579634Z",
"iopub.status.busy": "2025-02-27T23:21:20.579461Z",
"iopub.status.idle": "2025-02-27T23:21:20.585935Z",
"shell.execute_reply": "2025-02-27T23:21:20.585283Z"
},
"slideshow": {
"slide_type": ""
},
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"DecisionTreeClassifier(criterion='entropy', max_depth=2)In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
DecisionTreeClassifier(criterion='entropy', max_depth=2)
![\"data-layout\"](\"https://raw.githubusercontent.com/astro-informatics/course_mlbd_images/master/Lecture16_Images/iris_tree_entropy.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"source": [
"#### Gini tree (for reference)\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": "slide"
},
"tags": []
},
"source": [
"## CART training algorithm\n",
"\n",
"Classification And Regression Tree (CART) algorihtm can be used to train decision trees -- also called *growing trees* algorithm (used by SciKit Learn)."
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"source": [
"- Splits the sample into two subsets using a single feature $k$ at threshold $t_k$\n",
"- Chooses feature $k$ and threshold $t_k$ by finding pair that produces purest subset, weighted by their size.\n",
"\n",
"Cost function minimized for each split:\n",
"\n",
"$$\n",
"J(k,t_k)=\\frac{m_\\text{left}}{m}G_\\text{left}+\\frac{m_\\text{right}}{m}G_\\text{right} ,\n",
"$$\n",
"\n",
"where $G_\\text{left/right}$ measures the impurity of the left/right subset and $m_\\text{left/right}$ is the number of instances in the left/right subset."
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"source": [
"Note that the CART algorithm:\n",
"- Only splits data in two at each stage, i.e. is binary.\n",
"- Is a greedy algorithm. It searches for the optimal split at each level, then repeats for subsequent levels. There is no guarantee that the overall optimal tree is found. Nevertheless, usually produces a tree that is reasonably good."
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": "slide"
},
"tags": []
},
"source": [
"## Regularisation and hyperparameters\n",
"\n",
"Decision Trees are **non-parametric** classification algorithms since the number of parameters is not determined prior to training.\n",
"\n",
"Tends to overfit if not careful. "
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"source": [
"Need to regularise the problem. This can be done by the restricting the complexity of the tree, for example though the following SciKit Learn parameters.\n",
"\n",
"- `max_depth`: maximum depth of the tree.\n",
"- `max_features` maximum number of features used in splitting at each node.\n",
"- `max_leaf_nodes` maximum number of leaf nodes.\n",
"- `min_samples_split`: mimimum number of samples a node must have before it can be split.\n",
"- `min_samples_leaf`: minimum number a leaf can have to be created.\n",
"- `min_weight_fraction`: Same as `min_samples_leaf` but expressed as a fraction of total samples.\n",
"\n",
"Generally, increasing ```min_*``` hyperparameters or reducing ```max_*``` hyperparameters will regularise the model."
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"source": [
"Other algorithms _prune_, i.e. make a (relatively) unrestricted tree then remove statistically insignificant nodes."
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": "subslide"
},
"tags": [
"exercise_pointer"
]
},
"source": [
"**Exercises:** *You can now complete Exercise 1 in the exercises associated with this lecture.*"
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": "slide"
},
"tags": []
},
"source": [
"## Decision trees for regression \n",
"\n",
"Decision trees can also be used for regression tasks."
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"source": [
"### Train regression model"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"editable": true,
"execution": {
"iopub.execute_input": "2025-02-27T23:21:20.735593Z",
"iopub.status.busy": "2025-02-27T23:21:20.735405Z",
"iopub.status.idle": "2025-02-27T23:21:20.743106Z",
"shell.execute_reply": "2025-02-27T23:21:20.742456Z"
},
"slideshow": {
"slide_type": ""
},
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"DecisionTreeRegressor(max_depth=2, random_state=42)In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
DecisionTreeRegressor(max_depth=2, random_state=42)
![\"data-layout\"](\"https://raw.githubusercontent.com/astro-informatics/course_mlbd_images/master/Lecture16_Images/regression_tree.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"editable": true,
"slideshow": {
"slide_type": "subslide"
},
"tags": []
},
"source": [
"### Train a deeper model"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"editable": true,
"execution": {
"iopub.execute_input": "2025-02-27T23:21:20.898435Z",
"iopub.status.busy": "2025-02-27T23:21:20.898251Z",
"iopub.status.idle": "2025-02-27T23:21:20.904714Z",
"shell.execute_reply": "2025-02-27T23:21:20.904136Z"
},
"slideshow": {
"slide_type": ""
},
"tags": []
},
"outputs": [
{
"data": {
"text/html": [
"DecisionTreeRegressor(max_depth=3, random_state=42)In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
DecisionTreeRegressor(max_depth=3, random_state=42)