spce0038-machine-learning-with-big-data/week6/slides/Lecture17_EnsembleRFs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# Lecture 17: Ensemble learning and random forests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "skip"
    },
    "tags": []
   },
   "source": [
    "![](https://www.tensorflow.org/images/colab_logo_32px.png)\n",
    "[Run in colab](https://colab.research.google.com/drive/1SNSqzKBuOnHfK_G_agJQtmE8y3WM2EhI)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-02-27T23:21:28.589474Z",
     "iopub.status.busy": "2025-02-27T23:21:28.589245Z",
     "iopub.status.idle": "2025-02-27T23:21:28.595460Z",
     "shell.execute_reply": "2025-02-27T23:21:28.594870Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last executed: 2025-02-27 23:21:28\n"
     ]
    }
   ],
   "source": [
    "import datetime\n",
    "now = datetime.datetime.now()\n",
    "print(\"Last executed: \" + now.strftime(\"%Y-%m-%d %H:%M:%S\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Ensemble learning overview\n",
    "\n",
    "- Leverage *wisdom of the crowd*.\n",
    "- The average from many predictors (or people) may be more accurate that the result from any single given predictor.\n",
    "- Group of predictors called an *ensemble*.\n",
    "- Even if individual predictors/classifiers are *weak* (only slightly better than random), an ensemble can be *strong* (high accuracy)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "Two broad approaches:\n",
    "- Different predictors.\n",
    "- Same predictor, different training sets.\n",
    "\n",
    "An example of ensemble learning is *Random Forests* where *Decision Trees* are trained on random _*subsets*_ of the training data and then for each sample the assigned class is the one that gets the most *votes* from the ensemble."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Voting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Consider a number of classifiers.\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/astro-informatics/course_mlbd_images/master/Lecture17_Images/votes1.png\" alt=\"data-layout\" width=\"700\" style=\"display:block; margin:auto\"/>\n",
    "\n",
    "[Source: Geron]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Majority wins (hard voting) \n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/astro-informatics/course_mlbd_images/master/Lecture17_Images/votes2.png\" alt=\"data-layout\" width=\"700\" style=\"display:block; margin:auto\"/>\n",
    "\n",
    "[Source: Geron]\n",
    "\n",
    "*Majorty wins* is often called *hard voting*.\n",
    "\n",
    "Also _soft_ voting (see upcoming slides and example)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-02-27T23:21:28.628304Z",
     "iopub.status.busy": "2025-02-27T23:21:28.628127Z",
     "iopub.status.idle": "2025-02-27T23:21:29.475994Z",
     "shell.execute_reply": "2025-02-27T23:21:29.475366Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Build a voting classifier in Scikit using three weaker classifiers\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.datasets import make_moons\n",
    "\n",
    "# Use moons dataset\n",
    "X, y = make_moons(n_samples=500, noise=0.30, random_state=42) #X(features), y(classifications)\n",
    "\n",
    "# Split into training and test data \n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42) \n",
    "\n",
    "# Load three different classification algorithms, and initialise \n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.svm import SVC\n",
    "log_clf = LogisticRegression(random_state=42)\n",
    "rnd_clf = RandomForestClassifier(random_state=42)\n",
    "svm_clf = SVC(probability=True, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-02-27T23:21:29.478273Z",
     "iopub.status.busy": "2025-02-27T23:21:29.478018Z",
     "iopub.status.idle": "2025-02-27T23:21:29.628799Z",
     "shell.execute_reply": "2025-02-27T23:21:29.628226Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import VotingClassifier\n",
    "\n",
    "# Voting classifier \"=\" logistic + random forest + SVC\n",
    "# Set up the voting classifier\n",
    "voting_clf = VotingClassifier(\n",
    "        estimators=[('lr', log_clf), ('rf', rnd_clf), ('svc', svm_clf)],\n",
    "        voting='soft')\n",
    "\n",
    "# Classify using voting classifier (auto uses all the assigned invidiual classifiers)\n",
    "voting_clf.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-02-27T23:21:29.630674Z",
     "iopub.status.busy": "2025-02-27T23:21:29.630486Z",
     "iopub.status.idle": "2025-02-27T23:21:29.936574Z",
     "shell.execute_reply": "2025-02-27T23:21:29.936202Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LogisticRegression 0.864\n",
      "RandomForestClassifier 0.896\n",
      "SVC 0.896\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "VotingClassifier 0.92\n"
     ]
    }
   ],
   "source": [
    "# Let's see how each individual classifier did:\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "for clf in (log_clf, rnd_clf, svm_clf, voting_clf): #loop over classifiers\n",
    "    clf.fit(X_train, y_train) #fit each one individually \n",
    "    y_pred = clf.predict(X_test) #make prediction \n",
    "    print(clf.__class__.__name__, accuracy_score(y_test, y_pred)) #print the score    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Voting classifier did better than 3 individual ones!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Hard vs soft voting\n",
    "\n",
    "* Hard voting: select class with highest frequency across predictors, without weight taken into account.\n",
    "* Soft voting: if all classifiers can estimate class probabilities (i.e. they have a `predict_proba()` method), predict overall class probability, averaged over all individual classifiers, and select highest. \n",
    "\n",
    "Soft voting is often better than hard voting because it gives more weight to highly confident, but perhaps less frequent, votes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Bagging and pasting \n",
    "\n",
    "Instead of using different predictors, we can use same the predictor but different training sets. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "<img src=\"https://raw.githubusercontent.com/astro-informatics/course_mlbd_images/master/Lecture17_Images/bagging.png\" alt=\"data-layout\" width=\"700\" style=\"display:block; margin:auto\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "Two ways to create new training sets: \n",
    "- When sampling is performed *with replacement*, this method is called *bagging* (short for bootstrap aggregating).\n",
    "- When sampling is performed *without replacement*, it is called *pasting*.\n",
    "\n",
    "A big advantage is that trivially parallisable across nodes/CPU."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Bagging \n",
    "\n",
    "Given a standard training set $D$ of size $n$, **bagging** generates $m$ new training sets $D_{i}$, each of size $n'<n$, by sampling from $D$ uniformly and **with replacement**. \n",
    "\n",
    "By sampling *with replacement*, some observations may be repeated in each $D_{i}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Pasting\n",
    "\n",
    "Given a standard training set $D$ of size $n$, **pasting** generates $m$ new training sets $D_{i}$, each of size $n'<n$, by sampling from $D$ uniformly and **without replacement**. \n",
    "\n",
    "By sampling *without replacement*, observations **not** repeated in each $D_{i}$.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-02-27T23:21:29.938737Z",
     "iopub.status.busy": "2025-02-27T23:21:29.938552Z",
     "iopub.status.idle": "2025-02-27T23:21:29.944402Z",
     "shell.execute_reply": "2025-02-27T23:21:29.944039Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.856\n"
     ]
    }
   ],
   "source": [
    "# Example of Decision Trees using moons data from before\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "# Make a single prediction using a decision tree\n",
    "tree_clf = DecisionTreeClassifier(random_state=42) #setup classifier\n",
    "tree_clf.fit(X_train, y_train) #train\n",
    "y_pred_tree = tree_clf.predict(X_test) #predict\n",
    "print(accuracy_score(y_test, y_pred_tree)) #test score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-02-27T23:21:29.946181Z",
     "iopub.status.busy": "2025-02-27T23:21:29.946008Z",
     "iopub.status.idle": "2025-02-27T23:21:30.703206Z",
     "shell.execute_reply": "2025-02-27T23:21:30.702590Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.904\n"
     ]
    }
   ],
   "source": [
    "# SciKit Learn provides an easy way to do this using BaggingClassifier \n",
    "from sklearn.ensemble import BaggingClassifier #load the bagging classifier \n",
    "\n",
    "# Train ensemble of 500 Decision Tree classifiers\n",
    "# each using 100 training instances - randomly sampled from training set\n",
    "# with replacement. \n",
    "\n",
    "bag_clf = BaggingClassifier(\n",
    "        DecisionTreeClassifier(random_state=42), #classifier to use\n",
    "        n_estimators=500, #number of seperate classifiers\n",
    "        max_samples=100, #number of training instances \n",
    "        bootstrap=True, # set to False for pasting instead of bagging.\n",
    "        n_jobs=1, #number of cores to use -1 is all\n",
    "        random_state=42)\n",
    "\n",
    "bag_clf.fit(X_train, y_train) #fit data \n",
    "y_pred = bag_clf.predict(X_test) #compare to prediction \n",
    "print(accuracy_score(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Accuracy of bagging increased compared to a single classifier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "#### Plot decision boundary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-02-27T23:21:30.705203Z",
     "iopub.status.busy": "2025-02-27T23:21:30.705025Z",
     "iopub.status.idle": "2025-02-27T23:21:30.791583Z",
     "shell.execute_reply": "2025-02-27T23:21:30.790992Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from matplotlib.colors import ListedColormap\n",
    "\n",
    "def plot_decision_boundary(handle, clf, X, y, axes=[-1.5, 2.5, -1, 1.5], alpha=0.5, contour=True):\n",
    "    x1s = np.linspace(axes[0], axes[1], 100)\n",
    "    x2s = np.linspace(axes[2], axes[3], 100)\n",
    "    x1, x2 = np.meshgrid(x1s, x2s)\n",
    "    X_new = np.c_[x1.ravel(), x2.ravel()]\n",
    "    y_pred = clf.predict(X_new).reshape(x1.shape)\n",
    "    custom_cmap = ListedColormap(['#fafab0','#9898ff','#a0faa0'])\n",
    "    handle.contourf(x1, x2, y_pred, alpha=0.3, cmap=custom_cmap)\n",
    "    if contour:\n",
    "        custom_cmap2 = ListedColormap(['#7d7d58','#4c4c7f','#507d50'])\n",
    "        handle.contour(x1, x2, y_pred, cmap=custom_cmap2, alpha=0.8)\n",
    "    handle.plot(X[:, 0][y==0], X[:, 1][y==0], \"yo\", alpha=alpha)\n",
    "    handle.plot(X[:, 0][y==1], X[:, 1][y==1], \"bs\", alpha=alpha)\n",
    "    handle.axis(axes)\n",
    "    handle.set_xlabel(r\"$x_1$\", fontsize=18)\n",
    "    handle.set_ylabel(r\"$x_2$\", fontsize=18, rotation=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-02-27T23:21:30.793758Z",
     "iopub.status.busy": "2025-02-27T23:21:30.793528Z",
     "iopub.status.idle": "2025-02-27T23:21:31.420059Z",
     "shell.execute_reply": "2025-02-27T23:21:31.419539Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA7EAAAGTCAYAAAD+2JVNAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQABAABJREFUeJzsvXecXFd99/++ffpWSSutimXJci/IYHDDBgymmZhmSoILxECCQxLg4QnPC1IJfhKSh/4LJBTTQjHVBDAYjIMNMRjj3qSVZJXVarV16p1bz++PKdoyszu7O7P1vF8vYXbmlnPPvXM+53zvtyhCCIFEIpFIJBKJRCKRSCQrAHWpGyCRSCQSiUQikUgkEkmjyEWsRCKRSCQSiUQikUhWDHIRK5FIJBKJRCKRSCSSFYNcxEokEolEIpFIJBKJZMUgF7ESiUQikUgkEolEIlkxyEWsRCKRSCQSiUQikUhWDHIRK5FIJBKJRCKRSCSSFYNcxEokEolEIpFIJBKJZMUgF7ESiUQikUgkEolEIlkxyEWsRLKKePrpp1EUhVtuuWVO+11++eVcfvnlLWmTRCKRSFYWUktWDieddBLXX399w9u+/OUvb22DlgF/+7d/i6IoS90MSYuRi1iJpInccsstKIpS/ReJRNi0aRNXXnklH//4x8lms0vdxGXDSSedNKmv6v2b6yRKIpFIVjpSSxpHaslkHn/8cf72b/+Wp59+uunHntqn8XicM844gw9+8IMUCoWmn08imQlFCCGWuhESyWrhlltu4YYbbuDv//7v2b59O57ncezYMe666y7uuOMOtm7dym233cY555zTkvMLIXAcB8Mw0DSt4f1c1wXANM2WtKsW3/ve98jlctW/f/SjH/G1r32Nj3zkI3R3d1c/v+iiizj55JMXrV0SiUSy1EgtaZy1riWO46CqKoZhAPCtb32L1772tfziF7+Y9lb8pJNO4qyzzuK//uu/5nUuRVF44QtfyLXXXgtALpfj7rvv5j//8z95zWtew6233rqga2kWvu/j+z6RSGSpmyJpIfpSN0AiWY285CUv4ZnPfGb17/e9733ceeedvPzlL+cVr3gFTzzxBNFotOnnrVjs58piTjgqXH311ZP+PnbsGF/72te4+uqrOemkk+rul8/nicfjrW2cRCKRLAOklszOWtcSy7IW9Xy7du3ij/7oj6p/v/3tb8d1Xb7zne9QLBaXxcJR13V0XS5xVjvSnVgiWSSe//zn84EPfICDBw/yla98ZdJ3Tz75JK95zWvo7OwkEonwzGc+k9tuu23aMcbHx/nLv/xLTjrpJCzLYvPmzVx77bUMDw8DteOYjh07xg033MDmzZuxLIuNGzfyB3/wB5NcjWrFMR0/fpy3vOUtbNiwgUgkwrnnnssXv/jFSdtUzvcv//Iv/Pu//zs7duzAsiye9axncd999y2sw4Drr7+eRCLBvn37eOlLX0oymeQP//APAQjDkI9+9KOceeaZRCIRNmzYwNve9jbGxsamHefHP/4xl156KfF4nGQyycte9jIee+yxBbdPIpFIFhupJXNnMbWkkX6aym233YaiKDz88MPVz7797W+jKAqvetWrJm17+umn87rXva7698SY2FtuuYXXvva1ADzvec+ruv3eddddk45xzz33cMEFFxCJRDj55JP50pe+NGsfzkRPTw+KokxaON5999289rWvZevWrViWxZYtW/jLv/xLbNuetv+tt97KGWecQSQS4ayzzuK73/0u119//TQjxMjICG9605tIpVK0t7dz3XXX8dBDD017VmvFxCqKwk033cT3vvc9zjrrLCzL4swzz+T222+f1p677rqLZz7zmUQiEXbs2MFnPvMZGWe7DJFmColkEXnTm97E//k//4ef/vSn3HjjjQA89thjXHzxxfT29vJXf/VXxONxvvnNb3L11Vfz7W9/m1e+8pVAyW3n0ksv5YknnuDNb34zu3fvZnh4mNtuu40jR45McpuayKtf/Woee+wx/uzP/oyTTjqJ48ePc8cdd3Do0KG6Vmrbtrn88svp6+vjpptuYvv27dx6661cf/31jI+P8+d//ueTtv/P//xPstksb3vb21AUhX/+53/mVa96Ffv376+6OM0X3/e58sorueSSS/iXf/kXYrEYAG9729uqLnfvfOc7OXDgAJ/85Cd54IEH+NWvflU975e//GWuu+46rrzySv7pn/6JQqHAv/3bv3HJJZfwwAMPzGipl0gkkuWI1JK5s1haMp9+uuSSS1AUhV/+8pdVF/G7774bVVW55557qtsNDQ3x5JNPctNNN9U8znOf+1ze+c538vGPf5z/83/+D6effjpA9b8AfX19vOY1r+Etb3kL1113HZ///Oe5/vrrOf/88znzzDNn7cdisVg1duTzeX71q1/xxS9+kTe+8Y2TFrG33norhUKBP/mTP6Grq4vf/va3fOITn+DIkSOT3I5/+MMf8rrXvY6zzz6bm2++mbGxMd7ylrfQ29s76bxhGHLVVVfx29/+lj/5kz/htNNO4/vf/z7XXXfdrG2ucM899/Cd73yHP/3TPyWZTPLxj3+cV7/61Rw6dIiuri4AHnjgAV784hezceNG/u7v/o4gCPj7v/971q1b1/B5JIuEkEgkTeMLX/iCAMR9991Xd5u2tjbxjGc8o/r3C17wAnH22WeLYrFY/SwMQ3HRRReJU045pfrZX//1XwtAfOc735l2zDAMhRBCHDhwQADiC1/4ghBCiLGxMQGID3/4wzO2+7LLLhOXXXZZ9e+PfvSjAhBf+cpXqp+5risuvPBCkUgkRCaTmXS+rq4uMTo6Wt32+9//vgDED37wgxnPO5EPf/jDAhAHDhyofnbdddcJQPzVX/3VpG3vvvtuAYivfvWrkz6//fbbJ32ezWZFe3u7uPHGGydtd+zYMdHW1jbtc4lEIlkOSC0psdK0pNF+qsWZZ54prrnmmurfu3fvFq997WsFIJ544gkhhBDf+c53BCAeeuih6nbbtm0T1113XfXvW2+9VQDiF7/4xbRzbNu2TQDil7/8ZfWz48ePC8uyxLvf/e5Z2wjU/Hf11VdPeu6EEKJQKEzb/+abbxaKooiDBw9WPzv77LPF5s2bRTabrX521113CUBs27at+tm3v/1tAYiPfvSj1c+CIBDPf/7zJz2rQgjxN3/zN2LqEgcQpmmKvr6+6mcPPfSQAMQnPvGJ6mdXXXWViMVior+/v/rZ3r17ha7r044pWVqkO7FEssgkEolqZsnR0VHuvPNOrrnmGrLZLMPDwwwPDzMyMsKVV17J3r176e/vB0quReeee27Vmj6Rei4u0WgU0zS56667arpG1eNHP/oRPT09vOENb6h+ZhgG73znO8nlcvz3f//3pO1f97rX0dHRUf370ksvBWD//v0Nn3Mm/uRP/mTS37feeittbW288IUvrPbZ8PAw559/PolEgl/84hcA3HHHHYyPj/OGN7xh0naapvHsZz+7up1EIpGsNKSWzJ1Wa8l8+wlK13r33XcDkM1meeihh3jrW99Kd3d39fO7776b9vZ2zjrrrHn3wRlnnFHtV4B169Zx6qmnNtzHf/AHf8Add9zBHXfcwfe//33e9773cfvtt/PGN74RMSFX7MRY7Xw+z/DwMBdddBFCCB544AEAjh49yiOPPMK1115LIpGobn/ZZZdx9tlnTzrv7bffjmEYVc8DAFVVecc73tHwtV9xxRXs2LGj+vc555xDKpWqXnsQBPzsZz/j6quvZtOmTdXtdu7cyUte8pKGzyNZHKQ7sUSyyORyOdavXw+U3HqEEHzgAx/gAx/4QM3tjx8/Tm9vL/v27ePVr371nM5lWRb/9E//xLvf/W42bNjAc57zHF7+8pdz7bXX0tPTU3e/gwcPcsopp6Cqk+1cFZekgwcPTvp869atk/6uTELmKuK10HWdzZs3T/ps7969pNPpaj9O5fjx49XtoBRDVotUKrXg9kkkEslSILVkbiyGlsy3n6C0iP30pz9NX18f+/btQ1EULrzwwuri9sYbb+Tuu+/m4osvntafc2FqH0Opnxvt482bN3P
      "text/plain": [
       "<Figure size 1100x400 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "_, (ax1, ax2) = plt.subplots(1, 2, figsize=(11, 4))\n",
    "\n",
    "plot_decision_boundary(ax1, tree_clf, X, y), ax1.set_title(\"Decision Tree\")\n",
    "\n",
    "plot_decision_boundary(ax2, bag_clf, X, y), ax2.set_title(\"Decision Trees with Bagging\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Bagging versus pasting\n",
    "\n",
    "* Bootstrapping introduces a bit more diversity in the subsets that each predictor is trained on, so bagging ends up with a slightly higher bias than pasting.\n",
    "* But this also means that predictors end up being less correlated so the ensemble’s variance is reduced.\n",
    "* Overall, bagging often results in better models, which explains why it is generally preferred.\n",
    "* However, if you have spare time and CPU power you can use cross-validation to evaluate both bagging and pasting and select the one that works best.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Out-of-bag evaluation\n",
    "\n",
    "* With bagging, some instances may be sampled several times for any given predictor, while others may not be sampled at all. \n",
    "* By default a ```BaggingClassifier``` samples $m$ training instances with replacement (```bootstrap = True```), where $m$ is the size of the training set.\n",
    "* Only about ~63% of the training instances are sampled on average for any given predictor.\n",
    "\n",
    "* The remaining 37% of the training instances that are not sampled are called _out-of-bag (oob) instances_. Note that they are not the same 37% for all predictors.\n",
    "\n",
    "Since a predictor never sees the oob instances during training, it can be used to give a clean evaluation on the oob instances, without the need for a validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-02-27T23:21:31.422340Z",
     "iopub.status.busy": "2025-02-27T23:21:31.421969Z",
     "iopub.status.idle": "2025-02-27T23:21:34.657016Z",
     "shell.execute_reply": "2025-02-27T23:21:34.656250Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8986666666666666"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use oob_score=True in Scikit to do automatic oob evaluation after training.\n",
    "bag_clf = BaggingClassifier(\n",
    "    DecisionTreeClassifier(),\n",
    "    n_estimators=1000,\n",
    "    bootstrap=True, #using replacement \n",
    "    n_jobs=-1,\n",
    "    oob_score=True\n",
    ")\n",
    "bag_clf.fit(X_train, y_train)\n",
    "bag_clf.oob_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Should expect about 90% accuracy on the test set.  Let's see..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-02-27T23:21:34.659228Z",
     "iopub.status.busy": "2025-02-27T23:21:34.659028Z",
     "iopub.status.idle": "2025-02-27T23:21:34.984277Z",
     "shell.execute_reply": "2025-02-27T23:21:34.983579Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.88"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "y_pred = bag_clf.predict(X_test)\n",
    "accuracy_score(y_test,y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Sampling features\n",
    "\n",
    "Instead of just sampling training instances, we can also randomly sample the (set of) features themselves."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Random patches\n",
    "\n",
    "Sampling *both* instances and features is known as **random patches** method.\n",
    "\n",
    "In SciKit-Learn this is done by using the `max_features` and `bootstrap_features=True` key words in `BaggingClassifier`.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Random sub-spaces\n",
    "\n",
    "Keeping all training instances (i.e., ```bootstrap = False``` and ```max_samples = 1.0```) but sampling features only (i.e., ```bootstrap_features = True``` and/ or ```max_features``` smaller than 1.0) is called the **random subspaces** method."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Random forests\n",
    "\n",
    "A random forest is an ensemble of decision trees.\n",
    "\n",
    "We have actually been manually creating these in the previous examples explicitly!\n",
    "\n",
    "But we can use built-in SciKit-Learn functionality.\n",
    "* ```RandomForestClassifier```: specially designed for classification.\n",
    "* ```RandomForestRegressor```: specially designed for regression.\n",
    "\n",
    "Typically trained via bagging, although alternatives can be considered."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train a random forest directly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-02-27T23:21:34.986482Z",
     "iopub.status.busy": "2025-02-27T23:21:34.986292Z",
     "iopub.status.idle": "2025-02-27T23:21:35.726270Z",
     "shell.execute_reply": "2025-02-27T23:21:35.725659Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "rnd_clf = RandomForestClassifier(n_estimators=500, max_leaf_nodes=16,\n",
    "                                 n_jobs=-1, random_state=42)\n",
    "rnd_clf.fit(X_train, y_train)\n",
    "y_pred_rf = rnd_clf.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Equivalane to a bag of decision trees."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-02-27T23:21:35.728677Z",
     "iopub.status.busy": "2025-02-27T23:21:35.728236Z",
     "iopub.status.idle": "2025-02-27T23:21:35.733155Z",
     "shell.execute_reply": "2025-02-27T23:21:35.732576Z"
    }
   },
   "outputs": [],
   "source": [
    "bag_clf = BaggingClassifier(\n",
    "    DecisionTreeClassifier(max_features=\"sqrt\", max_leaf_nodes=16),\n",
    "    n_estimators=500, n_jobs=-1, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check consistent predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-02-27T23:21:35.734992Z",
     "iopub.status.busy": "2025-02-27T23:21:35.734794Z",
     "iopub.status.idle": "2025-02-27T23:21:36.552229Z",
     "shell.execute_reply": "2025-02-27T23:21:36.551813Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "np.True_"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bag_clf.fit(X_train, y_train)\n",
    "y_pred_bag = bag_clf.predict(X_test)\n",
    "np.all(y_pred_bag == y_pred_rf)  # same predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Extremely randomized trees\n",
    "\n",
    "When you are growing a tree in a random forest, at each node only a random subset of the features is considered for splitting (as discussed earlier). \n",
    "\n",
    "It is possible to make trees even more random by also using *random thresholds* for each feature rather than searching for the *best possible thresholds* (like regular decision trees do).\n",
    "\n",
    "Implemented using `ExtraTreesClassifier`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-02-27T23:21:36.554188Z",
     "iopub.status.busy": "2025-02-27T23:21:36.554005Z",
     "iopub.status.idle": "2025-02-27T23:21:37.131434Z",
     "shell.execute_reply": "2025-02-27T23:21:37.130848Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.912\n"
     ]
    }
   ],
   "source": [
    "from sklearn.ensemble import ExtraTreesClassifier\n",
    "\n",
    "rnd_clf = ExtraTreesClassifier(\n",
    "    n_estimators=500, \n",
    "    max_leaf_nodes=16, \n",
    "    n_jobs=-1, \n",
    "    random_state=42)\n",
    "\n",
    "rnd_clf.fit(X_train, y_train)\n",
    "y_pred_et = rnd_clf.predict(X_test)\n",
    "print(accuracy_score(y_test, y_pred_et))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Feature importance\n",
    "\n",
    "Can measure relative feature importance by looking at how much the tree nodes that use that feature reduce impurity on average.\n",
    "\n",
    "Specifically, a weighted average is computed, where each node's weight is equal to the number of associated training samples."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "\n",
    "Can access using ```feature_importances_``` variable.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-02-27T23:21:37.133484Z",
     "iopub.status.busy": "2025-02-27T23:21:37.133307Z",
     "iopub.status.idle": "2025-02-27T23:21:37.846441Z",
     "shell.execute_reply": "2025-02-27T23:21:37.845846Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sepal length (cm) = 0.11249225099876375\n",
      "sepal width (cm) = 0.02311928828251033\n",
      "petal length (cm) = 0.4410304643639577\n",
      "petal width (cm) = 0.4233579963547682\n"
     ]
    }
   ],
   "source": [
    "# Rank features by importance in iris data (note switching to Iris from Moons)\n",
    "\n",
    "from sklearn.datasets import load_iris\n",
    "iris = load_iris()\n",
    "\n",
    "X=iris[\"data\"]\n",
    "y=iris[\"target\"]\n",
    "\n",
    "rnd_clf = RandomForestClassifier(\n",
    "    n_estimators=500, \n",
    "    n_jobs=-1, \n",
    "    random_state=42)\n",
    "\n",
    "rnd_clf.fit(X,y)\n",
    "\n",
    "# Print out the importances \n",
    "for name, importance in zip(iris[\"feature_names\"], rnd_clf.feature_importances_): \n",
    "        print(name, \"=\", importance) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Boosting\n",
    "\n",
    "General term to mean combining several weak learners into a single strong learner.\n",
    "\n",
    "The general idea of most boosting methods is to train predictors sequentially, each trying to correct its predecessor. There are many boosting methods available but the most popular are *AdaBoost* (adaptive boosting) and *gradient boosting*. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### AdaBoost\n",
    "\n",
    "Give greater weight to training instances that were underfitted in predecessor.\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/astro-informatics/course_mlbd_images/master/Lecture17_Images/adaboost.png\" alt=\"data-layout\" width=\"600\" style=\"display:block; margin:auto\"/>\n",
    "\n",
    "[Source: Geron]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "-"
    },
    "tags": []
   },
   "source": [
    "AdaBoost pays more attention to training instances that predecessor underfitted, which forces new predictors to concentrate more on the \"hard cases\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "Disadvantage: results depend on previous classifier (sequential), so algorithm cannot be parallelized. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Gradient boosting \n",
    "\n",
    "Just like AdaBoost, Gradient Boosting works by sequentially adding predictors to an ensemble, each one correcting its predecessor. \n",
    "\n",
    "However, instead of tweaking the instance weights at every iteration like AdaBoost does, gradient boosting tries to fit the new predictor to the *residual error* made by the previous predictor."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Example\n",
    "\n",
    "Consider an example using decision trees as the base predictors for a regression problem. This is called Gradient Tree Boosting or Gradient Boosted Regression Trees (GBRT).\n",
    "\n",
    "First, let's implement this by hand."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-02-27T23:21:37.848645Z",
     "iopub.status.busy": "2025-02-27T23:21:37.848463Z",
     "iopub.status.idle": "2025-02-27T23:21:37.852092Z",
     "shell.execute_reply": "2025-02-27T23:21:37.851497Z"
    },
    "slideshow": {
     "slide_type": "-"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.tree import DecisionTreeRegressor\n",
    "\n",
    "# Training set: a noisy quadratic function\n",
    "np.random.seed(42)\n",
    "X = np.random.rand(100, 1) - 0.5\n",
    "y = 3*X[:, 0]**2 + 0.05 * np.random.randn(100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-02-27T23:21:37.853845Z",
     "iopub.status.busy": "2025-02-27T23:21:37.853671Z",
     "iopub.status.idle": "2025-02-27T23:21:37.857535Z",
     "shell.execute_reply": "2025-02-27T23:21:37.856961Z"
    },
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# Train Regressor\n",
    "tree_reg1 = DecisionTreeRegressor(max_depth=2, random_state=42)\n",
    "tree_reg1.fit(X, y);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-02-27T23:21:37.859257Z",
     "iopub.status.busy": "2025-02-27T23:21:37.859084Z",
     "iopub.status.idle": "2025-02-27T23:21:37.863265Z",
     "shell.execute_reply": "2025-02-27T23:21:37.862677Z"
    }
   },
   "outputs": [],
   "source": [
    "# now train 2nd Regressor using errors made by 1st one.\n",
    "y2 = y - tree_reg1.predict(X) #residual from the first fit (data-prediction)\n",
    "tree_reg2 = DecisionTreeRegressor(max_depth=2, random_state=42)\n",
    "tree_reg2.fit(X, y2);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-02-27T23:21:37.865063Z",
     "iopub.status.busy": "2025-02-27T23:21:37.864868Z",
     "iopub.status.idle": "2025-02-27T23:21:37.869942Z",
     "shell.execute_reply": "2025-02-27T23:21:37.869357Z"
    }
   },
   "outputs": [],
   "source": [
    "# now train 3rd Regressor using errors made by 2nd one.\n",
    "y3 = y2 - tree_reg2.predict(X)\n",
    "tree_reg3 = DecisionTreeRegressor(max_depth=2, random_state=42)\n",
    "tree_reg3.fit(X, y3)\n",
    "\n",
    "X_new = np.array([[0.8]])\n",
    "\n",
    "# now have ensemble w/ three trees.\n",
    "y_pred = sum(tree.predict(X_new) for tree in (tree_reg1, tree_reg2, tree_reg3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-02-27T23:21:37.871650Z",
     "iopub.status.busy": "2025-02-27T23:21:37.871479Z",
     "iopub.status.idle": "2025-02-27T23:21:37.875614Z",
     "shell.execute_reply": "2025-02-27T23:21:37.875034Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def plot_predictions(\n",
    "    regressors, X, y, axes,\n",
    "    ax,\n",
    "    label=None, \n",
    "    style=\"r-\", \n",
    "    data_style=\"b.\", \n",
    "    data_label=None):\n",
    "    \n",
    "    x1 = np.linspace(axes[0], axes[1], 500)\n",
    "    \n",
    "    y_pred = sum(\n",
    "        regressor.predict(x1.reshape(-1, 1)) for regressor in regressors)\n",
    "            \n",
    "    ax.plot(X[:, 0], y, data_style, label=data_label)\n",
    "    ax.plot(x1, y_pred, style, linewidth=2, label=label)\n",
    "    if label or data_label:\n",
    "        ax.legend(loc=\"upper center\", fontsize=16)\n",
    "    ax.axis(axes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-02-27T23:21:37.877341Z",
     "iopub.status.busy": "2025-02-27T23:21:37.877169Z",
     "iopub.status.idle": "2025-02-27T23:21:38.520264Z",
     "shell.execute_reply": "2025-02-27T23:21:38.519612Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA74AAAOsCAYAAABtYGqRAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQABAABJREFUeJzs3X18zXX/B/DXOWf3dmdmN7bZyJCbEDZzT6vpxl1kbtJo4SpcLkoojKu6dIN0lXLTQr/qIrqUIiptlUxiXCkqYhgbRm0zzHbO5/fHcc7O2bk/O/fn9Xw8zoN9z/fmcz7n5v19f7+fG4kQQoCIiIiIiIjIQ0mdXQAiIiIiIiIie2LiS0RERERERB6NiS8RERERERF5NCa+RERERERE5NGY+BIREREREZFHY+JLREREREREHo2JLxEREREREXk0Jr5ERERERETk0Zj4EhERERERkUdj4uumkpKSIJFItB7+/v6Ij4/H0KFD8dlnnzmtbPn5+ZBIJOjfv7/F26peizMVFRVBIpEgKSnJqeVwpPXr10MikWDChAnOLgpZoH///pBIJMjPz9daPmHCBEgkEqxfv97uZeBnh8g59J0H6Hs44nfA1Vl7bmHoN5YMn+s5+hxK9T0oKipyyPHIvfk4uwDUML169UKrVq0AAOXl5Th06BC2bduGbdu2YebMmVi+fLmTS0ieqqioCC1atEBiYiIDjofie0zk+jTPA/Qx9hyRK5swYQI2bNiAdevW8eIq2QQTXzf32GOPaf0Y1NbWYubMmXjjjTfw6quvYsyYMejevbtDy5SSkoJjx44hKCjIocclIqUlS5Zg7ty5iI2Ntfuxhg8fjh49eiAsLMzuxyIiXfXPA4icKS4uDseOHYOvr69Djrd7927U1NQgLi7OIccj98bE18P4+PjglVdewbvvvouKigp8+umnDk98g4KC0LZtW4cek4jqxMbGOiTpBYCwsDAmvUREBADw9fV16Dngbbfd5rBjkftjH18PFBAQgOTkZADAhQsX9K6ze/duPPjgg4iNjYWfnx+ioqIwfPhwFBQU6F3/+PHjePTRR9GiRQv4+/sjODgYiYmJuP/++7Fu3TqtdU318S0oKMC9996L8PBwBAcHo1u3bnjnnXcMvh5z+osY6uNx9OhR5OTkoFevXoiLi4Ofnx+aNGmC9PR0fPjhhwb3Z4gl9WDKf//7Xzz22GPo0KEDGjdujICAALRo0QKPPvoofvvtN73baPbdPHXqFMaPH4+YmBj4+/vjtttuw/z581FdXa1329raWqxYsQIdO3ZEQEAAmjZtihEjRuDIkSMW18OECRPQokULAMDp06d1+pSpLFq0CBKJBIsWLcKZM2eQnZ2NhIQE+Pr66tyh2LJlCwYNGoSmTZvCz88PcXFxePjhh3H06FGD5fjzzz+Rk5ODzp07IyQkBEFBQejYsSOef/55XLt2zaLXpNlX9fLly5g6dSqaN28Of39/JCYmYubMmfjzzz91ttP8vF+7dg0LFy7E7bffjqCgIJ3P7MGDBzFu3Dj1fiMiIpCRkYEdO3YYLNfZs2fx6KOPIjY2Vv3dfvbZZ3H9+nWD25jq43vw4EFkZWWhRYsWCAgIQEREBDp16oTZs2fj9OnT6n2Y8x6b6uO7f/9+jBo1Cs2aNVP/1gwePBhffvmlybJb8hlXKBRYs2YNevXqhfDwcPj6+iIqKgqdOnXC9OnT2VSb6BbN7/BHH32E3r17IzQ0FI0aNUKvXr0M/h6VlJRgxowZaN26NQICAhAUFISEhATcddddWLp0qd5tzp8/j1mzZql/E0NCQtC9e3e88cYbqK2t1Vlf8/v/22+/ITMzE1FRUWjUqBG6d++OTz75RL3uDz/8gCFDhqBp06YIDAxEWloadu/ebfL1r127Fl27dkWjRo0QHh6O++67D/v27TOn6nRYei5ljOZ5zNatW9XvS0hICPr372/wfdHsh/zdd99h8ODBaNq0KaRSqVYMuH79OpYtW4YePXogPDwcAQEBaNOmDZ5++mlcvnzZYLneffdddO/eHUFBQYiIiMCgQYPw3XffGVzf1DnbtWvXsGLFCvTu3RuNGzdWx9jBgwfjgw8+0NrHhg0bAAATJ07Uij+LFi3SW2/6jvXiiy/izjvvVJ8jtG/fHvPnz9cbzzXLLoTAmjVr1J+VsLAw3HPPPTY5TyYnEuSWEhMTBQCxbt06vc8nJycLAGLBggU6zz355JMCgJBKpSIlJUU89NBDIjU1VUgkEiGTycQ777yjtf6RI0dEaGioACDatGkjHnzwQfHQQw+JtLQ0ERwcLDp16qS1fl5engAg+vXrp3PsDz/8UMhkMgFAdOjQQYwZM0b07t1bSCQSMWvWLAFA1P9Ynjp1SgAQiYmJJuvj1KlTWsuzs7MFANG2bVuRkZEhMjMzRVpampBKpQKAmDlzps6+DB3P0nowRSaTiaCgINGtWzfx4IMPiiFDhoiWLVsKAKJRo0bi+++/19kmKytLABAzZswQoaGhIjExUYwaNUqkp6eLwMBAAUAMGzZMZzu5XC6GDRsmAAg/Pz9xzz33iMzMTJGUlCQCAgLEE088IQCIrKwss8q+du1aMWLECHVZs7KytB4qOTk5AoAYO3asiIiIEDExMWLEiBHiwQcfFE8++aQQQoiamhoxatQoAUD4+/uLnj17ioceekh06tRJABCBgYHi888/1ynDL7/8IhISEgQAERsbKwYNGiQGDx4soqOjBQDRuXNn8ddff5n3Zggh1q1bJwCIIUOGiNtuu02Eh4eLYcOGieHDh4vGjRur3/eLFy9qbaf6vKemporu3buLRo0aiXvvvVdkZmaK9PR09XorVqxQf+46d+4sRo4cKXr37i38/PwEALF48WKdMh07dkxERUWpX+NDDz0k7rvvPhEYGCjS0tJEWlqaACDy8vK0tlN9TvT9Prz88svqcrRu3VqMGjVKDB48WNx+++1a25j7HqvqTd9nZ82aNepjdenSRYwZM0b07NlT/T1ftGiRzjbWfsYnTpwoAIiAgACRnp4uxowZIzIyMtS/hVu3btXZhsidmToPMET1/Vu4cKGQSCSiV69eIjMzU/2bK5FIxH//+1+tbUpKSkSzZs0EANG8eXMxdOhQkZmZKfr06SMiIiJEWFiYznG++eYb9W9nUlKSGDJkiMjIyFAvu+eee8TNmze1tlF9/6dPny4aNWok2rRpI0aPHq3+rZNIJGLz5s1i69atwtfXV3Tp0kWr7D4+PuK7774z+JpnzpwpJBKJ6N27txgzZozo0KGDerv6r1kIIfr166f3N1YIy8+lTFG9nzNnzhQARLdu3cSYMWNESkqKuvz//ve/DZbxiSeeEFKpVLRr106MHj1a3HPPPeKDDz4QQghx7tw50bFjRwFAREREiPT0dDF8+HD1MZOSkkRRUZHOvv/+97+rX2Pfvn3F6NGjRbt27YRUKhUzZszQe65n7JztzJkzol27dgKACAoKEnfffbcYPXq06NOnjwgLC1Nvc+nSJZGVlSVuu+02AUD06tVLK/5o/p4bOv+7fPmy6Ny5swAgQkNDxZAhQ8SIESNEZGSkACBatGihs41m2bOysoSvr68YOHCgGDVqlGjdurX6PGXfvn1a29n6/JDsh4mvmzIW8I4ePapOLn/88Uet59asWSMAiFatWon//e9/Ws998803IiQkRPj5+Ynff/9dvVx1Qvn888/rHOvatWvim2++0VpmKPEtKSkRISEhAoBYvny51nNfffWVCAgIsHnim5+fL/744w+d9X/99VcRHx8vAIgffvjBrONZWg+mbNy4UVy9elVrmUKhECtXrhQARPv27YVCodB6XnVSAEA8++yzora2Vv3ckSNHRKNGjQQAsXfvXq3t3njjDQFAREdHi6NHj6qX19TUiMcff1y9T3MTXyHMe19UiS8A8fDDD4sbN27orPPMM8+oE8eTJ09qPbd582Yhk8lE48aNxZ9//qlefu3aNXVAnD9/vqiurlY/V1VVJcaMGSMAiIkTJ5r9elQJHADRo0cPcfnyZfVzf/75pzphGz16tNZ2qs87AHHHHXeIkpISnX3
      "text/plain": [
       "<Figure size 1100x1100 with 6 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, [[ax1, ax2], [ax3, ax4], [ax5, ax6]] = plt.subplots(3, 2, figsize=(11, 11))\n",
    "\n",
    "plot_predictions([tree_reg1], X, y, axes=[-0.5, 0.5, -0.1, 0.8], ax=ax1, label=\"$h_1(x_1)$\", style=\"g-\", data_label=\"Training set\")\n",
    "ax1.set_ylabel(\"$y$\", fontsize=16, rotation=0)\n",
    "ax1.set_title(\"Residuals and tree predictions\", fontsize=16)\n",
    "\n",
    "plot_predictions([tree_reg1], X, y, axes=[-0.5, 0.5, -0.1, 0.8], ax=ax2, label=\"$h(x_1) = h_1(x_1)$\", data_label=\"Training set\")\n",
    "ax2.set_ylabel(\"$y$\", fontsize=16, rotation=0)\n",
    "ax2.set_title(\"Ensemble predictions\", fontsize=16)\n",
    "\n",
    "plot_predictions([tree_reg2], X, y2, axes=[-0.5, 0.5, -0.5, 0.5], ax=ax3, label=\"$h_2(x_1)$\", style=\"g-\", data_style=\"k+\", data_label=\"Residuals\")\n",
    "ax3.set_ylabel(\"$y - h_1(x_1)$\", fontsize=16)\n",
    "\n",
    "plot_predictions([tree_reg1, tree_reg2], X, y, axes=[-0.5, 0.5, -0.1, 0.8], ax=ax4, label=\"$h(x_1) = h_1(x_1) + h_2(x_1)$\")\n",
    "ax4.set_ylabel(\"$y$\", fontsize=16, rotation=0)\n",
    "\n",
    "plot_predictions([tree_reg3], X, y3, axes=[-0.5, 0.5, -0.5, 0.5], ax=ax5, label=\"$h_3(x_1)$\", style=\"g-\", data_style=\"k+\")\n",
    "ax5.set_ylabel(\"$y - h_1(x_1) - h_2(x_1)$\", fontsize=16)\n",
    "ax5.set_xlabel(\"$x_1$\", fontsize=16)\n",
    "\n",
    "plot_predictions([tree_reg1, tree_reg2, tree_reg3], X, y, axes=[-0.5, 0.5, -0.1, 0.8], ax=ax6, label=\"$h(x_1) = h_1(x_1) + h_2(x_1) + h_3(x_1)$\")\n",
    "ax6.set_xlabel(\"$x_1$\", fontsize=16)\n",
    "ax6.set_ylabel(\"$y$\", fontsize=16, rotation=0);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "The ensemble’s predictions are equal to the sum of the predictions of the first two trees. Similarly, in the third row another tree is trained on the residual errors of the second tree. \n",
    "\n",
    "You can see that the ensemble’s predictions gradually get better as trees are added to the ensemble.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "Now, let's use in-built functionality rather than implementing by hand."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2025-02-27T23:21:38.522279Z",
     "iopub.status.busy": "2025-02-27T23:21:38.522098Z",
     "iopub.status.idle": "2025-02-27T23:21:38.532656Z",
     "shell.execute_reply": "2025-02-27T23:21:38.532079Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {\n",
       "  /* Definition of color scheme common for light and dark mode */\n",
       "  --sklearn-color-text: #000;\n",
       "  --sklearn-color-text-muted: #666;\n",
       "  --sklearn-color-line: gray;\n",
       "  /* Definition of color scheme for unfitted estimators */\n",
       "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
       "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
       "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
       "  --sklearn-color-unfitted-level-3: chocolate;\n",
       "  /* Definition of color scheme for fitted estimators */\n",
       "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
       "  --sklearn-color-fitted-level-1: #d4ebff;\n",
       "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
       "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
       "\n",
       "  /* Specific color for light theme */\n",
       "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
       "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-icon: #696969;\n",
       "\n",
       "  @media (prefers-color-scheme: dark) {\n",
       "    /* Redefinition of color scheme for dark theme */\n",
       "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
       "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-icon: #878787;\n",
       "  }\n",
       "}\n",
       "\n",
       "#sk-container-id-1 {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 pre {\n",
       "  padding: 0;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-hidden--visually {\n",
       "  border: 0;\n",
       "  clip: rect(1px 1px 1px 1px);\n",
       "  clip: rect(1px, 1px, 1px, 1px);\n",
       "  height: 1px;\n",
       "  margin: -1px;\n",
       "  overflow: hidden;\n",
       "  padding: 0;\n",
       "  position: absolute;\n",
       "  width: 1px;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-dashed-wrapped {\n",
       "  border: 1px dashed var(--sklearn-color-line);\n",
       "  margin: 0 0.4em 0.5em 0.4em;\n",
       "  box-sizing: border-box;\n",
       "  padding-bottom: 0.4em;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-container {\n",
       "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
       "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
       "     so we also need the `!important` here to be able to override the\n",
       "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
       "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
       "  display: inline-block !important;\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-text-repr-fallback {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       "div.sk-parallel-item,\n",
       "div.sk-serial,\n",
       "div.sk-item {\n",
       "  /* draw centered vertical line to link estimators */\n",
       "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
       "  background-size: 2px 100%;\n",
       "  background-repeat: no-repeat;\n",
       "  background-position: center center;\n",
       "}\n",
       "\n",
       "/* Parallel-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item::after {\n",
       "  content: \"\";\n",
       "  width: 100%;\n",
       "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
       "  flex-grow: 1;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel {\n",
       "  display: flex;\n",
       "  align-items: stretch;\n",
       "  justify-content: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:first-child::after {\n",
       "  align-self: flex-end;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:last-child::after {\n",
       "  align-self: flex-start;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:only-child::after {\n",
       "  width: 0;\n",
       "}\n",
       "\n",
       "/* Serial-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-serial {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  align-items: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  padding-right: 1em;\n",
       "  padding-left: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
       "clickable and can be expanded/collapsed.\n",
       "- Pipeline and ColumnTransformer use this feature and define the default style\n",
       "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
       "*/\n",
       "\n",
       "/* Pipeline and ColumnTransformer style (default) */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable {\n",
       "  /* Default theme specific background. It is overwritten whether we have a\n",
       "  specific estimator or a Pipeline/ColumnTransformer */\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "/* Toggleable label */\n",
       "#sk-container-id-1 label.sk-toggleable__label {\n",
       "  cursor: pointer;\n",
       "  display: flex;\n",
       "  width: 100%;\n",
       "  margin-bottom: 0;\n",
       "  padding: 0.5em;\n",
       "  box-sizing: border-box;\n",
       "  text-align: center;\n",
       "  align-items: start;\n",
       "  justify-content: space-between;\n",
       "  gap: 0.5em;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label .caption {\n",
       "  font-size: 0.6rem;\n",
       "  font-weight: lighter;\n",
       "  color: var(--sklearn-color-text-muted);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:before {\n",
       "  /* Arrow on the left of the label */\n",
       "  content: \"▸\";\n",
       "  float: left;\n",
       "  margin-right: 0.25em;\n",
       "  color: var(--sklearn-color-icon);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "/* Toggleable content - dropdown */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content {\n",
       "  max-height: 0;\n",
       "  max-width: 0;\n",
       "  overflow: hidden;\n",
       "  text-align: left;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content pre {\n",
       "  margin: 0.2em;\n",
       "  border-radius: 0.25em;\n",
       "  color: var(--sklearn-color-text);\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted pre {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
       "  /* Expand drop-down */\n",
       "  max-height: 200px;\n",
       "  max-width: 100%;\n",
       "  overflow: auto;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
       "  content: \"▾\";\n",
       "}\n",
       "\n",
       "/* Pipeline/ColumnTransformer-specific style */\n",
       "\n",
       "#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator-specific style */\n",
       "\n",
       "/* Colorize estimator box */\n",
       "#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label label.sk-toggleable__label,\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  /* The background is the default theme color */\n",
       "  color: var(--sklearn-color-text-on-default-background);\n",
       "}\n",
       "\n",
       "/* On hover, darken the color of the background */\n",
       "#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "/* Label box, darken color on hover, fitted */\n",
       "#sk-container-id-1 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator label */\n",
       "\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  font-family: monospace;\n",
       "  font-weight: bold;\n",
       "  display: inline-block;\n",
       "  line-height: 1.2em;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label-container {\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "/* Estimator-specific */\n",
       "#sk-container-id-1 div.sk-estimator {\n",
       "  font-family: monospace;\n",
       "  border: 1px dotted var(--sklearn-color-border-box);\n",
       "  border-radius: 0.25em;\n",
       "  box-sizing: border-box;\n",
       "  margin-bottom: 0.5em;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "/* on hover */\n",
       "#sk-container-id-1 div.sk-estimator:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
       "\n",
       "/* Common style for \"i\" and \"?\" */\n",
       "\n",
       ".sk-estimator-doc-link,\n",
       "a:link.sk-estimator-doc-link,\n",
       "a:visited.sk-estimator-doc-link {\n",
       "  float: right;\n",
       "  font-size: smaller;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1em;\n",
       "  height: 1em;\n",
       "  width: 1em;\n",
       "  text-decoration: none !important;\n",
       "  margin-left: 0.5em;\n",
       "  text-align: center;\n",
       "  /* unfitted */\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted,\n",
       "a:link.sk-estimator-doc-link.fitted,\n",
       "a:visited.sk-estimator-doc-link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "/* Span, style for the box shown on hovering the info icon */\n",
       ".sk-estimator-doc-link span {\n",
       "  display: none;\n",
       "  z-index: 9999;\n",
       "  position: relative;\n",
       "  font-weight: normal;\n",
       "  right: .2ex;\n",
       "  padding: .5ex;\n",
       "  margin: .5ex;\n",
       "  width: min-content;\n",
       "  min-width: 20ex;\n",
       "  max-width: 50ex;\n",
       "  color: var(--sklearn-color-text);\n",
       "  box-shadow: 2pt 2pt 4pt #999;\n",
       "  /* unfitted */\n",
       "  background: var(--sklearn-color-unfitted-level-0);\n",
       "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted span {\n",
       "  /* fitted */\n",
       "  background: var(--sklearn-color-fitted-level-0);\n",
       "  border: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link:hover span {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link {\n",
       "  float: right;\n",
       "  font-size: 1rem;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1rem;\n",
       "  height: 1rem;\n",
       "  width: 1rem;\n",
       "  text-decoration: none;\n",
       "  /* unfitted */\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "#sk-container-id-1 a.estimator_doc_link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>GradientBoostingRegressor(learning_rate=1.0, max_depth=2, n_estimators=3,\n",
       "                          random_state=42)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow\"><div><div>GradientBoostingRegressor</div></div><div><a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.6/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html\">?<span>Documentation for GradientBoostingRegressor</span></a><span class=\"sk-estimator-doc-link fitted\">i<span>Fitted</span></span></div></label><div class=\"sk-toggleable__content fitted\"><pre>GradientBoostingRegressor(learning_rate=1.0, max_depth=2, n_estimators=3,\n",
       "                          random_state=42)</pre></div> </div></div></div></div>"
      ],
      "text/plain": [
       "GradientBoostingRegressor(learning_rate=1.0, max_depth=2, n_estimators=3,\n",
       "                          random_state=42)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.ensemble import GradientBoostingRegressor\n",
    "\n",
    "gbrt = GradientBoostingRegressor(max_depth=2, n_estimators=3,\n",
    "                                 learning_rate=1.0, random_state=42)\n",
    "gbrt.fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "For GBRT, the learning rate plays the role of scaling the contribution for each tree.\n",
    "\n",
    "Low values (e.g. 0.1) need more trees in ensemble to fit training set but predictions usually generalize better. This is called shrinkage.\n",
    "\n",
    "Can also consider early stopping to regularise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-02-27T23:21:38.534498Z",
     "iopub.status.busy": "2025-02-27T23:21:38.534323Z",
     "iopub.status.idle": "2025-02-27T23:21:38.584825Z",
     "shell.execute_reply": "2025-02-27T23:21:38.584228Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-2 {\n",
       "  /* Definition of color scheme common for light and dark mode */\n",
       "  --sklearn-color-text: #000;\n",
       "  --sklearn-color-text-muted: #666;\n",
       "  --sklearn-color-line: gray;\n",
       "  /* Definition of color scheme for unfitted estimators */\n",
       "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
       "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
       "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
       "  --sklearn-color-unfitted-level-3: chocolate;\n",
       "  /* Definition of color scheme for fitted estimators */\n",
       "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
       "  --sklearn-color-fitted-level-1: #d4ebff;\n",
       "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
       "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
       "\n",
       "  /* Specific color for light theme */\n",
       "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
       "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-icon: #696969;\n",
       "\n",
       "  @media (prefers-color-scheme: dark) {\n",
       "    /* Redefinition of color scheme for dark theme */\n",
       "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
       "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-icon: #878787;\n",
       "  }\n",
       "}\n",
       "\n",
       "#sk-container-id-2 {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 pre {\n",
       "  padding: 0;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 input.sk-hidden--visually {\n",
       "  border: 0;\n",
       "  clip: rect(1px 1px 1px 1px);\n",
       "  clip: rect(1px, 1px, 1px, 1px);\n",
       "  height: 1px;\n",
       "  margin: -1px;\n",
       "  overflow: hidden;\n",
       "  padding: 0;\n",
       "  position: absolute;\n",
       "  width: 1px;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-dashed-wrapped {\n",
       "  border: 1px dashed var(--sklearn-color-line);\n",
       "  margin: 0 0.4em 0.5em 0.4em;\n",
       "  box-sizing: border-box;\n",
       "  padding-bottom: 0.4em;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-container {\n",
       "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
       "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
       "     so we also need the `!important` here to be able to override the\n",
       "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
       "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
       "  display: inline-block !important;\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-text-repr-fallback {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       "div.sk-parallel-item,\n",
       "div.sk-serial,\n",
       "div.sk-item {\n",
       "  /* draw centered vertical line to link estimators */\n",
       "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
       "  background-size: 2px 100%;\n",
       "  background-repeat: no-repeat;\n",
       "  background-position: center center;\n",
       "}\n",
       "\n",
       "/* Parallel-specific style estimator block */\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item::after {\n",
       "  content: \"\";\n",
       "  width: 100%;\n",
       "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
       "  flex-grow: 1;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel {\n",
       "  display: flex;\n",
       "  align-items: stretch;\n",
       "  justify-content: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item:first-child::after {\n",
       "  align-self: flex-end;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item:last-child::after {\n",
       "  align-self: flex-start;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item:only-child::after {\n",
       "  width: 0;\n",
       "}\n",
       "\n",
       "/* Serial-specific style estimator block */\n",
       "\n",
       "#sk-container-id-2 div.sk-serial {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  align-items: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  padding-right: 1em;\n",
       "  padding-left: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
       "clickable and can be expanded/collapsed.\n",
       "- Pipeline and ColumnTransformer use this feature and define the default style\n",
       "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
       "*/\n",
       "\n",
       "/* Pipeline and ColumnTransformer style (default) */\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable {\n",
       "  /* Default theme specific background. It is overwritten whether we have a\n",
       "  specific estimator or a Pipeline/ColumnTransformer */\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "/* Toggleable label */\n",
       "#sk-container-id-2 label.sk-toggleable__label {\n",
       "  cursor: pointer;\n",
       "  display: flex;\n",
       "  width: 100%;\n",
       "  margin-bottom: 0;\n",
       "  padding: 0.5em;\n",
       "  box-sizing: border-box;\n",
       "  text-align: center;\n",
       "  align-items: start;\n",
       "  justify-content: space-between;\n",
       "  gap: 0.5em;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 label.sk-toggleable__label .caption {\n",
       "  font-size: 0.6rem;\n",
       "  font-weight: lighter;\n",
       "  color: var(--sklearn-color-text-muted);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 label.sk-toggleable__label-arrow:before {\n",
       "  /* Arrow on the left of the label */\n",
       "  content: \"▸\";\n",
       "  float: left;\n",
       "  margin-right: 0.25em;\n",
       "  color: var(--sklearn-color-icon);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 label.sk-toggleable__label-arrow:hover:before {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "/* Toggleable content - dropdown */\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content {\n",
       "  max-height: 0;\n",
       "  max-width: 0;\n",
       "  overflow: hidden;\n",
       "  text-align: left;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content pre {\n",
       "  margin: 0.2em;\n",
       "  border-radius: 0.25em;\n",
       "  color: var(--sklearn-color-text);\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content.fitted pre {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
       "  /* Expand drop-down */\n",
       "  max-height: 200px;\n",
       "  max-width: 100%;\n",
       "  overflow: auto;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
       "  content: \"▾\";\n",
       "}\n",
       "\n",
       "/* Pipeline/ColumnTransformer-specific style */\n",
       "\n",
       "#sk-container-id-2 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator-specific style */\n",
       "\n",
       "/* Colorize estimator box */\n",
       "#sk-container-id-2 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-label label.sk-toggleable__label,\n",
       "#sk-container-id-2 div.sk-label label {\n",
       "  /* The background is the default theme color */\n",
       "  color: var(--sklearn-color-text-on-default-background);\n",
       "}\n",
       "\n",
       "/* On hover, darken the color of the background */\n",
       "#sk-container-id-2 div.sk-label:hover label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "/* Label box, darken color on hover, fitted */\n",
       "#sk-container-id-2 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator label */\n",
       "\n",
       "#sk-container-id-2 div.sk-label label {\n",
       "  font-family: monospace;\n",
       "  font-weight: bold;\n",
       "  display: inline-block;\n",
       "  line-height: 1.2em;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-label-container {\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "/* Estimator-specific */\n",
       "#sk-container-id-2 div.sk-estimator {\n",
       "  font-family: monospace;\n",
       "  border: 1px dotted var(--sklearn-color-border-box);\n",
       "  border-radius: 0.25em;\n",
       "  box-sizing: border-box;\n",
       "  margin-bottom: 0.5em;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-estimator.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "/* on hover */\n",
       "#sk-container-id-2 div.sk-estimator:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-estimator.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
       "\n",
       "/* Common style for \"i\" and \"?\" */\n",
       "\n",
       ".sk-estimator-doc-link,\n",
       "a:link.sk-estimator-doc-link,\n",
       "a:visited.sk-estimator-doc-link {\n",
       "  float: right;\n",
       "  font-size: smaller;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1em;\n",
       "  height: 1em;\n",
       "  width: 1em;\n",
       "  text-decoration: none !important;\n",
       "  margin-left: 0.5em;\n",
       "  text-align: center;\n",
       "  /* unfitted */\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted,\n",
       "a:link.sk-estimator-doc-link.fitted,\n",
       "a:visited.sk-estimator-doc-link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "/* Span, style for the box shown on hovering the info icon */\n",
       ".sk-estimator-doc-link span {\n",
       "  display: none;\n",
       "  z-index: 9999;\n",
       "  position: relative;\n",
       "  font-weight: normal;\n",
       "  right: .2ex;\n",
       "  padding: .5ex;\n",
       "  margin: .5ex;\n",
       "  width: min-content;\n",
       "  min-width: 20ex;\n",
       "  max-width: 50ex;\n",
       "  color: var(--sklearn-color-text);\n",
       "  box-shadow: 2pt 2pt 4pt #999;\n",
       "  /* unfitted */\n",
       "  background: var(--sklearn-color-unfitted-level-0);\n",
       "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted span {\n",
       "  /* fitted */\n",
       "  background: var(--sklearn-color-fitted-level-0);\n",
       "  border: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link:hover span {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
       "\n",
       "#sk-container-id-2 a.estimator_doc_link {\n",
       "  float: right;\n",
       "  font-size: 1rem;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1rem;\n",
       "  height: 1rem;\n",
       "  width: 1rem;\n",
       "  text-decoration: none;\n",
       "  /* unfitted */\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 a.estimator_doc_link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "#sk-container-id-2 a.estimator_doc_link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 a.estimator_doc_link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "</style><div id=\"sk-container-id-2\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>GradientBoostingRegressor(learning_rate=0.05, max_depth=2, n_estimators=500,\n",
       "                          n_iter_no_change=10, random_state=42)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" checked><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow\"><div><div>GradientBoostingRegressor</div></div><div><a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.6/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html\">?<span>Documentation for GradientBoostingRegressor</span></a><span class=\"sk-estimator-doc-link fitted\">i<span>Fitted</span></span></div></label><div class=\"sk-toggleable__content fitted\"><pre>GradientBoostingRegressor(learning_rate=0.05, max_depth=2, n_estimators=500,\n",
       "                          n_iter_no_change=10, random_state=42)</pre></div> </div></div></div></div>"
      ],
      "text/plain": [
       "GradientBoostingRegressor(learning_rate=0.05, max_depth=2, n_estimators=500,\n",
       "                          n_iter_no_change=10, random_state=42)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gbrt_best = GradientBoostingRegressor(\n",
    "    max_depth=2, learning_rate=0.05, n_estimators=500,\n",
    "    n_iter_no_change=10, # Early stopping (set to None to turn off)\n",
    "    random_state=42)\n",
    "gbrt_best.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-02-27T23:21:38.586570Z",
     "iopub.status.busy": "2025-02-27T23:21:38.586390Z",
     "iopub.status.idle": "2025-02-27T23:21:38.590237Z",
     "shell.execute_reply": "2025-02-27T23:21:38.589641Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "92"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gbrt_best.n_estimators_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-02-27T23:21:38.591962Z",
     "iopub.status.busy": "2025-02-27T23:21:38.591770Z",
     "iopub.status.idle": "2025-02-27T23:21:38.595850Z",
     "shell.execute_reply": "2025-02-27T23:21:38.595265Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def plot_predictions(regressors, X, y, axes, style,\n",
    "                     label=None, data_style=\"b.\", data_label=None):\n",
    "    x1 = np.linspace(axes[0], axes[1], 500)\n",
    "    y_pred = sum(regressor.predict(x1.reshape(-1, 1))\n",
    "                 for regressor in regressors)\n",
    "    plt.plot(X[:, 0], y, data_style, label=data_label)\n",
    "    plt.plot(x1, y_pred, style, linewidth=2, label=label)\n",
    "    if label or data_label:\n",
    "        plt.legend(loc=\"upper center\")\n",
    "    plt.axis(axes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-02-27T23:21:38.597668Z",
     "iopub.status.busy": "2025-02-27T23:21:38.597495Z",
     "iopub.status.idle": "2025-02-27T23:21:38.801956Z",
     "shell.execute_reply": "2025-02-27T23:21:38.801261Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1EAAAGKCAYAAADt48tvAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAitlJREFUeJzt3Xd8U1X/B/BP0tK9KC1taRkyBVlSoBRliJWCiKAgS1lPf4CjipbHgQPEwRBEFFEEC/qoCE70cYBYqYKULerDRilQoAOBFgq0kJzfH2mSm2Y0OzfJ5/165UVyc5Oc3IZ8873nnO9RCCEEiIiIiIiIyCpKTzeAiIiIiIjImzCJIiIiIiIisgGTKCIiIiIiIhswiSIiIiIiIrIBkygiIiIiIiIbMIkiIiIiIiKyAZMoIiIiIiIiGzCJIiIiIiIisgGTKCIiIiIiIhswifKg9957DwqFAoWFhZ5uikX5+flQKBTIz8/3dFPIz/GzSGQfxhsi12nWrBkmTJjg6WaQmzGJIr+yb98+PP/88x75IXHw4EE89thj6NmzJ0JCQuz6QbN//34MGDAAERERiI2NxdixY1FWVuaaBnvQW2+9hffee8/TzTDgyc+OIy5fvoysrCy0b98e0dHRiIiIQKdOnfD666/j6tWrnm4ekc/y9HfGyZMnMWLECMTExCAqKgpDhgzB33//bfXjt2zZgptvvhlhYWFITEzEI488gosXLxrso016TV22bt3q7LfkUVu2bMHzzz+P8+fPe7opBmbPno21a9d6uhl2efPNN9G2bVsEBwcjOTkZOTk5qKysNNjnwIEDeOKJJ9C5c2dERkYiKSkJgwYNws6dOz3Uar1ATzeA5K937964fPkygoKCPN0Uh+3btw+zZs1C37590axZM7e+dkFBAd544w20a9cObdu2xZ49e2x6fFFREXr37o3o6GjMnj0bFy9exIIFC/Dnn39i+/btPvH30XrrrbcQFxdndGbPk59FT352HHH58mXs3bsXt99+O5o1awalUoktW7bgsccew7Zt27Bq1SpPN5FIh/HGOS5evIhbbrkF5eXlePrpp1GvXj289tpr6NOnD/bs2YMGDRpYfPyePXtw6623om3btli4cCGKioqwYMECHD58GN9//73R/o888gi6detmsK1ly5ZOfU+etmXLFsyaNQsTJkxATEyMwX0HDx6EUumZfonZs2dj+PDhGDp0qEde315PPvkkXnnlFQwfPhxTp07Fvn37sHjxYuzduxfr16/X7ffuu+8iNzcXw4YNw4MPPojy8nK888476NGjB9atW4eMjAyPvQcmUX6osrIS4eHhVu+vVCoREhLiwhbZz9b34kl33nknzp8/j8jISCxYsMDmJGr27NmorKzErl270KRJEwBA9+7dcdttt+G9997D5MmTXdBqeZHzZ9Fely5dQlhYmMuePzY21uiM8P3334/o6Gi8+eabWLhwIRITE132+uTfGG8846233sLhw4exfft2XXIzcOBAtG/fHq+++ipmz55t8fFPP/006tevj/z8fERFRQHQDFmbNGkSfvjhB/Tv399g/169emH48OGueTNeIDg42NNNcCpXf9ZPnz6NhQsXYuzYsfjPf/6j2966dWs8/PDD+O9//4vBgwcDAEaPHo3nn38eERERuv3+9a9/oW3btnj++ec9mkRBkMesXLlSABBHjx412P7dd9+Jm2++WYSFhYmIiAhx++23i//9738G+/z+++9i/Pjx4rrrrhPBwcEiISFBTJw4UZw5c8Zgv5kzZwoAYu/evWL06NEiJiZGdO7cWQghRNOmTcWgQYPEpk2bRLdu3URwcLC47rrrxPvvv2/wHBs3bhQAxMaNG3Xb+vTpI2644Qaxd+9e0bdvXxEaGioaNWok5s2bZ/Q+CwsLxeDBg0VYWJiIj48Xjz76qFi3bp3Rc1p7vPLz88UDDzwg4uPjRUxMjO41HnjgAdG6dWsREhIiYmNjxfDhww2OrfbxtS/SNlhz7J1h/vz5Jv/2ljRs2FDcc889Rttbt24tbr31VrvaYcvf0RrWHL/Tp0+LCRMmiOTkZBEUFCQSExPFnXfeqTsWTZs2Nfob9enTRwhh+bP4+++/i969e4vQ0FDRokUL8emnnwohhMjPzxfdu3cXISEhonXr1mLDhg0G7XHWZ2fJkiWiXbt2IigoSCQlJYkHH3xQnDt3zuTx3rlzp+jVq5cIDQ0VU6dOFUIIsWPHDtG/f3/RoEEDERISIpo1ayYmTpxo19/BGgsWLBAAxP79+132GiQfjDf+FW+6desmunXrZrS9f//+okWLFhYfW15eLgIDA8Xjjz9usL2qqkpERESIrKws3Tbt3+vTTz8VFRUV4urVqw63XXvsNm/eLB577DERFxcnwsLCxNChQ0VpaanNz1dUVCQmTpwoGjZsKIKCgkS7du1Ebm6u0X5vvPGGaNeunQgNDRUxMTEiNTVVfPTRR0II/We79kUat8aPH2/0HjZt2iQefvhhERcXJ6Kjo8XkyZNFVVWVOHfunBg7dqyIiYkRMTEx4vHHHxdqtdqgPfPnzxfp6ekiNjZWhISEiC5duujimpapNknbsXv3bjFgwAARGRkpwsPDRb9+/URBQYHJ423qs15RUSGmTp0qmjZtKoKCgkR8fLzIyMgQu3btsvnvIPX5558LAOLbb7812F5WViYAiDFjxtT5HHfffbeIjY11qB2OYk+UzHzwwQcYP348MjMzMW/ePFy6dAlvv/02br75Zvz222+6IQEbNmzA33//jYkTJyIxMRF79+7FsmXLsHfvXmzduhUKhcLgee+55x60atUKs2fPhhBCt/3IkSMYPnw4srKyMH78eKxYsQITJkxAamoqbrjhBottPXfuHAYMGIC7774bI0aMwGeffYYnn3wSHTp0wMCBAwFozmb069cPp0+fxtSpU5GYmIhVq1Zh48aNdh+jBx98EPHx8ZgxY4Zu7OyOHTuwZcsWjBo1CikpKSgsLMTbb7+Nvn37Yt++fQgLC0Pv3r3xyCOP4I033sDTTz+Ntm3bAoDuX2uPfVVVFS5cuGBVW+Pi4ux+n1InT55EaWkpunbtanRf9+7d8d1339n93Nb8Ha1h7fEbNmwY9u7di4cffhjNmjVDaWkpNmzYgOPHj6NZs2ZYtGgRHn74YUREROCZZ54BACQkJNT5Hu644w6MGjUK99xzD95++22MGjUKH330ER599FHcf//9GDNmDObPn4/hw4fjxIkTiIyMBOCcz87zzz+PWbNmISMjAw888AAOHjyIt99+Gzt27MCvv/6KevXq6dr6zz//YODAgRg1ahTuu+8+JCQkoLS0FP3790d8fDyeeuopxMTEoLCwEF988YXR+1SpVHX+LcLCwox6t6qrq1FRUYHLly9j586dWLBgAZo2bepzQ27Ieow3dfPGeKNWq/HHH3/gX//6l9E+3bt3xw8//IALFy7ovgNr+/PPP3Ht2jWjeBMUFITOnTvjt99+M3rMxIkTcfHiRQQEBKBXr16YP3++yXhli4cffhj169fHzJkzUVhYiEWLFiE7Oxtr1qyx+jlKSkrQo0cPKBQKZGdnIz4+Ht9//z2ysrJQUVGBRx99FACwfPlyPPLII7qhZVeuXMEff/yBbdu2YcyYMbj77rtx6NAhfPzxx3jttdd0xzo+Pr7O95CYmIhZs2Zh69atWLZsGWJiYrBlyxY0adIEs2fPxnfffYf58+ejffv2GDdunO6xr7/+Ou68807ce++9qK6uxurVq3HPPffgm2++waBBgwBoPkf/93//h+7du+tGo7Ro0QIAsHfvXvTq1QtRUVF44oknUK9ePbzzzjvo27cvfv75Z6SlpRm01dRn/f7778dnn32G7OxstGvXDv/88w82b96M/fv3o0uXLgA0oykuXbpU598iICAA9evXB6D5XANAaGiowT7auLVr1646n6+4uNhpv7Hs5tEUzs/VPjN44cIFERMTIyZNmmSwX3FxsYiOjjbYfunSJaPn+/jjjwUA8csvv+i2ac+ejB492mh/7Rl/6f6lpaUiODhYTJs2TbfN3JlBAOI///m
      "text/plain": [
       "<Figure size 1000x400 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, axes = plt.subplots(ncols=2, figsize=(10, 4), sharey=True)\n",
    "\n",
    "plt.sca(axes[0])\n",
    "plot_predictions([gbrt], X, y, axes=[-0.5, 0.5, -0.1, 0.8], style=\"r-\",\n",
    "                 label=\"Ensemble predictions\")\n",
    "plt.title(f\"learning_rate={gbrt.learning_rate}, \"\n",
    "          f\"n_estimators={gbrt.n_estimators_}\")\n",
    "plt.xlabel(\"$x_1$\");\n",
    "plt.ylabel(\"$y$\", rotation=0);\n",
    "\n",
    "plt.sca(axes[1])\n",
    "plot_predictions([gbrt_best], X, y, axes=[-0.5, 0.5, -0.1, 0.8], style=\"r-\")\n",
    "plt.title(f\"learning_rate={gbrt_best.learning_rate}, \"\n",
    "          f\"n_estimators={gbrt_best.n_estimators_}\")\n",
    "plt.xlabel(\"$x_1$\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "exercise_pointer"
    ]
   },
   "source": [
    "**Exercises:** *You can now complete Exercise 1 in the exercises associated with this lecture.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Stacking (or meta learning)\n",
    "\n",
    "Stacking (stacked generalisation) introduces a learner to find the optimal combination, rather than a simple aggregation.\n",
    "\n",
    "Done by a _blender_ or _meta learner_.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "<img src=\"https://raw.githubusercontent.com/astro-informatics/course_mlbd_images/master/Lecture17_Images/blender.png\" alt=\"data-layout\" width=\"600\" style=\"display:block; margin:auto\"/>\n",
    "\n",
    "[Source: Geron]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "Can extend to multiple blenders.\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/astro-informatics/course_mlbd_images/master/Lecture17_Images/blender2.png\" alt=\"data-layout\" width=\"600\" style=\"display:block; margin:auto\"/>\n",
    "\n",
    "[Source: Geron]"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}