{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Lecture 7: Training III"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "![](https://www.tensorflow.org/images/colab_logo_32px.png)\n",
    "[Run in colab](https://colab.research.google.com/drive/151rKfOzxrWMK7fiViIGsosquddN07yJg )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:19:44.467687Z",
     "iopub.status.busy": "2024-01-10T00:19:44.467138Z",
     "iopub.status.idle": "2024-01-10T00:19:44.475581Z",
     "shell.execute_reply": "2024-01-10T00:19:44.475048Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last executed: 2024-01-10 00:19:44\n"
     ]
    }
   ],
   "source": [
    "import datetime\n",
    "now = datetime.datetime.now()\n",
    "print(\"Last executed: \" + now.strftime(\"%Y-%m-%d %H:%M:%S\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Polynomial regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "So far we have considered only linear regression.  Polynomial regression can also be performed with a model that is linear (in the parameters)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:19:44.516067Z",
     "iopub.status.busy": "2024-01-10T00:19:44.515424Z",
     "iopub.status.idle": "2024-01-10T00:19:44.928255Z",
     "shell.execute_reply": "2024-01-10T00:19:44.927538Z"
    }
   },
   "outputs": [],
   "source": [
    "# Common imports\n",
    "import os\n",
    "import numpy as np\n",
    "np.random.seed(42) # To make this notebook's output stable across runs\n",
    "\n",
    "# To plot pretty figures\n",
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams['axes.labelsize'] = 14\n",
    "plt.rcParams['xtick.labelsize'] = 12\n",
    "plt.rcParams['ytick.labelsize'] = 12"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Example data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:19:44.931698Z",
     "iopub.status.busy": "2024-01-10T00:19:44.931400Z",
     "iopub.status.idle": "2024-01-10T00:19:44.935887Z",
     "shell.execute_reply": "2024-01-10T00:19:44.935221Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy.random as rnd\n",
    "np.random.seed(42)\n",
    "m = 100\n",
    "X = 6 * np.random.rand(m, 1) - 3\n",
    "y = 0.5 * X**2 + X + 2 + np.random.randn(m, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:19:44.938729Z",
     "iopub.status.busy": "2024-01-10T00:19:44.938267Z",
     "iopub.status.idle": "2024-01-10T00:19:45.153988Z",
     "shell.execute_reply": "2024-01-10T00:19:45.153313Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 18388 characters]",
      "text/plain": [
       "<Figure size 1000x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(10,5))\n",
    "plt.plot(X, y, \"b.\")\n",
    "plt.xlabel(\"$x_1$\", fontsize=18)\n",
    "plt.ylabel(\"$y$\", rotation=0, fontsize=18)\n",
    "plt.axis([-3, 3, 0, 10]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly a straight line will not fit the data well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Construct new features\n",
    "\n",
    "Can use a linear model by constructing additional features that are powers of existing features:\n",
    "\n",
    "$$\\hat{y} = \\theta_0 + \\theta_1 x_1 + \\theta_2 x_1^2 + \\theta_3 x_2 + \\theta_4 x_2^2 + \\theta_5 x_1 x_2 + ... $$\n",
    "\n",
    "Model remains linear in the parameters $\\theta_j$.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Generate polynomial features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:19:45.157478Z",
     "iopub.status.busy": "2024-01-10T00:19:45.157015Z",
     "iopub.status.idle": "2024-01-10T00:19:45.487951Z",
     "shell.execute_reply": "2024-01-10T00:19:45.487213Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([-0.75275929]), array([-0.75275929,  0.56664654]))"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "poly_features = PolynomialFeatures(degree=2, include_bias=False)\n",
    "X_poly = poly_features.fit_transform(X)\n",
    "X[0], X_poly[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:19:45.491035Z",
     "iopub.status.busy": "2024-01-10T00:19:45.490576Z",
     "iopub.status.idle": "2024-01-10T00:19:45.495240Z",
     "shell.execute_reply": "2024-01-10T00:19:45.494597Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((100, 1), (100, 2))"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape, X_poly.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Fit model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:19:45.498108Z",
     "iopub.status.busy": "2024-01-10T00:19:45.497659Z",
     "iopub.status.idle": "2024-01-10T00:19:45.555973Z",
     "shell.execute_reply": "2024-01-10T00:19:45.555227Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([1.78134581]), array([[0.93366893, 0.56456263]]))"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "lin_reg = LinearRegression()\n",
    "lin_reg.fit(X_poly, y)\n",
    "lin_reg.intercept_, lin_reg.coef_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Parameters are close to the model used to generate the data (2, 1 and 0.5 respectively)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:19:45.559266Z",
     "iopub.status.busy": "2024-01-10T00:19:45.558700Z",
     "iopub.status.idle": "2024-01-10T00:19:45.735843Z",
     "shell.execute_reply": "2024-01-10T00:19:45.735234Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 35040 characters]",
      "text/plain": [
       "<Figure size 1000x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X_new = np.linspace(-3, 3, 100).reshape(100, 1)\n",
    "X_new_poly = poly_features.transform(X_new)\n",
    "y_new = lin_reg.predict(X_new_poly)\n",
    "plt.figure(figsize=(10,5))\n",
    "plt.plot(X, y, \"b.\")\n",
    "plt.plot(X_new, y_new, \"r-\", linewidth=2, label=\"Predictions\")\n",
    "plt.xlabel(\"$x_1$\", fontsize=18)\n",
    "plt.ylabel(\"$y$\", rotation=0, fontsize=18)\n",
    "plt.legend(loc=\"upper left\", fontsize=14)\n",
    "plt.axis([-3, 3, 0, 10]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "exercise_pointer"
    ]
   },
   "source": [
    "**Exercises:** *You can now complete Exercise 1 in the exercises associated with this lecture.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Learning curves"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How determine whether overfitting or underfitting?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Overfitting with high degree polynomials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:19:45.739185Z",
     "iopub.status.busy": "2024-01-10T00:19:45.738769Z",
     "iopub.status.idle": "2024-01-10T00:19:45.934670Z",
     "shell.execute_reply": "2024-01-10T00:19:45.934005Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 52312 characters]",
      "text/plain": [
       "<Figure size 700x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "plt.figure(figsize=(7,3))\n",
    "for style, width, degree in ((\"g-\", 1, 300), (\"b--\", 2, 2), (\"r-+\", 2, 1)):\n",
    "    polybig_features = PolynomialFeatures(degree=degree, include_bias=False)\n",
    "    std_scaler = StandardScaler()\n",
    "    lin_reg = LinearRegression()\n",
    "    polynomial_regression = Pipeline(\n",
    "        ((\"poly_features\", polybig_features), (\"std_scaler\", std_scaler), (\"lin_reg\", lin_reg)) )\n",
    "    polynomial_regression.fit(X, y)\n",
    "    y_newbig = polynomial_regression.predict(X_new)\n",
    "    plt.plot(X_new, y_newbig, style, label=str(degree), linewidth=width)\n",
    "\n",
    "plt.plot(X, y, \"b.\", linewidth=3)\n",
    "plt.legend(loc=\"upper left\")\n",
    "plt.xlabel(\"$x_1$\", fontsize=18)\n",
    "plt.ylabel(\"$y$\", rotation=0, fontsize=18)\n",
    "plt.axis([-3, 3, 0, 10]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Training data is underfitted for degree 1, fitted well for degree 2, and overfitted for degree 300."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Learning curves provide another way to determine whether model underfitted or overfitted."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider performance on training and validation set *as size of the training set increases*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Plotting learning curves"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:19:45.938531Z",
     "iopub.status.busy": "2024-01-10T00:19:45.937877Z",
     "iopub.status.idle": "2024-01-10T00:19:45.944402Z",
     "shell.execute_reply": "2024-01-10T00:19:45.943841Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import mean_squared_error\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "def plot_learning_curves(model, X, y):\n",
    "    X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=10)\n",
    "    train_errors, val_errors = [], []\n",
    "    for m in range(1, len(X_train)):\n",
    "        model.fit(X_train[:m], y_train[:m])\n",
    "        y_train_predict = model.predict(X_train[:m])\n",
    "        y_val_predict = model.predict(X_val)\n",
    "        train_errors.append(mean_squared_error(y_train_predict, y_train[:m]))\n",
    "        val_errors.append(mean_squared_error(y_val_predict, y_val))\n",
    "\n",
    "    plt.figure(figsize=(8,4))\n",
    "    plt.plot(np.sqrt(train_errors), \"r-+\", linewidth=2, label=\"Training set\")\n",
    "    plt.plot(np.sqrt(val_errors), \"b-\", linewidth=3, label=\"Validation set\")\n",
    "    plt.legend(loc=\"upper right\", fontsize=14)   \n",
    "    plt.xlabel(\"Training set size\", fontsize=14) \n",
    "    plt.ylabel(\"RMSE\", fontsize=14)              "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Underfitted learning curves"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Learning curve for linear model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:19:45.947504Z",
     "iopub.status.busy": "2024-01-10T00:19:45.946932Z",
     "iopub.status.idle": "2024-01-10T00:19:46.221925Z",
     "shell.execute_reply": "2024-01-10T00:19:46.221338Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 40904 characters]",
      "text/plain": [
       "<Figure size 800x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lin_reg = LinearRegression()\n",
    "plot_learning_curves(lin_reg, X, y)\n",
    "plt.axis([0, 80, 0, 3]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- RMSE on training set small for small training set size since model can fit data well for very few data points (perfect for one or two points).\n",
    "\n",
    "- RMSE performance on training and validation eventually similar but high since linear model cannot fit the data well (recall data generated by quadratic)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Overfitted learning curves  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Learning curve for poynomial model of degree 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:19:46.225436Z",
     "iopub.status.busy": "2024-01-10T00:19:46.224761Z",
     "iopub.status.idle": "2024-01-10T00:19:46.559315Z",
     "shell.execute_reply": "2024-01-10T00:19:46.558734Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 40172 characters]",
      "text/plain": [
       "<Figure size 800x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "polynomial_regression = Pipeline((\n",
    "        (\"poly_features\", PolynomialFeatures(degree=10, include_bias=False)),\n",
    "        (\"lin_reg\", LinearRegression()),\n",
    "    ))\n",
    "plot_learning_curves(polynomial_regression, X, y)\n",
    "plt.axis([0, 80, 0, 3]);  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- RMSE now much smaller.\n",
    "- But training and validation set errors remain quite different.\n",
    "\n",
    "Model performs much better on training set than validation set, suggesting overfitting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Well-fitted learning curves  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Learning curve for poynomial model of degree 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:19:46.562771Z",
     "iopub.status.busy": "2024-01-10T00:19:46.562122Z",
     "iopub.status.idle": "2024-01-10T00:19:46.882864Z",
     "shell.execute_reply": "2024-01-10T00:19:46.882258Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 38780 characters]",
      "text/plain": [
       "<Figure size 800x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "polynomial_regression = Pipeline((\n",
    "        (\"poly_features\", PolynomialFeatures(degree=2, include_bias=False)),\n",
    "        (\"lin_reg\", LinearRegression()),\n",
    "    ))\n",
    "plot_learning_curves(polynomial_regression, X, y)\n",
    "plt.axis([0, 80, 0, 3]);  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Bias-variance tradeoff"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bias-variance tradeoff refers to the problem of simultaneously reducing the two types of errors that prevents supervised learning algorithms from generalising to other data.\n",
    "\n",
    "- **Bias**: Expected difference between data and prediction.\n",
    "\n",
    "- **Variance**: Expected ability of the model to fluctuate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "One seeks a model that accurately fits the training data, while also generalising to unseen data.\n",
    "\n",
    "Typically impossible to do both simultaneously.\n",
    "\n",
    "- On one hand, high-variance models may fit training data well but typically overfit to noise or unrepresentative training data.\n",
    "\n",
    "- On the other hand, low-complexity models with a high bias typically underfit training data.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Contributions to the mean square error "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider underlying (true) model:\n",
    "$ y = f(x) + \\epsilon,$\n",
    "\n",
    "where\n",
    "- $y$ is the target and $x$ features.\n",
    "- $f$ is the true model, that we will approximate by $h$.\n",
    "- $\\epsilon$ is the noise, with zero mean and variance $\\sigma^2$.\n",
    "\n",
    "\n",
    "Approximate $f$ by $h$, which is fitted by a learning algorithm and training data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Expected value of the mean square error is given by\n",
    "\n",
    "$$\\text{E} \\left[ \\left(y - h(x)\\right)^2 \\right]\n",
    "= \n",
    "\\text{Bias}^2\\left[ h(x) \\right]\n",
    "+\n",
    "\\text{Var}\\left[ h(x) \\right]\n",
    "+\n",
    "\\sigma^2$$\n",
    "\n",
    "Three contributions to the error:\n",
    " \n",
    "1. Bias: $\\text{Bias}\\left[ h(x) \\right] = \\text{E} \\left[ h(x) - f(x) \\right].$\n",
    "\n",
    "2. Variance:  $\\text{Var}\\left[ h(x) \\right] =  \\text{E} \\left[ h^2(x)\\right] - \\text{E} \\left[ h(x)\\right]^2. $\n",
    "\n",
    "3. Irreducible error $\\sigma^2$ due to noise in observations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Tradeoff \n",
    "\n",
    "By choosing a complex model, the bias can be made small but the variance will be large.\n",
    "\n",
    "By choosing a simple model, the variance can be made small but the bias will be large.\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/astro-informatics/course_mlbd_images/master/Lecture07_Images/biasvariance.png\" width=\"700px\" style=\"display:block; margin:auto\"/> \n",
    "\n",
    "[Image source](http://francescopochetti.com/bias-v-s-variance-tradeoff/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Regularization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One approach to mitigate the bias-variance tradeoff is by *regularization*.\n",
    "\n",
    "Consider a complex model but place additional constraints to reduce its variance.\n",
    "\n",
    "As a consequence the bias is increased but can introduce a regularisation parameter to control the tradeoff."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Add regularization term $R(\\theta)$ to the cost function:\n",
    "\n",
    "$$C_\\lambda(\\theta) = C(\\theta) + \\lambda\\ R(\\theta).$$\n",
    "\n",
    "The regularization parameter $\\lambda$ controls the amount of regularization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Regularization should only be added when training.  When using fitted model to make predictions, should evaluate cost without regularization term."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Tikhonov regularization \n",
    "\n",
    "*Tikhonov* regularization adopts $\\ell_2$ regularising term (also called *Ridge regression*):\n",
    "\n",
    "$$ R(\\theta) = \\frac{1}{2} \\sum_{j=1}^n \\theta_j^2 = \\frac{1}{2}  \\theta^{\\rm T}\\theta.$$\n",
    "\n",
    "\n",
    "\n",
    "Acts to keep parameters small.\n",
    "\n",
    "Note that the bias term $\\theta_0$ is not regularized (i.e. sum starts from 1 not 0)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:19:46.887064Z",
     "iopub.status.busy": "2024-01-10T00:19:46.886396Z",
     "iopub.status.idle": "2024-01-10T00:19:46.893828Z",
     "shell.execute_reply": "2024-01-10T00:19:46.893284Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import Ridge\n",
    "\n",
    "np.random.seed(42)\n",
    "m = 20\n",
    "X = 3 * np.random.rand(m, 1)\n",
    "y = 1 + 0.5 * X + np.random.randn(m, 1) / 1.5\n",
    "X_new = np.linspace(0, 3, 100).reshape(100, 1)\n",
    "\n",
    "def plot_model(model_class, polynomial, alphas, **model_kargs): \n",
    "    # Use alpha for regularization parameter (lambda used already)\n",
    "    for alpha, style in zip(alphas, (\"b-\", \"g--\", \"r:\")):\n",
    "        model = model_class(alpha, **model_kargs) if alpha > 0 else LinearRegression()\n",
    "        if polynomial:\n",
    "            model = Pipeline((\n",
    "                    (\"poly_features\", PolynomialFeatures(degree=10, include_bias=False)),\n",
    "                    (\"std_scaler\", StandardScaler()),\n",
    "                    (\"regul_reg\", model),\n",
    "                ))\n",
    "        model.fit(X, y)\n",
    "        y_new_regul = model.predict(X_new)\n",
    "        lw = 2 if alpha > 0 else 1\n",
    "        plt.plot(X_new, y_new_regul, style, linewidth=lw, label=r\"$\\lambda = {}$\".format(alpha))\n",
    "    plt.plot(X, y, \"b.\", linewidth=3)\n",
    "    plt.legend(loc=\"upper left\", fontsize=15)\n",
    "    plt.xlabel(\"$x_1$\", fontsize=18)\n",
    "    plt.axis([0, 3, 0, 4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:19:46.896773Z",
     "iopub.status.busy": "2024-01-10T00:19:46.896231Z",
     "iopub.status.idle": "2024-01-10T00:19:47.338341Z",
     "shell.execute_reply": "2024-01-10T00:19:47.337595Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 61640 characters]",
      "text/plain": [
       "<Figure size 1000x500 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(10,5))\n",
    "plt.subplot(121)\n",
    "plot_model(Ridge, polynomial=False, alphas=(0, 10, 100), random_state=42)\n",
    "plt.ylabel(\"$y$\", rotation=0, fontsize=18)\n",
    "plt.subplot(122)\n",
    "plot_model(Ridge, polynomial=True, alphas=(0, 10**-5, 1), random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "exercise_pointer"
    ]
   },
   "source": [
    "**Exercises:** *You can now complete Exercises 2 and 3 in the exercises associated with this lecture.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Lasso regularization\n",
    "\n",
    "\n",
    "*Lasso* regularization adopts $\\ell_1$ regularising term:\n",
    "\n",
    "$$ R(\\theta) =\\sum_{j=1}^n \\left\\vert \\theta_j \\right\\vert .$$\n",
    "\n",
    "Acts to promote sparsity.\n",
    "\n",
    "Again, note that the bias term $\\theta_0$ is not regularized (i.e. sum starts from 1 not 0).\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:19:47.341641Z",
     "iopub.status.busy": "2024-01-10T00:19:47.341174Z",
     "iopub.status.idle": "2024-01-10T00:19:47.722342Z",
     "shell.execute_reply": "2024-01-10T00:19:47.721671Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 53228 characters]",
      "text/plain": [
       "<Figure size 1000x500 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.linear_model import Lasso\n",
    "\n",
    "plt.figure(figsize=(10,5))\n",
    "plt.subplot(121)\n",
    "plot_model(Lasso, polynomial=False, alphas=(0, 0.1, 1), random_state=42)\n",
    "plt.ylabel(\"$y$\", rotation=0, fontsize=18)\n",
    "plt.subplot(122)\n",
    "plot_model(Lasso, polynomial=True, alphas=(0, 10**-7, 1), tol=1, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Differentiability\n",
    "\n",
    "Note that the Lasso penality is non-differentiable at zero.\n",
    "\n",
    "Gradient descent can still be used but with gradients replaced by [sub-gradients](https://en.wikipedia.org/wiki/Subderivative) when any $\\theta_j=0$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Elastic Net regularization\n",
    "\n",
    "\n",
    "Provides a mix of Tikhonov and Lasso regularization, controlled by mix ratio $r$:\n",
    "\n",
    "$$R(\\theta)\n",
    "= \n",
    "r\\sum_{j=1}^n \\left\\vert \\theta_j \\right\\vert\n",
    "+\n",
    "\\frac{1-r}{2} \\sum_{j=1}^n \\theta_j^2.$$\n",
    "\n",
    "- For $r=0$, corresponds to Tikhonov regularization.\n",
    "- For $r=1$, corresponds to Lasso regularization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Stopping early\n",
    "\n",
    "Compute RMSE on validation set as train and stop when starts to increase."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:19:47.725983Z",
     "iopub.status.busy": "2024-01-10T00:19:47.725495Z",
     "iopub.status.idle": "2024-01-10T00:19:47.734570Z",
     "shell.execute_reply": "2024-01-10T00:19:47.733897Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import SGDRegressor\n",
    "np.random.seed(42)\n",
    "m = 100\n",
    "X = 6 * np.random.rand(m, 1) - 3\n",
    "y = 2 + X + 0.5 * X**2 + np.random.randn(m, 1)\n",
    "\n",
    "X_train, X_val, y_train, y_val = train_test_split(X[:50], y[:50].ravel(), test_size=0.5, random_state=10)\n",
    "\n",
    "poly_scaler = Pipeline((\n",
    "        (\"poly_features\", PolynomialFeatures(degree=90, include_bias=False)),\n",
    "        (\"std_scaler\", StandardScaler()),\n",
    "    ))\n",
    "\n",
    "X_train_poly_scaled = poly_scaler.fit_transform(X_train)\n",
    "X_val_poly_scaled = poly_scaler.transform(X_val)\n",
    "\n",
    "sgd_reg = SGDRegressor(max_iter=1,\n",
    "                       penalty=None,\n",
    "                       eta0=0.0005,\n",
    "                       warm_start=True,\n",
    "                       learning_rate=\"constant\",\n",
    "                       random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:19:47.737763Z",
     "iopub.status.busy": "2024-01-10T00:19:47.737190Z",
     "iopub.status.idle": "2024-01-10T00:19:48.426308Z",
     "shell.execute_reply": "2024-01-10T00:19:48.425611Z"
    }
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings(action='ignore')\n",
    "n_epochs = 500\n",
    "train_errors, val_errors = [], []\n",
    "for epoch in range(n_epochs):\n",
    "    sgd_reg.fit(X_train_poly_scaled, y_train)\n",
    "    y_train_predict = sgd_reg.predict(X_train_poly_scaled)\n",
    "    y_val_predict = sgd_reg.predict(X_val_poly_scaled)\n",
    "    train_errors.append(mean_squared_error(y_train_predict, y_train))\n",
    "    val_errors.append(mean_squared_error(y_val_predict, y_val))\n",
    "\n",
    "best_epoch = np.argmin(val_errors)\n",
    "best_val_rmse = np.sqrt(val_errors[best_epoch])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:19:48.429987Z",
     "iopub.status.busy": "2024-01-10T00:19:48.429495Z",
     "iopub.status.idle": "2024-01-10T00:19:48.654702Z",
     "shell.execute_reply": "2024-01-10T00:19:48.653840Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 46580 characters]",
      "text/plain": [
       "<Figure size 1000x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(10,5))\n",
    "plt.annotate('Best model',\n",
    "             xy=(best_epoch, best_val_rmse),\n",
    "             xytext=(best_epoch, best_val_rmse + 1),\n",
    "             ha=\"center\",\n",
    "             arrowprops=dict(facecolor='black', shrink=0.05),\n",
    "             fontsize=16,\n",
    "            )\n",
    "\n",
    "best_val_rmse -= 0.03  # just to make the graph look better\n",
    "plt.plot([0, n_epochs], [best_val_rmse, best_val_rmse], \"k:\", linewidth=2)\n",
    "plt.plot(np.sqrt(val_errors), \"b-\", linewidth=3, label=\"Validation set\")\n",
    "plt.plot(np.sqrt(train_errors), \"r--\", linewidth=2, label=\"Training set\")\n",
    "plt.legend(loc=\"upper right\", fontsize=14)\n",
    "plt.xlabel(\"Epoch\", fontsize=14)\n",
    "plt.ylabel(\"RMSE\", fontsize=14);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Note that for stochastic or mini-batch gradient descent the RMSE is noisy and may be difficult to know when reach minimum (can employ simple strategies to deal with that)."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
