{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Lecture 5: Training I"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "![](https://www.tensorflow.org/images/colab_logo_32px.png)\n",
    "[Run in colab](https://colab.research.google.com/drive/10z5cZZcHnp1cfCRiD4ESsdwLR7NFb5p0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:19:25.718881Z",
     "iopub.status.busy": "2024-01-10T00:19:25.718648Z",
     "iopub.status.idle": "2024-01-10T00:19:25.726287Z",
     "shell.execute_reply": "2024-01-10T00:19:25.725751Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last executed: 2024-01-10 00:19:25\n"
     ]
    }
   ],
   "source": [
    "import datetime\n",
    "now = datetime.datetime.now()\n",
    "print(\"Last executed: \" + now.strftime(\"%Y-%m-%d %H:%M:%S\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Linear regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Linear regression mode (scalar form)\n",
    "\n",
    "$$\\hat{y} = \\theta_0 + \\theta_1 x_1 + ... + \\theta_n x_n$$\n",
    "\n",
    "- $\\hat{y}$ is the predicted value.\n",
    "- $n$ is the number of features.\n",
    "- $x_j$ is the $j$th feature, for $j=0,1,...,n$.\n",
    "- $\\theta_j$ is the $j$th model parameter\n",
    "<br>(with bias $\\theta_0$ and feature weights $\\theta_1, ..., \\theta_n)$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Linear regression mode (vector form)\n",
    "\n",
    "$$\\hat{y} =\\theta^{\\rm T} x = h_\\theta(x)$$\n",
    "\n",
    "- $x$ is the instance feature vector $x=(x_0, x_1, ... x_n)^{\\rm T}$, where $x_0 = 1$.\n",
    "- $\\theta$ is the vector of model parameters $\\theta=(\\theta_0, \\theta_1, ... \\theta_n)^{\\rm T}$.\n",
    "- $\\cdot^{\\rm T}$ denotes transpose.\n",
    "- $h_\\theta(x)$ is the hypothesis function, with parameters $\\theta$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Train linear regression\n",
    "\n",
    "Give training data $\\{ x^{(i)}, y^{(i)} \\}$, for $m$ instances $i=1,2,...,m$.\n",
    "\n",
    "Minimise mean square error (MSE):\n",
    "\n",
    "$$\\text{MSE} = \\frac{1}{m} \\sum_{i=1}^{m} \\left(\\theta^{\\rm T} x^{(i)} - y^{(i)}\\right)^2 .$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Concise matrix-vector notation\n",
    "\n",
    "Recall:\n",
    "- $x_j^{(i)}$ is (scalar) value of feature $j$ in $i$th training example.\n",
    "- $x^{(i)}$ is the (column) vector of features of the $i$th training example.\n",
    "- $y^{(i)}$ is the (scalar) value of target of the $i$th training example.\n",
    "- $n$ features and $m$ training instances\n",
    "\n",
    "Define:\n",
    "- Feature matrix: $X_{m \\times n} = [ x^{(1)},\\ x^{(2)},\\ ...,\\ x^{(m)}]^{\\rm T}$.\n",
    "- Target vector: $y_{m \\times 1} = [ y^{(1)},\\ y^{(2)},\\ ...,\\ y^{(m)}]^{\\rm T}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Recall features matrix and target vector\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/astro-informatics/course_mlbd_images/master/Lecture05_Images/data-layout.png\" width=\"500\"  style=\"display:block; margin:auto\"/>\n",
    "\n",
    "[Image source](https://github.com/jakevdp/sklearn_tutorial)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Minimising the MSE is equivalent to minimising the cost function\n",
    "\n",
    "$$C(\\theta) = \\frac{1}{m} (X \\theta - y)^{\\rm T}(X \\theta - y),$$\n",
    "\n",
    "where for notational convenience we denote the dependence on $\\theta$ only and consider $X$ and $y$ fixed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Normal equations\n",
    "\n",
    "Minimise the cost function analytically \n",
    "\n",
    "$$\\min_\\theta\\ C(\\theta) = \\min_\\theta \\ (X \\theta - y)^{\\rm T}(X \\theta - y).$$\n",
    "\n",
    "Solution given by \n",
    "\n",
    "$$ \\hat{\\theta} = \\left( X^{\\rm T} X \\right)^{-1} X^{\\rm T} y. $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:19:25.767150Z",
     "iopub.status.busy": "2024-01-10T00:19:25.766614Z",
     "iopub.status.idle": "2024-01-10T00:19:26.182564Z",
     "shell.execute_reply": "2024-01-10T00:19:26.181848Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Common imports\n",
    "import os\n",
    "import numpy as np\n",
    "np.random.seed(42) # To make this notebook's output stable across runs\n",
    "\n",
    "# To plot pretty figures\n",
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams['axes.labelsize'] = 14\n",
    "plt.rcParams['xtick.labelsize'] = 12\n",
    "plt.rcParams['ytick.labelsize'] = 12"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:19:26.186110Z",
     "iopub.status.busy": "2024-01-10T00:19:26.185473Z",
     "iopub.status.idle": "2024-01-10T00:19:26.426019Z",
     "shell.execute_reply": "2024-01-10T00:19:26.425332Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 22856 characters]",
      "text/plain": [
       "<Figure size 900x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "m = 100\n",
    "X = 2 * np.random.rand(m, 1)\n",
    "y = 4 + 3 * X + np.random.randn(m, 1)\n",
    "plt.figure(figsize=(9,6))\n",
    "plt.plot(X, y, \"b.\")\n",
    "plt.xlabel(\"$x_1$\", fontsize=18)\n",
    "plt.ylabel(\"$y$\", rotation=0, fontsize=18)\n",
    "plt.axis([0, 2, 0, 15]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "exercise_pointer"
    ]
   },
   "source": [
    "**Exercises:** *You can now complete Exercise 1 in the exercises associated with this lecture.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Solve using Scikit-Learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:19:26.430231Z",
     "iopub.status.busy": "2024-01-10T00:19:26.428868Z",
     "iopub.status.idle": "2024-01-10T00:19:26.811959Z",
     "shell.execute_reply": "2024-01-10T00:19:26.811220Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([4.21509616]), array([[2.77011339]]))"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "lin_reg = LinearRegression()\n",
    "lin_reg.fit(X, y)\n",
    "lin_reg.intercept_, lin_reg.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:19:26.815257Z",
     "iopub.status.busy": "2024-01-10T00:19:26.814784Z",
     "iopub.status.idle": "2024-01-10T00:19:26.820509Z",
     "shell.execute_reply": "2024-01-10T00:19:26.819869Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[4.21509616],\n",
       "       [9.75532293]])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_new = np.array([[0], [2]])\n",
    "lin_reg.predict(X_new)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Computational complexity\n",
    "\n",
    "Solving the normal equation requires inverting $X^{\\rm T} X$, which is an $n \\times n$ matrix (where $n$ is the number of features).\n",
    "\n",
    "Inverting an $n \\times n$ matrix is naively of complexity $\\mathcal{O}(n^3)$ ($\\mathcal{O}(n^{2.4})$ if certain fast algorithms are used).\n",
    "\n",
    "Also, typically requires all training instances to be held in memory at once.\n",
    "\n",
    "Other methods are required to handle large data-sets, i.e. when considering large numbers of features and large numbers of training instances."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Batch gradient descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://raw.githubusercontent.com/astro-informatics/course_mlbd_images/master/Lecture05_Images/optimization_gd.png\" width=\"400px\" style=\"display:block; margin:auto\"/>\n",
    "\n",
    "[[Image source](http://www.holehouse.org/mlclass)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Gradient of the cost function is given by\n",
    "\n",
    "$$ \\frac{\\partial}{\\partial \\theta} C(\\theta)\n",
    "=\\frac{2}{m} X^{\\rm T}  \\left( X \\theta - y \\right) \n",
    "= \\left[\\frac{\\partial C}{\\partial \\theta_0},\\ \\frac{\\partial C}{\\partial \\theta_1},\\ ...,\\ \\frac{\\partial C}{\\partial \\theta_n} \\right]^{\\rm T} \n",
    ".\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Batch gradient descent algorithm defined by taking a step $\\alpha$ in the direction of the gradient:\n",
    "\n",
    "$$\\theta^{(t)} = \\theta^{(t-1)} - \\alpha  \\frac{\\partial C}{\\partial \\theta},$$\n",
    "\n",
    "where $t$ denotes the iteration number."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "This algorithm is called *batch* gradient descent since it uses the full *batch* of training data ($X$) at each iteration.  We will see other forms of gradient descent soon..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "exercise_pointer"
    ]
   },
   "source": [
    "**Exercises:** *You can now complete Exercise 2 in the exercises associated with this lecture.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Learning rate\n",
    "\n",
    "The step size $\\alpha$ is also called the *learning rate*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Good learning rate\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/astro-informatics/course_mlbd_images/master/Lecture05_Images/bgd_alpha_good.png\" width=\"600px\" style=\"display:block; margin:auto\"/>\n",
    "\n",
    "[Source: Geron]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Learning rate too small \n",
    "\n",
    "Convergence will be slow.\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/astro-informatics/course_mlbd_images/master/Lecture05_Images/bgd_alpha_small.png\" width=\"600px\" style=\"display:block; margin:auto\"/>\n",
    "\n",
    "[Source: Geron]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Learning rate too large \n",
    "\n",
    "May not converge.\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/astro-informatics/course_mlbd_images/master/Lecture05_Images/bgd_alpha_large.png\" alt=\"data-layout\" width=\"600px\" style=\"display:block; margin:auto\"/>\n",
    "\n",
    "[Source: Geron]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Evolution of fitted curve \n",
    "\n",
    "Consider the evolution of the curve corresponding to the best fit parameters for the first 10 iterations for different learning rates $\\alpha$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:19:26.824434Z",
     "iopub.status.busy": "2024-01-10T00:19:26.823834Z",
     "iopub.status.idle": "2024-01-10T00:19:26.831253Z",
     "shell.execute_reply": "2024-01-10T00:19:26.830646Z"
    }
   },
   "outputs": [],
   "source": [
    "theta_path_bgd = []\n",
    "\n",
    "X_b = np.c_[np.ones((m, 1)), X]  # add x0 = 1 to each instance\n",
    "X_new_b = np.c_[np.ones((2, 1)), X_new] \n",
    "\n",
    "def plot_gradient_descent(theta, alpha, theta_path=None):\n",
    "    m = len(X_b)\n",
    "    plt.plot(X, y, \"b.\")\n",
    "    n_iterations = 1000\n",
    "    for iteration in range(n_iterations):\n",
    "        if iteration < 10:\n",
    "            y_predict = X_new_b.dot(theta)\n",
    "            style = \"b-\" if iteration > 0 else \"r--\"\n",
    "            plt.plot(X_new, y_predict, style)\n",
    "        gradients = 2/m * X_b.T.dot(X_b.dot(theta) - y)\n",
    "        theta = theta - alpha * gradients\n",
    "        if theta_path is not None:\n",
    "            theta_path.append(theta)\n",
    "    plt.xlabel(\"$x_1$\", fontsize=18)\n",
    "    plt.axis([0, 2, 0, 15])\n",
    "    plt.title(r\"$\\alpha = {}$\".format(alpha), fontsize=16)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:19:26.834198Z",
     "iopub.status.busy": "2024-01-10T00:19:26.833760Z",
     "iopub.status.idle": "2024-01-10T00:19:27.369568Z",
     "shell.execute_reply": "2024-01-10T00:19:27.368907Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 70212 characters]",
      "text/plain": [
       "<Figure size 1000x400 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "theta = np.random.randn(2,1)  # random initialization\n",
    "\n",
    "plt.figure(figsize=(10,4))\n",
    "plt.subplot(131); plot_gradient_descent(theta, alpha=0.02)\n",
    "plt.ylabel(\"$y$\", rotation=0, fontsize=18)\n",
    "plt.subplot(132); plot_gradient_descent(theta, alpha=0.1)\n",
    "plt.subplot(133); plot_gradient_descent(theta, alpha=0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- $\\alpha=0.02$: learning rate is too small and will take a long time to reach the global optimum.\n",
    "- $\\alpha=0.1$: learning rate is good and global optimum will be found quickly.\n",
    "- $\\alpha=0.5$: learning rate is too large and parameters jump about (may not converge)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Convergence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cost function may not be nice with a single local minimum that coincides with the global minimum.\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/astro-informatics/course_mlbd_images/master/Lecture05_Images/bgd_pitfalls.png\" alt=\"data-layout\" width=\"600px\" style=\"display:block; margin:auto\"/>\n",
    "\n",
    "[Source: Geron]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Convex objective function\n",
    "\n",
    "MSE cost function for linear regression is convex (for any two points on the curve, the line segment connecting those points lies above the curve).\n",
    "\n",
    "Consequently, the cost function has a single local minimum (that therefore coincides with the global minimum).\n",
    "\n",
    "Furthermore, the cost function is smooth with a slope that never changes abruptly (i.e. it is Lipschitz continuous).  \n",
    "\n",
    "Gradient descent is therefore guaranteed to converge to the global minimum."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Feature scaling\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For gradient descent to work well it is critical for all features to have a similar scale."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://raw.githubusercontent.com/astro-informatics/course_mlbd_images/master/Lecture05_Images/bgd_feature_scaling.png\" alt=\"data-layout\" width=\"800px\" style=\"display:block; margin:auto\"/>\n",
    "\n",
    "[Source: Geron]\n",
    "\n",
    "In the case on the left, the features have the same scale and even intially the solution moves towards the minimum, thus reaching it quickly.\n",
    "\n",
    "In the case on the right, the features have different scales and initially the solution does not move directly towards the minimum.  Convergence is therefore slow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Feature scaling methods\n",
    "\n",
    "In general should ensure features have a similar scale since many machine learning methods do not work well when numerical attributes have very different scales.\n",
    "\n",
    "#### Min-max scaling\n",
    "\n",
    "Min-max scaling given by\n",
    "\\begin{align*}\n",
    "x_j \\rightarrow \\frac{x_j - \\min_i x_i}{\\max_i x_i - \\min_i x_i} .\n",
    "\\end{align*}\n",
    "See Scikit-Learn [`MinMaxScalar`](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html).\n",
    "\n",
    "\n",
    "#### Standardization\n",
    "\n",
    "Standardization scaling given by\n",
    "\\begin{align*}\n",
    "x_j \\rightarrow \\frac{x_j - \\mu_j}{\\sigma_j},\n",
    "\\end{align*}\n",
    "where $\\mu_j$ and $\\sigma_j$ are the mean and standard deviation of feature $j$ computed from the training instances.\n",
    "See Scikit-Learn [`StandardScaler`](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html)."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
