{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Lecture 9: Support vector machines (SVMs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "![](https://www.tensorflow.org/images/colab_logo_32px.png)\n",
    "[Run in colab](https://colab.research.google.com/drive/1t9EbYLj41BV5KnTU-zFIVM-K87yatl5e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:07.514586Z",
     "iopub.status.busy": "2024-01-10T00:20:07.514123Z",
     "iopub.status.idle": "2024-01-10T00:20:07.521951Z",
     "shell.execute_reply": "2024-01-10T00:20:07.521418Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last executed: 2024-01-10 00:20:07\n"
     ]
    }
   ],
   "source": [
    "import datetime\n",
    "now = datetime.datetime.now()\n",
    "print(\"Last executed: \" + now.strftime(\"%Y-%m-%d %H:%M:%S\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Large margin classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Logistic regression with Hinge loss\n",
    "\n",
    "Recall cost function for logistic regression, with $\\ell_2$ regularisation, is given by\n",
    "\n",
    "\\begin{align*}\n",
    "C(\\theta) = -\\frac{1}{m} \\sum_{i=1}^m \n",
    "\\left [ \n",
    "y^{(i)} \\log(\\hat{p}^{(i)})\n",
    "+\n",
    "(1 - y^{(i)}) \\log(1 - \\hat{p}^{(i)})\n",
    "\\right]\n",
    "+\n",
    "\\frac{\\lambda}{2m}\n",
    "\\sum_{j=1}^n \\theta_j^2,\n",
    "\\end{align*}\n",
    "\n",
    "where \n",
    "$\\hat{p} = h_\\theta(x) = \\sigma(\\theta^{\\rm T} x) \\quad \\text{and} \\quad \\sigma(t) = \\frac{1}{1+\\exp{(-t)}}.$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Recall, the bias $\\theta_0$ is not regularised, i.e. sum over $j$ starts at 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Plot sigmoid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:07.563118Z",
     "iopub.status.busy": "2024-01-10T00:20:07.562607Z",
     "iopub.status.idle": "2024-01-10T00:20:07.972568Z",
     "shell.execute_reply": "2024-01-10T00:20:07.971878Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:07.976305Z",
     "iopub.status.busy": "2024-01-10T00:20:07.975558Z",
     "iopub.status.idle": "2024-01-10T00:20:08.219979Z",
     "shell.execute_reply": "2024-01-10T00:20:08.219258Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 26932 characters]",
      "text/plain": [
       "<Figure size 800x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "t = np.linspace(-4, 4, 100)\n",
    "sig = 1 / (1 + np.exp(-t))\n",
    "plt.figure(figsize=(8, 4))\n",
    "plt.plot([-4, 4], [0, 0], \"k-\")\n",
    "plt.plot([-4, 4], [0.5, 0.5], \"k:\")\n",
    "plt.plot([-4, 4], [1, 1], \"k:\")\n",
    "plt.plot([0, 0], [-1.1, 1.1], \"k-\")\n",
    "plt.plot(t, sig, \"b-\", linewidth=2, label=r\"$\\sigma(t) = \\frac{1}{1 + e^{-t}}$\")\n",
    "plt.xlabel(\"t\")\n",
    "plt.legend(loc=\"upper left\", fontsize=20)\n",
    "plt.axis([-4, 4, -0.1, 1.1]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Hinge loss\n",
    "\n",
    "Logistic regression cost function for reference: \n",
    "\n",
    "\\begin{align*}\n",
    "C(\\theta) = -\\frac{1}{m} \\sum_{i=1}^m \n",
    "\\left [ \n",
    "y^{(i)} \\log(\\hat{p}^{(i)})\n",
    "+\n",
    "(1 - y^{(i)}) \\log(1 - \\hat{p}^{(i)})\n",
    "\\right]\n",
    "+\n",
    "\\frac{\\lambda}{2m}\n",
    "\\sum_{j=1}^n \\theta_j^2.\n",
    "\\end{align*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "#### Positive training instances\n",
    "\n",
    "For $y^{(i)}=1$, replace cost $-\\log(\\sigma(\\theta^{\\rm T} x))$ with *hinge loss* $\\max(1 - \\theta^{\\rm T} x, 0)$.\n",
    "\n",
    "For training, we want not just $\\theta^{\\rm T} x \\geq 0$ but $\\theta^{\\rm T} x \\geq 1$. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "inclass_exercise"
    ]
   },
   "source": [
    "#### Exercise: \n",
    "\n",
    "Plot cost $-\\log(\\sigma(\\theta^{\\rm T} x))$ and hinge loss $\\max(1 - \\theta^{\\rm T} x, 0)$ ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Logistic regression cost function for reference: \n",
    "\\begin{align*}\n",
    "C(\\theta) = -\\frac{1}{m} \\sum_{i=1}^m \n",
    "\\left [ \n",
    "y^{(i)} \\log(\\hat{p}^{(i)})\n",
    "+\n",
    "(1 - y^{(i)}) \\log(1 - \\hat{p}^{(i)})\n",
    "\\right]\n",
    "+\n",
    "\\frac{\\lambda}{2m}\n",
    "\\sum_{j=1}^n \\theta_j^2.\n",
    "\\end{align*}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:08.223354Z",
     "iopub.status.busy": "2024-01-10T00:20:08.222895Z",
     "iopub.status.idle": "2024-01-10T00:20:08.226877Z",
     "shell.execute_reply": "2024-01-10T00:20:08.226303Z"
    },
    "tags": [
     "inclass_exercise",
     "solution"
    ]
   },
   "outputs": [],
   "source": [
    "t = np.linspace(-4, 4, 100)\n",
    "sig = 1 / (1 + np.exp(-t))\n",
    "cost_one = -np.log(sig)\n",
    "cost_one_hinge = np.maximum(1 - t, np.zeros(t.size))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:08.229821Z",
     "iopub.status.busy": "2024-01-10T00:20:08.229175Z",
     "iopub.status.idle": "2024-01-10T00:20:08.462823Z",
     "shell.execute_reply": "2024-01-10T00:20:08.462159Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": [
     "inclass_exercise",
     "solution"
    ]
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 42844 characters]",
      "text/plain": [
       "<Figure size 800x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(8, 4))\n",
    "plt.plot([0, 0], [0, 5], \"k-\")\n",
    "plt.plot(t, cost_one, \"b-\", linewidth=2, label=r\"$-\\log{(\\sigma(\\theta^{\\rm T} x))}$\")\n",
    "plt.xlabel(r\"$\\theta^{\\rm T} x$\", fontsize=20)\n",
    "plt.axis([-4, 4, 0, 5]);\n",
    "plt.title('Cost for $y=1$', fontsize=20)\n",
    "plt.plot(t, cost_one_hinge, \"r-\", linewidth=2, label=r\"$\\max(1-\\theta^{\\rm T} x, 0)$\")\n",
    "plt.legend(loc=\"upper right\", fontsize=20);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Negative training instances\n",
    "\n",
    "For $y^{(i)}=0$, replace $-\\log(1-\\sigma(\\theta^{\\rm T} x))$ with *hinge loss* $\\max(1 + \\theta^{\\rm T} x, 0)$.\n",
    "\n",
    "For training, we want not just $\\theta^{\\rm T} x < 0$ but $\\theta^{\\rm T} x \\leq -1$. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "inclass_exercise"
    ]
   },
   "source": [
    "#### Exercise: \n",
    "\n",
    "Plot cost $-\\log(1-\\sigma(\\theta^{\\rm T} x))$ and hinge loss $\\max(1 + \\theta^{\\rm T} x, 0)$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:08.466146Z",
     "iopub.status.busy": "2024-01-10T00:20:08.465732Z",
     "iopub.status.idle": "2024-01-10T00:20:08.470442Z",
     "shell.execute_reply": "2024-01-10T00:20:08.469824Z"
    },
    "tags": [
     "inclass_exercise",
     "solution"
    ]
   },
   "outputs": [],
   "source": [
    "cost_zero = -np.log(1-sig)\n",
    "cost_zero_hinge = np.maximum(1 + t, np.zeros(t.size))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:08.473267Z",
     "iopub.status.busy": "2024-01-10T00:20:08.472901Z",
     "iopub.status.idle": "2024-01-10T00:20:08.747961Z",
     "shell.execute_reply": "2024-01-10T00:20:08.747253Z"
    },
    "tags": [
     "inclass_exercise",
     "solution"
    ]
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 47732 characters]",
      "text/plain": [
       "<Figure size 1000x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(10, 5))\n",
    "plt.plot([0, 0], [0, 5], \"k-\")\n",
    "plt.plot(t, cost_zero, \"b-\", linewidth=2, label=r\"$-\\log{(1-\\sigma(\\theta^{\\rm T} x))}$\")\n",
    "plt.xlabel(r\"$\\theta^{\\rm T} x$\", fontsize=20)\n",
    "plt.axis([-4, 4, 0, 5]);\n",
    "plt.title('Cost for $y=0$', fontsize=20);\n",
    "plt.plot(t, cost_zero_hinge, \"r-\", linewidth=2, label=r\"$max(1+\\theta^{\\rm T} x, 0)$\")\n",
    "plt.legend(loc=\"upper left\", fontsize=20);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Replace costs with hinge loss functions\n",
    "\n",
    "Recall logistic regression with $\\ell_2$ regularisation cost function is given by\n",
    "\n",
    "\\begin{align*}\n",
    "C(\\theta) = -\\frac{1}{m} \\sum_{i=1}^m \n",
    "\\left [ \n",
    "y^{(i)} \\log(\\hat{p}^{(i)})\n",
    "+\n",
    "(1 - y^{(i)}) \\log(1 - \\hat{p}^{(i)})\n",
    "\\right]\n",
    "+\n",
    "\\frac{\\lambda}{2m}\n",
    "\\sum_{j=1}^n \\theta_j^2.\n",
    "\\end{align*}\n",
    "\n",
    "Replace costs for $y^{(i)}=1$ and $y^{(i)}=0$ with hinge losses given above:\n",
    "\\begin{align*}\n",
    "\\min_\\theta\n",
    "\\sum_{i=1}^m \n",
    "\\left [ \n",
    "y^{(i)} \n",
    "\\max(1 - \\theta^{\\rm T} x^{(i)}, 0)\n",
    "+\n",
    "(1 - y^{(i)}) \n",
    "\\max(1 + \\theta^{\\rm T} x^{(i)}, 0)\n",
    "\\right]\n",
    "+\n",
    "\\frac{\\lambda}{2}\n",
    "\\sum_{j=1}^n \\theta_j^2.\n",
    "\\end{align*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Introduce $k^{(i)}=1$ for positive instances ($y^{(i)}=1$) and $k^{(i)}=-1$ for negative instances ($y^{(i)}=0$):\n",
    "\n",
    "\\begin{align*}\n",
    "\\Rightarrow\n",
    "\\min_\\theta\n",
    "C\n",
    "\\sum_{i=1}^m \n",
    "\\max(1 - k^{(i)} \\theta^{\\rm T} x^{(i)}, 0)\n",
    "+\n",
    "\\frac{1}{2}\n",
    "\\sum_{j=1}^n \\theta_j^2\n",
    "\\end{align*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The convention is to weight the fidelity term by C rather than regularisation term by $\\lambda$ (thus $C$ plays the role of $1/\\lambda$).\n",
    "- Large $C$ $\\rightarrow$ little regularisation \n",
    "- Small $C$ $\\rightarrow$ greater regularisation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Constrained objective problem\n",
    "\n",
    "So far we considered the *unconstrained* objective problem by adapting the logistic regression cost function:\n",
    "\\begin{align*}\n",
    "\\min_\\theta\n",
    "C\n",
    "\\sum_{i=1}^m \n",
    "\\max(1 - k^{(i)} \\theta^{\\rm T} x^{(i)}, 0)\n",
    "+\n",
    "\\frac{1}{2}\n",
    "\\sum_{j=1}^n \\theta_j^2\n",
    "\\end{align*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "We can also consider the *constrained* objective problem:\n",
    "\\begin{align*}\n",
    "\\min_\\theta\n",
    "\\frac{1}{2}\n",
    "\\sum_{j=1}^n \\theta_j^2\n",
    "\\quad\n",
    "\\text{subject to}\n",
    "\\quad\n",
    "k^{(i)} \\theta^{\\rm T} x^{(i)} \\geq 1 \\ \\text{for}\\ i=1,2,\\ldots,m\n",
    "\\end{align*}\n",
    "\n",
    "(Follows intuitively by considering large $C$.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Intuition for large margin classification\n",
    "\n",
    "Decision boundary defined by $\\theta^{\\rm T} x + b = 0$ (where here $\\theta$ does *not* include bias and $b=\\theta_0$; in notation above $\\theta$ did include bias, i.e. $\\theta_0$ term).\n",
    "\n",
    "Consequently, $\\theta$ is orthogonal to decision boundary."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Recall constrained objective:\n",
    "\n",
    "\\begin{align*}\n",
    "\\min_\\theta\n",
    "\\frac{1}{2}\n",
    "\\sum_{j=1}^n \\theta_j^2\n",
    "\\quad\n",
    "\\text{subject to}\n",
    "\\quad\n",
    "k^{(i)} \\theta^{\\rm T} x^{(i)} \\geq 1 \\ \\text{for}\\ i=1,2,\\ldots,m\n",
    "\\end{align*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Projection\n",
    "\n",
    "Note that the term $k^{(i)} \\theta^{\\rm T} x^{(i)}$ is related to the projection of $x^{(i)}$ onto $\\theta$:\n",
    "\n",
    "$$k^{(i)} \\theta^{\\rm T} x^{(i)} = p^{(i)} \\| \\theta \\|,$$\n",
    "\n",
    "where $ p^{(i)}$ is the projection of $x^{(i)}$ onto $\\theta$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Attempting to minimise $\\| \\theta \\|$, hence requires $p^{(i)}$ to be large $\\Rightarrow$ large margin classification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Graphical illustration\n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/astro-informatics/course_mlbd_images/master/Lecture09_Images/SVM_optimize_cropped.png\" width=\"600px\" style=\"display:block; margin:auto\"/>\n",
    "\n",
    "(Note difference notation used: $w=\\theta$ without bias.)\n",
    "\n",
    "[Image source](http://www.saedsayad.com/support_vector_machine.htm)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Training SVMs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:08.751989Z",
     "iopub.status.busy": "2024-01-10T00:20:08.751391Z",
     "iopub.status.idle": "2024-01-10T00:20:08.757625Z",
     "shell.execute_reply": "2024-01-10T00:20:08.757013Z"
    }
   },
   "outputs": [],
   "source": [
    "# Common imports\n",
    "import os\n",
    "import numpy as np\n",
    "np.random.seed(42) # To make this notebook's output stable across runs\n",
    "\n",
    "# To plot pretty figures\n",
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams['axes.labelsize'] = 14\n",
    "plt.rcParams['xtick.labelsize'] = 12\n",
    "plt.rcParams['ytick.labelsize'] = 12"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Load data and train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:08.760717Z",
     "iopub.status.busy": "2024-01-10T00:20:08.760092Z",
     "iopub.status.idle": "2024-01-10T00:20:09.136447Z",
     "shell.execute_reply": "2024-01-10T00:20:09.135746Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC\n",
    "from sklearn import datasets\n",
    "\n",
    "iris = datasets.load_iris()\n",
    "X = iris[\"data\"][:, (2, 3)]  # petal length, petal width\n",
    "y = iris[\"target\"]\n",
    "\n",
    "setosa_or_versicolor = (y == 0) | (y == 1)\n",
    "X = X[setosa_or_versicolor]\n",
    "y = y[setosa_or_versicolor]\n",
    "\n",
    "# SVM Classifier model\n",
    "svm_clf = SVC(kernel=\"linear\", C=np.finfo(float).max)\n",
    "svm_clf.fit(X, y);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Plot decision boundaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:09.140117Z",
     "iopub.status.busy": "2024-01-10T00:20:09.139447Z",
     "iopub.status.idle": "2024-01-10T00:20:09.143979Z",
     "shell.execute_reply": "2024-01-10T00:20:09.143319Z"
    }
   },
   "outputs": [],
   "source": [
    "# Bad models\n",
    "x0 = np.linspace(0, 5.5, 200)\n",
    "#pred_1 = 5*x0 - 20\n",
    "pred_2 = x0 - 1.8\n",
    "pred_3 = 0.1 * x0 + 0.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:09.147122Z",
     "iopub.status.busy": "2024-01-10T00:20:09.146671Z",
     "iopub.status.idle": "2024-01-10T00:20:09.152212Z",
     "shell.execute_reply": "2024-01-10T00:20:09.151513Z"
    }
   },
   "outputs": [],
   "source": [
    "def plot_svc_decision_boundary(svm_clf, xmin, xmax):\n",
    "    w = svm_clf.coef_[0]\n",
    "    b = svm_clf.intercept_[0]\n",
    "\n",
    "    # At the decision boundary, w0*x0 + w1*x1 + b = 0\n",
    "    # => x1 = -w0/w1 * x0 - b/w1\n",
    "    x0 = np.linspace(xmin, xmax, 200)\n",
    "    decision_boundary = -w[0]/w[1] * x0 - b/w[1]\n",
    "\n",
    "    # On the margin, w0*x0 + w1*x1 + b = +/- 1\n",
    "    margin = 1/w[1]\n",
    "    gutter_up = decision_boundary + margin\n",
    "    gutter_down = decision_boundary - margin\n",
    "\n",
    "    svs = svm_clf.support_vectors_\n",
    "    plt.scatter(svs[:, 0], svs[:, 1], s=200, facecolors='#FFAAAA')\n",
    "    plt.plot(x0, decision_boundary, \"k-\", linewidth=2)\n",
    "    plt.plot(x0, gutter_up, \"k--\", linewidth=2)\n",
    "    plt.plot(x0, gutter_down, \"k--\", linewidth=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:09.155462Z",
     "iopub.status.busy": "2024-01-10T00:20:09.155013Z",
     "iopub.status.idle": "2024-01-10T00:20:09.547975Z",
     "shell.execute_reply": "2024-01-10T00:20:09.547289Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 73520 characters]",
      "text/plain": [
       "<Figure size 1600x400 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(16,4))\n",
    "\n",
    "plt.subplot(121)\n",
    "#plt.plot(x0, pred_1, \"g--\", linewidth=2)\n",
    "plt.plot(x0, pred_2, \"m-\", linewidth=2)\n",
    "plt.plot(x0, pred_3, \"r-\", linewidth=2)\n",
    "plt.plot(X[:, 0][y==1], X[:, 1][y==1], \"bs\", label=\"Iris-Versicolor\")\n",
    "plt.plot(X[:, 0][y==0], X[:, 1][y==0], \"yo\", label=\"Iris-Setosa\")\n",
    "plt.xlabel(\"Petal length\", fontsize=14)\n",
    "plt.ylabel(\"Petal width\", fontsize=14)\n",
    "plt.legend(loc=\"upper left\", fontsize=14)\n",
    "plt.axis([0, 5.5, 0, 2])\n",
    "\n",
    "plt.subplot(122)\n",
    "plot_svc_decision_boundary(svm_clf, 0, 5.5)\n",
    "plt.plot(X[:, 0][y==1], X[:, 1][y==1], \"bs\")\n",
    "plt.plot(X[:, 0][y==0], X[:, 1][y==0], \"yo\")\n",
    "plt.xlabel(\"Petal length\", fontsize=14)\n",
    "plt.axis([0, 5.5, 0, 2]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Adding training instances outside the margin will not alter the decision boundary.  \n",
    "\n",
    "Boundary is defined by *support vectors* that are located on the edge of the margin."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "inclass_exercise"
    ]
   },
   "source": [
    "#### Exercise: what value would you guess for the weights $\\theta$ (excluding the bias)?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": [
     "solution",
     "inclass_exercise"
    ]
   },
   "source": [
    "From plot appears $w \\sim [1, 1]^{\\rm T}$ (or slightly more accurately $w \\sim [1.1, 0.9]^{\\rm T}$).\n",
    "\n",
    "Let's check:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:09.551525Z",
     "iopub.status.busy": "2024-01-10T00:20:09.551157Z",
     "iopub.status.idle": "2024-01-10T00:20:09.555816Z",
     "shell.execute_reply": "2024-01-10T00:20:09.555293Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.29411744, 0.82352928])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svm_clf.coef_[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Feature scaling\n",
    "\n",
    "SVMs are sensitive to feature scales, hence feature scaling is important if features not already of similar scale."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:09.558797Z",
     "iopub.status.busy": "2024-01-10T00:20:09.558365Z",
     "iopub.status.idle": "2024-01-10T00:20:09.771970Z",
     "shell.execute_reply": "2024-01-10T00:20:09.771227Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 28712 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "Xs = np.array([[1, 50], [5, 20], [3, 80], [5, 60]]).astype(np.float64)\n",
    "ys = np.array([0, 0, 1, 1])\n",
    "svm_clf = SVC(kernel=\"linear\", C=100)\n",
    "svm_clf.fit(Xs, ys)\n",
    "\n",
    "#plt.figure(figsize=(16,4))\n",
    "#plt.subplot(121)\n",
    "plt.plot(Xs[:, 0][ys==1], Xs[:, 1][ys==1], \"bo\")\n",
    "plt.plot(Xs[:, 0][ys==0], Xs[:, 1][ys==0], \"ms\")\n",
    "plot_svc_decision_boundary(svm_clf, 0, 6)\n",
    "plt.xlabel(\"$x_0$\", fontsize=20)\n",
    "plt.ylabel(\"$x_1$  \", fontsize=20, rotation=0)\n",
    "plt.title(\"Unscaled\", fontsize=16)\n",
    "plt.axis([0, 6, 0, 90]);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:09.775249Z",
     "iopub.status.busy": "2024-01-10T00:20:09.774779Z",
     "iopub.status.idle": "2024-01-10T00:20:09.988009Z",
     "shell.execute_reply": "2024-01-10T00:20:09.987234Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 45308 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "scaler = StandardScaler()\n",
    "X_scaled = scaler.fit_transform(Xs)\n",
    "svm_clf.fit(X_scaled, ys)\n",
    "\n",
    "#plt.subplot(122)\n",
    "plt.plot(X_scaled[:, 0][ys==1], X_scaled[:, 1][ys==1], \"bo\")\n",
    "plt.plot(X_scaled[:, 0][ys==0], X_scaled[:, 1][ys==0], \"ms\")\n",
    "plot_svc_decision_boundary(svm_clf, -2, 2)\n",
    "plt.xlabel(\"$x_0$\", fontsize=20)\n",
    "plt.title(\"Scaled\", fontsize=16)\n",
    "plt.axis([-2, 2, -2, 2]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Hard margin classification\n",
    "\n",
    "Hard margin classification corresponds to strictly imposing all training instances correctly classfied."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Problems with hard margin classification\n",
    "- Can fail if data not linearly separable.\n",
    "- Sensitive to outliers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:09.991474Z",
     "iopub.status.busy": "2024-01-10T00:20:09.990918Z",
     "iopub.status.idle": "2024-01-10T00:20:09.999255Z",
     "shell.execute_reply": "2024-01-10T00:20:09.998651Z"
    }
   },
   "outputs": [],
   "source": [
    "X_outliers = np.array([[3.4, 1.3], [3.2, 0.8]])\n",
    "y_outliers = np.array([0, 0])\n",
    "Xo1 = np.concatenate([X, X_outliers[:1]], axis=0)\n",
    "yo1 = np.concatenate([y, y_outliers[:1]], axis=0)\n",
    "Xo2 = np.concatenate([X, X_outliers[1:]], axis=0)\n",
    "yo2 = np.concatenate([y, y_outliers[1:]], axis=0)\n",
    "\n",
    "svm_clf2 = SVC(kernel=\"linear\", C=10**9)\n",
    "svm_clf2.fit(Xo2, yo2);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:10.002384Z",
     "iopub.status.busy": "2024-01-10T00:20:10.001828Z",
     "iopub.status.idle": "2024-01-10T00:20:10.200031Z",
     "shell.execute_reply": "2024-01-10T00:20:10.199283Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 35936 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plt.figure(figsize=(16,5))\n",
    "# plt.subplot(121)\n",
    "plt.plot(Xo1[:, 0][yo1==1], Xo1[:, 1][yo1==1], \"bs\")\n",
    "plt.plot(Xo1[:, 0][yo1==0], Xo1[:, 1][yo1==0], \"yo\")\n",
    "plt.text(0.3, 1.0, \"Impossible!\", fontsize=24, color=\"red\")\n",
    "plt.xlabel(\"Petal length\", fontsize=14)\n",
    "plt.ylabel(\"Petal width\", fontsize=14)\n",
    "plt.annotate(\"Outlier\",\n",
    "             xy=(X_outliers[0][0], X_outliers[0][1]),\n",
    "             xytext=(2.5, 1.7),\n",
    "             ha=\"center\",\n",
    "             arrowprops=dict(facecolor='black', shrink=0.1),\n",
    "             fontsize=16,\n",
    "            )\n",
    "plt.axis([0, 5.5, 0, 2]);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:10.203351Z",
     "iopub.status.busy": "2024-01-10T00:20:10.202872Z",
     "iopub.status.idle": "2024-01-10T00:20:10.408005Z",
     "shell.execute_reply": "2024-01-10T00:20:10.407286Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 61336 characters]",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plt.subplot(122)\n",
    "plt.plot(Xo2[:, 0][yo2==1], Xo2[:, 1][yo2==1], \"bs\")\n",
    "plt.plot(Xo2[:, 0][yo2==0], Xo2[:, 1][yo2==0], \"yo\")\n",
    "plot_svc_decision_boundary(svm_clf2, 0, 5.5)\n",
    "plt.xlabel(\"Petal length\", fontsize=14)\n",
    "plt.annotate(\"Outlier\",\n",
    "             xy=(X_outliers[1][0], X_outliers[1][1]),\n",
    "             xytext=(3.2, 0.08),\n",
    "             ha=\"center\",\n",
    "             arrowprops=dict(facecolor='black', shrink=0.1),\n",
    "             fontsize=16,\n",
    "            )\n",
    "plt.axis([0, 5.5, 0, 2]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Soft margin classification\n",
    "\n",
    "\n",
    "Allow some margin violations by varying hyperparameter $C$.\n",
    "\n",
    "\n",
    "Recall, large $C$ corresponds to small regularisation and thus few margin violations.  Small $C$ corresponds to greater regularisation and thus more margin violations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:10.411326Z",
     "iopub.status.busy": "2024-01-10T00:20:10.410852Z",
     "iopub.status.idle": "2024-01-10T00:20:10.418578Z",
     "shell.execute_reply": "2024-01-10T00:20:10.417946Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn import datasets\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.svm import LinearSVC\n",
    "\n",
    "iris = datasets.load_iris()\n",
    "X = iris[\"data\"][:, (2, 3)]  # petal length, petal width\n",
    "y = (iris[\"target\"] == 2).astype(np.float64)  # Iris-Virginica"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Train SVMs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:10.421567Z",
     "iopub.status.busy": "2024-01-10T00:20:10.421137Z",
     "iopub.status.idle": "2024-01-10T00:20:10.424523Z",
     "shell.execute_reply": "2024-01-10T00:20:10.423871Z"
    }
   },
   "outputs": [],
   "source": [
    "# disable convergence warning from early stopping\n",
    "from warnings import simplefilter\n",
    "from sklearn.exceptions import ConvergenceWarning\n",
    "simplefilter(\"ignore\", category=ConvergenceWarning)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:10.427696Z",
     "iopub.status.busy": "2024-01-10T00:20:10.427158Z",
     "iopub.status.idle": "2024-01-10T00:20:10.441192Z",
     "shell.execute_reply": "2024-01-10T00:20:10.440527Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/sklearn/svm/_classes.py:32: FutureWarning: The default value of `dual` will change from `True` to `'auto'` in 1.5. Set the value of `dual` explicitly to suppress the warning.\n",
      "  warnings.warn(\n",
      "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/sklearn/svm/_classes.py:32: FutureWarning: The default value of `dual` will change from `True` to `'auto'` in 1.5. Set the value of `dual` explicitly to suppress the warning.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "scaler = StandardScaler()\n",
    "svm_clf1 = LinearSVC(C=1, loss=\"hinge\", random_state=42)\n",
    "svm_clf2 = LinearSVC(C=100, loss=\"hinge\", random_state=42)\n",
    "\n",
    "scaled_svm_clf1 = Pipeline((\n",
    "        (\"scaler\", scaler),\n",
    "        (\"linear_svc\", svm_clf1),\n",
    "    ))\n",
    "scaled_svm_clf2 = Pipeline((\n",
    "        (\"scaler\", scaler),\n",
    "        (\"linear_svc\", svm_clf2),\n",
    "    ))\n",
    "\n",
    "scaled_svm_clf1.fit(X, y);\n",
    "scaled_svm_clf2.fit(X, y);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Compute support vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:10.444584Z",
     "iopub.status.busy": "2024-01-10T00:20:10.444121Z",
     "iopub.status.idle": "2024-01-10T00:20:10.451499Z",
     "shell.execute_reply": "2024-01-10T00:20:10.450894Z"
    }
   },
   "outputs": [],
   "source": [
    "# Convert to unscaled parameters\n",
    "b1 = svm_clf1.decision_function([-scaler.mean_ / scaler.scale_])\n",
    "b2 = svm_clf2.decision_function([-scaler.mean_ / scaler.scale_])\n",
    "w1 = svm_clf1.coef_[0] / scaler.scale_\n",
    "w2 = svm_clf2.coef_[0] / scaler.scale_\n",
    "svm_clf1.intercept_ = np.array([b1])\n",
    "svm_clf2.intercept_ = np.array([b2])\n",
    "svm_clf1.coef_ = np.array([w1])\n",
    "svm_clf2.coef_ = np.array([w2])\n",
    "\n",
    "# Find support vectors (LinearSVC does not do this automatically)\n",
    "t = y * 2 - 1 # t = +/-1\n",
    "support_vectors_idx1 = (t * (X.dot(w1) + b1) < 1).ravel()\n",
    "support_vectors_idx2 = (t * (X.dot(w2) + b2) < 1).ravel()\n",
    "svm_clf1.support_vectors_ = X[support_vectors_idx1]\n",
    "svm_clf2.support_vectors_ = X[support_vectors_idx2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:10.454748Z",
     "iopub.status.busy": "2024-01-10T00:20:10.454166Z",
     "iopub.status.idle": "2024-01-10T00:20:10.900018Z",
     "shell.execute_reply": "2024-01-10T00:20:10.899257Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 94620 characters]",
      "text/plain": [
       "<Figure size 1600x400 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(16,4))\n",
    "plt.subplot(121)\n",
    "plt.plot(X[:, 0][y==1], X[:, 1][y==1], \"g^\", label=\"Iris-Virginica\")\n",
    "plt.plot(X[:, 0][y==0], X[:, 1][y==0], \"bs\", label=\"Iris-Versicolor\")\n",
    "plot_svc_decision_boundary(svm_clf1, 4, 6)\n",
    "plt.xlabel(\"Petal length\", fontsize=14)\n",
    "plt.ylabel(\"Petal width\", fontsize=14)\n",
    "plt.legend(loc=\"upper left\", fontsize=14)\n",
    "plt.title(\"$C = {}$\".format(svm_clf1.C), fontsize=16)\n",
    "plt.axis([4, 6, 0.8, 2.8])\n",
    "\n",
    "plt.subplot(122)\n",
    "plt.plot(X[:, 0][y==1], X[:, 1][y==1], \"g^\")\n",
    "plt.plot(X[:, 0][y==0], X[:, 1][y==0], \"bs\")\n",
    "plot_svc_decision_boundary(svm_clf2, 4, 6)\n",
    "plt.xlabel(\"Petal length\", fontsize=14)\n",
    "plt.title(\"$C = {}$\".format(svm_clf2.C), fontsize=16)\n",
    "plt.axis([4, 6, 0.8, 2.8]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercises:** *You can now complete Exercise 1 in the exercises associated with this lecture.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Non-linear classification with polynomial features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "So far we have considered linear classification only.\n",
    "\n",
    "Most data-sets are not linearly separable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### 1D example \n",
    "\n",
    "Consider 1D feature space with $x_1$, which is clearly not linearly separable.\n",
    "\n",
    "However, if augment feature space with $x_2=(x_1)^2$ we see that the resulting 2D feature space is linearly separable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:10.903323Z",
     "iopub.status.busy": "2024-01-10T00:20:10.902893Z",
     "iopub.status.idle": "2024-01-10T00:20:10.908148Z",
     "shell.execute_reply": "2024-01-10T00:20:10.907548Z"
    }
   },
   "outputs": [],
   "source": [
    "X1D = np.linspace(-4, 4, 9).reshape(-1, 1)\n",
    "X2D = np.c_[X1D, X1D**2]\n",
    "y = np.array([0, 0, 1, 1, 1, 1, 1, 0, 0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:10.911084Z",
     "iopub.status.busy": "2024-01-10T00:20:10.910698Z",
     "iopub.status.idle": "2024-01-10T00:20:11.175999Z",
     "shell.execute_reply": "2024-01-10T00:20:11.175328Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 20732 characters]",
      "text/plain": [
       "<Figure size 1600x400 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(16, 4))\n",
    "plt.subplot(121)\n",
    "plt.grid(True, which='both')\n",
    "plt.axhline(y=0, color='k')\n",
    "plt.plot(X1D[:, 0][y==0], np.zeros(4), \"bs\", markersize=10)\n",
    "plt.plot(X1D[:, 0][y==1], np.zeros(5), \"g^\", markersize=10)\n",
    "plt.gca().get_yaxis().set_ticks([])\n",
    "plt.xlabel(r\"$x_1$\", fontsize=20)\n",
    "plt.axis([-4.5, 4.5, -0.2, 0.2])\n",
    "\n",
    "plt.subplot(122)\n",
    "plt.grid(True, which='both')\n",
    "plt.axhline(y=0, color='k')\n",
    "plt.axvline(x=0, color='k')\n",
    "plt.plot(X2D[:, 0][y==0], X2D[:, 1][y==0], \"bs\", markersize=10)\n",
    "plt.plot(X2D[:, 0][y==1], X2D[:, 1][y==1], \"g^\", markersize=10)\n",
    "plt.xlabel(r\"$x_1$\", fontsize=20)\n",
    "plt.ylabel(r\"$x_2$\", fontsize=20, rotation=0)\n",
    "plt.gca().get_yaxis().set_ticks([0, 4, 8, 12, 16])\n",
    "plt.plot([-4.5, 4.5], [6.5, 6.5], \"r--\", linewidth=3)\n",
    "plt.axis([-4.5, 4.5, -1, 17])\n",
    "\n",
    "plt.subplots_adjust(right=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### 2D example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:11.179245Z",
     "iopub.status.busy": "2024-01-10T00:20:11.178843Z",
     "iopub.status.idle": "2024-01-10T00:20:11.327960Z",
     "shell.execute_reply": "2024-01-10T00:20:11.327250Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 26164 characters]",
      "text/plain": [
       "<Figure size 1000x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.datasets import make_moons\n",
    "X, y = make_moons(n_samples=100, noise=0.15, random_state=42)\n",
    "\n",
    "def plot_dataset(X, y, axes):\n",
    "    plt.plot(X[:, 0][y==0], X[:, 1][y==0], \"bs\")\n",
    "    plt.plot(X[:, 0][y==1], X[:, 1][y==1], \"g^\")\n",
    "    plt.axis(axes)\n",
    "    plt.grid(True, which='both')\n",
    "    plt.xlabel(r\"$x_1$\", fontsize=20)\n",
    "    plt.ylabel(r\"$x_2$\", fontsize=20, rotation=0)\n",
    "\n",
    "plt.figure(figsize=(10, 5))\n",
    "plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:11.331500Z",
     "iopub.status.busy": "2024-01-10T00:20:11.330810Z",
     "iopub.status.idle": "2024-01-10T00:20:11.339010Z",
     "shell.execute_reply": "2024-01-10T00:20:11.338465Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/sklearn/svm/_classes.py:32: FutureWarning: The default value of `dual` will change from `True` to `'auto'` in 1.5. Set the value of `dual` explicitly to suppress the warning.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets import make_moons\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "\n",
    "polynomial_svm_clf = Pipeline((\n",
    "        (\"poly_features\", PolynomialFeatures(degree=3)),\n",
    "        (\"scaler\", StandardScaler()),\n",
    "        (\"svm_clf\", LinearSVC(C=10, loss=\"hinge\", random_state=42))\n",
    "    ))\n",
    "\n",
    "polynomial_svm_clf.fit(X, y);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:11.342040Z",
     "iopub.status.busy": "2024-01-10T00:20:11.341364Z",
     "iopub.status.idle": "2024-01-10T00:20:11.553315Z",
     "shell.execute_reply": "2024-01-10T00:20:11.552726Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 41332 characters]",
      "text/plain": [
       "<Figure size 1000x500 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_predictions(clf, axes):\n",
    "    x0s = np.linspace(axes[0], axes[1], 100)\n",
    "    x1s = np.linspace(axes[2], axes[3], 100)\n",
    "    x0, x1 = np.meshgrid(x0s, x1s)\n",
    "    X = np.c_[x0.ravel(), x1.ravel()]\n",
    "    y_pred = clf.predict(X).reshape(x0.shape)\n",
    "    y_decision = clf.decision_function(X).reshape(x0.shape)\n",
    "    plt.contourf(x0, x1, y_pred, cmap=plt.cm.brg, alpha=0.2)\n",
    "    plt.contourf(x0, x1, y_decision, cmap=plt.cm.brg, alpha=0.1)\n",
    "\n",
    "plt.figure(figsize=(10,5))\n",
    "plot_predictions(polynomial_svm_clf, [-1.5, 2.5, -1, 1.5])\n",
    "plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Adding polynomial features leads to a combinatorial increase in the dimensionality of the feature space and thus is computationally costly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "There are much better ways to perform non-linear classification with SVMs, where computational tricks can be exploited..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Non-linear classification with kernels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Similarity features and landmarks\n",
    "\n",
    "Compute new features based on proximity to landmarks.\n",
    "\n",
    "Consider landmarks $l^{(1)}$, $l^{(2)}$, $\\ldots$\n",
    "\n",
    "Then for each training instance $x$, compute features $f_j = \\text{sim}(x, l^{(j)})$, where $\\text{sim(.,.)}$ defines a *similarity* function.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Similarity functions\n",
    "\n",
    "The Gaussian radial basis function (RBF) is a common *similarity function*:\n",
    "\n",
    "$$\n",
    "\\phi_\\gamma(x,l) = exp( - \\gamma \\| x-l \\|^2),\n",
    "$$\n",
    "\n",
    "where $\\gamma$ controls the width of the kernel.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### 1D example (from above)\n",
    "\n",
    "Consider landmarks at $x_1=-2$ and $x_1=1$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:11.556833Z",
     "iopub.status.busy": "2024-01-10T00:20:11.556239Z",
     "iopub.status.idle": "2024-01-10T00:20:11.561660Z",
     "shell.execute_reply": "2024-01-10T00:20:11.561119Z"
    }
   },
   "outputs": [],
   "source": [
    "def gaussian_rbf(x, landmark, gamma):\n",
    "    return np.exp(-gamma * np.linalg.norm(x - landmark, axis=1)**2)\n",
    "\n",
    "gamma = 0.3\n",
    "\n",
    "x1s = np.linspace(-4.5, 4.5, 200).reshape(-1, 1)\n",
    "x2s = gaussian_rbf(x1s, -2, gamma)\n",
    "x3s = gaussian_rbf(x1s, 1, gamma)\n",
    "\n",
    "XK = np.c_[gaussian_rbf(X1D, -2, gamma), gaussian_rbf(X1D, 1, gamma)]\n",
    "yk = np.array([0, 0, 1, 1, 1, 1, 1, 0, 0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:11.564642Z",
     "iopub.status.busy": "2024-01-10T00:20:11.563993Z",
     "iopub.status.idle": "2024-01-10T00:20:11.979858Z",
     "shell.execute_reply": "2024-01-10T00:20:11.979177Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 76700 characters]",
      "text/plain": [
       "<Figure size 1600x400 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(16, 4))\n",
    "plt.subplot(121)\n",
    "plt.grid(True, which='both')\n",
    "plt.axhline(y=0, color='k')\n",
    "plt.scatter(x=[-2, 1], y=[0, 0], s=150, alpha=0.5, c=\"red\")\n",
    "plt.plot(X1D[:, 0][yk==0], np.zeros(4), \"bs\", markersize=10)\n",
    "plt.plot(X1D[:, 0][yk==1], np.zeros(5), \"g^\", markersize=10)\n",
    "plt.plot(x1s, x2s, \"g--\")\n",
    "plt.plot(x1s, x3s, \"b:\")\n",
    "plt.gca().get_yaxis().set_ticks([0, 0.25, 0.5, 0.75, 1])\n",
    "plt.xlabel(r\"$x_1$\", fontsize=20)\n",
    "plt.ylabel(r\"Similarity\", fontsize=14)\n",
    "plt.annotate(r'$\\mathbf{x}$',\n",
    "             xy=(X1D[3, 0], 0),\n",
    "             xytext=(-0.5, 0.20),\n",
    "             ha=\"center\",\n",
    "             arrowprops=dict(facecolor='black', shrink=0.1),\n",
    "             fontsize=18,\n",
    "            )\n",
    "plt.text(-2, 0.9, \"$x_2$\", ha=\"center\", fontsize=20)\n",
    "plt.text(1, 0.9, \"$x_3$\", ha=\"center\", fontsize=20)\n",
    "plt.axis([-4.5, 4.5, -0.1, 1.1])\n",
    "\n",
    "plt.subplot(122)\n",
    "plt.grid(True, which='both')\n",
    "plt.axhline(y=0, color='k')\n",
    "plt.axvline(x=0, color='k')\n",
    "plt.plot(XK[:, 0][yk==0], XK[:, 1][yk==0], \"bs\", markersize=10)\n",
    "plt.plot(XK[:, 0][yk==1], XK[:, 1][yk==1], \"g^\", markersize=10)\n",
    "plt.xlabel(r\"$x_2$\", fontsize=20)\n",
    "plt.ylabel(r\"$x_3$  \", fontsize=20, rotation=0)\n",
    "plt.annotate(r'$\\phi\\left(\\mathbf{x}\\right)$',\n",
    "             xy=(XK[3, 0], XK[3, 1]),\n",
    "             xytext=(0.65, 0.50),\n",
    "             ha=\"center\",\n",
    "             arrowprops=dict(facecolor='black', shrink=0.1),\n",
    "             fontsize=18,\n",
    "            )\n",
    "plt.plot([-0.1, 1.1], [0.57, -0.1], \"r--\", linewidth=3)\n",
    "plt.axis([-0.1, 1.1, -0.1, 1.1])\n",
    "    \n",
    "plt.subplots_adjust(right=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Training data *are* linearly separable in new feature space."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": [
     "inclass_exercise"
    ]
   },
   "source": [
    "### Exercise: What are new feature values corresponding to the instance at $x_1=-1$ ($\\gamma=0.3$)?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": [
     "solution",
     "inclass_exercise"
    ]
   },
   "source": [
    "$x_2 = \\exp(-0.3 \\times (1)^2) = 0.74$\n",
    "\n",
    "$x_3 = \\exp(-0.3 \\times (2)^2) = 0.30$\n",
    "\n",
    "Let's check:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:11.983440Z",
     "iopub.status.busy": "2024-01-10T00:20:11.982863Z",
     "iopub.status.idle": "2024-01-10T00:20:11.987586Z",
     "shell.execute_reply": "2024-01-10T00:20:11.987051Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Phi(-1.0, -2) = [0.74081822]\n",
      "Phi(-1.0, 1) = [0.30119421]\n"
     ]
    }
   ],
   "source": [
    "x1_example = X1D[3, 0]\n",
    "for landmark in (-2, 1):\n",
    "    k = gaussian_rbf(np.array([[x1_example]]), np.array([[landmark]]), gamma)\n",
    "    print(\"Phi({}, {}) = {}\".format(x1_example, landmark, k))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### How set landmarks?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Common approach is to set a landmark at the location of each instance in the training data-set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Kernel trick"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For SVMs it is not necessary to actually compute new features for each kernel.\n",
    "\n",
    "Instead, can be done implicitly, providing *considerable* computational saving."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "SVM are well suited for small to medium sized data-sets, with complex structure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The kernel trick is based on Mercer's theorem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Mercer's theorem\n",
    "\n",
    "For (feature) mapping function $\\phi(x)$, the inner product of two transformed vectors can be computed implicitly by the evaluation of the kernel function $K$ by\n",
    "$\n",
    "K(x,z) = \\langle \\phi(x), \\phi(z) \\rangle .\n",
    "$\n",
    "\n",
    "There is therefore no need to explicitly compute $\\phi(x)$.\n",
    "\n",
    "Moreoever, it is not necessary to even know the explicit form of the (feature) mapping function $\\phi(x)$.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "(We will not cover the kernel trick and Mercer's theorem in any further detail.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Common kernels\n",
    "\n",
    "Some common similarity function, or *kernel*, include the following.\n",
    "\n",
    "1. Gaussian radial basis function (RBF):\n",
    "$\n",
    "K_\\gamma(x,l) = exp( - \\gamma \\| x-l \\|^2),\n",
    "$\n",
    "where $\\gamma$ controls the width of the kernel.\n",
    "\n",
    "2. Polynomial kernel:\n",
    "$\n",
    "K_{c, d}(x,l) = (x^{\\rm T}l + c)^d,\n",
    "$\n",
    "for constant offset $c$ and degree $d$.\n",
    "\n",
    "3. Linear kernel, i.e. linear SVM:\n",
    "$\n",
    "K(x,l) = x^{\\rm T}l.\n",
    "$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### 2D example (from above)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:11.991176Z",
     "iopub.status.busy": "2024-01-10T00:20:11.990615Z",
     "iopub.status.idle": "2024-01-10T00:20:12.002315Z",
     "shell.execute_reply": "2024-01-10T00:20:12.001682Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC\n",
    "\n",
    "gamma1, gamma2 = 0.1, 5\n",
    "C1, C2 = 0.001, 1000\n",
    "hyperparams = (gamma1, C1), (gamma1, C2), (gamma2, C1), (gamma2, C2)\n",
    "\n",
    "svm_clfs = []\n",
    "for gamma, C in hyperparams:\n",
    "    rbf_kernel_svm_clf = Pipeline((\n",
    "            (\"scaler\", StandardScaler()),\n",
    "            (\"svm_clf\", SVC(kernel=\"rbf\", gamma=gamma, C=C))\n",
    "        ))\n",
    "    rbf_kernel_svm_clf.fit(X, y)\n",
    "    svm_clfs.append(rbf_kernel_svm_clf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:12.005352Z",
     "iopub.status.busy": "2024-01-10T00:20:12.004809Z",
     "iopub.status.idle": "2024-01-10T00:20:13.099222Z",
     "shell.execute_reply": "2024-01-10T00:20:13.098533Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 153324 characters]",
      "text/plain": [
       "<Figure size 1600x800 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(16, 8))\n",
    "\n",
    "for i, svm_clf in enumerate(svm_clfs):\n",
    "    plt.subplot(221 + i)\n",
    "    plot_predictions(svm_clf, [-1.5, 2.5, -1, 1.5])\n",
    "    plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])\n",
    "    gamma, C = hyperparams[i]\n",
    "    plt.title(r\"$\\gamma = {}, C = {}$\".format(gamma, C), fontsize=16)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercises:** *You can now complete Exercise 2 in the exercises associated with this lecture.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Can also use SVMs to perform regression, in addition to classification.\n",
    "\n",
    "General idea is to reverse the objective: rather than fitting the largest possible margin between two classes, SVM regression fits as many instances as possible within the margin. \n",
    "\n",
    "The width of the margin is controlled by the hyperparameter $\\epsilon$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Linear regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:13.103087Z",
     "iopub.status.busy": "2024-01-10T00:20:13.102462Z",
     "iopub.status.idle": "2024-01-10T00:20:13.107290Z",
     "shell.execute_reply": "2024-01-10T00:20:13.106674Z"
    }
   },
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "m = 50\n",
    "X = 2 * np.random.rand(m, 1)\n",
    "y = (4 + 3 * X + np.random.randn(m, 1)).ravel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:13.110249Z",
     "iopub.status.busy": "2024-01-10T00:20:13.109685Z",
     "iopub.status.idle": "2024-01-10T00:20:13.118040Z",
     "shell.execute_reply": "2024-01-10T00:20:13.117503Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/sklearn/svm/_classes.py:32: FutureWarning: The default value of `dual` will change from `True` to `'auto'` in 1.5. Set the value of `dual` explicitly to suppress the warning.\n",
      "  warnings.warn(\n",
      "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/sklearn/svm/_classes.py:32: FutureWarning: The default value of `dual` will change from `True` to `'auto'` in 1.5. Set the value of `dual` explicitly to suppress the warning.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "from sklearn.svm import LinearSVR\n",
    "\n",
    "svm_reg1 = LinearSVR(epsilon=1.5, random_state=42)\n",
    "svm_reg2 = LinearSVR(epsilon=0.5, random_state=42)\n",
    "svm_reg1.fit(X, y)\n",
    "svm_reg2.fit(X, y)\n",
    "\n",
    "def find_support_vectors(svm_reg, X, y):\n",
    "    y_pred = svm_reg.predict(X)\n",
    "    off_margin = (np.abs(y - y_pred) >= svm_reg.epsilon)\n",
    "    return np.argwhere(off_margin)\n",
    "\n",
    "svm_reg1.support_ = find_support_vectors(svm_reg1, X, y)\n",
    "svm_reg2.support_ = find_support_vectors(svm_reg2, X, y)\n",
    "\n",
    "eps_x1 = 1\n",
    "eps_y_pred = svm_reg1.predict([[eps_x1]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:13.120834Z",
     "iopub.status.busy": "2024-01-10T00:20:13.120604Z",
     "iopub.status.idle": "2024-01-10T00:20:13.125151Z",
     "shell.execute_reply": "2024-01-10T00:20:13.124515Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([6.52640746])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eps_y_pred"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:13.128852Z",
     "iopub.status.busy": "2024-01-10T00:20:13.128404Z",
     "iopub.status.idle": "2024-01-10T00:20:13.611003Z",
     "shell.execute_reply": "2024-01-10T00:20:13.610305Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 112472 characters]",
      "text/plain": [
       "<Figure size 1600x500 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_svm_regression(svm_reg, X, y, axes):\n",
    "    x1s = np.linspace(axes[0], axes[1], 100).reshape(100, 1)\n",
    "    y_pred = svm_reg.predict(x1s)\n",
    "    plt.plot(x1s, y_pred, \"k-\", linewidth=2, label=r\"$\\hat{y}$\")\n",
    "    plt.plot(x1s, y_pred + svm_reg.epsilon, \"k--\")\n",
    "    plt.plot(x1s, y_pred - svm_reg.epsilon, \"k--\")\n",
    "    plt.scatter(X[svm_reg.support_], y[svm_reg.support_], s=180, facecolors='#FFAAAA')\n",
    "    plt.plot(X, y, \"bo\")\n",
    "    plt.xlabel(r\"$x_1$\", fontsize=18)\n",
    "    plt.legend(loc=\"upper left\", fontsize=18)\n",
    "    plt.axis(axes)\n",
    "\n",
    "plt.figure(figsize=(16, 5))\n",
    "plt.subplot(121)\n",
    "plot_svm_regression(svm_reg1, X, y, [0, 2, 3, 11])\n",
    "plt.title(r\"$\\epsilon = {}$\".format(svm_reg1.epsilon), fontsize=18)\n",
    "plt.ylabel(r\"$y$\", fontsize=18, rotation=0)\n",
    "#plt.plot([eps_x1, eps_x1], [eps_y_pred, eps_y_pred - svm_reg1.epsilon], \"k-\", linewidth=2)\n",
    "plt.annotate(\n",
    "        '', xy=(eps_x1, eps_y_pred), xycoords='data',\n",
    "        xytext=(eps_x1, eps_y_pred - svm_reg1.epsilon),\n",
    "        textcoords='data', arrowprops={'arrowstyle': '<->', 'linewidth': 1.5}\n",
    "    )\n",
    "plt.text(0.91, 5.6, r\"$\\epsilon$\", fontsize=20)\n",
    "plt.subplot(122)\n",
    "plot_svm_regression(svm_reg2, X, y, [0, 2, 3, 11])\n",
    "plt.title(r\"$\\epsilon = {}$\".format(svm_reg2.epsilon), fontsize=18);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Non-linear regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:13.614588Z",
     "iopub.status.busy": "2024-01-10T00:20:13.614131Z",
     "iopub.status.idle": "2024-01-10T00:20:13.619620Z",
     "shell.execute_reply": "2024-01-10T00:20:13.618994Z"
    }
   },
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "m = 100\n",
    "X = 2 * np.random.rand(m, 1) - 1\n",
    "y = (0.2 + 0.1 * X + 0.5 * X**2 + np.random.randn(m, 1)/10).ravel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:13.622590Z",
     "iopub.status.busy": "2024-01-10T00:20:13.622133Z",
     "iopub.status.idle": "2024-01-10T00:20:13.634075Z",
     "shell.execute_reply": "2024-01-10T00:20:13.633427Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.svm import SVR\n",
    "\n",
    "svm_poly_reg1 = SVR(kernel=\"poly\", degree=2, C=100, epsilon=0.1)\n",
    "svm_poly_reg2 = SVR(kernel=\"poly\", degree=2, C=0.01, epsilon=0.1)\n",
    "svm_poly_reg1.fit(X, y);\n",
    "svm_poly_reg2.fit(X, y);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-01-10T00:20:13.637220Z",
     "iopub.status.busy": "2024-01-10T00:20:13.636764Z",
     "iopub.status.idle": "2024-01-10T00:20:14.087863Z",
     "shell.execute_reply": "2024-01-10T00:20:14.087257Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 132588 characters]",
      "text/plain": [
       "<Figure size 1600x500 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(16, 5))\n",
    "plt.subplot(121)\n",
    "plot_svm_regression(svm_poly_reg1, X, y, [-1, 1, 0, 1])\n",
    "plt.title(r\"$degree={}, C={}, \\epsilon = {}$\".format(svm_poly_reg1.degree, svm_poly_reg1.C, svm_poly_reg1.epsilon), fontsize=18)\n",
    "plt.ylabel(r\"$y$\", fontsize=18, rotation=0)\n",
    "plt.subplot(122)\n",
    "plot_svm_regression(svm_poly_reg2 , X, y, [-1, 1, 0, 1])\n",
    "plt.title(r\"$degree={}, C={}, \\epsilon = {}$\".format(svm_poly_reg2.degree, svm_poly_reg2.C, svm_poly_reg2.epsilon), fontsize=18);"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
