2288 lines
1.0 MiB
Plaintext
2288 lines
1.0 MiB
Plaintext
|
{
|
||
|
|
"cells": [
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "slide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"# Lecture 9: Support vector machines (SVMs)"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "skip"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"\n",
|
||
|
|
"[Run in colab](https://colab.research.google.com/drive/1t9EbYLj41BV5KnTU-zFIVM-K87yatl5e)"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 1,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:07.514586Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:07.514123Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:07.521951Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:07.521418Z"
|
||
|
|
},
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "skip"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"name": "stdout",
|
||
|
|
"output_type": "stream",
|
||
|
|
"text": [
|
||
|
|
"Last executed: 2024-01-10 00:20:07\n"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"import datetime\n",
|
||
|
|
"now = datetime.datetime.now()\n",
|
||
|
|
"print(\"Last executed: \" + now.strftime(\"%Y-%m-%d %H:%M:%S\"))"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "slide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"## Large margin classification"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"### Logistic regression with Hinge loss\n",
|
||
|
|
"\n",
|
||
|
|
"Recall cost function for logistic regression, with $\\ell_2$ regularisation, is given by\n",
|
||
|
|
"\n",
|
||
|
|
"\\begin{align*}\n",
|
||
|
|
"C(\\theta) = -\\frac{1}{m} \\sum_{i=1}^m \n",
|
||
|
|
"\\left [ \n",
|
||
|
|
"y^{(i)} \\log(\\hat{p}^{(i)})\n",
|
||
|
|
"+\n",
|
||
|
|
"(1 - y^{(i)}) \\log(1 - \\hat{p}^{(i)})\n",
|
||
|
|
"\\right]\n",
|
||
|
|
"+\n",
|
||
|
|
"\\frac{\\lambda}{2m}\n",
|
||
|
|
"\\sum_{j=1}^n \\theta_j^2,\n",
|
||
|
|
"\\end{align*}\n",
|
||
|
|
"\n",
|
||
|
|
"where \n",
|
||
|
|
"$\\hat{p} = h_\\theta(x) = \\sigma(\\theta^{\\rm T} x) \\quad \\text{and} \\quad \\sigma(t) = \\frac{1}{1+\\exp{(-t)}}.$"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "fragment"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"Recall, the bias $\\theta_0$ is not regularised, i.e. sum over $j$ starts at 1."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"#### Plot sigmoid"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 2,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:07.563118Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:07.562607Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:07.972568Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:07.971878Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"import numpy as np\n",
|
||
|
|
"%matplotlib inline\n",
|
||
|
|
"import matplotlib\n",
|
||
|
|
"import matplotlib.pyplot as plt"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 3,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:07.976305Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:07.975558Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:08.219979Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:08.219258Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAqQAAAFzCAYAAADolRujAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8WgzjOAAAACXBIWXMAAA9hAAAPYQGoP6dpAABOVElEQVR4nO3deVxU9f7H8ReggDiCO4jibotZWm5pi1qUbS63bLPUXLpaWpp1K8uyul1pz0rL1FIr/Wl2M7tltpBWlmbpuOaSC+IGigsIyOLM+f1xYoYJUIbtzMD7+Xicx8x8+c7Mh5lhfPs95/s9AYZhGIiIiIiIWCTQ6gJEREREpGpTIBURERERSymQioiIiIilFEhFRERExFIKpCIiIiJiKQVSEREREbGUAqmIiIiIWEqBVEREREQsVc3qAorD6XRy8OBBatWqRUBAgNXliIiIiMjfGIbByZMniY6OJjDQuzFPvwikBw8eJCYmxuoyREREROQs9u3bR5MmTby6j18E0lq1agHmLxgeHm5xNSIipZORkUF0dDRg/oe7Zs2aFlckIlJ6aWlpxMTEuHKbN/wikObtpg8PD1cgFRG/FxQU5LoeHh6uQCoilUpJDq/UpCYRERERsZQCqYiIiIhYSoFURERERCylQCoiIiIillIgFRERERFLKZCKiIiIiKUUSEVERETEUgqkIiIiImIpBVIRERERsZQCqYiIiIhYyi9OHVpWDMMgNzcXp9NpdSkiUgYCAwOpXr16iU5TJyIivqNKBFKHw0FKSgonT54kNzfX6nJEpAxVr16dWrVqUb9+fY9zxIuIiP+o9IHU4XCwb98+srOziYiIwGazERQUpBEVET9nGAYOh4P09HROnDjBqVOniImJUSgVEfFDlT6QpqSkkJ2dTdOmTalRo4bV5YhIGbPZbERERJCYmEhKSgqRkZFWlyQiIl6q1JOaDMPg5MmTREREKIyKVGI1atQgPDyckydPYhiG1eWIiIiXKnUgzc3NJTc3F5vNZnUpIlLOatWq5fqbFxER/1KpA2nebHodUyZS+eX9nWsVDRER/1OpA2keTWASqfz0dy4i4r+8DqQ//vgjffr0ITo6moCAAD777LOz3mfFihVccsklhISE0Lp1a+bMmVOCUkVERESkMvI6kGZkZNC+fXumTZtWrP579uzhxhtvpFevXqxfv55x48YxYsQIvv76a6+LFREREZHKx+tln66//nquv/76YvefPn06LVq04NVXXwXg/PPPZ+XKlbz++uv07t3bq+fOyMigVq1arl1zOTk55ObmUq1aNUJCQjz6gecuPKfTiWEYBAQEEBjozuEOhwMwz/iS19+bvoZh4HQ6S9U37/lK2xc8j5ct7Pfwtb5ne9296VsW72dFf07K+72vKp+TvNc972d5zvYdUaNGDddj5ObmkpOTQ1BQEKGhoSXqm5mZiWEYhIaGun6/06dPk52dTWBgYKGrfeR9Xs7U99SpUzidTkJCQqhWrZrrfllZWV71DQgIICwszNU3KysLh8NBcHAw1atX97qv0+nk1KlTANSsWdPVNzs7m9OnT1O9enWCg4O97msYBpmZmQCEhYUV+M73pm9x3vuy+JwU9t6Xxeck7/30pm9x3vvSfk6Kej9L+znJ/36W9nNS1PtZ0s9JRX9HlORzYvV3RN5rURLlfgzpqlWriI2N9Wjr3bs3q1atKvI+2dnZpKWleWwA0dHRpKSkuPq9/PLL2Gw2xowZ43H/hg0bYrPZOHTokKvtyJEj2O12EhISPPpu2rQJu91OVlaWq+3o0aPY7XZ2797t0XfLli3Y7XbXHwDAsWPHsNvt7Ny506Pv1q1bsdvtpKenu9pOnDiB3W5nx44dHn23bduG3W4nNTXV1Xby5Ensdjtbt2716Pvnn39it9s5fvy4qy0jIwO73c4ff/zh0XfXrl3Y7XaOHTvmajt16hR2u53Nmzd79N2zZw92u93j9c3OzsZut7Nx40aPvnv37sVut3P48GFXW25uLna7nfXr13v03bdvH3a7naSkJFebw+HAbrdjt9s9wsOBAwew2+0cOHDA1WYYhqtv/n+4k5KSsNvt7Nu3z+P51q9fj91u95hpffjwYex2O3v37vXou3HjRux2O9nZ2a62lJQU7HY7e/bs8ei7efNm7Ha760sV3O/9rl27PPr+8ccf2O12jz/M48ePY7fb+fPPPz365n1OTp486WpLTU3Fbrezbds2j747duzAbrdz4sQJV1t6enqhn5OdO3cWeO8zMzOx2+1s2bLFo+/u3bux2+0cPXrU1ZaVlYXdbmfTpk0efRMSErDb7Rw5csTVlpOTg91uZ8OGDR59ExMTsdvtJCcnu9pOnz7tej/z279/P3a73eNv1ul0uvrmn6h06NAh7HY7+/fv93iMrVu3sm/fPo+/jbN9RyQmJrrapk2bhs1mY/jw4R59mzdvjs1m83iN58yZg81m44477vDo27ZtW2w2G+vWrXO1LVy4EJvNRt++fSnMzz//7Lr+xRdfYLPZCnxnXnnlldhsNo89S99//z02m41u3bp59L3++uux2WwsXrzY1bZ69WpsNhvt27f36HvLLbdgs9mYN2+eq23Tpk3YbDbatGnj0XfQoEHYbDZmzJjhatu1axc2m43GjRt79B05ciQ2m4033njD1Xbo0CFsNhu1a9f26Dt+/HhsNhuTJ092taWmpmKz2bDZbJw+fdrV/uSTT2Kz2XjyySddbadPn3b1zf8dOnnyZGw2G+PHj/d4vtq1axf49+GNN97AZrMxcuRIj76NGzfGZrN5/I3PmDEDm83GoEGDPPq2adMGm83m8Tczb948bDYbt9xyi0ff9u3bY7PZWL16tatt8eLF2Gy2AgM+3bp1w2az8f3337vavv76a2w2G1deeaVH39jYWGw2G1988YWr7aeffsJms9G5c2ePvn379sVms7Fw4UJX27p167DZbLRt29aj7x133IHNZvM43G7r1q3YbDaaN2/u0Xf48OHYbDaPvaiJiYnYbDYaNmzo0XfMmDHYbDZefvllV1tKSorr/czvsccew2az8eyzz7raMjMzXX3z/7v87LPPYrPZeOyxxzweI6+vNznCyu+Izp07Y7PZ+Omnn1xtvv4dER0dTUmVeyBNSkoqsFB1ZGQkaWlpHv+45xcXF0dERIRri4mJKe8yRURERMQiAUYpVpEOCAhg8eLF9O/fv8g+55xzDkOHDmXChAmutqVLl3LjjTeSmZlZ6C6s7Oxsj1GrtLQ0YmJiOHjwIFFRUV7tst+7dy8tWrQgODhYu+x9sK922fvGLvuPPvqIlStXsm7dOjZt2kROTg7vvfceQ4YM8YvPCZh/9wkJCbRs2dL1veKru+MyMjJcI0CpqamEh4cX2Rd8Z3ecdtlrl7122WuX/Zn6Hj9+nOjoaI/vteIq91OHRkVFeeyyA0hOTiY8PLzIsyeFhIR4fDDy1KxZ0+O40ODgYNcH8+/9AI/d8Pn/4cqvsDVKvekbEBBQaLs3fQt7PvX1vi+U/v2s6M+Jr7yWTz/9NHv37qV+/fo0atSIvXv3EhgYWKC/r9RbmKCgII/gCmf/jsivevXqri/ckvbN/+Wcp1q1aq4v/aLqPlvfwr4rg4KCCq3Nm775/6EsSd/AwMBC+xb2He5N34CAgEL7FvZ+etMXCn8/y+JzUth7Xxafk8LeT1/4nBT1fpb2c1LU+1nazwmU33tfXt8Rpf2cWPEdUVh7cZX7Lvtu3boRHx/v0fbtt98WOKZBRKwza9YsEhISOHLkCKNGjbK6HBER8WGnT8OxY7B7N9jtsHw5fPYZ5DvU1Gtej5Cmp6d7TODZs2cP69evp27dujRt2pQJEyZw4MABPvjgAwBGjRrF1KlTefTRRxk2bBjff/89H3/8MV9++WXJqxaRMvX3A+RFRKRyMwzIzISjR83t2DHPLa/t+HFzO3HCfT3fPNwy43Ug/f333+nVq5frdt4MxiFDhjBnzhwOHTrkMSutRYsWfPnllzz00EO88cYbNGnShFmzZnm95JOIiIiIFM7pNMPi4cPmduRIwS0lxdyOHjUv8x3ZaDmvA2n
|
||
|
|
"text/plain": [
|
||
|
|
"<Figure size 800x400 with 1 Axes>"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"metadata": {},
|
||
|
|
"output_type": "display_data"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"t = np.linspace(-4, 4, 100)\n",
|
||
|
|
"sig = 1 / (1 + np.exp(-t))\n",
|
||
|
|
"plt.figure(figsize=(8, 4))\n",
|
||
|
|
"plt.plot([-4, 4], [0, 0], \"k-\")\n",
|
||
|
|
"plt.plot([-4, 4], [0.5, 0.5], \"k:\")\n",
|
||
|
|
"plt.plot([-4, 4], [1, 1], \"k:\")\n",
|
||
|
|
"plt.plot([0, 0], [-1.1, 1.1], \"k-\")\n",
|
||
|
|
"plt.plot(t, sig, \"b-\", linewidth=2, label=r\"$\\sigma(t) = \\frac{1}{1 + e^{-t}}$\")\n",
|
||
|
|
"plt.xlabel(\"t\")\n",
|
||
|
|
"plt.legend(loc=\"upper left\", fontsize=20)\n",
|
||
|
|
"plt.axis([-4, 4, -0.1, 1.1]);"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"### Hinge loss\n",
|
||
|
|
"\n",
|
||
|
|
"Logistic regression cost function for reference: \n",
|
||
|
|
"\n",
|
||
|
|
"\\begin{align*}\n",
|
||
|
|
"C(\\theta) = -\\frac{1}{m} \\sum_{i=1}^m \n",
|
||
|
|
"\\left [ \n",
|
||
|
|
"y^{(i)} \\log(\\hat{p}^{(i)})\n",
|
||
|
|
"+\n",
|
||
|
|
"(1 - y^{(i)}) \\log(1 - \\hat{p}^{(i)})\n",
|
||
|
|
"\\right]\n",
|
||
|
|
"+\n",
|
||
|
|
"\\frac{\\lambda}{2m}\n",
|
||
|
|
"\\sum_{j=1}^n \\theta_j^2.\n",
|
||
|
|
"\\end{align*}"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "fragment"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"#### Positive training instances\n",
|
||
|
|
"\n",
|
||
|
|
"For $y^{(i)}=1$, replace cost $-\\log(\\sigma(\\theta^{\\rm T} x))$ with *hinge loss* $\\max(1 - \\theta^{\\rm T} x, 0)$.\n",
|
||
|
|
"\n",
|
||
|
|
"For training, we want not just $\\theta^{\\rm T} x \\geq 0$ but $\\theta^{\\rm T} x \\geq 1$. "
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
},
|
||
|
|
"tags": [
|
||
|
|
"inclass_exercise"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"#### Exercise: \n",
|
||
|
|
"\n",
|
||
|
|
"Plot cost $-\\log(\\sigma(\\theta^{\\rm T} x))$ and hinge loss $\\max(1 - \\theta^{\\rm T} x, 0)$ ."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "-"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"Logistic regression cost function for reference: \n",
|
||
|
|
"\\begin{align*}\n",
|
||
|
|
"C(\\theta) = -\\frac{1}{m} \\sum_{i=1}^m \n",
|
||
|
|
"\\left [ \n",
|
||
|
|
"y^{(i)} \\log(\\hat{p}^{(i)})\n",
|
||
|
|
"+\n",
|
||
|
|
"(1 - y^{(i)}) \\log(1 - \\hat{p}^{(i)})\n",
|
||
|
|
"\\right]\n",
|
||
|
|
"+\n",
|
||
|
|
"\\frac{\\lambda}{2m}\n",
|
||
|
|
"\\sum_{j=1}^n \\theta_j^2.\n",
|
||
|
|
"\\end{align*}"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 4,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:08.223354Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:08.222895Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:08.226877Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:08.226303Z"
|
||
|
|
},
|
||
|
|
"tags": [
|
||
|
|
"inclass_exercise",
|
||
|
|
"solution"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"t = np.linspace(-4, 4, 100)\n",
|
||
|
|
"sig = 1 / (1 + np.exp(-t))\n",
|
||
|
|
"cost_one = -np.log(sig)\n",
|
||
|
|
"cost_one_hinge = np.maximum(1 - t, np.zeros(t.size))"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 5,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:08.229821Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:08.229175Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:08.462823Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:08.462159Z"
|
||
|
|
},
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "fragment"
|
||
|
|
},
|
||
|
|
"tags": [
|
||
|
|
"inclass_exercise",
|
||
|
|
"solution"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAApYAAAGjCAYAAACWrVWJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8WgzjOAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB88UlEQVR4nO3deZxN9R/H8ded3TZjJztl37KTIhUqJQopMZSi1I+ihcpWaCMtSCmEIqKihMpWspOtsmfLzowZY8xyfn983XtnMjNmuTPnzsz7+Xjcx9zzveee87mXxqfv8vk6LMuyEBERERHJIB+7AxARERGRnEGJpYiIiIh4hBJLEREREfEIJZYiIiIi4hFKLEVERETEI5RYioiIiIhHKLEUEREREY9QYikiIiIiHqHEUkREREQ8QomliIiIiHiEEksRERER8QglliKSqx06dIg+ffpw/fXXExQUhMPhwOFw8M0339gdmohItuNndwAikrUuX77M119/zeLFi1m/fj2nTp0iPDyckJAQypcvT+PGjXnggQe47bbb8PHJ2f/veejQIRo0aMDp06ftDiXXOnnyJOvXr2f9+vVs2LCBDRs2cObMGQBCQ0OZNm2avQGKSJoosRTJRebPn8/AgQM5ePDgVa+dOXOGM2fOsHnzZj766COqVKnCuHHjaNeuXdYHmoRp06bRq1cvAA4cOECFChUyfM3XX3+d06dP4+fnx6hRo2jRogX58+cHoHz58hm+vlxbiRIl7A5BRDxIiaVILvHaa68xdOhQ13Hr1q1p3749NWrUoGDBgpw9e5a///6bhQsXsmzZMnbv3s3LL7/sNYllZvjpp58A6NChAy+88ILN0Ui5cuWoVq0aS5cutTsUEUknJZYiucDUqVNdSWXx4sX56quvaNmy5VXn3XHHHfTr148dO3bw7LPPcurUqawONUsdPXoUgCpVqtgcSe41dOhQGjVqRKNGjShRogQHDx6kYsWKdoclIumkxFIkhzt69ChPP/00APny5WPlypVUq1YtxffUqlWLJUuW8MUXX2RFiLa5fPkyAP7+/jZHknuNGDHC7hBExINy9sx8EeHdd9/l4sWLAIwcOfKaSaWTj48PjzzySJKvXb58mYkTJ9KqVSuKFStGQEAAJUuW5O6772bmzJnEx8eneO1jx47x0ksvUb9+fUJCQvD396dEiRLUrl2bhx56iGnTphEeHg7AihUrcDgcrvmVABUrVnSt3nY+VqxYkarPNW3aNNd7nEaMGJHoWj179vToZx4+fHiie4aFhfHaa69Rr149ChYsiMPhSNMilTJlyuBwOLjtttuuee6OHTvw8/PD4XAwduzYVN9DRCQ91GMpkoNZlsX06dMB01v5+OOPZ/iaBw8e5K677uKvv/5K1H7ixAkWL17M4sWLmTx5Mt9++y2FCxe+6v2rV6/mnnvucSWOTidPnuTkyZPs2LGD2bNnU7RoUe65554Mx+sJGf3MCe3Zs4c2bdokuYAqtZo0acL8+fPZvHkzlmUlSpL/67nnniMuLo7rr7+eZ555Jt33FBFJDSWWIjnYzp07XaV0brnlFgoUKJCh60VERHD77bezf/9+wCx6efTRRylVqhQHDhzgww8/ZOXKlfz666/ce++9rFq1Cl9fX9f7o6Oj6dq1K+Hh4RQoUIAnn3ySVq1aUbx4cS5fvsyBAwdYs2YNCxYscL2nUaNGbN++nW+//ZZXXnkFgCVLllCqVKlEsaV2Xl6HDh1o2LAhALVr1wbgySef5KmnnnKdU6hQIY995v/q1KkTR48e5ZlnnqF9+/YUKlSIPXv2pGkVetOmTZk/fz5hYWHs3r2bqlWrJnmecyEWwFtvvUVAQECq7yEiki6WiORYM2fOtAALsF5++eUMX2/QoEGu673yyitXvR4fH29169bNdc7EiRMTvf7zzz+7Xlu4cGGy94mJibHCwsIStU2dOtX13gMHDmT4s1iW5bresGHDkj0no5/Zsixr2LBhrtd9fHysJUuWZCjuVatWua43c+bMJM+5fPmyVaVKFQuwWrRokeprO6+bkcfUqVPT/dkOHDjguk5oaGi6ryMi9tAcS5EczFloGsxq8IyIjo5mypQpANSsWZPhw4dfdY7D4WDixIkUKVIEgA8//DDR68ePH3c9b9GiRbL38vPzIzg4OEPxeoInPvN/9ezZkzZt2mQoroYNG+LnZwacNmzYkOQ5H374Ibt378bhcPDuu+9m6H4iIqmloXCRHOzChQuu5/ny5cvQtTZt2sT58+cBkxwlN9wbHBxMly5dmDRpErt27eLff//luuuuA3D9BFMCqX///hmKKbN54jP/V7du3TIcV548eahTpw6bN29OMrE8c+YMI0eOBKBHjx7Ur18/1dfevn17huMrU6ZMhq8hItmTEkuRHCzhnMrIyMgMXWvHjh2u502aNEnx3CZNmjBp0iTX+5xJ1s0330ylSpXYv38/AwYMYNasWXTs2JEWLVrQqFEjr5sD6InP/F916tTxSGxNmzZl8+bNbN26lbi4uERJ77Bhwzh//jz58uVj9OjRabpurVq1PBKfiOROGgoXycGcw7NgVjBnxNmzZ13PrzWsXrJkySTf5+/vz8KFC6levTpghnGHDBnCzTffTMGCBbnzzjv54osviIuLy1CsnuKJz/xfCRcGZUTTpk0BuHjxIjt37nS179q1i8mTJwPwwgsvXLXISUQkM6nHUiQHq1u3ruv55s2bPXbdlMrbXEuNGjXYvn07CxcuZOHChaxatYq9e/cSFRXFkiVLWLJkCePGjeOHH37I8LxQT8rIZ04opRXjaeFMLMEk6M6e0IEDBxIbG0uZMmUYNGhQmq+bsJc2vcqUKUPBggUzfB0RyX6UWIrkYDVr1qRo0aKcPn2a1atXEx4enu5FMQnrM544cSLFbRATLtJJqq6jr68vHTp0oEOHDgD8+++//Pjjj0yYMIFNmzaxadMm+vTpk6jskB08+Zk9rXLlyhQuXJizZ8+yceNGHnvsMX744Qd+/PFHAEaPHk3evHnTfF1nCaaMmDp1apJF5kUk59NQuEgO5nA4CA0NBcwcS+cK5/RIOPdu3bp1KZ67fv36JN+XnOuuu45evXrx+++/uxaaLFq0iKioKNc5nuoxTIvM/Mye4Jz3uWHDBmJjYxk4cCBgVo0nt2uSiEhmUmIpksM9++yzrp6roUOHXrV7THLi4+OZNWuW67hBgwau4c3p06cnu4XhhQsX+OqrrwAz7J3cIpak+Pv707JlSwBiY2NdK7IBgoKCXM+jo6NTfc2MyIrPnBHO4fDt27czfvx415/tu+++m+5E3LKsDD/UWymSeymxFMnhSpcu7aqtGBkZScuWLVm5cmWK79m1axd33nknb7/9tqstMDCQ3r17A2Ye3muvvXbV+yzL4umnn3bt9vP0008nen316tXs3bs32ftevnzZFVv+/PkpVqyY67WEydq+fftSjN9TPPGZM5Mzsbx8+TJDhgwBzM4+N998c5bFICKSkOZYiuQCvXr14siRIwwdOpSTJ09y66230qZNG+677z6qV69OwYIFOXv2LLt37+b777/nxx9/JC4uLtHiHzA9nvPnz2f//v0MHz6c7du306tXL6677jrX9oYrVqwAoFmzZjzxxBOJ3v/zzz/z2muvccstt9CuXTvq1KlDsWLFiIqKYvfu3Xz00UeuRUaPPfaYqwg4QL169QgKCuLSpUu8+uqr+Pv7U758eXx8zP8fly5dmjx58nj8u8voZ85MTZo0weFwYFkWMTExBAYG8tZbb2XZ/T3h119/TfQ/G84EHWDv3r1MmzYt0fnqDRXxclm70Y+I2Onrr7+2KlSokKpt+WrWrJnk1oMHDhywqlWrluJ7mzdvbp05c+aq9ybc2jClx3333WddvHjxqve/8MILyb5n+fLlaf4+nO9NaUvHjH7m/35uT6tevbrr2s8//7zHr5/ZQkND07RdpIh4N/VYiuQi999/P/fccw/z5s1j8eLFbNiwgZMnT3LhwgWCg4OpUKECTZs2pVOnTtx6661JztOrUKECf/zxB5988glz585lx44dhIeHU7hwYer
|
||
|
|
"text/plain": [
|
||
|
|
"<Figure size 800x400 with 1 Axes>"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"metadata": {},
|
||
|
|
"output_type": "display_data"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"plt.figure(figsize=(8, 4))\n",
|
||
|
|
"plt.plot([0, 0], [0, 5], \"k-\")\n",
|
||
|
|
"plt.plot(t, cost_one, \"b-\", linewidth=2, label=r\"$-\\log{(\\sigma(\\theta^{\\rm T} x))}$\")\n",
|
||
|
|
"plt.xlabel(r\"$\\theta^{\\rm T} x$\", fontsize=20)\n",
|
||
|
|
"plt.axis([-4, 4, 0, 5]);\n",
|
||
|
|
"plt.title('Cost for $y=1$', fontsize=20)\n",
|
||
|
|
"plt.plot(t, cost_one_hinge, \"r-\", linewidth=2, label=r\"$\\max(1-\\theta^{\\rm T} x, 0)$\")\n",
|
||
|
|
"plt.legend(loc=\"upper right\", fontsize=20);"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"#### Negative training instances\n",
|
||
|
|
"\n",
|
||
|
|
"For $y^{(i)}=0$, replace $-\\log(1-\\sigma(\\theta^{\\rm T} x))$ with *hinge loss* $\\max(1 + \\theta^{\\rm T} x, 0)$.\n",
|
||
|
|
"\n",
|
||
|
|
"For training, we want not just $\\theta^{\\rm T} x < 0$ but $\\theta^{\\rm T} x \\leq -1$. "
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
},
|
||
|
|
"tags": [
|
||
|
|
"inclass_exercise"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"#### Exercise: \n",
|
||
|
|
"\n",
|
||
|
|
"Plot cost $-\\log(1-\\sigma(\\theta^{\\rm T} x))$ and hinge loss $\\max(1 + \\theta^{\\rm T} x, 0)$."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 6,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:08.466146Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:08.465732Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:08.470442Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:08.469824Z"
|
||
|
|
},
|
||
|
|
"tags": [
|
||
|
|
"inclass_exercise",
|
||
|
|
"solution"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"cost_zero = -np.log(1-sig)\n",
|
||
|
|
"cost_zero_hinge = np.maximum(1 + t, np.zeros(t.size))"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 7,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:08.473267Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:08.472901Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:08.747961Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:08.747253Z"
|
||
|
|
},
|
||
|
|
"tags": [
|
||
|
|
"inclass_exercise",
|
||
|
|
"solution"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAzEAAAHwCAYAAACWveG4AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8WgzjOAAAACXBIWXMAAA9hAAAPYQGoP6dpAACLRElEQVR4nOzdd3hTZR/G8W9aaMssG0Q2MpQlskVliCDwqiBTkK2iooIMFUQKKOAWF4iggAKyBBUVEAdDRRmKskSRDbKhi+6e94/HJK20pSPtadL7c125mvPkJPkl0ZK7z3JYlmUhIiIiIiLiJfzsLkBERERERCQjFGJERERERMSrKMSIiIiIiIhXUYgRERERERGvohAjIiIiIiJeRSFGRERERES8ikKMiIiIiIh4FYUYERERERHxKgoxIiIiIiLiVRRiRERERETEqyjEiIiIiIiIV1GIERGRDDly5AhDhw6levXqBAUF4XA4cDgcfPLJJ3aXJiIieUQ+uwsQEcntYmNj+fjjj1m9ejVbtmzhzJkzhIWFERwcTOXKlWnatCndunWjbdu2+Pn59t+Gjhw5QqNGjTh79qzdpeR5hw8f5o033uCLL77g6NGjBAYGUr16dXr27MmwYcMoWLCg3SWKiGQbh2VZlt1FiIjkVitWrGDUqFEcOnToiufWrFmTV199lc6dO2d/Yekwb948Bg0aBMDBgwepUqVKlh/zgQceYPbs2eTLl48pU6Zwyy23ULhwYQAqV65MkSJFsvwccmWrVq3i3nvvJSwsLMXba9asyRdffME111yTw5WJiOQMhRgRkVQ8++yzTJgwwXV82223ceedd3LddddRrFgxzp8/z759+1i1ahXr1q0jMTGRBg0asGPHDvuKTiI7Qky1atU4ePAg3bt3Z9myZVl+PMm4X3/9lZYtWxIVFUXhwoUZO3Ysbdq0ISoqisWLFzN79mzABJlt27YpWIqIT9JwMhGRFMydO9cVYMqUKcPSpUtp1arVZee1a9eOYcOGsWvXLh5//HHOnDmT06XmqOPHjwPmC7LYY/jw4URFRZEvXz6++uorWrRo4bqtbdu21KhRgyeeeII///yTV155hYkTJ9pXrIhINvHtwdsiIplw/PhxHnnkEQAKFSrEhg0bUgwwSdWtW5e1a9cyevTonCjRNrGxsQDkz5/f5krypi1btrBp0yYAhgwZkizAOI0aNYprr70WgNdff524uLgcrVFEJCcoxIiI/Mdrr73GpUuXAJg8eTK1a9dO1/38/Py49957U7wtNjaWGTNm0KZNG0qXLk1AQADlypWjU6dOLFiwgMTExDQf+8SJEzz11FPccMMNBAcHkz9/fsqWLUu9evW45557mDdvnmt+xPr163E4HK6hZABVq1Z1rSLmvKxfvz5dr2vevHmu+zhNmjQp2WMNHDjQo6954sSJyZ4zNDSUZ599loYNG1KsWDEcDgfz5s1LV/0AFSpUwOFw0LZt2yueu2vXLvLly4fD4eCVV15J93PkhKQrwCX9fJPy8/Ojf//+AFy8eJHvvvsuJ0oTEclRGk4mIpKEZVnMnz8fML0w999/f5Yf89ChQ3Ts2JE//vgjWfupU6dYvXo1q1evZtasWXz66aeUKFHisvtv2rSJ//3vf5dN4j59+jSnT59m165dLF68mFKlSvG///0vy/V6QlZfc1J//fUX7du3T9fiCqlp1qwZK1as4JdffsGyrGSB7L9GjhxJQkIC1atX59FHH830c2aH77//HjD/bTZq1CjV85L2HP7www+0b98+22sTEclJCjEiIkns3r3btXzwzTffnOVJ0REREdx6660cOHAAgC5dujB48GDKly/PwYMHeeutt9iwYQPff/89d9xxBxs3bsTf3991/5iYGHr37k1YWBhFihThoYceok2bNpQpU4bY2FgOHjzIjz/+yMqVK133adKkCTt37uTTTz9l/PjxAKxdu5by5csnq61q1arpeg1dunShcePGANSrVw+Ahx56iIcffth1TvHixT32mv+re/fuHD9+nEcffZQ777yT4sWL89dff1G5cuV01Q/QvHlzVqxYQWhoKH/++Se1atVK8TznIg0AL774IgEBAel+jpywd+9eAK655hry5Uv9n/CkvYfO+4iI+BRLRERcFixYYAEWYD399NNZfrzRo0e7Hm/8+PGX3Z6YmGj17dvXdc6MGTOS3f7NN9+4blu1alWqzxMXF2eFhoYma5s7d67rvgcPHszya7Esy/V4ISEhqZ6T1ddsWZYVEhLiut3Pz89au3ZtlureuHGj6/EWLFiQ4jmxsbFWzZo1LcC65ZZb0v3YzsfNymXu3LlXfJ6oqCjX+Z07d77i+YUKFbIAq3nz5ul+LSIi3kJzYkREkjh37pzrepkyZbL0WDExMcyZMweAOnXqpLhKlMPhYMaMGZQsWRKAt956K9ntJ0+edF2/5ZZbUn2ufPnyUbRo0SzV6wmeeM3/NXDgwCwPh2rcuLGr52Lr1q0pnvPWW2/x559/4nA4eO2117L0fNkhPDzcdd25N09aChUqBJieMRERX6PhZCIiSST9ouj8EphZ27dv5+LFi4D5Ip7akKmiRYvSs2dPZs6cyZ49e/jnn3+46qqrAFw/wSz7PHz48CzVlN088Zr/q2/fvlmuq0CBAtSvX59ffvklxRBz7tw5Jk+eDED//v254YYb0v3YO3fuzHJ9FSpUuOI50dHRruvpGeYWGBgIQFRUVOYLExHJpRRiRESSSDoHJjIyMkuPtWvXLtf1Zs2apXlus2bNmDlzput+zi/0N910E9WqVePAgQOMGDGChQsX0rVrV2655RaaNGmS6+ZseOI1/1f9+vU9Ulvz5s355Zdf2LFjBwkJCckCVkhICBcvXqRQoUJMnTo1Q49bt25dj9R3JUFBQa7rzqWu0xITEwOYACci4ms0nExEJAnnECcwK2llxfnz513XrzQ0rVy5cineL3/+/Kxatcq178fWrVsZN24cN910E8WKFeP2229n0aJFJCQkZKlWT/HEa/6vpIsGZEXz5s0BuHTpErt373a179mzh1mzZgHwxBNPXLYAQm6RNGCnZ4iYM4SnZ+iZiIi3UU+MiEgSDRo0cF3/5ZdfPPa4aS3peyXXXXcdO3fuZNWqVaxatYqNGzeyf/9+oqKiWLt2LWvXruXVV1/lyy+/zPI8Hk/KymtOKq2VyzLCGWLAhEFnD8+oUaOIj4+nQoUKmdqsNGnvU2ZVqFCBYsWKpXlOUFAQJUuW5Ny5cxw7dizNcy9cuOAKMRUrVsxyfSIiuY1CjIhIEnXq1KFUqVKcPXuWTZs2ERYWlukJ80n3Pzl16hQ1a9ZM9dykE/hT2jfF39+fLl260KVLFwD++ecf1qxZw9tvv8327dvZvn07Q4cOTbbUsh08+Zo9rUaNGpQoUYLz58+zbds2hgwZwpdffsmaNWsAmDp1KgULFszw4zqXnc6KuXPnprhh6H9dd911bNq0if379xMfH5/qMstJ9+dx9uKJiPgSDScTEUnC4XAwYMAAwAzHca60lRlJ50r8/PPPaZ67ZcuWFO+XmquuuopBgwaxefNm1yT0zz//PNkkbk/1hGREdr5mT3DO09m6dSvx8fGMGjUKMKuX3XvvvTlSQ1bcdNNNgPlvc/v27amet2HDBtf1li1bZntdIiI5TSFGROQ/Hn/8cddf5CdMmHDZrvOpSUxMZOHCha7jRo0auYYIzZ8/n8TExBTvFx4eztKlSwHzl/bUJrinJH/+/K7d2ePj410rg0HyieDOSd7ZLSdec1Y4h5Tt3LmT6dOnuz7b1157LdOhz7KsLF/S0wsDuHriwPTepCQxMZEPPvgAgGLFitGmTZtMvS4RkdxMIUZE5D+uvvpq194lkZGRtGrVKtlftlOyZ88ebr/9dl566SVXW2BgIPfddx9g5k08++yzl93PsiweeeQRzp49C8AjjzyS7Hbn0KHUxMbGumorXLgwpUuXdt2WNBj8/fffadbvKZ54zdnJGWJiY2MZN24cAN27d3f1cOR2TZs25eabbwbgvffeY/PmzZed88orr7B3714Ahg8fTv78+XO0RhG
|
||
|
|
"text/plain": [
|
||
|
|
"<Figure size 1000x500 with 1 Axes>"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"metadata": {},
|
||
|
|
"output_type": "display_data"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"plt.figure(figsize=(10, 5))\n",
|
||
|
|
"plt.plot([0, 0], [0, 5], \"k-\")\n",
|
||
|
|
"plt.plot(t, cost_zero, \"b-\", linewidth=2, label=r\"$-\\log{(1-\\sigma(\\theta^{\\rm T} x))}$\")\n",
|
||
|
|
"plt.xlabel(r\"$\\theta^{\\rm T} x$\", fontsize=20)\n",
|
||
|
|
"plt.axis([-4, 4, 0, 5]);\n",
|
||
|
|
"plt.title('Cost for $y=0$', fontsize=20);\n",
|
||
|
|
"plt.plot(t, cost_zero_hinge, \"r-\", linewidth=2, label=r\"$max(1+\\theta^{\\rm T} x, 0)$\")\n",
|
||
|
|
"plt.legend(loc=\"upper left\", fontsize=20);"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"### Replace costs with hinge loss functions\n",
|
||
|
|
"\n",
|
||
|
|
"Recall logistic regression with $\\ell_2$ regularisation cost function is given by\n",
|
||
|
|
"\n",
|
||
|
|
"\\begin{align*}\n",
|
||
|
|
"C(\\theta) = -\\frac{1}{m} \\sum_{i=1}^m \n",
|
||
|
|
"\\left [ \n",
|
||
|
|
"y^{(i)} \\log(\\hat{p}^{(i)})\n",
|
||
|
|
"+\n",
|
||
|
|
"(1 - y^{(i)}) \\log(1 - \\hat{p}^{(i)})\n",
|
||
|
|
"\\right]\n",
|
||
|
|
"+\n",
|
||
|
|
"\\frac{\\lambda}{2m}\n",
|
||
|
|
"\\sum_{j=1}^n \\theta_j^2.\n",
|
||
|
|
"\\end{align*}\n",
|
||
|
|
"\n",
|
||
|
|
"Replace costs for $y^{(i)}=1$ and $y^{(i)}=0$ with hinge losses given above:\n",
|
||
|
|
"\\begin{align*}\n",
|
||
|
|
"\\min_\\theta\n",
|
||
|
|
"\\sum_{i=1}^m \n",
|
||
|
|
"\\left [ \n",
|
||
|
|
"y^{(i)} \n",
|
||
|
|
"\\max(1 - \\theta^{\\rm T} x^{(i)}, 0)\n",
|
||
|
|
"+\n",
|
||
|
|
"(1 - y^{(i)}) \n",
|
||
|
|
"\\max(1 + \\theta^{\\rm T} x^{(i)}, 0)\n",
|
||
|
|
"\\right]\n",
|
||
|
|
"+\n",
|
||
|
|
"\\frac{\\lambda}{2}\n",
|
||
|
|
"\\sum_{j=1}^n \\theta_j^2.\n",
|
||
|
|
"\\end{align*}"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "fragment"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"Introduce $k^{(i)}=1$ for positive instances ($y^{(i)}=1$) and $k^{(i)}=-1$ for negative instances ($y^{(i)}=0$):\n",
|
||
|
|
"\n",
|
||
|
|
"\\begin{align*}\n",
|
||
|
|
"\\Rightarrow\n",
|
||
|
|
"\\min_\\theta\n",
|
||
|
|
"C\n",
|
||
|
|
"\\sum_{i=1}^m \n",
|
||
|
|
"\\max(1 - k^{(i)} \\theta^{\\rm T} x^{(i)}, 0)\n",
|
||
|
|
"+\n",
|
||
|
|
"\\frac{1}{2}\n",
|
||
|
|
"\\sum_{j=1}^n \\theta_j^2\n",
|
||
|
|
"\\end{align*}"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "fragment"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"The convention is to weight the fidelity term by C rather than regularisation term by $\\lambda$ (thus $C$ plays the role of $1/\\lambda$).\n",
|
||
|
|
"- Large $C$ $\\rightarrow$ little regularisation \n",
|
||
|
|
"- Small $C$ $\\rightarrow$ greater regularisation"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"### Constrained objective problem\n",
|
||
|
|
"\n",
|
||
|
|
"So far we considered the *unconstrained* objective problem by adapting the logistic regression cost function:\n",
|
||
|
|
"\\begin{align*}\n",
|
||
|
|
"\\min_\\theta\n",
|
||
|
|
"C\n",
|
||
|
|
"\\sum_{i=1}^m \n",
|
||
|
|
"\\max(1 - k^{(i)} \\theta^{\\rm T} x^{(i)}, 0)\n",
|
||
|
|
"+\n",
|
||
|
|
"\\frac{1}{2}\n",
|
||
|
|
"\\sum_{j=1}^n \\theta_j^2\n",
|
||
|
|
"\\end{align*}"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "fragment"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"We can also consider the *constrained* objective problem:\n",
|
||
|
|
"\\begin{align*}\n",
|
||
|
|
"\\min_\\theta\n",
|
||
|
|
"\\frac{1}{2}\n",
|
||
|
|
"\\sum_{j=1}^n \\theta_j^2\n",
|
||
|
|
"\\quad\n",
|
||
|
|
"\\text{subject to}\n",
|
||
|
|
"\\quad\n",
|
||
|
|
"k^{(i)} \\theta^{\\rm T} x^{(i)} \\geq 1 \\ \\text{for}\\ i=1,2,\\ldots,m\n",
|
||
|
|
"\\end{align*}\n",
|
||
|
|
"\n",
|
||
|
|
"(Follows intuitively by considering large $C$.)"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"### Intuition for large margin classification\n",
|
||
|
|
"\n",
|
||
|
|
"Decision boundary defined by $\\theta^{\\rm T} x + b = 0$ (where here $\\theta$ does *not* include bias and $b=\\theta_0$; in notation above $\\theta$ did include bias, i.e. $\\theta_0$ term).\n",
|
||
|
|
"\n",
|
||
|
|
"Consequently, $\\theta$ is orthogonal to decision boundary."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "fragment"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"Recall constrained objective:\n",
|
||
|
|
"\n",
|
||
|
|
"\\begin{align*}\n",
|
||
|
|
"\\min_\\theta\n",
|
||
|
|
"\\frac{1}{2}\n",
|
||
|
|
"\\sum_{j=1}^n \\theta_j^2\n",
|
||
|
|
"\\quad\n",
|
||
|
|
"\\text{subject to}\n",
|
||
|
|
"\\quad\n",
|
||
|
|
"k^{(i)} \\theta^{\\rm T} x^{(i)} \\geq 1 \\ \\text{for}\\ i=1,2,\\ldots,m\n",
|
||
|
|
"\\end{align*}"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"#### Projection\n",
|
||
|
|
"\n",
|
||
|
|
"Note that the term $k^{(i)} \\theta^{\\rm T} x^{(i)}$ is related to the projection of $x^{(i)}$ onto $\\theta$:\n",
|
||
|
|
"\n",
|
||
|
|
"$$k^{(i)} \\theta^{\\rm T} x^{(i)} = p^{(i)} \\| \\theta \\|,$$\n",
|
||
|
|
"\n",
|
||
|
|
"where $ p^{(i)}$ is the projection of $x^{(i)}$ onto $\\theta$."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "fragment"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"Attempting to minimise $\\| \\theta \\|$, hence requires $p^{(i)}$ to be large $\\Rightarrow$ large margin classification."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"### Graphical illustration\n",
|
||
|
|
"\n",
|
||
|
|
"<img src=\"https://raw.githubusercontent.com/astro-informatics/course_mlbd_images/master/Lecture09_Images/SVM_optimize_cropped.png\" width=\"600px\" style=\"display:block; margin:auto\"/>\n",
|
||
|
|
"\n",
|
||
|
|
"(Note difference notation used: $w=\\theta$ without bias.)\n",
|
||
|
|
"\n",
|
||
|
|
"[Image source](http://www.saedsayad.com/support_vector_machine.htm)\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "slide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"## Training SVMs"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 8,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:08.751989Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:08.751391Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:08.757625Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:08.757013Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"# Common imports\n",
|
||
|
|
"import os\n",
|
||
|
|
"import numpy as np\n",
|
||
|
|
"np.random.seed(42) # To make this notebook's output stable across runs\n",
|
||
|
|
"\n",
|
||
|
|
"# To plot pretty figures\n",
|
||
|
|
"%matplotlib inline\n",
|
||
|
|
"import matplotlib\n",
|
||
|
|
"import matplotlib.pyplot as plt\n",
|
||
|
|
"plt.rcParams['axes.labelsize'] = 14\n",
|
||
|
|
"plt.rcParams['xtick.labelsize'] = 12\n",
|
||
|
|
"plt.rcParams['ytick.labelsize'] = 12"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"### Load data and train"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 9,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:08.760717Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:08.760092Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:09.136447Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:09.135746Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"from sklearn.svm import SVC\n",
|
||
|
|
"from sklearn import datasets\n",
|
||
|
|
"\n",
|
||
|
|
"iris = datasets.load_iris()\n",
|
||
|
|
"X = iris[\"data\"][:, (2, 3)] # petal length, petal width\n",
|
||
|
|
"y = iris[\"target\"]\n",
|
||
|
|
"\n",
|
||
|
|
"setosa_or_versicolor = (y == 0) | (y == 1)\n",
|
||
|
|
"X = X[setosa_or_versicolor]\n",
|
||
|
|
"y = y[setosa_or_versicolor]\n",
|
||
|
|
"\n",
|
||
|
|
"# SVM Classifier model\n",
|
||
|
|
"svm_clf = SVC(kernel=\"linear\", C=np.finfo(float).max)\n",
|
||
|
|
"svm_clf.fit(X, y);"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"### Plot decision boundaries"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 10,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:09.140117Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:09.139447Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:09.143979Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:09.143319Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"# Bad models\n",
|
||
|
|
"x0 = np.linspace(0, 5.5, 200)\n",
|
||
|
|
"#pred_1 = 5*x0 - 20\n",
|
||
|
|
"pred_2 = x0 - 1.8\n",
|
||
|
|
"pred_3 = 0.1 * x0 + 0.5"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 11,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:09.147122Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:09.146671Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:09.152212Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:09.151513Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"def plot_svc_decision_boundary(svm_clf, xmin, xmax):\n",
|
||
|
|
" w = svm_clf.coef_[0]\n",
|
||
|
|
" b = svm_clf.intercept_[0]\n",
|
||
|
|
"\n",
|
||
|
|
" # At the decision boundary, w0*x0 + w1*x1 + b = 0\n",
|
||
|
|
" # => x1 = -w0/w1 * x0 - b/w1\n",
|
||
|
|
" x0 = np.linspace(xmin, xmax, 200)\n",
|
||
|
|
" decision_boundary = -w[0]/w[1] * x0 - b/w[1]\n",
|
||
|
|
"\n",
|
||
|
|
" # On the margin, w0*x0 + w1*x1 + b = +/- 1\n",
|
||
|
|
" margin = 1/w[1]\n",
|
||
|
|
" gutter_up = decision_boundary + margin\n",
|
||
|
|
" gutter_down = decision_boundary - margin\n",
|
||
|
|
"\n",
|
||
|
|
" svs = svm_clf.support_vectors_\n",
|
||
|
|
" plt.scatter(svs[:, 0], svs[:, 1], s=200, facecolors='#FFAAAA')\n",
|
||
|
|
" plt.plot(x0, decision_boundary, \"k-\", linewidth=2)\n",
|
||
|
|
" plt.plot(x0, gutter_up, \"k--\", linewidth=2)\n",
|
||
|
|
" plt.plot(x0, gutter_down, \"k--\", linewidth=2)"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 12,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:09.155462Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:09.155013Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:09.547975Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:09.547289Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAABTMAAAGDCAYAAAD3S4SpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8WgzjOAAAACXBIWXMAAA9hAAAPYQGoP6dpAADW0ElEQVR4nOzdd1iTVxsG8DsgSy2ggLJFZLjr3ru1rlpHHXVUcaOETm1rW7W2zn4dWhDcotbRita96rbuXSeIgEwFRED2er8/IilIwASDb164f9fFJUlOTh4Sx+PznuccmSAIAoiIiIiIiIiIiIh0nJ7YARARERERERERERGpg8VMIiIiIiIiIiIikgQWM4mIiIiIiIiIiEgSWMwkIiIiIiIiIiIiSWAxk4iIiIiIiIiIiCSBxUwiIiIiIiIiIiKSBBYziYiIiIiIiIiISBJYzCQiIiIiIiIiIiJJYDGTiIiIiIiIiIiIJIHFTCIiIiIiIiIiIpIEnStmXrp0CXK5HI0aNUK1atXg6OiIYcOGITg4WK3nJyUlYfLkybCyskK1atXQvXt3XL16VeXY3bt3o0WLFjA2NoajoyPmzJmD3Nxcbf44RERERCQxzEeJiIiIdJdMEARB7CAKGzJkCM6cOYOhQ4eiadOmePToEXx9fZGamorz58+jcePGJT43Pz8fnTt3xo0bNzBjxgxYWlrCz88PkZGRuHLlClxdXZVjDxw4gH79+qFbt24YMWIEbt68iWXLlmHy5Mnw9/d/HT8qEREREekg5qNEREREukvniplnz55Fq1atYGhoqLzv/v37aNKkCYYMGYLff/+9xOf++eefGD58OLZt24YhQ4YAAOLj4+Hm5oY+ffpg8+bNyrGNGjWCgYEBLl++jCpVqgAAvv32WyxYsAB37txB/fr1y+knJCIiIiJdxnyUiIiISHfpXJt5hw4diiSOAODq6opGjRrh7t27pT43MDAQtWvXxuDBg5X3WVlZYdiwYdi1axeysrIAAHfu3MGdO3cwefJkZeIIANOmTYMgCAgMDNTiT0REREREUsJ8lIiIiEh36VwxUxVBEPD48WNYWlqWOu7atWto0aIF9PSK/lht2rRBenq6cp+ja9euAQBatWpVZJytrS3s7e2VjxMRERERAcxHiYiIiHRFlZcPEd+mTZsQHR2N77//vtRxsbGx6NKlS7H7bWxsAAAxMTFo0qQJYmNji9z/4tiYmJgSXyMrK0t5RR1Q7IuUmJgICwsLyGQytX4eIiIiIl0iCAKePXsGW1vbYkU4UtClfBRgTkpEREQViyb5qM4XM+/duwcvLy+0b98eY8eOLXVsRkYGjIyMit1vbGysfLzwryWNTUlJKfE1Fi5ciLlz56odPxEREZFUREZGwt7eXuwwdI6u5aMAc1IiIiKqmNTJR3W6mPno0SP069cPZmZmCAwMhL6+fqnjTUxMilyhLpCZmal8vPCvJY0teFyVmTNn4rPPPlPeTk5OhqOjIyIjI2FqavryH4qIiCqc1FupuN71OoRcAdAD3jz6Jkxb8N8Eko6UlBQ4ODjgjTfeEDsUnaOL+SjAnFQsffv2xZkzZwAAf/31F3r06CFyRNLx1Vdfwd/fHwDw/fff4+OPPxY5IunYvHkzpk6dCgAYOnQoVq9eLXJERETap0k+qrPFzOTkZPTp0wdJSUk4ffo0bG1tX/ocGxsbZctOYQX3FcxR0M4TGxsLBweHYmPbtGlT4msYGRmpvIJuamrKxJGIqBLKz8lHsDwYVXOrAgAcv3KEfTeubCNpYntyUbqajwLMScXyySefKIuZ69atw8CBA8UNSEI+/fRTZTFz7dq1+Prrr196cYAUPDw8MGvWLCQkJGDnzp347bffYG1tLXZYRETlQp18VCc3RcrMzET//v0RHByMvXv3omHDhmo9r1mzZrh69Sry8/OL3H/hwgVUrVoVbm5uynEAcPny5SLjYmJiEBUVpXyciIjoZSIWRyD1WioAoGqjqnCa7SRuQESkFcxHSZWBAwcqW9/27NmD8PBwcQOSEFdXV/Tp0wcAEB4ejn379okckXQYGxtj0qRJAICcnBysXLlS5IiIiMSlc8XMvLw8DB8+HOfOncO2bdvQvn17leNiY2Nx79495OTkKO8bMmQIHj9+jB07dijvS0hIwLZt29C/f3/l1etGjRqhfv36WLlyJfLy8pRj/f39IZPJMGTIkHL66YiIqCJJvZmKh98/VNzQB+oH1Ieekc7900pEGmI+SiWpUqUKPD09AQDW1tYICQkROSJpkcvlyu99fHxEjER6PD09oaenhypVqiAxMVHscIiIRCUTBEEQO4jCPvnkEyxduhT9+/fHsGHDij0+evRoAIql9uvXr0dYWBicnJwAKBLPTp064datW5gxYwYsLS3h5+eHiIgIXLp0Ce7u7sp59u7di/feew/du3fHBx98gFu3bsHX1xcTJkzQ6EpXSkoKzMzMkJyczJYeIqJKJD8nH1fbXUXqVcWqTMeZjnBe4CxyVERlw3ymKKnlowA/w9cpLi4Ox48fx+DBg2FgYCB2OJKSn58PNzc3PHjwAABw584dNGjQQOSopGPr1q3o2rWrcpsKIqKKRJNcRueKmd26dcPJkydLfLwgXFXJIwA8ffoUM2bMwM6dO5GRkYHWrVvjp59+QqtWrYrNtXPnTsydOxd3796FlZUVPDw8MHv2bI2SEiaORESV08P5DxH2bRgAoGrDqmh1tRVXZZJkMZ8pSmr5KMDPkKTj119/VR5eJZfLuUKTiIgASLyYKTVMHImIKp/UW6m40uIKhBzF6eUtzreAaWv+G0DSxXxG+vgZklQkJSXBzs4O6enpqF69OqKjo/l7loiINMpldPY084ouJyenyP5IRJWVvr4+W7RIUvJz8nHP456ikAnA8QtHFjKJiCqxhw8fIicnBy4uLmKHIgnm5ub48MMPsWLFCqSmpmLDhg1F9tIk9WRmZuLKlSvo2LGj2KEQEb12LGa+ZikpKUhISEBWVpbYoRDpDCMjI1haWvKqPElC5P8ikXrl+enlDavC6TsncQMiIiJRxMbGYtq0adi9ezeGDh2KrVu3ih2SZHh5eWHFihUAAF9fX0ybNg16etyqRV2LFi3Czz//jNTUVERFRcHCwkLskIiIXisWM1+jlJQUREdHo3r16rC0tISBgQFkMpnYYRGJRhAE5OTkIDk5GdHR0QDAgibptNRbqQj/LlxxQw+ov46nlxMRVVY1a9bEmTNnkJ+fj+3btyMmJga2trZihyUJTZo0Qbdu3XDixAkEBQXh6NGj6Nmzp9hhScbjx4+RkJAAAFizZg2++OILkSMiInq9uGfmK9Kkpz80NBQGBgawt7dnEZOoEEEQEBUVhZycHDg78zRo0k35ufm41v4anl1+BgBw+NIB9RbVEzkqIu3gfovSx89QHN9++y3mz58PAJg9ezbmzp0rckTSsX37dgwZMgQA8N5772HXrl0iRyQd9+/fh5ubGwCgTp06ePDgAfT19UWOiojo1WiSy3A5yWuSk5ODrKwsmJmZsZBJ9AKZTAYzMzNkZWUhJydH7HCIVIr8X6SykFm1AdvLiYgI8PT0VBaRVqxYgezsbJEjko4BAwbA3t4eALBnzx6EhYWJHJF0uLq6ok+fPgAUe7bu27dP5IiIiF4vFjNfk4LDfnjQCZFqBX82eDAW6aK022nF2sv1jbkCgoiosrO3t8egQYMAKFp/AwMDRY5IOqpUqYKpU6cCUHTp+Pv7ixyRtBQ+NMnHx0fESIiIXj8WM18zrsokUo1/NkhX5ec+P708W7Eri8N0B5i2ZQsnEREpFC4q+fr6ihiJ9EycOBGGhoYAgNWrVyM9PV3kiKSjd+/eqFdPsd3NkSNHcPfuXZEjIiJ6fVjMJCIiKkXkT4Xay+tXhdNcJ3EDIiIindKlSxc0adIEAHDu3DlcuXJF5Iiko1atWvjggw8AAE+fPsWWLVtEjkg69PT04OXlpbzt5+cnYjRERK8Xi5lEREQlSLudhvA54YobekD9ALaXExFRUTKZjKszX8GL7x3Pp1XfuHHjULVqVQBAQEAAUlJSRI6IiOj
|
||
|
|
"text/plain": [
|
||
|
|
"<Figure size 1600x400 with 2 Axes>"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"metadata": {},
|
||
|
|
"output_type": "display_data"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"plt.figure(figsize=(16,4))\n",
|
||
|
|
"\n",
|
||
|
|
"plt.subplot(121)\n",
|
||
|
|
"#plt.plot(x0, pred_1, \"g--\", linewidth=2)\n",
|
||
|
|
"plt.plot(x0, pred_2, \"m-\", linewidth=2)\n",
|
||
|
|
"plt.plot(x0, pred_3, \"r-\", linewidth=2)\n",
|
||
|
|
"plt.plot(X[:, 0][y==1], X[:, 1][y==1], \"bs\", label=\"Iris-Versicolor\")\n",
|
||
|
|
"plt.plot(X[:, 0][y==0], X[:, 1][y==0], \"yo\", label=\"Iris-Setosa\")\n",
|
||
|
|
"plt.xlabel(\"Petal length\", fontsize=14)\n",
|
||
|
|
"plt.ylabel(\"Petal width\", fontsize=14)\n",
|
||
|
|
"plt.legend(loc=\"upper left\", fontsize=14)\n",
|
||
|
|
"plt.axis([0, 5.5, 0, 2])\n",
|
||
|
|
"\n",
|
||
|
|
"plt.subplot(122)\n",
|
||
|
|
"plot_svc_decision_boundary(svm_clf, 0, 5.5)\n",
|
||
|
|
"plt.plot(X[:, 0][y==1], X[:, 1][y==1], \"bs\")\n",
|
||
|
|
"plt.plot(X[:, 0][y==0], X[:, 1][y==0], \"yo\")\n",
|
||
|
|
"plt.xlabel(\"Petal length\", fontsize=14)\n",
|
||
|
|
"plt.axis([0, 5.5, 0, 2]);"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "fragment"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"Adding training instances outside the margin will not alter the decision boundary. \n",
|
||
|
|
"\n",
|
||
|
|
"Boundary is defined by *support vectors* that are located on the edge of the margin."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
},
|
||
|
|
"tags": [
|
||
|
|
"inclass_exercise"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"#### Exercise: what value would you guess for the weights $\\theta$ (excluding the bias)?"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "fragment"
|
||
|
|
},
|
||
|
|
"tags": [
|
||
|
|
"solution",
|
||
|
|
"inclass_exercise"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"From plot appears $w \\sim [1, 1]^{\\rm T}$ (or slightly more accurately $w \\sim [1.1, 0.9]^{\\rm T}$).\n",
|
||
|
|
"\n",
|
||
|
|
"Let's check:"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 13,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:09.551525Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:09.551157Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:09.555816Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:09.555293Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"text/plain": [
|
||
|
|
"array([1.29411744, 0.82352928])"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"execution_count": 13,
|
||
|
|
"metadata": {},
|
||
|
|
"output_type": "execute_result"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"svm_clf.coef_[0]"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"### Feature scaling\n",
|
||
|
|
"\n",
|
||
|
|
"SVMs are sensitive to feature scales, hence feature scaling is important if features not already of similar scale."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 14,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:09.558797Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:09.558365Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:09.771970Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:09.771227Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAkIAAAHbCAYAAAAnJvUWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8WgzjOAAAACXBIWXMAAA9hAAAPYQGoP6dpAABTi0lEQVR4nO3de1xUZf4H8M8wAwMITNxUUBA172jgdSVFTSTTNFNBMzPLZG1TUrRc3UyzVSrFcmsr/WlaXjLFzLK8oMkmiSkprXkJUlFAUvHCAMIMM5zfHywThxmYAYSZ8Xzev9e8fs05z3POl1l1PjzPc86RCYIggIiIiEiCHKxdABEREZG1MAgRERGRZDEIERERkWQxCBEREZFkMQgRERGRZDEIERERkWQxCBEREZFkMQgRERGRZDEIERERkWQxCBGRWVlZWZDJZJDJZMjKyqq1bVBQEGQyGTZu3NgktVlD5WfRFCo/+6CgoCY5H5HUMAgRERGRZDEIERERkWQxCBEREZFkMQgRUaMbPHgwZDIZkpOTkZ6ejrFjx8LHxwdKpRJdu3ZFQkICBEEw6qfRaLBixQr06tUL7u7ucHJyQsuWLdGnTx+8+uqruHXrllGfu3fv4r333sOAAQPg6ekJpVKJNm3aYNSoUdi6dauo7eXLl/H222/jkUceQWBgIJRKJR544AEMGDAAa9asQXl5eZ1/Vp1Oh3Xr1mHw4MHw8vKCUqlE27Zt8eKLLyI7O7vGfnv27MGgQYPg7u4OlUqFgQMHYvfu3XU+PxHVjcLaBRCRdOzfvx+rVq1C+/btMWzYMOTl5SElJQXz5s1DdnY23nvvPUPb8vJyjBw5EocOHYKHhwcGDhyIBx54ADdu3EBmZiZWrFiBSZMmwcvLy9AnOzsbw4cPx9mzZ+Hq6oqHH34Y3t7eyM3NxZEjR3D69GlMmjTJ0H7Tpk1YtGgR2rZti44dO+Lhhx9GXl4eUlNT8eOPP+LAgQNITEy0eGF0YWEhRo8ejeTkZLi5uaFXr17w9fXF6dOn8fHHH2PHjh1ISkpCaGioqN+7776LuLg4AEDfvn3Rvn17ZGZmYsyYMYbtRNRIBCIiMy5duiQAEAAIly5dqrVtmzZtBADChg0bDNsGDRpk6P/xxx+L2h86dEiQyWSCXC4XsrOzDdv/85//CACE0NBQQa1WG53nxIkTQn5+vuG9Xq8XevfuLQAQIiMjhevXr4val5SUCN9++61o2/Hjx4XTp08bHTs3N1d46KGHBADC9u3bjfZX/izVTZo0SQAgPP7448K1a9dE+959910BgNChQwdBp9MZtv/yyy+CXC4XHBwchB07doj6bN68WZDJZAIAoU2bNkbnI6KG49QYETWZsWPH4q9//ato2yOPPIJHH30Uer0ehw8fNmy/du0aAGDgwIFwd3c3Olbv3r3h7e1teP/NN98gLS0Nfn5+2LlzJ3x9fUXtnZ2dMWLECNG2Pn36IDg42OjY/v7+eOeddwAAO3bssOhnO3fuHD7//HP4+/tj69ataN68uWj/7NmzMWLECGRmZmLv3r2G7e+//z70ej2ioqIwfvx4UZ+nn34ao0ePtuj8RFQ/nBojoiYzatQok9u7dOmCffv2ITc317CtZ8+ekMvl+OSTT9CxY0eMHTsWfn5+NR573759AIBJkybBzc3N4po0Gg0OHDiAEydO4Pr169BoNBAEAYWFhQCA3377zaLjfPfddxAEAY899pjJ4AZUrJX67rvvcPToUTz++OMAgOTkZADA5MmTTfZ59tlnuVaIqBExCBGRWVXXyAgmFjVXVbnf1LqawMBAk308PDwAAKWlpYZt7du3x7vvvotXXnkFM2fOxMyZM9GmTRv0798fjz/+OKKiouDk5GRof/nyZQBA586dLfypgGPHjmHChAm4cuVKjW3UarVFx7p48SIAYP369Vi/fn2tbW/cuGH475ycHABA27ZtTbataTsR3RsMQkRkVrNmzQz/XVxcXGvboqIiADA5KuPgULfZ+FmzZiE6Ohpff/01UlJSkJKSgm3btmHbtm1YvHgxjhw5UusoUW3u3r2LMWPG4Nq1a3juuefw4osv4sEHH4SHhwfkcjkyMjLQqVMns8GvUuUVZiEhIXjooYdqbduvX7961UxE9x6DEBGZ5eXlBTc3NxQVFeH33383ua4GAG7dumW4pL2m0Z+6atGiBaZPn47p06cDAM6fP4/nn38eqamp+Pvf/45PP/1UdL7z589bdNwffvgB165dQ8+ePfHJJ58Y7c/MzKxTnQEBAQCAhx9+GB988IHF/Vq1aoULFy4gKysL3bp1M9pv7pEmRNQwXCxNRGY5ODhg0KBBAICdO3fW2C4xMREA4OnpiZCQkEappXPnzpg/fz4AID093bB9+PDhAIDPP//c7KgVALOBbfPmzXWq67HHHgMAfP3116IpPnMqP9ctW7aY3P/ZZ5/VqQ4iqhsGISKyyKuvvgqZTIYtW7aYXAOTmpqKhQsXAgDmzp0LR0fHBp3v+++/x3fffYeysjLRdkEQsGfPHgBAmzZtDNtHjx6N0NBQXL16FVFRUbh586aoX2lpqehqrS5dugAADh06hLNnz4rarl27Fl988UWd6g0NDcW4ceOQnZ2NsWPHmhzJKS4uxpYtWwxXxAEV039yuRzbt2/Hrl27RO23bduGr776qk51EFEdWfPafSKyL6tXrxbkcrkAQGjXrp0wfvx4YeLEiULv3r0N97uZOHGi6D45gvDnfYQOHz5s8riLFy8WAAiLFy82bKu8746Hh4cwePBgYdKkScKTTz5puE+RSqUSTp06JTpOVlaW0KlTJwGA4OrqKkRGRgpPPfWUEB4eLqhUKqN78TzxxBMCAMHJyUmIjIwUJk6cKHTu3FmQyWTCP/7xjxrv34Ma7iOkVquFoUOHGo7Zp08fITo6WoiKihL69OkjODk5CQCEc+fOifq98847hmP269dPmDRpktCnTx8BgDBnzhzeR4ioETEIEVGdnDp1Spg2bZrQoUMHwdXVVXBychJatWoljBkzRti9e7fJPvUJQr///ruwZMkSYejQoUJgYKDg7OwseHp6Cj169BD+/ve/i26+WFVhYaHw9ttvC3369BHc3d0FpVIptGnTRhg9erSwbds2UVutViusWLFC6N69u+Dq6ip4eXkJkZGRwoEDBww3kaxLEBKEihs7bt26VRgxYoTQokULwdHRUfD29haCg4OF5557Tti1a5eg1WqN+u3evVsYMGCA0KxZM8HNzU0ICwsTEhMTa62DiBpOJggWXhJBREREdJ/hGiEiIiKSLJsNQj///DOGDx8ODw8PuLu7IzIyUnSFSFVHjx7FgAED4OrqipYtWyI2NtZwLxMiIiKimtjk1NjJkyfx8MMPIyAgAH/9619RXl6ODz/8ELdu3cLx48fRqVMnQ9v09HT0798fXbp0QUxMDHJycrBy5UoMGTJEdIUIERERUXU2GYRGjhyJ1NRUZGZmGh6qmJeXh44dOyIyMlJ0H5MRI0YgPT0d58+fN9ymf926dZg+fTr279+PyMhIq/wMREREZPtscmrsyJEjiIiIED1Z2s/PD4MGDcKePXsM015qtRpJSUmYPHmyIQQBwJQpU+Dm5obt27c3ee1ERERkP2wyCGk0Gri4uBhtd3V1hVarxa+//goAOH36NHQ6HXr37i1q5+TkhJCQEJw6dapJ6iUiIiL7ZJPPGuvUqROOHTsGvV4PuVwOANBqtfjpp58AALm5uQAqpssAmHzoop+fH44cOVLjOTQaDTQajeF9eXk5bt26BW9vb5NPzSYiIiLbIwgCCgsL4e/vX+cHOwM2GoT+9re/4cUXX8S0adPw6quvory8HP/85z8NwaekpET0/5VKpdExnJ2dDftNiY+PxxtvvNEI1RMREVFTy87ORuvWrevczyaD0IwZM5CdnY0VK1YYnizdu3dvvPrqq1i2bBnc3NwAwDB9VnVkp1JpaanJ6bVKCxYsQFxcnOF9QUEBAgMDkZ2dLVpvRERERLZLrVYjICAA7u7u9epvk0EIAJYtW4Z58+bhzJkzUKlU6N69u+GBjh07dgTw55R
|
||
|
|
"text/plain": [
|
||
|
|
"<Figure size 640x480 with 1 Axes>"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"metadata": {},
|
||
|
|
"output_type": "display_data"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"Xs = np.array([[1, 50], [5, 20], [3, 80], [5, 60]]).astype(np.float64)\n",
|
||
|
|
"ys = np.array([0, 0, 1, 1])\n",
|
||
|
|
"svm_clf = SVC(kernel=\"linear\", C=100)\n",
|
||
|
|
"svm_clf.fit(Xs, ys)\n",
|
||
|
|
"\n",
|
||
|
|
"#plt.figure(figsize=(16,4))\n",
|
||
|
|
"#plt.subplot(121)\n",
|
||
|
|
"plt.plot(Xs[:, 0][ys==1], Xs[:, 1][ys==1], \"bo\")\n",
|
||
|
|
"plt.plot(Xs[:, 0][ys==0], Xs[:, 1][ys==0], \"ms\")\n",
|
||
|
|
"plot_svc_decision_boundary(svm_clf, 0, 6)\n",
|
||
|
|
"plt.xlabel(\"$x_0$\", fontsize=20)\n",
|
||
|
|
"plt.ylabel(\"$x_1$ \", fontsize=20, rotation=0)\n",
|
||
|
|
"plt.title(\"Unscaled\", fontsize=16)\n",
|
||
|
|
"plt.axis([0, 6, 0, 90]);"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 15,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:09.775249Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:09.774779Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:09.988009Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:09.987234Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAkMAAAHbCAYAAADI5J4oAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8WgzjOAAAACXBIWXMAAA9hAAAPYQGoP6dpAACEKklEQVR4nO3dd1hU19YG8Hco0kQsEEWqBexRVLAXorHGjmCvsTOTGPUa04zRxJsviWVAxMSa2EGNXWPvIhbsFUSKqCjSFFCG8/3B5cRxUEGGOQzz/p5nnntn73Nm1nEysNjr7L1lgiAIICIiIjJQRlIHQERERCQlJkNERERk0JgMERERkUFjMkREREQGjckQERERGTQmQ0RERGTQmAwRERGRQWMyRERERAaNyRAREREZNCZDRFRkt2/fhr+/P+rWrQsrKyuYm5vD0dERnp6e8Pf3x6ZNmySJy9XVFTKZDNHR0Tp5P5lMBplMppP3IiLtkXE7DiIqis2bN2PQoEHIyspCpUqV0LhxY9jZ2eHp06eIiIhAQkICKlWqhMePH+s8NldXV9y7dw93796Fq6trsb9fXiLEH6tE+sVE6gCISH89fPgQw4cPR1ZWFqZMmYI5c+bA3Nxc7Zhz584hNDRUogiJiN6NyRARvbcdO3YgPT0dVatWxa+//prvMU2aNEGTJk10HBkRUcHxniEiem8PHz4EANjZ2RX63OzsbCxfvhwdO3aEra0tzMzM4OjoiI4dOyIgIEDt2MTERCiVSnTr1g3VqlWDhYUFypUrh6ZNm+Lnn39GZmbme8UfGhqKLl26wM7ODmXKlIGDgwOGDBmCa9euvfGcU6dOoWvXrihfvjzKli2Lpk2bYvny5e/1/kRUMvCeISJ6b6tXr8bQoUNhbGyMvXv3okOHDgU6LyUlBZ988gmOHz8OU1NTtGzZElWrVsWDBw9w5coVJCYmqt13k/c+Dg4OqFmzJuzt7ZGYmIiwsDCkp6ejRYsWOHToEMzMzNTe5033DGVnZ2Pw4MHYuHEjzMzM0KRJEzg4OODWrVu4ePEiLCwssHnzZnTp0kXt9UJCQjBw4ECoVCrUr18fDRo0QGxsLE6cOIHJkydj3rx5AHjPEJHeEYiI3lNaWprg4OAgABBkMpnQvn17Yfbs2cLOnTuFR48evfG8vn37CgAEDw8P4e7du2p9L1++FP7++2+1tmvXrgmnTp3SeJ2kpCShU6dOAgDh//7v/zT6XVxcBAAa7/HVV18JAIRmzZoJUVFRan0hISGCsbGxUKFCBeHp06die0JCgmBtbS0AEObNm6d2zv79+wVzc3MBgMAfq0T6h99aIiqSGzduCM2aNRMTgVcfjRo1EhYvXixkZ2eLx0dERAgABHNzcyEuLq7I73/z5k0BgODp6anRl18y9OTJE8HCwuKt7z9x4kQBgBAQECC2zZkzRwAgNG/ePN9zPvvsMyZDRHqKN1ATUZHUqlULp0+fxpkzZ7Bz506EhYXh/PnzSExMREREBCZMmIBNmzZh586dKFOmDPbs2QMA6N69OxwcHAr8PiqVCocPH8bJkyeRkJCAjIwMCLl/0AEAbt68WaDXOXToEDIyMtChQ4c3vn/79u0RFBSEkydPwt/fHwBw+PBhAMDgwYPzPWf48OFYuHBhga+HiEoOJkNEpBVeXl7w8vICkHvPzIULF/DLL79g/fr12L9/PxYuXIhp06bh3r17AIDatWsX+LVv376NPn364OrVq288JjU1tUCvFRUVBQA4cODAOxdITExMFP9/XFwcAKBatWr5HvumdiIq+ZgMEZHWyWQyNG7cGOvWrcPz58+xbds2/P3335g2bdp7vZ6Pjw+uXr2KTz75BP/5z39Qt25dlCtXDqampnjx4oXGjdNvk5OTAwCoWbMmWrVq9dZjC5OwEZH+YjJERMWqU6dO2LZtm7gCtbOzMwDgxo0bBTr/xo0buHTpEj744ANs2bIFJibqP7Zu375dqHicnJwA5Jb3Vq5cWeDzHBwccOPGjTdu7aGrLT+ISPu4zhARvTehAFPIY2JiAACOjo4AIE5X37VrF+7fv//O85OSkgAAVatW1UiEgNxp94XRoUMHlClTBocPH8ajR48KfF67du0AAGvWrMm3/88//yxUHERUcjAZIqL3FhQUhOHDh+PkyZMafYIgYPPmzQgMDAQADBgwAADQqFEj9OrVCxkZGejVq5eYLOXJzs7Gtm3bxOfu7u4wNjbG5cuXxZuY82zfvh3z588vVMyVK1eGXC7Hs2fP0KNHD1y+fFnjmKysLGzbtk1t9Gr06NEoW7YsTp06BaVSqXb84cOHERwcXKg4iKjk4KKLRPTeFixYgMmTJwPIXYXaw8MDtra2SE5OxrVr18TS0ZAhQ7Bq1SoYGeX+/fX06VN069YNp0+fRpkyZdQWXbx8+bLGoouff/45Fi5cCCMjI7Rp0wZVq1bFzZs3cf78eXzzzTeYM2cOAM2Rqrctujh8+HCsXbsWRkZGaNiwIapXrw4TExPExcUhIiICz549w+7du9UWXly/fj2GDBkClUqFBg0aoH79+oiPj8exY8fw+eefi4kZf6wS6RcmQ0T03tLS0nDw4EEcOHAAYWFhuH//Ph4+fAgTExNUrVoVXl5eGDZsmMZKzgDw4sULLF++HGvXrsXly5fx/PlzfPDBB6hTpw569+6NiRMniscKgoAVK1YgKCgIN2/ehLGxMRo0aAB/f3/4+fm9cbf4d+1av3v3bixduhRhYWF49OgRrKysYG9vj4YNG6Jnz57o1asXLC0t1c45fvw45syZg1OnTiE7Oxu1atXC+PHjMXbsWO5aT6SnmAwRERGRQeM9Q0RERGTQdJIMhYeHw9/fH/Xq1YOVlRWcnZ3h6+uLW7duFej85ORkjB07FnZ2drCysoK3tzfOnz9fzFETERGRIdBJmczHxwcnTpxA//798eGHH+LBgwcIDAxEeno6Tp8+jfr167/x3JycHLRp0wYXL17EtGnTYGtri6CgIMTGxuLcuXNwc3Mr7vCJiIioFNNJMnTy5Ek0bdoUZcqUEdtu376NBg0awMfH563rhGzcuBF+fn4ICQmBj48PgNwl8t3d3dG1a1esXbu2uMMnIiKiUkzSG6ibNGkCADh37twbj/H19cXRo0dx//59cVouAIwbNw6rV69GUlJSoZbiJyIiInqVZDdQC4KAhw8fwtbW9q3HXbhwAY0bN1ZLhIDcTSGfP39e4PuOiIiIiPIj2d5ka9asQXx8PH744Ye3HpeQkIC2bdtqtNvb2wMA7t+/jwYNGuR7blZWFrKyssTnOTk5SEpKQqVKld65WzURERGVDIIgIC0tDVWrVtUYHNEGSZKhGzduYNKkSWjRogWGDx/+1mMzMjLyLYOZm5uL/W8yd+5czJo1q2jBEhERUYkQGxsr7nOoTTpPhh48eIDu3bvDxsYGoaGhMDY2fuvxFhYWaqM7eTIzM8X+N5kxYwa++OIL8XlKSgqcnZ0RExOD8+fPIzg4GHv27NE4r2fPnvjrr78KeklERERUjFJTU+Hk5ARra+tieX2dJkMpKSno2rUrkpOTcezYMVStWvWd59jb2yMhIUGjPa/tba9hZmaW76iSjY0NevXqhV69eiEqKgqLFi3CsmXLkJKSAgD48MMPUa5cuYJeFhEREelAcd3iorMbqDMzM9GjRw/cunULO3bsQN26dQt0XqNGjXD+/Hnk5OSotYeFhcHS0hLu7u5Fiqt69er47bffEBcXh6CgIHz44YcYP3682jHx8fEYPXo0Lly4UKT3IiIiopJHJ8mQSqWCn58fTp06hZCQELRo0SLf4xISEnDjxg28fPlSbPPx8cHDhw+xefNmse3x48cICQlBjx49tDatvmzZspgwYQIuXrwIBwcHtb7g4GAsX74cjRs3Rps2bRASEqIWIxEREekvnawz9Pnnn2PhwoXo0aMHfH19NfqHDBkCABgxYgRWrVqltsO0SqVC69atceXKFbUVqGNiYhAeHo5atWoVOI7U1FTY2NggJSWlwGUwQRBQu3ZtjSn8Dg4OmDhxIsaMGQM7O7sCx0BERESF8z6/vwtDJ8l
|
||
|
|
"text/plain": [
|
||
|
|
"<Figure size 640x480 with 1 Axes>"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"metadata": {},
|
||
|
|
"output_type": "display_data"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"from sklearn.preprocessing import StandardScaler\n",
|
||
|
|
"scaler = StandardScaler()\n",
|
||
|
|
"X_scaled = scaler.fit_transform(Xs)\n",
|
||
|
|
"svm_clf.fit(X_scaled, ys)\n",
|
||
|
|
"\n",
|
||
|
|
"#plt.subplot(122)\n",
|
||
|
|
"plt.plot(X_scaled[:, 0][ys==1], X_scaled[:, 1][ys==1], \"bo\")\n",
|
||
|
|
"plt.plot(X_scaled[:, 0][ys==0], X_scaled[:, 1][ys==0], \"ms\")\n",
|
||
|
|
"plot_svc_decision_boundary(svm_clf, -2, 2)\n",
|
||
|
|
"plt.xlabel(\"$x_0$\", fontsize=20)\n",
|
||
|
|
"plt.title(\"Scaled\", fontsize=16)\n",
|
||
|
|
"plt.axis([-2, 2, -2, 2]);"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"### Hard margin classification\n",
|
||
|
|
"\n",
|
||
|
|
"Hard margin classification corresponds to strictly imposing all training instances correctly classfied."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"#### Problems with hard margin classification\n",
|
||
|
|
"- Can fail if data not linearly separable.\n",
|
||
|
|
"- Sensitive to outliers."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 16,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:09.991474Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:09.990918Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:09.999255Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:09.998651Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"X_outliers = np.array([[3.4, 1.3], [3.2, 0.8]])\n",
|
||
|
|
"y_outliers = np.array([0, 0])\n",
|
||
|
|
"Xo1 = np.concatenate([X, X_outliers[:1]], axis=0)\n",
|
||
|
|
"yo1 = np.concatenate([y, y_outliers[:1]], axis=0)\n",
|
||
|
|
"Xo2 = np.concatenate([X, X_outliers[1:]], axis=0)\n",
|
||
|
|
"yo2 = np.concatenate([y, y_outliers[1:]], axis=0)\n",
|
||
|
|
"\n",
|
||
|
|
"svm_clf2 = SVC(kernel=\"linear\", C=10**9)\n",
|
||
|
|
"svm_clf2.fit(Xo2, yo2);"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 17,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:10.002384Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:10.001828Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:10.200031Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:10.199283Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAksAAAHACAYAAACyIiyEAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8WgzjOAAAACXBIWXMAAA9hAAAPYQGoP6dpAABotUlEQVR4nO3deVxU5RoH8N8wygyggAooCLjhFmq4gRuKds2lXFLUSq9Lmpl7rlGumVfNrDTMm2WiqZWRlmabmRiJGy7ljjvIosjqAgMM5/5x7oyMszAMw8ww/L6fz3xy3vOe9zznaPL4nvc8RyIIggAiIiIi0snB2gEQERER2TImS0REREQGMFkiIiIiMoDJEhEREZEBTJaIiIiIDGCyRERERGQAkyUiIiIiA5gsERERERnAZImIiIjIACZLRERERAbYXLJ04sQJTJ06FYGBgXBxcYG/vz+GDx+OhIQEo/bPzs7GxIkT4enpCRcXF/Ts2ROnTp3S2XfPnj1o164d5HI5/P39sXjxYhQVFZnzdIiIiKiSk9jau+HCw8Nx+PBhDBs2DG3atEFaWhoiIyPx4MEDHD16FK1atdK7b3FxMUJDQ/H3339j7ty58PDwwCeffIKkpCScPHkSTZs2Vff9+eef8dxzzyEsLAwvvfQSzp49i/Xr12PixInYsGGDJU6ViIiIKgGbS5bi4uLQoUMHODo6qtuuXLmC1q1bIzw8HNu2bdO7786dOzFixAh8++23CA8PBwCkp6ejWbNm6NevH3bs2KHuGxgYiOrVqyM+Ph7VqlUDACxYsAD/+c9/cOHCBbRo0aKCzpCIiIgqE5u7DdelSxeNRAkAmjZtisDAQFy8eNHgvtHR0ahbty6GDBmibvP09MTw4cPxww8/QKFQAAAuXLiACxcuYOLEiepECQAmT54MQRAQHR1txjMiIiKiyqxa6V2sTxAE3LlzB4GBgQb7nT59Gu3atYODg2YOGBwcjI0bNyIhIQGtW7fG6dOnAQAdOnTQ6Ofj4wNfX1/1dl0UCoU66QLEW3+ZmZmoU6cOJBJJWU+NiIiIrEAQBNy/fx8+Pj5aecOTKkWytH37diQnJ+Odd94x2C81NRXdu3fXavf29gYApKSkoHXr1khNTdVof7JvSkqK3mOsWLECS5cuLUv4REREZKOSkpLg6+trsI/NJ0uXLl3ClClT0LlzZ4wZM8Zg37y8PMhkMq12uVyu3l7yv/r65ubm6j1GREQEZs2apf6ek5MDf39/JCUlwdXVtfQTIiIiIqvLzc2Fn58fatasWWpfm06W0tLS8Nxzz8HNzQ3R0dGQSqUG+zs5OWncIlPJz89Xby/5X319Vdt1kclkOpMsV1dXJktERESVjDFLaGxugbdKTk4O+vXrh+zsbPzyyy/w8fEpdR9vb2/1LbaSVG2qMVS33/T1NeZYREREVDXYZLKUn5+PAQMGICEhAT/++COeeuopo/YLCgrCqVOnUFxcrNF+7NgxODs7o1mzZup+ABAfH6/RLyUlBbdv31ZvJyIiIrK5ZEmpVGLEiBE4cuQIvv32W3Tu3Flnv9TUVFy6dAmFhYXqtvDwcNy5cwe7du1St927dw/ffvstBgwYoL59FhgYiBYtWmDjxo1QKpXqvhs2bIBEIlHXaCIiIiKyuTVLs2fPxp49ezBgwABkZmZqFaEcNWoUAHGh9ZYtW3Djxg00bNgQgJgsderUCePGjcOFCxfUFbyVSqXWE2yrV6/GwIED8eyzz+LFF1/EuXPnEBkZiQkTJqBly5YWOVciIiKyfTZXwTssLAyHDh3Su10V7tixY7WSJQDIysrC3Llz8f333yMvLw8dO3bE+++/r1VTCQC+//57LF26FBcvXoSnpyfGjh2LRYsWoXr16kbHm5ubCzc3N+Tk5HCBNxERUSVRlp/fNpcsVTZMloiIiCqfsvz8trk1S0RERES2hMkSERERkQFMloiIiIgMYLJEREREZACTJSIiIiIDmCwRERERGcBkiYiIiMgAJktEREREBjBZIiIiIjKAyRIRERGRAUyWiIiIiAxgskRERERkAJMlIiIiIgOYLBEREREZwGSJiIiIyAAmS0REREQGMFkiIiIiMoDJEhEREZEBTJaIiIiIDGCyRERERGQAkyUiIiIiA5gsERERERnAZImIiIjIgGrWDoCIiIgsJzERuHdP/3YPD8Df33LxVAZMloiIiKqIxESgeXMgP19/H7kcuHyZCVNJvA1HRERURdy7ZzhRAsTthmaeqiImS0REREQGMFkiIiIiMoDJEhEREZEBTJaIiIiIDGCyRERERGQAkyUiKhdBEPDNN99gyJAh8PPzg1wuR61atRAUFIR58+YhMTHR4jEtWbIEEokES5Ys0WiPiYmBRCJBWFiYxWMiosrLJpOlBw8eYPHixejbty9q164NiUSCqKgoo/YNCwuDRCLR+alevbpG34YNG+rsN2nSpAo4KyL7k5KSgk6dOuHFF1/E999/j3r16mHw4MEIDQ1FcnIyVq9ejWbNmmH9+vVmOyYTHiLTeXiIdZQMkcvFfvSYTRalvHfvHt555x34+/vj6aefRkxMjNH7vv3225gwYYJG28OHDzFp0iQ8++yzWv2DgoIwe/ZsjbZmzZqZFDdRVZKVlYXQ0FBcv34dbdu2xZdffonAwED19qKiIqxduxbz58/H1KlToVQqMX36dCtGDAQHB+PixYtwdna2ahxE1uLvLxacZAXvsrHJZMnb2xupqamoV68e4uPj0bFjR6P37d27t1bbtm3bAAAjR47U2la/fn2MGjXK9GCJqqipU6fi+vXraNSoEf744w+4u7trbK9WrRpmz54NuVyOqVOnYs6cOejduzdatmxpnYABODs7o0WLFlY7PpEt8PdnMlRWNnkbTiaToV69emYbb8eOHXBxccGgQYN0bi8oKMDDhw/Ndjwie3f9+nV8/fXXAID3339fK1EqafLkyXj66adRWFiI9957T90+duxYg7fYo6KiIJFIMHbsWHVbWFgYevbsCQA4dOiQxu3zhg0blhp3abfwsrKysHjxYgQFBaFmzZpwdnZG69at8e677+LRo0da/UuujUpMTMT48ePh5+eH6tWra8RNRJWbTc4smVN6ejr279+PESNGwMXFRWv7H3/8AWdnZyiVSjRo0ABvvPEGZsyYoXc8hUIBhUKh/p6bm1shcRPZsr1796K4uBju7u4YOHCgwb4SiQT//ve/8ffff2Pv3r0QBAESicSk4/bt2xdyuRy//vor6tati759+6q3eZRzkcWFCxfQt29fJCUlwdvbG926dUP16tVx/PhxLFy4EN999x1iYmLg5uamte+VK1fQtm1bODo6omvXrhAEodzxEJHtsPtk6ZtvvkFRUZHOW3Bt2rRBt27d0Lx5c2RkZCAqKgozZ85ESkoKVq1apXO8FStWYOnSpRUdNpFNO3nyJACgbdu2qFat9L9GVLfSMzIycPPmTTRq1Mik47755pvo1KkTfv31V7Ro0cLoBz9Kk5eXh4EDByIpKQkLFizAwoUL4ejoCAB49OgRJkyYgK+++gpvvPEGvvjiC639d+zYgVGjRuHzzz+HTCYzS0xEZDts8jacOe3YsQOenp461zLt2bMH8+bNw6BBg/DKK6/g0KFD6NOnDz744APcvn1b53gRERHIyclRf5KSkir6FIhsTnp6OgCgbt26RvUv2U+1ry3ZsmULrl27hueffx7Lli1TJ0qAuM5p48aN8PLywpdffomsrCyt/WvXro3IyEgmSkR2yq6TpevXr+PIkSMYMWKEUf/6lUgkeOONN1BUVKT3CTyZTAZXV1eNDxEZJgiCtUMwaN++fQCAESNG6Nxeo0YNdOjQAUVFRThx4oTW9n/96186b88RkX2w62Rpx44dAHQ/BaePn58fACAzM7NCYiKyB6r1OHfu3DGq/927d9W/9vT0rJCYyuP69esAgH//+99667T99NNPAHTPjBmzuJyIKi+7XrO0Y8cONGnSBJ06dTJ6H9Vfmrb4FzqRrWjfvj22bduGU6dOoaioqNSZ2+PHjwMA6tSpY3RiUVxcXN4wjaY6Vt++fUu9tdi
|
||
|
|
"text/plain": [
|
||
|
|
"<Figure size 640x480 with 1 Axes>"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"metadata": {},
|
||
|
|
"output_type": "display_data"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"# plt.figure(figsize=(16,5))\n",
|
||
|
|
"# plt.subplot(121)\n",
|
||
|
|
"plt.plot(Xo1[:, 0][yo1==1], Xo1[:, 1][yo1==1], \"bs\")\n",
|
||
|
|
"plt.plot(Xo1[:, 0][yo1==0], Xo1[:, 1][yo1==0], \"yo\")\n",
|
||
|
|
"plt.text(0.3, 1.0, \"Impossible!\", fontsize=24, color=\"red\")\n",
|
||
|
|
"plt.xlabel(\"Petal length\", fontsize=14)\n",
|
||
|
|
"plt.ylabel(\"Petal width\", fontsize=14)\n",
|
||
|
|
"plt.annotate(\"Outlier\",\n",
|
||
|
|
" xy=(X_outliers[0][0], X_outliers[0][1]),\n",
|
||
|
|
" xytext=(2.5, 1.7),\n",
|
||
|
|
" ha=\"center\",\n",
|
||
|
|
" arrowprops=dict(facecolor='black', shrink=0.1),\n",
|
||
|
|
" fontsize=16,\n",
|
||
|
|
" )\n",
|
||
|
|
"plt.axis([0, 5.5, 0, 2]);"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 18,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:10.203351Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:10.202872Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:10.408005Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:10.407286Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAjIAAAHACAYAAAC4foLWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8WgzjOAAAACXBIWXMAAA9hAAAPYQGoP6dpAACzHUlEQVR4nOzdd1gT2RoG8De0UFSwYBdZ+0roCFiwd9feRcGKDbuIvfeuoKIuiqy969obVkSpQlDX3lFRIArSOfePyFwjxaCciPj9nmeevZxM5p24e83HzJnziRhjDIQQQgghvyC1n30ChBBCCCHfiwoZQgghhPyyqJAhhBBCyC+LChlCCCGE/LKokCGEEELIL4sKGUIIIYT8sqiQIYQQQsgviwoZQgghhPyyqJAhhBBCyC+LChlCCCGE/LLyVMgEBgbC1dUVJiYm0NPTg5GREXr06IF79+4p9f64uDi4uLjA0NAQenp6aNKkCUJCQrLd9+jRo7CysoK2tjaMjIwwa9YspKWl5eV0CSGEEFLIifLSa6lbt264du0aunfvDjMzM7x+/Rqenp6Ij49HQEAAJBJJju/NyMiAg4MDbt26BTc3N5QqVQrr16/H8+fPERwcjOrVqwv7njx5Eu3atUPjxo3Ru3dvREREYN26dXBxccGGDRt+7BMTQgghpNDIUyHj7+8PGxsbaGlpCWP379+HqakpunXrhu3bt+f43r1796Jnz57Yt28funXrBgCIjo5GjRo10KZNG+zcuVPY18TEBJqamggKCoKGhgYAYPr06Vi4cCFu376NWrVq5fmDEkIIIaTwydOtpXr16ikUMQBQvXp1mJiY4M6dO7m+d//+/ShTpgy6dOkijBkaGqJHjx44cuQIkpOTAQC3b9/G7du34eLiIhQxADBixAgwxrB///68nDIhhBBCCjGNb++SO8YY3rx5AxMTk1z3Cw0NhZWVFdTUFGsnW1tbbNq0Cffu3YOpqSlCQ0MBADY2Ngr7lS9fHhUrVhRez0lycrJQFAHyW1oxMTEoWbIkRCJRXj4aIYQQQn4Sxhg+fvyI8uXLZ6kdvvTDhcyOHTvw8uVLzJ07N9f9oqKi0LBhwyzj5cqVAwC8evUKpqamiIqKUhj/et9Xr17lmrNo0SLMmTNH2dMnhBBCSAH2/PlzVKxYMcfXf6iQuXv3LkaOHIm6devC2dk5130TExMhFouzjGtrawuvf/nPnPb98OFDrjlTpkzB+PHjhZ9lMhmMjIzw/PlzFCtWLPcP9AM+fPiAqVOn4p9//hHGihcvjhUrVqBr167ccgkhhJDC6MOHD6hUqRKKFi2a637fXci8fv0a7dq1g76+Pvbv3w91dfVc99fR0VG45ZMpKSlJeP3Lf+a0b+brORGLxdkWQcHBwejYsWOu7/0RxYoVg6+vL3r06IHBgwfjzZs3iI2NxcCBA3H69GmsW7cOJUuW5JZPCCGEFEbfmhbyXQviyWQytGnTBnFxcTh16hTKly//zfeUK1dOuG30pcyxzGNk3lLKaV9lsrLTqVMnuLq6IiEh4bver6y//voLkZGR6NGjhzC2Z88eSCQSHD9+nGs2IYQQ8rvJcyGTlJSE9u3b4969ezh27Bhq166t1PssLCwQEhKCjIwMhfEbN25AV1cXNWrUEPYDgKCgIIX9Xr16hRcvXgivf49169bBwsIC/v7+330MZZQsWRJ79uzBrl27ULx4cQDyK1hdu3bNtkAjhBBCyPfJUyGTnp6Onj174vr169i3bx/q1q2b7X5RUVG4e/cuUlNThbFu3brhzZs3OHjwoDD27t077Nu3D+3btxduB5mYmKBWrVrYtGkT0tPThX03bNgAkUgkrEGTV5lzcR48eAAHBwdMnjw529tX+alXr16QSqVo06YNAGD+/PnZTmImhBBCyPfJ04J4Y8eOxZo1a9C+fXuFWyeZ+vbtCwDo378/tm3bhsePH8PY2BiAvAhq0KABpFKpwsq+z549Q2BgIGrWrCkc59ixY+jQoQOaNGkiFAOenp4YNGgQNm3alKcP+OHDB+jr6yM4OBgjR45EQECA8JqpqSl8fX1/6CqPMhhjOHz4MDp06KAwlyg5ORkZGRnfnPdDCCGE/G4yv79lMlnuD+uwPGjUqBEDkOOWydnZmQFgjx8/Vnh/TEwMGzRoECtZsiTT1dVljRo1YoGBgdlmHTp0iFlYWDCxWMwqVqzIpk+fzlJSUvJyuowxxmQyGQPAZDIZS0tLY4sWLWKamprCOWtqarL58+ez1NTUPB/7R02aNInVqlWL3bx5U+XZhBBCSEH25fd3bvJ0ReZXlF1FFx4ejn79+iE8PFzYz9bWFr6+vgpXhni6du0aHBwcwBiDuro6pk6diunTp2dZOZkQQgj5HSl7Rea7nlr61ZmZmSEwMBBTp04VVgu8efMmLCwssGbNmiwTknkoWbIkrK2tAchvu82bNw/29vaQSqXcswkhhJDC4rcsZABAS0sLCxYswLVr14QnppKSkjB27Fg0b94cT58+5Zpfq1Yt+Pv7Y86cOUJPqdDQUFhbW2Pp0qUKE50JIYQQkr3ftpDJZG9vj9DQUIwePVoY8/Pzg6mpKbZs2QKed940NTUxc+ZMBAQECI+xp6SkwN3dHY0aNcKDBw+4ZRNCCCGFwW9fyACArq4u1qxZg/Pnz8PIyAgA8PHjRwwaNAgdOnTA69evueZbW1sjODgYbm5uwgqG165dg7m5OQIDA7lmE0IIIb8yKmS+0LRpU4SHh2PAgAHC2LFjx2BiYoK9e/dyzdbW1sbSpUtx+fJlVKlSBQDw559/cn80nBBCCPmVUSHzFX19fWzZsgVHjhxBmTJlAAAxMTHo2bMnevfujZiYGK75DRo0wK1btzBy5Ej4+vpCU1OTax4hhBDyK6NCJgcdOnSAVCpVWEl49+7dkEgkOHnyJNfsIkWKwNPTM0v7h9DQUPTs2RPR0dFc8wkhhJBfBRUyuShVqhT27t2LnTt3Cj2ToqKi0LZtW7i4uODjx48qO5fk5GQ4OTlh7969MDExweHDh1WWTQghhBRUVMh8g0gkQu/evSGVStG6dWthfPPmzTA3N8fly5dVch7//fefMOk4OjoanTt3hrOzM+Li4lSSTwghhBREVMgoqXz58jhx4gQ2btwIPT09AMDjx4/RuHFjjB8/HomJiVzzzczMIJVK0bFjR2HM19cXpqamOHfuHNdsQgghpKCiQiYPRCIRXFxcEB4eDgcHBwDyhpCrVq2CtbU1goKCuOaXKVMGhw4dgo+Pj7Bc84sXL9CiRQu4uroiISGBaz4hhBBS0FAh8x2qVKkCPz8/LF++HGKxGABw584d2NvbY9asWUhNTeWWLRKJ4OzsjIiICDRr1kwYX7duHSwsLGjdGUIIIb8VKmS+k7q6OiZMmICQkBCFnklz586Fvb09IiMjueYbGRnhzJkz8PDwgI6ODgDgyZMnQu8oQggh5HdA33o/qHbt2rh+/Tpmz54NdXV1ABCKm+XLl3PtmaSmpgZXV1eEhYXB3t4e06dPF4oqQggh5HcgYjybCRUAmW3A4+LioK+vzzUrODgYTk5OuH37tjDWoEED+Pj4oGrVqlyz09LSAEBoQJk5tmXLFgwcOFBhnBBCCCnoMr+/ZTKZMC80O7/NFZnevXvjzZs3XDMyeyZNmDBB6Jl09epVmJubw8vLi2sDSg0NjSzFyqJFizB06FDUr18fd+/e5ZZNCCGE/Cy/zRUZQL7AnZeXF7p27co99/Lly+jfvz8eP34sjLVq1Qre3t6oUKEC9/y3b9+icuXKSEpKAiDv5bR48WKMGjWK5tEQQggp8OiKTDbevXuHbt26wdHREbGxsVyzGjZsiFu3bsHFxUUYO336NCQSCXbs2MH16gwAlC5dGn5+fqhRowYAICkpCWPHjkWzZs3w5MkTrtmEEEKIqvw2hUz79u2F/71z505IJBKcOnWKa2bRokWxceNGnDhxAuXKlQMAxMXFoW/fvujevTv3nkn29vYIDQ3FqFGjhLGLFy/CzMwM3t7e3IspQgghhLffppD5559/sH37dhgYGAAAXr16hTZt2mDYsGGIj4/
|
||
|
|
"text/plain": [
|
||
|
|
"<Figure size 640x480 with 1 Axes>"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"metadata": {},
|
||
|
|
"output_type": "display_data"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"# plt.subplot(122)\n",
|
||
|
|
"plt.plot(Xo2[:, 0][yo2==1], Xo2[:, 1][yo2==1], \"bs\")\n",
|
||
|
|
"plt.plot(Xo2[:, 0][yo2==0], Xo2[:, 1][yo2==0], \"yo\")\n",
|
||
|
|
"plot_svc_decision_boundary(svm_clf2, 0, 5.5)\n",
|
||
|
|
"plt.xlabel(\"Petal length\", fontsize=14)\n",
|
||
|
|
"plt.annotate(\"Outlier\",\n",
|
||
|
|
" xy=(X_outliers[1][0], X_outliers[1][1]),\n",
|
||
|
|
" xytext=(3.2, 0.08),\n",
|
||
|
|
" ha=\"center\",\n",
|
||
|
|
" arrowprops=dict(facecolor='black', shrink=0.1),\n",
|
||
|
|
" fontsize=16,\n",
|
||
|
|
" )\n",
|
||
|
|
"plt.axis([0, 5.5, 0, 2]);"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"### Soft margin classification\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"Allow some margin violations by varying hyperparameter $C$.\n",
|
||
|
|
"\n",
|
||
|
|
"\n",
|
||
|
|
"Recall, large $C$ corresponds to small regularisation and thus few margin violations. Small $C$ corresponds to greater regularisation and thus more margin violations."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"#### Load data"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 19,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:10.411326Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:10.410852Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:10.418578Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:10.417946Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"import numpy as np\n",
|
||
|
|
"from sklearn import datasets\n",
|
||
|
|
"from sklearn.pipeline import Pipeline\n",
|
||
|
|
"from sklearn.preprocessing import StandardScaler\n",
|
||
|
|
"from sklearn.svm import LinearSVC\n",
|
||
|
|
"\n",
|
||
|
|
"iris = datasets.load_iris()\n",
|
||
|
|
"X = iris[\"data\"][:, (2, 3)] # petal length, petal width\n",
|
||
|
|
"y = (iris[\"target\"] == 2).astype(np.float64) # Iris-Virginica"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"#### Train SVMs"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 20,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:10.421567Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:10.421137Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:10.424523Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:10.423871Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"# disable convergence warning from early stopping\n",
|
||
|
|
"from warnings import simplefilter\n",
|
||
|
|
"from sklearn.exceptions import ConvergenceWarning\n",
|
||
|
|
"simplefilter(\"ignore\", category=ConvergenceWarning)"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 21,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:10.427696Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:10.427158Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:10.441192Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:10.440527Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"name": "stderr",
|
||
|
|
"output_type": "stream",
|
||
|
|
"text": [
|
||
|
|
"/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/sklearn/svm/_classes.py:32: FutureWarning: The default value of `dual` will change from `True` to `'auto'` in 1.5. Set the value of `dual` explicitly to suppress the warning.\n",
|
||
|
|
" warnings.warn(\n",
|
||
|
|
"/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/sklearn/svm/_classes.py:32: FutureWarning: The default value of `dual` will change from `True` to `'auto'` in 1.5. Set the value of `dual` explicitly to suppress the warning.\n",
|
||
|
|
" warnings.warn(\n"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"scaler = StandardScaler()\n",
|
||
|
|
"svm_clf1 = LinearSVC(C=1, loss=\"hinge\", random_state=42)\n",
|
||
|
|
"svm_clf2 = LinearSVC(C=100, loss=\"hinge\", random_state=42)\n",
|
||
|
|
"\n",
|
||
|
|
"scaled_svm_clf1 = Pipeline((\n",
|
||
|
|
" (\"scaler\", scaler),\n",
|
||
|
|
" (\"linear_svc\", svm_clf1),\n",
|
||
|
|
" ))\n",
|
||
|
|
"scaled_svm_clf2 = Pipeline((\n",
|
||
|
|
" (\"scaler\", scaler),\n",
|
||
|
|
" (\"linear_svc\", svm_clf2),\n",
|
||
|
|
" ))\n",
|
||
|
|
"\n",
|
||
|
|
"scaled_svm_clf1.fit(X, y);\n",
|
||
|
|
"scaled_svm_clf2.fit(X, y);"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"#### Compute support vectors"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 22,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:10.444584Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:10.444121Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:10.451499Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:10.450894Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"# Convert to unscaled parameters\n",
|
||
|
|
"b1 = svm_clf1.decision_function([-scaler.mean_ / scaler.scale_])\n",
|
||
|
|
"b2 = svm_clf2.decision_function([-scaler.mean_ / scaler.scale_])\n",
|
||
|
|
"w1 = svm_clf1.coef_[0] / scaler.scale_\n",
|
||
|
|
"w2 = svm_clf2.coef_[0] / scaler.scale_\n",
|
||
|
|
"svm_clf1.intercept_ = np.array([b1])\n",
|
||
|
|
"svm_clf2.intercept_ = np.array([b2])\n",
|
||
|
|
"svm_clf1.coef_ = np.array([w1])\n",
|
||
|
|
"svm_clf2.coef_ = np.array([w2])\n",
|
||
|
|
"\n",
|
||
|
|
"# Find support vectors (LinearSVC does not do this automatically)\n",
|
||
|
|
"t = y * 2 - 1 # t = +/-1\n",
|
||
|
|
"support_vectors_idx1 = (t * (X.dot(w1) + b1) < 1).ravel()\n",
|
||
|
|
"support_vectors_idx2 = (t * (X.dot(w2) + b2) < 1).ravel()\n",
|
||
|
|
"svm_clf1.support_vectors_ = X[support_vectors_idx1]\n",
|
||
|
|
"svm_clf2.support_vectors_ = X[support_vectors_idx2]"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"#### Plot"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 23,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:10.454748Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:10.454166Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:10.900018Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:10.899257Z"
|
||
|
|
},
|
||
|
|
"scrolled": true
|
||
|
|
},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAABUUAAAGUCAYAAAASz0NVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8WgzjOAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOzdd1gU1/oH8O/SFhHBAihSRFAp9oLYBXuJsibW6C9qenLTb5KbmGiMmmjqTYxJ7k3VRE0suS52EcXeUNSoIBZQkCYIgnRY5vcH2QnD7tJh2/fzPDy6M2dmzuwMy7tnzjmvTBAEAURERERERERERERmwkLfFSAiIiIiIiIiIiJqTmwUJSIiIiIiIiIiIrPCRlEiIiIiIiIiIiIyK2wUJSIiIiIiIiIiIrPCRlEiIiIiIiIiIiIyK2wUJSIiIiIiIiIiIrPCRlEiIiIiIiIiIiIyK2wUJSIiIiIiIiIiIrPCRlEiIiIiIiIiIiIyK2wUJSJqZNHR0XjzzTfRv39/dOjQATY2Nmjbti2GDBmCpUuXIikpSS/1iouLw1dffYUFCxagZ8+esLKygkwmw4oVK/RSHyIiIiKqninHlVu2bEFwcDDatGmDli1bonfv3vj4449RWlraJNsREVUlEwRB0HcliIhMQVZWFl5++WVs2LABgiDA09MTfn5+aNu2LVJSUnD69GkUFxfD1tYWYWFhGDduXLPW75VXXsGXX36psXz58uV49913m7UuRERERKSbqceV6u2trKwwatQo2Nvb4+DBg7h//z6GDRuG8PBwtGjRotG2IyLShj1FiYgaQXp6OoKDg7F+/Xp069YNe/fuxa1bt7Bv3z789ttvOHz4MDIyMvD+++9DEAS9BGs9evTA66+/jg0bNiA2Nhb/93//1+x1ICIiIqLqmXpcqVQq8eWXX8Le3h6nT5/Gvn378Mcff+D69evo2bMnjh07hsWLFzfadkREurCnKBFRA5WVlWHo0KE4c+YMhg8fjh07dsDR0VFn+ZMnT6Jnz56wt7dvxlpqWrBgAdatW8eeokREREQGwhziyoEDByIqKgorVqzAO++8I1l37NgxDB8+HHK5HOnp6ZJzr+92RES6sKcoEVEDLV++HGfOnIGzszM2bdpUYxA2ePBgvQeuRERERGR4TD2uTE5ORlRUFADg0Ucf1Vg/bNgweHh4oLi4GLt3727wdkRE1WGjKBFRA2RlZeHzzz8HAHz00UdwdXXVc42IiIiIyBiZQ1x5/vx5AEDbtm3RuXNnrWUGDBggKduQ7YiIqsNGUSKiBvjpp5+Ql5cHFxeXRpmjc8GCBZDJZHX+OXToUMNPhoiIiIj0xhziyoSEBACAp6enzjIeHh6Ssg3ZjoioOlb6rgARkTHbs2cPACA0NBRWVg3/SB02bFi9tuvQoUODj01ERERE+mMOceWDBw8AAC1bttRZRj0dQG5uboO3IyKqDhtFiYgaIDo6GgAQFBTUKPt78skn8eSTTzbKvoiIiIjIeDCuJCJqXhw+T0RUTyUlJbh//z4AwM3NTb+VISIiIiKjZS5xZatWrQAA+fn5Osvk5eUBABwcHBq8HRFRddhTlIionmQymfh/QRAaZZ8//PADjh07Vuft3nrrLfj5+TVKHYiIiIioeZlLXOnl5QUASEpK0llGvU5dtiHbERFVh42iRET1ZG1tDQ8PDyQlJeHq1auYOHFig/d57NgxrFu3rs7bLViwgI2iREREREbKXOLKvn37AgDu3buHhIQErZnkz549CwDo169fg7cjIqoOh88TETXA1KlTAQA//vgjSktLqy2bnZ2NlJSUasusXbsWgiDU+Sc4OLixTomIiIiI9MAc4kp3d3cEBgYCADZu3Kix/tixY0hKSoJcLsekSZMavB0RUXXYKEpE1ABvvfUW2rRpgytXrmD27NnIzMzUKJORkYGPP/4Yvr6+KCkp0UMtiYiIiMjQmUtcuWjRIgDAqlWrxORSQEUv0Oeffx4A8MILL8DR0bFRtiMi0kUmNNaEJUREZurkyZN45JFHkJqaCltbWwwaNAgdO3ZEQUEB4uPjcenSJQiCAGdnZ9y9e1dv9YyOjhYDRgC4efMmMjMz4e7uLpnQf9u2bXB1ddVHFYmIiIjMmrnElS+//DJWr14Na2trjB49Gi1btsSBAwdw//59DB06FPv370eLFi0abTsiIm3YKEpE1Aiys7Px3//+Fzt37kRMTAxyc3Nha2uLjh07olevXhg9ejQeeeQRuLi46K2Ohw4dQkhISI3lEhISOEE9ERERkZ6YS1y5efNmfP3117hw4QJKS0vh4+ODefPm4dVXX4WNjY3OfdZ3OyKiqtgoSkRERERERERERGaFc4oSERERERERERGRWWGjKBEREREREREREZkVNooSERERERERERGRWWGjKBEREREREREREZkVNooSERERERERERGRWWGjKBEREREREREREZkVK31XwNiVl5cjJSUFrVq1gkwm03d1iIiIiOpMEAQ8ePAAHTt2hIUFn5kbI8akREREZMz0EY+yUbSBUlJS4OHhoe9qEBERETVYUlIS3N3d9V0NqgfGpERERGQKmjMeZaNoA7Vq1QpAxUVzcHDQc22IiIiI6i43NxceHh5iXEPGhzEpERERGTN9xKMG1ygaFRWFdevWITIyErdu3UK7du0waNAgrFixAt26dat22+DgYBw+fFjrOisrK5SWloqvvby8cPv2bY1yzzzzDP7zn//Uur7q4UkODg4MQImIiMiocdh1BWOLRwHGpERERGQamjMeNbhG0Y8++gjHjx/HjBkz0KtXL6SlpWHNmjXo168fTp06hR49eujc9p133sGTTz4pWZafn49nn30W48aN0yjfp08f/POf/5QsqynQJSIiIiLTxniUiIiIyPQZXKPoa6+9ho0bN8LGxkZcNmvWLPTs2ROrVq3C+vXrdW47duxYjWXq8nPnztVY5+bmhnnz5jVCrYmIiIjIVDAeJSIiIjJ9BtcoOmTIEI1lXbt2Rffu3REbG1vn/W3cuBEtW7ZEaGio1vUlJSUoLS1Fy5Yt67xvIiIiIjI9jEeJiIiITF/z5LhvIEEQkJ6eDicnpzptl5GRgf3790OhUGgNMg8ePAg7OzvY29vDy8sLX375ZY37LC4uRm5uruSHiIiIiEybIcWjAGNSIiIiooYyuJ6i2mzYsAHJyclYtmxZnbbbtGkTysrKtA5V6tWrF4YNGwZfX1/cu3cPa9euxSuvvIKUlBR89NFHOve5cuVKvP/++3U+ByIiIiIyXoYUjwKMSYmIiIgaSiYIgqDvSlTn6tWrCAoKQvfu3XH06FFYWlrWetshQ4bgxo0bSElJgZVV9e2/giBg4sSJOHDgABISEuDu7q61XHFxMYqLi8XXubm58PDwQE5OTp0yfZaWlkKlUtW6PJGpsrS0hLW1tb6rQURk1nJzc+Ho6FjneMZcGFo8CjReTEpERERkCPQRjxp0T9G0tDRMnjwZjo6O2Lp1a50C0Pj4eJw8eRIvvPBCjQEoAMhkMrz66qvYt28fDh06pHPCe7lcDrlcXut6VJWbm4vMzExJEEtk7uRyOZycnPgljoiIDI4hxqNAw2NSIiIiInNnsI2iOTk5mDhxIu7fv4+jR4+iY8eOddp+48aNALRn+dTFw8MDAJCVlVWnY9VWbm4ukpOTYW9vDycnJ1hbW0MmkzXJsYiMgSAIKC0tRU5ODpKTkwGADaNERGQwTDEeJSIiIqIKBtkoWlRUhClTpuDatWuIiIhAQEBAnfexceNG+Pj4YNCgQbXeJj4+HgDg7Oxc5+PVRmZmJuzt7eHu7s7GUKK/tGjRAq1atcKdO3eQmZnJRlEiIjIIphqPEhEREVEFg8s+r1KpMGvWLJw8eRJbtmzB4MGDtZZLTU3F1atXUVpaqrHu/PnziI2NxaOPPqp126ysLI35PEtLS7Fq1SrY2NggJCSk4SdSRWlpKYqLi+Ho6MgGUaIqZDIZHB0dUVxcrPV3moiIqDmZajxKRERERH8zuJ6i//znP7F9+3ZMmTIFWVlZWL9+vWS9em6lt99+G+vWrUNCQgK8vLw
|
||
|
|
"text/plain": [
|
||
|
|
"<Figure size 1600x400 with 2 Axes>"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"metadata": {},
|
||
|
|
"output_type": "display_data"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"plt.figure(figsize=(16,4))\n",
|
||
|
|
"plt.subplot(121)\n",
|
||
|
|
"plt.plot(X[:, 0][y==1], X[:, 1][y==1], \"g^\", label=\"Iris-Virginica\")\n",
|
||
|
|
"plt.plot(X[:, 0][y==0], X[:, 1][y==0], \"bs\", label=\"Iris-Versicolor\")\n",
|
||
|
|
"plot_svc_decision_boundary(svm_clf1, 4, 6)\n",
|
||
|
|
"plt.xlabel(\"Petal length\", fontsize=14)\n",
|
||
|
|
"plt.ylabel(\"Petal width\", fontsize=14)\n",
|
||
|
|
"plt.legend(loc=\"upper left\", fontsize=14)\n",
|
||
|
|
"plt.title(\"$C = {}$\".format(svm_clf1.C), fontsize=16)\n",
|
||
|
|
"plt.axis([4, 6, 0.8, 2.8])\n",
|
||
|
|
"\n",
|
||
|
|
"plt.subplot(122)\n",
|
||
|
|
"plt.plot(X[:, 0][y==1], X[:, 1][y==1], \"g^\")\n",
|
||
|
|
"plt.plot(X[:, 0][y==0], X[:, 1][y==0], \"bs\")\n",
|
||
|
|
"plot_svc_decision_boundary(svm_clf2, 4, 6)\n",
|
||
|
|
"plt.xlabel(\"Petal length\", fontsize=14)\n",
|
||
|
|
"plt.title(\"$C = {}$\".format(svm_clf2.C), fontsize=16)\n",
|
||
|
|
"plt.axis([4, 6, 0.8, 2.8]);"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"**Exercises:** *You can now complete Exercise 1 in the exercises associated with this lecture.*"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "slide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"## Non-linear classification with polynomial features"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"So far we have considered linear classification only.\n",
|
||
|
|
"\n",
|
||
|
|
"Most data-sets are not linearly separable."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"### 1D example \n",
|
||
|
|
"\n",
|
||
|
|
"Consider 1D feature space with $x_1$, which is clearly not linearly separable.\n",
|
||
|
|
"\n",
|
||
|
|
"However, if augment feature space with $x_2=(x_1)^2$ we see that the resulting 2D feature space is linearly separable."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 24,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:10.903323Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:10.902893Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:10.908148Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:10.907548Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"X1D = np.linspace(-4, 4, 9).reshape(-1, 1)\n",
|
||
|
|
"X2D = np.c_[X1D, X1D**2]\n",
|
||
|
|
"y = np.array([0, 0, 1, 1, 1, 1, 1, 0, 0])"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 25,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:10.911084Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:10.910698Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:11.175999Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:11.175328Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAABYwAAAGFCAYAAABe77cAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8WgzjOAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA8KklEQVR4nO3de3RedZkv8G+Spklb2mCp0HZouTnoCAwwM8KAFRWhIBxZoCC48IIVkJkBZBCOgi2FKiDeBo4MAkeGq2eQQfA24mm5DZbhQEGcBbOQMpVeaItQRpoSmjRN3vNH00jJu0uStnnfJJ/PWl2r2Xu/uw9Pd3ee98sv+60plUqlAAAAAAAw7NVWugAAAAAAAKqDwBgAAAAAgCQCYwAAAAAAugiMAQAAAABIIjAGAAAAAKCLwBgAAAAAgCQCYwAAAAAAugiMAQAAAABIkozozUGdnZ1ZsWJFxo4dm5qamm1dEwAAA6xUKmXNmjWZPHlyamutKRhuzPsAAENbX+b9XgXGK1asyJQpU7ZKcQAAVK9ly5Zl5513rnQZDDDzPgDA8NCbeb9XgfHYsWO7Tzhu3Lgtr2wLtLe3Z+7cuZk+fXrq6+srWku10ZtielOevhTTm2J6U0xvytOXYtXUm+bm5kyZMqV77mN4Me8PDnpTnr4U05vyWlpaMnny5CTJkiVLsv3221e2oCrjuilPX4rpTbFq6k1f5v1eBcYbfyxt3LhxVTFAjh49OuPGjat4o6uN3hTTm/L0pZjeFNObYnpTnr4Uq8beeBzB8GTeHxz0pjx9KaY35dXV1XX/vhrue9XGdVOevhTTm2LV2JvezPseUAcAAAAAQBKBMQAAAAAAXQTGAAAAAAAkERgDAAAAANBFYAwAAAAAQBKBMQAAAAAAXUZUugAAAGD4Wbo0WbWq5/b165NFi5ry5JPJiDLvViZMSKZO3fb1AQD01lCbawTGAADAgFq6NHnnO5PW1nJ765N8oPC1jY3Js89W55srAGD4GYpzjUdSAABAlXnttdcye/bsHHnkkRk/fnxqampy0003lT22s7Mz3/ve97Lffvtl1KhR2WGHHXLooYfmP/7jPwa26D5YtaroTdVba20tv4IHAKAShuJcY4UxAABUmVWrVmXOnDmZOnVq9t133zz44IOFx86YMSM/+MEP8ulPfzpnnnlmWlpa8uSTT+all14auIIBABgyBMYAAFBlJk2alJUrV2bixIl5/PHH8573vKfscXfccUduvvnm3HXXXTnuuOMGuEoAAIYij6QAAIAq09DQkIkTJ77lcd/5zndywAEH5LjjjktnZ2daWloGoDoAAIYygTEAAAxCzc3Neeyxx/Ke97wnF154YZqamrLddttl9913zx133FHp8gAAGKQ8kgIAAAahRYsWpVQq5fbbb8+IESPyjW98I01NTbnqqqty0kknZdy4cTnyyCPLvratrS1tbW3dXzc3NydJ2tvb097evs1rX78+2fCp4f19fXsGoMyqs/HvZiD+jgYTfSmmN+W9sR8Ddd8bTFw35elLseHem8Ey1/Tl70dgDAAAg9Brr72WJHnllVfy//7f/8uBBx6YJDnmmGOy22675Wtf+1phYHz55Zfnkksu6bF97ty5GT169LYrusuiRU1JPtDv18+f/3BWrly91eoZbObNm1fpEqqSvhTTm021trZ2//7+++9PY2NjBaupXq6b8vSl2HDtzWCZa15//fVeHyswBgCAQWjUqFFJkt122607LE6S7bbbLh/5yEdy2223Zf369RkxoufIf8EFF+Tcc8/t/rq5uTlTpkzJ9OnTM27cuG1e+5NPbtnrp017b/bff+vUMpi0t7dn3rx5Ofzww1Nf3/+VTEONvhTTm/Le+Lz3Qw89NNtvv33liqlCrpvy9KXYcO/NYJlrNv5EWW8IjAEAYBCaPHlykmSnnXbqsW/HHXdMe3t7Wlpa0tTU1GN/Q0NDGhoaemyvr68fkDd6ZTLsPr6+PsPw/Wi3gfp7Gmz0pZjebOqNvdCbYnpTnr4UG669GSxzTV/+bnzoHQAADEKTJ0/OxIkTs3z58h77VqxYkcbGxowdO7YClQEAMJgJjAEAYJA68cQTs2zZsk2eGbhq1ar85Cc/yaGHHpraWuM+AAB945EUAABQha6++uq8+uqrWbFiRZLkZz/7WV544YUkyVlnnZWmpqZccMEFueOOO/Kxj30s5557bpqamnLttdemvb09l112WSXLBwBgkBIYAwBAFfrWt76VJUuWdH9911135a677kqSfPKTn0xTU1N22mmnzJ8/P+edd17+4R/+Ie3t7TnooINy2223Zd99961U6QAADGICYwAAqEKLFy/u1XG77757d5A8WEyYkDQ2Jq2tfX9tY+OG1wMAVIOhONcIjAEAgAE1dWry7LPJqlU9961f35758x/OtGnvzYgRPT/Ne8KEDa8HAKgGQ3GuERgDAAADburU8m+Q2tuTlStXZ//9k/qe76sAAKrOUJtrfGwyAAAAAABJBMYAAAAAAHQRGAMAAAAAkERgDAAAAABAF4ExAAAAAABJBMYAAAAAAHQRGAMAAAAAkERgDAAAAABAF4ExAAAAAABJBMYAAAAAAHQRGAMAAAAAkERgDAAAAABAF4ExAAAAAABJBMYAAAAAAHQRGAMAAAAAkERgDAAAAABAF4ExAAAAAABJBMYAAAAAAHQRGAMAAAAAkERgDAAAAABAF4ExAAAAAABJBMYAAAAAAHQRGAMAAAAAkERgDAAAAABAF4ExAAAAAABJBMYAAAAAAHQRGAMAAAAAkERgDAAAAABAF4ExAAAAAABJBMYAAAAAAHQRGAMAAAAAkERgDAAAAABAF4ExAAAAAABJBMYAAAAAAHQRGAMAAAAAkERgDAAAAABAF4ExAAAAAABJBMYAAAAAAHQRGAMAAAAAkERgDAAAAABAF4ExAAAAAABJBMYAAAAAAHQRGAMAAAAAkERgDAAAAABAF4ExAAAAAABJBMYAAAAAAHQRGAMAAAAAkERgDAAAAABAF4ExAAAAAABJBMYAAAAAAHQRGAMAAAAAkERgDAAAAABAF4ExAAAAAABJBMYAAFB1XnvttcyePTtHHnlkxo8fn5qamtx0002bHNPZ2ZmbbropxxxzTKZMmZIxY8Zk7733zte+9rW0trZWpnAAAAY9gTEAAFSZVatWZc6cOXnmmWey7777lj3m9ddfz2c/+9m8/PLLOeOMM3LllVfmgAMOyOzZs/PhD384pVJpgKsGAGAoGFHpAgAAgE1NmjQpK1euzMSJE/P444/nPe95T49jRo4cmYcffjgHH3xw97bTTjstu+66a2bPnp377rsvhx122ECWDQDAEGCFMQAAVJmGhoZMnDhxs8eMHDlyk7B4o+OOOy5J8swzz2yT2gAAGNqsMAYAgCHkxRdfTJJMmDCh8Ji2tra0tbV1f93c3JwkaW9vT3t7+7Yt8C1s/PMrXUc10pvy9KWY3pT3xn5Uw32v2rhuytOXYnpTrJp605caBMYAADCEfOMb38i4cePy4Q9/uPCYyy+/PJdcckmP7XPnzs3o0aO3ZXm9Nm/evEqXULX0pjx9KaY3m3rjB4Pef//9aWxsrGA11ct1U56+FNObYtXQm9dff73XxwqMAQBgiLjsssty77335pprrsn2229feNwFF1yQc889t/vr5ubmTJkyJdOnT8+4ceMGoNJi7e3tmTdvXg4//PDU19dXtJZqozfl6UsxvSmvpaWl+/eHHnroZu+Xw5Hrpjx9KaY3xaqpNxt/oqw3BMYAADAE/PCHP8zMmTPzuc99Ln/zN3+z2WMbGhrS0NDQY3t9fX3F38xsVE21VBu9KU9fiunNpt7YC70ppjfl6UsxvSlWDb3py5/vQ+8AAGCQmzdvXj796U/n6KOPzrXXXlvpcgAAGMQExgAAMIg9+uijOe644/JXf/VXueOOOzJihB8iBACg/wTGAAAwSD3zzDM5+uijs+uuu+bnP/95Ro0aVemSAAAY5Cw/AACAKnT11Vfn1VdfzYoVK5IkP/vZz/LCCy8kSc4666zU1tbmiCOOyB/+8Iecf/75+dd
|
||
|
|
"text/plain": [
|
||
|
|
"<Figure size 1600x400 with 2 Axes>"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"metadata": {},
|
||
|
|
"output_type": "display_data"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"plt.figure(figsize=(16, 4))\n",
|
||
|
|
"plt.subplot(121)\n",
|
||
|
|
"plt.grid(True, which='both')\n",
|
||
|
|
"plt.axhline(y=0, color='k')\n",
|
||
|
|
"plt.plot(X1D[:, 0][y==0], np.zeros(4), \"bs\", markersize=10)\n",
|
||
|
|
"plt.plot(X1D[:, 0][y==1], np.zeros(5), \"g^\", markersize=10)\n",
|
||
|
|
"plt.gca().get_yaxis().set_ticks([])\n",
|
||
|
|
"plt.xlabel(r\"$x_1$\", fontsize=20)\n",
|
||
|
|
"plt.axis([-4.5, 4.5, -0.2, 0.2])\n",
|
||
|
|
"\n",
|
||
|
|
"plt.subplot(122)\n",
|
||
|
|
"plt.grid(True, which='both')\n",
|
||
|
|
"plt.axhline(y=0, color='k')\n",
|
||
|
|
"plt.axvline(x=0, color='k')\n",
|
||
|
|
"plt.plot(X2D[:, 0][y==0], X2D[:, 1][y==0], \"bs\", markersize=10)\n",
|
||
|
|
"plt.plot(X2D[:, 0][y==1], X2D[:, 1][y==1], \"g^\", markersize=10)\n",
|
||
|
|
"plt.xlabel(r\"$x_1$\", fontsize=20)\n",
|
||
|
|
"plt.ylabel(r\"$x_2$\", fontsize=20, rotation=0)\n",
|
||
|
|
"plt.gca().get_yaxis().set_ticks([0, 4, 8, 12, 16])\n",
|
||
|
|
"plt.plot([-4.5, 4.5], [6.5, 6.5], \"r--\", linewidth=3)\n",
|
||
|
|
"plt.axis([-4.5, 4.5, -1, 17])\n",
|
||
|
|
"\n",
|
||
|
|
"plt.subplots_adjust(right=1)"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"### 2D example"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 26,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:11.179245Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:11.178843Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:11.327960Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:11.327250Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA24AAAHZCAYAAAD+AWiDAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8WgzjOAAAACXBIWXMAAA9hAAAPYQGoP6dpAABMFElEQVR4nO3de3yT9d3/8XfaxrRFWqRlUO61KOchTGACt44b6Q8FlLk5B6iTOTzObc7TwAmIBebk4Zi62wNOh1JEvd3odN7b0EG1ICg3lJOHiYAUDEJRCoMi0BCa6/dHl66HJE2aw3Vdyev5eOShTa4r/SbfJuSd7/f7+ToMwzAEAAAAALCsNLMbAAAAAAAIjeAGAAAAABZHcAMAAAAAiyO4AQAAAIDFEdwAAAAAwOIIbgAAAABgcQQ3AAAAALA4ghsAAAAAWBzBDQAAAAAsjuAGAAAAABZn2+D25ZdfqqSkROPHj1fnzp3lcDhUWloa1rmlpaVyOBwBLwcOHIhvwwEAAAAgQhlmN6C9ampqNG/ePBUVFem8887TqlWrIr6PefPm6Zxzzml2XadOnWLTQAAAAACIEdsGt4KCAlVXV6tbt27auHGjhg0bFvF9XHrppTr//PPj0DoAAAAAiB3bTpV0uVzq1q1b1Pdz7Ngx1dfXx6BFAAAAABAftg1usVBcXKycnBxlZ2fr29/+tnbu3Gl2kwAAAACgFdtOlYxGdna2pk6d2hjcNm3apEceeUQXXnihNm/erMLCwoDneTweeTyexp99Pp8OHz6svLw8ORyORDUfAAAAgMUYhqFjx46pe/fuSkuLw/iYkQQqKysNScbixYvbfR9r1qwxHA6H8aMf/SjoMSUlJYYkLly4cOHChQsXLly4cAl42bt3b7szSSgpOeIWyMiRIzVixAiVl5cHPWbGjBm6++67G38+evSoioqKtGPHDnXu3DkRzUQLXq9XFRUVKi4ultPpNLs5KYk+MB99YD76wHz0gfnoA3Px/Jvv8OHD6tu3rzp27BiX+ye4NVFYWKjt27cHvd3lcsnlcrW6vnPnzsrLy4tn0xCE1+tVdna28vLyeJMyCX1gPvrAfPSB+egD89EH5uL5t454LaFK6eIkLVVVValLly5mNwMAAAAAmkn64FZdXa2PP/5YXq+38bqDBw+2Om758uXatGmTxo8fn8jmAQAAAECbbD1V8oknntCRI0e0f/9+SdJf/vIXffbZZ5Kkn/3sZ8rNzdWMGTO0ZMkS7d69W2effbYk6cILL9SQIUN0/vnnKzc3V5s3b9Zzzz2nwsJCzZw506yHAwAAAAAB2Tq4/eY3v9Gnn37a+PMrr7yiV155RZI0ZcoU5ebmBjzvqquu0t/+9jetWLFCJ06cUEFBgW6++WaVlJSoa9euCWk7AAAAAITL1sFtz549bR5TWlqq0tLSZtc98MADeuCBB+LTKAAAAACIsaRf4wYAAAAAdkdwAwAAAACLI7gBAAAAgMUR3AAAAADA4ghuAAAAAGBxBDcAAAAAsDiCGwAAAABYHMENAAAAACyO4AYAAAAAFkdwAwAAAACLI7gBAAAAgMUR3AAAAADA4ghuAAAAAGBxBDcAAAAAsDiCGwAAAABYHMENAAAAACyO4AYAAAAAFkdwAwAAAACLI7gBAAAAgMUR3AAAAADA4ghuAAAAAGBxBDcAAAAAsLgMsxsAAHbjdks1NcFvz8+XiooS1x4AAJD8CG4AEAG3W+rXT6qrC35MZqa0fTvhDQAAxA5TJQEgAjU1oUOb1HB7qBE5AACASBHcAAAAAMDiCG4AAAAAYHEENwAAAACwOIIbAAAAAFgcwQ0AAAAALI7gBgAAAAAWR3ADgAjk5zfs0xZKZmbDcQAAALHCBtwAEIGioobNtUPt05afz+bbAAAgtghuABChoiKCGQAASCymSgIAAACAxRHcAAAAAMDiCG4AAAAAYHEENwAAAACwOIIbAAAAAFgcwQ0AAAAALI7gBgAAAAAWR3ADAAAAAIsjuAEAAACAxRHcAAAAAMDiMsxuAADA+txuqaYm+O35+VJRUeLaAwBAqiG4AQBCcrulfv2kurrgx2RmStu3E94AAIgXpkoCAEKqqQkd2qSG20ONyAEAgOgQ3AAAAADA4ghuAAAAAGBxBDcAAAAAsDiCGwAAAABYHMENAAAAACyO4AYAAAAAFkdwAwCElJ/fsE9bKJmZDccBAID4YANuAEBIRUUNm2uH2qctP5/NtwEAiCeCGwCgTUVFBDMAAMzEVEkAAAAAsDiCGwAAAABYHMENAAAAACyONW4AAFtxuymUAgBIPQQ3AIBtuN1Sv35SXV3wYzIzG6pgEt4AAMmE4AYAYhTHLmpqQoc2qeH2mhr6CwCQXAhuAFIeozgAAMDqbFuc5Msvv1RJSYnGjx+vzp07y+FwqLS0NOzzjxw5oltuuUVdunRRhw4dVFxcrM2bN8evwQAsK5JRHAAAADPYNrjV1NRo3rx52rZtm84777yIzvX5fJowYYJeeukl3Xbbbfr1r3+tL774QqNHj9bOnTvj1GIAAAAAaB/bTpUsKChQdXW1unXrpo0bN2rYsGFhn1tWVqZ3331Xy5Yt08SJEyVJkydPVt++fVVSUqKXXnopXs0GAAAAgIjZdsTN5XKpW7du7Tq3rKxMXbt21ZVXXtl4XZcuXTR58mS99tpr8ng8sWomAAAAAETNtsEtGlu2bNHQoUOVltb84Q8fPlwnTpzQjh07TGoZAAAAALRm26mS0aiurtaoUaNaXV9QUCBJ2r9/vwYNGtTqdo/H02w0rra2VpLk9Xrl9Xrj1FqE4n/eef7Nkwx9cPq0JDnDOM4rKz7MZOiDcOXmSpmZGaqrcwQ9JjPTUG7u6YT2VSr1gVXRB+ajD8zF82++eD/3KRncTp48KZfL1er6zMzMxtsDmT9/vubOndvq+oqKCmVnZ8e2kYjIypUrzW5CyrNzH+zalStpdJvHrV37jqqrj8a9Pe1l5z6IxOOPZ6m29oygt+fknNKHH57Uhx8msFH/kip9YGX0gfnoA3Px/JvnxIkTcb3/lAxuWVlZAdex1f2rHnhWVlbA82bMmKG777678efa2loVFhaquLhYeXl58WksQvJ6vVq5cqUuueQSOZ1tj5gg9pKhD9xuadYso81RnO9855uW3MctGfogkdxu6dCh4Lfn5UW+Xx99YD76wHz0gbl4/s13KNQ/LjGQksHNX5GyJf913bt3D3iey+UKOFLndDp5gZiMPjCfnfugV6+GzbVD7dOWn+9QUZG1H5+d+yBR3G5p4MD4bbZOH5iPPjAffWAunn/zxPt5T8ngNnjwYK1Zs0Y+n69ZgZL169crOztbffv2NbF1AOLB7W4rmElDhyauPTBHJJutW3F0FQCQupI+uFVXV+vo0aPq1atXYwqeOHGiysrK9MorrzTu41ZTU6Nly5bp8ssvDziqBsC+3G6pX7/4jbIA0QjnSwX+LgEAtg5uTzzxhI4cOaL9+/dLkv7yl7/os88+kyT97Gc/U25urmbMmKElS5Zo9+7dOvvssyU1BLf//M//1PXXX6+PPvpI+fn5Wrhwoerr6wMWHwFgb4yywKr4UgEAEC5bB7ff/OY3+vTTTxt/fuWVV/TKK69IkqZMmaLc3NyA56Wnp2v58uWaPn26HnvsMZ08eVLDhg1TaWmp+vXrl5C2A0AojMKkBr5UAACEy9bBbc+ePW0eU1paqtLS0lbXn3XWWVq0aJEWLVoU+4YBJkm1D/vJ+ngZhQEAAC3ZOrgB+LdU+7AfyeO1m2QdhUnWoA0AQCIQ3IAkkawf9oOJ5PHCfKn2xQIAALGW1vYhAABExypBOz+/ISCGkpnZcBwAAFbCiBsAIGUUFYWz2TqjfgAA6yG4AYgLK61n8o+ytDVNj1GW1FBURDADANgPwQ1AzFltPZN/lOWDD6QjRwIf06lT/NsBtMSXCgCAcBHcAMScVQulTJxonTAJSEzdBACEj+AGICVYNUwGwihMamHqJgAgHAQ3IEmk2of9ZH68jMIAAICWCG6ATQUq/lFW9u81XJ06SQUFzW9Ppg/7yR5ukm0UJpmDNgAAiUB
|
||
|
|
"text/plain": [
|
||
|
|
"<Figure size 1000x500 with 1 Axes>"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"metadata": {},
|
||
|
|
"output_type": "display_data"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"from sklearn.datasets import make_moons\n",
|
||
|
|
"X, y = make_moons(n_samples=100, noise=0.15, random_state=42)\n",
|
||
|
|
"\n",
|
||
|
|
"def plot_dataset(X, y, axes):\n",
|
||
|
|
" plt.plot(X[:, 0][y==0], X[:, 1][y==0], \"bs\")\n",
|
||
|
|
" plt.plot(X[:, 0][y==1], X[:, 1][y==1], \"g^\")\n",
|
||
|
|
" plt.axis(axes)\n",
|
||
|
|
" plt.grid(True, which='both')\n",
|
||
|
|
" plt.xlabel(r\"$x_1$\", fontsize=20)\n",
|
||
|
|
" plt.ylabel(r\"$x_2$\", fontsize=20, rotation=0)\n",
|
||
|
|
"\n",
|
||
|
|
"plt.figure(figsize=(10, 5))\n",
|
||
|
|
"plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])\n",
|
||
|
|
"plt.show()"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 27,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:11.331500Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:11.330810Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:11.339010Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:11.338465Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"name": "stderr",
|
||
|
|
"output_type": "stream",
|
||
|
|
"text": [
|
||
|
|
"/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/sklearn/svm/_classes.py:32: FutureWarning: The default value of `dual` will change from `True` to `'auto'` in 1.5. Set the value of `dual` explicitly to suppress the warning.\n",
|
||
|
|
" warnings.warn(\n"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"from sklearn.datasets import make_moons\n",
|
||
|
|
"from sklearn.pipeline import Pipeline\n",
|
||
|
|
"from sklearn.preprocessing import PolynomialFeatures\n",
|
||
|
|
"\n",
|
||
|
|
"polynomial_svm_clf = Pipeline((\n",
|
||
|
|
" (\"poly_features\", PolynomialFeatures(degree=3)),\n",
|
||
|
|
" (\"scaler\", StandardScaler()),\n",
|
||
|
|
" (\"svm_clf\", LinearSVC(C=10, loss=\"hinge\", random_state=42))\n",
|
||
|
|
" ))\n",
|
||
|
|
"\n",
|
||
|
|
"polynomial_svm_clf.fit(X, y);"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 28,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:11.342040Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:11.341364Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:11.553315Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:11.552726Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA24AAAHZCAYAAAD+AWiDAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8WgzjOAAAACXBIWXMAAA9hAAAPYQGoP6dpAAB4g0lEQVR4nO3deXhb5Z02/vtIlmTZ1mLLsiw7dkwSBxdISVISljJALt6ZUjJ0Om3YBqZNKWW6TgsFSugSYGjzUspSyoQpDU0ClB9TMtC+HWiHMoWypSE7NGRxEive5E2WLHmRJeuc3x9Gxotky7aks92f68oF1vqVj7Xcep7n+wiSJEkgIiIiIiIixTLIXQARERERERFNj8GNiIiIiIhI4RjciIiIiIiIFI7BjYiIiIiISOEY3IiIiIiIiBSOwY2IiIiIiEjhGNyIiIiIiIgUjsGNiIiIiIhI4RjciIiIiIiIFI7BjYiIiIiISOFUG9z6+/uxceNGXHbZZSgrK4MgCNi2bVtG1922bRsEQUj5r6OjI7eFExERERERzVKB3AXMVU9PD+655x7U1tbi7LPPxmuvvTbr27jnnntw2mmnTTjN6XRmp0AiIiIiIqIsUW1w83q98Pv9qKysxJ49e7Bq1apZ38YnP/lJnHPOOTmojoiIiIiIKHtUO1XSYrGgsrJy3rcTiUSQSCSyUBEREREREVFuqDa4ZcOaNWtgt9tRVFSET33qU2hsbJS7JCIiIiIioilUO1VyPoqKirB+/fqx4LZ37148+OCDuOCCC7Bv3z7U1NSkvN7w8DCGh4fHfhZFEb29vXC5XBAEIV/lExERERGRwkiShEgkgqqqKhgMORgfkzRg9+7dEgBp69atc76NN954QxIEQfqXf/mXtJfZuHGjBID/+I//+I//+I//+I//+I//+C/lv5aWljlnkunocsQtlQsvvBDnnnsuXnnllbSX2bBhA2655Zaxn/v6+lBbW4v/3P4uioudaa831Bsd+39HmTEr9dIoURpBb3wvykwfg0HI3Z/zSG94ymnWMkvO7k9NRCmBzvh78JiWwSDw71sOPAby08MxEHp7pz3fXFaSp0pSEyURzfEjqDU1wCDoeiWIbHgM5DWX33/wZB/K0Y0KdCEudcK4Yhk6IyfRUwGU2xfluGLtifQN4BMXfR42my0nt8/gNk5NTQ2OHj2a9nyLxQKLZeqH9eJiJ0qKy6acPhgYGj2/yIpSlzbfyOUmSiOIxopgM5dmNbiNBPomnlDkRJGrMGu3ryWiNIJIrAh2szOn4ZnS4zGQnxaOgRDonvZ8c82CPFUyN6KUQFGsCHazQ7PhWel4DOQ1l99/vFCCA4Nw9EchLXBBsBUjKhUiagfszuIcV6xduVpCpc53lxw5efIk3G73vG4jGdYAMKypzOSwxqBGRFoyUzADALPLnodKiEgJAo1BiM2tQNmHn3/84Ub4jM2wolbGyigdzQc3v9+Pvr4+LF68GCaTCQDQ3d09JaC99NJL2Lt3L/71X/91TvfDwKY+U0bVwLBGROo144gZQxkRYVxgA7Bw6ADM4SD8SwzwVY2eb3XXosJZL2OFlI6qg9ujjz6KUCiE9vZ2AMDvfvc7tLaO/iF+4xvfgMPhwIYNG7B9+3Y0NTWhrq4OAHDBBRdgxYoVOOecc+BwOLBv3z788pe/RE1NDe68885Z1zHUG+V0SJXgqBoRqRFHy4hovsYHtjo0wVTgQ99pURxeXgSAgU0NVB3cfvKTn+DUqVNjPz///PN4/vnnAQDXX389HA5HyutdffXVePHFF/Hyyy9jcHAQXq8XX/rSl7Bx40Z4PJ5Z1+EoM8JWzNCmRBxVIyIlyySQAQxlRDR3ycBW0nkc5R4jzPYAQkXdaK0BRlxFDGwqourg5vP5ZrzMtm3bsG3btgmn3Xvvvbj33ntzUxTJjqNqRKQEDGVEJLdkaKtDE0pXOZEoGURnZQVaI90wNTSgmoFNVVQd3IiSRnrDMODDUU+GNSLKlVSBTIAIlABCbwACRttwM5ARkRJ4awtQChuK6qsR8TcCoRBMZzZwlE2FGNxItUYCfRCRAEqAojKLaltwE5EyZDpCBkwNZaKUAGKAuczGNuhEpFhiJATwu23V4iddUo1UUyBFaQTBmEwFEZEqzCeQERFpRSIUAABIrtxsDk25x+BGisb1akSUDgMZEVFmxP4g4PbCUMrQpmYMbqQ4DGtE+sZARkSURf2jI21GlxPoyPz1lZSHwY0UgWGNSPsYyIiI5CE4+JqqBQxuJBuGNSLtYOt7IiLlMfR3AQ4ngA65S6EsYHCjvGJYI1IXBjIiInUK7j8Kqc0PcziIQVsA7c1HAAA+I2BFrczV0VwwuFHOMawRKRNDGRGR9vSeCMFw5D0sQAuihiBCtj5YPKXw1QFWdy0Wcv821WJwo5wZH9gY1ojyL5NgxlBGRKQNvSdCQA1Q3fgHjAy2IrQkBounFP66UgY2jWBwo6xiWCPKj/GhTIAIlABCbwACDGOnM5QREWlfoDEIsbkVNUYfQjU1CNuOofCjbvjrihnYNIbBjeaNUyGJsm82o2WilABigLnMBoNgzHVpRESkEIHGIDzoAGoL4BRKAACmvzkHnV6JgU2DGNxoThjWiOaH0xiJiChbKuodGDl4BECp3KVQDjG40axwKiRR5mYKZwxmRESUDWKIG2vrAYMbzYhhjSg9hjMiIpJVfwBw2yDY+X6jdQxulBYDG9Go6cIZgxkREeVboDEIABCbW2EeasKwNIxAZRzA+Wg2tqII1fIWSDnB4EYTMKyRXjGcERGR0o0PbHVogqnAh25bJ9rqDEBFFTAA1C6+GAajIHOllAsMbgSAgY30gwGNiIjUZnJgM9sDCBV1o8XSg8RHalHkrkW5bQlOvhuTuVLKJQY3HWNYIy1jQCMiIrVLGdg8URyNNUGoKIe54UJUfND2X0xIcpZKecDgpkMMbKQl6QIawxkREWnBwnoToihAKWxIlBghVAJWsx2u+pVyl0Z5xuCmEwxrpHYcQSMiIgIQCiFxpk3uKkgGDG4ax8BGasRRNCIioqnESAjgxzndYnDTKAY2UgO1BjR/lxHBsDHt+aX2BLwViTxWREREWif2B4FiQHJxtE2vGNw0hGGNlEytIW0yf5cRa2+sQixuSHsZs0nEi1vaGd6IiCirDKUMbXrG4KYBDGykNFoJaakEw8ZpQxsAxOIGBMNGBjciIsqO/gAEhx0SBuWuhGTE4KZiDGykJEJvAAI+DDRaCGlERERyk0JdcpdACsHgpkIMbCSnyaNpAkSgBDCX2WAQ0q/7IiIiotkx9HdhuL8b5nA3BqUACm0VcpdEMmJwUxEGNsq3VFMeJ4+kiVICiOWrIiIiIu0LNAYhNrfC1rwXoicMS2URQp4ofFXNsLpr5S6PZMLgpgLJwMawRrmWSVAjIiKi3EgGtpLO4ygpOAbbIiv6lhhwoC4Kq7sWVgAVznq5yySZMLgpGAMb5RqDGhERkfwmBLYSP2zVI+grF3BwuYGBjcYwuCkMp0NSrjCkERERKU8ytNWhCeb6EEIeAYecUZgaGhjYaAIGN4VgYKNsY1DLjVJ7AmaTOOM+bqV2bgVARESZ8dYWoBQ2JEqMEApDMDVUM7DRFAxuMmNgo2yaHNYY1LLPW5HAi1vaEQyn76BZak9wDzciIpo1MRIC+HGQ0mBwkxHXsNF8MajJw1vBYEZERNkl9geBYkBy2eQuhRSKwU0GDGw0VwxqRERE2mUoZWij9Bjc8oiBjWaL69SIiIh0oD8gdwWkAgxuecDARrPBUTUiIiL9MPR3AQAEhx0SBmWuhpSMwS2H2HiEMsGgRkREpHMOJ4AOuasghWNwywEGNpoJwxoREZG+BRqDMPR3QWrzwxwOYtAWQMgShc8IWFErd3mkQAxuWcTARtNhWCMiIqJAYxAAIBw9iGq0wFI1jE740V4Xg+DxwOqu5R5ulBKDW5ZwHRulMj6sMagRERHpVzKwic2tqEMTTDYfumx9Y4GtiIGNZsDglgUjvWEU1VTKXQYpAEfViIiIaLzJgc1sD6Br8Dg6q+NIfKSWgY0yxuCWBdYyi9w
|
||
|
|
"text/plain": [
|
||
|
|
"<Figure size 1000x500 with 1 Axes>"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"metadata": {},
|
||
|
|
"output_type": "display_data"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"def plot_predictions(clf, axes):\n",
|
||
|
|
" x0s = np.linspace(axes[0], axes[1], 100)\n",
|
||
|
|
" x1s = np.linspace(axes[2], axes[3], 100)\n",
|
||
|
|
" x0, x1 = np.meshgrid(x0s, x1s)\n",
|
||
|
|
" X = np.c_[x0.ravel(), x1.ravel()]\n",
|
||
|
|
" y_pred = clf.predict(X).reshape(x0.shape)\n",
|
||
|
|
" y_decision = clf.decision_function(X).reshape(x0.shape)\n",
|
||
|
|
" plt.contourf(x0, x1, y_pred, cmap=plt.cm.brg, alpha=0.2)\n",
|
||
|
|
" plt.contourf(x0, x1, y_decision, cmap=plt.cm.brg, alpha=0.1)\n",
|
||
|
|
"\n",
|
||
|
|
"plt.figure(figsize=(10,5))\n",
|
||
|
|
"plot_predictions(polynomial_svm_clf, [-1.5, 2.5, -1, 1.5])\n",
|
||
|
|
"plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"Adding polynomial features leads to a combinatorial increase in the dimensionality of the feature space and thus is computationally costly."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "fragment"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"There are much better ways to perform non-linear classification with SVMs, where computational tricks can be exploited..."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "slide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"## Non-linear classification with kernels"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"### Similarity features and landmarks\n",
|
||
|
|
"\n",
|
||
|
|
"Compute new features based on proximity to landmarks.\n",
|
||
|
|
"\n",
|
||
|
|
"Consider landmarks $l^{(1)}$, $l^{(2)}$, $\\ldots$\n",
|
||
|
|
"\n",
|
||
|
|
"Then for each training instance $x$, compute features $f_j = \\text{sim}(x, l^{(j)})$, where $\\text{sim(.,.)}$ defines a *similarity* function.\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"### Similarity functions\n",
|
||
|
|
"\n",
|
||
|
|
"The Gaussian radial basis function (RBF) is a common *similarity function*:\n",
|
||
|
|
"\n",
|
||
|
|
"$$\n",
|
||
|
|
"\\phi_\\gamma(x,l) = exp( - \\gamma \\| x-l \\|^2),\n",
|
||
|
|
"$$\n",
|
||
|
|
"\n",
|
||
|
|
"where $\\gamma$ controls the width of the kernel.\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"### 1D example (from above)\n",
|
||
|
|
"\n",
|
||
|
|
"Consider landmarks at $x_1=-2$ and $x_1=1$."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 29,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:11.556833Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:11.556239Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:11.561660Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:11.561119Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"def gaussian_rbf(x, landmark, gamma):\n",
|
||
|
|
" return np.exp(-gamma * np.linalg.norm(x - landmark, axis=1)**2)\n",
|
||
|
|
"\n",
|
||
|
|
"gamma = 0.3\n",
|
||
|
|
"\n",
|
||
|
|
"x1s = np.linspace(-4.5, 4.5, 200).reshape(-1, 1)\n",
|
||
|
|
"x2s = gaussian_rbf(x1s, -2, gamma)\n",
|
||
|
|
"x3s = gaussian_rbf(x1s, 1, gamma)\n",
|
||
|
|
"\n",
|
||
|
|
"XK = np.c_[gaussian_rbf(X1D, -2, gamma), gaussian_rbf(X1D, 1, gamma)]\n",
|
||
|
|
"yk = np.array([0, 0, 1, 1, 1, 1, 1, 0, 0])"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 30,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:11.564642Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:11.563993Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:11.979858Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:11.979177Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAABdMAAAGFCAYAAAAW8GrFAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8WgzjOAAAACXBIWXMAAA9hAAAPYQGoP6dpAADgIUlEQVR4nOzdeZxN9R/H8dedxSwYS8jQCGXNGiF7NAgV2bNkyZo9EbJGKqVCKeuQpBItSj9ji+yyREL2ZabsDWP2e39/nGaYZl/PvXfez8fjPsw96/ucc2fc+7nf8/1abDabDRERERERERERERERSZKL2QFEREREREREREREROydiukiIiIiIiIiIiIiIilQMV1EREREREREREREJAUqpouIiIiIiIiIiIiIpEDFdBERERERERERERGRFKiYLiIiIiIiIiIiIiKSAhXTRURERERERERERERSoGK6iIiIiIiIiIiIiEgK3MwO4OisVitBQUHkzZsXi8VidhwRERERyQI2m41bt25RrFgxXFzUHiUn0ft9EREREeeWlvf6KqZnUFBQEH5+fmbHEBEREZFscOHCBR544AGzY0g20vt9ERERkZwhNe/1VUzPoLx58wLGyfbx8TE5DURFRbF+/XqaNWuGu7u72XHshs5L0nRukqZzkzidl6Tp3CRN5yZxOi9Js7dzExISgp+fX9x7P8k57O39fkaFhoZSrFgxAM6dO0f+/PnNDSRJsre/g5I4XSfHoOvkGHSdHIMzXqe0vNdXMT2DYm/19PHxsYs311FRUXh7e+Pj4+M0L+jMoPOSNJ2bpOncJE7nJWk6N0nTuUmczkvS7PXcqJuPnMfe3u9nlKura9zPznJMzspe/w5KfLpOjkHXyTHoOjkGZ75OqXmvrw4fRURERERERERERERSoGK6iIiIiIiIiIiIiEgKVEwXEREREREREREREUmBiukiIiIiIiIiIiIiIilQMV1EREREREREREREJAUqpouIiIiIiIiIiIiIpEDFdBERERERERERERGRFKiYLiIiIiIiIiIiIiKSAhXTRURERERERERERERSoGK6iIiIiIiIiIiIiEgKVEwXEREREREREREREUmBiukiIiIiIiIiIiIiIilQMV1EREREREREREREJAUqpouIiIiIiIiIiIiIpEDFdBERERERERERERGRFKiYLiIiIiIiIqbYcHoDFT+syIbTG8yOIiIiIpIiFdNFREREREQk29lsNsZtHMcfV/9g3MZx2Gw2syOJiIiIJMvuium3b99m0qRJtGjRgoIFC2KxWAgICEj1+jdv3qRfv34ULlyY3Llz88QTT7B///5El/3uu+949NFH8fT0pESJEkyaNIno6OhMOhIREREREcmI7PxsINlv/an17A3aC8DeoL2sP7Xe5EQiIiIiybO7YvrVq1eZOnUqf/zxB1WrVk3TularlVatWrFixQoGDx7M22+/zeXLl2ncuDF//vlnvGXXrVtHmzZtyJ8/P3PmzKFNmzZMmzaNIUOGZObhiIiIiIhIOmXXZwPJfjabjQmbJ+BqcQXA1eLKhM0T1DpdRERE7Jqb2QH+y9fXl+DgYIoWLcq+fft47LHHUr3uqlWr2LFjB1999RXt27cHoGPHjpQtW5ZJkyaxYsWKuGVHjRpFlSpVWL9+PW5uxmnw8fHhjTfeYNiwYZQvXz5zD0xERERERNIkuz4bSPa7t1U6QIwtJq51evOHm5uYTERERCRpdtcy3cPDg6JFi6Zr3VWrVnH//ffz3HPPxU0rXLgwHTt25NtvvyUiIgKAo0ePcvToUfr16xdXSAcYNGgQNpuNVatWZewgREREREQkw7Ljs4Fkv/+2So+l1ukiIiJi7+yuZXpGHDhwgEcffRQXl/jfEdSqVYv58+dz4sQJKleuzIEDBwCoWbNmvOWKFSvGAw88EDc/MREREfHeeIeEhAAQFRVFVFRUZh1KusVmsIcs9kTnBSKiIzh+/Ti/X/md4FvBFPIuRI8qPeLOyWubXiPaFs0DPg9QIl8JSuYvSYX7KuDu6m5ycvPodZM4nZe7QkJg2TIXrl2DSZOscedk7doYdu50oVEjG82aqSAAet0kReclafZ2buwlh6Reaj8b/Je9v9/PqHuPwaxjWn86fqv0WLGt03888SPNSjfL9lz2xt7+DkridJ0cg66TY9B1cgzOeJ3ScixOVUwPDg6mYcOGCab7+voCEBQUROXKlQkODo43/b/LBgUFJbmPGTNmMGXKlATT169fj7e3d3qjZ7rAwECzI9ilnHRerDYrv4b8ysFbBzl8+zCXwi8RQ0zc/HLe5Sh0sVDc8wX7FnAj+ka8beSy5OIh74eomLsi3Xy7YbFYsi2/PclJr5u0yGnn5dixAmzfXpxKla5Su/ZfAISEuDNyZEsAqlb9kVy5rAAEBFzku+8e5s8//yQ6+igAViuMHduAMmVu0LHjcXx8nOeNR1rktNdNaum8JM1ezs2dO3fMjiBplNrPBv/lKO/30ys8PDzu502bNuHp6Zmt+7fZbLxy4hVccMGKNcF8F1wY/u1wZpadmWPfe/6XvfwdlOTpOjkGXSfHoOvkGJzpOqXlvb5TFdPDwsLw8PBIMD32DWJYWFi8f5NaNrb1SWLGjh3LyJEj456HhITg5+dHs2bN8PHxyVD+zBAVFUVgYCD+/v64u+fcFsX/lVPOi81mi/vQYbPZGDJ3CJduXYqbn88jH5UKV+LB/A9S/r7ytKzXMu7cvFzvZa6EXeH8P+c5H3Kek9dP8k/EP/wR+gdFCxWlVatWcds58NcBqhSpgquLa4IMziSnvG7SKqecF6sV7m3MuHWrC99/78p995WiZUvjiymbDdautVK0qI0mTVrg5WWcm27d/HjwwRiaNClFy5YlAfjzTzh+3J0LFwrw+ecliP0vKCgIChcGJz6VQM553aSVzkvS7O3cJPf+UOxTaj8b/Je9v9/PqNDQ0LifmzRpQv78+bN1/+tPr+fkoZNJzrdi5WTYSdwruOf41un29ndQEqfr5Bh0nRyDrpNjcMbrlJb3+k5VTPfy8kq078PY1hdeXl7x/k1q2dj5ifHw8Ej0Tbm7u7tdvYDsLY+9cNbzcuGfC7y/6302nNnA/n7744rcL1R9gWth1/Av7U/tB2pTPG/xJFv4jK43Ot65sdqsnLh2gl0Xd/GAzwNx84JvBVMvoB4P+DzAgBoD6F29N4VzF876gzSRs75uMspZz4vNBm+8AQsWwIYN8PDDxvROneDmTWjXzgV397tV9i++iP3Jldg7w9q0caFDh/hfNvn5wVdfwV9/WciT5+55GzAA9u+HefPgnm59nZazvm4ySuclafZybuwhg6RNaj8b/JejvN9Pr3uPIbuPyWazMWXrFFwtrsTYYpJcztXiypStU2hZtqVap+M8rz1np+vkGHSdHIOuk2NwpuuUluNwqmK6r69vXBcu94qdVqxYsbjlYqf7+fklWLZWrVpZnFQkc/xx5Q/e2v4Wnx3+jGhrNABbzm6haemmAExvOj3d23axuFC+UHnKFyofb/rhy4fJmysvZ2+e5dWNrzLl5ym89NhLvFLvFYrkLpL+gxGxExYLbN8O587BkiUw/d9fo1q1jEd65c0L7dvHn3bnDvz2G1y9Cvf2NGCzGTlERCT9UvvZQLLP+lOJ95X+X7F9p68/tZ7mDzfPhmQiIiIiqeOS8iKOo1q1auzfvx+rNX7fe7t378bb25uyZcvGLQewb9++eMsFBQVx8eLFuPki9uqv23/R//v+VJpXiaWHlhJtjaZxycb88PwPNC7ZOEv33eyhZlwceZHFzyymhm8NwqLDeGfnO5T6oBSvbniVkAjdBi+O5eZNeO01uPdu/3HjYPlymDAha/ft7Q1nz8LOnVCmzN3p48fDSy8ZRXYREUmf1H42kOxhs9mYsHkCrpbUdRPoanFlwuYJ2GwayFtERETsh8MW04ODgzl27Fi80Vbbt2/P33//zerVq+OmXb16la+++oqnn3467nbNRx55hPLlyzN//nxiYu7eXjhv3jwsFgvt/9t0UMSOnLh2gjJzyjB//3ysNivPlnuW3S/
|
||
|
|
"text/plain": [
|
||
|
|
"<Figure size 1600x400 with 2 Axes>"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"metadata": {},
|
||
|
|
"output_type": "display_data"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"plt.figure(figsize=(16, 4))\n",
|
||
|
|
"plt.subplot(121)\n",
|
||
|
|
"plt.grid(True, which='both')\n",
|
||
|
|
"plt.axhline(y=0, color='k')\n",
|
||
|
|
"plt.scatter(x=[-2, 1], y=[0, 0], s=150, alpha=0.5, c=\"red\")\n",
|
||
|
|
"plt.plot(X1D[:, 0][yk==0], np.zeros(4), \"bs\", markersize=10)\n",
|
||
|
|
"plt.plot(X1D[:, 0][yk==1], np.zeros(5), \"g^\", markersize=10)\n",
|
||
|
|
"plt.plot(x1s, x2s, \"g--\")\n",
|
||
|
|
"plt.plot(x1s, x3s, \"b:\")\n",
|
||
|
|
"plt.gca().get_yaxis().set_ticks([0, 0.25, 0.5, 0.75, 1])\n",
|
||
|
|
"plt.xlabel(r\"$x_1$\", fontsize=20)\n",
|
||
|
|
"plt.ylabel(r\"Similarity\", fontsize=14)\n",
|
||
|
|
"plt.annotate(r'$\\mathbf{x}$',\n",
|
||
|
|
" xy=(X1D[3, 0], 0),\n",
|
||
|
|
" xytext=(-0.5, 0.20),\n",
|
||
|
|
" ha=\"center\",\n",
|
||
|
|
" arrowprops=dict(facecolor='black', shrink=0.1),\n",
|
||
|
|
" fontsize=18,\n",
|
||
|
|
" )\n",
|
||
|
|
"plt.text(-2, 0.9, \"$x_2$\", ha=\"center\", fontsize=20)\n",
|
||
|
|
"plt.text(1, 0.9, \"$x_3$\", ha=\"center\", fontsize=20)\n",
|
||
|
|
"plt.axis([-4.5, 4.5, -0.1, 1.1])\n",
|
||
|
|
"\n",
|
||
|
|
"plt.subplot(122)\n",
|
||
|
|
"plt.grid(True, which='both')\n",
|
||
|
|
"plt.axhline(y=0, color='k')\n",
|
||
|
|
"plt.axvline(x=0, color='k')\n",
|
||
|
|
"plt.plot(XK[:, 0][yk==0], XK[:, 1][yk==0], \"bs\", markersize=10)\n",
|
||
|
|
"plt.plot(XK[:, 0][yk==1], XK[:, 1][yk==1], \"g^\", markersize=10)\n",
|
||
|
|
"plt.xlabel(r\"$x_2$\", fontsize=20)\n",
|
||
|
|
"plt.ylabel(r\"$x_3$ \", fontsize=20, rotation=0)\n",
|
||
|
|
"plt.annotate(r'$\\phi\\left(\\mathbf{x}\\right)$',\n",
|
||
|
|
" xy=(XK[3, 0], XK[3, 1]),\n",
|
||
|
|
" xytext=(0.65, 0.50),\n",
|
||
|
|
" ha=\"center\",\n",
|
||
|
|
" arrowprops=dict(facecolor='black', shrink=0.1),\n",
|
||
|
|
" fontsize=18,\n",
|
||
|
|
" )\n",
|
||
|
|
"plt.plot([-0.1, 1.1], [0.57, -0.1], \"r--\", linewidth=3)\n",
|
||
|
|
"plt.axis([-0.1, 1.1, -0.1, 1.1])\n",
|
||
|
|
" \n",
|
||
|
|
"plt.subplots_adjust(right=1)"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"Training data *are* linearly separable in new feature space."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
},
|
||
|
|
"tags": [
|
||
|
|
"inclass_exercise"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"### Exercise: What are new feature values corresponding to the instance at $x_1=-1$ ($\\gamma=0.3$)?"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "fragment"
|
||
|
|
},
|
||
|
|
"tags": [
|
||
|
|
"solution",
|
||
|
|
"inclass_exercise"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"$x_2 = \\exp(-0.3 \\times (1)^2) = 0.74$\n",
|
||
|
|
"\n",
|
||
|
|
"$x_3 = \\exp(-0.3 \\times (2)^2) = 0.30$\n",
|
||
|
|
"\n",
|
||
|
|
"Let's check:"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 31,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:11.983440Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:11.982863Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:11.987586Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:11.987051Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"name": "stdout",
|
||
|
|
"output_type": "stream",
|
||
|
|
"text": [
|
||
|
|
"Phi(-1.0, -2) = [0.74081822]\n",
|
||
|
|
"Phi(-1.0, 1) = [0.30119421]\n"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"x1_example = X1D[3, 0]\n",
|
||
|
|
"for landmark in (-2, 1):\n",
|
||
|
|
" k = gaussian_rbf(np.array([[x1_example]]), np.array([[landmark]]), gamma)\n",
|
||
|
|
" print(\"Phi({}, {}) = {}\".format(x1_example, landmark, k))"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"### How set landmarks?"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "fragment"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"Common approach is to set a landmark at the location of each instance in the training data-set."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"### Kernel trick"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"For SVMs it is not necessary to actually compute new features for each kernel.\n",
|
||
|
|
"\n",
|
||
|
|
"Instead, can be done implicitly, providing *considerable* computational saving."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "fragment"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"SVM are well suited for small to medium sized data-sets, with complex structure."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "fragment"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"The kernel trick is based on Mercer's theorem."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"#### Mercer's theorem\n",
|
||
|
|
"\n",
|
||
|
|
"For (feature) mapping function $\\phi(x)$, the inner product of two transformed vectors can be computed implicitly by the evaluation of the kernel function $K$ by\n",
|
||
|
|
"$\n",
|
||
|
|
"K(x,z) = \\langle \\phi(x), \\phi(z) \\rangle .\n",
|
||
|
|
"$\n",
|
||
|
|
"\n",
|
||
|
|
"There is therefore no need to explicitly compute $\\phi(x)$.\n",
|
||
|
|
"\n",
|
||
|
|
"Moreoever, it is not necessary to even know the explicit form of the (feature) mapping function $\\phi(x)$.\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "fragment"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"(We will not cover the kernel trick and Mercer's theorem in any further detail.)"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"### Common kernels\n",
|
||
|
|
"\n",
|
||
|
|
"Some common similarity function, or *kernel*, include the following.\n",
|
||
|
|
"\n",
|
||
|
|
"1. Gaussian radial basis function (RBF):\n",
|
||
|
|
"$\n",
|
||
|
|
"K_\\gamma(x,l) = exp( - \\gamma \\| x-l \\|^2),\n",
|
||
|
|
"$\n",
|
||
|
|
"where $\\gamma$ controls the width of the kernel.\n",
|
||
|
|
"\n",
|
||
|
|
"2. Polynomial kernel:\n",
|
||
|
|
"$\n",
|
||
|
|
"K_{c, d}(x,l) = (x^{\\rm T}l + c)^d,\n",
|
||
|
|
"$\n",
|
||
|
|
"for constant offset $c$ and degree $d$.\n",
|
||
|
|
"\n",
|
||
|
|
"3. Linear kernel, i.e. linear SVM:\n",
|
||
|
|
"$\n",
|
||
|
|
"K(x,l) = x^{\\rm T}l.\n",
|
||
|
|
"$\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"### 2D example (from above)"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 32,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:11.991176Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:11.990615Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:12.002315Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:12.001682Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"from sklearn.svm import SVC\n",
|
||
|
|
"\n",
|
||
|
|
"gamma1, gamma2 = 0.1, 5\n",
|
||
|
|
"C1, C2 = 0.001, 1000\n",
|
||
|
|
"hyperparams = (gamma1, C1), (gamma1, C2), (gamma2, C1), (gamma2, C2)\n",
|
||
|
|
"\n",
|
||
|
|
"svm_clfs = []\n",
|
||
|
|
"for gamma, C in hyperparams:\n",
|
||
|
|
" rbf_kernel_svm_clf = Pipeline((\n",
|
||
|
|
" (\"scaler\", StandardScaler()),\n",
|
||
|
|
" (\"svm_clf\", SVC(kernel=\"rbf\", gamma=gamma, C=C))\n",
|
||
|
|
" ))\n",
|
||
|
|
" rbf_kernel_svm_clf.fit(X, y)\n",
|
||
|
|
" svm_clfs.append(rbf_kernel_svm_clf)"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 33,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:12.005352Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:12.004809Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:13.099222Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:13.098533Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAABT8AAALTCAYAAADQEXZNAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8WgzjOAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOzdeXhb5Z02/vtIlmzLsi1vkhUvMSEmAZKGLQmBFOgwb6ctw7QvQ2k73ZgO7bR0lk4HGGinQ0uZ0kLpQhnoQCAJtH3bFwrtr0DnbYchNCFp9kDYgpPYseNV1mJ5l6xzfn8ox5Zk7ZbOpvtzXb4gsiw/Pl7O99zneZ6vIEmSBCIiIiIiIiIiIiKDMak9ACIiIiIiIiIiIqJiYPhJREREREREREREhsTwk4iIiIiIiIiIiAyJ4ScREREREREREREZEsNPIiIiIiIiIiIiMiSGn0RERERERERERGRIDD+JiIiIiIiIiIjIkBh+EhERERERERERkSEx/CQiIiIiIiIiIiJDYvhJREREREREREREhsTwk4iIiIiIiIiIiAyJ4ScRGc5TTz2Fq666CnV1daiqqsK6detw7733IhwO5/xax44dw49+9CPceOONWLt2LcrKyiAIAu6+++4ijHzBoUOHcNttt+Hiiy9Gc3MzrFYr6uvrcdlll+HrX/86+vr6ivr5s1HI47zU18z149T6vhIREVFpYn1aXEs9JkrVoEv9OCLKk0REZCD/+I//KAGQysrKpPe+973SddddJzkcDgmAtHnzZmlqaiqv10t8++Y3v1mU8Xu9XukTn/iEJAiCBEBqb2+X3vve90of/ehHpSuuuEIqLy+XAEgVFRXS//t//68oY8hGoY/zUl4zn49T+vtKREREpYv1afEt5ZgoWYMu5eOIKH8MP4nIMJ599lkJgGS326WDBw/OP+7xeKS1a9dKAKR//ud/zuk1H330UemWW26RfvrTn0pvvfWW9MlPfrJoxeXQ0ND8OFetWiX913/9lySKYtxzgsGg9I1vfEMqLy+X/vCHPxR8DNkoxnHO9zXz/Tglv69ERERUulifKiPfY6J0DVqMnwciyozhJxEZxvr16yUA0t13373ofTt37pQASOXl5VIgEMj7c3z6058uSnEZDoelDRs2SACkd7/73RnHuHv3bml8fLygY8hWMY5zvq9ZqLEU6/tKREREpY31qTqyPSZK16BK/DwQ0WLc85OIstLb2wuTyYS2tjZs2bIl5fOee+45CIKACy+8EHNzc4qNr7+/H/v37wcA/NVf/dWi92/evBltbW2YnZ3FCy+8oNi4svXNb34T+/btQ1NTE37xi1+gtrY27fM3bdoEu92u0OgWFOM45/uaev+eExER0dKwPi0uvdSn+VK6BtX7zwORnjH8JKKsjI2N4X3vex9Onz6Nz3/+8/B4PIueMzk5iS9+8YswmUx45JFHUFZWptj4Dh8+DACor6/HWWedlfQ5l1xySdxztcLn8+F73/seAOA73/kO3G63yiNKrRjHOd/X1PP3nIiIiJaO9Wnx6Kk+zZfSNaiefx6I9I7hJxFlZe3atXjhhRfwgQ98AJFIBHv37l30nH/7t39Db28v/u7v/g7r169P+jo33ngjBEHI+W3Hjh1px9fd3Q0AaG9vT/mctra2uOdqxeOPP46JiQk4nU588pOfLMhr6uk45/uaev6eExER0dKxPi0ePdWn+VK6BtXzzwOR3il324uIDGHz5s144YUX8MYbb+DP//zP5x8/fPgwfvjDH6K1tRV333132o/PR3Nzc9r3j4+PAwCqqqpSPkdehhMMBvMaQ7H89re/BQB88IMfLNhsBD0d53xfU8/fcyIiIioc1qeFp6f6NF9K16B6/nkg0juGn0SUk/POOw8A8MYbb8w/JooiPve5zyESieBHP/oRqqurU378TTfdhJtuuqno49STQ4cOAQA2btxYsNfkcSYiIqJSwfq08FifEpGRcNk7EeUkWXH54IMP4sCBA/jQhz6ED33oQ6qMSy5oJycnUz5nYmICAFBTU6PImLIRCoUQCAQAAC0tLeoOJgvFOM75vqZev+dERERUWKxPC0tv9Wm+lK5B9frzQGQEnPlJRDlZsWIFKioq8NZbb0EURQwODuJf//VfUV1djR/96EcZP37Lli3YtWtXzp/39ttvx+rVq1O+v6OjAwDQ19eX8jny++TnaoEgCPP/L0lSwV5XT8c539fU6/eciIiICov1aWHprT7Nl9I1qF5/HoiMgOEnEeXEbDbjnHPOwWuvvYbu7m7ceuutGB8fxwMPPIDW1taMH79r1y5s374958974403pi16LrzwQgCA1+tFd3d30g6KBw4cAABcdNFFOX/+YrFYLGhra0NfXx/efvttvP/97y/I6+rpOOf7mnr9nhMREVFhsT4tLL3Vp/lSugbV688DkRFw2TsR5UxeWvStb30Lzz77LNavX48vfvGLWX3stm3bIElSzm9XXXVV2tdtbW2d7+D5s5/9bNH7d+3ahb6+PpSXl+MDH/hAbl9wkf3FX/wFAOCxxx5DOBxO+1y/34+BgYGMr6mn45zva+r5e05ERESFxfq0sPRUn+ZL6RpUzz8PRHrH8JOIciYXl48//jjKysrwyCOPwGRS/8/JV77yFQDAt7/97flN2oHo3dWbb74ZAPB3f/d3qK2tjfu4O+64A6tXr8Ydd9xRsLFs27YNgiBktWTl9ttvR11dHd544w189KMfxejo6KLneDwe3HvvvVi1ahVCoVDBxpmPYhznfF8z348jIiIiY2F9mpmR69N8KV2DsnYlUolERJSjX/7ylxIACYB0yy23qD2cOP/wD/8gAZAsFov0vve9T/rLv/xLyeFwSACkyy+/XJqamlr0MZ/+9KclANKnP/3pRe87ePCgtHHjxvm3xsZGCYDU2toa9/jAwEDcxz322GMSAGnlypVZjXv37t2S2+2WAEgVFRXSVVddJf3VX/2V9KEPfUh617veJQmCIAGQmpqa8jouhVbo45zva+b7cfl+X4mIiEibWJ+Wdn26lNpOyRp0KR9HRPlj+ElEOTty5IgEQHK5XNLk5KTaw1nkF7/4hXTFFVdINTU1UmVlpbRmzRrp29/+tjQ7O5v0+emKy5deemm+kE731t3dHfdxN998swRAuueee7Iet8/nk+655x7p8ssvl+rq6iSz2SxVVVVJnZ2d0l/+5V9KDz30kDQ8PJzLoSiqQh7nfF8z34/L9/tKRERE2sT6tLTr06XWdkrVoEv9OCLKjyBJBWzfRkQl4emnn8aHP/xhXH/99XjqqafUHo4mdXZ2Ynp6Gl1dXaisrFR7OERERESGxvo0M9anRFSq2O2diHK2f/9+AMAll1yi8ki0qaenB8ePH8ejjz7KwpKIiIhIAaxP02N9SkSlTP0doPM0MTGBO++8E+973/tQX18PQRCwbdu2rD5W3ug52dvQ0FBxB05kAAcOHACA+W6FFK+jowOSJOGmm25SeyhERFRErEeJtIP1aXqsT4molOl25ufo6CjuuusutLe3Y926ddixY0fOr3HXXXfhrLPOinvM4XAUZoBEBiVJEg4ePAhBEHDxxRerPRwiIiLVsB4l0gbWp0RElI5uw0+3243BwUE0NzfjwIEDed3he//7389lEUQ5EgQBgUBA7WEQERGpjvUokTawPiUionR0u+y9vLwczc3NS36d8fFxRCKRAoyIiIiIiEoJ61EiIiIi7dNt+FkI73nPe1BTUwObzYa/+Iu/QFdXl9pDIiIiIqISwnqUiIiIqLh0u+x9KWw2G2688cb5YvPgwYP43ve+h8suuwyHDh1CW1tb0o+bnZ3F7Ozs/L9FUYTP50NDQwMEQVBq+EREREQFI0kSxsfHsWzZMphMJX1fXFH51qMAa1IiIiIylqLXo5IB7N+/XwIgbd26Ne/X2LlzpyQIgvS3f/u3KZ9z5513SgD4xje+8Y1vfOMb3wz31tfXl3cdRcrVo5LEmpRvfOMb3/jGN74Z861Y9WhJzvxMZvPmzdi4cSP++7//O+Vz7rjjDnz5y1+e//fY2Bja29ux7Yf7UFXpQGV9hRJ
|
||
|
|
"text/plain": [
|
||
|
|
"<Figure size 1600x800 with 4 Axes>"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"metadata": {},
|
||
|
|
"output_type": "display_data"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"plt.figure(figsize=(16, 8))\n",
|
||
|
|
"\n",
|
||
|
|
"for i, svm_clf in enumerate(svm_clfs):\n",
|
||
|
|
" plt.subplot(221 + i)\n",
|
||
|
|
" plot_predictions(svm_clf, [-1.5, 2.5, -1, 1.5])\n",
|
||
|
|
" plot_dataset(X, y, [-1.5, 2.5, -1, 1.5])\n",
|
||
|
|
" gamma, C = hyperparams[i]\n",
|
||
|
|
" plt.title(r\"$\\gamma = {}, C = {}$\".format(gamma, C), fontsize=16)"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"**Exercises:** *You can now complete Exercise 2 in the exercises associated with this lecture.*"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "slide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"## Regression"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "fragment"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"Can also use SVMs to perform regression, in addition to classification.\n",
|
||
|
|
"\n",
|
||
|
|
"General idea is to reverse the objective: rather than fitting the largest possible margin between two classes, SVM regression fits as many instances as possible within the margin. \n",
|
||
|
|
"\n",
|
||
|
|
"The width of the margin is controlled by the hyperparameter $\\epsilon$."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"### Linear regression"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 34,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:13.103087Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:13.102462Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:13.107290Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:13.106674Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"np.random.seed(42)\n",
|
||
|
|
"m = 50\n",
|
||
|
|
"X = 2 * np.random.rand(m, 1)\n",
|
||
|
|
"y = (4 + 3 * X + np.random.randn(m, 1)).ravel()"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 35,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:13.110249Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:13.109685Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:13.118040Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:13.117503Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"name": "stderr",
|
||
|
|
"output_type": "stream",
|
||
|
|
"text": [
|
||
|
|
"/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/sklearn/svm/_classes.py:32: FutureWarning: The default value of `dual` will change from `True` to `'auto'` in 1.5. Set the value of `dual` explicitly to suppress the warning.\n",
|
||
|
|
" warnings.warn(\n",
|
||
|
|
"/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/site-packages/sklearn/svm/_classes.py:32: FutureWarning: The default value of `dual` will change from `True` to `'auto'` in 1.5. Set the value of `dual` explicitly to suppress the warning.\n",
|
||
|
|
" warnings.warn(\n"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"from sklearn.svm import LinearSVR\n",
|
||
|
|
"\n",
|
||
|
|
"svm_reg1 = LinearSVR(epsilon=1.5, random_state=42)\n",
|
||
|
|
"svm_reg2 = LinearSVR(epsilon=0.5, random_state=42)\n",
|
||
|
|
"svm_reg1.fit(X, y)\n",
|
||
|
|
"svm_reg2.fit(X, y)\n",
|
||
|
|
"\n",
|
||
|
|
"def find_support_vectors(svm_reg, X, y):\n",
|
||
|
|
" y_pred = svm_reg.predict(X)\n",
|
||
|
|
" off_margin = (np.abs(y - y_pred) >= svm_reg.epsilon)\n",
|
||
|
|
" return np.argwhere(off_margin)\n",
|
||
|
|
"\n",
|
||
|
|
"svm_reg1.support_ = find_support_vectors(svm_reg1, X, y)\n",
|
||
|
|
"svm_reg2.support_ = find_support_vectors(svm_reg2, X, y)\n",
|
||
|
|
"\n",
|
||
|
|
"eps_x1 = 1\n",
|
||
|
|
"eps_y_pred = svm_reg1.predict([[eps_x1]])"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 36,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:13.120834Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:13.120604Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:13.125151Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:13.124515Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"text/plain": [
|
||
|
|
"array([6.52640746])"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"execution_count": 36,
|
||
|
|
"metadata": {},
|
||
|
|
"output_type": "execute_result"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"eps_y_pred"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 37,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:13.128852Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:13.128404Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:13.611003Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:13.610305Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAABSsAAAHpCAYAAABqXWmJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8WgzjOAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOzde3zO9f/H8ce1AzannHKawyZnC0kIhfoWKTQ6UKhElFPOdj4jxFCpKCmk0EE6/KKUkkNJKMcNMYcct7HZ4drn98fFxcUws+1zbZ732203u96fz+e6Xuta22uvz/v9elsMwzAQERERERERERERMZmL2QGIiIiIiIiIiIiIgIqVIiIiIiIiIiIi4iRUrBQRERERERERERGnoGKliIiIiIiIiIiIOAUVK0VERERERERERMQpqFgpIiIiIiIiIiIiTkHFShEREREREREREXEKKlaKiIiIiIiIiIiIU1CxUkRERERERERERJyCipUiIiIiIiIiIiLiFFSsFBHJZcnJyXzzzTdERkbi5+dHjRo1sFgsWCwWQkNDb/r5582bZ3++a32sXLny5r8YERERESnQkpKSCA0NxdfXlxIlSlC6dGmaN2/O1KlTSUtLy/HzKicVkbziZnYAIiKFzYYNG3jkkUfy/HVcXFyoUKHCVY8XLVo0z2MQEREREee1f/9+2rVrx759+wDw9PQkNTWV33//nd9//50FCxawatUqypQpk+PXUE4qIrlNxUoRkTxQpkwZ7rrrLvvHq6++ypEjR3L1NapVq2ZPPEVERERELpWRkcFjjz3Gvn37qFy5MvPnz+fBBx8kMzOTTz/9lP79+/Pnn3/y7LPPsmLFihy/jnJSEcltKlaKiOSytm3bcvLkSYexcePGmRSNiIiIiNyKPvjgA7Zu3QrA0qVLadWqFWCbCfnUU0+RmZlJr169+Prrr1m1ahUPPPCAmeGKiNipZ6WIFBinTp1ixowZtG3blmrVqlG0aFHKly+Pr68vzzzzDF9//bXZIQLg6upqdggiIiIikkcKSk76wQcfANC+fXt7ofJSTz/9NN7e3gDMnz8/X2MTEbkWzawUkQLht99+o1u3bvz3338AuLu7U7JkSRISEjhx4gTbtm2jWbNm+dIrUkRERERuTQUlJ01OTubXX38FoFOnTlmeY7FY6NixI2+99Rb/93//l5/hiYhck2ZWiojTO3v2rD0p7NGjB3/88QdpaWmcOHGC9PR04uPj+eyzz+jWrZvZoearY8eO0axZM0qUKIGHhwc+Pj48++yzrF692uzQRERERAqdgpSTbt++nczMTAAaNWp01fMuHDty5MgVbYyySzmpiOQ2FStFxOmtWrWK//77jxo1avDJJ59w1113ORyvUqUK3bp1w8fH55rPM2/ePCwWS44/nC3hSk5OZtOmTRQpUoTMzEz27t3LggULaN++PS+88AIZGRlmhygiIiJSaBSknPTQoUP2z6tWrXrV8y49duk1N0I5qYjkNi0DFxGndyHBOXToEAsXLuTJJ5/E3d39hp/Hw8ODihUr5jiOIkWK5Pja3FSlShVCQkLw8/Ojbt26FC1aFKvVyvr16wkJCWHlypW8//77FC9enJkzZ5odroiIiEihUJBy0qSkJPvnnp6eVz3v0mOXXpMdyklFJK9YDMMwzA5CRORaUlNTeeCBB+x9d1xdXSlTpgyurq7MnTuXzp07mxzh9dWsWZP9+/cTEhJCaGhonr1OZmYmfn5+fPHFF7i4uLBjxw5q166dZ68nIiIicqsoSDnpwoULeeaZZwDYvXs3d9xxR5bnff/99zz00EMArF27NsuNeHJCOamI3AwtAxcRp1e0aFEWLlxobw5utVo5fvw4R48evWridatycXFhypQpgC1JXL58uckRiYiIiBQOBSknLVmypP3z5OTkq5536bFLr7lZyklF5GaoWCkiTm/y5MnUrVuXc+fO8d133/Hff/9hGAaGYVC3bl2zw3M6d9xxB+XLlwcgLi7O5GhERERECoeClJNWqVLF/nl8fPxVz7v02KXX5AblpCKSU+pZKSJObc6cOYwZM4aOHTuyYsUKXFxyfo9l8eLFDBs2LMfXL1u2jHvvvTfH14uIiIhIwVTQctL69evj4uJCZmYm27Zts88Gvdy2bdsAqFSpEmXLls1xTCIiuUnFShFxalOnTgXglVdeuamkECAlJYWjR4/m+Pq0tLSbev38Ehsby/HjxwHw9vY2ORoRERGRgq+g5aSenp60bt2aNWvW8O233zJ69OgrzjEMg++++w7A3rcyNyknFZGc0jJwEXFqu3btAuC///676ed67rnn7Et1cvLRrl27m47hZl1vTzTDMOzJqIuLC48++mh+hCUiIiJSqBXEnLRv374A/Pjjj6xfv/6K459++ql9eXafPn1u6GtQTioieUnFShFxahd654wdO5YFCxbYm4BnZmZy5MgRFi9ezKBBg8wMMUunTp3i+PHj9o/MzEzA1sT80vEzZ85cce28efOwWCxYLBZWr17tcGz//v3cc889vP3228TFxdkTxczMTNatW0enTp347LPPAHjppZecrn+SiIiISEFUEHPSvn374uvri2EYdO/enVWrVgG2mD/99FP69+8PQKdOnXjggQeuuF45qYiYRcvARcSpRUdH07dvX44fP86zzz4LwG233caZM2fIyMgAcMo7tU2bNmX//v1XjE+ePJnJkyfbH/ft25d58+bd0HNv3LiRjRs3ArZdKUuWLElSUhKpqan2c55//nlmzJiRs+BFRERExEFBzEnd3Nz48ssvad++Pfv27ePBBx/E09OTzMxMzp07B9hy1gULFuTo+ZWTikheUbFSRJxa7969qV27Nm+++SZr167l0KFDnD17lgoVKlCjRg1at27N008/bXaY+aZixYrMnDmT3377jc2bN3Ps2DFOnTpFsWLF8Pb25t577+WFF16gdevWZocqIiIiUmgU1Jy0Zs2abNmyhSlTprBs2TL27t2Lu7s7DRs2pGfPngwZMoQiRYrc8PMqJxWRvGQxrtdsQkRERERERERERCQfqGeliIiIiIiIiIiIOIV8K1aeOXOGkJAQOnbsSNmyZbFYLFn2aduwYQMvv/wyzZo1w93dHYvFkl8hioiIiEghp5xURERExLnlW7Hy+PHjhIeHs337dho3bnzV877++mvmzJmDxWLBx8cnv8ITERERkVuAclIRERER55ZvxcrKlStz+PBh9u/f77AT7uUGDRpEQkICv//+O//73//yKzwRERERuQUoJxURERFxbvm2G3jRokWpVKnSdc+rWLFiPkQjIiIiIrci5aQiIiIizk0b7IiIiIiIiIiIiIhTyLeZlXktNTWV1NRU++PMzExOnjxJuXLl1BBdRERECiTDMEhKSqJKlSq4uOgec0GgnFREREQKEzPy0UJTrJwwYQJhYWFmhyEiIiKS6w4cOICXl5fZYUg2KCcVERGRwig/89FCU6wcP348I0aMsD9OSEigevXqHDhwgFKlSpkYmYiIiEjWTpw4QUxMDKmpqUyaNMk+vn37durXr09iYiLVqlWjZMmSJkYpN0I5qYiIiBQmZuSjhaZYWbRoUYoWLXrFeKlSpZQYioiIiFM5deoUr7/+OtOnT+fMmTO4uroybtw4atSoAUCLFi0cztfy4YJDOamIiIgURvmZjxaaYqWIiIiIs0tMTCQmJoapU6eSkJAAQNOmTQkPD6d69eomRyciIiIiYj4VK0VERETywU8//YSfnx8nT54EoFGjRoSFhfH4449r5qSIiIiIyHn5WqycNWsWp0+f5tChQwAsX76cgwcPAjBkyBBKly7N/v37+fDDDwH4/fffAYiMjASgRo0a9O7dOz9DFhEREckVvr6+ZGRkULduXUJDQ3nyySe1w7dJlJOKiIiIOC+LYRhGfr1YzZo12b9/f5bH9u7dS82aNVm9ejXt27fP8pz777+f1atXZ+u1EhMTKV26NAkJCeoPJCIiIvkqNTWVd999lzVr1vDxxx/bZ05u2bKFBg0a4OaWvfvFymfyhnJSERERkewxI5fJ12JlflJiKCIiIvktLS2N999
|
||
|
|
"text/plain": [
|
||
|
|
"<Figure size 1600x500 with 2 Axes>"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"metadata": {},
|
||
|
|
"output_type": "display_data"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"def plot_svm_regression(svm_reg, X, y, axes):\n",
|
||
|
|
" x1s = np.linspace(axes[0], axes[1], 100).reshape(100, 1)\n",
|
||
|
|
" y_pred = svm_reg.predict(x1s)\n",
|
||
|
|
" plt.plot(x1s, y_pred, \"k-\", linewidth=2, label=r\"$\\hat{y}$\")\n",
|
||
|
|
" plt.plot(x1s, y_pred + svm_reg.epsilon, \"k--\")\n",
|
||
|
|
" plt.plot(x1s, y_pred - svm_reg.epsilon, \"k--\")\n",
|
||
|
|
" plt.scatter(X[svm_reg.support_], y[svm_reg.support_], s=180, facecolors='#FFAAAA')\n",
|
||
|
|
" plt.plot(X, y, \"bo\")\n",
|
||
|
|
" plt.xlabel(r\"$x_1$\", fontsize=18)\n",
|
||
|
|
" plt.legend(loc=\"upper left\", fontsize=18)\n",
|
||
|
|
" plt.axis(axes)\n",
|
||
|
|
"\n",
|
||
|
|
"plt.figure(figsize=(16, 5))\n",
|
||
|
|
"plt.subplot(121)\n",
|
||
|
|
"plot_svm_regression(svm_reg1, X, y, [0, 2, 3, 11])\n",
|
||
|
|
"plt.title(r\"$\\epsilon = {}$\".format(svm_reg1.epsilon), fontsize=18)\n",
|
||
|
|
"plt.ylabel(r\"$y$\", fontsize=18, rotation=0)\n",
|
||
|
|
"#plt.plot([eps_x1, eps_x1], [eps_y_pred, eps_y_pred - svm_reg1.epsilon], \"k-\", linewidth=2)\n",
|
||
|
|
"plt.annotate(\n",
|
||
|
|
" '', xy=(eps_x1, eps_y_pred), xycoords='data',\n",
|
||
|
|
" xytext=(eps_x1, eps_y_pred - svm_reg1.epsilon),\n",
|
||
|
|
" textcoords='data', arrowprops={'arrowstyle': '<->', 'linewidth': 1.5}\n",
|
||
|
|
" )\n",
|
||
|
|
"plt.text(0.91, 5.6, r\"$\\epsilon$\", fontsize=20)\n",
|
||
|
|
"plt.subplot(122)\n",
|
||
|
|
"plot_svm_regression(svm_reg2, X, y, [0, 2, 3, 11])\n",
|
||
|
|
"plt.title(r\"$\\epsilon = {}$\".format(svm_reg2.epsilon), fontsize=18);"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {
|
||
|
|
"slideshow": {
|
||
|
|
"slide_type": "subslide"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"source": [
|
||
|
|
"### Non-linear regression"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 38,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:13.614588Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:13.614131Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:13.619620Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:13.618994Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"np.random.seed(42)\n",
|
||
|
|
"m = 100\n",
|
||
|
|
"X = 2 * np.random.rand(m, 1) - 1\n",
|
||
|
|
"y = (0.2 + 0.1 * X + 0.5 * X**2 + np.random.randn(m, 1)/10).ravel()"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 39,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:13.622590Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:13.622133Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:13.634075Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:13.633427Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"from sklearn.svm import SVR\n",
|
||
|
|
"\n",
|
||
|
|
"svm_poly_reg1 = SVR(kernel=\"poly\", degree=2, C=100, epsilon=0.1)\n",
|
||
|
|
"svm_poly_reg2 = SVR(kernel=\"poly\", degree=2, C=0.01, epsilon=0.1)\n",
|
||
|
|
"svm_poly_reg1.fit(X, y);\n",
|
||
|
|
"svm_poly_reg2.fit(X, y);"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 40,
|
||
|
|
"metadata": {
|
||
|
|
"execution": {
|
||
|
|
"iopub.execute_input": "2024-01-10T00:20:13.637220Z",
|
||
|
|
"iopub.status.busy": "2024-01-10T00:20:13.636764Z",
|
||
|
|
"iopub.status.idle": "2024-01-10T00:20:14.087863Z",
|
||
|
|
"shell.execute_reply": "2024-01-10T00:20:14.087257Z"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAABTAAAAHrCAYAAAD427KdAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8WgzjOAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOzdd3zM5wMH8M83l8gSQQRBbGrTojEapVrbT5FW7b1K7VGzFFVbtFo1KmZqxa49KhSlRO0ZRMSIkYRExt3z++PrTpK7JJfLzeTzfr3yivuu57m7xH3yfJ8hCSEEiIiIiIiIiIiIiKyQnaUrQERERERERERERJQWNmASERERERERERGR1WIDJhEREREREREREVktNmASERERERERERGR1WIDJhEREREREREREVktNmASERERERERERGR1WIDJhEREREREREREVktNmASERERERERERGR1WIDJhEREREREREREVktNmASERERERERERGR1WIDJhEREREREREREVktNmBSjrBy5UpIkgRvb29LV4WIiIiIcghmUCIiIuNgAyblCP/++y8A4IMPPrBwTSgjz549w8qVK9GlSxdUqlQJrq6ucHR0RLFixfD5559j69atZqnHf//9h++++w716tVDiRIl4OzsDDc3N7z33nvo3r07du3aBaVSaZa6ZCQ2NhZ79uzB9OnT0a5dO5QoUQKSJEGSJEyZMkXv68TExGDKlCmoWrUqcufODXd3d9SuXRvz5s1DQkKCyc/PCUz1GhnrZ4CIiIyLGdR2MIMaztQZ0NDrMx/pjxmVbIIgygHq1q0rAIgpU6ZYuiqUAXt7ewFA8+Xk5CRcXV1TbGvevLl4/fq1ScoPDw8XHTt2TFEeAJE3b16hUChSbKtUqZK4cuWKSeqRGUeOHNGqr/rru+++0+sad+/eFSVLltSc5+LiIhwdHTWP33//ffH8+XOTnZ8TmPI1MsbPABERGR8zqO1gBjWMqTNgVq7PfKQfZlSyFeyBSdmeSqXChQsXAPDuty1ISkrChx9+iF9++QW3b99GXFwcXr16hdDQUPTu3RsAsGfPHvTv39/oZV+5cgW1atVCYGAgnJyc0K9fP/zzzz+IjY3FixcvkJCQgJCQEIwdOxYuLi64cuUKYmNjjV4PQ+TLlw+NGzfG6NGjERgYiMKFC+t9blJSElq3bo27d+/Cy8sLBw4cwOvXrxEbG4s//vgDbm5uOH/+PLp06WKS83MCc7xGWfkZICIi42MGtS3MoJln6nxjjOszH6WPGZVsiqVbUIlM7fLly5q7PA8ePLB0dSgDhw8fTnd///79Ne/n/fv3jVbuzZs3RYECBQQAUaxYMXHmzJl0jw8LCxMfffSRiI+PN1odDJWUlKS1rUSJEnrf2Vy+fLnmNf3777+19q9fv16z/+DBg0Y/Pycw9WuU1Z8BIiIyPmZQ28IMmnmmzjdZvT7zUcaYUcmWsAcmZQsnTpzAl19+icKFC8PZ2RlVqlTB4sWLIYTQzD1UsGBBFC1aVOvc+/fv49tvv8UHH3yAvHnzwtHREWXLlsU333yDhw8fZlj2gQMH0LZtWxQsWBDOzs6oWrUqfv75ZwghsHDhQkiShPfff1/rvFGjRkGSJLRq1QpCCKxfvx6NGzeGh4cHJEnCjBkzjFbPrD5Hc2rUqFG6+9V3wAHg7NmzRikzMTERnTp1QmRkJNzc3HD06FHUqlUr3XOKFSuGv/76C7ly5TJKHbJCoVBk6fxVq1YBkF/7unXrau3/6quvUKpUKQDA6tWrjX5+Vrx48QKLFi2Cr68vvL294ejoiAIFCqBq1aro3Lkz/vzzT6OWZyhTv0ZZ/RkgIiLDMIMyg2aFrWdQU+ebrF7fkvmIGVXGjEpGZcnWU6KsUqlUYtSoUSnm0siTJ4/m3wMHDhTDhg0TAESzZs20zv/pp59Erly5NMe7uroKBwcHzWNPT88055dJTEwUffv21RwrSZJwd3fXPO7bt6/o16+fACB69eqldf4nn3wiAIghQ4aIJk2aCABCoVCIfPnyCUmSxJ9//mmUemblXGv033//aeq+adMmo1xz/vz5mmuuXbvWKNe0NH3vbL5+/VrY2dkJAGL27NlpHjdw4EABQBQuXNio52fF33//LQoWLKh57xwcHET+/PlTzGE1b948o5VnKEu9Rry7TURkOsygzKDGYMsZ1NT5xlTXN0c+YkZNHzMqGYoNmGTTJk+eLAAIR0dHMWPGDBEZGSmEEOLx48eiU6dOmv9oAYjx48enOHfRokUCgMifP7+YNWuWiIiIEEIIoVQqxbFjx0SlSpUEAFGnTh2dZY8YMUIA8gTfc+fOFS9evBBCCPHo0SPNBNzqD66ffvpJ6/z8+fMLACJ37twib968YunSpeLVq1dCCCEiIyM1/85KPbP6HK2R+jkBENeuXcvy9d68eSOKFClic69DRvQNBmfPntW8nsn/YElt8eLFmuOePXtmtPMN9erVK83vl5+fn/j3339T7A8PDxdbt24Vt2/fznJZWWWp14jhkIjIdJhBmUGzytYzqKnzjamub+p8xIyaMWZUMhQbMMlmHTp0SEiSJOzs7MSePXu09icmJooKFSpo/rPdvHmzZl9wcLCws7MTRYsWFVevXtV5/WvXrmnOvXv3bop9+/bt09zx3rt3r9a5CQkJokyZMprzT5w4kWL/3bt3NfucnZ3FhQsXdNYhK/XM6nPUZeXKlSl6GmT268iRIxmWkZ4XL14ILy8vAUD4+vpm6VpqW7Zs0dQvMDDQKNdUs+TrpW8w2LFjh6a8tH4OhRBi27ZtmuMuXrxotPMNtX37dgFAlChRQqhUKoOvY473yFKvEcMhEZFpMIOmX09mUP3YegY1db4x1fVNnY+YUTPGjEqGsgeRjRo7diyEEOjfvz+aNWumtd/e3h6tWrXCtWvXALxb/VEIgeHDh0OlUuGXX35BhQoVdF7/vffeg7u7O6KionD//n2UKFFCs2/8+PEAgK+//hpNmzbVOtfBwQFNmjTBr7/+Cjs7O1SvXj3F/vPnz2v+PXPmTFSrVk3rGlmppzGeoy7Ozs4oVKhQusekJytz9ahUKnTt2hURERFwcnLCzz//bPC1ktu7dy8A+T1r3bq1Ua6pZsnXS18xMTGaf7u4uKR5XPJ9yc/J6vmGSkpKAgA8fPgQ69evx5dffgkHB4dMX8cc75GlXiMiIjINZtC068kMqj9bz6Cmzje2mp+YUYlMhw2YZJP+/vtvnD17FnZ2dhg7dmyax3l5eQEA8ubNq5l8+MSJE5qJt/v164d+/fqleX5UVBSAlP9p//333/j333+hUCj0Krt8+fJwdXVNsU8dHt3d3TFw4ECd52elnll9jmnp0KEDOnTokOFxpjB06FDs2rULALB48WKdgdsQ6gn2K1WqpPU+ZZUlX6/srmXLlqhfvz5OnDiBLl26oHv37siXLx8UCgVWrFiBli1b6nUdvkdERJQZzKDp15MZVH/MoNkTMyqR6bABk2ySOkTUrFkTJUuWTPO4p0+fAkCKFRh37typ+ffjx4/1Kq9MmTJaZdeqVQve3t5pnqMOZbpWf1SHx5YtW6Z5dywr9czqc7Q2o0aN0tztXrBgAXr16mW0az969AgA0n0vszM3NzfNv2NjY9M8Lvm+5Odk9XxDOTo6Yv369RgwYAD27NkDpVKJyMhIAEDZsmWzfH1jstRrRERExscMmn49mUH1Z+sZ1NT5xlbzEzMqkenYWboCRIZQh68PP/ww3eNOnz4N4N3QHQD477//AACzZs2CkOeBzfArb968WmXXqlUr3bIvXbqkVbZaSEgIAKBRo0Zpnp+Vemb1OVqTMWPGYN68eQCAuXPnYtiwYUa9fnx8PADzDNe2RkWKFNH8Ozw8PM3jku9Lfk5WzzfUnDlz8N577+HNmzfYt28fnjx5ovlZfu+997J8fWOy1GtERETGxwzKDGostp5BTZ1vbDU/MaMSmQ57YJJNUt/RzZ8/f5rHREREIDg4GEDKAKc+19nZ2aCy1XdL0ys7KioKx44dA6B99/vZs2cICwvTqldqWalnVp9jWjZs2IChQ4cafH5
|
||
|
|
"text/plain": [
|
||
|
|
"<Figure size 1600x500 with 2 Axes>"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"metadata": {},
|
||
|
|
"output_type": "display_data"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"plt.figure(figsize=(16, 5))\n",
|
||
|
|
"plt.subplot(121)\n",
|
||
|
|
"plot_svm_regression(svm_poly_reg1, X, y, [-1, 1, 0, 1])\n",
|
||
|
|
"plt.title(r\"$degree={}, C={}, \\epsilon = {}$\".format(svm_poly_reg1.degree, svm_poly_reg1.C, svm_poly_reg1.epsilon), fontsize=18)\n",
|
||
|
|
"plt.ylabel(r\"$y$\", fontsize=18, rotation=0)\n",
|
||
|
|
"plt.subplot(122)\n",
|
||
|
|
"plot_svm_regression(svm_poly_reg2 , X, y, [-1, 1, 0, 1])\n",
|
||
|
|
"plt.title(r\"$degree={}, C={}, \\epsilon = {}$\".format(svm_poly_reg2.degree, svm_poly_reg2.C, svm_poly_reg2.epsilon), fontsize=18);"
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"metadata": {
|
||
|
|
"celltoolbar": "Slideshow",
|
||
|
|
"kernelspec": {
|
||
|
|
"display_name": "Python 3 (ipykernel)",
|
||
|
|
"language": "python",
|
||
|
|
"name": "python3"
|
||
|
|
},
|
||
|
|
"language_info": {
|
||
|
|
"codemirror_mode": {
|
||
|
|
"name": "ipython",
|
||
|
|
"version": 3
|
||
|
|
},
|
||
|
|
"file_extension": ".py",
|
||
|
|
"mimetype": "text/x-python",
|
||
|
|
"name": "python",
|
||
|
|
"nbconvert_exporter": "python",
|
||
|
|
"pygments_lexer": "ipython3",
|
||
|
|
"version": "3.13.1"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"nbformat": 4,
|
||
|
|
"nbformat_minor": 4
|
||
|
|
}
|