{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f756c5c5",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# Lecture 19: Unsupervised learning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05e057af",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "skip"
    },
    "tags": []
   },
   "source": [
    "![](https://www.tensorflow.org/images/colab_logo_32px.png)\n",
    "[Run in colab](https://colab.research.google.com/drive/1N2lOE8YOXNuekxMtflQCB3eM3I6Bx7-w)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "bd0c5278",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:18.896747Z",
     "iopub.status.busy": "2025-03-07T05:32:18.896515Z",
     "iopub.status.idle": "2025-03-07T05:32:18.902967Z",
     "shell.execute_reply": "2025-03-07T05:32:18.902404Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last executed: 2025-03-07 05:32:18\n"
     ]
    }
   ],
   "source": [
    "import datetime\n",
    "now = datetime.datetime.now()\n",
    "print(\"Last executed: \" + now.strftime(\"%Y-%m-%d %H:%M:%S\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa389c67-03a7-46cc-b849-7963cd86ebc9",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Unsupervised learning overview"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d792ecf7-e0dd-4a17-a021-7cec1e0d5f83",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Labelled data is scarce\n",
    "\n",
    "The majority of data is unlabeled: we have input features $\\mathbf{X}$, but no labels $\\mathbf{y}$."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "01ba1b5a-00a3-443c-aad2-3b01d0a90b3e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Imagine that you want to detect defective items on a manufacturing line. You can collect images of the  items on the  manufacturing line easily and automatically, but if you want to train a classifier to predict whether an item is defective or not, you will need to label every image.\n",
    "\n",
    "This labelling generally requires human experts to sit down and go through the images by hand, which is time consuming, tedious, and prone to errors. As a result, labelling is often only done on a small subset of the available images.\n",
    "\n",
    "With unsupervised learning, you do not need labelled data!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b67f95a-003a-4d7e-9138-bff388491356",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Types of unsupervised learning\n",
    "\n",
    "We will cover three types of unsupervised learning.\n",
    "\n",
    "1. *Clustering*: Group similar data instances into clusters.\n",
    "2. *Density estimation*: Estimate the probability distribution of data (for use as a generative model to generate new data instances or to estimate probability densities).\n",
    "3. *Anomaly detection*: Detect anomalous data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1aedd59-5c76-4866-ae55-0745009652fe",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Clustering with K-means"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d879db6b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Clustering overview\n",
    "\n",
    "*Clustering* is the task of identifying similar instances and assigning them to *clusters*, or groups of similar instances.\n",
    "\n",
    "Consider the Iris dataset, which classifies each instance into one of three classes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "43b2f3d8",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:18.905064Z",
     "iopub.status.busy": "2025-03-07T05:32:18.904894Z",
     "iopub.status.idle": "2025-03-07T05:32:19.276111Z",
     "shell.execute_reply": "2025-03-07T05:32:19.275469Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a4916a09",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:19.278360Z",
     "iopub.status.busy": "2025-03-07T05:32:19.278122Z",
     "iopub.status.idle": "2025-03-07T05:32:20.063169Z",
     "shell.execute_reply": "2025-03-07T05:32:20.062447Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['setosa', 'versicolor', 'virginica'], dtype='<U10')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.datasets import load_iris\n",
    "\n",
    "data = load_iris()\n",
    "X = data.data\n",
    "y = data.target\n",
    "data.target_names"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7e3dedd",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "Let's plot the petal width against the petal length, both with and without different markers for the classes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "010507ab",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:20.065475Z",
     "iopub.status.busy": "2025-03-07T05:32:20.065147Z",
     "iopub.status.idle": "2025-03-07T05:32:20.220141Z",
     "shell.execute_reply": "2025-03-07T05:32:20.219437Z"
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 37992 characters]",
      "text/plain": [
       "<Figure size 900x350 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(9, 3.5))\n",
    "\n",
    "plt.subplot(121)\n",
    "plt.plot(X[y==0, 2], X[y==0, 3], \"yo\", label=\"Iris setosa\")\n",
    "plt.plot(X[y==1, 2], X[y==1, 3], \"bs\", label=\"Iris versicolor\")\n",
    "plt.plot(X[y==2, 2], X[y==2, 3], \"g^\", label=\"Iris virginica\")\n",
    "plt.xlabel(\"Petal length\", fontsize=14)\n",
    "plt.ylabel(\"Petal width\", fontsize=14)\n",
    "plt.legend(fontsize=12)\n",
    "\n",
    "plt.subplot(122)\n",
    "plt.scatter(X[:, 2], X[:, 3], c=\"k\", marker=\".\")\n",
    "plt.xlabel(\"Petal length\", fontsize=14)\n",
    "plt.tick_params(labelleft=False)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c34d799b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "When the instances are plotted without different markers to differentiate the classes, as in the figure on the right, it is not obvious that the upper-right cluster is composed of two distinct sub-clusters."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ade9d090",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "There is no universal definition of what a cluster is. It depends on the context: different algorithms will capture different kinds of clusters.\n",
    "\n",
    "Some algorithms look for instances centered around a particular point, called a *centroid*. Others look for continuous regions of densely packed instances."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dddd23bc-a156-4ae1-bd94-5da2129779d7",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### K-means clustering overview"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c73d2a5-ec56-4ea6-a496-10f66084984f",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Cluster data into different subsets.  \n",
    "\n",
    "First, let's consider how K-means represents clusters, then we'll consider the algorithm to find clusters.\n",
    "\n",
    "- Each cluster is defined by a centroid.\n",
    "- Each data instance is allocated a cluster (equivalently, associated with a centroid)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "258dfb21",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "Let's generate some blobs to use as a dataset, and run the K-Means algorithm for 1, 2, and 3 iterations to see how the centroids move around:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "52682b37-a58a-4647-9b52-2ec50a4ed106",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:20.222313Z",
     "iopub.status.busy": "2025-03-07T05:32:20.222127Z",
     "iopub.status.idle": "2025-03-07T05:32:20.308963Z",
     "shell.execute_reply": "2025-03-07T05:32:20.308289Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.cluster import KMeans\n",
    "from sklearn.datasets import make_blobs\n",
    "blob_centers = np.array([[ 0.2,  2.3], [-1.5 ,  2.3], [-2.8,  1.8],\n",
    "                         [-2.8,  2.8], [-2.8,  1.3]])\n",
    "blob_std = np.array([0.4, 0.3, 0.1, 0.1, 0.1])\n",
    "X, y = make_blobs(n_samples=2000, centers=blob_centers, cluster_std=blob_std,\n",
    "                  random_state=7)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d70af36-80ec-460d-b2ef-6f47b656dde7",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Now let's plot them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d6ea16db-a3a4-4c10-bc45-c4e312ce118f",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:20.311254Z",
     "iopub.status.busy": "2025-03-07T05:32:20.311003Z",
     "iopub.status.idle": "2025-03-07T05:32:20.452710Z",
     "shell.execute_reply": "2025-03-07T05:32:20.451956Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 38664 characters]",
      "text/plain": [
       "<Figure size 800x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_clusters(X, y=None):\n",
    "    plt.scatter(X[:, 0], X[:, 1], c=y, s=1)\n",
    "    plt.xlabel(\"$x_1$\")\n",
    "    plt.ylabel(\"$x_2$\", rotation=0)\n",
    "\n",
    "plt.figure(figsize=(8, 4))\n",
    "plot_clusters(X)\n",
    "plt.gca().set_axisbelow(True)\n",
    "plt.grid()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8cb1420-daa7-4e6c-8439-780eb9ee4b20",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "#### Cluster with Scikit-Learn K-means"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b3bb454-6d2a-4d6c-823f-a6481ace87a8",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Must set number of clusters to consider."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "9b1a11fe-7599-469f-ba66-8aec5bd5e464",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:20.455045Z",
     "iopub.status.busy": "2025-03-07T05:32:20.454858Z",
     "iopub.status.idle": "2025-03-07T05:32:20.516904Z",
     "shell.execute_reply": "2025-03-07T05:32:20.516260Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "k = 5\n",
    "kmeans = KMeans(n_clusters=k, n_init=10, random_state=42)\n",
    "y_pred = kmeans.fit_predict(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "263eeb26-1ef1-408b-828e-027b12b5e65a",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Each instance assigned to one of the five clusters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "473e5159-d2e4-460f-9369-8a4ddd660d64",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:20.520073Z",
     "iopub.status.busy": "2025-03-07T05:32:20.519823Z",
     "iopub.status.idle": "2025-03-07T05:32:20.526424Z",
     "shell.execute_reply": "2025-03-07T05:32:20.525803Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 4, ..., 3, 1, 0], dtype=int32)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4ea4e83-42f4-438f-b05d-32d86e099b28",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "A *centroid* is estimated for each cluster (i.e., cluster centers).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f166dd90-cb2a-4344-83d9-24428a788a16",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:20.528872Z",
     "iopub.status.busy": "2025-03-07T05:32:20.528668Z",
     "iopub.status.idle": "2025-03-07T05:32:20.535050Z",
     "shell.execute_reply": "2025-03-07T05:32:20.534458Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-2.80214068,  1.55162671],\n",
       "       [ 0.08703534,  2.58438091],\n",
       "       [-1.46869323,  2.28214236],\n",
       "       [-2.79290307,  2.79641063],\n",
       "       [ 0.31332823,  1.96822352]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kmeans.cluster_centers_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "403c4770-d2c2-489c-a399-ebf9d88b8f63",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:20.538443Z",
     "iopub.status.busy": "2025-03-07T05:32:20.537552Z",
     "iopub.status.idle": "2025-03-07T05:32:20.543515Z",
     "shell.execute_reply": "2025-03-07T05:32:20.542928Z"
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def plot_centroids(centroids, weights=None, circle_color='w', cross_color='k'):\n",
    "    if weights is not None:\n",
    "        centroids = centroids[weights > weights.max() / 10]\n",
    "    plt.scatter(centroids[:, 0], centroids[:, 1],\n",
    "                marker='o', s=35, linewidths=8,\n",
    "                color=circle_color, zorder=10, alpha=0.9)\n",
    "    plt.scatter(centroids[:, 0], centroids[:, 1],\n",
    "                marker='x', s=2, linewidths=12,\n",
    "                color=cross_color, zorder=11, alpha=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "cc0d3c9d-01ba-4bf4-a619-b6aaa58ce4c1",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:20.546972Z",
     "iopub.status.busy": "2025-03-07T05:32:20.546031Z",
     "iopub.status.idle": "2025-03-07T05:32:20.698656Z",
     "shell.execute_reply": "2025-03-07T05:32:20.697966Z"
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 41680 characters]",
      "text/plain": [
       "<Figure size 800x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(8, 4))\n",
    "plot_clusters(X)\n",
    "plot_centroids(kmeans.cluster_centers_)\n",
    "plt.gca().set_axisbelow(True)\n",
    "plt.grid()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85d52600-bd88-42a3-ab87-534bd5aaf799",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "#### Predictions\n",
    "\n",
    "Can make predictions for new instances:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "4aaa517b-6e61-4420-a67a-eaa6f5b5e4a3",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:20.700640Z",
     "iopub.status.busy": "2025-03-07T05:32:20.700446Z",
     "iopub.status.idle": "2025-03-07T05:32:20.703441Z",
     "shell.execute_reply": "2025-03-07T05:32:20.702851Z"
    },
    "slideshow": {
     "slide_type": "skip"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "7dd1bbaa-2397-4bd2-b498-1015bbb12d51",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:20.705280Z",
     "iopub.status.busy": "2025-03-07T05:32:20.705104Z",
     "iopub.status.idle": "2025-03-07T05:32:20.710360Z",
     "shell.execute_reply": "2025-03-07T05:32:20.709768Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([4, 4, 3, 3], dtype=int32)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_new = np.array([[0, 2], [3, 2], [-3, 3], [-3, 2.5]])\n",
    "kmeans.predict(X_new)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "475155e2-673a-4621-85de-c75110cba1eb",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "#### Decision boundaries"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e13f0010-97f7-4f01-9587-849b86d200ff",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Let's plot the model's decision boundaries. This gives us a _Voronoi diagram_:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "9dd5caf8-5690-4205-bec6-273d580d4bdf",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:20.712259Z",
     "iopub.status.busy": "2025-03-07T05:32:20.712086Z",
     "iopub.status.idle": "2025-03-07T05:32:20.996132Z",
     "shell.execute_reply": "2025-03-07T05:32:20.995470Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 57568 characters]",
      "text/plain": [
       "<Figure size 800x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_data(X):\n",
    "    plt.plot(X[:, 0], X[:, 1], 'k.', markersize=2)\n",
    "\n",
    "def plot_decision_boundaries(clusterer, X, resolution=1000, show_centroids=True,\n",
    "                             show_xlabels=True, show_ylabels=True):\n",
    "    mins = X.min(axis=0) - 0.1\n",
    "    maxs = X.max(axis=0) + 0.1\n",
    "    xx, yy = np.meshgrid(np.linspace(mins[0], maxs[0], resolution),\n",
    "                         np.linspace(mins[1], maxs[1], resolution))\n",
    "    Z = clusterer.predict(np.c_[xx.ravel(), yy.ravel()])\n",
    "    Z = Z.reshape(xx.shape)\n",
    "\n",
    "    plt.contourf(Z, extent=(mins[0], maxs[0], mins[1], maxs[1]),\n",
    "                cmap=\"Pastel2\")\n",
    "    plt.contour(Z, extent=(mins[0], maxs[0], mins[1], maxs[1]),\n",
    "                linewidths=1, colors='k')\n",
    "    plot_data(X)\n",
    "    if show_centroids:\n",
    "        plot_centroids(clusterer.cluster_centers_)\n",
    "\n",
    "    if show_xlabels:\n",
    "        plt.xlabel(\"$x_1$\")\n",
    "    else:\n",
    "        plt.tick_params(labelbottom=False)\n",
    "    if show_ylabels:\n",
    "        plt.ylabel(\"$x_2$\", rotation=0)\n",
    "    else:\n",
    "        plt.tick_params(labelleft=False)\n",
    "\n",
    "plt.figure(figsize=(8, 4))\n",
    "plot_decision_boundaries(kmeans, X)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c8361ca-5628-425d-a00b-1d5aee46f941",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Typically, each data instance is asigned to the cluster with the closest centroid (*hard clustering*)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "356ba5e6-5c23-44a9-9449-17997312fbf0",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "#### Hard versus soft clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d65afee3-559f-4da1-8c9a-99460188b91a",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "*Hard clustering* corresponds to selecting the closest cluster for each instance.\n",
    "\n",
    "Alternatively, with *soft clustering* compute a similarly score based on the confidence of being associated with each cluster.  This is given by the (Euclidean) distance between a data instance and the cluster centroid."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df56c34f-97b0-4816-aa1f-1c87a694eceb",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Can be computed by the `transform()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "0925b7bf-5559-4566-a5e3-b82d6d166ce8",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:20.998235Z",
     "iopub.status.busy": "2025-03-07T05:32:20.998053Z",
     "iopub.status.idle": "2025-03-07T05:32:21.002930Z",
     "shell.execute_reply": "2025-03-07T05:32:21.002348Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[2.84, 0.59, 1.5 , 2.9 , 0.31],\n",
       "       [5.82, 2.97, 4.48, 5.85, 2.69],\n",
       "       [1.46, 3.11, 1.69, 0.29, 3.47],\n",
       "       [0.97, 3.09, 1.55, 0.36, 3.36]])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kmeans.transform(X_new).round(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a98903e2-4bda-40a9-bf74-b316c08cf67b",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Can verify that this is indeed the Euclidian distance between each instance and each centroid:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "ee7eef95-6760-4b40-9819-fe32350dd2a3",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:21.004670Z",
     "iopub.status.busy": "2025-03-07T05:32:21.004503Z",
     "iopub.status.idle": "2025-03-07T05:32:21.008883Z",
     "shell.execute_reply": "2025-03-07T05:32:21.008283Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[2.84, 0.59, 1.5 , 2.9 , 0.31],\n",
       "       [5.82, 2.97, 4.48, 5.85, 2.69],\n",
       "       [1.46, 3.11, 1.69, 0.29, 3.47],\n",
       "       [0.97, 3.09, 1.55, 0.36, 3.36]])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.linalg.norm(np.tile(X_new, (1, k)).reshape(-1, k, 2)\n",
    "               - kmeans.cluster_centers_, axis=2).round(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d041388d-b3f4-45eb-870d-0dcd48199e4c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### K-means algorithm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aee9b2d6-cfda-4a47-bac9-2170d42dd111",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "If know centroids..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eeff7cbf-ff59-4e55-b5d1-231d88e2e6dc",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "can cluster data instances by finding closest centroid."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c6b3a87-8436-48a0-8bd6-920a3032c198",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "If know clusters... "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba85317f-d84b-4601-95cb-bee4366a5816",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "can compute centroid of data instances in cluster."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "679f6ba5-1f5a-40b4-8843-4435a7fc507c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "K-means finds centroids and clusters iteratively following this approach."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d851e64a-7063-464b-8d4d-b54359f90cdf",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "#### Algorithm\n",
    "\n",
    "Must set number of clusters to consider, $k$.\n",
    "\n",
    "Then run iterative algorithm to continually update estimates of centroids and cluster allocations.\n",
    "\n",
    "1. Randomly initialise centroids.\n",
    "2. Label instances by closest centroid.\n",
    "3. Calculate new centroid of each cluster by taking mean of instances in the cluster.\n",
    "4. Repeat steps 2 to 3. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2ab11e4-6d47-4bab-a4cb-de4c0f2f5358",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "#### Algorithm visually\n",
    "\n",
    "Let's run K-Mean algorithm for 1, 2 and 3 iterations, to see how the centroids move."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "f313bf6e-4fd2-425e-8592-a2050f07590e",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:21.010924Z",
     "iopub.status.busy": "2025-03-07T05:32:21.010760Z",
     "iopub.status.idle": "2025-03-07T05:32:22.385407Z",
     "shell.execute_reply": "2025-03-07T05:32:22.384850Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 185596 characters]",
      "text/plain": [
       "<Figure size 1000x800 with 6 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "starting_random_state = 6\n",
    "kmeans_iter1 = KMeans(n_clusters=5, init=\"random\", n_init=1, max_iter=1,\n",
    "                      random_state=starting_random_state)\n",
    "kmeans_iter2 = KMeans(n_clusters=5, init=\"random\", n_init=1, max_iter=2,\n",
    "                      random_state=starting_random_state)\n",
    "kmeans_iter3 = KMeans(n_clusters=5, init=\"random\", n_init=1, max_iter=3,\n",
    "                      random_state=starting_random_state)\n",
    "kmeans_iter1.fit(X)\n",
    "kmeans_iter2.fit(X)\n",
    "kmeans_iter3.fit(X)\n",
    "\n",
    "plt.figure(figsize=(10, 8))\n",
    "\n",
    "plt.subplot(321)\n",
    "plot_data(X)\n",
    "plot_centroids(kmeans_iter1.cluster_centers_, circle_color='r', cross_color='w')\n",
    "plt.ylabel(\"$x_2$\", rotation=0)\n",
    "plt.tick_params(labelbottom=False)\n",
    "plt.title(\"Update the centroids (initially randomly)\")\n",
    "\n",
    "plt.subplot(322)\n",
    "plot_decision_boundaries(kmeans_iter1, X, show_xlabels=False,\n",
    "                         show_ylabels=False)\n",
    "plt.title(\"Label the instances\")\n",
    "\n",
    "plt.subplot(323)\n",
    "plot_decision_boundaries(kmeans_iter1, X, show_centroids=False,\n",
    "                         show_xlabels=False)\n",
    "plot_centroids(kmeans_iter2.cluster_centers_)\n",
    "\n",
    "plt.subplot(324)\n",
    "plot_decision_boundaries(kmeans_iter2, X, show_xlabels=False,\n",
    "                         show_ylabels=False)\n",
    "\n",
    "plt.subplot(325)\n",
    "plot_decision_boundaries(kmeans_iter2, X, show_centroids=False)\n",
    "plot_centroids(kmeans_iter3.cluster_centers_)\n",
    "\n",
    "plt.subplot(326)\n",
    "plot_decision_boundaries(kmeans_iter3, X, show_ylabels=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22b6d0c4-1524-4f57-917d-a2e5bb182684",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "#### Convergence\n",
    "\n",
    "Convergence is guaranteed since mean squared distance between instances and their closest centroid can only go down at each step.\n",
    "\n",
    "Usually only a relatively small number of iterations are required."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f97d2836-fa6d-44b6-9381-a79ff0c71d1a",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "#### Variability\n",
    "\n",
    "While the algorithm is guaranteed to converge, it might not converge to the desired solution.\n",
    "\n",
    "Final clusters are highly dependent on initial random centroid selection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "9f229355-f37d-4759-8880-93da14aade52",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:22.387747Z",
     "iopub.status.busy": "2025-03-07T05:32:22.387288Z",
     "iopub.status.idle": "2025-03-07T05:32:23.117782Z",
     "shell.execute_reply": "2025-03-07T05:32:23.117092Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 83080 characters]",
      "text/plain": [
       "<Figure size 1000x320 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_clusterer_comparison(clusterer1, clusterer2, X, title1=None,\n",
    "                              title2=None):\n",
    "    clusterer1.fit(X)\n",
    "    clusterer2.fit(X)\n",
    "\n",
    "    plt.figure(figsize=(10, 3.2))\n",
    "\n",
    "    plt.subplot(121)\n",
    "    plot_decision_boundaries(clusterer1, X)\n",
    "    if title1:\n",
    "        plt.title(title1)\n",
    "\n",
    "    plt.subplot(122)\n",
    "    plot_decision_boundaries(clusterer2, X, show_ylabels=False)\n",
    "    if title2:\n",
    "        plt.title(title2)\n",
    "\n",
    "kmeans_rnd_init1 = KMeans(n_clusters=5, init=\"random\", n_init=1, random_state=2)\n",
    "kmeans_rnd_init2 = KMeans(n_clusters=5, init=\"random\", n_init=1, random_state=9)\n",
    "\n",
    "plot_clusterer_comparison(kmeans_rnd_init1, kmeans_rnd_init2, X,\n",
    "                          \"Solution 1\",\n",
    "                          \"Solution 2 (with a different random init)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "161b0c65-e0ae-474a-aedb-0f3a4d4c2c96",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "#### Manual initialisation\n",
    "\n",
    "If have a rough idea of good clustering, can set manual initialisation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "111f75d2-01d6-4216-9b4c-95f433895eeb",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:23.119729Z",
     "iopub.status.busy": "2025-03-07T05:32:23.119556Z",
     "iopub.status.idle": "2025-03-07T05:32:23.128790Z",
     "shell.execute_reply": "2025-03-07T05:32:23.128193Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {\n",
       "  /* Definition of color scheme common for light and dark mode */\n",
       "  --sklearn-color-text: #000;\n",
       "  --sklearn-color-text-muted: #666;\n",
       "  --sklearn-color-line: gray;\n",
       "  /* Definition of color scheme for unfitted estimators */\n",
       "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
       "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
       "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
       "  --sklearn-color-unfitted-level-3: chocolate;\n",
       "  /* Definition of color scheme for fitted estimators */\n",
       "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
       "  --sklearn-color-fitted-level-1: #d4ebff;\n",
       "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
       "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
       "\n",
       "  /* Specific color for light theme */\n",
       "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
       "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-icon: #696969;\n",
       "\n",
       "  @media (prefers-color-scheme: dark) {\n",
       "    /* Redefinition of color scheme for dark theme */\n",
       "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
       "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-icon: #878787;\n",
       "  }\n",
       "}\n",
       "\n",
       "#sk-container-id-1 {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 pre {\n",
       "  padding: 0;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-hidden--visually {\n",
       "  border: 0;\n",
       "  clip: rect(1px 1px 1px 1px);\n",
       "  clip: rect(1px, 1px, 1px, 1px);\n",
       "  height: 1px;\n",
       "  margin: -1px;\n",
       "  overflow: hidden;\n",
       "  padding: 0;\n",
       "  position: absolute;\n",
       "  width: 1px;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-dashed-wrapped {\n",
       "  border: 1px dashed var(--sklearn-color-line);\n",
       "  margin: 0 0.4em 0.5em 0.4em;\n",
       "  box-sizing: border-box;\n",
       "  padding-bottom: 0.4em;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-container {\n",
       "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
       "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
       "     so we also need the `!important` here to be able to override the\n",
       "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
       "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
       "  display: inline-block !important;\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-text-repr-fallback {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       "div.sk-parallel-item,\n",
       "div.sk-serial,\n",
       "div.sk-item {\n",
       "  /* draw centered vertical line to link estimators */\n",
       "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
       "  background-size: 2px 100%;\n",
       "  background-repeat: no-repeat;\n",
       "  background-position: center center;\n",
       "}\n",
       "\n",
       "/* Parallel-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item::after {\n",
       "  content: \"\";\n",
       "  width: 100%;\n",
       "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
       "  flex-grow: 1;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel {\n",
       "  display: flex;\n",
       "  align-items: stretch;\n",
       "  justify-content: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:first-child::after {\n",
       "  align-self: flex-end;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:last-child::after {\n",
       "  align-self: flex-start;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-parallel-item:only-child::after {\n",
       "  width: 0;\n",
       "}\n",
       "\n",
       "/* Serial-specific style estimator block */\n",
       "\n",
       "#sk-container-id-1 div.sk-serial {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  align-items: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  padding-right: 1em;\n",
       "  padding-left: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
       "clickable and can be expanded/collapsed.\n",
       "- Pipeline and ColumnTransformer use this feature and define the default style\n",
       "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
       "*/\n",
       "\n",
       "/* Pipeline and ColumnTransformer style (default) */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable {\n",
       "  /* Default theme specific background. It is overwritten whether we have a\n",
       "  specific estimator or a Pipeline/ColumnTransformer */\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "/* Toggleable label */\n",
       "#sk-container-id-1 label.sk-toggleable__label {\n",
       "  cursor: pointer;\n",
       "  display: flex;\n",
       "  width: 100%;\n",
       "  margin-bottom: 0;\n",
       "  padding: 0.5em;\n",
       "  box-sizing: border-box;\n",
       "  text-align: center;\n",
       "  align-items: start;\n",
       "  justify-content: space-between;\n",
       "  gap: 0.5em;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label .caption {\n",
       "  font-size: 0.6rem;\n",
       "  font-weight: lighter;\n",
       "  color: var(--sklearn-color-text-muted);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:before {\n",
       "  /* Arrow on the left of the label */\n",
       "  content: \"▸\";\n",
       "  float: left;\n",
       "  margin-right: 0.25em;\n",
       "  color: var(--sklearn-color-icon);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "/* Toggleable content - dropdown */\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content {\n",
       "  max-height: 0;\n",
       "  max-width: 0;\n",
       "  overflow: hidden;\n",
       "  text-align: left;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content pre {\n",
       "  margin: 0.2em;\n",
       "  border-radius: 0.25em;\n",
       "  color: var(--sklearn-color-text);\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-toggleable__content.fitted pre {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
       "  /* Expand drop-down */\n",
       "  max-height: 200px;\n",
       "  max-width: 100%;\n",
       "  overflow: auto;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
       "  content: \"▾\";\n",
       "}\n",
       "\n",
       "/* Pipeline/ColumnTransformer-specific style */\n",
       "\n",
       "#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator-specific style */\n",
       "\n",
       "/* Colorize estimator box */\n",
       "#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label label.sk-toggleable__label,\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  /* The background is the default theme color */\n",
       "  color: var(--sklearn-color-text-on-default-background);\n",
       "}\n",
       "\n",
       "/* On hover, darken the color of the background */\n",
       "#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "/* Label box, darken color on hover, fitted */\n",
       "#sk-container-id-1 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator label */\n",
       "\n",
       "#sk-container-id-1 div.sk-label label {\n",
       "  font-family: monospace;\n",
       "  font-weight: bold;\n",
       "  display: inline-block;\n",
       "  line-height: 1.2em;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-label-container {\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "/* Estimator-specific */\n",
       "#sk-container-id-1 div.sk-estimator {\n",
       "  font-family: monospace;\n",
       "  border: 1px dotted var(--sklearn-color-border-box);\n",
       "  border-radius: 0.25em;\n",
       "  box-sizing: border-box;\n",
       "  margin-bottom: 0.5em;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "/* on hover */\n",
       "#sk-container-id-1 div.sk-estimator:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-1 div.sk-estimator.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
       "\n",
       "/* Common style for \"i\" and \"?\" */\n",
       "\n",
       ".sk-estimator-doc-link,\n",
       "a:link.sk-estimator-doc-link,\n",
       "a:visited.sk-estimator-doc-link {\n",
       "  float: right;\n",
       "  font-size: smaller;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1em;\n",
       "  height: 1em;\n",
       "  width: 1em;\n",
       "  text-decoration: none !important;\n",
       "  margin-left: 0.5em;\n",
       "  text-align: center;\n",
       "  /* unfitted */\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted,\n",
       "a:link.sk-estimator-doc-link.fitted,\n",
       "a:visited.sk-estimator-doc-link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "/* Span, style for the box shown on hovering the info icon */\n",
       ".sk-estimator-doc-link span {\n",
       "  display: none;\n",
       "  z-index: 9999;\n",
       "  position: relative;\n",
       "  font-weight: normal;\n",
       "  right: .2ex;\n",
       "  padding: .5ex;\n",
       "  margin: .5ex;\n",
       "  width: min-content;\n",
       "  min-width: 20ex;\n",
       "  max-width: 50ex;\n",
       "  color: var(--sklearn-color-text);\n",
       "  box-shadow: 2pt 2pt 4pt #999;\n",
       "  /* unfitted */\n",
       "  background: var(--sklearn-color-unfitted-level-0);\n",
       "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted span {\n",
       "  /* fitted */\n",
       "  background: var(--sklearn-color-fitted-level-0);\n",
       "  border: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link:hover span {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link {\n",
       "  float: right;\n",
       "  font-size: 1rem;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1rem;\n",
       "  height: 1rem;\n",
       "  width: 1rem;\n",
       "  text-decoration: none;\n",
       "  /* unfitted */\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "#sk-container-id-1 a.estimator_doc_link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "#sk-container-id-1 a.estimator_doc_link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>KMeans(init=array([[-3,  3],\n",
       "       [-3,  2],\n",
       "       [-3,  1],\n",
       "       [-1,  2],\n",
       "       [ 0,  2]]),\n",
       "       n_clusters=5, n_init=1, random_state=42)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" checked><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow\"><div><div>KMeans</div></div><div><a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.6/modules/generated/sklearn.cluster.KMeans.html\">?<span>Documentation for KMeans</span></a><span class=\"sk-estimator-doc-link fitted\">i<span>Fitted</span></span></div></label><div class=\"sk-toggleable__content fitted\"><pre>KMeans(init=array([[-3,  3],\n",
       "       [-3,  2],\n",
       "       [-3,  1],\n",
       "       [-1,  2],\n",
       "       [ 0,  2]]),\n",
       "       n_clusters=5, n_init=1, random_state=42)</pre></div> </div></div></div></div>"
      ],
      "text/plain": [
       "KMeans(init=array([[-3,  3],\n",
       "       [-3,  2],\n",
       "       [-3,  1],\n",
       "       [-1,  2],\n",
       "       [ 0,  2]]),\n",
       "       n_clusters=5, n_init=1, random_state=42)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "good_init = np.array([[-3, 3], [-3, 2], [-3, 1], [-1, 2], [0, 2]])\n",
    "kmeans = KMeans(n_clusters=5, init=good_init, n_init=1, random_state=42)\n",
    "kmeans.fit(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "1102abdb-5e5e-4200-b147-4b989002ef71",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:23.130470Z",
     "iopub.status.busy": "2025-03-07T05:32:23.130290Z",
     "iopub.status.idle": "2025-03-07T05:32:23.415955Z",
     "shell.execute_reply": "2025-03-07T05:32:23.415286Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 49400 characters]",
      "text/plain": [
       "<Figure size 800x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(8, 4))\n",
    "plot_decision_boundaries(kmeans, X)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b289ad4-aacb-4ee7-a5ee-b6f77a5a60ae",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "#### Performance measure\n",
    "\n",
    "In order to assess whether a set of clusters is good, need to have some measure of performance.\n",
    "\n",
    "Goodness can be estimated by *sum of squared distances between instances and their clustered centroid* (low good).\n",
    "\n",
    "Score given by negative of this qualtity (so high good)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "b18c5ff7-0a7c-44f9-be65-ccbed864eca7",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:23.417941Z",
     "iopub.status.busy": "2025-03-07T05:32:23.417765Z",
     "iopub.status.idle": "2025-03-07T05:32:23.422662Z",
     "shell.execute_reply": "2025-03-07T05:32:23.422070Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-211.59853725816828"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kmeans.score(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbdd8b7f-86bd-4a66-bf74-7986609f996c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "#### Multiple initialisations\n",
    "\n",
    "Now have measure of performance, can run multiple times with different centroid initialisations and then select clustering with highest score.\n",
    "\n",
    "Controlled by `n_init` variable in Scikit-Learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "59c62d7b-1a33-43a9-a348-72190b6894fc",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:23.424453Z",
     "iopub.status.busy": "2025-03-07T05:32:23.424262Z",
     "iopub.status.idle": "2025-03-07T05:32:23.437608Z",
     "shell.execute_reply": "2025-03-07T05:32:23.437018Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-2 {\n",
       "  /* Definition of color scheme common for light and dark mode */\n",
       "  --sklearn-color-text: #000;\n",
       "  --sklearn-color-text-muted: #666;\n",
       "  --sklearn-color-line: gray;\n",
       "  /* Definition of color scheme for unfitted estimators */\n",
       "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
       "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
       "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
       "  --sklearn-color-unfitted-level-3: chocolate;\n",
       "  /* Definition of color scheme for fitted estimators */\n",
       "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
       "  --sklearn-color-fitted-level-1: #d4ebff;\n",
       "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
       "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
       "\n",
       "  /* Specific color for light theme */\n",
       "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
       "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-icon: #696969;\n",
       "\n",
       "  @media (prefers-color-scheme: dark) {\n",
       "    /* Redefinition of color scheme for dark theme */\n",
       "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
       "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-icon: #878787;\n",
       "  }\n",
       "}\n",
       "\n",
       "#sk-container-id-2 {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 pre {\n",
       "  padding: 0;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 input.sk-hidden--visually {\n",
       "  border: 0;\n",
       "  clip: rect(1px 1px 1px 1px);\n",
       "  clip: rect(1px, 1px, 1px, 1px);\n",
       "  height: 1px;\n",
       "  margin: -1px;\n",
       "  overflow: hidden;\n",
       "  padding: 0;\n",
       "  position: absolute;\n",
       "  width: 1px;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-dashed-wrapped {\n",
       "  border: 1px dashed var(--sklearn-color-line);\n",
       "  margin: 0 0.4em 0.5em 0.4em;\n",
       "  box-sizing: border-box;\n",
       "  padding-bottom: 0.4em;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-container {\n",
       "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
       "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
       "     so we also need the `!important` here to be able to override the\n",
       "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
       "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
       "  display: inline-block !important;\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-text-repr-fallback {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       "div.sk-parallel-item,\n",
       "div.sk-serial,\n",
       "div.sk-item {\n",
       "  /* draw centered vertical line to link estimators */\n",
       "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
       "  background-size: 2px 100%;\n",
       "  background-repeat: no-repeat;\n",
       "  background-position: center center;\n",
       "}\n",
       "\n",
       "/* Parallel-specific style estimator block */\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item::after {\n",
       "  content: \"\";\n",
       "  width: 100%;\n",
       "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
       "  flex-grow: 1;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel {\n",
       "  display: flex;\n",
       "  align-items: stretch;\n",
       "  justify-content: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item:first-child::after {\n",
       "  align-self: flex-end;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item:last-child::after {\n",
       "  align-self: flex-start;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-parallel-item:only-child::after {\n",
       "  width: 0;\n",
       "}\n",
       "\n",
       "/* Serial-specific style estimator block */\n",
       "\n",
       "#sk-container-id-2 div.sk-serial {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  align-items: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  padding-right: 1em;\n",
       "  padding-left: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
       "clickable and can be expanded/collapsed.\n",
       "- Pipeline and ColumnTransformer use this feature and define the default style\n",
       "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
       "*/\n",
       "\n",
       "/* Pipeline and ColumnTransformer style (default) */\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable {\n",
       "  /* Default theme specific background. It is overwritten whether we have a\n",
       "  specific estimator or a Pipeline/ColumnTransformer */\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "/* Toggleable label */\n",
       "#sk-container-id-2 label.sk-toggleable__label {\n",
       "  cursor: pointer;\n",
       "  display: flex;\n",
       "  width: 100%;\n",
       "  margin-bottom: 0;\n",
       "  padding: 0.5em;\n",
       "  box-sizing: border-box;\n",
       "  text-align: center;\n",
       "  align-items: start;\n",
       "  justify-content: space-between;\n",
       "  gap: 0.5em;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 label.sk-toggleable__label .caption {\n",
       "  font-size: 0.6rem;\n",
       "  font-weight: lighter;\n",
       "  color: var(--sklearn-color-text-muted);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 label.sk-toggleable__label-arrow:before {\n",
       "  /* Arrow on the left of the label */\n",
       "  content: \"▸\";\n",
       "  float: left;\n",
       "  margin-right: 0.25em;\n",
       "  color: var(--sklearn-color-icon);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 label.sk-toggleable__label-arrow:hover:before {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "/* Toggleable content - dropdown */\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content {\n",
       "  max-height: 0;\n",
       "  max-width: 0;\n",
       "  overflow: hidden;\n",
       "  text-align: left;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content pre {\n",
       "  margin: 0.2em;\n",
       "  border-radius: 0.25em;\n",
       "  color: var(--sklearn-color-text);\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-toggleable__content.fitted pre {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
       "  /* Expand drop-down */\n",
       "  max-height: 200px;\n",
       "  max-width: 100%;\n",
       "  overflow: auto;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
       "  content: \"▾\";\n",
       "}\n",
       "\n",
       "/* Pipeline/ColumnTransformer-specific style */\n",
       "\n",
       "#sk-container-id-2 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator-specific style */\n",
       "\n",
       "/* Colorize estimator box */\n",
       "#sk-container-id-2 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-label label.sk-toggleable__label,\n",
       "#sk-container-id-2 div.sk-label label {\n",
       "  /* The background is the default theme color */\n",
       "  color: var(--sklearn-color-text-on-default-background);\n",
       "}\n",
       "\n",
       "/* On hover, darken the color of the background */\n",
       "#sk-container-id-2 div.sk-label:hover label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "/* Label box, darken color on hover, fitted */\n",
       "#sk-container-id-2 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator label */\n",
       "\n",
       "#sk-container-id-2 div.sk-label label {\n",
       "  font-family: monospace;\n",
       "  font-weight: bold;\n",
       "  display: inline-block;\n",
       "  line-height: 1.2em;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-label-container {\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "/* Estimator-specific */\n",
       "#sk-container-id-2 div.sk-estimator {\n",
       "  font-family: monospace;\n",
       "  border: 1px dotted var(--sklearn-color-border-box);\n",
       "  border-radius: 0.25em;\n",
       "  box-sizing: border-box;\n",
       "  margin-bottom: 0.5em;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-estimator.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "/* on hover */\n",
       "#sk-container-id-2 div.sk-estimator:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-2 div.sk-estimator.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
       "\n",
       "/* Common style for \"i\" and \"?\" */\n",
       "\n",
       ".sk-estimator-doc-link,\n",
       "a:link.sk-estimator-doc-link,\n",
       "a:visited.sk-estimator-doc-link {\n",
       "  float: right;\n",
       "  font-size: smaller;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1em;\n",
       "  height: 1em;\n",
       "  width: 1em;\n",
       "  text-decoration: none !important;\n",
       "  margin-left: 0.5em;\n",
       "  text-align: center;\n",
       "  /* unfitted */\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted,\n",
       "a:link.sk-estimator-doc-link.fitted,\n",
       "a:visited.sk-estimator-doc-link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "/* Span, style for the box shown on hovering the info icon */\n",
       ".sk-estimator-doc-link span {\n",
       "  display: none;\n",
       "  z-index: 9999;\n",
       "  position: relative;\n",
       "  font-weight: normal;\n",
       "  right: .2ex;\n",
       "  padding: .5ex;\n",
       "  margin: .5ex;\n",
       "  width: min-content;\n",
       "  min-width: 20ex;\n",
       "  max-width: 50ex;\n",
       "  color: var(--sklearn-color-text);\n",
       "  box-shadow: 2pt 2pt 4pt #999;\n",
       "  /* unfitted */\n",
       "  background: var(--sklearn-color-unfitted-level-0);\n",
       "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted span {\n",
       "  /* fitted */\n",
       "  background: var(--sklearn-color-fitted-level-0);\n",
       "  border: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link:hover span {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
       "\n",
       "#sk-container-id-2 a.estimator_doc_link {\n",
       "  float: right;\n",
       "  font-size: 1rem;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1rem;\n",
       "  height: 1rem;\n",
       "  width: 1rem;\n",
       "  text-decoration: none;\n",
       "  /* unfitted */\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 a.estimator_doc_link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "#sk-container-id-2 a.estimator_doc_link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "#sk-container-id-2 a.estimator_doc_link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "</style><div id=\"sk-container-id-2\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>KMeans(init=&#x27;random&#x27;, n_clusters=5, n_init=10, random_state=2)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" checked><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow\"><div><div>KMeans</div></div><div><a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.6/modules/generated/sklearn.cluster.KMeans.html\">?<span>Documentation for KMeans</span></a><span class=\"sk-estimator-doc-link fitted\">i<span>Fitted</span></span></div></label><div class=\"sk-toggleable__content fitted\"><pre>KMeans(init=&#x27;random&#x27;, n_clusters=5, n_init=10, random_state=2)</pre></div> </div></div></div></div>"
      ],
      "text/plain": [
       "KMeans(init='random', n_clusters=5, n_init=10, random_state=2)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kmeans_rnd_10_inits = KMeans(n_clusters=5, init=\"random\", n_init=10,\n",
    "                             random_state=2)\n",
    "kmeans_rnd_10_inits.fit(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4524c4c7-8f7c-49d9-943d-412a24ab77b1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Recover good clusters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "eda66172-ee1c-439c-b602-c5a9de875e28",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:23.439340Z",
     "iopub.status.busy": "2025-03-07T05:32:23.439172Z",
     "iopub.status.idle": "2025-03-07T05:32:23.711937Z",
     "shell.execute_reply": "2025-03-07T05:32:23.711261Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 49308 characters]",
      "text/plain": [
       "<Figure size 800x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(8, 4))\n",
    "plot_decision_boundaries(kmeans_rnd_10_inits, X)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c842d05a-f644-4eaa-ad76-020ca86928cf",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "#### Extensions\n",
    "\n",
    "- Acceleration\n",
    "- Mini-batch\n",
    "- Selecting number of clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a63242f-1e3c-44c8-b2a9-6894639579b0",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "#### Limitations\n",
    "\n",
    "- Need to run several times with different initialisations\n",
    "- Need to specify number of clusters\n",
    "- Does not behave well when clusters have varying sizes, different densities, or non-spherical shapes."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f0a0e6d-0d88-488e-b524-0d489ca7cbce",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Density estimation with Gaussian mixture models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a75ccc62-75ea-4731-a770-d435582217be",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "Aim of density estimation is to estimate a *probability distribution* from which data were drawn.  \n",
    "\n",
    "Can then generate new data instances (generative AI) and also perform probability density estimation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5698c35f-e9dd-411e-a27c-7ad557863b14",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "A *Gaussian mixture model* (GMM) is a probabilistic model that assumes that the instances were generated from a mixture of a number of Gaussian distributions whose parameters are unknown."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d99d4e4d-7c89-4b0b-82a3-01a89421fa7c",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Gaussian mixture model description"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95422c17-1259-4fab-9ec3-61c9128272d2",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Instances generated from a Gaussian distribution form clusters that look like ellipsoids. Each cluster can have a different ellipoidal shape, size, density, and orientation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a15daff3-3eca-446e-b8a1-c21749f34d8e",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Scikit-Learn contains several GMM variants. In the simplist, the `GaussianMixture` class, you must know in advance the number of Gaussian distributions $k$."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "724d96ff-26bb-42dc-879a-9841813029c1",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "The dataset $\\mathbf{X}$ is assumed to have been generated through the following probabilistic process:\n",
    "\n",
    "- For each instance, a cluster is picked randomly from among $k$ clusters. The probability of choosing the $j^{\\text{th}}$ cluster is defined by the clusters weight, $\\phi^{(j)}$. The index of the cluster chosen for the $i^{\\text{th}}$ instance is noted $z^{(i)}$.\n",
    "- If $z^{(i)}=j$, meaning the $i^{\\text{th}}$ instance has been assigned to the $j^{\\text{th}}$ cluster, the location $\\mathbf{x}^{(i)}$  of this instance is sampled randomly from the Gaussian distribution with mean $\\mathbf{\\mu}^{(j)}$ and covariance matrix ${\\sum}^{(j)}$. This is noted as:\n",
    "\n",
    "$$\\mathbf{x}^{(i)} \\sim \\mathcal{N} \\left( \\mathbf{\\mu}^{(j)}, {\\sum}^{(j)} \\right)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28acc2bd-d0f4-48f1-95f9-25ac4b28fea7",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "Let's generate some random blobs to use as a dataset again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "684e1980-71eb-4748-a70c-f7cffcd914d5",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:23.714311Z",
     "iopub.status.busy": "2025-03-07T05:32:23.714134Z",
     "iopub.status.idle": "2025-03-07T05:32:23.719858Z",
     "shell.execute_reply": "2025-03-07T05:32:23.719306Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "X1, y1 = make_blobs(n_samples=1000, centers=((4, -4), (0, 0)), random_state=42)\n",
    "X1 = X1.dot(np.array([[0.374, 0.95], [0.732, 0.598]]))\n",
    "X2, y2 = make_blobs(n_samples=250, centers=1, random_state=42)\n",
    "X2 = X2 + [6, -8]\n",
    "X = np.r_[X1, X2]\n",
    "y = np.r_[y1, y2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "29b9362a-9ae4-49d9-93b9-7483a6d8dbbc",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:23.721625Z",
     "iopub.status.busy": "2025-03-07T05:32:23.721457Z",
     "iopub.status.idle": "2025-03-07T05:32:23.823411Z",
     "shell.execute_reply": "2025-03-07T05:32:23.822731Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 22156 characters]",
      "text/plain": [
       "<Figure size 800x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(8, 4))\n",
    "plt.plot(X[:, 0], X[:, 1], 'k.', markersize=2)\n",
    "plt.xlabel(\"$x_1$\")\n",
    "plt.ylabel(\"$x_2$\", rotation=0);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ef1235d-28d2-4fdd-834a-52d948f3111d",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Training\n",
    "\n",
    "Let's train a Gaussian mixture model.  \n",
    "\n",
    "The model is trainined by the *expectation-maximisation* (EM) algorithm.  We won't go into technical details but has similarites with the K-means algorithm.  It initialises clusters randomly, then repeats an *expectation* step (cf. assign instances to clusters) and a *maximisation* step (cf. updating clusters)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "21208a04-d9ff-431d-ad74-843a079b2100",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:23.825362Z",
     "iopub.status.busy": "2025-03-07T05:32:23.825184Z",
     "iopub.status.idle": "2025-03-07T05:32:23.830771Z",
     "shell.execute_reply": "2025-03-07T05:32:23.830186Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.mixture import GaussianMixture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "effa4aaf-6129-4226-b59a-a97b8d32a4ef",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:23.832627Z",
     "iopub.status.busy": "2025-03-07T05:32:23.832316Z",
     "iopub.status.idle": "2025-03-07T05:32:23.902677Z",
     "shell.execute_reply": "2025-03-07T05:32:23.902140Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-3 {\n",
       "  /* Definition of color scheme common for light and dark mode */\n",
       "  --sklearn-color-text: #000;\n",
       "  --sklearn-color-text-muted: #666;\n",
       "  --sklearn-color-line: gray;\n",
       "  /* Definition of color scheme for unfitted estimators */\n",
       "  --sklearn-color-unfitted-level-0: #fff5e6;\n",
       "  --sklearn-color-unfitted-level-1: #f6e4d2;\n",
       "  --sklearn-color-unfitted-level-2: #ffe0b3;\n",
       "  --sklearn-color-unfitted-level-3: chocolate;\n",
       "  /* Definition of color scheme for fitted estimators */\n",
       "  --sklearn-color-fitted-level-0: #f0f8ff;\n",
       "  --sklearn-color-fitted-level-1: #d4ebff;\n",
       "  --sklearn-color-fitted-level-2: #b3dbfd;\n",
       "  --sklearn-color-fitted-level-3: cornflowerblue;\n",
       "\n",
       "  /* Specific color for light theme */\n",
       "  --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, white)));\n",
       "  --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, black)));\n",
       "  --sklearn-color-icon: #696969;\n",
       "\n",
       "  @media (prefers-color-scheme: dark) {\n",
       "    /* Redefinition of color scheme for dark theme */\n",
       "    --sklearn-color-text-on-default-background: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-background: var(--sg-background-color, var(--theme-background, var(--jp-layout-color0, #111)));\n",
       "    --sklearn-color-border-box: var(--sg-text-color, var(--theme-code-foreground, var(--jp-content-font-color1, white)));\n",
       "    --sklearn-color-icon: #878787;\n",
       "  }\n",
       "}\n",
       "\n",
       "#sk-container-id-3 {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 pre {\n",
       "  padding: 0;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 input.sk-hidden--visually {\n",
       "  border: 0;\n",
       "  clip: rect(1px 1px 1px 1px);\n",
       "  clip: rect(1px, 1px, 1px, 1px);\n",
       "  height: 1px;\n",
       "  margin: -1px;\n",
       "  overflow: hidden;\n",
       "  padding: 0;\n",
       "  position: absolute;\n",
       "  width: 1px;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-dashed-wrapped {\n",
       "  border: 1px dashed var(--sklearn-color-line);\n",
       "  margin: 0 0.4em 0.5em 0.4em;\n",
       "  box-sizing: border-box;\n",
       "  padding-bottom: 0.4em;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-container {\n",
       "  /* jupyter's `normalize.less` sets `[hidden] { display: none; }`\n",
       "     but bootstrap.min.css set `[hidden] { display: none !important; }`\n",
       "     so we also need the `!important` here to be able to override the\n",
       "     default hidden behavior on the sphinx rendered scikit-learn.org.\n",
       "     See: https://github.com/scikit-learn/scikit-learn/issues/21755 */\n",
       "  display: inline-block !important;\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-text-repr-fallback {\n",
       "  display: none;\n",
       "}\n",
       "\n",
       "div.sk-parallel-item,\n",
       "div.sk-serial,\n",
       "div.sk-item {\n",
       "  /* draw centered vertical line to link estimators */\n",
       "  background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));\n",
       "  background-size: 2px 100%;\n",
       "  background-repeat: no-repeat;\n",
       "  background-position: center center;\n",
       "}\n",
       "\n",
       "/* Parallel-specific style estimator block */\n",
       "\n",
       "#sk-container-id-3 div.sk-parallel-item::after {\n",
       "  content: \"\";\n",
       "  width: 100%;\n",
       "  border-bottom: 2px solid var(--sklearn-color-text-on-default-background);\n",
       "  flex-grow: 1;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-parallel {\n",
       "  display: flex;\n",
       "  align-items: stretch;\n",
       "  justify-content: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  position: relative;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-parallel-item {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-parallel-item:first-child::after {\n",
       "  align-self: flex-end;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-parallel-item:last-child::after {\n",
       "  align-self: flex-start;\n",
       "  width: 50%;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-parallel-item:only-child::after {\n",
       "  width: 0;\n",
       "}\n",
       "\n",
       "/* Serial-specific style estimator block */\n",
       "\n",
       "#sk-container-id-3 div.sk-serial {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  align-items: center;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  padding-right: 1em;\n",
       "  padding-left: 1em;\n",
       "}\n",
       "\n",
       "\n",
       "/* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is\n",
       "clickable and can be expanded/collapsed.\n",
       "- Pipeline and ColumnTransformer use this feature and define the default style\n",
       "- Estimators will overwrite some part of the style using the `sk-estimator` class\n",
       "*/\n",
       "\n",
       "/* Pipeline and ColumnTransformer style (default) */\n",
       "\n",
       "#sk-container-id-3 div.sk-toggleable {\n",
       "  /* Default theme specific background. It is overwritten whether we have a\n",
       "  specific estimator or a Pipeline/ColumnTransformer */\n",
       "  background-color: var(--sklearn-color-background);\n",
       "}\n",
       "\n",
       "/* Toggleable label */\n",
       "#sk-container-id-3 label.sk-toggleable__label {\n",
       "  cursor: pointer;\n",
       "  display: flex;\n",
       "  width: 100%;\n",
       "  margin-bottom: 0;\n",
       "  padding: 0.5em;\n",
       "  box-sizing: border-box;\n",
       "  text-align: center;\n",
       "  align-items: start;\n",
       "  justify-content: space-between;\n",
       "  gap: 0.5em;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 label.sk-toggleable__label .caption {\n",
       "  font-size: 0.6rem;\n",
       "  font-weight: lighter;\n",
       "  color: var(--sklearn-color-text-muted);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 label.sk-toggleable__label-arrow:before {\n",
       "  /* Arrow on the left of the label */\n",
       "  content: \"▸\";\n",
       "  float: left;\n",
       "  margin-right: 0.25em;\n",
       "  color: var(--sklearn-color-icon);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 label.sk-toggleable__label-arrow:hover:before {\n",
       "  color: var(--sklearn-color-text);\n",
       "}\n",
       "\n",
       "/* Toggleable content - dropdown */\n",
       "\n",
       "#sk-container-id-3 div.sk-toggleable__content {\n",
       "  max-height: 0;\n",
       "  max-width: 0;\n",
       "  overflow: hidden;\n",
       "  text-align: left;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-toggleable__content.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-toggleable__content pre {\n",
       "  margin: 0.2em;\n",
       "  border-radius: 0.25em;\n",
       "  color: var(--sklearn-color-text);\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-toggleable__content.fitted pre {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 input.sk-toggleable__control:checked~div.sk-toggleable__content {\n",
       "  /* Expand drop-down */\n",
       "  max-height: 200px;\n",
       "  max-width: 100%;\n",
       "  overflow: auto;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {\n",
       "  content: \"▾\";\n",
       "}\n",
       "\n",
       "/* Pipeline/ColumnTransformer-specific style */\n",
       "\n",
       "#sk-container-id-3 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator-specific style */\n",
       "\n",
       "/* Colorize estimator box */\n",
       "#sk-container-id-3 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-label label.sk-toggleable__label,\n",
       "#sk-container-id-3 div.sk-label label {\n",
       "  /* The background is the default theme color */\n",
       "  color: var(--sklearn-color-text-on-default-background);\n",
       "}\n",
       "\n",
       "/* On hover, darken the color of the background */\n",
       "#sk-container-id-3 div.sk-label:hover label.sk-toggleable__label {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "/* Label box, darken color on hover, fitted */\n",
       "#sk-container-id-3 div.sk-label.fitted:hover label.sk-toggleable__label.fitted {\n",
       "  color: var(--sklearn-color-text);\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Estimator label */\n",
       "\n",
       "#sk-container-id-3 div.sk-label label {\n",
       "  font-family: monospace;\n",
       "  font-weight: bold;\n",
       "  display: inline-block;\n",
       "  line-height: 1.2em;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-label-container {\n",
       "  text-align: center;\n",
       "}\n",
       "\n",
       "/* Estimator-specific */\n",
       "#sk-container-id-3 div.sk-estimator {\n",
       "  font-family: monospace;\n",
       "  border: 1px dotted var(--sklearn-color-border-box);\n",
       "  border-radius: 0.25em;\n",
       "  box-sizing: border-box;\n",
       "  margin-bottom: 0.5em;\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-0);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-estimator.fitted {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-0);\n",
       "}\n",
       "\n",
       "/* on hover */\n",
       "#sk-container-id-3 div.sk-estimator:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-2);\n",
       "}\n",
       "\n",
       "#sk-container-id-3 div.sk-estimator.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-2);\n",
       "}\n",
       "\n",
       "/* Specification for estimator info (e.g. \"i\" and \"?\") */\n",
       "\n",
       "/* Common style for \"i\" and \"?\" */\n",
       "\n",
       ".sk-estimator-doc-link,\n",
       "a:link.sk-estimator-doc-link,\n",
       "a:visited.sk-estimator-doc-link {\n",
       "  float: right;\n",
       "  font-size: smaller;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1em;\n",
       "  height: 1em;\n",
       "  width: 1em;\n",
       "  text-decoration: none !important;\n",
       "  margin-left: 0.5em;\n",
       "  text-align: center;\n",
       "  /* unfitted */\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted,\n",
       "a:link.sk-estimator-doc-link.fitted,\n",
       "a:visited.sk-estimator-doc-link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "div.sk-estimator:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link:hover,\n",
       ".sk-estimator-doc-link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover,\n",
       "div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,\n",
       ".sk-estimator-doc-link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "/* Span, style for the box shown on hovering the info icon */\n",
       ".sk-estimator-doc-link span {\n",
       "  display: none;\n",
       "  z-index: 9999;\n",
       "  position: relative;\n",
       "  font-weight: normal;\n",
       "  right: .2ex;\n",
       "  padding: .5ex;\n",
       "  margin: .5ex;\n",
       "  width: min-content;\n",
       "  min-width: 20ex;\n",
       "  max-width: 50ex;\n",
       "  color: var(--sklearn-color-text);\n",
       "  box-shadow: 2pt 2pt 4pt #999;\n",
       "  /* unfitted */\n",
       "  background: var(--sklearn-color-unfitted-level-0);\n",
       "  border: .5pt solid var(--sklearn-color-unfitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link.fitted span {\n",
       "  /* fitted */\n",
       "  background: var(--sklearn-color-fitted-level-0);\n",
       "  border: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "\n",
       ".sk-estimator-doc-link:hover span {\n",
       "  display: block;\n",
       "}\n",
       "\n",
       "/* \"?\"-specific style due to the `<a>` HTML tag */\n",
       "\n",
       "#sk-container-id-3 a.estimator_doc_link {\n",
       "  float: right;\n",
       "  font-size: 1rem;\n",
       "  line-height: 1em;\n",
       "  font-family: monospace;\n",
       "  background-color: var(--sklearn-color-background);\n",
       "  border-radius: 1rem;\n",
       "  height: 1rem;\n",
       "  width: 1rem;\n",
       "  text-decoration: none;\n",
       "  /* unfitted */\n",
       "  color: var(--sklearn-color-unfitted-level-1);\n",
       "  border: var(--sklearn-color-unfitted-level-1) 1pt solid;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 a.estimator_doc_link.fitted {\n",
       "  /* fitted */\n",
       "  border: var(--sklearn-color-fitted-level-1) 1pt solid;\n",
       "  color: var(--sklearn-color-fitted-level-1);\n",
       "}\n",
       "\n",
       "/* On hover */\n",
       "#sk-container-id-3 a.estimator_doc_link:hover {\n",
       "  /* unfitted */\n",
       "  background-color: var(--sklearn-color-unfitted-level-3);\n",
       "  color: var(--sklearn-color-background);\n",
       "  text-decoration: none;\n",
       "}\n",
       "\n",
       "#sk-container-id-3 a.estimator_doc_link.fitted:hover {\n",
       "  /* fitted */\n",
       "  background-color: var(--sklearn-color-fitted-level-3);\n",
       "}\n",
       "</style><div id=\"sk-container-id-3\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>GaussianMixture(n_components=3, n_init=10, random_state=42)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator fitted sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" checked><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label fitted sk-toggleable__label-arrow\"><div><div>GaussianMixture</div></div><div><a class=\"sk-estimator-doc-link fitted\" rel=\"noreferrer\" target=\"_blank\" href=\"https://scikit-learn.org/1.6/modules/generated/sklearn.mixture.GaussianMixture.html\">?<span>Documentation for GaussianMixture</span></a><span class=\"sk-estimator-doc-link fitted\">i<span>Fitted</span></span></div></label><div class=\"sk-toggleable__content fitted\"><pre>GaussianMixture(n_components=3, n_init=10, random_state=42)</pre></div> </div></div></div></div>"
      ],
      "text/plain": [
       "GaussianMixture(n_components=3, n_init=10, random_state=42)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gm = GaussianMixture(n_components=3, n_init=10, random_state=42)\n",
    "gm.fit(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91965fbb-57e5-4693-ae1f-76fb4a5099dc",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "Let's look at the parameters of the GMM estimated:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "b1397df2-f564-431d-8999-d0335f9b1bfc",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:23.904988Z",
     "iopub.status.busy": "2025-03-07T05:32:23.904805Z",
     "iopub.status.idle": "2025-03-07T05:32:23.909338Z",
     "shell.execute_reply": "2025-03-07T05:32:23.908881Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.40005972, 0.20961444, 0.39032584])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gm.weights_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "4e58e66f-59dd-4068-8fb2-4f52f9c43b7c",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:23.911681Z",
     "iopub.status.busy": "2025-03-07T05:32:23.911504Z",
     "iopub.status.idle": "2025-03-07T05:32:23.915938Z",
     "shell.execute_reply": "2025-03-07T05:32:23.915368Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-1.40764129,  1.42712848],\n",
       "       [ 3.39947665,  1.05931088],\n",
       "       [ 0.05145113,  0.07534576]])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gm.means_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "d3d1a385-4db6-49d6-aa12-0adc52725905",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:23.918013Z",
     "iopub.status.busy": "2025-03-07T05:32:23.917721Z",
     "iopub.status.idle": "2025-03-07T05:32:23.922273Z",
     "shell.execute_reply": "2025-03-07T05:32:23.921776Z"
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[ 0.63478217,  0.72970097],\n",
       "        [ 0.72970097,  1.16094925]],\n",
       "\n",
       "       [[ 1.14740131, -0.03271106],\n",
       "        [-0.03271106,  0.95498333]],\n",
       "\n",
       "       [[ 0.68825143,  0.79617956],\n",
       "        [ 0.79617956,  1.21242183]]])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gm.covariances_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25d8790e-4879-416b-9255-b152fdbba228",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "806fbeb6-33d9-4c4b-85f4-a8d016832704",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "The model can be used to predict which cluster each instance belongs to (hard clustering) or to estimate probabilities that it came from each cluster. \n",
    "\n",
    "For this, just use `predict()` method or the `predict_proba()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "c1514c98-36c0-4c41-8e66-013b74f0ef7d",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:23.924979Z",
     "iopub.status.busy": "2025-03-07T05:32:23.924161Z",
     "iopub.status.idle": "2025-03-07T05:32:23.928952Z",
     "shell.execute_reply": "2025-03-07T05:32:23.928484Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2, 2, 0, ..., 1, 1, 1])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gm.predict(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "043027db-4724-4eb2-af6a-fa520086dbe0",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:23.930889Z",
     "iopub.status.busy": "2025-03-07T05:32:23.930714Z",
     "iopub.status.idle": "2025-03-07T05:32:23.935904Z",
     "shell.execute_reply": "2025-03-07T05:32:23.935097Z"
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.   , 0.023, 0.977],\n",
       "       [0.001, 0.016, 0.983],\n",
       "       [1.   , 0.   , 0.   ],\n",
       "       ...,\n",
       "       [0.   , 1.   , 0.   ],\n",
       "       [0.   , 1.   , 0.   ],\n",
       "       [0.   , 1.   , 0.   ]])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gm.predict_proba(X).round(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83733bbf-e1a9-41a1-9dee-112c70e82593",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Generative model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e024b7b9-e8a1-4b0a-b2d2-bf2a20e121fc",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "It is a generative model, so you can sample new instances from it (and get their labels)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "ae6987db-a0c1-4dbe-ac1c-550b12d37764",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:23.937941Z",
     "iopub.status.busy": "2025-03-07T05:32:23.937765Z",
     "iopub.status.idle": "2025-03-07T05:32:23.944065Z",
     "shell.execute_reply": "2025-03-07T05:32:23.943632Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-2.32491052,  1.04752548],\n",
       "       [-1.16654983,  1.62795173],\n",
       "       [ 1.84860618,  2.07374016],\n",
       "       [ 3.98304484,  1.49869936],\n",
       "       [ 3.8163406 ,  0.53038367],\n",
       "       [ 0.38079484, -0.56239369]])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_new, y_new = gm.sample(6)\n",
    "X_new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "d41f7c1f-6e4f-4f75-a0d3-970b5a6904b1",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:23.946209Z",
     "iopub.status.busy": "2025-03-07T05:32:23.946036Z",
     "iopub.status.idle": "2025-03-07T05:32:23.951024Z",
     "shell.execute_reply": "2025-03-07T05:32:23.950557Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 1, 1, 1, 2])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_new"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be0b4200-83f1-492f-9939-95cee8231fa6",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Probability density estimation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "298a478e-845a-41b7-a5f0-be616ae62184",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Can estimate the log of the _probability density function_ (PDF) at any location using the `score_samples()` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "f63f5892-3af5-4a40-9b09-16a252ea7b70",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:23.953173Z",
     "iopub.status.busy": "2025-03-07T05:32:23.953002Z",
     "iopub.status.idle": "2025-03-07T05:32:23.959035Z",
     "shell.execute_reply": "2025-03-07T05:32:23.958565Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-2.61, -3.57, -3.33, ..., -3.51, -4.4 , -3.81])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gm.score_samples(X).round(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22eeb02b-7f46-4441-973d-50b251e45906",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "A valid probability density must integrate to one over the parameters space.  Let's check!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "137358fd-658c-4a63-a50d-75a6b6cd616f",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:23.961456Z",
     "iopub.status.busy": "2025-03-07T05:32:23.961119Z",
     "iopub.status.idle": "2025-03-07T05:32:25.128194Z",
     "shell.execute_reply": "2025-03-07T05:32:25.127575Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "np.float64(0.9999999999225088)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "resolution = 100\n",
    "grid = np.arange(-10, 10, 1 / resolution)\n",
    "xx, yy = np.meshgrid(grid, grid)\n",
    "X_full = np.vstack([xx.ravel(), yy.ravel()]).T\n",
    "\n",
    "pdf = np.exp(gm.score_samples(X_full))\n",
    "pdf_probas = pdf * (1 / resolution) ** 2\n",
    "pdf_probas.sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c159cf5-a9e1-4d04-bfd5-6784a15180ec",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "### Visualise"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "578a9285-d07e-4f08-8659-816353416937",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "Let's plot the resulting probability densities and decision boundaries for clustering (dashed lines)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "fce7a3dc-9e32-4eb4-bb5f-7ebdb21ef90f",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:25.130371Z",
     "iopub.status.busy": "2025-03-07T05:32:25.130180Z",
     "iopub.status.idle": "2025-03-07T05:32:26.007076Z",
     "shell.execute_reply": "2025-03-07T05:32:26.006418Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 136112 characters]",
      "text/plain": [
       "<Figure size 800x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib.colors import LogNorm\n",
    "\n",
    "def plot_gaussian_mixture(clusterer, X, resolution=1000, show_ylabels=True):\n",
    "    mins = X.min(axis=0) - 0.1\n",
    "    maxs = X.max(axis=0) + 0.1\n",
    "    xx, yy = np.meshgrid(np.linspace(mins[0], maxs[0], resolution),\n",
    "                         np.linspace(mins[1], maxs[1], resolution))\n",
    "    Z = -clusterer.score_samples(np.c_[xx.ravel(), yy.ravel()])\n",
    "    Z = Z.reshape(xx.shape)\n",
    "\n",
    "    plt.contourf(xx, yy, Z,\n",
    "                 norm=LogNorm(vmin=1.0, vmax=30.0),\n",
    "                 levels=np.logspace(0, 2, 12))\n",
    "    plt.contour(xx, yy, Z,\n",
    "                norm=LogNorm(vmin=1.0, vmax=30.0),\n",
    "                levels=np.logspace(0, 2, 12),\n",
    "                linewidths=1, colors='k')\n",
    "\n",
    "    Z = clusterer.predict(np.c_[xx.ravel(), yy.ravel()])\n",
    "    Z = Z.reshape(xx.shape)\n",
    "    plt.contour(xx, yy, Z,\n",
    "                linewidths=2, colors='r', linestyles='dashed')\n",
    "    \n",
    "    plt.plot(X[:, 0], X[:, 1], 'k.', markersize=2)\n",
    "    plot_centroids(clusterer.means_, clusterer.weights_)\n",
    "\n",
    "    plt.xlabel(\"$x_1$\")\n",
    "    if show_ylabels:\n",
    "        plt.ylabel(\"$x_2$\", rotation=0)\n",
    "    else:\n",
    "        plt.tick_params(labelleft=False)\n",
    "\n",
    "plt.figure(figsize=(8, 4))\n",
    "plot_gaussian_mixture(gm, X)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11feb53b-8b19-4390-8d09-8e5ecb565e97",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Anomaly detection with Gaussian mixture models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cf47de9-0cc0-4a7e-84a4-4a0f40acde21",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "GMMs can be used for *anomaly detection*.\n",
    "\n",
    "Instances located in low density regions can be considered anomalies. \n",
    "\n",
    "Must define what density threshold constitutes an anomaly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "12e7c124-913e-42a6-8e3a-23e586284b52",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:26.009077Z",
     "iopub.status.busy": "2025-03-07T05:32:26.008900Z",
     "iopub.status.idle": "2025-03-07T05:32:26.013481Z",
     "shell.execute_reply": "2025-03-07T05:32:26.012915Z"
    },
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "densities = gm.score_samples(X)\n",
    "density_threshold = np.percentile(densities, 1) # Find threshold corresponding to 1% of instances\n",
    "anomalies = X[densities < density_threshold]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "009242db-3837-403f-ae46-8bcb40c50e9d",
   "metadata": {
    "editable": true,
    "execution": {
     "iopub.execute_input": "2025-03-07T05:32:26.015220Z",
     "iopub.status.busy": "2025-03-07T05:32:26.015054Z",
     "iopub.status.idle": "2025-03-07T05:32:26.907039Z",
     "shell.execute_reply": "2025-03-07T05:32:26.906342Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 138296 characters]",
      "text/plain": [
       "<Figure size 800x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(8, 4))\n",
    "\n",
    "plot_gaussian_mixture(gm, X)\n",
    "plt.scatter(anomalies[:, 0], anomalies[:, 1], color='r', marker='*')\n",
    "plt.ylim(top=5.1);"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
