{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true,
    "slideshow": {
     "slide_type": ""
    },
    "tags": []
   },
   "source": [
    "# Coursework\n",
    "# SPCE0038: Machine Learning with Big-Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "This coursework is provided as a Jupyter notebook, which you will need to complete.  \n",
    "\n",
    "Throughout the notebook you will need to complete code, analytic exercises (if equations are required please typeset your solutions using latex in the markdown cell provided) and descriptive answers. Much of the grading of the coursework will be performed automatically, so it is critical you name your variables as requested."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before you turn this coursework in, make sure everything runs as expected. First, **restart the kernel** (in the menubar, select Kernel$\\rightarrow$Restart) and then **run all cells** (in the menubar, select Cell$\\rightarrow$Run All).\n",
    "\n",
    "Make sure you fill in any place that says \"YOUR ANSWER HERE\" or `YOUR CODE HERE` and remove remove the `raise NotImplementedError()` exceptions that are thrown before you have added your answers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please also:\n",
    "- Make sure you use a python environment using the `requirements.txt` files provided by the course.\n",
    "- Make sure your notebook executes without errors.\n",
    "- Do not add and remove cells but only provide your answers in the spaces given.\n",
    "- Do not add or change code in the cells other than the ones marked with `# YOUR CODE HERE`.\n",
    "- Do not overwrite or rename any existing variables.\n",
    "- Do not install code or packages in the notebooks.\n",
    "- Do not import any libraries other than modules from `sklearn`.\n",
    "- Always label your plots.\n",
    "- Answer the questions concisely and show your work/derivations/reasoning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Please rename the notebook filename to include your candidate number in the filename.  And please also add your candidate number below:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "CANDIDATE_NUMBER = \"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will be able to run some basic tests in the notebook to check the basic operation of your code is as expected.  Although do not assume your responses are complete or fully correct just because the basic tests pass."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you have renamed the notebook file and completed the exercises, please upload the notebook to Moodle.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## AstroML\n",
    "\n",
    "The data used is this coursework is obtained using [AstroML](http://www.astroml.org), a python package for machine learning for astronomy.  Although we take data from AstroML, this coursework is not based on standard AstroML examples.  So you will *not* find the solutions in AstroML examples!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SDSS\n",
    "\n",
    "The data obtained through AstroML was observed by the [Sloan Digital Sky Survey](https://www.sdss.org/) (SDSS), which began observations in 2000.  SDSS data have lead to many scientific advances and the experiment is widely seen as one of the most successful surveys in the history of astronomy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dependencies\n",
    "\n",
    "- Standard course dependencies (e.g. numpy, scikit-learn, etc.)\n",
    "- [AstoML](http://www.astroml.org)\n",
    "- [AstroPy](http://www.astropy.org/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "ebf563e5f38beef45736bc1921c6c8ca",
     "grade": false,
     "grade_id": "cell-60b5947d6f57e1e5",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "62f0bf3c5af939aa8d31f358766e54f1",
     "grade": false,
     "grade_id": "cell-ea880cd0d16868fc",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "def check_var_defined(var):\n",
    "    try:\n",
    "        exec(var)\n",
    "    except NameError:\n",
    "        raise NameError(var + \" not defined.\")\n",
    "    else:\n",
    "        print(var + \" defined.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "15eddda2a8295d028fd507afa72dce3b",
     "grade": false,
     "grade_id": "cell-b2775724006d2978",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "## Part 1: Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "a9cff16a3014318dbe8e070dc4dac087",
     "grade": false,
     "grade_id": "cell-b3bc03ae580e8edb",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "In these exercises we will consider the regression problem of the astonomical distance modulus vs redshift relationship.\n",
    "\n",
    "In astronomy, the [distance modulus](https://en.wikipedia.org/wiki/Distance_modulus) specifies the difference between the apparent and absolute magnitudes of an astronomnical object.  It provides a way of expressing astrophysical distances. \n",
    "\n",
    "Astronomical [redshift](https://en.wikipedia.org/wiki/Redshift) specifies the shift in wavelength that astronomical objects undergo due to the expansion of the Universe.  Due to Hubble's Law, more distance objects experience a greater redshift.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "a6a25d04743d3fe93a33d1b4411a8516",
     "grade": false,
     "grade_id": "cell-72d05aca43e8358d",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "from astroML.datasets import generate_mu_z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "169eb83a4ad31a60a51d6de794445254",
     "grade": false,
     "grade_id": "cell-b5b33781a14baffd",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Load data\n",
    "m = 150\n",
    "z_sample, mu_sample, dmu = generate_mu_z(m, random_state=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "208758e42789bf2d1470c30a5fb356e0",
     "grade": false,
     "grade_id": "cell-acd86dc17cc2dd53",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Plot the distance modulus ($\\mu$) vs redhift ($z$), including error bars.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "a6759ec46b4d7447c6b0873e63a52ec2",
     "grade": true,
     "grade_id": "cell-ae7f16835ab51af3",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# Plot data\n",
    "def plot_dist_mod():\n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()\n",
    "    plt.xlabel('$z$')\n",
    "    plt.ylabel('$\\mu$')\n",
    "    plt.title('Distance modulus vs redshift')\n",
    "    plt.ylim(36, 50)\n",
    "    plt.xlim(0, 1.5)\n",
    "plot_dist_mod()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "596c8005f2265096bb8d4dcfb6756856",
     "grade": false,
     "grade_id": "cell-cdbb1766c5be9049",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "Recall the normal equations for linear regression follow by analytically minimising the cost function: \n",
    "\n",
    "$$\\min_\\theta\\ C(\\theta) = \\min_\\theta \\ (X \\theta - y)^{\\rm T}(X \\theta - y).$$\n",
    "\n",
    "Show analytically that the solution is given by \n",
    "\n",
    "$$ \\hat{\\theta} = \\left( X^{\\rm T} X \\right)^{-1} X^{\\rm T} y. $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "c509e0ca35e2e737d727917ad9699ad8",
     "grade": false,
     "grade_id": "cell-3b18a50412c27c56",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "[Matrix calculus identities](https://en.wikipedia.org/wiki/Matrix_calculus) may be useful (note that we use the denominator layout convention)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "3a99014a4147c496f8f9de0535f06e6a",
     "grade": false,
     "grade_id": "cell-4701b02e60d7683c",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Expand the cost function and drop terms that do not depend on $\\theta$ (use latex mathematics expressions):*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "b97fe78f2b4bc7502cb9ebdfe1d17874",
     "grade": true,
     "grade_id": "cell-a114fd93ba74dac5",
     "locked": false,
     "points": 3,
     "schema_version": 3,
     "solution": true
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "2e5cd989e6893a3cefd6a3020e615c8d",
     "grade": false,
     "grade_id": "cell-9a33de31635ab257",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Calculate the derivative, set it to zero, and solve for $\\theta$ (use latex mathematics expressions):*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "72da939ee35de54049492e263cc444f8",
     "grade": true,
     "grade_id": "cell-0f0b521f765f4826",
     "locked": false,
     "points": 3,
     "schema_version": 3,
     "solution": true
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "37c7febbf20f95d2bd6ef750ed2d11f3",
     "grade": false,
     "grade_id": "cell-dd5113a0fc89a960",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Solve for $\\theta$ by numerically implementing the analytic solution given above.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "0993e1f7ab7124438012231f05c51e9a",
     "grade": false,
     "grade_id": "cell-d71c2644693323b2",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "def compute_theta_lin_reg(X, y):\n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()\n",
    "    return theta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "ce1006c8dcbd4682472e45a7a6e54960",
     "grade": true,
     "grade_id": "cell-f024710582a10726",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "assert compute_theta_lin_reg(z_sample, mu_sample).shape == (2,)\n",
    "theta = compute_theta_lin_reg(z_sample, mu_sample)\n",
    "(theta_c, theta_m) = theta\n",
    "print(\"Linear regression parameters recovered analytically: intercept={0:.4f}, slope={1:.4f}\".format(theta_c, theta_m))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "0276019b59b620bff4ba748f13afb83e",
     "grade": true,
     "grade_id": "cell-52c5a2bcccb2010d",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('theta_c')\n",
    "check_var_defined('theta_m')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "96ed9fc16f94ad4965a2b03c463b7cd7",
     "grade": false,
     "grade_id": "cell-883f0e6586934a9f",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Write a method to make a prediction for a given redshift.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "475f9a392bb5bb1dee3671f1e4e4c5cf",
     "grade": false,
     "grade_id": "cell-f5341cc7da485877",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "def predict_lin_reg(theta, x):\n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()\n",
    "    return y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "3cfc6f1f228d2759b5869de17f4c5754",
     "grade": false,
     "grade_id": "cell-c71c338c46a8bb93",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Predict the distance modulus for a range of redshift values between 0.01 and 1.5 and plot the predicted curve overlayed on data (make a new plot; do not revise the plot above).  Call the variable used to store the predictions for your polynomial model `mu_pred_lin`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "cd987dfd33566ad55e8f9bfe8f8997aa",
     "grade": false,
     "grade_id": "cell-fc6aa680d6528ab1",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "z = np.linspace(0.01, 1.5, 1000)\n",
    "plot_dist_mod()\n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "08cf8664e67c3940e4b3acc27b0d325e",
     "grade": true,
     "grade_id": "cell-f8e904006f960080",
     "locked": true,
     "points": 2,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('mu_pred_lin')\n",
    "assert mu_pred_lin.shape == (len(z),), \"Make sure the shape of your predictions is correct\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "78269320b540a1d943b94e85f8887bdf",
     "grade": false,
     "grade_id": "cell-c55d51874c74517a",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Solve for the parameters $\\theta$ using Scikit-Learn.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "dcb822e86c7cfb2cc413af696ebf6b12",
     "grade": false,
     "grade_id": "cell-21b18d58a127b96c",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "lin_reg = LinearRegression()\n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "2dc255e53a5d0c973dfca577078d77c7",
     "grade": true,
     "grade_id": "cell-7ac2cec6a7505062",
     "locked": true,
     "points": 2,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "assert lin_reg.coef_.shape == (1,), \"Make sure your features have the right shape, such that we have 1 fitted coefficient\"\n",
    "print(\"Linear regression parameters recovered by scikit-learn: intercept={0:.4f}, slope={1:.4f}\"\n",
    "      .format(lin_reg.intercept_, lin_reg.coef_[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "7e98e11722002c12cbf55f1e5039e6dd",
     "grade": false,
     "grade_id": "cell-59443e08cb021f49",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Extend your model to include polynomial features up to degree 15 (using Scikit-Learn).  Use variable `lin_reg_poly` for your revised model.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "c48074367c1bb4a2704f931153372c1b",
     "grade": false,
     "grade_id": "cell-8ea9ea3be9d57ce5",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "degree = 15\n",
    "bias = False\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "def compute_poly_features(degree, bias):\n",
    "    # Return polynomial features of samples and class\n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()\n",
    "    return z_sample_poly, poly_features\n",
    "z_sample_poly, poly_features = compute_poly_features(degree, bias)\n",
    "# Train model\n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "01cfbed87f8a26ed0ec8be34e437801a",
     "grade": true,
     "grade_id": "cell-d46184658a6b600b",
     "locked": true,
     "points": 2,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('lin_reg_poly')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "735fd8a02427e9a36d74685def71342c",
     "grade": false,
     "grade_id": "cell-5ccd955a752b81ea",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Plot the data and the predictions of your models considered so far (linear and polynomial regression).  Call the variable used to store the predictions for your polynomial model `mu_pred_poly`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "dd2a4e0385cd8ed4673e4aad5b632280",
     "grade": false,
     "grade_id": "cell-24f6bf2cfd9a7f71",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "11efe12f85bf5edc3719a470150c480e",
     "grade": true,
     "grade_id": "cell-55abac99185aa2df",
     "locked": true,
     "points": 2,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('mu_pred_poly')\n",
    "assert mu_pred_poly.shape == (len(z),)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "0dfd352eae6049c8302271e9173b5b86",
     "grade": false,
     "grade_id": "cell-dc45782c437fb3b3",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Comment on the accuracy of your models.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "7a9b6ddcab190716cd17c327abb8da2a",
     "grade": true,
     "grade_id": "cell-c37fd48a9be1bd84",
     "locked": false,
     "points": 2,
     "schema_version": 3,
     "solution": true
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "7b3d585af5547dd71593690acb89f54a",
     "grade": false,
     "grade_id": "cell-642b8f293c37d087",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Think about methods that could be used to improve the performance of your models. Improve your polynomial model and use the improved model to make predictions. Call the variable used to store the polynomial model `ridge_reg_poly`. Call the variable used to store the predictions for your polynomial model `mu_pred_poly_improved`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "809a92a774fec689d2048a896f1f4357",
     "grade": true,
     "grade_id": "cell-249f8b263075ba25",
     "locked": false,
     "points": 3,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "23cbdcf0308d7d2f474ccecf681ddd73",
     "grade": false,
     "grade_id": "cell-5f2d7d6ea319189b",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('ridge_reg_poly')\n",
    "check_var_defined('mu_pred_poly_improved')\n",
    "assert mu_pred_poly_improved.shape == (len(z),), \"Make sure the shape of your predictions is correct\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "8bb3a9403b75f4dc8b3a3e38664cd50a",
     "grade": false,
     "grade_id": "cell-d7ee6a9b415c62a9",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Plot the predictions made with new model and all previous models considered.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "c882d9ba097863626e9c85736d3dab30",
     "grade": true,
     "grade_id": "cell-14a8a955921c3d9c",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "180a54403a99d155688f8a290fc1015b",
     "grade": false,
     "grade_id": "cell-62604dceb287f6e4",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Compute the RMS error between your predictions and the data samples.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "6e268ea0e1117f31d0a367335c3ab18a",
     "grade": false,
     "grade_id": "cell-2ee5e0675e003f12",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# Define a general function to compute the RMS error\n",
    "def compute_rms(mu_1, mu_2):\n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "6f8072de8469e1f06df363cc72011cc8",
     "grade": true,
     "grade_id": "cell-946369d338039825",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "assert np.isclose(compute_rms(mu_pred_lin, mu_pred_lin), 0.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "7621e2e41683215f9f485a1161168a5c",
     "grade": false,
     "grade_id": "cell-9c52009b6ad3fd17",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# Compute the RMS error between the data and the predictions for each model.\n",
    "# Use variables rms_sample_lin, rms_sample_poly and rms_sample_poly_improved.\n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f309e5fc58b77e4cda8bc8f3ee78ce1d",
     "grade": false,
     "grade_id": "cell-579ae5f1089bfb46",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Print RMS values computed.\n",
    "print(\"rms_sample_lin = {0:.4f}\".format(rms_sample_lin))\n",
    "print(\"rms_sample_poly = {0:.4f}\".format(rms_sample_poly))\n",
    "print(\"rms_sample_poly_improved = {0:.4f}\".format(rms_sample_poly_improved))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "7d71e55b0de3ea49b8e285df5a75e860",
     "grade": true,
     "grade_id": "cell-09beb7cceccb40be",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('rms_sample_lin')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "3c12f33d96207e0b339d5a7d7e28146a",
     "grade": true,
     "grade_id": "cell-6000d96ec4b0de43",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('rms_sample_poly')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "c61d68d34d7d1760459d619d00f2a03c",
     "grade": true,
     "grade_id": "cell-06d3a7838b6ad7e7",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('rms_sample_poly_improved')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "d2cfab16a3b881eb9666957d282df2eb",
     "grade": false,
     "grade_id": "cell-a4632d53a04563a6",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Comment on what models you believe are best.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "65df151fe8f001ee61ec5607e15a0b3d",
     "grade": true,
     "grade_id": "cell-d34e017b234e387c",
     "locked": false,
     "points": 2,
     "schema_version": 3,
     "solution": true
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "92e241c8e04b9942db80a362a82b6429",
     "grade": false,
     "grade_id": "cell-d9ef9bfe07f359d0",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "Using our cosmological concordance model we can predict the theoretical distance modulus vs redshift relationship using our understanding of the physics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "9df4b3203c1281eb061f9a3d159457f7",
     "grade": false,
     "grade_id": "cell-19fd3ac1f2d371eb",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "from astroML.cosmology import Cosmology\n",
    "cosmo = Cosmology()\n",
    "mu_cosmo = np.array(list(map(cosmo.mu, z)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "463b8b27c0408d32b519fd702dff0a63",
     "grade": false,
     "grade_id": "cell-22c310e26cfe2d41",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Plot the data, predictions made with all regression models, and the values predicted by the cosmological model.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "27b490f88536997467462e8d6ce09aa7",
     "grade": true,
     "grade_id": "cell-62cecf73f9f9a228",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "b277d71c70a41ca213e62c916b98207d",
     "grade": false,
     "grade_id": "cell-285160edd2e418e2",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Compute the RMS error between the predictions made by the cosmological model and each of the regression models, over the sample array `z`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "0034720cf8c44e319df90de05466ef84",
     "grade": false,
     "grade_id": "cell-939141aa04822e33",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# Compute the RMS error between the data and the predictions for each model.\n",
    "# Use variables rms_cosmo_lin, rms_cosmo_poly and rms_cosmo_poly_improved.\n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f4a351596164e691fb7bb6359e3de55b",
     "grade": false,
     "grade_id": "cell-770a716893bdb639",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Print RMS values computed.\n",
    "print(\"rms_cosmo_lin = {0:.4f}\".format(rms_cosmo_lin))\n",
    "print(\"rms_cosmo_poly = {0:.4f}\".format(rms_cosmo_poly))\n",
    "print(\"rms_cosmo_poly_improved = {0:.4f}\".format(rms_cosmo_poly_improved))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "6f439fe717a169e6d0658974b1dcbc7b",
     "grade": true,
     "grade_id": "cell-e8a9f757965b2069",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('rms_cosmo_lin')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "33956cf40ac1a6064c599cca42bb3520",
     "grade": true,
     "grade_id": "cell-30ed009401260759",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('rms_cosmo_poly')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "4a4fc0aef5b79d11ac43990c71b12d93",
     "grade": true,
     "grade_id": "cell-e6ee872c00472aa2",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('rms_cosmo_poly_improved')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "f67fca72235a566438e6daed262ecc04",
     "grade": false,
     "grade_id": "cell-cfa4d93afc081e93",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Comment on the RMS values computed and the implications for the accuracy of the different regression models considered.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "f25a554b6d3d42a22139116a83b3c1c7",
     "grade": true,
     "grade_id": "cell-a993842383ec9778",
     "locked": false,
     "points": 2,
     "schema_version": 3,
     "solution": true
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "e28b2898e145c2fb9c843928795b54f2",
     "grade": false,
     "grade_id": "cell-c8ac035dcf2c47fc",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "900204de52da9d3c8b85d7604102a518",
     "grade": false,
     "grade_id": "cell-2d2591b309fcc3d2",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "## Part 2: Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "0e5cd46344bce92e33149bb5f42b9485",
     "grade": false,
     "grade_id": "cell-bb13c563fd6784c9",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "In these exercises we will consider classification of [RR Lyrae](https://en.wikipedia.org/wiki/RR_Lyrae_variable) variable stars.  RR Lyrae variables are often used as standard candles to measure astronomical distances since their period of pulsation can be related to their absolute magnitude.\n",
    "\n",
    "Observations of star magnitudes are made in each [SDSS filter band](http://skyserver.sdss.org/dr2/en/proj/advanced/color/sdssfilters.asp): u, g, r, i, z.\n",
    "\n",
    "We will consider the space of astronomical \"colours\" to distinguish RR Lyraes from background stars.  Astronomical colours are simply differences in magnitudes between bands, e.g. u-g, g-r, r-i, i-z.  You can find further background [here](https://en.wikipedia.org/wiki/Color%E2%80%93color_diagram)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "c8a74d7f460b94e02dde82bb72bb5eaf",
     "grade": false,
     "grade_id": "cell-4f6b1f1dc074f5cc",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "First, download the data.  (This may take some time on first execution.  Subsequently executions will read from cached data on your system.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "df9d97d24e9496cc877839350420247b",
     "grade": false,
     "grade_id": "cell-73597701131bc8e2",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "# Load data\n",
    "from astroML.datasets import fetch_rrlyrae_combined\n",
    "X, y = fetch_rrlyrae_combined()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "8bf835a66bdebf03f21f25c2038156eb",
     "grade": false,
     "grade_id": "cell-2b739257efd6fbdf",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "You can learn more about the format of the returned data [here](http://www.astroml.org/modules/generated/astroML.datasets.fetch_rrlyrae_combined.html).  In particular, note that the columns of `X` are u-g, g-r, r-i, i-z."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "cbba628ec0398b2d517363e7ba5b669b",
     "grade": false,
     "grade_id": "cell-1d6b876f89c05942",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Construct a Pandas DataFrame for the `X` data and a Series for the `y` data.  Call your Pandas objects `X_pd` and `y_pd` respectively.*\n",
    "\n",
    "Be sure to give your colums the correct colour name, e.g. 'u-g'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "17f47902f34b6b5623f6d387902a536e",
     "grade": false,
     "grade_id": "cell-7250404d5b9e0c13",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "cols=['u-g', 'g-r', 'r-i', 'i-z']\n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "32251036f16850a2b13b1f78e0cf6ca8",
     "grade": true,
     "grade_id": "cell-a913d8acabdfba5c",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('X_pd')\n",
    "print(X_pd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "1a57773c7e968f9c71d9be3229d9e2a8",
     "grade": false,
     "grade_id": "cell-0b438d5f8dcac8e9",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "097b3a38b8aec4ddf4a649990924e628",
     "grade": true,
     "grade_id": "cell-d1392b89a707b35a",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('y_pd')\n",
    "print(y_pd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "a2a814f95845ae40b8d2e46d3f3b02d2",
     "grade": false,
     "grade_id": "cell-dba1c66617cc789e",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Combine your data and targets into a single Pandas DataFrame, labelling the target column 'target'.  Call the resulting Pandas DataFrame `X_pd_all`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "c9bc7cc9a6cb4270148403c68e9cd022",
     "grade": false,
     "grade_id": "cell-f80ca9f2573d06fb",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "9b150964210efff48af195349090fff3",
     "grade": true,
     "grade_id": "cell-694c65584675c6c6",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('X_pd_all')\n",
    "print(X_pd_all)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "2a59e1592037847daee2a1d59e66e82e",
     "grade": false,
     "grade_id": "cell-b2cdd1e8ff443c4f",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Add a 'target description' column to your existing `X_pd_all` DataFrame, with fields 'Background' and 'RR Lyrae' to specify the target type.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "0175b69fda7663e0c7f1e816703b8273",
     "grade": false,
     "grade_id": "cell-f94161f729fadf8c",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "9b02148198742613e8a4ca524e073b71",
     "grade": true,
     "grade_id": "cell-cb2480e79d82c641",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "print(X_pd_all)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "696a530a5fa11fd74e886aad53c4730a",
     "grade": false,
     "grade_id": "cell-add4e81373265098",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*How many RR Lyrae variable stars are there in the dataset (i.e compute `n_rrlyrae`)?*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "9e45a92c9efcd4a5dc60ae91cfaece89",
     "grade": false,
     "grade_id": "cell-753a59a39e3df18c",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "e4fa379f2a74a6855884a165bc422e87",
     "grade": true,
     "grade_id": "cell-c7fa425ec227dd04",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('n_rrlyrae')\n",
    "print(\"n_rrlyrae = {0}\".format(n_rrlyrae))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "8836626b58900a14fc8d5cca59c33d20",
     "grade": false,
     "grade_id": "cell-a267bf2d5be875a6",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*How many background stars are there in the dataset (i.e. compute `n_background`)?*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "9390ffb5aa4c435a65609e8d9fbbea17",
     "grade": false,
     "grade_id": "cell-f902e74120d04b39",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "766f1ee43bdca36d9c5b2eba3f20be3e",
     "grade": true,
     "grade_id": "cell-dd77ae406ebc1e36",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('n_background')\n",
    "print(\"n_background = {0}\".format(n_background))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "0ad0f46079adada0deefc29a592e1c23",
     "grade": false,
     "grade_id": "cell-494facc20b7778b6",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Plot scatter plot pairs for all colour combinations using `seaborn`.  Colour the points by target type. Make sure the distribution plots are normalised to have an area of 1 under the curve for each of the classes.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "2198169c65059d8e1fd6d52582651c9f",
     "grade": false,
     "grade_id": "cell-ef57ad845334bbaa",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import seaborn as sns; sns.set()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "9a5b0a8a06c988cba3bac3fb41ee6ca6",
     "grade": true,
     "grade_id": "cell-6f8c0ce750628d0e",
     "locked": false,
     "points": 2,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "cf6915bf6aa3ce7f12e619edc09b06e1",
     "grade": false,
     "grade_id": "cell-149d6b589054b26b",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "Let's separate the data into training and test sets, keeping 25% of the data for testing.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "386593a707ee1226d8721ea92970f653",
     "grade": false,
     "grade_id": "cell-22b31f7602338d7f",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "55ce997145aece349fa6ae7401d4cb1d",
     "grade": false,
     "grade_id": "cell-34fc23b040e948f7",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "First let's consider 1D classification for the zeroth colour, i.e. $u-g$. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "7c77a8433e4898de59b775624cf05767",
     "grade": false,
     "grade_id": "cell-53b81bcac2b85a55",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "ind = 0\n",
    "col=cols[ind]\n",
    "col"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "b3578a7b38fba9d5a52458bf8df86c7f",
     "grade": false,
     "grade_id": "cell-7a87a3946325c16c",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "X_train_1d = X_train[:, ind]\n",
    "X_train_1d = X_train_1d.reshape(-1,1)\n",
    "X_test_1d = X_test[:, ind]\n",
    "X_test_1d = X_test_1d.reshape(-1,1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "838b5dad42b1e778c11cd408b6987daf",
     "grade": false,
     "grade_id": "cell-bb6c2985470ef60d",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "To get some further intuition about the 1D classiciation problem consider a 1D plot of\n",
    "class against colour."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "c32035c4ae5b83d6b7f73e39bda0d7a6",
     "grade": false,
     "grade_id": "cell-aac19aaa4019fefc",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "def plot_scatter():\n",
    "    plt.figure(figsize=(10,5))\n",
    "    plt.scatter(X_train_1d[y_train==1], y_train[y_train==1], c='m', marker='^', label='RR Lyrae')\n",
    "    plt.scatter(X_train_1d[y_train==0], y_train[y_train==0], c='c', marker='v', label='Background')\n",
    "    plt.xlabel('$' + col + '$')\n",
    "    plt.ylabel('Probability of type RR Lyrae')\n",
    "plot_scatter()    \n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "e9cb694a42ffb3138b18c2e405645ce1",
     "grade": false,
     "grade_id": "cell-bd01fa3c7086288f",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Given the plot shown above, comment on how well you expect logistic regression to perform.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "cf842b131e1b494992c17edc41569b9d",
     "grade": true,
     "grade_id": "cell-1cad643fb7816037",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "639ffbefef37614d5671b1df92842700",
     "grade": false,
     "grade_id": "cell-00dca71454bb5330",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Where would you guess the decision bounary should lie?  Set the variable `decision_boundary_guess` to your guess.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "1f8f91ff116877b36b5f768c4e7bbf53",
     "grade": false,
     "grade_id": "cell-5eef717d4fab1828",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "59d1034042c8531f42243eb3de1e6de7",
     "grade": true,
     "grade_id": "cell-00ef975a7880050f",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('decision_boundary_guess')\n",
    "print(\"decision_boundary_guess = {0:.4f}\".format(decision_boundary_guess))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "f6e3c3c0ceece7871bf22c9255a51295",
     "grade": false,
     "grade_id": "cell-4c704d78b7b22e68",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "Use Scikit-Learn to perform logistic regression to classify the two classes for this 1D problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "1acac14798c5dc7542fc6ea9406938bc",
     "grade": false,
     "grade_id": "cell-618989081fddad31",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "First, set the inverse regularation strength `C` such that regularisation is effecitvely not performed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "58f98ba91242ad172c6817655bedd385",
     "grade": false,
     "grade_id": "cell-d7b94ebcadcc6111",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "C = 1e10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "c9845c6ff70b8fbeaf82ce21e36fc34b",
     "grade": false,
     "grade_id": "cell-70bcad6835ff868d",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Second, fit the model using Scikit-Learn. Use the variable `clf` for your classification model.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "c1620a833ae3b05de4b1a57a1ce9ab03",
     "grade": false,
     "grade_id": "cell-f1790c24720c07d8",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "4af20705b1b5e8ba2ccb810b4ac4a8f6",
     "grade": true,
     "grade_id": "cell-f6edcb4e5f610518",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('clf')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "8241cadb1bf5fa6ca1e650cf83a16181",
     "grade": false,
     "grade_id": "cell-1aafef5deaf49404",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Compute the decision boundary of the logistic regression model fitted by Scikit-Learn.  User variable `decision_boundary_sklearn` for your result.*\n",
    "\n",
    "(Ensure your result is a scalar and not an array of length 1.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "3ada1a115b287bb9f88afb2c54886805",
     "grade": false,
     "grade_id": "cell-01cd8a3ebc69de43",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "d8f8933baaf6b9eec389763501544d46",
     "grade": true,
     "grade_id": "cell-0ed39065189e2fae",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "assert not hasattr(decision_boundary_sklearn, \"__len__\")\n",
    "print(\"decision_boundary_sklearn = {0:.4f}\".format(decision_boundary_sklearn))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "61747e5a45a3f44ed6fecb3d4840739f",
     "grade": false,
     "grade_id": "cell-b634a6057f675df8",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Evaluate the probabilities predicted by your logistic regression model over the domain specified by the variable `X_1d_new`. Use variable `y_1d_proba` for your computed probabilities.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "713631c608bc415a882d456ebd580a72",
     "grade": false,
     "grade_id": "cell-b3f7cdf5d4698ad6",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "X_1d_new = np.linspace(0.3, 2.0, 1000).reshape(-1, 1)\n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "8d32cd84ae52ee452b0b5d3d026c81e7",
     "grade": true,
     "grade_id": "cell-bb76289d4e36fcb0",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('y_1d_proba')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "d257001f04351ccc61372e6e3ef4ed24",
     "grade": false,
     "grade_id": "cell-84d06c82f79d0657",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Plot the probability of a star being of type RR Lyrae against the colour variable considered.  Also plot the probability of being a Background star.  Overlay these plots on the scatter plot of class types.  Also plot the decision boundary that you guessed previously and the one computed by Scikit-Learn.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "9406aeb0f224d10e25c85c23229913ed",
     "grade": true,
     "grade_id": "cell-1c623b6df631aa69",
     "locked": false,
     "points": 3,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "plot_scatter()\n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "38c50ceb30eb397312f8c8781e61c8ce",
     "grade": false,
     "grade_id": "cell-47b93e984622610a",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*From inspection of your plot, how would all objects in the training set be classified?*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "935b844dc811a7bb002362530ea1382c",
     "grade": true,
     "grade_id": "cell-dff437e03665f571",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "57c4ff384e0f250284b57a31fdd4d676",
     "grade": false,
     "grade_id": "cell-8bd241aeb91446bd",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Use your logistic regression model fitted by Scikit-Learn to predict the class of all objects in the test set. Use variable `y_test_1d_pred` to specify your answer.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "2c6b24344bcddc3f7b486346219e61af",
     "grade": false,
     "grade_id": "cell-bf444b0d8690c876",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "78f7aec829aca85dea759adc10e7c1dc",
     "grade": true,
     "grade_id": "cell-d69905ed477cb96f",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('y_test_1d_pred')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "a6947196762b67730c6fc399918e0f5b",
     "grade": false,
     "grade_id": "cell-71d78cb3b65a5d2d",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*How many objects are classified as of type RR Lyrae?  Use variable `n_rrlyrae_pred` to specify your answer.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "8346a9ea30670eef34617e29278b7e57",
     "grade": false,
     "grade_id": "cell-16f880b76044c462",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "63011f49012188bb2c43c31b5583d0ab",
     "grade": true,
     "grade_id": "cell-357fb80562d278c5",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('n_rrlyrae_pred')\n",
    "assert n_rrlyrae_pred % 1 == 0 # check integer\n",
    "print(\"n_rrlyrae_pred = {0}\".format(n_rrlyrae_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "60515433771f989e3f319af605095ce3",
     "grade": false,
     "grade_id": "cell-1c52ae8c8d62b5c1",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*How many objects are classified as of type Background?  Use variable `n_background_pred` to specify your answer.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "d27d505271b0382875458a6a858a5a99",
     "grade": false,
     "grade_id": "cell-ba43adb513abebcd",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "92dc794912b509ba3c7666f5adfc5121",
     "grade": true,
     "grade_id": "cell-5280ae78f8605c96",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('n_background_pred')\n",
    "assert n_background_pred % 1 == 0 # check integer\n",
    "print(\"n_background_pred = {0}\".format(n_background_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "5a4bf6f4b7b42873c50a01847c6a3ccd",
     "grade": false,
     "grade_id": "cell-8c852f5f04910102",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "Let's check the Scikit-Learn result by solving the logistic regression problem (without regularisation) manually."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "80cfc52c440f87820d3df9ce756d0420",
     "grade": false,
     "grade_id": "cell-297b86c040caaaa6",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "Recall that the cost function for logistic regression is given by\n",
    "$$\n",
    "C(\\theta) = -\\frac{1}{m} \\sum_{i=1}^m \n",
    "\\left [ \n",
    "y^{(i)} \\log(\\hat{p}^{(i)})\n",
    "+\n",
    "(1 - y^{(i)}) \\log(1 - \\hat{p}^{(i)})\n",
    "\\right],\n",
    "$$\n",
    "\n",
    "\n",
    "where\n",
    "\n",
    "$$\\hat{p} = \\sigma(\\theta^\\text{T} x) = \\frac{1}{1+\\exp{(-\\theta^\\text{T} x)}}. $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "0be2a2a142b58454e9eee6c6a629faf3",
     "grade": false,
     "grade_id": "cell-4f5546691b1d0ce4",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "Show analytically that the derivative of the cost function is given by\n",
    "$$\\begin{eqnarray}\n",
    "\\frac{\\partial C}{\\partial \\theta} \n",
    "&=& \n",
    "\\frac{1}{m} \\sum_{i=1}^m \n",
    "\\left[ \\sigma\\left(\\theta^{\\rm T} x^{(i)} \\right) - y^{(i)} \\right]\n",
    "x^{(i)}\\\\\n",
    "&=&\n",
    "\\frac{1}{m} \n",
    "X^{\\rm T}\n",
    "\\left[ \\sigma\\left(X \\theta \\right) - y \\right]\n",
    "\\end{eqnarray}$$\n",
    "\n",
    "(use latex mathematics expressions)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "aac5b115296dd81b6f6b134cdbfc8ab9",
     "grade": false,
     "grade_id": "cell-e9f16916c6a0b264",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*First, simplify the cost function terms $\\log(\\hat{p})$ and $\\log(1-\\hat{p})$ to express in terms linear in $\\log\\left({1+{\\rm e}^{-\\theta^{\\rm T}x}}\\right)$.*\n",
    "\n",
    "(You may drop $i$ superscripts for notational brevity.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "460abf3f41632681b67e865162d4cef3",
     "grade": true,
     "grade_id": "cell-5fc3a8343ec24488",
     "locked": false,
     "points": 2,
     "schema_version": 3,
     "solution": true
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "460d773d236b0ccb40e769762703284e",
     "grade": false,
     "grade_id": "cell-171037df1a01a3f4",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Next, substitute these terms into the cost function and simplify to also express the cost function in terms linear in $\\log\\left({1+{\\rm e}^{-\\theta^{\\rm T}x}}\\right)$.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "e9a37a1c8dc9250fa7070f97ba5adf58",
     "grade": true,
     "grade_id": "cell-dcdb0de863dc8931",
     "locked": false,
     "points": 2,
     "schema_version": 3,
     "solution": true
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "e1fdb7cfcd2cb7f1b88deb585ec01297",
     "grade": false,
     "grade_id": "cell-6f608ac000ec6c3b",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Now compute the derivative of the cost function with respect to variable $\\theta_j$, i.e. compute $\\partial C / \\partial \\theta_j$.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "6d001523ce8324ab2ed170b153b96003",
     "grade": true,
     "grade_id": "cell-c386ea220c086ace",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "0a496a1ff30721c92959260555798802",
     "grade": false,
     "grade_id": "cell-89b9177d7dde5e70",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Combine terms for all $\\theta_j$ to give the overall derivative with respect to $\\theta$, i.e. $\\partial C / \\partial \\theta$.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "d456331004b76307432816c30e8c1fdf",
     "grade": true,
     "grade_id": "cell-331a74ac412db42b",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "54da6d5b63818d49cf17905b39e8bfab",
     "grade": false,
     "grade_id": "cell-722790463e0f0312",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "Using the analytically expression for the derivative of the cost function, we will solve the logistic regression problem by implementing a gradient descent algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "56b427758004808ef8810be4f28e3f57",
     "grade": false,
     "grade_id": "cell-c4321f09bf73ba33",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*First, define the sigmoid function.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "529913786cc0781f24e5c75054c8261b",
     "grade": false,
     "grade_id": "cell-e12fc0aa65b673b1",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "def sigmoid(x):\n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "797ba69ea2672a081856123e1d4cab2f",
     "grade": true,
     "grade_id": "cell-a5a50f4ec07d05fd",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "assert np.isclose(sigmoid(0), 0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "915639b69935959b525f62d5bd0b3c00",
     "grade": false,
     "grade_id": "cell-ba2bb8821f4e75ad",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Next, extend the training data to account for a bias term in your model. Use variable `X_train_1d_b` to specify your result.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "e23f14ea753621e64da00f591268a851",
     "grade": false,
     "grade_id": "cell-463d94ffced62fba",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "b3f12428f02c09090a800fe5d46ef8f5",
     "grade": true,
     "grade_id": "cell-c6f32c5137a9f302",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('X_train_1d_b')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "9e30b76258ae09d0d57ee57525cab5fa",
     "grade": false,
     "grade_id": "cell-15322bd5d7e6c8bf",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Implement batch gradient descent to fit the parameters of your logistic regression model.  Consider `n_iterations = 4000` iterations and a learning rate of `alpha = 100.0`. Consider a starting point of $\\theta_0 = (1, 1)$, i.e. `theta = np.array([[1], [1]])`. Use variable `theta` to specify your estimated parameters.*\n",
    "\n",
    "*(Make sure your implementation is reasonably efficient. If it takes longer than 2 minutes to execute when running on our server it may not complete and you will not be awarded grades. The solution answer runs in under 10 seconds.)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "e09887c1b2b8d4411a068063095582ec",
     "grade": false,
     "grade_id": "cell-aee503a999e27cf6",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "n_iterations = 4000\n",
    "alpha = 100.0\n",
    "theta = np.array([[1], [1]])\n",
    "\n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "b29368af60c38637709a67ba63372233",
     "grade": true,
     "grade_id": "cell-db0d3866ab6ed5aa",
     "locked": true,
     "points": 4,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('theta')\n",
    "print(\"theta[0] = {0:.4f}\".format(theta[0][0]))\n",
    "print(\"theta[1] = {0:.4f}\".format(theta[1][0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "2c3275c68a087f4b57825087f2c0e024",
     "grade": false,
     "grade_id": "cell-d6efe104a72bb532",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Compute the difference between the logistic regression model intercept computed by Scikit-Learn and manually.  Use variable `intercept_diff` for your result.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "609ee4a21d8b86b1bbf931c8313b38d9",
     "grade": false,
     "grade_id": "cell-b761dbdc7668fb7d",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "77cebdb4b7cf0653da556a19347d7921",
     "grade": true,
     "grade_id": "cell-eda44b051be24b4c",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('intercept_diff')\n",
    "print(\"intercept_diff = {0:.4E}\".format(intercept_diff))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "59efb6665ec993c52e2262eee838fc25",
     "grade": false,
     "grade_id": "cell-3ff8e6906407e9fe",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Compute the difference between the logistic regression model* slope *(i.e. coefficient) computed by Scikit-Learn and manually.  Use variable `coeff_diff` for your result.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "ec3e8455318b44f96b4cc4ff259aaaad",
     "grade": false,
     "grade_id": "cell-17bd3970318abda0",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "bcc12d94a1fa9b22a5710d5cc64b9399",
     "grade": true,
     "grade_id": "cell-830185c3c51f3f91",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('coeff_diff')\n",
    "print(\"coeff_diff = {0:.4E}\".format(coeff_diff))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "8eebb530913c2d78ea43e8d6cfccb9a7",
     "grade": false,
     "grade_id": "cell-0d8a45598ebad1aa",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "You should find that the solution from your gradient descent algorithm is close (although not identical) to that recovered by Scikit-Learn. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "ebb30f7db5829cfb68123d51796994cd",
     "grade": false,
     "grade_id": "cell-8b404d163f645ffd",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "Both fitted logistic regression models, however, are not effective. The reason for this is because of class imbalance.  *Describe the class imbalance problem in your own words and how it manifests itself in the classification problem at hand.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "bbf0e320d4f9d168daa3b4b6397a6a6b",
     "grade": true,
     "grade_id": "cell-73126eae7fcd4d45",
     "locked": false,
     "points": 3,
     "schema_version": 3,
     "solution": true
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "d2c5242206ed7f763289951a7d5b1acd",
     "grade": false,
     "grade_id": "cell-32339ef70667c4de",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "The class imbalance problem can be addressed by weighting the training data in a manner that is inversely proportional to their frequency.\n",
    "\n",
    "*Repeat the fitting of your linear regression model but this time perform class weighting.  Use variable `clf_balanced` for your new model.*\n",
    "\n",
    "See the `class_weight` argument of the Scikit-Learn [Logistic Regression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) classifier for further details on how to perform class weighting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "ca93af45fe489f2970d9465be9f1f227",
     "grade": false,
     "grade_id": "cell-043f89d606f8da67",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "1fea7d2b3c6c89155867794f7c43b1b3",
     "grade": true,
     "grade_id": "cell-5dea5e84c6b3f90f",
     "locked": true,
     "points": 0,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('clf_balanced')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "56698708c8d07b85c23e96e74a5f7d02",
     "grade": false,
     "grade_id": "cell-0e177c0c3236c200",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Compute the decision boundary of the logistic regression model fitted by Scikit-Learn when weighting classes.* \n",
    "\n",
    "(Ensure your result is a scalar and not an array of length 1.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "4e68b25af62c3c3b9ce3d88fe30e7246",
     "grade": false,
     "grade_id": "cell-8789a822ce94928b",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "b1830b90407271f2c1a3d024750d1368",
     "grade": true,
     "grade_id": "cell-6d9e2c731edfff2f",
     "locked": true,
     "points": 2,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('decision_boundary_sklearn_balanced')\n",
    "assert not hasattr(decision_boundary_sklearn_balanced, \"__len__\")\n",
    "print(\"decision_boundary_sklearn_balanced = {0:.4f}\".format(decision_boundary_sklearn_balanced))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "5c9b36e7743bfde53557b5426c4fb77f",
     "grade": false,
     "grade_id": "cell-3361e275fac9beac",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Evaluate the probabilities prediced by your new logistic regression model over the domain specified by the variable `X_1d_new`. Use variable `y_1d_proba_balanced` for your computed probabilities.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "e9f94c6ad6d1ed961f39c11ae5b8ac9e",
     "grade": false,
     "grade_id": "cell-3db9585a121a321b",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "344d65536b5d89cf8b1fd6e93f527f66",
     "grade": true,
     "grade_id": "cell-b5df6903e536bc3e",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('y_1d_proba_balanced')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "acc5329742bc3574c6acb59d20110cb6",
     "grade": false,
     "grade_id": "cell-cef66593b1ed2e90",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*For your new balanced model, plot the probability of a star being of type RR Lyrae against the colour variable considered.  Also plot the probability of being a Background star.  Overlay these plots on the scatter plot of class types.  Also plot the decision boundary that you guessed previously, the one computed by Scikit-Learn initially, and the one computed by Scikit-Learn for your new balanced model.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "48606796161d04d156d3cba41a082cba",
     "grade": true,
     "grade_id": "cell-9ce627c16d3996e6",
     "locked": false,
     "points": 2,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "plot_scatter()\n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "a29c1deb02f416bb5859ea868d5f1ecf",
     "grade": false,
     "grade_id": "cell-b91a8e6743a0e23f",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Comment on the decision boundary of the balanced model compared to the unbalanced models fitted previously.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "963c36d881b371659186c7739c8a1a43",
     "grade": true,
     "grade_id": "cell-45725a6e2c43d0f1",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "f1f1759a78aff4f424286f2f577be5ec",
     "grade": false,
     "grade_id": "cell-a5126bea92958ffe",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "Now that we've built up good intuition surrounding the subtleties of the classification problem at hand in 1D, let's consider the 2D problem (we will keep to 2D for plotting convenience)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "b853b8b820ed7b06e86b709642fba4c3",
     "grade": false,
     "grade_id": "cell-46205b5da6dd0e77",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "For the 2D case we consider the following colours."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "92a30f4f2db3010902ab7d521054bbff",
     "grade": false,
     "grade_id": "cell-11a94502070606e6",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "ind = 1\n",
    "cols[:ind+1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "24955f013f80e7be059375dcbbecec30",
     "grade": false,
     "grade_id": "cell-ea66a8b2540e3455",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "Consider the following training and test data for the 2D problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "62e64cbf875c25da99467cb26c1f6c7a",
     "grade": false,
     "grade_id": "cell-374dd2ec4c108d9c",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "X_train_2d = X_train[:, :ind+1]\n",
    "X_train_2d = X_train_2d.reshape(-1,ind+1)\n",
    "X_test_2d = X_test[:, :ind+1]\n",
    "X_test_2d = X_test_2d.reshape(-1,ind+1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "be00d6c3427598d2d4a80466e56e4221",
     "grade": false,
     "grade_id": "cell-ead79764fa5bdc91",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Train a logistic regression model for this 2D problem.  Use variable `clf_2d_logistic` for your classifier.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "084e3730086edc9f3a88b357a32f13b3",
     "grade": false,
     "grade_id": "cell-807bce2068c8c513",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "e54724d2562c049fb17447ee0955fdcd",
     "grade": true,
     "grade_id": "cell-6d3421df624a8839",
     "locked": true,
     "points": 0,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('clf_2d_logistic')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "7b0cbcffdf52227ee932131fe0594f4d",
     "grade": false,
     "grade_id": "cell-d8af8aa1d07b0c2e",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Compute the precision and recall of your 2D logistic regression model. Use variables `precision_logistic` and `recall_logistic` for your results.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "a497d2e0107dba9c7bf6b8d6890b5d75",
     "grade": false,
     "grade_id": "cell-5aa20025d9dcd3de",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "89d58d5042e1fc392b449b7f0cb03e3b",
     "grade": true,
     "grade_id": "cell-901a95a71017f77a",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('precision_logistic')\n",
    "print(\"precision_logistic = {0:.6f}\".format(precision_logistic))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "a8c0b5936c41a742cddef03152ffa57a",
     "grade": true,
     "grade_id": "cell-c829df4fb5b5646d",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('recall_logistic')\n",
    "print(\"recall_logistic = {0:.6f}\".format(recall_logistic))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "9ba63808879ae626d04958a07df7be2e",
     "grade": false,
     "grade_id": "cell-e7757c51155e4dfc",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "Consider the following meshgrid defining the u-g and g-r colour domain of interest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "936ae179349f4afb9f2e84f901b1ac39",
     "grade": false,
     "grade_id": "cell-c1045a44e953f715",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "xlim = (0.7, 1.45)  # u-g\n",
    "ylim = (-0.15, 0.4) # g-r\n",
    "xx, yy = np.meshgrid(np.linspace(xlim[0], xlim[1], 100),\n",
    "                     np.linspace(ylim[0], ylim[1], 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "f4172020874b0ca0808cef832529bb1c",
     "grade": false,
     "grade_id": "cell-7b3fd3dd9fb85196",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Over the domain specified above plot the predicted classification probability.  Overlay on your plot the data instances, highlighting whether a RR Lyrae or background star, and the decision boundary.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f86ab2a5e0c7bbae351fa52f20f7f928",
     "grade": true,
     "grade_id": "cell-cb93ed5cd3864d37",
     "locked": false,
     "points": 5,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "9d7c9da1a39851de5856c8e6353c3aa7",
     "grade": false,
     "grade_id": "cell-a39275a441142214",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Now train an SVM classifier that can support a non-linear decision boundary on the same problem. Use the variable `clf_2d_svm` for your model.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "e95438410123c8be26e3caeee498b988",
     "grade": true,
     "grade_id": "cell-c15b2b7d9de2fc9c",
     "locked": false,
     "points": 3,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_var_defined('clf_2d_svm')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "fb4bec941b61ff878b444ad7d0845e2e",
     "grade": false,
     "grade_id": "cell-5b337b0e95730f71",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Replicate for the SVM your plot above for the 2D logistic regression model.  Over the domain specified above plot the decision function score.  Overlay on your plot the data instances, highlighting whether a RR Lyrae or background star, and the decision boundary.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "139c53d2625a5b080ec97d77d07c64d4",
     "grade": true,
     "grade_id": "cell-3f1d0097c1a9a231",
     "locked": false,
     "points": 4,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "1d44b6983a7391f6baec9783e8f9c3b9",
     "grade": false,
     "grade_id": "cell-9396defb6a2fd540",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Compute the precision and recall of your 2D SVM model. Use variables `precision_svm` and `recall_svm` for your results.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "676c64172c9ee6735a3554f3a19e2787",
     "grade": false,
     "grade_id": "cell-4bb00fa2958c34b5",
     "locked": false,
     "schema_version": 3,
     "solution": true
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "242d477110906928300aca9370ecae60",
     "grade": true,
     "grade_id": "cell-8a0f183fa2697432",
     "locked": true,
     "points": 2,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('precision_svm')\n",
    "print(\"precision_svm = {0:.6f}\".format(precision_svm))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "4b9d01eaa4ee806dc2c07205c2d46660",
     "grade": true,
     "grade_id": "cell-c88d680adad83470",
     "locked": true,
     "points": 2,
     "schema_version": 3,
     "solution": false
    }
   },
   "outputs": [],
   "source": [
    "check_var_defined('recall_svm')\n",
    "print(\"recall_svm = {0:.6f}\".format(recall_svm))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "991b3f11093789127b5b6fbf01e75552",
     "grade": false,
     "grade_id": "cell-6e323ca87d06c0b4",
     "locked": true,
     "schema_version": 3,
     "solution": false
    }
   },
   "source": [
    "*Comment on the difference in decision boundary between your logistic regression and SVM models and how this impacts the effectiveness of the models.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "3d6cb3a273d55b60497f2ed2047bfc2d",
     "grade": true,
     "grade_id": "cell-5429ab62dc4857e2",
     "locked": false,
     "points": 4,
     "schema_version": 3,
     "solution": true
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "MLBDenv",
   "language": "python",
   "name": "mlbdenv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}

