MODEL SELECTION AND VALIDATION - COMPUTER SCIENCE

Do not use MINST digit dataset. Use dataset.load_digits() from sklearn
Due: Sep 28, 2021 11:59 PM
For this assignment, you are to determine which model is best for prediction, report the
right hyperparameters, and the resulting accuracy for the Digit Recognition data set that
was used in the previous assignment.
As before, create a PDF of your notebook showing your steps and include the table
below as mentioned. Attach both a jupyter notebook and the PDF version. Also, be sure
to include your name at the start of the notebook and at the top of the PDF.
Specifically, you are to test the following models
Model Hyperparameter Testing range Notes
Support Vector
Machine
Gamma – size of
the kernel
C – slack variable
10-x for x = -5 to 5 use the ‘rbf’: radial
basis function
kernel
K-nearest
neighbors
k – number of
neighbors
1,3,5,7,9 use the sklearn
function
Decision Trees min_samples_split 3,5,7,9 (1 was
removed)
use the defaults for
the other
hyperparameters.
Logistic Regression C – inverse of the
regularization
strength (smaller =
more regularization)
10-x for x = -5 to 5 with the L1 penalty
(Lasso)
Steps are as follows:
1. Separate your data into training and testing. We will use cross-validation over the
training set to select the right parameters
a. Use train_test_split to create a separate training and test set.
X_train, X_test, y_train, y_test = train_test_split(X,
y, stratify=True, test_size=0.20)
b. For the training set, you have two choices to perform hyperparameter
selection.
i. Use cross-validation to evaluate each model variant and select the
best hyperparameters (standard practice, most recommended)
ii. Create a hold-out validation set and train on one portion of the data
and use the accuracy on the hold-out validation set to pick the right
hyperparameters (also valid)
2. Steps to turn in for the assignment
a. Train the four models with their default parameters. Report the resulting
accuracy of each model using the default parameters.
b. For each of the four models, find the hyperparameters giving the highest
accuracy on the validation set by performing an exhaustive grid search.
Report the hyperparameter values and accuracy on the validation
set.
i. Consider using sklearn.model_selection.GridSearchCV
ii. For the models with two hyperparameters, you will need to search
both simultaneously to find the optimum combination
c. Now apply the highest accuracy trained models to the test set. Report the
accuracy of each model.
Fill the following table with the information.
Model Default
validation
accuracy
Tuned
validation
accuracy
Selected
hyperparameter
s
Final test set
accuracy
SVM
k-NN
Decision Trees
Logistic
Regression

{
“nbformat”: 4,
“nbformat_minor”: 5,
“metadata”: {
“kernelspec”: {
“display_name”: “Python 3 (ipykernel)”,
“language”: “python”,
“name”: “python3”
},
“language_info”: {
“codemirror_mode”: {
“name”: “ipython”,
“version”: 3
},
“file_extension”: “.py”,
“mimetype”: “text/x-python”,
“name”: “python”,
“nbconvert_exporter”: “python”,
“pygments_lexer”: “ipython3”,
“version”: “3.9.6”
},
“colab”: {
“name”: “ModelSelectionAndValidation (1).ipynb”,
“provenance”: []
}
},
“cells”: [
{
“cell_type”: “code”,
“metadata”: {
“colab”: {
“base_uri”: “https://localhost:8080/”,
“height”: 907
},
“id”: “79ac7e6e”,
“outputId”: “1508f570-ddb4-4d5a-868a-585899772bc0”
},
“source”: [
“# Model Selection And Validation with SVM for Machine Learningn”,
“n”,
“import matplotlib.pyplot as pltn”,
“n”,
“import matplotlib as mpln”,
“n”,
“from sklearn.datasets import load_digitsn”,
“n”,
“import numpy as npn”,
“n”,
“import randomn”,
“n”,
“n”,
“digits = load_digits()n”,
“n”,
“X,y = digits.data, digits.targetn”,
“n”,
“n”,
“%matplotlib inlinen”,
“n”,
“random_examples = [random.randint(0,len(digits.images))n”,
“n”,
“for i in range(10)]n”,
“n”,
“for n,number in enumerate(random_examples):n”,
” plt.subplot(2, 5, n+1)n”,
” plt.imshow(digits.images[number],cmap=’binary’,interpolation=’none’, extent=[0,8,0,8])n”,
” plt.grid()n”,
” plt.show()”
],
“id”: “79ac7e6e”,
“execution_count”: 1,
“outputs”: [
{
“output_type”: “display_data”,
“data”: {
“image/png”: “iVBORw0KGgoAAAANSUhEUgAAAFUAAABZCAYAAABPN2ghAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAD6UlEQVR4nO2cv0tcWRTHv9/MMiBBHJgYiyRowEVIEwVJtYUpsmyqtJtC0MYqhWWw0v/AJs0UURvZLrBFSDbNkMYiBpT8IFmCPEhsglOIhWQxni0cYRLm3nkzvO9zRs6n0Xln7rzDh3udc9+9V5oZnGy5cNYJnEdcqgCXKsClCnCpAlyqgF8UH1oqlWx0dLTtdltbW9F4X19fMFYoFIKxcrmM4+PjprH+/v7oPYvFYtPrSZJgb2+PzWISqUNDQ9jc3Gy7XalUisbHx8c7ajs9PY2Dg4Omsampqeg9R0ZGml6fnJwMtvHhL8ClCnCpAlyqAJcqQPLtH6NarQZj+/v70baxkitWGdRqNayvr7edDwCsrq5G483wnirApQpwqQJcqgCXKsClCsi9pIqVRcPDwx23jT1QqVarLUunLPGeKsClCnCpAlL9TSWZADgA8B3AkZmFn9A6bX1R3TazPVkm5wgf/gLSSjUA/5B8TXJOmdB5IO3w/83MdkleBvCC5Acze9n4hrrsOQAYHBwM1oWxenJhYSGaxMbGRjAWWvUEgMPDw1zr1FRSzWy3/vMryScAbgF4+dN7KgAqADA2NmahVcqZmZngfZIkieaxvLwcjIVWPYGTSUOrVdMsaTn8SV4k2X/6O4DfAbxVJ9bLpOmpQwCekDx9/7qZPZNm1eO0lGpmOwBu5pDLucFLKgEuVUDuj/5i39Jra2vRthMTE8FY7LFhpVIJVhaxfDrFe6oAlyrApQpwqQJcqgCXKiD3kmp+fl7yuUtLS8FYrVYLbjRbXFzMPBfvqQJcqgCXKsClCnCpAlyqgNxLqtjCX6tyK7a+NTAwEIyVy+Vo26zxnirApQpwqQJcqgCXKsClCnCpAnKvU2O02u+0vb0djK2srARjxWJRsmoawnuqAJcqIJVUkn+Q/EjyE8mH6qR6nTRbKQsAHgG4C+AGgPskb6gT62XS9NRbAD6Z2Y6Z/QfgLwD3tGn1NmmkXgHwueH1l/o1J0BmJVXjnn8A30jmutt6dnY2Fr4EIOvjSsEdcWmk7gK41vD6av3aDzTu+Se52U0H2PLOJ83wfwXgV5LXSRYB/Angb21avU2a7elHJB8AeA6gAOCxmb2TZ9bDpD3y8xTA0zY+t9JZOjJyzYf+39Ozx6epAjKV2m3TWZIJyTckt0i2/w9dO71vVsO/Pp39F8AdnEwQXgG4b2bvM7lBZzklACbzPlKfZU/16WydLKV243T2TI7Ud9WTfwEtj9QryLKnpprO5knjkXoAp0fq5WQptaums2d5pD6z4d+F09kzO1LvMyoBPqMS4FIFuFQBLlWASxXgUgW4VAEuVcD//9gbceKUPKcAAAAASUVORK5CYII=n”,
“text/plain”: [
“

MODEL SELECTION AND VALIDATION – COMPUTER SCIENCE

Why Choose Us

How it Works

Submit a Comment Cancel reply

Recent Posts