3B_Convolutional_Neural_Networks_Assignment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PyTorch Assignment: Convolutional Neural Network (CNN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[Duke Community Standard](http://integrity.duke.edu/standard.html): By typing your name below, you are certifying that you have adhered to the Duke Community Standard in completing this assignment.**\n",
    "\n",
    "Name: "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convolutional Neural Network\n",
    "\n",
    "Adapt the CNN example for MNIST digit classfication from Notebook 3A. \n",
    "Feel free to play around with the model architecture and see how the training time/performance changes, but to begin, try the following:\n",
    "\n",
    "Image ->  \n",
    "convolution (32 3x3 filters) -> nonlinearity (ReLU) ->  \n",
    "convolution (32 3x3 filters) -> nonlinearity (ReLU) -> (2x2 max pool) ->  \n",
    "convolution (64 3x3 filters) -> nonlinearity (ReLU) ->  \n",
    "convolution (64 3x3 filters) -> nonlinearity (ReLU) -> (2x2 max pool) -> flatten ->\n",
    "fully connected (256 hidden units) -> nonlinearity (ReLU) ->  \n",
    "fully connected (10 hidden units) -> softmax \n",
    "\n",
    "Note: The CNN model might take a while to train. Depending on your machine, you might expect this to take up to half an hour. If you see your validation performance start to plateau, you can kill the training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torchvision import datasets, transforms\n",
    "from tqdm.notebook import tqdm, trange"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x_cnn = torch.randn(100, 1, 28, 28)\n",
    "\n",
    "test1 = nn.Conv2d(1, 32, kernel_size=3, padding=1)\n",
    "test2 = nn.Conv2d(32, 32, kernel_size=3, padding=1)\n",
    "test3 = nn.Conv2d(32, 64, kernel_size = 3, padding = 1)\n",
    "test4 = nn.Conv2d(64, 64, kernel_size = 3, padding = 1)\n",
    "\n",
    "x_cnn = test1(x_cnn)\n",
    "x_cnn = F.relu(x_cnn)\n",
    "x_cnn = test2(x_cnn)\n",
    "x_cnn = F.relu(x_cnn)\n",
    "x_cnn = F.max_pool2d(x_cnn, kernel_size = 2)\n",
    "x_cnn = test3(x_cnn)\n",
    "x_cnn = test4(x_cnn)\n",
    "x_cnn = F.max_pool2d(x_cnn, kernel_size = 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([100, 64, 7, 7])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_cnn.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "lin1 = nn.Linear(7*7*64, 256)\n",
    "x_test = lin1(x_cnn.view(-1,7*7*64))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6400.0"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "44800/(7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "### YOUR CODE HERE ###\n",
    "\n",
    "class MNIST_CNN(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.conv1 = nn.Conv2d(1, 32, kernel_size=3, padding=1)\n",
    "        self.conv2 = nn.Conv2d(32, 32, kernel_size=3, padding=1)\n",
    "        self.conv3 = nn.Conv2d(32, 64, kernel_size = 3, padding = 1)\n",
    "        self.conv4 = nn.Conv2d(64, 64, kernel_size = 3, padding = 1)\n",
    "        self.fc1 = nn.Linear(7*7*64, 256)\n",
    "        self.fc2 = nn.Linear(256, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        # conv layer 1\n",
    "        x = self.conv1(x)\n",
    "        x = F.relu(x)\n",
    "        \n",
    "        # conv layer 2\n",
    "        x = self.conv2(x)\n",
    "        x = F.relu(x)\n",
    "        x = F.max_pool2d(x, kernel_size=2)\n",
    "        \n",
    "        # conv layer 3\n",
    "        x = self.conv3(x)\n",
    "        x = F.relu(x)\n",
    "        \n",
    "        # conv layer 4\n",
    "        x = self.conv4(x)\n",
    "        x = F.relu(x)\n",
    "        x = F.max_pool2d(x, kernel_size=2)\n",
    "        \n",
    "        # fc layer 1\n",
    "        x = x.view(-1, 7*7*64)\n",
    "        x = self.fc1(x)\n",
    "        x = F.relu(x)\n",
    "        \n",
    "        # fc layer 2\n",
    "        x = self.fc2(x)\n",
    "        return x        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "96ce7e8fccc54e049fcb50bff9af8e5b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/4 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0991c23cef8c48e4bff43dae3cf4068b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/600 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d23111c895ff45d09fccdae6b50bc9b3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/600 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8419e740b144455cbf8c92d96082af0d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/600 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9aad09602561496bb17197dc1440cfd2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/600 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "980fcae43a154a10a104fc18558a75ff",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/100 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy: 0.9860000014305115\n"
     ]
    }
   ],
   "source": [
    "# Load the data\n",
    "mnist_train = datasets.MNIST(root=\"./datasets\", train=True, transform=transforms.ToTensor(), download=True)\n",
    "mnist_test = datasets.MNIST(root=\"./datasets\", train=False, transform=transforms.ToTensor(), download=True)\n",
    "train_loader = torch.utils.data.DataLoader(mnist_train, batch_size=100, shuffle=True)\n",
    "test_loader = torch.utils.data.DataLoader(mnist_test, batch_size=100, shuffle=False)\n",
    "\n",
    "## Training\n",
    "# Instantiate model  \n",
    "model = MNIST_CNN()  # <---- change here\n",
    "\n",
    "# Loss and Optimizer\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=0.001)  # <---- change here\n",
    "\n",
    "# Iterate through train set minibatchs \n",
    "for epoch in trange(4):  # <---- change here\n",
    "    for images, labels in tqdm(train_loader):\n",
    "        # Zero out the gradients\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        # Forward pass\n",
    "        x = images  # <---- change here \n",
    "        y = model(x)\n",
    "        loss = criterion(y, labels)\n",
    "        # Backward pass\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "## Testing\n",
    "correct = 0\n",
    "total = len(mnist_test)\n",
    "\n",
    "with torch.no_grad():\n",
    "    # Iterate through test set minibatchs \n",
    "    for images, labels in tqdm(test_loader):\n",
    "        # Forward pass\n",
    "        x = images  # <---- change here \n",
    "        y = model(x)\n",
    "\n",
    "        predictions = torch.argmax(y, dim=1)\n",
    "        correct += torch.sum((predictions == labels).float())\n",
    "\n",
    "print('Test accuracy: {}'.format(correct/total))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "l = 0.005, epoch = 2 -> accuracy:0.9810\n",
    "\n",
    "l = 0.005, epoch = 3 -> accuracy:0.9833\n",
    "\n",
    "l = 0.005, epoch = 4 -> accuracy:0.9843\n",
    "\n",
    "l = 0.001, epoch = 2 -> accuracy:0.9881\n",
    "\n",
    "l = 0.001, epoch = 3 -> accuracy:0.9900\n",
    "\n",
    "l = 0.001, epoch = 4 -> accuracy:0.9902\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Short answer\n",
    "\n",
    "1\\. How does the CNN compare in accuracy with yesterday's logistic regression and MLP models? How about training time?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`[Your answer here]`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2\\. How many trainable parameters are there in the CNN you built for this assignment?\n",
    "\n",
    "*Note: The total of trainable parameters counts each element in a tensor. For example, a weight matrix that is 10x5 has 50 trainable parameters.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`[Your answer here]`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3\\. When would you use a CNN versus a logistic regression model or an MLP?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`[Your answer here]`"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.12 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "23574fd47338620f5fa9034c1801f73296c9ae358dd5ea70f96243aff333c802"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}