2A_Logistic_Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Logistic Regression in PyTorch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we're going to build a very simple neural network in PyTorch to do handwritten digit classification.\n",
    "First, we'll start with some exploration of the MNIST dataset, explaining how we load and format the data.\n",
    "We'll then jump into motivating and then implementing the logistic regression model, including the forward and backwards pass, loss functions, and optimizers.\n",
    "After training the model, we'll evaluate how we did and visualize what we've learned.\n",
    "Finally, we'll refactor our code in an object-oriented manner, using higher level APIs.\n",
    "\n",
    "Before we get started, some imports for the packages we'll be using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import torch\n",
    "from tqdm.notebook import tqdm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MNIST Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The MNIST dataset is very popular machine learning dataset, consisting of 70000 grayscale images of handwritten digits, of dimensions 28x28. \n",
    "We'll be using it as our example dataset for this section of the tutorial, with the goal being to predict which digit is in each image.\n",
    "\n",
    "![mnist](Figures/mnist.png)\n",
    "\n",
    "The first (and often most important) step in machine learning is preparing the data.\n",
    "This can include downloading, organizing, formatting, shuffling, pre-processing, augmenting, and batching examples so that they can be fed to a model.\n",
    "The `torchvision` package makes this easy by implementing many of these, allowing us to put these datasets into a usable form in only a few lines of code.\n",
    "First, let's download the train and test sets of MNIST:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchvision import datasets, transforms\n",
    "\n",
    "mnist_train = datasets.MNIST(root=\"./datasets\", train=True, transform=transforms.ToTensor(), download=True)\n",
    "mnist_test = datasets.MNIST(root=\"./datasets\", train=False, transform=transforms.ToTensor(), download=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of MNIST training examples: 60000\n",
      "Number of MNIST test examples: 10000\n"
     ]
    }
   ],
   "source": [
    "print(\"Number of MNIST training examples: {}\".format(len(mnist_train)))\n",
    "print(\"Number of MNIST test examples: {}\".format(len(mnist_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we'd expect, 60000 of the MNIST examples are in the train set, and the rest are in the test set.\n",
    "We added the transform `ToTensor()` when formatting the dataset, to convert the input data from a Pillow `Image` type into a PyTorch `Tensor`. Tensors will eventually be the input type that we feed into our model. \n",
    "\n",
    "Let's look at an example image from the train set and its label.\n",
    "Notice that the `image` tensor defaults to something 3-dimensional.\n",
    "The \"1\" in the first dimension indicates that the image only has one channel (i.e. grayscale).\n",
    "We need to get rid of this to visualize the image with `imshow`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Default image shape: torch.Size([1, 28, 28])\n",
      "Reshaped image shape: torch.Size([28, 28])\n",
      "The label for this image: 1\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAMdUlEQVR4nO3dX6gc5R3G8eeJVYRUNBqMUVPTFi9aik1LkIKhpDSGKELSC6W5KJGWnl6oWKgQsYJKKYRaLSJaOKL5U6wiRJsg0lZC1JageJRUo0nUhtgmOZxTEdFcpXp+vTiTcoy7s8edmZ1Nft8PHHZ33t2ZH0OevO/M7M7riBCAU9+ctgsAMBiEHUiCsANJEHYgCcIOJPGFQW7MNqf+gYZFhDstr9Sz215le7/td2zfWmVdAJrlfq+z2z5N0luSrpR0SNLLktZGxJsln6FnBxrWRM9+uaR3IuJARByT9Lik1RXWB6BBVcJ+kaR/z3h9qFj2KbZHbI/ZHquwLQAVVTlB12mo8JlhekSMShqVGMYDbarSsx+StGjG64slHalWDoCmVAn7y5Iutf1l22dI+qGk7fWUBaBufQ/jI+Jj2zdK+ouk0yQ9EhFv1FYZgFr1femtr41xzA40rpEv1QA4eRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQRN9TNgNNu/3220vb77rrrtL2OXO692XLly8v/ezzzz9f2n4yqhR22wclfSTpE0kfR8TSOooCUL86evbvRcR7NawHQIM4ZgeSqBr2kPRX26/YHun0Btsjtsdsj1XcFoAKqg7jr4iII7bPl/Ss7X0R8cLMN0TEqKRRSbIdFbcHoE+VevaIOFI8Tkp6StLldRQFoH59h932XNtnHX8uaaWkPXUVBqBeVYbxCyQ9Zfv4ev4YEX+upSqkcP3115e2r1+/vrR9amqq721H5Dui7DvsEXFA0jdrrAVAg7j0BiRB2IEkCDuQBGEHkiDsQBL8xBWtueSSS0rbzzzzzAFVkgM9O5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kwXV2NGrFihVd22666aZK6963b19p+zXXXNO1bWJiotK2T0b07EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBNfZUcmyZctK2zdu3Ni17eyzz6607bvvvru0/d133620/lMNPTuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJMF1dlSybt260vYLL7yw73U/99xzpe1btmzpe90Z9ezZbT9ie9L2nhnLzrX9rO23i8d5zZYJoKrZDOM3SVp1wrJbJe2IiEsl7SheAxhiPcMeES9Iev+ExaslbS6eb5a0pt6yANSt32P2BRExLkkRMW77/G5vtD0iaaTP7QCoSeMn6CJiVNKoJNmOprcHoLN+L71N2F4oScXjZH0lAWhCv2HfLun4NZd1krbVUw6ApjiifGRt+zFJyyXNlzQh6Q5Jf5L0hKQvSfqXpGsj4sSTeJ3WxTD+JDN//vzS9l73X5+amura9sEHH5R+9rrrritt37lzZ2l7VhHhTst7HrNHxNouTd+vVBGAgeLrskAShB1IgrADSRB2IAnCDiTBT1yTW7x4cWn71q1bG9v2/fffX9rOpbV60bMDSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBJcZ09u1aoT7yX6aZdddlml9e/YsaNr23333Vdp3fh86NmBJAg7kARhB5Ig7EAShB1IgrADSRB2IImet5KudWPcSnrg1qxZU9q+adOm0va5c+eWtu/atau0vex20L1uQ43+dLuVND07kARhB5Ig7EAShB1IgrADSRB2IAnCDiTB79lPAWX3fm/yvu+SdODAgdJ2rqUPj549u+1HbE/a3jNj2Z22D9veXfxd3WyZAKqazTB+k6ROtzP5XUQsKf6eqbcsAHXrGfaIeEHS+wOoBUCDqpygu9H2a8Uwf163N9kesT1me6zCtgBU1G/Yfy/pq5KWSBqXdE+3N0bEaEQsjYilfW4LQA36CntETETEJxExJekhSZfXWxaAuvUVdtsLZ7z8gaQ93d4LYDj0vM5u+zFJyyXNt31I0h2SltteIikkHZT0s+ZKRC/r16/v2jY1NdXotjds2NDo+lGfnmGPiLUdFj/cQC0AGsTXZYEkCDuQBGEHkiDsQBKEHUiCn7ieBJYsWVLavnLlysa2vW3bttL2/fv3N7Zt1IueHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSYMrmk8Dk5GRp+7x5Xe8K1tOLL75Y2n7VVVeVth89erTvbaMZTNkMJEfYgSQIO5AEYQeSIOxAEoQdSIKwA0nwe/aTwHnnnVfaXuV20Q8++GBpO9fRTx307EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBNfZh8DGjRtL2+fMae7/5F27djW2bgyXnv+KbC+yvdP2Xttv2L65WH6u7Wdtv1089n8HBQCNm02X8bGkX0TE1yR9R9INtr8u6VZJOyLiUkk7itcAhlTPsEfEeES8Wjz/SNJeSRdJWi1pc/G2zZLWNFQjgBp8rmN224slfUvSS5IWRMS4NP0fgu3zu3xmRNJIxToBVDTrsNv+oqStkn4eER/aHe9p9xkRMSpptFgHN5wEWjKr07y2T9d00B+NiCeLxRO2FxbtCyWV3wIVQKt69uye7sIflrQ3Iu6d0bRd0jpJG4rH8rl9E+s15fKKFStK23v9hPXYsWNd2x544IHSz05MTJS249Qxm2H8FZJ+JOl127uLZbdpOuRP2P6JpH9JuraRCgHUomfYI+LvkrodoH+/3nIANIWvywJJEHYgCcIOJEHYgSQIO5AEP3EdgHPOOae0/YILLqi0/sOHD3dtu+WWWyqtG6cOenYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1Igt+zD8C+fftK23tNm7xs2bI6y0FS9OxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kIQjovwN9iJJWyRdIGlK0mhE3Gf7Tkk/lfSf4q23RcQzPdZVvjEAlUVEx1mXZxP2hZIWRsSrts+S9IqkNZKuk3Q0In472yIIO9C8bmGfzfzs45LGi+cf2d4r6aJ6ywPQtM91zG57saRvSXqpWHSj7ddsP2J7XpfPjNgesz1WrVQAVfQcxv//jfYXJT0v6dcR8aTtBZLekxSSfqXpof6Pe6yDYTzQsL6P2SXJ9umSnpb0l4i4t0P7YklPR8Q3eqyHsAMN6xb2nsN425b0sKS9M4NenLg77geS9lQtEkBzZnM2fpmkv0l6XdOX3iTpNklrJS3R9DD+oKSfFSfzytZFzw40rNIwvi6EHWhe38N4AKcGwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKDnrL5PUnvzng9v1g2jIa1tmGtS6K2ftVZ2yXdGgb6e/bPbNwei4ilrRVQYlhrG9a6JGrr16BqYxgPJEHYgSTaDvtoy9svM6y1DWtdErX1ayC1tXrMDmBw2u7ZAQwIYQeSaCXstlfZ3m/7Hdu3tlFDN7YP2n7d9u6256cr5tCbtL1nxrJzbT9r++3iseMcey3Vdqftw8W+22376pZqW2R7p+29tt+wfXOxvNV9V1LXQPbbwI/ZbZ8m6S1JV0o6JOllSWsj4s2BFtKF7YOSlkZE61/AsP1dSUclbTk+tZbt30h6PyI2FP9RzouI9UNS2536nNN4N1Rbt2nGr1eL+67O6c/70UbPfrmkdyLiQEQck/S4pNUt1DH0IuIFSe+fsHi1pM3F882a/scycF1qGwoRMR4RrxbPP5J0fJrxVvddSV0D0UbYL5L07xmvD2m45nsPSX+1/YrtkbaL6WDB8Wm2isfzW67nRD2n8R6kE6YZH5p918/051W1EfZOU9MM0/W/KyLi25KuknRDMVzF7Pxe0lc1PQfguKR72iymmGZ8q6SfR8SHbdYyU4e6BrLf2gj7IUmLZry+WNKRFuroKCKOFI+Tkp7S9GHHMJk4PoNu8TjZcj3/FxETEfFJRExJekgt7rtimvGtkh6NiCeLxa3vu051DWq/tRH2lyVdavvLts+Q9ENJ21uo4zNszy1OnMj2XEkrNXxTUW+XtK54vk7SthZr+ZRhmca72zTjannftT79eUQM/E/S1Zo+I/9PSb9so4YudX1F0j+Kvzfark3SY5oe1v1X0yOin0g6T9IOSW8Xj+cOUW1/0PTU3q9pOlgLW6ptmaYPDV+TtLv4u7rtfVdS10D2G1+XBZLgG3RAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kMT/AB1U3JBTXNyMAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Pick out the 4th (0-indexed) example from the training set\n",
    "image, label = mnist_train[3]\n",
    "\n",
    "# Plot the image\n",
    "print(\"Default image shape: {}\".format(image.shape))\n",
    "image = image.reshape([28,28])\n",
    "print(\"Reshaped image shape: {}\".format(image.shape))\n",
    "plt.imshow(image, cmap=\"gray\")\n",
    "\n",
    "# Print the label\n",
    "print(\"The label for this image: {}\".format(label))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While we could work directly with the data as a `torchvision.dataset`, we'll find it useful to use a `DataLoader`, which will take care of shuffling and batching:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_loader = torch.utils.data.DataLoader(mnist_train, batch_size=100, shuffle=True)\n",
    "test_loader = torch.utils.data.DataLoader(mnist_test, batch_size=100, shuffle=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An example of a minibatch drawn from a `DataLoader`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of the minibatch of images: torch.Size([100, 1, 28, 28])\n",
      "Shape of the minibatch of labels: torch.Size([100])\n"
     ]
    }
   ],
   "source": [
    "data_train_iter = iter(train_loader)\n",
    "images, labels = data_train_iter.next()\n",
    "\n",
    "print(\"Shape of the minibatch of images: {}\".format(images.shape))\n",
    "print(\"Shape of the minibatch of labels: {}\".format(labels.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logistic Regression Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a good feel for how to load our data, let's start putting together our model. \n",
    "In this tutorial, we'll be building a logistic regression model, which is essentially a fully-connected neural network without any hidden layers. \n",
    "While fairly basic, logistic regression can perform surprisingly well on many simple classification tasks.\n",
    "\n",
    "#### The forward pass\n",
    "\n",
    "While our data inputs (which we'll call `x`) are images (i.e. 2-dimensional), MNIST digits are pretty small, and the model we're using is very simple.\n",
    "Thus, we're going to be treating the input as flat vectors.\n",
    "To convert our inputs into row vectors (a.k.a. flattening), we can use `view()`, the equivalent of NumPy's `reshape()`.\n",
    "Also like NumPy, we can replace one of the dimensions of the reshaping with a `-1`, which tells PyTorch to infer this dimension based on the original dimensions and the other specified dimensions.\n",
    "Let's do try this flattening on the minibatch of 100 images we drew in the previous section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The shape of input x: torch.Size([100, 784])\n"
     ]
    }
   ],
   "source": [
    "x = images.view(-1, 28*28)\n",
    "print(\"The shape of input x: {}\".format(x.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get our predicted probabilities of each digit, let's first start with the probability of a digit being a 1 like the image above. \n",
    "For our simple model, we can start by applying a linear transformation. \n",
    "That is, we multiply each pixel $x_i$ of the input row vector by a weight $w_{i,1}$, sum them all together, and then add a bias $b_1$.\n",
    "This is equivalent to a dot product between the class \"1\" weights and the input:\n",
    "\n",
    "\\begin{align}\n",
    "y_1 = \\sum_i x_i w_{i,1} + b_1\n",
    "\\end{align}\n",
    "\n",
    "The magnitude of this result $y_1$, we'll take as being correlated to our belief in how likely we think the input digit was a 1. \n",
    "The higher the value of $y_1$, the more likely we think the input image $x$ was a 1 (i.e., we'd hope we'd get a relatively large value for $y_1$ for the above image). \n",
    "Remember though, our original goal was to identify all 10 digits, so we actually have:\n",
    "\n",
    "\\begin{align*}\n",
    "y_0 =& \\sum_i x_i w_{i,0} + b_0 \\\\\n",
    "y_1 =& \\sum_i x_i w_{i,1} + b_1 \\\\\n",
    "y_2 =& \\sum_i x_i w_{i,2} + b_2 \\\\\n",
    "y_3 =& \\sum_i x_i w_{i,3} + b_3 \\\\\n",
    "y_4 =& \\sum_i x_i w_{i,4} + b_4 \\\\\n",
    "y_5 =& \\sum_i x_i w_{i,5} + b_5 \\\\\n",
    "y_6 =& \\sum_i x_i w_{i,6} + b_6 \\\\\n",
    "y_7 =& \\sum_i x_i w_{i,7} + b_7 \\\\\n",
    "y_8 =& \\sum_i x_i w_{i,8} + b_8 \\\\\n",
    "y_9 =& \\sum_i x_i w_{i,9} + b_9\n",
    "\\end{align*}\n",
    "\n",
    "We can express this in matrix form as:\n",
    "\n",
    "\\begin{align}\n",
    "y = x W + b \n",
    "\\end{align}\n",
    "\n",
    "To take advantage of parallel computation, we commonly process multiple inputs $x$ at once, in a minibatch.\n",
    "We can stack each input $x$ into a matrix $X$, giving us \n",
    "\n",
    "\\begin{align}\n",
    "Y = X W + b \n",
    "\\end{align}\n",
    "\n",
    "Visualizing the dimensions:\n",
    "\n",
    "<img src=\"Figures/mnist_matmul.PNG\" width=\"500\"/>\n",
    "\n",
    "In our specific example, the minibatch size $m$ is $100$, the dimension of the data is $28 \\times 28=784$, and the number of classes $c$ is $10$.\n",
    "While $X$ and $Y$ are matrices due to the batching, conventionally, they are often given lowercase variable names, as if they were for a single example.\n",
    "We will use `x` and `y` throughout."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The weight $W$ and bias $b$ make up the parameters of this model.\n",
    "When we say that we want to \"learn the model,\" what we're really trying to do is find good values for every element in $W$ and $b$.\n",
    "Before we begin learning, we need to initialize our parameters to some value, as a starting point.\n",
    "Here, we don't really know what the best values are, so we going to initialize $W$ randomly (using something called [Xavier initialization](http://proceedings.mlr.press/v9/glorot10a.html)), and set $b$ to a vector of zeros."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Randomly initialize weights W\n",
    "W = torch.randn(784, 10)/np.sqrt(784)\n",
    "W.requires_grad_()\n",
    "\n",
    "# Initialize bias b as 0s\n",
    "b = torch.zeros(10, requires_grad=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As both `W` and `b` are parameters we wish to learn, we set `requires_grad` to `True`. \n",
    "This tells PyTorch's autograd to track the gradients for these two variables, and all the variables depending on `W` and `b`.\n",
    "\n",
    "With these model parameters, we compute $y$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Linear transformation with W and b\n",
    "y = torch.matmul(x, W) + b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see for example what the predictions look like for the first example in our minibatch. Remember, the bigger the number, the more the model thinks the input $x$ is of that class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([-0.2289, -0.0704, -0.5158, -0.4413,  0.0985,  0.8360, -0.8170,  0.5376,\n",
      "         0.1400,  0.3876], grad_fn=<SliceBackward0>)\n"
     ]
    }
   ],
   "source": [
    "print(y[0,:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can interpret these values (aka logits) $y$ as probabilities if we normalize them to be positive and add up to 1. In logistic regression, we do this with a softmax:\n",
    "\n",
    "\\begin{align}\n",
    "p(y_i) = \\text{softmax}(y_i) = \\frac{\\text{exp}(y_i)}{\\sum_j\\text{exp}(y_j)}\n",
    "\\end{align}\n",
    "\n",
    "Notice that because the range of the exponential function is always non-negative, and since we're normalizing by the sum, the softmax achieves the desired property of producing values between 0 and 1 that sum to 1. If we look at the case with only 2 classes, we see that the softmax is the multi-class extension of the binary sigmoid function: \n",
    "\n",
    "<img src=\"Figures/Logistic-curve.png\" width=\"300\"/>\n",
    "\n",
    "We can compute the softmax ourselves using the above formula if we'd like, but PyTorch already has the softmax function in `torch.nn.functional`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "py[0] from equation: tensor([0.0713, 0.0835, 0.0535, 0.0577, 0.0989, 0.2068, 0.0396, 0.1535, 0.1031,\n",
      "        0.1321], grad_fn=<SelectBackward0>)\n",
      "py[0] with torch.nn.functional.softmax: tensor([0.0713, 0.0835, 0.0535, 0.0577, 0.0989, 0.2068, 0.0396, 0.1535, 0.1031,\n",
      "        0.1321], grad_fn=<SelectBackward0>)\n"
     ]
    }
   ],
   "source": [
    "# Option 1: Softmax to probabilities from equation\n",
    "py_eq = torch.exp(y) / torch.sum(torch.exp(y), dim=1, keepdim=True)\n",
    "print(\"py[0] from equation: {}\".format(py_eq[0]))\n",
    "\n",
    "# Option 2: Softmax to probabilities with torch.nn.functional\n",
    "import torch.nn.functional as F\n",
    "py = F.softmax(y, dim=1)\n",
    "print(\"py[0] with torch.nn.functional.softmax: {}\".format(py[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've now defined the forward pass of our model: given an input image, the graph returns the probabilities the model thinks the input is each of the 10 classes. Are we done?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The cross-entropy loss\n",
    "\n",
    "This tutorial isn't done yet, so you can probably guess that the answer is not quite. \n",
    "We don't know the values of $W$ and $b$ yet! \n",
    "Remember how we initialized them randomly? \n",
    "Before we adjust any of the weights, we need a way to measure how the model is doing.\n",
    "Specifically, we're going to measure how badly the model is doing.\n",
    "We do this with a *loss* function, which takes the model's prediction and returns a single number (i.e. a scalar) summarizing model performance.\n",
    "This loss will inform how we adjust the parameters of the model.\n",
    "\n",
    "The loss we commonly use in classification is cross-entropy, a concept from information theory.\n",
    "Explaining exactly what the cross-entropy represents goes slightly beyond the scope of this course, but you can think of it as a way of quantifying how far apart one distribution $y'$ is from another $y$.\n",
    "\n",
    "\\begin{align}\n",
    "H_{y'}(y)=-\\sum_i y'_i \\text{log}(y_i)\n",
    "\\end{align}\n",
    "\n",
    "In our case, $y$ is the set of probabilities predicted by the model ($py$ above); $y'$ is the target distribution.\n",
    "What is the target distribution?\n",
    "It's the true label, which is what we wanted the model to predict.\n",
    "\n",
    "Cross-entropy not only captures how *correct* (max probability corresponds to the right answer) the model's answers are, it also accounts for how *confident* (high confidence in correct answers) they are. This encourages the model to produce very high probabilities for correct answers while driving down the probabilities for the wrong answers, instead of merely being satisfied with it being the argmax. \n",
    "\n",
    "We focus here on supervised learning, a setting in which we have the labels.\n",
    "Our `DataLoader` automatically includes the corresponding labels for each of our inputs.\n",
    "Here are the labels from the first time we retrieved a minibatch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([100])\n"
     ]
    }
   ],
   "source": [
    "print(labels.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like the softmax operation, we can implement the cross-entropy directly from the equation, using the softmax output.\n",
    "However, as with the softmax, `torch.nn.functional` already has the cross-entropy loss implemented as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cross entropy from equation: 2.342674970626831\n",
      "cross entropy with torch.nn.functional.cross_entropy: 2.342674970626831\n"
     ]
    }
   ],
   "source": [
    "# Cross-entropy loss from equation\n",
    "cross_entropy_eq = torch.mean(-torch.log(py_eq)[range(labels.shape[0]),labels])\n",
    "print(\"cross entropy from equation: {}\".format(cross_entropy_eq))\n",
    "\n",
    "# Option 2: cross-entropy loss with torch.nn.functional\n",
    "cross_entropy = F.cross_entropy(y, labels)\n",
    "print(\"cross entropy with torch.nn.functional.cross_entropy: {}\".format(cross_entropy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that PyTorch's cross-entropy loss combines the softmax operator and cross-entropy into a single operation, for numerical stability reasons.\n",
    "Don't do the softmax twice!\n",
    "Make sure to feed in the pre-softmax logits `y`, not the post-softmax probabilities `py`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The backwards pass\n",
    "\n",
    "Now that we have the loss as a way of quantifying how badly the model is doing, we can improve our model by changing the parameters in a way that minimizes the loss.\n",
    "For neural networks, the common way of doing this is with backpropagation: we take the gradient of the loss with respect to $W$ and $b$ and take a step in the direction that reduces our loss.\n",
    "\n",
    "If we were not using a deep learning framework like PyTorch, we would have to go through and derive all the gradients ourselves by hand, then code them into our program. \n",
    "We certainly still could.\n",
    "However, with modern auto-differentiation libraries, it's much faster and easier to let the computer do it.\n",
    "\n",
    "First, we need to create an optimizer.\n",
    "There are many choices, but since logistic regression is fairly simple, we'll use standard stochastic gradient descent (SGD), which makes the following update:\n",
    "\n",
    "\\begin{align}\n",
    "\\theta_{t+1} = \\theta_t - \\alpha \\nabla_\\theta \\mathcal{L}\n",
    "\\end{align}\n",
    "\n",
    "where $\\theta$ is a parameter, $\\alpha$ is our learning rate (step size), and $\\nabla_\\theta \\mathcal{L}$ is the gradient of our loss with respect to $\\theta$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Optimizer\n",
    "optimizer = torch.optim.SGD([W,b], lr=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we created our parameters $W$ and $b$, we indicated that they require gradients.\n",
    "To compute the gradients for $W$ and $b$, we call the `backward()` function on the cross-entropy loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "cross_entropy.backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each of the variables that required gradients have now accumulated gradients.\n",
    "We can see these for example on `b`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([-0.0025, -0.0078, -0.0085, -0.0410, -0.0495,  0.0299, -0.0237,  0.0421,\n",
       "         0.0292,  0.0320])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0., 0., 0.,  ..., 0., 0., 0.],\n",
       "        [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "        [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "        ...,\n",
       "        [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "        [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "        [0., 0., 0.,  ..., 0., 0., 0.]])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "W.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To apply the gradients, we could manually update $W$ and $b$ using the update rule $\\theta_{t+1} = \\theta_t - \\alpha \\nabla_\\theta \\mathcal{L}$, but since we have an optimizer, we can tell it to perform the update step for us:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer.step()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We set our learning rate to 0.1, so `b` has been updated by `-0.1*b.grad`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 0.0003,  0.0008,  0.0008,  0.0041,  0.0050, -0.0030,  0.0024, -0.0042,\n",
       "        -0.0029, -0.0032], requires_grad=True)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've now successfully trained on a minibatch!\n",
    "However, one minibatch probably isn't enough.\n",
    "At this point, we've trained the model on 100 examples out of the 60000 in the training set.\n",
    "We're going to need to repeat this process, for more of the data.\n",
    "\n",
    "One more thing to keep in mind though: gradients calculated by `backward()` don't override the old values; instead, they accumulate.\n",
    "Therefore, you'll want to clear the gradient buffers before you compute gradients for the next minibatch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b.grad before zero_grad(): tensor([-0.0025, -0.0078, -0.0085, -0.0410, -0.0495,  0.0299, -0.0237,  0.0421,\n",
      "         0.0292,  0.0320])\n",
      "b.grad after zero_grad(): tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])\n"
     ]
    }
   ],
   "source": [
    "print(\"b.grad before zero_grad(): {}\".format(b.grad))\n",
    "optimizer.zero_grad()\n",
    "print(\"b.grad after zero_grad(): {}\".format(b.grad))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train the model, we just need repeat what we just did for more minibatches from the training set.\n",
    "As a recap, the steps were:\n",
    "1. Draw a minibatch\n",
    "2. Zero the gradients in the buffers for `W` and `b`\n",
    "3. Perform the forward pass (compute prediction, calculate loss)\n",
    "4. Perform the backward pass (compute gradients, perform SGD step)\n",
    "\n",
    "Going through the entire dataset once is referred to as an epoch.\n",
    "In many cases, we train neural networks for multiple epochs, but here, a single epoch is enough.\n",
    "We also wrap the train_loader with `tqdm`.\n",
    "This isn't neccessary, but it adds a handy progress bar so we can track our training progress."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f1956cb13de04eb69cbc9e178efe6ded",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/600 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Iterate through train set minibatchs \n",
    "for images, labels in tqdm(train_loader):\n",
    "    # Zero out the gradients\n",
    "    optimizer.zero_grad()\n",
    "    \n",
    "    # Forward pass\n",
    "    x = images.view(-1, 28*28)\n",
    "    y = torch.matmul(x, W) + b\n",
    "    cross_entropy = F.cross_entropy(y, labels)\n",
    "    # Backward pass\n",
    "    cross_entropy.backward()\n",
    "    optimizer.step()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's see how we did! \n",
    "For every image in our test set, we run the data through the model, and take the digit in which we have the highest confidence as our answer. \n",
    "We then compute an accuracy by seeing how many we got correct.\n",
    "We're going to wrap evaluation with `torch.no_grad()`, as we're not interested in computing gradients during evaluation.\n",
    "By turning off the autograd engine, we can speed up evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cf97fbe01dad4b5a99fe8e010f215be6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/100 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test accuracy: 0.9002000093460083\n"
     ]
    }
   ],
   "source": [
    "correct = 0\n",
    "total = len(mnist_test)\n",
    "\n",
    "with torch.no_grad():\n",
    "    # Iterate through test set minibatchs \n",
    "    for images, labels in tqdm(test_loader):\n",
    "        # Forward pass\n",
    "        x = images.view(-1, 28*28)\n",
    "        y = torch.matmul(x, W) + b\n",
    "        \n",
    "        predictions = torch.argmax(y, dim=1)\n",
    "        correct += torch.sum((predictions == labels).float())\n",
    "    \n",
    "print('Test accuracy: {}'.format(correct/total))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not bad for a simple model and a few lines of code.\n",
    "Before we conclude this example, there's one more interesting thing we can do. \n",
    "Normally, it can be difficult to inspect exactly what the filters in a model are doing, but since this model is so simple, and the weights transform the data directly to their logits, we can actually visualize what the model's learning by simply plotting the weights. \n",
    "The results look pretty reasonable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABH4AAACACAYAAAB9Yq5jAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABelElEQVR4nO29W7BmZXnv+4zZdDeCeECJUUCFiBxsQJqzJxTFc4xJuatcFyljWZVK1d6VrMq2arn2vshFKlVr74tV5a3JSlxW5bR15UBKFBXlJNA0chIUFeIhGCIeohGRPswe+4L5DX7vr/sbdPccPXvOj/+/iuL9eow5xjve5/SO73v+z9P1fV9BEARBEARBEARBEATB4mHpaE8gCIIgCIIgCIIgCIIgODLIFz9BEARBEARBEARBEAQLinzxEwRBEARBEARBEARBsKDIFz9BEARBEARBEARBEAQLinzxEwRBEARBEARBEARBsKDIFz9BEARBEARBEARBEAQLilV98dN13du7rvtG13UPdl33kakmFawtIseNj8hwMRA5bnxEhouByHHjIzJcDESOGx+R4WIgctz46Pq+P7w/7LpNVfXNqrqqqh6uqp1V9Z/6vv/adNMLjjQix42PyHAxEDlufESGi4HIceMjMlwMRI4bH5HhYiByXAwcs4q/vaSqHuz7/p+rqrqu+5uq+o2qmqsAJ5xwQn/SSSdVVdWmTZuaY0888cQw3rx5c3Ns3759w3hp6akkpb179zbn8ZivsWvXrgNe75hj2iXgNT1HYnl5eRhv3bp17jU4J/6N771nz57mGO/N+fqLumc961lzr89zv/3tb/+o7/uTDvAohyTHrVu39scdd1ytnNsc47P6ebZs2TKM+Wye8zxZ+3nmjf13Y19s8t6W9Tw98Hz5XL4XP/PvrHMEn7+q1eNHH310EhlWPWmLL3jBC6pq3BYtY57L5/M1Hn/88bnHuGbUEz/7PNnx76vG5chr8tju3bub88Z0hp9nuu+5+7N1l58ffvjhSeR4/PHH989//vOrav+14xp5Lger28cee+ww9nrxM/XZvpB+l/Aac07WuRNOOGEYP/bYY8OYvs/3sh7w2bg2Yz7T1+A6PvTQQ5PZ4rOf/ez+xBNPrKr9Y9q8+3vuXjOCxxwXKUeeN+ajiLG4dbBzGrvG2N/x+a1PnL91d7YGP/nJT+oXv/jFvEkekhyPO+64/rnPfe4B58+5jdkiZeO147P6eQjem/bra/zyl78cxvanvMaYPnKN6e+r2uekz6xq5X2wPzyOxcWp/GlV1bOe9az+Oc95zgGPUSZje595+7+q9nnHfPaYH5rn5+zLiLF7cb6Wx5g985qUh3XG9yZm6/Ozn/2sHn/88UlskXHR60/ds96P7SkJroOfldekTThW0Ya5PmPvEmN7m7H153ztO2jDY+9P1C1fn2v1yCOPTGqLs9g/tjf0XOf5W8eZZz/72cP4P/7jP5pjlCPvbRujXMf2f2OxmsfoK+1TKX/u0avaPRL9g+dBeftZZvN/7LHHateuXZPZ4vOe97wDzpnxyfbGcw/2/Wvs/XrMn1Kf5+0Tq8btaJ7Oje2BPN95OjK2D7VO89iPf/zjeba4qi9+Tq6qf8Hnh6vq0rE/OOmkk+pP/uRPqqpV1KqqBx54YBj/6q/+anOMi0zDePTRR5vzeMzX+Od//udhzE3PC1/4wua8H//4x8OYGwAb0E9/+tNh/LKXvWzuMSr3z372s+a82ZdgVVXf//73m2MzY6lqjcAKd8455wzjn//8580xOoDf/u3f/m4dGIckx+OOO67e9KY3VdX+Cnn88ccP40ceeaQ59tKXvnQY89nscMc2pHTAXJMxZzzmcCknBgEfox54vnwuzr2qNUq+sFrnCMvw5JNPHsYf/ehHJ5FhVdULXvCC+qM/+qOqap+vqur+++8fxt6I8Fw+32yjNcMdd9wxjCnvqnbNfvCDHwxjrlHV/I3sqaee2nz+93//97n3okx4jP6gqtVdb2xoc9u3bx/G//Zv/9ac96//+q/D2C87/PzhD394Ejk+//nPr9///d+vqv11j3rjufzwhz8cxmO2eNZZZw3j73znO82x733ve8P4V37lV4bxK17xiua8Bx98cBiPvfT86Ec/Gsa20ze+8Y3D+JZbbhnGr3rVq5rzKNPZS/gM9L0vf/nLh7HtbZ6+VLU68t73vncyWzzxxBPrwx/+cFW1sqlqNy/2h5QXzxv7ovXFL35xc4xy5N/ZR817KbIN0D+MbZw4X9pNVetLrAvc5NNH22ZnX6RV7R9bZ3H3ox/9aI3gkOT43Oc+tz74wQ9W1f57G85tzBZf8pKXDGN/acN9CWVW1a4RdfbMM89szqOu33vvvcP4lFNOac7j+nuPRR2Z/XBQVXXnnXc25zEGX3TRRc0xymPsyzvuufwSxP3dH/7hH05mi895znPq/e9//wGPUSbeD9I30JdZ3nzeX/ziF80xxkXu3ezL6Iu5b7Evo4/1vegDqYPeX/Le1knGa8Yb68zYD0mza/7FX/xFjeCQ4+If/MEfHHDO9DWnnXZac4y+n7po30IfRLusamVImzj//POb87797W8PY66P58S9jf0KbYLrzz1VVeuHH3744eYYbZhr5fcn6hb1u6rdB/7xH//xZLZ4wgkn1Pve976q2t/+KRPPlcfohxxnXv/61w/jL3zhC80xvtMxztrGzjvvvGHMtfX7BP/O8Zl7kwsvvHAY26fSt/N9uaqGd7Kqds/l/Tv3an6Wf/mXJ8Vz7bXX1ggOSY7Pe97z6vd+7/eqquqb3/xmc+yMM84Yxraxb33rW8OYe4qf/OQnzXn0Y9wPVLXv1/Rr9qe0YV6D167a78fb5hivTzv1l5J8Tr/vUM/4d54v7dT7L8r7E5/4xDxbXFWNnwN9lbXf1+Rd1/1u13V3dF13hxUtWBd4WjlShvN+wQ+OKg7ZFu10gnWBQ7JFb+aDdYHY4mLgkGzRLybBusAh26K/QA/WBRIXNz5ii4uB2OICYDUZPw9XFX96P6Wq/tUn9X3/sar6WFXVaaed1js1aQb+EuBftOalbftbQv6aOfbLKX8N9DWYTcBviP1tO7+ZH0snve2224bxtm3bmvP4q4Kvz285aTz8hr6q/ebX1zhIx/m0cqQMTzzxxH72C8LYr1n+Fp7fZvJbVesDfzHhr5JV7S+TlOdYhsYYuMb+Fp6ZC9QDPxe/rXc6L5+Nc3c2FO/Nb+SrxlMFgUO2xZe97GX97Es8/vpU1a6tf9GirvPXKMuAa8tv3qtafea68Fv+qtZOKVPLl+vJX82rWvnwF1Bms1S1z2k5UgbUXWfwvehFLxrGpso4zXUODskWTz/99H72vP61g2vsXwzoQ+iDvK7f/e5TPxj4C1/6Wv7CyzWoan+p4zyYVVbVZvU4G+hrX3sqi5i/evmZ6f+ceXTjjTcOY+qZ/Q/tz/MYoywAh2yLp556aj/znf6FiJ+dkTXvFy3/IstffK2H/GWSmRiWN/WEMXKMwuBf2ykv6oWzi3jM2YiM67y3ZfPVr351GNsnzPT6aX7EOCRbfNnLXtbPYhf9YlWrY9Z7+j9mjXgPxBhhf8rP9OW+BufBbCB/8Ui/NhaP+JzWA675zp07ax4YT53xw18zPQ/v7+bgkG3xRS96UT9ba/uGMfufRw/zL9T8RfbXfu3XmmO0D96L2RxVrd7SPzjmjO0bmR1JWTlLj3HR/of+lnZvu+K9/SI4m/PT7HMOOS7O4hOzHzw378Nof/ST/lL37LPP5n2bY1yvc889dxh7HvMyBLwOvL5lQ7ulvnB+VW2mhe2IeyzaPWNuVeunrOvz3umEQ7bFF77whf1MDo5bzIy59dZbm2Pc29H+/P5F/+VjtA+ukd+p6G95X9sKZeXsr5tuumkYc29iu6ftvOc972mOMXYzW8k+i+8yr3zlK5tjs3h9/fXX1wgOyRZPPvnkfqbTfl/ks/odmvsS+jjHPmY/Os5T71UyozmPsYsx077q9NNPH8beU/NcvgebtcD9q/fblDevN2Zv1M2q/X3EPKwm42dnVZ3Rdd1pXddtqar3V9XVq7hecHQQOW58RIaLgchx4yMyXAxEjhsfkeFiIHLc+IgMFwOR4wLgsDN++r7f23Xd/1FV11bVpqr6877v73+aPwvWGSLHjY/IcDEQOW58RIaLgchx4yMyXAxEjhsfkeFiIHJcDKyG6lV9319TVddMNJfgKCFy3PiIDBcDkePGR2S4GIgcNz4iw8VA5LjxERkuBiLHjY9VffFzqNi7d+/AgTWvnxW0zcEjP4/8NnLuqlrOpjnN5GKyjoy5o+RZk4/tDi/klV56aVvUnPxEPpc5iOR9mjPINSAv2bVUyOseq7Y/FTZv3jzw8i0n8lrNLWX3MVZEd80cdiSgnHxNjl1BnzIda2vN83yMNQTY5cS8TMrGXExen/xWd1HhNa2Prt4/Ffq+n6sf1DHXSiEvfl4No6q2JoVrSHCtyX03v52c+7EuQO985zuH8Re/+MWaB3L13e2A/sLz4GfWvXEnMz6zueAH2x77ULC0tDSsEdeqqq2P4Q5mrO+xY8eOYex6FKxR4Pnz+eiH3VGEfp66404/YzVqqGesQzDWAcV6QHlTnq5Dw3u7hpC5+VNh7969c2uWMH54rpQXY6TjInXBnHPGHfrRu+++uzmP/HPe13pOW7fdM/7xGq6JQ11z/TGuAWOy63XYNomZL7bPXy1meuVae6zT4RpYbHgx5gsZ71xPh3VFKE/LhvOiHbnpBvXc9TO4/qy34FhNX+u6Ivy7sXp/rFUy1h55SjAueg9JvbSvZBdLrpFrMLD+hus9scsNa1d4n0Xd4Dq4Jgjn67o73BdxTq4LNa+uYlW7V7nnnnuGsevnURfG2sVPhd27dw8ditzFkDK1DtEO6E/td+mr/R7DWiu0Sz8nZX3fffcNY9cwoU92rSH6TdrOWA1E68i8Gk7eU/OZvdcY61K7GmzatGnwZ3x/qGr3NO7yyv0r/QbXuartyMV9XVVb/4Z26n0L/47+2z6an1l/rqqVHeXv+pr0le4qNa9mnvcw1P957zK20dVgeXl5mIP3a1wv19Ohj+O+wR06Gavskxl3qLNeV/rG1772tcPYew/6ZPuVeQ06HENoY57vBRdccMBj9h1jXc7Y9XYM0+58giAIgiAIgiAIgiAIgnWDfPETBEEQBEEQBEEQBEGwoFhTqtfS0tKQcu/0VaanOR2aKaVMW3eqGtOtvvzlLzfHmBLGdH6naJGSw1RQp00z5c+t+0h3YEoY517VpmyNtWNmmprTprmOpuW4heQU2L1795AObPob4bZyTGfkGj/00EPNeaR+MbWuql0Hpq2bGsJ15Txm6b8zMIXXdBi2WqX+ud0yU0ndjvySSy4ZxmMtr5l66HR3p9xOhT179gwpwG7ryBRiU3e4ZkzdtKyo974GZcy1Nd2RNkc5koLkezt9++KLLx7GpK+4la71kGC6Pudruhjna79yJFLal5eX96NpzMBUWaeI0w74DGypXtXqpVPEb7755mFMe3M679e//vVhTAqS08WZCu1WsqTSco1vuOGG5jz6Fcq9qk13Zwrv5Zdf3pzHtTE9wv5jKmzevHlIZ7YPoQ8n7auqTTOnPpsKSV9pShn92Tx6a1VrV/Rl9k9MVXcaMufLv3NaPPXVtJFvfOMbw5j+wq1iuR5OrZ/FyYNtfXoweOKJJwZarmlanJup2owF9Lum61H3nObPmEHag9eE+kz/ZCoLfaFlQ/um7lhvOX/SyjwPynqsXbp1yRSqqbC0tDQ8M9sBV7WUA/t+0qC4J7B9kEpFipWvSb1wTOO6816WFfeKngfXmmvLPUtVS3Px9al33AtaP1/1qlcN43k03inpJVVPycP7duqen4f+hPp7++23N+fRb/hZua6Uk+Mi9/vbt2+fez3K1/sox4MZrJuM3X4noE5zb2PqJnXQ+35TqKbC8vLy4DtMz+FcXS6Ce3m+E7kUA/cE3p+xxALl6NIL9LdcB/tUxmTbIu3jzjvvHMaOT/xsf0iZcH8z9o5s/zM7NiWN9oknnhhitveh/Oz9Je2UcdGUMH7mXrNqPq3W+kJd4tp5vvPKXFS1+kN94X6laj61vqqNu7y3qeOMtWM06jEk4ycIgiAIgiAIgiAIgmBBkS9+giAIgiAIgiAIgiAIFhT54icIgiAIgiAIgiAIgmBBsaY1frquG3i15h6yTod58PxMLqCvwXoX5M1WtTxz8mbNkSMHmbw9c5DJCzQvl20CWQ/B3Ft+NofVNTBmcP0LtoDzupk3OQW6rtuvzs0M5DebJ0pu+1j7R66/78PP5LiaA0mZ8l5uR8r5eq14jLIfa9Vr7ui8Fn/mfVIPzB1128CpcMwxxwzr5Nol5Aub58rnJ+fV8yS3nq1Lq9qaQjzP8mGdEeqFZUX7e9vb3tYco71s27ZtGLuFNPVkjCtM7rZ1nPxa1/lwDakp8MQTTwx1KMjVr2r11zWRqGOsNeD1J8/fnPU3v/nNw5h+0s/NukGs8eI5/c3f/M0wZgvhqpafTdt2HRT62nvvvbc5Rr2lPF3/hTHFfsp12KbCrl27Bjlat93ymWCNCs57rBWvaxnQL5N/bt1mbYODrXPF2gVV7fpRn1zvjz7VbdqvuOKKA87JdSc4R9fXmO0FpqxlsHXr1qGukH0L7c0xge1euSau4UC5ufYD/451AlwzjTUu5rVxnj3LDJdddllzjM/GfZT3K1xb753o1zkn1zChvvhZjlS9rb7vh/oSjuf0j9SvqlYG9C+uHcfzrCes2cJnt09lvKP+uNUxY7drntHHcm1tR3wW7z3pU+nPvZflvtzPMpvHlPW2uq4bdNMxjT7dNZx47Itf/OIw9prweez/KIMxP0b94buPayBxTzFW34R/Zz3gHF33i76Jtuh6W5S119R6PBW2bNkyxEO/23APTT9U1a4Z5zZW88z7V9rpPffcM4ztE1gziXrh9wTWhB2rC8XaZfaHjJOuy0k/QN/hurKcx+c+97nm2Ez+fq9eDY477rjhnmPvGY4fXEvGPuse9zau4UQZ8l6uc8h7Uc9sb7ye6yjy3ZQx3nW4uH/1u95dd901jCkn6xJ9qHWfNafGkIyfIAiCIAiCIAiCIAiCBUW++AmCIAiCIAiCIAiCIFhQrCnVa9OmTUPaklt8MtXTraHZYpIt80wJYKqm2xwzrYrpUWPtbZl251RDUqxMZeGzcU5+ZqbFO4WQ6dekIzhllymPTql2GupUmN3HrTIpQ6fCkTbC1GK3TCa85vOu73Q3psnzGk5fZfqt147phfw7UwOZkudn5rMx/dZ6y3V0mvSRSqPdvHnzoJumWnCuTp+k7JgSahlQ700voR1wLbwupEfRLn29sVbWBKkxTrdlWrxbvdPGOA/bPdNqTU0wVWoK7Nu3b0hHtZzoM9xq+dWvfvUwJg3Gc+Sz2k55LmXNlOOqVoZMtzVtYB79r6qVN3XTekAdtB6Q/kQbHqMYmD7nOU+FLVu2DJQK+6Hzzz9/GNvGaLeMY74G041N62FaPOkHpIAZlLF99Bg19ZxzzhnGXHfTn0h3czo0262y5bVlw9hnStfM1sfahx8qaIveU/B5TBFi6jrnY5ul7djWaZtnnXXWMPaegj6BdCvTCzkn+0lSfzgPrzF9o2VDPeDcbc+kr/j6jhVTYd++fcOcTA2Z16rb86PPs82S7ko/XNXGeuqJaUL0lZSP9Zm2bX1iDGZcNJ2VMvGzMGZS5x0DuBc3VWNGlRnbBx4OZvM2XYM66+ehb+R5fA+oav0p22/7mqQim7bDGEfKmd9b6ONtA9QRzt1UHdqbfQdlTz/svQDpTpavadtTYdOmTfvto2Y488wzh7FpsZQX18z7XNJibDu33HLLMOae3O2yL7/88mFM+/XegX/nUg+XXnrpMP7Upz41jO3jGJP9LklKONfMdHXKzlT52brdfvvtNRUee+yxuuGGG6pqf9oTY5r9Kf0Qn9XXoP8z/Y3yoI54f0RbJOWMvrqqtQmXbqD+cF35fUVVW0aCtlfVPpvfAwnGJe81/O4yD8n4CYIgCIIgCIIgCIIgWFDki58gCIIgCIIgCIIgCIIFxZpSvZaXl4cUrrGuCE6BZVVrpig7LZVpna4gzvQ3pl2y60xVm2rJ6zndkynoTsljijVT0dzRgGlaTq0kNYGplKbB8ZruPuBU5akwS6N1eiRTTF1dnMeYfulU8nmdR6raFD3K0Omr8zqPuDo6ZT1GW+O6Oi2eqb5eD6Y1c+5O5yVdwnowJR3B95zZz1jKoTsEMXV0rIMMUw5NV6PseG+nwM5LwbRezKNWVs3XOz8X5+80WsqR6e2mMNBHeI72H1Pg+OOPrwsvvLCq9vcL1HV3V6IuUrfdwYfUkNl9ZmDq8le+8pVh7C5w9JOkWHk96MvdCYc0AKYE22ZJDXF3t+uuu24YUycch8Y6WzEle2rMo88yjjFlv6r1UfQbpq/xGZ3uzlT1u+++exg7hZjX55wcg+nPr7zyyuYYZUf6iv2PKUoEqUZM37Z+7tixYxib8jxLw5/XnfJwsHfv3mE+Y907TYWbFyMcB3jMMZO2yLH193Wve90wpk+2f6ZfM52Vqeu0MXeP4XPZJ5OGyThhygjjgf2bY/5U2Lp165Cq7w5LtE1THLlOXE/7MvriMVoV45PPI92Rft6dDPl3pMZUtR3xGKvcXYfxwfuWefHOexbamambsz321LTLGW3JNkD/5z0L159z9n7gpptuGsbeU87rOuz9AOMk6TdjcdGg/+N7kd8D+I7geEfQD1seY136HFOmxGzuY52e/V7FTlv0o5Y3ZWJqFv0S6db2CZQj18UxgDJ2DPinf/qnYUz5W7d4jB2gqlp/aBsj6DtMCZxRyabs6rV169Z6xSteUVWtjla1ewDLkH6Idsn5V7XrSrusanWEMchxn3sKnufvBkgtt31wT0Q/7Oei/njfw9g6jxZf1cYG7xNMl56HZPwEQRAEQRAEQRAEQRAsKPLFTxAEQRAEQRAEQRAEwYIiX/wEQRAEQRAEQRAEQRAsKNa0xg9bnrotHlvVkXNX1XIByZ00F5EcbLd9Y70b8jl9DV6frVHNYSdXz/VCyLMj1911bzgP82TJR73xxhuHsfm7fE5zkX2/KdD3/cATNs+ea2muIet0sDWk9YBcYvNk2V6Px9gOs6rlfZKLab49eaRu+8m/43m+F2sI+ZnZuo/XMO+TsvY1PK+p0Pf9wAX2PVmzynrPui/8O/N32T52jL9LDrtbH5Kjy1aNbidKW7z11lubY9RRztHtvnmvGSf5QMfI6TZvn+th2/AaTIF9+/YNPs8cdbZmNaeZXHQ+g5+HNnDbbbc1x8hPZitZ6wHrXfB6bus7xn0m152643oF9D+ug+LaQzOMtR62T3bdm6mwtLQ0xKexVuCOVawVQJmaB099dt23nTt3DmPKf4yPTz/qGhS8xic/+cnmGGtIseWpaxmwnstYzS7GO9c14NpYT2b27FoVq8HS0tIQk9yenjUnrPeUx1gLb+rB2Wef3RzjszIeuTYAazhxH+FW04xHlg3rL9InuD4Y45avwTnS/myLnL/rvzmWT4W9e/cOdm9Z+TkIxsl5NdSqWn/oeoGs2cI6EfbtrKnGObk2Fv0+ZV/V7pVps2M12jxf3o/xbawu4Pe///3m2Kz2kGtVrAb79u0bbMl+jHHZeyt+ZpxhjRef57jOZ6WsXb+K68+9kn085cs4W9Xu/ek7+C5V1dqbdZrzYNx1vSi+k43V25kSjz/++KDrfk/gM7mOE/evlAFrDFa1+03X1mM9F+7rXH+WsZZxy/UfaUfeXzKGcl963nnnNecxrrz0pS9tjtE2acOORWxD7mscCZ/add3gx90enf7J73qMSZS997LUe9sHZc/YYnumrf/O7/zOMHZNKNoi32erqj796U8P49NOO20YX3311c15vKZlw/0xY4H9Lvc99m98PxtDMn6CIAiCIAiCIAiCIAgWFPniJwiCIAiCIAiCIAiCYEGxplSvvu+HdCanNDKFy6nqTHViip9TW3mMbbarqrZt2zaMmXJGCklVm2LFlEm3WRyjfDDFjGnZfi6mKJqmxXmwBajT5Zga6DS4O+64o6bGpk2b9ksvm4EpjKbtMOWZKXNO/R1LGyUdhKnBToVjyirXmC0aq9o1HmuVfPvttw9jt2JkqqHb4s5Lvxyjc1mXrJ9Toeu6wV7cmpA65XRHUhCol7fccktzHltfm2pBXacMTENheiz1nq0Tq9rU8rF0d+qP0+cpO8uA8qEem5LEtTLVlCm2U2Hv3r3DXG2T9IUG01753GOp7/YlvAbT502FZEosr/fa1762OY/rairLDTfcMIyZxm1fwXR06xztiPN1m2jGCacfHynaJVuBj9G57CtJC2N88rz5jNZ7pneTBmDqFPWEem9qCG3RvoM+doyyR7qEqQlM5yalxr6XczTlafZ5zEYOFV3XDb7GLbFJHzRdiOn2fAbT3UldNL2EPo7yMD2CPonx2ZQ3zuOaa65pjvGa9P+mCdCeHcOo05Sv4yL1wnQOf54Ss3nYp5JuZsoH6Tqcm1ugU/6kAFbNL0dgG2OMozwctxjHbc+k25MCY6rpW97ylmHsuEsZ0887jtPO3A57ZitjLagPFccee+ygc9xfVLVx7Morr2yO8dnpx2wf9CcsB1HV7lPok+0LqRekSZpaP6PCPR0Ym2xvlLX1kc/Ga1xyySXNeYwhtgvTpKbCMcccM+i39ZJ+0/t/2gGfyXuw17zmNcPYMYI2zOe1zfLv6Petd/QPjuO0j8svv3wYe3/D5+I+qKqNraQXudzHBRdcMIxNJ37zm99cVfvT+leDTZs2DfstU7HG9gDUN/o77wf4DHy2qnafR3u2/vJdki3hvYfnnsIUd8ZMUnH5HlRV9ZnPfGYYe69Hv8k4Ydrl9u3bh7HpYtaLeUjGTxAEQRAEQRAEQRAEwYIiX/wEQRAEQRAEQRAEQRAsKPLFTxAEQRAEQRAEQRAEwYJiTWv8HHvssUMdGNdWIKfNnFG21SUv0HV8yAV0PRdyAdkmb4y/S571Aw880JzHOgRu3UcOHvmcrr1AXrRbDZKrx7m7FSBrarhmwZQtMmdgO3dzue+///5h7For5KtSnmeccUZzHrmTrmdEHWGNEesS146t+8y3pJ65xhLXlXx+83PJv3frYdZ0IjfVdR5YY8etBl27Yir0fT/ovmuXUK7m5ZoLPYPrC5BL79pTrJXAWhaumeOW3DOYt0y7YhvOqtZ2yLlnG+uqlu9tPjnlxfoarqFBOZo/Pa8u1mpw7LHHDtxurz856vYL5FPTFt0Glhxk12ZgLQJymm0D9K+sh2B7e8973jOMvXZcZ3K1XXOC/HvXVKA/uvHGGw84J1/fdcpcd2VKzNbJHHZ+doygvbCNp2ME/ZxlzHWifdjvsB0xW626Lgvjs1sY027pN62fnKNrqtHWqeOu38DY7Zo4M702x341GKtfyNoD1lnaB23RNU94DbcIp56yRoRBH8S467VjjRDX0uG8WNfHPp7ytU5TZ/h3ji2sX+DrHwl/WvWkD5/FQz877cp7DuolfaXnTftw/GAdkHl12A70dzO47tBYzUvuDTn23oS1SmyL1MPTTz99GFt3eX0fm63VlC3B9+zZM+zTvKegHbkeHfWea2wfN1azi/cbq0HGY1wT1ynhXsz+n9fgXsn2zD2RfS0/02ZZ17Kqredlu/C8pgJt0fVW+J7AWi5V7Xqy9o3r29E2XZeNz8hj1POqNlbR77uWEvXbuk75sFama1Dx3q7jRP266qqrhrHbjv/Zn/3ZMHZtz1ktGe/NVouZnnqNx2q4MT5xPq61y/2Lvzeg7+I7oevicI04D9bjqWpjjt/5qWfci7s+G78bcL0o+tN57z5VrS3s2LGjOeb7zcPTZvx0XffnXdc92nXdffi3E7uu+3zXdd9a+f+RicLBlHh55LjhERkuBiLHjY/IcAGQ/c1CILa4AIgtLgRiiwuA2OJi42CoXh+vqrfr3z5SVdf1fX9GVV238jlY3/hRRY4bHZHhYiBy3PiIDBcDH6/IcaMjtrgY+HhFjhsdscXFwMcrclxYPC3Vq+/7G7uue7n++Teq6o0r4/9ZVddX1X95umstLy8PaVtuwce0YafUM9WLqdKmfDBVymlfTItkOtdY2h1TsZwuTuqX70V6DFPRmObl+ToFn/eb1xLZx5yKK8rEY1XV9hg8DDmy9bBbT1IepANUtemgnKdTYPl81hGuCe992mmnNedR1kwf93nE5z73ueYz0zaZ1u1UOqaVkiZS1aYlMs3RqcNMDzZlR6nRk8iw6sn0+5n+mbLH9TNlhumTpGw53Zby8TMxDZbptraBeenQbv3ItrWmGvEz9dNprtTDsZaRTOk3JYnzYqp+1X70m0nk+Pjjjw+UNdM/5qUqV7XPx7aoX/rSl5rzaItOKeU1KF9TCtiqli3cndJOmoupdrQPpum67TvnyBa2VS3dgM/stqtM4fWaKlZMZotVT/k2U2Sp96SOVrU+lTZs3WZ8MhWSz8S/s3xIQ6FeOC6SYm2qF+Mf6QiUvedkW2dspY+2jpOCYX3inKfa35CW4DR/ys1+knGS8zJth8dMGWeaOXXCNBf6K66J91tsaWtqDvdfY7bI/ZypJ7feeuswnrUQrtqfXsIYYr+udP3JbHHz5s372cgM1FPHetMHZhijkFiO8+gIprkwZtJHWc+5b/F+jHsw6ytBHTJthDGO17fN0qdZ12b2vHnz5knfNWbwfo32Z51ijJhXXqKqlc0XvvCF5hj3S/RjpoRRVtQ37ylI7/G6kqZFX2i7Z2wwpYbXoB5Yv2nP3i8Kk+5RZ/7Mz865Wu9Jg6KsvO+mLjiOUWf57NanM888cxjTBkxJoj6Z4kMKGuOuS3VQt6yTXAPq2jXXXNOc94Y3vGEYez8xW+MpbZFx0aD/8J6bsYD7RtPQ+E5nvaQMaFeOwaRVXXfddcPY68/YZ6od9YWyMX2Zz2mdo83xOX0NxkU/87w4ZBxucecX9X3/SFXVyv+PDMkzONKIHDc+IsPFQOS48REZLgYix42PyHAxEDlufESGi4HIcUFwxLt6dV33u13X3dF13R0uFBtsDFCGUxbEDNYWlOPYr3zB+gVlOO+XlGD9I3Lc+KAMnfUbbBzEFjc+IsPFAOV4sIVqg/WFxMX1j8Pt6vWDrute3Pf9I13XvbiqHp13Yt/3H6uqj1VVveIVr+hn6UzumDKvc05Vm1rONEZWmq9qU/Kc1sc0WqaYOfWRdBMqrdOV+SXWXXfd1Rz7rd/6rWHMNE7TiUiDcCoj07u2bds2jJ0Wz79zutxBdKE5KDlShieddFI/o0r4y4Ox1HumQTJV2V8kMZ3YHQh4TabWueMQU3iZrud0dHYGcMoc5T2vI1lVm6bplDzOlzQK6yarxbuT2bwuWsBh2eJZZ53Vz+hHt912W3Me9cjzoT7ffffdw9hp00wzt57QdnjMHW+4LkyLZzpmVdU73vGOYWxaGWXHdFincdLWnabLFEzqialAlJVt0bTIA+CQbfHlL3/5YIukL1Xt75MIdlzjutoH0Rc6xZabMqbbuusFaSj0R+6mSEqJKZ6kr5DCZX2hX/GGg3RAXs8dcign663Tew+Aw7LFk08+uZ/5OvqkqnY9TVVl+jJ1z5QM6qJpAKRm0V+ZRs2YTNu232ScNbV5Hi3Wqfrz6BJVrUwYO9zFk/rpDlAz3fA+AzhkWzz11FP7mW984xvf2JzHWDJG15jXba+q9UmmYdD38jy/OF100UXDmP7CHUpoA7NOLwe6F/3D9ddf35xHvfVegP7/05/+9DDmPqeqla8px+4GegAcli2+5CUv6Wd66o5c3HNYd2hzXHdTN/lM3nvSTmnrtKmqNsbRrxmUo+2Z9sLr2Y6on/bLpExQxqaV0ed4PzFbR68FcMi2eMopp/SzPZb3l4zZXjuuA/dotlmuq2kj3IvymbwvmUdzcbdR0oLYebSq9aeMp57vWDkCPifHfq4xeJ9+AByWLZ5yyin97L3QcXpeuYCqVubelxLck3s/wrWgrbtjJvcgF1544TC2/6a8/SxcP/pv2yLLEdi38/2ZFC5ThtmVeJ4ujMjzkG3xec97Xj+LE94bUi+972HnbcraNHbajteEexPu9233//AP/zCM6eNYyqKqfX+0P6VMSf8zxZP+x13DuAa8vsuhUB9dUuJgf9A/3Iyfq6vqAyvjD1TVPx7mdYKji8hx4yMyXAxEjhsfkeFiIHLc+IgMFwOR48ZHZLgYiBwXBAfTzv2vq+rWqjqz67qHu677UFX9t6q6quu6b1XVVSufg/WN0ypy3OiIDBcDkePGR2S4AMj+ZiEQW1wAxBYXArHFBUBscbFxMF29/tOcQ2+e8+/B+sS3+76/6AD/HjluHESGi4HIceMjMlwAZH+zEIgtLgBiiwuB2OICILa42DjcGj+HheXl5YHHZm4aOdKuv0HOKjly5r6RTz1W4I31H8zBJn+X/E3XkyAv3i3U2BaZvL23ve1tzXlsJUxOY1VbD4P1Nb7+9a8355HXaN65a5BMga7rBl6lecCUm2uCkE9K2bsWCbnt5liS/8raA26LSrAVqnWOcnOdINayIS/W9V+4/q6pRD0jD5l1gQzX9HF9iqnw05/+tK6++uqq2t8GyJs19578WPJQzdGlnrquCGt68O/YVrGqrQfAOZpnzZoRrN9S1fKdWdfCcuQ87Fd4DV7fOnPVVVcNY7cT95ynwkxW9gvkT9uOqIvkTLMeT1Wr92P1HehnXHeNvGhy1r121Hvz4+fpiGsZ0Ee7jhZrgtDuHWsoJ9cV8eep0HXdwKu3L6N/NK+cfoTyds0D+jLXCaJdUaYPPPBAcx5reJA7b/9EHbLds94Ca/ewtXRVWw/DrW85Dz6L/RTl6ppHM969dWQ12Lt37zBvtiuvautHeK/ANaENWNaUr/citDn6btekoq1zjV3Hh3D9JdYsYC0Ix9avfOUrw5h+pKqNyZyT6z7SX3se9h9TYdeuXYPOuY0672k7Ym0F1uSxj6IfdV0R+lHqguMR7YrXv+mmm5rz2LrZ8Y42wWOuI8Nntp3SD3BO3sMw/thfzPz5SI2fQwb9qet+cC62f+7faB+uK8J9tusZ0Sa+9a1vDWPX3mB9LOq5ayD95m/+5jB2jUraHOuI8b2iql1bN9i54YYbhjF9t2sjMSZbD6yfU+Gxxx6rW265paqqLr/88uYY7ci+gM/EVuCOi4znrvvCNeQ7oXWB9dwYZ//yL/+yOe/tb3/7MPZeme8J9Jv23/w715ydV7PLa8M9A/Wz6im/NWVc3LJly6Av3lOwFo7rDVH/qPdc46o2Fvr9izUQabPeU9COuL943/ve15zHOfodjrGC8Zh1UKvafbrr//A7C/qLg61zWLW/js/DEe/qFQRBEARBEARBEARBEBwd5IufIAiCIAiCIAiCIAiCBcWaUr327ds3pCY5bZ7poU5HZBobU/CcdsdrOoWfabRMZTMlg2mRPOZ2jEwdcyozUyaZzuv0PNIPvvnNbzbH5qVW+l5M7XJK+8GmfR0Kdu3aNVBf3JaQqWpef86FKYu+BuXrlEOmx/L6TqMlfYFyd9o108fdbpvp+pS101w5f6fxM5WPKZeWIalEV155ZXPMc54KS0tLQ6q2bZG653RTyof6bN2mDKyHtAmmuV522WXNeUyt5BzdopP+gpSIqpaaxTk5tZfn3X777c0xypGp6tbdr371q8PYNBenX0+BTZs2DWtk3ePcnO5LKgJT1a2XTJM37YHPwzU3BYJp7PStpr797d/+7TB2G1PSRmn3pn/QTh0bqJ+8vik1lL115Pzzz68jgaWlpSHF2NQErp/py9RLpgPbPigTU8nmtYG/+OKLm/P+9E//dBjPo3hUtbZuKjDPNa2TYIq2W1lzPUgNcgo4fYJpLrN5TZnS3nXdcD0/N+kfTkenXtLXmipNvTddhtd3WjhBH0G/63brN99889zr8TNpKG4J/5a3vGUYm/rGNaDPOffcc5vzaM+Ue9X+NKmpcNxxx9X27duran/KEvVybB/GuXr9aDumN8yjlFt/ud85WPqBn4V7MPoE+zzarPWE9+P+2CUGeH1SEaueWivPbzVYXl4efKVjNNffbatpc6SlOPbxGayH3/nOd4Yx1/i6665rzqPveeihh4axKZOkhngfNdPTqtZ3u+QDY4p900UXPVWOh+thujXjtd/PXCpgKmzdunWIXX4m2oqPUU/5zuA9KuOkYyt1mO8GflauGfdEY7R5x3jaN6l+pv1xv3P//fc3xygT7n28l+I8fGz2d17P1WDLli2DjdN/VrUUQb9XcU1Iz7Qt8r2AMa2q1QPSu2xH9GN8J6evqGr3ivZj3GNzP+zzuAb+TmFeyRL7Ka6b93pen3lIxk8QBEEQBEEQBEEQBMGCIl/8BEEQBEEQBEEQBEEQLCjWlOp1zDHHDGnKTrtjWpZTsZjOOq+DRVWbEmXaE9OomGLlytqf+tSnhjHTLJ2yy3Qu0xZYaZvzGKMuuRvRvA4PZ511VnMeU0udHu603SnArl5OCRxL5Z/XBchUAdKenGpHWgopUJY150Xd8XyZOum1Y+qn0xAJysaUKVacJ2XBqdtM73VnjtNPP33uvVeDvXv3DumJXmfahNNSuYY8zxQfpsf6eUlp4POZrsSUedqvuwMQTkMmrYAy9TPTHzkdmunclLH9D9Mux7pWTYVdu3YN9m874vOZrkddZ9qo09bpa0055Gemlzrdluv/5S9/+YBzqGrt1GnMpFgxLZ5p6lVVf/VXfzWMnWpNH0G/a/oCdc4ydDelqbBv375B921v7PbIeVe1esoUdNuiaVsE09MZg/ystFPqvaly/OzUZerJu971rmHs7iKkbTk9nPbNWOgYQHuwrc98k2mhqwHpJfYLnJvXhOn2jG+mWtA+7J8Ya/lM9gn0cezk5fWnHzBlm3ssxs93v/vdzXm8JtPnq1ofSjt16jtT8Me6vkyJ3bt3D3bmdabPcqznnobydkzj3taxivGDcdb7FsqYftg+j/sKX4M6yrg4jxZZtT+Fiz6CftNlC6iHpuXP1nFKegnfM7wPffWrX92cR7CDIjsJuQsT96je05NSwv23YyvXnNfw+wj9hd8R6CcZG+z/P/OZzxzwelWtbfL5/Q5Gu/c8HMunxOy+psCR/uMYxL02Oyzb33/2s58dxu6Y9slPfnIYszOYdYY+i+vufRD/zjT0nTt3DmP6hOuvv745j7ZpahptjNewnxyjMM72iVNSoPft2zesBbt4VbXyMCWTOkUf5DjP+GQ7NVVuBj8f701/6r0Ybcc2xu8A/u7v/m4Yu+vfnXfeOYwdQ/hslKHjIt9HHUO8X5qHZPwEQRAEQRAEQRAEQRAsKPLFTxAEQRAEQRAEQRAEwYIiX/wEQRAEQRAEQRAEQRAsKNa0xs/y8vLAeT755JObY2M8+Hltes3VY7tg1qeoavl/5K+6lgFr+bD+A+v9eI6uBXHVVVcNY/IH3XKWz2xuL7mB5Bqaw0eOozmJ5oFOgc2bNw8cQ9c9Ig/fLWfJ02T9F/PGyWNl/ZGqVvbkYrru0eWXX37AObnm0Y033jiMXVuHIE/WXFTWbnGLVz7zWE0izss82Hkt/laLLVu2DNxo6x51zFxlcsJpR54neaiusUG+M23M9YzIQSZf94orrmjOo/25VgD5+Kwdc+GFFzbnsRW7ZUD5UI7mjL/uda8bxq6NYd2YAqy3Rd9X1XKk3facdkRZmA/OY+bx0w54Pa53VdUtt9wyjKlLtkXahOshcF6sg2Fb/NCHPnTAv6lq/Tx9judBnbYPO1ItpPu+H67te7Iui+sRzeOju60z6z84RhDUUdeYYSzhHM0/p95v27atOUZ/TttxS1Je0zVHGBdZL8Rc97F9gus0TIVZPByrQ+DahvSNrLdkWfNZXUOFz849xj333NOcx7oYjE2umUZ5+Floc6xrMFa7yPueeW2JvWejbToOca2mxKZNm4baC65rxnpb9hus5UM78j6X62Tdpo1xnd12+eqrrx7GrGHDfWdVu2audUO94z7OsqLfdw0N1nphzSjHPj6zax7Nrun4tRrs3bt30A/rFHXPfoF10uirrHuUzY4dO5pjjDv0M67xx7o0XNdLL720OY97WdcVoR+gnnlfwlp4rntC2+ecWEuvql0D24X3d1NhaWlpuLbr29EWWee0qt2DMNa73gr9nN81qAuMY/bfv/7rvz6MqU+OR4ynlgGfjTWiXJuR9WEcM2+77bZh/Na3vvWA/17V1pWZt8+duvbdTL9dp4k+ye/hPJfz4X6yqtV77z35rsLYZ19D/8c19l6JtSbtJ2lH9P/+boA+3jrC56TeUterWl31PsG+ah6S8RMEQRAEQRAEQRAEQbCgyBc/QRAEQRAEQRAEQRAEC4o1b+c+SzVz2ijbBfoYU7Z4zO3omLLlFGWm8DP9yun7TAdkSt5Yy/bXvOY1zTGmqTHV1OnQTEd3+0RSVu64444D/ntVm4421nJ+KvR9P1zX6atjrfC4zqQiOM2c16ROVLWps0xTZFpmVZt6ymuY/kFZe13n0WHc+pbp2m7TyLaSnAdpRVWtHpuW41a4U2HPnj3D87vtL1tw+3mZssrnc3o30yn9TExhZdq008CZzkqbcron0xudgsln4dippZSB24NSBrQxU2roL0hnrBqn2KwGM50jta5qfx9KMGWcFBLbIlv0MlW2qk03ZYqt09GpB5Svaahcn/e9733NMdoVr+92mLyG15u0BPp/PqPn6xRyt3eeErP72qfSb/p5ue5M+bUfooytl/axMzh+MqWatsO27FWtzVrGvObXvva1YezUd87JlBquD1PrfS/aqakIszjgdVoN2LbW8YjPYyrfvDmYQkcfahoG4xrv7ZR2ppZTno593kcQpAUzbpj+cd999w1jUxZIvyYNxTR2ytr0PO/9psLu3buHVH1Tj+++++5h7PbyfH7T14kxyi+vSZ2xT2Dcoezsrygr+3buDd/whjcMY9si9zeWAfWQ83BZBPoEU85m8XpKWyRFiFTCqvl7w6o2ZtDvem9DipX3APPguEgbu/jii4ex6TDnnHPOMLbu0P8x3nsfRZ9viiTXg8/FvZ3/zrH1YNfgcDDTCz8T5zNGB6ZduqQB93WmgV1wwQXDmPr76le/ujmP/vbaa68dxqQ+8jl8Xx+jvlre1AW/23Hfe9dddw1j00Tpm+znZ2s1pS2yvIv9E/2d338Z9+n/TI/iO7XtlFRnvof7GtwfUbf9LjFPr6pa/0q6nvWKemxaFnWJ+zm/q5B6aCqZqV/zkIyfIAiCIAiCIAiCIAiCBUW++AmCIAiCIAiCIAiCIFhQ5IufIAiCIAiCIAiCIAiCBcWa1vjZs2fPwFU3n5s8cLcOZj0J8jndvo21OVzPhbVEOHbLU/I52UrRLVTJnTQnkvx58ij9zOScsuZBVdvWkeeZ68raFa7rYf7fFOi6buBjug4B18Q1FzhvPwNBmboeAutx8LndivHcc8894LVvvfXW5jN5mq71xBoC5mkS5JWax801oGzcYpR8Zbdp9DpOheOOO25oT2g50iZYF6eqlQFrjlj3WHPGa0vOLmsGvec972nOu/7664cx6wuY0037cB0Ctsqkf3DdD87JMiDvnn7ANVfGeOfmBE+BzZs3D+tsn0muLznMVa1vZJ0O+0zWcDAfmRxz+lPXhGJtEtqKa8iQe+5jtHXKwvUiOA/X42FdLeqw66CQT+1aLZzHlKAcbYuUieMMP3PNHKtYr8F2Sp49Ofi2I9b+oE/weZyTW/DyM9fdujvm8zhH1kqwHClj17WY2YbbOa8WMz83ptu+J2MhY46fhzUQ3FKY59L+xmo10HZc64R1LNxemn6A/tQ1wBjj7E/nyd7+h8/s2iesYzEljj322Dr77LOrav91ZkyzjFkbgvsA17u58MILh7FrfTBmUC9ss6ynw7F9B+3Ideuoh7yG4xZl4n0W5UO7tO7u3LlzGLvmyJGqQzmTj3WPcdjrRfujfP3c3G+4/gafj7biPdD27duHMWuJuK4I199+knsW6p/1hS29WSemqo1p9M+ul0IdOeuss5pj9vNTYc+ePcP+0DVWWQPGtXBop/RtliPX9sorr2yOsY4Kr2f5cG/FuGsZ0C+7DgtrzrCu2Fg9V9sz68pcddVVw9h6x3bl3C/x3Klr/MzWwvsp2offMyjfHTt2DGPPmfsX78/oW/ieYZ9DuTFWe2/DWr6ulcjr873Ve1T6Xdsi96jcv7oOJevnWQ8OVnbJ+AmCIAiCIAiCIAiCIFhQ5IufIAiCIAiCIAiCIAiCBcWaUr327t07pFc6dY/pek5tmpfG6HabTI1zqj9Trpgi6dRWzov0K8+X6bxupcx0qzEaCtNOnZLKVDKmxLm1J1Pi3Jb7kksuqamxe/fuYS29JqREmWbGFDfSBpy6xxRTps9VtdQapukyLbOqTYllCrLpPVw7t4fl/Hl9p9sy7dWUGtL8SD9zmiBTIN2q90i2ypylwToFlimT1jfqMNOh3/zmNzfnMcXdFEfaC9PFLW/KhCm7tqMxSiPTakkjtN7NSw+uau30TW9609z5UsdN7TINcAosLy8POmdaAqmz9gtMLaavdRtipuyb1kYKF/WH7TWr2jWiLjsFmW2dnb5KH8r7uqUy0509Dz4L18MtRqn7pgKMtbleDZjS7lRd2oepzUxVp0ydZs7YajoN9dT0FYK+k2NTlDlf0394L1IkTO1iDDA1inpoX0zw73yNWTydMqV969atA3XHa0wbMFWAc+O6um0wn9V0GbYA5h7Ifp0+77LLLhvGXgfaKdPPq6rOO++8YUx7M/2DMrWtc89F320/RV1ym3HvPabC448/XrfffntVtT6pqt0vOH7QL5E64FjPY9ZL+h4+n+MM7YNr5BhDf8HyA1WtHY1RDOkD+YxVrT9k7LA9szW212Pm2x1zV4O+7/ezwRm4d5hR+mag/+N5tiP6Wreh5rOT8mEqC69Je/N7C3XO7xlsH05d8j6U9mbqIefB9wzTt6ibbuduGuZUOOGEE+qKK66oqv0p5PzsvSfXjLZi+hptzDQk+mnqpmnU9Lds4U46TlVbNsTHaLd8p53Xbr1q/xhP/aJPtY8hvcg2MrP9KW1xy5Ytw9zsW6hvlFlV6/O4/7MMP//5zw9jx0yuA2Pml7/85eY82vpYuRjajm2MNkHbcUyjfVuGjJNcD9+LumnbY0weQzJ+giAIgiAIgiAIgiAIFhT54icIgiAIgiAIgiAIgmBBsaZUr67rhjQyV7tmSqlpBUxVZFqyU9KY5mTaAlOsmHJm2ggpOfNSP6va9Fh3n+J8maJsOhHTXj3feZXyTWFgOqBTbF3tfgp0XTfQf9zph+l5ToGlrJiyamoTn8dUDqb8UUec/sc0OdLPnKLKFEtTPlj5netoSh7/zvpIefM5TakgHcm6f6ToJUyHtu4xLdydBTh32tTdd9/dnMcU04svvrg5xhRHPrtlwHmRNmIKDmkQ7pjA1E3qnVMw2fnHKdVcD+qF9ZNUDadx2m6nwJYtW4a5+bnp40xxZJcnHrOcKFN3vZjRIZ7uXvRjXMcxCo/pAPQzTIU2hYSp204dJvhcngd9qO35SPjTqidj2qyLj30U4527VnBdmJbsOEAaoCkq1FP6b1OgGf+oy6ZbMS3Z60eZkDZiOiltzNQHynxed6Oq9plNC5rtPaakevV9P9iBu4ZwLd0hkhQDpvJ7v0Fqln0XbYK2Y/oCaXmkl5hCyPk73nHNOA/7f/pux2f6TV7D60Y9s1+xnUyFrVu3DrQA2zvlYxoA9yY8z7GVGIt33Ie6+ws7flE+9pv0baYHcs9Kuo9tkXHRVA36i3ldwjwvy3Em/ynpJUtLS0M8NM2c9C7Ll/Pk+rsrIH2oaeZ8dvph74cZr+kffC/6V9vpvPcHz2mM3kb/T3vzPpS+m1Q3X39K/PznP68bbrihqvYv6/HWt751GNv3cG0pY8e+eTTbqvZdknbqvSf39Vw/vxP+/d///TB+97vf3Ry74447hjF9tq9B27Q989kY+7wXoA2bQjTr8julPPu+H95hvFcY86d8Hv6daZfUU9OcaB+kLHvfyPdAronfAxifPA+C9Cv7U38m5pVdMJ1rrGuY4/U8JOMnCIIgCIIgCIIgCIJgQfG0X/x0XXdq13Vf6rru613X3d913R+s/PuJXdd9vuu6b638//lPd63g6GDlV4RXRoYbHptjixsbscWFQWxxg2N5ebkiw4VAbHGDI7a4MIgtbnDEFhcfB5Pxs7eq/s++78+uqsuq6n/vuu6cqvpIVV3X9/0ZVXXdyudgHWIlpfHhyHAhEFvcwIgtLhRiixsfkeFiIHLc+IgMFwOR48ZHZLjAeFoyX9/3j1TVIyvjn3dd9/WqOrmqfqOq3rhy2v+squur6r+MXYut3cxNY/0Cc+/JQyX/0JxXcvnNwSP3jbURzOMjl5S8XHOF2S7ZPEpyC8nfNaebx2677bbm2IUXXjiM+SxjLWz9zDNO4grv+PGq1ctw06ZNAx/WnG/Cx8jzJz/ftU8oD/OdKQ9ycs055t+Rh26uPOstuN0pOftjfE5e03Vd5l3Pa0MeslsS7ty5kx/39H1/Z9Xq5bh3795hbVyDgevJegJVLUeVNsU2v1Vtu2bXcZpXk8Jta1kfhtezXrAOiOsLsP0mfYz53uTRWj7kkJsfTHDd7MNm11ypbzGJLe7atWvgc7tdLFs8v+1tb2uOka9PrrjrSc0431X7c9vntYP2PN74xjcOY8ravOVLL710GLteEf0k/b9bibPNpbnUfGbWGHGdMvp5t5VX7YTJbHFpaWmwQddvod+wfZAHznoF9kN8RtcBmdf61jUe6ANpz9YZ2qLjIufBuY/FCte6m9fG23rHedmHzWLrpz71qfrhD384iQz7vh/iLWVW1T6r9xGsKcCYNlanw7GK12d9B68rawPw+mN1A13rifeijbkuBGs2+Bh1iXN64IEH5p7nGCVMZot93w8+3vV5OAc/E2tIcG2tC6xrwVprVa0/Y2y1LfLvWNfCdT+4R/X6sWYe7dT2TH21PbMmBWu+2X+7PgsxW4+u62oqGR5zzDHD+nlvyM9uzf2ud71rGN94443DmM9Z1cbCM844oznG/TljnPclZ5111gGP2XfT/9l30L7t4+bBNcYo7zF9oR9wfS3tWSezxa1btw6643nzXcD7euoiazW5lf28ujg+xrqWtgG2lWedKusd5X3zzTc3xxi7WBfUey7qmm2M/of65P0E92reA8/mv2nTpslsseu6YT5j9W1cb4jzZFy0HtDevJfjNRnjHFt5ffpaxyPagGUzrxbjWI23sX0P4fjMmlDbtm1rjo29qxKHVOOn67qXV9UFVbWjql608qXQ7MuhX5nzN7/bdd0dXdfdYeEGa4/VypAbl+DoYbVydAAL1h6rleHBbvaCI4vY4sbHamV4pIoNB4eG1crRLw/B2iPvGYuBxMWNj/jTxcRBf/HTdd2zq+p/VdV/7vv+P57u/Bn6vv9Y3/cX9X1/kX/9CNYWU8jwaX55C9YAU8jRvxgEa4spZOiuYsHaI7a48TGFDN0VJlh7TCFH/0IerC3ynrEYSFzc+Ig/XVwcVN+2rus215MK8Jd93//dyj//oOu6F/d9/0jXdS+uqkfnX+FJ7Nq1a0hNdatRpkf5m16mP45RZpia5fRMto/l373+9a9vziO1hmlfpniQtuW0X/7iwG883a6Uv9i7bS0za/hrolPH+Fxnnnlmc0ypoV1NIEOmtJtqx1RRpzoy/ZApf6baUb6W4bz0P9MGSUXgvUwRIpXDFAjKlDJ0S2WuP8dVbTqqZUNwrTwPv9xPZYtd1w3zsxyZUmo7JajnTptmSuZFF13UHOP9+OzOJqO8aCu2RdqEU0GpM0y39S8RPM+pldRr2pRThymr7du3N8fUXnwSW1xaWhpSeZnqW9WmONuf0sZIETLNkH/nX1GZXk3/ZPrCZz/72QP+jW2b15hHWfV8/bJNKhSpDFVtSjbXxm2EeS/TXHiNqulska3A7d+5LqZyMM2ZccvZJ9RLt02lXyLl1NQExifGRVMTeH3HNN5rXnv4qlb+ni//jr7dqfqkEDFVv+opG96zZ89kMty3b99gL9TRqqp77rlnGDsG0b+Szuo0fOr6GOWDFCFvuq+//vphPEaBHmsJ381pV20KHtfcbbPpv+lX7BN4L+uBf4CaMi7O5Od4zra/jtOk33Hd5febNTMF+p3vfOcwvvbaa4fxjh07mvOuuOKKYUz52HewfIB1gTLh/sl7Adqw/SEp0PQJjuPUNfsVvtxPJcOlpaXB5zkrlvs30mqqWj9EO/KaEI6tXGf6YT837fu73/3uMPZelnpuiif9P/dRfh8h1W6Mwsv9qtvP005f9apXNccs7yltcfYsnvdLXvKSYWzfT/oLj11wwQXNeVwn7yXuvffeYUzf41j1mte8ZhiT/mNZUcaO4/Not74XY4f3N9xjk0pmKhB18q677mqOzXRt7969k8mw6ql46PIH9Fd+X6QdcY/h90Ve0/QrXp8ydMtz2tzll18+jFkuoar1Y34W2gspnt6L8VkcW+mP+H7i7zlYFsF75bFSFMTBdPXqqup/VNXX+77/7zh0dVV9YGX8gar6x4O6Y7DmWFGGl1VkuAiILW5gxBYXCrHFDYwVW4wMFwOR4wZGbHGhEDluYMQWFx8Hk/Hz2qr67ar6atd1d6/82/9VVf+tqv6/rus+VFXfq6r/7YjMMFg1VrKTXlBVV0aGGxrPrtjihsbKr7yxxY2P2OIGx8qvgJHhxkdscYNjJdMyMtz4iC1ucMQWFx8H09Xr5nqSmnAgvHna6QRHAispyl/p+/6iAxyODDcOHuv7Pra4gbFCiYgtbnzEFjc4tm7dWpHhQiC2uMGxZcuW2OJiILa4wRFbXHwcVI2fqXD88cfXxRdfXFX782ZZm8OcUfJhydVza3PWbjCvn9ckT9C1Ky655JJhTK6+udrkWLoIGfnT5Cp6vnfeeecwNt+Y3EDylN0Sj9c33881FqbA0tLSwIdl+2TD/EXKY6zVK+sNWDbUGdY08TXIvyev91DqZ5CLSW67W/xR9uQdV7X1QihD8z4pJ+s+a1VMCdZqcm0FrvuXvvSl5tgHP/jBYfyZz3xmGLu2C9fTtXC4tuS+kwdd1XJ2uX5eI9qm2zrz2cZqmLB+g+fB+hVsueu6E7y+uf/mxU+F2XO4phB5454n+cPktptLTE65fRyfj5xmc7Df//73D2PajteHa2xfS/0Zqw1DDrZ9H/36GGecsvc87COmAmsZuOYQfZnr6ZBnzrpcrjHDenSuGcb4wfV0TRDWQ6N/GGsX67oJ8+qAuJ4EfYlrO3CfQD47dbCq1XH7N+v5FNi3b99wz1m7+BloR45B9Gt8Btf6YM0I1zJgrKX+sGaQ58X6Mrbts88+exi7lg5lz1pDrnnANtFuEU4bpjx9L9ZiYJ2kqv39/FR44oknhnp1rmnDZ3esZ4076pvlTV9m38M9JXXU9S/oR2k79qn0HZSpr0k9sf9h7LBfoc+mjN0umZ9tG7O9kO+7GuzevbvxeQR1ys/Dmh6sB8i4UtXuFcbqV3F/7NobrBfHtbMvZBzzMe5h6E+tc1x/x5B59dq8R6Xu2049r6mwvLw8vDO5NhrnZ/ugf+H6uS4X18X+hLGQewLuZ6paPaE8+G7n6429w51//vnD2DXAKGPHReoo7dLvhIwxfF/mNeyvV4O+74fref1ZU9LvxvNqMjoOjNVfOu+884YxbZt7mar23fTWW28dxn4np567/hvnwfc+xwnuiWxHlC91zrU36a+9z7VuzcN03jYIgiAIgiAIgiAIgiBYV8gXP0EQBEEQBEEQBEEQBAuKNaV6PfHEE0MKl+kaTHdz2ifT+pjC7ZQ0pu45ndtpnTM4ZY7pjmyD+JWvfKU5j+nRpq0xZZdp2GyjXNW2t3XKFtMVmbrnVq5MJzU1asr02Rn27ds3pMY5jY1r7rbaXGfKySnIpFc4hZOpcaQ2mJLBdEamEJrmwJRYp0TOa8nqNFqmV5sK4lT7Gdyql8/lVFymDU6JY489dqAf3XHHHc0x0uOcgvnggw8OY6a5Oo2TKZluqUr5cz2dWs/zmLptWyY90HrHe5NCYnoDZeX0beohdcjXoL2ZSub28VNg9+7dAyWEbWqr2jk79f5d73rXMB5rtUt5WIZcS/ph2ywpILQP+6pPf/rTw9j2wTR50ihMh6EtOj2fMn3LW94yjL1uMypy1f7PYt2aCktLS0NscFykvjne0aeOxcWxWMXYwtRm02dpf1wz0wipF2MxjT7PMmCavH0Cn4W+yanXpJRYF2a+3nTw1eDYY48dYtINN9yw37EZnF5PX0Y6g+MR15mxr6rqe9/73jDmHuOVr3xlcx7jM/c2pnXQlzt9nj7OsZCgnGw3pNswfd50WO51uFeqmlZ2xLOe9axhHqZO0Q+5hTRT+Ok3vEa0D/sXxn7qtvcEpHrx+o6L9KM33XRTc4x7JO7jHANIR3MMmEffdytr+gFS4qqe0kPfdzXo+36Yj30QY4Ypk5Qh/bBt1hRWgj6Uum3qGfd13GuOtVu3LRK0y7H5mRLLeZAmYhoK32P8LI7XU2H37t0D5cWlI/iMpN9XtbT3v/7rvx7GnufYfo1UG8YSU2loO/S9tvs3vOENw9jlIihj2r3vxRjn91vGSe6tLG/Gf9O0Zz57yvfG5eXlwX95Tfjc3kvTTzIemfa0c+fOYTwWF/lM9us8xnca75spX5e24N8x3pmiSJqo15/+gvtm723om7Zt29YcMxVuHpLxEwRBEARBEARBEARBsKDIFz9BEARBEARBEARBEAQLijWleu3evXugVDh1j1QLp/wytZApbU4DZ0q3UxWZpssK30x1r5pf4dvpb0wTdXo+U6yZdu+0O6bwOo2T6ddM3fO9nCJHOM1sCnRdN6Skef2Z4mZqCyupM53RVCKuiSkLlC9TLE1LoAyZnuw0Y+qg0yp5DVIRnGLOv3NKNufIKvVj9AXL80h1TNizZ8+Q9uwq9zfeeOMwfsc73tEco0xI+7K9kaJineXzM43RqaC0OXZdcDcU2soYHYbpk/fdd19zHukHYzbGFGDrDG3Y6dZTdkrgPWa0PM+FOut1ZQos7cM+iGnBY10vqOdOo2V6LFNnSc+ravXHKbaUDe3IPobpt35m+g7GGvsf0tFMzTWtaSqwq5efifJxqjrT00nzMI2OsjNlhtQH6qgppqRMcE6mtzIt2ynKtFOuralGlKPpfJwXn8t+kjZs25t9dqed1YAdaBznqTdO0ac+c10tJ66r6d68H9fS5zG1nr7VHbloO6YG8l70oV5/nme9pQ/lvUzZ5hwdX0yFmwrLy8vDtU0Npx55/0qfRfr9GM2N46o2nrKbp/cLPHb77bcPY/tU0ltf+9rXNscYA8biFuXj+GBff6BrV7XPaSrTbI86ZXzsum7QF8dF6ro7pvJc6rP9Pp+HlMmq1k/S/7ksAulw1HPTqOjj7JPpL3g9+0zK0M9MP8nY43tRPtbbI7G3qXqSCjPrcuUOV8R73/ve5jPtiLZjf0jauPWecuS73l133dWcR/u78sorh7GpoLQVrx+vQSqT3zUY491RmXrDuGY5XnrppcPYlLOZzfh9ajU45phjBv/v9efey7RX+l7Kxvsj0jC956Ov4Xu+r0EfzTjr7wb8XkDQ/mhjB7uPqmr9BeO45cHrm656sBToZPwEQRAEQRAEQRAEQRAsKPLFTxAEQRAEQRAEQRAEwYIiX/wEQRAEQRAEQRAEQRAsKNa0xg9rUriOA/nD5vzPazdpPhu5pm5tSc4lOXNuuU1+JMeuvcKaET7GZ+E1zOljTRi3w2Z7OPIEzXUnt9OtK81RnwJd1w2cZLatr2p50a4bQ5Bnbd6k6+QQ5DZyXd0Wj9xd1hqwzlGXzFOmrFiPwnpFjqX55HwWtvm0rDlH1zkwl3cqsN6W26/S3q655prmGPnoXGe3w6Rcr7jiiubYxz/+8WFMPXfdBtapoQysI6wzMtY2/d577z3gfava2hvme5Pjz7Wx3fOZrSdHop37L3/5y+GZXPeI93fNBdbc4FqeffbZzXnkpZurTP939913D2O326TfZe0Z+136BOsBueG0ddcOIU/cdUuot7yGZUi+/Vg78imxvLw8zN215FgPwDVz6GN5nmtS8LOvQf9C/fV55ONTF+yvGPvsU+n3aGO+Bn2q6yxxvow3bgnPezk+zORqG10NWOPHz02bcP0NgjUFrL+sV2D/R72nH3bdBD4v45trHfGzn4V1A1lTwfsS2rp9E2sg0jfZFul/PA/H2qmwvLw8rI3nQ9u075lXK8U+hGvmvQ/tmfrs2hiU8VVXXXXA+VW1Mc7H6OsZ+7zOrH/h2iT0h9wLuJYibdg1KWY6an1fDZaWlgab9xrTF9j++az0M9Y1fvZejrrOGkuuW0qfxzbU3nvQFh3HuRehDO0LOV/Xf2NsYD0c128aq8cypR8lHnvssbr55puran+95J7V7yHEZZddNoxtR9Rf7g2rWr1hnSXXxeFnzmOs/qNrwNEncC/l/RjnS92qalvOs6W37f7OO+884PWqnnq/8l5+NeD7IluZV7V1m970pjc1x7g/5Pq41hPfCb1XmPfO4Fbs8/aUrsfH8yxD6iev730U19Z6Sz2jrrq+JvfHtkXb/jwk4ycIgiAIgiAIgiAIgmBBkS9+giAIgiAIgiAIgiAIFhRrSvViq0yn0TJd0JQcpi8xNdGtIZlG5bRIUm2YTvm1r32tOY8peUxpN9WB6VdOi2QK3VjL9rFWmUzFJmXLbVOZVuz0M7ZNnRIzWXkuTPc2rY0pseedd94w9hypB14T6gFT5seodkwhduthpvY6XZXtWqlXTu3lvU1RYRoi/85tV5nW53TFI9XOfWlpaZi7KZO0K9sYU5S5zl4Xpso61Z9tL6kzTvdmyip129QN6ppTbDl/psOSqlTVplRbF5guS9/ha1AX7BOccjsFtm7dOqSJ33///c0x0gas90yXZbq4W5VynWcp1zPQd3EdPvGJTzTnMV15Xmv3qjZV1vRG6hapE6ay0kebYkH6AY9ZX5iO7PTyg22VeTiY6b51jzrl1sFf/epXhzGfyT6E8cjrQv/Ldbfd8zP9plONeX3HzHmtasda6ToGMP2aMcA6w3hPelLVUz5nSopC3/eDDN1WlindliFTwWnDTounDL0mpG/QjzkeUR6kldkWaVekHlS1bcFpO06LJ73X9AjKg/Nga/uqNu5aR7zXmQpbtmwZaEumJdEGTDkgfY3+ytQ+Ugm87oyFtAFTMmg7lP1Yu3XHH8Za7kfuu+++5jzGAFNlaHO33XZbHQzsf2bPPOU+hzR2U6wYq7797W83x2atw6vaZzWNkfrr+EEdpi16b8NyFtxT+V5jMYdrRvsznYt+znbKfcLY/p37Yev0kdqjHnfccXXRRRdV1f7rx/2l957btm074Hl+T6B87E9IbePzUUeqqi6++OIDzsO2wvUz/YdzpA3bpkghsk+44YYbDnivsXeo66+/vjl2JHzqnj17hv2C15/x6cEHH2yOUd70tbZnvud7Teatq8su8B2dtuM9PO1jjApJPbA9z/ubqnafQJ3zOzLjkKmspvnPQzJ+giAIgiAIgiAIgiAIFhT54icIgiAIgiAIgiAIgmBBkS9+giAIgiAIgiAIgiAIFhRrWuNn06ZNA8eXbX6rWl4iawhUtZw28pHNP2dNFXL8fU1y+tjC0NcnF9At5sk/Z0v1qpbDSR6027CSj+djrI3BOgrmh/IabsNnHuIUWF5eHubqGhvkD5tLTM4mZW8OJGVoPji5xeSsu80oj421MuZ5rovBtSOPe6wuhGtaENQDc8u5NpahObpTYfPmzcOcrHvkwR9sq1/LgBxxc8JpV7Rtrx9bxPKYubz87LoJlBc5xq7fwhoSrofA9SH31jx46rL5u+SdT4nZc4y1l3R7Xfo8rittr6ptqclaAFXz28BecMEFzXmsGUE5sbZHVatX9v+8F+srWE7UJestfQn9tfV2+/btw9h1k45UC+m9e/cOz/X617++OUbeN+VR1cpurJU9fYjrPzBOcs1cW4L6xXu5Zg2v77XlNemL7VPpA22nrJ9DXXB8dlwnZnVLpqy7tWnTpiGG+Lpcf9eGYdxhPRX7D66J4/qZZ545jFlDxtdg7QH6Scdxzt/7KLYRZu02+3jqgf0KbZFzck2dsfo/rKkzJXbv3j3UjXDdKK6TfT/rcbD+o2sTcf28v6E+cz1dk4IyGdtzURccW2l/1BPvN+izXbuKNsf6I675Qp33MV9zCmzZsmWYj/WG63D55Zc3x7gmrH1nXzjm41iXkHFrx44dzXls4U69t237+gTjAf2ka6lwv33uuec2x26//fZhTBm6dgj1wHsN28JU2L1791DfxPsnru273/3u5hhthzbhPT7fH+0DKR/qhd8hWPOMtVg8X/o575Fof5SHbYN66P0T9YT7d/sfzv+SSy5pjs3W7WDrxBwMlpaWhn2x/RP1zftLPiv3XV4TxirXkuO51FnHKuo6907eQ1J/vKdmjJvnR6pan2n/z1qclOHOnTub81jnyP7CejwPyfgJgiAIgiAIgiAIgiBYUOSLnyAIgiAIgiAIgiAIggVF53SmI3qzrvthVX23ql5YVT96mtOPNNbDHKrWbh4v6/v+pKc/bRzrTIZVz6x5TCLDqnUnx/Uwh6rY4mrxTJpHbPHIIjJcHZ5J81hUOa6HOVRFhqvFM2keiyrH9TCHqshwtXgmzWOuHNf0i5/hpl13R9/3F635jdfZHNbTPA4V62XemcfqsB7mvR7msJ7mcahYL/POPFaH9TDv9TCH9TSPQ8V6mXfmsTqsh3mvhzmsp3kcKtbLvDOP1WE9zHs9zGE9zeNQsV7mnXk8iVC9giAIgiAIgiAIgiAIFhT54icIgiAIgiAIgiAIgmBBcbS++PnYUbovsR7mULV+5nGoWC/zzjxWh/Uw7/Uwh6r1M49DxXqZd+axOqyHea+HOVStn3kcKtbLvDOP1WE9zHs9zKFq/czjULFe5p15rA7rYd7rYQ5V62ceh4r1Mu/Mo45SjZ8gCIIgCIIgCIIgCILgyCNUryAIgiAIgiAIgiAIggXFmn7x03Xd27uu+0bXdQ92XfeRNbzvn3dd92jXdffh307suu7zXdd9a+X/z1+DeZzadd2Xuq77etd193dd9wdHay6rwTNZjpHhqu971GW4cs/IcXX3PepyjAxXfd+jLsOVe0aOq7vvUZdjZLjq+x51Ga7cM3Jc3X2Puhwjw1Xf96jLcOWekePq7nvU5bhuZdj3/Zr8V1Wbquqhqjq9qrZU1T1Vdc4a3fsNVbW9qu7Dv/2/VfWRlfFHqur/WYN5vLiqtq+MT6iqb1bVOUdjLpFjZPhMlWHkuBhyjAw3vgwjx8WQY2S48WUYOS6GHCPDjS/DyHEx5LheZbiWCnB5VV2Lz/+1qv7rGt7/5VKAb1TViyGcb6zlwq/c9x+r6qr1MJfIMTJ8psowclwMOUaGG1+GkeNiyDEy3PgyjBwXQ46R4caXYeS4GHJcLzJcS6rXyVX1L/j88Mq/HS28qO/7R6qqVv7/K2t5867rXl5VF1TVjqM9l0NE5LiCyHAyxBYPD5HjCiLDyRBbPDxEjiuIDCdDbPHwEDmuIDKcDLHFw0PkuIL1JMO1/OKnO8C/9Wt4/3WDruueXVX/q6r+c9/3/3G053OIiBwrMlwURI4bH5HhYiBy3PiIDBcDkePGR2S4GIgcNz7WmwzX8oufh6vqVHw+par+dQ3vb/yg67oXV1Wt/P/Rtbhp13Wb60kF+Mu+7//uaM7lMPGMl2NkODlii4eHZ7wcI8PJEVs8PDzj5RgZTo7Y4uHhGS/HyHByxBYPD894Oa5HGa7lFz87q+qMrutO67puS1W9v6quXsP7G1dX1QdWxh+oJ7l3RxRd13VV9T+q6ut93//3ozmXVeAZLcfI8Iggtnh4eEbLMTI8IogtHh6e0XKMDI8IYouHh2e0HCPDI4LY4uHhGS3HdSvDNS5s9M56sqr1Q1X1f6/hff+6qh6pqj315DeQH6qqF1TVdVX1rZX/n7gG83hdPZnmdm9V3b3y3zuPxlwix8jwmSrDyHEx5BgZbnwZRo6LIcfIcOPLMHJcDDlGhhtfhpHjYshxvcqwW5lcEARBEARBEARBEARBsGBYS6pXEARBEARBEARBEARBsIbIFz9BEARBEARBEARBEAQLinzxEwRBEARBEARBEARBsKDIFz9BEARBEARBEARBEAQLinzxEwRBEARBEARBEARBsKDIFz9BEARBEARBEARBEAQLinzxEwRBEARBEARBEARBsKDIFz9BEARBEARBEARBEAQLiv8fVYZ8YekJrlIAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 1440x144 with 10 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Get weights\n",
    "fig, ax = plt.subplots(1, 10, figsize=(20, 2))\n",
    "\n",
    "for digit in range(10):\n",
    "    ax[digit].imshow(W[:,digit].detach().view(28,28), cmap='gray')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the model learned a template for each digit.\n",
    "Remember that our model takes a dot product between the weights of each digit and input.\n",
    "Therefore, the more the input matches the template for a digit, the higher the value of the dot product for that digit will be, which makes the model more likely to predict that digit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The Full Code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The entire model, with the complete model definition, training, and evaluation (but minus the weights visualization) as independently runable code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "import torch.nn.functional as F\n",
    "from torchvision import datasets, transforms\n",
    "from tqdm.notebook import tqdm\n",
    "\n",
    "# Load the data\n",
    "mnist_train = datasets.MNIST(root=\"./datasets\", train=True, transform=transforms.ToTensor(), download=True)\n",
    "mnist_test = datasets.MNIST(root=\"./datasets\", train=False, transform=transforms.ToTensor(), download=True)\n",
    "train_loader = torch.utils.data.DataLoader(mnist_train, batch_size=100, shuffle=True)\n",
    "test_loader = torch.utils.data.DataLoader(mnist_test, batch_size=100, shuffle=False)\n",
    "\n",
    "## Training\n",
    "# Initialize parameters\n",
    "W = torch.randn(784, 10)/np.sqrt(784)\n",
    "W.requires_grad_()\n",
    "b = torch.zeros(10, requires_grad=True)\n",
    "\n",
    "# Optimizer\n",
    "optimizer = torch.optim.SGD([W,b], lr=0.1)\n",
    "\n",
    "# Iterate through train set minibatchs \n",
    "for images, labels in tqdm(train_loader):\n",
    "    # Zero out the gradients\n",
    "    optimizer.zero_grad()\n",
    "    \n",
    "    # Forward pass\n",
    "    x = images.view(-1, 28*28)\n",
    "    y = torch.matmul(x, W) + b\n",
    "    cross_entropy = F.cross_entropy(y, labels)\n",
    "    # Backward pass\n",
    "    cross_entropy.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "## Testing\n",
    "correct = 0\n",
    "total = len(mnist_test)\n",
    "\n",
    "with torch.no_grad():\n",
    "    # Iterate through test set minibatchs \n",
    "    for images, labels in tqdm(test_loader):\n",
    "        # Forward pass\n",
    "        x = images.view(-1, 28*28)\n",
    "        y = torch.matmul(x, W) + b\n",
    "        \n",
    "        predictions = torch.argmax(y, dim=1)\n",
    "        correct += torch.sum((predictions == labels).float())\n",
    "    \n",
    "print('Test accuracy: {}'.format(correct/total))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: The accuracy from the full version directly above might return a slightly different test accuracy from the step-by-step version we first went through. \n",
    "We trained our model with stochastic gradient descent (SGD), with the word \"stochastic\" highlighting that training is an inherently random process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Higher level APIs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far, we've primarily been building neural networks with fairy basic PyTorch operations.\n",
    "We did this to provide a clearer picture of how models actually work and what's going on under the hood.\n",
    "This can be important as you're learning concepts and the various frameworks, and sometimes the low-level control is necessary if you're trying to build something novel.\n",
    "\n",
    "However, most of the time, we do find ourselves repeating the same fairly standard lines of code, which can slow us down. \n",
    "Worse, it clutters up our code unnecessarily and introduces room for bugs and typos.\n",
    "And finally, as researchers or engineers, we would like to spend most of our time thinking on the highest levels of abstractions: I want to add a convolution layer here, then a fully-connected there, etc.\n",
    "Having to code all the small details are distractions that can detract from our ability to translate ideas into code.\n",
    "For this reason, PyTorch has higher level abstractions to help speed up implementation and improve model organization.\n",
    "While there are many ways to organize PyTorch code, one common paradigm is with `torch.nn.Module`.\n",
    "\n",
    "#### Object-oriented Refactorization\n",
    "\n",
    "It often makes sense for us to code our models in an [object-oriented manner](https://realpython.com/python3-object-oriented-programming/).\n",
    "To understand why, let's look back at the linear transformation $y = xW + b$ that we used for logistic regression. \n",
    "We can see that while the operation consisted of a matrix multiplication and addition, also associated with this operation was the instantiation of two parameters `W` and `b`, and these two parameters conceptually *belong* to the transform.\n",
    "As such, it would make sense to bundle up the instantiation of the two parameters with the actual transformation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note: illustrative example only; see below for torch.nn usage\n",
    "class xW_plus_b:\n",
    "    def __init__(self, dim_in, dim_out):\n",
    "        self.W = torch.randn(dim_in, dim_out)/np.sqrt(dim_in)\n",
    "        self.W.requires_grad_()\n",
    "        self.b = torch.zeros(dim_out, requires_grad=True)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        return torch.matmul(x, self.W) + self.b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use what we just wrote, we can create an `xW_plus_b` instance using its `__init__()` method (the constructor).\n",
    "In this case, we're going to set the dimensions to be 784 and 10, as we did in our logisitic regression example above.\n",
    "This creates an `xW_plus_b` instance with two parameters `W` and `b`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "W: torch.Size([784, 10])\n",
      "b: torch.Size([10])\n"
     ]
    }
   ],
   "source": [
    "# Note: illustrative example only; see below for torch.nn usage\n",
    "lin_custom = xW_plus_b(784, 10)\n",
    "print(\"W: {}\".format(lin_custom.W.shape))\n",
    "print(\"b: {}\".format(lin_custom.b.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After instantiating the instance, we can perform the actual linear transform of our custom `xW_plus_b` class by calling the instance's `forward()` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 10])\n"
     ]
    }
   ],
   "source": [
    "# Note: illustrative example only; see below for torch.nn usage\n",
    "x_rand = torch.randn(1,784)\n",
    "y = lin_custom.forward(x_rand)\n",
    "print(y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using  `torch.nn`\n",
    "\n",
    "While we can certainly implement our own classes for the operations we'd like to use, we don't have to, as PyTorch already has them in the `torch.nn` sublibrary. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, the linear transform example we just went through is called `torch.nn.Linear`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Linear parameters: [torch.Size([10, 784]), torch.Size([10])]\n",
      "torch.Size([1, 10])\n"
     ]
    }
   ],
   "source": [
    "lin = nn.Linear(784, 10)\n",
    "print(\"Linear parameters: {}\".format([p.shape for p in lin.parameters()]))\n",
    "\n",
    "y = lin(x_rand)\n",
    "print(y.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The implementation for `nn.Linear` has a few more things under the hood (notice for example that the `forward()` function is aliased with calling the instance itself), but in spirit, it operates in much the same way as our custom `xW_plus_b` class.\n",
    "In the first line, we instantiate a `Linear` object, which automatically creates weight and bias variables of the specified dimensions.\n",
    "The fourth line then calls the `forward()` function (aliased with the object call), which performs the linear transformation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using `torch.nn.Module`\n",
    "\n",
    "The `torch.nn.Linear` class we just saw is a subclass of `torch.nn.Module`.\n",
    "However, `Module`s do not have to just describe a single operation; they can also define a chain of operations, each of which may also be `Module`s.\n",
    "As such, we can place our entire neural network within a `Module`.\n",
    "In this case, the module can track all of its associated parameters, some of which may also be associated with a submodule (e.g. `nn.Linear`), while also defining the `forward()` function, in one place."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MNIST_Logistic_Regression(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.lin = nn.Linear(784, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.lin(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this particular example, we didn't need to chain any operations, but we'll see this come in handy as we move on to more complex models.\n",
    "Additionally, the `nn.Module` that we subclassed has a few other nice features.\n",
    "For example:\n",
    "- The `forward()` function of a `nn.Module` will call the `forward()` function of any child `nn.Module`s.\n",
    "- `print()` will print out a formatted summary of our model, recursively summarizing any child `nn.Module`s as well.\n",
    "- The `parameters()` function will return a generator that returns all parameters of a `nn.Module` (including those of any children)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The model: \n",
      "MNIST_Logistic_Regression(\n",
      "  (lin): Linear(in_features=784, out_features=10, bias=True)\n",
      ")\n",
      "\n",
      "Parameters: \n",
      "[Parameter containing:\n",
      "tensor([[ 0.0303,  0.0342, -0.0052,  ...,  0.0004, -0.0050,  0.0322],\n",
      "        [-0.0186,  0.0259,  0.0133,  ..., -0.0243,  0.0293, -0.0271],\n",
      "        [-0.0170, -0.0293, -0.0183,  ..., -0.0044, -0.0041, -0.0052],\n",
      "        ...,\n",
      "        [ 0.0109, -0.0350,  0.0196,  ..., -0.0039, -0.0178,  0.0181],\n",
      "        [-0.0221, -0.0280, -0.0260,  ..., -0.0229,  0.0061, -0.0091],\n",
      "        [-0.0318, -0.0175, -0.0053,  ..., -0.0220, -0.0048,  0.0251]],\n",
      "       requires_grad=True), Parameter containing:\n",
      "tensor([-0.0012,  0.0022, -0.0059, -0.0324, -0.0029,  0.0221,  0.0199, -0.0169,\n",
      "        -0.0180, -0.0212], requires_grad=True)]\n",
      "\n",
      "Output shape: \n",
      "torch.Size([1, 10])\n"
     ]
    }
   ],
   "source": [
    "model = MNIST_Logistic_Regression()\n",
    "y = model(x_rand)\n",
    "print(\"The model: \\n{}\".format(model))\n",
    "print(\"\\nParameters: \\n{}\".format(list(model.parameters())))\n",
    "print(\"\\nOutput shape: \\n{}\".format(y.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Full code with `nn.Module`\n",
    "\n",
    "Refactoring our previous complete logistic regression code to use a `nn.Module`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "213f3a7ee9964da5abe899184cf0a271",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=600.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a5e46aea78c3447eae9901387280bc0e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Test accuracy: 0.9031999707221985\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torchvision import datasets, transforms\n",
    "from tqdm.notebook import tqdm\n",
    "\n",
    "class MNIST_Logistic_Regression(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.lin = nn.Linear(784, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.lin(x)\n",
    "\n",
    "# Load the data\n",
    "mnist_train = datasets.MNIST(root=\"./datasets\", train=True, transform=transforms.ToTensor(), download=True)\n",
    "mnist_test = datasets.MNIST(root=\"./datasets\", train=False, transform=transforms.ToTensor(), download=True)\n",
    "train_loader = torch.utils.data.DataLoader(mnist_train, batch_size=100, shuffle=True)\n",
    "test_loader = torch.utils.data.DataLoader(mnist_test, batch_size=100, shuffle=False)\n",
    "\n",
    "## Training\n",
    "# Instantiate model\n",
    "model = MNIST_Logistic_Regression()\n",
    "\n",
    "# Loss and Optimizer\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=0.1)\n",
    "\n",
    "# Iterate through train set minibatchs \n",
    "for images, labels in tqdm(train_loader):\n",
    "    # Zero out the gradients\n",
    "    optimizer.zero_grad()\n",
    "    \n",
    "    # Forward pass\n",
    "    x = images.view(-1, 28*28)\n",
    "    y = model(x)\n",
    "    loss = criterion(y, labels)\n",
    "    # Backward pass\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "## Testing\n",
    "correct = 0\n",
    "total = len(mnist_test)\n",
    "\n",
    "with torch.no_grad():\n",
    "    # Iterate through test set minibatchs \n",
    "    for images, labels in tqdm(test_loader):\n",
    "        # Forward pass\n",
    "        x = images.view(-1, 28*28)\n",
    "        y = model(x)\n",
    "        \n",
    "        predictions = torch.argmax(y, dim=1)\n",
    "        correct += torch.sum((predictions == labels).float())\n",
    "    \n",
    "print('Test accuracy: {}'.format(correct/total))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the benefits of organizing a model as a `nn.Module` may not be as obvious for a simple logistic regression model, such a programming style allows for much quicker and cleaner implementations for more complex models, as we'll see in later notebooks."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.12 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "23574fd47338620f5fa9034c1801f73296c9ae358dd5ea70f96243aff333c802"
   }
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {
     "00ab3361f9b948b48b0976623b0a83a7": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "02ba6fc556654c0bad4a145c4b17418c": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "03770380477d40069927db8dba2ecc34": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "04a01f520eaf4d1a8ebc8689fc58c824": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_26d5c364d9f04653ac0ae8767c252a89",
       "style": "IPY_MODEL_5081bd58b2c548079ae99dce7948daa1",
       "value": " 600/600 [00:19&lt;00:00, 30.41it/s]"
      }
     },
     "05372d8d36294452a3a083d0e7844c96": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_cbc29e26fbbd461ab309214dac56a9ce",
       "style": "IPY_MODEL_232cd012d52d437daabfba407e796fbc",
       "value": " 100/100 [00:03&lt;00:00, 29.56it/s]"
      }
     },
     "0645f6470aad4651bab08c97952a0a24": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_55001bfcd7dd4a0db1f8170c55fc99a9",
        "IPY_MODEL_04a01f520eaf4d1a8ebc8689fc58c824"
       ],
       "layout": "IPY_MODEL_f045f24016f5412696ccd4d42d3ac4a6"
      }
     },
     "070afb362477408c98244dcab5f6f78d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_5ebbf75c554e4ada93d74270c4550786",
       "style": "IPY_MODEL_3258e34f678f45839bdea8d59f2aa1c3",
       "value": " 100/100 [00:02&lt;00:00, 46.13it/s]"
      }
     },
     "08127475b3c0456db3db286c913b7271": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "0932d5c51b9b43eb96c0de08e3f7912d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_70837d601e064314987516a48f246429",
       "max": 600,
       "style": "IPY_MODEL_d47fa458994a4c2d9922b6daf4779539",
       "value": 600
      }
     },
     "0b73de309f0e4622960445b5e67b0f23": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "0b90a0686fef47dc8985949c412ebf43": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_53ae2d18383a43ebb7a2327b62efdbfe",
       "style": "IPY_MODEL_7e87dbe6f0484a6d891c4359f362f2ba",
       "value": 100
      }
     },
     "0c18433b48ac4be0b0e037d5be0c63f8": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "0dcf2e7c51d04f5da15ede8dd49f0eac": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "1110c89bcb7948b8a4081f188c8f053f": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "13366e3937e24dc4bf9254771e0ac514": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "139a26066e2a453a92a5741ac7440a3f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "180345d08d3e43fa88bf97bc5cb3f4b1": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "19ad38959222414cab16faf7bd42713f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_ab3028b53595467b92c4906fb61ba183",
       "max": 600,
       "style": "IPY_MODEL_22d2a9019d9c4417883d571b24405d10",
       "value": 600
      }
     },
     "1b2e84ba16de427fa17a661d4033d276": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_56f455eeb9624937980c91d59d0a72ed",
       "style": "IPY_MODEL_e988071311e640fd84d509864d1c64b6",
       "value": 100
      }
     },
     "1cb1d5db956f4c1599f362159649c5cb": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "1e340c6bb86f4a068e41cbb423949885": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_19ad38959222414cab16faf7bd42713f",
        "IPY_MODEL_d1e25145f12b42a3be57a6aace611870"
       ],
       "layout": "IPY_MODEL_6029a556c8d543f9929a6e553bf4a9a5"
      }
     },
     "1ebda3e2449a4577be0b36950eff50ae": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_cd4ee4fec9a845c3a17cbc5516345a56",
        "IPY_MODEL_a52554c238f84cefa9e52f4ea05c75e7"
       ],
       "layout": "IPY_MODEL_e3e090911b134c2f9f59b1f8cdddcf4b"
      }
     },
     "21606ea13c9c4afa8ef95cd727230bdc": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "22d2a9019d9c4417883d571b24405d10": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "232cd012d52d437daabfba407e796fbc": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "25f6dea15b0448fd941133106d8d1d37": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "26d5c364d9f04653ac0ae8767c252a89": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "28859aee2b35402d9a371a808b0f77af": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "30309ea05c174448943ecb5ba92ae0e5": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "312e9fe46c29463999edfd8af1ade5b9": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_6dadd75dd8224017a21fea4302e957aa",
       "style": "IPY_MODEL_a4d777e8f0994329912efaab8f50d747",
       "value": 100
      }
     },
     "3258e34f678f45839bdea8d59f2aa1c3": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "32e9a047ea3a41439e5d83ecc5310ddb": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_a52ede1b19f84fa1887ca5c33686b97e",
       "style": "IPY_MODEL_acd64aff4bd24ec49aef585414303fab",
       "value": " 100/100 [00:02&lt;00:00, 35.70it/s]"
      }
     },
     "33724df2f1a3433bbb26aa18bc88b126": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_e1d28e912a1d40d8890d9946c1f86981",
       "style": "IPY_MODEL_cabcc13852074e07a8d322cd99f24e5e",
       "value": 100
      }
     },
     "3877f7bc979a4999a78e0ca12a2efd2f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_fb696dfe7bae4a00b375d9c7c6053293",
       "style": "IPY_MODEL_42473977bd5248308eca1210a128bdae",
       "value": " 100/100 [00:01&lt;00:00, 52.80it/s]"
      }
     },
     "3b47462712754c9484f4bb97183b127f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "3bf1ea9535ad49c189eafc90c872c079": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "4112ba3474384d839e66ffaaab8f2355": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_c0971f8cf1284aaca0068c0b57d6cab0",
       "style": "IPY_MODEL_28859aee2b35402d9a371a808b0f77af",
       "value": " 600/600 [00:16&lt;00:00, 35.95it/s]"
      }
     },
     "42473977bd5248308eca1210a128bdae": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "42c21b2eb4834c7b84838c3831e2f34b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "45759a4191104054b18f78f161e4114b": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "46fbadc7425740fd9a72f8034a4e6753": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "479ef10b25b44072bf9b62bb3244e1f8": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "48e29dbfe2f64b07adb36bf03501699d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_ffeae8ce8dde457eb2ba3ec954e2216e",
        "IPY_MODEL_d393ae32b0a94f1d8591995c66f71243"
       ],
       "layout": "IPY_MODEL_fb6e26311b32443391e4a1ce19de0678"
      }
     },
     "4b92c8e1cdcb4ea2a3622474cab1a875": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "4bec82b3e28441f78808b30d623eded8": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_f65f7b566392443eab4238881599bb42",
       "max": 600,
       "style": "IPY_MODEL_5d83d8f891bb47489b5eebd8331f97d1",
       "value": 600
      }
     },
     "4c2af1030c8246eda9ccc184abd348d1": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "4c83d21fbe2546b1bc75b37618873f65": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_0932d5c51b9b43eb96c0de08e3f7912d",
        "IPY_MODEL_bfb0ba4f81f14080b9924db914dcf40c"
       ],
       "layout": "IPY_MODEL_630457eddaf54d7cafe66f26ab1d46a7"
      }
     },
     "4e39f76cfaa941b0a5ad44075c7ef198": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_7cbcf84618fd43efa672966c819226c5",
        "IPY_MODEL_d291924bdbb349e69a084d7d91c76ebe"
       ],
       "layout": "IPY_MODEL_f8d0a0172da942d0bf4db39113cfef42"
      }
     },
     "4ed8e96625e24a919dcdc548e12086e9": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "4f35ff781dcf40c6b03ce427f220dfe4": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "5081bd58b2c548079ae99dce7948daa1": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "53606d6c4d93409c82e68a7fdca9610c": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "53ae2d18383a43ebb7a2327b62efdbfe": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "546189e2848f4b218d68788c06d6ea85": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "55001bfcd7dd4a0db1f8170c55fc99a9": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_61b8676138324dd3831ebaed08572e4f",
       "max": 600,
       "style": "IPY_MODEL_b1fa6a5c2a2443f7b0561ad65e10468e",
       "value": 600
      }
     },
     "56f455eeb9624937980c91d59d0a72ed": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "5d83d8f891bb47489b5eebd8331f97d1": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "5dfeb1724b7340a8806aa3e944db9956": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_d89d4ffb0770466292129554c7369cae",
        "IPY_MODEL_dd2b6ac6de5a4907bde927b72e888208"
       ],
       "layout": "IPY_MODEL_e5bb8e91222b466791970de816a0187b"
      }
     },
     "5e838fe82fa549aaab1fde139b26121a": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "5ebbf75c554e4ada93d74270c4550786": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "5ebf9c13b75741a3950b6461bee442af": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "5f3b481c4c934ee3ab2a2086412d6cc8": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "6029a556c8d543f9929a6e553bf4a9a5": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "61b8676138324dd3831ebaed08572e4f": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "630457eddaf54d7cafe66f26ab1d46a7": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "64630cc14f9b474287401e4bac647c22": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "64777a4ec2d0417a949bba49edaedd94": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "6779d64d71cc4f80a18e3d7edb9fe77b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_479ef10b25b44072bf9b62bb3244e1f8",
       "max": 600,
       "style": "IPY_MODEL_7903dd30475249cdb0b4e61de937e7c1",
       "value": 600
      }
     },
     "6943475d270b4209873dc4835b0300bd": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_6fd0da636b74404abc93e483a3bebfb3",
        "IPY_MODEL_8032b7629d5943febafa8d25e59c175c"
       ],
       "layout": "IPY_MODEL_e28d4876f752451a8d81c93b107bbcd3"
      }
     },
     "69e506b09b4c4f0db02641e724e78020": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "6ac175ccc9174632802d40c527afdea3": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_f0874107f14b455ca59d0efd8bea5db1",
       "style": "IPY_MODEL_9613febf823e490487115eaacc3791a6",
       "value": " 600/600 [00:16&lt;00:00, 36.04it/s]"
      }
     },
     "6ad9032d612f454b85321ed603ca3815": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_b29d4784e1194b61851631bbb06c4a32",
       "max": 600,
       "style": "IPY_MODEL_4b92c8e1cdcb4ea2a3622474cab1a875",
       "value": 600
      }
     },
     "6d69e250974440d2aaa5d90d5371c959": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "6dadd75dd8224017a21fea4302e957aa": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "6dd5a25c673f4a4faf8c7d0b2e398abf": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_ee64dbf4666c4ca790c82b352e2bd3dc",
       "style": "IPY_MODEL_a8abaf603e14436c9556618d0a599e82",
       "value": 100
      }
     },
     "6fd0da636b74404abc93e483a3bebfb3": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_eef6e87e976447aea9d6ec869e115173",
       "style": "IPY_MODEL_53606d6c4d93409c82e68a7fdca9610c",
       "value": 100
      }
     },
     "703e062492724a06bdf1510053ccbc6a": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_ef59d86997614b18a8c1311689822f03",
       "style": "IPY_MODEL_02ba6fc556654c0bad4a145c4b17418c",
       "value": " 600/600 [00:24&lt;00:00, 24.88it/s]"
      }
     },
     "70837d601e064314987516a48f246429": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "709f252ba28243698ffde792b2b7297c": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_69e506b09b4c4f0db02641e724e78020",
       "style": "IPY_MODEL_ad8cee115d6749df944c36a3564fef46",
       "value": " 100/100 [00:03&lt;00:00, 31.10it/s]"
      }
     },
     "71f74cf044db43979c7a297ff8be572e": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "74c67621af0148fe92defe8e4513c56c": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "752ef1aef0804eabbe76ba2a4f5a56c8": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "75657820ee9648bca89a17c4def23516": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "759a293be651417982a43ab19f7ef9be": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "75a748fff88147efbf2c40e75150f667": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_de4d634ca87f438e9c4aaf69072cb2e1",
        "IPY_MODEL_6ac175ccc9174632802d40c527afdea3"
       ],
       "layout": "IPY_MODEL_c93a4f2634ab4334a9bc86f864ec9d03"
      }
     },
     "77411f173eaa4ba6ac63ed381d2374e4": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "779e7d516c99412f84a0be9af866c266": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_781e1aed06e348ac827b4339bc448f75",
        "IPY_MODEL_32e9a047ea3a41439e5d83ecc5310ddb"
       ],
       "layout": "IPY_MODEL_21606ea13c9c4afa8ef95cd727230bdc"
      }
     },
     "7813e38f58f242f5868953bd4c8003fc": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "781e1aed06e348ac827b4339bc448f75": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_5f3b481c4c934ee3ab2a2086412d6cc8",
       "style": "IPY_MODEL_da6915df7a4e4525a369586d4e45111a",
       "value": 100
      }
     },
     "7838ea755ce5480eab9de43af72d6a69": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "7903dd30475249cdb0b4e61de937e7c1": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "794168f4d4594941aa2441f5a662d1df": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "7953b46a197646e085cd429c4777ec30": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "79ef0f8bd70f4164a11f0afe3ec669c2": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_794168f4d4594941aa2441f5a662d1df",
       "max": 600,
       "style": "IPY_MODEL_3b47462712754c9484f4bb97183b127f",
       "value": 600
      }
     },
     "7cbcf84618fd43efa672966c819226c5": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_74c67621af0148fe92defe8e4513c56c",
       "style": "IPY_MODEL_139a26066e2a453a92a5741ac7440a3f",
       "value": 100
      }
     },
     "7e2a7cb7b012443d80dd1e19e3f9934e": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "7e87dbe6f0484a6d891c4359f362f2ba": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "8032b7629d5943febafa8d25e59c175c": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_dfbdeaf4bae14bde9dca3e0d579bafad",
       "style": "IPY_MODEL_ed8e61c1bb23474d90831e61954f4985",
       "value": " 100/100 [00:03&lt;00:00, 29.53it/s]"
      }
     },
     "809bbad7968743fab4783f7d0d36bdf0": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_8886b0dda5ea42adb4a2800057aa3b4b",
       "max": 600,
       "style": "IPY_MODEL_7838ea755ce5480eab9de43af72d6a69",
       "value": 600
      }
     },
     "82a4101b67bf48eda5c119e282f9e167": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_1b2e84ba16de427fa17a661d4033d276",
        "IPY_MODEL_05372d8d36294452a3a083d0e7844c96"
       ],
       "layout": "IPY_MODEL_77411f173eaa4ba6ac63ed381d2374e4"
      }
     },
     "83e68222f60c48dca841c8836e582553": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "85a0c592d9b54b44a240e2dff7947ebd": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_f3149d5c75fa45999c4797beb1efb134",
        "IPY_MODEL_efd0d8f6ffee421b9c372c76ed9ef1a8"
       ],
       "layout": "IPY_MODEL_0c18433b48ac4be0b0e037d5be0c63f8"
      }
     },
     "86aabc046a7d4feeaec002c17836cf74": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_6779d64d71cc4f80a18e3d7edb9fe77b",
        "IPY_MODEL_d9f45d42e93e4a65b23d8fe9eeff9680"
       ],
       "layout": "IPY_MODEL_546189e2848f4b218d68788c06d6ea85"
      }
     },
     "87443c75233049f2b0d63527c639d788": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "8886b0dda5ea42adb4a2800057aa3b4b": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "8a5088b74cf642f687cf6d1d67cb229a": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "8c5e48995dd2474cb1ac1ca557aee46c": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_30309ea05c174448943ecb5ba92ae0e5",
       "style": "IPY_MODEL_42c21b2eb4834c7b84838c3831e2f34b",
       "value": " 600/600 [00:17&lt;00:00, 34.84it/s]"
      }
     },
     "8c69ddf36e724015b78f9cc894a6f21f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "8dd74db4495047afa57108a8322cc955": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_46fbadc7425740fd9a72f8034a4e6753",
       "style": "IPY_MODEL_0dcf2e7c51d04f5da15ede8dd49f0eac",
       "value": " 100/100 [00:02&lt;00:00, 40.64it/s]"
      }
     },
     "8e51c26c3ff44704a9d7a8a1569d80e2": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "9613febf823e490487115eaacc3791a6": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "96ad99d2f3484aa6a7d5e826b948ad7f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_4c2af1030c8246eda9ccc184abd348d1",
       "style": "IPY_MODEL_4f35ff781dcf40c6b03ce427f220dfe4",
       "value": " 100/100 [00:02&lt;00:00, 39.49it/s]"
      }
     },
     "9b3be61161934791a0843b79608a7bac": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "9e6e9e41a88c48be86597fd4c6d32411": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_0b90a0686fef47dc8985949c412ebf43",
        "IPY_MODEL_96ad99d2f3484aa6a7d5e826b948ad7f"
       ],
       "layout": "IPY_MODEL_87443c75233049f2b0d63527c639d788"
      }
     },
     "a040d319ed724ba887065525487ff446": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_809bbad7968743fab4783f7d0d36bdf0",
        "IPY_MODEL_703e062492724a06bdf1510053ccbc6a"
       ],
       "layout": "IPY_MODEL_e551461ea6654cfe8133172eff4de7dd"
      }
     },
     "a1599327408f4216a3c7726870784b8a": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_8a5088b74cf642f687cf6d1d67cb229a",
       "style": "IPY_MODEL_180345d08d3e43fa88bf97bc5cb3f4b1",
       "value": " 600/600 [00:20&lt;00:00, 29.41it/s]"
      }
     },
     "a4d777e8f0994329912efaab8f50d747": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "a52554c238f84cefa9e52f4ea05c75e7": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_c9b74f1c8796436dbe487ebd9e428079",
       "style": "IPY_MODEL_c8e1306d346b4496bae3e5d745db9b7b",
       "value": " 600/600 [00:16&lt;00:00, 35.42it/s]"
      }
     },
     "a52ede1b19f84fa1887ca5c33686b97e": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "a6f0c7ea1aff42cdb9eeab5ac77328be": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "a8a79b095f874fe0bb8764e960bdbe57": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_5e838fe82fa549aaab1fde139b26121a",
       "style": "IPY_MODEL_64630cc14f9b474287401e4bac647c22",
       "value": " 100/100 [00:02&lt;00:00, 48.19it/s]"
      }
     },
     "a8abaf603e14436c9556618d0a599e82": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "a9cb23521c114698aec111175b0fdded": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_be90841470e448f398c1d84b58b1e5f0",
        "IPY_MODEL_a8a79b095f874fe0bb8764e960bdbe57"
       ],
       "layout": "IPY_MODEL_ede80f846a4a4f21b637697aa2ece42a"
      }
     },
     "a9fc6a779ac44d4293dbc9bb1471128e": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "ab3028b53595467b92c4906fb61ba183": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "acd64aff4bd24ec49aef585414303fab": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "ad5dee1503b446ddb28cccbf183fc363": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_4bec82b3e28441f78808b30d623eded8",
        "IPY_MODEL_8c5e48995dd2474cb1ac1ca557aee46c"
       ],
       "layout": "IPY_MODEL_75657820ee9648bca89a17c4def23516"
      }
     },
     "ad8cee115d6749df944c36a3564fef46": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "b0ae3ea99c9849768dbdea29a2c8e13b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "b1fa6a5c2a2443f7b0561ad65e10468e": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "b29d4784e1194b61851631bbb06c4a32": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "b3c0849c8fbb49b989ca8ac1e00f2ca7": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_312e9fe46c29463999edfd8af1ade5b9",
        "IPY_MODEL_070afb362477408c98244dcab5f6f78d"
       ],
       "layout": "IPY_MODEL_752ef1aef0804eabbe76ba2a4f5a56c8"
      }
     },
     "b914085c48e34577b65a00e1b65949a8": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "bc8f5b6c8f534b79940e05b0bf9a7ba4": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "be90841470e448f398c1d84b58b1e5f0": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_b914085c48e34577b65a00e1b65949a8",
       "style": "IPY_MODEL_759a293be651417982a43ab19f7ef9be",
       "value": 100
      }
     },
     "bfb0ba4f81f14080b9924db914dcf40c": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_83e68222f60c48dca841c8836e582553",
       "style": "IPY_MODEL_7813e38f58f242f5868953bd4c8003fc",
       "value": " 600/600 [00:20&lt;00:00, 29.87it/s]"
      }
     },
     "c0971f8cf1284aaca0068c0b57d6cab0": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "c2890ce3286f4f73af27b73b3bfbeb66": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_79ef0f8bd70f4164a11f0afe3ec669c2",
        "IPY_MODEL_a1599327408f4216a3c7726870784b8a"
       ],
       "layout": "IPY_MODEL_caead25870804f11aa1e279e7d4b9603"
      }
     },
     "c8253a11811f4b0cbc812051b6de6f6e": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "c8561fa56818435f94707ee41181197c": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "c8e1306d346b4496bae3e5d745db9b7b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "c93a4f2634ab4334a9bc86f864ec9d03": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "c9b74f1c8796436dbe487ebd9e428079": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "cabcc13852074e07a8d322cd99f24e5e": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "caead25870804f11aa1e279e7d4b9603": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "cbc29e26fbbd461ab309214dac56a9ce": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "cd44a9cf38fa4433a0662235ebd1a5f0": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "cd4ee4fec9a845c3a17cbc5516345a56": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_f0ad882691bd4012aa89810f86c52621",
       "max": 600,
       "style": "IPY_MODEL_1cb1d5db956f4c1599f362159649c5cb",
       "value": 600
      }
     },
     "d045ee6e5ade4f938d6b76862ca5dc33": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "d0ba99eac67041c2a449a80b71e1a0a5": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "d1901b27e0554752ae4addf37d4acfd4": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "d1e25145f12b42a3be57a6aace611870": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_4ed8e96625e24a919dcdc548e12086e9",
       "style": "IPY_MODEL_0b73de309f0e4622960445b5e67b0f23",
       "value": " 600/600 [00:19&lt;00:00, 31.36it/s]"
      }
     },
     "d291924bdbb349e69a084d7d91c76ebe": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_00ab3361f9b948b48b0976623b0a83a7",
       "style": "IPY_MODEL_7953b46a197646e085cd429c4777ec30",
       "value": " 100/100 [00:02&lt;00:00, 39.69it/s]"
      }
     },
     "d3440bde35cd4306b00d651fb9606e47": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "d393ae32b0a94f1d8591995c66f71243": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_a9fc6a779ac44d4293dbc9bb1471128e",
       "style": "IPY_MODEL_b0ae3ea99c9849768dbdea29a2c8e13b",
       "value": " 600/600 [00:19&lt;00:00, 30.64it/s]"
      }
     },
     "d47fa458994a4c2d9922b6daf4779539": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "d89d4ffb0770466292129554c7369cae": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_7e2a7cb7b012443d80dd1e19e3f9934e",
       "style": "IPY_MODEL_25f6dea15b0448fd941133106d8d1d37",
       "value": 100
      }
     },
     "d9f45d42e93e4a65b23d8fe9eeff9680": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_5ebf9c13b75741a3950b6461bee442af",
       "style": "IPY_MODEL_d3440bde35cd4306b00d651fb9606e47",
       "value": " 600/600 [00:20&lt;00:00, 28.95it/s]"
      }
     },
     "da6915df7a4e4525a369586d4e45111a": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "db35d1a9069a4d86b37b3f3d85ff05f5": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_cd44a9cf38fa4433a0662235ebd1a5f0",
       "style": "IPY_MODEL_a6f0c7ea1aff42cdb9eeab5ac77328be",
       "value": " 600/600 [00:20&lt;00:00, 29.41it/s]"
      }
     },
     "dc1ec29fb4054067a283cc37d19debeb": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_6ad9032d612f454b85321ed603ca3815",
        "IPY_MODEL_4112ba3474384d839e66ffaaab8f2355"
       ],
       "layout": "IPY_MODEL_df2f1c9f45dd4974bc218e0f571b020c"
      }
     },
     "dd2b6ac6de5a4907bde927b72e888208": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_d045ee6e5ade4f938d6b76862ca5dc33",
       "style": "IPY_MODEL_3bf1ea9535ad49c189eafc90c872c079",
       "value": " 100/100 [00:02&lt;00:00, 34.76it/s]"
      }
     },
     "de4d634ca87f438e9c4aaf69072cb2e1": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_6d69e250974440d2aaa5d90d5371c959",
       "max": 600,
       "style": "IPY_MODEL_c8561fa56818435f94707ee41181197c",
       "value": 600
      }
     },
     "df2f1c9f45dd4974bc218e0f571b020c": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "dfbdeaf4bae14bde9dca3e0d579bafad": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "e01fab92b3e94bc6992d25581ab86245": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_f1dcaff9e6d34c208aa3601a740471e1",
        "IPY_MODEL_8dd74db4495047afa57108a8322cc955"
       ],
       "layout": "IPY_MODEL_d1901b27e0554752ae4addf37d4acfd4"
      }
     },
     "e085bf0edc8f405ea1e200394a2d699d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_6dd5a25c673f4a4faf8c7d0b2e398abf",
        "IPY_MODEL_709f252ba28243698ffde792b2b7297c"
       ],
       "layout": "IPY_MODEL_13366e3937e24dc4bf9254771e0ac514"
      }
     },
     "e1d28e912a1d40d8890d9946c1f86981": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "e28d4876f752451a8d81c93b107bbcd3": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "e2ee4488d9fb48949fa658478b188d8a": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_d0ba99eac67041c2a449a80b71e1a0a5",
       "max": 600,
       "style": "IPY_MODEL_9b3be61161934791a0843b79608a7bac",
       "value": 600
      }
     },
     "e3e090911b134c2f9f59b1f8cdddcf4b": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "e551461ea6654cfe8133172eff4de7dd": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "e5bb8e91222b466791970de816a0187b": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "e988071311e640fd84d509864d1c64b6": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": "initial"
      }
     },
     "ed8e61c1bb23474d90831e61954f4985": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "ede80f846a4a4f21b637697aa2ece42a": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "ee0d392ddfec4748b5b2926f1ffcc739": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_33724df2f1a3433bbb26aa18bc88b126",
        "IPY_MODEL_3877f7bc979a4999a78e0ca12a2efd2f"
       ],
       "layout": "IPY_MODEL_71f74cf044db43979c7a297ff8be572e"
      }
     },
     "ee64dbf4666c4ca790c82b352e2bd3dc": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "eef6e87e976447aea9d6ec869e115173": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "ef59d86997614b18a8c1311689822f03": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "efd0d8f6ffee421b9c372c76ed9ef1a8": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_1110c89bcb7948b8a4081f188c8f053f",
       "style": "IPY_MODEL_03770380477d40069927db8dba2ecc34",
       "value": " 100/100 [00:03&lt;00:00, 29.83it/s]"
      }
     },
     "f045f24016f5412696ccd4d42d3ac4a6": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "f0874107f14b455ca59d0efd8bea5db1": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "f0ad882691bd4012aa89810f86c52621": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "f1dcaff9e6d34c208aa3601a740471e1": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_8e51c26c3ff44704a9d7a8a1569d80e2",
       "style": "IPY_MODEL_08127475b3c0456db3db286c913b7271",
       "value": 100
      }
     },
     "f3149d5c75fa45999c4797beb1efb134": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_64777a4ec2d0417a949bba49edaedd94",
       "style": "IPY_MODEL_8c69ddf36e724015b78f9cc894a6f21f",
       "value": 100
      }
     },
     "f65f7b566392443eab4238881599bb42": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "f8d0a0172da942d0bf4db39113cfef42": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "fb696dfe7bae4a00b375d9c7c6053293": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "fb6e26311b32443391e4a1ce19de0678": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "fd0373f64152443cb49b3cfe0bdaea65": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_e2ee4488d9fb48949fa658478b188d8a",
        "IPY_MODEL_db35d1a9069a4d86b37b3f3d85ff05f5"
       ],
       "layout": "IPY_MODEL_45759a4191104054b18f78f161e4114b"
      }
     },
     "ffeae8ce8dde457eb2ba3ec954e2216e": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "IntProgressModel",
      "state": {
       "bar_style": "success",
       "description": "100%",
       "layout": "IPY_MODEL_c8253a11811f4b0cbc812051b6de6f6e",
       "max": 600,
       "style": "IPY_MODEL_bc8f5b6c8f534b79940e05b0bf9a7ba4",
       "value": 600
      }
     }
    },
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}