Exploratory_Data_Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5e25ea09-74d6-47e6-bad6-e922f52c3532",
   "metadata": {},
   "source": [
    "# Personal Information\n",
    "Name: Yuan Su\n",
    "\n",
    "StudentID: 14534053\n",
    "\n",
    "Email: yuan.su2@student.uva.nl\n",
    "\n",
    "Submitted on: **19.03.2023**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3cf6243-adfe-4eb8-bba3-bb2835079abd",
   "metadata": {},
   "source": [
    "# Data Context\n",
    "**To serve the purpose of the testing SET-alternative algorithm. The data is important to be simple and refined. Therefore, clean and balanced datasets are the first choice. Conveniently, the datasets that come with machine learning APi in python have provided some exceptional ones. For the purpose to test universality of this algorithm, the data selections are mainly chosen to train artificial neural networks(ANN) and convolutional neural networks(CNN). Thus, some regression data and picture data are chosen.**\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a833d964-56e1-49c7-8172-7435357624aa",
   "metadata": {},
   "source": [
    "# Data Description\n",
    "\n",
    "**For regression tasks, in the API ‘sklearn’, there are some good datasets for regression purposes. ‘California housing dataset’, ‘diabetes dataset’ and ‘wine dataset’ are chosen as those data have various dimensionalities indicating different complexity. Additionally, the total number of instances also vary, which is suitable to test the running time difference between ANNs and sparse neural networks(SNNs) in different algorithms. More importantly, those dataset do not contain missing values and the correlations amongst features also vary, indicating a good universality if training on different architectures of neural networks. For CNNs classification tasks, in ‘torch’ API, ‘MNIST’ dataset is chosen as this dataset has enough training instance to construct a CNN but the complexity of this CNN is capable of being run on a personal device without using transfer learning. Also, ‘FMNIST’ dataset is chosen as this dataset is much more complex than ‘MNIST’ in terms of features difference and label difference. ** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "534317db-d881-4e33-a358-754e2881e8bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Imports\n",
    "import os\n",
    "import numpy as np \n",
    "import pandas as pd\n",
    "from sklearn.datasets import fetch_california_housing,load_diabetes,load_wine,fetch_openml\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt \n",
    "import torch\n",
    "from torchvision import datasets\n",
    "import torchvision"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b582b299-f599-4140-a454-bcbfdeeb273f",
   "metadata": {},
   "source": [
    "### Data Loading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9f0cf9be-2cac-4227-957f-ad893212e70c",
   "metadata": {},
   "outputs": [],
   "source": [
    "california = fetch_california_housing()\n",
    "diabetes = load_diabetes()\n",
    "wine = load_wine()\n",
    "mnist = fetch_openml('mnist_784',version = 1)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29aa0061-eb6a-41ca-97cd-515b8aea3302",
   "metadata": {},
   "outputs": [],
   "source": [
    "#MNIST data\n",
    "train_dataset_m = torchvision.datasets\\\n",
    ".MNIST(root='./data',\n",
    "       train=True, \n",
    "       transform=None,\n",
    "       target_transform=None,\n",
    "       download=True)\n",
    "test_dataset_m = torchvision.datasets\\\n",
    ".MNIST(root='./data',\n",
    "       train=False,\n",
    "       transform=None,\n",
    "       target_transform=None,\n",
    "       download=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a966159b-655e-465b-96e1-d6dc9ef09acb",
   "metadata": {},
   "outputs": [],
   "source": [
    "#FMNIST data\n",
    "train_dataset_f = torchvision.datasets\\\n",
    ".FashionMNIST(root='./data', \n",
    "              train=True,\n",
    "              transform=None,\n",
    "              target_transform=None,\n",
    "              download=True)\n",
    "test_dataset_f = torchvision.datasets\\\n",
    ".FashionMNIST(root='./data',\n",
    "              train=False,\n",
    "              transform=None, \n",
    "              target_transform=None,\n",
    "              download=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "313af93b-9d69-48f9-abc7-3f38053154c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "california_df = pd.DataFrame(california.data, columns=california.feature_names)\n",
    "diabetes_df = pd.DataFrame(diabetes.data, columns=diabetes.feature_names)\n",
    "wine_df = pd.DataFrame(wine.data, columns=wine.feature_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4df9546-a6d7-4678-aca6-cd13d5f3c79a",
   "metadata": {},
   "source": [
    "### Analysis for regression tasks: \n",
    "The general description of the dataset for regression tasks is presented, indicating the amount of instances and features.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a889a6c7-aed8-4a0f-9925-c4f8e2fce1d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "overview = {'data_name':['california','diabetes','wine_df'],\n",
    "            'instance':[california_df.shape[0],diabetes_df.shape[0],wine_df.shape[0]],\n",
    "            'features':[california_df.shape[1],diabetes_df.shape[1],wine_df.shape[1]]\n",
    "           }\n",
    "overview = pd.DataFrame(overview)\n",
    "overview"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2b796dc-f69d-4686-b802-bd0d8f679ee8",
   "metadata": {},
   "source": [
    "### Analysis correlations of regression tasks:\n",
    "\n",
    "correlations amoungst features with targets are vary. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c33b453f-1bc2-4cad-8021-e548d307f154",
   "metadata": {},
   "outputs": [],
   "source": [
    "co_california = california_df.corr()\n",
    "co_diabetes = diabetes_df.corr()\n",
    "co_wine = wine_df.corr()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63367907-e881-47b1-bcd5-1da93711632c",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axs = plt.subplots(nrows=3, ncols=1, figsize=(10, 5))\n",
    "\n",
    "sns.heatmap(co_california, cmap='coolwarm', ax=axs.flat[0])\n",
    "axs.flat[0].set_title('Correlation California')\n",
    "\n",
    "sns.heatmap(co_diabetes, cmap='coolwarm', ax=axs.flat[1])\n",
    "axs.flat[1].set_title('Correlation Diabetes')\n",
    "\n",
    "sns.heatmap(co_wine, cmap='coolwarm', ax=axs.flat[2])\n",
    "axs.flat[2].set_title('Correlation Wine')\n",
    "plt.subplots_adjust(hspace=0.8, wspace=0.8)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f273bea3-ecaa-4fac-83d0-5fe547b7873d",
   "metadata": {},
   "source": [
    "### Analysis for classification tasks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60074a1b-1ae5-46e8-971f-100199861c9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "mnist_x = mnist.data\n",
    "mnist_y =mnist.target\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "851b6b35-5bed-4c9c-8f47-d6211ec674fb",
   "metadata": {},
   "source": [
    "label distribution of MNIST"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35aa5c25-17fa-4079-b5fb-f6e516d535e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "label_counts = np.bincount(mnist_y.astype(int))\n",
    "sns.countplot(x=mnist_y)\n",
    "plt.title('Label distribution')\n",
    "plt.xlabel('Label')\n",
    "plt.ylabel('Count')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56295bb6-7342-4b56-ab7d-fb9d31486c1d",
   "metadata": {},
   "source": [
    "Some sample images from MNIST and instances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f34010cd-1655-46bd-b3ea-7d1a5e8d3af7",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Training set size:', len(train_dataset_m))\n",
    "print('Test set size:', len(test_dataset_m))\n",
    "\n",
    "fig, ax = plt.subplots(1, 5, figsize=(15, 3))\n",
    "for i in range(5):\n",
    "    img, label = train_dataset_m[i]\n",
    "    ax[i].imshow(img, cmap='gray')\n",
    "    ax[i].set_title('Label: {}'.format(label))\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9ee9b94-1bfb-44fb-9543-637a2bed16b7",
   "metadata": {},
   "source": [
    "Label distribution of FMNIST"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7577b50-a3ec-4deb-8551-434631525066",
   "metadata": {},
   "outputs": [],
   "source": [
    "fmnist_trainset = datasets.FashionMNIST(root='./data', train=True, download=True)\n",
    "fmnist_train_labels = fmnist_trainset.targets\n",
    "labels, counts = torch.unique(fmnist_train_labels, return_counts=True)\n",
    "label_dist = dict(zip(labels.numpy(), counts.numpy()))\n",
    "plt.bar(label_dist.keys(), label_dist.values())\n",
    "plt.xlabel('Label')\n",
    "plt.ylabel('Count')\n",
    "plt.title('Fashion-MNIST Label Distribution')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6cb9753-81a1-47d7-8a79-df05cb5b97b0",
   "metadata": {},
   "source": [
    "Some sample images from FMNIST and instances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0167436-0ccd-426f-9115-79f466ccdad2",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Training set size:', len(train_dataset_f))\n",
    "print('Test set size:', len(test_dataset_f))\n",
    "fig, ax = plt.subplots(1, 5, figsize=(15, 3))\n",
    "for i in range(5):\n",
    "    img, label = train_dataset_f[i]\n",
    "    ax[i].imshow(img, cmap='gray')\n",
    "    ax[i].set_title('Label: {}'.format(label))\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "f4bc7a400e35f160b13ed52195005e41b219907c1be09b125a1c17e685484faa"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}