HW3-2-distribution_of_word_frequencies.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Python for Research Homework: Week 3, Case Study 2\n",
    "\n",
    "In this case study, we will find and plot the distribution of word frequencies for each translation of Hamlet.  Perhaps the distribution of word frequencies of Hamlet depends on the translation --- let's find out!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# DO NOT EDIT THIS CODE!\n",
    "import os\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from collections import Counter\n",
    "\n",
    "def count_words_fast(text):\n",
    "    text = text.lower()\n",
    "    skips = [\".\", \",\", \";\", \":\", \"'\", '\"', \"\\n\", \"!\", \"?\", \"(\", \")\"]\n",
    "    for ch in skips:\n",
    "        text = text.replace(ch, \"\")\n",
    "    word_counts = Counter(text.split(\" \"))\n",
    "    return word_counts\n",
    "\n",
    "def word_stats(word_counts):\n",
    "    num_unique = len(word_counts)\n",
    "    counts = word_counts.values()\n",
    "    return (num_unique, counts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1 \n",
    "\n",
    "In this case study, we will find and visualize summary statistics of the text of different translations of Hamlet. For this case study, functions `count_words_fast` and `word_stats` are already defined as in the Case 2 Videos (Videos 3.2.x).\n",
    "\n",
    "#### Instructions \n",
    "- Read in the data as a pandas dataframe using `pd.read_csv`. Use the `index_col` argument to set the first column in the csv file as the index for the dataframe. The data can be found at https://courses.edx.org/asset-v1:HarvardX+PH526x+2T2019+type@asset+block@hamlets.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "hamlets = pd.read_csv('./hamlets.csv', index_col=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2 \n",
    "\n",
    "In this exercise, we will summarize the text for a single translation of Hamlet in a `pandas` dataframe. \n",
    "\n",
    "#### Instructions\n",
    "- Find the dictionary of word frequency in `text` by calling `count_words_fast()`. Store this as `counted_text`.\n",
    "- Create a `pandas` dataframe named `data`.\n",
    "- Using `counted_text`, define two columns in data:\n",
    "    - `word`, consisting of each unique word in text.\n",
    "    - `count`, consisting of the number of times each word in `word` is included in the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "language, text = hamlets.iloc[0]\n",
    "counted_text = count_words_fast(text)\n",
    "data = pd.DataFrame(columns= (\"word\", \"count\"))\n",
    "num = 1\n",
    "for (word, count) in counted_text.items():\n",
    "    data.loc[num] = word, count\n",
    "    num += 1\n",
    "    \n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 3\n",
    "\n",
    "In this exercise, we will continue to define summary statistics for a single translation of Hamlet. \n",
    "\n",
    "#### Instructions\n",
    "- Add a column to data named `length`, defined as the length of each word.\n",
    "- Add another column named `frequency`, which is defined as follows for each word in `data`:\n",
    "    - If `count > 10`, `frequency` is \"frequent\".\n",
    "    - If `1 < count <= 10`, `frequency` is \"infrequent\".\n",
    "    - If `count == 1`, `frequency` is \"unique\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "# write your code here!\n",
    "lengths = []\n",
    "frequencies = []\n",
    "\n",
    "for word in data[\"word\"]:\n",
    "    lengths.append(len(word))\n",
    "\n",
    "for count in data[\"count\"]:\n",
    "    if count > 10:\n",
    "        frequencies.append(\"frequent\")\n",
    "    elif 1 < count <= 10:\n",
    "        frequencies.append(\"infrequent\")\n",
    "    elif count == 1:\n",
    "        frequencies.append(\"unique\")\n",
    "\n",
    "data[\"length\"] = lengths\n",
    "data[\"frequency\"] = frequencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3348"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "frequencies.count(\"unique\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 4\n",
    "\n",
    "In this exercise, we will summarize the statistics in data into a smaller pandas dataframe. \n",
    "\n",
    "#### Instructions \n",
    "- Create a `pandas` dataframe named `sub_data` including the following columns:\n",
    "    - `language`, which is the language of the text (defined in Exercise 2).\n",
    "    - `frequency`, which is a list containing the strings \"frequent\", \"infrequent\", and \"unique\".\n",
    "    - `mean_word_length`, which is the mean word length of each value in frequency.\n",
    "    - `num_words`, which is the total number of words in each frequency category."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "sub_data = pd.DataFrame(columns= ('language', 'frequency', 'mwl', 'num_words'))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 5\n",
    "\n",
    "In this exercise, we will join all the data summaries for text Hamlet translation.\n",
    "\n",
    "#### Instructions \n",
    "- The previous code for summarizing a particular translation of Hamlet is consolidated into a single function called `summarize_text`. Create a pandas dataframe` grouped_data` consisting of the results of `summarize_text` for each translation of Hamlet in `hamlets`.\n",
    "    - Use a `for` loop across the row indices of `hamlets` to assign each translation to a new row.\n",
    "    - Obtain the `ith` row of `hamlets` to variables using the `.iloc` method, and assign the output to variables `language` and `text`.\n",
    "    - Call `summarize_text` using `language` and `text`, and assign the output to `sub_data`.\n",
    "    - Use the pandas `.append()` function to append to pandas dataframes row-wise to `grouped_data`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "           language   frequency  mean_word_length  num_words\n",
      "frequency                                                   \n",
      "frequent    English    frequent          4.371517        323\n",
      "infrequent  English  infrequent          5.825243       1442\n",
      "unique      English      unique          7.005675       3348\n",
      "           language   frequency  mean_word_length  num_words\n",
      "frequency                                                   \n",
      "frequent     German    frequent          4.528053        303\n",
      "infrequent   German  infrequent          6.481830       1596\n",
      "unique       German      unique          9.006987       5582\n",
      "              language   frequency  mean_word_length  num_words\n",
      "frequency                                                      \n",
      "frequent    Portuguese    frequent          4.417625        261\n",
      "infrequent  Portuguese  infrequent          6.497870       1643\n",
      "unique      Portuguese      unique          8.669778       5357\n"
     ]
    }
   ],
   "source": [
    "def summarize_text(language, text):\n",
    "    counted_text = count_words_fast(text)\n",
    "\n",
    "    data = pd.DataFrame({\n",
    "        \"word\": list(counted_text.keys()),\n",
    "        \"count\": list(counted_text.values())\n",
    "    })\n",
    "    \n",
    "    data.loc[data[\"count\"] > 10,  \"frequency\"] = \"frequent\"\n",
    "    data.loc[data[\"count\"] <= 10, \"frequency\"] = \"infrequent\"\n",
    "    data.loc[data[\"count\"] == 1,  \"frequency\"] = \"unique\"\n",
    "    \n",
    "    data[\"length\"] = data[\"word\"].apply(len)\n",
    "    \n",
    "    sub_data = pd.DataFrame({\n",
    "        \"language\": language,\n",
    "        \"frequency\": [\"frequent\",\"infrequent\",\"unique\"],\n",
    "        \"mean_word_length\": data.groupby(by = \"frequency\")[\"length\"].mean(),\n",
    "        \"num_words\": data.groupby(by = \"frequency\").size()\n",
    "    })\n",
    "    \n",
    "    return(sub_data)\n",
    "    \n",
    "# write your code here!\n",
    "grouped_data = pd.DataFrame(columns = ('language', 'frequency', 'mean_word_length', 'num_words'))\n",
    "\n",
    "for row in [0, 1, 2]:\n",
    "    language = hamlets.iloc[row][0]\n",
    "    text = hamlets.iloc[row][1]\n",
    "    sub_data = summarize_text(language, text)\n",
    "    grouped_data.append(sub_data)\n",
    "    print(sub_data)\n",
    "#     for i in [0, 1, 2]:\n",
    "# #         print(sub_data.iloc[i])\n",
    "#         grouped_data.append(sub_data.iloc[i])\n",
    "\n",
    "# sub_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 6\n",
    "\n",
    "In this exercise, we will plot our results and look for differences across each translation.\n",
    "\n",
    "#### Instructions \n",
    "- Plot the word statistics of each translations on a single plot. Note that we have already done most of the work for you.\n",
    "- Consider: do the word statistics differ by translation?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "colors = {\"Portuguese\": \"green\", \"English\": \"blue\", \"German\": \"red\"}\n",
    "markers = {\"frequent\": \"o\",\"infrequent\": \"s\", \"unique\": \"^\"}\n",
    "import matplotlib.pyplot as plt\n",
    "for i in range(grouped_data.shape[0]):\n",
    "    row = grouped_data.iloc[i]\n",
    "    plt.plot(row.mean_word_length, row.num_words,\n",
    "        marker=markers[row.frequency],\n",
    "        color = colors[row.language],\n",
    "        markersize = 10\n",
    "    )\n",
    "\n",
    "color_legend = []\n",
    "marker_legend = []\n",
    "for color in colors:\n",
    "    color_legend.append(\n",
    "        plt.plot([], [],\n",
    "        color=colors[color],\n",
    "        marker=\"o\",\n",
    "        label = color, markersize = 10, linestyle=\"None\")\n",
    "    )\n",
    "for marker in markers:\n",
    "    marker_legend.append(\n",
    "        plt.plot([], [],\n",
    "        color=\"k\",\n",
    "        marker=markers[marker],\n",
    "        label = marker, markersize = 10, linestyle=\"None\")\n",
    "    )\n",
    "plt.legend(numpoints=1, loc = \"upper left\")\n",
    "\n",
    "plt.xlabel(\"Mean Word Length\")\n",
    "plt.ylabel(\"Number of Words\")\n",
    "# write your code to display the plot here!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}