NLP_with_Python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "de439f0e-fabb-4919-8985-4efce29f46af",
   "metadata": {
    "tags": []
   },
   "source": [
    "<img src=\"https://spacy.io/static/social_default-1d3b50b1eba4c2b06244425ff0c49570.jpg\" align='right' width=200>\n",
    "\n",
    "# Natural Language Processing with Python\n",
    "## ... and spaCy\n",
    "\n",
    "This notebook is an exercise-based introductory demo of how to use Python for Natural Language Processing (NLP). It uses open data and the package spaCy, which comes with a lot of functionality for interacting with text data. Similar things can be done with packages like `nltk`. At some point, some machine learning will be done, for which scikit-learn is used. \n",
    "\n",
    "Let's start by importing the pacakges that are used:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "dc7ef533-b329-4573-80c6-d18f76674868",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This notebook uses the following packages (and versions):\n",
      "---------------------------------------------------------\n",
      "python 3.9.7 \n",
      "numpy 1.21.2\n",
      "pandas 1.3.4\n",
      "regex 2.5.103\n",
      "spacy 3.1.3\n",
      "sklearn 1.0\n",
      "gensim 4.1.2\n",
      "matplotlib 3.4.3\n"
     ]
    }
   ],
   "source": [
    "# General imports\n",
    "import sys, os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# NLP related\n",
    "import string\n",
    "import regex as re\n",
    "import spacy\n",
    "\n",
    "# Machine learning\n",
    "import sklearn\n",
    "import gensim\n",
    "\n",
    "# Visualisation\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Print which versions are used\n",
    "print(\"This notebook uses the following packages (and versions):\")\n",
    "print(\"---------------------------------------------------------\")\n",
    "print(\"python\", sys.version[:6])\n",
    "print('\\n'.join(f'{m.__name__} {m.__version__}' for m in globals().values() if getattr(m, '__version__', None)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62cfd6fa-ccca-43aa-8040-fa7be58fc672",
   "metadata": {},
   "source": [
    "## Text data\n",
    "\n",
    "Text is unstructured data, which means that we don't have something like a nice set of features (e.g. columns in a pandas DataFrame) for a set of observations (e.g. rows in that same DataFrame). The information is enclosed in human-readable text, but needs to be made quantitative in order for machine learning methods to be able to handle them. That process of getting quantitative information out of text is called NLP. SpaCy will help us out. \n",
    "\n",
    "\n",
    "## Simple string operations\n",
    "\n",
    "The first step might often be to use Python's rich collection of string operations. For example, making everything lower case, removing punctuation or splitting a document into its consecutive sentences are operations that we wouldn't need anything else than core python for:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4d5ea738-69c5-44eb-ac7c-1f6fbfd8d565",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['This workshop is about language, and Python', \"Let's Go!\"]\n",
      "text text \n",
      "this workshop is about language  and python\n",
      "let s go \n"
     ]
    }
   ],
   "source": [
    "my_text = \"This workshop is about language, and Python. Let's Go!\"\n",
    "sentences = my_text.split('. ')\n",
    "print(sentences)\n",
    "\n",
    "# Removing punctuation with regular expressions\n",
    "def remove_punctuation(text):\n",
    "    pattern = \"[\" + string.punctuation + \"]+\"\n",
    "    result = re.sub(pattern,\" \",text)\n",
    "    return result\n",
    "\n",
    "print(remove_punctuation(\"text!!!text??\"))\n",
    "\n",
    "\n",
    "for s in sentences:\n",
    "    print(remove_punctuation(s.lower()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ecb8146-f87d-4c2b-ad62-7b24e1742abd",
   "metadata": {},
   "source": [
    "After simple operations like that, your results will no longer be case sensitive (but if uppercase is used to find names later on, be careful!). Note that for more complex string operations, it may be very useful to get familiar with [regular expressions](https://regex101.com/)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9dc51c8e-3b9d-49dd-b1a4-18ade8939d47",
   "metadata": {
    "tags": []
   },
   "source": [
    "## SpaCy language models\n",
    "\n",
    "Much of what's here is adapted from the [spaCy documentation](https://spacy.io/).\n",
    "\n",
    "There are many complications. In most applications, you will be after something like *the meaning*, *the context* or *the intent* of text. These can be hard to extract, and we will look at the qunatification of text in steps.\n",
    "\n",
    "From spaCy you can import [pre-trained language models](https://spacy.io/usage/models) in a number of languages, that enable you to digest the \"documents\" (this can be just that example sentence, or a whole collection of books). The examples below show what you can do with such \"NLP models\".\n",
    "\n",
    "### Part-of-Speech Tagging\n",
    "POS tagging can be helpful for understanding the build-up of the text you're dealing with. See below for an example.\n",
    "\n",
    "Let's start with a simple example sentence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "804ed881-ffdb-455d-91bc-70859596be85",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This           DET    nsubj\n",
      "is             AUX    ROOT\n",
      "an             DET    det\n",
      "example        NOUN   compound\n",
      "sentence       NOUN   attr\n",
      "by             ADP    prep\n",
      "Marcel         PROPN  pobj\n",
      "with           ADP    prep\n",
      "a              DET    det\n",
      "somewhat       ADV    advmod\n",
      "obvouis        ADJ    amod\n",
      "spelling       NOUN   compound\n",
      "mistake        NOUN   pobj\n",
      ".              PUNCT  punct\n"
     ]
    }
   ],
   "source": [
    "sentence = \"This is an example sentence by Marcel with a somewhat obvouis spelling mistake.\"\n",
    "\n",
    "nlp = spacy.load('en_core_web_sm')\n",
    "\n",
    "doc = nlp(sentence)\n",
    "for token in doc:\n",
    "    print(f\"{token.text:14s} {token.pos_:6s} {token.dep_}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c513202f-8d31-4d59-9ac4-776a4c5b85af",
   "metadata": {},
   "source": [
    "And if you need to know what any of those abbreviations mean, you can invoke"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "cc711a23-14a2-48a1-9f18-7c49162bf6e1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'adjective'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spacy.explain(\"ADJ\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b12e3a83-3220-49c8-991e-078a0a346d71",
   "metadata": {},
   "source": [
    "Which shows that even a spelling mistake gets correctly interpreted. The interplay of words within a sentence is also known to the `doc` object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "14550c3b-72f1-40ca-bd6a-082dff5d861d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" xml:lang=\"en\" id=\"b4da1500c1944be4b49a69369c7733ba-0\" class=\"displacy\" width=\"2325\" height=\"574.5\" direction=\"ltr\" style=\"max-width: none; height: 574.5px; color: #000000; background: #ffffff; font-family: Arial; direction: ltr\">\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"484.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">This</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">DET</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"484.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"225\">is</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"225\">AUX</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"484.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"400\">an</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"400\">DET</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"484.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"575\">example</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"575\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"484.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"750\">sentence</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"750\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"484.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"925\">by</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"925\">ADP</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"484.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1100\">Marcel</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1100\">PROPN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"484.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1275\">with</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1275\">ADP</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"484.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1450\">a</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1450\">DET</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"484.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1625\">somewhat</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1625\">ADV</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"484.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1800\">obvouis</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1800\">ADJ</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"484.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1975\">spelling</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1975\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"484.5\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"2150\">mistake.</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"2150\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-b4da1500c1944be4b49a69369c7733ba-0-0\" stroke-width=\"2px\" d=\"M70,439.5 C70,352.0 205.0,352.0 205.0,439.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-b4da1500c1944be4b49a69369c7733ba-0-0\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">nsubj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M70,441.5 L62,429.5 78,429.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-b4da1500c1944be4b49a69369c7733ba-0-1\" stroke-width=\"2px\" d=\"M420,439.5 C420,264.5 735.0,264.5 735.0,439.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-b4da1500c1944be4b49a69369c7733ba-0-1\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">det</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M420,441.5 L412,429.5 428,429.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-b4da1500c1944be4b49a69369c7733ba-0-2\" stroke-width=\"2px\" d=\"M595,439.5 C595,352.0 730.0,352.0 730.0,439.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-b4da1500c1944be4b49a69369c7733ba-0-2\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">compound</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M595,441.5 L587,429.5 603,429.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-b4da1500c1944be4b49a69369c7733ba-0-3\" stroke-width=\"2px\" d=\"M245,439.5 C245,177.0 740.0,177.0 740.0,439.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-b4da1500c1944be4b49a69369c7733ba-0-3\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">attr</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M740.0,441.5 L748.0,429.5 732.0,429.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-b4da1500c1944be4b49a69369c7733ba-0-4\" stroke-width=\"2px\" d=\"M770,439.5 C770,352.0 905.0,352.0 905.0,439.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-b4da1500c1944be4b49a69369c7733ba-0-4\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">prep</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M905.0,441.5 L913.0,429.5 897.0,429.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-b4da1500c1944be4b49a69369c7733ba-0-5\" stroke-width=\"2px\" d=\"M945,439.5 C945,352.0 1080.0,352.0 1080.0,439.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-b4da1500c1944be4b49a69369c7733ba-0-5\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">pobj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1080.0,441.5 L1088.0,429.5 1072.0,429.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-b4da1500c1944be4b49a69369c7733ba-0-6\" stroke-width=\"2px\" d=\"M770,439.5 C770,177.0 1265.0,177.0 1265.0,439.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-b4da1500c1944be4b49a69369c7733ba-0-6\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">prep</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1265.0,441.5 L1273.0,429.5 1257.0,429.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-b4da1500c1944be4b49a69369c7733ba-0-7\" stroke-width=\"2px\" d=\"M1470,439.5 C1470,89.5 2145.0,89.5 2145.0,439.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-b4da1500c1944be4b49a69369c7733ba-0-7\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">det</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1470,441.5 L1462,429.5 1478,429.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-b4da1500c1944be4b49a69369c7733ba-0-8\" stroke-width=\"2px\" d=\"M1645,439.5 C1645,352.0 1780.0,352.0 1780.0,439.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-b4da1500c1944be4b49a69369c7733ba-0-8\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">advmod</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1645,441.5 L1637,429.5 1653,429.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-b4da1500c1944be4b49a69369c7733ba-0-9\" stroke-width=\"2px\" d=\"M1820,439.5 C1820,264.5 2135.0,264.5 2135.0,439.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-b4da1500c1944be4b49a69369c7733ba-0-9\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">amod</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1820,441.5 L1812,429.5 1828,429.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-b4da1500c1944be4b49a69369c7733ba-0-10\" stroke-width=\"2px\" d=\"M1995,439.5 C1995,352.0 2130.0,352.0 2130.0,439.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-b4da1500c1944be4b49a69369c7733ba-0-10\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">compound</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1995,441.5 L1987,429.5 2003,429.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-b4da1500c1944be4b49a69369c7733ba-0-11\" stroke-width=\"2px\" d=\"M1295,439.5 C1295,2.0 2150.0,2.0 2150.0,439.5\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-b4da1500c1944be4b49a69369c7733ba-0-11\" class=\"displacy-label\" startOffset=\"50%\" side=\"left\" fill=\"currentColor\" text-anchor=\"middle\">pobj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M2150.0,441.5 L2158.0,429.5 2142.0,429.5\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "</svg></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "spacy.displacy.render(doc, style='dep')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d9132fa-aa77-46ef-9f06-755bdafdce46",
   "metadata": {},
   "source": [
    "### Named entity recognition\n",
    "\n",
    "SpaCy understands that my name is a \"named entity\" and it can try to figure out what kind of an entity I am:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "719c54dc-a296-4d32-849b-d49fe16854dd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Marcel is a PRODUCT and appears in the sentence at position 31\n"
     ]
    }
   ],
   "source": [
    "for ent in doc.ents: print(f\"{ent} is a {ent.label_} and appears in the sentence at position {ent.start_char}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25c7bb4f-a4b2-48cb-9666-7fd1420583d2",
   "metadata": {},
   "source": [
    "--- \n",
    "#### Exercise\n",
    "Just to get familiar with this type of exercise and solution loading: \n",
    "\n",
    "As you can see, my name isn't totally obvious for spaCy. Try with \"Steve\" and see if it gets better. Also, use the displacy renderer with `style='ent` to see what it recognizes in the sentence \"Steve worked for Apple until January 2011\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5291805f-0a1f-4988-bdd1-d90f974d0042",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "883d3ed6-e446-46ee-a1be-40cd3f1a0557",
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you want solutions, uncomment and run the next two lines.\n",
    "# to_include = os.path.join('solutions', 'NER.py')\n",
    "# %load $to_include"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d385dd00-7501-4466-8ed9-f7e7c24730ca",
   "metadata": {},
   "source": [
    "In many real-world applications, paying special attention to pre-defined entities is very valuable!\n",
    "\n",
    "### Stopwords\n",
    "\n",
    "In many cases, tere is little to no information in super common words like *the* or *is*. Note that **this depends on your use case!!**. In general, the most common words in a language do'n add information because they appear all over the place, but their actual meaning might be important in your context. SpaCy comes with lists of stopwords that are useful for most use cases:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "10b301ba-6f52-4fb6-a817-2093019a9c2f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I know 326 stopwords.\n"
     ]
    }
   ],
   "source": [
    "stopwords = spacy.lang.en.stop_words.STOP_WORDS\n",
    "print(f\"I know {len(stopwords)} stopwords.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3e8973e-ec51-4ae6-a2d7-7369e5b67bd7",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "#### Exercise\n",
    "`stopwords` is a set. Can you think of a reason why?\n",
    "\n",
    "What is the longest stopword in English included in spaCy?\n",
    "\n",
    "Add a few more stopwords: \"and\", \"market\" and \"people\". How many of them were already in the collection? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c650231f-85c4-4c18-a8b0-93282d13e2ec",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d0a21e9-1297-4e9b-96ed-a1ff9ae639a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you want solutions, uncomment and run the next two lines.\n",
    "# to_include = os.path.join('solutions', 'stopwords.py')\n",
    "# %load $to_include"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d38b68ec-26fe-4cec-b961-981f1debdd59",
   "metadata": {},
   "source": [
    "## Text Normalization: Stemming and Lemmatization\n",
    "\n",
    "Often, the information content of a text does not depend on verb conjugations, single vs plural, etc. In such cases we want to use text normalization in order to tell our future model that \"is\" and \"was\" are both represnetations of \"be\". In the same vein, you could also map synonyms to the same word. This is less common, for reasons you can probably think of. This type of word normalization can be done in a variety of ways.\n",
    "\n",
    "Stemming means to cut-off parts of the word (typically a suffix) to get back to the root of the word (e.g. reading -> read, played -> play etc.). This is a simple procedure. \n",
    "\n",
    "Lemmatization, on the other hand, is an organized and step-by-step procedure of obtaining the root form of the word. It makes use of vocabulary (dictionary importance of words) and morphological analysis (word structure and grammar relations). It typically results in more useful features for our future predictive models. It can be done with spaCy like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "3b95ffec-aaee-4c6b-8c35-b73d901ab884",
   "metadata": {},
   "outputs": [],
   "source": [
    "from_the_news = \"Belarus has been accused of taking revenge for EU sanctions by offering migrants tourist visas, and helping them across its border. The BBC has tracked one group trying to reach Germany.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "bd927d49-1695-426a-a136-899727bb160d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Belarus have be accuse of take revenge for EU sanction by offer migrant tourist visa , and help they across its border . the BBC have track one group try to reach Germany .'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc = nlp(from_the_news)\n",
    "\n",
    "lemma_word1 = [] \n",
    "for token in doc:\n",
    "    lemma_word1.append(token.lemma_)\n",
    "' '.join(lemma_word1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c5f5b9f-d164-4832-8c98-29eb5ebaadb3",
   "metadata": {},
   "source": [
    "## Preprocessing pipelines\n",
    "\n",
    "Before to jump to learning on our text data, let's create a pipeline for preprocessing the data in way that we would want to. We will use pandas pipes to combine the functions into a pipeline. They work in a dataframe containing the data, so let's first create a simple data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "4d637884-2706-41bb-a599-961f8fdc350b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>My first text ingredient.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>More text. In the DataFrame.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           text\n",
       "0     My first text ingredient.\n",
       "1  More text. In the DataFrame."
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame({'text':['My first text ingredient.', 'More text. In the DataFrame.']})\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "0be311a8-e7e7-425d-b8e1-9b716eb77cc1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      my first text ingredient\n",
       "1    more text in the dataframe\n",
       "Name: text, dtype: object"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def remove_period(text):\n",
    "    return text.str.replace(\".\", \"\", regex=False)\n",
    "\n",
    "def to_lower(text):\n",
    "    return text.str.lower()\n",
    "\n",
    "processed = (df.text.pipe(remove_period)\n",
    "                    .pipe(to_lower)\n",
    "            )\n",
    "\n",
    "processed"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fc4800a-d044-4890-b931-bdabe7bf8dde",
   "metadata": {},
   "source": [
    "For the cases below, we will be using a subset of the \"20 newsgroup\" dataset that comes along with scikit-learn. These are kind of discussion forums on which people get questions answered. We will load the subset here and quickly look at it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a7ad8578-56f8-4ff9-8ff7-f67b00eb6816",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2261,)\n",
      "2261\n"
     ]
    }
   ],
   "source": [
    "from sklearn.datasets import fetch_20newsgroups\n",
    "\n",
    "# We will load only 4 of the categories\n",
    "cats = ['sci.space', 'sci.med', 'rec.autos', 'alt.atheism']\n",
    "data = fetch_20newsgroups(categories=cats, \n",
    "                          remove=('headers', 'footers'))\n",
    "\n",
    "print(data.target.shape)\n",
    "print(len(data.data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "6ef68957-af6a-4a18-8027-ce52cc118f35",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sci.med\n",
      "\n",
      "Does anyone on this newsgroup happen to know WHY morphine was\n",
      "first isolated from opium?  If you know why, or have an idea for where I\n",
      "could look to find this info, please mail me.\n",
      "\tCSH\n",
      "any suggestionas would be greatly appreciated\n",
      "\n",
      "--\n",
      " \"Kilimanjaro is a pretty tricky climb. Most of it's up, until you reach\n",
      "the very, very top, and then it tends to slope away rather sharply.\"\n",
      "\t\t\t\t\tSir George Head, OBE (JC)\n"
     ]
    }
   ],
   "source": [
    "# Get a random one\n",
    "random_index = np.random.randint(0, high=len(data.data))\n",
    "print(data.target_names[data.target[random_index]])\n",
    "print()\n",
    "print(data.data[random_index])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98ef2277-bcaf-4e51-b295-00d666f26241",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "#### Exercise\n",
    "\n",
    "Create a pre-processing pipeline that cleans the data of the newsgroups. You can think of your own steps and order (order matters!), or you can take these steps (these might well be sub-optimal!):\n",
    "1. Transform to lower case\n",
    "2. Remove punctuation\n",
    "3. Lemmatize\n",
    "4. Remove stop words -- Look at the stop words list: is this lemmatized?\n",
    "\n",
    "Are you ready? Or do you need to remove more?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ebcf000-eae3-4869-b3f9-3e5d580079ac",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "id": "96d785e6-cb36-4c19-b132-f9d612fce73b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you want solutions, uncomment and run the next two lines.\n",
    "# to_include = os.path.join('solutions', 'preprocessor.py')\n",
    "# %load $to_include"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c0ec68d-cf33-4182-8c7e-5242f49e2007",
   "metadata": {},
   "source": [
    "## Vectorize the preprocessed text data\n",
    "\n",
    "In order for machine learning methods to deal with this cleaned-up text data, we are going to build a \"bag of words\" matrix out of these. This is a huge feature space where every observations is a document, and every single word that is in at least one of the documents is a feature. This will be a very sparse matrix.\n",
    "\n",
    "We can use either the `CountVectorizer` or the `TfidfVectorizer` from scikit-learn for this. \n",
    "\n",
    "---\n",
    "\n",
    "#### Exercise\n",
    "\n",
    "Give that a go, look at the various 'hyperparameters\" for the vectorizers and play with it a bit. Down below, we will use these in supervised and unsupervised learning.\n",
    "\n",
    "Note that these vectorizers can take preprocessor functions as well! This will need to be done just slightly differently than above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ed3405f-de10-4653-858f-2473182bb2b5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "id": "b0353148-4bbe-4067-9c05-2626c25e6eee",
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you want solutions, uncomment and run the next two lines.\n",
    "# to_include = os.path.join('solutions', 'vectorizer.py')\n",
    "# %load $to_include"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "923c64ca-2ce0-44b3-a62f-8540e3ca8911",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Another example, calling pre-processors from the vectorizer.\n",
    "\n",
    "# If you want solutions, uncomment and run the next two lines.\n",
    "# to_include = os.path.join('solutions', 'vectorizer_preprocessor.py')\n",
    "# %load $to_include"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f79732c-af5b-43a2-813e-acd514e8a543",
   "metadata": {},
   "source": [
    "Just like with more common data sets, we can do both supervised and unsupervised machine learning with text data, after the vectorization described above. After all, we created numeric features (based on the occurence of words) for all documents, which serve as the observations of our model. Hence, we can use our data set to train a machine learning model like we are used to. \n",
    "\n",
    "Below are a supervised learning example, in which the label (the category in the 20 newsgroups data set) is predicted based on the bags-of-words. We can also pretend that we do not yet know these labels, or that there are 4 and do an unsupervised, clustering-like analysis. In this case, that is known as topic modeling and is described after the supervised example.\n",
    "\n",
    "The two examples will be followed by a brief discussion of more elaborate machine learning techniques based on the *context* of words, rather than the words themselves, through word vectors.\n",
    "\n",
    "## Supervised learning: text classification\n",
    "\n",
    "We have a feature matrix (the result of the Count- or TfIdf-Vectorizer above) as well as a label (the category the text came from) for a subset of the 20 newsgroups data set. Building a predictive algorithm, that based on the occurence of words will determine which of the 4 labels fit best can be made in a way completely analogously to how we would do this with a feature matrix of another origin.\n",
    "\n",
    "---\n",
    "#### Exercise\n",
    "\n",
    "If you follow the these steps, your predictive model will be built:\n",
    "- Split your feature matrix and target vector in a train and a test set (e.g. a random 20% of your data can go in the test set) using `sklearn.model_selection.train_test_split`\n",
    "- Instantiate a supervised classification algorithm. For example, use `sklearn.naive_bayes.MultinomialNB` with the default settings\n",
    "- Train on the train set and evaluate the predictions on the independent test, using a visualization of the confusion matrix (`sklearn.metrics.confusion_matrix`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "415be8c5-aba8-4990-bad5-a0e873d8f1b0",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "973461e1-907a-4f44-a6cc-247b9f46fc6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you want solutions, uncomment and run the next two lines.\n",
    "# to_include = os.path.join('solutions', 'naive_bayes.py')\n",
    "# %load $to_include"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "288421f4-2ce4-4e8b-87c9-4477ae0b3647",
   "metadata": {
    "tags": []
   },
   "source": [
    "The above example is as simple as the example below. By no means do I pretend that this is all there is to machine learning! I do hope that it shows you how to use a bag-of-words to do machine learning on text data.\n",
    "\n",
    "## Unsupervised learning: topic modeling with LDA\n",
    "\n",
    "In the unsupervised setting we look for structure present in the data that we do not have a \"target variable\" for. We do not know the correct answer, if that even means something. \n",
    "\n",
    "In this particular example, we would hope that 4 clusters are present, which in reality are described by the 4 different labels that we predicted above. Here we have a look at the data and try to find 4 topics, described by a form of soft clustering through [Latent Dirichlet Allocation](https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation).\n",
    "\n",
    "The procedure is very similar to the supervised learning example above. It doesn't make much sense to split off a test set, as there is nothing to test.\n",
    "\n",
    "---\n",
    "#### Exercise\n",
    "Run an LDA clustering with 4 components (`sklearn.decomposition.LatentDirichletAllocation`).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "875a03d8-e73d-48c8-a94b-d12a5afdb1a6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a525dbb7-6565-4a62-8c8f-bc349b273c67",
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you want solutions, uncomment and run the next two lines.\n",
    "# to_include = os.path.join('solutions', 'lda.py')\n",
    "# %load $to_include"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "801f5ffa-0866-4d26-a987-17d2d225bc2c",
   "metadata": {},
   "source": [
    "We can easily visualize the results with `pyLDAvis.sklearn` to investigate our topics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "id": "2a6586f3-8c7f-49bb-844b-c89a0465ad31",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/marcel/miniforge-pypy3/envs/nlp/lib/python3.9/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function get_feature_names is deprecated; get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use get_feature_names_out instead.\n",
      "  warnings.warn(msg, category=FutureWarning)\n",
      "/home/marcel/miniforge-pypy3/envs/nlp/lib/python3.9/site-packages/pyLDAvis/_prepare.py:246: FutureWarning: In a future version of pandas all arguments of DataFrame.drop except for the argument 'labels' will be keyword-only\n",
      "  default_term_info = default_term_info.sort_values(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<link rel=\"stylesheet\" type=\"text/css\" href=\"https://cdn.jsdelivr.net/gh/bmabey/pyLDAvis@3.3.1/pyLDAvis/js/ldavis.v1.0.0.css\">\n",
       "\n",
       "\n",
       "<div id=\"ldavis_el139251399679162056642841328478\"></div>\n",
       "<script type=\"text/javascript\">\n",
       "\n",
       "var ldavis_el139251399679162056642841328478_data = {\"mdsDat\": {\"x\": [-0.1378278220385027, 0.18748998323933333, -0.030039823688888766, -0.019622337511941947], \"y\": [-0.0555401212641172, -0.036335311662497295, -0.09218575025104578, 0.18406118317766024], \"topics\": [1, 2, 3, 4], \"cluster\": [1, 1, 1, 1], \"Freq\": [27.435133879286518, 27.31592260559262, 26.720585791642858, 18.528357723478006]}, \"tinfo\": {\"Term\": [\"space\", \"car\", \"god\", \"nasa\", \"launch\", \"atheist\", \"orbit\", \"patient\", \"medical\", \"satellite\", \"disease\", \"religion\", \"atheism\", \"health\", \"msg\", \"moon\", \"mission\", \"argument\", \"shuttle\", \"earth\", \"lunar\", \"drive\", \"com\", \"buy\", \"jesus\", \"people\", \"pitt\", \"belief\", \"program\", \"doctor\", \"god\", \"atheist\", \"religion\", \"atheism\", \"jesus\", \"belief\", \"islam\", \"bible\", \"christian\", \"islamic\", \"faith\", \"muslim\", \"god exist\", \"fallacy\", \"believe god\", \"innocent\", \"theist\", \"jaeger\", \"evil\", \"christianity\", \"alt atheism\", \"liar\", \"muslims\", \"mathew\", \"satan\", \"qur\", \"prophecy\", \"verse\", \"rushdie\", \"existence god\", \"religious\", \"existence\", \"argument\", \"claim\", \"statement\", \"matthew\", \"evidence\", \"believe\", \"exist\", \"accept\", \"people\", \"truth\", \"true\", \"think\", \"woman\", \"example\", \"war\", \"book\", \"say\", \"mean\", \"thing\", \"don\", \"edu\", \"know\", \"science\", \"man\", \"point\", \"way\", \"like\", \"time\", \"read\", \"good\", \"question\", \"come\", \"try\", \"post\", \"orbit\", \"satellite\", \"mission\", \"moon\", \"shuttle\", \"lunar\", \"spacecraft\", \"launch\", \"planetary\", \"digex\", \"access digex\", \"henry\", \"jpl\", \"mar\", \"space station\", \"payload\", \"space shuttle\", \"sci space\", \"astronomy\", \"titan\", \"gif\", \"orbiter\", \"propulsion\", \"astronaut\", \"solar\", \"space\", \"alaska\", \"alaska edu\", \"mars\", \"jsc\", \"nasa\", \"rocket\", \"pat\", \"nasa gov\", \"earth\", \"flight\", \"ftp\", \"datum\", \"station\", \"technology\", \"commercial\", \"program\", \"gov\", \"access\", \"request\", \"project\", \"software\", \"cost\", \"send\", \"dc\", \"fund\", \"system\", \"available\", \"edu\", \"year\", \"include\", \"information\", \"work\", \"time\", \"like\", \"use\", \"new\", \"com\", \"know\", \"think\", \"msg\", \"dealer\", \"tire\", \"keyboard\", \"wheel\", \"tek\", \"tek com\", \"honda\", \"truck\", \"wagon\", \"ico\", \"ico tek\", \"vice ico\", \"mustang\", \"sho\", \"brake\", \"rec auto\", \"glutamate\", \"mot com\", \"mot\", \"beauchaine\", \"bobbe\", \"bobbe vice\", \"needle\", \"vw\", \"drain\", \"clutch\", \"ford\", \"integra\", \"boyle\", \"car\", \"auto\", \"oil\", \"rear\", \"buy\", \"driver\", \"drive\", \"uiuc edu\", \"mph\", \"uiuc\", \"food\", \"price\", \"speed\", \"engine\", \"road\", \"com\", \"door\", \"owner\", \"taste\", \"get\", \"good\", \"mile\", \"like\", \"edu\", \"want\", \"look\", \"don\", \"go\", \"ve\", \"know\", \"time\", \"new\", \"thank\", \"problem\", \"year\", \"think\", \"use\", \"thing\", \"work\", \"people\", \"say\", \"lot\", \"find\", \"right\", \"pitt\", \"cancer\", \"gordon\", \"pitt edu\", \"gordon bank\", \"geb\", \"livesey\", \"infection\", \"hiv\", \"bank n3jxp\", \"chastity intellect\", \"n3jxp\", \"n3jxp skepticism\", \"skepticism chastity\", \"shameful\", \"cadre\", \"cadre dsl\", \"dsl\", \"dsl pitt\", \"edu shameful\", \"geb cadre\", \"intellect geb\", \"shameful surrender\", \"surrender soon\", \"sgi com\", \"sgi\", \"com jon\", \"frank\", \"diagnose\", \"tobacco\", \"medical\", \"patient\", \"yeast\", \"intellect\", \"health\", \"jon\", \"disease\", \"drug\", \"bank\", \"treatment\", \"medicine\", \"water\", \"doctor\", \"objective\", \"chastity\", \"morality\", \"edu\", \"system\", \"year\", \"child\", \"physician\", \"caltech edu\", \"com\", \"dr\", \"study\", \"moral\", \"10\", \"center\", \"don\", \"know\", \"cause\", \"information\", \"think\", \"say\", \"use\", \"case\", \"people\", \"time\"], \"Freq\": [1125.0, 1141.0, 724.0, 541.0, 439.0, 339.0, 315.0, 248.0, 236.0, 287.0, 307.0, 273.0, 261.0, 227.0, 244.0, 238.0, 236.0, 273.0, 211.0, 315.0, 207.0, 333.0, 1365.0, 260.0, 196.0, 1171.0, 140.0, 178.0, 341.0, 232.0, 724.0578168624595, 338.720539757753, 272.7434893528648, 260.31266577232657, 195.295863244525, 178.06727875085454, 158.0072755541922, 157.04968165940537, 155.09283627205093, 108.282139464871, 99.67976285288331, 95.85203845882846, 78.64518443083108, 73.86406413209417, 73.86052158550913, 73.84151306981346, 69.08061943071291, 64.29939987346633, 64.28842666697871, 63.3426910276449, 63.340526802914965, 59.520803954511386, 59.52066016958064, 58.55915248304118, 57.607531522998656, 55.696659936039374, 53.78452945701498, 49.95323611632529, 49.00315777915004, 48.991641019536154, 161.36746006045257, 119.03416762068377, 262.42707038376506, 261.1494228322475, 137.71634304755676, 91.64752936344938, 242.59497800874442, 360.0094374653838, 251.55557536637173, 136.77498972110874, 687.2072849111646, 90.0355660916519, 202.38651866651838, 607.7612759854377, 140.56911661223626, 209.36446822146678, 100.04733297032534, 187.33172596227874, 399.7403737023941, 246.273922593285, 355.1419453507479, 441.0242463312652, 640.1479182595317, 419.71182473421777, 215.23038879733056, 187.27316037986634, 237.30205506589536, 258.5710527262157, 358.0247752041579, 311.9141832841666, 213.82165710246713, 259.133926206518, 219.4745276214004, 214.72300397062554, 216.5075213465116, 212.8993883798804, 315.0870331036333, 287.1761966310884, 236.1317299217864, 238.02428059821503, 211.10759071576794, 206.2913382139786, 122.51931237253072, 436.1027230546448, 87.85566033996535, 86.88771038446687, 86.88771038446686, 85.91293925241344, 79.18652850854168, 75.34172397137291, 72.45449665423413, 72.45398613445326, 71.48962400352939, 71.48827646675454, 71.4881639494451, 68.60338436830477, 68.57999130236857, 64.75251557569415, 64.75104888274191, 64.74981389374071, 128.51853064053205, 1112.6835341093272, 68.52289440850727, 61.8537557623939, 57.04832327077278, 55.11452350018664, 529.9932240172649, 174.86333207205394, 88.47884947871967, 163.73260166969698, 270.3010072182419, 155.27822059259495, 92.32694020393025, 208.6794435013089, 131.63417187599484, 169.78855140299947, 112.34334454451789, 260.12908010086954, 183.9587889651053, 140.71099033277235, 143.1151375793595, 150.26189576444426, 125.34229123927761, 217.94552150201696, 176.61725587054582, 118.08622908713676, 107.45151361903143, 295.9455461249254, 162.492159098135, 471.93521607888493, 290.6059540776789, 174.9708140909699, 166.15908035501107, 189.9830734301129, 201.0980003795239, 202.98382734099746, 181.05922386817576, 177.4125277440435, 192.68651369841876, 181.76044583978305, 176.94402836838174, 243.69161959854026, 145.35655251336377, 117.67091371965, 107.15036589809348, 96.65999344934718, 70.86928102669735, 70.86928102669735, 66.11073828134839, 59.420384192028386, 55.61021340664116, 55.59803549826227, 55.59803549826227, 55.59803549826227, 54.65199057965264, 51.78879822041906, 121.46655463691538, 54.56341897931524, 45.10416915653836, 47.900238286581846, 47.90023828591302, 44.139265861626924, 44.139265861626924, 44.139265861626924, 43.19098733734297, 41.26732690490849, 38.42106700424799, 37.47246440696192, 94.56984267441733, 35.563820741905346, 35.55604225551941, 1109.3133030207782, 144.05075808853442, 158.12671961370003, 68.7217089217857, 235.46690057702756, 131.79203853571485, 286.86858536001284, 93.72102783333843, 76.54526915623002, 103.6261546999222, 253.82196592369573, 186.16850903997306, 170.53318891138002, 220.26457091427534, 127.17907664871288, 735.4531809221128, 78.39496337891417, 75.27184185434633, 64.66104100866474, 286.71209417098305, 396.9614359531328, 105.50274167959292, 459.29576655634736, 704.7965659989137, 263.90430522385304, 268.8202144159414, 394.76992115054674, 252.40222179425362, 218.78823591047762, 375.42948162044775, 333.8048879653266, 256.1838359899552, 158.48801553354676, 218.92881835953682, 258.4354346126078, 291.66169629170054, 220.5206243535814, 230.72309837248775, 209.03920315585486, 243.76544459916076, 222.20706577286498, 173.5572220147824, 180.89456041384128, 177.35753517880582, 139.5723456466372, 132.98307551323404, 126.38689092179472, 113.21075988293248, 113.20922279849717, 105.67967953397131, 99.07859780013551, 100.94067328361886, 95.3263749166636, 74.61097177183835, 74.61097177183835, 74.61097177183835, 74.61097177183835, 74.61097177183835, 74.60724305258938, 72.72843864913382, 72.72843864913382, 72.72843864913382, 72.72843864913382, 72.72843864913382, 72.72843864913382, 72.72843864913382, 72.72843864913382, 72.72843864913382, 65.18491181998495, 67.0207517916022, 56.71686070892529, 60.41831865280837, 55.752693572100576, 52.952995762347804, 228.72258552904594, 238.58593417625752, 82.73453457071372, 75.51425840783442, 206.09199354243577, 83.75021674561219, 240.66135803795402, 104.52136779201251, 136.02631267221014, 136.00835415119096, 103.94027384297944, 138.74218671940702, 164.0175171818709, 122.61840055016302, 74.59050327545633, 107.35496047742481, 420.0761728097707, 228.55973122176795, 230.27321710941533, 118.98898813836479, 81.40992329699414, 82.37041922553668, 241.6498756263653, 98.24525131780177, 132.3694218909758, 109.60203121682696, 131.72907348533866, 120.6285914165673, 186.03911226745143, 181.7644019426411, 136.0286725808593, 128.89708637175576, 162.06100219869796, 147.84952342164019, 138.71028466557743, 126.34705630382672, 138.41581575097996, 122.6878606103398], \"Total\": [1125.0, 1141.0, 724.0, 541.0, 439.0, 339.0, 315.0, 248.0, 236.0, 287.0, 307.0, 273.0, 261.0, 227.0, 244.0, 238.0, 236.0, 273.0, 211.0, 315.0, 207.0, 333.0, 1365.0, 260.0, 196.0, 1171.0, 140.0, 178.0, 341.0, 232.0, 724.7788579209127, 339.44001087243515, 273.4638947719854, 261.0336109248296, 196.01365197128254, 178.802340577347, 158.7228171944186, 157.76663283594314, 155.85389376304235, 109.00162748492797, 100.39608266947698, 96.5713417312824, 79.36023460874259, 74.57935268571659, 74.57934253285063, 74.57934544853101, 69.79844128568801, 65.01756254684076, 65.01763679507131, 64.06139705533052, 64.06141344862719, 60.23671989614888, 60.23671541400514, 59.2804761168708, 58.32436129246054, 56.41201830869702, 54.499669290692545, 50.67500282425565, 49.718779522270744, 49.71876706900357, 164.42356465712044, 121.4067750458503, 273.41550543827935, 312.5408377727961, 156.7808934495072, 99.48936172297654, 308.95421919960637, 500.5860429063604, 333.67123885042605, 163.41390791137337, 1171.4146954474795, 99.33162647511139, 282.90249557435914, 1238.4280028442179, 180.22609733974284, 320.1744017315323, 114.76270207640646, 284.804315166882, 856.612334276884, 433.34570221699045, 762.2099051789171, 1154.403469052176, 2236.9558731471006, 1158.6661541370897, 374.7490203045526, 293.63847325316954, 463.4145811631885, 575.2963669436746, 1138.64067771845, 969.504932239357, 426.1334493018823, 855.5002929261864, 495.2527618993217, 477.5681423887411, 495.40703002560315, 509.797666062659, 315.81439611237994, 287.89111241507385, 236.85845741930208, 238.78353196926832, 211.82382527259946, 207.0093979294233, 123.23909044990077, 439.0450192567416, 88.57547887325438, 87.61256578828971, 87.61256578828974, 86.64963597476113, 79.90954607131341, 76.05818938555441, 73.16956745115083, 73.16956481453923, 72.20667605183394, 72.20666926293572, 72.2066196612559, 69.31806587619745, 69.31783699517915, 65.46656162905377, 65.46653808384094, 65.46653368050491, 129.96275605514248, 1125.5065378640281, 69.31738117160442, 62.57784150806183, 57.763536952675324, 55.83768823918402, 541.0396251677965, 185.59876931219523, 91.44499694302083, 177.05283372649407, 315.4614530680622, 171.27589471629906, 97.14837518149324, 245.78922259183744, 148.14272216009974, 200.69614542401408, 123.11055387188173, 341.1426419650765, 228.66535695932373, 165.17615505974243, 172.05285692466038, 183.31224520325765, 150.64737371184634, 339.622738056025, 254.27952540322318, 141.02921120432194, 124.85831087402957, 719.3264752521561, 268.223896510083, 2236.9558731471006, 856.6916626298837, 358.84588920923863, 361.49100935185186, 595.5755813065973, 969.504932239357, 1138.64067771845, 665.2218599719347, 629.1467638652546, 1365.7149447090064, 1158.6661541370897, 1238.4280028442179, 244.41326754537053, 146.0747473093887, 118.38722679648407, 107.88513644436324, 97.38291366799568, 71.60480654681011, 71.60480654681011, 66.83105416349127, 60.14778404481837, 56.32892241950394, 56.32889278882115, 56.32889278882115, 56.32889278882115, 55.374225299504644, 52.50996525749414, 123.16927812224698, 55.37297549431507, 45.82674303186464, 48.69151941771224, 48.69151941771794, 44.87196260338483, 44.87196260338483, 44.87196260338483, 43.917252562537335, 42.007851858436304, 39.14345151892733, 38.1888145232108, 96.4123235271378, 36.27931833359254, 36.27921910313104, 1141.727314991778, 149.93718416384638, 166.11342627582883, 71.57511076347762, 260.823836259066, 142.25881306007335, 333.2739579889567, 100.23241505603562, 81.09878729211569, 113.54661414776062, 308.0133054707174, 225.58850126707324, 208.36224632491331, 282.14416153532954, 155.53630174104518, 1365.7149447090064, 87.8470622238476, 84.04445190062434, 68.74559311396737, 525.2151576433456, 855.5002929261864, 134.859752367879, 1138.64067771845, 2236.9558731471006, 540.5603033106155, 590.2886220442166, 1154.403469052176, 576.1930922342656, 457.1656684084025, 1158.6661541370897, 969.504932239357, 629.1467638652546, 273.6481027465414, 577.3944409728693, 856.6916626298837, 1238.4280028442179, 665.2218599719347, 762.2099051789171, 595.5755813065973, 1171.4146954474795, 856.612334276884, 359.686434316482, 591.415310408697, 515.1257301514343, 140.29626933382744, 133.70571759487876, 127.11530799503653, 113.93419361612796, 113.93422003109413, 106.40216112740035, 99.81179523983543, 101.695120358946, 96.04557154737358, 75.33256777158591, 75.33256777158591, 75.33256777158591, 75.33256777158591, 75.33256777158591, 75.332615715516, 73.44955225482583, 73.44955225482583, 73.44955225482583, 73.44955225482583, 73.44955225482583, 73.44955225482583, 73.44955225482583, 73.44955225482583, 73.44955225482583, 65.9177039671559, 67.80150529815286, 57.444098341361276, 61.211402337130004, 56.50292741206358, 53.67804696144563, 236.43681162372152, 248.71332122518623, 84.76770577808159, 77.22935381649765, 227.19890825385943, 86.66286013023725, 307.95666107212605, 115.02578404172127, 164.21603827291213, 165.19442273332984, 116.94720702637066, 181.38118733435198, 232.58378259766255, 158.84631106867562, 76.28906254669332, 140.79836109284793, 2236.9558731471006, 719.3264752521561, 856.6916626298837, 204.65969451634936, 91.48163691604432, 95.28236202576477, 1365.7149447090064, 147.8585236714921, 315.80813156686054, 201.97076938992484, 342.9924738941056, 279.2903461856952, 1154.403469052176, 1158.6661541370897, 426.2195285799812, 361.49100935185186, 1238.4280028442179, 856.612334276884, 665.2218599719347, 410.2730212503146, 1171.4146954474795, 969.504932239357], \"Category\": [\"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Default\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic1\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic2\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic3\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\", \"Topic4\"], \"logprob\": [30.0, 29.0, 28.0, 27.0, 26.0, 25.0, 24.0, 23.0, 22.0, 21.0, 20.0, 19.0, 18.0, 17.0, 16.0, 15.0, 14.0, 13.0, 12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0, -4.7582, -5.5179, -5.7346, -5.7812, -6.0686, -6.1609, -6.2805, -6.2865, -6.2991, -6.6584, -6.7411, -6.7803, -6.9782, -7.0409, -7.0409, -7.0412, -7.1078, -7.1796, -7.1797, -7.1945, -7.1946, -7.2568, -7.2568, -7.2731, -7.2894, -7.3232, -7.3581, -7.432, -7.4512, -7.4515, -6.2594, -6.5637, -5.7731, -5.778, -6.4179, -6.8252, -5.8517, -5.457, -5.8154, -6.4248, -4.8105, -6.8429, -6.0329, -4.9333, -6.3974, -5.999, -6.7375, -6.1102, -5.3523, -5.8367, -5.4706, -5.254, -4.8814, -5.3035, -5.9714, -6.1105, -5.8738, -5.7879, -5.4625, -5.6004, -5.978, -5.7858, -5.9519, -5.9738, -5.9655, -5.9823, -5.5859, -5.6787, -5.8744, -5.8664, -5.9864, -6.0095, -6.5305, -5.2609, -6.8631, -6.8741, -6.8741, -6.8854, -6.9669, -7.0167, -7.0558, -7.0558, -7.0692, -7.0692, -7.0692, -7.1104, -7.1107, -7.1682, -7.1682, -7.1682, -6.4827, -4.3242, -7.1116, -7.214, -7.2948, -7.3293, -5.0659, -6.1747, -6.856, -6.2405, -5.7392, -6.2935, -6.8134, -5.9979, -6.4587, -6.2042, -6.6172, -5.7776, -6.124, -6.392, -6.3751, -6.3264, -6.5077, -5.9545, -6.1648, -6.5673, -6.6617, -5.6486, -6.2481, -5.1819, -5.6668, -6.1741, -6.2258, -6.0918, -6.035, -6.0256, -6.1399, -6.1603, -6.0777, -6.1361, -6.1629, -5.8208, -6.3375, -6.5488, -6.6425, -6.7455, -7.0559, -7.0559, -7.1254, -7.2321, -7.2983, -7.2986, -7.2986, -7.2986, -7.3157, -7.3695, -6.5171, -7.3173, -7.5077, -7.4476, -7.4476, -7.5294, -7.5294, -7.5294, -7.5511, -7.5966, -7.6681, -7.6931, -6.7674, -7.7454, -7.7456, -4.3052, -6.3465, -6.2533, -7.0866, -5.8551, -6.4355, -5.6577, -6.7764, -6.9788, -6.6759, -5.7801, -6.0901, -6.1778, -5.9219, -6.4711, -4.7162, -6.955, -6.9956, -7.1476, -5.6582, -5.3329, -6.658, -5.187, -4.7588, -5.7411, -5.7227, -5.3384, -5.7857, -5.9286, -5.3886, -5.5062, -5.7708, -6.251, -5.928, -5.7621, -5.6411, -5.9207, -5.8755, -5.9742, -5.8205, -5.9131, -6.1602, -6.1188, -6.1385, -6.012, -6.0604, -6.1112, -6.2213, -6.2213, -6.2902, -6.3547, -6.336, -6.3933, -6.6383, -6.6383, -6.6383, -6.6383, -6.6383, -6.6383, -6.6638, -6.6638, -6.6638, -6.6638, -6.6638, -6.6638, -6.6638, -6.6638, -6.6638, -6.7734, -6.7456, -6.9125, -6.8493, -6.9297, -6.9812, -5.5181, -5.4759, -6.5349, -6.6263, -5.6223, -6.5227, -5.4672, -6.3012, -6.0377, -6.0379, -6.3068, -6.018, -5.8506, -6.1415, -6.6386, -6.2744, -4.9101, -5.5188, -5.5113, -6.1715, -6.5511, -6.5394, -5.4631, -6.3631, -6.065, -6.2537, -6.0698, -6.1579, -5.7246, -5.7479, -6.0377, -6.0916, -5.8626, -5.9544, -6.0182, -6.1115, -6.0203, -6.1409], \"loglift\": [30.0, 29.0, 28.0, 27.0, 26.0, 25.0, 24.0, 23.0, 22.0, 21.0, 20.0, 19.0, 18.0, 17.0, 16.0, 15.0, 14.0, 13.0, 12.0, 11.0, 10.0, 9.0, 8.0, 7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0, 1.2924, 1.2912, 1.2907, 1.2906, 1.2897, 1.2892, 1.2888, 1.2888, 1.2885, 1.2867, 1.2862, 1.2859, 1.2843, 1.2837, 1.2837, 1.2834, 1.283, 1.2822, 1.2821, 1.2821, 1.282, 1.2814, 1.2814, 1.2811, 1.281, 1.2806, 1.2801, 1.279, 1.2788, 1.2786, 1.2746, 1.2736, 1.2523, 1.1137, 1.1637, 1.2112, 1.0515, 0.9637, 1.0109, 1.1154, 0.76, 1.1951, 0.9584, 0.5815, 1.0448, 0.8686, 1.1561, 0.8744, 0.5312, 0.7283, 0.5296, 0.3311, 0.0422, 0.2779, 0.7388, 0.8436, 0.6241, 0.4936, 0.1364, 0.1593, 0.6037, 0.099, 0.4795, 0.494, 0.4656, 0.4202, 1.2954, 1.2952, 1.2946, 1.2945, 1.2943, 1.2942, 1.2918, 1.291, 1.2895, 1.2894, 1.2894, 1.2892, 1.2886, 1.2882, 1.2879, 1.2879, 1.2877, 1.2877, 1.2877, 1.2873, 1.287, 1.2867, 1.2867, 1.2867, 1.2865, 1.2862, 1.2862, 1.2861, 1.2852, 1.2847, 1.2771, 1.2381, 1.2647, 1.2195, 1.1432, 1.1996, 1.2468, 1.134, 1.1796, 1.1305, 1.2062, 1.0266, 1.0802, 1.1374, 1.1135, 1.0989, 1.1138, 0.8541, 0.9333, 1.1201, 1.1476, 0.4096, 0.7965, -0.2583, 0.2166, 0.5794, 0.5204, 0.1551, -0.2753, -0.4268, -0.0036, 0.0318, -0.6607, -0.5546, -0.6481, 1.3168, 1.3148, 1.3137, 1.3129, 1.3123, 1.3094, 1.3094, 1.3089, 1.3076, 1.3069, 1.3067, 1.3067, 1.3067, 1.3066, 1.3059, 1.3058, 1.305, 1.3038, 1.3034, 1.3034, 1.3033, 1.3033, 1.3033, 1.3031, 1.302, 1.3011, 1.3008, 1.3004, 1.2998, 1.2996, 1.2909, 1.2797, 1.2705, 1.2791, 1.2175, 1.2433, 1.1698, 1.2526, 1.262, 1.2283, 1.1262, 1.1277, 1.1194, 1.0721, 1.1185, 0.7008, 1.2059, 1.2095, 1.2585, 0.7144, 0.5519, 1.0742, 0.4118, 0.1648, 0.6027, 0.5332, 0.2467, 0.4943, 0.5828, 0.1928, 0.2535, 0.4213, 0.7736, 0.35, 0.1213, -0.1263, 0.2156, 0.1247, 0.2727, -0.25, -0.0296, 0.591, 0.1351, 0.2535, 1.6807, 1.6804, 1.6801, 1.6795, 1.6795, 1.6791, 1.6785, 1.6784, 1.6784, 1.6762, 1.6762, 1.6762, 1.6762, 1.6762, 1.6762, 1.676, 1.676, 1.676, 1.676, 1.676, 1.676, 1.676, 1.676, 1.676, 1.6747, 1.6743, 1.6731, 1.6728, 1.6725, 1.6723, 1.6527, 1.6443, 1.6616, 1.6634, 1.5884, 1.6517, 1.4393, 1.5901, 1.4975, 1.4915, 1.568, 1.4179, 1.3366, 1.427, 1.6634, 1.4147, 0.0134, 0.5394, 0.3721, 1.1436, 1.5692, 1.5402, -0.0461, 1.2771, 0.8163, 1.0746, 0.7289, 0.8463, -0.1395, -0.1664, 0.5438, 0.6546, -0.3478, -0.0709, 0.1181, 0.5081, -0.4498, -0.3813]}, \"token.table\": {\"Topic\": [1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 4, 2, 2, 2, 1, 1, 3, 4, 2, 2, 1, 1, 2, 3, 1, 2, 3, 4, 1, 3, 4, 4, 3, 1, 1, 2, 3, 4, 1, 1, 3, 3, 1, 2, 3, 4, 3, 2, 3, 1, 2, 3, 4, 4, 2, 3, 4, 4, 1, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 4, 4, 1, 3, 4, 1, 1, 1, 2, 3, 4, 3, 1, 2, 3, 4, 4, 1, 2, 3, 4, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 2, 4, 3, 4, 2, 1, 3, 4, 1, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 3, 1, 2, 3, 4, 2, 3, 4, 1, 4, 4, 4, 1, 2, 4, 1, 2, 3, 4, 4, 1, 2, 3, 1, 2, 3, 4, 1, 1, 2, 3, 4, 1, 2, 3, 4, 1, 4, 1, 1, 1, 1, 2, 3, 4, 1, 2, 1, 3, 4, 3, 4, 4, 2, 4, 2, 3, 4, 4, 4, 1, 2, 3, 4, 2, 3, 1, 2, 3, 4, 1, 1, 1, 2, 3, 4, 4, 4, 2, 3, 4, 1, 3, 4, 2, 4, 3, 3, 3, 1, 2, 3, 4, 4, 1, 2, 3, 4, 1, 3, 3, 4, 4, 1, 1, 1, 1, 1, 3, 4, 2, 2, 3, 1, 2, 3, 4, 1, 2, 3, 1, 1, 2, 3, 4, 4, 1, 2, 3, 4, 1, 2, 3, 4, 2, 1, 2, 3, 4, 2, 2, 1, 1, 2, 1, 2, 3, 4, 1, 4, 1, 4, 2, 3, 2, 2, 1, 2, 3, 4, 1, 4, 3, 3, 3, 4, 3, 1, 1, 3, 4, 4, 2, 3, 2, 3, 3, 1, 2, 3, 4, 1, 2, 4, 1, 3, 4, 2, 2, 1, 2, 3, 2, 3, 1, 4, 2, 1, 2, 3, 4, 1, 4, 4, 4, 2, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 1, 2, 3, 4, 1, 1, 2, 3, 4, 3, 4, 3, 1, 1, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 2, 4, 1, 1, 2, 1, 2, 3, 4, 2, 1, 2, 4, 1, 2, 3, 4, 4, 4, 4, 4, 3, 2, 4, 1, 2, 4, 2, 4, 1, 2, 3, 2, 2, 2, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 1, 2, 3, 4, 4, 1, 2, 3, 4, 1, 3, 2, 3, 4, 3, 3, 1, 2, 3, 4, 1, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 3, 2, 4, 1, 3, 4, 3, 1, 2, 3, 4, 1, 2, 4, 1, 2, 3, 4, 2, 3, 4, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 3, 3, 3, 1, 2, 3, 4, 1, 2, 3, 1, 3, 4, 1, 2, 3, 4, 3, 1, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 4], \"Freq\": [0.017493095203751963, 0.3906791262171272, 0.20991714244502357, 0.38484809448254326, 0.8383619347399806, 0.061194301805838, 0.0428360112640866, 0.061194301805838, 0.04237899833343484, 0.8536341092877591, 0.10292042452405606, 0.9930082427928207, 0.9954213334918308, 0.9907660364413914, 0.983431313930056, 0.9582485074502979, 0.02925949641069612, 0.010972311154011045, 0.9928737073085048, 0.9832893484431686, 0.9960403148040302, 0.9987037153595882, 0.03334729825615616, 0.9604021897772975, 0.05219520028661471, 0.6039730318879702, 0.20878080114645883, 0.13048800071653677, 0.0060895391858016385, 0.16441755801664423, 0.8281773292690228, 0.9955853386998, 0.9805677631911947, 0.995512695332979, 0.7191570861821683, 0.055934440036390864, 0.15581736867280313, 0.06792039147276034, 0.9922318632321618, 0.9951407162454919, 0.9805677631911947, 0.9805677631911947, 0.6565911752089386, 0.18258150326665673, 0.04915655857179219, 0.10884666540896842, 0.9923036076841317, 0.008118907695533362, 0.9823878311595369, 0.01533602165113068, 0.08051411366843608, 0.9009912720039275, 0.993879441861454, 0.993879441861454, 0.01049512185402788, 0.11544634039430668, 0.8605999920302861, 0.9947218592624659, 0.016641451728898003, 0.9713352614393624, 0.011386256446088108, 0.39242161112458607, 0.09018384851931481, 0.20961651277462362, 0.3071125652279369, 0.3378540642653309, 0.05396280193126812, 0.28858367989330347, 0.319084394028368, 0.00716100655577345, 0.5263339818493487, 0.03580503277886725, 0.43324089662429377, 0.01310803890646765, 0.9831029179850738, 0.9955853386998, 0.346917355504643, 0.06840623911359157, 0.5814530324655284, 0.9945211906970987, 0.9834315655899015, 0.8350908695961706, 0.04799372813771095, 0.07678996502033753, 0.04159456438601616, 0.9688700857030205, 0.14351457510173313, 0.14131792344201272, 0.5381796566314992, 0.177196567217446, 0.9922690345190515, 0.450197533957342, 0.13401228917799948, 0.31827918679774875, 0.09841527486509337, 0.9097514102369791, 0.056859463139811196, 0.02436834134563337, 0.04711109771855323, 0.6418887064152877, 0.2649999246668619, 0.04711109771855323, 0.028479686481715033, 0.850322067811206, 0.004068526640245005, 0.11391874592686013, 0.8367060908327891, 0.15599605083323187, 0.9926424838708612, 0.9910990910542415, 0.9930082427928211, 0.201327030187144, 0.016236050821543873, 0.7825776495984147, 0.06019336276862452, 0.23217439925040886, 0.7051222495753158, 0.38201548403357055, 0.11521101899425144, 0.34216806393029564, 0.1611221769393291, 0.05691709971198859, 0.034150259827193157, 0.887906755507022, 0.022766839884795435, 0.04734255304450627, 0.22994954335903045, 0.06086899677150806, 0.6627957426230877, 0.9707881784932934, 0.08401496525248398, 0.03600641367963599, 0.8611533938379607, 0.018003206839817994, 0.04217664882010724, 0.9278862740423595, 0.028117765880071498, 0.08693702966956415, 0.9128388115304236, 0.993879441861454, 0.993879441861454, 0.1331382950009373, 0.8558890392917397, 0.009509878214352663, 0.2861030955874891, 0.2110010329957732, 0.31516044123309345, 0.1877551564792897, 0.993879441861454, 0.010632862234947738, 0.20556866987565628, 0.7797432305628342, 0.7865242968020603, 0.0032367255012430464, 0.03884070601491656, 0.17154645156588147, 0.984348296166488, 0.6527692372335482, 0.16241148486193543, 0.08745233800258062, 0.09369893357419352, 0.7552344063821557, 0.14984809650439596, 0.026972657370791273, 0.07192708632211006, 0.9801759412113421, 0.016473545230442727, 0.9855433448700366, 0.9960547995604474, 0.9922317281546003, 0.32295410118485035, 0.17077677601921407, 0.30604550949977966, 0.19952138188383423, 0.0934165314185184, 0.9049726481168969, 0.08116532486086622, 0.8246397005864008, 0.09415177683860482, 0.9853512136678226, 0.01037211803860866, 0.9802095313801497, 0.9470050304817244, 0.04117413176007497, 0.8569713882158237, 0.040045391972702044, 0.09610894073448491, 0.9962203669254535, 0.993879441861454, 0.23990168251305874, 0.12756676768551536, 0.5464427212797449, 0.0875831539333389, 0.995414787752231, 0.9819593761815065, 0.24818069138159646, 0.21694116379510178, 0.4373533862109252, 0.0971896413802056, 0.9989253854297752, 0.9954607668372126, 0.30274682795736557, 0.12857973388150662, 0.4640559486450739, 0.1052016004485054, 0.9912260135098749, 0.9918003561104015, 0.8046693318425622, 0.15743530405615347, 0.03498562312358966, 0.004401429600544803, 0.08362716241035127, 0.9066944977122295, 0.992502727017222, 0.9891137974346077, 0.9875648502946215, 0.9941612062205061, 0.9941612062205061, 0.18113625362474, 0.4876745289896846, 0.18113625362474, 0.15048242608824552, 0.9931646635896345, 0.09682121849380014, 0.45920920771345214, 0.08575593638022298, 0.3568553481628634, 0.9922318244408456, 0.9923008935552708, 0.012948444478456599, 0.9840817803627014, 0.993879441861454, 0.9954460410469326, 0.9908108942220476, 0.9843494202645988, 0.9948286664674201, 0.011538968348115866, 0.011538968348115866, 0.9692733412417327, 0.9886178045548937, 0.984997798698332, 0.9917955663445843, 0.3624857759936836, 0.1570771695972629, 0.32364801428007467, 0.1570771695972629, 0.002277670753885098, 0.9930644486939026, 0.004555341507770196, 0.9960701728686921, 0.3144099864035616, 0.1782827576534162, 0.4031122451375273, 0.10363234188720745, 0.9918667404199595, 0.22700759424423142, 0.27444201692213055, 0.4557092750126735, 0.044046249629477736, 0.16959216189489965, 0.17515256064555207, 0.4837546913067629, 0.17237236127022587, 0.9951239028782286, 0.6368375299335253, 0.22476618703536183, 0.06129986919146232, 0.07492206234512061, 0.9860871078564566, 0.9867816793611361, 0.9952686595108, 0.9247219844084404, 0.07035928142238133, 0.5676761041853363, 0.11076606910933391, 0.17537960942311204, 0.14538046570600077, 0.029606218895982167, 0.9685463038828451, 0.1111612695211148, 0.8892901561689184, 0.21503821185205768, 0.7860017398730383, 0.9963756522411932, 0.9967186515635879, 0.4406578252329998, 0.009902423038943816, 0.004951211519471908, 0.5446332671419098, 0.2343777281486857, 0.7599520276336174, 0.9857979494994705, 0.9857979494995859, 0.9494593269643853, 0.049322562439708326, 0.9983091443867964, 0.994083734148872, 0.9960702469851119, 0.9932418865007943, 0.9955853386998, 0.9955853386998, 0.979595533017803, 0.020331228043765723, 0.9262771826252851, 0.07342441081785797, 0.9791140722832972, 0.12556688603888227, 0.28133340289724257, 0.40690028893612484, 0.1859661476778383, 0.1196124724091675, 0.10702168583978144, 0.7743333740172422, 0.03611989791864922, 0.9511573118577628, 0.012039965972883072, 0.9974212824924861, 0.9928732834374684, 0.07139079218571748, 0.03569539609285874, 0.8923849023214684, 0.962327114022789, 0.021871070773245207, 0.04020693363242073, 0.9609457138148554, 0.9840156926243351, 0.5864703615806753, 0.0870742021560828, 0.20829515025572748, 0.1178062735052885, 0.10931155516136343, 0.8854235968070437, 0.9978882593583264, 0.9918005860534241, 0.9935029549873746, 0.5114211111034117, 0.14242106891487413, 0.26326318799416126, 0.08415790435878927, 0.41781281904460554, 0.3001190672010547, 0.1883100029496814, 0.09611656400556653, 0.03546280043116575, 0.13741835167076727, 0.8245101100246036, 0.3030857029124444, 0.16106840211918474, 0.37929010821614473, 0.15587264721211427, 0.008793975396095796, 0.7621445343283023, 0.03224457645235125, 0.19639878384613946, 0.027275864710816083, 0.8182759413244824, 0.027275864710816083, 0.12546897766975398, 0.990831700500284, 0.9928736405269595, 0.4421984425894424, 0.13730362600950724, 0.254415542311734, 0.16557201959969992, 0.9926962671953629, 0.5021901011304974, 0.15957442465828892, 0.24170831970299644, 0.09621399133808596, 0.9640222594696757, 0.02794267418752683, 0.9932643046362325, 0.9983036342974191, 0.9791783819779133, 0.018245559912631926, 0.06393384101036317, 0.8311399331347213, 0.07555817573952012, 0.029060836822892355, 0.3785483593348653, 0.11065259734403755, 0.34360543385780085, 0.1669495328348637, 0.03214683610212475, 0.0642936722042495, 0.8165296369939686, 0.08358177386552436, 0.9428941832347656, 0.05387966761341518, 0.985543098016942, 0.9944386653317288, 0.99690468938899, 0.46695568578015295, 0.10156286165718327, 0.2591604056079849, 0.1727736037386566, 0.9832886729819691, 0.5737173103888915, 0.39759943836253414, 0.026684526064599607, 0.07865359968831563, 0.6960843572415933, 0.15730719937663126, 0.07078823971948406, 0.9881786503908978, 0.9860780350053886, 0.9955847050795092, 0.993879441861454, 0.9902882194837987, 0.996110799757585, 0.9955853386998, 0.01991405443110019, 0.8297522679625079, 0.1460363991614014, 0.9925920618770658, 0.007694512107574154, 0.009773377257208704, 0.988888080661208, 0.0017769776831288552, 0.9832885805328067, 0.9840156571660527, 0.9980599463284909, 0.01439800181133533, 0.15357868598757685, 0.8206861032461138, 0.01439800181133533, 0.8802092969603099, 0.03826996943305695, 0.06378328238842826, 0.025513312955371303, 0.013500494461270763, 0.8910326344438704, 0.09450346122889534, 0.21848709106273229, 0.22165357064335162, 0.13932510154724959, 0.41797530464174876, 0.993879441861454, 0.15292082772490764, 0.4114960455142969, 0.118166094151065, 0.3183533595363986, 0.058185547884774896, 0.945515153127592, 0.8470516443693434, 0.0548092240474281, 0.09965313463168746, 0.9915535482046903, 0.9915535482046903, 0.08770387866430525, 0.14251880282949603, 0.5773838678733428, 0.19002507043932804, 0.9885607576475819, 0.4657509664830047, 0.13250942990079853, 0.30306612185232135, 0.09839809151049395, 0.49094497104688, 0.14292312479489763, 0.23578278214751475, 0.13081099557499107, 0.3218137315499202, 0.20732230782542935, 0.34450572544126074, 0.1268688749379493, 0.996729150542991, 0.9954115009965004, 0.987368263194586, 0.1029090432879975, 0.07264167761505706, 0.82327234630398, 0.9809172679751075, 0.7140269285708919, 0.08130009582737878, 0.1555306181045507, 0.04948701485144796, 0.9060558373374714, 0.010067287081527461, 0.08053829665221969, 0.4380236590279819, 0.11909399024263104, 0.310855499955342, 0.1332237856951466, 0.026420867081919645, 0.9159233921732144, 0.0616486898578125, 0.029930437157708205, 0.9378203642748572, 0.029930437157708205, 0.18790723444547308, 0.27208967547704505, 0.33221999049959644, 0.20895284470336606, 0.26904907454712823, 0.12468127844866919, 0.4790385961448869, 0.12686866929864585, 0.9866797674072835, 0.9941612062205061, 0.9760080124584162, 0.9941606832622445, 0.2867395164807991, 0.13689499496502666, 0.4883821441995546, 0.0887967534908281, 0.8713632407628589, 0.05228179444577153, 0.0784226916686573, 0.07167226210751525, 0.16539752794041981, 0.7663418794572784, 0.45020273876570105, 0.19642049992480393, 0.2537822388408971, 0.0990793672187064, 0.9960679583966738, 0.7823506255822761, 0.04993727397333678, 0.17200616590816, 0.2082019543648245, 0.3190191236235214, 0.35092103598587354, 0.12257050539219506, 0.08988064592997713, 0.33967880474835516, 0.3011585279212221, 0.2684746566739577, 0.023593890876744014, 0.9791464713848765], \"Term\": [\"10\", \"10\", \"10\", \"10\", \"accept\", \"accept\", \"accept\", \"accept\", \"access\", \"access\", \"access\", \"access digex\", \"alaska\", \"alaska edu\", \"alt atheism\", \"argument\", \"argument\", \"argument\", \"astronaut\", \"astronomy\", \"atheism\", \"atheist\", \"auto\", \"auto\", \"available\", \"available\", \"available\", \"available\", \"bank\", \"bank\", \"bank\", \"bank n3jxp\", \"beauchaine\", \"belief\", \"believe\", \"believe\", \"believe\", \"believe\", \"believe god\", \"bible\", \"bobbe\", \"bobbe vice\", \"book\", \"book\", \"book\", \"book\", \"boyle\", \"brake\", \"brake\", \"buy\", \"buy\", \"buy\", \"cadre\", \"cadre dsl\", \"caltech edu\", \"caltech edu\", \"caltech edu\", \"cancer\", \"car\", \"car\", \"car\", \"case\", \"case\", \"case\", \"case\", \"cause\", \"cause\", \"cause\", \"cause\", \"center\", \"center\", \"center\", \"center\", \"chastity\", \"chastity\", \"chastity intellect\", \"child\", \"child\", \"child\", \"christian\", \"christianity\", \"claim\", \"claim\", \"claim\", \"claim\", \"clutch\", \"com\", \"com\", \"com\", \"com\", \"com jon\", \"come\", \"come\", \"come\", \"come\", \"commercial\", \"commercial\", \"commercial\", \"cost\", \"cost\", \"cost\", \"cost\", \"datum\", \"datum\", \"datum\", \"datum\", \"dc\", \"dc\", \"dealer\", \"diagnose\", \"digex\", \"disease\", \"disease\", \"disease\", \"doctor\", \"doctor\", \"doctor\", \"don\", \"don\", \"don\", \"don\", \"door\", \"door\", \"door\", \"door\", \"dr\", \"dr\", \"dr\", \"dr\", \"drain\", \"drive\", \"drive\", \"drive\", \"drive\", \"driver\", \"driver\", \"driver\", \"drug\", \"drug\", \"dsl\", \"dsl pitt\", \"earth\", \"earth\", \"earth\", \"edu\", \"edu\", \"edu\", \"edu\", \"edu shameful\", \"engine\", \"engine\", \"engine\", \"evidence\", \"evidence\", \"evidence\", \"evidence\", \"evil\", \"example\", \"example\", \"example\", \"example\", \"exist\", \"exist\", \"exist\", \"exist\", \"existence\", \"existence\", \"existence god\", \"faith\", \"fallacy\", \"find\", \"find\", \"find\", \"find\", \"flight\", \"flight\", \"food\", \"food\", \"food\", \"ford\", \"ford\", \"frank\", \"ftp\", \"ftp\", \"fund\", \"fund\", \"fund\", \"geb\", \"geb cadre\", \"get\", \"get\", \"get\", \"get\", \"gif\", \"glutamate\", \"go\", \"go\", \"go\", \"go\", \"god\", \"god exist\", \"good\", \"good\", \"good\", \"good\", \"gordon\", \"gordon bank\", \"gov\", \"gov\", \"gov\", \"health\", \"health\", \"health\", \"henry\", \"hiv\", \"honda\", \"ico\", \"ico tek\", \"include\", \"include\", \"include\", \"include\", \"infection\", \"information\", \"information\", \"information\", \"information\", \"innocent\", \"integra\", \"intellect\", \"intellect\", \"intellect geb\", \"islam\", \"islamic\", \"jaeger\", \"jesus\", \"jon\", \"jon\", \"jon\", \"jpl\", \"jsc\", \"keyboard\", \"know\", \"know\", \"know\", \"know\", \"launch\", \"launch\", \"launch\", \"liar\", \"like\", \"like\", \"like\", \"like\", \"livesey\", \"look\", \"look\", \"look\", \"look\", \"lot\", \"lot\", \"lot\", \"lot\", \"lunar\", \"man\", \"man\", \"man\", \"man\", \"mar\", \"mars\", \"mathew\", \"matthew\", \"matthew\", \"mean\", \"mean\", \"mean\", \"mean\", \"medical\", \"medical\", \"medicine\", \"medicine\", \"mile\", \"mile\", \"mission\", \"moon\", \"moral\", \"moral\", \"moral\", \"moral\", \"morality\", \"morality\", \"mot\", \"mot com\", \"mph\", \"mph\", \"msg\", \"muslim\", \"muslims\", \"mustang\", \"n3jxp\", \"n3jxp skepticism\", \"nasa\", \"nasa\", \"nasa gov\", \"nasa gov\", \"needle\", \"new\", \"new\", \"new\", \"new\", \"objective\", \"objective\", \"objective\", \"oil\", \"oil\", \"oil\", \"orbit\", \"orbiter\", \"owner\", \"owner\", \"owner\", \"pat\", \"pat\", \"patient\", \"patient\", \"payload\", \"people\", \"people\", \"people\", \"people\", \"physician\", \"physician\", \"pitt\", \"pitt edu\", \"planetary\", \"point\", \"point\", \"point\", \"point\", \"post\", \"post\", \"post\", \"post\", \"price\", \"price\", \"price\", \"problem\", \"problem\", \"problem\", \"problem\", \"program\", \"program\", \"program\", \"program\", \"project\", \"project\", \"project\", \"project\", \"prophecy\", \"propulsion\", \"question\", \"question\", \"question\", \"question\", \"qur\", \"read\", \"read\", \"read\", \"read\", \"rear\", \"rear\", \"rec auto\", \"religion\", \"religious\", \"religious\", \"request\", \"request\", \"request\", \"request\", \"right\", \"right\", \"right\", \"right\", \"road\", \"road\", \"road\", \"road\", \"rocket\", \"rocket\", \"rushdie\", \"satan\", \"satellite\", \"say\", \"say\", \"say\", \"say\", \"sci space\", \"science\", \"science\", \"science\", \"send\", \"send\", \"send\", \"send\", \"sgi\", \"sgi com\", \"shameful\", \"shameful surrender\", \"sho\", \"shuttle\", \"skepticism chastity\", \"software\", \"software\", \"software\", \"solar\", \"solar\", \"space\", \"space\", \"space\", \"space shuttle\", \"space station\", \"spacecraft\", \"speed\", \"speed\", \"speed\", \"speed\", \"statement\", \"statement\", \"statement\", \"statement\", \"station\", \"station\", \"station\", \"study\", \"study\", \"study\", \"study\", \"surrender soon\", \"system\", \"system\", \"system\", \"system\", \"taste\", \"taste\", \"technology\", \"technology\", \"technology\", \"tek\", \"tek com\", \"thank\", \"thank\", \"thank\", \"thank\", \"theist\", \"thing\", \"thing\", \"thing\", \"thing\", \"think\", \"think\", \"think\", \"think\", \"time\", \"time\", \"time\", \"time\", \"tire\", \"titan\", \"tobacco\", \"treatment\", \"treatment\", \"treatment\", \"truck\", \"true\", \"true\", \"true\", \"true\", \"truth\", \"truth\", \"truth\", \"try\", \"try\", \"try\", \"try\", \"uiuc\", \"uiuc\", \"uiuc\", \"uiuc edu\", \"uiuc edu\", \"uiuc edu\", \"use\", \"use\", \"use\", \"use\", \"ve\", \"ve\", \"ve\", \"ve\", \"verse\", \"vice ico\", \"vw\", \"wagon\", \"want\", \"want\", \"want\", \"want\", \"war\", \"war\", \"war\", \"water\", \"water\", \"water\", \"way\", \"way\", \"way\", \"way\", \"wheel\", \"woman\", \"woman\", \"woman\", \"work\", \"work\", \"work\", \"work\", \"year\", \"year\", \"year\", \"year\", \"yeast\", \"yeast\"]}, \"R\": 30, \"lambda.step\": 0.01, \"plot.opts\": {\"xlab\": \"PC1\", \"ylab\": \"PC2\"}, \"topic.order\": [3, 1, 2, 4]};\n",
       "\n",
       "function LDAvis_load_lib(url, callback){\n",
       "  var s = document.createElement('script');\n",
       "  s.src = url;\n",
       "  s.async = true;\n",
       "  s.onreadystatechange = s.onload = callback;\n",
       "  s.onerror = function(){console.warn(\"failed to load library \" + url);};\n",
       "  document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
       "}\n",
       "\n",
       "if(typeof(LDAvis) !== \"undefined\"){\n",
       "   // already loaded: just create the visualization\n",
       "   !function(LDAvis){\n",
       "       new LDAvis(\"#\" + \"ldavis_el139251399679162056642841328478\", ldavis_el139251399679162056642841328478_data);\n",
       "   }(LDAvis);\n",
       "}else if(typeof define === \"function\" && define.amd){\n",
       "   // require.js is available: use it to load d3/LDAvis\n",
       "   require.config({paths: {d3: \"https://d3js.org/d3.v5\"}});\n",
       "   require([\"d3\"], function(d3){\n",
       "      window.d3 = d3;\n",
       "      LDAvis_load_lib(\"https://cdn.jsdelivr.net/gh/bmabey/pyLDAvis@3.3.1/pyLDAvis/js/ldavis.v3.0.0.js\", function(){\n",
       "        new LDAvis(\"#\" + \"ldavis_el139251399679162056642841328478\", ldavis_el139251399679162056642841328478_data);\n",
       "      });\n",
       "    });\n",
       "}else{\n",
       "    // require.js not available: dynamically load d3 & LDAvis\n",
       "    LDAvis_load_lib(\"https://d3js.org/d3.v5.js\", function(){\n",
       "         LDAvis_load_lib(\"https://cdn.jsdelivr.net/gh/bmabey/pyLDAvis@3.3.1/pyLDAvis/js/ldavis.v3.0.0.js\", function(){\n",
       "                 new LDAvis(\"#\" + \"ldavis_el139251399679162056642841328478\", ldavis_el139251399679162056642841328478_data);\n",
       "            })\n",
       "         });\n",
       "}\n",
       "</script>"
      ],
      "text/plain": [
       "PreparedData(topic_coordinates=              x         y  topics  cluster       Freq\n",
       "topic                                                \n",
       "2     -0.137828 -0.055540       1        1  27.435134\n",
       "0      0.187490 -0.036335       2        1  27.315923\n",
       "1     -0.030040 -0.092186       3        1  26.720586\n",
       "3     -0.019622  0.184061       4        1  18.528358, topic_info=         Term         Freq        Total Category  logprob  loglift\n",
       "12011   space  1125.000000  1125.000000  Default  30.0000  30.0000\n",
       "2235      car  1141.000000  1141.000000  Default  29.0000  29.0000\n",
       "5706      god   724.000000   724.000000  Default  28.0000  28.0000\n",
       "8728     nasa   541.000000   541.000000  Default  27.0000  27.0000\n",
       "7410   launch   439.000000   439.000000  Default  26.0000  26.0000\n",
       "...       ...          ...          ...      ...      ...      ...\n",
       "11342     say   147.849523   856.612334   Topic4  -5.9544  -0.0709\n",
       "13641     use   138.710285   665.221860   Topic4  -6.0182   0.1181\n",
       "2343     case   126.347056   410.273021   Topic4  -6.1115   0.5081\n",
       "9624   people   138.415816  1171.414695   Topic4  -6.0203  -0.4498\n",
       "13090    time   122.687861   969.504932   Topic4  -6.1409  -0.3813\n",
       "\n",
       "[303 rows x 6 columns], token_table=       Topic      Freq    Term\n",
       "term                          \n",
       "36         1  0.017493      10\n",
       "36         2  0.390679      10\n",
       "36         3  0.209917      10\n",
       "36         4  0.384848      10\n",
       "694        1  0.838362  accept\n",
       "...      ...       ...     ...\n",
       "14450      2  0.339679    year\n",
       "14450      3  0.301159    year\n",
       "14450      4  0.268475    year\n",
       "14483      1  0.023594   yeast\n",
       "14483      4  0.979146   yeast\n",
       "\n",
       "[530 rows x 3 columns], R=30, lambda_step=0.01, plot_opts={'xlab': 'PC1', 'ylab': 'PC2'}, topic_order=[3, 1, 2, 4])"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pyLDAvis.sklearn\n",
    "\n",
    "pyLDAvis.enable_notebook()\n",
    "dash = pyLDAvis.sklearn.prepare(lda, bow, vectorizer)\n",
    "dash"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fd8c00e-caa8-42c0-a3af-7060aeb5237c",
   "metadata": {},
   "source": [
    "---\n",
    "#### Exercise\n",
    "Play with the number of topics and see if you understand what happens!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f602a53d-16bc-48e9-b70b-6cee7aaedcc4",
   "metadata": {},
   "source": [
    "## Word vectors\n",
    "\n",
    "In order to capture the 'meaning' or 'context' of a word, people often use word vectors. These are an $N$-dimensional representation of a word in an abstract space, in which words with a similar meaning are supposed to be near each other.\n",
    "\n",
    "The `nlp` object defined above comes with 96-dimensional word vectors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "bb9afeb1-a6e0-4b36-be1d-e36108fe4d55",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(96,)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mango = nlp('mango')\n",
    "mango.vector.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31566c41-bf0e-43ea-b8af-719d37b8401b",
   "metadata": {},
   "source": [
    "The numbers by themselves hardly mean anything, but proximity in this high-dimensional space does. \n",
    "\n",
    "---\n",
    "#### Exercise\n",
    "\n",
    "Get the vectors for \"mango\", \"strawberry\" and \"brick\" and verify that the fruits are indeed the closest pair.\n",
    "\n",
    "Use the `similarity` method of tokens as well, to get a measure of all pairwise similarities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4931277-7ebd-416b-b066-ecb7e212f3fc",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "239ac307-f25e-4da3-8be7-20c0087c2078",
   "metadata": {},
   "outputs": [],
   "source": [
    "# If you want solutions, uncomment and run the next two lines.\n",
    "# to_include = os.path.join('solutions', 'vector_dist.py')\n",
    "# %load $to_include"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36442989-e09a-4eec-b081-ef39fe53a676",
   "metadata": {},
   "source": [
    "With the larger language model, the word vectors are 300-dimensional.\n",
    "\n",
    "Word vectors can be incredibly powerful. You can use the pre-trained models in spaCy, or you can train your own, with e.g. `word2vec`, `GenSim` or `FastText`. It can also be useful to take existing word embeddings and \"re-train\" them, which is supposed to make the existing embeddings more relevant for your domain of application, while you can still use the versatile pre-trained models, which are typically trained on massive amounts of data (more than you're likely to have at hand).\n",
    "\n",
    "With the vectors representing words, you can also do machine learning. In that case you do not need the bag-of-words methods any longer, which is nice for several reasons, e.g.:\n",
    "- Bag-of-words methods are unaware of the contexts of words\n",
    "- Word vectors are less sensitive to the use of synonyms and are more versatile in large diverse corpora of text\n",
    "- Word vectors trivially combine into document vectors (through averaging), allowing you to treat the documents in much the same way\n",
    "\n",
    "When you create an `nlp()` object ot of a document, using one of the \"larger\" language models (see above), it is possible to assess the similarity of two documents using the `.similarity()` method. This uses the `.vector` attribute and calculates the similarity based on the [cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity), a well-known distance metric that is insensitive to the length (L2-norm) of the vector, but only to its \"direction\".\n",
    "\n",
    "SpaCy has lots of functionalitie with word vectors, transformers and other sophisticated tooling see e.g. [their documentation](https://spacy.io/). The key difference between word/document-vectors and contextual language models such as transformers is that word vectors model lexical types, rather than tokens. If you have a list of terms with no context around them, a transformer model like BERT can’t really help you. BERT is designed to understand language in context, which isn’t what you have. A word vectors table will be a much better fit for your task. However, if you do have words in context – whole sentences or paragraphs of running text -- word vectors will only provide a very rough approximation of what the text is about.\n",
    "\n",
    "Transformer models are usually trained with PyTorch, and is greatly helped by the use of GPUs. These are beyond the scope of this workshop.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3af82dfa-2cd4-4972-98e8-847c8cc642b5",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc8a2b95-4671-41fe-9d46-fbd36b1f3786",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}