notebooks/Python_03_Data_Collection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python Immersion Course\n",
    "\n",
    "### Data Collection with Python - Part 3\n",
    "\n",
    "### Joe Blankenship - Just some dude"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you have a good grasp of Python's basic fucntionality, you can interact with a number of data sources. This section will focus on the basics of extracting, tranforming, and loading data formats into dataframes for analysis. Data manipulation inside of the dataframes will be saved for Part 4."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "## Basics\n",
    "\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are several key terms and concepts to be aware of when collecting data for analysis and visualization:\n",
    "\n",
    "* Primary Sources - collected directly from the original source\n",
    "* Secondary Sources - collected by an intermediary\n",
    "* Explicitly Spatial - for data where location patterns are directly analyzed\n",
    "* Implicitly Spatial - for data that represents location, but is not directly analyzed spatially\n",
    "* Individual Data - data that represents an single unit of something\n",
    "* Aggregate Data - data that represents a sum of single units of something\n",
    "* Discrete Data - a data type representing a count of something and values are finite\n",
    "* Continuous Data - a data type representing an interval/measure of something and values are potential infinite\n",
    "* Qualitative Data - attributes, labels, non-numerical entries\n",
    "* Quantitative Data - numerical measurements, counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "## Web Scraping\n",
    "\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Urllib and IO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first scraper we'll build will use core Python libraries to:\n",
    "\n",
    "* Go to a HTTP website\n",
    "* Gather the source code\n",
    "* Print the output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here we'll import urllib, io, and pprint modules to obtain out data\n",
    "\n",
    "from urllib.request import Request, urlopen\n",
    "from io import TextIOWrapper\n",
    "from pprint import pprint\n",
    "\n",
    "# Declare the URL\n",
    "url = 'https://en.wikipedia.org/wiki/Department_of_Geography,_University_of_Kentucky'\n",
    "\n",
    "# Open the URL\n",
    "page = Request(url)\n",
    "page_content = urlopen(page)\n",
    "# page_content.read()\n",
    "\n",
    "# Buffer our text stream from the website\n",
    "page_data = TextIOWrapper(page_content)\n",
    "\n",
    "# pprint out our data\n",
    "for row in page_data:\n",
    "    pprint(row)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Requests and BeautifulSoup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, we may want something a bit more elegant. This is where `requests` and `beautifulsoup` comes in to help us out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import requests and beautifulsoup\n",
    "# Import pandas, we'll use that at the end\n",
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import pandas as pd\n",
    "\n",
    "# we are going to scrape crime data from the UK crime http://www.uky.edu/crimelog/\n",
    "# substitute variables to fill in REST query criteria\n",
    "start_month, start_day, start_year = 1, 1, 2018\n",
    "end_month, end_day, end_year = 10, 4, 2018\n",
    "crime_data_raw = requests.get('http://www.uky.edu/crimelog/log?field_log_category_value=All' +\n",
    "                              '&field_log_report_value%5Bmin%5D%5Bmonth%5D=' + str(start_month) +\n",
    "                              '&field_log_report_value%5Bmin%5D%5Bday%5D=' + str(start_day) +\n",
    "                              '&field_log_report_value%5Bmin%5D%5Byear%5D=' + str(start_year) +\n",
    "                              '&field_log_report_value%5Bmax%5D%5Bmonth%5D=' + str(end_month) +\n",
    "                              '&field_log_report_value%5Bmax%5D%5Bday%5D=' + str(end_day) +\n",
    "                              '&field_log_report_value%5Bmax%5D%5Byear%5D=' + str(end_year)\n",
    "                             )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a soup object \n",
    "crime_bs_proc = BeautifulSoup((crime_data_raw.text), \"html5lib\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a filter for our soup object to pull out the table\n",
    "crime_data_table = crime_bs_proc.find('table', {'class': 'views-table cols-8'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# find the table header in the data\n",
    "crime_data_header = crime_data_table.find('thead')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# find all the table headers\n",
    "crime_data_heads = crime_data_header.find_all('th')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create an empty list for the header\n",
    "header = []\n",
    "\n",
    "# iterate through the header element to get text\n",
    "for col in crime_data_heads:\n",
    "    cols = col.find_all('a')\n",
    "    cols = [ele.text.strip() for ele in cols]\n",
    "    header.append([ele for ele in cols if ele])\n",
    "\n",
    "# flatten the list to a single list\n",
    "header = [item for sublist in header for item in sublist]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# find the table rows in the data\n",
    "crime_data_body = crime_data_table.find('tbody')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# find all table rows\n",
    "crime_data_rows = crime_data_body.find_all('tr')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create an empty list for the rows of data\n",
    "data = []\n",
    "\n",
    "# iterate through the header element to get the rows\n",
    "for row in crime_data_rows:\n",
    "    cols = row.find_all('td')\n",
    "    cols = [ele.text.strip() for ele in cols]\n",
    "    data.append([ele for ele in cols if ele])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a dataframe with our data using our header list\n",
    "uk_crime_data = pd.DataFrame(data, columns=header)\n",
    "uk_crime_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is also the `scrapy` library in Python for more complex scraping projects."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "## APIs\n",
    "\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "APIs often have 'wrappers' in Python that you can use to interface with the underlying data.\n",
    "\n",
    "Here we will use the data.world API to import some data\n",
    "\n",
    "  * docs at https://github.com/datadotworld/data.world-py\n",
    "\n",
    "Prior to this, you should load your API credentials from data.world into your active virtual env (in the terminal)\n",
    "\n",
    "`export DW_AUTH_TOKEN=<YOUR_TOKEN>`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import our API library\n",
    "\n",
    "import datadotworld as dw"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load our data sets from the API using a known user data collection\n",
    "\n",
    "lex_business_health = dw.load_dataset('inform8n/most-recent-lexington-ky-health-department-inspection-scores')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# list the dataframes available in the data set collection\n",
    "\n",
    "lex_business_health.dataframes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load a data set into a dataframe from the data collection\n",
    "\n",
    "food_scores = lex_business_health.dataframes.get('most_recent_food_scores')\n",
    "food_scores.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "## Flat files\n",
    "\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are several ways to import flat files for analysis.\n",
    "\n",
    "The simplest method is to use `pandas` as it supports several well known formats\n",
    "\n",
    "However, for each of the following files, there are core and 3rd party libraries you can also use to load your data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import pandas\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CSV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read csv with pandas\n",
    "\n",
    "census_ky_csv = pd.read_csv('data/census_2010_ky.csv')\n",
    "census_ky_csv.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can use the csv library to import/manipulate csv files\n",
    "\n",
    "import csv\n",
    "\n",
    "with open('data/census_2010_ky.csv') as census_ky_csv_2:\n",
    "    reader = csv.DictReader(census_ky_csv_2)  # You can also use csv.reader\n",
    "    for row in reader:\n",
    "        print(row)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Excel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read excel in xls format with pandas\n",
    "\n",
    "census_ky_xls = pd.read_excel('data/census_2010_ky.xls')\n",
    "census_ky_xls.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read excel in xlsx format with pandas\n",
    "\n",
    "census_ky_xlsx = pd.read_excel('data/census_2010_ky.xlsx')\n",
    "census_ky_xlsx.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### JSON"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read json with pandas\n",
    "\n",
    "census_ky_json = pd.read_json('data/census_2010_ky.json')\n",
    "census_ky_json.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can also use the core json library to import json data\n",
    "\n",
    "import json\n",
    "\n",
    "with open('data/census_2010_ky.json') as census_ky_json_2:\n",
    "    reader = json.load(census_ky_json_2)\n",
    "    for row in reader:\n",
    "        print(row)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### XML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read xml into dataframe using core xml library\n",
    "\n",
    "import xml.etree.ElementTree as et\n",
    "\n",
    "root = et.parse('data/census_2010_ky.xml')  # use element tree to parse the xml data\n",
    "rows = root.findall('row')  # find all row elements in xml\n",
    "# iterate and select elements in row\n",
    "data = [[row.find('geoid').text, row.find('label').text, row.find('totpop').text] for row in rows]\n",
    "# push above data into pandas dataframe\n",
    "census_ky_xml = pd.DataFrame(data, columns=['geoid', 'label', 'totpop'])\n",
    "census_ky_xml.head(2)  # check your dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read xml into dataframe using lxml library\n",
    "\n",
    "from lxml import objectify\n",
    "# use objectify to parse xml data\n",
    "xml_data = objectify.parse(open('data/census_2010_ky.xml'))\n",
    "root = xml_data.getroot()  # select root tree in xml data\n",
    "# create an empty list as destination for our data\n",
    "data = []\n",
    "# for the row data in our root data\n",
    "for elt in root.row:\n",
    "    # create and empty dictionary\n",
    "    el_data = {}\n",
    "    # for each child element in row, extract the tag with data and append the list 'data'\n",
    "    for child in elt.getchildren():\n",
    "        el_data[child.tag] = child.pyval\n",
    "    data.append(el_data)\n",
    "# create a pandas dataframe for data list\n",
    "census_ky_xml_2 = pd.DataFrame(data)\n",
    "# check your dataframe\n",
    "census_ky_xml_2.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Binary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import binary with pandas\n",
    "\n",
    "census_ky_binary = pd.read_pickle('data/census_2010_ky')\n",
    "census_ky_binary.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Know that `pandas` also supports many other file formats such as `hdf5`, `stata`, `SQL`, `html`, `sas`, and even data from your `clipboard`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PDF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read pdf files... may god have mercy on your soul.\n",
    "\n",
    "import pdfx\n",
    "import pprint\n",
    "# after pdfx import, create a PDFx object for our PDF\n",
    "census_ky_pdf = pdfx.PDFx('data/census_2010_ky.pdf')\n",
    "# extract metadata for PDF\n",
    "census_ky_pdf_metadata = census_ky_pdf.get_metadata()\n",
    "# extract references and place them in a dictionary, hyperlink extraction also possible\n",
    "census_ky_pdf_refs = census_ky_pdf.get_references_as_dict()\n",
    "# extract the body of text from PDF\n",
    "census_ky_pdf_text = census_ky_pdf.reader.get_text()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a great starting point for extracting text, metadata, and references (with hyperlinks) from PDFs (very useful for social scientists). However, there are a few ways to extract tabular data from PDFs and none are very easy. The techniques through which the tabular text can be restructured for a dataframe will be covered in Part 4."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DOCX"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read docs files...\n",
    "\n",
    "import docx\n",
    "# create a document object for our docx file\n",
    "doc = docx.Document('data/census_2010_ky.docx')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check the number of paragraphs\n",
    "len(doc.paragraphs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pull the text out of the first paragraph\n",
    "doc.paragraphs[0].text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# extract and output contents of tables\n",
    "table = doc.tables[0]\n",
    "# create empty list for preprocessing\n",
    "data = []\n",
    "# for each row in the table\n",
    "# for each cell in row\n",
    "# add the cell to the list 'data'\n",
    "for row in table.rows:\n",
    "    for cell in row.cells:\n",
    "        data.append(cell.text)\n",
    "# create a function to split our long list into n size chunks equal to # of headers\n",
    "def sublist_gen(l, n):\n",
    "    for i in range(0, len(l), n):\n",
    "        yield l[i:i + n]\n",
    "# use our function to create list of list\n",
    "# first list == headers\n",
    "sub_data = list(sublist_gen(data, 6))\n",
    "# extract our headers\n",
    "headers = sub_data.pop(0)\n",
    "# create a dataframe from lists\n",
    "docx_table_dataframe = pd.DataFrame(sub_data, columns=headers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check your dataframe\n",
    "docx_table_dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "## Databases\n",
    "\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SQL Databases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create and write data from a sqlite3 database\n",
    "# Python3 comes with sqlite and can be a power tool for initial data exploration\n",
    "\n",
    "import sqlite3\n",
    "import csv\n",
    "\n",
    "# create the database \n",
    "connection = sqlite3.connect('data/census_2010_ky.db')\n",
    "# create a cursor to interact with the database\n",
    "cursor = connection.cursor()\n",
    "# use the cursor to execute SQL commands on the database. here we're creating a table.\n",
    "cursor.execute('CREATE TABLE kycensus2010 (id, geoid, label, totpop, medage, sindads, sinmoms);')\n",
    "# with the CSV we intend to use to append out table, iterate through the CSV rows\n",
    "with open('data/census_2010_ky.csv', 'r') as data:\n",
    "    ky_data = csv.DictReader(data)\n",
    "    to_database = [(i['id'],\n",
    "                    i['geoid'],\n",
    "                    i['label'],\n",
    "                    i['totpop'],\n",
    "                    i['medage'],\n",
    "                    i['sindads'],\n",
    "                    i['sinmoms'])\n",
    "                   for i in ky_data\n",
    "                  ]\n",
    "# take the output of the above iterations and place the CSV row data into the database table\n",
    "cursor.executemany('INSERT INTO kycensus2010 (id, geoid, label, totpop, medage, sindads, sinmoms) VALUES (?, ?, ?, ?, ?, ?, ?);', to_database)\n",
    "# commit your changes to the database and tables\n",
    "connection.commit()\n",
    "# close your connection to the database\n",
    "connection.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read data from sqlite3 database\n",
    "\n",
    "connection = sqlite3.connect('data/census_2010_ky.db')\n",
    "# use pandas to read a table from the database connection to create a dataframe\n",
    "census_2010_ky_sql = pd.read_sql_query(\"SELECT * FROM kycensus2010\", connection)\n",
    "# close the database connection once you're done creating your pandas dataframe\n",
    "connection.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# test your dataframe\n",
    "census_2010_ky_sql.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use sqlalchemy to manipulate sql databases\n",
    "# if you plan to move your sqlite3 database to another sql database, use this approach\n",
    "\n",
    "import sqlalchemy as sqla\n",
    "# create a sqlachemy database connection (or engine)\n",
    "census_2010_ky_database = sqla.create_engine('sqlite:///data/census_2010_ky.db')\n",
    "# read a table from the database usng the above engine\n",
    "census_2010_ky_sql_2 = pd.read_sql('SELECT * FROM kycensus2010', census_2010_ky_database)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# test your dataframe\n",
    "census_2010_ky_sql_2.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NoSQL Databases"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Honestly the easiest way to import `CSV`, `JSON`, and other flat files is to use the `mongoimport` CLI tool.\n",
    "\n",
    "At the terminal, type a `mongoimport` command like the one below to import your data:\n",
    "\n",
    "`mongoimport -d census2010 -c kycensus2010 --type csv --file data/census_2010_ky.csv --headerline`\n",
    "\n",
    "`-d` is the new database mongo will create to store your collections<br>\n",
    "`-c` is the new collection for your flat file import<br>\n",
    "`--type` denotes the type of file (e.g., CSV, JSON)<br>\n",
    "`--file` is the relative path to your flat file<br>\n",
    "`--headerline` denotes that the flat file has a headerline (specific to CSV)<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Connect to and access a mongo database\n",
    "\n",
    "from pymongo import MongoClient\n",
    "# create a MongoDB connection called a client\n",
    "client = MongoClient()\n",
    "# create a database named census2010\n",
    "database = client.census2010\n",
    "# create a mongo collection named kycensus2010\n",
    "kycensus2010 = database.kycensus2010\n",
    "# create a pandas dataframe for a list of the kycensus2010 record search outputs\n",
    "census_2010_ky_mongo = pd.DataFrame(list(kycensus2010.find()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check your dataframe\n",
    "census_2010_ky_mongo.head(2)\n",
    "\n",
    "# notice how Mongo has assigned each record a unique _id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Graph Databases"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given this is an intro course, I'll save graph databases until a future social network analysis in Python class. Just know that they exist and have some interesting properties as compared to SQL and NoSQL databases.\n",
    "\n",
    "There are links below for `Neo4j` and the `py2neo` library."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<hr>\n",
    "\n",
    "## Additional Materials (for future versions)\n",
    "\n",
    "<hr>\n",
    "\n",
    "* [Newspaper](https://github.com/codelucas/newspaper/)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<hr>\n",
    "\n",
    "## Resources\n",
    "\n",
    "<hr>\n",
    "\n",
    "**Note:** A lot of the open-source materials are provided by people who develop those materials for a living. So please consider sending them a thank you and if you can, a few buck to support their efforts. Thanks! :)    \n",
    "\n",
    "* [Pandas](https://pandas.pydata.org/pandas-docs/stable/)\n",
    "* [urllib](https://docs.python.org/3/library/urllib.html)\n",
    "* [io](https://docs.python.org/3/library/io.html)\n",
    "* [pprint](https://docs.python.org/3/library/pprint.html)\n",
    "* [requests](http://docs.python-requests.org/en/latest/)\n",
    "* [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/)\n",
    "* [datadotworld](https://github.com/datadotworld/data.world-py)\n",
    "* [csv](https://docs.python.org/3/library/csv.html)\n",
    "* [json](https://docs.python.org/3/library/json.html)\n",
    "* [xml](https://docs.python.org/3/library/xml.html)\n",
    "* [lxml](https://lxml.de/)\n",
    "* [pdfx](https://github.com/metachris/pdfx)\n",
    "* [python-docx](https://python-docx.readthedocs.io/en/latest/)\n",
    "* [sqlite](https://docs.python.org/3/library/sqlite3.html)\n",
    "* [sqlalchemy](https://docs.sqlalchemy.org/en/latest/)\n",
    "* [pymongo](https://api.mongodb.com/python/current/)\n",
    "* [py2neo](https://py2neo.org/v4/)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}