week01-introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sampling and statistics: Introduction\n",
    "\n",
    "### Key concepts\n",
    "* types of data\n",
    "* types of error\n",
    "* mean\n",
    "* variance\n",
    "* normal distribution \n",
    "* skewed distribution\n",
    "* standard deviation\n",
    "* standard error\n",
    "\n",
    "### Reading\n",
    "\n",
    "Glover, Jenkins and Doney – Modeling Methods (GJD)\n",
    "\n",
    "* Sections 2.1-2 - Measurement Theory\n",
    "\n",
    "Statistical Thinking for the 21st Century (ST21): \n",
    "\n",
    "* Chapter 1: Introduction\n",
    "* Chapter 2: Working with data\n",
    "* Chapter 4: Summarizing data\n",
    "\n",
    "### Focus questions\n",
    "\n",
    "* What is the difference between accuracy and precision? \n",
    "* What are sources of error in the data you work with?\n",
    "* What is the difference between the standard deviation and standard error?\n",
    "\n",
    "## Types of Data \n",
    "* __Discrete__ - represented by an integer (whole number)- counts, presence/absence <br>\n",
    "* __Continuous__ - represented by real<sup>1</sup> numbers (e.g. temperature, wave speed, length of an organism, concentration)  <br>\n",
    "* __Categorical__ - examples: species, sediment type, hair color, site # <br>\n",
    "* __Metadata__ - \"data describing data\"\n",
    "\n",
    "<sup>1</sup> *Actually, vectors can be represented as complex numbers with an imaginary part*\n",
    "\n",
    "## Types of Measurements\n",
    "__Nominal__ - categories of equal rank<br>\n",
    "* Species Description (phytoplankton types: diatoms, coccolithophores, etc.) \n",
    "\n",
    "\n",
    "__Ordinal__ - Categories have a logically defined rank. Steps arent equal in size or quantifiable<br>\n",
    "\n",
    "* How sediment grains are categorized: angularity and sphericity\n",
    "* Hurricane scale: ranking is not equivalent to strength\n",
    "* Beaufort Wind Scale: Mariner estimate of wind from wave climate, logically defined, but not quantifiable\n",
    "\n",
    "__Scale__: __interval__ and __ratio__\n",
    "\t\n",
    "* __interval scale__: constant succesive intervals, but the reference point is arbitrary e.g. temperature scale <br>\n",
    "* __ratio scale__: natural zero point (ex: length, mass)\n",
    "\n",
    "## Types of Error\n",
    "* __Systematic error__  - systematically repeatable biased, i.e. some pattern in how they vary <br>\n",
    "* __Random error__ - Impercise, but unbiased. Noise.\n",
    "\n",
    "![error](images/error_type.png)\n",
    "\n",
    "__Measurement precision__<br>\n",
    "You would not record a value from a ruler to 3.7567453 cm if your ruler only has mm hash marks\n",
    "\n",
    "Rounding introduces error into your calculations, so in general it is better to use all of the digits that you have and round off to the significant digit when reporting the value\n",
    "\n",
    "__Drift__ - a systematic error that changes over time\n",
    "\n",
    "![Nitrate Profile](images/nitrate_profile_smooth.png)\n",
    "\n",
    "Johnson, K. S., and L. J. Coletti, 2002: In situ ultraviolet spectrophotometry for high resolution and long-term monitoring of nitrate, bromide and bisulfide in the ocean. _Deep Sea Res. Part I Oceanogr. Res. Pap._, 49, 1291–1305, doi:10.1016/S0967-0637(02)00020-1.\n",
    "\n",
    "Bottle data (squares): chemically derived values of nitrate (low resolution, high precision)<br>\n",
    "ISUS Nitrate Sensor (High resolution, systematic error (underreported values compared to bottle casts) , instrument noise (small scale random error))\n",
    "\n",
    "Smoothing reduces noise, but there is a trade-off: lowers resolution<br>\n",
    "In this example, the bias is reduced by correcting for the temperature dependence of the sensor.\n",
    "\n",
    "### Sample distributions, probability density \n",
    "\n",
    "![wave_height_density_normal](images/waveheight_04_density_normal.png)\n",
    "\n",
    "Bars: Example distribution of wave heights during an extreme wave event (N = 200). Synthetic (computer-generated) data based on a similar data set in Emery and Thomson, Section 3.8.4. The sample mean $\\bar{x}$ is 5.08 m and the sample standard deviation $s$ is 1.18 m.\n",
    "\n",
    "Wave height categories are divided in to bins of with $\\Delta x$\n",
    "\n",
    "Probability density = fraction/$\\Delta x$ <br>\n",
    "\n",
    "Orange line: Normal distribution (hypothetical), allows the use of mathematical theories to use statistics. The shape of the normal distribution is defined by two parameters - the mean and standard deviation. This theoretical curve has the same mean and standard deviation as the sample distribution.  <br>\n",
    "\n",
    "Integral of area under the curve = 1 (all probability falls under the curve) for both the sample distribution and the theoretical normal distribution.\n",
    "\n",
    "### Cumulative distribution\n",
    "\n",
    "![wave_height_cumulative_normal](images/waveheight_06_cumulative_normal.png)\n",
    "\n",
    "Same samples and theoretical probability, expressed as cumulative probability. This is the probability that the data fall with a given interval or less.\n",
    "\n",
    "## Descriptive statistics\n",
    "\n",
    "### Mean ###\n",
    "\n",
    "With a finite number of $N$ samples, the __true mean__ of a population, $\\mu$ (\"mu\"), can be _estimated_ by the sample mean $\\bar{x}$,\n",
    "\n",
    "$$ \\bar{x} = \\frac{1}{N}\\sum_{i=1}^{N}{x_i} = \\frac{1}{N}(x_1+x_2+x_3+....+x_N) $$\n",
    "\n",
    "\n",
    "### Variance ###\n",
    "__Variance__ describes the spread of the data. The sample variance is equal to the sample __standard deviation__ squared ($s^2$). The sample variance is an estimate of the true variance ($\\sigma^2$)\n",
    "\n",
    "$$s^2 = \\frac{1}{N-1}\\sum_{i=1}^N(x_i-\\bar{x})^2,$$\n",
    "\n",
    "where $N-1$ is the __degrees of freedom__. Degrees of freedom are the number of independent pieces of information. The variance of one sample ($N=1$) is essentially meaningless because it is always equal to zero, no matter what the value of the sample is. In this case, there are no degrees of freedom. Calculating the variance/standard deviation with $N-1$ in the denominator is called the __unbiased__ estimate of the true standard deviation $\\sigma$.\n",
    "\n",
    "The variance is __positive definite__ because it is the sum of squared values and therefore cannot be negative.\n",
    "\n",
    "### Standard Error ###\n",
    "\n",
    "The __standard error__ describes how well the sample mean describes the true mean\n",
    "\n",
    "$$ SE = \\frac{s}{\\sqrt{N}}. $$\n",
    "\n",
    "The standard error does not describe the spread of the data, it describes the how well the sample mean, $\\bar{x}$, represents the true mean, $\\mu$. The standard error can thought of as an estimate of the standard deviation of $(\\bar{x} - \\mu)$, after many repeated experiments with $N$ samples. \n",
    "\n",
    "#### Example revisited: repreated trials\n",
    "\n",
    "![wave_height_trial_dist_normal](images/waveheight_08_trial_dist_normal.png)\n",
    "\n",
    "Results from 100 repeated trials during same storm (N = 200 each trial). The $x$-axis is the same as that shown for the sample distribution above, to emphasize how the distribution of means clusters more tightly together. We can think of these repeated trials as different instruments measuring different waves in the same storm. Note that this is easy to do with a computer-generated example dataset, but not so easy to do in practices.\n",
    "\n",
    "Blue bars:\n",
    "Probability density of the mean of each trial (mean of all sample means = 4.99m)\n",
    "\n",
    "Orange curve: \n",
    "Normal distribution\n",
    "Mean = Mean of trial 1 (example distribution shown above).\n",
    "Standard deviation = Standard error of trial 1, SE = $s/\\sqrt{N}$\n",
    "\n",
    "The standard error from 1 set of samples, gives an estimate of standard deviation (spread) of the means."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Programming tutorial: Unix shell\n",
    "\n",
    "See notes on the [Software Carpentry](https://software-carpentry.org) website.\n",
    "\n",
    "[http://swcarpentry.github.io/shell-novice/](http://swcarpentry.github.io/shell-novice/)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}