Anomaly Detection with LSTM and Cross Correlation Input

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "hide_input": false,
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\" style=\"margin-top: 1em;\"><ul class=\"toc-item\"><li><span><a href=\"#Imports\" data-toc-modified-id=\"Imports-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Imports</a></span></li><li><span><a href=\"#Data-Prep\" data-toc-modified-id=\"Data-Prep-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Data Prep</a></span><ul class=\"toc-item\"><li><span><a href=\"#Workflow\" data-toc-modified-id=\"Workflow-2.1\"><span class=\"toc-item-num\">2.1&nbsp;&nbsp;</span>Workflow</a></span></li><li><span><a href=\"#Build-Sqlite-Data-Base\" data-toc-modified-id=\"Build-Sqlite-Data-Base-2.2\"><span class=\"toc-item-num\">2.2&nbsp;&nbsp;</span>Build Sqlite Data Base</a></span></li><li><span><a href=\"#Clean-Data\" data-toc-modified-id=\"Clean-Data-2.3\"><span class=\"toc-item-num\">2.3&nbsp;&nbsp;</span>Clean Data</a></span></li><li><span><a href=\"#Create-SVID-Columns\" data-toc-modified-id=\"Create-SVID-Columns-2.4\"><span class=\"toc-item-num\">2.4&nbsp;&nbsp;</span>Create SVID Columns</a></span></li><li><span><a href=\"#Normalize-Signal-Data\" data-toc-modified-id=\"Normalize-Signal-Data-2.5\"><span class=\"toc-item-num\">2.5&nbsp;&nbsp;</span>Normalize Signal Data</a></span></li><li><span><a href=\"#Generate-Files-with-Different-Sampling-Rates\" data-toc-modified-id=\"Generate-Files-with-Different-Sampling-Rates-2.6\"><span class=\"toc-item-num\">2.6&nbsp;&nbsp;</span>Generate Files with Different Sampling Rates</a></span></li><li><span><a href=\"#Calculate-Cross-Correlation-Matrix-(CCM)\" data-toc-modified-id=\"Calculate-Cross-Correlation-Matrix-(CCM)-2.7\"><span class=\"toc-item-num\">2.7&nbsp;&nbsp;</span>Calculate Cross Correlation Matrix (CCM)</a></span></li><li><span><a href=\"#Shape-SigCCM-Data-for-LSTM-Input\" data-toc-modified-id=\"Shape-SigCCM-Data-for-LSTM-Input-2.8\"><span class=\"toc-item-num\">2.8&nbsp;&nbsp;</span>Shape SigCCM Data for LSTM Input</a></span></li><li><span><a href=\"#Save-Files\" data-toc-modified-id=\"Save-Files-2.9\"><span class=\"toc-item-num\">2.9&nbsp;&nbsp;</span>Save Files</a></span><ul class=\"toc-item\"><li><span><a href=\"#Save-Pickled-Files\" data-toc-modified-id=\"Save-Pickled-Files-2.9.1\"><span class=\"toc-item-num\">2.9.1&nbsp;&nbsp;</span>Save Pickled Files</a></span></li><li><span><a href=\"#Save-to-Dill-File\" data-toc-modified-id=\"Save-to-Dill-File-2.9.2\"><span class=\"toc-item-num\">2.9.2&nbsp;&nbsp;</span>Save to Dill File</a></span></li><li><span><a href=\"#Save-to-HDF5-File\" data-toc-modified-id=\"Save-to-HDF5-File-2.9.3\"><span class=\"toc-item-num\">2.9.3&nbsp;&nbsp;</span>Save to HDF5 File</a></span></li><li><span><a href=\"#Save-to-PyArrow-Parquet-File\" data-toc-modified-id=\"Save-to-PyArrow-Parquet-File-2.9.4\"><span class=\"toc-item-num\">2.9.4&nbsp;&nbsp;</span>Save to PyArrow-Parquet File</a></span></li></ul></li><li><span><a href=\"#Load-Files\" data-toc-modified-id=\"Load-Files-2.10\"><span class=\"toc-item-num\">2.10&nbsp;&nbsp;</span>Load Files</a></span></li></ul></li><li><span><a href=\"#Data-Generator\" data-toc-modified-id=\"Data-Generator-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Data Generator</a></span><ul class=\"toc-item\"><li><span><a href=\"#Generator-in-a-Python-Class\" data-toc-modified-id=\"Generator-in-a-Python-Class-3.1\"><span class=\"toc-item-num\">3.1&nbsp;&nbsp;</span>Generator in a Python Class</a></span></li><li><span><a href=\"#Generator-Based-on-Numpy-IO\" data-toc-modified-id=\"Generator-Based-on-Numpy-IO-3.2\"><span class=\"toc-item-num\">3.2&nbsp;&nbsp;</span>Generator Based on Numpy IO</a></span></li><li><span><a href=\"#Generator-Based-on-HD5-IO\" data-toc-modified-id=\"Generator-Based-on-HD5-IO-3.3\"><span class=\"toc-item-num\">3.3&nbsp;&nbsp;</span>Generator Based on HD5 IO</a></span></li></ul></li><li><span><a href=\"#LSTM\" data-toc-modified-id=\"LSTM-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>LSTM</a></span><ul class=\"toc-item\"><li><span><a href=\"#Build-the-LSTM-Model\" data-toc-modified-id=\"Build-the-LSTM-Model-4.1\"><span class=\"toc-item-num\">4.1&nbsp;&nbsp;</span>Build the LSTM Model</a></span><ul class=\"toc-item\"><li><span><a href=\"#Step-12,-Sampled-Data,-100-Epochs\" data-toc-modified-id=\"Step-12,-Sampled-Data,-100-Epochs-4.1.1\"><span class=\"toc-item-num\">4.1.1&nbsp;&nbsp;</span>Step 12, Sampled Data, 100 Epochs</a></span><ul class=\"toc-item\"><li><span><a href=\"#Make-Predictions-and-Plot-ROC-AUC-Metric\" data-toc-modified-id=\"Make-Predictions-and-Plot-ROC-AUC-Metric-4.1.1.1\"><span class=\"toc-item-num\">4.1.1.1&nbsp;&nbsp;</span>Make Predictions and Plot ROC-AUC Metric</a></span></li></ul></li><li><span><a href=\"#5000-Records-of-Attack-Data,-100-Epochs\" data-toc-modified-id=\"5000-Records-of-Attack-Data,-100-Epochs-4.1.2\"><span class=\"toc-item-num\">4.1.2&nbsp;&nbsp;</span>5000 Records of Attack Data, 100 Epochs</a></span><ul class=\"toc-item\"><li><span><a href=\"#Make-Predictions-and-Plot-ROC-AUC-Metric\" data-toc-modified-id=\"Make-Predictions-and-Plot-ROC-AUC-Metric-4.1.2.1\"><span class=\"toc-item-num\">4.1.2.1&nbsp;&nbsp;</span>Make Predictions and Plot ROC-AUC Metric</a></span></li></ul></li></ul></li><li><span><a href=\"#Stacked-LSTMs\" data-toc-modified-id=\"Stacked-LSTMs-4.2\"><span class=\"toc-item-num\">4.2&nbsp;&nbsp;</span>Stacked LSTMs</a></span><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><span><a href=\"#Make-Predictions-and-Plot-ROC-AUC-Metric\" data-toc-modified-id=\"Make-Predictions-and-Plot-ROC-AUC-Metric-4.2.0.1\"><span class=\"toc-item-num\">4.2.0.1&nbsp;&nbsp;</span>Make Predictions and Plot ROC-AUC Metric</a></span></li></ul></li></ul></li></ul></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Imports  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "import pandas as pd\n",
    "import pickle as pkl\n",
    "import pyarrow as pa\n",
    "import pyarrow.parquet as pq\n",
    "import collections\n",
    "import itertools\n",
    "import keras\n",
    "import math\n",
    "import h5py\n",
    "import os\n",
    "\n",
    "# os\n",
    "from os import mkdir\n",
    "\n",
    "# pandas\n",
    "# from pandas import read_csv \n",
    "# from pandas import datetime \n",
    "from pandas.plotting import autocorrelation_plot\n",
    "\n",
    "# scipy\n",
    "from scipy import sparse\n",
    "from scipy import signal\n",
    "from scipy.signal import correlate\n",
    "from scipy.signal import correlate2d\n",
    "from scipy.sparse import coo_matrix, vstack\n",
    "\n",
    "# statsmodels\n",
    "# from statsmodels.tsa.arima_model import ARIMA \n",
    "\n",
    "# sci-kit learn\n",
    "from sklearn.metrics import auc\n",
    "from sklearn.metrics import roc_curve\n",
    "from sklearn.metrics import mean_squared_error\n",
    "from sklearn.datasets import make_classification\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# matplotlib\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "# sqlite\n",
    "# from sqlalchemy import create_engine\n",
    "# from flask.ext.sqlalchemy import SQLAlchemy\n",
    "# from flask_sqlalchemy import SQLAlchemy\n",
    "# from sqlalchemy import create_engine\n",
    "\n",
    "# sci-kit learn\n",
    "# from sklearn.decomposition import PCA\n",
    "from sklearn.metrics import mean_squared_error\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "\n",
    "# keras\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from keras.callbacks import EarlyStopping\n",
    "from keras.layers import TimeDistributed\n",
    "from keras.models import Sequential\n",
    "from keras.models import Model\n",
    "from keras.layers import Dense\n",
    "from keras.layers import LSTM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Prep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Workflow \n",
    "  \n",
    "1. Extract Data on TORGI\n",
    "   * Table join on jupyterhub to merge AGC, CN0 and local timestamp info\n",
    "   * Generate CSV file\n",
    "2. Build Sqlite Data Base\n",
    "   * Read CSV file and load into sqlite db in chunks to avoid using enormous CSV files and make data more accessible\n",
    "3. Clean Data\n",
    "   * Load dataframe, change time stamps to 64 bit integers\n",
    "   * Sort data by local time \n",
    "   * Reindex dataframe\n",
    "   * Remove NaN Values from Attack Column\n",
    "4. Data Wrangling\n",
    "   * Limit Data to Days with Jamming\n",
    "   * Create SVID Columns, one for each AGC & CN0 signal\n",
    "   * Normalize Signal Data\n",
    "   * Generate files with different sampling rates\n",
    "     (because memory is a **_huge_** problem).\n",
    "5. Number Crunching and More Data Wrangling\n",
    "   * Select various lookahead time windows in which to do Cross Correlation\n",
    "   * Calculate normalized Cross Correlation Matrix (CCM) \n",
    "   * Flatten CCM and append to each row in separate column\n",
    "6. Shape SigCCM Data for LSTM Input (i.e. more Data Wrangling)\n",
    "   * In addition to the CCM time window, create time sequence windows for the LSTM\n",
    "   * Wrangle data, converting column with CCMs into a column of nested sequences of CCMs of predetermined length. With 5,000 records of jamming data, this produces a huge 20GB file\n",
    "7. Store LSTM-Ready Data in HDF5 for Use by Data Generators \n",
    "8. Create Data Generator that can feed chunks of the 20GB file into Keras, batch by batch\n",
    "9. Define various LSTM models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.2.0'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scipy.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build Sqlite Data Base"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "filepath = 'CSV Files/combined.csv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# csv_database = create_engine('sqlite:///csv_database.db')\n",
    "csv_database = create_engine('sqlite:///combined.db')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "chunksize = 100000\n",
    "i, j = 0, 1\n",
    "for df in pd.read_csv(filepath, chunksize=chunksize, iterator=True):\n",
    "    df = df.rename(columns={c: c.replace(' ', '_') for c in df.columns}) \n",
    "    # shift up all index values by j\n",
    "    df.index += j\n",
    "    i += 1\n",
    "    df.to_sql('table', csv_database, if_exists='append')\n",
    "    # take highest index value and add one\n",
    "    # (don't know the index of the highest index, so use -1)\n",
    "    j = df.index[-1] + 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NOTE: Sqlite apparently requires you to put the table name between quotes. See this [Stackoverflow article](https://stackoverflow.com/questions/25387537/inserting-a-table-name-into-a-query-gives-sqlite3-operationalerror-near-sy)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# id, svid, constellation, cn0, agc, has_agc, sat_time_nanos\n",
    "# fields = \"id, svid, constellation, cn0, agc, has_agc, sat_time_nanos\"\n",
    "# fields = \"constellation, cn0, agc, has_agc, sat_time_nanos\"\n",
    "# fields = \"cn0, agc\"\n",
    "# sql_string = 'SELECT ' + fields + ' FROM table'\n",
    "# sql_string = 'SELECT * FROM \"table\" LIMIT 5'\n",
    "flds = \"svid, constellation, cn0, agc, local_time, sat_time_nanos, attack\"\n",
    "sql_string = 'SELECT ' + flds + ' FROM \"table\"'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_combined = pd.read_sql_query(sql_string, csv_database)\n",
    "\n",
    "df_combined[df_combined[\"Attack\"] == 1].count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clean Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert Time Stamps and SVID from Float to 64bit Integer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sat_time_nanos = df_combined[\"sat_time_nanos\"]\n",
    "localtime = df_combined[\"local_time\"]\n",
    "svids = df_combined[\"svid\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sat_time_nanos = sat_time_nanos.astype('int64')\n",
    "localtime = localtime.astype('int64') \n",
    "svids = svids.astype('int64')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_combined[\"sat_time_nanos\"] = sat_time_nanos\n",
    "df_combined[\"local_time\"] = localtime\n",
    "df_combined[\"svid\"] = svids"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sort Data by Local Time Stamp and Reindex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# colheader = 'sat_time_nanos'\n",
    "colheader = 'local_time'\n",
    "df_combined_sorted = df_combined.sort_values(colheader).copy()\n",
    "df_combined_sorted.index = range(len(df_combined_sorted))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remove NaN Values from Attack Column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "values = {\"Attack\": False}\n",
    "df_combined_sorted = df_combined_sorted.fillna(value = values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_combined_sorted[\"Attack\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Limit Days to Days with Jamming.\n",
    "\n",
    "Reasoning:\n",
    "* Assuming this will probably not cause data imbalance, based on Tracey's statement that pervasive jamming might represent actual conditions and \n",
    "* We don't know how to label much of the data we have. We cannot assume the absence of a True Attack label means Attack = False necessarily. \n",
    "\n",
    "Hao's calculated range of local time stamps on jamming days:\n",
    "* local time for 10/3/2018 00:00 is 1538264040000 \n",
    "* local time for 10/6/2018 00:00 is 1538523240000 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "(df_combined_sorted[\"local_time\"].min(), \n",
    " df_combined_sorted[\"local_time\"].max())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_jamming = df_combined_sorted[\n",
    "    df_combined_sorted[\"local_time\"] >= 1538264040000 # 1538264040000 \n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_jamming = df_combined_sorted[\n",
    "    df_combined_sorted[\"local_time\"] >= 0 # 1538264040000 \n",
    "]\n",
    "df_jamming[\"Attack\"].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_jamming = df_jamming[\n",
    "     (df_jamming[\"local_time\"] <= 1538523240000) # 1538523240000\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create SVID Columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_headers(df): \n",
    "    \n",
    "    svids = df[\"svid\"].unique()\n",
    "    svids = list(svids)\n",
    "\n",
    "    svidhdrs = []\n",
    "    \n",
    "    # create separate AGC & CN0 labels for each\n",
    "    # of the 106 AGC & CN0 signals.\n",
    "    for svid in svids:       \n",
    "        agc_nm = \"agc_\" + \"{:003d}\".format(svid)  \n",
    "        cn0_nm = \"cn0_\" + \"{:003d}\".format(svid) \n",
    "        svidhdrs.append(agc_nm)\n",
    "        svidhdrs.append(cn0_nm)\n",
    "\n",
    "    svidhdrs.sort()\n",
    "\n",
    "    # Since we have separate AGC & CN0 signals for\n",
    "    # each of the 106 signals, eliminate the original\n",
    "    # AGC & CN0 labels (without number suffixes)\n",
    "    dfcols = list(df.columns)\n",
    "    dfcols.remove('svid')\n",
    "    dfcols.remove('cn0')\n",
    "    dfcols.remove('agc')    \n",
    "    dfcols.remove('constellation')\n",
    "    dfcols.remove('sat_time_nanos')    \n",
    "    \n",
    "    colhdrs = dfcols + svidhdrs\n",
    "    \n",
    "    # Return signal IDs and (new) revised column headers\n",
    "    return svids, colhdrs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def create_svid_cols(df):\n",
    "\n",
    "    svids, colhdrs = get_headers(df) \n",
    "    df_new = pd.DataFrame(columns = colhdrs)\n",
    "    # Create a first row of zeros\n",
    "    d = {k:0 for k in colhdrs}    \n",
    "    df_new.loc[0] = d # first row\n",
    "    \n",
    "    lt_previous = -1      \n",
    "    \n",
    "    for rowidx, row in df.iterrows():\n",
    "       \n",
    "        svid = row[\"svid\"]         \n",
    "        agc_nm = \"agc_\" + \"{:003d}\".format(svid)  \n",
    "        cn0_nm = \"cn0_\" + \"{:003d}\".format(svid)        \n",
    "\n",
    "        lt = row[\"local_time\"]\n",
    "                    \n",
    "        if lt != lt_previous and lt_previous != -1:\n",
    "            \n",
    "            # new time stamp\n",
    "                                        \n",
    "            # open up a new row in dataframe\n",
    "            nrow = len(df_new)\n",
    "            \n",
    "            # initialize new row to have same values as last row\n",
    "            lastrow = df_new.loc[nrow - 1].to_dict()\n",
    "            df_new.loc[nrow] = lastrow\n",
    "                           \n",
    "        else:\n",
    "            # load data into current (already existing) last row\n",
    "            nrow = len(df_new) - 1\n",
    "            \n",
    "        df_new.loc[nrow][agc_nm] = row[\"agc\"]\n",
    "        df_new.loc[nrow][cn0_nm] = row[\"cn0\"]\n",
    "        df_new.loc[nrow][\"Attack\"] = row[\"Attack\"]\n",
    "        df_new.loc[nrow][\"local_time\"] = lt  \n",
    "        \n",
    "        lt_previous = lt\n",
    "    \n",
    "    return df_new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_signals = create_svid_cols(df_jamming)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pickle the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "path = PickledDir + \"df_signals.pkl\"\n",
    "fd = open(path, \"wb\")\n",
    "pkl.dump(df_signals, fd)\n",
    "fd.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Normalize Signal Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do the signal values need to be normalized?\n",
    "\n",
    "Take a peak at max signal values to find out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "local_time    1.538464e+12\n",
       "Attack        1.000000e+00\n",
       "agc_001       1.711000e+01\n",
       "agc_002       3.270000e+00\n",
       "agc_003       3.160000e+01\n",
       "agc_004       3.368000e+01\n",
       "agc_005       3.368000e+01\n",
       "agc_006       3.368000e+01\n",
       "agc_007       3.368000e+01\n",
       "agc_008       2.203000e+01\n",
       "dtype: float64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_cols = list(df_signals.columns)\n",
    "\n",
    "df_signals[data_cols].max()[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def normalize_signals(df):\n",
    "    \n",
    "    # initialize scaler object\n",
    "    scaler = MinMaxScaler(feature_range=(0, 1))\n",
    "    \n",
    "    # extract column names and remove \n",
    "    # those that are irrelevant \n",
    "    data_cols = list(df_signals_in_cols.columns)\n",
    "    data_cols.remove(\"local_time\")\n",
    "    data_cols.remove(\"Attack\")\n",
    "    \n",
    "    # extract signal values\n",
    "    vals = df[data_cols].values\n",
    "    vals = vals.astype('float32')\n",
    "    vals_norm = scaler.fit_transform(vals)\n",
    "    \n",
    "    df_norm = df.copy()\n",
    "    for idx, colname in enumerate(data_cols):\n",
    "        df_norm[colname] = vals_norm[:, idx]\n",
    "\n",
    "    return df_norm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_norm = normalize_signals(df_signals)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pickle the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "path = PickledDir + \"df_norm.pkl\"\n",
    "fd = open(path, \"wb\")\n",
    "pkl.dump(df_norm, fd)\n",
    "fd.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate Files with Different Sampling Rates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Filter DataFrame, selecting every *samplerate* rows (records)\n",
    "def samplerecords(df, samplerate):\n",
    "\n",
    "    indices = list(range(0, len(df), samplerate))    \n",
    "    df_filtered = df.iloc[indices]\n",
    "    \n",
    "    return df_filtered"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2713, 90)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_attack_step12 = samplerecords(df_norm_attackperiod, 12)\n",
    "df_attack_step12.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Go ahead and reindex this (sparsely sampled) dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_attack_step12.index = range(len(df_attack_step12))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plots of labels VS. local time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7ff9e1f92908>]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAH+NJREFUeJzt3XuYXXV97/H3d26ZJOQCzORCbhOSAAkRuYwQA9VQogY8B9RDzwMtFSwWHy1aqz2neLSK2NbWtqfP41OqcizeTgWRWo02NiKiWI5cJiCXCUSGhCRDIJmQy+SeTOZ7/tgryc6ePbPXzF5rr8t8Xs+TzN5rr1m/7/qttb977bW+81vm7oiISL7UJR2AiIhET8ldRCSHlNxFRHJIyV1EJIeU3EVEckjJXUQkh5TcRURySMldRCSHlNxFRHKoIamGW1pavK2tLanmRUQyac2aNdvdvbXSfIkl97a2Njo6OpJqXkQkk8xsY5j5dFpGRCSHlNxFRHJIyV1EJIeU3EVEckjJXUQkhyomdzO728y2mdlzg7xuZvZFM+sys2fM7MLowxQRkeEIc+T+dWDFEK9fCSwI/t0CfKn6sEREpBoV69zd/WEzaxtilmuAb3rhfn2PmtlkM5vu7q9GFONJnnh5B3/+/efo2raXvv7CLQLPPWMiZ02dwDPdu9i+9zC7DxzhqjdM45cvbseA3oN9x39/QnMDfUedhjoDwKzw+sTmBswMMzizZTy9B/tYdlYr313TTX+/g8Hhvn7GNNThgDv0u1NfZ+w52MfclvHsPnCEI0f7AXjbwqn8bN025rWewks9e9m1/wjjmuqpD9rtO+rMPm0c86aM53Cfs2nHPi4/Zwrfe/IVpk1sZtueg2ztPcSys1s52u8c6utn7ZZepk9qZtqkZtZs3MmE5ga29h463gdnT5sAwKbX99MVtDm3ZTwHDh9l4tgGVpw7jZ69h+jatpc3n3k6azbt5JGu1wGYPK6RXfuPMKG5gasWT8cM7n1iMwCLpk9k+cIpAPzzf25g3+GjACyeMZEjfc6yc1oZU1/Hmk07eXrzbsY11bNtz6Hjfb5gyim8uG0vUyaMYcmZp/Pwiz1MbG7kDTMmsevA4eMxvOWsVta91svW3kOcP2sy50ybwL1PbOb3LpnNtx/fxMJpE7lg9mROH990fNlf/FnXgH2kvs442u8sO7uVxWdM4p7HN9FYX8eeg0e4dH4Lj7+8gxsumcM/PlT43RuWzGbTjgM8/Jsezj1jIp1bek9a3vmzJrNw+gTWvbaHl3r2sfvAkeOvveeCGcw8dSy9B/t4atNOls5v4Us/f4nfXzKHzi27eXLTLgA+8NYz+cov1rNo+kTWvtrLKWMauPaimTy2YQd9R/uZOrGZjo07uPmyuTz8m+28uvsgb5w5iT2H+th/uI/tew6zdc9Bjt0Vc/nCKfz0+W3Ht/2G7fvYH2wXgN+5aCYPv9jD/sNH2VOy/x/td+qCfd2AQ339NDXUgcPbzp3K9EnN/Pi51/gvb5jOsTfJ1W88g3f/0yMnLavY2MZ6Dhw5Wva14Zh12ljeff6MqpczmH9/9lUa6+u4dH4L45vqT34xWM/5U07B3fnumm4WTDmFz6zsZP6UU+h8pZd3nDt1WO2NaaznhiVzmDS2ke8/9Qrre/YCcMXCqbxx1uSoVqssC3MP1SC5/8jdF5d57UfAX7v7fwbPHwT+zN0H/IWSmd1C4eie2bNnX7RxY6ha/JP82f3P8J2OzcP+PYmXGdTqdrxW+HysWXuV1HLday2pdTu2jaNUuh6lbbjD7y+Zw+fetZiVT2/hI/c8VVVsx9r74vUX8F/Pm868/7WKfi/8/ueuWcwNS+YMcw2OtW9r3L290nxR/IVquVUtuzu4+13AXQDt7e0j2mXec+EMJfcR+tr73sT7vvYEAM/c/nbOu/0noX/3hiWzeVPbafzxvb8e8NpvLWjhWzdfQttt/x5ZrEPZ8Pl3ArC+Zy+//fe/GHLem5a28fX/93Ls8Sz+zGr2Hupj0tjGk47sh+uqN0xj1bOvRRjd8P3uJbP59mOb+It3FRLQ+Xf8hF37R75Ow/XAn7yFBVMnRL7c//voRj71/cKlw3mt43nw48tOer39L37K0SAjD7YN7/zdC3nnedNDtbdh+z4u/7ufF775A/0OH12+gI8uP2uEazA8UVTLdAOzip7PBLZEsFwRERmhKJL7SuC9QdXMEmB3XOfbRUQknIqnZczsHmAZ0GJm3cBngEYAd/8ysAq4CugC9gPviytYEREJJ0y1zPUVXnfgjyKLSEREqqa/UBURySEldwltsJK4NJcBhin1zVI7tZS/Naosjs3oeCLvESX3USqGMmKh+vps05ZJTNS19eUWV8vtq+QuoQ2288fxBydhWIiGw8wThWPt5Ck1W8nPmrWb4k5Mc2yllNxFRHJIyV1EJIeU3EVEcihzyX00XsGPTBWd565qmTS0U0v5W6MTBl+36NfaPZm+zFxyFxGRyjKX3DN0sTp9ijpvuFUkWaoSSFLV1Tnq55ooX6YYcRtlFljL91HmkrskJ3WlkGHmUSnkiB0vhaz5Bk5vL6Y3soGU3EVEckjJXUQkh5TcRURyKHPJPc/lWbFTKWTm26ml/K3RCYOtWywDh3ky+0fmkruIiFSWueSepavVqWNlH4b7VXV8KKqEzIaypZCRjwo5cIG13L6ZS+6SnNSVQoZot9alkHlKzxoVcqA0x1ZKyV1EJIeU3EVEckjJXUQkhzKX3PNcnhU7lUJmvp1ayt8anVDTUsgh2otT5pK7iIhUlrnknqGL1elz0qiQw/xVdXwoVZdCqqNrohY3r9aokJIZqSuFDPFm1KiQI3diVMhk2k2ndEdXTMldRCSHlNxFRHJIyV1EJIcyl9zzXJ6VdiqFjL+dNJ3RTfFmjY3HcoNsT+Q9krnkLtGIujJAClTskl212Ha1rIYKldzNbIWZrTOzLjO7rczrs83sITN7ysyeMbOrog81aCuuBYuI5EjF5G5m9cCdwJXAIuB6M1tUMtungPvc/QLgOuCfog5Ukpe6UsgUjgqZp29EVuZRTdpN8defFIc2QJgj94uBLndf7+6HgXuBa0rmcWBi8HgSsCW6EEVEZLgaQswzA9hc9LwbuKRkntuBn5jZh4HxwPJIohMRkREJc+Re7otI6bXf64Gvu/tM4CrgW2Y2YNlmdouZdZhZR09Pz/CjFRGJUXwDh6XzHqrdwKyi5zMZeNrlZuA+AHf/FdAMtJQuyN3vcvd2d29vbW0dUcCjsTwrLVQKGX87aTqnm+LNKiGESe5PAAvMbK6ZNVG4YLqyZJ5NwBUAZraQQnLXoXmKpSmJ5In6Nbui3nRJ7wsVk7u79wG3AquB5ylUxXSa2R1mdnUw28eBPzSzp4F7gJs8pkMmvXdERCoLc0EVd18FrCqZ9umix2uBS6MNTdImbaWQYWhUyJHTqJADpTm2UvoLVRGRHFJyFxHJocwld13Br4LuoZr5dmopf2t0wqD3UI2pMQ0cJjLKZemcbp7lYTsouY8mVeyxab5omiZpHhdFTih7D9WibRfFgXa5fUH3UB2C3joiIpVlLrlLctJWCpnGUSHzxEp+1qzdFHdllrazkruISA4puYuI5FDmknuey7Nip1LIzLdTS/lboxMGLYWMZVTIZHoyc8ldJM+ydE43z/KwFZTcR5OiPXa4OUQ5Jxz1UzZU2kyRlEKWnZayG2Snid47IiKVZS65S3LSVwpZueGajwqZo6OP5EaFTG8npjeygZTcRURySMldRCSHMpfc81yeFTuVQma+nVrK3xqdMPiokNGv9VDvnThlLrmL5FmWzunmWR62g5L7aFJcCjnM3TdPFwrjlOaLgXJC+VEhTzyOZlTIcNPikrnkrreOiEhlmUvukpzUlUKGmUelkCN2YlTI2q5UmvswzbGVUnIXEckhJXcRkRzKXHLPc3lW2qkUsgbtpOhrf4o3a3xiGRUymZEhM5fcJRpZOneYJdV2axqqbZKPIB6V9nk7qZosgvbKLKWWfavkLpIiSY39fXIM+TScL1d56IPMJfe8HlWIiEQpc8ldkpO6UsgU3iA7TzfbSGpUyDTLUl8ouYuI5JCSu4hIII5z7Ro4LKQ8XOjIKpVCxt9OGqpljknxZpUQQiV3M1thZuvMrMvMbhtknv9uZmvNrNPMvh1tmBK19KSQfKm6FDIFGyYFIcSiYilk0ZpHUgqZ8MBhDZVmMLN64E7gbUA38ISZrXT3tUXzLAA+AVzq7jvNbEpcAYvkWRq+BaUghFioFHKgi4Eud1/v7oeBe4FrSub5Q+BOd98J4O7bog3zhLweVYiIRClMcp8BbC563h1MK3YWcJaZPWJmj5rZinILMrNbzKzDzDp6enpGFrEkJnWlkCE+6mtdCpmnow8r+VmzdlPch2m6JlJJmORebm1Kv7U0AAuAZcD1wFfNbPKAX3K/y93b3b29tbV1uLGKiEhIYZJ7NzCr6PlMYEuZeX7g7kfcfQOwjkKyFxHJjDiqqzyhQSXCJPcngAVmNtfMmoDrgJUl83wfuBzAzFoonKZZH2Wgx+ThQkdWqRQy/nbSdEoixZtVQqiY3N29D7gVWA08D9zn7p1mdoeZXR3Mthp43czWAg8B/8PdX48raKlenv5MPk2qHxUyeWmIIQ61HxWy3LTa9W7FUkgAd18FrCqZ9umixw58LPgnIiOUhqPlNMQQB5VCplxejypERKKUueQuyUldKaRGhYzViVEha32D7BT3YYpDK6XkLiKSQ5lL7nk4F5aYKjpvqJHtVC1Tu3ZqKX9rdMJg6xbfqJC6h6rIqJahb/25loftoOQ+mlRR6pXm06BpolEh06tSqWPk653wqJBK7iIiJfJwSkrJXSRF0nDqPgUhxCINfVtLmUvuef3KmAWpK4UMM0/NSyFr0lxNJLUqae7CNMdWKnPJXUREKstcch9l36yipVLIzLdTS/lboxMGLYWMYaV9iPbilLnkLpJneTqtk2V52AxK7qNJcSnYMPdeJZ1wqh31Lw13+kk+gnhULIWMeCdPelsquYuIlMjDKSkld5EUSeaePaUx5FMOL4sMKXPJPa9fGbMgbaWQYXYGlUKO3IlRIWvcbor7MNUjVpbIXHIXEZHKMpfcR9k3q1RRKWT87SR9Ea5YUps1yf0plqbdE1mnzCV3iUaWvl6OJmnYLCkIIRYV76E6yOMo26vl+07JXUQkh5TcRURKpPhMY2hK7iIpkobrFykIIRZp6Ntaylxyz+v5wCxIWylkmIuPukH2yKkUcqAUhzZA5pK7iEhc4qiu8uP/1Vbmkvso+2aVKiqFjL+dNB21jsZSyDzJXHIXSTPdQzW9KvZtFfcYrrC4SJcblpK7iEgOKbmLiJTIw5khJXeRFEnD+eYUhBCLNPRtLWUuuef1fGAWpK4UUqNCxup4KWSN33Vp7sM0x1YqVHI3sxVmts7MuszstiHmu9bM3MzaowtRRKQ24ji4d09mnP6Kyd3M6oE7gSuBRcD1ZraozHwTgI8Aj0UdZLFR9s0qVVQKGX87aToyVClktoU5cr8Y6HL39e5+GLgXuKbMfJ8DvgAcjDA+kUypPjknn92TjyAetR8VcuBSavnhHSa5zwA2Fz3vDqYdZ2YXALPc/UcRxiYiIiMUJrmX+6w5/sXJzOqAfwA+XnFBZreYWYeZdfT09ISPUkSkhvJwZihMcu8GZhU9nwlsKXo+AVgM/NzMXgaWACvLXVR197vcvd3d21tbW0cetUhuJZ9Wko8gHqPtXH6Y5P4EsMDM5ppZE3AdsPLYi+6+291b3L3N3duAR4Gr3b0jjoDzej4wC1JXChlmnlqXQuZoD9WokANlaftWTO7u3gfcCqwGngfuc/dOM7vDzK6OO8AB8dS6wTypovPcVS2ThnZqKc/VMoM2EUPbntA9VBvCzOTuq4BVJdM+Pci8y6oPS0QkOdk5Ph9c5v5CVapQxR6b5q/KaaJSyPQq3jZlR2yMeCfXqJAiIhI5JXcRkRJ5uIKi5C6SKsmnleQjiEcOr3kPKXPJPa/nA7MgdaWQIRqufSlkfljJz5q1m+JOTHNspTKX3EfZh2+0clYKGab8UKWQIzcaSyHjGL3Rh2gvTplL7iIiccvQAfqglNxHE5VCxq/qjkq+o5OPIB4VSyFjbO/EtNr1rpK7iEgOKbmLiJTIwxUUJXeRVEk+rSQfQTxyeM17SJlL7nk9H5gFKoWs3E6e9s8To0LqBtnHpDi0ATKX3EfZh2+qqBQy+XZqKc+lkLVsu1BGnMIbZItIeBo4LL0q3kO1QjXNsNsrs5S03UNVREQyRsldRCSHlNxFRErk4QqKkrtIqiSfVpKPIB45vOY9pMwl97xe7MmCPJVCRh1zrksha91ugp1Y8QNgBLFp4LCQRtmHb6qoFDL5dmppNJZC5knmkrtIFOI6OKz+D36SP/ZPPoJ4VCyFLFrzSPqg3MBhUSw3JCV3EZEcUnIXEckhJXcRkRJ5OO2v5C6SKsmnleQjiMdou1CbueSe14s9WZCnUsio5bkUstYrlWgpZIWPtnLjxVRcpnsiHyyZS+6j7MM3VVQKmXw7taRSyGzLXHIXiUJcR/QaFTK9aj4qZNkbteoeqiIiUgUldxGRHFJyFxEpkYfT/qGSu5mtMLN1ZtZlZreVef1jZrbWzJ4xswfNbE70oYqMBsmnleQjiMdou1BbMbmbWT1wJ3AlsAi43swWlcz2FNDu7ucB9wNfiDrQ4/HEtWCpSKWQybdTSxoVcqCRxlapxDIOYY7cLwa63H29ux8G7gWuKZ7B3R9y9/3B00eBmdGGWdRWXAuWilQKmXw7taRSyGwLk9xnAJuLnncH0wZzM/Djci+Y2S1m1mFmHT09PeGjFIlYbKNCVr3k5L8BJB9BPIZz1B3NDbLjWW5YYZJ7uXjKfraa2Q1AO/C35V5397vcvd3d21tbW8NHKSIiw9IQYp5uYFbR85nAltKZzGw58Engre5+KJrwRERkJMIcuT8BLDCzuWbWBFwHrCyewcwuAL4CXO3u26IPU0REhqNicnf3PuBWYDXwPHCfu3ea2R1mdnUw298CpwDfNbNfm9nKQRYnIkNK/mpi8hHEYzgXavPQB2FOy+Duq4BVJdM+XfR4ecRxiUhK5LHMczAx3B+78KGiUSFFaiO2fKWBw1Kr8sBh0d5DtdyHYi0/J5XcRURySMldRCSHlNxFRHJIyV1EJIeU3EVSJfkivOQjiMdoK4VUcheRIeW1eqacyqNCjuAG2YmMCankLqNU9QN8Dbbc5JeQ/QjiUbEUcpDHI26v7DTdQ1VERKqg5C4ikkNK7iIiOaTkLiKSQ0ruIqmSfBFe8hHEI1wppBf9H127Sdw6UMldRIY0igaFrChLfaHkLqNTTG/S6t/8yWeP5COIR+VRIYsex9SeRoUUEZGqKLmLiOSQkruISA4puYuI5JCSu0iqJF+ImHwE8QhTjnhsnkhLISGRocOU3EVkSLUc7CrtstQTSu4yKsV3f+xql5x8+kg+gnjUvBSyzFJq2bdK7iIiOaTkLiKSQ0ruIiI5pOQuIpJDSu4iqZJ8IWLyEcQjVClkyc+o2tWokCKSOlkaCTFuWeoLJXcZleJ6k2pUyPSqfINsK3ocT3saFVJERKoSKrmb2QozW2dmXWZ2W5nXx5jZd4LXHzOztqgDFRGR8ComdzOrB+4ErgQWAdeb2aKS2W4Gdrr7fOAfgL+JOlAREQkvzJH7xUCXu69398PAvcA1JfNcA3wjeHw/cIVZli49iIjkS0OIeWYAm4uedwOXDDaPu/eZ2W7gdGB7FEEWq6/TZ8ZI1VfxeTumoX7Qvh/TkMylmzDHD02DxNbcWM/BI/2RxTK2qYF9h4/S3Fhf1XKa6pPfv4/1WUNC77W4Dgsb607sC4Ntp0fXv87b/vcvePn1fZG1+9Vfrue+js2VZ4xYmORerqtLqzbDzIOZ3QLcAjB79uwQTQ900exTufXy+fzbU6/wyq4DNNXXcdOlbbz1rFZ+0vkanVt6ef7VXv7qPW/gvo7NHOlzHn95B2Ma6pg4tpEzW8az+8ARmhrqmNDcwMEj/azZuJPzZk6iqb4OM7hsfiv7j/RxwyVz+MT3nsVxDhw+yr5DR5k4toHG+jp6Dx7haD9MHtvIuq17WDrvdPYd6mPf4aM01hvvv+xM7n5kAxe3ncbjL+/gly9u55xpEzh1XBP97uw+cISl81q4bMHpHDjcT+eW3dywZA6f/WEnbS3j6d5xgAee38qn3rmQ8U0NbNtziAef30p722nMmNzMtx/fzKxTx/Lkpl1s33uImy+by4rF03CHxze8ziNdr/PCa71cfvYUdh84QlvLeC6b38KPPnwZT27aCcBnrz6Xz6zsPKl/l847nf+54hx6DxzhvXc/DsCFsydz45vbOGNy8/H1AfjAW86ksb6OG5e2AfCFa8/jSz9/icnjGnlq0y6gkCBuvmwuX3l4PTNPHcun3rmQz/5wLZPGNrJi8TQ6t/TywNqtANy0tI2Nr+/joXU93LS0jd9a0MIH/+VJvv3+S7j2y79i+cKpfPztZx2P9bTxTTQ31g1I0gunT+T5V3v50LJ5fOjy+azv2cfWPQd57pVerr94Ng+s3cqqP76Mi//yQQBWfeS3WLNxB3/+g05+75LZ/Mtjm05a3qSxjXzkigWs3dLLvz7ZfdJrn3vXYgDu+8ASVndu5bo3zeKSv3qQH374Mn6+bhuf//ELAPz0Y2/l3Xc+wrJzpvDDp7dwwezJ/PV7zuNfn+ymvs6YMXks33uymz99x9mMH9PAb7bu4aI5p7G19yBNDXV079zPL9b1sO/wUQDeOGsyT28u9PHyhVNZv30v63sKCWl8Uz133/QmvvmrjWzasZ9nX9nN2MZ6xjXVs2DqKew+0Ed9HUxsbuTI0X56D/Qxfkw9Y5vq+ejyszhn2gSa6uv4bxfOBOAPLp3Lz17Yxn90vjbo+3Je63he6qk+Ib7r/DOYNrG56uWUXfYFM3ipZy/97sf32WI3LW3jwRcK++K81lP4j87XmD/lFLq27QVgXFM950ybGLq95sZ6PrhsHhuDD4o3zpzE0nkt1a9ISOYVquvN7M3A7e7+juD5JwDc/fNF86wO5vmVmTUArwGtPsTC29vbvaOjI4JVEBEZPcxsjbu3V5ovzPfpJ4AFZjbXzJqA64CVJfOsBG4MHl8L/GyoxC4iIvGqeFomOId+K7AaqAfudvdOM7sD6HD3lcA/A98ysy5gB4UPABERSUiYc+64+ypgVcm0Txc9Pgj8TrShiYjISOkvVEVEckjJXUQkh5TcRURySMldRCSHlNxFRHKo4h8xxdawWQ+wMcJFthDDcAc1otiTkdXYsxo3KPYozHH31kozJZbco2ZmHWH+aiuNFHsyshp7VuMGxV5LOi0jIpJDSu4iIjmUp+R+V9IBVEGxJyOrsWc1blDsNZObc+4iInJCno7cRUQkkMrkbmYvm9mzZvZrM+sIpp1mZg+Y2YvBz1OD6WZmXwxuzv2MmV1YtJwbg/lfNLMbi6ZfFCy/K/jdSO79Mkjct5vZK8G0X5vZVUXzfyKIYZ2ZvaNoetkbkgfDLj8WrM93giGYI2Fmk83sfjN7wcyeN7M3Z6HPh4g99f1uZmcXxfdrM+s1s4+mvd+HiDv1fR4s+0/MrNPMnjOze8ysebD2zGxM8LwreL1tpOtUc+6eun/Ay0BLybQvALcFj28D/iZ4fBXwYwp3g1oCPBZMPw1YH/w8NXh8avDa48Cbg9/5MXBljHHfDvxpmXkXAU8DY4C5wEsUhlSuDx6fCTQF8ywKfuc+4Lrg8ZeBD0bY598A3h88bgImZ6HPh4g9E/1eFFc9hZvczMlKv5eJO/V9TuGWoBuAsUXt3DRYe8CHgC8Hj68DvjPSdar1v1QeuQ+i+Cbc3wDeVTT9m17wKDDZzKYD7wAecPcd7r4TeABYEbw20d1/5YWt9M2iZdXSNcC97n7I3TcAXRRuRl72huTBEddvU7gBOZzcB1Uxs4nAWyiMy4+7H3b3XWSgz4eIfTCp6fcSVwAvuftGMtDvg8Q9mLT1eQMw1gp3jRsHvDpEe8Xb4n7giiC+Ya1ThLGHltbk7sBPzGyNFe67CjDV3V8FCH5OCaaXu4H3jArTu8tMjytugFuDr9F3H/uKPYK4Twd2uXtfDHGfCfQAXzOzp8zsq2Y2nmz0+WCxQ/r7vdh1wD3B4yz0e7m4IeV97u6vAH8HbKKQ1HcDa4Zo73iMweu7g/iGu041l9bkfqm7XwhcCfyRmb1liHkHuzn3cKdHoVzcXwLmAedT2Jn+Ppg3TXE3ABcCX3L3C4B9FE4HDCYLsWeh3wsBFc7vXg18t9Ksg8SSSOxl4k59nwcfONdQOJVyBjCewvt1sPZSE/twpTK5u/uW4Oc24N8ofNXZGnzNJPi5LZi9G5hV9OszgS0Vps8sMz2WuN19q7sfdfd+4P8E6zKSuLdT+BreUDI9Ct1At7s/Fjy/n0LCTH2fDxZ7Rvr9mCuBJ919a/A8C/0+IO6M9PlyYIO797j7EeB7wNIh2jseY/D6JAq3Eh3uOtVc6pK7mY03swnHHgNvB57j5Jtw3wj8IHi8EnhvUEmwBNgdfJVdDbzdzE4NPq3fDqwOXttjZkuCc2fvLVpW5HEfe5MG3h2sy7G4rwuuxs8FFlC4+FX2huTBOdOHKNyAvLQPquLurwGbzezsYNIVwFpS3udDxZ6Ffi9yPSef2kh9v5eLOyN9vglYYmbjgj45tq8P1l7xtrgW+FkQ37DWKaLYhyfuK7bD/UfhHOrTwb9O4JPB9NOBB4EXg5+nBdMNuJPCFepngfaiZf0BhQsdXcD7iqa3U9jxXgL+keCPuWKK+1tBXM9Q2MjTi37nk0EM6yiqYqBQFfGb4LVPlrTxeLA+3wXGRNjv5wMdQZzfp1B1keo+rxB7Vvp9HPA6MKloWur7fZC4s9LnnwVeCPrlWxQqXsq2BzQHz7uC188c6TrV+p/+QlVEJIdSd1pGRESqp+QuIpJDSu4iIjmk5C4ikkNK7iIiOaTkLiKSQ0ruIiI5pOQuIpJD/x905id2WGMuMQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(df_attack_step12.index,\n",
    "         df_attack_step12[\"Attack\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f17265b28d0>]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJztnXvwJNV137+nu2dBQkKAWUmYXbSLvEq0SlmB/IKQSUXIIPGww6ZSjgLllJCsGJcj8pLLDooS4pBUpSw5UkoJtkxJfkixhDFxrC1llY3KJrbLBZjFkrEAL/wCQmxAYi1hhG12d6b75I/uO9PTc/t9e6an5/up2tqZnu7bt2dnT58+j+8VVQUhhJBh4a16AoQQQtxD404IIQOExp0QQgYIjTshhAwQGndCCBkgNO6EEDJAaNwJIWSA0LgTQsgAoXEnhJABEqzqxOeee67u2bNnVacnhJC15MEHH/xTVd1Ztt/KjPuePXtw5MiRVZ2eEELWEhF5qsp+DMsQQsgAoXEnhJABQuNOCCEDhMadEEIGCI07IYQMkFLjLiK/KCLPichXcz4XEfm4iGyLyEMicrH7aRJCCKlDFc/9lwFcXfD5NQD2JX9uAvDz7adFCCGkDaV17qr6uyKyp2CXAwA+rfF6ffeJyFkicp6qPutojnM88LVv4/ceO97F0IT0gvPOehluuOSChe3fOTHGZ+59CifHYe0xTxv5eM/37cEZp83/l1dV/Lf7nsLxF09aj/M9D9dfshuvOfP0hc++8NAzeOwbL9aeS5YzTgvw3sv2Ykcw72uGkeKXfv9JfOelce0xPU/wD/7mbpz3qpcV7vetPz+Jz97/dYzDqPY52nDFG1+DN+8+q9NzuGhiOh/A06n3x5JtC8ZdRG5C7N3jggsWf7xV+MOnnsd/uWe70bGE9B2zpPHVb3otzj5jx9xnv3P0OD5y+CgAQKT+mPte/Qq8802vnfvsm985iX/z+YetY5rjdgQefvzy1y+M+y/vfgh/cSqsNZe8uV10wdm4ZO85c589/tyL+A//81Hr3KqMOfI9vP/t31O47/96+Bv4T196rPY52vLqM09fC+Nu+0qsq26r6h0A7gCAra2tRitz/9jbXo8fe9viD42QIfCr9z+FD/2Pr+KUxZM8OYm3/d5PvR27z3l55TEf++aLeOfHfjdnzPgp4KPvejP+3sW75j4LI8Xr/9UhnJrYvdqTkwjvf/vr8ZNX/dXKc8nyB09+G+/6hXut5zg5jrd96sYtXPHG11QeM4oUFxbM23aOr9z6Dpz18h0le68XLqpljgHYnXq/C8AzDsYlZOMIvNhXmkSLvk8YxYbI9+q5mGb/0DKmOY9tTLPJnDeNqmISKXyvnQnxp9e7eI6iuRXheQJP7NebJWx4jnXAhXE/CODdSdXMpQBe6CreTsjQCRJjObF42eMwNkSBX88QjZIxzfFpJsm2kb9oCkQEI18wtt5okuNaGsVRci0T69yi3LmVEfgexpYbRhazT5Nz9J3SsIyIfA7A5QDOFZFjAP4tgBEAqOonABwCcC2AbQB/CeC9XU2WkKFjDLfdc0+Me01v2feN527zjoufBnxPij3+mjca2/jp8dK08aoDTxBabhgL5wiH67lXqZa5oeRzBfB+ZzMiZIOZee6LhslUdNT33CU5vshzt4858jxrJclk6rm383iNx2wLy5gnhry5FRF4Yr1h5J0jGKBxH96zCCFrjD81xPkGta4hmnrH1jGN5243Bb4vhSGTth7vbG5F52gYlqlQ3jgJI/ieQJZZKrMkaNwJ6REjPz/52TQsE0y94wLPPcdIB55nP66FV53GeP5F52jiVQc54aQsYaSD9NoBGndCekVQFKYwYZmaxmhUEMefGtCchOLIF7vHHxYfV5VpjqHgHE2SnSPfs4ahsoxDHWQyFaBxJ6RXBAVhijBSeBKX+tWhaSmk2W4/zk1Ypqj0s8054nmXh2XCKBpkMhWgcSekVxQZu3GojTzlWSmkzTs2pYA5CVXfs5ZCliViqzJ9Uin03BuEZXJKOLOMI219DX2Fxp2QHlFcChk1ig97nkBymnqqee5FDUaumphWVwpJz50Q0jllTUxNk39xSWO9JqZ4PmI/zjT/uGpisj2ptGgwihPB1ZqY6iao14VhXhUha4rx3PMMatMEZpCXGC2Ja498rzBk0j7mXh6WaXJDG/n2m5LtHAzLEEI6xxi7vFLIpp67n9PUMyuFzKlzzzsual7JkqY4odqs9BPITwRnCSOGZQghS2AWc7eHZZoa05FvD1OYbXldr6OSJqa63bJZjMiX63NUbWIahxFLIQkh3VNWCtnUyyzTiMl7IijVlnHg9ZY1SjVOqNJzJ4T0hbImpqae8igvMVrSjDTKUVds02CUJTcf0OIcQU4JZ5Zx1Ky8dB0Y5lURsqYYDzrPEDeOuecY0HGJRozv2cMyY0dNTNNzFDQxNTlF4Nmvd+EcYbPy0nWAxp2QHmEMTV4opGnZ3ign9DHVZc95IsgLmYQlidhac8vNB8SVLE1EvaqGZSbUliGELAMTIrCrQkaNy/aCvMRoSUVKrrZMSSK21txyng5ir7p5ArmqKiQTqoSQziny3NslVHOSliW15E0TsXXI015v41WzFJLGnZBeUSQ/ECdUm3qyklsKWSRGVpZQdZGMDAoapZo+GQQVm5jGbGIihCyDopWY2jYxNYnj+zkaLcvy3Jtq17AUksadkF7hJyJfeU1MjT33vOXySsor8xbIdtXEZMbIi7k3zzHU0JZhzJ0QsgxyRb4aqkIC+QZ0XKKKGHj2kIkx+O6amOzVMq7r+hfOEWpr8bO+QuNOSM/IldltU+eeE/oIo2JJg9zjjA68g1LIwC9KqDYb3/e8GmGZYZrBYV4VIWtMXjJwUmKIiyjSlim6YeRqy0yX5+u6FLKh5+5LDW0Zeu6EkCWQlwxsqy1jN6DFTwN5HnAbxcYsRWEZ11o6WZhQJYQsjbxkYCttmaLQR8HTQJxQzV+er/uEalP9+riuX7XYwFMVkhCyNIpEvhonVPMSoyWhj8DzoLrYVDUuaX6qNbe8dVpbJlTNGEVQfoAQsjR8v6AmvaknW5BQLTKgefryJpzRRPfFNjfnCWQ/v9N37hyRTvcdGjTuhPSM3Jr0KGpctldUClkUN8/Tlx+3KMu0ncOetG2hLePla/TMnSOMnFT89JFhXhUha0xeMjAMm5ft5WnLhFFxHN/PCW+ELbzqLIWlkA29ajPvIs89ihSRuqnV7yM07oT0jHiJOLuGetOyvXxtmWIjbZKN2Xh9mxBRlrx8QJuwjPmeihqZZuvA0rgTQpZAXgy69TJ7uUna4iYmc+6541yHZRxry5jjiiQIJtMFR4ZpBitdlYhcLSJHRWRbRG6xfH6BiNwjIl8WkYdE5Fr3UyVkM7CFKVS1nbZMnrpjSVhm6gFnjXsLxcYs3WjL5K9FOx1/0z13EfEB3A7gGgD7AdwgIvszu/1rAHep6kUArgfwc64nSsimYEuoGs+5eSlkc20ZYDEsU5aIrTW3gpWY2kgcmzHyKNOyX3eqfHOXANhW1SdU9RSAOwEcyOyjAM5MXr8KwDPupkjIZmFLqLZt9zehj2xTT5m2TJ6+fFkitsncsrQJ/ZhQiy28lR4fAPyBNjEFFfY5H8DTqffHALwls89PA/jfIvJPAJwB4EonsyNkAwl8wUtju3FvWrZnPOBsXXuVJibAVgrpsFrG8xpJIxRhSkYLE6rTdWA313O3XXn2G7sBwC+r6i4A1wL4jIgsjC0iN4nIERE5cvz48fqzJWQDsGnLmGRom4QqYPPAi2Pns+MyYSKnYZmCSp4OSyHNZ5tcCnkMwO7U+11YDLu8D8BdAKCq9wI4HcC52YFU9Q5V3VLVrZ07dzabMSEDJ7As7mySoW1KIYFF414mqzvKSUyWJWLr0NUC2UBxE5P5bJO1ZR4AsE9E9orIDsQJ04OZfb4O4AoAEJE3IjbudM0JaYBN5Gsy9dybyg/kJUZLwjK+vaRw7LSJyS7y1XYNVaAkoepQtriPlP5SVHUC4GYAhwE8irgq5mERuU1Erkt2+wkAPyoifwTgcwDeo2VybIQQKzaZXWNcXRu7Um2ZHPmB0GkTU14tfbvFSYCSUsiBV8tUSahCVQ8BOJTZdmvq9SMALnM7NUI2k1gVMtMR2tIQ5SZGSyQNgpxY/TiMnMWq0zeewJ9tn7RY33SU88SRZnrD3OQmJkLI8iguhWyuChmPk62fL24UKvL4XTX/5N1AnHjuFcIyVIUkhCwFm7aMMcptVCEBS2K0VBUyJ1bfYn3TKucII4Vqc696lPOkkmZWCjlMMzjMqyJkjbFpy0w6KoUsKzfML6F0qC1jeTpom2OYxdwLwjJhNLfv0KBxJ6Rn2LRWZjoobmPQ5Qtk2z1gp9oyFi+7bY6hkvzApmvLEEKWi03kq62Xaat6mYqRFZZC2mP14xY16HnnSCeRp8a9xRqqQMWE6gbXuRNClkgn2jIWT9a8LDJuxaWQbhOqoS0s4/BmlmXopZA07oT0DLNAdrpVZJr8a1wts5i0NJ5y8RqqRU1M7lQhs+fo4maWZeObmAghy8XUnaft0mxhiZaebGrQKjLCeWWKoVPhMFtCtZ1XXacUkp47IWQpFMWg26pC2pOW9cMyrrVlFufWrsFolFPCmabtOfrOMK+KkDXGVunR2nOfrqiUCstUKDcMcgS4XGrL2ES+xqGjsEyVmDvDMoSQZTBdaCJcDFM0XnbOJC1DW1im3HNfkCB2qC1jk+etMrcipjmGSmGZYZrBYV4VIWvMyOJlt21imhk7S0K1UinkoraM6yamdFdulWRvlTELwzItG6X6Do07IT3D5sm2bWKyGemwQrVInuCY21LI2SpRC3NzmEDOwlJIQshSMcnA+YRqS0/WkrQcV2gU8j2ByLzHr6qli3zUmpvtSaVlg5GIwPfsKzy5OkffGeZVEbLGFHnu7cMy9b3j7ALWbb1q2/hAJsfgwKv2c1Z4MozpuRNCloktBm089+alkIsx6Cox9/hzb+64tvLDtvHjcS3naGF4R97iilZpXN+k+gaNOyE9wyby1VZ7fBb6sDQKlYwZ+GJPdjorhSxKqDY3UYHvVapzpyokIWQp2JaImyZUW5YGhnOa6dWaeIKM1k2VRGwdikshm58jG07KYhYDEaFxJ4QsAWsTk6vSwMiWUC3z3L1MCaXbsEwXTUzm2MImJocVP32Exp2QnjFtYnIYgy7Wlin33CfW5qdleO4twjKeV1oKOdQGJoDGnZDeYZbSG2eqR/wWIYSmqpDm83mP323M3Z4PaN9gFM+7uBSSnjshZGnYRL7GUdQq8Rfk3DDSn+Ux8rz5mnvHMXebyJeZZ5v1TbNPHFlcyhb3keFeGSFrykyuNpX8DLXx4tgA4HkCT+y182UGLrt4SNVEbFVMBZDtHE2rgwATlsn33F2uA9tHaNwJ6RnThGqmWqZtyV6QWb6vaugj8L1MmaLbmPusI3fxHG1uaKUJVYfrwPYRGndCeoZtoYlJFDXWlTEEnuSoQpZ3qKaTu7NSSNeqkIvnaBuKqlIKOVRo3AnpGdYmJgdeZtbYTb3jEiOdm1B1FXPvsompNKE6XBM43CsjZE2xLxjdPvk3yhi7aVy7QkK1y1JII/JlK4Vsql8PxPMblyZU6bkTQpZEYIlBT8L2ZXtZIa2qjUJZdcVZzN2d+fA9yeQDHIRl/PkbRhaXssV9hMadkJ5hFflykFAdZRKjVdcQzWrLdLHIxWjhxtNOKA1YFDzLEi84MlwTONwrI2RNsXaThtrK0AGmpLF+vXpWW8aFYqN9bvNhGZG4hLMpZQnVkAlVQsgymTUxzZctti+FFLsqZFm1jJ9pYuogLDPKnGPs4GbGUsgKiMjVInJURLZF5Jacfd4lIo+IyMMi8lm30yRkc7CXQmqr5CJQVApZVxXSfVjG1ijV+mbmzdf1ZxlHww7LBGU7iIgP4HYA7wBwDMADInJQVR9J7bMPwAcBXKaqz4vIq7uaMCFDx64Kqa3L9rIdm5UX6/C9nBJKhzF3S6NU69JPJlRLuQTAtqo+oaqnANwJ4EBmnx8FcLuqPg8Aqvqc22kSsjnkiXy1T6jKgraMVyGuPfIkoy1jSijdeb1ZkS83TVsetWVKOB/A06n3x5Jtad4A4A0i8vsicp+IXG0bSERuEpEjInLk+PHjzWZMyMDJk+dt6ylnQx+xnnm5CVg4roO1R33LOq3twzLFqpDUlgFsV5+9HQYA9gG4HMANAD4pImctHKR6h6puqerWzp07686VkI3AiHzNq0Jqa095MTFazbhltWVcq0ICplEqm1BtH5ZhQrWYYwB2p97vAvCMZZ/Pq+pYVZ8EcBSxsSeENCC70EQYRe2Nnc1zr2Lc80ooHTcxZUsh2yhCAtSWqfKv8wCAfSKyV0R2ALgewMHMPr8J4O0AICLnIg7TPOFyooRsErHX6VhbxvcWFsSoEtfOesBmXm4TqouLcLcvhSxfIHujtWVUdQLgZgCHATwK4C5VfVhEbhOR65LdDgP4log8AuAeAD+pqt/qatKEDJ2s1+lEW8ai7lglrj3ys08R7aUBsgS+t+C5u6iWGRd47mMHeYw+U1oKCQCqegjAocy2W1OvFcAHkj+EkJZkFQ270pap4rnnacu0rWbJniPbxNQ6x+CVl0K6vEH1jeE+kxCyxmSXiBuHrrRl5m8YlTz3RF0x9uFmYRmXhnHkZ59UIgdNW/HTgJl3FmrLEEKWTjYsE0autGWypZDlBtR40ObQbrRlFkM/LkohAeQmVaktQwhZOtlk4CSK2lePWJqYqpVCmsU0otlcPIGIa1XI+aYtFwlVALnlkC66fvvMcK+MkDXGllDtphSyQrVMZvGQLkoIraWQjjz3PH2ZMZuYCCHLZrH80IG2TDZJWzGunfWAJxUTsXWwqUK6qJYBMCeWZogiharbRqy+QeNOSA+xiXy19TJNYtQwqZikna5xmsynaiK2Dtl1Wp1oyyTH2zx3s831TapPDPfKCFljssbORd2373mZsEy1Jh7fEpZxXR+eLdOseuMpwrYWraGLWv2+QeNOSA9Jl0KqKiYOtGXiLtBM12ulUkizpqvx3N3Xh4+8bMjIjX49YE+oztaBpXEnhCyRdFjGeJltE6ptVCHT83DRLbtwDt+WUG2/EhNgL4UMOyjn7Bs07oT0kHRC1Rin9qWQcS35tBmpYrXIrBQydZzjsEw2HxCXQrZvYgJg1ZeZLg7OmDshZJmkVz+aTD339toyQMoDrxqWSQxgl6WQXWjLjDI3pTRGc2bI2jI07oT0kPRCEzMvs2VYJhOmiOPa1cMys5h7+0qWLEEH2jLmeGtCNdS5fYbIcK+MkDUmnVB1lfxbTIxW1JbJ3hQ6SKjaSyHd1LkXl0LScyeELJG0sZsm/1p6y9bEaA1tGSMXXDURW29u8yJfIUshW0PjTkgPCVLLzo0dqTBmY9DVSyEtCVXnpZDzTwdjRwtkA5gL9xjMNqpCEkKWSlpbJnSU/MvGoBuXQla8KdSamz9/DifaMiacZEmodrHId9+gcSekh8yXQrrxMm3qjlXKDadt/OEsLONcWyblZauqmwWyC8IyXSzy3Tdo3AnpIWmRL2cJVWtitNwEjDIecFfaMuYcrnIMIz8/LDNbB3a4JnC4V0bIGmMLy7RPqGYTo9UqUvxMPLwLbZn0whoTR8nObDgpDROqhJCVECdUTSmkCcu0V4WMx6tX0mi821ndfRelkLNzTBzlGGZqlmxiIoT0hLgUcl5bxsUC2WY8I0bWTFummppkk7lNQnXWYJR9UkljtrGJiRCyVGxNTC4WyI7Hi2oJZ82Snd3JD6TzAa4ajILMk0oaqkISQlZCWuRrVgrpplomTMW1qzwNzI6bhWVc14cHKS97duNxk1AtKoVkQpUQslTSZXzjyE3M3U95snXEyLIesAtpgKJzuMoxzMJJlmqZyE1jWJ+hcSekh6S1yGcNN4482SialgJWMW6zNVSXkFANU9fboSrkzHOncSeELJF0aaDxPF0lVNPlhs1LId2rQsZjRyyFdASNOyE9JL3QhGtVyLR3XKuJadr81GETU6TTkEnbG8g0gcwFsgkhfSFt7NyrQkYzSYManntdNckmc5u/8Th6UikIy9BzJ4QslSDlZTtrYkrFoOvEnBd04CNtvSrU4tzcNzEFmXBSGlerW/WZ4V4ZIWtMWuTLlcjVfBdo9SYezxN4Et9oTGmm87DMnOfuRihNRJJ+gaI1VDfccxeRq0XkqIhsi8gtBfv9kIioiGy5myIhm0e6FHLiqO57zoBOPddqxi3wvNqJ2HpzMzee9PW2P4fvSaEq5EaHZUTEB3A7gGsA7Adwg4jst+z3SgD/FMD9ridJyKYRWMoW2xo7W3llVeMWSxBHtRKxjeaWOocLiYOR75WUQg43eFHlyi4BsK2qT6jqKQB3Ajhg2e/fA/gwgBMO50fIRjJfCulWW2beA69m3PxEpXLiSBogy7wqpLsGo9hzz9eWGbDjXsm4nw/g6dT7Y8m2KSJyEYDdqvoFh3MjZGNJh1DGrpqYUuWVdZqYgPgmED9FdBPOmH9ScRf6GfmSqwo58gUiw7XuVX4ttqufflsi4gH4GICfKB1I5CYROSIiR44fP159loRsGLPqEXdNTE21ZYD4ZjN/XEdNTKG7HIMZIy+hOuT1U4Fqxv0YgN2p97sAPJN6/0oAfw3A/xGRrwG4FMBBW1JVVe9Q1S1V3dq5c2fzWRMycGY12u6amIKUumPdmHPgSaJJk4RlltDE5KKSxYSTsnShbNk3qvzLPgBgn4jsFZEdAK4HcNB8qKovqOq5qrpHVfcAuA/Adap6pJMZE7IBBOma9CjuCG0bQkgnLcc149qB72USqh1Vy6QTqg7OMUqtRZtmErpvxOobpcZdVScAbgZwGMCjAO5S1YdF5DYRua7rCRKyicwkcNWZlzmXpA3rNfEE04RqN1UmtmX2XIRNikohh7xQBwAEVXZS1UMADmW23Zqz7+Xtp0XIZjP13JMEowvjLiJJmCKqXZESJB5w3URsVdILZLtsMIpLIe0x9yErQgLsUCWklxiPOgzjUkhXCcxFD7wfTUzpBLKrjlwzRm7MncadELJsZjXpEcZh5Cz5Z5bva9TEFHXXxJROILuSH4jH9QoSqsM2f8O+OkLWlLTIV+jQy5wmRmvGzqc3BYeVLNnxgbj+3KXnPirQlmG1DCFk6aRldscO1yydhmVqxrXjsEzqpuDY6zX5gDB1DlfaMnme+5B1ZQAad0J6yXShiTBOfrrz3GMP3HRt1k2ojjtKqALpkJG7sMzIz29iGrKuDEDjTkgvSXvuLhtuTGI0DE0zUj1tmbCjhGo8N8mUQtJzbwONOyE9ZFYKqU5b5aeJUeO5VzTSXWvLxHObNUp5EuvIt6WoiYmlkISQpTMrhYzcJlSzpZB1mpjC7pqY4jFnc3NX+ulNk8BpJhG1ZQghK8B41BOTUHVp7NLlhlUTqn6zRGwd/FTM3VUYymedOyGkT4zSIl+Rwzr3aWK0Xlzb3BTGDuPhtnOMk5CRq/FHXoG2DGPuhJBlM0uoupMfAGZhGbMOalUxstlx7ipZFs7hG1nhyNmTiu95G6stM+yrI2RNSTcxuQwhBElidJwoTVY/bt7j77YU0qHn7gu1ZQgh/WHW1GNKIR02MYWxKmQdTfb4ppAuhewioTprlHI1frG2zLDN37CvjpA1xvckUYV0HHNPbhi1PHejJtlhE1M6oepq/NyVmBzmMfoKjTshPWXkSUoV0m0T0ySq16EZeB7CsDtVSGD2dOA0DJXXxMSEKiFkVZjuynHoLsEYJEJadVciCnyZasvH7ztIqE6fDtwZXpZCEkJ6h1lowqn8QCoxWieOb2L142m1TDcJ1VnppxvTNOIC2YSQvjFNqDpVhYyTlmFNMbKZmmSHde5+KoHsyKv2PUGkQJTx3ichtWUIISsi9tw7aGKKYlXIeqWQsak4NelSFXKmLeOyFBLAQmgmrsihcSeErIDYk3WtLeOlSiFrhGWS858YhwhqND/VYaYt4zDHMF2+bz404/IcfWXYV0fIGhOXQpr4uMvqkVgjvm4pJACcmISdhTNmpZBurxewe+6sliGErIRRSuTLnSc7S6jWCUuYmP9Lp7pb5CLwY22ZseMFwQHM6cuEkUK1GwmFPjHsqyNkjZnrUHVc9x3WNKDTsMwk7KyEMEiuN3SYY/AtYZmu1oHtGzTuhPSUWBfF8UpMyYIY45pdoMbLPZnE3LvA5AOcJlQtnnuXFT99gsadkJ5iRL5Cl9oy/sxzrxWWmSZUu6sPH6VWiXKnLZN47jbjzoQqIWQV+J7g5Nht05AJy4xrSt5OE6rjZSRUXWrLmISqJSxDz50QsgpGvuDEJATgzssMEn3zSRjVVoUE4ph7V/Xhoy60ZSx17tMFuBlzJ4SsAt/zcCLx3F0ZVOOtnpzU71AFkrBMR+EMP61749pzDy3GnZ47IWQVjDzBS6diz91ZmMI3JY1hbW2Z2XEdJVT9uK7faROTZ6mWCbtbTapPDPvqCFljfE9w0nlYxnju9Uoazb51j6s7t9niJO5UIQFMV5BKv2ZYhhCyEkb+LCzjUlsGiMMrTUoh4+M6amJK8gHjiVtVSABz66iG07DMsM1fpasTkatF5KiIbIvILZbPPyAij4jIQyLyWyLyOvdTJWSzCHzBiXHiuTsOy5wYh421ZeokYusw8mf5AGc5BpNQTcn+mjVVN95zFxEfwO0ArgGwH8ANIrI/s9uXAWyp6vcCuBvAh11PlJBNwyzWAbgzRGmtFb+B/EDd5fnq4HdwDpu2TMiE6pRLAGyr6hOqegrAnQAOpHdQ1XtU9S+Tt/cB2OV2moRsHmnP2uUC2bPx68fcgW4Wx47HnZ2jS1XImfwAwzLnA3g69f5Ysi2P9wH4ou0DEblJRI6IyJHjx49XnyUhG4g/Z1Ddhini1/WrZbJjuCTtrXdaCkn5gSm2b2BxUUIAIvIPAWwB+Ijtc1W9Q1W3VHVr586d1WdJyAaS9qxdJTGDuaeB+mGZusfVIX2zWUoT08CNe1Bhn2MAdqfe7wLwTHYnEbkSwIcAvE1VT7qZHiGbSxc9OjvIAAAK1ElEQVTGbj70UX3MueO60paZCxm5vZmNrQlVhmUeALBPRPaKyA4A1wM4mN5BRC4C8AsArlPV59xPk5DNI+ggTJF+AqjzNJAOmdRJxNZh7hyOwzL2Ushhe+6l/7qqOgFwM4DDAB4FcJeqPiwit4nIdcluHwHwCgC/LiJfEZGDOcMRQioSdOAtzyVGaxi3dBK1u1LI1Dmcl0JuXhNTlbAMVPUQgEOZbbemXl/peF6EbDxpz9q1tgxQLyzhNzyuDn6HOQZ7KSTDMoSQFTDqJEzRMKHquw8RZWmaDyhillDlSkyEkJ4QzIUpHLXjN02oeu6Tu1mCDp5URtOE6mJYxlXStq8M++oIWWOCDjz3prXk/hKqZdLncBWWMWOGKc/dvO4qMdwXaNwJ6SlBB01Mo7nyyur//UdLqHNPn8NV0tbM1e6507gTQlZA0EGCsWm54RBLIbvSyOkLNO6E9JS5JiZHhmjU8GlgXpOme20ZVzkGY8DtqpDDNn/DvjpC1pi0Z+nO2DUTI/M8gZnOMrRlXHnVIoLAi1d4MmyK/ACNOyE9ZeR3F6bIvq50bHKD6a4UspuKnMAXe4cqE6qEkFXQSWlgw4QqMEtAdhXO6EpWeOR5Vm0ZlkISQlZCn0oh08d2pgrZwfUCcQI467mLxKGmIUPjTkhP6VMTU3oO3Rn3dCmkO9MUeN5CKeTQvXaAxp2Q3tK15153TLO/vxRtGYcxd08WmpiGXgYJ0LgT0lu60HMZtXgaMPuvkyokEH+PWVXIoSdTARp3QnqLCVMEnkCkD9Uyy0uoujzHyPcypZBRZ+vA9onhXyEha4oxdk6Ti3PGvd5//2UmVF2ew18IyyjDMoSQ1WEMnEsv0zT1AA0SquZJoitVyK7q3D2xJFRp3AkhK8J41q69zKYe+LqWQtqamIauCAnQuBPSW4z36jK5GI9nPPC6CVVj3DuKuXekXxNYmphYCkkIWRnT8IljQ9Tac1+GtozjsEy6WmYSMuZOCFkhxrN2bYimHnhNAzrTlunGbKTzAU4990xYZhLp4BUhARp3QnrLLKHq1rjPSiwbhmU6jFcHHZwjLoWcX0PV9XfaR2jcCekpXZRCpserH5bpVn4gHtv9OeJSyExClWEZQsiqMIbOdcNNUw+8a1XIeGyB77BpC7BpyzChSghZIcZ77c5z71cTkxnb9fUuasvQcyeErJCu2v1npZANVSG7jLl7nvMGI2rLEEJ6RVdCXdObRlNtmQ5DGoEvndzMFhOqwzd9w79CQtYUv7OwjNcorr2ssIzr8X1PELLOnRDSF7rQlgHiJ4EmBrRrbZl4bM/5+CN/cYFslkISQlaGSJxc7CKh2sS4+8sIy3jSSUfuYink8E3f8K+QkDUm8KQTbZkmce1ZKWS3TUyux7dry9BzBwCIyNUiclREtkXkFsvnp4nIryWf3y8ie1xPlJBNpCtPtskNw3i7XdaIm3yAS6gtk4OI+ABuB3ANgP0AbhCR/Znd3gfgeVX9HgAfA/AzridKyCYS+J5zedqR3yzUY24IXcrljjxxfvMIfI/aMjlcAmBbVZ9Q1VMA7gRwILPPAQC/kry+G8AV4rLFjJANZeSL+1JIz2v0NNC0hLLuObpJqG6etkxQYZ/zATyden8MwFvy9lHViYi8AOC7APypi0kSsqkEnuc8+dfUgHah+5Jl5HudlEKqAld+9HcgAF54abwRYZkqxt32LWiDfSAiNwG4CQAuuOCCCqcmZLP5wDvegAt3nuF0zB9+y+vw7Asv1T7uB7/3PJxxmt9pSONHLtuLE+PQ6ZhXvem12H7uzxFpbJLe8NpX4u+8+budnqOPiOqCDZ7fQeStAH5aVa9K3n8QAFT1P6b2OZzsc6+IBAC+AWCnFgy+tbWlR44ccXAJhBCyOYjIg6q6VbZflVvwAwD2icheEdkB4HoABzP7HARwY/L6hwD8dpFhJ4QQ0i2lYZkkhn4zgMMAfAC/qKoPi8htAI6o6kEAnwLwGRHZBvBtxDcAQgghK6JKzB2qegjAocy2W1OvTwD4+26nRgghpCnDL/YkhJANhMadEEIGCI07IYQMEBp3QggZIDTuhBAyQEqbmDo7schxAE81PPxcUNogD343+fC7yYffjZ0+fi+vU9WdZTutzLi3QUSOVOnQ2kT43eTD7yYffjd21vl7YViGEEIGCI07IYQMkHU17nesegI9ht9NPvxu8uF3Y2dtv5e1jLkTQggpZl09d0IIIQWsnXEvW6x7CIjIbhG5R0QeFZGHReSfJdvPEZEvicjjyd9nJ9tFRD6efCcPicjFqbFuTPZ/XERuTG3/GyLyx8kxH1+3ZRFFxBeRL4vIF5L3e5PF2R9PFmvfkWzPXbxdRD6YbD8qIleltq/tb0xEzhKRu0XkT5Lfz1v5uwFE5F8k/5e+KiKfE5HTB/+bUdW1+YNYcvj/ArgQwA4AfwRg/6rn1cF1ngfg4uT1KwE8hnhx8g8DuCXZfguAn0leXwvgi4hXxLoUwP3J9nMAPJH8fXby+uzksz8A8NbkmC8CuGbV113zO/oAgM8C+ELy/i4A1yevPwHgx5PX/xjAJ5LX1wP4teT1/uT3cxqAvcnvyl/33xjitYz/UfJ6B4CzNv13g3gZ0CcBvCz1W3nP0H8z6+a5V1mse+1R1WdV9Q+T1y8CeBTxDzS9EPmvAPi7yesDAD6tMfcBOEtEzgNwFYAvqeq3VfV5AF8CcHXy2Zmqeq/Gv9pPp8bqPSKyC8APAPhk8l4AfD/ixdmBxe/Gtnj7AQB3qupJVX0SwDbi39fa/sZE5EwAfxvx+gpQ1VOq+mfg7waI5c1fJvFKcS8H8CwG/ptZN+NuW6z7/BXNZSkkj4QXAbgfwGtU9VkgvgEAeHWyW973UrT9mGX7uvCfAfwUALOk/XcB+DNVnSTv09czt3g7ALN4e93vbB24EMBxAL+UhKw+KSJnYMN/N6r6/wD8LICvIzbqLwB4EAP/zaybca+0EPdQEJFXAPjvAP65qn6naFfLNm2wvfeIyA8CeE5VH0xvtuyqJZ8N7rtB7J1eDODnVfUiAH+BOAyTx0Z8N0mO4QDiUMp3AzgDwDWWXQf1m1k3434MwO7U+10AnlnRXDpFREaIDfuvqupvJJu/mTwaI/n7uWR73vdStH2XZfs6cBmA60Tka4gff78fsSd/VvLIDcxfz/Q7SD5/FeKlIOt+Z+vAMQDHVPX+5P3diI39pv9urgTwpKoeV9UxgN8A8H0Y+G9m3Yx7lcW6154kvvcpAI+q6kdTH6UXIr8RwOdT29+dVD9cCuCF5PH7MIB3isjZiffyTgCHk89eFJFLk3O9OzVWr1HVD6rqLlXdg/jf/7dV9YcB3IN4cXZg8buxLd5+EMD1SWXEXgD7ECcL1/Y3pqrfAPC0iPyVZNMVAB4BfzdfB3CpiLw8mbf5Xob9m1l1RrfuH8QZ/scQZ6c/tOr5dHSNfwvxY91DAL6S/LkWcdzvtwA8nvx9TrK/ALg9+U7+GMBWaqwfQZz42Qbw3tT2LQBfTY75r0ga2tbpD4DLMauWuRDxf7RtAL8O4LRk++nJ++3k8wtTx38ouf6jSFV9rPNvDMBfB3Ak+e38JuJql43/3QD4dwD+JJn7ZxBXvAz6N8MOVUIIGSDrFpYhhBBSARp3QggZIDTuhBAyQGjcCSFkgNC4E0LIAKFxJ4SQAULjTgghA4TGnRBCBsj/B00zkSlqb6/ZAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(df_norm_step1000.index,\n",
    "         df_norm_step1000[\"Attack\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculate Cross Correlation Matrix (CCM)\n",
    "\n",
    "Calculate the cross correlation matrix on a sliding time window and append the contents of the ccm to the signal data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def mycorrelate(ar_signals, normalized = False):\n",
    "    \n",
    "    # initialize cross correlation matrix with zeros\n",
    "    nrows = ar_signals.shape[0]    \n",
    "    ccm = np.zeros(shape=(nrows, nrows), dtype=list)\n",
    "    \n",
    "    for i, outer_row in enumerate(ar_signals):\n",
    "        for j, inner_row in enumerate(ar_signals): \n",
    "\n",
    "            if(not normalized):\n",
    "                x = np.correlate(inner_row, outer_row) \n",
    "            else: \n",
    "                len_inner = int(len(inner_row))\n",
    "                mean_inner = np.mean(inner_row)\n",
    "                std_inner = np.std(inner_row)\n",
    "                a = (inner_row - mean_inner) / (std_inner * len_inner) \n",
    "                b = (outer_row - np.mean(outer_row)) / (\n",
    "                     np.std(outer_row)) \n",
    "                x = np.correlate(a, b) \n",
    "                    \n",
    "            ccm[i][j] = x[0] \n",
    "            \n",
    "    return ccm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**concat_ccm**\n",
    "\n",
    "* Look ahead the number of time slices specified in window parameter _win_ \n",
    "* Take the cross correlation matrix (CCM) in the window of all signals.\n",
    "* Flatten each CCM into a single (nested 2D) list\n",
    "* Each row will have a flatted CCM. Store it in the **sigccm** column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#############################################################\n",
    "# concat_ccm: \n",
    "# Description: Takes DataFrame of signals as input \n",
    "# and concatenates a flattened ccm onto each row,\n",
    "# but unlike prior version of this routine, concat_ccm\n",
    "# will save the results at regular intervals as a precaution.\n",
    "#############################################################\n",
    "\n",
    "def concat_ccm(df, win, batchID, batch_size):\n",
    "\n",
    "    DataDir = \"data/\"\n",
    "    BatchDir = DataDir + batchID + \"/\"    \n",
    "\n",
    "    # If directory does not already exist\n",
    "    # for this batch, create it.\n",
    "    if not os.path.isdir(BatchDir):\n",
    "        mkdir(BatchDir)    \n",
    "    \n",
    "    result = []    \n",
    "    width = df.shape[1]\n",
    "    length = df.shape[0]\n",
    "    \n",
    "    # Isolate signal columns in order to calculate CCM\n",
    "    value_cols = list(df.columns)\n",
    "    value_cols.remove(\"local_time\")\n",
    "    value_cols.remove(\"Attack\")\n",
    "    \n",
    "    df_values = df[value_cols]\n",
    "    \n",
    "    # Cross correlation matrix (ccm) will be a \n",
    "    # square matrix of dimensions (win X win)\n",
    "    \n",
    "    # Length of augmented row will be:\n",
    "    #    length of current row + flattened ccm (win X win)    \n",
    "    # First win-1 rows will be extended by zeros\n",
    "    for i in range(win - 1):\n",
    "        row = list(df_values.iloc[i].values)\n",
    "        zeros = np.zeros(win * win).tolist()\n",
    "        result.append(row + zeros)\n",
    "    \n",
    "    for idx, _ in enumerate(df_values.iterrows()):\n",
    "        \n",
    "        if idx + win > length:\n",
    "            break\n",
    "\n",
    "        view = df_values.iloc[idx : idx + win].values        \n",
    "        ccm = mycorrelate(view, normalized = True)\n",
    "        \n",
    "        flatmatrix = ccm.ravel().tolist()\n",
    "\n",
    "        # Attach the flattened CCM to the **end** of the window\n",
    "        # (Careful: indices change within the view)\n",
    "        # Last line of view will always have index of win - 1\n",
    "        concatenated_line = list(view[win - 1, :]) + flatmatrix \n",
    "        result.append(concatenated_line)\n",
    "        \n",
    "        if idx != 0 and idx % batch_size == 0:\n",
    "            \n",
    "            fnm = 'df_{:s}_id{:d}.pkl'.format(batchID, idx)\n",
    "            \n",
    "            df_view = df.iloc[0 : idx + win].copy()        \n",
    "            df_view[\"sigccm\"] = pd.Series(result).values\n",
    "\n",
    "            df_view[\"sigccm\"] = df_view.sigccm.apply(\n",
    "                                   lambda x: [list(x)])\n",
    "    \n",
    "            # reindex output DataFrame\n",
    "            df_view.index = range(len(df_view))\n",
    "\n",
    "            path = BatchDir + fnm\n",
    "            fd = open(path, \"wb\")\n",
    "            pkl.dump(df_view, fd)\n",
    "            fd.close()\n",
    "\n",
    "    df_augmented = df.copy()\n",
    "    df_augmented[\"sigccm\"] = pd.Series(result).values\n",
    "    \n",
    "    # Column \"sigccm\" now contains list elements.\n",
    "    # Encapsulate each list element with an extra set\n",
    "    # of list brackets to set the stage for creating\n",
    "    # a list of lists that represents the time steps\n",
    "    # of an LSTM\n",
    "    \n",
    "    df_augmented[\"sigccm\"] = df_augmented.sigccm.apply(\n",
    "                                    lambda x: [list(x)])\n",
    "\n",
    "    # reindex output DataFrame\n",
    "    df_augmented.index = range(len(df_augmented))\n",
    "    \n",
    "    return df_augmented, result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate SigCCM Training Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_augmented, res = concat_ccm(df_norm_attackperiod, \n",
    "                          win = 100,\n",
    "                          batchID = \"norm_attack\",\n",
    "                          batch_size = 1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_augmented, res = concat_ccm(df_attack_step12, \n",
    "                          win = 100,\n",
    "                          batchID = \"attack_step12\",\n",
    "                          batch_size = 1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Split SigCCM data into train and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "length = len(df_augmented)\n",
    "len_train = int(length * 0.9)\n",
    "len_test = length - len_train\n",
    "df_sigccm_train = df_augmented[ : len_train]\n",
    "df_sigccm_test = df_augmented[len_train : length]\n",
    "# reindex the test data\n",
    "df_sigccm_test.index = range(len(df_sigccm_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate SigCCM Test Data for `id_attack_5000` from separate dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_attack_5000to6000 = df_norm_attackperiod[5000:6000]\n",
    "\n",
    "df_augmented, res = concat_ccm(df_attack_5000to6000, \n",
    "                          win = 100,\n",
    "                          batchID = \"norm_attack\",\n",
    "                          batch_size = 1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Shape SigCCM Data for LSTM Input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Add LSTM Sequences**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just as we had a lookahead window for purposes of calculating the CCM, likewise, we now calculate a moving lookahead window to create LSTM time steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def add_time_steps_df(df, win = 3):\n",
    "\n",
    "    sigccm = df[\"sigccm\"].copy()\n",
    "    \n",
    "    # Calculate the length of the list elements\n",
    "    # so we'll know how big the padding has to be\n",
    "    listlen = len(df[\"sigccm\"][0][0])\n",
    "    padding = list(np.repeat(0, listlen))\n",
    "    \n",
    "    if win < 2:\n",
    "        return sigccm\n",
    "    \n",
    "    # # First merge requires slightly different syntax\n",
    "    # # in call to combine method\n",
    "    # s = df[\"sigccm\"].apply(lambda x: x[0]).shift(-1).fillna(0)\n",
    "    # sigccm = sigccm.combine(s, lambda x1, x2: x1 + [x2])\n",
    "    # \n",
    "    # if win < 3:\n",
    "    #     return sigccm\n",
    "    \n",
    "    # Subsequent merges\n",
    "    for i in range(1, win):\n",
    "        \n",
    "        s = df[\"sigccm\"].apply(\n",
    "            lambda x: x[0]).shift(-i).fillna(0)\n",
    "            \n",
    "        sigccm = sigccm.combine(s, lambda x1, x2: x1 + [x2])\n",
    "    \n",
    "    df[\"sigccm\"] = sigccm\n",
    "\n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add time steps to training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = add_time_steps_df(df_sigccm_train, win = 100)\n",
    "df_train_rows_5000to6000_ccm_100_lstm_100 = df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = add_time_steps_df(df_sigccm_train, win = 100)\n",
    "df_train_attack_step12_ccm_100_lstm_100 = df "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add time steps to test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = add_time_steps_df(df_sigccm_test, win = 100)\n",
    "df_test_rows_5000to6000_ccm_100_lstm_100 = df  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = add_time_steps_df(df_sigccm_test, win = 100)\n",
    "df_test_attack_step12_ccm_100_lstm_100 = df "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**IO Tools**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def extracth5toarray(h5path, dataID):\n",
    "    f = h5py.File(h5path)\n",
    "    s = f[dataID]\n",
    "    return np.array(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def store_series_of_lists_in_h5file(df, dataID, colID1 = None, \n",
    "                                                colID2 = None):\n",
    "    \n",
    "    DataDir = \"data/\"\n",
    "    DataDir = DataDir + dataID + \"/\"  \n",
    "    \n",
    "    # If directory does not already exist\n",
    "    # for this batch, create it.\n",
    "    if not os.path.isdir(DataDir):\n",
    "        mkdir(DataDir)\n",
    "\n",
    "    col1 = df[colID1]\n",
    "    col2 = df[colID2]\n",
    "    \n",
    "    # get the dimensions of \n",
    "    length = len(col1)\n",
    "    col1_x, col1_y = np.array(col1[0]).shape\n",
    "    # col2_x, col2_y = np.array(col2).shape\n",
    "    \n",
    "    path = DataDir + dataID + \".h5\"\n",
    "    f = h5py.File(path) \n",
    "    \n",
    "    if colID1 != None:\n",
    "        dataset1 = f.create_dataset(colID1, (length, col1_x, col1_y), \n",
    "                                     dtype='f', \n",
    "                                     chunks=(1, col1_x, col1_y))    \n",
    "\n",
    "    if colID2 != None:        \n",
    "        dataset2 = f.create_dataset(colID2, (length,), \n",
    "                                     dtype='f', \n",
    "                                     chunks=(1,))          \n",
    "    \n",
    "    chunkshape1 = (col1_x, col1_y)\n",
    "    # chunkshape2 = (col2_x, col2_y)\n",
    "    for i, row in enumerate(col1):\n",
    "        shape1 = np.array(col1[i]).shape\n",
    "        # shape2 = np.array(col2[i]).shape\n",
    "        # if shape1 != chunkshape1 and shape2 != chunkshape2:\n",
    "        if shape1 != chunkshape1:    \n",
    "            print(\"Mismatch with element %d:\\n\" % (i))\n",
    "            print(\"chunkshape = %s\" % str(chunkshape1)) \n",
    "            continue\n",
    "        else:\n",
    "            dataset1[i] = col1[i]\n",
    "            dataset2[i] = col2[i]\n",
    "\n",
    "    f.close()\n",
    "    \n",
    "    return path     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Store LSTM-Ready Data in HDF5 for Use by Generators**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Time steps have been added. \n",
    "\n",
    "Store SigCCMLSTM training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df_train_attack_step12_ccm_100_lstm_100\n",
    "dataID = \"train_attack_step12_ccm_100_lstm_100\"\n",
    "h5path = store_series_of_lists_in_h5file(df, dataID, \n",
    "                                         \"sigccm\", \"Attack\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Store SigCCMLSTM test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df_test_attack_step12_ccm_100_lstm_100\n",
    "dataID = \"test_attack_step12_ccm_100_lstm_100\"\n",
    "h5path = store_series_of_lists_in_h5file(df, dataID, \n",
    "                                         \"sigccm\", \"Attack\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save Files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save Pickled Files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Data after each phone's signal data has been assigned a column**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "path = PickledDir + \"df_signals.pkl\"\n",
    "fd = open(path, \"wb\")\n",
    "pkl.dump(df_signals, fd)\n",
    "fd.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Data after signals have been normalized**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Data for time period in which jamming occurred**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# This file is 1.4GB, i.e. it's **huge**\n",
    "PickledDir = \"Pickled Files/\"\n",
    "path = PickledDir + \"df_attack_id5000.pkl\"\n",
    "fd = open(path, \"wb\")\n",
    "pkl.dump(df_attack_id5000, fd)\n",
    "fd.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Data after column with CCM matrix has been added**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "path = PickledDir + \"df_signals_ccm.pkl\"\n",
    "fd = open(path, \"wb\")\n",
    "pkl.dump(df_signals_ccm, fd) \n",
    "fd.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "path = PickledDir + \"df_attack_step12_sigccm_train.pkl\"\n",
    "fd = open(path, \"wb\")\n",
    "pkl.dump(df_attack_step12_sigccm_train, fd)\n",
    "fd.close() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "path = PickledDir + \"df_attack_step12_sigccm_test.pkl\"\n",
    "fd = open(path, \"wb\")\n",
    "pkl.dump(df_attack_step12_sigccm_test, fd) \n",
    "fd.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save to Dill File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import dill\n",
    "\n",
    "PickledDir = \"Pickled Files/\"\n",
    "path = PickledDir + \"df_attack_rows_5000_ccm_100_lstm_100.dill\"\n",
    "fd = open(path, \"wb\")\n",
    "dill.dump(df_attack_rows_5000_ccm_100_lstm_100, fd)\n",
    "fd.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save to HDF5 File "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "DataDir = \"data/rows_5000_ccm_100_lstm_100/\"\n",
    "fnm = \"rows_5000_ccm_100_lstm_100.h5\"\n",
    "path = DataDir + fnm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Using a DataFrame method**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "store = pd.HDFStore(path)\n",
    "store.is_open"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`store.put()` warns that performance might suffer, because PyTables is just going to pickle the objects (i.e. may as well use Pickle)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    ">> store.put('df', df_rows_5000_ccm_100_lstm_100)\n",
    ">> store.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Using `h5py`**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = df_rows_5000_ccm_100_lstm_100.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save to PyArrow-Parquet File"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df = df_rows_5000_ccm_100_lstm_100.copy()\n",
    "df[\"sigccm\"] = df.sigccm.apply(lambda x: np.array(x)) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "table = pa.Table.from_pandas(df_rows_5000_ccm_100_lstm_100)\n",
    "pq.write_table(table, 'parquet.example1')\n",
    "table2 = pq.read_table('parquet.example1')\n",
    "df4 = table.to_pandas() \n",
    "df5 = table2.to_pandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "PickledDir = \"Pickled Files/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Raw data** (after some cleaning)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "path = PickledDir + \"df_jamming.pkl\"\n",
    "fd = open(path, \"rb\")\n",
    "df_jamming = pkl.load(fd)\n",
    "fd.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Data after each phone's signal data has been assigned a column**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "path = PickledDir + \"df_signals.pkl\"\n",
    "fd = open(path, \"rb\")\n",
    "df_signals = pkl.load(fd)\n",
    "fd.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Data after signals have been normalized**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "PickledDir = \"Pickled Files/\"\n",
    "path = PickledDir + \"df_norm.pkl\"\n",
    "fd = open(path, \"rb\")\n",
    "df_norm = pkl.load(fd)\n",
    "fd.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Data for time period in which jamming occurred**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "PickledDir = \"Pickled Files/\"\n",
    "path = PickledDir + \"df_norm_attackperiod.pkl\"\n",
    "fd = open(path, \"rb\")\n",
    "df_norm_attackperiod = pkl.load(fd)\n",
    "fd.close() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Attack period data that has been sampled (every 12 datapoints)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "PickledDir = \"Pickled Files/\"\n",
    "path = PickledDir + \"df_attack_step12.pkl\"\n",
    "fd = open(path, \"rb\")\n",
    "df_attack_step12 = pkl.load(fd)\n",
    "fd.close() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "PickledDir = \"Pickled Files/\"\n",
    "path = PickledDir + \"df_attack_step12_sigccm_train.pkl\"\n",
    "fd = open(path, \"rb\")\n",
    "df_sigccm_train = pkl.load(fd)\n",
    "df_attack_step12_sigccm_train = df_sigccm_train \n",
    "fd.close() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "PickledDir = \"Pickled Files/\"\n",
    "path = PickledDir + \"df_attack_step12_sigccm_test.pkl\"\n",
    "fd = open(path, \"rb\")\n",
    "df_sigccm_test = pkl.load(fd)\n",
    "df_attack_step12_sigccm_test = df_sigccm_test \n",
    "fd.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Data after column with CCM matrix has been added**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "DataDir = \"data/attack_period1/\"\n",
    "path = DataDir + \"df_attack_id5000.pkl\"\n",
    "fd = open(path, \"rb\")\n",
    "df_attack_id5000 = pkl.load(fd)\n",
    "fd.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Data after LSTM time steps have been added**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "PickledDir = \"Pickled Files/\"\n",
    "path = PickledDir + \"df_train_attack_step12_ccm_100_lstm_100.pkl\"\n",
    "fd = open(path, \"rb\")\n",
    "df_train_attack_step12_ccm_100_lstm_100 = pkl.load(fd)\n",
    "fd.close() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "PickledDir = \"Pickled Files/\"\n",
    "path = PickledDir + \"df_test_attack_step12_ccm_100_lstm_100.pkl\"\n",
    "fd = open(path, \"rb\")\n",
    "df_test_attack_step12_ccm_100_lstm_100 = pkl.load(fd)\n",
    "fd.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Generator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generator in a Python Class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Using Python Class `getitem` as Generator**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class DataGenerator(keras.utils.Sequence):\n",
    "    'Generates data for Keras'\n",
    "    def __init__(self, batchID, \n",
    "                 batch_size = 32, \n",
    "                 dim = (32,32,32)):\n",
    "\n",
    "        self.dim = dim\n",
    "        self.batch_size = batch_size\n",
    "        self.datafile_IDs = []\n",
    "        self.labelfile_IDs = []\n",
    "        self.batchID = batchID    \n",
    "        self.DataDir = \"data/\"\n",
    "        self.BatchDir = \"\"\n",
    "        \n",
    "        DataDir = self.DataDir\n",
    "        BatchDir = DataDir + batchID + \"/\"\n",
    "        self.BatchDir = BatchDir\n",
    "        \n",
    "        path = BatchDir + \"datafilenames_\" + batchID + \".pkl\"                                    \n",
    "        fd = open(path, \"rb\")\n",
    "        self.datafile_IDs = pkl.load(fd)\n",
    "        fd.close()\n",
    "        \n",
    "        path = BatchDir + \"labelfilenames_\" + batchID + \".pkl\"        \n",
    "        fd = open(path, \"rb\")\n",
    "        self.labelfile_IDs = pkl.load(fd)\n",
    "        fd.close()\n",
    "\n",
    "    def __len__(self):\n",
    "        'Denotes the number of batches per epoch'\n",
    "        return int(\n",
    "            np.floor(len(self.datafile_IDs) / self.batch_size))\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "        \n",
    "        'Generate one batch of data'\n",
    "        \n",
    "        datafn = self.datafile_IDs[index]        \n",
    "        labelfn = self.labelfile_IDs[index]  \n",
    "        \n",
    "        print(\"In getitem: index = %d, datafn = %s, labelfn = %s\" % (\n",
    "               index, datafn, labelfn))\n",
    "\n",
    "        batch_size = self.batch_size\n",
    "        \n",
    "        # Initialize data arrays for this batch     \n",
    "        X = np.empty((self.batch_size, *self.dim))\n",
    "        y = np.empty((self.batch_size), dtype=int)         \n",
    "\n",
    "        BatchDir = self.BatchDir\n",
    "\n",
    "        # Read data\n",
    "        datafn = BatchDir + datafn\n",
    "        X = np.load(datafn)\n",
    "\n",
    "        # Read label\n",
    "        labelfn = BatchDir + labelfn\n",
    "        y = np.load(labelfn)\n",
    "        \n",
    "        return X, y        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generator Based on Numpy IO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Batch Generation Using Numpy IO**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def genbatchfiles(df, batchID, batch_size = 5):\n",
    "\n",
    "    DataDir = \"data/\"\n",
    "    BatchDir = DataDir + batchID + \"/\"\n",
    "    \n",
    "    # If directory does not already exist\n",
    "    # for this batch, create it.\n",
    "    if not os.path.isdir(BatchDir):\n",
    "        mkdir(BatchDir)\n",
    "\n",
    "    sigccm = df[\"sigccm\"]\n",
    "    attack = df[\"Attack\"]\n",
    "        \n",
    "    idcnt = 0\n",
    "    datafiles = []\n",
    "    labelfiles = []\n",
    "    \n",
    "    # number of records to process\n",
    "    nrecs = len(sigccm)\n",
    "    print(\"Number of records: %d\" % nrecs)\n",
    "    \n",
    "    for i in range(0, nrecs, batch_size):\n",
    "        \n",
    "        if i + batch_size > nrecs:\n",
    "            # upperbound = nrecs \n",
    "            \n",
    "            # if we don't have enough records left \n",
    "            # to make a complete batch, just bail out\n",
    "            break\n",
    "        else: \n",
    "            upperbound = i + batch_size            \n",
    "        \n",
    "        x = np.stack(sigccm[i : upperbound]) \n",
    "        y = np.stack(attack[i : upperbound]) \n",
    "        \n",
    "        fnm = 'data_{:s}{:02d}.npy'.format(batchID, idcnt)\n",
    "        datafiles.append(fnm)        \n",
    "        np.save(BatchDir + fnm, x)\n",
    "               \n",
    "        fnm = 'labels_{:s}{:02d}.npy'.format(batchID, idcnt)\n",
    "        labelfiles.append(fnm)        \n",
    "        np.save(BatchDir + fnm, y)\n",
    "        \n",
    "        idcnt = idcnt + 1            \n",
    "        \n",
    "    path = BatchDir + \"datafilenames_\" + batchID + \".pkl\"\n",
    "    \n",
    "    fd = open(path, \"wb\")\n",
    "    pkl.dump(datafiles, fd)\n",
    "    fd.close() \n",
    "\n",
    "    path = BatchDir + \"labelfilenames_\" + batchID + \".pkl\"\n",
    "    \n",
    "    fd = open(path, \"wb\")\n",
    "    pkl.dump(labelfiles, fd)\n",
    "    fd.close()    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of records: 5100\n"
     ]
    }
   ],
   "source": [
    "data = df_rows_5000_ccm_100_lstm_100\n",
    "batchID = \"rows_5000_ccm_100_lstm_100\" \n",
    "batch_size = 200\n",
    "\n",
    "genbatchfiles(data,\n",
    "              batchID = batchID, \n",
    "              batch_size = batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Data Generator Based on Numpy IO**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def data_generator(batchID, \n",
    "                   batch_size = 32, \n",
    "                   dim = (20, 100)):\n",
    "\n",
    "    print(\"In data_generator.\")   \n",
    "    \n",
    "    outputshape = (batch_size, *dim)\n",
    "    \n",
    "    DataDir = \"data/\"\n",
    "    BatchDir = DataDir + batchID + \"/\"\n",
    "    \n",
    "    datafile_IDs = []\n",
    "    labelfile_IDs = []\n",
    "\n",
    "    path = BatchDir + \"datafilenames_\" + batchID + \".pkl\"                                    \n",
    "    fd = open(path, \"rb\")\n",
    "    datafile_IDs = pkl.load(fd)\n",
    "    fd.close()\n",
    "\n",
    "    path = BatchDir + \"labelfilenames_\" + batchID + \".pkl\"        \n",
    "    fd = open(path, \"rb\")\n",
    "    labelfile_IDs = pkl.load(fd)\n",
    "    fd.close()       \n",
    "\n",
    "    while True:\n",
    "        for index in range(len(datafile_IDs)):    \n",
    "\n",
    "            'Generate one batch of data'\n",
    "\n",
    "            datafn = datafile_IDs[index]        \n",
    "            labelfn = labelfile_IDs[index]  \n",
    "\n",
    "            # Initialize data arrays for this batch     \n",
    "            # X = np.empty((batch_size, *dim))\n",
    "            # y = np.empty((batch_size), dtype=int)         \n",
    "\n",
    "            # Read data\n",
    "            datapath = BatchDir + datafn\n",
    "            X = np.load(datapath)\n",
    "\n",
    "            # Read label\n",
    "            labelpath = BatchDir + labelfn\n",
    "            y = np.load(labelpath) \n",
    "\n",
    "            if outputshape != X.shape:\n",
    "                msg = \"Wrong shape: \"\n",
    "                idx = \"index = {:d}, \".format(index)\n",
    "                fnm = \"datafn = {:s}, \".format(datafn)\n",
    "                shp = \"X.shape = {:s}\".format(str(X.shape))\n",
    "                msg = msg + idx + fnm + shp\n",
    "                print(msg)\n",
    "\n",
    "            else: \n",
    "\n",
    "                yield X, y                 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generator Based on HD5 IO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Target data must have the shape `(num_samples, num_classes)` according fchollet himself. See [this StackOverflow post](https://github.com/keras-team/keras/issues/9233)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def data_generator(dataID,\n",
    "                   batch_size, \n",
    "                   dim = (20, 100)):      \n",
    "    \n",
    "    print(\"\\nIn data_generator.\\n\")      \n",
    "\n",
    "    DataDir = \"data/\"\n",
    "    BatchDir = DataDir + dataID + \"/\"\n",
    "    h5path = BatchDir + dataID + \".h5\"\n",
    "    \n",
    "    f = h5py.File(h5path, \"r\")\n",
    "    data = f[\"sigccm\"]\n",
    "    labels = f[\"Attack\"]\n",
    "\n",
    "    # number of records to process\n",
    "    nrecs = len(data)\n",
    "\n",
    "    f.close()\n",
    "    \n",
    "    outputshape = (batch_size, *dim)\n",
    "    \n",
    "    while True:            \n",
    "\n",
    "        f = h5py.File(h5path, \"r\")\n",
    "        data = f[\"sigccm\"]\n",
    "        labels = f[\"Attack\"] \n",
    "        \n",
    "        for i in range(0, nrecs, batch_size):\n",
    "\n",
    "            'Generate one batch of data'\n",
    "            \n",
    "            if i + batch_size > nrecs:\n",
    "                \n",
    "                # upperbound = nrecs \n",
    "                \n",
    "                # If we can get a complete batch\n",
    "                # out of the remaining data, go\n",
    "                # ahead and wrap up this epoch and\n",
    "                # start the next one.\n",
    "                \n",
    "                break\n",
    "                \n",
    "            else: \n",
    "                upperbound = i + batch_size\n",
    "                # print(\"\\ndata[%d : %d]\\n\" % (i, upperbound))\n",
    "                X = np.array(data[i : upperbound]) \n",
    "                y = np.array(labels[i : upperbound])             \n",
    "            \n",
    "            if outputshape != X.shape:\n",
    "                msg = \"Wrong shape: \"\n",
    "                idx = \"index = {:d}, \".format(index)\n",
    "                shp = \"X.shape = {:s}\".format(str(X.shape))\n",
    "                msg = msg + idx + shp\n",
    "                print(msg)\n",
    "\n",
    "            else:\n",
    "\n",
    "                # Label (Attack) field has an extra\n",
    "                # nested array dimension. Get rid of it.\n",
    "                \n",
    "                u = np.array(y)\n",
    "                y = np.resize(u, (batch_size, 1))\n",
    "                yield X, y  \n",
    "                \n",
    "        f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Generator for Data Sampled Every 12 Steps**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create generator for training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'batch_size': 90,\n",
       " 'dataID': 'train_attack_step12_ccm_100_lstm_100',\n",
       " 'dim': (100, 10088)}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params_train = {'dataID': \"train_attack_step12_ccm_100_lstm_100\",\n",
    "          'batch_size': 90,\n",
    "          'dim': (100, 10088)}\n",
    "params_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dg_train = data_generator(**params_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create generator for test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'batch_size': 90,\n",
       " 'dataID': 'test_attack_step12_ccm_100_lstm_100',\n",
       " 'dim': (100, 10088)}"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params_test = {'dataID': \"test_attack_step12_ccm_100_lstm_100\",\n",
    "               'batch_size': 90,\n",
    "               'dim': (100, 10088)}\n",
    "params_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dg_validation = data_generator(**params_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Generator for First 5,000 Records of Attack Period**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create generator for training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'batch_size': 200,\n",
       " 'dataID': 'train_attack_5000_ccm_100_lstm_100',\n",
       " 'dim': (100, 10088)}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params_train = {'dataID': \"train_attack_5000_ccm_100_lstm_100\",\n",
    "          'batch_size': 200,\n",
    "          'dim': (100, 10088)}\n",
    "params_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dg_train = data_generator(**params_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create generator for test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'batch_size': 200,\n",
       " 'dataID': 'rows_5000to6000_ccm_100_lstm_100',\n",
       " 'dim': (100, 10088)}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params_test = {'dataID': \"rows_5000to6000_ccm_100_lstm_100\",\n",
    "               'batch_size': 200,\n",
    "               'dim': (100, 10088)}\n",
    "params_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dg_validation = data_generator(**params_test) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LSTM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build the LSTM Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(100, 10088, 1)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "time_series_length, input_dim, output_dim = 100, 10088, 1\n",
    "time_series_length, input_dim, output_dim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Build the model\n",
    "model = Sequential()\n",
    "\n",
    "# I arbitrarily picked the output dimensions as 20\n",
    "model.add(LSTM(200, \n",
    "            input_shape=(time_series_length, input_dim))) \n",
    "\n",
    "# model.add(Dense(output_dim, activation='relu'))\n",
    "model.add(Dense(output_dim, activation='sigmoid'))\n",
    "\n",
    "model.compile(loss='mean_squared_error',\n",
    "              optimizer='sgd',\n",
    "              metrics=['accuracy'])\n",
    "# model.compile(loss='mae', optimizer='adam') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example datafile: `data_rows_5000_ccm_100_lstm_100_id00.npy`\n",
    "\n",
    "**steps_per_epoch** = the number of samples in your dataset divided by the batch size, or 1 if that cannot be determined."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 12, Sampled Data, 100 Epochs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_datarecs, num_testrecs = 2441, 272"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(27, 3)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_size = params_train[\"batch_size\"]\n",
    "# subtract 100 to account for bad records that were discarded\n",
    "steps_per_epoch = int(np.floor((num_datarecs) / batch_size))\n",
    "validation_steps = int(np.floor((num_testrecs) / batch_size))\n",
    "steps_per_epoch, validation_steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# removing spew to make notebook render on Github\n",
    "history = model.fit_generator(generator = dg_train,\n",
    "                    validation_data = dg_validation,\n",
    "                    validation_steps = validation_steps,\n",
    "                    steps_per_epoch = steps_per_epoch,\n",
    "                    epochs=100, \n",
    "                    verbose=1,\n",
    "                    workers=1,\n",
    "                    max_queue_size=1, \n",
    "                    use_multiprocessing=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the training and test loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xt8VeWd7/HPb+/sJEC4hiiXAEFFBW9cwkUt3motaIvtaWu90NEzbemrPZ4zczp1qtOxPbVnztg601pnrCO1trVOtWovQ0etqIPVWlCCd0DlKgRUIjdBIGQnv/PHs3ayEwLZhCSb7PV9v17rtfda61krz3ot+P328zzrYu6OiIjEQyLfFRARkZ6joC8iEiMK+iIiMaKgLyISIwr6IiIxoqAvIhIjCvoiIjGioC8iEiMK+iIiMVKU7wq0NXToUK+qqsp3NUREepVly5a95+4VHZU76oJ+VVUVNTU1+a6GiEivYmZv5VJO3TsiIjGioC8iEiMK+iIiMXLU9emLiHRGQ0MDtbW17Nu3L99V6ValpaVUVlaSSqU6tb2CvogUhNraWvr3709VVRVmlu/qdAt3Z+vWrdTW1jJ27NhO7UPdOyJSEPbt20d5eXnBBnwAM6O8vPyIWjMK+iJSMAo54Gcc6TEWTtBvaoKFfw/bc7pUVUQklgon6G9bCy/cAz++ADYsyXdtRCRmduzYwY9+9KPD3u7iiy9mx44d3VCj9hVO0B96AnzhSSgdCD//OLx8f75rJCIxcrCg39jYeMjtHnnkEQYNGtRd1TpA4QR9gKHj4AtPwOgZ8NsvwSPXwf49+a6ViMTA9ddfz5o1a5g4cSJTp07l/PPP58orr+S0004D4BOf+ARTpkzhlFNOYf78+c3bVVVV8d5777F+/XrGjx/PF7/4RU455RQuuugi9u7d2+X1LLxLNvsOgbm/gce/BUtuh9VPwifvhFFT810zEekh3/79clZsfr9L9zlhxAC+9fFTDrr+5ptv5rXXXuOll17iqaee4pJLLuG1115rvrTy7rvvZsiQIezdu5epU6fyqU99ivLy8lb7WLVqFffddx8//vGPueyyy/j1r3/N3Llzu/Q4CuuXfkYyBbP+H1z9e2hsgLsvgke/Dnu25btmIhIT06ZNa3Ut/W233cYZZ5zBjBkz2LhxI6tWrTpgm7FjxzJx4kQApkyZwvr167u8Xjn90jezWcAPgSRwl7vf3Gb9V4EvAGmgDvhLd38rWjcauAsYBThwsbuv76oDOKSx58CXn4UnvgXPz4eX7oNz/gamfQlSpT1SBRHpeYf6Rd5T+vXr1/z9qaee4oknnmDx4sX07duX8847r91r7UtKSpq/J5PJbune6fCXvpklgduB2cAE4Aozm9Cm2ItAtbufDjwEfC9r3T3ALe4+HpgGbOmKiuesdAB87Afw5T/D6Onw+Dfhh2fAs7dB/a4erYqIFK7+/fuza1f7MWXnzp0MHjyYvn378vrrr7NkSf6uMMyle2casNrd17r7fuB+4NLsAu6+yN0zI6ZLgEqAKDkUufvjUbndWeV61jHj4aoHQ5dPxYnw+I3wg1Pgif8D29blpUoiUjjKy8s5++yzOfXUU7nuuutarZs1axbpdJrTTz+dG2+8kRkzZuSplmDufugCZp8GZrn7F6L5zwHT3f3ag5T/V+Add/+/ZvYJQrfPfmAs8ARwvbsf9Bqm6upq75GXqNQug2d/AK8/DN4Ex18Ak6+Gk2ZDUUnH24vIUWXlypWMHz8+39XoEe0dq5ktc/fqjrbNpU+/vXt+280UZjYXqAbOzdr/TGASsAH4FXAN8JM2280D5gGMHj06hyp1gcop8Nl7YecmePEX4cauB6+GPkPgtM/AxCtg+ESIwW3dIhIfuXTv1BIGYTMqgc1tC5nZhcA3gDnuXp+17YtR11Aa+B0wue227j7f3avdvbqiosNXPHatgSPhvOvhr1+Fq34Nx50Ly34K88+DW0+HP9wQ7vDtoEUkItIb5PJLfykwzszGApuAy4ErswuY2STgTkI30JY22w42swp3rwMuAI7OF+AmkjDuwjDt2QZvPAorfw9LfwJLfgSDq+D0y+GMz8KQ4/JdWxGRTukw6Lt72syuBR4jXLJ5t7svN7ObgBp3XwDcApQBD0ZPgNvg7nPcvdHMvgY8aWHFMuDH3XUwXabvEJh0VZjqd4V+/5d+CX/8LvzxZhh2OkyYA+MvDYPCIiK9RIcDuT2txwZyO2NnLSz/LaxYALXPh2XlJ8BJF8PJl0Dl1NBiEJEep4HcrhvIlYyBlXDW/wzT+5tDC+D1h0P3z59vg77lMO6j4Qqg48+Hkv75rrGISCuF+RiGnjBgBEz7IvzF7+Bv18KnfhIu+3zjYXjgc/DdsXDPpbD4R7B1Tb5rKyLdrLOPVga49dZb2bOnZ25hUtDvCqUD4bRPw6fuguvWwjUPw4wvw6534LEb4F8mw22TwvN/Vj0BDV1/a7WI5FdvCfrq3ulqySKo+lCYLvoObF8Pqx6HVQth2c/guX+DolIYczac8GE4/sNQcZLuBxDp5bIfrfyRj3yEY445hgceeID6+no++clP8u1vf5sPPviAyy67jNraWhobG7nxxht599132bx5M+effz5Dhw5l0aJF3VpPBf3uNrgqdANN+2L4hf/Ws+Fxz6seh8f+LpQZUBkSwAkXhvsESgfmtcoivd6j18M7r3btPoedBrNvPujq7EcrL1y4kIceeojnn38ed2fOnDk8/fTT1NXVMWLECB5++GEgPJNn4MCBfP/732fRokUMHTq0a+vcDgX9npTqEwL7CRfCrH+EHRtCAlj9RLgq6IWfgyXDVUDHnRemyurwqGgR6TUWLlzIwoULmTRpEgC7d+9m1apVzJw5k6997Wt8/etf52Mf+xgzZ87s8bop6OfToNFQ/d/D1NgAtTWw+nFY+xQ8/b1wT0BxWXhE9PEXwHHnQ/nx6goS6cghfpH3BHfnhhtu4Etf+tIB65YtW8YjjzzCDTfcwEUXXcQ3v/nNHq2bgv7RIpmCMWeG6cPfhL3bYd0zsHZRaA288Ugo139ESAJjzwktgYEj81lrEYlkP1r5ox/9KDfeeCNXXXUVZWVlbNq0iVQqRTqdZsiQIcydO5eysjJ+9rOftdpW3Ttx1mdwuOt3wpzw3J9ta2HdH2Hd06E18Er04vehJ4bgP/bcMHjcp+desCwiLbIfrTx79myuvPJKzjzzTADKysq49957Wb16Nddddx2JRIJUKsUdd9wBwLx585g9ezbDhw/v9oFc3ZHbGzU1wZbloRtozSJ468+Q3guWgOFnhFZA1TnhBfElZfmurUiP0B25uiO3cCUS4UqCYaeFu4PT9WE8INMSWPwjePaHkCiCEZNh7EyomgmjpkNx33zXXkTySEG/EBSVQNXZYTr/72D/B7DxuTAmsP4Z+NOt8Mw/QyIFI6dE9xGcDZXT1BIQiRkF/UJU3C9c7XP8BWG+fld4J8D6P4X7BP70A3jmn0JLYPjEkADGfCh0B5UOyG/dRY6Au2MFfnXbkXbJK+jHQUl/GPeRMAHU7w4tgbeehfXPtnQHWSIkgbEzozGB6XponPQapaWlbN26lfLy8oIN/O7O1q1bKS0t7fQ+NJArsH8P1C4NSWDdM+F7U0NIAsNOg9FnwZizYPSZUNbDbzYTyVFDQwO1tbXs27cv31XpVqWlpVRWVpJKtb5pM9eBXAV9OdD+PaElsGFxuDKodimko/9IQ08MwX9MlAgG9dA7jUXkkHT1jnRecd/wPoDjzw/z6f2w+UXY8OeQBJb/LjwyAsJzg8ac2ZIIhp4Uri4SkaOSfunL4WtqhC0r4K3FLYlg97thXemgcGno6BkhEYyYBKnO9z+KSG70S1+6TyLZcp/A9Hktdww3dwkthlWPhbLJ4hD4R00PSWDUdOhXnt/6i8SYfulL9/hga0sS2PgcbHohDA4DlI+LksB0GDUDho7TQ+REjlCX/tI3s1nAD4EkcJe739xm/VeBLwBpoA74S3d/K2v9AGAl8Ft3vzbno5Deq185nHxxmAAa9kXjAlESeONheOnesK7P4JAEKqeGz5GTw70GItLlOgz6ZpYEbgc+AtQCS81sgbuvyCr2IlDt7nvM7MvA94DPZq3/DvDHrqu29Dqp0paniELoEnpvVUgAG5fAhufgzT+EdRZ1H2VaA5XTwkvp1RoQOWK5/NKfBqx297UAZnY/cCnQHPTdPfuxcEuAuZkZM5sCHAv8Aeiw6SExYQYVJ4Zp8ufCsj3bwjOENi6Bjc/DC/fA83eGdf2Hw6hpIQGMmhYeLFdUkr/6i/RSuQT9kcDGrPlaYPohyn8eeBTAzBLAPwOfAz58sA3MbB4wD2D0aF33HVt9h8CJF4UJwotl3nk13CdQuzS0Clb8R1iXLA53D1dOhVFTo9aA3i0g0pFcgn57bep2R3/NbC7h1/y50aKvAI+4+8ZD3Rbt7vOB+RAGcnOok8RBMhX690dOhunRG4h2vQu1z4eWQO1SqPkJLLk9rBswMkoC02BkdWgN6HJRkVZyCfq1wKis+Upgc9tCZnYh8A3gXHevjxafCcw0s68AZUCxme129+uPrNoSW/2PhfEfDxOEG8fefRU2Lo2SwVJY8buwLpGC4aeHsYFMItDYgMRch5dsmlkR8Cahe2YTsBS40t2XZ5WZBDwEzHL3VQfZzzWEwd5DXr2jSzbliO16J4wNZLqFNi1reYxEnyGhBVBZHb1jYFp4Yb1IL9dll2y6e9rMrgUeI1yyebe7Lzezm4Aad18A3EL4Jf9g1I2zwd3nHNERiHRW/2Ew/mNhgtAaeOdV2PwCvP0yvP1SeL/A07e0jA0cOwEqxoeWwcgpGiSWgqWbsySe9r0f7hlY9zRsfim8fnLv9rCuqDS0AEafBceMD9OQ48IYg8hRSo9hEDmU0gFw4kfDBOG+gd1bYFNNeNnMumfgj9+l+ZqFotLwPKGx54QXzhw7Qe8akF5JQV8EwuBu/2Ph5EvCBOER0++9CXWvh9bA+mfgyZtathk4KrQCRp8ZksHwiZDUfyk5uulfqMjBFPeFERPDdMblYdkH74X7BbashLo34J1XYNXCqHwZVJwU3jkwdFx4rlBltcYH5KiioC9yOPoNbd0aANhdF1oBGxaHRLDuaXj5vrCuqE8YHxh2GpSfEJLBsNP1LmLJGwV9kSNVVgGn/rcwZezdHh4xvf6ZMC29q+Wy0cy7iKs+FC4fHXIclB8PpQPzU3+JFQV9ke7QZ3Drp4w2NcH7tVD3ZugeWv8nWHJHy+OmIbx6cszZYRoxCYaM1dNGpcsp6Iv0hEQiBPVBo2HchWFZw77w8plta2Dr6nAT2aqFLV1DAGXDWgaLx5wJIyZDSVl+jkEKgoK+SL6kSsOln8dOaFnmHsYFtqyIEsLacEPZU/9I8+WjZcfC4KqWweIxZ4UuIj1eQnKgoC9yNDGDY04OU7a9O0K30DuvwvZ1sP0teP0ReDHrRTRDjg9dQuUnhKuGKqdpwFgOoKAv0hv0GdT6ZjII4wRbV4Wrhja/CNvWhcTw2q/Bm8KAccV4GHpCaBkMOT4kg4rxobtJYklBX6S3SiTCfQEVJ8GUa1qW1+8KD5zbsCQ8b+jd5aFVkBk0LhkY3kFQPi4kg8FV4V6E/sPycBDS0xT0RQpNSX84/vwwZTQ1wvb14amjGxaHpPDWYmj4oKXM4KowRnDMyTBoTJiOOVlXEBUYBX2ROEgkw70A5ce33F3sHu4w3rYmahkshjX/Ba/c37KdJWHYqWF84NhTYPCYloSgR070SjprInFlFm4sK6sID5M7K3rVRf0u2LEhjBG8/VIYJ3j5Pti/u2XbVN9w+eioqeFu40FjwuWo/Sp0FdFRTkFfRFor6R9+1R97Sss7CZoaYdfb4aqh7evg7VfCm8r+/C/QlG7ZtnRgeB9B5dTwuIlBo8PbyvoMVjI4Sijoi0jHEskQvAdWQtXZMCla3rA33EuwY0OYtqwIXUVP3xKuIMooHRhaBpXRu4sHjYFBo6B0kJJBD1PQF5HOS/VpaRVkq98dbjJ7vxZ21oZHVNcuC28sy04GJQNhxBmhZTBichhMHlgZkoSSQbdQ0BeRrldSBpVTgCmtl+//ALa8Djs3hmSwbU14/MSfbgVvzNp+QOgeGjk5TIPHhvcX9B2iZHCEFPRFpOcU9wvJoLJtMtgT3lGwc0OUDKJB5Of+DRr3t5Qr6hMeWzFySmgZDB0HA0aER1Mkkj17LL1UTkHfzGYBPyS8GP0ud7+5zfqvAl8A0kAd8Jfu/paZTQTuAAYAjcA/uPuvurD+IlIIivu2nwzS9SEZ7NgA728Kn2+/Ai/+Ozw/v6WcJUMCGBG1DCpOggEjQ0JI9enZYznKdfhidDNLAm8CHwFqgaXAFe6+IqvM+cBz7r7HzL4MnOfunzWzEwF391VmNgJYBox39x0H+3t6MbqIdKipMYwTbH8rJIOdteHO480vwAd1rcsOqIzegDYJjpkQBpAHjgqPtiggXfli9GnAandfG+34fuBSoDnou/uirPJLgLnR8jezymw2sy1ABXDQoC8i0qFEMjxy+pjxrZe7R91Da+H9zWEgecvr4dlEr/9n67J9y6OWwZRwr0Hm6qS+5QU9bpBL0B8JbMyarwWmH6L854FH2y40s2lAMbDmcCooIpIzs/BLftCoA9ft3QFb10SDyBujZPACrH6C5sdWA6T6hbuQh50W3YVcBQOj+w1SpT11JN0ml6DfXsprt0/IzOYC1cC5bZYPB34BXO2efb1W8/p5wDyA0aNH51AlEZHD1GdQ++MG9bvhvTdg56bQOti2NjzC+uVfwf5dLeUsEV56P/yMlpZB/+FhGjiq1zy5NJegXwtkp81KYHPbQmZ2IfAN4Fx3r89aPgB4GPh7d1/S3h9w9/nAfAh9+jnXXkTkSJWUhS6ekW2SQeYVlzuilsG2tWEQed0z8Eqb61GKy0IiGHZ6eD5R/+FhIHnouHCZ6VEkl6C/FBhnZmOBTcDlwJXZBcxsEnAnMMvdt2QtLwZ+C9zj7g92Wa1FRLpb9isu29qzLbQKdr0TEsO7y8Mbzl68t/WTSyF0DQ07Lbzgpv/w8AjrIWPDew2K+/bMsWTpMOi7e9rMrgUeI1yyebe7Lzezm4Aad18A3AKUAQ9aGADZ4O5zgMuAc4ByM7sm2uU17v5S1x+KiEgP6TskTMNObb3cHfZujwaRN4XHUrzzapjWPAnpfVmFLbzmsvyEaBB5JFScDCdf0q1V7/CSzZ6mSzZFpCC5w76d4cF1W1eH1sG7y8MD7HZugr3bYNR0+PzCTu2+Ky/ZFBGRI2UWBpP7DAqXmo7/eOv1+z8Ij7XuZgr6IiJHg+J+PfKWst5xjZGIiHQJBX0RkRhR0BcRiREFfRGRGFHQFxGJEQV9EZEYUdAXEYkRBX0RkRhR0BcRiREFfRGRGFHQFxGJEQV9EZEYUdAXEYkRBX0RkRhR0BcRiREFfRGRGFHQFxGJEQV9EZEYySnom9ksM3vDzFab2fXtrP+qma0ws1fM7EkzG5O17mozWxVNV3dl5UVE5PB0GPTNLAncDswGJgBXmNmENsVeBKrd/XTgIeB70bZDgG8B04FpwLfMbHDXVV9ERA5HLr/0pwGr3X2tu+8H7gcuzS7g7ovcfU80uwSojL5/FHjc3be5+3bgcWBW11RdREQOVy5BfySwMWu+Nlp2MJ8HHj2cbc1snpnVmFlNXV1dDlUSEZHOyCXoWzvLvN2CZnOBauCWw9nW3ee7e7W7V1dUVORQJRER6Yxcgn4tMCprvhLY3LaQmV0IfAOY4+71h7OtiIj0jFyC/lJgnJmNNbNi4HJgQXYBM5sE3EkI+FuyVj0GXGRmg6MB3IuiZSIikgdFHRVw97SZXUsI1kngbndfbmY3ATXuvoDQnVMGPGhmABvcfY67bzOz7xASB8BN7r6tW45EREQ6ZO7tds/nTXV1tdfU1OS7GiIivYqZLXP36o7K6Y5cEZEYUdAXEYkRBX0RkRhR0BcRiREFfRGRGFHQFxGJEQV9EZEYUdAXEYkRBX0RkRhR0BcRiREFfRGRGFHQFxGJEQV9EZEYUdAXEYkRBX0RkRhR0BcRiREFfRGRGFHQFxGJEQV9EZEYySnom9ksM3vDzFab2fXtrD/HzF4ws7SZfbrNuu+Z2XIzW2lmt1n05nQREel5HQZ9M0sCtwOzgQnAFWY2oU2xDcA1wC/bbHsWcDZwOnAqMBU494hrLSIinVKUQ5lpwGp3XwtgZvcDlwIrMgXcfX20rqnNtg6UAsWAASng3SOutYiIdEou3TsjgY1Z87XRsg65+2JgEfB2ND3m7isPt5IiItI1cgn67fXBey47N7MTgPFAJSFRXGBm57RTbp6Z1ZhZTV1dXS67FhGRTsgl6NcCo7LmK4HNOe7/k8ASd9/t7ruBR4EZbQu5+3x3r3b36oqKihx3LSIihyuXoL8UGGdmY82sGLgcWJDj/jcA55pZkZmlCIO46t4REcmTDoO+u6eBa4HHCAH7AXdfbmY3mdkcADObama1wGeAO81sebT5Q8Aa4FXgZeBld/99NxyHiIjkwNxz6p7vMdXV1V5TU5PvaoiI9CpmtszdqzsqpztyRURiREFfRCRGFPRFRGJEQV9EJEYU9EVEYkRBX0QkRhT0RURiREFfRCRGFPRFRGJEQV9EJEYU9EVEYkRBX0QkRhT0RURiREFfRCRGFPRFRGJEQV9EJEYU9EVEYkRBX0QkRhT0RURiREFfRCRGcgr6ZjbLzN4ws9Vmdn07688xsxfMLG1mn26zbrSZLTSzlWa2wsyquqbqIiJyuDoM+maWBG4HZgMTgCvMbEKbYhuAa4BftrOLe4Bb3H08MA3YciQVFhGRzivKocw0YLW7rwUws/uBS4EVmQLuvj5a15S9YZQcitz98ajc7q6ptoiIdEYu3TsjgY1Z87XRslycCOwws9+Y2YtmdkvUcmjFzOaZWY2Z1dTV1eW4axEROVy5BH1rZ5nnuP8iYCbwNWAqcByhG6j1ztznu3u1u1dXVFTkuGsRETlcuQT9WmBU1nwlsDnH/dcCL7r7WndPA78DJh9eFUVEpKvkEvSXAuPMbKyZFQOXAwty3P9SYLCZZX6+X0DWWICIiPSsDoN+9Av9WuAxYCXwgLsvN7ObzGwOgJlNNbNa4DPAnWa2PNq2kdC186SZvUroKvpx9xyKiIh0xNxz7Z7vGdXV1V5TU5PvaoiI9CpmtszdqzsqpztyRURiREFfRCRGFPRFRGJEQV9EJEYU9EVEYkRBX0QkRhT0RURiREFfRCRGFPRFRGJEQV9EJEYU9EVEYkRBX0QkRhT0RURiREFfRCRGFPRFRGJEQV9EJEYU9EVEYkRBX0QkRhT0RURiJKegb2azzOwNM1ttZte3s/4cM3vBzNJm9ul21g8ws01m9q9dUWkREemcDoO+mSWB24HZwATgCjOb0KbYBuAa4JcH2c13gD92vpoiItIVcvmlPw1Y7e5r3X0/cD9waXYBd1/v7q8ATW03NrMpwLHAwi6or4iIHIFcgv5IYGPWfG20rENmlgD+Gbju8KsmIiJdLZegb+0s8xz3/xXgEXffeKhCZjbPzGrMrKauri7HXYuIyOEqyqFMLTAqa74S2Jzj/s8EZprZV4AyoNjMdrt7q8Fgd58PzAeorq7ONaGIiMhhyiXoLwXGmdlYYBNwOXBlLjt396sy383sGqC6bcAXEZGe02H3jrungWuBx4CVwAPuvtzMbjKzOQBmNtXMaoHPAHea2fLurLSIiHSOuR9dvSnV1dVeU1OT72qIiPQqZrbM3as7Kqc7ckVEYkRBX0QkRhT0RURipKCC/r1L3uK93fX5roaIyFGrYIL+uvc+4Jv/8Rozv7uIf3xkpYK/iEg7CurqndVbdvOv/7WKBS9vpqQoybhjy+hXXES/kiR9iovok0rQJ5WkNJWkJJWMvicoKQqfpakkJUVhviSVoDT6LE4msj5DmaKEYdbezcoiIj0v16t3CiroZ6zespufPruOTTv28kF9mt31jexraGTv/kb27E+zL93E/vQBz4Y7LAmDkqIkxUWJkCgySSFrWfO6aFlxMizLXl+ctb4kmSBVZGE+q2xm25I288VFCVLRciUgkXjLNejnckdur3PCMWX8wydPO2SZpianPt3EvoZG9qUbqW9oav7MLN+fzvre2BTNt1kefa9Pt6zPfN9dn2br7qYDts1sk27quoSbSlqrRJFKHpgsUsnWnyXJ9pcXJ+2A5SXRvlLJBKnmpGOtt836m6miRHOdlJBEjh4FGfRzkUgYfYqT9ClO5q0OTU3O/saWhLG/sYn6rATTnCDazLdaH23fcLAyja3X7a5PH7CPhlZlu77ll0qG5JCZspNKqlWisKwyiebtWiegljIlWftItdlnJuGkilrPF2Wvy0pOqUSCRELJSQpfbIP+0SCRMEoTYYzhaOHuNDSGZNRwkOQRPlvKNDS2KZeVQDLL0tF8JkG17Mubv9c3NLF7X7q5FZRJSummpqz9eJe2kLIVJaw5QTQnj0wiSoTvRYn2E1RR0loltVSUYIqTFiWalsRUlMiUP3AfRYnQgipKZBKdZZUPY0mprFZXUolKDpOCvrRiZhQXheBESb5r077GJm9OHNlJI5N80lHS2p8O3xsaM91p0TZRwsokorYJLbPPhqx9tf2+r6GJXfvSUVLyA9Y3/+2mJrpz2MwMUonspJNJOJnEEX0WJUglrFVyyiS5VmWbE09YX5TMbJfZl0X7CtslEy0tqKKoxZRJaEWJkJRSbda1+t78qQsjeoqCvvQ6yYSRPMpaSIeSSVJtE1FDY0vCyE4eDY3enJCyk09DU2hFpZsyrSynsamJhiZvszx8Tzcvb9lXutHZnU431yE7YWXPZ+rR2E2tqvYkEyH4N7d6kpnEEyWHrOSTTFirxJOdxJLN22TtI9GyXXYiK0pktjeSWQkrmQhlku38/Uw9s1tfRYkEyaRlbdOy78zn0ZLUFPRFullvS1LZ3EPyyCSddJtE1CppROvTjSERpaMkEsplJbPMumibxmg/6aaW1lJm3837aGpJhOmmppZE2pzg2tkmKtvqezd2D3akOVlP1krvAAAEiklEQVQckHBaEtUpIwbyL1dM6tZ6KOiLyEGZZcYeoA+9L2m1JzuRNXpLMskkknRTaEG1JIuWMk1+YALKlGlsatlPdvJpanIaMvts9Ob9ZJJlpg7pJmfU4D7dfvwK+iISK9mJLI4K5jEMIiLSMQV9EZEYUdAXEYkRBX0RkRhR0BcRiREFfRGRGFHQFxGJEQV9EZEYOepeomJmdcBbR7CLocB7XVSd3iKOxwzxPO44HjPE87gP95jHuHtFR4WOuqB/pMysJpe3xxSSOB4zxPO443jMEM/j7q5jVveOiEiMKOiLiMRIIQb9+fmuQB7E8Zghnscdx2OGeB53txxzwfXpi4jIwRXiL30RETmIggn6ZjbLzN4ws9Vmdn2+69NdzGyUmS0ys5VmttzM/ipaPsTMHjezVdHn4HzXtauZWdLMXjSz/4zmx5rZc9Ex/8rMivNdx65mZoPM7CEzez0652cW+rk2s/8d/dt+zczuM7PSQjzXZna3mW0xs9eylrV7bi24LYpvr5jZ5M7+3YII+maWBG4HZgMTgCvMbEJ+a9Vt0sDfuPt4YAbwP6JjvR540t3HAU9G84Xmr4CVWfPfBX4QHfN24PN5qVX3+iHwB3c/GTiDcPwFe67NbCTwv4Bqdz8VSAKXU5jn+mfArDbLDnZuZwPjomkecEdn/2hBBH1gGrDa3de6+37gfuDSPNepW7j72+7+QvR9FyEIjCQc78+jYj8HPpGfGnYPM6sELgHuiuYNuAB4KCpSiMc8ADgH+AmAu+939x0U+LkmvNGvj5kVAX2BtynAc+3uTwPb2iw+2Lm9FLjHgyXAIDMb3pm/WyhBfySwMWu+NlpW0MysCpgEPAcc6+5vQ0gMwDH5q1m3uBX4W6Apmi8Hdrh7OpovxHN+HFAH/DTq1rrLzPpRwOfa3TcB/wRsIAT7ncAyCv9cZxzs3HZZjCuUoG/tLCvoy5LMrAz4NfDX7v5+vuvTnczsY8AWd1+WvbidooV2zouAycAd7j4J+IAC6sppT9SHfSkwFhgB9CN0bbRVaOe6I132771Qgn4tMCprvhLYnKe6dDszSxEC/r+7+2+ixe9mmnvR55Z81a8bnA3MMbP1hK67Cwi//AdFXQBQmOe8Fqh19+ei+YcISaCQz/WFwDp3r3P3BuA3wFkU/rnOONi57bIYVyhBfykwLhrhLyYM/CzIc526RdSX/RNgpbt/P2vVAuDq6PvVwH/0dN26i7vf4O6V7l5FOLf/5e5XAYuAT0fFCuqYAdz9HWCjmZ0ULfowsIICPteEbp0ZZtY3+reeOeaCPtdZDnZuFwB/EV3FMwPYmekGOmzuXhATcDHwJrAG+Ea+69ONx/khQrPuFeClaLqY0Mf9JLAq+hyS77p20/GfB/xn9P044HlgNfAgUJLv+nXD8U4EaqLz/TtgcKGfa+DbwOvAa8AvgJJCPNfAfYRxiwbCL/nPH+zcErp3bo/i26uEq5s69Xd1R66ISIwUSveOiIjkQEFfRCRGFPRFRGJEQV9EJEYU9EVEYkRBX0QkRhT0RURiREFfRCRG/j+3P7T6virM+gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot history\n",
    "plt.plot(history.history['loss'], label='train')\n",
    "plt.plot(history.history['val_loss'], label='test')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Make Predictions and Plot ROC-AUC Metric"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate ROC-AUC for a single batch of test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_test, y_test = next(dg_validation)\n",
    "y_pred = model.predict(X_test).ravel()\n",
    "fpr, tpr, thresholds = roc_curve(y_test, y_pred)\n",
    "aucurve = auc(fpr, tpr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd4VGX2wPHvMQERxQaoYOg1BWQhEBBp0l2RoigWihvgh65gWVdlQVBUVhFEQbogiKIigqKyorKyKlVAQAgCMZSEIkWqIKSc3x9zE2NIyACZuZmZ83meeXLvzDtzzyUhJ+/73nteUVWMMcYYgIvcDsAYY0zhYUnBGGNMFksKxhhjslhSMMYYk8WSgjHGmCyWFIwxxmSxpGCMMSaLJQUTdERku4icFJHjIrJXRKaLyGU52twoIv8VkWMickREPhGRqBxtLheRV0Vkp/NZic5+Kf+ekTH+Y0nBBKsOqnoZUAf4CzAw8wURaQR8AXwMlAUqAeuAJSJS2WlTFFgERAPtgMuBG4GDQANfBS0i4b76bGO8YUnBBDVV3QssxJMcMo0A3lLV11T1mKr+qqqDgeXAM06bHkB5oLOqJqhqhqruU9XnVHVBbscSkWgR+VJEfhWRX0TkX87z00Xk+WztmotISrb97SLypIisB34TkcEiMifHZ78mImOc7StEZKqI7BGRXSLyvIiEXeA/lTGAJQUT5EQkAmgPJDr7xfH8xf9BLs1nA62d7VbA56p63MvjlAC+Aj7H0/uoiqen4a27gb8CVwIzgVtE5HLns8OAO4FZTtsZQJpzjL8AbYDe53AsY/JkScEEq49E5BiQDOwDhjrPX43n535PLu/ZA2TOF5TMo01ebgX2quooVf3d6YGsOIf3j1HVZFU9qao7gDVAJ+e1m4ETqrpcRK7Fk+QeUdXfVHUfMBrodg7HMiZPlhRMsOqkqiWA5kBN/vhlfwjIAMrk8p4ywAFn+2AebfJSDvj5vCL1SM6xPwtP7wHgHv7oJVQAigB7ROSwiBwGJgHXXMCxjcliScEENVX9HzAdGOns/wYsA7rm0vxO/hjy+QpoKyKXenmoZKBKHq/9BhTPtn9dbqHm2P8AaO4Mf3Xmj6SQDJwCSqnqlc7jclWN9jJOY87KkoIJBa8CrUUkc7L5KaCniAwQkRIicpUzEdwIeNZpMxPPL+APRaSmiFwkIiVF5F8icksux/gUuE5EHhGRi53PjXNeW4tnjuBqEbkOeCS/gFV1P7AYeBPYpqqbnOf34LlyapRzyexFIlJFRJqdx7+LMWewpGCCnvML9i3gaWf/O6At0AXPvMEOPBO2N6nqVqfNKTyTzT8BXwJHgZV4hqHOmCtQ1WN4Jqk7AHuBrUAL5+WZeC553Y7nF/r7XoY+y4lhVo7newBFgQQ8w2FzOLehLmPyJLbIjjHGmEzWUzDGGJPFkoIxxpgslhSMMcZksaRgjDEmS8AV3ypVqpRWrFjR7TCMMSagrF69+oCqls6vXcAlhYoVK7Jq1Sq3wzDGmIAiIju8aWfDR8YYY7JYUjDGGJPFkoIxxpgsATenkJvU1FRSUlL4/fff3Q7FOIoVK0ZERARFihRxOxRjzDkIiqSQkpJCiRIlqFixIiLidjghT1U5ePAgKSkpVKpUye1wjDHnwGfDRyIyTUT2iciGPF4XERnjLIa+XkTqnu+xfv/9d0qWLGkJoZAQEUqWLGk9N2MCkC/nFKbjWfA8L+2Bas6jLzDhQg5mCaFwse+HMYHJZ8NHqvqNiFQ8S5OOeBZPV2C5iFwpImWcevHGGD+YtWInH6/d5XYYJh8ZGemcPp1K3crXMLSDb9dTcvPqo+v58xKEKc5zZxCRviKySkRW7d+/3y/BBYPVq1dTq1YtqlatyoABA8irTPrixYupU6cO0dHRNGv2x1othw8f5o477qBmzZpERkaybNkyANatW0ejRo2oVasWHTp04OjRo345H1PwPl67i4Q99v0rzA4fPsz3369i48aNef4fLlCq6rMHUBHYkMdrn+FZ1CRzfxFQL7/PrFevnuaUkJBwxnOFUVpaml+PV79+fV26dKlmZGRou3btdMGCBWe0OXTokEZGRuqOHTtUVfWXX37Jeq1Hjx46ZcoUVVU9deqUHjp0SFVVY2NjdfHixaqqOnXqVB08eHCuxw+U70sou3PiUr1z4lK3wzC5OHTokPbu3VsBrVq1atb/ufMFrFIvfm+72VNIwbPYeaYIYLdLsVywTp06Ua9ePaKjo5k8eXLW85dddhlDhgwhLi6OZcuWsXr1apo1a0a9evVo27Yte/Z4RsumTJlC/fr1ueGGG7j99ts5ceLEBcWzZ88ejh49SqNGjRARevTowUcffXRGu1mzZtGlSxfKly8PwDXXeNZ/P3r0KN988w3x8fEAFC1alCuvvBKAzZs307RpUwBat27Nhx9+eEGxGmP+LD09nRtvvJFp06bxxBNPsH79+j/14n3JzUtS5wMPich7QBxwRAtgPuHZTzaSsLtgu8NRZS/Pdxxv2rRpXH311Zw8eZL69etz++23U7JkSX777TdiYmIYNmwYqampNGvWjI8//pjSpUvz/vvvM2jQIKZNm0aXLl3o06cPAIMHD2bq1Kn079//T8f4+uuvefTRR884dvHixVm6dOmfntu1axcRERFZ+xEREezadebY8ZYtW0hNTaV58+YcO3aMhx9+mB49epCUlETp0qW5//77WbduHfXq1eO1117j0ksvJSYmhvnz59OxY0c++OADkpOTz/hcY8y5O3jwIFdffTVhYWG88MILlCtXjtjYWL/G4LOkICLvAs2BUiKSAgwFigCo6kRgAXALkAicAO73VSz+MGbMGObNmwdAcnIyW7dupWTJkoSFhXH77bcDnr+wN2zYQOvWrQHPXwNlyniW1t2wYQODBw/m8OHDHD9+nLZt255xjBYtWrB27Vqv4tFcxh5zuyIoLS2N1atXs2jRIk6ePEmjRo1o2LAhaWlprFmzhrFjxxIXF8fDDz/Miy++yHPPPce0adMYMGAAw4YN47bbbqNo0aLe/SMZY3KlqrzzzjtZ/8/69OlD586dXYnFl1cf3Z3P6wr8vaCP6+uZ+dwsXryYr776imXLllG8eHGaN2+edY1+sWLFCAsLAzzf+Ojo6KwJ2+x69erFRx99xA033MD06dNZvHjxGW3OpacQERFBSkpK1n5KSgply5Y9470RERGUKlWKSy+9lEsvvZSmTZuybt06mjRpQkREBHFxcQDccccdvPjiiwDUrFmTL774AvD0ND777DNv/pmMMblITk6mX79+LFiwgIYNG9K4cWNX47HaRwXgyJEjXHXVVRQvXpyffvqJ5cuX59quRo0a7N+/PysppKamsnHjRgCOHTtGmTJlSE1N5Z133sn1/Zk9hZyPnAkBoEyZMpQoUYLly5ejqrz11lt07NjxjHYdO3bk22+/JS0tjRMnTrBixQoiIyO57rrrKFeuHJs3bwZg0aJFREVFAbBv3z4AMjIyeP755+nXr985/osZYwDeffddoqOjWbx4Ma+++irfffdd1v8ztwRFmQu3tWvXjokTJ1K7dm1q1KhBw4YNc21XtGhR5syZw4ABAzhy5AhpaWk88sgjREdH89xzzxEXF0eFChWoVasWx44du+C4JkyYQK9evTh58iTt27enffv2AEycOBGAfv36ERkZSbt27ahduzYXXXQRvXv3JiYmBoCxY8dy7733cvr0aSpXrsybb74JeH6Qx40bB0CXLl24//6AHvkzxjVXXXUVcXFxTJ48udCUhJHcxp4Ls9jYWM25yM6mTZuIjIx0KSKTF/u+FH53TfL0Wt//v0YuRxIa0tLSGD16NKdPn2bQoEGAZ1jZHxUARGS1quY7a23DR8YY4wfr1q2jYcOGWZeYZv5BXthKwlhSMMYYHzp16hRPP/00sbGxJCcn88EHH/Dee+8VumSQKWiSQqANgwU7+34Y47F161Zeeukl7rnnHhISErjjjjsKbUKAIEkKxYoV4+DBg/aLqJBQZz2FYsWKuR2KMa44fvx41lWEMTEx/PTTT8yYMYOSJUu6HFn+guLqo8xr8q1YXuGRufKaMaHmyy+/pG/fvuzYsYO6desSGRlJ5cqV3Q7La0GRFIoUKVJoLucyxoSmQ4cO8fjjjzNt2jSqV6/O//73v4C8+i4okoIxxrgpPT2dxo0bs2XLFgYOHMiQIUMCdvjUkoIxxpynAwcOZBWwGz58OOXLl6du3fNeWbhQCIqJZmOM8afM0jHVq1fnjTfeADzl8wM9IYAlBWOMOSc7duygffv29OzZk8jIyKy1RYKFJQVjjPHS22+/TUxMDN999x1jx47l22+/pWbNmm6HVaBsTsEYY7xUunRpGjduzKRJk6hQoYLb4fiEJQVjjMlDamoqo0aNIjU1laeffpq2bdvSpk2bQn1H8oWy4SNjjMnFDz/8QFxcHAMHDiQhIaHQFrAraJYUjDEmm99//51//etf1K9fn927d/Phhx/y7rvvBn0yyGRJwRhjsklMTGTkyJH06NGDTZs20aVLF7dD8iubUzDGhLzjx48zb948unfvTkxMDJs3bw7Z0jnWUzDGhLSFCxcSHR1Nz5492bRpE0DIJgSwpGCMCVEHDx6kZ8+etGvXjuLFi/Ptt98GZAG7gmbDR8aYkJNZwC4xMZFBgwYxePDggC1gV9AsKRhjQsb+/fspWbIkYWFhvPTSS1SoUIE6deq4HVahYsNHxpigp6q8+eabVK9enSlTpgDQsWNHSwi5sKRgjAlq27dvp23btvztb3+jVq1atGjRwu2QCjVLCsaYoDVz5kxiYmJYtmwZ48ePZ/HixVSvXt3tsAo1m1MwxgSta6+9lqZNmzJx4kTKly/vdjgBwZKCMSZopKamMmLECNLT0xkyZAht2rShTZs2bocVUGz4yBgTFNasWUP9+vUZPHgwmzdvzipgZ86N9RSMCXCzVuzk47W7zuu9CXuOElXm8gKOyL9OnjzJs88+y8iRIyldujTz5s2jU6dObocVsHzaUxCRdiKyWUQSReSpXF4vLyJfi8gPIrJeRG7xZTzGBKOP1+4iYc/R83pvVJnL6Vjn+gKOyL+SkpJ45ZVX6NWrFwkJCZYQLpDPegoiEgaMA1oDKcD3IjJfVROyNRsMzFbVCSISBSwAKvoqJmOCVVSZy3n//xq5HYbfHD16lLlz59KrVy+io6PZunVr0K6E5m++7Ck0ABJVNUlVTwPvAR1ztFEgs+96BbDbh/EYY4LAggULiImJIT4+PquAnSWEguPLpHA9kJxtP8V5LrtngPtEJAVPL6F/bh8kIn1FZJWIrNq/f78vYjXGFHIHDhyge/fu/PWvf6VEiRIsWbLECtj5gC+TQm7LFOW8HOBuYLqqRgC3ADNF5IyYVHWyqsaqamzp0qV9EKoxpjDLLGD33nvvMWTIENasWUPDhg3dDiso+fLqoxSgXLb9CM4cHooH2gGo6jIRKQaUAvb5MC5jTID45ZdfKF26NGFhYYwcOZIKFSpQu3Ztt8MKar7sKXwPVBORSiJSFOgGzM/RZifQEkBEIoFigI0PGRPiVJWpU6dSo0YNJk+eDECHDh0sIfiBz5KCqqYBDwELgU14rjLaKCLDROQ2p9k/gD4isg54F+ildseJMSEtKSmJVq1a0bt3b+rUqUOrVq3cDimk+PTmNVVdgGcCOftzQ7JtJwCNfRmDMSZwzJgxgwcffJCwsDAmTpxInz59uOgiK7zgT3ZHszGm0Chbtiw333wzEyZMICIiwu1wQpIlBWMKgVAtVXH69GlefPFFMjIyeOaZZ2jdujWtW7d2O6yQZv0yYwqBUCxV8f3331OvXj2GDh1KUlKSFbArJKynYEwhESqlKk6cOMGQIUMYPXo0ZcqUYf78+XTo0MHtsIzDegrGGL/atm0bY8eOpU+fPmzcuNESQiFjPQVjjM8dOXKEuXPncv/99xMdHU1iYiLlypXL/43G76ynYIzxqc8++4zo6Gh69+7NTz/9BGAJoRCznoIxBeBCrh6CwL6CKC/79+/nkUceYdasWcTExDB37lxq1qzpdlgmH5YUjCkAmVcPne8v9kC9gigv6enp3HTTTWzbto1nn32Wp556iqJFi7odlvGCJQVjCkioXD10Nnv37uWaa64hLCyMUaNGUbFiRWJiYtwOy5wDm1MwxlywjIwMJk2aRPXq1Zk0aRIAt956qyWEAJRvUhCRS0RkoIhMdParikh734dmjAkEiYmJtGzZkn79+lG/fn3atm3rdkjmAnjTU5iGZ8Gcm5z93cBwn0VkjAkYb775JrVq1WLNmjVMmTKFr776isqVK7sdlrkA3iSFaqo6HEgFUNUT5L6qmjEmxJQvX562bduSkJBA7969EbFfDYHOm4nm086KaAogIpWA0z6NyhhTKJ06dYp///vfZGRkMGzYMFq2bEnLli3dDssUIG+SwnPA50CEiMwAmgG9fRqVMS4I1Uql3lqxYgXx8fFs3LiRnj17oqrWMwhC+Q4fqep/gK5AH2Ae0EBVv/J1YMb4WyhWKvXGb7/9xmOPPUajRo04cuQIn376KdOnT7eEEKTy7SmIyBeq2gb4OJfnjAkqdq/BmXbs2MH48ePp168fL774IpdfHtw9olCXZ1IQkaJAMeBaESnBH5PLlwPl/RCbMcYlhw8fZs6cOfTu3ZuoqCgSExNtJbQQcbbho78DG4GaztfMx0Jgou9DM8a44eOPPyYqKop+/fplFbCzhBA68kwKqjpaVcsBT6pqeVUt5zyiVfVVP8ZojPGDffv20a1bNzp16kTp0qVZvny5FbALQfnOKajqqyJSE4jCM5yU+fwsXwZmjPGf9PR0GjduzM6dO3n++ed54oknKFKkiNthGRd4M9E8GGiDZxhpIdAW+A6wpGBMgNu9ezfXXXcdYWFhvPbaa1SsWJGoqCi3wzIu8uaO5ruAFsAeVe0O3IBVVzUmoGVkZDBhwgRq1qzJxImeKcJbbrnFEoLxKimcVNV0IM25CmkvYMVNjAlQW7ZsoUWLFjz44IPExcXRvr3VtzR/8CYp/CAiV+IpjLcKWAms8WlUxhifmDp1KjfccAPr169n2rRpfPHFF1SqVMntsEwhctZhIPHcsviMqh4GxonIQuByVbWkYEwAqlixIu3bt2fcuHGUKVPG7XBMIXTWpKCqKiKfAvWc/US/RGWMKRCnTp3iueeeA+D555+3AnYmX94MH60Ukbo+j8QYU6CWLl1KnTp1eOGFF9izZw+q6nZIJgB4kxRuwpMYNovIGhH5QURs+MiYQur48eM8/PDD3HTTTZw4cYLPP/+cqVOnWgE74xVvLi3tdL4fLiLtgNeAMOANVX0xlzZ3As/gWa9hnarec77HM8bAzp07mTRpEn//+98ZPnw4JUqUcDskE0C8uaP55/P5YBEJA8YBrYEU4HsRma+qCdnaVAMGAo1V9ZCIXHM+xzIm1B06dIgPPviAvn37EhUVRVJSEmXLlnU7LBOAvBk+Ol8NgERVTVLV08B7QMccbfoA41T1EICq7vNhPMYEpXnz5hEVFcWDDz7I5s2bASwhmPPmy6RwPZCcbT/FeS676kB1EVkiIsud4aYziEhfEVklIqv279/vo3CNCSx79+6la9eudOnSheuuu46VK1dSo0YNt8MyAc6rchUiEgFUU9WvReRiIFxVf8vvbbk8l/Pyh3CgGtAciAC+FZEY576IP96kOhmYDBAbG2uXUJiQl56eTpMmTUhOTmb48OE8/vjjVsDOFAhvCuL9DXgIuAKoAlQAxgOt8nlrClAu234EsDuXNstVNRXYJiKb8SSJ772K3pgcgn2d5ZSUFMqWLUtYWBhjxoyhUqVKVt7aFChvho8GAA2BowCqugXwZkL4e6CaiFRyVnHrBszP0eYjPMX2EJFSeIaTkrwL3ZgzBes6yxkZGYwdO5aaNWsyYcIEANq3b28JwRQ4b4aPflfV05nXODtXFeV7wbOqponIQ3jKbYcB01R1o4gMA1ap6nzntTYikgCkA/9U1YPneS7GAMG3zvJPP/1E7969WbJkCW3btuXWW291OyQTxLxJCktE5AmgmIi0wLNM56fefLiqLgAW5HhuSLZtBR5zHsaYHN544w0eeughihcvzowZM+jevbvdhGZ8ypvhoyeAY8BPwMPAImCQL4MyxnhUqVKFDh06sGnTJnr06GEJwficNz2FW/DcjTzB18EYE+p+//13hg0bBsDw4cNp0aIFLVq0cDkqE0q86SncCSSKyJsi0taZUzDGFLAlS5ZQp04d/v3vf7N//34rYGdckW9ScJbgrA58AvwNSBKRib4OzJhQcezYMfr370+TJk04deoUCxcuZMqUKTZUZFzh1R3NqnoK+BiYjudS0zt9GJMxISUlJYU33niD/v378+OPP9KmTRu3QzIhLN+kICKtROQN4GfgPuAt4DpfB2ZMMDt48GDW/QaRkZEkJSXx2muvcdlll7kcmQl13vQU+gGfA5Gqeq+qzncK3BljzpGqMmfOHKKiohgwYEBWATtbGtMUFt7MKdyhqnNU9aQ/AjImWO3Zs4fbb7+drl27Uq5cOVatWmUF7Eyhk+clqSLyP1VtJiKH+HMhO8Fz39nVPo/OmCCRWcBu165djBgxgkcffZTwcK/qURrjV2f7qcy8OLqUPwIxJhglJydz/fXXExYWxrhx46hUqRLVq1d3Oyxj8pTn8JGqZjibU1U1PfsDmOqf8IwJTOnp6YwZM+ZPBezatm1rCcEUet70X2tn33FuXqvvm3CMCXybNm0iPj6eZcuW0b59ezp06OB2SMZ4Lc+egog86cwn1BaRX53HIWA/OYrcGWM8Jk+eTJ06ddiyZQszZ87ks88+o3z58m6HZYzXznb10QigNDDa+VoaKKWqV6vqP/0RnDGBplq1anTu3JmEhATuu+8+uyvZBJyzDR9VVdWtIjITiM58MvOHXFXX+zg2Ywq9kydP8swzzyAivPjii1bAzgS8syWFp4B4YFwurynQ1CcRGRMgvvnmG3r37s3WrVvp168fqmo9AxPw8kwKqhrvfG3iv3CMKfzrLB89epSnnnqKCRMmULlyZRYtWsTNN9/s02Ma4y/e1D7qIiIlnO2nRGS2iNzg+9BMqCrs6yzv3r2b6dOn89hjj7F+/XpLCCaoeHNJ6jOqOldEbgQ6AK8Ak4CGPo3MhLTCts7ygQMHmD17Ng8++CA1a9Zk27ZtXHvttW6HZUyB8yYppDtfbwXGq+qHIjLYhzGZIFDYh4C8parMnj2b/v37c/jwYVq1akX16tUtIZig5U2V1D0iMg7oBiwQkaJevs+EsMI+BOSN3bt306lTJ7p160aFChVYvXq13ZFsgp43PYU78azTPFZVD4lIWTxXJhlzVoVtCOhcpKen07RpU3bt2sXIkSN5+OGHrYCdCQn5/pSr6nERSQCai0hz4FtV/Y/PIzPGBTt27CAiIoKwsDDGjx9P5cqVqVq1qtthGeM33lx99BAwGyjvPGaLyIO+DswYf0pPT+eVV14hMjIyq4BdmzZtLCGYkONNf7gv0EBVjwOIyHBgKTDel4EZ4y8bNmwgPj6elStXcuutt9KpUye3QzLGNd5MGAuQmm0/1XnOmIA3ceJE6tatS1JSErNmzWL+/PlERES4HZYxrvGmpzATWC4iH+JJBp2AGT6NyhgfyyxJERkZSdeuXXn11VcpXbq022EZ4zpvJppHiMjXQGa5i36q+r1vwzJuu5D7DKBw3WuQ3YkTJxgyZAhhYWG89NJLNGvWjGbNmrkdljGFhrf3G5xyHiedrybIXch9BlB47jXIbvHixdSuXZtRo0Zx/PhxVDX/NxkTYvLtKYjIIOAeYB6e4aNZIvKOqv7b18EZdwXyfQbZHTlyhCeeeILJkydTpUoV/vvf/1p5a2Py4M2cwn1APVU9ASAiLwCrAUsKJiDs2bOHt99+m8cff5xnn32W4sWLux2SMYWWN8NHO/hz8ggHkrz5cBFpJyKbRSRRRPK8C1pE7hARFZFYbz7XmPzs37+fsWPHAlCzZk22b9/Oyy+/bAnBmHx4kxROABtF5A0RmQL8CBwWkVdE5JW83iQiYXgW6GkPRAF3i0hULu1KAAOAFedzAsZkp6rMmjWLyMhI/vGPf7BlyxYAu7LIGC95M3z0mfPItNzLz24AJKpqEoCIvAd0BBJytHsOz3rQj3v5ucbkKjk5mQceeIDPPvuMuLg4pk6dagXsjDlH3lySOvU8P/t6IDnbfgoQl72BiPwFKKeqn4pInklBRPriubOa8uXLn2c4JpilpaXRvHlz9u7dy+jRo+nfvz9hYWFuh2VMwPFl2cfc7nrOugZQRC4CRgO98vsgVZ0MTAaIjY216whNlu3bt1OuXDnCw8OZNGkSlStXpnLlym6HZUzA8uW6CClAuWz7EcDubPslgBhgsYhsx7OS23ybbDbeSEtLY+TIkURGRjJ+vKcMV6tWrSwhGHOBvO4piMjFqnouN659D1QTkUrALjyL9NyT+aKqHgFKZfv8xcDjqrrqHI5hQtD69euJj49n1apVdOzYkdtvv93tkIwJGt6Uzm4gIj8CW539G0RkbH7vU9U04CFgIbAJmK2qG0VkmIjcdoFxmxA1fvx46tWrx44dO3j//feZN28eZcuWdTssY4KGNz2FMXjWZ/4IQFXXiYhXt4Oq6gJgQY7nhuTRtrk3n2lCU2YBu5iYGLp168bo0aMpVapU/m80xpwTb5LCRaq6Q+RP88bpPorHmD/57bffGDx4MOHh4bz88ss0bdqUpk2buh2WMUHLm6SQLCINAHVuSOsPbPFtWKYgXEil08JQ5XTRokX06dOHbdu20b9//6zegjHGd7y5+ugB4DE8S3H+gucqoQd8GZQpGBdS6dTNKqeHDx+md+/etGrVivDwcL755hvGjBljCcEYP/Dm5rV9eK4cMgEoECud/vLLL7z33ns8+eSTDB06lEsuucTtkIwJGd6Uzp5CtpvOMqlqX59EZEJSZiJ4+OGHqVGjBtu3b7eJZGNc4M3w0VfAIuexBLgGW2jHFBBV5e233yYqKoonnniCrVu3AlhCMMYl3gwfvZ99X0RmAl/6LCITMnZlEFWdAAASTElEQVTu3Em/fv34z3/+Q6NGjZg6dSrVqlVzOyxjQtr51D6qBFQo6EBM7gL9CqK8ZBaw27dvH2PGjOHBBx+0AnbGFALezCkc4o85hYuAX4E8F8wxBSvzCqLz+eVeGNdJTkpKokKFCoSHhzNlyhSqVKlCxYoV3Q7LGOM4a1IQzzWAN+CpXQSQobbaud8F4hVEOaWlpTFq1CiGDh3KiBEjGDBgAC1btnQ7LGNMDmdNCqqqIjJPVev5KyATfNauXUt8fDxr1qyhc+fOdO3a1e2QjDF58Obqo5UiUtfnkZig9Prrr1O/fn127drFnDlzmDt3LmXKlHE7LGNMHvLsKYhIuFPp9Cagj4j8DPyGZ/EcVVVLFCZPmSUpateuzb333ssrr7zC1Vdf7XZYxph8nG34aCVQF+jkp1iCVrBeQZSb48ePM2jQIIoUKcLIkSOtgJ0xAeZsw0cCoKo/5/bwU3xBIVBrEJ2rL774gpiYGMaOHUtqaip2TYIxgedsPYXSIvJYXi+q6is+iCdoBcMVRHk5dOgQjz32GNOnT6dGjRp888033HTTTW6HZYw5D2frKYQBl+FZSzm3hzEA7Nu3jzlz5jBw4EDWrl1rCcGYAHa2nsIeVR3mt0hMQNm7dy/vvvsujz76aFYBu5IlS7odljHmAuU7p2BMdqrKjBkziIqKYuDAgVkF7CwhGBMczpYU7HZT8yfbt2+nXbt29OrVi6ioKNauXWsF7IwJMnkOH6nqr/4MxBRuaWlptGjRggMHDjBu3Dj69evHRRd5c++jMSaQnE+VVBNCEhMTqVSpEuHh4UybNo3KlStToYIVyTUmWNmfeiZXqampDB8+nOjoaMaNGwdAixYtLCEYE+Ssp2DOsGbNGuLj41m7di1du3blrrvucjskY4yfWE/B/MmYMWNo0KABe/fuZe7cucyePZtrr73W7bCMMX5iScEAZJWk+Mtf/kKPHj1ISEigc+fOLkdljPE3Gz4KcceOHWPgwIFcfPHFjBo1iiZNmtCkSRO3wzLGuMSSgpeCsdLp559/zv/93/+RnJzMI488klXu2hgTumz4yEvBVOn04MGD9OzZk/bt23PppZeyZMkSXnnlFUsIxhjrKZyLYKl0evDgQebNm8fTTz/NoEGDuPjii90OyRhTSPi0pyAi7URks4gkishTubz+mIgkiMh6EVkkInYRvI/s2bOHkSNHoqpUr16dHTt2MGzYMEsIxpg/8VlSEJEwYBzQHogC7haRqBzNfgBiVbU2MAcY4at4QpWqMm3aNCIjI3n66adJTEwE4KqrrnI5MmNMYeTL4aMGQKKqJgGIyHtARyAhs4Gqfp2t/XLgPl8FcyETxVB4J4vPZtu2bfTt25evvvqKpk2bMmXKFCtgZ4w5K18OH10PJGfbT3Gey0s88J/cXhCRviKySkRW7d+//7yCuZCJYih8k8X5SUtL4+abb2bFihVMmDCBr7/+murVq7sdljGmkPNlTyG3S1lyXbRXRO4DYoFmub2uqpOByQCxsbHnvfBvsEwUn83WrVupXLky4eHhvPnmm1SpUoVy5cq5HZYxJkD4sqeQAmT/bRQB7M7ZSERaAYOA21T1lA/jCWqpqak8//zzxMTE8PrrrwPQvHlzSwjGmHPiy57C90A1EakE7AK6AfdkbyAifwEmAe1UdZ8PYwlqq1atIj4+nvXr19OtWzfuvvtut0MyxgQon/UUVDUNeAhYCGwCZqvqRhEZJiK3Oc1eBi4DPhCRtSIy31fxBKvXXnuNuLg4Dhw4wMcff8y7777LNddc43ZYxpgA5dOb11R1AbAgx3NDsm238uXxg1lmSYrY2Fji4+MZMWIEV155pdthGWMCnN3RHGCOHj3Kk08+SbFixRg9ejSNGzemcePGbodljAkSVvsogCxYsIDo6GgmT55MeHh4VrlrY4wpKJYUAsCBAwe47777+Otf/8oVV1zB0qVLefnll62AnTGmwFlSCACHDh3ik08+YejQoaxZs4a4uDi3QzLGBCmbUyikdu3axTvvvMM///lPqlWrxo4dO2wi2Rjjc9ZTKGRUlSlTphAVFcUzzzzDzz//DGAJwRjjF5YUCpGff/6Zli1b0rdvX+rWrcv69eupWrWq22EZY0KIDR8VEmlpabRs2ZJff/2VSZMm0bt3by66yHK2Mca/LCm4bPPmzVSpUoXw8HBmzJhBlSpViIiIcDssY0yIsj9FXXL69GmeffZZatWqxbhx4wBo1qyZJQRjjKusp+CClStXEh8fz4YNG7jnnnu499573Q7JGGMA6yn43auvvkqjRo2y7j145513KFWqlNthGWMMYEnBbzJLUjRo0IA+ffqwceNGbr31VpejMsaYP7PhIx87cuQITzzxBJdccgmvvvoqN954IzfeeKPbYRljTK6sp+BDn3zyCVFRUbzxxhtcfPHFVsDOGFPoWVLwgf3793PPPfdw2223UbJkSZYvX85LL71kBeyMMYWeJQUfOHLkCAsWLODZZ59l1apV1K9f3+2QjDHGKzanUECSk5N5++23eeqpp6hatSo7duzgiiuucDssY4w5J9ZTuEAZGRlMnDiR6Ohonn/++awCdpYQjDGByJLCBdi6dSs333wzDzzwAA0aNODHH3+0AnbGmIBmw0fnKS0tjdatW3P48GGmTp3K/fffbxPJxpiAZ0nhHG3atIlq1aoRHh7OzJkzqVKlCmXLlnU7LGOMKRA2fOSlU6dOMXToUGrXrs3rr78OQJMmTSwhGGOCivUUvLB8+XLi4+NJSEige/fudO/e3e2QjDHGJ6ynkI9Ro0Zx4403cuzYMRYsWMBbb71FyZIl3Q7LGGN8wpJCHjIyMgBo1KgR/fr1Y8OGDbRv397lqIwxxrds+CiHw4cP849//IPixYszduxYK2BnjAkp1lPI5qOPPiIqKooZM2ZQokQJK2BnjAk5lhSAffv2ceedd9K5c2euvfZaVq5cyfDhw+2+A2NMyLGkABw9epQvv/ySF154gZUrV1K3bl23QzLGGFeE7JzCzp07mTlzJv/617+oWrUqO3fupESJEm6HZYwxrvJpT0FE2onIZhFJFJGncnn9YhF533l9hYhU9FUsUWUvJ6rs5WRkZDB+/Hiio6MZPnx4VgE7SwjGGOPDpCAiYcA4oD0QBdwtIlE5msUDh1S1KjAaeMlX8QztEE236uE0b96cv//97zRq1IiNGzdaATtjjMnGlz2FBkCiqiap6mngPaBjjjYdgRnO9hygpfhodjctLY22bdvy448/8uabb7Jw4UIqVqzoi0MZY0zA8uWcwvVAcrb9FCAurzaqmiYiR4CSwIHsjUSkL9AXoHz58ucVTHh4OG+//TZVqlShTJky5/UZxhgT7HzZU8jtL/6cF/570wZVnayqsaoaW7p06fMO6KabbrKEYIwxZ+HLpJAClMu2HwHszquNiIQDVwC/+jAmY4wxZ+HLpPA9UE1EKolIUaAbMD9Hm/lAT2f7DuC/arcRG2OMa3w2p+DMETwELATCgGmqulFEhgGrVHU+MBWYKSKJeHoI3XwVjzHGmPz59OY1VV0ALMjx3JBs278DXX0ZgzHGGO9ZmQtjjDFZLCkYY4zJYknBGGNMFksKxhhjskigXQEqIvuBHef59lLkuFs6BNg5hwY759BwIedcQVXzvfs34JLChRCRVaoa63Yc/mTnHBrsnEODP87Zho+MMcZksaRgjDEmS6glhcluB+ACO+fQYOccGnx+ziE1p2CMMebsQq2nYIwx5iwsKRhjjMkSlElBRNqJyGYRSRSRp3J5/WIRed95fYWIVPR/lAXLi3N+TEQSRGS9iCwSkQpuxFmQ8jvnbO3uEBEVkYC/fNGbcxaRO53v9UYRmeXvGAuaFz/b5UXkaxH5wfn5vsWNOAuKiEwTkX0isiGP10VExjj/HutFpG6BBqCqQfXAU6b7Z6AyUBRYB0TlaPMgMNHZ7ga873bcfjjnFkBxZ/uBUDhnp10J4BtgORDrdtx++D5XA34ArnL2r3E7bj+c82TgAWc7CtjudtwXeM5NgbrAhjxevwX4D56VKxsCKwry+MHYU2gAJKpqkqqeBt4DOuZo0xGY4WzPAVqKSG5LgwaKfM9ZVb9W1RPO7nI8K+EFMm++zwDPASOA3/0ZnI94c859gHGqeghAVff5OcaC5s05K3C5s30FZ67wGFBU9RvOvgJlR+At9VgOXCkiBbbOcDAmheuB5Gz7Kc5zubZR1TTgCFDSL9H5hjfnnF08nr80Alm+5ywifwHKqeqn/gzMh7z5PlcHqovIEhFZLiLt/Badb3hzzs8A94lICp71W/r7JzTXnOv/93Pi00V2XJLbX/w5r7v1pk0g8fp8ROQ+IBZo5tOIfO+s5ywiFwGjgV7+CsgPvPk+h+MZQmqOpzf4rYjEqOphH8fmK96c893AdFUdJSKN8KzmGKOqGb4PzxU+/f0VjD2FFKBctv0IzuxOZrURkXA8Xc6zddcKO2/OGRFpBQwCblPVU36KzVfyO+cSQAywWES24xl7nR/gk83e/mx/rKqpqroN2IwnSQQqb845HpgNoKrLgGJ4CscFK6/+v5+vYEwK3wPVRKSSiBTFM5E8P0eb+UBPZ/sO4L/qzOAEqHzP2RlKmYQnIQT6ODPkc86qekRVS6lqRVWtiGce5TZVXeVOuAXCm5/tj/BcVICIlMIznJTk1ygLljfnvBNoCSAikXiSwn6/Rulf84EezlVIDYEjqrqnoD486IaPVDVNRB4CFuK5cmGaqm4UkWHAKlWdD0zF08VMxNND6OZexBfOy3N+GbgM+MCZU9+pqre5FvQF8vKcg4qX57wQaCMiCUA68E9VPehe1BfGy3P+BzBFRB7FM4zSK5D/yBORd/EM/5Vy5kmGAkUAVHUinnmTW4BE4ARwf4EeP4D/7YwxxhSwYBw+MsYYc54sKRhjjMliScEYY0wWSwrGGGOyWFIwxhiTxZKCKbREJF1E1mZ7VDxL24p5VZX0NxGJFZExznZzEbkx22v9RKSHH2OpE+hVQ41/Bd19CiaonFTVOm4Hca6cG+Qyb5JrDhwHljqvTSzo44lIuFPDKzd18JQ1WVDQxzXByXoKJqA4PYJvRWSN87gxlzbRIrLS6V2sF5FqzvP3ZXt+koiE5fLe7SLyktNupYhUdZ6vIJ51KDLXoyjvPN9VRDaIyDoR+cZ5rrmIfOr0bPoBjzrHbCIiz4jI4yISKSIrc5zXeme7noj8T0RWi8jC3Cpgish0EXlFRL4GXhKRBiKyVDxrCiwVkRrOHcDDgLuc498lIpeKp17/907b3CrLmlDmdu1we9gjrweeO3LXOo95znPFgWLOdjU8d7UCVMSpPw+MBe51tosClwCRwCdAEef58UCPXI65HRjkbPcAPnW2PwF6Ott/Az5ytn8Erne2r3S+Ns/2vmeAx7N9fta+c16Vne0ngcF47lxdCpR2nr8Lz128OeOcDnwKhDn7lwPhznYr4ENnuxfwerb3DQfuy4wX2AJc6vb32h6F52HDR6Ywy234qAjwuojUwZM0qufyvmXAIBGJAOaq6lYRaQnUA753ynxcAuRVA+rdbF9HO9uNgC7O9kw8azQALAGmi8hsYO65nByeIm53Ai/i+eV/F1ADTyG/L504w4C86tp8oKrpzvYVwAynV6Q4ZRFy0Qa4TUQed/aLAeWBTecYuwlSlhRMoHkU+AW4Ac/w5xmL56jqLBFZAfwVWCgivfGUG56hqgO9OIbmsX1GG1XtJyJxzrHWOsnKW+/jqUU11/NRulVEagEbVbWRF+//Ldv2c8DXqtrZGbZanMd7BLhdVTefQ5wmhNicggk0VwB71FMrvzuev6T/REQqA0mqOgZPRcnawCLgDhG5xmlzteS9TvVd2b4uc7aX8kfhxHuB75zPqaKqK1R1CHCAP5c0BjiGp4z3GVT1Zzy9nafxJAjwlLouLZ51ARCRIiISnUec2V0B7HK2e53l+AuB/uJ0Q8RTPdeYLJYUTKAZD/QUkeV4ho5+y6XNXcAGEVkL1MSzdGECnjH7L5wJ3S+BvJYwvNjpaTyMp2cCMAC433lvd+c1gJdF5Efncthv8KwhnN0nQOfMieZcjvU+cB9/rAdwGk8595dEZB2eeYczJtNzMQL4t4gs4c+J8msgKnOiGU+Pogiw3on5OS8+24QQq5JqTDbiWZAnVlUPuB2LMW6wnoIxxpgs1lMwxhiTxXoKxhhjslhSMMYYk8WSgjHGmCyWFIwxxmSxpGCMMSbL/wOC9e5S6NNZkQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(1)\n",
    "plt.plot([0, 1], [0, 1], 'k--')\n",
    "plt.plot(fpr, tpr, \n",
    "         label='area = {:.3f}'.format(aucurve))\n",
    "plt.xlabel('False positive rate')\n",
    "plt.ylabel('True positive rate')\n",
    "plt.title('ROC curve')\n",
    "plt.legend(loc='best')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5000 Records of Attack Data, 100 Epochs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "num_datarecs, num_testrecs = 5000, 1000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25, 5)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_size = params_train[\"batch_size\"]\n",
    "# subtract 100 to account for bad records that were discarded\n",
    "steps_per_epoch = int(np.floor((num_datarecs) / batch_size))\n",
    "validation_steps = int(np.floor((num_testrecs) / batch_size))\n",
    "steps_per_epoch, validation_steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Removing spew to make notebook render on Github\n",
    "history = model.fit_generator(generator = dg_train,\n",
    "                    validation_data = dg_validation,\n",
    "                    validation_steps = validation_steps,\n",
    "                    steps_per_epoch = steps_per_epoch,\n",
    "                    epochs=100, \n",
    "                    verbose=1,\n",
    "                    workers=1,\n",
    "                    max_queue_size=1, \n",
    "                    use_multiprocessing=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the training and test loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3X2UXHWd5/H399ZDP6S703noQEjAZIRFATFgyODDMOismugM6Kg8KIO468b1jGfd2cER9iAOnJmzzJk5LsMuoujgDiIiD8PILmFAXFD3yEMCRkwIkISJpAmQJkkn3emnqq7v/nHv7b5dXdVd3emHpO/ndU6duvW7D3UvFerTv9/v/n5l7o6IiEgw2ycgIiJHBwWCiIgACgQREYkoEEREBFAgiIhIRIEgIiKAAkFERCIKBBERARQIIiISyc72CUzE4sWLfcWKFbN9GiIix5RnnnnmTXdvG2+7YyoQVqxYwaZNm2b7NEREjilm9ttatlOTkYiIAAoEERGJKBBERARQIIiISESBICIigAJBREQiCgQREQGOsXEIk/bru2D/yxDkIMhAkIVMLnweWo7WJcvjbYMsWLwclL1OlI14JNfnwm1ERI5i6QiELffB9kdm9xwsCptMPnquG36drRsOpUweMtkKy7mR4TUq0DKJ8tzo/UbsG4dUtuy45duVHzcKOhGZk9IRCJ+5B9yhNAilIpQK4fNgcjl6jh+DRfDB4XU+CKVS+DouLxWHy5Ll5fvGz4MDief+cLvBgcSjED4GesLzil+XCsPnOuJ1VOalGfyPaaMDpNLrIBsFTzJ8EsuVamqVAm6816OCrFKoZYeDuHxdefiJpFg6AgHALPxCymSB+tk+m6k1FEqFkQE06nVx5LpkKI4KxEKVbasFarw8WOE4UXmhFwYPResGKx9nxPtE5TPGouCIwiObqMUN1ezyw49sXfjIN0Nd9Mjmh5sTy7dPvs7GZcn3KH+v6LWaHGWGpCcQ5rIggCAP5Gf7TKaee1gDqhRiIwIkDrXBCjWpuFaW2G5woCw4y7YbqsVFZcX+aPuoJjdwGHr2heUD3dDfFT7w6fnvkGxyrFrLqaGZcFQTYbLvrFJzYnnNLNnnVqEWGGTCx1AfW2b0e5XX6Mym57+ZTJgCQY5uZtGXS4ajvmZXKo1sToyb/KoGywAUk02GySbF5HKitldteVStcBCKfRXWVagRxss+ODv/3SyoECqVbtAof06sj/+NDN3wkbz5I1pvQVhuQaI8qPxIhtuIm0MqnUPZ+STPZcR7ZIYD14Lo38pguJxrgFxjGPjx518qRrXQ+vA5N2/aa4oKBJGpEgRAEP7leyxK9ofFzXxVmxcTzYPJ14NRn1b8ZVceOsmmwmR/24j1ySbDwZHvVRos65+L9+1NrC+N3K40GNU0B4fXx8v4cJDHtdF4v6PNnz4NbadO61soEEQkNJebHidjREAWywKrMBwuQzedlN1QUipGAVMaDqk4PL00XGvxUlibK/SE6+O7EC0YrlUW+6BpybRfsgJBRKSSFAakbl0QERFAgSAiIhEFgoiIAAoEERGJKBBERARQIIiISESBICIigAJBREQiNQWCma01sxfNbIeZXVVh/Xlm9qyZFc3sk4nyt5jZM2a22cy2mtl/TKx7l5n9JjrmTWaa4UpEZDaNGwhmlgFuBtYBpwGXmtlpZZu9AlwB3FlW/hrwHndfBfwucJWZnRCtuwVYD5wSPdZO8hpERGQK1FJDWAPscPeX3X0AuAu4MLmBu+9y9+eAUln5gLv3Ry/r4vczs6VAi7s/4e4O3A587MguRUREjkQtgbAM2J143R6V1cTMTjSz56Jj/I2774n2b5/sMUVEZOrVEgiV2vZr/hUQd9/t7mcCJwOfNbPjJnJMM1tvZpvMbFNHR0etbzvCLY/v5G8ffmFS+4qIpEUtgdAOnJh4vRzYM9E3imoGW4Hfi465vJZjuvut7r7a3Ve3tbVN9G0B2LhrPz97aXJhIiKSFrUEwkbgFDNbaWZ54BLggVoObmbLzawhWl4AvBd40d1fA7rM7Nzo7qLLgR9P6gpqkA2M4uA0/bShiMgcMW4guHsR+BLwMLANuNvdt5rZ9WZ2AYCZnWNm7cCngG+b2dZo97cDT5nZr4GfAX/n7r+J1n0R+C6wA9gJPDSF1zVCLhNQGCyNv6GISIrV9AM57r4B2FBWdm1ieSMjm4Di8p8AZ1Y55ibgjImc7GRlM0axpBqCiMhYUjFSORsEajISERlHKgIhlzE1GYmIjCMVgaAmIxGR8aUjEIKAQlE1BBGRsaQiEPLZgEJJgSAiMpZUBILGIYiIjC8dgZAJKJaccB49ERGpJBWBkAvCqZPUsSwiUl0qAiGbCS9TzUYiItWlIhBymbCGoI5lEZHqUhEI2bjJSDUEEZGq0hEIQ01GqiGIiFSTikAYbjJSDUFEpJpUBEI2UA1BRGQ86QiEuIagQBARqSoVgZCP+hAK6lQWEakqFYGgcQgiIuNLSSBoHIKIyHhSEQi5QDUEEZHxpCIQ4hqC7jISEakuFYGgcQgiIuNLRSBoHIKIyPjSEQhD4xBUQxARqSYVgZCLbzvVXUYiIlWlIhDi2U41UllEpLpUBEJOI5VFRMaVqkDQOAQRkepSEQhD4xDUhyAiUlVNgWBma83sRTPbYWZXVVh/npk9a2ZFM/tkonyVmT1hZlvN7Dkzuzix7g+ifTab2f8zs5On5pJGi0cqq8lIRKS6cQPBzDLAzcA64DTgUjM7rWyzV4ArgDvLynuAy939dGAtcKOZtUbrbgE+4+6rov2umexFjEcjlUVExpetYZs1wA53fxnAzO4CLgSejzdw913RuhHfuO7+UmJ5j5ntBdqATsCBlmj1fGDPpK9iHMNNRqohiIhUU0sgLAN2J163A7870TcyszVAHtgZFX0e2GBmvcAh4NyJHrNWw01GqiGIiFRTSx+CVSib0J/aZrYU+D7wOXePv5X/DPiIuy8Hvgd8o8q+681sk5lt6ujomMjbDgkCIzDdZSQiMpZaAqEdODHxejkTaN4xsxbgQeAad38yKmsD3unuT0Wb/Qh4T6X93f1Wd1/t7qvb2tpqfdtRsplAv4cgIjKGWgJhI3CKma00szxwCfBALQePtr8fuN3d70msOgDMN7N/E73+ILCt9tOeuFxgqiGIiIxh3D4Edy+a2ZeAh4EMcJu7bzWz64FN7v6AmZ1D+MW/APgjM7suurPoIuA8YJGZXREd8gp332xm/wG4L+qIPgD8uym/uoRsJlAfgojIGGrpVMbdNwAbysquTSxvJGxKKt/vDuCOKse8nzBEZkQuYxqHICIyhlSMVIZw+gqNQxARqS41gZDNmMYhiIiMITWBkAvUhyAiMpbUBEI2o7uMRETGkp5ACALNdioiMobUBILuMhIRGVtqAiGbUQ1BRGQs6QmEQDUEEZGxpCYQNA5BRGRsqQmErPoQRETGlJ5A0DgEEZExpSYQ8lmNVBYRGUtqAiEbqA9BRGQs6QkE9SGIiIwpNYGQ00hlEZExpSYQNJeRiMjYUhMIOf1imojImFITCNlAdxmJiIwlPYGQCdRkJCIyhtQEQi5jDAyWcFcoiIhUkppAyAbhpQ6q2UhEpKLsbJ/ATMlmDIBiyclmZvlkRGTGFAoF2tvb6evrm+1TmXb19fUsX76cXC43qf1TEwj5TFhDKAyWqM8pEUTSor29nebmZlasWIGZzfbpTBt3Z9++fbS3t7Ny5cpJHSM9TUZxDUEdyyKp0tfXx6JFi+Z0GACYGYsWLTqimlCKAiGqIWi0skjqzPUwiB3pdaYmEHKBaggiMvM6Ozv55je/OeH9PvKRj9DZ2TkNZ1RdagIhriEoEERkJlULhMHBwTH327BhA62trdN1WhWlplM5F/UhqMlIRGbSVVddxc6dO1m1ahW5XI6mpiaWLl3K5s2bef755/nYxz7G7t276evr48tf/jLr168HYMWKFWzatInu7m7WrVvH+973Pn75y1+ybNkyfvzjH9PQ0DDl51pTIJjZWuDvgQzwXXe/oWz9ecCNwJnAJe5+b1S+CrgFaAEGgb929x9F6wz4K+BT0bpb3P2mqbioSuJxCKohiKTXdf97K8/vOTSlxzzthBa+/kenV11/ww03sGXLFjZv3szjjz/ORz/6UbZs2TJ0J9Btt93GwoUL6e3t5ZxzzuETn/gEixYtGnGM7du388Mf/pDvfOc7XHTRRdx3331cdtllU3odUEMgmFkGuBn4INAObDSzB9z9+cRmrwBXAFeW7d4DXO7u283sBOAZM3vY3Tuj7U8E3ubuJTNbcsRXM4b4LiNNcCcis2nNmjUjbgu96aabuP/++wHYvXs327dvHxUIK1euZNWqVQC8613vYteuXdNybrXUENYAO9z9ZQAzuwu4EBgKBHffFa0b8W3r7i8llveY2V6gDegEvgh82t1L0fq9R3Ql48gpEERSb6y/5GfKvHnzhpYff/xxHn30UZ544gkaGxs5//zzK942WldXN7ScyWTo7e2dlnOrpVN5GbA78bo9KpsQM1sD5IGdUdFbgYvNbJOZPWRmp0z0mBMx1GSkqStEZAY1NzfT1dVVcd3BgwdZsGABjY2NvPDCCzz55JMzfHYj1VJDqHRj64S+Vc1sKfB94LNxjQCoA/rcfbWZ/TFwG/B7FfZdD6wHOOmkkybytiOoyUhEZsOiRYt473vfyxlnnEFDQwPHHXfc0Lq1a9fyrW99izPPPJNTTz2Vc889dxbPtLZAaCds648tB/bU+gZm1gI8CFzj7sn4awfui5bvB75XaX93vxW4FWD16tWT/vM+r9tORWSW3HnnnRXL6+rqeOihhyqui/sJFi9ezJYtW4bKr7yyvKt26tTSZLQROMXMVppZHrgEeKCWg0fb3w/c7u73lK3+Z+AD0fLvAy8xjYbGIei2UxGRisYNBHcvAl8CHga2AXe7+1Yzu97MLgAws3PMrJ3wFtJvm9nWaPeLgPOAK8xsc/RYFa27AfiEmf0G+G/A56f0yspkg7jJSDUEEZFKahqH4O4bgA1lZdcmljcSNiWV73cHcEeVY3YCH53IyR6JnJqMRETGlKKpK+LfQ1CTkYhIJakJhFwQ/x6CaggiIpWkJhCGfw9BNQQRkUpSFwgFDUwTkRk02emvAW688UZ6enqm+IyqS00gDDUZFVVDEJGZcywFQmqmv1ansojMhuT01x/84AdZsmQJd999N/39/Xz84x/nuuuu4/Dhw1x00UW0t7czODjI1772Nd544w327NnD+9//fhYvXsxjjz027eeamkCIbztVp7JIij10Fbz+m6k95vHvgHU3VF2dnP76kUce4d577+Xpp5/G3bngggv4+c9/TkdHByeccAIPPvggEM5xNH/+fL7xjW/w2GOPsXjx4qk95yrS02SkcQgiMsseeeQRHnnkEc466yzOPvtsXnjhBbZv38473vEOHn30Ub761a/yi1/8gvnz58/K+aWmhpAJDDM1GYmk2hh/yc8Ed+fqq6/mC1/4wqh1zzzzDBs2bODqq6/mQx/6ENdee22FI0yv1NQQIOxYVpORiMyk5PTXH/7wh7ntttvo7u4G4NVXX2Xv3r3s2bOHxsZGLrvsMq688kqeffbZUfvOhNTUECDsWNY4BBGZScnpr9etW8enP/1p3v3udwPQ1NTEHXfcwY4dO/jKV75CEATkcjluueUWANavX8+6detYunTpjHQqm/ux8xfz6tWrfdOmTZPe/8y/fJg/Pns5f3nB7P9qkojMjG3btvH2t799tk9jxlS6XjN7xt1Xj7dvupqMMoF+IEdEpIpUBULYZHTs1IhERGZSugIhUA1BRKSaVAVCLmOay0gkhY6lvtIjcaTXmapAyGYC3WUkkjL19fXs27dvzoeCu7Nv3z7q6+snfYx03XYamMYhiKTM8uXLaW9vp6OjY7ZPZdrV19ezfPmoH6+sWaoCIZ8NNFJZJGVyuRwrV66c7dM4JqSrySjQXUYiItWkKxA0DkFEpKpUBUIuYxR1l5GISEWpCoRsoLuMRESqSVUg5DK6y0hEpJpUBUI20F1GIiLVpCsQVEMQEakqVYGg2U5FRKpLVSBoHIKISHU1BYKZrTWzF81sh5ldVWH9eWb2rJkVzeyTifJVZvaEmW01s+fM7OIK+/4PM+s+ssuoTTajPgQRkWrGDQQzywA3A+uA04BLzey0ss1eAa4A7iwr7wEud/fTgbXAjWbWmjj2aqCVGZJXH4KISFW11BDWADvc/WV3HwDuAi5MbuDuu9z9OaBUVv6Su2+PlvcAe4E2GAqavwX+4oivokaa7VREpLpaAmEZsDvxuj0qmxAzWwPkgZ1R0ZeAB9z9tYkea7Ky+j0EEZGqapnt1CqUTehb1cyWAt8HPuvuJTM7AfgUcH4N+64H1gOcdNJJE3nbUXIaqSwiUlUtNYR24MTE6+XAnlrfwMxagAeBa9z9yaj4LOBkYIeZ7QIazWxHpf3d/VZ3X+3uq9va2mp924qyGaPkUFItQURklFpqCBuBU8xsJfAqcAnw6VoObmZ54H7gdne/Jy539weB4xPbdbv7yRM58cnIZcL8K5RK1AWZ6X47EZFjyrg1BHcvErb3PwxsA+52961mdr2ZXQBgZueYWTthM9C3zWxrtPtFwHnAFWa2OXqsmpYrqUE2CFu/NBZBRGS0mn4xzd03ABvKyq5NLG8kbEoq3+8O4I4ajt9Uy3kcqWxcQ1A/gojIKKkaqZzLhDUEjUUQERktVYGQDcLL1WhlEZHR0hUIGfUhiIhUk6pAyKsPQUSkqlQFwlANQeMQRERGSVcgBKohiIhUk6pAyKkPQUSkqlQFQjwOQXcZiYiMlqpAyAUahyAiUk2qAkEjlUVEqktZIKgPQUSkmlQFQk53GYmIVJWqQNA4BBGR6lIVCMOT26mGICJSLmWBEN12qj4EEZFRUhUIGocgIlJdqgJB4xBERKpLVSAM1RDUhyAiMkrKAkF3GYmIVJOqQBgeh6BAEBEpl6pAyOq2UxGRqtIVCEE8dYUCQUSkXKoCwczIBkZBfQgiIqOkKhAgbDZSDUFEZLTUBUIuCNSpLCJSQfoCIRtopLKISAWpC4RsYJrLSESkgtQFQi6jJiMRkUpqCgQzW2tmL5rZDjO7qsL688zsWTMrmtknE+WrzOwJM9tqZs+Z2cWJdT+IjrnFzG4zs9zUXNLYshlTk5GISAXjBoKZZYCbgXXAacClZnZa2WavAFcAd5aV9wCXu/vpwFrgRjNrjdb9AHgb8A6gAfj8JK9hQtRkJCJSWbaGbdYAO9z9ZQAzuwu4EHg+3sDdd0XrRvzp7e4vJZb3mNleoA3odPcN8TozexpYPvnLqF3YZKQagohIuVqajJYBuxOv26OyCTGzNUAe2FlWngP+BPiXiR5zMrIZUyCIiFRQSyBYhbIJtbmY2VLg+8Dn3L382/ibwM/d/RdV9l1vZpvMbFNHR8dE3raibBBotlMRkQpqCYR24MTE6+XAnlrfwMxagAeBa9z9ybJ1XydsQvov1fZ391vdfbW7r25ra6v1bavKqYYgIlJRLYGwETjFzFaaWR64BHigloNH298P3O7u95St+zzwYeDSCrWGaZMNAnUqi4hUMG4guHsR+BLwMLANuNvdt5rZ9WZ2AYCZnWNm7cCngG+b2dZo94uA84ArzGxz9FgVrfsWcBzwRFR+7dReWmXZjCa3ExGppJa7jIjuCNpQVnZtYnkjFe4Scvc7gDuqHLOm955q+Uygye1ERCpI3UjlcLZT1RBERMqlMBACChqpLCIyyqw028ymumzAyx2Heed1jzC/IUdrY475DTlaGnK0Rq8XNOaZ35Bj4bw8rY15Fs7Ls7AxT3N9liCodBeuiMixL3WB8MXffyvLWhs42FsY8Xj1QC+d0fJglU7nTGAsiAJjQRQSC+blWTgvLGttzLOgMTciRFoaspgpRETk6Je6QDjluGb+/EOnVl3v7nT1F+k8XOBAzwD7ewY4cHiAAz0FDhweYN/hATp7Bth/eICdHd0c+G243XghkgyL1kTNZKh20pinpT47oiyXSV2LnojMotQFwnjMjJb6HC31OU5a1FjTPu7Oob4inT1RcEQhsj96HOgpROsG2L2/hy294TZ9hbH7MhpyGVoasjTX52iuH35uiZfrsjTXZ2kaWp+luS4XlYWv67KZqfjPIiIpoECYAmY29Jf9WxbVvl9/cZCDvQUOlTVfHewp0NVX5FBfgUO9Rbr6w+fOngHa9/dwqK9IV1+B/uL4neP5bEBz3XBANNWVh0mWprpcuL4uXN80tN3wawWLyNynQJhFddkMS5ozLGmun9T+A8US3f1hOHT1FaNHge7+YlQehkp3tK67v0h3X5Hd+3tGbFOtuSspnwmGgiIZGvFyHCbzytbPSwRLvC6jjnmRo5IC4RiWzwYszIYd2JPl7vQVSnT1h6HS3VfkcH+RQ9FzHDjd/YN094fhEofJ3q4+dnaE23X1FWuqsUDYFBYHxby6TBQsOZrqMsM1lri8PjeitpIMmMZ8Rh32IlNIgZByZkZDPkNDPsOS5iM7VmGwNBQO3f1RUPQPh0wcJIcTtZN4uf1AD4cHikOBU8vPnAZGGBBRM1hT/cgaS9zv0jS0PLI/Ju6LyWfVeS8CCgSZQrlMEN5F1Tj5Gkusvzg4ojYyYjkRJnFTWRg+YWf97gM9Q81kvYXBcd+rLhvQ0jAyKFoahp/j/qHhTv3wLrHWqDyru8FkjlAgyFGpLpuhrinDoqa6IzpOcbA0oj8l2dcSP8ed9Id6o478viKvdvZyqDcsGxhn7qvmuizz4wGO9bmhO8Pi24vDW47D247nN8ZjVnI05NTkJUcXBYLMadkjrLW4O/3FEgd7C3T1FTgYh0ZvYWh8SvJOsUN9BXa92cOhvgKdPYUxayi5jDG/IT9U2wjHpuSHxqgkR9HHtZRWjVGRaaRAEBmDmVGfy1Cfy3Bcy8TvBusrhLcWH+gZoDMaj9LZUxgaFd/ZMxA9F3i1s49tr3XR2TPA4YGxm7rm5TPhYMaG4aarOERaEmHSmgiY+Y3h7caqlUg1CgSRaTTZMBkoljjUVxgKi0MjAqQYLvcODJXv7OgeGscy1t1emcCGahrzo5rJgsZ8tJxnwbwwPOJmrfi5SUGSCgoEkaNQPhuwuKmOxZPoQ4lrJQejZq0wPMIBj3GQHIhC5s3uAbbv7Q4HQ/YXqx4zG1iiPySaz6sxT+u83PCcXkNze4WPlnqFyLFGgSAyx0y2VlIYLNHZU+Bgb9isFU/DcjCejiUxBctv9/WweXcnnT2Fqp3uYYgMT/64cN7ox6J5dSycl2dxUxgm6huZXQoEEQHC24bbmutoa669VuLudPcXowAJJ3+M5/GKJ4I8cLjA/sNhTSTsiB+g2uD4+Q05Fs3Ls6gpDItFTXkWNdXRFj2HtaY8bc11asaaBgoEEZk0M4vGZ+Q4cWFtk0EOlpyDvQX2H+5nX/dweLzZ3T+83NXPzo5unt4VBohXCJC67HCALW4Kn9vi5+ixJFpXn9NcXLVQIIjIjMoENtRkdPKS8bcvDpbY3zPAm11haMSPjq5+3uweoKOrn1f29fDMbw+w//BAxWO01GcTQVHPcc11LGmpY0lzPUvi5Zb61N+FpUAQkaNaNhNEX9zj94kUBkvsPzzA3kPDobG3q4+Orn46uvvZe6ifX+/uZG9XX8Xp5+tzAce11HNccz1LWurC5eh5SXO4fPz8ehrzc/Orc25elYikUi4TfaGP06Ee933s7QpDYm9XH3sP9fPGoT7e6Opn76E+tu45xE+37a04uLC5LsuSKBzC8Kjn+Ph1Sz3Hz6+nranumJvWRIEgIqmT7Pt4a1tT1e3i4HgjDotDfWXLfTz1r/vZ29U3akLGwKCtuY6l8xtYOr+epfMbOKG1nmWtDSxtDcsWN9UdVdPBKxBERKpIBsfJS6oHR6nk7O8Z4PWDYUi8fqiP1w+Gj9cO9vHiG1387KUOespGoGcC47jmOpYtaOCE1oYwLObXc3wUIstaG2htzM1Yv4YCQUTkCAWBDQ0kPGPZ/IrbuId3V+3p7GNPZ+9QaOzp7OXVzl6efeUADz73GsWye3Ln5TOc0NrAt/7kXWPWZqaCAkFEZAaY2dBEi6ed0FJxm1LJefNwP6919vHawV7aD4Rh8eqBXlobctN+jgoEEZGjRBDY0B1V7zyxdebff8bfUUREjko1BYKZrTWzF81sh5ldVWH9eWb2rJkVzeyTifJVZvaEmW01s+fM7OLEupVm9pSZbTezH5nZkf/MloiITNq4gWBmGeBmYB1wGnCpmZ1WttkrwBXAnWXlPcDl7n46sBa40czietDfAP/d3U8BDgD/frIXISIiR66WGsIaYIe7v+zuA8BdwIXJDdx9l7s/B5TKyl9y9+3R8h5gL9Bm4T1UHwDujTb9R+BjR3QlIiJyRGoJhGXA7sTr9qhsQsxsDZAHdgKLgE53jydgn9QxRURk6tQSCJVGRFSZvLbKAcyWAt8HPufupYkc08zWm9kmM9vU0dExkbcVEZEJqCUQ2oETE6+XA3tqfQMzawEeBK5x9yej4jeBVjOLb3utekx3v9XdV7v76ra2tlrfVkREJqiWQNgInBLdFZQHLgEeqOXg0fb3A7e7+z1xubs78BgQ35H0WeDHEzlxERGZWuaVfnmifCOzjwA3AhngNnf/azO7Htjk7g+Y2TmEX/wLgD7gdXc/3cwuA74HbE0c7gp332xmv0PYQb0Q+BVwmbv3j3MeHcBvJ3yVocWENZO0SeN1p/GaIZ3XrWuuzVvcfdwmlpoCYS4ws03uvnq2z2OmpfG603jNkM7r1jVPLY1UFhERQIEgIiKRNAXCrbN9ArMkjdedxmuGdF63rnkKpaYPQURExpamGoKIiIwhFYEw3mytc4GZnWhmj5nZtmh22S9H5QvN7CfRrLI/MbMFs32uU83MMmb2KzP7P9HrOT+Trpm1mtm9ZvZC9Jm/e65/1mb2Z9G/7S1m9kMzq5+Ln7WZ3WZme81sS6Ks4mdroZui77bnzOzsI3nvOR8INc7WOhcUgT9397cD5wJ/Gl3nVcBPo1llfxq9nmu+DGxLvE7DTLp/D/yLu78aeBayAAAClUlEQVQNeCfh9c/Zz9rMlgH/CVjt7mcQjom6hLn5Wf8vwtmhk6p9tuuAU6LHeuCWI3njOR8I1DBb61zg7q+5+7PRchfhF8Qywmv9x2izOTerrJktBz4KfDd6Pedn0o2mgzkP+AcAdx9w907m+GdN+AuPDdGUN43Aa8zBz9rdfw7sLyuu9tleSDgThEdTA7VGc8dNShoCYUpmaz2WmNkK4CzgKeA4d38NwtAAlszemU2LG4G/YHjq9TTMpPs7QAfwvaip7LtmNo85/Fm7+6vA3xH+9sprwEHgGeb+Zx2r9tlO6fdbGgLhiGdrPZaYWRNwH/Cf3f3QbJ/PdDKzPwT2uvszyeIKm861zzsLnA3c4u5nAYeZQ81DlURt5hcCK4ETgHmEzSXl5tpnPZ4p/feehkA4otlajyVmliMMgx+4+z9FxW/EVcjoee9snd80eC9wgZntImwK/ABhjaGmmXSPYe1Au7s/Fb2+lzAg5vJn/W+Bf3X3DncvAP8EvIe5/1nHqn22U/r9loZAmPRsrceSqO38H4Bt7v6NxKoHCGeThTk2q6y7X+3uy919BeHn+n/d/TPM8Zl03f11YLeZnRoV/QHwPHP4syZsKjrXzBqjf+vxNc/pzzqh2mf7AHB5dLfRucDBuGlpMlIxMK3SbK2zfEpTzszeB/wC+A3D7en/lbAf4W7gJML/qT7l7uUdVsc8MzsfuNLd/3AyM+kea8xsFWFHeh54Gfgc4R94c/azNrPrgIsJ76j7FfB5wvbyOfVZm9kPgfMJZzV9A/g68M9U+GyjcPyfhHcl9RD+CNmmSb93GgJBRETGl4YmIxERqYECQUREAAWCiIhEFAgiIgIoEEREJKJAEBERQIEgIiIRBYKIiADw/wHFE8qFRz0FDAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot history\n",
    "plt.plot(history.history['loss'], label='train')\n",
    "plt.plot(history.history['val_loss'], label='test')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Make Predictions and Plot ROC-AUC Metric"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate ROC-AUC for a single batch of test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_test, y_test = next(dg_validation)\n",
    "y_pred = model.predict(X_test).ravel()\n",
    "fpr, tpr, thresholds = roc_curve(y_test, y_pred)\n",
    "aucurve = auc(fpr, tpr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd4VNXWwOHfMkGKgkpRg6E3SQBRI4h06SqCBcWGaADRq6CIihel6eVSBUF6EUREFKXo5YqKevWjiIiAdGIEEjpIr0lY3x9zEkNIyACZOZmZ9T7PPDlnZs+cdUjIyt77nLVFVTHGGGMALnM7AGOMMbmHJQVjjDFpLCkYY4xJY0nBGGNMGksKxhhj0lhSMMYYk8aSgjHGmDSWFEzQEZEtInJCRI6KyC4RmSIiV2Zoc4eIfCciR0TkkIh8ISJRGdoUEpHhIrLN+aw4Z7+of8/IGP+xpGCCVUtVvRKoDtwMvJ76gojUAr4G5gLFgTLAKmCRiJR12lwOLASigeZAIeAOYD9Qw1dBi0i4rz7bGG9YUjBBTVV3AQvwJIdUg4APVPVdVT2iqn+p6hvAUqCP06YdUBK4T1XXqeoZVd2jqm+p6vzMjiUi0SLyjYj8JSK7ReSfzvNTROTtdO0aiEhiuv0tIvKaiKwGjonIGyIyK8NnvysiI5ztq0RkkojsFJHtIvK2iIRd4j+VMYAlBRPkRCQSaAHEOfsF8PzF/2kmzT8BmjjbjYGvVPWol8cpCHwLfIWn91EeT0/DW48AdwNXA9OAu0SkkPPZYcBDwEdO26lAsnOMm4GmQIcLOJYxWbKkYILVHBE5AiQAe4DezvOF8fzc78zkPTuB1PmCIlm0yco9wC5VHaqqJ50eyM8X8P4RqpqgqidUdSuwAmjtvHYncFxVl4rIdXiS3IuqekxV9wDDgLYXcCxjsmRJwQSr1qpaEGgA3Mjfv+wPAGeAiEzeEwHsc7b3Z9EmKyWAPy4qUo+EDPsf4ek9ADzK372EUkAeYKeIHBSRg8A44NpLOLYxaSwpmKCmqv8DpgBDnP1jwBKgTSbNH+LvIZ9vgWYicoWXh0oAymXx2jGgQLr96zMLNcP+p0ADZ/jrPv5OCgnAKaCoql7tPAqparSXcRpzXpYUTCgYDjQRkdTJ5h7AkyLSRUQKisg1zkRwLaCv02Yanl/An4nIjSJymYgUEZF/ishdmRzjS+B6EXlRRPI6n1vTeW0lnjmCwiJyPfBidgGr6l7gB+B94E9VXe88vxPPlVNDnUtmLxORciJS/yL+XYw5hyUFE/ScX7AfAG86+/8HNAPuxzNvsBXPhG0dVd3stDmFZ7J5A/ANcBhYhmcY6py5AlU9gmeSuiWwC9gMNHRenobnktcteH6hz/Qy9I+cGD7K8Hw74HJgHZ7hsFlc2FCXMVkSW2THGGNMKuspGGOMSWNJwRhjTBpLCsYYY9JYUjDGGJMm4IpvFS1aVEuXLu12GMYYE1B+/fXXfapaLLt2AZcUSpcuzfLly90OwxhjAoqIbPWmnQ0fGWOMSWNJwRhjTBpLCsYYY9IE3JxCZpKSkkhMTOTkyZNuh2Ic+fLlIzIykjx58rgdijHmAgRFUkhMTKRgwYKULl0aEXE7nJCnquzfv5/ExETKlCnjdjjGmAvgs+EjEZksIntEZE0Wr4uIjHAWQ18tIrdc7LFOnjxJkSJFLCHkEiJCkSJFrOdmTADy5ZzCFDwLnmelBVDBeXQCxlzKwSwh5C72/TAmMPksKajqj8Bf52nSCs/i6aqqS4GrRcTK/xpjTAbHjh3j5elL6PvFWp8fy82rj27g7CUIE53nziEinURkuYgs37t3r1+CCwa//vorVatWpXz58nTp0oXMyqQfOnSIli1bctNNNxEdHc37778PwNatW7n11lupXr060dHRjB07Nu09M2fOpFq1akRHR/Pqq6/67XyMCUXfffcd1apVY+7/fmXdjsM+P56bSSGz8YVMF3dQ1fGqGqOqMcWKZXuXdq6VkpLi1+M9++yzjB8/ns2bN7N582a++uqrc9qMGjWKqKgoVq1axQ8//MDLL7/M6dOniYiIYPHixaxcuZKff/6ZAQMGsGPHDvbv388rr7zCwoULWbt2Lbt372bhwoWZHN0YcykOHjxIx44dadSoEZdddhnly2e12mvOcvPqo0Q8i52nigR2uBTLJWvdujUJCQmcPHmSrl270qlTJwCuvPJKunXrxoIFCxg6dCj58+enW7duHD16lKJFizJlyhQiIiKYMGEC48eP5/Tp05QvX55p06ZRoECBbI6atZ07d3L48GFq1aoFQLt27ZgzZw4tWrQ4q52IcOTIEVSVo0ePUrhwYcLDw7nssr//Xjh16hRnzpwBID4+nooVK5KanBs3bsxnn31Go0aNLjpWE9o++nkbc1dudzuMXEb55ZflHD9ejphXP6R06dJs2H2Uq67y/ZHdTArzgOdF5GOgJnDIWX/2kvT9Ym2Od7Giiheid8vzr4s+efJkChcuzIkTJ7jtttt44IEHKFKkCMeOHaNKlSr069ePpKQk6tevz9y5cylWrBgzZ86kZ8+eTJ48mfvvv5+OHTsC8MYbbzBp0iReeOGFs47x/fff89JLL51z7AIFCrB48eKzntu+fTuRkZFp+5GRkWzffu5/vOeff557772X4sWLc+TIEWbOnJmWEBISErj77ruJi4tj8ODBFC9enPz587Nhwwa2bNlCZGQkc+bM4fTp0979Qxq/CpRftj//6Zl6rFmmsMuRuC85OYnw8DyAUKZMGfLmzUvBggUBiIooRKvqmY6w5yifJQURmQE0AIqKSCLQG8gDoKpjgfnAXUAccBx4ylex+MOIESOYPXs24PllunnzZooUKUJYWBgPPPAAABs3bmTNmjU0adIE8AwnRUR45tbXrFnDG2+8wcGDBzl69CjNmjU75xgNGzZk5cqVXsWT2fxBZlcELViwgOrVq/Pdd9/xxx9/0KRJE+rWrUuhQoUoUaIEq1evZseOHbRu3ZoHH3yQ6667jjFjxvDwww9z2WWXcccddxAfH+/dP5Lxq7krt7Nu52GiIgq5Hcp51SxTmFbVb+DRmiXdDsU1qsr06dPp2rUrAwYMSPsD0Q0+Swqq+kg2ryvwj5w+bnZ/0fvCDz/8wLfffsuSJUsoUKAADRo0SLtGP1++fISFhQGeb3x0dDRLliw55zPat2/PnDlzuOmmm5gyZQo//PDDOW0upKcQGRlJYmJi2n5iYiLFixc/573vv/8+PXr0QEQoX748ZcqUYcOGDdSoUSOtTfHixYmOjuann37iwQcfpGXLlrRs2RKA8ePHp52f8Q9vewCpCWHmM7X8EJW5WAkJCXTu3Jn58+dz++23U7t2bVfjCYo7mt126NAhrrnmGgoUKMCGDRtYunRppu0qVarE3r17WbJkCbVq1SIpKYlNmzYRHR3NkSNHiIiIICkpienTp3PDDed2Ey+kpxAREUHBggVZunQpNWvW5IMPPjhnOAqgZMmSLFy4kLp167J79242btxI2bJlSUxMpEiRIuTPn58DBw6waNEiunXrBsCePXu49tprOXDgAKNHj+aTTz65gH8tk5kLGerxdrjFX8MN5uLNmDGDZ555hpSUFIYPH87zzz/v+h9ZlhRyQPPmzRk7dizVqlWjUqVK3H777Zm2u/zyy5k1axZdunTh0KFDJCcn8+KLLxIdHc1bb71FzZo1KVWqFFWrVuXIkSOXHNeYMWNo3749J06coEWLFmmTzKmXl3bu3Jk333yT9u3bU7VqVVSVgQMHUrRoUb755htefvllRARVpXv37lStWhWArl27smrVKgB69epFxYoVLznWQOKLsfoLGVe34Zbgcc0111CzZk3Gjx+fa0rCSGZjz7lZTEyMZlxkZ/369VSuXNmliExWgvX78vC4JT4Zq7df9MEvOTmZYcOGcfr0aXr27Al4hpX9UQFARH5V1Zjs2llPwZiLYGP15kKtWrWK2NhYfv31Vx566KG0ZJDbSsLYegrGGONDp06d4s033yQmJoaEhAQ+/fRTPv7441yXDFIFTVIItGGwYGffD2M8Nm/ezMCBA3n00UdZt24dDz74YK5NCBAkSSFfvnzs37/ffhHlEqnrKeTLl8/tUIxxxdGjR5k+fToAVapUYcOGDUydOpUiRYq4HFn2gmJOIfWafCuWl3ukrrxmTKj55ptv6NSpE1u3buWWW26hcuXKlC1b1u2wvBYUSSFPnjy55nIuY0xoOnDgAN27d2fy5MlUrFiR//3vfwF59V1QJAVjjHFTSkoKtWvXZtOmTbz++uv06tUrYIdPLSkYY8xF2rdvH4ULFyYsLIz+/ftTsmRJbrnlolcWzhWCYqLZGGP8SVX54IMPqFixIhMnTgQ85fMDPSGAJQVjjLkgW7dupUWLFjz55JNUrlyZevXquR1SjrLhIxPUfFGnKBDKURvf+PDDD3n22WdRVUaOHMlzzz131oJUwcCSgsk13C405y2rPhq6ihUrRu3atRk3bhylSpVyOxyfCIqCeCY4WKE5k9skJSUxdOhQkpKSePPNNwH/FbDLaVYQzwQkKzRncovffvuN2NhYfvvtN9q2bZtrC9jltOAaDDPGmEt08uRJ/vnPf3LbbbexY8cOPvvsM2bMmBH0ySCV9RSMz13o8pHGuCkuLo4hQ4bQrl07hg4dyjXXXON2SH5lPQXjc6kLyGfHJnCNW44ePcq0adMATwG7jRs3Mnny5JBLCGA9BeMnNldgcqsFCxbQqVMnEhISiImJoXLlyiFdS816CsaYkLR//36efPJJmjdvToECBfjpp58CsoBdTrOegjEm5KQWsIuLi6Nnz5688cYbAVvALqdZUjDGhIy9e/dSpEgRwsLCGDhwIKVKlaJ69epuh5WrWFIwF+VC7j62q4qM21SVKVOm0K1bNwYMGMAzzzxDq1at3A4rV7I5BXNRvL2iCOyqIuOuLVu20KxZM55++mmqVq1Kw4YN3Q4pV7OegrlodkWRye2mTZvGs88+i4gwevRonnnmmaArYJfTLCkYY4LWddddR7169Rg7diwlS1r9K29YUjDGBI2kpCQGDRpESkoKvXr1omnTpjRt2tTtsAKKJQVzFitJYQLVihUrePrpp1m1ahWPPvpowFYzdZsNrpmzWEkKE2hOnDhBjx49qFGjBrt372b27NlMnz7dEsJF8mlPQUSaA+8CYcBEVR2Q4fWSwFTgaqdND1Wd78uYTPZsAtkEkvj4eN555x3at2/P4MGDQ7JeUU7yWU9BRMKAUUALIAp4RESiMjR7A/hEVW8G2gKjfRWPMSZ4HD58mClTpgAQHR3N5s2bmThxoiWEHODL4aMaQJyqxqvqaeBjIOPdIgqkDkxfBezwYTzGmCAwf/58qlSpQmxsLOvXrwcI2qUx3eDL4aMbgIR0+4lAzQxt+gBfi8gLwBVA48w+SEQ6AZ0Au6zsItjdxyYY7Nu3j5deeokPP/yQqKgoFi1aZAXsfMCXPYXMZnkyLgj9CDBFVSOBu4BpInJOTKo6XlVjVDWmWLFiPgg1uNndxybQpRaw+/jjj+nVqxcrVqzg9ttvdzusoOTLnkIiUCLdfiTnDg/FAs0BVHWJiOQDigJ7fBhX0LjQy0dt8tgEmt27d1OsWDHCwsIYMmQIpUqVolq1am6HFdR82VP4BaggImVE5HI8E8nzMrTZBjQCEJHKQD5grw9jCip2+agJVqrKpEmTqFSpEuPHjwegZcuWlhD8wGc9BVVNFpHngQV4LjedrKprRaQfsFxV5wEvAxNE5CU8Q0vtVTXjEJM5D+sBmGATHx9Px44d+e6776hfvz6NG2c61Wh8xKf3KTj3HMzP8FyvdNvrgNq+jMEYEzimTp3Kc889R1hYGGPHjqVjx45WwM7PrMyFMSbXKF68OHfeeSdjxowhMjLS7XBCkiWFXMYuHzWh5PTp0wwYMIAzZ87Qp08fmjRpQpMmTdwOK6RZvyyXsctHTaj45ZdfuPXWW+nduzfx8fHYdGLuYD2FXMgmj00wO378OL169WLYsGFEREQwb948WrZs6XZYxmE9BWOMX/3555+MHDmSjh07snbtWksIuYz1FIwxPnfo0CE+//xznnrqKaKjo4mLi6NEiRLZv9H4nfUUjDE+9Z///Ifo6Gg6dOjAhg0bACwh5GKWFIwxPrF3714ee+wx7rnnHq655hqWLFnCjTfe6HZYJhs2fGSMyXEpKSnUqVOHP//8k759+9KjRw8uv/xyt8MyXrCkYIzJMbt27eLaa68lLCyMoUOHUrp0aapUqeJ2WOYC2PCRMeaSnTlzhnHjxlGxYkXGjRsHwD333GMJIQBlmxREJL+IvC4iY5398iLSwvehGWMCQVxcHI0aNaJz587cdtttNGvWzO2QzCXwpqcwGc+COXWc/R1Af59FZIwJGO+//z5Vq1ZlxYoVTJgwgW+//ZayZcu6HZa5BN4khQqq2h9IAlDV42S+qpoxJsSULFmSZs2asW7dOjp06ICI/WoIdN5MNJ92VkRTABEpA5z2aVTGmFzp1KlT/Pvf/+bMmTP069ePRo0a0ahRI7fDMjnIm57CW8BXQKSITAW+B/7p06iMMbnOzz//zK233krfvn3Ztm2bFbALUtkmBVX9L9AG6AjMBmqo6re+DswYkzscO3aMbt26UatWLQ4dOsSXX37JlClTbKgoSHlz9dHXqrpXVeeq6hxV3SMiX/sjOGOM+7Zu3cro0aPp3Lkza9eu5e6773Y7JONDWc4piMjlQD7gOhEpyN+Ty4WAkn6IzRjjkoMHDzJr1iw6dOhAVFQUcXFxthJaiDhfT+EfwFrgRudr6mMBMNb3oRlj3DB37lyioqLo3LlzWgE7SwihI8ukoKrDVLUE8JqqllTVEs4jWlWH+zFGY4wf7Nmzh7Zt29K6dWuKFSvG0qVLrYBdCMr2klRVHS4iNwJReIaTUp//yJeBGWP8JyUlhdq1a7Nt2zbefvttXn31VfLkyeN2WMYF2SYFEXkDaIpnGGkB0Az4P8CSgjEBbseOHVx//fWEhYXx7rvvUrp0aaKiotwOy7jIm/sUHgYaAjtV9QngJqy6qjEB7cyZM4wZM4Ybb7yRsWM9U4R33XWXJQTjVVI4oaopQLJzFdIuwIqbGBOgNm3aRMOGDXnuueeoWbMmLVpYfUvzN2+Swm8icjWewnjLgWXACp9GZYzxiUmTJnHTTTexevVqJk+ezNdff02ZMmXcDsvkIucdBhLPLYt9VPUgMEpEFgCFVNWSgjEBqHTp0rRo0YJRo0YRERHhdjgmFzpvUlBVFZEvgVud/Ti/RGWMyRGnTp3irbfeAuDtt9+2AnYmW94MHy0TkVt8HokxJkctXryY6tWr869//YudO3daATvjFW+SQh08iWGjiKwQkd9ExIaPjMmljh49SteuXalTpw7Hjx/nq6++YtKkSVbAznjFm0tLW1/sh4tIc+BdIAyYqKoDMmnzENAHz3oNq1T10Ys9njEGtm3bxrhx4/jHP/5B//79KViwoNshmQDizR3Nf1zMB4tIGDAKaAIkAr+IyDxVXZeuTQXgdaC2qh4QkWsv5ljGhLoDBw7w6aef0qlTJ6KiooiPj6d48eJuh2UCkDfDRxerBhCnqvGqehr4GGiVoU1HYJSqHgBQ1T0+jMeYoDR79myioqJ47rnn2LhxI4AlBHPRfJkUbgAS0u0nOs+lVxGoKCKLRGSpM9x0DhHpJCLLRWT53r17fRSuMYFl165dtGnThvvvv5/rr7+eZcuWUalSJbfDMgHOq3IVIhIJVFDV70UkLxCuqseye1smz2W8/CEcqAA0ACKBn0SkinNfxN9vUh0PjAeIiYmxSyhMyEtJSaFu3bokJCTQv39/unfvbgXsTI7wpiDe08DzwFVAOaAUMBponM1bE4ES6fYjgR2ZtFmqqknAnyKyEU+S+MWr6I0JMYmJiRQvXpywsDBGjBhBmTJlrLy1yVHeDB91AW4HDgOo6ibAmwnhX4AKIlLGWcWtLTAvQ5s5eIrtISJF8QwnxXsXujGh48yZM4wcOZIbb7yRMWPGANCiRQtLCCbHeZMUTjoTxUDaVUXZXvCsqsl4ehgLgPXAJ6q6VkT6ici9TrMFwH4RWQd8D7yiqvsv9CSMCWYbNmygXr16dOnShTp16nDPPfe4HZIJYt7MKSwSkVeBfCLSEM8ynV968+GqOh+Yn+G5Xum2FejmPIwxGUycOJHnn3+eAgUKMHXqVJ544gm7Cc34lDc9hVeBI8AGoCuwEOjpy6CMMR7lypWjZcuWrF+/nnbt2llCMD7nTU/hLjx3I4/xdTDGhLqTJ0/Sr18/APr370/Dhg1p2LChy1GZUOJNT+EhIE5E3heRZs6cgjEmhy1atIjq1avz73//m71791oBO+OKbJOCswRnReAL4GkgXkTG+jowY0LFkSNHeOGFF6hbty6nTp1iwYIFTJgwwYaKjCu8uqNZVU8Bc4EpeC41fciHMRkTUhITE5k4cSIvvPACv//+O02bNnU7JBPCsk0KItJYRCYCfwCPAx8A1/s6MGOC2f79+9PuN6hcuTLx8fG8++67XHnllS5HZkKdNz2FzsBXQGVVfUxV56W/b8EY4z1VZdasWURFRdGlS5e0Ana2NKbJLbyZU3hQVWep6gl/BGRMsNq5cycPPPAAbdq0oUSJEixfvtwK2JlcJ8tLUkXkf6paX0QOcHYhO8Fz31lhn0dnTJBILWC3fft2Bg0axEsvvUR4uFf1KI3xq/P9VKZeHF3UH4EYE4wSEhK44YYbCAsLY9SoUZQpU4aKFSu6HZYxWcpy+EhVzzibk1Q1Jf0DmOSf8IwJTCkpKYwYMeKsAnbNmjWzhGByPW/6r9XS7zg3r93mm3CMCXzr168nNjaWJUuW0KJFC1q2bOl2SMZ4Lcuegoi85swnVBORv5zHAWAvGYrcGWM8xo8fT/Xq1dm0aRPTpk3jP//5DyVLlnQ7LGO8dr6rjwYBxYBhztdiQFFVLayqr/gjOGMCTYUKFbjvvvtYt24djz/+uN2VbALO+YaPyqvqZhGZBkSnPpn6Q66qq30cmzG53okTJ+jTpw8iwoABA6yAnQl450sKPYBYYFQmrylQzycRGRMgfvzxRzp06MDmzZvp3Lkzqmo9AxPwskwKqhrrfK3rv3CMyf0OHz5Mjx49GDNmDGXLlmXhwoXceeedbodlTI7wpvbR/SJS0NnuISKfiMhNvg/NmNxpx44dTJkyhW7durF69WpLCCaoeFP7qI+qHhGRO4CWwExgnG/DMiZ32bdvH6NHjwbgxhtv5M8//2To0KFcccUVLkdmTM7yJimkOF/vAUar6mdAXt+FZEzuoarMnDmTqKgoXnzxRTZt2gTAdddd53JkxviGN0lhp4iMAtoC80Xkci/fZ0xA27FjB61bt6Zt27aUKlWKX3/91e5INkHPmzuaH8KzTvNIVT0gIsXxXJlkTNBKSUmhXr16bN++nSFDhtC1a1crYGdCQrY/5ap6VETWAQ1EpAHwk6r+1+eRGeOCrVu3EhkZSVhYGKNHj6Zs2bKUL1/e7bCM8Rtvrj56HvgEKOk8PhGR53wdmDH+lJKSwjvvvEPlypXTCtg1bdrUEoIJOd70hzsBNVT1KICI9AcWA6N9GZgx/rJmzRpiY2NZtmwZ99xzD61bt3Y7JGNc482EsQBJ6faTnOeMCXhjx47llltuIT4+no8++oh58+YRGRnpdljGuMabnsI0YKmIfIYnGbQGpvo0KmN8LLUkReXKlWnTpg3Dhw+nWLFibodljOu8mWgeJCLfA6nlLjqr6i++DcsY3zh+/Di9evUiLCyMgQMHUr9+ferXr+92WMbkGt7eb3DKeZxwvhoTcH744QeqVavG0KFDOXr0KKqa/ZuMCTHeXH3UE5gBRACRwEci8rqvAzMmpxw6dIhnnnkmraT1d999x6hRo6yiqTGZ8GZO4XHgVlU9DiAi/wJ+Bf7ty8CMySk7d+7kww8/pHv37vTt25cCBQq4HZIxuZY3w0dbOTt5hAPx3ny4iDQXkY0iEiciWd4FLSIPioiKSIw3n2tMdvbu3cvIkSMBTwG7LVu2MHjwYEsIxmTDm57CcWCtiCzAs7hOU+D/ROQdAFXtltmbRCQMzwI9TYBE4BcRmaeq6zK0Kwh0AX6+6LMIAB/9vI25K7dn227dzsNERRTyQ0TBSVWZMWMGXbp04fDhwzRr1oyKFSvalUXGeMmbnsJ/gD7AEmAp0A/4DljrPLJSA4hT1XhVPQ18DLTKpN1beNaDPul92IFn7srtrNt5ONt2URGFaFX9Bj9EFHwSEhJo2bIljz32GOXLl+e3336zAnbGXCBvLkmddJGffQOQkG4/EaiZvoGI3AyUUNUvRaR7Vh8kIp3w3FlNyZIlLzIc90VFFGLmM7XcDiMoJScn06BBA3bt2sWwYcN44YUXCAsLczssYwKOL8s+ZnZpR9o1gCJyGTAMaJ/dB6nqeGA8QExMjF1HaNJs2bKFEiVKEB4ezrhx4yhbtixly5Z1OyxjApYv10VIBEqk248EdqTbLwhUAX4QkS3A7cA8m2w23khOTmbIkCFUrlw5bUW0xo0bW0Iw5hJ53VMQkbyqeiE3rv0CVBCRMsB2PIv0PJr6oqoeAoqm+/wfgO6quvwCjmFC0OrVq4mNjWX58uW0atWKBx54wO2QjAka3ty8VkNEfgc2O/s3icjI7N6nqsnA88ACYD3wiaquFZF+InLvJcZtQtTo0aO59dZb2bp1KzNnzmT27NkUL17c7bCMCRre9BRG4FmfeQ6Aqq4SkYbefLiqzgfmZ3iuVxZtG3jzmSY0pRawq1KlCm3btmXYsGEULVo0+zcaYy6IN0nhMlXdmqEkQIqP4jHmLMeOHeONN94gPDycwYMHU69ePerVq+d2WMYELW8mmhNEpAagIhImIi8Cm3wclzEsXLiQqlWrMnz4cE6dOmUF7IzxA2+SwrNANzxLce7Gc5XQs74MyoS2gwcP0qFDBxo3bkx4eDg//vgjI0aMsAJ2xviBNzev7cFz5ZAxfrF7924+/vhjXnvtNXr37k3+/PndDsmYkJFtUhCRCaS76SyVqnbySUQmJKUmgq5du1KpUiW8D3G1AAASkklEQVS2bNliE8nGuMCb4aNvgYXOYxFwLbbQjskhqsqHH35IVFQUr776Kps3bwawhGCMS7wZPpqZfl9EpgHf+CwiEzK2bdtG586d+e9//0utWrWYNGkSFSpUcDssY0LaxdQ+KgOUyulAApG35bDBSmJnlFrAbs+ePYwYMYLnnnvOCtgZkwt4M6dwgL/nFC4D/gKyXDAnlKSWw/bml72VxPaIj4+nVKlShIeHM2HCBMqVK0fp0qXdDssY4zhvUhDPNYA34aldBHBG7WLxs1g5bO8kJyczdOhQevfuzaBBg+jSpQuNGjVyOyxjTAbnTQqqqiIyW1Vv9VdAJvisXLmS2NhYVqxYwX333UebNm3cDskYkwVvrj5aJiK3+DwSE5Tee+89brvtNrZv386sWbP4/PPPiYiIcDssY0wWsuwpiEi4U+m0DtBRRP4AjuFZPEdV1RKFyVJqAbtq1arx2GOP8c4771C4cGG3wzLGZON8w0fLgFuA1n6KxQSBo0eP0rNnT/LkycOQIUOsgJ0xAeZ8w0cCoKp/ZPbwU3wmgHz99ddUqVKFkSNHkpSUZAXsjAlA5+spFBORblm9qKrv+CCeXMHb+w/s3gOPAwcO0K1bN6ZMmUKlSpX48ccfqVOnjtthGWMuwvl6CmHAlXjWUs7sEbRS7z/Ijt174LFnzx5mzZrF66+/zsqVKy0hGBPAztdT2Kmq/fwWSS5j9x+c365du5gxYwYvvfRSWgG7IkWKuB2WMeYSZTunYEx6qsrUqVOJiori9ddfTytgZwnBmOBwvqRgt5uas2zZsoXmzZvTvn17oqKiWLlypRWwMybIZDl8pKp/+TMQk7slJyfTsGFD9u3bx6hRo+jcuTOXXebNvY/GmEByMVVSTQiJi4ujTJkyhIeHM3nyZMqWLUupUlYk15hgZX/qmUwlJSXRv39/oqOjGTVqFAANGza0hGBMkLOegjnHihUriI2NZeXKlbRp04aHH37Y7ZCMMX5iPQVzlhEjRlCjRg127drF559/zieffMJ1113ndljGGD+xpGAA0kpS3HzzzbRr145169Zx3333uRyVMcbfbPgoxB05coTXX3+dvHnzMnToUOrWrUvdunXdDssY4xLrKYSwr776iipVqjB69GhU1QrYGWMsKYSi/fv38+STT9KiRQuuuOIKFi1axDvvvINn9VVjTCizpBCC9u/fz+zZs3nzzTf57bffqFXLajwZYzx8mhREpLmIbBSROBHpkcnr3URknYisFpGFImIXwfvIzp07GTJkCKpKxYoV2bp1K/369SNv3rxuh2aMyUV8lhREJAwYBbQAooBHRCQqQ7PfgBhVrQbMAgb5Kp5QpapMnjyZypUr8+abbxIXFwfANddc43JkxpjcyJc9hRpAnKrGq+pp4GOgVfoGqvq9qh53dpcCkT6MJ+T8+eefNG3alNjYWG666SZWrVplBeyMMefly0tSbwAS0u0nAjXP0z4W+G9mL4hIJ6ATQMmSJXMqvqCWnJzMnXfeyf79+xkzZgydOnWyAnbGmGz5MilkdilLptc8isjjQAxQP7PXVXU8MB4gJibGrps8j82bN1O2bFnCw8N5//33KVeuHCVKlHA7LGNMgPDln46JQPrfRpHAjoyNRKQx0BO4V1VP+TCeoJaUlMTbb79NlSpVeO+99wBo0KCBJQRjzAXxZU/hF6CCiJQBtgNtgUfTNxCRm4FxQHNV3ePDWILa8uXLiY2NZfXq1bRt25ZHHnnE7ZCMMQHKZz0FVU0GngcWAOuBT1R1rYj0E5F7nWaDgSuBT0VkpYjM81U8werdd9+lZs2a7Nu3j7lz5zJjxgyuvfZat8MyxgQon9Y+UtX5wPwMz/VKt93Yl8cPZqqKiBATE0NsbCyDBg3i6quvdjssY0yAs4J4Aebw4cO89tpr5MuXj2HDhlG7dm1q167tdljGmCBh1ygGkPnz5xMdHc348eMJDw+3AnbGmBxnSSEA7Nu3j8cff5y7776bq666isWLFzN48GArYGeMyXGWFALAgQMH+OKLL+jduzcrVqygZs3z3QNojDEXL2TmFD76eRtzV273qu26nYeJiijk44jOb/v27UyfPp1XXnmFChUqsHXrVptINsb4XMj0FOau3M66nYe9ahsVUYhW1W/wcUSZU1UmTJhAVFQUffr04Y8//gCwhGCM8YuQ6SmA55f9zGdy79oBf/zxBx07duT777+nQYMGTJgwgfLly7sdljEmhIRUUsjNkpOTadSoEX/99Rfjxo2jQ4cOVsDOGON3lhRctnHjRsqVK0d4eDhTp06lXLlyREZaBXFjjDvsT1GXnD59mr59+1K1alVGjRoFQP369S0hGGNcZT0FFyxbtozY2FjWrFnDo48+ymOPPeZ2SMYYA1hPwe+GDx9OrVq10u49mD59OkWLFnU7LGOMASwp+E1qSYoaNWrQsWNH1q5dyz333ONyVMYYczYbPvKxQ4cO8eqrr5I/f36GDx/OHXfcwR133OF2WMYYkynrKfjQF198QVRUFBMnTiRv3rxWwM4Yk+tZUvCBvXv38uijj3LvvfdSpEgRli5dysCBA62AnTEm17Ok4AOHDh1i/vz59O3bl+XLl3Pbbbe5HZIxxnjF5hRySEJCAh9++CE9evSgfPnybN26lauuusrtsIwx5oJYT+ESnTlzhrFjxxIdHc3bb7+dVsDOEoIxJhBZUrgEmzdv5s477+TZZ5+lRo0a/P7771bAzhgT0Gz46CIlJyfTpEkTDh48yKRJk3jqqadsItkYE/AsKVyg9evXU6FCBcLDw5k2bRrlypWjePHibodljDE5woaPvHTq1Cl69+5NtWrVeO+99wCoW7euJQRjTFCxnoIXli5dSmxsLOvWreOJJ57giSeecDskY4zxCespZGPo0KHccccdHDlyhPnz5/PBBx9QpEgRt8MyxhifsKSQhTNnzgBQq1YtOnfuzJo1a2jRooXLURljjG/Z8FEGBw8e5OWXX6ZAgQKMHDnSCtgZY0KK9RTSmTNnDlFRUUydOpWCBQtaATtjTMixpADs2bOHhx56iPvuu4/rrruOZcuW0b9/f7vvwBgTciwpAIcPH+abb77hX//6F8uWLeOWW25xOyRjjHFFyM4pbNu2jWnTpvHPf/6T8uXLs23bNgoWLOh2WMYY4yqf9hREpLmIbBSROBHpkcnreUVkpvP6zyJS2pfxgOeqotGjRxMdHU3//v3TCthZQjDGGB8mBREJA0YBLYAo4BERicrQLBY4oKrlgWHAQF/FA3DixHEaNGjAP/7xD2rVqsXatWutgJ0xxqTjy55CDSBOVeNV9TTwMdAqQ5tWwFRnexbQSHw0u1v5+itZ/b/5/P7777z//vssWLCA0qVL++JQxhgTsHw5p3ADkJBuPxGomVUbVU0WkUNAEWBf+kYi0gnoBFCyZMmLCqZPq6o0LnIf5UZ3JyIi4qI+wxhjgp0vewqZ/cWf8cJ/b9qgquNVNUZVY4oVK3bRAdWpU8cSgjHGnIcvk0IiUCLdfiSwI6s2IhIOXAX85cOYjDHGnIcvk8IvQAURKSMilwNtgXkZ2swDnnS2HwS+U7uN2BhjXOOzOQVnjuB5YAEQBkxW1bUi0g9YrqrzgEnANBGJw9NDaOureIwxxmTPpzevqep8YH6G53ql2z4JtPFlDMYYY7xnZS6MMcaksaRgjDEmjSUFY4wxaSwpGGOMSSOBdgWoiOwFtl7k24uS4W7pEGDnHBrsnEPDpZxzKVXN9u7fgEsKl0JElqtqjNtx+JOdc2iwcw4N/jhnGz4yxhiTxpKCMcaYNKGWFMa7HYAL7JxDg51zaPD5OYfUnIIxxpjzC7WegjHGmPOwpGCMMSZNUCYFEWkuIhtFJE5EemTyel4Rmem8/rOIlPZ/lDnLi3PuJiLrRGS1iCwUkVJuxJmTsjvndO0eFBEVkYC/fNGbcxaRh5zv9VoR+cjfMeY0L362S4rI9yLym/PzfZcbceYUEZksIntEZE0Wr4uIjHD+PVaLyC05GoCqBtUDT5nuP4CywOXAKiAqQ5vngLHOdltgpttx++GcGwIFnO1nQ+GcnXYFgR+BpUCM23H74ftcAfgNuMbZv9btuP1wzuOBZ53tKGCL23Ff4jnXA24B1mTx+l3Af/GsXHk78HNOHj8Yewo1gDhVjVfV08DHQKsMbVoBU53tWUAjEclsadBAke05q+r3qnrc2V2KZyW8QObN9xngLWAQcNKfwfmIN+fcERilqgcAVHWPn2PMad6cswKFnO2rOHeFx4Ciqj9y/hUoWwEfqMdS4GoRybF1hoMxKdwAJKTbT3Sey7SNqiYDh4AifonON7w55/Ri8fylEciyPWcRuRkooapf+jMwH/Lm+1wRqCgii0RkqYg091t0vuHNOfcBHheRRDzrt7zgn9Bcc6H/3y+ITxfZcUlmf/FnvO7WmzaBxOvzEZHHgRigvk8j8r3znrOIXAYMA9r7KyA/8Ob7HI5nCKkBnt7gTyJSRVUP+jg2X/HmnB8BpqjqUBGphWc1xyqqesb34bnCp7+/grGnkAiUSLcfybndybQ2IhKOp8t5vu5abufNOSMijYGewL2qespPsflKdudcEKgC/CAiW/CMvc4L8Mlmb3+256pqkqr+CWzEkyQClTfnHAt8AqCqS4B8eArHBSuv/r9frGBMCr8AFUSkjIhcjmcieV6GNvOAJ53tB4Hv1JnBCVDZnrMzlDIOT0II9HFmyOacVfWQqhZV1dKqWhrPPMq9qrrcnXBzhDc/23PwXFSAiBTFM5wU79coc5Y357wNaAQgIpXxJIW9fo3Sv+YB7ZyrkG4HDqnqzpz68KAbPlLVZBF5HliA58qFyaq6VkT6ActVdR4wCU8XMw5PD6GtexFfOi/PeTBwJfCpM6e+TVXvdS3oS+TlOQcVL895AdBURNYBKcArqrrfvagvjZfn/DIwQURewjOM0j6Q/8gTkRl4hv+KOvMkvYE8AKo6Fs+8yV1AHHAceCpHjx/A/3bGGGNyWDAOHxljjLlIlhSMMcaksaRgjDEmjSUFY4wxaSwpGGOMSWNJweRaIpIiIivTPUqfp23prKpK+puIxIjICGe7gYjcke61ziLSzo+xVA/0qqHGv4LuPgUTVE6oanW3g7hQzg1yqTfJNQCOAoud18bm9PFEJNyp4ZWZ6njKmszP6eOa4GQ9BRNQnB7BTyKywnnckUmbaBFZ5vQuVotIBef5x9M9P05EwjJ57xYRGei0WyYi5Z3nS4lnHYrU9ShKOs+3EZE1IrJKRH50nmsgIl86PZvOwEvOMeuKSB8R6S4ilUVkWYbzWu1s3yoi/xORX0VkQWYVMEVkioi8IyLfAwNFpIaILBbPmgKLRaSScwdwP+Bh5/gPi8gV4qnX/4vTNrPKsiaUuV073B72yOqB547clc5jtvNcASCfs10Bz12tAKVx6s8DI4HHnO3LgfxAZeALII/z/GigXSbH3AL0dLbbAV86218ATzrbTwNznO3fgRuc7audrw3Sva8P0D3d56ftO+dV1tl+DXgDz52ri4FizvMP47mLN2OcU4AvgTBnvxAQ7mw3Bj5zttsD76V7X3/g8dR4gU3AFW5/r+2Rex42fGRys8yGj/IA74lIdTxJo2Im71sC9BSRSOBzVd0sIo2AW4FfnDIf+YGsakDNSPd1mLNdC7jf2Z6GZ40GgEXAFBH5BPj8Qk4OTxG3h4ABeH75PwxUwlPI7xsnzjAgq7o2n6pqirN9FTDV6RUpTlmETDQF7hWR7s5+PqAksP4CYzdBypKCCTQvAbuBm/AMf56zeI6qfiQiPwN3AwtEpAOecsNTVfV1L46hWWyf00ZVO4tITedYK51k5a2ZeGpRfe75KN0sIlWBtapay4v3H0u3/Rbwvare5wxb/ZDFewR4QFU3XkCcJoTYnIIJNFcBO9VTK/8JPH9Jn0VEygLxqjoCT0XJasBC4EERudZpU1iyXqf64XRflzjbi/m7cOJjwP85n1NOVX9W1V7APs4uaQxwBE8Z73Oo6h94ejtv4kkQ4Cl1XUw86wIgInlEJDqLONO7CtjubLc/z/EXAC+I0w0RT/VcY9JYUjCBZjTwpIgsxTN0dCyTNg8Da0RkJXAjnqUL1+EZs//amdD9BshqCcO8Tk+jK56eCUAX4CnnvU84rwEMFpHfncthf8SzhnB6XwD3pU40Z3KsmcDj/L0ewGk85dwHisgqPPMO50ymZ2IQ8G8RWcTZifJ7ICp1ohlPjyIPsNqJ+S0vPtuEEKuSakw64lmQJ0ZV97kdizFusJ6CMcaYNNZTMMYYk8Z6CsYYY9JYUjDGGJPGkoIxxpg0lhSMMcaksaRgjDEmzf8DiLeyK94XqikAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(1)\n",
    "plt.plot([0, 1], [0, 1], 'k--')\n",
    "plt.plot(fpr, tpr, \n",
    "         label='area = {:.3f}'.format(aucurve))\n",
    "plt.xlabel('False positive rate')\n",
    "plt.ylabel('True positive rate')\n",
    "plt.title('ROC curve')\n",
    "plt.legend(loc='best')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stacked LSTMs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(100, 10088, 1)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "time_series_length, input_dim, output_dim = 100, 10088, 1\n",
    "time_series_length, input_dim, output_dim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "\n",
    "# NOTE how batch_input_shape has one more parameter\n",
    "# than input_shape. The tuple describing the input\n",
    "# shape has 3 elements instead of 2, the first of which\n",
    "# is the new one.\n",
    "model.add(LSTM(20, return_sequences=True, stateful = True,\n",
    "            batch_input_shape=(\n",
    "                batch_size, time_series_length, input_dim))) \n",
    "model.add(LSTM(20, return_sequences=True, stateful = True)) \n",
    "model.add(LSTM(20, stateful = True))\n",
    "\n",
    "# model.add(Dense(output_dim, activation='relu'))\n",
    "model.add(Dense(output_dim, activation='sigmoid'))\n",
    "\n",
    "model.compile(loss='mean_squared_error',\n",
    "              optimizer='sgd',\n",
    "              metrics=['accuracy'])\n",
    "# model.compile(loss='mae', optimizer='adam') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "num_datarecs, num_testrecs = 2441, 272"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(27, 3)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_size = params_train[\"batch_size\"]\n",
    "# subtract 100 to account for bad records that were discarded\n",
    "steps_per_epoch = int(np.floor((num_datarecs) / batch_size))\n",
    "validation_steps = int(np.floor((num_testrecs) / batch_size))\n",
    "steps_per_epoch, validation_steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Removing spew to make notebook render on Github\n",
    "history = model.fit_generator(generator = dg_train,\n",
    "                    validation_data = dg_validation,\n",
    "                    validation_steps = validation_steps,\n",
    "                    steps_per_epoch = steps_per_epoch,\n",
    "                    epochs=10, \n",
    "                    verbose=1,\n",
    "                    workers=1,\n",
    "                    max_queue_size=1, \n",
    "                    use_multiprocessing=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the training and test loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xt4HPV97/H3V7u632xL8k2ysQADvoENwkASQkgg2CExJm0ppKQlTeu0KadpU3IKITdITpIT2jRJH5JAUve0TRNKCBcnmEtITJI2kCBjA5JtYmPAloRtIWNZ98vqe/6YlbSS12htSx5p9/N6nn12Z+Y3o+8u5vOb+c3srLk7IiKSGbLCLkBERE4ehb6ISAZR6IuIZBCFvohIBlHoi4hkEIW+iEgGUeiLiGQQhb6ISAZR6IuIZJBo2AWMVl5e7gsWLAi7DBGRKWXz5s2vu3vFWO0mXegvWLCA2trasMsQEZlSzOzVVNppeEdEJIMo9EVEMohCX0Qkg6QU+ma2ysxeNLNdZnZzkuUfN7NtZva8mf3MzE5JWDbfzB43s+3xNgvGr3wRETkWY4a+mUWAO4HVwGLgOjNbPKrZFqDG3c8G7gO+krDs34E73H0RsBI4MB6Fi4jIsUtlT38lsMvdd7t7L3APcFViA3ff5O6d8cmngSqAeOcQdfefxtu1J7QTEZGTLJXQrwT2Jkw3xOcdzYeBR+KvzwAOmdn9ZrbFzO6IHzmIiEgIUrlO35LMS/obi2Z2PVADXJKw/YuBFcAe4L+AG4B/GbXeOmAdwPz581MoSURkEhsYgP5u6OuCvs74607oG3zugv6u+PKER9FMqPnQhJaWSug3APMSpquAptGNzOwy4FbgEnfvSVh3i7vvjrd5ELiQUaHv7ncDdwPU1NToR3tFZOLE+qG3HXo74s/x12MF8hHzOhOCfdS8/u7jq61q5aQI/WeAhWZWDTQC1wIfSGxgZiuAu4BV7n5g1LrTzazC3ZuBdwL6uq2IpMY9CNLBgO4ZFdZD023x5474vIQwHz19rIFsWZBdANG84Dk7D7Lzg9c5RVA4c+S8Ee0KgvnR/PjyhMfQvHjbaD5EJv4mCWP+BXfvN7MbgceACLDe3evN7Hag1t03AHcARcAPzQxgj7uvcfeYmd0E/MyCBZuB70zUmxGRScI92PPtbk3yOBR/xKfHCuvko8lHysqG3KIgiHMKh58LK+LTg/OK4u0SpnMKhgN6dEhHcsCSjXJPTeY+uUZTampqXPfeEZkE+nug61CSwE72Oskj1vvm24/mQ15JQggnBHay8M4tHhXUhSPXi+acnM9lkjKzze5eM1a7SXfDNREZRwMD0NMKHS3QOeoxVniPNQwSyYG8aZBXGjzyp8P0BcPTIx7TRrbNK4Fo7kn5CGQkhb7IVOEeDHsMBffBhNevH2X+QfBY8u1lRUcFcSmUVMYDfNqowB4d4KXBOLRMOQp9kbD09yQJ7oNH7pF3tgzvqcd6km/LIlBQNvwoP2PkdEEZFCa8zp8RDI+k0Vi1pEahLzKeYv3QcQDa9kH7/uHn9v3Qth/a90FHPNx7246+nbxpwwFdUgVzzjkyxIceMyC3FLJ0/0QZm0JfJBW9nUFgDwZ32/6EME8I+M4Wkl5tUlAGRbOheBaUnT4c1gVlUFA+MsTzp5+US/ckM+lflmQud+h6IyG4DyQJ9vj8nsNHrp8VhaJZwWPafKiqGQ72xOfCioy/skQmD4W+pKeBWBDkh/ZA697g+XDTqKGW/cnHyLMLgiAvng2zlsLps4eni2bGA312MC6uIRWZYhT6MjXF+uBwIxzamxDse+HQq8Hr1gYY6B+5Tv704T3wU94SBHhxYqDHl+UWh/OeRE4Chb5MTv09QXAfejUI88G99cGQb2sCHxi5TvEcKJ0HlTWw5OpgyKV0PkybF8zPKQjnvYhMIgp9CUdvx6gw3zMy2Nv3jWxvWcFVLNPmwYK3BYE+bV482OdBaZW+7COSAoW+TIxYP7Tugdd3wRsvHxnsnS0j22dlB8E9bT4svCy+h54Q7MVzdUWLyDjQ/0Vy/NyhoxladsHrO4PnwcfBl2Ggb7htNG94r3zO8viQS0KwF83WSVGRk0ChL2PraYeDL8XDfdfIcE+8lDGSCzNOhYoz4awrg+vRyxbCjOrgskV9+1MkdAp9CcT6gmGXZHvtba8lNLRgb73sNDjn2niwnxaEe2kVZOnXMEUmM4V+JnEPvmjUsjMh3ON78G+8PPISx/zpQZCfeimUnx4P99ODPfns/PDeg4icEIV+OurvDYL9wPbhvfXBgE+830skN9hLn7kIFr0vCPXyhfHbBMwIr34RmTAK/anMPRh62V8P++tg/7bg9esvJuy1W3CitOx0mHfB8HBM+cLgEkidPBXJKAr9qaK3I9hz31+f8KgLfgBjUEkVzFoCZ7w7uH3AzEUw4zTd91xEhij0J5uBgWB8fTDYD8SfD77M0N0bc4qCQF+yNgj3WUuC6fzpoZYuIpOfQj9MnQfjwb4tPjxTH+zN93XGG1gwFDN7GZx9bRDus5bAtFM0LCMix0WhfzIMnlgdGnuvD8bf25qG2+TPgNlL4dw/GQ73irN0vxgRGVcK/fHW3gyvbT36idVIDpSfCdVvj4f74mCIpmiWvrwkIhNOoX8iug5B05b441lo2hrcW2bQ6BOrs5YEV89EssOrWUQyWkqhb2argK8DEeC77v7lUcs/DvwZ0A80A3/q7q8mLC8BtgMPuPuN41T7ydXTDvueh8Znh0P+4O7h5dOrYd5KuOAjMHdFEPA6sSoik8yYoW9mEeBO4HKgAXjGzDa4+7aEZluAGnfvNLO/BL4C/GHC8s8Dvxi/sidYX3cwPDMU8FuCIZrB+7eXVMHc5bDi+iDg5yzXl5lEZEpIZU9/JbDL3XcDmNk9wFXAUOi7+6aE9k8D1w9OmNl5wCzgUaBmHGoeX7G+4OqZpi3DIX9g2/AYfGEFzD03uDxy7orgUTQz3JpFRI5TKqFfCSQMVNMAXPAm7T8MPAJgZlnAPwIfBN51nDWOn4EYvP67kQG/74Xh30nNmxaE+lv+GirPDV6XVOoEq4ikjVRCP1niedKGZtcT7M1fEp/1UWCju++1NwlOM1sHrAOYP39+CiWlwD0Ycx8cnml8Fl57Dvo6guU5RcGwzMo/Hw746dUKeBFJa6mEfgMwL2G6Cmga3cjMLgNuBS5x9/iuMxcBF5vZR4EiIMfM2t395sR13f1u4G6AmpqapB3KmHo7YNfP4lfRxIO+uzVYFs0LvuA0OAZfeW5wFY1uAywiGSaV0H8GWGhm1UAjcC3wgcQGZrYCuAtY5e4HBue7+x8ltLmB4GTviMAfN72dcO8HISsaXDmz5P3DAV9xli6TFBEhhdB3934zuxF4jOCSzfXuXm9mtwO17r4BuINgT/6H8WGcPe6+ZgLrPlJRBaz7RRDwusGYiEhS5n58oykTpaamxmtra8MuQ0RkSjGzze4+5hWSumuXiEgGUeiLiGQQhb6ISAZR6IuIZBCFvohIBlHoi4hkEIW+iEgGUeiLiGQQhb6ISAZR6IuIZBCFvohIBlHoi4hkEIW+iEgGUeiLiGQQhb6ISAZR6IuIZBCFvohIBlHoi4hkEIW+iEgGUeiLiGQQhb6ISAZR6IuIZBCFvohIBlHoi4hkkJRC38xWmdmLZrbLzG5OsvzjZrbNzJ43s5+Z2Snx+cvN7Ckzq48v+8PxfgMiIpK6MUPfzCLAncBqYDFwnZktHtVsC1Dj7mcD9wFfic/vBP7Y3ZcAq4Cvmdm08SpeRESOTSp7+iuBXe6+2917gXuAqxIbuPsmd++MTz4NVMXn/87dd8ZfNwEHgIrxKl5ERI5NKqFfCexNmG6IzzuaDwOPjJ5pZiuBHOClJMvWmVmtmdU2NzenUJKIiByPVELfkszzpA3NrgdqgDtGzZ8D/AfwIXcfOGJj7ne7e42711RU6EBARGSiRFNo0wDMS5iuAppGNzKzy4BbgUvcvSdhfgnwMPApd3/6xMoVEZETkcqe/jPAQjOrNrMc4FpgQ2IDM1sB3AWscfcDCfNzgAeAf3f3H45f2SIicjzGDH137wduBB4DtgP3unu9md1uZmvize4AioAfmtlWMxvsFK4B3g7cEJ+/1cyWj//bEBGRVJh70uH50NTU1HhtbW3YZYiITClmttnda8Zqp2/kiohkEIW+iEgGUeiLiGQQhb6ISAZR6IuIZBCFvohIBlHoi4hkEIW+iEgGUeiLiGQQhb6ISAZR6IuIZBCFvohIBlHoi4hkEIW+iEgGSeWXs0REJr2+vj4aGhro7u4Ou5QJlZeXR1VVFdnZ2ce1vkJfRNJCQ0MDxcXFLFiwALNkP+099bk7LS0tNDQ0UF1dfVzb0PCOiKSF7u5uysrK0jbwAcyMsrKyEzqaUeiLSNpI58AfdKLvUaEvIjIODh06xDe/+c1jXu8973kPhw4dmoCKklPoi4iMg6OFfiwWe9P1Nm7cyLRp0yaqrCPoRK6IyDi4+eabeemll1i+fDnZ2dkUFRUxZ84ctm7dyrZt21i7di179+6lu7ubj33sY6xbtw6ABQsWUFtbS3t7O6tXr+Ztb3sbv/71r6msrOShhx4iPz9/XOtU6ItI2rntx/Vsazo8rttcPLeEz75vyVGXf/nLX6auro6tW7fy5JNPcuWVV1JXVzd0lc369euZMWMGXV1dnH/++fze7/0eZWVlI7axc+dOfvCDH/Cd73yHa665hh/96Edcf/314/o+FPoiIhNg5cqVIy6r/MY3vsEDDzwAwN69e9m5c+cRoV9dXc3y5csBOO+883jllVfGva6UQt/MVgFfByLAd939y6OWfxz4M6AfaAb+1N1fjS/7E+BT8aZfcPd/G6faRUSSerM98pOlsLBw6PWTTz7JE088wVNPPUVBQQHveMc7kl52mZubO/Q6EonQ1dU17nWNeSLXzCLAncBqYDFwnZktHtVsC1Dj7mcD9wFfia87A/gscAGwEvismU0fv/JFRCaH4uJi2traki5rbW1l+vTpFBQUsGPHDp5++umTXN2wVPb0VwK73H03gJndA1wFbBts4O6bEto/DQwOQl0B/NTdD8bX/SmwCvjBiZcuIjJ5lJWV8da3vpWlS5eSn5/PrFmzhpatWrWKb3/725x99tmceeaZXHjhhaHVmUroVwJ7E6YbCPbcj+bDwCNvsm7lsRQoIjJVfP/73086Pzc3l0ceeSTpssFx+/Lycurq6obm33TTTeNeH6QW+sm+/uVJG5pdD9QAlxzLuma2DlgHMH/+/BRKEhGR45HKl7MagHkJ01VA0+hGZnYZcCuwxt17jmVdd7/b3WvcvaaioiLV2kVE5BilEvrPAAvNrNrMcoBrgQ2JDcxsBXAXQeAfSFj0GPBuM5seP4H77vg8EREJwZjDO+7eb2Y3EoR1BFjv7vVmdjtQ6+4bgDuAIuCH8ZsB7XH3Ne5+0Mw+T9BxANw+eFJXREROvpSu03f3jcDGUfM+k/D6sjdZdz2w/ngLFBGR8aMbromIZBCFvojIODjeWysDfO1rX6Ozs3OcK0pOoS8iMg6mSujrhmsiIuMg8dbKl19+OTNnzuTee++lp6eHq6++mttuu42Ojg6uueYaGhoaiMVifPrTn2b//v00NTVx6aWXUl5ezqZNm8b+YydAoS8i6eeRm2HfC+O7zdnLYPWXj7o48dbKjz/+OPfddx+//e1vcXfWrFnDL3/5S5qbm5k7dy4PP/wwENyTp7S0lK9+9ats2rSJ8vLy8a05CQ3viIiMs8cff5zHH3+cFStWcO6557Jjxw527tzJsmXLeOKJJ/j7v/97fvWrX1FaWnrSa9OevoiknzfZIz8Z3J1bbrmFj3zkI0cs27x5Mxs3buSWW27h3e9+N5/5zGeSbGHiaE9fRGQcJN5a+YorrmD9+vW0t7cD0NjYyIEDB2hqaqKgoIDrr7+em266iWefffaIdSea9vRFRMZB4q2VV69ezQc+8AEuuugiAIqKivje977Hrl27+MQnPkFWVhbZ2dl861vfAmDdunWsXr2aOXPmTPiJXHNPesPM0NTU1HhtbW3YZYjIFLN9+3YWLVoUdhknRbL3amab3b1mrHXTaninua2HgYHJ1YmJiEwmaRP6u5vbeec/PMk9z+wdu7GISIZKm9CvLi9kSWUJX3pkOwcOH/mDwyIikkahb2Z88epl9PQPcNtPto29goikncl2jnIinOh7TJvQBzi1oogbLz2dh59/jZ/v2B92OSJyEuXl5dHS0pLWwe/utLS0kJeXd9zbSLtLNv/iktP48XNNfPrBei742zIKc9PuLYpIElVVVTQ0NNDc3Bx2KRMqLy+Pqqqq414/7RIxJ5rFF9+/jD/49lP8009/x6feuzjskkTkJMjOzqa6ujrsMia9tBreGXT+ghlct3I+6//nZeoaW8MuR0Rk0kjL0Ae4efVZlBXlcvP9z9MfGwi7HBGRSSFtQ780P5vPvm8xdY2H+X+/fiXsckREJoW0DX2AK5fN4dIzK/jqT39Hwxsn51dpREQms7QOfTPj82uX4g6feag+rS/lEhFJRVqHPkDV9AL+7t1n8PMdB9j4wr6wyxERCVVKoW9mq8zsRTPbZWY3J1n+djN71sz6zez3Ry37ipnVm9l2M/uGmdl4FZ+qG96ygKWVJXzux/W0dvWd7D8vIjJpjBn6ZhYB7gRWA4uB68xs9MXve4AbgO+PWvctwFuBs4GlwPnAJSdc9TGKRrL40tVn09Lew/99dMfJ/vMiIpNGKnv6K4Fd7r7b3XuBe4CrEhu4+yvu/jww+tpIB/KAHCAXyAZCuT/CsqpSPvTWar7/mz3UvnIwjBJEREKXSuhXAon3K26IzxuTuz8FbAJeiz8ec/ftx1rkePn45WdQOS2fW+5/gd5+XbsvIpknldBPNgaf0mUwZnY6sAioIugo3mlmb0/Sbp2Z1ZpZ7UTeN6MwN8rn1y5h54F27vrFSxP2d0REJqtUQr8BmJcwXQU0pbj9q4Gn3b3d3duBR4ALRzdy97vdvcbdayoqKlLc9PF551mzuHLZHP550y52N7dP6N8SEZlsUgn9Z4CFZlZtZjnAtcCGFLe/B7jEzKJmlk1wEje04Z1Bn33fYnKjWdz6QJ2u3ReRjDJm6Lt7P3Aj8BhBYN/r7vVmdruZrQEws/PNrAH4A+AuM6uPr34f8BLwAvAc8Jy7/3gC3scxmVmSx82rz+Kp3S3ct7kh7HJERE4am2x7ujU1NV5bWzvhf2dgwLnmrqfY1dzOzz5+CWVFuRP+N0VEJoqZbXb3mrHapf03co8mK8v44vuX0dHTzxceDn3ESUTkpMjY0Ac4Y1Yxf3HJaTywpZFf7UzvX9sREYEMD32Av7r0dKrLC7n1gTq6emNhlyMiMqEyPvTzsiP8n6uXsudgJ9/4+c6wyxERmVAZH/oAbzmtnN8/r4rv/HI3O/YdDrscEZEJo9CPu/U9iyjJz+bmH71AbGByXdEkIjJeFPpx0wtz+PR7F7F17yH+8zevhl2OiMiEUOgnWLu8kosXlvOVR19kX2t32OWIiIw7hX4CM+MLa5fSFxvgcxvqx15BRGSKUeiPckpZIR+7bCGP1u/j8Xr9vKKIpBeFfhJ/fvGpnDW7mM9uqKe9pz/sckRExo1CP4nsSBZffP8y9h3u5h8eezHsckRExo1C/yjOnT+dD154Cv/21Cts3Xso7HJERMaFQv9NfOKKM5lZnMst979AX0w/rygiU59C/00U52Vz25olbH/tMOv/++WwyxEROWEK/TFcsWQ2ly+exT898Tv2HuwMuxwRkROi0B+DmXHbmiVEzLj1Qf28oohMbQr9FMydls9NV5zJL3/XzIbnUv1NeBGRyUehn6I/vmgB51SVcvuPt3GoszfsckREjotCP0WRLONL7z+bQ119fGnjjrDLERE5Lgr9Y7B4bgl/dnE1/1W7l6d3t4RdjojIMVPoH6O/edcZzJuRzycfeIGefv28oohMLQr9Y5SfE+ELa5exu7mDb256KexyRESOiUL/OFxyRgVXLZ/LN5/cxa4DbWGXIyKSspRC38xWmdmLZrbLzG5OsvztZvasmfWb2e+PWjbfzB43s+1mts3MFoxP6eH69HsXU5AT5ZP31zGgn1cUkSlizNA3swhwJ7AaWAxcZ2aLRzXbA9wAfD/JJv4duMPdFwErgQMnUvBkUV6Uyyffcxa/feUg99buDbscEZGUpLKnvxLY5e673b0XuAe4KrGBu7/i7s8DI+5KFu8cou7+03i7dndPm3sZXFMzjwuqZ/DFjds50KafVxSRyS+V0K8EEndlG+LzUnEGcMjM7jezLWZ2R/zIIS2YGV98/zK6+wb4/E+2h12OiMiYUgl9SzIv1UHsKHAxcBNwPnAqwTDQyD9gts7Mas2strm5OcVNTw6nVRTx0UtP48fPNbHpxbQYuRKRNJZK6DcA8xKmq4BUb0DTAGyJDw31Aw8C545u5O53u3uNu9dUVFSkuOnJ4y/fcRqnVRTyqQfq6OzVzyuKyOSVSug/Ayw0s2ozywGuBTakuP1ngOlmNpjk7wS2HXuZk1tuNMKX3n82jYe6+NoTO8MuR0TkqMYM/fge+o3AY8B24F53rzez281sDYCZnW9mDcAfAHeZWX183RjB0M7PzOwFgqGi70zMWwnXyuoZXHv+PP7lv1+mrrE17HJERJKyyXZ/+JqaGq+trQ27jOPS2tnHu776C+ZOy+M/PnwBpfnZYZckIhnCzDa7e81Y7fSN3HFUWpDN59Ys5vmGVs7/whN85D9qebTuNbr7dI8eEZkcomEXkG7ee/ZcTplRyP1bGvjxc008Vr+f4rwo71k6h6tWzOXC6jKyspJdECUiMvE0vDOB+mMD/M9LLTy0pZHH6vfR0RtjdkkeVy2fy1XLK1k0pxgzdQAicuJSHd5R6J8kXb0xfrp9Pw9taeQXv2umf8A5Y1YRVy2v5Krlc6maXhB2iSIyhSn0J7GDHb08/MJrPLSlkdpX3wDg/AXTWbuikiuXzWFaQU7IFYrIVKPQnyL2Huzkoa2NPLi1iV0H2smOGJecMZO1K+Zy2aJZ5GWnzV0rRGQCKfSnGHenvukwD21tZMNzTew/3ENRbpQrlsxm7Yq5vOW0ciI6ASwiR6HQn8JiA87Tu1t4cEsjj9bto62nn5nFubzvnLmsXV7J0soSnQAWkREU+mmiuy/Gz3cc4MEtjWx68QB9MefUikLWLq9k7fJK5pfpBLCIKPTTUmtnHxvrXuOBLY389uWDAJw7f9rQCeCyotyQKxSRsCj001zjoS42bG3ioa2N7NjXRjTLuHhhOWtXVHL54lkU5Oh7dyKZRKGfQXbsO8yDW5rYsLWRptZuCnIiXLFkNlcum8Py+dMo1xGASNpT6GeggQHnmVcO8uDWJh5+vonD3cG9/eeU5rFkbilLK0tYOreUpZWlzCrJ1clgkTSi0M9wPf0xnn31EPVNrdQ1tlLXdJiXmtsZ/M9dXpRzREdQNT1fHYHIFJVq6GvgN03lRiNcdFoZF51WNjSvo6efHfsOU9d4mLrGVl5obOW/d71ObCDoCUrzs4c6gSWVpSydW8KCskLdIE4kjSj0M0hhbpTzTpnBeafMGJrX3RfjxX1t1DW1Utd4mPqmVv71f16hNzYQrJMTYcncUpYkHBGcVlFINKK7cotMRQr9DJeXHeGcedM4Z960oXl9sQF27m+PDwsFw0M/+O0euvuCjiA3msWiOSUsqwyGh5bMLeWMWcXkRNURiEx2GtOXlMQGnN3N7UNHBHWNrdQ3Haa9JzhZnB0xzpxdPGJoaNGcEt07SOQk0YlcmXADA86rBzuHjgjqGw9T19TKoc4+ACJZxukVRZw5u5h5M/Kpml5A1fTgee60PHKj6hBExotO5MqEy8oyqssLqS4v5H3nzAWCG8c1HuoaOj9Q19jKs3ve4OEXXhs6YQxgBjOLc4c6gnkJHULV9HzmqFMQmRAKfRlXZhYP7gJWLZ09NL8/NsC+w900vNEVf3QOPW9+9Q1+8vyRncKs4rx4RzDcGcybEe8USvN1DkHkOCj05aSIRrKGOoNkknUKew8Gz8+88gYbnmsioU/ADGaX5I3oEEYcKahTEElKoS+TwlidQl9sgH2t3aOOEoLXv335IA9t7RrRKWQNdQrDHcKs0jzKCnOpKM6hrDCXsqIcinKj+kKaZJSUQt/MVgFfByLAd939y6OWvx34GnA2cK273zdqeQmwHXjA3W8cj8Ils2RHspg3o4B5MwqAsiOWD3YKe0d1CA1vdPGblw/y4KhOYVBuNIvyoqADKCvMoawol/KiXMqLcuLzhqenF+aQre8nyBQ3ZuibWQS4E7gcaACeMbMN7r4todke4AbgpqNs5vPAL06sVJGjG9kpHKkvNkBLey+vt/fQ0tHL6209tHT0xOcF85vbe9ixr42W9t6hL6eNNq0gO+gkCnOGOovhTiPoHAandRQhk1Eqe/orgV3uvhvAzO4BrgKGQt/dX4kvO+L/FDM7D5gFPAqMeTmRyETIjmQxuzSP2aV5Y7Z1dw5399MS7yBa2ntobg+eW9p7aeno4fW2XrbvO0xLey+tXX1Jt5MTzaK8MIfy4tyho4iyohym5edQnBelOC9KSV52/HU2JfnBc2FORJ2FTJhUQr8S2Jsw3QBckMrGzSwL+Efgg8C7jrk6kRCYGaX52ZTmZ3Nqxdjte/sHONgxfBTR0t4TvI4fRbR0pHYUMSjLoCg36ACK86KU5GdTkjc8PdxZDE8X5wVtSvKDefnZ6jgkuVRCP9m/nFS/0fVRYKO7732zf4Bmtg5YBzB//vwUNy0yOeREj+0oortvgLbuPg5393O4u4+27n7ajnju53BX0Katu4+mQ9209bQNzU92fiJRJMuGO4Tc4aOIxKOLgpwohbkR8rMjFOZGyc+JUJgTpSAnMvQ6eI7oXktpJJXQbwDmJUxXAU0pbv8i4GIz+yhQBOSYWbu735zYyN3vBu6G4Bu5KW5bZMoxM/LjoTqz5Pi24e509saGOonDo56TdR5t3f3sPdg5tKy9p3/MjiNRTiSLgtwIBdkRCnKDjiF4DHcMBTkj5xfkxNtmR4b34TcyAAAE4klEQVRfj1qeG83SEclJlkroPwMsNLNqoBG4FvhAKht39z8afG1mNwA1owNfRI6NmVGYG6UwN5rS0UUyg0ccnb39dPbG6OyN0dHbT1dvjI6efrr6YnT0xEYs7+rtp6M3FrSJzz/Q1h0sT2jbfwy9SZYFN/3Lyw6OOHKzs8iLBp1iXvx1Xk4keM7OIj/eNi87a2i9wXWH5x1lWTSi24STQui7e7+Z3Qg8RnDJ5np3rzez24Fad99gZucDDwDTgfeZ2W3uvmRCKxeR45Z4xHHkBbAnprc/sTMJnjt6YnT19Y/oIAY7kO6+GF19Mbr7Bujuj9HdGwue+wZ4o6OP7v4YPX0D8TbB41iOUhLlRLPIiwadQn5CZ5KbHRx15ESyyInGH4mvo1nkHrEsMmI6N3qUdZMsC3O4TDdcE5Epxd3pjQ3Q3TdAT7yzSOwQBjuQnv7hTqW7fyB4He9AhtvFO5u+GL2xAXr744/E1/0D9MSnx0uWMaLzGOwUllaW8s/XrTiubeqGayKSlsyM3GgkuCFffvZJ+7vuTl/Mj+gQemMxevqP0mHEBt50WW9/wvLYAPOm50/4+1Doi4ikwMzIiVpwT6fcsKs5froOS0Qkgyj0RUQyiEJfRCSDKPRFRDKIQl9EJIMo9EVEMohCX0Qkgyj0RUQyyKS7DYOZNQOvnsAmyoHXx6mcqU6fxUj6PEbS5zEsHT6LU9x9zF+AmHShf6LMrDaV+09kAn0WI+nzGEmfx7BM+iw0vCMikkEU+iIiGSQdQ//usAuYRPRZjKTPYyR9HsMy5rNIuzF9ERE5unTc0xcRkaNIm9A3s1Vm9qKZ7TKzjP4dXjObZ2abzGy7mdWb2cfCrilsZhYxsy1m9pOwawmbmU0zs/vMbEf838hFYdcUJjP72/j/J3Vm9gMzO74fHp4i0iL0zSwC3AmsBhYD15nZ4nCrClU/8Hfuvgi4EPirDP88AD4GbA+7iEni68Cj7n4WcA4Z/LmYWSXw10CNuy8l+B3wa8OtamKlRegDK4Fd7r7b3XuBe4CrQq4pNO7+mrs/G3/dRvA/dWW4VYXHzKqAK4Hvhl1L2MysBHg78C8A7t7r7ofCrSp0USDfzKJAAdAUcj0TKl1CvxLYmzDdQAaHXCIzWwCsAH4TbiWh+hrwv4Hx+2XrqetUoBn41/hw13fNrDDsosLi7o3APwB7gNeAVnd/PNyqJla6hL4lmZfxlyWZWRHwI+Bv3P1w2PWEwczeCxxw981h1zJJRIFzgW+5+wqgA8jYc2BmNp1gVKAamAsUmtn14VY1sdIl9BuAeQnTVaT5IdpYzCybIPD/093vD7ueEL0VWGNmrxAM+73TzL4XbkmhagAa3H3wyO8+gk4gU10GvOzuze7eB9wPvCXkmiZUuoT+M8BCM6s2sxyCEzEbQq4pNGZmBGO22939q2HXEyZ3v8Xdq9x9AcG/i5+7e1rvyb0Zd98H7DWzM+Oz3gVsC7GksO0BLjSzgvj/N+8izU9sR8MuYDy4e7+Z3Qg8RnD2fb2714dcVpjeCnwQeMHMtsbnfdLdN4ZYk0we/wv4z/gO0m7gQyHXExp3/42Z3Qc8S3DV2xbS/Nu5+kauiEgGSZfhHRERSYFCX0Qkgyj0RUQyiEJfRCSDKPRFRDKIQl9EJIMo9EVEMohCX0Qkg/x/e4ZTJDXKpf4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot history\n",
    "plt.plot(history.history['loss'], label='train')\n",
    "plt.plot(history.history['val_loss'], label='test')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "90/90 [==============================] - 1s 7ms/step\n",
      "\n",
      "test loss:  0.1466633528470993\n",
      "test accuracy:  0.7777777910232544\n"
     ]
    }
   ],
   "source": [
    "test_loss, test_acc = model.evaluate(X_test, y_test,\n",
    "                                        batch_size = batch_size)\n",
    "print(\"\\ntest loss: \", test_loss)\n",
    "print(\"test accuracy: \", test_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Make Predictions and Plot ROC-AUC Metric"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate ROC-AUC for a single batch of test data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((90, 100, 10088), (90, 1))"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test, y_test = next(dg_validation)\n",
    "X_test.shape, y_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that with a stateful LSTM, you have to specify the batch_size with the `predict()` method as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = model.predict(X_test, batch_size = batch_size).ravel()\n",
    "fpr, tpr, thresholds = roc_curve(y_test, y_pred)\n",
    "aucurve = auc(fpr, tpr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmcTfX/wPHXuxmSomSLxr7OYkljl4iIFCqlha8a/LQq3xaiQuUb2SJriLSolFIprb71LZIkWTOEGUuW7OvMeP/+uMdtGjPmYs6cufe+n4/Hfcw5937uPe9jxrzn8/mc8/6IqmKMMcYAnOd1AMYYY/IOSwrGGGP8LCkYY4zxs6RgjDHGz5KCMcYYP0sKxhhj/CwpGGOM8bOkYEKOiGwUkSMiclBEtovIdBG5KEObRiLytYgcEJF9IvKRiMRkaFNYREaLyGbnsxKd/WK5e0bG5B5LCiZU3aCqFwG1gSuAfidfEJGGwOfAh0BpoALwK/C9iFR02uQHvgJigeuAwkAjYDdQz62gRSTSrc82JhCWFExIU9XtwHx8yeGkYcBrqvqSqh5Q1b9UdQCwCBjotOkKlAU6quoqVT2hqjtU9VlVnZfZsUQkVkS+EJG/RORPEXnSeX66iDyXrl0zEUlOt79RRJ4QkeXAIREZICKzM3z2SyIyxtm+WESmisg2EdkiIs+JSMQ5/lMZA1hSMCFORKKANkCis18Q31/872bS/B3gWme7JfCZqh4M8DiFgC+Bz/D1Pirj62kE6nbgeuASYCbQVkQKO58dAdwKvOm0nQGkOse4AmgFdD+DYxmTJUsKJlR9ICIHgCRgB/CM8/yl+H7ut2Xynm3AyfmColm0yUo7YLuqjlDVo04P5MczeP8YVU1S1SOquglYCnRwXrsGOKyqi0SkJL4k97CqHlLVHcAooPMZHMuYLFlSMKGqg6oWApoB1fn7l/0e4ARQKpP3lAJ2Odu7s2iTlTLA+rOK1Ccpw/6b+HoPAHfwdy+hHJAP2CYie0VkLzAJKHEOxzbGz5KCCWmq+l9gOjDc2T8ELAQ6ZdL8Vv4e8vkSaC0iFwZ4qCSgUhavHQIKptu/LLNQM+y/CzRzhr868ndSSAKOAcVU9RLnUVhVYwOM05jTsqRgwsFo4FoROTnZ3Bf4l4g8JCKFRKSIMxHcEBjktJmJ7xfweyJSXUTOE5GiIvKkiLTN5BgfA5eJyMMicr7zufWd15bhmyO4VEQuAx7OLmBV3QksAF4F/lDV1c7z2/BdOTXCuWT2PBGpJCJXn8W/izGnsKRgQp7zC/Y14Cln/39Aa+AmfPMGm/BN2DZR1XVOm2P4JpvXAF8A+4HF+IahTpkrUNUD+CapbwC2A+uA5s7LM/Fd8roR3y/0twMM/U0nhjczPN8VyA+swjccNpszG+oyJktii+wYY4w5yXoKxhhj/CwpGGOM8bOkYIwxxs+SgjHGGL+gK75VrFgxLV++vNdhGGNMUPn55593qWrx7NoFXVIoX748S5Ys8ToMY4wJKiKyKZB2NnxkjDHGz5KCMcYYP0sKxhhj/IJuTiEzKSkpJCcnc/ToUa9DMY4CBQoQFRVFvnz5vA7FGHMGQiIpJCcnU6hQIcqXL4+IeB1O2FNVdu/eTXJyMhUqVPA6HGPMGXBt+EhEponIDhFZkcXrIiJjnMXQl4tInbM91tGjRylatKglhDxCRChatKj13IwJQm7OKUzHt+B5VtoAVZxHT2DCuRzMEkLeYt8PY4KTa8NHqvqtiJQ/TZP2+BZPV2CRiFwiIqWcevHGBJWjKWlM/2Ejh4+leh2KCUHHU1I4fPgwNzeoSq0yl7h6LC/nFC7nn0sQJjvPnZIURKQnvt4EZcuWzZXgQsHPP/9Mt27dOHLkCG3btuWll1465S/4PXv2cM8997B+/XoKFCjAtGnTiIuL87+elpZGfHw8l19+OR9//DEAV111FQcOHABgx44d1KtXjw8++CD3TiwP+nnTHl74dA0A1kkyOUpB9QQAVaJKhHRSyOy/TqaLO6jqZGAyQHx8fNAuAJGWlkZERESuHe/ee+9l8uTJNGjQgLZt2/LZZ5/Rpk2bf7QZMmQItWvXZs6cOaxZs4b777+fr776yv/6Sy+9RHR0NPv37/c/99133/m3b775Ztq3b+/+yeRxaSd8P5bv3duQK8td6nE0JhTs3buXxx57jClTplC5cmWmTJnC1Q3Lu35cL5NCMr7Fzk+KArZ6FMs569ChA0lJSRw9epTevXvTs2dPAC666CL69OnD/PnzGTFiBBdccAF9+vTh4MGDFCtWjOnTp1OqVCleeeUVJk+ezPHjx6lcuTIzZ86kYMGC2Rw1a9u2bWP//v00bNgQgK5du/LBBx+ckhRWrVpFv379AKhevTobN27kzz//pGTJkiQnJ/PJJ5/Qv39/Ro4cecoxDhw4wNdff82rr7561nFmZtw3ifyatDdHP9Ntuw4e8zoEE0LS0tJo1KgRa9eu5fHHH2fgwIFccMEFuXJsL5PCXOABEZkF1Af25cR8wqCPVrJq6/7sG56BmNKFeeaG06+LPm3aNC699FKOHDlC3bp1ufnmmylatCiHDh0iLi6OwYMHk5KSwtVXX82HH35I8eLFefvtt+nfvz/Tpk3jpptuokePHgAMGDCAqVOn8uCDD/7jGN988w2PPPLIKccuWLAgP/zwwz+e27JlC1FRUf79qKgotmzZcsp7a9Wqxfvvv0+TJk1YvHgxmzZtIjk5mZIlS/Lwww8zbNgw/1BRRnPmzKFFixYULlz4tP82Z2rytxs4T6Bk4QI5+rluq1u+CBWKXeR1GCaI7d69m0svvZSIiAief/55ypQpQ3x8fK7G4FpSEJG3gGZAMRFJBp4B8gGo6kRgHtAWSAQOA3e7FUtuGDNmDHPmzAEgKSmJdevWUbRoUSIiIrj55psBWLt2LStWrODaa68FfH8NlCrlW1p3xYoVDBgwgL1793Lw4EFat259yjGaN2/OsmXLAoons2VWM7siqG/fvvTu3ZvatWtTo0YNrrjiCiIjI/n4448pUaIEV155JQsWLMj0GG+99Rbdu3cPKJ4z1b725Qy88fSJ2JhQoaq88cYb9O7dmxdeeIEePXrQsWNHT2Jx8+qj27N5XYH7c/q42f1F74YFCxbw5ZdfsnDhQgoWLEizZs381+gXKFDAP4+gqsTGxrJw4cJTPqNbt2588MEH1KpVi+nTp2f6i/hMegpRUVEkJyf795OTkylduvQp7y1cuLB/+EdVqVChAhUqVGDWrFnMnTuXefPmcfToUfbv389dd93F66+/Dvj+olm8eLE/ERpjzk5SUhK9evVi3rx5NGjQgMaNG3saj9U+ygH79u2jSJEiFCxYkDVr1rBo0aJM21WrVo2dO3f6k0JKSgorV64EfOPzpUqVIiUlhTfeeCPT95/sKWR8ZEwIAKVKlaJQoUIsWrQIVeW1117LdEJ47969HD9+HIApU6bQtGlTChcuzH/+8x+Sk5PZuHEjs2bN4pprrvEnBIB3332Xdu3aUaBAcA3xGJOXvPXWW8TGxrJgwQJGjx7N//73P2JiYjyNKSTKXHjtuuuuY+LEidSsWZNq1arRoEGDTNvlz5+f2bNn89BDD7Fv3z5SU1N5+OGHiY2N5dlnn6V+/fqUK1eOGjVqZDmOfyYmTJjgvyS1TZs2/knmiRMnAtCrVy9Wr15N165diYiIICYmhqlTpwb02bNmzaJv377nHKMx4axIkSLUr1+fyZMn55mSMJLZ2HNeFh8frxkX2Vm9ejXR0dEeRWSycrbfl1qDPqfjFTanYEJPamoqo0aN4vjx4/Tv3x/wDdvmRgUAEflZVbOdtbbhI2OMyQW//vorDRo04PHHH2f58uX+i0HyWkkYSwrGGOOiY8eO8dRTTxEfH09SUhLvvvsus2bNynPJ4KSQSQrBNgwW6uz7YYzPunXrGDp0KHfccQerVq3illtuybMJAUIkKRQoUIDdu3fbL6I84uR6CnZlkglXBw8e9F9FGBcXx5o1a5gxYwZFixb1OLLshcTVRyevyd+5c6fXoYQ9VSUlTTmUJmxNKcjiXZvP+DOOp55wITJjcscXX3xBz5492bRpE3Xq1CE6OpqKFSt6HVbAQiIp5MuXL89czhVOdh08xupt+1m1db/v67b9rN95yF8c7lwUL3R+DkRoTO7Zs2cPjz76KNOmTaNq1ar897//DcqrIkMiKRh3pZ1Q/th1iFXb9v8jCew48HcRuFIXFyCmVGFaxVxGTOnCVC1ZiIL5z64irAhcFmR1j0x4S0tLo3Hjxvz+++/069ePp59+OmiHTy0phLDV2/bz7pJkNPOK5Nk6fCyNNX8eYO32/RxN8Q3pRJ4nVClZiCZVihFTqjAxpQoTXaowRS7Mn5OhGxMUdu3a5S9gN2TIEMqWLUudOme9snCeYEkhhL3542ZmLtpEoQJn920+P/I8qpQoxB31yhFT2pcAKpe4iPyRIXF9gjFnTVWZOXMmDz/8MC+88AI9e/akQ4cOXoeVIywphDBFKXphfn5+6lqvQzEmZGzatIn/+7//Y/78+TRq1IimTZt6HVKOsqSQx736/R98serPs3rv+p0HczgaY8Lb66+/zr333ouqMnbsWO677z7OOy+0es6WFPK495Ymk/TXEaqWPPPFW8peWpA65Yq4EJUx4al48eI0btyYSZMmUa5cOa/DcYUlhSAQX64IU7vV9ToMY8JOSkoKI0aMICUlhaeeeorWrVvTqlWrPH1H8rmypOAyVaXTxIX8sevQWb1/75EUuzzTGA/88ssvJCQk8Msvv9C5c2d/NdNQTghgScF1qSeUJZv2UKvMJdS4/OzWMr6+xqkrphlj3HH06FEGDx7MsGHDKFasGO+99x433XST12HlGksKATpwNMV/rf6ZSD3he8+10SV44JoqOR2WMSaHJSYmMnz4cLp27cqIESMoUiS85uUsKQRg/c6DtBr17TmVb8gXEVpXKBgTSg4ePMicOXPo0qULcXFxrF27NmxL51hSCMDug8dJO6F0a1SeSiXO/CqgCBHaxF3mQmTGmHM1f/58evbsSVJSEvHx8URHR4dtQgBLCmfk2piSNK5czOswjDE5YPfu3fTp04fXXnuN6tWr89133wVlAbucZknBGBN2ThawS0xMpH///gwYMCBoC9jlNEsKxpiwsXPnTooWLUpERARDhw6lXLly1K5d2+uw8hSb/TTGhDxV5dVXX6Vq1aq88sorALRv394SQiYsKRhjQtrGjRtp3bo199xzDzVq1KB58+Zeh5SnWVIwxoSsmTNnEhcXx8KFCxk/fjwLFiygatWqXoeVp9mcgjEmZJUsWZKmTZsyceJEypYt63U4QcGSgjEmZKSkpDBs2DDS0tJ4+umnadWqFa1atfI6rKBiw0fGmJCwdOlS6taty4ABA1i7di2qZ1+BIJxZUjDGBLUjR47Qt29f6tWrx59//smcOXN44403Qr6aqVtcTQoicp2IrBWRRBHpm8nrZUXkGxH5RUSWi0hbN+MxxoSeDRs2MHLkSLp168aqVatCZq1kr7iWFEQkAhgHtAFigNtFJCZDswHAO6p6BdAZGO9WPMaY0LF//36mT58OQGxsLOvWrWPKlClhV9HUDW72FOoBiaq6QVWPA7OA9hnaKHBykYGLga0uxmOMCQHz5s0jLi6OhIQEVq9eDRCyS2N6wc2kcDmQlG4/2XkuvYHAXSKSDMwDHszsg0Skp4gsEZElO3fudCNWY0wet2vXLrp06cL1119PoUKF+P77762AnQvcTAqZzfJkvBzgdmC6qkYBbYGZInJKTKo6WVXjVTW+ePHiLoRqjMnLThawmzVrFk8//TRLly6lQYMGXocVkty8TyEZKJNuP4pTh4cSgOsAVHWhiBQAigE7XIzLGBMk/vzzT4oXL05ERATDhw+nXLly1KxZ0+uwQpqbPYWfgCoiUkFE8uObSJ6boc1moAWAiEQDBQAbHzImzKkqU6dOpVq1akyePBmAG264wRJCLnAtKahqKvAAMB9Yje8qo5UiMlhEbnSa/RvoISK/Am8B3dTuODEmrG3YsIGWLVvSvXt3ateuTcuWLb0OKay4WuZCVefhm0BO/9zT6bZXAY3djMEYEzxmzJjBfffdR0REBBMnTqRHjx6cd57dY5ubrPaRMSbPKF26NNdccw0TJkwgKirK63DCkiUFY4xnjh8/zgsvvMCJEycYOHAg1157Lddee63XYYU165cZYzzx008/ceWVV/LMM8+wYcMGK2CXR1hSMMbkqsOHD/Poo4/SoEED9uzZw9y5c3nttdesgF0eYUnBGJOr/vjjD8aOHUuPHj1YuXIlN9xwg9chmXRsTsEY47p9+/bx/vvvc/fddxMbG0tiYiJlypTJ/o0m11lPwRjjqk8++YTY2Fi6d+/OmjVrACwh5GGWFIwxrti5cyd33nkn7dq1o0iRIixcuJDq1at7HZbJhg0fGWNyXFpaGk2aNOGPP/5g0KBB9O3bl/z583sdlgmAJQVjTI7Zvn07JUqUICIighEjRlC+fHni4uK8DsucARs+MsacsxMnTjBp0iSqVq3KpEmTAGjXrp0lhCCUbVIQkQtEpJ+ITHT2K4tIG/dDM8YEg8TERFq0aEGvXr2oW7curVu39jokcw4C6SlMw7dgThNnfyswxLWIjDFB49VXX6VGjRosXbqUV155hS+//JKKFSt6HZY5B4EkhSqqOgRIAVDVw2S+qpoxJsyULVuW1q1bs2rVKrp37253JYeAQCaajzsroimAiFQAjrsalTEmTzp27Bj/+c9/OHHiBIMHD6ZFixa0aNHC67BMDgqkp/As8BkQJSIzgG+AJ12NyhiT5/z4449ceeWVDBo0iM2bN1sBuxCVbVJQ1U+BTkAPYA5QT1W/dDswY0zecOjQIfr06UPDhg3Zt28fH3/8MdOnT7ehohAVyNVHn6vqTlX9UFU/UNUdIvJ5bgRnjPHepk2bGD9+PL169WLlypVcf/31XodkXJTlnIKI5AcKACVFpBB/Ty4XBsrmQmzGGI/s3buX2bNn0717d2JiYkhMTLSV0MLE6XoK9wMrgerO15OP+cBE90Mzxnjhww8/JCYmhl69evkL2FlCCB9ZJgVVHaWqZYAnVLWsqpZxHrGqOjoXYzTG5IIdO3bQuXNnOnToQPHixVm0aJEVsAtD2V6SqqqjRaQ6EINvOOnk82+6GZgxJvekpaXRuHFjNm/ezHPPPcfjjz9Ovnz5vA7LeCDbpCAiA4BW+IaR5gOtgf8BlhSMCXJbt27lsssuIyIigpdeeony5csTExPjdVjGQ4Hcp3Ab0BzYpqpdgFpYdVVjgtqJEyeYMGEC1atXZ+JE3xRh27ZtLSGYgJLCEVVNA1Kdq5C2A1bcxJgg9fvvv9O8eXPuu+8+6tevT5s2Vt/S/C2QpPCLiFyCrzDeEmAxsNTVqIwxrpg6dSq1atVi+fLlTJs2jc8//5wKFSp4HZbJQ047DCS+WxYHqupeYJyIzAcKq6olBWOCUPny5WnTpg3jxo2jVKlSXodj8qDTJgVVVRH5GLjS2U/MlaiMMTni2LFjPPvsswA899xzVsDOZCuQ4aPFIlLH9UiMMTnqhx9+oHbt2jz//PNs27bNCtiZgASSFJrgSwxrRWSpiPwiIjZ8ZEwedfDgQXr37k2TJk04fPgwn332GVOnTrUCdiYggVxa2uFsP1xErgNeAiKAKar6QiZtbgUG4luv4VdVveNsj2eMgc2bNzNp0iTuv/9+hgwZQqFChbwOyQSRQO5oXn82HywiEcA44FogGfhJROaq6qp0baoA/YDGqrpHREqczbGMCXd79uzh3XffpWfPnsTExLBhwwZKly7tdVgmCAUyfHS26gGJqrpBVY8Ds4D2Gdr0AMap6h4AVd3hYjzGhKQ5c+YQExPDfffdx9q1awEsIZiz5mZSuBxISref7DyXXlWgqoh8LyKLnOGmU4hITxFZIiJLdu7c6VK4xgSX7du306lTJ2666SYuu+wyFi9eTLVq1bwOywS5gMpViEgUUEVVvxGR84FIVT2U3dsyeS7j5Q+RQBWgGRAFfCcicc59EX+/SXUyMBkgPj7eLqEwYS8tLY2rrrqKpKQkhgwZwqOPPmoF7EyOCKQg3j3AA8DFQCWgHDAeaJnNW5OBMun2o4CtmbRZpKopwB8ishZfkvgpoOiNCTPJycmULl2aiIgIxowZQ4UKFay8tclRgQwfPQQ0APYDqOrvQCATwj8BVUSkgrOKW2dgboY2H+ArtoeIFMM3nLQhsNCNCR8nTpxg7NixVK9enQkTJgDQpk0bSwgmxwWSFI46E8WA/6qibC94VtVUfD2M+cBq4B1VXSkig0XkRqfZfGC3iKwCvgEeU9XdZ3oSxoSyNWvW0LRpUx566CGaNGlCu3btvA7JhLBA5hS+F5HHgQIi0hzfMp0fB/LhqjoPmJfhuafTbSvQx3kYYzKYMmUKDzzwAAULFmTGjBl06dLFbkIzrgqkp/A4cABYA/QGvgL6uxmUMcanUqVK3HDDDaxevZquXbtaQjCuC6Sn0Bbf3cgT3A7GmHB39OhRBg8eDMCQIUNo3rw5zZs39zgqE04CSQq3Ai+LyNf4bkD70ll0J6h8tfpPHnrrF1JPnPkVrSecQmL2R5px0/fff09CQgJr166le/fuqKr1DEyuC6TMRRfn3oTrgXuAySLyqar2cj26HJS44yCHjqdxT+MK5Is88/9oF+aPpE7ZIi5EZsLdgQMHePLJJxk3bhzlypVj/vz5tGrVyuuwTJgK6OY1VT0mIh8CR/AVt7sVCKqkcNKjratSML8tMW3yjuTkZKZMmcKDDz7I888/z0UXXeR1SCaMZTvRLCItRWQKsB64C3gNuMztwIwJZbt37/bfbxAdHc2GDRt46aWXLCEYzwVy9VEv4DMgWlXvVNW56e9bMMYETlWZPXs2MTExPPTQQ/4CdrY0pskrsk0KqnqLqs5W1SO5EZAxoWrbtm3cfPPNdOrUiTJlyrBkyRIrYGfynCwH10Xkv6p6tYjs4Z+F7ATffWeXuh6dMSHiZAG7LVu2MGzYMB555BEiI21uy+Q9p/upPHlxdLHcCMSYUJSUlMTll19OREQE48aNo0KFClStWtXrsIzJUpbDR6p6wtmcqqpp6R/A1NwJz5jglJaWxpgxY/5RwK5169aWEEyeF0j/tWb6HacgXl13wjEm+K1evZqEhAQWLlxImzZtuOGGG7wOyZiAZdlTEJEnnPmEmiLyl/PYA+wkQ5E7Y4zP5MmTqV27Nr///jszZ87kk08+oWzZsl6HZUzATnf10TCgODDK+VocKKaql6rqY7kRnDHBpkqVKnTs2JFVq1Zx1113WZkKE3RON3xUWVXXichMIPbkkyd/yFV1ucuxGZPnHTlyhIEDByIivPDCC1bAzgS90yWFvkACMC6T1xRo6kpExgSJb7/9lu7du7Nu3Tp69eplBexMSMgyKahqgvP1qtwLx5i8b//+/fTt25cJEyZQsWJFvvrqK6655hqvwzImRwRS++gmESnkbPcVkXdEpJb7oRmTN23dupXp06fTp08fli9fbgnBhJRAah8NVNUDItIIuAF4G5jkbljG5C27du1i/PjxAFSvXp0//viDESNGcOGFF3ocmTE5K5CkcHJBnXbAeFV9DzjfvZCMyTtUlbfffpuYmBgefvhhfv/9dwBKlizpcWTGuCOQpLBNRMYBnYF5IpI/wPcZE9S2bt1Khw4d6Ny5M+XKlePnn3+2O5JNyAt0Oc62wFhV3SMipfFdmWRMyEpLS6Np06Zs2bKF4cOH07t3bytgZ8JCIMtxHhSRVUAzEWkGfKeqn7oemTEe2LRpE1FRUURERDB+/HgqVqxI5cqVvQ7LmFwTyNVHDwDvAGWdxzsicp/bgRmTm9LS0hg5ciTR0dH+AnatWrWyhGDCTiD94Z5APVU9CCAiQ4AfgPFuBmZMblmxYgUJCQksXryYdu3a0aFDB69DMsYzgUwYC5CSbj/Fec6YoDdx4kTq1KnDhg0bePPNN5k7dy5RUVFeh2WMZwLpKcwEFonIe/iSQQdghqtRGeOykyUpoqOj6dSpE6NHj6Z48eJeh2WM5wKZaB4mIt8AJ8td9FLVn9wNyxh3HD58mKeffpqIiAiGDh3K1VdfzdVXX+11WMbkGYHeb3DMeRxxvhoTdBYsWEDNmjUZMWIEBw8eRFWzf5MxYSaQq4/6A28BpYAo4E0R6ed2YMbklH379vF///d//pLWX3/9NePGjbOKpsZkIpA5hbuAK1X1MICIPA/8DPzHzcCMySnbtm3j9ddf59FHH2XQoEEULFjQ65CMybMCGT7axD+TRySwIZAPF5HrRGStiCSKSJZ3QYvILSKiIhIfyOcak52dO3cyduxYwFfAbuPGjbz44ouWEIzJRiBJ4TCwUkSmiMgrwG/AXhEZKSIjs3qTiETgW6CnDRAD3C4iMZm0KwQ8BPx4NidgTHqqyptvvkl0dDT//ve//QXs7MoiYwITyPDRJ87jpEUBfnY9IFFVNwCIyCygPbAqQ7tn8a0H/WiAn2tMppKSkrj33nv55JNPqF+/PlOnTrUCdsacoUAuSZ16lp99OZCUbj8ZqJ++gYhcAZRR1Y9FJMukICI98d1ZTdmyZc8yHBPKUlNTadasGdu3b2fUqFE8+OCDREREeB2WMUHHzbKPmV3a4b8GUETOA0YB3bL7IFWdDEwGiI+Pt+sIjd/GjRspU6YMkZGRTJo0iYoVK1KxYkWvwzImaLm5LkIyUCbdfhSwNd1+ISAOWCAiG4EGwFybbDaBSE1NZfjw4URHR/tXRGvZsqUlBGPOUcA9BRE5X1XP5Ma1n4AqIlIB2IJvkZ47Tr6oqvuAYuk+fwHwqKouOYNjmDC0fPlyEhISWLJkCe3bt+fmm2/2OiRjQkYgN6/VE5HfgHXOfi0RGZvd+1Q1FXgAmA+sBt5R1ZUiMlhEbjzHuE2YGj9+PFdeeSWbNm3i7bffZs6cOZQuXdrrsIwJGYH0FMbgW5/5AwBV/VVEmgfy4ao6D5iX4bmns2jbLJDPNOHpZAG7uLg4OnfuzKhRoyhWrFj2bzTGnJFAksJ5qropQ0nH2C5RAAAS+UlEQVSANJfiMeYfDh06xIABA4iMjOTFF1+kadOmNG3a1OuwjAlZgUw0J4lIPUBFJEJEHgZ+dzkuY/jqq6+oUaMGo0eP5tixY1bAzphcEEhSuBfog28pzj/xXSV0r5tBmfC2d+9eunfvTsuWLYmMjOTbb79lzJgxVsDOmFwQyM1rO/BdOWRMrvjzzz+ZNWsWTzzxBM888wwXXHCB1yEZEzayTQpOvaNT+u2q2tOViExYOpkIevfuTbVq1di4caNNJBvjgUCGj74EvnIe3wMlsIV2TA5RVV5//XViYmJ4/PHHWbduHYAlBGM8Esjw0dvp90VkJvCFaxGZsLF582Z69erFp59+SsOGDZk6dSpVqlTxOixjwtrZ1D6qAJTL6UBMeDlZwG7Hjh2MGTOG++67zwrYGZMHBDKnsIe/5xTOA/4Cslwwx5jT2bBhA+XKlSMyMpJXXnmFSpUqUb58ea/DMsY4TjunIL5rAGsBxZ1HEVWtqKrv5EZwJnSkpqYydOhQYmJiGDduHAAtWrSwhGBMHnPanoKqqojMUdUrcysgE3qWLVtGQkICS5cupWPHjnTq1MnrkIwxWQjk6qPFIlLH9UhMSHr55ZepW7cuW7ZsYfbs2bz//vuUKlXK67CMMVnIsqcgIpFOpdMmQA8RWQ8cwrd4jqqqJQqTpZMF7GrWrMmdd97JyJEjufTSS70OyxiTjdMNHy0G6gAdcikWEwIOHjxI//79yZcvH8OHD7cCdsYEmdMNHwmAqq7P7JFL8Zkg8vnnnxMXF8fYsWNJSUmxAnbGBKHT9RSKi0ifrF5U1ZEuxGOC0J49e+jTpw/Tp0+nWrVqfPvttzRp0sTrsIwxZ+F0PYUI4CJ8ayln9jAGgB07djB79mz69evHsmXLLCEYE8RO11PYpqqDcy0SE1S2b9/OW2+9xSOPPOIvYFe0aFGvwzLGnKNs5xSMSU9VmTFjBjExMfTr189fwM4SgjGh4XRJoUWuRWGCwsaNG7nuuuvo1q0bMTExLFu2zArYGRNishw+UtW/cjMQk7elpqbSvHlzdu3axbhx4+jVqxfnnRfIvY/GmGByNlVSTRhJTEykQoUKREZGMm3aNCpWrEi5clYk15hQZX/qmUylpKQwZMgQYmNj/QXsmjdvbgnBmBBnPQVziqVLl5KQkMCyZcvo1KkTt912m9chGWNyifUUzD+MGTOGevXqsX37dt5//33eeecdSpYs6XVYxphcYknBAPhLUlxxxRV07dqVVatW0bFjR4+jMsbkNhs+CnMHDhygX79+nH/++YwYMYKrrrqKq666yuuwjDEesZ5CGPvss8+Ii4tj/PjxqKoVsDPGWFIIR7t37+Zf//oXbdq04cILL+T7779n5MiR+FZfNcaEM0sKYWj37t3MmTOHp556il9++YWGDRt6HZIxJo9wNSmIyHUislZEEkWkbyav9xGRVSKyXES+EhG7CN4l27ZtY/jw4agqVatWZdOmTQwePJjzzz/f69CMMXmIa0lBRCKAcUAbIAa4XURiMjT7BYhX1ZrAbGCYW/GEK1Vl2rRpREdH89RTT5GYmAhAkSJFPI7MGJMXudlTqAckquoGVT0OzALap2+gqt+o6mFndxEQ5WI8YeePP/6gVatWJCQkUKtWLX799VcrYGeMOS03L0m9HEhKt58M1D9N+wTg08xeEJGeQE+AsmXL5lR8IS01NZVrrrmG3bt3M2HCBHr27GkF7Iwx2XIzKWR2KUum1zyKyF1APHB1Zq+r6mRgMkB8fLxdN3ka69ato2LFikRGRvLqq69SqVIlypQp43VYxpgg4eafjslA+t9GUcDWjI1EpCXQH7hRVY+5GE9IS0lJ4bnnniMuLo6XX34ZgGbNmllCMMacETd7Cj8BVUSkArAF6Azckb6BiFwBTAKuU9UdLsYS0pYsWUJCQgLLly+nc+fO3H777V6HZIwJUq71FFQ1FXgAmA+sBt5R1ZUiMlhEbnSavQhcBLwrIstEZK5b8YSql156ifr167Nr1y4+/PBD3nrrLUqUKOF1WMaYIOVq7SNVnQfMy/Dc0+m2W7p5/FCmqogI8fHxJCQkMGzYMC655BKvwzLGBDkriBdk9u/fzxNPPEGBAgUYNWoUjRs3pnHjxl6HZYwJEXaNYhCZN28esbGxTJ48mcjISCtgZ4zJcZYUgsCuXbu46667uP7667n44ov54YcfePHFF62AnTEmx1lSCAJ79uzho48+4plnnmHp0qXUr3+6ewCNMebs2ZxCHrVlyxbeeOMNHnvsMapUqcKmTZtsItkY4zrrKeQxqsorr7xCTEwMAwcOZP369QCWEIwxucKSQh6yfv16WrRoQc+ePalTpw7Lly+ncuXKXodljAkjNnyUR6SmptKiRQv++usvJk2aRPfu3a2AnTEm11lS8NjatWupVKkSkZGRzJgxg0qVKhEVZRXEjTHesD9FPXL8+HEGDRpEjRo1GDduHABXX321JQRjjKesp+CBxYsXk5CQwIoVK7jjjju48847vQ7JGGMA6ynkutGjR9OwYUP/vQdvvPEGxYoV8zosY4wBLCnkmpMlKerVq0ePHj1YuXIl7dq18zgqY4z5Jxs+ctm+fft4/PHHueCCCxg9ejSNGjWiUaNGXodljDGZsp6Ciz766CNiYmKYMmUK559/vhWwM8bkeZYUXLBz507uuOMObrzxRooWLcqiRYsYOnSoFbAzxuR5lhRcsG/fPubNm8egQYNYsmQJdevW9TokY4wJiM0p5JCkpCRef/11+vbtS+XKldm0aRMXX3yx12EZY8wZsZ7COTpx4gQTJ04kNjaW5557zl/AzhKCMSYYWVI4B+vWreOaa67h3nvvpV69evz2229WwM4YE9Rs+Ogspaamcu2117J3716mTp3K3XffbRPJxpigZ0nhDK1evZoqVaoQGRnJzJkzqVSpEqVLl/Y6LGOMyRE2fBSgY8eO8cwzz1CzZk1efvllAK666ipLCMaYkGI9hQAsWrSIhIQEVq1aRZcuXejSpYvXIRljjCusp5CNESNG0KhRIw4cOMC8efN47bXXKFq0qNdhGWOMKywpZOHEiRMANGzYkF69erFixQratGnjcVTGGOMuGz7KYO/evfz73/+mYMGCjB071grYGWPCivUU0vnggw+IiYlhxowZFCpUyArYGWPCjiUFYMeOHdx666107NiRkiVLsnjxYoYMGWL3HRhjwo4lBWD//v188cUXPP/88yxevJg6dep4HZIxxngibOcUNm/ezMyZM3nyySepXLkymzdvplChQl6HZYwxnnK1pyAi14nIWhFJFJG+mbx+voi87bz+o4iUdyuWSwrmo2LxC9ETyvjx44mNjWXIkCH+AnaWEIwxxsWkICIRwDigDRAD3C4iMRmaJQB7VLUyMAoY6lY8t9Uty4R2pWjTqgX3338/DRs2ZOXKlVbAzhhj0nGzp1APSFTVDap6HJgFtM/Qpj0ww9meDbQQl2Z3U1NTad26Nb/99huvvvoq8+fPp3z58m4cyhhjgpabcwqXA0np9pOB+lm1UdVUEdkHFAV2pW8kIj2BngBly5Y9q2AiIyN5/fXXqVSpEqVKlTqrzzDGmFDnZk8hs7/4M174H0gbVHWyqsaranzx4sXPOqAmTZpYQjDGmNNwMykkA2XS7UcBW7NqIyKRwMXAXy7GZIwx5jTcTAo/AVVEpIKI5Ac6A3MztJkL/MvZvgX4Wu02YmOM8YxrcwrOHMEDwHwgApimqitFZDCwRFXnAlOBmSKSiK+H0NmteIwxxmTP1ZvXVHUeMC/Dc0+n2z4KdHIzBmOMMYGzMhfGGGP8LCkYY4zxs6RgjDHGz5KCMcYYPwm2K0BFZCew6SzfXowMd0uHATvn8GDnHB7O5ZzLqWq2d/8GXVI4FyKyRFXjvY4jN9k5hwc75/CQG+dsw0fGGGP8LCkYY4zxC7ekMNnrADxg5xwe7JzDg+vnHFZzCsYYY04v3HoKxhhjTsOSgjHGGL+QTAoicp2IrBWRRBHpm8nr54vI287rP4pI+dyPMmcFcM59RGSViCwXka9EpJwXceak7M45XbtbRERFJOgvXwzknEXkVud7vVJE3sztGHNaAD/bZUXkGxH5xfn5butFnDlFRKaJyA4RWZHF6yIiY5x/j+UiUidHA1DVkHrgK9O9HqgI5Ad+BWIytLkPmOhsdwbe9jruXDjn5kBBZ/vecDhnp10h4FtgERDvddy58H2uAvwCFHH2S3gddy6c82TgXmc7BtjoddzneM5NgTrAiixebwt8im/lygbAjzl5/FDsKdQDElV1g6oeB2YB7TO0aQ/McLZnAy1EJLOlQYNFtuesqt+o6mFndxG+lfCCWSDfZ4BngWHA0dwMziWBnHMPYJyq7gFQ1R25HGNOC+ScFSjsbF/MqSs8BhVV/ZbTr0DZHnhNfRYBl4hIjq0zHIpJ4XIgKd1+svNcpm1UNRXYBxTNlejcEcg5p5eA7y+NYJbtOYvIFUAZVf04NwNzUSDf56pAVRH5XkQWich1uRadOwI554HAXSKSjG/9lgdzJzTPnOn/9zPi6iI7HsnsL/6M190G0iaYBHw+InIXEA9c7WpE7jvtOYvIecAooFtuBZQLAvk+R+IbQmqGrzf4nYjEqepel2NzSyDnfDswXVVHiEhDfKs5xqnqCffD84Srv79CsaeQDJRJtx/Fqd1JfxsRicTX5Txddy2vC+ScEZGWQH/gRlU9lkuxuSW7cy4ExAELRGQjvrHXuUE+2Rzoz/aHqpqiqn8Aa/EliWAVyDknAO8AqOpCoAC+wnGhKqD/72crFJPCT0AVEakgIvnxTSTPzdBmLvAvZ/sW4Gt1ZnCCVLbn7AylTMKXEIJ9nBmyOWdV3aeqxVS1vKqWxzePcqOqLvEm3BwRyM/2B/guKkBEiuEbTtqQq1HmrEDOeTPQAkBEovElhZ25GmXumgt0da5CagDsU9VtOfXhITd8pKqpIvIAMB/flQvTVHWliAwGlqjqXGAqvi5mIr4eQmfvIj53AZ7zi8BFwLvOnPpmVb3Rs6DPUYDnHFICPOf5QCsRWQWkAY+p6m7voj43AZ7zv4FXROQRfMMo3YL5jzwReQvf8F8xZ57kGSAfgKpOxDdv0hZIBA4Dd+fo8YP4384YY0wOC8XhI2OMMWfJkoIxxhg/SwrGGGP8LCkYY4zxs6RgjDHGz5KCybNEJE1ElqV7lD9N2/JZVZXMbSISLyJjnO1mItIo3Wu9RKRrLsZSO9irhprcFXL3KZiQckRVa3sdxJlybpA7eZNcM+Ag8IPz2sScPp6IRDo1vDJTG19Zk3k5fVwTmqynYIKK0yP4TkSWOo9GmbSJFZHFTu9iuYhUcZ6/K93zk0QkIpP3bhSRoU67xSJS2Xm+nPjWoTi5HkVZ5/lOIrJCRH4VkW+d55qJyMdOz6YX8IhzzKtEZKCIPCoi0SKyOMN5LXe2rxSR/4rIzyIyP7MKmCIyXURGisg3wFARqSciP4hvTYEfRKSacwfwYOA25/i3iciF4qvX/5PTNrPKsiaceV073B72yOqB747cZc5jjvNcQaCAs10F312tAOVx6s8DY4E7ne38wAVANPARkM95fjzQNZNjbgT6O9tdgY+d7Y+Afznb9wAfONu/AZc725c4X5ule99A4NF0n+/fd86rorP9BDAA352rPwDFnedvw3cXb8Y4pwMfAxHOfmEg0tluCbznbHcDXk73viHAXSfjBX4HLvT6e22PvPOw4SOTl2U2fJQPeFlEauNLGlUzed9CoL+IRAHvq+o6EWkBXAn85JT5uADIqgbUW+m+jnK2GwI3Odsz8a3RAPA9MF1E3gHeP5OTw1fE7VbgBXy//G8DquEr5PeFE2cEkFVdm3dVNc3ZvhiY4fSKFKcsQiZaATeKyKPOfgGgLLD6DGM3IcqSggk2jwB/ArXwDX+esniOqr4pIj8C1wPzRaQ7vnLDM1S1XwDH0Cy2T2mjqr1EpL5zrGVOsgrU2/hqUb3v+yhdJyI1gJWq2jCA9x9Kt/0s8I2qdnSGrRZk8R4BblbVtWcQpwkjNqdggs3FwDb11crvgu8v6X8QkYrABlUdg6+iZE3gK+AWESnhtLlUsl6n+rZ0Xxc62z/wd+HEO4H/OZ9TSVV/VNWngV38s6QxwAF8ZbxPoarr8fV2nsKXIMBX6rq4+NYFQETyiUhsFnGmdzGwxdnudprjzwceFKcbIr7qucb4WVIwwWY88C8RWYRv6OhQJm1uA1aIyDKgOr6lC1fhG7P/3JnQ/QLIagnD852eRm98PROAh4C7nfd2cV4DeFFEfnMuh/0W3xrC6X0EdDw50ZzJsd4G7uLv9QCO4yvnPlREfsU373DKZHomhgH/EZHv+Wei/AaIOTnRjK9HkQ9Y7sT8bACfbcKIVUk1Jh3xLcgTr6q7vI7FGC9YT8EYY4yf9RSMMcb4WU/BGGOMnyUFY4wxfpYUjDHG+FlSMMYY42dJwRhjjN//A/+RXt4G752lAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(1)\n",
    "plt.plot([0, 1], [0, 1], 'k--')\n",
    "plt.plot(fpr, tpr, \n",
    "         label='area = {:.3f}'.format(aucurve))\n",
    "plt.xlabel('False positive rate')\n",
    "plt.ylabel('True positive rate')\n",
    "plt.title('ROC curve')\n",
    "plt.legend(loc='best')\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:py36]",
   "language": "python",
   "name": "conda-env-py36-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}