hands-on/080_imdb_rnn_complete.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# IMDB: recursive neural networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Required imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.datasets import imdb\n",
    "from keras.preprocessing import sequence\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the training and test data.  To limit computation time, we restrict the number of words to 5,000."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_words = 5_000\n",
    "(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=num_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the review vary in length, and we prefer to limit the computation time, we will base the classification on the first 100 features of each input sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_length = 100\n",
    "x_train = sequence.pad_sequences(x_train, maxlen=feature_length)\n",
    "x_test = sequence.pad_sequences(x_test, maxlen=feature_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the training and test input are 2D arrays. We split the training set into a subset for actual training, and one for validation.  First we seed the random number generator to ensure reproducibility. In this case, we will use part of the 25000 test examples as valiation data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(1234)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train, x_val, y_train, y_val = train_test_split(x_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GRU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Required imports & model definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers import Activation, Dense, Dropout\n",
    "from keras.layers.embeddings import Embedding\n",
    "from keras.layers.recurrent import GRU\n",
    "from keras.models import Sequential\n",
    "from keras.optimizers import Adam"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, to limit training times, we restrict ourselfs to using a limited number of features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "vector_length = 64\n",
    "num_units = 64\n",
    "model = Sequential()\n",
    "model.add(Embedding(num_words, vector_length, mask_zero=True,\n",
    "                    input_length=feature_length))\n",
    "model.add(GRU(num_units))\n",
    "model.add(Dropout(rate=0.5))\n",
    "model.add(Dense(1))\n",
    "model.add(Activation('sigmoid'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_2 (Embedding)      (None, 100, 64)           320000    \n",
      "_________________________________________________________________\n",
      "gru_2 (GRU)                  (None, 64)                24768     \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 64)                0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 1)                 65        \n",
      "_________________________________________________________________\n",
      "activation_2 (Activation)    (None, 1)                 0         \n",
      "=================================================================\n",
      "Total params: 344,833\n",
      "Trainable params: 344,833\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss='binary_crossentropy', optimizer=Adam(),\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###    Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/gjb/miniconda3/envs/prace_ml_cpu/lib/python3.7/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "Train on 18750 samples, validate on 6250 samples\n",
      "Epoch 1/10\n",
      "18750/18750 [==============================] - 58s 3ms/step - loss: 0.4824 - acc: 0.7543 - val_loss: 0.3566 - val_acc: 0.8427\n",
      "Epoch 2/10\n",
      "18750/18750 [==============================] - 61s 3ms/step - loss: 0.3146 - acc: 0.8704 - val_loss: 0.3526 - val_acc: 0.8432\n",
      "Epoch 3/10\n",
      "18750/18750 [==============================] - 52s 3ms/step - loss: 0.2646 - acc: 0.8971 - val_loss: 0.3469 - val_acc: 0.8451\n",
      "Epoch 4/10\n",
      "18750/18750 [==============================] - 68s 4ms/step - loss: 0.2210 - acc: 0.9173 - val_loss: 0.3670 - val_acc: 0.8386\n",
      "Epoch 5/10\n",
      "18750/18750 [==============================] - 56s 3ms/step - loss: 0.1790 - acc: 0.9338 - val_loss: 0.4016 - val_acc: 0.8344\n",
      "Epoch 6/10\n",
      "18750/18750 [==============================] - 75s 4ms/step - loss: 0.1386 - acc: 0.9510 - val_loss: 0.4807 - val_acc: 0.8282\n",
      "Epoch 7/10\n",
      "18750/18750 [==============================] - 59s 3ms/step - loss: 0.1082 - acc: 0.9614 - val_loss: 0.5543 - val_acc: 0.8298\n",
      "Epoch 8/10\n",
      "18750/18750 [==============================] - 57s 3ms/step - loss: 0.0896 - acc: 0.9698 - val_loss: 0.6586 - val_acc: 0.8267\n",
      "Epoch 9/10\n",
      "18750/18750 [==============================] - 78s 4ms/step - loss: 0.0719 - acc: 0.9759 - val_loss: 0.7059 - val_acc: 0.8294\n",
      "Epoch 10/10\n",
      "18750/18750 [==============================] - 58s 3ms/step - loss: 0.0755 - acc: 0.9733 - val_loss: 0.6660 - val_acc: 0.8250\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(x_train, y_train, batch_size=64, epochs=10,\n",
    "                    validation_data=(x_val, y_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training accuracy is much better than the validation accurcy, so the model is likely heavily overtrained."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25000/25000 [==============================] - 38s 2ms/step\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.6901801983165741, 0.82452]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LSTM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Required imports & model definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.layers.recurrent import LSTM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, to limit training times, we restrict ourselfs to using a limited number of features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "vector_length = 64\n",
    "num_units = 64\n",
    "model = Sequential()\n",
    "model.add(Embedding(num_words, vector_length, mask_zero=True,\n",
    "                    input_length=feature_length))\n",
    "model.add(LSTM(num_units))\n",
    "model.add(Dropout(rate=0.5))\n",
    "model.add(Dense(1))\n",
    "model.add(Activation('sigmoid'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_3 (Embedding)      (None, 100, 64)           320000    \n",
      "_________________________________________________________________\n",
      "lstm_1 (LSTM)                (None, 64)                33024     \n",
      "_________________________________________________________________\n",
      "dropout_3 (Dropout)          (None, 64)                0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 1)                 65        \n",
      "_________________________________________________________________\n",
      "activation_3 (Activation)    (None, 1)                 0         \n",
      "=================================================================\n",
      "Total params: 353,089\n",
      "Trainable params: 353,089\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss='binary_crossentropy', optimizer=Adam(),\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###    Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 18750 samples, validate on 6250 samples\n",
      "Epoch 1/10\n",
      "18750/18750 [==============================] - 75s 4ms/step - loss: 0.4644 - acc: 0.7726 - val_loss: 0.3517 - val_acc: 0.8480\n",
      "Epoch 2/10\n",
      "18750/18750 [==============================] - 77s 4ms/step - loss: 0.3029 - acc: 0.8783 - val_loss: 0.3404 - val_acc: 0.8510\n",
      "Epoch 3/10\n",
      "18750/18750 [==============================] - 80s 4ms/step - loss: 0.2565 - acc: 0.8969 - val_loss: 0.3488 - val_acc: 0.8434\n",
      "Epoch 4/10\n",
      "18750/18750 [==============================] - 79s 4ms/step - loss: 0.2082 - acc: 0.9203 - val_loss: 0.3682 - val_acc: 0.8459\n",
      "Epoch 5/10\n",
      "18750/18750 [==============================] - 78s 4ms/step - loss: 0.1712 - acc: 0.9357 - val_loss: 0.4161 - val_acc: 0.8368\n",
      "Epoch 6/10\n",
      "18750/18750 [==============================] - 83s 4ms/step - loss: 0.1411 - acc: 0.9502 - val_loss: 0.5028 - val_acc: 0.8318\n",
      "Epoch 7/10\n",
      "18750/18750 [==============================] - 81s 4ms/step - loss: 0.1191 - acc: 0.9562 - val_loss: 0.5534 - val_acc: 0.8248\n",
      "Epoch 8/10\n",
      "18750/18750 [==============================] - 81s 4ms/step - loss: 0.1156 - acc: 0.9581 - val_loss: 0.6960 - val_acc: 0.8243\n",
      "Epoch 9/10\n",
      "18750/18750 [==============================] - 78s 4ms/step - loss: 0.0964 - acc: 0.9659 - val_loss: 0.6730 - val_acc: 0.8179\n",
      "Epoch 10/10\n",
      "18750/18750 [==============================] - 82s 4ms/step - loss: 0.0770 - acc: 0.9732 - val_loss: 0.6445 - val_acc: 0.8118\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(x_train, y_train, batch_size=64, epochs=10,\n",
    "                    validation_data=(x_val, y_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training accuracy is much better than the validation accurcy, so the model is likely heavily overtrained."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25000/25000 [==============================] - 55s 2ms/step\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.6544363550662995, 0.81168]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(x_test, y_test)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}