appendix4_NN1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **附录4：关于神经网络（Neural Networks, NN）1**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Madmom** 节拍分析算法使用的是 **深度学习（Deep Learning）** 领域内的**循环神经网络（Recurrent Neural Network, RNN）**技术，更具体地说是 **双向长短时记忆（Bidirectional Long Short-Term Memory）循环网络**。为了更好地理解 RNN，首先需要理解基本的 **神经网络（Neural Networks，NN）**，附录 4 对应的内容基本全都是依据阅读材料 2 所做的梳理。\n",
    "\n",
    "&emsp;\n",
    "\n",
    "## **阅读材料推荐**\n",
    "\n",
    "神经网络到底指什么，下面这几篇教程或文档讲得必定比我好，所以先把他们放上来，供阅读参考：\n",
    "\n",
    "&emsp;\n",
    "\n",
    "1. **伯克利机器学习速成课程系列（Machine Learning Crash Course）**\n",
    "\n",
    "    这是 Machine Learning @ Berkeley 发表在 Medium 上的一系列文章，共 5 篇，说是速成课程，更准确地描述可能是通识课程，以 **不使用数学公式** 的方式、生动活泼地讲解机器学习相关的一些概念，具体到神经网络主题，前 3 篇相关度最高。非常适合作为最初的入门阅读材料。\n",
    "\n",
    "    + [系列 1：回归（Regression）/分类（Classification），损失函数（Cost Function）和梯度下降（Gradient Descent）](https://medium.com/@ml.at.berkeley/machine-learning-crash-course-part-1-9377322b3042?source=---------5----------------------------)\n",
    "\n",
    "    + [系列2：支持向量机（Support Vector Machines, SVM）, 感知机（Perceptron）和逻辑蒂斯回归（Logistic Regression）](https://medium.com/@ml.at.berkeley/machine-learning-crash-course-part-2-3046b4a7f943)\n",
    "\n",
    "    + [系列3：神经网络（Neural Networks）](https://medium.com/@ml.at.berkeley/machine-learning-crash-course-part-3-4a5bfd72294a)\n",
    "\n",
    "&emsp;\n",
    "\n",
    "2. **[神经网络与深度学习（Neural Networks and Deep Learning)](http://neuralnetworksanddeeplearning.com/index.html)**\n",
    "\n",
    "    **五星强烈推荐！**\n",
    "    \n",
    "    说是书，其实是由 6 篇长文构成，所以先不要被体量吓到；其次不要被文章内的数学公式和篇幅吓到，都是有充分原因的。（我最初就被吓到了，多人推荐阅读但一直拖延没有看。）**一切都是为了把神经网络是什么讲得清清楚楚**（后面深度学习的部分还没看）。我自己的体会是看懂了这个系列的文章，对神经网络就有了初步的认识与理解。\n",
    "\n",
    "    中文版在[这里](https://github.com/Xovee/simplified-chinese-translation-of-neural-networks-and-deep-learning)。\n",
    "    \n",
    "    这个笔记本的内容主要基于第 1、2 章：\n",
    "    \n",
    "    + [第 1 章：使用神经网络识别手写数字](http://neuralnetworksanddeeplearning.com/chap1.html)\n",
    "    \n",
    "    + [第 2 章：反向传播算法是如何工作的](http://neuralnetworksanddeeplearning.com/chap2.html)\n",
    "\n",
    "&emsp;    \n",
    "\n",
    "3. **[计算图上的微积分：反向传播（Calculus on Computational Graphs: Backpropagation）](http://colah.github.io/posts/2015-08-Backprop/)**\n",
    "\n",
    "    **五星推荐！**\n",
    "    \n",
    "    这位作者同样也非常善于清晰友好地讲解复杂概念，这篇文章旨在帮助读者从直觉上理解反向传播这一核心概念，对数学依赖很少。\n",
    "\n",
    "\n",
    "&emsp;\n",
    "\n",
    "4. **[从零开始使用 Python 构建一个神经网络(Implementing a Neural Network from Scratch in Python)](http://www.wildml.com/2015/09/implementing-a-neural-network-from-scratch/)**\n",
    "\n",
    "    这也是一个系列的指南，从基本的神经网络构建开始，后面还有循环神经网络的示例。在阅读 2 之前，先跟着这篇教程的代码操作了一遍，但比较之下，还是 2 讲解得更清楚细致，这个也还是放在这里供参考。\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;\n",
    "\n",
    "## **关于神经网络**\n",
    "\n",
    "![NeuralNetworks1](img/neuralnetworks1.png)\n",
    "\n",
    "![NeuralNetworks3](img/neuralnetworks3.png)\n",
    "\n",
    "![NeuralNetworks2](img/neuralnetworks2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;\n",
    "\n",
    "## **神经网络构建示例：使用神经网络识别手写数字**\n",
    "\n",
    "以下代码是 **[神经网络与深度学习（Neural Networks and Deep Learning)](http://neuralnetworksanddeeplearning.com/index.html)** 教材中第一、二章配套的示例 [Python 代码](https://github.com/mnielsen/neural-networks-and-deep-learning)，因原代码是 python 2.0 所以改动了几处使符合 python 3.0 语法规范。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**以便利的数据格式和结构调用训练和测验数据**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "mnist_loader\n",
    "~~~~~~~~~~~~\n",
    "这部分代码的功能是以适当的数据结构和格式加载 MNIST 图像数据。这些图片全是手写数字（0-9），要训练的神经网络模型\n",
    "就是要根据图片判断具体是哪个数字。其中 50000 张图片用作神经网络训练数据集（training set），10000 张用作训练时\n",
    "的验证数据集（validation set），另还有 10000 张图片用作测试数据集（testing set）。\n",
    "\n",
    "数据文件 mnist.pkl.gz 存放在仓库内 data 文件夹下。\n",
    "\"\"\"\n",
    "\n",
    "import pickle\n",
    "import gzip\n",
    "import numpy as np\n",
    "\n",
    "def load_data():\n",
    "    \"\"\"\n",
    "    以元组的形式返回训练数据、验证数据和测试数据\n",
    "    \n",
    "    training_data 返回的是由两个元素构成的元组：第一个元素存放的是 50000 张图片数据，即一个由\n",
    "    50000 个元素构成的数组（numpy ndarry），每一元素又是一个包含 28*28=784 个像素值的数组；元组中的第二个元素也是\n",
    "    一个由 50000 个元素构成的数组，存放的是每张图片（元组的第一个元素）对应的数字（0-9）\n",
    "    \n",
    "    validation_data 以及 test_data 的结构是相同的，只是数量分别都是 10000 个\n",
    "    \n",
    "    为了方便神经网络使用，后面 load_data_wrapper 函数又进一步对数据格式做了一点调整\n",
    "\n",
    "    \"\"\"\n",
    "    f = gzip.open('data/mnist.pkl.gz', 'rb')\n",
    "    training_data, validation_data, test_data = pickle.load(f, encoding='latin1')\n",
    "    f.close()\n",
    "    return training_data, validation_data, test_data\n",
    "\n",
    "\n",
    "def load_data_wrapper():\n",
    "    \"\"\"\n",
    "    基于 load_data 函数，返回一个元组，由 (training_data、validation_data、test_data) 三个元素构成，\n",
    "    更方便后面构建神经网络时使用。\n",
    " \n",
    "    具体地，training_data 是一个列表，包含 50000 个二元元组(x,y)。x 是一个 784 维的数组（numpy.ndarray），存放图片\n",
    "    数据；y 是一个 10 维的数组，存放表示 x 对应数字的单位向量。\n",
    "    \n",
    "    validation_data 以及 test_data 是包含 10000 个二元元组(x,y) 的列表。x 都是一个 784 维的数组，存放图片数据；y \n",
    "    对应的则是分类，即 x 对应的数字的值（整数）。\n",
    "    \n",
    "    所以，训练数据与验证/测试数据的格式略有不同，都是为了方便构建的神经网络使用。\n",
    "    \"\"\"\n",
    "    tr_d, va_d, te_d = load_data()\n",
    "    training_inputs = [np.reshape(x, (784, 1)) for x in tr_d[0]]\n",
    "    training_results = [vectorized_result(y) for y in tr_d[1]]\n",
    "    training_data = zip(training_inputs, training_results)\n",
    "    validation_inputs = [np.reshape(x, (784, 1)) for x in va_d[0]]\n",
    "    validation_data = zip(validation_inputs, va_d[1])\n",
    "    test_inputs = [np.reshape(x, (784, 1)) for x in te_d[0]]\n",
    "    test_data = zip(test_inputs, te_d[1])\n",
    "    # 下面这三行是从 python2.0 至 python 3.0 做的改动\n",
    "    test_data = list(test_data)\n",
    "    training_data = list(training_data)\n",
    "    validation_data = list(validation_data)\n",
    "    return training_data, validation_data, test_data\n",
    "\n",
    "def vectorized_result(j):\n",
    "    \"\"\"\n",
    "    返回一个 10 维的单位向量，j 位放 1.0，其余位置为 0。这是为了将某一个数字（0-9）转换成后面神经网络所需的输出数据。\n",
    "    即用[1.0, 0, 0, 0, 0, 0, 0, 0, 0, 0] 表示数字 0，用 [0, 1.0, 0, 0, 0, 0, 0, 0, 0, 0]表示数字 1，依次类推。\n",
    "    \"\"\"\n",
    "    e = np.zeros((10, 1))\n",
    "    e[j] = 1.0\n",
    "    return e"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "training_data, validation_data, test_data = load_data_wrapper()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;\n",
    "\n",
    "**构建神经网络**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import random\n",
    "\n",
    "\n",
    "class Network(object):\n",
    "    \n",
    "    def __init__(self, sizes):\n",
    "        self.num_layers = len(sizes)  #sizes 是一个列表，每一个元素对应神经网络的一层，交待的是每一层分别有多少个神经元\n",
    "        self.sizes = sizes\n",
    "        # w 和 b 的初始值是从均值为 0，标准差为 1 的正态分布中随机抽取的\n",
    "        self.biases = [np.random.randn(y, 1) for y in sizes[1:]]\n",
    "        self.weights = [np.random.randn(y, x) for x, y in zip(sizes[:-1], sizes[1:])]\n",
    "      \n",
    "                        \n",
    "    def feedforward(self, a):\n",
    "        \"\"\"\n",
    "        数据在神经网络内正向传播的数学表达式\n",
    "        即数据 a 经加权求和即 wa 再加上偏差 b，作为某一层的输入数据，经激活函数 sigmoid，输出值又成为下一层的输入数据 a\n",
    "        \"\"\"\n",
    "        for b, w in zip(self.biases, self.weights):\n",
    "            a = sigmoid(np.dot(w, a)+b)\n",
    "        return a\n",
    "    \n",
    "   \n",
    "    def SGD(self, training_data, epochs, mini_batch_size, eta, test_data=None):\n",
    "        \"\"\"\n",
    "        随机梯度下降（Stochastic Gradient Descent)定义\n",
    "        eta 是学习速率（learning rate）\n",
    "        epochs 可以理解为以完整训练数据进行训练的次数，mini_batch_size 则是每一次完整训练分割为多大的单位进行\n",
    "        \"\"\"\n",
    "        if test_data: n_test = len(test_data)    # 每一个 epoch 结束后，是否在测验数据中做测验\n",
    "        n = len(training_data)\n",
    "        for j in range(epochs):\n",
    "            random.shuffle(training_data)  # 每一个 epoch 初始时都先将全部数据随机化\n",
    "            mini_batches = [               # 按 mini_batch_size 将全部数据分成 mini_batches\n",
    "                training_data[k:k+mini_batch_size]\n",
    "                for k in range(0, n, mini_batch_size)\n",
    "            ]\n",
    "            for mini_batch in mini_batches:   # 在每一个 mini_batch 内通过反向传播更新所有 w 和 b\n",
    "                self.update_mini_batch(mini_batch, eta)\n",
    "            if test_data:\n",
    "                print(\"Epoch {0}: {1} / {2}\".format(j, self.evaluate(test_data), n_test))\n",
    "            else:\n",
    "                print(\"Epoch {0} complete\".format(j))\n",
    "    \n",
    "                        \n",
    "    def update_mini_batch(self, mini_batch, eta):\n",
    "        \"\"\"\n",
    "        通过反向传播算法实现梯度下降，更新 b 和 w 值\n",
    "        \"\"\"\n",
    "        nabla_b = [np.zeros(b.shape) for b in self.biases]\n",
    "        nabla_w = [np.zeros(w.shape) for w in self.weights]\n",
    "        for x, y in mini_batch:\n",
    "            delta_nabla_b, delta_nabla_w = self.backprop(x, y)\n",
    "            nabla_b = [nb+dnb for nb, dnb in zip(nabla_b, delta_nabla_b)]\n",
    "            nabla_w = [nw+dnw for nw, dnw in zip(nabla_w, delta_nabla_w)]\n",
    "        self.weights = [w-(eta/len(mini_batch))*nw for w, nw in zip(self.weights, nabla_w)]\n",
    "        self.biases = [b-(eta/len(mini_batch))*nb for b, nb in zip(self.biases, nabla_b)]\n",
    "    \n",
    "                        \n",
    "    def backprop(self, x, y):\n",
    "        \"\"\"\n",
    "        反向传播算法\n",
    "        \n",
    "        先正向传播（feedforward），输入数据从输入层到隐藏层，再从隐藏层到输出层，得到输出数据\n",
    "        activations[-1] 即为输出层输出的结果，会拿来与 training_data 中的 y 算损失函数\n",
    "        \n",
    "        再反向传播（backpropagation），首先根据损失函数与输出层激活函数求得输出层神经元误差，即 delta，进一步求得\n",
    "        输出层的 nabla_b 和 nabla_w；再根据另一个公式继续反向推进，根据输出层（即本层的下一层）的 w、误差以及隐藏层的\n",
    "        激活函数求得本曾的 nabla_b 和 nabla_w，以此继续反向推进（本例只含有一层隐藏层，因此到此为止）\n",
    "        \"\"\"\n",
    "        nabla_b = [np.zeros(b.shape) for b in self.biases]\n",
    "        nabla_w = [np.zeros(w.shape) for w in self.weights]\n",
    "        activation = x\n",
    "        activations = [x]\n",
    "        zs = []\n",
    "        # 正向传播\n",
    "        for b, w in zip(self.biases, self.weights):\n",
    "            z = np.dot(w, activation)+b\n",
    "            zs.append(z)\n",
    "            activation = sigmoid(z)\n",
    "            activations.append(activation)\n",
    "        # 反向传播\n",
    "        # 下面 delta 计算的是输出层神经元的误差，即损失函数的导数（a-y）乘以输出层激活函数的导数\n",
    "        delta = self.cost_derivative(activations[-1], y)*sigmoid_prime(zs[-1])   \n",
    "        # nabla_b 和 delta 相等，直接使用其值\n",
    "        # nabla_w 等于 delta 乘以上一层神经元的输出值\n",
    "        nabla_b[-1] = delta\n",
    "        nabla_w[-1] = np.dot(delta, activations[-2].transpose())\n",
    "        # 继续反向向隐藏层推进根据神经元误差计算 nabla_b 和 nabla_w\n",
    "        for l in range(2, self.num_layers):\n",
    "            z = zs[-l]\n",
    "            sp = sigmoid_prime(z)\n",
    "            # 公式改变，下一层的 w 乘以下一层的误差再与本层的激活函数的导数相乘\n",
    "            delta = np.dot(self.weights[-l+1].transpose(), delta)*sp\n",
    "            nabla_b[-l] = delta\n",
    "            nabla_w[-l] = np.dot(delta, activations[-l-1].transpose())\n",
    "        return nabla_b, nabla_w\n",
    "                        \n",
    "    \n",
    "    def evaluate(self, test_data):\n",
    "        \"\"\"\n",
    "        在测验数据中测试神经网络的效果\n",
    "        \"\"\"\n",
    "        test_results = [(np.argmax(self.feedforward(x)), y) for (x, y) in test_data]\n",
    "        return sum(int(x==y) for (x, y) in test_results)\n",
    "    \n",
    "                        \n",
    "    def cost_derivative(self, output_activations, y):\n",
    "        \"\"\"\n",
    "        输入数据经正向传播从输出层输出的结果与实际 y 之间的偏差\n",
    "        \"\"\"\n",
    "        return (output_activations-y)\n",
    "                        \n",
    "\n",
    "        \n",
    "def sigmoid(z):\n",
    "    \"\"\"\n",
    "    激活函数 sigmoid 的表达式，作用是平滑 0/1 二分分类/判断\n",
    "    \"\"\"\n",
    "    return 1.0/(1.0+np.exp(-z))\n",
    "                        \n",
    "def sigmoid_prime(z):\n",
    "    \"\"\"\n",
    "    sigmoid 的导数\n",
    "    \"\"\"\n",
    "    return sigmoid(z)*(1-sigmoid(z))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 输入层 784 个神经元，中间层 30 个神经元，输出层 10 个神经元\n",
    "\n",
    "net = Network([784, 30, 10])   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 对全体训练数据做 SGD，重复 30 次（epochs），以 10 个数据为 1 个 mini_batch，学习速率是 3.0\n",
    "net.SGD(training_data, 30, 10, 3.0, test_data=test_data)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}