6_tensorflow_优化器操作.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-07-12T17:03:03.442298Z",
     "start_time": "2019-07-12T17:03:02.505759Z"
    }
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-07-12T17:04:23.284074Z",
     "start_time": "2019-07-12T17:04:23.215719Z"
    }
   },
   "outputs": [],
   "source": [
    "# 梯度下降\n",
    "tf.train.GradientDescentOptimizer(0.01).minimize(cross_entropy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-07-12T17:05:51.402177Z",
     "start_time": "2019-07-12T17:05:51.396641Z"
    }
   },
   "outputs": [],
   "source": [
    "# SGD 随机梯度下降"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### MomentumOpitimizer\n",
    "Momentum动量算法引入了变量v充当速度角色，它代表参数在参数空间移动的方向和速率。可以使SGD不至于陷入局部鞍点震荡，同时起到一定加速作用\n",
    "Momentum最开始有可能会偏离目标较远，但是通常会慢慢矫正回来\n",
    "v = mu*v - learning_rate*dx\n",
    "x+=v\n",
    "tf.train.MomentumOpitimizer(learning_rate,momentum)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### AdaGrad\n",
    "使用每个变量的历史梯度值累加作为更新的分母，起到平衡不同变量梯度数值差异过大的问题\n",
    "* cache += dx**2 \n",
    "* x += -learning_rate*dx/(np.sqrt(cache) + 1e-1)\n",
    "* tf.train.AdagradOptimizer(learning_rate,initial_accumulator_value=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### RMSProp\n",
    "* 在AdaGrad基础上加入了decay fator(衰减因子),防止历史梯度求和过大\n",
    "* cache = decay_rate*cache + (1-decay_rate)*dx**2\n",
    "* x += -learning_rate*dx/(np.sqrt(cache)+1e-7)\n",
    "* tf.train.RMSPropOptimizer(learning_rate,decay=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### AdaDelta\n",
    "* 参考：https://zh.d2l.ai/chapter_optimization/adadelta.html\n",
    "* AdaDelta算法没有学习率这一超参数\n",
    "* AdaDelta算法也像RMSProp算法一样，使用了小批量随机梯度g(t)，按元素平方的指数加权移动平均变量s(t), 给定超参数p,\n",
    "* ps(t-1) + (1-p)g(t)*g(t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Adam算法\n",
    "* 参考：https://zh.d2l.ai/chapter_optimization/adam.html\n",
    "* Adam算法在RMSProp算法基础上对小批量随机梯度也做了指数加权移动平均。\n",
    "* Adam算法使用了动量变量v和RMSProp算法中小批量随机梯度按元素平方的指数加权移动平均变量s(t)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}