General_Q-learning_agents.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "504efc34",
   "metadata": {},
   "source": [
    "# REINFORCEMENT LEARNING\n",
    "\n",
    "### Author: Hyungjoo Kim\n",
    "\n",
    "##### All data was provided by University College London, Department of Computer Science, Reinforcement Learning Module\n",
    "\n",
    "##### Supervisor: Prof. Hado Van Hasselt, Matteo Hessel, and Diana Borsa"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0f580f8",
   "metadata": {},
   "source": [
    "**Context**\n",
    "\n",
    "In this assignment, we will take a first look at learning algorithms for sequential decision problems.\n",
    "\n",
    "**Background reading**\n",
    "\n",
    "* Sutton and Barto (2018), Chapters 3 - 6\n",
    "\n",
    "**Overview of this assignment**\n",
    "\n",
    "You will use Python to implement several reinforcement learning algorithms.\n",
    "\n",
    "You will then run these algorithms on a few problems, to understand their properties."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6963c4da",
   "metadata": {},
   "source": [
    "# Setup\n",
    "\n",
    "### Import Useful Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f77038c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.collections as mcoll\n",
    "import matplotlib.path as mpa\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from collections import namedtuple"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f021414",
   "metadata": {},
   "source": [
    "### Set options"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "3fc5f452",
   "metadata": {},
   "outputs": [],
   "source": [
    "np.set_printoptions(precision=3, suppress=1)\n",
    "plt.style.use('seaborn-colorblind')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d67bb261",
   "metadata": {},
   "source": [
    "### Some grid worlds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ace7445f",
   "metadata": {},
   "outputs": [],
   "source": [
    "W = -100  # wall\n",
    "G = 100  # goal\n",
    "\n",
    "GRID_LAYOUT = np.array([\n",
    "  [W, W, W, W, W, W, W, W, W, W, W, W],\n",
    "  [W, W, 0, W, W, W, W, W, W, 0, W, W],\n",
    "  [W, 0, 0, 0, 0, 0, 0, 0, 0, G, 0, W],\n",
    "  [W, 0, 0, 0, W, W, W, W, 0, 0, 0, W],\n",
    "  [W, 0, 0, 0, W, W, W, W, 0, 0, 0, W],\n",
    "  [W, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, W],\n",
    "  [W, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, W],\n",
    "  [W, W, 0, 0, 0, 0, 0, 0, 0, 0, W, W],\n",
    "  [W, W, W, W, W, W, W, W, W, W, W, W]\n",
    "])\n",
    "\n",
    "class Grid(object):\n",
    "\n",
    "  def __init__(self, noisy=False):\n",
    "    # -1: wall\n",
    "    # 0: empty, episode continues\n",
    "    # other: number indicates reward, episode will terminate\n",
    "    self._layout = GRID_LAYOUT\n",
    "    self._start_state = (2, 2)\n",
    "    self._state = self._start_state\n",
    "    self._number_of_states = np.prod(np.shape(self._layout))\n",
    "    self._noisy = noisy\n",
    "\n",
    "  @property\n",
    "  def number_of_states(self):\n",
    "      return self._number_of_states\n",
    "\n",
    "  def get_obs(self):\n",
    "    y, x = self._state\n",
    "    return y*self._layout.shape[1] + x\n",
    "\n",
    "  def obs_to_state(self, obs):\n",
    "    x = obs % self._layout.shape[1]\n",
    "    y = obs // self._layout.shape[1]\n",
    "    s = np.copy(grid._layout)\n",
    "    s[y, x] = 4\n",
    "    return s\n",
    "\n",
    "  def step(self, action):\n",
    "    y, x = self._state\n",
    "    \n",
    "    if action == 0:  # up\n",
    "      new_state = (y - 1, x)\n",
    "    elif action == 1:  # right\n",
    "      new_state = (y, x + 1)\n",
    "    elif action == 2:  # down\n",
    "      new_state = (y + 1, x)\n",
    "    elif action == 3:  # left\n",
    "      new_state = (y, x - 1)\n",
    "    else:\n",
    "      raise ValueError(\"Invalid action: {} is not 0, 1, 2, or 3.\".format(action))\n",
    "\n",
    "    new_y, new_x = new_state\n",
    "    reward = self._layout[new_y, new_x]\n",
    "    if self._layout[new_y, new_x] == W:  # wall\n",
    "      discount = 0.9\n",
    "      new_state = (y, x)\n",
    "    elif self._layout[new_y, new_x] == 0:  # empty cell\n",
    "      reward = -1.\n",
    "      discount = 0.9\n",
    "    else:  # a goal\n",
    "      discount = 0.\n",
    "      new_state = self._start_state\n",
    "\n",
    "    if self._noisy:\n",
    "      width = self._layout.shape[1]\n",
    "      reward += 10*np.random.normal(0, width - new_x + new_y)\n",
    "\n",
    "    self._state = new_state\n",
    "    return reward, discount, self.get_obs()\n",
    "\n",
    "  def plot_grid(self):\n",
    "    plt.figure(figsize=(4, 4))\n",
    "    plt.imshow(self._layout != W, interpolation=\"nearest\", cmap='pink')\n",
    "    plt.gca().grid(0)\n",
    "    plt.xticks([])\n",
    "    plt.yticks([])\n",
    "    plt.title(\"The grid\")\n",
    "    plt.text(2, 2, r\"$\\mathbf{S}$\", ha='center', va='center')\n",
    "    plt.text(9, 2, r\"$\\mathbf{G}$\", ha='center', va='center')\n",
    "    h, w = self._layout.shape\n",
    "    for y in range(h-1):\n",
    "      plt.plot([-0.5, w-0.5], [y+0.5, y+0.5], '-k', lw=2)\n",
    "    for x in range(w-1):\n",
    "      plt.plot([x+0.5, x+0.5], [-0.5, h-0.5], '-k', lw=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e8484fc3",
   "metadata": {},
   "outputs": [],
   "source": [
    "SMALL_GRID_LAYOUT = np.array([\n",
    "  [W, W, W, W, W],\n",
    "  [W, W, 3, W, W],\n",
    "  [W, W, 0, W, W],\n",
    "  [W, 1, 0, W, W],\n",
    "  [W, W, 0, W, W],\n",
    "  [W, W, 0, 4, W],\n",
    "  [W, W, W, W, W]\n",
    "])\n",
    "\n",
    "\n",
    "def plot_small_grid(version=1):\n",
    "  plt.imshow(SMALL_GRID_LAYOUT < -1, interpolation='nearest', cmap='pink_r',\n",
    "             vmin=-0.2, vmax=1.2)\n",
    "  if version == 1:\n",
    "    plt.text(2, 1, '$+3$', ha='center', va='center', fontsize=12)\n",
    "    plt.text(1, 3, '$+1$', ha='center', va='center', fontsize=12)\n",
    "    plt.text(3, 5, '$+4$', ha='center', va='center', fontsize=12)\n",
    "  else:\n",
    "    plt.text(2, 1, '$+2$', ha='center', va='center', fontsize=12)\n",
    "    plt.text(1, 3, '$+1$', ha='center', va='center', fontsize=12)\n",
    "    plt.text(3, 5, '$+5$', ha='center', va='center', fontsize=12)\n",
    "  plt.text(2, 3, '$S$', ha='center', va='center', fontsize=12)\n",
    "  h, w = SMALL_GRID_LAYOUT.shape\n",
    "  for r in np.arange(0.5, h):\n",
    "    plt.plot([-0.5, w - 0.5], [r, r], '-k', lw=3, alpha=0.4)\n",
    "  for c in np.arange(0.5, w):\n",
    "    plt.plot([c, c], [-0.5, h - 0.5], '-k', lw=3, alpha=0.4)\n",
    "  plt.xticks([]); plt.yticks([]);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62ee63b6",
   "metadata": {},
   "source": [
    "### Helper functions\n",
    "(You should not have to change, or even look at, these.  Do run the cell to make sure the functions are loaded though.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ed4b6cb8",
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_experiment(env, agent, number_of_steps):\n",
    "    mean_reward = 0.\n",
    "    try:\n",
    "      action = agent.initial_action()\n",
    "    except AttributeError:\n",
    "      action = 0\n",
    "    for i in range(number_of_steps):\n",
    "      reward, discount, next_state = grid.step(action)\n",
    "      action = agent.step(reward, discount, next_state)\n",
    "      mean_reward += reward\n",
    "    return mean_reward/float(number_of_steps)\n",
    "\n",
    "map_from_action_to_subplot = lambda a: (2, 6, 8, 4)[a]\n",
    "map_from_action_to_name = lambda a: (\"up\", \"right\", \"down\", \"left\")[a]\n",
    "\n",
    "def plot_values(grid, values, colormap='pink', vmin=0, vmax=10):\n",
    "  plt.imshow(values - 1000*(grid<0), interpolation=\"nearest\", cmap=colormap, vmin=vmin, vmax=vmax)\n",
    "  plt.yticks([])\n",
    "  plt.xticks([])\n",
    "  plt.colorbar(ticks=[vmin, vmax])\n",
    "\n",
    "def plot_action_values(grid, action_values, vmin=-5, vmax=5):\n",
    "  q = action_values\n",
    "  fig = plt.figure(figsize=(10, 10))\n",
    "  fig.subplots_adjust(wspace=0.3, hspace=0.3)\n",
    "  for a in [0, 1, 2, 3]:\n",
    "    plt.subplot(4, 3, map_from_action_to_subplot(a))\n",
    "    plot_values(grid, q[..., a], vmin=vmin, vmax=vmax)\n",
    "    action_name = map_from_action_to_name(a)\n",
    "    plt.title(r\"$q(s, \\mathrm{\" + action_name + r\"})$\")\n",
    "    \n",
    "  plt.subplot(4, 3, 5)\n",
    "  v = np.max(q, axis=-1)\n",
    "  plot_values(grid, v, colormap='summer', vmin=vmin, vmax=vmax)\n",
    "  plt.title(\"$v(s)$\")\n",
    "  \n",
    "  # Plot arrows:\n",
    "  plt.subplot(4, 3, 11)\n",
    "  plot_values(grid, grid==0, vmax=1)\n",
    "  for row in range(len(grid)):\n",
    "    for col in range(len(grid[0])):\n",
    "      if grid[row][col] == 0:\n",
    "        argmax_a = np.argmax(q[row, col])\n",
    "        if argmax_a == 0:\n",
    "          x = col\n",
    "          y = row + 0.5\n",
    "          dx = 0\n",
    "          dy = -0.8\n",
    "        if argmax_a == 1:\n",
    "          x = col - 0.5\n",
    "          y = row\n",
    "          dx = 0.8\n",
    "          dy = 0\n",
    "        if argmax_a == 2:\n",
    "          x = col\n",
    "          y = row - 0.5\n",
    "          dx = 0\n",
    "          dy = 0.8\n",
    "        if argmax_a == 3:\n",
    "          x = col + 0.5\n",
    "          y = row\n",
    "          dx = -0.8\n",
    "          dy = 0\n",
    "        plt.arrow(x, y, dx, dy, width=0.02, head_width=0.4, head_length=0.4, length_includes_head=True, fc='k', ec='k')\n",
    "\n",
    "def plot_rewards(xs, rewards, color):\n",
    "  mean = np.mean(rewards, axis=0)\n",
    "  p90 = np.percentile(rewards, 90, axis=0)\n",
    "  p10 = np.percentile(rewards, 10, axis=0)\n",
    "  plt.plot(xs, mean, color=color, alpha=0.6)\n",
    "  plt.fill_between(xs, p90, p10, color=color, alpha=0.3)\n",
    "\n",
    "def parameter_study(parameter_values, parameter_name,\n",
    "  agent_constructor, env_constructor, color, repetitions=10, number_of_steps=int(1e4)):\n",
    "  mean_rewards = np.zeros((repetitions, len(parameter_values)))\n",
    "  greedy_rewards = np.zeros((repetitions, len(parameter_values)))\n",
    "  for rep in range(repetitions):\n",
    "    for i, p in enumerate(parameter_values):\n",
    "      env = env_constructor()\n",
    "      agent = agent_constructor()\n",
    "      if 'eps' in parameter_name:\n",
    "        agent.set_epsilon(p)\n",
    "      elif 'alpha' in parameter_name:\n",
    "        agent._step_size = p\n",
    "      else:\n",
    "        raise NameError(\"Unknown parameter_name: {}\".format(parameter_name))\n",
    "      mean_rewards[rep, i] = run_experiment(grid, agent, number_of_steps)\n",
    "      agent.set_epsilon(0.)\n",
    "      agent._step_size = 0.\n",
    "      greedy_rewards[rep, i] = run_experiment(grid, agent, number_of_steps//10)\n",
    "      del env\n",
    "      del agent\n",
    "\n",
    "  plt.subplot(1, 2, 1)\n",
    "  plot_rewards(parameter_values, mean_rewards, color)\n",
    "  plt.yticks=([0, 1], [0, 1])\n",
    "  plt.ylabel(\"Average reward over first {} steps\".format(number_of_steps), size=12)\n",
    "  plt.xlabel(parameter_name, size=12)\n",
    "\n",
    "  plt.subplot(1, 2, 2)\n",
    "  plot_rewards(parameter_values, greedy_rewards, color)\n",
    "  plt.yticks=([0, 1], [0, 1])\n",
    "  plt.ylabel(\"Final rewards, with greedy policy\".format(number_of_steps), size=12)\n",
    "  plt.xlabel(parameter_name, size=12)\n",
    "\n",
    "def epsilon_greedy(q_values, epsilon):\n",
    "  if epsilon < np.random.random():\n",
    "    return np.argmax(q_values)\n",
    "  else:\n",
    "    return np.random.randint(np.array(q_values).shape[-1])\n",
    "\n",
    "\n",
    "def colorline(x, y, z):\n",
    "    \"\"\"\n",
    "    Based on:\n",
    "    http://nbviewer.ipython.org/github/dpsanders/matplotlib-examples/blob/master/colorline.ipynb\n",
    "    http://matplotlib.org/examples/pylab_examples/multicolored_line.html\n",
    "    Plot a colored line with coordinates x and y\n",
    "    Optionally specify colors in the array z\n",
    "    Optionally specify a colormap, a norm function and a line width\n",
    "    \"\"\"\n",
    "    segments = make_segments(x, y)\n",
    "    lc = mcoll.LineCollection(segments, array=z, cmap=plt.get_cmap('copper_r'),\n",
    "                              norm=plt.Normalize(0.0, 1.0), linewidth=3)\n",
    "\n",
    "    ax = plt.gca()\n",
    "    ax.add_collection(lc)\n",
    "    return lc\n",
    "\n",
    "\n",
    "def make_segments(x, y):\n",
    "    \"\"\"\n",
    "    Create list of line segments from x and y coordinates, in the correct format\n",
    "    for LineCollection: an array of the form numlines x (points per line) x 2 (x\n",
    "    and y) array\n",
    "    \"\"\"\n",
    "    points = np.array([x, y]).T.reshape(-1, 1, 2)\n",
    "    segments = np.concatenate([points[:-1], points[1:]], axis=1)\n",
    "    return segments\n",
    "\n",
    "\n",
    "def plotting_helper_function(_x, _y, title=None, ylabel=None):\n",
    "  z = np.linspace(0, 0.9, len(_x))**0.7\n",
    "  colorline(_x, _y, z)\n",
    "  plt.plot(0, 0, '*', color='#000000', ms=20, alpha=0.7, label='$w^*$')\n",
    "  plt.plot(1, 1, '.', color='#ee0000', alpha=0.7, ms=20, label='$w_0$')\n",
    "  min_y, max_y = np.min(_y), np.max(_y)\n",
    "  min_x, max_x = np.min(_x), np.max(_x)\n",
    "  min_y, max_y = np.min([0, min_y]), np.max([0, max_y])\n",
    "  min_x, max_x = np.min([0, min_x]), np.max([0, max_x])\n",
    "  range_y = max_y - min_y\n",
    "  range_x = max_x - min_x\n",
    "  max_range = np.max([range_y, range_x])\n",
    "  plt.arrow(_x[-3], _y[-3], _x[-1] - _x[-3], _y[-1] - _y[-3], color='k',\n",
    "            head_width=0.04*max_range, head_length=0.04*max_range,\n",
    "            head_starts_at_zero=False)\n",
    "  plt.ylim(min_y - 0.2*range_y, max_y + 0.2*range_y)\n",
    "  plt.xlim(min_x - 0.2*range_x, max_x + 0.2*range_x)\n",
    "  ax = plt.gca()\n",
    "  ax.ticklabel_format(style='plain', useMathText=True)\n",
    "  plt.legend(loc=2)\n",
    "  plt.xticks(rotation=12, fontsize=10)\n",
    "  plt.yticks(rotation=12, fontsize=10)\n",
    "  plt.locator_params(nbins=3)\n",
    "  if title is not None:\n",
    "    plt.title(title, fontsize=20)\n",
    "  if ylabel is not None:\n",
    "    plt.ylabel(ylabel, fontsize=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24cfcfe8",
   "metadata": {},
   "source": [
    "## Task 1 - Implement a general Q-learning agent \n",
    "Implement a  **general Q-learning** agent that learns action values from experience.   The agent must act according to an $\\epsilon$-greedy policy over its action values.  It must be configurable so as to implement any of **Sarsa**, **Expected Sarsa**, **Q-learning**,  and **double Q-learning**.\n",
    "\n",
    "The `__init__` must accept two functions `target_policy` and `behaviour_policy` as arguments.   The function `behaviour_policy(action_values)` should map `action_values` to a single action. \n",
    "\n",
    "For instance, the random policy can be implemented as:\n",
    "```\n",
    "def behaviour_policy(action_values):\n",
    "  return np.random.randint(len(action_values))\n",
    "```\n",
    "\n",
    "The target policy is defined by a function `target_policy(action_values, action)`, which returns **a vector** with one probability per action.  The `action` argument is used for instance by Sarsa (where the target policy is a greedy distribution with its peak on the selected action).\n",
    "\n",
    "For instance, the random target policy is:\n",
    "```\n",
    "def target_policy(action_values, unused_action):\n",
    "  number_of_actions = len(action_values)\n",
    "  return np.ones((number_of_actions,))/number_of_actions\n",
    "```\n",
    "\n",
    "The `__init__` must also accept  a `double` boolean flag. Note that this is compatible with any choice of `target_policy` and `behaviour_policy`. For instance, if the `target_policy` is the policy described above for Sarsa and `double=True`, the algorithm should implement **double Sarsa**. Note that we then need two action-value functions.\n",
    "\n",
    "You should be able to use the `__init__` as provided below, so you just have to implement the `step` function.  We store the initial state in the constructor because you need its value on the first `step` in order to compute the update on the first transition in the agent's lifetime.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "0464db46",
   "metadata": {},
   "outputs": [],
   "source": [
    "class GeneralQ(object):\n",
    "\n",
    "  def __init__(self, number_of_states, number_of_actions, initial_state,\n",
    "               target_policy, behaviour_policy, double, step_size=0.1):\n",
    "    # Settings.\n",
    "    self._number_of_actions = number_of_actions\n",
    "    self._step_size = step_size\n",
    "    self._behaviour_policy = behaviour_policy\n",
    "    self._target_policy = target_policy\n",
    "    self._double = double\n",
    "    # Initial state.\n",
    "    self._s = initial_state\n",
    "    # Tabular q-estimates.\n",
    "    self._q = np.zeros((number_of_states, number_of_actions))\n",
    "    if double:\n",
    "      self._q2 = np.zeros((number_of_states, number_of_actions))\n",
    "    # The first action in an agent's lifetime is always 0(=up) in our setup.\n",
    "    self._last_action = 0\n",
    "\n",
    "  @property\n",
    "  def q_values(self):\n",
    "    return (self._q + self._q2)/2 if self._double else self._q\n",
    "\n",
    "  def step(self, reward, discount, next_state):\n",
    "    # we need to consider two kinds of case, which are SARSA and Q-learning\n",
    "    # This system implement the random policy, so we need to select two action-value function (q and q')\n",
    "    # because of the over/under-estimated values as shown in Lecture note 6: Model-Free Control page 29-33\n",
    "    action = self._behaviour_policy(self.q_values[next_state])\n",
    "    \n",
    "    # Define the Double Q-learning\n",
    "    if self._double == True:\n",
    "        if np.random.rand() >= 0.5: # For the action-value function q\n",
    "            self._q[self._s, self._last_action] = self._q[self._s, self._last_action] + self._step_size * (reward + discount * np.dot(self._q2[next_state], self._target_policy(self._q[next_state], action)) - self._q[self._s, self._last_action])\n",
    "\n",
    "        else:  # For the action-value function q'\n",
    "            self._q2[self._s, self._last_action] = self._q2[self._s, self._last_action] + self._step_size * (reward + discount * np.dot(self._q[next_state], self._target_policy(self._q2[next_state], action)) - self._q2[self._s, self._last_action])          \n",
    "            \n",
    "    else:  # Define SARSA\n",
    "        self._q[self._s, self._last_action] = self._q[self._s, self._last_action] + self._step_size * (reward + discount * np.dot(self._q[next_state], self._target_policy(self._q[next_state], action)) - self._q[self._s, self._last_action])\n",
    "     \n",
    "    # Update the action and new state\n",
    "    self._last_action = action\n",
    "    self._s = next_state\n",
    "    return action"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2c2b2be",
   "metadata": {},
   "source": [
    "## Experiment 1: Run the cells below\n",
    "\n",
    "This will train Q-learning, Sarsa, Expected Sarsa, and double Q-learning agents on the deterministic version of the Grid problem.\n",
    "\n",
    "The agents will be trained  with a step size $\\alpha=\\frac{1}{10}$ and $\\epsilon$-greedy behaviour, with $\\epsilon=\\frac{1}{4}$.\n",
    "\n",
    "The plots will show action values for each of the actions, as well as a state value defined by $v(s) = \\max_a q(s, a)$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "08efe848",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Agent settings.\n",
    "# Do not modify this cell.\n",
    "epsilon = 0.25\n",
    "step_size = 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "46803afd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkoAAAI9CAYAAADFBFfgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABEFElEQVR4nO3deZxcVZ3///enO1uHBBLWQBIwgIMsMhoWRQ2gYUyUJcKo4CiDLEKE4UsMiywj+JMRHAGZcckXIxDgyyISAhoQMRgFJk7IggQCsgQNSSQmNCEhay9Vn98fVd1UdbqqblfX7XtP9+v5eNxHum6duvdUp26fd51z77nm7gIAAMD26pKuAAAAQFoRlAAAAEogKAEAAJRAUAIAACiBoAQAAFACQQkAAKAEghIAAEAJBCUAAIASCEoAipjZ9WY2JaF9LzCzg5PYNwB0xpiZG0AbM9tN0nOS9nf3rQns/4uSTnX3f+7pfQNAZ+hRAlDoq5J+nURIyvuVpE+a2Z4J7R8AihCUgD7GzOrM7AozW2Fmb5nZ+WbWku9N+oykJzuU39fMHjGzRjPbYGZzKmzfzWz/gsd3mNl/FDxent//S2b2jpnNMLNBkuTu2yQtlvTpWr7n0JjZ7Wa21syWFqzb2czmmNlr+X+HFzx3hZktM7NXzGxCMrUGeieCEtD3XC3pBEnjJO0v6V8lve3ub0n6oKRXOpS/S9JjkvbIL9+uQR2+LGmCpP0k/YOkfy947s+S/rEG+wjZHZImdlh3uaTfufv7Jf0u/1hmdpCk0yQdnH/NNDOr77mqAr0bQQnoQ/K9RlMlfdnd33D3DZIelfRCvsgwSRs7vGw/SfWS6t19m7vPq0FVfuzuK919naTvSvpSwXMb8/Xos9z9KUnrOqyeJOnO/M93Svpcwfqfu3uTu/9V0jJJR/ZEPYG+gKAE9C3jJf3Z3ZcXrNtF7wWldyQN7fCaLyvXGL9pZreZ2c41qMfKgp/fkLRXweOhktbXYB+9zR7uvlqS8v/unl8/UsW/z1X5dQBqoF/SFQDQo3aV9FbbAzPrJ+lESW3nED2v3FDYwrYy7j5X0lwz213Sr5U74fsHZfaxRdLggscjlGu8C40u+HlvSW8WPD5Q0t2V30r6TZw40RsbG4vWLV68+EVJ2wpWTXf36d3YjXWyjsuZgRohKAF9y8uSvmNmY5Trtfm+pH0ltZ00/GtJx0i6R5LM7BTlepuWKdfTM1y56QNkZndIkrt/tcM+npP0L2b2oqR/ym9vUYcyF5jZI8qFqisl3Z/f5kBJh0k6o9vvNAUaGxu1aNHConVmddvc/fAqNrfGzPZ099X5qwLX5tevUnHwHKXi4AmgGxh6A/oQd39C0gOSlkh6RrkQlJX0Yr7IXZI+a2YN+cefUO4quI3Khajv5XuYpFzj3Nn5Shcp10u1Xrlhu4c7KXOvpN9K+kt+aevROknSH9y9lzT0LvdM0dINv9J7AfIMSb8sWH+amQ3MB+D3S1rQnR0BeA8TTgJ9mJl9WtJP8ldSta27TtJad/+vMq8boFzYOtTdW7q4z+WSzsmHto7PPSPpbHdfut0LA3T44Yf5ggX/W7Suvn7g4ko9SmZ2n6RjlRsqXSPpGuUC5y+UG6pcIekL+ZPhZWZXSTpLUqukKe7+WE3fCNCHMfQG9G0H6r0TuSVJ7n5lpRe5e3P+tTXl7h+p9TaT5p6t4jX+pRJPjS9R/rvKXT0IoMYISkDfdqDeOz8JNebu3R1uA5AwghLQh7n75AT2+b6e3meyut6jBCA9CEoAEBuvaugNQHoQlAAgNgy9AaEjKAE1VG/mHFR9R6ukjHtnEz62IygBYeNvOlBD/ZSbhhp9w98rlnBxjhIQNoISAMSIc5SAsBGUACAmTA8AhI+gBAAxIigBYSMoAUBsmB4ACB1BCQBiRI8SEDaCEgDExiURlICQEZQAIEYMvQFhIygBCN7XPv/5SOV+NnNmzDUpxlVvQPgISgAQI4ISEDaCEgDEhpm5gdARlAAgNq5stjXpSgDoBoISAMSKoTcgZAQlAIhJ7mRuht6AkBGUACBGnMwNhI2gBACxcblzjhIQMoISAMSKoTcgZAQlAIgNE04CoSMoAUBsGHoDQkdQAlLuobv/K1K5k78yJdZ69LRPfvzjNd/mjrvsErnsu2+/3e39uUueZegNCBlBCQBi5O5JVwFANxCUACAu7vIMQQkIGUEJAGLkWYISEDKCEgDEiaE3IGgEJQCIi0ue4WRuIGQEJQCIicsZegMCR1ACgLg45ygBoSMoAUCMmB4ACBtBCQBixPQAQNgISkAC7rvxPyKX7T90QKRyj//+tsjbnPDJsyOXTcotT8yMXHbQoBGRyg05/fTI27z57rsjly2JoTcgeAQlAIiNSwQlIGgEJQCIibuUZXoAIGgEJQCIEz1KQNAISgAQF85RAoJHUAKA2HBTXCB0BCUAiAs9SkDwCEoAEBMXQQkIHUEJAGJEUALCRlACgLi4y5keAAgaQQkAYkSPEhA2ghKQgMXPvxq57JENB0UqN3jUjtVWJ5VWPPt45LJD9x4eqdyWpqZqq1MdF/MoAYEjKAFATFzOzNxA4AhKABAXepSA4BGUACBGnKMEhI2gBABxcclbGXoDQkZQAoDYOD1KQOAISgAQF25hAgSPoAQAMXGG3oDgEZQAIEb0KAFhIygBQFzc5RmCEhAyghKQgPWbN0cu+9ziaLN47/5a75qZ+/Gfzo1cdv8RIyKVy3rPhxZ6lICwEZQAIC6cowQEj6AEADFx0aMEhI6gBABxcZdzrzcgaAQlAIgLQ29A8AhKABAjht6AsBGUACAuTA8ABI+gBAAxYWZuIHwEJQCIET1KQNgISgAQF656A4JHUAKAuLjkLQQlIGQEJSABLZlM5LJ/W7cuUrnV77wTeZtnnnxy5LJRmFlNtyd18TYvy5dHKufcwgRAFxGUACAmLimTQDgDUDsEJQCIi7uyWYbegJARlAAgJi4pQ1ACgkZQAoAYZRl6A4JGUAKAmDhDb0DwCEoAECOG3oCwEZQAICZc9QaEj6AEAHFh6A0IHkEJAGLCVW9A+AhKQALufPjhyGVrPYt2b9QacabzWx98MOaaFHN3ghIQOIISAMSI6QGAsBGUACAmrui9XQDSiaAEAHFxp0cJCBxBCQBiwsncQPgISgAQI6YHAMJGUAKAmLi7WglKQNAISgAQE4begPARlAAgLszMDQSPoAQAMWF6ACB8BCUAiAkzcwPhIygBKTfjoYeSrgK6gaAEhI2gBAAxcXe1MPQGBI2gBAAx4ao3IHwEJQCIC+coAcEjKAFATFxi6A0IHEEJAGLCVW9A+AhKABATl5ShRwkIGkEJAOLCvd6A4BGUACAmLqmltTXpagDoBoISAMTE3bmFCRA4ghJQQ81S4wrpjaTrgR6zT7knXWLoDQgcQQmoIXffLek6ID3cXU0tLUlXA0A3EJQAICYMvQHhIygBQEy41xsQPoISAMQk665tzc1JVwNAN9QlXQEA6K3cXc2trUVLd5nZDWb2spk9b2YPmdmwgueuMLNlZvaKmU3o9s4AEJQAIC7uruaWlqKlBuZIOsTdD5X0qqQrJMnMDpJ0mqSDJU2UNM3M6muxQ6AvIygBQEyy7tqybVvR0l3u/lt3b+uami9pVP7nSZJ+7u5N7v5XScskHdntHQJ9HOcoAUBMspmMtmzcGOcuzpJ0f/7nkcoFpzar8usAdANBCQBisi2bffzljRt37bB6kJktKng83d2nFxYwsyckjehkk1e5+y/zZa6S1CrpnraXdVLeq6s5gDYEJQCIibtPrPJ1x5V73szOkHSCpPHu3haGVkkaXVBslKQ3q9k/gPdwjhIABMTMJkr6pqST3H1LwVO/knSamQ00szGS3i9pQRJ1BHoTepQAICw/ljRQ0hwzk6T57j7Z3V80s19Iekm5IbkL3J3ZLoFusvd6bQEAAFCIoTcAAIASCEoAAAAlEJQAAABKICgBAACUQFACAAAogaAEAABQAkEJAACgBIISAABACQQlAACAEghKAAAAJRCUAAAASiAoAQAAlEBQAgAAKIGgBAAAUAJBCQAAoASCEgAAQAkEJQAAgBIISgAAACUQlAAAAEogKAEAAJRAUAIAACihzwQlM7vezKZ04/XLzey4iGUPMLM/mdlGM/s/JcosMLODq60PkAZRjys+7whRd9uNEtt80cyOjVg2cruTL89xFoM+EZTMbDdJ/yrppz20y8sk/cHdh7r7D0t82G+U9J0eqg9Qc108rvi8IyhxtRvufrC7/6EW2+qkbeE4i0GfCEqSvirp1+6+tYf2t4+kFyuU+ZWkT5rZnj1QHyAOX1X044rPO0LzVdWw3TCzfrXYTgUcZzHoNUHJzOrM7AozW2Fmb5nZ+WbWkv9W8BlJT3Yov6+ZPWJmjWa2wczmdHF/e5nZg/l9/bVtiM3M5kr6pKQfm9kmM7tP0t6SZucfXyZJ7r5N0mJJn+7+uw+Hmd1uZmvNbGnBup3NbI6ZvZb/d3jBc1eY2TIze8XMJiRT677JzHYyMzezXQvWHWRma8xsR3U4rsodU331814tjpOeEXe7ke/x+aaZPS9ps5n1K+wFMrOxBadpPGBm95vZf3TYzIfM7Pn8/u43s0H51/4/dWhb+tJx1pPHSK8JSpKulnSCpHGS9leuy/Rtd39L0gclvdKh/F2SHpO0R375dtQdmVmdpNmSlkgaKWm8pClmNsHdPyXpaUn/5u5D3P1LklZIOjH/+PsFm/qzpH/s6hsN3B2SJnZYd7mk37n7+yX9Lv9YZnaQpNMkHZx/zTQzq++5qvZt7r5B0ipJBxWs/q6k/3T3d7X9cVXpmOqLn/dq3SGOk57QE+3GlyQdL2mYu7e2rTSzAZIeUu7/emdJ90k6uZPXf1G5/9cxkg5VrqdL7n66Om9b+spxdod66BjpFUEpn/6nSvqyu7+R/wP/qKQX8kWGSdrY4WX7SaqXVO/u29x9Xhd2eYSk3dz9O+7e7O5/kfQz5f4jumJjvm59hrs/JWldh9WTJN2Z//lOSZ8rWP9zd29y979KWibpyJ6oJ9otlXSgJJnZkZLGSvpJ/rlhKj6uKh1Tfe7zXi2Ok/j1YLvxQ3df2ckQ3kcl9cs/3+LusyQtKPH6N919nXJf0D9UYX994jjryWOkVwQl5Xp0/uzuywvW7aL3PvDvSBra4TVfVu6X96aZ3WZmO3dhf/tI2svM1rctkq5U7htGVwyVtL6Lr+mN9nD31ZKU/3f3/PqRklYWlFuVX4ees1Tv9ShdL+nb7t6Uf9zxuKp0TPF57x6Ok9rqqXZjZYn1e0n6m7t7hbJ/L/h5i6QhFfbXl4+zWI6Rnji5rCfsKumttgeWO2nuREltY73PS/oHSQvbyrj7XElzzWx3Sb9WrjvzBxH3t1LSX/Pde1F4ifUHSro74jZSZeLEid7Y2Ljd+sWLF78oaVvBqunuPr3K3Vgn60r9LhGPpZK+kj+nYk/lhh7aFB1XEY6pYD/v1Zo4cbQ3Nm4rWrd4cWMtjxGJ46RaPdVulPq/WC1ppJlZQVgaLen1LryHzrYd1HEWQlvSW4LSy5K+Y2ZjlEvS35e0r3J/5KXcB/oYSfdIkpmdoty3hmXKpe/hkp7LP3eHJLn7V8vsb4Gkd83sm5J+KKlZuQ9ng7sv7KT8mnx92pnZQEmHSTqjC+8zNRobG7Vo0fZv1axum7sf3sXNrTGzPd19teWu1libX79KuT8cbUZJerOqCqNabT1K10m60t0zBc+1H1fljikp/M97tRobt2nholOK1tXZ9GqOEYnjpNZ6ut3o6H8lZST9m5n9X+XOYzpS0h+6sI2itiXE4yyEtqRXDL25+xOSHlDu5OpnlPswZ/XeJfp3SfqsmTXkH39CuasZNip3MHwv/01Byv0yy4475xuLE5UbK/6rpEZJt0raqcRLrpf07/lhukvy605Sbq6lYP+guWe3W6r0K713YJ8h6ZcF608zs4H5P2bvV+dj+IjPS5JGSMq4+8Mdnis8rsodU1Iv+LxXK+vFSzdwnNRQT7cbney/WdIpks5WLqh9RdIjkprKvKyjjm1LkMdZ6tsSd+91i3KXRr7WYd11kqZUeN0A5a4Y6N8DdXxG0iFJ/66qXQ47bKxnMs3bLZIWVXjf9ynX5dyiXMo/W7nzAn4n6bX8vzsXlL9Kua7oVyR9Jun3zbLd/2fF4ypfLujPe7XL2MN29ebMuUVLpWMk//viOOnhJQ3tRv44ObObrw/qOAuhLbH8BnoVM7tI0jHufkrFwqjK4Ycf5s88s/0XqH79GhZ7dcMKQK9z2GG7+bxniv8MNfSfzjGSQkm0G2Z2jHINd6NyJ4rfImlfz5+Q3BeE0Jb0lnOUOjpQ740zIxauXC81gFJc3R5uQ89Jot04QNIvlLuS7XVJn+9LISkn/W1JrwxK7j456Tr0du65cWUA5RGUwpBEu+G5q7i6c8Vj8EJoS3plUEJPcBVfAAWgI5eUISgBZaS/LSEooWpp/xYApAE9SkB5aW9LuhSU6s2cZNV3tErKuHc2UZfy35V7sjpBsMGDXMM6TuaLXmv9RvmWbSWOkdywAkFpe7QlfUuz1Ojuu3X+bPrbki59VvspN6EK+oa/l302/d2liRg2VDqXiy37jOmzKhbJpPvLciJoS/qWFdIbpZ9Nf1tCqEfV0t5dCiSNq96AytLelhCUUJUQrlQA0oCgBJQWQltCUEKVXO6tSVcCSDV3rnoDykt/W0JQQpU89d8CgDSgRwkoJ/1tCUEJ3ZDuDzeQNM5RAqJId1tCUELV0v4tAEgDghJQXtrbEoISqpT+cWUgae5MDwCUl/62hKCEqrinf1wZSAN6lIDSQmhLCEoB+NrnPx+p3M9mzoy5Jh2le5Kw3uK746OVu+p38dYjzQ4qMedvRy+9FW89OkNQQhqktx2R0t6WEJRQpfTPpgokjZviApWkvy0hKKFqaf9wA4njXm9ARWlvSwhKqFL6x5WBpDE9AFBJ+tsSghKqkjsBL91XKgBpwFVvQGkhtCUEJVQt7d2lQNLoUQIqS3tbQlBClVxpn00VSBznKAEVpL8tISihamnvLgWSxlVvQGVpb0sISqhS+i/pBNKAHiWgnPS3JQQlVCn9H24gaZyjBFSS/raEoISquKf/RoZAGhCUgNJCaEsSD0oP3f1fkcr1Hzog8jb77zgw4jajlRuw06DI+x644+Bo2xwwIvI2f/y16yKV++HUqZG3+X9+8IPIZUvxbLo/3GkW9bYkkhT1oz/jc9G3OSziR3p4xHK7RPvYS5J2i1h2jyHRt3nwT6KVO/GA6Nuc/Ur0sqVwU9zueeSX0yKX7U1/9wcN2jPyvgcNirbN//jCFyKV+/CHPhR533967rnIZctJe1uSeFBCqFzOV2WgIg4ToJz0tyUEJVSHky+ASDhMgDICaEsISqha2r8FAEljegCgsrS3JQQlVMfT/+EGEseEk0B5AbQlBCVUxZX+DzeQtABGFYBEhdCWEJRQHXc5l/MAFXGYAGUE0JYQlFA193R/CwCSRo8SUFna2xKCEqpHCwCUxzlKQGUpP0gISqiOS1ku5wHK4qo3oIIA2pLEg1Lrxqaab/OESefXfJtJeuCW/4xULrO5JeaaFOKrcne8uTF62b2G1n7/Zz5c+23W2t8viV72xQuilfv6I9XVpTs4TKq3efn6yGV3eN+wSOUmfPLs6iqTUi9vWRWp3G477hipXK1m244u/W1J4kEJYfIALukEksY5SkB5IbQlBCVULe0fbiANOEyA8tLelhCUUB1X6i/pBJLGTXGBCgJoSwhKqFL6b2QIpAGHCVBO+tsSghKqlvYPN5A0zlECKkt7W0JQQlXcJU/5JZ1AGnCYAKWF0JYQlFC1tH8LAJLm6b/yGUhc2tsSghKqQwsARMJhApQRQFtCUELVsim/UgFImour3oBK0t6WJB6Unn56SaRye++6a8w16Vk77rJL5LLP/PHFSOX2Gj682up0HWepdsvazdHL1lvEgkOqqkpqPfxy9LJjIn7031hfVVW6hcOkeu+ufjdy2b8vfyvGmqTXnx98IFK5rc3Nkco1DIn+h2Trpk2Ry5YUQFuSeFBCuNI+rgwkLv2jCkDi0t6WEJRQFXdP/SRhQNIYegPKC6EtISihamn/FgCkQbqbACB5aW9LCEqoTgBzXwBJC+D0CyBZAbQlBCVULe3fAoA04DABykt7W0JQQnUCGFcGksZNcYEKAmhLCEqoiiv93aVAGqT8yzKQqBDaEoISquPp7y4FksY5SkAFAbQlBCVUx13emu7uUiANUv5lGUhWAG0JQQlVS/u3ACBpAdzGCkhc2tuSxIPShi1bIpV7ceXKyNs855//OVK5TDZaijWLeg+J6Oq6sM2ov6Oo5WrCJY/4+8P2XlgTvezKiHdx2KUh+jY/EPGOQFH/fNX+CJFufTZ62dE7RSv3xobq6tIdKW8DUu1/Xo5+H5sB/aI1Z2d87nORynXlb3RUUduSrux77q8XRCq3raUl8jZ7VABtSeJBCWFySd5KCwCUw8zcQHkhtCUEJVQngEs6gTSgRwkoI4C2hKCE6tECAOVxjhJQWcoPEoISquNSNuVXKgBJY+gNqCCAtoSghOq4p36SMCANUv5lGUhWAG0JQQnVS/mHG0iaS0r3d2UgBVLelhCUUBV3pf4EPCAN6FECSguhLSEooTruqR9XBpLGTXGBCgJoSwhKqFrax5WBNKBHCSgv7W1JrwxKUWfcDoV7Cj9ErtSPK6dZVxrPdREnXH9na3V1SatNzdHLvvZ2tHI9HVq4KW7PaW5tjVQujhm3ay3bhb/5W5u7cKBEMHyXXaLve9Om7u8wgLakVwYlxM8ltfayQArEIeVtAJCoENoSghKq465syj/cQNK4KS5QQQBtCUEJVckNKdACAJUQlIDSQmhLCEqoWm87FwyoNWbmBipLe1tCUEJVPIDuUiAN6FECSguhLSEooWpp7y4FksY5SkBlaW9LCEqoSghXKgBpwFVvQGkhtCUEJVQngO5SIGkuicMEKCOAtoSghKqEcKUCkAYMvQGlhdCWJB6U7nz44Ujlzjz55Hgr0gvc+uCDPbq/tH8LSLNXI84kLUkH7BpfPXqLqLeKerkx3np0hqOkelHbB0k643Ofi60evUFPtw9dkfa2JPGghDC5e+rHlYGkcVNcoLwQ2hKCEqqW9m8BQBow9AaUl/a2hKCEqrh76icJA5LGTXGB8kJoSwhKqEoIl3QCiXOmBwDKCaEtISihamnvLgWSRo8SUFna2xKCEqri7sqk/JJOIA0ISkBpIbQlBCVUxSW1ZjJJVwNINW6KC5QXQltCUEJ1AphNFUgc93oDygugLSEooSq5b8rp/nADSeMcJaC8ENoSghKq4u6p7y4F0oCr3oDSQmhLgglKMx56KOkqoIO0fwvoLV5J4LYbqA16lHpOV253gnRJe1sSTFBCuoTQXQqkAUEJKC2EtoSghOoEMJsqkDiXOEyAMgJoSwhKqIq7qyXl48pA0lycowSUE0JbQlBCVULoLgXSgKE3oLQQ2hKCEqoSwocbSBoncwPlhdCWEJRQFXdXS2tr0tUA0o2b4gJlhdCWEJRQHffU3/EZSBo9SkAFAbQlBCVUJYT78wBpQFACSguhLSEooSohdJcCSeOmuEB5IbQlXQpKzVLjCumNuCqD1Nmn1BMupb67NBGrGxv1/03nGOk7Sh4jkrgpbgm0JX1O0G1Jl4KSu+8WV0UQlhDmvkgCxwgKcY5S5zhO0CaEtoShN1TF3dXc0pJ0NYDU46o3oLQQ2hKCEqoSwrcAIGn0KAHlhdCWEJRQFZdSfwIekAYEJaC0ENqSuqQrgDBl3dXc2rrd0h1mdoOZvWxmz5vZQ2Y2rOC5K8xsmZm9YmYTult/oEfkT+YuXLqL4wS9SQhtCUEJVfFsVlubmrZbummOpEPc/VBJr0q6QpLM7CBJp0k6WNJESdPMrL67OwPi1jY9QOFSAxwn6DVCaEsISqhK1l1NLS3bLd3h7r9197avEvMljcr/PEnSz929yd3/KmmZpCO7tTOgh9S6R4njBL1JCG0J5yihKp7NatuWLXHu4ixJ9+d/Hqnch73Nqvw6INV64GRujhMELYS2hKCEqmzNZB5/4e23d+3kqUFmtqjg8XR3n972wMyekDSik9dd5e6/zJe5SlKrpHvaXtZJeU6RRfq92fh45prpHY+TsseIxHGCviOEtoSghKq4+8QqX3dcuefN7AxJJ0ga7+5tH+BVkkYXFBsl6c1q9g/0JI4ToLwQjhHOUUJqmNlESd+UdJK7F/bF/krSaWY20MzGSHq/pAVJ1BFIGscJUF6tjxF6lJAmP5Y0UNIcM5Ok+e4+2d1fNLNfSHpJuW7UC9w93TOUAfHhOAHKq+kxYu/1SAEAAKAQQ28AAAAlEJQAAABKICgBAACUQFACAAAogaAEAABQAkEJAACgBIISAABACQQlAACAEghKAAAAJRCUAAAASiAoAQAAlEBQAgAAKIGgBAAAUAJBCQAAoASCEgAAQAkEJQAAgBIISgAAACUQlAAAAEogKAEAAJRAUAIAACiBoAQAAFACQQkAAKAEghLQx5jZ9WY2pYbbu8PM/qNW24u4zwVmdnBP7hNA30RQAvoQM9tN0r9K+mnSdemmGyV9J+lKAOj9CEpA3/JVSb92961JV6SbfiXpk2a2Z9IVAdC7EZSAXsbM6szsCjNbYWZvmdn5ZtaS7036jKQnO5Tf18weMbNGM9tgZnMqbP/DZvasmW00s/slDerw/IFm9gczW29mL5rZSfn1Z5rZ7IJyy8zsFwWPV5rZh/I/LzezS8zs+Xyd7jez9v24+zZJiyV9utrfU5qZ2e1mttbMlhas29nM5pjZa/l/hxc8d0X+9/mKmU1IptZA70RQAnqfqyWdIGmcpP2VG2p7293fkvRBSa90KH+XpMck7ZFfvl1qw2Y2QNLDkv6fpJ0lPSDpnwue7y9ptqTfStpd0oWS7jGzA5QLaOPyQW5PSf0lfTz/un0lDZH0fMHuvihpoqQxkg5Vrjes0J8l/WP5X0Ww7lDuvRe6XNLv3P39kn6XfywzO0jSaZIOzr9mmpnV91xVgd6NoAT0Ivleo6mSvuzub7j7BkmPSnohX2SYpI0dXrafpHpJ9e6+zd3nldnFR5ULOP/l7i3uPlPSwg7PD5H0PXdvdve5kh6R9CV3/0t+3x+SdIykxyX9zcw+kH/8tLtnC7b1Q3d/093XKRe+PtShLhvz76fXcfenJK3rsHqSpDvzP98p6XMF63/u7k3u/ldJyyQd2RP1BPoCghLQu4yX9Gd3X16wbhe9F5TekTS0w2u+rFxj+6aZ3WZmO5fZ/l6S/ubuXrDujQ7Pr+wQeN6QNDL/85OSjpV0dP7nPygXko5RhyFBSX8v+HmLcgGs0FBJ68vUtbfZw91XS1L+393z60dKWllQbpXe+30D6KZ+SVcAQE3tKumttgdm1k/SiZLaLt9/XtI/qKAXKN/rM9fMdpf0a+WGuH5QYvurJY00MysIS3tLej3/85uSRptZXUFY2lvSq/mfn8zXZ4yk65QLOl+WdJSkH3fxvR4o6e4uvqZHTZw40RsbG4vWLV68+EVJ2wpWTXf36d3YjXWyzjtZB6AKBCWgd3lZ0nfMbIxyIeT7kvaV1HZS8K+V6725R5LM7BTlepuWKddDM1zSc/nn7pAkd/9qwfb/V1KrpP9jZj+RdJJywzy/zz//jKTNki4zs5uUOwfpRElH5J9/UrkQtsbdV5nZu8qd79RP0p+ivkkzGyjpMElnRH1NEhobG7Vw4TNF6+rq+m1z98Or2NwaM9vT3Vfnz/Fam1+/StLognKjlAusAGqAoTegF3H3J5Q7wXqJcqHlBUlZSS/mi9wl6bNm1pB//AnlwstG5ULU9/I9TFKu8S06X8ndmyWdolyv0zuSTpU0q8PzJyl3dV2jpGmS/tXdX84//6qkTZKezj9+V9JfJM1z90wX3upJkv7g7ikPBC73bNHSDb/Se8HwDEm/LFh/mpkNzAfk90ta0J0dAXiPFZ9qAKA3MbNPS/pJ/kqptnXXSVrr7v9V5nUDlAtbh7p7S+wV7SIze0bS2e6+tGLhBB1++GG+YMEfi9bV1w9aXKlHyczuU+5crl0lrZF0jXJXG/5CuaHMFZK+kD/RXWZ2laSzlOvtm+Luj9X0jQB9GEEJ6MXM7CJJx7j7KUnXpS86/PCx/swz/1O0rl+/HSoGJQDpwTlKQO92oN47Pwk9zF3dHW4DkDCCEtCLufvkpOvQtzlBCQgcQQkAYkVQAkJGUAKA2LjcW5OuBIBuICgBNVRv5hxUfUerpIx7ZxM+tmPoDQgbf9OBGuonaUTSlUCP+XvFEpyjBISOoAQAMcld9daVeTQBpA1BCQBiQ48SEDqCEgDEih4lIGQEJQCIjTP0BgSOoAQAMWLoDQgbQQkAYkOPEhA6ghIAxMSdoASEjqAEpNwjv5wWqdwJk86PuSbp9bXPfz5SuZ/NnBlzTTpDUAJCRlACgNgwPQAQOoISAMSIoTcgbAQlAIgN5ygBoSMoAUBMuIUJED6CEgDExiVxjhIQMoISAMTG5d6adCUAdANBCQBixNAbEDaCEgDEhpO5gdARlAAgNgy9AaEjKAFATHJXvXEyNxAyghKQgAf+739GLtuysTlSuai3OpHCuN3JkneejVz2x1+7LlK5qLc6kWp4uxP32mwHQCIISgAQG1c2Q48SEDKCEgDExSVl6VECQkZQAoAYOUEJCBpBCQBi5JyjBASNoAQAMXGXPENQAkJGUAKA2DhDb0DgCEoAEBdO5gaCR1ACgBh5lukBgJARlAAgLs5Vb0DoCEpAAhbOfyly2SM+elC0gr3s6qotG1dELnv4vvtGKven5currE21OEcJCB1BCQBiwlVvQPgISgAQI3qUgLARlAAgRgQlIGwEJQCIi7ucm+ICQSMoAUCM6FECwkZQAoC4MOEkEDyCEgDExEWPEhA6ghIAxMWd6QGAwBGUACBG3MIECBtBCQDiwi1MgOARlIAErNmwIXLZx+Y8E6nc0IaGaquTSndcfE/ksgP6RftTlkmgd4ehNyBsBCUAiIk793oDQkdQAoAYcY4SEDaCEgDExSVvpUcJCBlBCQBixNAbEDaCEgDEhXOUgOARlAAgJi7JWzlHCQgZQQkA4sI8SkDwCEoAEBuXuOoNCBpBCQDi4kw4CYSOoASkXNTZpNdv3hx5m2eefHK11emUmdV0e1LXZtHe1tJS8/3XhEvZDD1KQMgISgAQJ3qUgKARlAAgJu7O0BsQOIISAMTIGXoDgkZQAoC4cDI3EDyCEgDEhaAEBI+gBACxcWbmBgJHUAKAmLhzjhIQOoISAMSFoTcgeAQlAIiNK8vQGxA0ghIAxMXFhJNA4AhKQALufPjhyGXP+NznIpWri+E2IqFwjxZGbn3wwZhrUswlZbkpLhA0ghIAxMVdmYghDkA6EZQAICaurt3cF0D6EJQAIEYMvQFhIygBQEzcXVmG3oCgEZQAIEYMvQFhIygBQEy46g0IH0EJAOLCVW9A8AhKABATrnoDwkdQAoAYMfQGhI2gBKRcV2bx7qv2/4d/SLoKnXKG3oDgEZQAIEaZTCbpKgDoBoISAMTE3TlHCQgcQQkAYuISE04CgSMoAUCM6FECwkZQAoCYuLtaCUpA0AhKABAjpgcAwkZQAoCYMOEkED6CEgDExN3VyvQAQNAISgAQI3qUgLARlAAgJsyjBISPoAQgeMtefTXpKnTKJYbegMARlAAgLvQoAcEjKAFATLjqDQgfQQkAYuLuamHoDQgaQQkAYkKPEhA+ghIAxIVbmADBIygBQExcUoahNyBoBCUAiIm7q7m1NelqAOgGghIAxIQJJ4HwEZQAICZMOAmEj6AE1FCz1LhCeiPpeqDH7FPuSXdXE0NvQNAISkANuftuSdcB6eHu9CgBgSMoAUBMXFILPUpA0AhKABATd1dTS0vS1QDQDQQlAIhJlukBgODVJV0BAOit3F0tra1FS3eZ2Q1m9rKZPW9mD5nZsILnrjCzZWb2iplN6PbOABCUACAu2WxWm7dtK1pqYI6kQ9z9UEmvSrpCkszsIEmnSTpY0kRJ08ysvhY7BPoyghIAxMSzWW3bsqVo6fY23X/r7m1dU/Mljcr/PEnSz929yd3/KmmZpCO7vUOgj+McJQCISTab1ZaNG+PcxVmS7s//PFK54NRmVX4dgG4gKAFATJrcH3+9uXnXDqsHmdmigsfT3X16YQEze0LSiE42eZW7/zJf5ipJrZLuaXtZJ+W9upoDaENQAoCYuPvEKl93XLnnzewMSSdIGu/ubWFolaTRBcVGSXqzmv0DeA/nKAFAQMxsoqRvSjrJ3QtPevqVpNPMbKCZjZH0fkkLkqgj0JvQowQAYfmxpIGS5piZJM1398nu/qKZ/ULSS8oNyV3g7tw/Begme6/XFgAAAIUYegMAACiBoAQAAFACQQkAAKAEghIAAEAJBCUAAIASCEoAAAAlEJQAAABKICgBAACUQFACAAAogaAEAABQAkEJAACgBIISAABACQQlAACAEghKAAAAJRCUAAAASiAoAQAAlEBQAgAAKIGgBAAAUAJBCQAAoASCEgAAQAkEJQAAgBIISgAAACUQlAAAAEogKAEAAJRAUAKAlDGz281srZktTbouQF9HUAKA9LlD0sSkKwGAoAQAqePuT0lal3Q9ABCUAAAASuqXdAUAoLeaOHGiNzY2Fq1bvHjxi5K2Faya7u7Te7RiACIjKAFATBobG7Vw4YKidXV19dvc/fCEqgSgiwhKABAbl5RNuhIAuoFzlAAgRu6ZoiUKM7tP0v9KOsDMVpnZ2bFWEkBJ9CgBQGxc7l3vUXL3L8VQGQBVICgBQEzcVVVQApAeBCUAiI1HHm4DkE4EJQCIFT1KQMgISgAQm+rOUQKQHgQloIbqzZyDqu9olZRxt3Jl3Ft7qDYA4sDfdKCG+kkakXQl0GP+XrEEPUpA6AhKABArghIQMoISAMTE3ZXNctUbEDKCEgDEiqAEhIygBACx4RwlIHQEJQCIEUEJCBtBCQBi40wPAASOoAQAsaJHCQgZQQlIuTfcix6/9dZbam5u1siRI4vW72Nl5z3s1Tr+jkrp6d+RO+coAaEjKAGB+ad/+ie9+eabWrt2bdJVQUUMvQGhIygBgVm+fLk2bNiQdDUQkTvTAwAhq0u6AgBQSzNnztSgQYO0evXqpKuitukBChcAYSEoAYFobm7WokWL2nuTFi1apK1btyZcq3TJZrO69NJL1draqmuvvTbp6sg9d1PcwgVAWAhKQCBmzZqlj370o6qryx22Rx11lGbMmJFwrdJl1qxZamxsVCaT0YwZM1LQq+TKXfVWuAAICUEJCMTEiRM1aNAgZbO5xrZfv3465ZRTEq5VerT1Jm3atEmSlMlkUtKrlClaAISFoAQEYtiwYZo6daoaGho0YMAAnXXWWRoxYkTS1UqNpqYmNTQ0aOjQoZKk4cOHp+Ckd2foDQgcQQkIyNSpU1VXV6e6ujp961vfSro6qdLQ0KCXXnpJDz74oCRpzZo1uueeexKuldOjBASO6QGAgAwbNkw/+MEPtGHDBnqTAsGVbkDYCEpAYM4999ykq4DIXBK9SEDICEpAAqLeciPpbYai8NYkLZ2sa9OV31EtbnfiLnmWHiUgZAQlAL1Kf0l7J12JAp7tuwEW6A0ISgAQF3eCEhA4rnoDUmDy5Mm68MILtWbNmrLlfvOb32j8+PGaP39+2XKNjY06+uijddttt6mlpaVs2d60767IZDK67777NG7cOK1cubJm2+3IM9miBUBYCEpACjz66KP66U9/qjFjxpQNDvPnz9fcuXM1fvx4jRs3rmRwWLNmjRYsWKApU6Zo1KhRZUNLb9p3FG0BacyYMTr33HO1aNEivfHGG1VvrxJ3L1oABKbjQczCwlL9MkDyvSMsQ4cOdeUuidpuqa+v9wEDBnhzc7Nff/31Jcu1LQ8++KC//PLLZcvU1dX5pEmT3N171b7bRPmd7y25u/uZZ57pdXV1Zbe9cOHCSNsbkNtmyc/DoYfs56tef7BokbQo6c8pCwtL9IVzlIAEPPXUU1q2bFn747POOksbN27UkCFDNGTIEF1//fXq16+fzjnnHO2///7t5e666y7Nnj1bAwcOVF1dnc477zwde+yxGj58uGbPnq1t27ZJkpYvX65rrrlGW7duVUNDg4444ghdffXVvW7f1bj44ou1evVqPfnkk2pqatLgwYM1depUffCDH5SUuzXMhz/84aq23ZHLlc3QiwQELemkxsLSm5aoPUodHXLIIT5ixAifMWOGt7S0bPd8m5tuusn79+/vU6ZM8bVr15Ys9/rrr3t9fb0fc8wxvmDBgpLlesu+u9Kj1GbJkiU+ceJEr6ur82effbaqbVbqUfrgwfv6ipcfKFpEjxILS1CLufNtB6iVgWYeZb7sjvP5bN68WQMHDlS/fuU7ed1d7777rnbaaaeK+9iwYUOkcr1h31HnPOpsHqVSdY2yzb9LanIvWfDQQ/bzR2Z+r3i7B35xsbsfHqG6AFKAoTcgBXbYYYdI5cwsUgCRFLlcb9t3V8W1XUmSiyvdgMARlIA+qBazTsetK7Nop3dWcmceJSBwBCUAiIlLBCUgcMyjBATmpZde0p/+9KeK5d5991098sgjPVCjnhHk+/ZcUCpcAISFoASkwJw5c/TMM89ULJfNZnX88cdrwoQJam5uLlv2hhtu0Iknnqg//vGPtapmYuJ63+vWrdPtt9+urVu31qqq2/GMFy0AwkJQAlLgrLPO0rhx43T00UeXDUyzZs1SY2OjtmzZojvvvLNkufXr1+vmm2+WJF166aU1r29Pq/X7XrdunS6//HKNHj1a5513nhYvXlzzOkvKjb1lvXgBEBSCEpCAr3/96zKz9mXVqlVqaWnR//zP/+hTn/qUjj76aGUymaLXZLNZXXrppdq0aZM2b96sq666qmTvyk033aRsNne11XPPPRd0r1Kt3/ctt9yi0aNH67//+7+1ZcsWubvGjRvX/n+x9957a9WqVTWqvTP0BgSOoAQkwCpcddbZ87Nnz9aKFSvaH69bt0733nvvduU2b96sG2+8sX04acuWLbriiiu6WePk1Pp9V/rd19XVVSwTWX56AG6KC4SLoAQkYNq0aUUzv44aNUr9+/fXuHHj9Pvf/15PPvmk6uvri15z2GGH6Rvf+Eb74ylTpugTn/jEdtsePHiwrr76ak2aNElS7pYdF154YbxvKEa1ft/nnXeeVq5cqSlTpmjw4MGSpKeffrr9/2L58uUaOXJkTeredtUbPUpAuJiZG6ihamfmfuKJJ7TjjjvqyCOPrPjaYcOGacOGDap07M6ZM0ef/vSnOy0X4jxKtXjfHa1bt06//OUvddppp6mhoaHouVrMzH3IAe/zB6dfU7TuA8eexczcQECYRwlIgeOOOy7pKvRJO++8s84888xY98FwGxA2ghIQmEr3ReutgnzfzoSTQOgC/MsD9G333HOPNm7cWLHc2LFjgz6Ju6NQ3zdBCQgb5ygBNVTtOUo9LcRzlHpaLc5ROvj9+/gvfnhV0bpDPnse5ygBAeGqNyAFNm3apJaWlkhlTzjhBH3sYx+rWG7mzJkaNGiQVq9e3d3qpUJc73v9+vXdrFl5ns0WLQDCQlACUuCoo47S6NGjdfvtt5cNTEuXLtXcuXO1ZMkSzZs3r2S5tkkaW1tbde2118ZR5R4Vx/t+7rnnNGHCBO2yyy6R7iFXLaYHAMJGUAIS8Oyzz+r+++9vX5YvX641a9booosuag9MnQ2LX3755WpqatKWLVvK3qKj7ZYfmUxGM2bMCL5XqZbve+nSpZowYYI+9rGP6YknntDgwYM1a9as9v+LWbNmbTcretVcUsaLFwBB4RwloIainqP0ztChJU9M7tevn+rr67Vx40b179+/ff3SpUs1duzY9h6nAQMGaO7cufr4xz9e9PpsNqv99ttPy5cvlyT1799f55xzjqZNm9ZeJqRzlGr5viXp7LPP1h133NF+q5POLFy4UP98xBEV61jxHKX99vZ7v//NonUf+vy/cY4SEBB6lIAEvPvuu53OzN3Q0KALLrhAK1asKApJktTY2Kjhw4e3Px42bJjWrl273babmprU0NCgoUOHSpKGDx+uDRs2xPuGYlTr9z19+nTde++92nvvvTVkyBANGjSoaGZud9fhh9cwx2SzxQuAoNCjBNRQtVe9nX/++RowYICuvPJK7b777mVfy8zctZmZO5PJaObMmZo2bZruvvtujR49uuj5Wlz1dtC+e/u9111StO7DX7qIHiUgIMyjBKRAx+EhxK++vl6nnnqqTj311Bj34szMDQSOoAQEpq/2Agf5vl1yTuAGgkZQAgJz+umnd3qOTkcjR44s+VzSkzlWoxbvOwlMCQCEjaAEBObHP/5xpHIHHXRQmL0wJQT5vl3yVobegJBx1RuQAtlsVqeffroWLlxYsewtt9yi73//+xXLvfTSSzr11FPV3NxciyomLsT37e7yTPECICwEJSAFZs2apXvvvVcXXXRR2XLr16/XJZdcomuuuabiJJKXXXaZZs6cqTvvvLNsuUwmo/vuu0/jxo3TypUrS5b7zW9+o/Hjx2v+/Pllt9fY2Kijjz5at912W8XbskyePFkXXnih1qxZU7Zcku+7u5iZGwgbQQlIWNttN7LZbMVbdNx4443KZDLKZrNlb9HRdsuPbDarq666qtPelbagMGbMGJ177rlatGiR3njjjZLbnD9/vubOnavx48dr3LhxJQPTmjVrtGDBAk2ZMkWjRo0qG5geffRR/fSnP9WYMWPKBqYk33e35IfeChcAgSmcZI2FhaV7ywDJ946wFHrggQd8yJAhrtwNL/yoo47yzrzzzju+ww47tJcbNGiQv/nmm52WPf74472urs4l+Q477ODTp0/frsyZZ57ZXqba5cEHH/SXX365bJm6ujqfNGmSu7sPHTq0ZLn6+nofMGCANzc3J/6+Fy5cGOn/cUDu/7Lk5+EDo0b6/958XdEiaVG517CwsKRr4WRuIGG33npre4/L4MGDNX/+fK1du3a7iSd/85vfaNu2bTIzubtaW1v10EMP6fzzzy8qt3nzZv32t79VQ0ODNm/erEwmo1tvvVVf+9rXispdfPHFWr16tZ588kk1NTVp8ODBmjp1qj74wQ9KkrZs2aJBgwapri7X8XzXXXdp9uzZGjhwoOrq6nTeeefp2GOP1fDhwzV79mxt27ZNkrR8+XJdc8012rp1qxoaGnTEEUfo6quvliQ99dRTWrZsWXsdzjrrLG3cuFFDhgzRkCFDdP3116tfv+I/Sz39vvv166cPf/jDEf/3KnHmUQJCl3RSY2HpTUs1PUpr1qzxRYsWuSS/5JJLfOnSpd6Z5ubm9nKSfNGiRb5ly5ZOy7788sv+ox/9qL3c3/72t07LubsvWbLEJ06c6HV1df7ss8+WLHfTTTd5//79fcqUKb527dqS5V5//XWvr6/3Y445xhcsWFCynLv7IYcc4iNGjPAZM2Z4S0tLp2WSet+16FE6YORePu/71xYtokeJhSWohVuYADVU7S1MJMnMdMstt+i8884r+9pa38qjzYYNG7TTTjuVfN7d9e6775YtE3VbbTZv3qyBAwdu14vUmZ5+37W4hckHRo702y4o/v/8xFXXcAsTICAMvQGQpIrBxswihZ8o22qzww47RCoXp6h1rY4zJQAQOK56A1KkoaGhYpkxY8ZUvHGulDvHqNYee+wxPfDAAzXfbhRR33f//v17oDYRueSZbNECICz0KAEp8cILL+jAAw+sWO63v/1txfmJpGihq6tOP/10vf3225GHtWop6vs+5phjtGTJkh6oUWUuKUs4AoJGjxKQEn/729/0zjvvVCy32267aa+99qpYziKcY9Pm8ccfV2NjY8Vyra2tkbdZa2+99VbFiSklaePGjVqxYkWkbUZ931VzV6bDAiAsBCUgBZYuXaqJEye2X0ZfysqVK3XPPfdE6lmp9b6jam5u1owZMyKFmjlz5uiZZ56pWC6bzer444/XhAkTKt6a5IYbbtCJJ56oP/7xj2XL1fp9d8YlZd2LFgBhISgBKXD55Zerrq5OM2bMKHuLjttuu01nnHGGRo8erdtvv73TwOTuuu6669pvInvxxRdr5syZ3dr3qlWrdPHFF2vDhg3t23zttdc6Lfvaa6/pa1/7WsXZtqXcPErjxo3T0UcfXTYwzZo1S42NjdqyZUvZW5OsX79eN998syTp0ksvLVlOiv47765MJlO0AAhM0vMTsLD0pqWaeZReeOEFb2hocEk+YMAA//rXv97+3GOPPeb9+/fvdPboIUOG+B577OGLFi0q2t6mTZvat9e2HH300d6Zcvsu9PDDDxfNZl1fX+8zZsxwd/e33nrLDzjggE7rOHDgQG9oaPDvfe977u4+efLkTsuZmQ8ePNjHjRvnra2tRfvOZDL+vve9r73sbrvt5k1NTZ3W89///d/b38/gwYN93rx5Vb/vWsyjtP8ee/ivL7usaBHzKLGwBLXQowQk7Fvf+lb7rNbNzc362c9+1n7ejJm1z4zdGTPb7lykHXbYQZdcckn7ydyDBw/W9ddf3+V9FzrxxBO1zz77tD/eeeed9S//8i9F9Sin7fmo5QrNnj276JyjdevW6d57792u3ObNm3XjjTdq69atknJX/V1xxRWd7ifq+64Fht6AsBGUgISdc845mjp1qiTpIx/5iK666irtvPPOkqQJEyZo27Zt7d9srrnmGtXV1WnEiBH60Y9+pJUrV2rs2LHbbXPq1KntAetDH/qQPvaxj3V534Xq6up0ww03aMiQIdphhx303e9+VwMGDJAk7brrrvrzn//cXselS5eqvr5eDQ0Nmjx5spYvX67LLrtMkjRt2rSib2qjRo1S//79NW7cOP3+97/Xk08+qfr6+qJ9H3bYYfrGN77R/njKlCn6xCc+sV0dBw8erKuvvlqTJk2SlBsevPDCC7v1vrvLxdAbEDpm5gZqKO6ZuVetWqWnnnpKX/ziFyvOZn311Vfr2muv1bx580oGpa7sO5vNav/999emTZu0atWq9qDUUUtLi+6991595jOfqTjv0RNPPKEdd9xRRx55ZNlyUjwzc5d737WYmXv/PfbwG7785aJ1p9x8MzNzAwFhHiUgIKNGjSoa8irnkksu0Uc+8pGKISmquro6Pfroo2pubi4ZkqTchI9nnHFGpG0ed9xxNalbWrlyARNAuAhKQC+144476vjjj6/pNqNMiBmXKPeDSyPOSwLCFuZfHiBwnQ3r7CTpO5Mn67rJk4vWdzZMV2vf+c53ah6qau2ee+7Rxo0bK5YbO3ZsyZO4O/7eS/3Oa8Xd1UqPEhA0zlECaijqOUpd0RNBqa+Ict5RV1Q6R2nf3Xf3a7/whaJ1X5k2jXOUgIBw1RsQmBNOOCHSeUfr16/v0najlt+wYUPF826ibmvTpk2RZhl398jbnDlzpgYNGhTrJJJRuaRMNlu0AAgLQQkIyNKlSzV37lwtWbJE8+bNK1nuL3/5i3bddVeNGzdO8+fPL7vN5557ThMmTNAuu+yiP/3pTyXLrVu3Tpdffrn22GOPkkNbXd33UUcdVXaW8TY333yzdt9994ozfWezWV166aVqbW3VtddeW3bfPcHdCUpA4DhHCQjI5ZdfrqampvZA0HY/M3fX7Nmz2ydbXL58uQYOHKh58+Zp/PjxGjt2rG6++WYdfvh7Iz5Lly7VxRdfrKefflpNTU0aPHiwZs2apVdffVVS7uq1SZMmadOmTbr++uv1ox/9SNlsVk1NTZo3b57uv//+9n1v3bpVgwcPrrjvZ599tujWJ8uXL9emTZt00UUX6corr9R1112nM888U42NjZo7d257ublz56qlpUU/+9nPdNttt+nss8/Wt7/9be2yyy5Fv5+2W51kMhnNmDFD3/rWt7TnnnvG8D8RHeEICBvnKAE1FOc5SkuXLtXYsWPbe14GDBiguXPn6uMf/7heeeUVfeADHyi5jbq6Op100kl66KGH2tedffbZuuOOO8oOoy1cuFCbN2/WscceKzOLNDdRuX3vuOOOJU/I7tevn+rr67Vx40bddNNNZXutpFwoOvnkk9sfZ7NZ7bffflq+fLmkXNA755xzNG3atPYyPX2O0j677upX5SfAbHPe7bdzjhIQEIbegEA0NjZq+PDh7Y+HDRumtWvXSpIOOOCAohmvly5dqoEDB2rIkCEaMWKEbrvtNj3wwANF25s+fbruvfde7b333hoyZIgGDRqkp59+umg7hx9+uI455hgtWbJEEyZMaL8tyrnnnlvyvkjl9v3uu+92OjN3Q0ODLrjgAq1YsUL9+/fX5ZdfXlTummuukZSbffuYY47RggULikKSJDU1NamhoUFDhw6VJA0fPrz9Jr5Jas1mixYAYWHoDQjEscceqzVr1rTPUF3uXJ099thDRx11lM444wx95Stf6XQOovr6ep166qn6/Oc/r5kzZ2ratGlF93MrdOihh+qxxx7T888/ryuvvFKf+tSnurXvNieeeKIGDBigK6+8suws3kcddZSOO+44XXfddTriiCM6LdPQ0KCXXnqpfWbucr+fntJ2MjeAcDH0BtRQT0wPEPVWHn1VuVuY9PTQ2+hddvGLP/vZonXfuPtuht6AgNCjBASGgBSWVm6ECwSNoAQE5vTTT28/NwnbGzlyZNJVaNc2PQCAcBGUgJQrNVxU62Gk3mRvpeP3wzlKQPgISgAQF3qUgOARlAAgJi7OUQJCR1ACgJhwjhIQPoISAMSISSaBsBGUACAm7q4Wht6AoBGUACAmLilDUAKCRlACgJi4O0NvQOAISgAQE3dXS2tr0tUA0A0EJQCICdMDAOEjKAE11Cw1rpDeSLoe6DH7lH3WnaAEBI6gBNSQu++WdB2QHll3NTH0BgSNoAQAMXGJc5SAwBGUACAmzKMEhI+gBAAx4ao3IHx1SVcAAHqrrLu2NTcXLVGY2UQze8XMlpnZ5TFXE0AZ9CgBQEzcXU0Rw1EbM6uX9BNJ/yRplaSFZvYrd38phioCqICgBAAxcXc1tbR09WVHSlrm7n+RJDP7uaRJkghKQAIISgAQk0xrqzY0Nnb1ZSMlrSx4vErSR2pWKQBdQlACgJg0uT/+enPzrh1WDzKzRQWPp7v79ILH1smmvPa1AxAFQQkAYuLuE6t42SpJowsej5L0Zm1qBKCruOoNANJloaT3m9kYMxsg6TRJv0q4TkCfRY8SAKSIu7ea2b9JelxSvaTb3f3FhKsF9FnmztA3AABAZxh6AwAAKIGgBAAAUAJBCQAAoASCEgAAQAkEJQAAgBIISgAAACUQlAAAAEogKAEAAJTw/wNQausvjTrLewAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x720 with 12 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Q-learning\n",
    "# Do not modify this cell.\n",
    "def behaviour_policy(q):\n",
    "  return epsilon_greedy(q, epsilon)\n",
    "def target_policy(q, a):\n",
    "  return np.eye(len(q))[np.argmax(q)]\n",
    "\n",
    "learned_qs = []\n",
    "for _ in range(5):\n",
    "  grid = Grid()\n",
    "  agent = GeneralQ(grid._layout.size, 4, grid.get_obs(), target_policy, \n",
    "                   behaviour_policy, double=False, step_size=step_size)\n",
    "  run_experiment(grid, agent, int(1e5))\n",
    "  learned_qs.append(agent.q_values.reshape(grid._layout.shape + (4,)))\n",
    "  \n",
    "avg_qs = sum(learned_qs)/len(learned_qs)\n",
    "plot_action_values(GRID_LAYOUT, avg_qs, vmin=-20, vmax=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "41b29c81",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkoAAAI9CAYAAADFBFfgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABFfElEQVR4nO3deZxcVZ3///enO0l3h4QskAAmQYO4AAFlVfhNwJkgCSORkUFAkYmGVRjHfMMXhUFEZQRGZHFURqOs82VRY1TCIgTZcSAkyhIiCEggkZjQhCSdrZeqz++Pqm6qOl1Vt6vr9r2n+/V8PO4jXbdO3XuqU7fPu86591xzdwEAAGB7dUlXAAAAIK0ISgAAACUQlAAAAEogKAEAAJRAUAIAACiBoAQAAFACQQkAAKAEghIAAEAJBCUARczsMjObk9C+F5vZPknsGwB6YszMDaCTmY2T9LSkPd19awL7P0HSie7+z/29bwDoCT1KAAp9XtLdSYSkvDsk/b2Z7ZbQ/gGgCEEJGGTMrM7MLjCz183sTTM728za871JR0t6uFv5PczsTjNrNrMNZraowvbdzPYseHyjmf1HweMV+f0vN7O3zewGM2uUJHffJmmppKNq+Z5DY2bXm9laM1tWsG6smS0ys5fy/44peO4CM3vZzF40s+nJ1BoYmAhKwODzdUnHSJoqaU9J/yLpLXd/U9K+kl7sVv5mSfdI2iW/fKMGdThZ0nRJ75X0fklfK3juT5I+VIN9hOxGSTO6rTtf0u/c/X2Sfpd/LDPbW9JJkvbJv+ZaM6vvv6oCAxtBCRhE8r1GcyWd7O6vufsGSXdJei5fZLSklm4ve6+kekn17r7N3R+vQVV+4O4r3X2dpG9L+kzBcy35egxa7v6IpHXdVh8r6ab8zzdJ+qeC9be7e6u7vyrpZUmH9Ec9gcGAoAQMLtMk/cndVxSs20nvBKW3JY3s9pqTlWuM3zCz68xsbA3qsbLg59ckvavg8UhJ62uwj4FmF3dfLUn5f8fn109Q8e9zVX4dgBoYknQFAPSrnSW92fnAzIZImimp8xyiZ5UbCnuqs4y7PyDpATMbL+lu5U74vqrMPrZIGl7weFflGu9Ckwp+3l3SGwWP95L0/yq/lfSbMWOGNzc3F61bunTp85K2Faya5+7z+rAb62EdlzMDNUJQAgaXFyR9y8wmK9dr8x1Je0jqPGn4bklHSLpFkszsOOV6m15WrqdnjHLTB8jMbpQkd/98t308LemzZva8pI/nt7ekW5lzzOxO5ULVv0v6WX6bDZIOlDSrz+80BZqbm7VkyVNF68zqtrn7QVVsbo2Z7ebuq/NXBa7Nr1+l4uA5UcXBE0AfMPQGDCLufr+kX0h6RtKTyoWgrKTn80VulvSPZtaUf/x3yl0F16JciLo838Mk5Rrnns5X+rJyvVTrlRu2+3UPZW6VdJ+kv+SXzh6tT0p6yN0HSEPvcs8ULX1wh94JkLMk/aZg/Ulm1pAPwO+TtLgvOwLwDiacBAYxMztK0g/zV1J1rrtU0lp3v6bM64YpF7b2c/f2Xu5zhaTT8qGt+3NPSjrV3Zdt98IAHXTQgb548f8Wrauvb1haqUfJzG6T9DHlhkrXSLpYucD5c+WGKl+X9On8yfAyswslzZbUIWmOu99T0zcCDGIMvQGD215650RuSZK7/3ulF7l7W/61NeXuH6n1NpPmnq3iNf6ZEk9NK1H+28pdPQigxghKwOC2l945Pwk15u59HW4DkDCCEjCIuftZCezzPf29z2T1vkcJQHoQlAAgNl7V0BuA9CAoAUBsGHoDQkdQAmqo3sw5qAaPDkkZ954mfOxCUALCxt90oIaGKDcNNQaHv1Us4eIcJSBsBCUAiBHnKAFhIygBQEyYHgAIH0EJAGJEUALCRlACgNgwPQAQOoISAMSIHiUgbAQlAIiNSyIoASEjKAFAjBh6A8JGUAIQvNOPPz5SuZ/Mnx9zTYpx1RsQPoISAMSIoASEjaAEALFhZm4gdAQlAIiNK5vtSLoSAPqAoAQAsWLoDQgZQQkAYpI7mZuhNyBkBCUAiBEncwNhIygBQGxc7pyjBISMoAQAsWLoDQgZQQkAYsOEk0DoCEoAEBuG3oDQEZQAICbukmcZegNCRlACgBi5e9JVANAHBCUAiIu7PENQAkJGUAKAGHmWoASEjKAEAHFi6A0IGkEJAOLikmc4mRsIGUEJAGLicobegMARlAAgLs45SkDoCEoAECOmBwDCRlACgBgxPQAQNoISkIDd99gjctl93/OeSOXWtbRE3ub/PvVU5LJJefKNOyKX3XW3mZHKnfD4TZG3+fG/+3zksiUx9AYEj6AEALFxiaAEBI2gBAAxcZeyTA8ABI2gBABxokcJCBpBCQDiwjlKQPAISgAQG26KC4SOoAQAcaFHCQgeQQkAYuIiKAGhIygBQIwISkDYCEoAEBd3OdMDAEEjKAFAjOhRAsJGUAISsH8vbmEyduTISOXaOjqqrU4qtW3cFrns5h1XRCrXMKapytpUycU8SkDgCEoAEBOXMzM3EDiCEgDEhR4lIHgEJQCIEecoAWEjKAFAXFzyDobegJARlAAgNk6PEhA4ghIAxIVbmADBIygBQEycoTcgeAQlAIgRPUpA2AhKABAXd3mGoASEjKAEJCDr0RvPpmHDIpXbY5ddqq1OKm15oyVy2YbRyyKV27RyfZW1qR49SkDYCEoAEBfOUQKCR1ACgJi46FECQkdQAoC4uMu51xsQNIISAMSFoTcgeAQlAIgRQ29A2AhKABAXpgcAgkdQAoCYMDM3ED6CEgDEiB4lIGwEJQCIC1e9AcEjKAFAXFzydoISEDKCEpCAoUOiH3qNEW9h0tCLbV502mnR9j10aKRyE8aOjbzvhjFNkcpl2joib3PLmmi3O2l9c0vkbdYKV70BYSMoAUBMXFKmF/f1A5A+BCUAiIu7slmG3oCQEZQAICYuKUNQAoJGUAKAGGUZegOCRlACgJg4Q29A8AhKABAjht6AsBGUACAmXPUGhI+gBABxYegNCB5BCQBiwlVvQPgISkACFtx7b+SyX5k1K1K5HRobI28z6ozbIyJuc0hD9D8l9U3Ryrav3xZ5m9vq6yKV+9Qp/yfyNmvB3QlKQOAISgAQI6YHAMJGUAKAmLikjkwm6WoA6AOCEgDExZ0eJSBwBCUAiAkncwPhIygBQIyYHgAIG0EJAGLi7uogKAFBIygBQEwYegPCR1ACgLgwMzcQPIISAMSE6QGA8BGUACAmzMwNhI+gBKTcd266KekqoA8ISkDYCEoAEBN3VztDb0DQCEoAEBOuegPCR1ACgLhwjhIQPIISAMTEJYbegMARlAAgJlz1BoSPoAQAMXFJGXqUgKARlAAgLtzrDQgeQQkAYuKS2js6kq4GgD4gKAFATNydW5gAgSMoATXUJjW/Lr2WdD3Qb95d7kmXGHoDAkdQAmrI3cclXQekh7urtb096WoA6AOCEgDEhKE3IHwEJQCICfd6A8JHUAKAmGTdta2tLelqAOiDuqQrAAADlburraOjaOkrM7vCzF4ws2fN7FdmNrrguQvM7GUze9HMpvd5ZwAISgAQF3dXW3t70VIDiyRNcff9JP1Z0gWSZGZ7SzpJ0j6SZki61szqa7FDYDAjKAFATLLu2rJtW9HSV+5+n7t3dk09IWli/udjJd3u7q3u/qqklyUd0ucdAoMc5ygBQEyymYy2tLTEuYvZkn6W/3mCcsGp06r8OgB9QFACgJhsy2bvfaGlZeduqxvNbEnB43nuPq+wgJndL2nXHjZ5obv/Jl/mQkkdkm7pfFkP5b26mgPoRFACgJi4+4wqX3dkuefNbJakYyRNc/fOMLRK0qSCYhMlvVHN/gG8g3OUACAgZjZD0lclfdLdtxQ8dYekk8yswcwmS3qfpMVJ1BEYSOhRAoCw/EBSg6RFZiZJT7j7We7+vJn9XNJy5YbkznF3ZrsE+sje6bUFAABAIYbeAAAASiAoAQAAlEBQAgAAKIGgBAAAUAJBCQAAoASCEgAAQAkEJQAAgBIISgAAACUQlAAAAEogKAEAAJRAUAIAACiBoAQAAFACQQkAAKAEghIAAEAJBCUAAIASCEoAAAAlEJQAAABKICgBAACUQFACAAAogaAEAABQAkEJAACghEETlMzsMjOb04fXrzCzIyOW/YCZ/dHMWszs30qUWWxm+1RbHyANoh5XfN4Ror62GyW2+byZfSxi2cjtTr48x1kMBkVQMrNxkv5F0o/7aZdfkfSQu4909/8q8WH/rqRv9VN9gJrr5XHF5x1BiavdcPd93P2hWmyrh7aF4ywGgyIoSfq8pLvdfWs/7e/dkp6vUOYOSX9vZrv1Q32AOHxe0Y8rPu8IzedVw3bDzIbUYjsVcJzFYMAEJTOrM7MLzOx1M3vTzM42s/b8t4KjJT3crfweZnanmTWb2QYzW9TL/b3LzH6Z39ernUNsZvaApL+X9AMz22Rmt0naXdLC/OOvSJK7b5O0VNJRfX/34TCz681srZktK1g31swWmdlL+X/HFDx3gZm9bGYvmtn0ZGo9OJnZKDNzM9u5YN3eZrbGzHZUt+Oq3DE1WD/v1eI46R9xtxv5Hp+vmtmzkjab2ZDCXiAzO6DgNI1fmNnPzOw/um3mw2b2bH5/PzOzxvxr/0fd2pbBdJz15zEyYIKSpK9LOkbSVEl7Ktdl+pa7vylpX0kvdit/s6R7JO2SX74RdUdmVidpoaRnJE2QNE3SHDOb7u7/IOlRSf/q7iPc/TOSXpc0M//4OwWb+pOkD/X2jQbuRkkzuq07X9Lv3P19kn6Xfywz21vSSZL2yb/mWjOr77+qDm7uvkHSKkl7F6z+tqT/dPeN2v64qnRMDcbPe7VuFMdJf+iPduMzkj4habS7d3SuNLNhkn6l3P/1WEm3SfpUD68/Qbn/18mS9lOup0vufop6blsGy3F2o/rpGBkQQSmf/udKOtndX8v/gb9L0nP5IqMltXR72Xsl1Uuqd/dt7v54L3Z5sKRx7v4td29z979I+oly/xG90ZKv26Dh7o9IWtdt9bGSbsr/fJOkfypYf7u7t7r7q5JelnRIf9QTXZZJ2kuSzOwQSQdI+mH+udEqPq4qHVOD7vNeLY6T+PVju/Ff7r6yhyG8j0oakn++3d0XSFpc4vVvuPs65b6gf7jC/gbFcdafx8iACErK9ej8yd1XFKzbSe984N+WNLLba05W7pf3hpldZ2Zje7G/d0t6l5mt71wk/bty3zB6Y6Sk9b18zUC0i7uvlqT8v+Pz6ydIWllQblV+HfrPMr3To3SZpG+4e2v+cffjqtIxxee9bzhOaqu/2o2VJda/S9Jf3d0rlP1bwc9bJI2osL/BfJzFcoz0x8ll/WFnSW92PrDcSXMzJXWO9T4r6f2Snuos4+4PSHrAzMZLulu57syrIu5vpaRX8917UXiJ9XtJ+n8Rt5EqM2bM8Obm5u3WL1269HlJ2wpWzXP3eVXuxnpYV+p3iXgsk/S5/DkVuyk39NCp6LiKcEwF+3mv1owZk7y5eVvRuqVLm2t5jEgcJ9Xqr3aj1P/FakkTzMwKwtIkSa/04j30tO2gjrMQ2pKBEpRekPQtM5usXJL+jqQ9lPsjL+U+0EdIukWSzOw45b41vKxc+h4j6en8czdKkrt/vsz+FkvaaGZflfRfktqU+3A2uftTPZRfk69PFzNrkHSgpFm9eJ+p0dzcrCVLtn+rZnXb3P2gXm5ujZnt5u6rLXe1xtr8+lXK/eHoNFHSG1VVGNXq7FG6VNK/u3um4Lmu46rcMSWF/3mvVnPzNj215LiidXU2r5pjROI4qbX+bje6+19JGUn/amb/rdx5TIdIeqgX2yhqW0I8zkJoSwbE0Ju73y/pF8qdXP2kch/mrN65RP9mSf9oZk35x3+n3NUMLcodDJfnvylIuV9m2XHnfGMxU7mx4lclNUv6qaRRJV5ymaSv5Yfp/m9+3SeVm2sp2D9o7tntlirdoXcO7FmSflOw/iQza8j/MXufeh7DR3yWS9pVUsbdf93tucLjqtwxJQ2Az3u1sl689AHHSQ31d7vRw/7bJB0n6VTlgtrnJN0pqbXMy7rr3rYEeZylvi1x9wG3KHdp5Evd1l0qaU6F1w1T7oqBof1QxyclTUn6d1XtcuCBB3gm07bdImlJhfd9m3Jdzu3KpfxTlTsv4HeSXsr/O7ag/IXKdUW/KOnopN83y3b/nxWPq3y5oD/v1S4HHLizt2XOKFoqHSP53xfHST8vaWg38sfJF/r4+qCOsxDaEstvYEAxsy9LOsLdj6tYGFU56KAD/cknt/8CNWRI01KvblgBGHAOPHCcP/5k8Z+hpqHzOEZSKIl2w8yOUK7hblbuRPEfSdrD8yckDwYhtCUD5Ryl7vbSO+PMiIUr10sNoBRXn4fb0H+SaDc+IOnnyl3J9oqk4wdTSMpJf1syIIOSu5+VdB0GOvfcuDKA8ghKYUii3fDcVVx9ueIxeCG0JQMyKKE/uIovgALQnUvKEJSAMtLflhCUULW0fwsA0oAeJaC8tLclvQpK9WZOsho8OiRl3HuaqEv578r9WZ0g2PBG1+juk/liwFrfIt+yrcQxkhtWIChtj7ZkcGmTmt19XM/Ppr8t6dVndYhyE6pgcPhb2WfT312aiNEjpTO42HLQmLegYpFMur8sJ4K2ZHB5XXqt9LPpb0sI9aha2rtLgaRx1RtQWdrbEoISqhLClQpAGhCUgNJCaEsISqiSy70j6UoAqebOVW9AeelvSwhKqJKn/lsAkAb0KAHlpL8tISihD9L94QaSxjlKQBTpbksISqha2r8FAGlAUALKS3tbQlBCldI/rgwkzZ3pAYDy0t+WEJRQFff0jysDaUCPElBaCG0JQSkApx9/fKRyP5k/P+aadJfuScIGikmjopVbuSHeeqTZ3iXm/O1u+Zvx1qMnBCWkQXrbESntbQlBCVVK/2yqQNK4KS5QSfrbEoISqpb2DzeQOO71BlSU9raEoIQqpX9cGUga0wMAlaS/LSEooSq5E/DSfaUCkAZc9QaUFkJbQlBC1dLeXQokjR4loLK0tyUEJVTJlfbZVIHEcY4SUEH62xKCEqqW9u5SIGlc9QZUlva2hKCEKqX/kk4gDehRAspJf1tCUEKV0v/hBpLGOUpAJelvSwhKqIp7+m9kCKQBQQkoLYS2JJigdNz06ZHLrl63LlK5HYcPj1TuXWPHRt531G0esvf7I2/zoT8+F6ncM28vjbzND405MHLZUjyb7g93mr1ndPSyjRGP0t0j3uqkN9uMemuQ/XeLvu+D3xWt3NHvi77NfX4YrdyzX4y+zf3+O3rZUrgpbt+ccPTRkcuu2RDtHj7DGxoilRs/KvoBdciee0YqN+nQ90QqN+6gCZH3/a7d/ylSuf/49Kcjb7O/pb0tCSYoIW1czldloCIOE6Cc9LclBCVUh5MvgEg4TIAyAmhLCEqoWtq/BQBJY3oAoLK0tyUEJVTH0//hBhLHhJNAeQG0JQQlVMWV/g83kLQARhWARIXQlhCUUB13OZfzABVxmABlBNCWEJRQNfd0fwsAkkaPElBZ2tsSghKqRwsAlMc5SkBlKT9ICEqojktZLucByuKqN6CCANqSxIPS8TNmRCo3sqkp8jYX3HtvtdVJpd+/uiBSuaFDR8dbkSJ8Ve6LqDNjS9Kw+mjlevO/8WJztHJ/fitauV+/0IudR/TqnOhlnz8nWrkV66upSd9wmFQv6p0OJGnDli2Ryt3z4IPVVqekm2q+xeiWb3olUrlZl50UqdxP5s/vS3WqkP62JPGghDB5AJd0AknjHCWgvBDaEoISqpb2DzeQBhwmQHlpb0sISqiOK/WXdAJJ46a4QAUBtCUEJVQp/TcyBNKAwwQoJ/1tCUEJVUv7hxtIGucoAZWlvS0hKKEq7pKn/JJOIA04TIDSQmhLCEqoWtq/BQBJ8/Rf+QwkLu1tCUEJ1aEFACLhMAHKCKAtISihatmUX6kAJM3FVW9AJWlvSxIPSq3t7ZHKjR81KuaapNfmv26MVM7qn4u5JgU4S7VPos62LUlDI5YdMay6uqTVq29HL1tn0cq9EHFG8lriMKlew9Chkcu+a+zYGGuSXps2LY9UruXVdTHXpEoBtCWJByWEK+3jykDi0j+qACQu7W0JQQlVcffUTxIGJI2hN6C8ENoSghKqlvZvAUAapLsJAJKX9raEoITqBDD3BZC0AE6/AJIVQFtCUELV0v4tAEgDDhOgvLS3JQQlVCeAcWUgadwUF6gggLaEoISquNLfXQqkQcq/LAOJCqEtISihOp7+7lIgaZyjBFQQQFtCUEJ13OUd6e4uBdIg5V+WgWQF0JYQlFC1tH8LAJIWwG2sgMSlvS1JPCiNGTEiUrnhDQ2Rt/nvs2dXW50emUW8P4KkCRGn0d9pwpjI22zf2BqpXOu6LZG32WcueTbd3wLSLOptSSRpaF3ttzljz2jlmiLeQWJsU/R97z0uWrnXNkTfZlTN/XiIdEp5G5BqvbmFyY5N0T6E/+dzn4tUriOTibzvSTvvHKncxN2ilRtz0G6R971pVbQDpb2lLfI2+1UAbUniQQlhckneQQsAlMPM3EB5IbQlBCVUJ4BLOoE0oEcJKCOAtoSghOrRAgDlcY4SUFnKDxKCEqrjUjblVyoASWPoDagggLaEoITquKd+kjAgDVL+ZRlIVgBtCUEJ1Uv5hxtImktK93dlIAVS3pYQlFAVd6X+BDwgDehRAkoLoS0hKKE67qkfVwaSxk1xgQoCaEsISqha2seVgTSgRwkoL+1tSeJBqbW9PVK5ul7Mjt0YcTbXpmHDIpWLOuOrJO3Q1BitYPS3o/aWaDNz/+3B5ugb7StX6seV06y+F///jRGP0oZeHM07RPvoa0TEcsOjT6Cs9duilevNZL1DIs5evvzN6NusBW6K2zdD6iL+xyr63RvGRrwbRK/+7g+PVnbIjtHq2LZua+R9R52sccvrMUx1XwsBtCWJByWEySV1pHzaeSANUt4GAIkKoS0hKKE67sqm/MMNJI2b4gIVBNCWEJRQldyQAi0AUAlBCSgthLaEoISqZVL+LQBIGjNzA5WlvS0hKKEqHkB3KZAG9CgBpYXQlhCUULW0d5cCSeMcJaCytLclBCVUJYQrFYA04Ko3oLQQ2hKCEqoTQHcpkDRX7+aDAgadANoSghKqEsKVCkAaMPQGlBZCW5J4UPrZ3XdHKnfC0UdH3uauY8ZEKrd+8+ZI5XYZNSryvqPO+tq4IeKUx5LaXnojUrmv/uAHkbdZC2n/FpBmi/8avewhE6KV68Xk9drcFq1c1Ea+F7vuVdmo6qMdyrrs0Rh2XgFHSfW+c9NNkcuePHNmpHK7RWwf3ty4MfK+R0acxbvxrWhT2O/0xsjI+476d/i0yy+PvM3+lva2JPGghDC5e+rHlYGkcVNcoLwQ2hKCEqqW9m8BQBow9AaUl/a2hKCEqrh76icJA5LGTXGB8kJoSwhKqEoIl3QCiXOmBwDKCaEtISihamnvLgWSRo8SUFna2xKCEqri7sqk/JJOIA0ISkBpIbQlBCVUxSV1ZDJJVwNINW6KC5QXQltCUEJ1AphNFUgc93oDygugLSEooSq5b8rp/nADSeMcJaC8ENoSghKq4u6p7y4F0oCr3oDSQmhLgglKP7/nnqSrgG7S/i1goOjN7U6QLvQo9Z9bFi5MugqoUtrbkmCCEtIlhO5SIA0ISkBpIbQlBCVUJ4DZVIHEucRhApQRQFtCUEJV3F3tKR9XBpLm4hwloJwQ2hKCEqoSQncpkAYMvQGlhdCWEJRQlRA+3EDSOJkbKC+EtoSghKq4u9o7OpKuBpBu3BQXKCuEtoSghOq4p/6Oz0DS6FECKgigLSEooSoh3J8HSAOCElBaCG0JQQlVCaG7FEgaN8UFyguhLelVUGqTml+XXourMkidd5d6wqXUd5cmYnVzs745j2Nk8Ch5jEjiprgl0JYMOkG3Jb0KSu4+Lq6KICwhzH2RBI4RFOIcpZ5xnKBTCG0JQ2+oirurrb096WoAqcdVb0BpIbQlBCVUJYRvAUDS6FECyguhLSEooSoupf4EPCANCEpAaSG0JXVJVwBhyrqrraNju6UvzOwKM3vBzJ41s1+Z2eiC5y4ws5fN7EUzm97X+gP9In8yd+HSVxwnGEhCaEsISqiKZ7Pa2tq63dJHiyRNcff9JP1Z0gWSZGZ7SzpJ0j6SZki61szq+7ozIG6d0wMULjXAcYIBI4S2hKCEqmTd1drevt3SF+5+n7t3fpV4QtLE/M/HSrrd3Vvd/VVJL0s6pE87A/pJrXuUOE4wkITQlnCOEqri2ay2bdkS5y5mS/pZ/ucJyn3YO63KrwNSrR9O5uY4QdBCaEsISqjK1kzm3ufeemvnHp5qNLMlBY/nufu8zgdmdr+kXXt43YXu/pt8mQsldUi6pfNlPZTnFFmk3xvN92Yuntf9OCl7jEgcJxg8QmhLCEqoirvPqPJ1R5Z73sxmSTpG0jR37/wAr5I0qaDYRElvVLN/oD9xnADlhXCMcI4SUsPMZkj6qqRPunthX+wdkk4yswYzmyzpfZIWJ1FHIGkcJ0B5tT5G6FFCmvxAUoOkRWYmSU+4+1nu/ryZ/VzScuW6Uc9x93TPUAbEh+MEKK+mx4i90yMFAACAQgy9AQAAlEBQAgAAKIGgBAAAUAJBCQAAoASCEgAAQAkEJQAAgBIISgAAACUQlAAAAEogKAEAAJRAUAIAACiBoAQAAFACQQkAAKAEghIAAEAJBCUAAIASCEoAAAAlEJQAAABKICgBAACUQFACAAAogaAEAABQAkEJAACgBIISAABACQQlAACAEghKwCBjZpeZ2Zwabu9GM/uPWm0v4j4Xm9k+/blPAIMTQQkYRMxsnKR/kfTjpOvSR9+V9K2kKwFg4CMoAYPL5yXd7e5bk65IH90h6e/NbLekKwJgYCMoAQOMmdWZ2QVm9rqZvWlmZ5tZe7436WhJD3crv4eZ3WlmzWa2wcwWVdj+/mb2BzNrMbOfSWrs9vxeZvaQma03s+fN7JP59V8ws4UF5V42s58XPF5pZh/O/7zCzP6vmT2br9PPzKxrP+6+TdJSSUdV+3tKMzO73szWmtmygnVjzWyRmb2U/3dMwXMX5H+fL5rZ9GRqDQxMBCVg4Pm6pGMkTZW0p3JDbW+5+5uS9pX0YrfyN0u6R9Iu+eUbpTZsZsMk/VrS/0gaK+kXkv654PmhkhZKuk/SeElfknSLmX1AuYA2NR/kdpM0VNL/l3/dHpJGSHq2YHcnSJohabKk/ZTrDSv0J0kfKv+rCNaNyr33QudL+p27v0/S7/KPZWZ7SzpJ0j7511xrZvX9V1VgYCMoAQNIvtdorqST3f01d98g6S5Jz+WLjJbU0u1l75VUL6ne3be5++NldvFR5QLONe7e7u7zJT3V7fkRki539zZ3f0DSnZI+4+5/ye/7w5KOkHSvpL+a2Qfzjx9192zBtv7L3d9w93XKha8Pd6tLS/79DDju/oikdd1WHyvppvzPN0n6p4L1t7t7q7u/KullSYf0Rz2BwYCgBAws0yT9yd1XFKzbSe8Epbcljez2mpOVa2zfMLPrzGxsme2/S9Jf3d0L1r3W7fmV3QLPa5Im5H9+WNLHJB2e//kh5ULSEeo2JCjpbwU/b1EugBUaKWl9mboONLu4+2pJyv87Pr9+gqSVBeVW6Z3fN4A+GpJ0BQDU1M6S3ux8YGZDJM2U1Hn5/rOS3q+CXqB8r88DZjZe0t3KDXFdVWL7qyVNMDMrCEu7S3ol//MbkiaZWV1BWNpd0p/zPz+cr89kSZcqF3ROlnSopB/08r3uJen/9fI1/WrGjBne3NxctG7p0qXPS9pWsGqeu8/rw26sh3XewzoAVSAoAQPLC5K+ZWaTlQsh35G0h6TOk4LvVq735hZJMrPjlOttelm5Hpoxkp7OP3ejJLn75wu2/7+SOiT9m5n9UNInlRvmeTD//JOSNkv6ipldqdw5SDMlHZx//mHlQtgad19lZhuVO99piKQ/Rn2TZtYg6UBJs6K+JgnNzc166qkni9bV1Q3Z5u4HVbG5NWa2m7uvzp/jtTa/fpWkSQXlJioXWAHUAENvwADi7vcrd4L1M8qFluckZSU9ny9ys6R/NLOm/OO/Uy68tCgXoi7P9zBJuca36Hwld2+TdJxyvU5vSzpR0oJuz39SuavrmiVdK+lf3P2F/PN/lrRJ0qP5xxsl/UXS4+6e6cVb/aSkh9w95YHA5Z4tWvrgDr0TDGdJ+k3B+pPMrCEfkN8naXFfdgTgHVZ8qgGAgcTMjpL0w/yVUp3rLpW01t2vKfO6YcqFrf3cvT32ivaSmT0p6VR3X1axcIIOOuhAX7z490Xr6usbl1bqUTKz25Q7l2tnSWskXazc1YY/V24o83VJn86f6C4zu1DSbOV6++a4+z01fSPAIEZQAgYwM/uypCPc/bik6zIYHXTQAf7kk48VrRsyZIeKQQlAenCOEjCw7aV3zk9CP3NXX4fbACSMoAQMYO5+VtJ1GNycoAQEjqAEALEiKAEhIygBQGxc7h1JVwJAHxCUgBqqN3MOqsGjQ1LGvacJH7sw9AaEjb/pQA0NkbRr0pVAv/lbxRKcowSEjqAEADHJXfXWm3k0AaQNQQkAYkOPEhA6ghIAxIoeJSBkBCUAiI0z9AYEjqAEADFi6A0IG0EJAGJDjxIQOoISAMTEnaAEhI6gBKTcPx11VKRyv77vvphrkl6nH398pHI/mT8/5pr0hKAEhIygBACxYXoAIHQEJQCIEUNvQNgISgAQG85RAkJHUAKAmHALEyB8BCUAiI1L4hwlIGQEJQCIjcu9I+lKAOgDghIAxIihNyBsBCUAiA0ncwOhIygBQGwYegNCR1ACgJjkrnrjZG4gZAQlIAEf2m+/yGXHjhgRqdyUffeNvM1lzz0XuWxSfnXzVZHL3n3H7yOVu/a88yJv8+wrrohctiz32mwHQCIISgAQG1c2Q48SEDKCEgDExSVl6VECQkZQAoAYOUEJCBpBCQBi5JyjBASNoAQAMXGXPENQAkJGUAKA2DhDb0DgCEoAEBdO5gaCR1ACgBh5lukBgJARlAAgLs5Vb0DoCEpAAhqGDYtctili2V1Gj468zWWRSyancddoM5JL0qlXnxKp3NvL10SvQE0m5uYcJSB0BCUAiAlXvQHhIygBQIzoUQLCRlACgBgRlICwEZQAIC7ucm6KCwSNoAQAMaJHCQgbQQkA4sKEk0DwCEoAEBMXPUpA6AhKABAXd6YHAAJHUAKAGHELEyBsBCUAiAu3MAGCR1ACEjB5/PjIZRuGDo1U7qPvf3/kbf7u0Ucjl02KmUUuu3VtS6RybW9vrbY6VWPoDQgbQQkAYuLOvd6A0BGUACBGnKMEhI2gBABxcck76FECQkZQAoAYMfQGhI2gBABx4RwlIHgEJQCIiUvyDs5RAkJGUAKAuDCPEhA8ghIAxMYlrnoDgkZQAoC4OBNOAqEjKAEJ2NLaGrlsfcQZqhuGRD+cv/tv/xap3PCGhkjldtxheOR9D9u5KVK51re2RN6me7QwsvH55sjbrAmXshl6lICQEZQAIE70KAFBIygBQEzcnaE3IHAEJQCIkTP0BgSNoAQAceFkbiB4BCUAiAtBCQgeQQkAYuPMzA0EjqAEADFx5xwlIHQEJQCIC0NvQPAISgAQG1eWoTcgaAQlAIiLiwkngcARlIAELPzd76IXnjYtUrE9d9st8iabW1oildtl1KhI5d7etCnyvsdsHhGp3OhNIyNvs7V5a6Ryp1xySeRt1oJLynJTXCBoBCUAiIu7MhHvQwcgnQhKABATl5ShRwkIGkEJAGLE0BsQNoISAMTE3ZVl6A0IGkEJAGLE0BsQNoISAMSEq96A8BGUACAuXPUGBI+gBAAx4ao3IHwEJQCIEUNvQNgISkDK9WoWb6SKM/QGBI+gBAAxymQySVcBQB8QlAAgJu7OOUpA4AhKABATl5hwEggcQQkAYkSPEhA2ghIAxMTd1UFQAoJGUAKAGDE9ABA2ghIAxIQJJ4HwEZQAICburg6mBwCCRlACgBjRowSEjaAEADFhHiUgfAQlAIiJSwy9AYEjKAFAXOhRAoJHUAKAmHDVGxA+ghIAxMTd1c7QGxA0ghIAxIQeJSB8BCUAiAu3MAGCR1ACgJi4pAxDb0DQCEoAEBN3V1tHR9LVANAHBCUAiAkTTgLhIygBQEyYcBIIH0EJqKE2qfl16bWk64F+8+5yT7q7Whl6A4JGUAJqyN3HJV0HpIe706MEBI6gBAAxcUnt9CgBQSMoAUBM3F2t7e1JVwNAHxCUACAmWaYHAIJXl3QFAGCgcne1d3QULX1lZleY2Qtm9qyZ/crMRhc8d4GZvWxmL5rZ9D7vDABBCQDiks1mtXnbtqKlBhZJmuLu+0n6s6QLJMnM9pZ0kqR9JM2QdK2Z1ddih8BgRlACgJh4NqttW7YULX3epvt97t7ZNfWEpIn5n4+VdLu7t7r7q5JelnRIn3cIDHKcowQAMclms9rS0hLnLmZL+ln+5wnKBadOq/LrAPQBQQkAYtLqfu8rbW07d1vdaGZLCh7Pc/d5hQXM7H5Ju/awyQvd/Tf5MhdK6pB0S+fLeijv1dUcQCeCEgDExN1nVPm6I8s9b2azJB0jaZq7d4ahVZImFRSbKOmNavYP4B2cowQAATGzGZK+KumT7l540tMdkk4yswYzmyzpfZIWJ1FHYCChRwkAwvIDSQ2SFpmZJD3h7me5+/Nm9nNJy5UbkjvH3bl/CtBH9k6vLQAAAAox9AYAAFACQQkAAKAEghIAAEAJBCUAAIASCEoAAAAlEJQAAABKICgBAACUQFACAAAogaAEAABQAkEJAACgBIISAABACQQlAACAEghKAAAAJRCUAAAASiAoAQAAlEBQAgAAKIGgBAAAUAJBCQAAoASCEgAAQAkEJQAAgBIISgAAACUQlAAAAEogKAEAAJRAUAIAACiBoAQAKWNm15vZWjNblnRdgMGOoAQA6XOjpBlJVwIAQQkAUsfdH5G0Lul6ACAoAQAAlDQk6QoAwEA1Y8YMb25uLlq3dOnS5yVtK1g1z93n9WvFAERGUAKAmDQ3N+uppxYXraurq9/m7gclVCUAvURQAoDYuKRs0pUA0AecowQAMXLPFC1RmNltkv5X0gfMbJWZnRprJQGURI8SAMTG5d77HiV3/0wMlQFQBYISAMTEXVUFJQDpQVACgNh45OE2AOlEUAKAWNGjBISMoAQAsanuHCUA6UFQAmqo3sw5qAaPDkkZdytXxr2jn2oDIA78TQdqaIikXZOuBPrN3yqWoEcJCB1BCQBiRVACQkZQAoCYuLuyWa56A0JGUAKAWBGUgJARlAAgNpyjBISOoAQAMSIoAWEjKAFAbJzpAYDAEZQAIFb0KAEhIygBKfeae9HjN998U21tbZowYULR+ndb2XkPB7Tuv6NS+vt35M45SkDoCEpAYD7+8Y/rjTfe0Nq1a5OuCipi6A0IHUEJCMyKFSu0YcOGpKuBiNyZHgAIWV3SFQCAWpo/f74aGxu1evXqpKuizukBChcAYSEoAYFoa2vTkiVLunqTlixZoq1btyZcq3TJZrM677zz1NHRoUsuuSTp6sg9d1PcwgVAWAhKQCAWLFigj370o6qryx22hx56qG644YaEa5UuCxYsUHNzszKZjG644YYU9Cq5cle9FS4AQkJQAgIxY8YMNTY2KpvNNbZDhgzRcccdl3Ct0qOzN2nTpk2SpEwmk5JepUzRAiAsBCUgEKNHj9bcuXPV1NSkYcOGafbs2dp1112TrlZqtLa2qqmpSSNHjpQkjRkzJgUnvTtDb0DgCEpAQObOnau6ujrV1dXpoosuSro6qdLU1KTly5frl7/8pSRpzZo1uuWWWxKuldOjBASO6QGAgIwePVpXXXWVNmzYQG9SILjSDQgbQQkIzOmnn550FRCZS6IXCQgZQQlIQNRbbvTknHPO0dq1azV//vyabTN0hbcmae9hXafe/I5qcbsTd8mz9CgBISMoAYG59dZbU3CScnoNlbR70pUo4NnBG2CBgYCgBABxcScoAYHjqjcgBc466yx96Utf0po1a2q2zXXr1umrX/2qTj755Jrs+7e//a2mTZumJ554omy55uZmHX744bruuuvU3t5etmyt990bmUxGt912m6ZOnaqVK1fWbLvdeSZbtAAIC0EJSIG77rpLP/7xjzV58uSSweHBBx/U+PHju4bdxo0bpwULFmxXrjMgTZo0Sd/73vd011139XnfkvTEE0/ogQce0LRp0zR16tSSoWXNmjVavHix5syZo4kTJ5YNTLXedxSdAWny5Mk644wztGTJEr322mtVb68Sdy9aAITFOHCB2mkw8ygX7b89cqRaWlp6fK6+vl719fXatGmThg4d2rV+2bJlOuCAA7pCx7Bhw/Tggw/qsMMOK3r9mDFjtGnTJnV09H5yw8J9X3nllbrgggvKlv/lL3+pffbZRx/84AdLlqmrq9PMmTP161//WjvuuGOk9x11350zk0c98fo1d82ePVs33XRT1wznPXnqqaf0zwcfXHF7f5PU6l5y5x/ad0+/+zffKVo38b3/vNTdD4pUYQCJ4xwlIAGPPPKIXn755a7Hs2fPVktLi0aMGKERI0bosssu05AhxYfnlClTdNRRR+mee+5RNpvVQQcdtF1IkqQf//jHOu+887Ru3Tpt2rRJQ4YM0W233db1/LZt29TY2Fhx36eddpr23HPPrnI333yzFi5cqIaGBtXV1enMM8/Uxz72MY0ZM0YLFy7Utm3bJEkrVqzQxRdfrK1bt6qpqUkHH3ywvv71r/fqfUfddzXOPfdcrV69Wg8//LBaW1s1fPhwzZ07V/vuu6+k3K1h9t9//6q23Z3Llc3wZRQIWvduYRYWluqXYZLvHmHpbsqUKb7rrrv6DTfc4O3t7ds93+m5557zpqYmHz58uD/++OMly3V0dPjtt9/ukyZN8kmTJpUs15t9X3nllT506FCfM2eOr127tmS5V155xevr6/2II47wxYsX98u+o/zOu//en3nmGZ8xY4bX1dX5H/7wh6q2OSy3zZKfh3332cNff+EXRYukJeVew8LCkq6FoTeghqIOvXWfz2fz5s1qaGjYrhepJzNnztTbb7+txx57rGLZTCajLVu2dN3/rCdR9+3u2rhxo0aNGlVxvxs2bIhUrlb77s3QW9S6RtlmpaG3/aa81++cf3nxdvc6gaE3ICAMvQEpsMMOO0Quu3Dhwshl6+vry4ak3uzbzCKFH0mRy8Wx796Ka7uSJBdXugGBIygBgbnnnnu0adMmffrTn656G7WYdTpuvZlFO72zkjvzKAGBIygBgTnllFP01ltviWHz9HMxMzcQOuZRAgJTzWX/g8nGjRt15513Jl2NHM8FpcIFQFgISkAKLFq0SE8++WTNt/v000/36pymtFq5cqVuueWWijN9S9IVV1yhmTNn6ve//33FsuvWrdP111+vrVu31qKaPfKMFy0AwkJQAlJg9uzZmjp1qg4//PCSgWnVqlU699xzu2bmPvfcc/XSSy/1WPbpp5/W9OnT9ZGPfESnnHJKbPXuL9ddd51mzZqlSZMm6frrry8ZmNavX6+rr75aknTeeeeV3N66det0/vnna9KkSTrzzDO1dOnSWOotl5T14gVAUAhKQAK++MUvysy6llWrVqm9vV2PPfaY/uEf/kGHH364MplM0WuWLl2qa665puvx9773PT3++OPbbfsTn/iEDjvsMN1///1qa2vThg0bivZ14oknKu3N9VblZh7vrPM3v/lNZTIZrVmzRl/+8pc1adKkHsPNlVde2TXj9tNPP91jr9KPfvSjrtu7bNmyRe6uqVOndu1r991316pVq2r0TpyhNyBwBCUgAVbhqrOenp85c6be/e53dz0eO3asPvvZz25Xrq6u/GFd6fm0KFfPzlBTaPPmzfrud7/bNYy2ZcuWHm+DUul3X1dXV7FMZM5NcYHQhfEXExhgrr322qKZXydOnKihQ4dq6tSpevDBB/Xwww+rvr6+6DV1dXW64oorNGLECO2www769re/rWHDhm237YULF+qJJ57QUUcdpWHDhmnUqFFF+7rtttuU9skBmpS71UpnnS+++GLV1dVp11131fe//32tXLlSBxxwQNFrhg8frq9//es69thjJeWGJr/0pS9tt+0zzzxTK1eu1Jw5czR8+HBJ0qOPPtq1rxUrVmjChAk1eR+dV73RowSEi5m5gRqqdmbu+++/XzvuuKMOOeSQsq/LZrPac889tWnTJq1atarHoFTo2Wef1euvv65jjjmmaH1o8yitWrVKjzzyiE444YSKs3gvWrRIRx11VKTpE9atW6ff/OY3Oumkk9TU1FT0XC1m5p7ygff4L+ddXLTugx+bzczcQECYRwlIgSOPPDJSubq6Ot11111qa2urGJIkab/99tN+++3X1+olbuLEiT0OM/bV2LFj9YUvfKHm2y3EcBsQNobegMDsvPPO2nnnnSuWc3c9++yz/VAjlMQ8SkDwCEpAYD7+8Y9r//33r1juoYce0oc+9KF+qFG6HHDAAT2exJ0UghIQNobegMCsWLGiay6lcgbrDN477bSTLr300qSrISnXq8ckk0DY6FECUmDTpk2RZp0erNxd69evj1R2/vz5amxs1OrVqyOVj7rdank2W7QACAtBCUiBQw89tOKs021tbVqyZElXb9KSJUtK3nrjhRde0AsvvNBV7o033oin4v3k6quv1vjx4/WlL31Ja9asKVkum83qvPPOU0dHhy655JKy2+ycvXynnXbSH//4x1pXuQtDb0DYmB4AqKGo0wP8aunSotuPnHbaadq0aVPXHEmXXnqpvvCFLxRNfHj77bfrc5/7nNxd2WxWQ4YM0fe+9z2dffbZRdvevHmzxowZo2HDhmnz5s1qbGzUfvvtV3RrlLRPD5CRdOXtt3c9/p//+R/dddddamhoUF1dnU499VR94xvf0E477VT0uvnz5+sLX/iCNm3apMbGRv3lL3/RbrvtVlRm2bJlOvfcc/Xoo4+qtbVVw4cP15w5czRlyhRJ0tChQ3XsscdqjwrTEEiVpwfYZ893++3f/WrRuv0+dQ7TAwABISgBNRQ1KL09cqRaWlp6fG7IkCGqr69XS0uLhg4d2rV+/fr1mjhxojZv3ixJamxs1Kuvvqpdd91+jzNnztTdd9+tbDarHXbYQddcc41OO+20rufTHpQ25JdyFixYoE996lNdj7PZrN773vdqxYoVknKB57TTTtO1115b9LpTTz1VN954Y9etTnry1FNP6Z8PPrhiPSsGpffu7rd+pzgoffj4fyUoAQFh6A1IwMaNG3ucmbupqUnnnHOOXn/99aKQJEmjR4/W3Llz1dTUpGHDhmn27Nk9hiRJuuyyy9TQ0CApN2P1rFmzYn9PtTRKKvr9XHxxbtLG4cOH64gjjtDixYuLQpIktba2qqmpSSNHjpQkjRkzpseT3ufNm6dbb71Vu+++u0aMGKHGxsaimbndXQcdVMMck80WLwCCQlACUmDmzJk6++yztWLFCl1zzTUaP358j+Xmzp2ruro61dXV6aKLLiq5vSlTpmjatGmqq6vTpZdeul3oCs2hhx6qI488Ug899JAeeughHdxDb09TU5OWL1+uX/7yl5KkNWvW6JZbbtmuXH19vU488UT95S9/0U9/+lMdcsghRffQqyV3yTNetAAIC9MDACnQfXiolNGjR+uqq67Shg0bSvYmdfrP//xPjRgxIrjepJ5Mnz5d06dPr+k2OwPTiSeeWNPtFnNm5gYCR1ACAnPGGWdEKrf33nvr1ltvLTohvBSXIt0ot7M/JN1nOKVIvkcJQLgISsAAtXz5cu2zzz493hy2+015GxoadN999+mII44ou82zzz5ba9eu1fz582ta11qaMGFC0lUowpQAQNgISsAA9de//jVy2ba2Nr3wwgsVg9Ktt94aaVbwJO299949hsNEuOQdDL0BIeNkbgADyvLly3XiiSeqra0t6ap03cKEk7mBcBGUgBQ466yzKs46LUm//e1vNW3aND3xxBP9VLPea25u1uGHH67rrruu4m1Z4njfX/nKVzR//nzddNNNZctlMhnddtttmjp1qlauXFlxu9ViZm4gbAQlIAXuuusu/fjHP9bkyZPLBocnnnhCDzzwgKZNm6apU6f2GBy2bt2qD37wg13zDI0bN06f+cxnetzeF7/4RY0bN06SdM4552jy5Ml6++23tyv34IMPavz48V3DbuPGjdOCBQt63OaaNWu0ePFizZkzRxMnTiwbmGr5vqXcrNsPPPCAstmsLrzwwh57lToD0uTJk3XGGWdoyZIleu2113rcXp/lh94KFwBhISgBCdhxxx1lZl3LqlWr1N7erq1bt+q///u/tfvuu6u9vV2XX355UblvfvObkqQtW7boscce06GHHrpdYGloaFBra2vXDN4bNmzQ2LFje6zHuHHjumYIz2Qy2rx5s3bYYYceyxXePHbjxo1d0xO8+OKLRXWcMmWKWltbtWnTJq1du1ZnnHGGPv3pT8f+viXp/PPPV2tra1fZnnqVTj/9dH3uc5/TypUrtWnTJm3btk1Tp04t2t+SJUtK/+f1BvMoAcHjZG4gAY888ohefvnlrsezZ89WS0uLRowYoREjRuiyyy7TkCFDdNppp2nPPffsKnfzzTdr4cKFXfc8O/PMM/Wxj32saNt1dXW64ooruu55Vl9fX3Jyyrlz5+qqq65Sa2urRowYoW9/+9saNmzYduWmTJmio446Svfcc4+y2awOOuggHXbYYZKk97///Vq4cKG2bdsmSVqxYoUuvvhibd26VU1NTTr44IP19a9/Pfb3vXnzZt13331qamrS5s2blclk9NOf/lSnn356Ublzzz1Xq1ev1sMPP9x1r7e5c+dq3333lZS7hcz+++/f4++r95hHCQhe4bT9LCwsfVuGSb57hKW7KVOm+K677uo33HCDt7e3b/d8pyuvvNKHDh3qc+bM8bVr15Ysl8lk/D3veY/X19f72WefXbKcu/tFF13kQ4cO9XHjxnlra2vJcs8995w3NTX58OHD/fHHHy9Z7pVXXvH6+no/4ogjfPHixWX3Xev3/cILL/j3v/99l+RLlizxv/71ryXLPvPMMz5jxgyvq6vzP/zhD9s9H+X/cVju/7Lk5+EDE97lj3/nkqJF0pJyr2FhYUnXwk1xgRqKelPc7vMYbd68WQ0NDRpS4Y717q6NGzdq1KhRFfcxf/58nXLKKSVvnNtpw4YNmjRpkq688srtel+6mzlzpt5++2099thjZctt2LAhUh3jeN+LFi3SUUcdpah/20rVNcqNgyvdFPeDEyb4deecWbTu7y68mJviAgFh6A1IgZ7OC+qJmUUKC5J0/PHH6/jjj69YbtSoUdq4cWOkbS5cuDBSuah1jON991Zc281xzksCAkdQAjCgpOoGwC7OUQICR1ACMKAcccQReuaZZ5KuhqTcvfGyBCUgaEwPACAIy5cv1x//+MeK5VpaWvT6669H2ua9996r5ubmvlatNHdlui0AwkJQAlJg0aJFevLJJyuWW7lypW655ZaKM163tbXphhtuqDjjdSj7zmaz+sQnPqHp06dXvDXJFVdcoZkzZ+r3v/992XLLli3TjBkzuqYuiINLyroXLQDCQlACUmD27NmaOnWqDj/88LLB4brrrtOsWbM0adIkXX/99SVDy0svvaTTTz+94ozXoex7wYIFam5uLjmJZKf169fr6quvliSdd955JctJuckp6+rqdMMNN2j16tVly/ZFJpMpWgAEJun5CVhYBtISdR6ls846y5XrcChazMyHDx/uU6dO9Y6ODr/nnnt86NChPZYdMWKE77LLLr5kyRJ/8803/QMf+ECP5RoaGrypqckvv/xyd/cg9l2oc06ozrLl5nv62te+5k1NTS6p7HxPnXNCSfJhw4b5F7/4xe3K1GIepT132cXv/spXihYxjxILS1ALPUpAAqzCHD2dz5uZ6upKH6adt9zo7TbTvu9CCxcuLDrnaN26dbr11lu3K7d582Z997vf1datWyXlbmFywQUX9Lifiy66qGsm8ba2Nv3kJz+J7Vwlht6AwCWd1FhYBtJS7czcEydO9KFDh/rhhx/uTz755HbPd7r44ou9rq6u4mzWy5Yt8/r6em9qavIvf/nLvmbNmpLbTPu+V65c6eeee25Xj9K5557rL7300nblstmsX3rppX7sscd2lfvFL37R4zbvvPPOrm1+5CMf8YsvvtgzmUxRmVr0KL13l138N3PnFi2iR4mFJaiFmbmBGqp2Zu77779fO+64ow455JCyr1u1apUeeeQRnXDCCWVns25vb9ett96qo48+WuPHjy+7zRD2LUmjR4/Whg0bVOlvVm9m5jYz/ehHP9KZZ5653XO1mJl7z1128StOPrlo3XFXX83M3EBAmEcJSIEjjzwyUrmJEyfqs5/9bMVyQ4cO1axZswbMvkPlyl2xByBcBCUAQah0P7i04rwkIGxh/uUBAhdlWKdT92G6weqWW25RS0tLxXIHHHBAyZO4u//eR0n61lln6dKzzqpFFbfj7uqgRwkIGkEJQBCmT58eqdxOO+2kSy+9NFLZOG+H24mhNyBsTA8ABOaYY47RYYcdVrHc/Pnz1djYWHEyxfXr12vkyJGaN2/eoNp3f3BJmWy2aAEQFoISEJBly5bpgQce0DPPPKPHH3+8ZLlsNqvzzjtPHR0duuSSS8pu88orr1Rra6u+9rWvlb09yEDbd39wd4ISEDiCEhCQ888/X62trdqyZUvZW3R03vIjk8mUvUVH5y0/2tvbtWXLFt14442DYt/9iaAEhI2gBARi2bJluu+++7rOeVm6dGmPvSudvSqbNm2SlLvXWKnelSuvvFIdHR2ScjNbl+rZGWj77i/urmw2W7QACAtBCQhEc3OzxowZ0/V49OjRWrt27XblWltb1dTUpJEjR0qSxowZow0bNvS4zbfeekujRuVOaR4yZIhGjhypLVu2DPh996eObLZoARAWZuYGaijqzNy90X16gP6eoXog7bs30zJEUWlm7t133tnP+8Qnitb92803MzM3EBCmBwCAmHSezA0gXAQlIDBJ9gIP1n33RUcmk3QVAPQBQQkIzCmnnNLjOTrdTZgwIfI2Gxsbtddeew26fceNHiUgfJyjBNRQHOcoIb0qnaM0YexYP6fbjX8v/MUvOEcJCAg9SgAQF3qUgOARlAAgJi7OUQJCR1ACgJhwjhIQPoISAMSISSaBsBGUACAm7q52ht6AoBGUACAmrtw95wCEi6AEADFxd4begMARlAAgJu6u9o6OpKsBoA8ISgAQE6YHAMJHUAJqqE1qfl16Lel6oN+8u+yz7gQlIHAEJaCG3H1c0nVAemTd1crQGxA0ghIAxMQlzlECAkdQAoCYMI8SED6CEgDEhKvegPDVJV0BABiosu7a1tZWtERhZjPM7EUze9nMzo+5mgDKoEcJAGLi7mqNGI46mVm9pB9K+rikVZKeMrM73H15DFUEUAFBCQBi4u5qbW/v7csOkfSyu/9FkszsdknHSiIoAQkgKAFATDIdHdrQ3Nzbl02QtLLg8SpJH6lZpQD0CkEJAGLS6n7vK21tO3db3WhmSwoez3P3eQWPrYdNee1rByAKghIAxMTdZ1TxslWSJhU8nijpjdrUCEBvcdUbAKTLU5LeZ2aTzWyYpJMk3ZFwnYBBix4lAEgRd+8ws3+VdK+keknXu/vzCVcLGLTMnaFvAACAnjD0BgAAUAJBCQAAoASCEgAAQAkEJQAAgBIISgAAACUQlAAAAEogKAEAAJRAUAIAACjh/wdDqarlZzH/dgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x720 with 12 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Sarsa\n",
    "# Do not modify this cell.\n",
    "def behaviour_policy(q):\n",
    "  return epsilon_greedy(q, epsilon)\n",
    "def target_policy(q, a):\n",
    "  return np.eye(len(q))[a]\n",
    "\n",
    "learned_qs = []\n",
    "for _ in range(5):\n",
    "  grid = Grid()\n",
    "  agent = GeneralQ(grid._layout.size, 4, grid.get_obs(), target_policy, \n",
    "                   behaviour_policy, double=False, step_size=step_size)\n",
    "  run_experiment(grid, agent, int(1e5))\n",
    "  learned_qs.append(agent.q_values.reshape(grid._layout.shape + (4,)))\n",
    "  \n",
    "avg_qs = sum(learned_qs)/len(learned_qs)\n",
    "plot_action_values(GRID_LAYOUT, avg_qs, vmin=-20, vmax=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "9e1f4729",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkoAAAI9CAYAAADFBFfgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABEmElEQVR4nO3de5wcVZ3///dnJplLLpCEJASTgOGiAgEVAYXfBlSQRCGy8kWBVTYSuQnrml/4orAoqKyBFbmtihqFILtc1BAXI6AEkasLIVGuMUiAQCIxYQi5J3Prz/eP7hm6J9PdNT1dU3VmXs/Hox6Zrj5ddXrSNefd51SdMncXAAAAdlaTdAUAAADSiqAEAABQBEEJAACgCIISAABAEQQlAACAIghKAAAARRCUAAAAiiAoAQAAFEFQAlDAzK4ws1kJ7XuxmR2YxL4BoDvGzNwAOpjZGElPSdrX3bcnsP/PSDrF3f9PX+8bALpDjxKAfJ+XdE8SISnn15I+YmZ7JLR/AChAUAIGGDOrMbOLzew1M3vDzM4zs9Zcb9LHJT3UpfzeZvYbM2sys41mtqjM9t3M9s17fLOZ/Xve45W5/S8zs7fMbJ6ZNUiSu++QtFTScdV8z6Exs5vMbJ2ZPZe3bpSZLTKzF3P/jsx77mIzW2FmL5jZ1GRqDfRPBCVg4LlU0gmSpkjaV9I/S3rT3d+QdJCkF7qUv0XSvZJ2zy3fqEIdPitpqqR9JL1L0tfynvuLpPdWYR8hu1nStC7rLpL0e3ffT9Lvc49lZgdIOlXSgbnX3GBmtX1XVaB/IygBA0iu12i2pM+6+6vuvlHS3ZKezRUZIWlzl5ftI6lWUq2773D3x6pQle+7+yp3Xy/p25JOy3tuc64eA5a7PyxpfZfVJ0r6We7nn0n6x7z1d7h7s7u/ImmFpMP7op7AQEBQAgaWYyT9xd1X5q3bTW8HpbckDe/yms8q2xi/bmY3mtmoKtRjVd7Pr0p6R97j4ZI2VGEf/c3u7r5GknL/js2tH6/C3+fq3DoAVTAo6QoA6FOjJb3R8cDMBkmaLqnjHKJnlB0Ke7KjjLs/IOkBMxsr6R5lT/i+psQ+tkkakvd4nLKNd76JeT/vKen1vMf7S/rv8m8l/aZNm+ZNTU0F65YuXfq8pB15q+a6+9xe7Ma6WcflzECVEJSAgWW5pG+Z2SRle22+I2lvSR0nDd8j6WhJt0qSmZ2kbG/TCmV7ekYqO32AzOxmSXL3z3fZx1OS/snMnpf0sdz2lnQpc76Z/UbZUPVvkn6e22a9pA9ImtHrd5oCTU1NWrLkyYJ1ZjU73P3QCja31sz2cPc1uasC1+XWr1Zh8JygwuAJoBcYegMGEHe/X9IvJT0t6QllQ1BG0vO5IrdI+oSZNeYe/4OyV8FtVjZEXZnrYZKyjXN35yt9Wdleqg3KDtv9TzdlbpN0n6SXc0tHj9YnJT3o7v2koXe5txcsvfBrvR0gZ0i6K2/9qWZWnwvA+0la3JsdAXgbE04CA5iZHSfpB7krqTrWzZG0zt2vK/G6OmXD1sHu3trDfa6UdGYutHV97glJX3D353Z6YYAOPfQDvnjx/xasq62tX1quR8nMbpf0YWWHStdKukzZwPkLZYcqX5P06dzJ8DKzSyTNlNQmaZa731vVNwIMYAy9AQPb/nr7RG5Jkrv/W7kXuXtL7rVV5e4frPY2k+aeqeA1flqRp44pUv7byl49CKDKCErAwLa/3j4/CVXm7r0dbgOQMIISMIC5+7kJ7POdfb3PZPW8RwlAehCUACA2XtHQG4D0ICgBQGwYegNCR1ACqqjWzDmoBo42Se3u3U342ImgBISNv+lAFQ1SdhpqDAx/L1vCxTlKQNgISgAQI85RAsJGUAKAmDA9ABA+ghIAxIigBISNoAQAsWF6ACB0BCUAiBE9SkDYCEoAEBuXRFACQkZQAoAYMfQGhI2gBCB4Z518cqRyP5k/P+aaFOKqNyB8BCUAiBFBCQgbQQkAYsPM3EDoCEoAEBtXJtOWdCUA9AJBCQBixdAbEDKCEgDEJHsyN0NvQMgISgAQI07mBsJGUAKA2LjcOUcJCBlBCQBixdAbEDKCEgDEhgkngdARlAAgNgy9AaEjKAFATNwlzzD0BoSMoAQAMXL3pKsAoBcISgAQF3d5O0EJCBlBCQBi5BmCEhAyghIAxImhNyBoBCUAiItL3s7J3EDICEoAEBOXM/QGBI6gBABxcc5RAkJHUAKAGDE9ABA2ghIAxIjpAYCwEZSABLx7//0jl33fpEmRym3YujXyNn/30EORyyZl6RsPRC47evRHIpU75X9vibzNY4+cEblsUQy9AcEjKAFAbFwiKAFBIygBQEzcpQzTAwBBIygBQJzoUQKCRlACgLhwjhIQPIISAMSGm+ICoSMoAUBc6FECgkdQAoCYuAhKQOgISgAQI4ISEDaCEgDExV3O9ABA0AhKABAjepSAsBGUgATsPW5c5LJDGxoildu0fXul1UmlHZveilx2S8NfI5Wr2zXa77JqXMyjBASOoAQAMXE5M3MDgSMoAUBc6FECgkdQAoAYcY4SEDaCEgDExSVvY+gNCBlBCQBi4/QoAYEjKAFAXLiFCRA8ghIAxMQZegOCR1ACgBjRowSEjaAEAHFxl7cTlICQEZSABDTW1VW97LgRIyqsTTrteGNr5LJbhkSbmXv7ui2VVqdi9CgBYSMoAUBcOEcJCB5BCQBi4qJHCQgdQQkA4uIu515vQNAISgAQF4begOARlAAgRgy9AWEjKAFAXJgeAAgeQQkAYsLM3ED4CEoAECN6lICwEZQAIC5c9QYEj6AEAHFxyVsJSkDICEpAAuoHRT/0ot7CZMTQoZG3+ZUZMyKVG97YGKncHj24fcqISaMilWvd2hJ5m9vXbY5UbvHPn4y8zWrhqjcgbAQlAIiJS2p3ghIQMoISAMTFXZkMQ29AyAhKABATl9ROUAKCRlACgBhlGHoDgkZQAoCYOENvQPAISgAQI4begLARlAAgJlz1BoSPoAQAcWHoDQgeQQkAYsJVb0D4CEpAAm6/++7IZS8+44xI5aLO4C1Jo4cPj1RuWENDtH0PjVZOyp7gHEXLm9sibzPT2h6p3Nd/9KPI26wGdycoAYEjKAFAjJgeAAgbQQkAYuKS2tqj9XYBSCeCEgDExZ0eJSBwBCUAiAkncwPhIygBQIyYHgAIG0EJAGLi7mojKAFBIygBQEwYegPCR1ACgLgwMzcQPIISAMSE6QGA8BGUACAmzMwNhI+gBKTcFfPmJV0F9AJBCQgbQQkAYuLuamXoDQgaQQkAYsJVb0D4CEoAEBfOUQKCR1ACgJi4xNAbEDiCEgDEhKvegPARlAAgJi6pnR4lIGgEJQCIC/d6A4JHUAKAmLik1ra2pKsBoBcISgAQE3fnFiZA4AhKQBW1SE2vSa8mXQ/0mb1KPekSQ29A4AhKQBW5+5ik64D0cHc1t7YmXQ0AvUBQAoCYMPQGhI+gBAAx4V5vQPgISgAQk4y7drS0JF0NAL1Qk3QFAKC/cne1tLUVLL1lZleZ2XIze8bMfmVmI/Keu9jMVpjZC2Y2tdc7A0BQAoC4uLtaWlsLlipYJGmyux8s6a+SLpYkMztA0qmSDpQ0TdINZlZbjR0CAxlBCQBiknHXth07Cpbecvf73L2ja+pxSRNyP58o6Q53b3b3VyStkHR4r3cIDHCcowQAMcm0t2vb5s1x7mKmpJ/nfh6vbHDqsDq3DkAvEJQAICY7MpnfLd+8eXSX1Q1mtiTv8Vx3n5tfwMzulzSum01e4u535cpcIqlN0q0dL+umvFdWcwAdCEoAEBN3n1bh644t9byZzZB0gqRj3L0jDK2WNDGv2ARJr1eyfwBv4xwlAAiImU2T9FVJn3T3bXlP/VrSqWZWb2aTJO0naXESdQT6E3qUACAs35dUL2mRmUnS4+5+rrs/b2a/kLRM2SG5892d2S6BXrK3e20BAACQj6E3AACAIghKAAAARRCUAAAAiiAoAQAAFEFQAgAAKIKgBAAAUARBCQAAoAiCEgAAQBEEJQAAgCIISgAAAEUQlAAAAIogKAEAABRBUAIAACiCoAQAAFAEQQkAAKAIghIAAEARBCUAAIAiCEoAAABFEJQAAACKICgBAAAUQVACAAAoYsAEJTO7wsxm9eL1K83s2Ihl321mfzazzWb2r0XKLDazAyutD5AGUY8rPu8IUW/bjSLbfN7MPhyxbOR2J1ee4ywGAyIomdkYSf8s6cd9tMuvSHrQ3Ye7+38W+bB/V9K3+qg+QNX18Lji846gxNVuuPuB7v5gNbbVTdvCcRaDARGUJH1e0j3uvr2P9reXpOfLlPm1pI+Y2R59UB8gDp9X9OOKzztC83lVsd0ws0HV2E4ZHGcx6DdBycxqzOxiM3vNzN4ws/PMrDX3reDjkh7qUn5vM/uNmTWZ2UYzW9TD/b3DzO7M7euVjiE2M3tA0kckfd/MtpjZ7ZL2lLQw9/grkuTuOyQtlXRc7999OMzsJjNbZ2bP5a0bZWaLzOzF3L8j85672MxWmNkLZjY1mVoPTGa2q5m5mY3OW3eAma01s13U5bgqdUwN1M97pThO+kbc7Uaux+erZvaMpK1mNii/F8jMDsk7TeOXZvZzM/v3Lpt5n5k9k9vfz82sIffa/1KXtmUgHWd9eYz0m6Ak6VJJJ0iaImlfZbtM33T3NyQdJOmFLuVvkXSvpN1zyzei7sjMaiQtlPS0pPGSjpE0y8ymuvtHJT0i6V/cfZi7nybpNUnTc4+/k7epv0h6b0/faOBuljSty7qLJP3e3feT9PvcY5nZAZJOlXRg7jU3mFlt31V1YHP3jZJWSzogb/W3Jf2Hu2/SzsdVuWNqIH7eK3WzOE76Ql+0G6dJOl7SCHdv61hpZnWSfqXs//UoSbdL+lQ3r/+Msv+vkyQdrGxPl9z9dHXftgyU4+xm9dEx0i+CUi79z5b0WXd/NfcH/m5Jz+aKjJC0ucvL9pFUK6nW3Xe4+2M92OVhksa4+7fcvcXdX5b0E2X/I3pic65uA4a7PyxpfZfVJ0r6We7nn0n6x7z1d7h7s7u/ImmFpMP7op7o9Jyk/SXJzA6XdIikH+SeG6HC46rcMTXgPu+V4jiJXx+2G//p7qu6GcL7kKRBuedb3X2BpMVFXv+6u69X9gv6+8rsb0AcZ315jPSLoKRsj85f3H1l3rrd9PYH/i1Jw7u85rPK/vJeN7MbzWxUD/a3l6R3mNmGjkXSvyn7DaMnhkva0MPX9Ee7u/saScr9Oza3frykVXnlVufWoe88p7d7lK6Q9A13b8497npclTum+Lz3DsdJdfVVu7GqyPp3SPqbu3uZsn/P+3mbpGFl9jeQj7NYjpG+OLmsL4yW9EbHA8ueNDddUsdY7zOS3iXpyY4y7v6ApAfMbKyke5Ttzrwm4v5WSXol170XhRdZv7+k/464jVSZNm2aNzU17bR+6dKlz0vakbdqrrvPrXA31s26Yr9LxOM5SZ/LnVOxh7JDDx0KjqsIx1Swn/dKTZs20ZuadhSsW7q0qZrHiMRxUqm+ajeK/V+skTTezCwvLE2U9FIP3kN32w7qOAuhLekvQWm5pG+Z2SRlk/R3JO2t7B95KfuBPlrSrZJkZicp+61hhbLpe6Skp3LP3SxJ7v75EvtbLGmTmX1V0n9KalH2w9no7k92U35trj6dzKxe0gckzejB+0yNpqYmLVmy81s1q9nh7of2cHNrzWwPd19j2as11uXWr1b2D0eHCZJer6jCqFRHj9IcSf/m7u15z3UeV6WOKSn8z3ulmpp26MklJxWsq7G5lRwjEsdJtfV1u9HV/0pql/QvZvZDZc9jOlzSgz3YRkHbEuJxFkJb0i+G3tz9fkm/VPbk6ieU/TBn9PYl+rdI+oSZNeYe/4OyVzNsVvZguDL3TUHK/jJLjjvnGovpyo4VvyKpSdJPJe1a5CVXSPpabpju/+bWfVLZuZaC/YPmntlpqdCv9faBPUPSXXnrTzWz+twfs/3U/Rg+4rNM0jhJ7e7+P12eyz+uSh1TUj/4vFcq44VLL3CcVFFftxvd7L9F0kmSvqBsUPucpN9Iai7xsq66ti1BHmepb0vcvd8tyl4a+WKXdXMkzSrzujplrxgY3Ad1fELS5KR/V5UuH/jAId7e3rLTImlJmfd9u7Jdzq3KpvwvKHtewO8lvZj7d1Re+UuU7Yp+QdLHk37fLDv9f5Y9rnLlgv68V7oc8oHR3tJ+dsFS7hjJ/b44Tvp4SUO7kTtOzujl64M6zkJoSyy3gX7FzL4s6Wh3P6lsYVTk0EM/4E88sfMXqEGDGpd6ZcMKQL/zgQ+M8ceeKPwz1Dh4LsdICiXRbpjZ0co23E3Knij+I0l7e+6E5IEghLakv5yj1NX+enucGbFwZXupARTj6vVwG/pOEu3GuyX9Qtkr2V6SdPJACklZ6W9L+mVQcvdzk65Df+eeHVcGUBpBKQxJtBuevYqrN1c8Bi+EtqRfBiX0BVfhBVAAunJJ7QQloIT0tyUEJVQs7d8CgDSgRwkoLe1tSY+CUq2Zk6wGjjZJ7e7dTdSl3HflvqxOEGxIg2tE18l80W9t2CzftqPIMZIdViAo7Yy2ZGBpkZrcfUz3z6a/LenRZ3WQshOqYGD4e8ln099dmogRw6WzudhywJi7oGyR9nR/WU4EbcnA8pr0avFn09+WEOpRsbR3lwJJ46o3oLy0tyUEJVQkhCsVgDQgKAHFhdCWEJRQIZd7W9KVAFLNnavegNLS35YQlFAhT/23ACAN6FECSkl/W0JQQi+k+8MNJI1zlIAo0t2WEJRQsbR/CwDSgKAElJb2toSghAqlf1wZSJo70wMApaW/LSEooSLu6R9XBtKAHiWguBDaEoJSAM46+eRI5X4yf37MNekq3ZOE9RfvHBmt3Mq34q1Hmh1QZM7frpa9EW89ukNQQhqktx2R0t6WEJRQofTPpgokjZviAuWkvy0hKKFiaf9wA4njXm9AWWlvSwhKqFD6x5WBpDE9AFBO+tsSghIqkj0BL91XKgBpwFVvQHEhtCUEJVQs7d2lQNLoUQLKS3tbQlBChVxpn00VSBznKAFlpL8tISihYmnvLgWSxlVvQHlpb0sISqhQ+i/pBNKAHiWglPS3JQQlVCj9H24gaZyjBJST/raEoISKuKf/RoZAGhCUgOJCaEsSD0rv2GuvSOWmHHhg5G3uaGmJVG7U8OGRyu0WsZwkjRsxIlK5fQ7fO/I27/mfRyOVe7H5zcjb3K9+t8hli/FMuj/caRb1tiSS1FAbrdy+PfgvHR/xI713xHoeNDb6vo+cGK3cBydE3+aBP4hWbv1Xo29z1H9EL1sMN8XtnVM+8YnIZTds3Rqp3O4R/0bvO25c5H3vs+c7IpXb7fBo5ca8712R9z169FGRyv37pz8deZt9Le1tSeJBCaFyOV+VgbI4TIBS0t+WEJRQGU6+ACLhMAFKCKAtISihYmn/FgAkjekBgPLS3pYQlFAZT/+HG0gcE04CpQXQlhCUUBFX+j/cQNICGFUAEhVCW0JQQmXc5VzOA5TFYQKUEEBbQlBCxdzT/S0ASBo9SkB5aW9LCEqoHC0AUBrnKAHlpfwgISihMi5luJwHKImr3oAyAmhLEg9KB73znZHKDW1oiLzNn99zT4W1Sac/vrIgUrmWlqaYa5KPr8q9EXW2bUmqi1q2B7dLemhldcvFYcW/Ri/7/PnRyr20vrK69AaHSeV68nd/0/btkcrdctddlVYnlZ7f/EKkcp+/8rRI5X4yf35vqlOB9LcliQclhMkDuKQTSBrnKAGlhdCWEJRQsbR/uIE04DABSkt7W0JQQmVcqb+kE0gaN8UFygigLSEooULpv5EhkAYcJkAp6W9LCEqoWNo/3EDSOEcJKC/tbQlBCRVxlzzll3QCacBhAhQXQltCUELF0v4tAEiap//KZyBxaW9LCEqoDC0AEAmHCVBCAG0JQQkVy6T8SgUgaS6uegPKSXtbknhQaqyvj1auri7mmqTX9rVbIpWrbfhrzDXJw1mqvRJ5tu0elB3Wzw6RVzeGsc1yOEwq15O/++NGjIivIim2ZUu0v/vb10VrR/pcAG1J4kEJ4Ur7uDKQuPSPKgCJS3tbQlBCRdw99ZOEAUlj6A0oLYS2hKCEiqX9WwCQBuluAoDkpb0tISihMgHMfQEkLYDTL4BkBdCWEJRQsbR/CwDSgMMEKC3tbQlBCZUJYFwZSBo3xQXKCKAtISihIq70d5cCaZDyL8tAokJoSwhKqIynv7sUSBrnKAFlBNCWEJRQGXd5W7q7S4E0SPmXZSBZAbQlBCVULO3fAoCkBXAbKyBxaW9LEg9KQ2O4hcm/zZwZqVxLW1ukclHrKEkTdtstUrmRk0ZF3mbr1pZI5ba93of3Z3DJM+n+FpBmcdzCpL4HR/PH94tWbujgaOUm7hp933tFLPv65ujbjGoVtzAJSk/+7o8YOjRSua/MmBGp3LCGhsj73m348Ejldt9vbLR97xu9fdi+LtqBsm1NDAdUNQTQliQelBAml+RttABAKczMDZQWQltCUEJlArikE0gDepSAEgJoSwhKqBwtAFAa5ygB5aX8ICEooTIuZVJ+pQKQNIbegDICaEsISqiMe+onCQPSIOVfloFkBdCWEJRQuZR/uIGkuaR0f1cGUiDlbQlBCRVxV+pPwAPSgB4loLgQ2hKCEirjnvpxZSBp3BQXKCOAtoSghIqlfVwZSAN6lIDS0t6WJB6UMhFn5BxcG30q4yERZ9Ies8sukcr1ZIbWIUOjlXWP/sFobtoWqVymtT3yNnvNlfpx5TQbVBO9bNQZtxt6cDQPizjhcdRyPfg4660d0ctW2xtb+3Z/3BS3d3rydz/qLN6jI86i3aO/+8MbI5WL+ne/5c1of/MlRb5P2uZlTZG32acCaEsSD0oIk0tqS/m080AapLwNABIVQltCUEJl3CP3BgIDFTfFBcoIoC0hKKEi2SEFWgCgHIISUFwIbQlBCRVrT/m3ACBpzMwNlJf2toSghIp4AN2lQBrQowQUF0JbQlBCxdLeXQokjXOUgPLS3pYQlFCREK5UANKAq96A4kJoSwhKqEwA3aVA0lwShwlQQgBtCUEJFQnhSgUgDRh6A4oLoS1JPCjdfvfdkcqddvzxkbe569ChkcpFnXl1SMQZXyVp6JZo2xy2Mfqsr2s3bIhU7ls//WnkbVZD2r8FpNnjq6OX/dCE+OrRX0SdlP7CRfHWozscJZW7Yt68yGXP+NSnIpUbOWxYpHLDezAz99Ct0coOfSvaXSOGRry7hBQ9ZMycMyfyNvta2tuSxIMSwuTuqR9XBpLGTXGB0kJoSwhKqFjavwUAacDQG1Ba2tsSghIq4u6pnyQMSBo3xQVKC6EtISihIiFc0gkkzpkeACglhLaEoISKpb27FEgaPUpAeWlvSwhKqIi7qz3ll3QCaUBQAooLoS0hKKEiLqmtPeI12cAAxU1xgdJCaEsISqhMALOpAonjXm9AaQG0JQQlVCT7TTndH24gaZyjBJQWQltCUEJF3D313aVAGnDVG1BcCG1JMEEp6q1O0HfS/i2gv+jJ7U6QLvQo9Z15v/pV0lVAhdLelgQTlJAuIXSXAmlAUAKKC6EtISihMgHMpgokziUOE6CEANoSghIq4u5qTfm4MpA0F+coAaWE0JYQlFCRELpLgTRg6A0oLoS2hKCEioTw4QaSxsncQGkhtCUEJVTE3dXa1pZ0NYB046a4QEkhtCUEJVTGPfV3fAaSRo8SUEYAbQlBCRUJ4f48QBoQlIDiQmhLCEqoSAjdpUDSuCkuUFoIbUmPglKL1PSa9GpclUHq7FXsCZdS312aiDVNTfrmXI6RgaPoMSKJm+IWQVsy4ATdlvQoKLn7mLgqgrCEMPdFEjhGkI9zlLrHcYIOIbQlDL2hIu6ultbWpKsBpB5XvQHFhdCWEJRQkRC+BQBJo0cJKC2EtoSghIq4lPoT8IA0ICgBxYXQltQkXQGEKeOulra2nZbeMLOrzGy5mT1jZr8ysxF5z11sZivM7AUzm9rb+gN9Incyd/7SWxwn6E9CaEsISqiIZzLa3ty809JLiyRNdveDJf1V0sWSZGYHSDpV0oGSpkm6wcxqe7szIG4d0wPkL1XAcYJ+I4S2hKCEimTc1dzautPSG+5+n7t3fJV4XNKE3M8nSrrD3Zvd/RVJKyQd3qudAX2k2j1KHCfoT0JoSzhHCRXxTEY7tm2LcxczJf089/N4ZT/sHVbn1gGp1gcnc3OcIGghtCUEJVRke3v77559883R3TzVYGZL8h7Pdfe5HQ/M7H5J47p53SXufleuzCWS2iTd2vGybspziizS7/Wm37VfNrfrcVLyGJE4TjBwhNCWEJRQEXefVuHrji31vJnNkHSCpGPcveMDvFrSxLxiEyS9Xsn+gb7EcQKUFsIxwjlKSA0zmybpq5I+6e75fbG/lnSqmdWb2SRJ+0lanEQdgaRxnAClVfsYoUcJafJ9SfWSFpmZJD3u7ue6+/Nm9gtJy5TtRj3f3dM9QxkQH44ToLSqHiP2do8UAAAA8jH0BgAAUARBCQAAoAiCEgAAQBEEJQAAgCIISgAAAEUQlAAAAIogKAEAABRBUAIAACiCoAQAAFAEQQkAAKAIghIAAEARBCUAAIAiCEoAAABFEJQAAACKICgBAAAUQVACAAAogqAEAABQBEEJAACgCIISAABAEQQlAACAIghKAAAARRCUAAAAiiAoAQOMmV1hZrOquL2bzezfq7W9iPtcbGYH9uU+AQxMBCVgADGzMZL+WdKPk65LL31X0reSrgSA/o+gBAwsn5d0j7tvT7oivfRrSR8xsz2SrgiA/o2gBPQzZlZjZheb2Wtm9oaZnWdmrbnepI9LeqhL+b3N7Ddm1mRmG81sUZntv9/M/mRmm83s55Iaujy/v5k9aGYbzOx5M/tkbv0ZZrYwr9wKM/tF3uNVZva+3M8rzez/mtkzuTr93Mw69+PuOyQtlXRcpb+nNDOzm8xsnZk9l7dulJktMrMXc/+OzHvu4tzv8wUzm5pMrYH+iaAE9D+XSjpB0hRJ+yo71Pamu78h6SBJL3Qpf4ukeyXtnlu+UWzDZlYn6X8k/ZekUZJ+Ken/5D0/WNJCSfdJGivpS5JuNbN3KxvQpuSC3B6SBkv6/3Kv21vSMEnP5O3uM5KmSZok6WBle8Py/UXSe0v/KoJ1s7LvPd9Fkn7v7vtJ+n3usczsAEmnSjow95obzKy276oK9G8EJaAfyfUazZb0WXd/1d03Srpb0rO5IiMkbe7ysn0k1Uqqdfcd7v5YiV18SNmAc527t7r7fElPdnl+mKQr3b3F3R+Q9BtJp7n7y7l9v0/S0ZJ+J+lvZvae3ONH3D2Tt63/dPfX3X29suHrfV3qsjn3fvodd39Y0vouq0+U9LPczz+T9I956+9w92Z3f0XSCkmH90U9gYGAoAT0L8dI+ou7r8xbt5veDkpvSRre5TWfVbaxfd3MbjSzUSW2/w5Jf3N3z1v3apfnV3UJPK9KGp/7+SFJH5Z0VO7nB5UNSUery5CgpL/n/bxN2QCWb7ikDSXq2t/s7u5rJCn379jc+vGSVuWVW623f98AemlQ0hUAUFWjJb3R8cDMBkmaLqnj8v1nJL1Leb1AuV6fB8xsrKR7lB3iuqbI9tdIGm9mlheW9pT0Uu7n1yVNNLOavLC0p6S/5n5+KFefSZLmKBt0PivpCEnf7+F73V/Sf/fwNX1q2rRp3tTUVLBu6dKlz0vakbdqrrvP7cVurJt13s06ABUgKAH9y3JJ3zKzScqGkO9I2ltSx0nB9yjbe3OrJJnZScr2Nq1QtodmpKSncs/dLEnu/vm87f+vpDZJ/2pmP5D0SWWHef6Qe/4JSVslfcXMrlb2HKTpkg7LPf+QsiFsrbuvNrNNyp7vNEjSn6O+STOrl/QBSTOiviYJTU1NevLJJwrW1dQM2uHuh1awubVmtoe7r8md47Uut361pIl55SYoG1gBVAFDb0A/4u73K3uC9dPKhpZnJWUkPZ8rcoukT5hZY+7xPygbXjYrG6KuzPUwSdnGt+B8JXdvkXSSsr1Ob0k6RdKCLs9/Utmr65ok3SDpn919ee75v0raIumR3ONNkl6W9Ji7t/fgrX5S0oPunvJA4HLPFCy98Gu9HQxnSLorb/2pZlafC8j7SVrcmx0BeJsVnmoAoD8xs+Mk/SB3pVTHujmS1rn7dSVeV6ds2DrY3Vtjr2gPmdkTkr7g7s+VLZygQw/9gC9e/MeCdbW1DUvL9SiZ2e3Knss1WtJaSZcpe7XhL5QdynxN0qdzJ7rLzC6RNFPZ3r5Z7n5vVd8IMIARlIB+zMy+LOlodz8p6boMRIceeog/8cSjBesGDRpaNigBSA/OUQL6t/319vlJ6GPu6u1wG4CEEZSAfszdz026DgObE5SAwBGUACBWBCUgZAQlAIiNy70t6UoA6AWCElBFtWbOQTVwtElqd+9uwsdODL0BYeNvOlBFgySNS7oS6DN/L1uCc5SA0BGUACAm2aveejKPJoC0ISgBQGzoUQJCR1ACgFjRowSEjKAEALFxht6AwBGUACBGDL0BYSMoAUBs6FECQkdQAoCYuBOUgNARlICUO+UTn4hU7uf33BNzTdLrrJNPjlTuJ/Pnx1yT7hCUgJARlAAgNkwPAISOoAQAMWLoDQgbQQkAYsM5SkDoCEoAEBNuYQKEj6AEALFxSZyjBISMoAQAsXG5tyVdCQC9QFACgBgx9AaEjaAEALHhZG4gdAQlAIgNQ29A6AhKABCT7FVvnMwNhIygBCTg4x/5SOSyQxsaIpU7/qMfjbzNux94IHLZpNz7u7mRyy74yX2Ryl07a1bkbf7/110XuWxJ7tXZDoBEEJQAIDauTDs9SkDICEoAEBeXlKFHCQgZQQkAYuQEJSBoBCUAiJFzjhIQNIISAMTEXfJ2ghIQMoISAMTGGXoDAkdQAoC4cDI3EDyCEgDEyDNMDwCEjKAEAHFxrnoDQkdQAhLQWFdX9bKjd9ml0uqkUm3j4Mhlz/zejEjl1j+7JnoFrotetDjOUQJCR1ACgJhw1RsQPoISAMSIHiUgbAQlAIgRQQkIG0EJAOLiLuemuEDQCEoAECN6lICwEZQAIC5MOAkEj6AEADFx0aMEhI6gBABxcWd6ACBwBCUAiBG3MAHCRlACgLhwCxMgeAQlIAH1g6IfelFvYTJi6NBKq5NKmZa2yGW3r9scrdyaLZVWp2IMvQFhIygBQEzcudcbEDqCEgDEiHOUgLARlAAgLi55Gz1KQMgISgAQI4begLARlAAgLpyjBASPoAQAMXFJ3sY5SkDICEoAEBfmUQKCR1ACgNi4xFVvQNAISgAQF2fCSSB0BCUg5QbX1kYqF3UGb0m6dtasSOWG1tdHKjeksSHyvut3HxKpXMv67ZG3GTWMbF3xVuRtVoVLmXZ6lICQEZQAIE70KAFBIygBQEzcnaE3IHAEJQCIkTP0BgSNoAQAceFkbiB4BCUAiAtBCQgeQQkAYuPMzA0EjqAEADFx5xwlIHQEJQCIC0NvQPAISgAQG1eGoTcgaAQlAIiLiwkngcARlIAE3H733ZHLnnb88ZHKjRo+PPI2h0S83cnwxsZo5ZqjlZOk4TuaI5Ub8taOyNv0iFnk9Msvj7zNanBJGW6KCwSNoAQAcXFXe9QUByCVCEoAEBOX1E6PEhA0ghIAxIihNyBsBCUAiIm7K8PQGxA0ghIAxIihNyBsBCUAiAlXvQHhIygBQFy46g0IHkEJAGLCVW9A+AhKABAjht6AsBGUgJTrySzeSBdn6A0IHkEJAGLU3t6edBUA9AJBCQBi4u6cowQEjqAEADFxiQkngcARlAAgRvQoAWEjKAFATNxdbQQlIGgEJQCIEdMDAGEjKAFATJhwEggfQQkAYuLuamN6ACBoBCUAiBE9SkDYCEoAEBPmUQLCR1ACgJi4xNAbEDiCEgDEhR4lIHgEJQCICVe9AeEjKAFATNxdrQy9AUEjKAFATOhRAsJHUAKAuHALEyB4BCUAiIlLamfoDQgaQQkAYuLuamlrS7oaAHqBoAQAMWHCSSB8BCUAiAkTTgLhIygBVdQiNb0mvZp0PdBn9ir1pLurmaE3IGgEJaCK3H1M0nVAerg7PUpA4AhKABATl9RKjxIQNIISAMTE3dXc2pp0NQD0AkEJAGKSYXoAIHg1SVcAAPord1drW1vB0ltmdpWZLTezZ8zsV2Y2Iu+5i81shZm9YGZTe70zAAQlAIhLJpPR1h07CpYqWCRpsrsfLOmvki6WJDM7QNKpkg6UNE3SDWZWW40dAgMZQQkAYuKZjHZs21aw9Hqb7ve5e0fX1OOSJuR+PlHSHe7e7O6vSFoh6fBe7xAY4DhHCQBikslktG3z5jh3MVPSz3M/j1c2OHVYnVsHoBcISgAQk2b3373U0jK6y+oGM1uS93iuu8/NL2Bm90sa180mL3H3u3JlLpHUJunWjpd1U94rqzmADgQlAIiJu0+r8HXHlnrezGZIOkHSMe7eEYZWS5qYV2yCpNcr2T+At3GOEgAExMymSfqqpE+6e/5JT7+WdKqZ1ZvZJEn7SVqcRB2B/oQeJQAIy/cl1UtaZGaS9Li7n+vuz5vZLyQtU3ZI7nx35/4pQC/Z2722AAAAyMfQGwAAQBEEJQAAgCIISgAAAEUQlAAAAIogKAEAABRBUAIAACiCoAQAAFAEQQkAAKAIghIAAEARBCUAAIAiCEoAAABFEJQAAACKICgBAAAUQVACAAAogqAEAABQBEEJAACgCIISAABAEQQlAACAIghKAAAARRCUAAAAiiAoAQAAFEFQAgAAKIKgBAAAUARBCQAAoAiCEgCkjJndZGbrzOy5pOsCDHQEJQBIn5slTUu6EgAISgCQOu7+sKT1SdcDAEEJAACgqEFJVwAA+qtp06Z5U1NTwbqlS5c+L2lH3qq57j63TysGIDKCEgDEpKmpSU8+ubhgXU1N7Q53PzShKgHoIYISAMTGJWWSrgSAXuAcJQCIkXt7wRKFmd0u6X8lvdvMVpvZF2KtJICi6FECgNi43Hveo+Tup8VQGQAVICgBQEzcVVFQApAeBCUAiI1HHm4DkE4EJQCIFT1KQMgISgAQm8rOUQKQHgQloIpqzZyDauBok9TubqXKuLf1UW0AxIG/6UAVDZI0LulKoM/8vWwJepSA0BGUACBWBCUgZAQlAIiJuyuT4ao3IGQEJQCIFUEJCBlBCQBiwzlKQOgISgAQI4ISEDaCEgDExpkeAAgcQQkAYkWPEhAyghKQcq+6Fzx+44031NLSovHjxxes38tKznvYr3X9HRXT178jd85RAkJHUAIC87GPfUyvv/661q1bl3RVUBZDb0DoCEpAYFauXKmNGzcmXQ1E5M70AEDIapKuAABU0/z589XQ0KA1a9YkXRV1TA+QvwAIC0EJCERLS4uWLFnS2Zu0ZMkSbd++PeFapUsmk9GFF16otrY2XX755UlXR+7Zm+LmLwDCQlACArFgwQJ96EMfUk1N9rA94ogjNG/evIRrlS4LFixQU1OT2tvbNW/evBT0KrmyV73lLwBCQlACAjFt2jQ1NDQok8k2toMGDdJJJ52UcK3So6M3acuWLZKk9vb2lPQqtRcsAMJCUAICMWLECM2ePVuNjY2qq6vTzJkzNW7cuKSrlRrNzc1qbGzU8OHDJUkjR45MwUnvztAbEDiCEhCQ2bNnq6amRjU1Nfr617+edHVSpbGxUcuWLdOdd94pSVq7dq1uvfXWhGvl9CgBgWN6ACAgI0aM0DXXXKONGzfSmxQIrnQDwkZQAgJz1llnJV0FROaS6EUCQkZQAhIQ9ZYb3Tn//PO1bt06zZ8/v2rbDF3+rUlau1nXoSe/o2rc7sRd8gw9SkDICEpAYG677bYUnKScXoMl7Zl0JfJ4ZuAGWKA/ICgBQFzcCUpA4LjqDUiBc889V1/60pe0du3aqmyvqalJRx11lG688Ua1traWLBt137/97W91zDHH6PHHH0/tvnuivb1dt99+u6ZMmaJVq1ZVbbtdeXumYAEQFoISkAJ33323fvzjH2vSpElFg8Mf/vAHjR07tnPYbcyYMVqwYEG321u7dq0WL16sWbNmacKECSVDS5R9S9Ljjz+uBx54QMccc4ymTJlSNLQkue8oOgLSpEmTdPbZZ2vJkiV69dVXK95eOe5esAAITNeDmIWFpfKlTvI9IyzDhw93ZS+J2mmpra31uro6b2lp8XzPPvusDx48uLNcXV2dP/bYY+7uvnz58qLbk+Q1NTV+4oknurtH3vcVV1xRcpuS/M4770x03x2i/M73lNzd/YwzzvCampqS237yyScjba8uu82in4eDJ+/jq1+6s2CRtCTpzykLC0v0hXOUgAQ8/PDDWrFiRefjmTNnavPmzRo2bJiGDRumK664QoMGFR6ekydP1nHHHad7771XmUxGhx56qI488khJ0rve9S4tXLhQO3bskCStXLlSl112mbZv367GxkYddthhuvTSS3u07zPPPFP77rtvZ7lbbrlFCxcuVH19vWpqanTOOefowx/+sEaOHJnYvitxwQUXaM2aNXrooYfU3NysIUOGaPbs2TrooIMkZW8N8/73v7+ibXflcmXa6UUCgpZ0UmNh6U9L1B6lriZPnuzjxo3zefPmeWtr607Pd3j22We9sbHRhwwZ0tmb1J2XXnrJa2tr/eijj/bFixcXLdeTfV999dU+ePBgnzVrlq9bty5V++5Jj1KHp59+2qdNm+Y1NTX+pz/9qaJtlutROujAvf215b8sWESPEgtLUIu5820HqJZ6M48yX3bX+Xy2bt2q+vr6nXqRujN9+nS99dZbevTRR0uW27hxo3bdddey24u6b3fXpk2bIm2zr/cddc6j7uZRKlbXKNv8u6Rm96IFD568j/9m/pWF293/M0vd/dAI1QWQAgy9ASkwdOjQyGUXLlwYqVyUoNKTfZtZ5G0mue+eimu7kiQXV7oBgSMoAYG59957tWXLFn3605+ueBvVmHU6bj2ZRTu9s5I78ygBgSMoAYE5/fTT9eabb4ph8/RziaAEBI55lIDAtLW1JV2FVNu0aZN+85vfJF2NLM8GpfwFQFgISkAKLFq0SE888UTVt/vUU09FPqcpzVatWqVbb7217EzfknTVVVdp+vTp+uMf/1i27Pr163XTTTdp+/bt1ahmt7zdCxYAYSEoASkwc+ZMTZkyRUcddVTRwLR69WpdcMEFnTNzX3DBBXrxxRe7LfvUU09p6tSp+uAHP6jTTz89tnr3lRtvvFEzZszQxIkTddNNNxUNTBs2bNC1114rSbrwwguLbm/9+vW66KKLNHHiRJ1zzjlaunRpLPWWS8p44QIgKAQlIAFf/OIXZWady+rVq9Xa2qpHH31UH/3oR3XUUUepvb294DVLly7Vdddd1/n4+uuv12OPPbbTto8//ngdeeSRuv/++9XS0qKNGzcW7OuUU05R2pvr7ZLq6uo66/zNb35T7e3tWrt2rb785S9r4sSJ3Yabq6++WplM9iqzp556qttepR/96EeaOHGirr/+em3btk3urilTpnTua88999Tq1aur9E6coTcgcAQlIAFW5qqz7p6fPn269tprr87Ho0aN0j/90z/tVK6mpvRhXe75tChVz45Qk2/r1q367ne/2zmMtm3bNl188cXdvrbcfsuViSw3PQA3xQXCFcZfTKCfueGGGwpmfp0wYYIGDx6sKVOm6A9/+IMeeugh1dbWFrympqZGV111lYYNG6ahQ4fq29/+turq6nba9sKFC/X444/ruOOOU11dnXbdddeCfd1+++1K++QAjZJ27NjRWefLLrtMNTU1GjdunL73ve9p1apVOuSQQwpeM2TIEF166aU68cQTJWWHJr/0pS/ttO1zzjlHq1at0qxZszRkyBBJ0iOPPNK5r5UrV2r8+PFVeR8dV73RowSEi5m5gSqqdGbu+++/X7vssosOP/zwkq/LZDLad999tWXLFq1evbrboJTvmWee0WuvvaYTTjihYH1o8yitXr1aDz/8sD7zmc+UncV70aJFOu644yJNn7B+/XrdddddOvXUU9XY2FjwXDVm5p787nf6nXMvK1j3ng/PZGZuICDMowSkwLHHHhupXE1Nje6++261tLSUDUmSdPDBB+vggw/ubfUSN2HChG6HGXtr1KhROuOMM6q+3XwMtwFhY+gNCMzo0aM1evTosuXcXc8880wf1AhFMY8SEDyCEhCYj33sY3r/+99fttyDDz6o9773vX1Qo3Q55JBDuj2JOykEJSBsDL0BgVm5cmXnXEqlDNQZvHfbbTfNmTMn6WpIyvbqMckkEDZ6lIAU2LJlS6RZpwcqd9eGDRsilZ0/f74aGhq0Zs2aSOWjbrdSnskULADCQlACUuCII44oO+t0S0uLlixZ0tmbtGTJkqK33li+fLmWL1/eWe7111+Pp+J95Nprr9XYsWP1pS99SWvXri1aLpPJ6MILL1RbW5suv/zyktvsmL18t91205///OdqV7kTQ29A2JgeAKiiqNMD/Grp0oLbj5x55pnasmVL5xxJc+bM0RlnnFEw8eEdd9yhz33uc3J3ZTIZDRo0SNdff73OO++8gm1v3bpVI0eOVF1dnbZu3aqGhgYdfPDBBbdGSfv0AO2Srr7jjs7H//Vf/6W7775b9fX1qqmp0Re+8AV94xvf0G677Vbwuvnz5+uMM87Qli1b1NDQoJdffll77LFHQZnnnntOF1xwgR555BE1NzdryJAhmjVrliZPnixJGjx4sE488UTtXWYaAqn89AAH7ruX3/HdrxasO/hT5zM9ABAQghJQRVGD0lvDh2vz5s3dPjdo0CDV1tZq8+bNGjx4cOf6DRs2aMKECdq6daskqaGhQa+88orGjdt5j9OnT9c999yjTCajoUOH6rrrrtOZZ57Z+Xzag9LG3FLKggUL9KlPfarzcSaT0T777KOVK1dKygaeM888UzfccEPB677whS/o5ptv7rzVSXeefPJJ/Z/DDitbz7JBaZ89/bbvFAal9538LwQlICAMvQEJ2LRpU7czczc2Nur888/Xa6+9VhCSJGnEiBGaPXu2GhsbVVdXp5kzZ3YbkiTpiiuuUH19vaTsjNUzZsyI/T1V065Swe/nssuykzYOGTJERx99tBYvXlwQkiSpublZjY2NGj58uCRp5MiR3Z70PnfuXN12223ac889NWzYMDU0NBTMzO3uOvTQKuaYTKZwARAUghKQAtOnT9d5552nlStX6rrrrtPYsWO7LTd79mzV1NSopqZGX//614tub/LkyTrmmGNUU1OjOXPm7BS6QnPEEUfo2GOP1YMPPqgHH3xQh3XT29PY2Khly5bpzjvvlCStXbtWt956607lamtrdcopp+jll1/WT3/6Ux1++OEF99CrJnfJ271gARAWpgcAUqDr8FAxI0aM0DXXXKONGzcW7U3q8B//8R8aNmxYcL1J3Zk6daqmTp1a1W12BKZTTjmlqtst5MzMDQSOoAQE5uyzz45U7oADDtBtt91WcEJ4MS5FulFuR39Ius9wSpFcjxKAcBGUgH5q2bJlOvDAA7u9OWzXm/LW19frvvvu09FHH11ym+edd57WrVun+fPnV7Wu1TR+/Pikq1CAKQGAsBGUgH7qb3/7W+SyLS0tWr58edmgdNttt0WaFTxJBxxwQLfhMBEueRtDb0DIOJkbQL+ybNkynXLKKWppaUm6Kp23MOFkbiBcBCUgBc4999yys05L0m9/+1sdc8wxevzxx/uoZj3X1NSko446SjfeeGPZ27LE8b6/8pWvaP78+frZz35Wslx7e7tuv/12TZkyRatWrSq73UoxMzcQNoISkAJ33323fvzjH2vSpEklg8Pjjz+uBx54QMccc4ymTJnSbXDYvn273vOe93TOMzRmzBiddtpp3W7vi1/8osaMGSNJOv/88zVp0iS99dZbO5X7wx/+oLFjx3YOu40ZM0YLFizodptr167V4sWLNWvWLE2YMKFkYKrm+5ays24/8MADymQyuuSSS7rtVeoISJMmTdLZZ5+tJUuW6NVXX+12e72WG3rLXwCEhaAEJGCXXXaRmXUuq1evVmtrq7Zv364f/vCH2nPPPdXa2qorr7yyoNw3v/lNSdK2bdv06KOP6ogjjtgpsNTX16u5ublzBu+NGzdq1KhR3dZjzJgxnTOEt7e3a+vWrRo6dGi35fJvHrtp06bO6QleeOGFgjpOnjxZzc3N2rJli9atW6ezzz5bn/70p2N/35J00UUXqbm5ubNsd71KZ511lj73uc9p1apV2rJli3bs2KEpU6YU7G/JkiXF//N6gnmUgOBxMjeQgIcfflgrVqzofDxz5kxt3rxZw4YN07Bhw3TFFVdo0KBBOvPMM7Xvvvt2lrvlllu0cOHCznuenXPOOfrwhz9csO2amhpdddVVnfc8q62tLTo55ezZs3XNNdeoublZw4YN07e//W3V1dXtVG7y5Mk67rjjdO+99yqTyejQQw/VkUceKUl617vepYULF2rHjh2SpJUrV+qyyy7T9u3b1djYqMMOO0yXXnpp7O9769atuu+++9TY2KitW7eqvb1dP/3pT3XWWWcVlLvgggu0Zs0aPfTQQ533eps9e7YOOuggSdlbyLz//e/v9vfVc8yjBAQvf9p+FhaW3i11ku8ZYelq8uTJPm7cOJ83b563trbu9HyHq6++2gcPHuyzZs3ydevWFS3X3t7u73znO722ttbPO++8ouXc3b/+9a/74MGDfcyYMd7c3Fy03LPPPuuNjY0+ZMgQf+yxx4qWe+mll7y2ttaPPvpoX7x4ccl9V/t9L1++3L/3ve+5JF+yZIn/7W9/K1r26aef9mnTpnlNTY3/6U9/2un5KP+Pddn/y6Kfh3ePf4c/9p3LCxZJS0q9hoWFJV0LN8UFqijqTXG7zmO0detW1dfXa1CZO9a7uzZt2qRdd9217D7mz5+v008/veiNczts3LhREydO1NVXX71T70tX06dP11tvvaVHH320ZLmNGzdGqmMc73vRokU67rjjFPVvW7G6RrlxcLmb4r5n/Hi/8fxzCtb9wyWXcVNcICAMvQEp0N15Qd0xs0hhQZJOPvlknXzyyWXL7brrrtq0aVOkbS5cuDBSuah1jON991Rc281yzksCAkdQAtCvpOoGwC7OUQICR1AC0K8cffTRevrpp5OuhqTsvfEyBCUgaEwPACAIy5Yt05///Oey5TZv3qzXXnutD2oUgbvauywAwkJQAlJg0aJFeuKJJ8qWW7VqlW699dayM163tLRo3rx5ZWe8DmXfmUxGxx9/vKZOnVr21iRXXXWVpk+frj/+8Y9lt7t+/XrddNNN2r59e9mylXBJGfeCBUBYCEpACsycOVNTpkzRUUcdVTI43HjjjZoxY4YmTpyom266qWhoefHFF3XWWWeVnfE6lH0vWLBATU1NRSeR7LBhwwZde+21kqQLL7ywaLn169froosu0sSJE3XOOedo6dKlRcv2Vnt7e8ECIDBJz0/AwtKflqjzKJ177rmubIdDwWJmPmTIEJ8yZYq3tbX5vffe64MHD+627LBhw3z33Xf3JUuW+BtvvOHvfve7uy1XX1/vjY2NfuWVV7q7B7HvfB1zQnWULTXf09e+9jVvbGx0SUXne/rhD3/oQ4YM8YaGBpfktbW1BXWZOHGir1q1qirzKO27++5+z1e+UrCIeZRYWIJa6FECEmBl5ujpeN7MVFNT/DDtuOVGT7eZ9n3nW7hwYcE5R+vXr9dtt922U7mtW7fqu9/9bucw2rZt23TxxRdH2ke+mpqasmV6gqE3IHBJJzUWlv60VDoz94QJE3zw4MF+1FFH+RNPPLHT8x0uu+wyr6mpKTub9XPPPee1tbXe2NjoX/7yl33t2rVFt5n2fa9atcovuOCCzh6fCy64wF988cWdymUyGZ8zZ46feOKJneV++ctfdrvNN9980y+66CIfMmSI19bW+iOPPLJTmWr0KO2z++5+1+zZBYvoUWJhCWphZm6giiqdmfv+++/XLrvsosMPP7zk61avXq2HH35Yn/nMZ0rOZt3a2qrbbrtNH//4xzV27NiS2wxh35I0YsQIbdy4UeX+ZvVkZu7169frrrvu0qmnnqrGxsaC56oxM/e+u+/uV332swXrTrr2WmbmBgJCUAKqqNKghPLiCEqlVCMo7bP77v6d004rWHfy9dcTlICAMOEkgCCUux9cWnFeEhC2MP/yAIGL0lvRgd6nrFtvvVWbN28uW+6QQw7p9iRuqWe/92pwd7VlmJkbCBlBCUAQpk6dGqncbrvtpjlz5sRcm+gyBCUgaEwPAATmhBNO0JFHHlm23Pz589XQ0KA1a9aULLdhwwYNHz5cc+fOHVD77gsuqT2TKVgAhIWgBATkueee0wMPPKCnn35ajz32WNFymUxGF154odra2nT55ZeX3ObVV1+t5uZmfe1rXyt5e5D+tu++4O4EJSBwBCUgIBdddJGam5u1bdu2krfo6LjlR3t7u+bNm1e0d6Xjlh+tra3atm2bbr755gGx775EUALCRlACAvHcc8/pvvvu6zznZenSpd32rnT0qmzZskVS9l5jxXpXrr76arW1tUnKzmxdrGenv+27r7i7MplMwQIgLAQlIBBNTU0aOXJk5+MRI0Zo3bp1O5Vrbm5WY2Ojhg8fLkkaOXKkNm7c2O0233zzTe26666SspffDx8+XNu2bev3++5LbZlMwQIgLEw4CVRR1Akne6Lr9ABxTLxoZvrRj36kc845p2S50Pdd7ekByk04uefo0X7h8ccXrPvXW25hwkkgIEwPAAAx6TiZG0C4CEpAYJLsBR6o++6Ntvb2pKsAoBcISkBgTj/99G7P0elq/PjxkbfZ0NCg/ffff8DtO270KAHh4xwloIriOEcJ6VXuHKXxo0b5+cceW7Dukl/+knOUgIDQowQAcaFHCQgeQQkAYuLiHCUgdAQlAIgJ5ygB4SMoAUCMmGQSCBtBCQBi4u5qZegNCBpBCQBi4srecw5AuAhKABATd2foDQgcQQkAYuLuam1rS7oaAHqBoAQAMWF6ACB8BCWgilqkptekV5OuB/rMXiWfdScoAYEjKAFV5O5jkq4D0iPjrmaG3oCgEZQAICYucY4SEDiCEgDEhHmUgPARlAAgJlz1BoSvJukKAEB/lXHXjpaWgiUKM5tmZi+Y2QozuyjmagIogR4lAIiJu6s5YjjqYGa1kn4g6WOSVkt60sx+7e7LYqgigDIISgAQE3dXc2trT192uKQV7v6yJJnZHZJOlERQAhJAUAKAmLS3tWljU1NPXzZe0qq8x6slfbBqlQLQIwQlAIhJs/vvXmppGd1ldYOZLcl7PNfd5+Y9tm425dWvHYAoCEoAEBN3n1bBy1ZLmpj3eIKk16tTIwA9xVVvAJAuT0raz8wmmVmdpFMl/TrhOgEDFj1KAJAi7t5mZv8i6XeSaiXd5O7PJ1wtYMAyd4a+AQAAusPQGwAAQBEEJQAAgCIISgAAAEUQlAAAAIogKAEAABRBUAIAACiCoAQAAFAEQQkAAKCI/weLEHbidKTF2wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x720 with 12 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Expected Sarsa\n",
    "# Do not modify this cell.\n",
    "grid = Grid()\n",
    "def behaviour_policy(q):\n",
    "  return epsilon_greedy(q, epsilon)\n",
    "def target_policy(q, a):\n",
    "  greedy = np.eye(len(q))[np.argmax(q)]\n",
    "  return greedy - greedy*epsilon + epsilon/4 \n",
    "\n",
    "learned_qs = []\n",
    "for _ in range(5):\n",
    "  grid = Grid()\n",
    "  agent = GeneralQ(grid._layout.size, 4, grid.get_obs(), target_policy, \n",
    "                   behaviour_policy, double=False, step_size=step_size)\n",
    "  run_experiment(grid, agent, int(1e5))\n",
    "  learned_qs.append(agent.q_values.reshape(grid._layout.shape + (4,)))\n",
    "  \n",
    "avg_qs = sum(learned_qs)/len(learned_qs)\n",
    "plot_action_values(GRID_LAYOUT, avg_qs, vmin=-20, vmax=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "262f09c6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkoAAAI9CAYAAADFBFfgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABD/0lEQVR4nO3deZwcVb3///enZ8tkgSSGGEgCsriw6FfDoigR/aEmCohyr4LXBYFIUOQao3CDyCJcga9s97qgBgmLsiiQqwQRBFHgh0AWBAwI14AhiYSEISSZbLN0f75/dGfomUx31/R0TdWZeT0fj3pkuvp01elJ15x3n1N1ytxdAAAA2FEm6QoAAACkFUEJAACgBIISAABACQQlAACAEghKAAAAJRCUAAAASiAoAQAAlEBQAgAAKIGgBKAbM7vYzGYltO+FZrZ/EvsGgN4YM3MD2M7MdpH0hKR93H1rAvv/tKTj3P1fBnrfANAbepQAFPuipLuSCEkFd0j6oJntmtD+AaAbghIwxJhZxszOMrMVZvaKmX3FzDoKvUkflfRAj/J7mdmdZtZiZhvM7N4K23cz26fo8XVm9p9Fj5cX9v+Mmb1mZtea2TBJcvdtkpZI+kgt33NozGyema01s6VF68aa2b1m9vfCv2OKnjvLzJaZ2XNmNi2ZWgODE0EJGHrOlXSUpKmS9pH0BUmvuvsrkt4u6bke5W+Q9DtJbyws59egDp+VNE3S3pLeIunbRc/9TdL/qcE+QnadpOk91s2R9Ad3f7OkPxQey8z2k3S8pP0Lr7nKzOoGrqrA4EZQAoaQQq/RbEmfdfcX3X2DpN9K+muhyGhJrT1etrekOkl17r7N3R+uQVV+6O4r3X2dpO9K+kzRc62FegxZ7v6gpHU9Vh8j6frCz9dL+kTR+lvcvc3d/yFpmaRDBqKewFBAUAKGliMk/c3dlxete4NeD0qvSRrV4zWfVb4xfsnMrjGzsTWox8qin1+UtFvR41GS1tdgH4PNG919tSQV/h1fWD9R3X+fqwrrANRAfdIVADCgxkl6ZfsDM6uXdLSk7ecQPaX8UNii7WXc/X5J95vZeEl3KX/C9xVl9rFF0vCixxOUb7yLTS76eXdJLxU93lfSLyq/lfSbPn26t7S0dFu3ZMmSpyVtK1o1193n9mM31ss6LmcGaoSgBAwtz0q6wMz2VL7X5nuS9pK0/aThuyQdLulGSTKzY5XvbVqmfE/PGOWnD5CZXSdJ7v7FHvt4QtK/mdnTkj5c2N7iHmVOM7M7lQ9V35L0y8I2myQdKOmEfr/TFGhpadHixYu6rTPLbHP3g6rY3Boz29XdVxeuClxbWL9K3YPnJHUPngD6gaE3YAhx9/sk3SrpSUmPKR+CcpKeLhS5QdLHzKy58Pgw5a+Ca1U+RF1S6GGS8o1zb+crfU35Xqr1yg/b/bqXMjdJ+r2kFwrL9h6tj0v6k7sPkobe5Z7ttvTDHXo9QJ4g6TdF6483s6ZCAH6zpIX92RGA1zHhJDCEmdlHJP2ocCXV9nUXSVrr7v9V5nWNyoetd7h7Rx/3uVzSjEJo6/ncY5JOdvelO7wwQAcddKAvXPhIt3V1dU1LKvUomdnNkj6g/FDpGknnKR84f6X8UOUKSZ8qnAwvMztb0kmSOiXNcvff1fSNAEMYQ2/A0LavXj+RW5Lk7t+q9CJ3by+8tqbc/d213mbS3HNVvMY/U+KpI0qU/67yVw8CqDGCEjC07avXz09Cjbl7f4fbACSMoAQMYe5+agL7fNNA7zNZfe9RApAeBCUAiI1XNfQGID0ISgAQG4begNARlIAaqjNzDqqho1NS1r23CR+7EJSAsPE3HaiheuWnocbQ8HLFEi7OUQLCRlACgBhxjhIQNoISAMSE6QGA8BGUACBGBCUgbAQlAIgN0wMAoSMoAUCM6FECwkZQAoDYuCSCEhAyghIAxIihNyBsBCUAwbt6zpxI5b50ySUx16Q7rnoDwkdQAoAYEZSAsBGUACA2zMwNhI6gBACxceVynUlXAkA/EJQAIFYMvQEhIygBQEzyJ3Mz9AaEjKAEADHiZG4gbAQlAIiNy51zlICQEZQAIFYMvQEhIygBQGyYcBIIHUEJAGLD0BsQOoISkHJ3/vpHkcod9YnTIm9z9732ilTu0Le9LVK5Uc3Nkfc9esSISOXGRCzXF//55S9HLvvtH/+43/tzlzzH0BsQMoISAMTI3ZOuAoB+ICgBQFzc5VmCEhAyghIAxMhzBCUgZAQlAIgTQ29A0AhKABAXlzzLydxAyAhKABATlzP0BgSOoAQAcXHOUQJCR1ACgBgxPQAQNoISAMSI6QGAsBGUgATc+uP/G7lspqEuUrl7/nhN5G1O++DJkcqteOGFyNustee2vhy5bFPT+EjlHv7uBdVWpzoMvQHBIygBQGxcIigBQSMoAUBM3KUc0wMAQSMoAUCc6FECgkZQAoC4cI4SEDyCEgDEhpviAqEjKAFAXOhRAoJHUAKAmLgISkDoCEoAECOCEhA2ghIAxMVdzvQAQNAISgAQI3qUgLARlIAEbFq1MXLZ+hENkco1N46qtjqp1Nr6VOSynZ17RCrXMGZYtdWpjot5lIDAEZQAICYuZ2ZuIHAEJQCICz1KQPAISgAQI85RAsJGUAKAuLjknQy9ASEjKAFAbJweJSBwBCUAiAu3MAGCR1ACgJg4Q29A8AhKABAjepSAsBGUACAu7vIsQQkIGUEJSEBHZ2fkspufXx+p3NZVrVXWprQ999knUrlJ48ZF3uYuO+8cqVzLky9G3uaY/bdFKrf15U2Rt1kr9CgBYSMoAUBcOEcJCB5BCQBi4qJHCQgdQQkA4uIu515vQNAISgAQF4begOARlAAgRgy9AWEjKAFAXJgeAAgeQQkAYsLM3ED4CEoAECN6lICwEZQAIC5c9QYEj6AEAHFxyTsISkDICEpAArZ1dEQu29zYGKlcrg/nwtx00Xcilcs01UUqVze8IfK+60dEK5vrQ8BYt/TlSOX6cuuYWuGqNyBsBCUAiIlLyjpBCQgZQQkA4uKuXI6hNyBkBCUAiIlLyhKUgKARlAAgRjmG3oCgEZQAICbO0BsQPIISAMSIoTcgbAQlAIgJV70B4SMoAUBcGHoDgkdQAoCYcNUbED6CEpCAf7/iishl//vrX49UbnhTU+Rt1mUykco1NzRHKpdrz0bed64h2r63rd0UeZsb/r4uUrkZl1wSeZu14O4EJSBwBCUAiBHTAwBhIygBQExcUmc2em8bgPQhKAFAXNzpUQICR1ACgJhwMjcQPoISAMSI6QGAsBGUACAm7q5OghIQNIISAMSEoTcgfAQlAIgLM3MDwSMoAUBMmB4ACB9BCQBiwszcQPgISkDKfe3KK5OuAvqBoASEjaAEADFxd3Uw9AYEjaAEADHhqjcgfAQlAIgL5ygBwSMoAUBMXGLoDQgcQQkAYsJVb0D4CEoAEBOXlKVHCQgaQQkA4sK93oDgEZQAICYuqaOzM+lqAOgHghIAxMTduYUJEDiCElBD7VLLCunFpOuBAbNHuSddYugNCBxBCaghd98l6TogPdxdbR0dSVcDQD8QlAAgJgy9AeEjKAFATLjXGxA+ghIAxCTnrm3t7UlXA0A/ZJKuAAAMVu6u9s7Obkt/mdn5ZvZPM3uisHys6LmzzGyZmT1nZtP6vTMA9CgBQFzcXe3xnMx9pbtfVrzCzPaTdLyk/SXtJuk+M3uLuzP2B/QDQQkAYpJz15Zt2wZqd8dIusXd2yT9w8yWSTpE0iMDVQFgMGLoDQBikstmtaW1tdtSI181s6fMbJ6ZjSmsmyhpZVGZVYV1APqBHiUAiMm2XO6eZ1tbx/VYPczMFhc9nuvuc4sLmNl9kib0ssmzJf1Y0oXKz2d5oaTLJZ0kyXop79XWHUAeQQkAYuLu06t83YeilDOzqyXdWXi4StLkoqcnSXqpmv0DeB1DbwAQEDPbtejhJyUtLfx8h6TjzazJzPaU9GZJCwe6fsBgQ48SAITle2b2TuWH1ZZLmilJ7v60mf1K0jOSOiWdxhVvQP+ZO0PYAAAAvWHoDQAAoASCEgAAQAkEJQAAgBIISgAAACUQlAAAAEogKAEAAJRAUAIAACiBoAQAAFACQQkAAKAEghIAAEAJBCUAAIASCEoAAAAlEJQAAABKICgBAACUQFACAAAogaAEAABQAkEJAACgBIISAABACQQlAACAEghKAAAAJRCUAAAAShgyQcnMLjazWf14/XIz+1DEsm81s7+YWauZ/XuJMgvNbP9q6wOkQdTjis87QtTfdqPENp82sw9ELBu53SmU5ziLwZAISma2i6QvSPrpAO3yTEl/cvdR7v79Eh/2yyRdMED1AWquj8cVn3cEJa52w933d/c/1WJbvbQtHGcxGBJBSdIXJd3l7lsHaH97SHq6Qpk7JH3QzHYdgPoAcfiioh9XfN4Rmi+qhu2GmdXXYjsVcJzFYNAEJTPLmNlZZrbCzF4xs6+YWUfhW8FHJT3Qo/xeZnanmbWY2QYzu7eP+9vNzG4v7Osf24fYzOx+SR+U9EMz22RmN0vaXdKCwuMzJcndt0laIukj/X/34TCzeWa21syWFq0ba2b3mtnfC/+OKXruLDNbZmbPmdm0ZGo9NJnZzmbmZjauaN1+ZrbGzHZSj+Oq3DE1VD/v1eI4GRhxtxuFHp//MLOnJG02s/riXiAzm1J0msatZvZLM/vPHpt5p5k9VdjfL81sWOG1P1ePtmUoHWcDeYwMmqAk6VxJR0maKmkf5btMX3X3VyS9XdJzPcrfIOl3kt5YWM6PuiMzy0haIOlJSRMlHSFplplNc/f/T9JDkr7q7iPd/TOSVkg6uvD4e0Wb+puk/9PXNxq46yRN77FujqQ/uPubJf2h8Fhmtp+k4yXtX3jNVWZWN3BVHdrcfYOkVZL2K1r9XUn/1903asfjqtIxNRQ/79W6ThwnA2Eg2o3PSDpS0mh379y+0swaJf2P8v/XYyXdLOmTvbz+08r/v+4p6R3K93TJ3T+v3tuWoXKcXacBOkYGRVAqpP/Zkj7r7i8W/sD/VtJfC0VGS2rt8bK9JdVJqnP3be7+cB92ebCkXdz9Andvd/cXJF2t/H9EX7QW6jZkuPuDktb1WH2MpOsLP18v6RNF629x9zZ3/4ekZZIOGYh6ostSSftKkpkdImmKpB8Vnhut7sdVpWNqyH3eq8VxEr8BbDe+7+4rexnCe4+k+sLzHe4+X9LCEq9/yd3XKf8F/Z0V9jckjrOBPEYGRVBSvkfnb+6+vGjdG/T6B/41SaN6vOazyv/yXjKza8xsbB/2t4ek3cxs/fZF0reU/4bRF6Mkre/jawajN7r7akkq/Du+sH6ipJVF5VYV1mHgLNXrPUoXSzrf3dsKj3seV5WOKT7v/cNxUlsD1W6sLLF+N0n/dHevUPblop+3SBpZYX9D+TiL5RgZiJPLBsI4Sa9sf2D5k+aOlrR9rPcpSW+RtGh7GXe/X9L9ZjZe0l3Kd2deEXF/KyX9o9C9F4WXWL+vpF9E3EaqTJ8+3VtaWnZYv2TJkqclbStaNdfd51a5G+tlXanfJeKxVNLnCudU7Kr80MN23Y6rCMdUsJ/3ak2fPtlbWrZ1W7dkSUstjxGJ46RaA9VulPq/WC1poplZUViaLOn5PryH3rYd1HEWQlsyWILSs5IuMLM9lU/S35O0l/J/5KX8B/pwSTdKkpkdq/y3hmXKp+8xkp4oPHedJLn7F8vsb6GkjWb2H5K+L6ld+Q9ns7sv6qX8mkJ9uphZk6QDJZ3Qh/eZGi0tLVq8eMe3apbZ5u4H9XFza8xsV3dfbfmrNdYW1q9S/g/HdpMkvVRVhVGt7T1KF0n6lrtni57rOq7KHVNS+J/3arW0bNOixcd2W5exudUcIxLHSa0NdLvR0yOSspK+amY/Vv48pkMk/akP2+jWtoR4nIXQlgyKoTd3v0/SrcqfXP2Y8h/mnF6/RP8GSR8zs+bC48OUv5qhVfmD4ZLCNwUp/8ssO+5caCyOVn6s+B+SWiT9TNLOJV5ysaRvF4bpvllY93Hl51oK9g+ae26HpUp36PUD+wRJvylaf7yZNRX+mL1ZvY/hIz7PSJogKevuv+7xXPFxVe6YkgbB571aOe++9APHSQ0NdLvRy/7bJR0r6WTlg9rnJN0pqa3My3rq2bYEeZylvi1x90G3KH9p5N97rLtI0qwKr2tU/oqBhgGo42OSDkj6d1XtcuCBUzybbd9hkbS4wvu+Wfku5w7lU/7Jyp8X8AdJfy/8O7ao/NnKd0U/J+mjSb9vlh3+PyseV4VyQX/eq12mHDjO27OndFsqHSOF3xfHyQAvaWg3CsfJif18fVDHWQhtiRU2MKiY2dckHe7ux1YsjKocdNCB/thjO36Bqq9vXuLVDSsAg86BB+7iDz/W/c9Qc8NcjpEUSqLdMLPDlW+4W5Q/UfwnkvbywgnJQ0EIbclgOUepp331+jgzYuHK91IDKMXV7+E2DJwk2o23SvqV8leyPS/pX4dSSMpLf1syKIOSu5+adB0GO/f8uDKA8ghKYUii3fD8VVz9ueIxeCG0JYMyKGEguLpfAAWgJ5eUJSgBZaS/LSEooWpp/xYApAE9SkB5aW9L+hSU6sycZDV0dErKuvc2UZcK35UHsjpBsOHDXKN7TuaLQWt9q3zLthLHSH5YgaC0I9qSoaVdanH3XXp/Nv1tSZ8+q/XKT6iCoeHlss+mv7s0EaNHSadwseWQMXd+xSLZdH9ZTgRtydCyQnqx9LPpb0sI9aha2rtLgaRx1RtQWdrbEoISqhLClQpAGhCUgNJCaEsISqiSy70z6UoAqebOVW9AeelvSwhKqJKn/lsAkAb0KAHlpL8tISihH9L94QaSxjlKQBTpbksISqha2r8FAGlAUALKS3tbQlBCldI/rgwkzZ3pAYDy0t+WEJRQFff0jysDaUCPElBaCG0JQSkAV8+ZE6ncly65JOaa9JTuScIGi0s/HK3cGffGW480+9T+0crd+nS89egNQQlpkN52REp7W0JQQpXSP5sqkDRuigtUkv62hKCEqqX9ww0kjnu9ARWlvS0hKKFK6R9XBpLG9ABAJelvSwhKqEr+BLx0X6kApAFXvQGlhdCWEJRQtbR3lwJJo0cJqCztbQlBCVVypX02VSBxnKMEVJD+toSghKqlvbsUSBpXvQGVpb0tISihSum/pBNIA3qUgHLS35YQlFCl9H+4gaRxjhJQSfrbEoISquKe/hsZAmlAUAJKC6EtSTwo3fnrH0Uql2moi7xNq8/UdJtWZ5H33Th6WKRyo3abEHmbj1/++0jlfn3Tf0Xe5if+bVbksqV4Lt0f7jSLelsSSRrWEK3c9z8afZtRD6eRjdHK7TUm+r6jlp0wMvo2P31rtHJnvz/6Nr/7YPSypXBT3P656+6fRi5bH/FAcY+WXPvS5jREPFBG7vbGaOVG7ht5383Nu0Yq9/uzzopU7tLTT4+87zN+8IPIZctJe1uSeFBCqFzOV2WgIg4ToJz0tyUEJVSHky+ASDhMgDICaEsISqha2r8FAEljegCgsrS3JQQlVMfT/+EGEseEk0B5AbQlBCVUxZX+DzeQtABGFYBEhdCWEJRQHXc5l/MAFXGYAGUE0JYQlFC1qJfZAkMVPUpAZWlvSwhKqB4tAFAe5ygBlaX8ICEooTou5bicByiLq96ACgJoSxIPSh2bOiKVa9gp+uzYRx15arXVGTDf/MIXIpc9avb0SOXqhg3kfydflftjVWv0srvvHK1cQ7QJ6SVJX74zetmkrPlm9LK//Ndo5eb/rbq69AeHSfVan301ctmd9hsXqdxHP3JKtdVJpb9tfjFSuUkf3CtSuRW3LuxPdaqQ/rYk8aCEMHkAl3QCSeMcJaC8ENoSghKqlvYPN5AGHCZAeWlvSwhKqI4r9Zd0AknjprhABQG0JQQlVCn9NzIE0oDDBCgn/W0JQQlVS/uHG0ga5ygBlaW9LSEooSrukqf8kk4gDThMgNJCaEsISqha2r8FAEnz9F/5DCQu7W0JQQnVoQUAIuEwAcoIoC0hKKFquZRfqQAkzcVVb0AlaW9LEg9Krz2zNlK55p2bY67JwHpl48bIZTevila2aewA/o44S7Vf1m6OXrY54lE6fkR1dUmr/10XvWxre7Ryr26tri79wWFSva2vRD9Q6v6ReHOWiFdXRZtJu2NTtIMk+j0waiSAtmRofrJQE2kfVwYSl/5RBSBxaW9LCEqoirunfpIwIGkMvQHlhdCWEJRQtbR/CwDSIN1NAJC8tLclBCVUJ4C5L4CkBXD6BZCsANoSghKqlvZvAUAacJgA5aW9LSEooToBjCsDSeOmuEAFAbQlBCVUxZX+7lIgDVL+ZRlIVAhtCUEJ1fH0d5cCSeMcJaCCANoSghKq4y7vTHd3KZAGKf+yDCQrgLaEoISqpf1bAJC0AG5jBSQu7W1J4kHJPdovqDPi9OuS9IsLzotULtNYF61cQybyvuuGN0QqVz+qKfI2s1uivfe2iL/LmnDJc+n+FpBmfTl38eVN0cq1bIm+zVMPilauKeJfiDHDou97p4gf/Yh37pEktXVGK7eOW5gEpTObjVx225potzu56ZILIpWL4+9+07jhkco1jo5+QGXbo/2O2iLeN6m9M+LBVCsBtCWJByWEySV5Jy0AUA4zcwPlhdCWEJRQnQAu6QTSgB4loIwA2hKCEqpHCwCUxzlKQGUpP0gISqiOS7mUX6kAJI2hN6CCANoSghKq4576ScKANEj5l2UgWQG0JQQlVC/lH24gaS4p3d+VgRRIeVtCUEJV3JX6E/CANKBHCSgthLaEoITquKd+XBlIGjfFBSoIoC0hKKFqaR9XBtKAHiWgvLS3JbEEpfe9+92Ry27YEm064Z2amyNvs0nRZkl1s2jl6qPP0JprizZLajYTfabx9lejzZTauTH6NvvNlfpx5TTr6MMXqMi/5j5sM/L/XMRJel/bFn3fEQ8R9eVL5kut0cr976vRt1kL3BS3f9r7MDN3e2vEtmR4tLbEs9Hu3CDV/hYc2agHiaTNK6NNYb/hxdcilRvW2Bh53zURQFtCjxKq4pI6Uz7tPJAGKW8DgESF0JYQlFAdd+VS/uEGksZNcYEKAmhLCEqoSn5IgRYAqISgBJQWQltCUELVsin/FgAkjZm5gcrS3pYQlFAVD6C7FEgDepSA0kJoSwhKqFrau0uBpHGOElBZ2tsSghKqEsKVCkAacNUbUFoIbQlBCdUJoLsUSJpL4jABygigLSEooSohXKkApAFDb0BpIbQlsQSlhx97LHLZfXbdNVK5d77pTZG3GXU21/r2aDNu122JPjN3XSZa2UzEcpJU3xztv+mz55wfeZu1kPZvAWn2P3+LXvYTb4tWrjH6RMKqi/jxGxVxkt6+zKLdHnHS4ZZoEy1Lkl7dGq3ctX+Jvs1a4Sip3lcuvTRy2R9985uRym1ri3YHg7rOPszMvTlaQ59ZH20K+760Dx4xZDz30kuRyl16ww2R910raW9L6FFCVdw99ePKQNK4KS5QXghtCUEJVUv7twAgDRh6A8pLe1tCUEJV3D31k4QBSeOmuEB5IbQlBCVUJYRLOoHEOdMDAOWE0JYQlFC1tHeXAkmjRwmoLO1tCUEJVXF3ZVN+SSeQBgQloLQQ2hKCEqrikjojTsMADFXcFBcoL4S2hKCE6gQwmyqQOO71BpQXQFtCUEJV8t+U0/3hBpLGOUpAeSG0JQQlVMXdU99dCqQBV70BpYXQliQelK7/9a+jlYu3GqhC2r8FDBa/fjbpGqBa9CgNnNMuuyzpKqBKaW9LEg9KCFMI3aVAGhCUgNJCaEsISqhOALOpAolzicMEKCOAtoSghKq4uzpSPq4MJM3FOUpAOSG0JQQlVCWE7lIgDRh6A0oLoS0hKKEqIXy4gaRxMjdQXghtCUEJVXF3dXR2Jl0NIN24KS5QVghtCUEJ1XFP/R2fgaTRowRUEEBbQlBCVUK4Pw+QBgQloLQQ2hKCEqoSQncpkDRuiguUF0Jb0qeg1C61rJBejKsySJ09Sj3hUuq7SxOxuqVF35nLMTJ0lDxGJHFT3BJoS4acoNuSPgUld98lroogLCHMfZEEjhEU4xyl3nGcYLsQ2hKG3lAVd1d7R0fS1QBSj6vegNJCaEsISqhKCN8CgKTRowSUF0JbQlBCVVxK/Ql4QBoQlIDSQmhLMklXAGHKuau9s3OHpT/M7Hwz+6eZPVFYPlb03FlmtszMnjOzaf1+A8BAKJzMXbz0F8cJBpMQ2hJ6lFAVz+W0ta0tjk1f6e6XFa8ws/0kHS9pf0m7SbrPzN7i7unur8WQF+P0ABwnGBRCaEvoUUJVcu5q6+jYYYnJMZJucfc2d/+HpGWSDolrZ0At1bpHqQyOEwQnhLaEoISqeC6nbVu27LDUwFfN7Ckzm2dmYwrrJkpaWVRmVWEdkGrbT+aOIShxnGBQCKEtYegNVdmazd7z11dfHdfLU8PMbHHR47nuPnf7AzO7T9KEXl53tqQfS7pQ+fblQkmXSzpJkvVSnlNkkX4vtdyTPW9uz+Ok7DEicZxg6AihLSEooSruPr3K130oSjkzu1rSnYWHqyRNLnp6kqSXqtk/MJA4ToDyQjhGGHpDapjZrkUPPylpaeHnOyQdb2ZNZranpDdLWjjQ9QPSgOMEKK/Wxwg9SkiT75nZO5XvCl0uaaYkufvTZvYrSc9I6pR0GlfyYAjjOAHKq+kxYu4MYQMAAPSGoTcAAIASCEoAAAAlEJQAAABKICgBAACUQFACAAAogaAEAABQAkEJAACgBIISAABACQQlAACAEghKAAAAJRCUAAAASiAoAQAAlEBQAgAAKIGgBAAAUAJBCQAAoASCEgAAQAkEJQAAgBIISgAAACUQlAAAAEogKAEAAJRAUAIAACiBoAQAAFACQQkYYszsYjObVcPtXWdm/1mr7UXc50Iz238g9wlgaCIoAUOIme0i6QuSfpp0XfrpMkkXJF0JAIMfQQkYWr4o6S5335p0RfrpDkkfNLNdk64IgMGNoAQMMmaWMbOzzGyFmb1iZl8xs45Cb9JHJT3Qo/xeZnanmbWY2QYzu7fC9t9lZo+bWauZ/VLSsB7P72tmfzKz9Wb2tJl9vLD+RDNbUFRumZn9qujxSjN7Z+Hn5Wb2TTN7qlCnX5pZ137cfZukJZI+Uu3vKc3MbJ6ZrTWzpUXrxprZvWb298K/Y4qeO6vw+3zOzKYlU2tgcCIoAYPPuZKOkjRV0j7KD7W96u6vSHq7pOd6lL9B0u8kvbGwnF9qw2bWKOnXkn4uaaykWyX9S9HzDZIWSPq9pPGSTpd0o5m9VfmANrUQ5HaV1CDpfYXX7SVppKSninb3aUnTJe0p6R3K94YV+5uk/1P+VxGs65R/78XmSPqDu79Z0h8Kj2Vm+0k6XtL+hddcZWZ1A1dVYHAjKAGDSKHXaLakz7r7i+6+QdJvJf21UGS0pNYeL9tbUp2kOnff5u4Pl9nFe5QPOP/l7h3ufpukRT2eHynpEndvd/f7Jd0p6TPu/kJh3++UdLikeyT908zeVnj8kLvnirb1fXd/yd3XKR++3tmjLq2F9zPouPuDktb1WH2MpOsLP18v6RNF629x9zZ3/4ekZZIOGYh6AkMBQQkYXI6Q9Dd3X1607g16PSi9JmlUj9d8VvnG9iUzu8bMxpbZ/m6S/unuXrTuxR7Pr+wReF6UNLHw8wOSPiDp/YWf/6R8SDpcPYYEJb1c9PMW5QNYsVGS1pep62DzRndfLUmFf8cX1k+UtLKo3Cq9/vsG0E/1SVcAQE2Nk/TK9gdmVi/paEnbL99/StJbVNQLVOj1ud/Mxku6S/khritKbH+1pIlmZkVhaXdJzxd+fknSZDPLFIWl3SX9b+HnBwr12VPSRcoHnc9KOlTSD/v4XveV9Is+vmZATZ8+3VtaWrqtW7JkydOSthWtmuvuc/uxG+tlnfeyDkAVCErA4PKspAvMbE/lQ8j3JO0laftJwXcp33tzoySZ2bHK9zYtU76HZoykJwrPXSdJ7v7Fou0/IqlT0r+b2Y8kfVz5YZ4/Fp5/TNJmSWea2eXKn4N0tKSDC88/oHwIW+Puq8xso/LnO9VL+kvUN2lmTZIOlHRC1NckoaWlRYsWPdZtXSZTv83dD6pic2vMbFd3X104x2ttYf0qSZOLyk1SPrACqAGG3oBBxN3vU/4E6yeVDy1/lZST9HShyA2SPmZmzYXHhykfXlqVD1GXFHqYpHzj2+18JXdvl3Ss8r1Or0k6TtL8Hs9/XPmr61okXSXpC+7+bOH5/5W0SdJDhccbJb0g6WF3z/bhrX5c0p/cPeWBwOWe67b0wx16PRieIOk3ReuPN7OmQkB+s6SF/dkRgNdZ91MNAAwmZvYRST8qXCm1fd1Fkta6+3+VeV2j8mHrHe7eEXtF+8jMHpN0srsvrVg4QQcddKAvXPjnbuvq6oYtqdSjZGY3K38u1zhJaySdp/zVhr9SfihzhaRPFU50l5mdLekk5Xv7Zrn772r6RoAhjKAEDGJm9jVJh7v7sUnXZSg66KAp/thj/3+3dfX1IyoGJQDpwTlKwOC2r14/PwkDzF39HW4DkDCCEjCIufupSddhaHOCEhA4ghIAxIqgBISMoAQAsXG5dyZdCQD9QFACaqjOzDmoho5OSVn33iZ87MLQGxA2/qYDNVQvaULSlcCAebliCc5RAkJHUAKAmOSveuvLPJoA0oagBACxoUcJCB1BCQBiRY8SEDKCEgDExhl6AwJHUAKAGDH0BoSNoAQAsaFHCQgdQQkAYuJOUAJCR1ACUu6uu38aqdzHps+MuSbpdfWcOZHKfemSS2KuSW8ISkDICEoAEBumBwBCR1ACgBgx9AaEjaAEALHhHCUgdAQlAIgJtzABwkdQAoDYuCTOUQJCRlACgNi43DuTrgSAfiAoAUCMGHoDwkZQAoDYcDI3EDqCEgDEhqE3IHQEJQCISf6qN07mBkJGUAISMP+6yyOX9Y5oDe3d910deZv/MfsHkcrtOWFCpHK7jRkTed9NDQ2Ryn3uyi9F3ubjl/8uUrmotzqRani7E/fabAdAIghKABAbVy5LjxIQMoISAMTFJeXoUQJCRlACgBg5QQkIGkEJAGLknKMEBI2gBAAxcZc8S1ACQkZQAoDYOENvQOAISgAQF07mBoJHUAKAGHmO6QGAkBGUACAuzlVvQOgISkAC1j37SuSy494RbXbsvlxd9eRTT9W0XByO/96nI5fd51MHRCq35k/Lq6xNtThHCQgdQQkAYsJVb0D4CEoAECN6lICwEZQAIEYEJSBsBCUAiIu7nJviAkEjKAFAjOhRAsJGUAKAuDDhJBA8ghIAxMRFjxIQOoISAMTFnekBgMARlAAgRtzCBAgbQQkA4sItTIDgEZSABJhZ5LKtf3s12jYbMtVWJ5XWProictm6YQ2RyuU6Br53h6E3IGwEJQCIiTv3egNCR1ACgBhxjhIQNoISAMTFJe+kRwkIGUEJAGLE0BsQNoISAMSFc5SA4BGUACAmLsk7OUcJCBlBCQDiwjxKQPAISgAQG5e46g0IGkEJAOLiTDgJhI6gBCRg49atkcvu1NwcqVxTXWPkbd500Xcilcs01UUqVzci2szYkpRpiLbN7LbOyNtse2VLpHKb10crVzMu5bL0KAEhIygBQJzoUQKCRlACgJi4O0NvQOAISgAQI2foDQgaQQkA4sLJ3EDwCEoAEBeCEhA8ghIAxMaZmRsIHEEJAGLizjlKQOgISgAQF4begOARlAAgNq4cQ29A0AhKABAXFxNOAoEjKAEJ+NXDD0cu+8n3vCdSuZ2z0W51Ikn1bdFuI9LUHu22KNkt0W83khkWbd9Rb3UiSVte2Ryp3MqWlsjbrAWXlOOmuEDQCEoAEBd3ZZ0eJSBkBCUAiIlLytKjBASNoAQAMWLoDQgbQQkAYuLuyjH0BgSNoAQAMWLoDQgbQQkAYsJVb0D4CEoAEBeuegOCR1ACgJhw1RsQPoISAMSIoTcgbAQlIAGPLFoUuexbJk6MVO76X/+6ytr039FHHBG57NhRoyKVG9HUFHmbY0aOjFTuu9dcE3mbteAMvQHBIygBQIyy2WzSVQDQDwQlAIiJu3OOEhA4ghIAxMQlJpwEAkdQAoAY0aMEhI2gBAAxcXd1EpSAoBGUACBGTA8AhI2gBAAxYcJJIHwEJQCIiburk+kBgKARlAAgRvQoAWEjKAFATJhHCQgfQQlIuSRvTRLVgj/8IekqpJJLDL0BgSMoAUBc6FECgkdQAoCYcNUbED6CEgDExN3VwdAbEDSCEgDEhB4lIHwEJQCIC7cwAYJHUAKAmLikLENvQNAISgAQE3dXe2dn0tUA0A8EJQCICRNOAuEjKAFATJhwEggfQQmooXapZYX0YtL1wIDZo9yT7q42ht6AoBGUgBpy912SrgPSw93pUQICR1ACgJi4pA56lICgEZQAICburraOjqSrAaAfCEoAEJMc0wMAwcskXQEAGKzcXR2dnd2W/jKz883sn2b2RGH5WNFzZ5nZMjN7zsym9XtnAOhRAoC45HI5bd62LY5NX+nulxWvMLP9JB0vaX9Ju0m6z8ze4u6cTQ70A0EJAGLiuZy2bdkyULs7RtIt7t4m6R9mtkzSIZIeGagKAIMRQ28AEJNcLqctra3dlhr5qpk9ZWbzzGxMYd1ESSuLyqwqrAPQD/QoAUBM2tzveb69fVyP1cPMbHHR47nuPre4gJndJ2lCL5s8W9KPJV2o/OwDF0q6XNJJkqyX8l5t3QHkEZQAICbuPr3K130oSjkzu1rSnYWHqyRNLnp6kqSXqtk/gNcx9AYAATGzXYseflLS0sLPd0g63syazGxPSW+WtHCg6wcMNvQoAUBYvmdm71R+WG25pJmS5O5Pm9mvJD0jqVPSaVzxBvSfuTOEDQAA0BuG3gAAAEogKAEAAJRAUAIAACiBoAQAAFACQQkAAKAEghIAAEAJBCUAAIASCEoAAAAlEJQAAABKICgBAACUQFACAAAogaAEAABQAkEJAACgBIISAABACQQlAACAEghKAAAAJRCUAAAASiAoAQAAlEBQAgAAKIGgBAAAUAJBCQAAoASCEgAAQAkEJQAAgBIISgAAACUQlAAgZcxsnpmtNbOlSdcFGOoISgCQPtdJmp50JQAQlAAgddz9QUnrkq4HAIISAABASfVJVwAABqvp06d7S0tLt3VLlix5WtK2olVz3X3ugFYMQGQEJQCISUtLixYtWthtXSZTt83dD0qoSgD6iKAEALFxSbmkKwGgHzhHCQBi5J7ttkRhZjdLekTSW81slZmdHGslAZREjxIAxMbl3vceJXf/TAyVAVAFghIAxMRdVQUlAOlBUAKA2Hjk4TYA6URQAoBY0aMEhIygBACxqe4cJQDpQVACaqjOzDmoho5OSVl3K1fGvXOAagMgDvxNB2qoXtKEpCuBAfNyxRL0KAGhIygBQKwISkDICEoAEBN3Vy7HVW9AyAhKABArghIQMoISAMSGc5SA0BGUACBGBCUgbAQlAIiNMz0AEDiCEgDEih4lIGQEJSDlXnTv9viVV15Re3u7Jk6c2G39HlZ23sNBrefvqJSB/h25c44SEDqCEhCYD3/4w3rppZe0du3apKuCihh6A0JHUAICs3z5cm3YsCHpaiAid6YHAEKWSboCAFBLt912m4YNG6bVq1cnXRVtnx6geAEQFoISEIj29nYtXry4qzdp8eLF2rp1a8K1SpdcLqczzjhDnZ2duvDCC5OujtzzN8UtXgCEhaAEBGL+/Pl6z3veo0wmf9geeuihuvbaaxOuVbrMnz9fLS0tymazuvbaa1PQq+TKX/VWvAAICUEJCMT06dM1bNgw5XL5xra+vl7HHntswrVKj+29SZs2bZIkZbPZlPQqZbstAMJCUAICMXr0aM2ePVvNzc1qbGzUSSedpAkTJiRdrdRoa2tTc3OzRo0aJUkaM2ZMCk56d4begMARlICAzJ49W5lMRplMRuecc07S1UmV5uZmPfPMM7r99tslSWvWrNGNN96YcK2cHiUgcEwPAARk9OjRuuKKK7RhwwZ6kwLBlW5A2AhKQGBOOeWUpKuAyFwSvUhAyAhKQAKi3nIj6W2GovjWJB29rNuuL7+jWtzuxF3yHD1KQMgISgAGlQZJuyddiSKeG7oBFhgMCEoAEBd3ghIQOK56A1Lg1FNP1emnn641a9aULXf33XfriCOO0KOPPlq2XEtLi97//vfrmmuuUUdHR9myg2nffZHNZnXzzTdr6tSpWrlyZc2225Nnc90WAGEhKAEp8Nvf/lY//elPteeee5YNDo8++qjuv/9+HXHEEZo6dWrJ4LBmzRotXLhQs2bN0qRJk8qGlsG07yi2B6Q999xTp5xyihYvXqwXX3yx6u1V4u7dFgCB6XkQs7CwVL80Sr57hGXUqFGu/CVROyx1dXXe2Njo7e3tfvHFF5cst325/fbb/dlnny1bJpPJ+DHHHOPuPqj2vV2U3/nukru7n3jiiZ7JZMpue9GiRZG215jfZsnPwzsO2NtXPX97t0XS4qQ/pywsLNEXzlECEvDggw9q2bJlXY9POukktba2auTIkRo5cqQuvvhi1dfXa8aMGdpnn326yt1www1asGCBmpqalMlkNHPmTH3gAx/QmDFjtGDBAm3btk2StHz5cp133nnaunWrmpubdfDBB+vcc88ddPuuxje+8Q2tXr1aDzzwgNra2jR8+HDNnj1bb3/72yXlbw3zrne9q6pt9+Ry5bL0IgFBSzqpsbAMpiVqj1JPBxxwgE+YMMGvvfZa7+jo2OH57S6//HJvaGjwWbNm+dq1a0uWe/75572urs4PP/xwX7hwYclyg2XffelR2u7JJ5/06dOneyaT8ccff7yqbVbqUXr7/nv5imdv7baIHiUWlqAWc+fbDlArTWYeZb7snvP5bN68WU1NTaqvL9/J6+7auHGjdt5554r72LBhQ6Ryg2HfUec86m0epVJ1jbLNlyW1uZcs+I4D9vY7b7uk+3b3/fQSdz8oQnUBpABDb0AKjBgxIlI5M4sUQCRFLjfY9t1XcW1XkuTiSjcgcAQlYAiqxazTcevLLNrpnZXcmUcJCBxBCQBi4hJBCQgc8ygBKXHPPfeopaWlZtvbuHGj7rzzzpptL2nPPPOM/vKXv1Qsl6r37fmgVLwACAtBCUiBpUuXavr06V2X0ZeycuVK3XjjjRVnvJakSy+9VEcffbT+/Oc/16qaicnlcjryyCM1bdo0tbe3ly3bl/e9bt06zZs3T1u3bq1VVXfgWe+2AAgLQQlIgTlz5iiTyejaa6/V6tWrS5a75pprdMIJJ2jy5MmaN29eycC0fv16XXnllZKkM844I5Y6D6T58+erpaVFW7Zs0fXXX1+yXNT3vW7dOs2ZM0eTJ0/WzJkztWTJkprXWVJ+7C3n3RcAQSEoAQlbunSp7r//fuVyOeVyOV144YVdz919991qbGyUmcnM9J3vfEfZbFZr1qzR1772NU2ePLnXRv7yyy9XLpe/2uqJJ54Iulcpl8vpjDPO0KZNm7R582adffbZJXuVorzvn/zkJ5o8ebL++7//W1u2bJG7a+rUqV2/4913312rVq2qUe2doTcgcAQlIGHnnHNO16zW7e3tuvrqq7vOVTIzZTKlD9PtjXuxzZs367LLLusaTtqyZYvOOuusmGofvwULFmjFihVdj9etW6ebbrpph3JR33fP31dPmUymYpnICtMDcFNcIFwEJSBhM2bM0OzZsyVJ7373u3X22Wdr7NixkqRp06Zp27ZtXTPEnnfeecpkMpowYYJ+8IMfaOXKlZoyZUq37Q0fPlznnnuujjnmGEn5W3acfvrpA/umaujAAw/U17/+9a7Hs2bN0mGHHbZDuajve+bMmVq5cqVmzZql4cOHS5Ieeuihrt/x8uXLNXHixJrUfftVb/QoAeFiZm6ghqqdmVvK93T85Cc/0cyZM0u+btWqVXrwwQf16U9/uuJs1vfee68+8pGPqLdjPMR5lEaPHq0NGzb0+n6KlXvfPa1bt06/+c1vdPzxx6u5ubnbc7WYmfuAt77Jb597Xrd1b/vASczMDQSEeZSAgEyaNEn/9m//lnQ1Bo2xY8fqxBNPjHUfDLcBYSMoAQhCpR60VHImnARCF+BfHmBwuuCCC3TkkUfWbHtTpkwJ+iTunm688Ua1trZWLJe2901QAsLGOUpADfXnHKWBFOI5SgOtFuco7f/mPfxX3z+727oDPjaTc5SAgHDVG5ACmzZtijTbtiQdddRReu9731ux3G233aZhw4aVncAyJHG97/Xr1/ezZuV5LtdtARAWghKQAoceemjF2bal1yenfPLJJ/Xwww+XLLd9ksbOzs5uE1iGKo73/cQTT2jatGl6wxveEOkectViegAgbAQlIAGPP/64fvnLX3Yty5cv7zbb9rx583q9vH3OnDlqa2vTli1byt6iY/stP7LZbMXbooSglu976dKlmjZtmt773vfqvvvu0/DhwzV//vyu/4v58+crm83WpuIuKevdFwBB4RwloIainqP02qhRJU9Mrq+vV11dnVpbW9XQ0NC1funSpZoyZUpXj1NjY6Puv/9+ve997+v2+lwup7333lvLly+XJDU0NGjGjBm66qqrusqEdI5SLd+3JJ188sm67rrrum510ptFixbpXw4+uGIdK56jtPfuftP3/qPbunf+61c5RwkICD1KQAI2btzYNRO0u2vSpElqaGhQc3OzTjvtNK1YsaJbSJKklpYWjRkzpuvx6NGjtXbt2h223dbWpubmZo0aNUqSNGbMGG3YsCHeNxSjWr/vuXPn6qabbtLuu++ukSNHatiwYd1m5nZ3HXRQDXNMLtd9ARAUepSAGqr2qrevfOUramxs1Le+9S2NHz++7GtrMUN1SD1K29V6Zu5sNqvbbrtNV111lX7xi19o8uTJ3Z6vxVVv++21u9900Te7rXvXZ75GjxIQEOZRAlKg5/AQ4ldXV6fjjjtOxx13XIx7cWbmBgJHUAICM1R7gYN83y45J3ADQSMoAYH5/Oc/3+s5Oj1NnDix5HNJT+ZYjVq87yQwJQAQNoISEJgf/vCHkcrtt99+YfbClBDk+3bJOxl6A0LGVW8ABpVnnnlGxx13nNrb25OuSv4qumz3BUBYCEpACpx66qk6/fTTtWbNmrLl7r77bh1xxBF69NFHy5ZraWnR+9//fl1zzTUVb40ymPYtSWeeeaZuu+02XX/99WXLZbNZ3XzzzZo6dapWrlxZcbvVYmZuIHDFc4ewsLD0b2mUfPcIS0+TJk3yhoYGb25u9q9+9av+8ssv71DG3f28885zST58+HA/7LDD/JFHHum13NKlS72pqclHjhzp48eP95/97Gfe3t7ea9nBtO+//vWv3tzc7JJ8l1128ba2th3KdHZ2+k033eSTJ0/2kSNH+rBhw/yhhx7aoVyU/8fG/P9lyc/D2yZP9Ee/f3G3RdLicq9hYWFJ15J4BVhYBtMSNSiNGjXKlb/BxQ5LXV2dNzY2ent7u1988cUly21fbr/9dn/22WfLlslkMn7MMce4uw+qffd05JFHeiaTcUk+YsQInzt37g5lTjzxxK4ypZZFixbVJihNmuiPXHlRt4WgxMIS1sLJ3EACHnzwQS1btqzr8UknnaTW1laNHDlSI0eO1MUXX6z6+nrNmDFD++yzT1e5G264QQsWLFBTU5MymYxmzpypD3zgAxozZowWLFigbdu2SZKWL1+u8847T1u3blVzc7MOPvhgnXvuuYNu38U2b96s3//+92pubtbmzZuVzWb1s5/9TF/60pe6lfvGN76h1atX64EHHlBbW5uGDx+u2bNn6+1vf7uk/C1k3vWud/X9P7VXzjxKQOiSTmosLINpqXbo7YADDvAJEyb4tdde6x0dHTs8v93ll1/uDQ0NPmvWLF+7dm3Jcs8//7zX1dX54Ycf7gsXLixZbrDt+9lnn/Uf/OAHLskXL17s//znP0uWffLJJ3369OmeyWT88ccf3+H5WvQovXXibv7w9y7stogeJRaWoBZuYQLUULW3MNm8ebOamppUX1++k9fdtXHjRu28884V97Fhw4ZI5QbbvqPewqRSXWtxC5O3TZzo15w2s9u6w84+j1uYAAFh6A1IgREjRkQqZ2aRwoKkyOUG2777Kq7t5jlTAgCBIygBGFQaGhqSrsLrXJyjBASOoARgUDn88MP15JNPJl0NSflL6HIEJSBoTDgJBOaZZ57RX/7yl4rlNm7cqDvvvDPSNu+55x61tLQksu+oou67tbVVK1asqOm+q+aubI8FQFgISkAK3HvvvXrssccqlsvlcjryyCM1bdq0irfouPTSS3X00Ufrz3/+c9lyS5cu1fTp07su4R/IfSf5viVp3bp1mjdvnrZu3VqxbDVcUs692wIgLAQlIAVOOukkTZ06Ve9///vLBof58+erpaVFW7ZsKXuLjvXr1+vKK6+UJJ1xxhll9z1nzhxlMhlde+21Wr169YDuO6n3vW7dOs2ZM0eTJ0/WzJkztWTJkrL17I9sNtttARCYpOcnYGEZTEvUeZROPfXUXmeENjMfPny4T5061Ts7O71YNpv1N73pTV1lS92iw93929/+dtetPIYPH+4PP/xwr+WKb/nR2NjoX/7yl3stV6t9p+F9//jHP/bhw4f7sGHDumYEL67L5MmTfeXKlTWZR2mfN77R7zrzzG6LmEeJhSWohR4lIAFWYY6e3p5fsGBBt3Nv1q1bp5tuummHcps3b9Zll13WNZy0ZcsWnXXWWb3u55xzzumaUbu9vV1XX311r+cq1WrfaXjfleqQyWQqlukLht6AsBGUgARcddVV3b6xTJo0SQ0NDZo6dar++Mc/6oEHHlBdXV231xx44IH6+te/3vV41qxZOuyww3bY9vDhw3XuuefqmGOOkZS/Zcfpp5/eaz1mzJih2bNnS5Le/e536+yzz9bYsWN3KFerfafhfc+cOVMrV67UrFmzNHz4cEnSQw891FWn5cuXa+LEib3+vvrKxdAbEDpm5gZqqNqZue+77z7ttNNOOuSQQyq+dvTo0dqwYYMqHbt9maHazPSTn/xEM2fOLFuu1vtO+n2vW7dOv/nNb3T88cerubm523O1mJl7nze+0S/97Ge7rTv2yiuZmRsICPMoASnwoQ99KOkqJCLp9z127FideOKJsW3flb9iD0C4CEpAYCrdF419pwvnJQFhC/MvDxC4KMM62/UcprvxxhvV2tpa8XVTpkwpeRJ3TxdccIGOPPLIiuXi2HdUtdh3X37vteDu6qRHCQga5ygBNRT1HKW+6BmUUL1aB6VK5yjtNX68X/ipT3Vb97mrruIcJSAgXPUGBMTdtX79+khlo5bbtGmTOjo6IpU96qij9N73vrdiudtuu03Dhg0rO4GllK/jqFGjNHfu3JrtW8pfabZx48ZIZePkkrK5XLcFQFgISkBArrzySo0fP16nn3661qxZU7LcCy+8oHHjxmnq1Kl69NFHy27z0EMP1eTJkzVv3ryygWnp0qW6//779eSTT+rhhx8uWS6Xy+mMM85QZ2enLrzwwrL7vvzyy9XW1qZvf/vbZW9NEnXf2WxWN998s/bcc08dcMABZfc9ENydoAQEjqE3oIZqPfSWlXT5Lbd0Pf75z3+u3/72t2pqalImk9HJJ5+s888/X2PHjtWCBQu6Jltcvny5LrjgAm3dulXNzc2aMmWKrrzySh100EF6/PHH9fe//71rmzNmzNCmTZs0cuRIjRgxQhdddJFOPPHEHSZdPOqoo/S73/1OuVxOhx56aMl7qd1222068cQTtWnTJg0bNkwvvPCCdt111x3KrV+/XpMmTdLmzZs1YsQIXXHFFTrllFN63WaUfd9yyy0688wz9dprr2nTpk2qr6/XL37xi67n99hjDx136KGlf9lVqDT09qZddvFvf+IT3dZ96Wc/Y+gNCAhBCaihWgelDYWlnPnz52u//fbT2972tpJlMpmMPv7xj+t//ud/tNNOO5U8Kbq+vl51dXVqbW1VQ0ND1/qlS5dqypQpXT1OjY2Nuv/++/W+972v2+tzuZz23ntvLV++XJLU0NCgGTNm6KqrrtphX+ecc44uvfRStbW1SZJ22WUXrVq1So2Njd3KRd33mDFjtGnTJnV2dpb8PUyWVMuzlCoFpT3GjfOzCxNgbjdz3jyCEhAQht6AFNtZ3e/HeN5550nKz0J9+OGHa+HChfrkJz+pt771rd3KLV26VE1NTRo5cqQmTJiga665RrfeeqskaePGjb3Ojt3c3KzTTjtNK1as6BaSJKmlpUVjxozpejx69GitXbt2h/q2tbWpublZo0aNkpQPLxs29B71Xn31Ve28886S8gFt1KhR2rJlyw7lou77hRde0BlnnKHhw4erqalJO++88w73bBrYa97yOnO5bguAsNCjBNRQ3Fe93XPPPbrssst00UUX6eCDDy75mpaWFn3qU5/SCSecoM997nNl5yD6yle+osbGRn3rW9/S+PHjy9YlhFnBX3vtNV166aVauXKlfv7zn3d7bqCvett93Dg/o8e0C/9+ww30KAEBYR4lICDTpk3TtGnTKpYbN26c/vjHP0baZm/DYiEbM2aMLrrooqSrIen1k7kBhIugBCCyJHugQ+397uRGuEDQCEoAIvv85z/f6/lBPU2cODHyNocNG6Z99923ZvtOE3qUgPARlICUG+jbbkQRpU67Ryw3XtLnDz+8pvtOi+0TTgIIF0EJAOJCjxIQPIISAMTExTlKQOgISgAQE85RAsJHUAKAGDHJJBA2ghIAxMTd1cHQGxA0ghIAxMQlZQlKQNAISgAQE3dn6A0IHEEJAGLi7uro7Ey6GgD6gaAEADFhegAgfAQloIbapZYV0otJ1wMDZo+yz7oTlIDAEZSAGnL3XZKuA9Ij5642ht6AoBGUACAmLnGOEhA4ghIAxIR5lIDwEZQAICZc9QaEL5N0BQBgsMq5a1t7e7clCjObbmbPmdkyM5sTczUBlEGPEgDExN3VFjEcbWdmdZJ+JOnDklZJWmRmd7j7MzFUEUAFBCUAiIm7q62jo68vO0TSMnd/QZLM7BZJx0giKAEJICgBQEyynZ3a0NLS15dNlLSy6PEqSe+uWaUA9AlBCQBi0uZ+z/Pt7eN6rB5mZouLHs9197lFj62XTXntawcgCoISAMTE3adX8bJVkiYXPZ4k6aXa1AhAX3HVGwCkyyJJbzazPc2sUdLxku5IuE7AkEWPEgCkiLt3mtlXJd0jqU7SPHd/OuFqAUOWuTP0DQAA0BuG3gAAAEogKAEAAJRAUAIAACiBoAQAAFACQQkAAKAEghIAAEAJBCUAAIASCEoAAAAl/D9rKJDsQEBjNgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x720 with 12 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Double Q-learning\n",
    "# Do not modify this cell.\n",
    "def behaviour_policy(q):\n",
    "  return epsilon_greedy(q, epsilon)\n",
    "def target_policy(q, a):\n",
    "  # Place equal probability on all actions that achieve the `max` value.\n",
    "  # This is equivalent to `return np.eye(len(q))[np.argmax(q)]` for Q-learning\n",
    "  # But results in slightly lower variance updates for double Q.\n",
    "  max_q = np.max(q)\n",
    "  pi = np.array([1. if qi == max_q else 0. for qi in q])\n",
    "  return pi / sum(pi)\n",
    "\n",
    "learned_qs = []\n",
    "for _ in range(5):\n",
    "  grid = Grid()\n",
    "  agent = GeneralQ(grid._layout.size, 4, grid.get_obs(), target_policy, \n",
    "                   behaviour_policy, double=True, step_size=step_size)\n",
    "  run_experiment(grid, agent, int(1e5))\n",
    "  learned_qs.append(agent.q_values.reshape(grid._layout.shape + (4,)))\n",
    "  \n",
    "avg_qs = sum(learned_qs)/len(learned_qs)\n",
    "plot_action_values(GRID_LAYOUT, avg_qs, vmin=-50, vmax=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd083e3a",
   "metadata": {},
   "source": [
    "## Task 2 - Analyse results\n",
    "\n",
    "Consider the greedy policy with respect to the estimated values learned by each of the four agents.\n",
    "\n",
    "**Task 2.1** How and why do the policies found by Q-learning, Sarsa, Expected Sarsa, and double Q-learning differ? Explain notable qualitative differences in at most four sentences.\n",
    "\n",
    "> - **Q-learning**: The value function ($v(s)$) of the Q-learning algorithm looks more effective on the upper half of the grid that it seems to be bright and explore/learn towards yellow (goal).\n",
    "\n",
    "> - **Sarsa**: The algorithm of Sarsa shows that the path goes around the middle of the rectangle (wall) in the most cases, which means the algorithm would slowly approach the goal and surely reach the goal.\n",
    "\n",
    "> - **Expected Sarsa**: Although the algorithm of expected Sarsa seems to be significantly similar to the Sarsa and they represent the same path as shown on the bottom side of each figure, the expected Sarsa looks smoother than the Sarsa, which means the values do not rapidly/dramatically change.\n",
    "\n",
    "> - **Double Q-learning**: The algorithm of double Q-learning finds a more efficient path than other algorithms in many cases and obtains higher rewards that are more balanced than other algorithms, as shown in the grid, which means that the performance is good not only in the middle region but also at close to the wall.\n",
    "\n",
    "**Task 2.2** Which of the algorithms, out of Q-learning, Sarsa, Expected Sarsa, and double Q-learning with the learning parameters (exploration, step size) as discussed above, will *in general* yield higher returns on average during learning?  You are allowed to specify a partial (rather than a full) ordering over the algorithms, but try to be as specfic as you can. Explain your answer in at most four sentences.\n",
    "\n",
    "> Specify a partial ordering over the algorithms = **1st** : Expected Sarsa, **2nd** : Sarsa, **3rd** : Double Q-learning, **4th** : Q-learning\n",
    "\n",
    "> The algorithms of Q-learning and Double Q-learning can achieve good results by choosing an action that can get a good Q-value for the action. However, it can be highly possible to have a state that goes to the wall if the policy is stochastic, such as epsilon greedy. On the other hand, Sarsa and Expected Sarsa are likely to be relatively stable because they update the values by exploring all the overall Q-value and tries to get higher rewards, which means it avoids close to the wall. Therefore, these ordering can be justified by this question based on the above qualitatively answers and these mentioned."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d63daf95",
   "metadata": {},
   "source": [
    "## Experiment 2: Run the cells below\n",
    "\n",
    "This will train Q-learning and double Q-learning agents on the stochastic version of the Grid problem.\n",
    "\n",
    "In the noisy version, a zero-mean Gaussian is added to all rewards.  \n",
    "\n",
    "The variance of this noise is higher the further to the left you go, and the further down.\n",
    "\n",
    "The agents will be trained  with a step size $\\alpha=\\frac{1}{10}$ and $\\epsilon$-greedy behaviour, with $\\epsilon=\\frac{1}{4}$.\n",
    "\n",
    "The plots will show action values for each of the actions, as well as a state value defined by $v(s) = \\max_a q(s, a)$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "eaa2aa58",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Agent settings.\n",
    "# Do not modify this cell.\n",
    "epsilon = 0.25\n",
    "step_size = 0.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "97108b3b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAFUCAYAAADYlmA2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA3gUlEQVR4nO3dd5xkZZX/8c+pznF6pntyYAJxyOw4gqDACoijCLrKgroqq6IiBpTFFXZdQFnTKroqCiKCu7pmfoqOkkEBRYbMMMDkHHump3M+vz+e21A0HW53V3WF/r5fr3p11b1VdQ9MV5960nnM3REREYkjkekAREQkdyhpiIhIbEoaIiISm5KGiIjEpqQhIiKxKWmIiEhshZkOIN3q6up8/vz5mQ5DRCRnPProo3vcfepA5/I+acyfP58VK1ZkOgwRkZxhZhsHO6fuKRERiU1JQ0REYlPSEBGR2JQ0REQkNiUNERGJTUlDRERiU9IQEZHY8n6dhojkh1Oue2jM79HT6/T0jn4PocICI2E25jjuu+g1Y36PTFHSEJEJwd1pbO9isJyxrr4VgIW15YO+R2HCmFRaCClIHLlKSUNEcsJYv52v2NzA9sb2Qc9fvvw5AP5z2aFDvs+BdRUcNr1qTLHkMo1piEjeW7OnZciEkan3ykVKGiKS17Y3trNqZ1NK3/PxrftpaOtK6XvmilhJw8wSA93SHZyIyFjsae7gsS37U/6+Pb3Owxv30dzRnfL3znZx//B3A139b2bWYWbrzexrZlaZriBFREaqvqWTv21uoNdHP1tqKJ09vfxlw8RLHHGTxseAe4AzgMOANwB3A5cBHwFeA3wjDfGJiIzY7uYOHt60b0zTa+No7+7hoQ17aWyfOF1VcWdPfQo4zt372nkvmNkK4FF3X2RmTwOPpiVCEZER2LyvjSe3N+JpamH019Hdy4Pr97Jkbg1TK0vG5ZqZFLelUQ30n7xcDkyK7u8AylIVlIjISLk7z+5o4olt+8ctYfTp7nUe3tTA+mitRz6L29L4EXCnmX0T2AzMAT4B3BKdPwN4PvXhiYgMr72rh8e27Ke+tTNjMbg7z+xoZF9bJ0fNrKawID/nCsVNGv8CrAbOA2YB24HvAN+Pzt8L3DeaAMzsHcCVhLGSpe4+4N6sZrYBaAJ6gG53XzKa64lIftnZ1METW/fT2dOb6VAA2Lq/nYa2bo6dXc3k8uJMh5NysZKGu/cC34tuA50fy0qXZ4C3AdfHeO6p7r5nDNcSkTzR1dPLszub2LSvLdOhvEJLZzcPbtjHgbXlHDy1kkQif8qOxC4jYmZnAMcAL5ta6+6fG0sA7r4qev+xvI2ITCA7Gtt5ensT7d09mQ5lUO7O6j0tbG/q4OhZ1UzJk1ZHrKRhZt8GziV0QyWP9IznaJMDd5iZA9e7+w2DPdHMLgQuBJg3b944hSci6dba2c0zO5rY2dSR6VBia+7o5sH1e5lbU8bi6VUUF+b2WEfclsb5wDHuvnk0FzGzu4AZA5y6wt1/E/NtTnT3bWY2jTAo/5y7/2mgJ0YJ5QaAJUuWjO80ChFJue6eXtbsaWFtfWvaFuul2+aGNnY0dXDw1ArmTy7P2S6ruEmjHmgY7UXc/bTRvjbpPbZFP3eZ2a3AUmDApCEi+cHd2dzQxnO7munozo6B7rHo6ull5Y4mNuxtY/H0SmZUl2Y6pBGLmzS+BvzYzL4I7Ew+4e7rUh5VP2ZWASTcvSm6fwZwdbqvKyKZs6upg2d3NtGUh2U6Wjq7eWRzA7XlxRw2vTKnZlnFTRrfjX6+ud9xBwrGEoCZvRX4FjAV+L2ZPeHubzCzWcCN7r4MmA7cGg2WFwI/cfc/juW6IpKd9rV2smpnc0bXXIyX+tZOHli/l5nVpRw6rZLKkuzf4ijulNu0jdy4+63ArQMc3wYsi+6vA45OVwwiknnNHd2s2tnMjqaJt1fF9sZ2djR1MLemlEOmVlJaNKbv4mmV/WlNRPJaW1cPz+9qZsv+9nEv/5FN3J1N+9rY0tDOginlHDS1gqIsXFU+aNIwsz+6+5nR/T8zyPRad39dmmITkTzW2d3L6j0tbNibuzOi0qHXnbX1LWxqaGNRbTkLaysoyKKZVkO1NH6UdP/GdAciIhNDb6+zfm8rq/e00JUlpT+yUVdPL8/tambD3jYOnVbJnJrSrFgEPWjScPefJN2/ZbDniYjEtb2xnWd3NNHalb0rubNNe3cPT2zbz7q9LRwxo5raiszOtMp4GRERyX/NHd08vb2RPS35PyMqXRrbu3low15mTypl8fSqjA2Wj7WMiIjIoHp7Q/2lNXtaNG6RIlv3t7OzqYPF06uYN7ls3LusxqWMiIhMPE3t3Ty2tYHG9vxbnJdp3b3OU9sb2d7UwTGzqse11RF3PteYyoiIyMSypaGNP6+vV8JIs93NHdy/tp7dzeNXwDFu0ugrI3KCmS1MvqUzOBHJPat2NvH41v309Ko7ajx09vTy8KYGNu4dn5GDjJcREZH88cz2RtaP0x8veYl76K7qcWdhbUVarzVsS8PCKMtBQLG7J/rdlDBEBIC1e1qUMDJs5Y4mdjSmtwzLsEnDw7r+pwCtwhGRATW2d7FqV3OmwxDgiW2NdKaxjHzcMY3HgYPTFoWI5LTnd7VM6LpR2aQr2rAqXeKOadwH/NHMbgY2k1SHyt1vSn1YIpIrunp62TmOs3dkeFv3t7N4RlVa3jtu0jgRWA+c3O+4A0oaIhNYU0e3WhlZpr27h47uHkoKUz/sHHc/jVNTfuWImX0VOAvoBNYCF7h7wwDPOxP4JmG21o3u/qV0xSQi8XX3KGFko+4eJx17Oo24WLsFib5bCmK4EzjC3Y8CXgA+O8A1C4DvAG8EFgPnm9niFFxbRMaopDD79nwQKE7Tv0usdzWz2WZ2q5nVA91AV9JtTNz9DnfvWzb6V2DOAE9bCqxx93Xu3gn8FDh7rNcWkbGrLCkkkQUlu+Ul5UUFadvAKe67fo/QffR6oBk4Dvgt8OEUx/PPwB8GOD6bMADfZ0t0TEQyrCBh1GW4XLe83PSqkrS9d9wer9cA89y9xczc3Z80s/cDDwHfH+7FZnYXMGOAU1e4+2+i51xBaMX8eKC3GODYoB2pZnYhcCHAvHnzhgtPRMZoTk0puzSDKmvMqSlL23vHTRo9hD/oAA1mNhVoJOa3fXc/bajzZvZeQomS1/vA0zC2AHOTHs8Btg1xvRuAGwCWLFmiUTqRNJtZVUpxQROd2okv4yaVFlFTVpS294/bPfUwsCy6fzvwM+DXwIqxBhDNivoM8BZ3H6wGwSPAQWa2wMyKgfMI3WMikgUSCWP+lPJMhyHAwtr0/jvETRr/BNwf3f8kcA/wDPDOFMTwbaAKuNPMnjCz7wGY2SwzWw4QDZRfTEhYq4Cfu/vKFFxbRFJk/pQyDYhnWGlhAbOqS9N6jbjrNBqS7rcBX0hVAO5+4CDHt/FS6wZ3Xw4sT9V1RSS1SgoLmFNTyqZ9bZkOZcJaUFtOIpHexB13ym2JmV1jZuvMbH907Awzuzit0YlITlmgLqqMKUgYB0xO3wB4n7jdU9cCRwDv4qVZSyuBj6QjKBHJTdWlRZp+myGzJ5WmbW1Gsrizp94KHBhNue0FcPetZqa1EiLyMgumlLOnpTPTYUw449XKi5uWOumXYKJpt/Upj0hEctr0qhLKi7Q/23iqqyimujR902yTxU0avwBuMbMFAGY2kzDr6afpCkxEcpOZsaguvVuOyssdOI7/v+MmjcuBDcDTQA2wmrC47qq0RCUiOW1eTZlaG+OktryYqZXpKxvSX6yk4e6d7v5Jd68EpgNV7n5JVDxQRORlEgnjiJnVmQ4j7yXMOGJmejZbGsygA+FmtnCI11VZtIjH3delOigRyX3Tq0o4YHI5G/cNVuhBxurQaZXjNpbRZ6jZU2sI02uHWinihE2RRERe4YgZVbR0dms2VRrMrSnLyNjRoN1T7p5w94Lo52A3JQwRGVQiYSydN1lrN1Jsbk0ZR8/KTPefttwSkbQqSBivnjeZAyZrtXgqHDqtkmNmT8IyVOcrDTvIioi8XCJhHDWrmtqKIp7e3kSXSqiPWFlRAcfMqqZuHGdKDURJQ0TGzexJZdSWF/Pszia27m/PdDg5wcxYMKWcQ6ZWUDgOZUKGo6QhIuOqtKiA4+bUMH9KJyt3NNHQ1pXpkLLWtMoSFk+voqo0e/5UZ08kIjKhTCkv5rULa9ne2M4Lu5tpbO8e/kUTRG15MYdMq6Q2CycQxEoaZraZgffk7iBsxfpr4LvRZkkiIrHNrC5lRlUJO5s6WL2nZUK3PKZVlnBgXUVWJos+cVsa/w28O/q5GZgHfJRQk2ov8GnCHt6XjTQAM/sqcBahKOJa4ILkTZ+SnrcBaCLar9zdl4z0WiKSncyMGdWlzKgupb6lk7X1Lexs6sh0WOPCzJhdXcqiuvJxX6g3GnGTxvuA06Pd9AAwsz8Ad7j74WZ2L3AXo0gawJ3AZ92928y+DHyWsGf4QE519z2juIaI5IjaimJqK4pp7uhm/d5WNje00dM7UEdHbisqSDB/chnzp5RTmkN1uuImjZlAc79jLcCs6P4LhEKGI+budyQ9/Cvw9tG8j4jkl8qSQo6cWc2h0yrZuK+NDXtbaevqyXRYY1ZVUsiCKeXMqSmjIM1bs6ZD3KRxG/AbM7uGMIYxh9AiuC06fwKhCu5Y/TPws0HOOXCHmTlwvbvfMNibmNmFwIUA8+bNS0FYIpIpRQUJDqyrYFFtOTuaOlhX38re1twrSzK9qoQFU8rHtSJtOsRNGh8CrgSuJ7QutgM/B66Ozq8D3jTYi83sLmDGAKeucPffRM+5AugGfjzI25zo7tvMbBpwp5k95+5/GuiJUUK5AWDJkiX5164VmYDMjJnVpcysLqWhrYt19S1sa+zAPXs/4gkz5taUsbC2nMqS/JisGuu/wt3bgX+NbgOd3zHM608b6ryZvRd4M/B6H+Q3oG88xd13mdmtwFJgwKQhIvmtpqyI4+bUcGhnN+vqW9mUZeMeRQUJFkwpZ8GUcooLM78gL5Vipz4zOwQ4GqhMPu7uN40lADM7kzDwfbK7D1hD2cwqgIS7N0X3z+ClVo6ITFDlxYUcMbOag6ZWsK6+lfV7WzOaPIoLEiyqq2D+5LKsWL2dDnHXaVwOfA54Ekj+w+7AmJIGYdvYEkKXE8Bf3f3DZjYLuNHdlxE2fro1Ol8I/MTd/zjG64pInigpLOCw6VUsrC1n9e4WNuxrG9duq4KEcWBdBQunlOdtsugTt6XxSWCpuz+V6gDc/cBBjm8DlkX31xFaOSIigyopLOCImdUsmFLOMzua2NWc/rUec2vKOGx6JSWFuTNtdiziJo024Ll0BiIikioVJYW8+oDJbNvfztPbG+lMQ1Xd8qICjpk9KatXb6dD3HbUvwPfMrOZZpZIvqUzOBGRsZg1qZSTF9VSW57aP+yz+953giUMiJ80bgY+SFij0RXduqOfIiJZq7SogOMPmMy8yWUpeb9DplVy3JyavB+7GEzc7qkFaY1CRCSNEgnj6FmTMIyN+wacpBnL4ulVGdmXO5vEXaexMd2BiIik25Ezq2jp7GZPy8hXlM+bXDbhEwYMkTTM7AZ3vzC6/z8MXBodd39PmmITEUkpM+PY2ZO4d80eukewnqO8qIAjZlSnMbLcMVRLY33S/TXpDkREZDyUFhWwsLaCF3b3r8E6uEOmVeZkccF0GDRpuPsXAcysgLCHxk+iciIiIjlt/pQyVu9pibUAsLggwazq0nGIKjcMO/zv7j3A15UwRCRflBQWMKUs3oZHM6pLSKiV8aK4c8ZuM7Oz0hqJiMg4qquMt8aibgKuxRhK3Cm3pcAvzewvhK6qF9t0GggXkVw0KebWqnGfN1HETRrPRDcRkbxQVTJ8rSgzo6J4YtSUiivuOo2r0h2IiMh4KisqIGFG7xCD4eVFBUTVtSUStzT63w92zt3vSV04IiIDO+W6h1L+nm2dPXjU2963Uvzzd77w4vmEGaVFqW9p3HfRa1L+nuMlbvfUD/o9ngoUE2pRLUxpRCIi46QsqevpuDk1mQskh8TtnnpZ7alo7ca/AU1jDcDMPg+cDfQCu4D39W3t2u95ZwLfBAoImzN9aazXFpHckcvfzvPJqMo0Rms3rgEuS0EMX3X3o9z9GOB3hB0CXyZKUt8B3ggsBs43s8UpuLaIiIzAWGr7nk5oHYyJuzcmPaxg4BpXS4E17r7O3TuBnxJaJyIiMo7iDoS/bG0GUE5Yu3FRKoIws2uA9wD7gVMHeMpswvqQPluAV6fi2iIiEl/cgfB393vcArzQr5UwKDO7C5gxwKkr3P037n4FcIWZfRa4GPiP/m8xwGsHnSdnZhcCFwLMmzcvTogiIhJD3IHw+8dyEXc/LeZTfwL8nlcmjS3A3KTHc4BXDJYnXe8G4AaAJUuWxK9/LCIiQxpqP41B99BINtYyImZ2kLuvjh6+BXhugKc9AhxkZguArcB5wDvHcl0RERm5oQbC1wBro9t+4BzCdNct0evOBhpSEMOXzOwZM3sKOAP4BICZzTKz5QDu3k3otrodWAX83N1XpuDaIiIyAkPtp/Fi6RAzux14k7v/OenYScC/jzUAd/+HQY5vA5YlPV4OLB/r9UREZPTiTrk9Hvhrv2MPAyekNhwREclmcZPG48B/mlkZQPTzGuCJNMUlIiJZKG7SeB9wIrDfzHYSxjhOIqytEBGRCSLulNsNwGvMbC4wC9ju7pvSGZiIiGSfuIv7AHD3zbx8ZbaIiEwgY6k9JSIiE4yShoiIxKakISIisQ1VRiTWjnzuvi514YiISDYbaiB8DaH2lPHyGlT9H6d+A10REclKg3ZPuXvC3QvcPQF8gLDx0aGEfTQOJVSkff+4RCkiIlkh7pTbzwMHuXtb9Hi1mX0IeAG4OR2BiYhI9ok7EJ4A5vc7dgDqmhIRmVDitjSuBe4xsx8SFvfNJZQWuTZNcYmISBaKW0bkq2b2NPAO4FhgO/DP7v7HdAYnIiLZZdikYWYFhLGLxelIEmb2ecKGTr3ALuB90V4a/Z+3AWgCeoBud1+S6lhERGRow45puHsP4Q91aZpi+Kq7H+XuxwC/Az43xHNPdfdjlDBERDIj7pjGN4Cfm9l/ErZ7fXGdxlgX97l7Y9LDCmLsSy4iIpkRN2l8O/p5er/jTgpmUJnZNYS9OfYDpw7yNAfuMDMHrnf3G8Z6XRERGZlYU26jhX4D3WIlDDO7y8yeGeB2dvT+V7j7XODHwMWDvM2J7n4c8Ebgo2b2uiGud6GZrTCzFbt3744TooiIxGDu2dMbZGYHAL939yOGed6VQLO7/9dw77lkyRJfsWJFiiIUEcl/ZvboYGPHsbqnzKwQuAg4Gagj1J8CwN0H/cYf870PcvfV0cO3AM8N8JwKIOHuTdH9M4Crx3JdEREZubgrwq8FPgT8Cfg74FfANOCeFMTwpair6ilCMvgEgJnNMrPl0XOmAw+Y2ZPA3witEa0REREZZ7G6p8xsK3CCu28yswZ3rzGzQwkD0ienPcoxUPeUiMjIDNU9FbelUc5Le4O3mVm5uz9HWB0uIiITRNwpt6uAVxG6hlYAV5pZI7A1XYGJiEj2iZs0PkFYFQ7wKeC7QBVwYTqCEhGR7BS3YOEjSfdXA6elLSIREclascY0zOzXZvYJMzsmzfGIiEgWi9s99XvgdcAlZlYNPADcD/wpuRUiI3fKdQ9lOoSUue+i12Q6BBFJs7jdUz8AfgAvrtq+kFCNthLt3jcuOrt76ezpHdVrn9/VDMAh0ypH/NqiAqOkUP/EIhLEXRF+KGE1+MnAScAO4HpCa0PGYLhv5/vbunh2ZxN7WjpHfY3Ll4dF9v9xxsGjev2U8mIOn1FFTVnRqGMQkfwQt3vqWWAt8EXgg+7ekr6QBKCtq4fndjazZX9bpkNhb2snf15Xz+xJpRw6rZLy4ri/NiKSb+J++t8DvBa4FLjMzP7ES2Mam4d8pYxIZ3cvq/e0sGFvK71ZVEwSYOv+drY3djBvchkHT61Qt5XIBBR3TON/gf8FMLPpwMeB69CYRsp09fSydk8L6/e20t2bXckiWa87G/a2srmhjQVTyllUW0FxYdzCAiKS6+KOaRwLnEIY03gt0EbYmlVjGmPU1dPL+r2trKtvpWuUA92Z0NPrrIlaRAumlLOoroKiAiUPkXwXt3vqVkKC+C3waXdfm76QJobuKFmszbFk0V93r7M6aiEtrK1gYW25kodIHovbPTU/zXFMGPmSLPrr7nVe2N0cJY9yFkxR8hDJR3G7pwz4AHA+UOfuR0Xbrc5w95+nM8B80d3Ty4Z9bazd0zLq9Ra5oKunl+d3NbOuPiSPhVPKKVTyEMkbcbunrgZOB74BfC86toWwOZOSxhB6e50N+1pZvTu/k0V/ycnjwLoKFkwppyBhw79QRLJa3K+A7wPe7O4/Bfqm9qwHFqYqEDO71MzczOoGOX+mmT1vZmvM7F9Tdd10cXc27Wvl7tV7WLmjaUIljGRdPb2s2tnE3at3h2nEWTwzTESGF7elUQA0R/f7PvWVScfGxMzmEloymwY5XwB8J3rOFuARM/utuz+biuun2s6mDp7d2URzR3emQ8kaHd29PL29kXX1rRw2vZKZ1aWZDklERiFuS+MPwNfNrAReHOP4PHBbiuK4FriMlxJSf0uBNe6+zt07gZ8CZ6fo2inT3NHNXzfu42+b9ilhDKKls5sVmxt4aP1eGtu7Mh2OiIxQ3KRxCTAT2A9MIrQwDgA+M9YAzOwtwFZ3f3KIp83mpe1mIbQ2Zg/xnhea2QozW7F79+6xhjisnl5n1c4m7ltbz+7mjrRfLx/Ut3byp3V7WbmjiR51WYnkjGG7p6KuobcTZk5VE5LFZnffEfciZnYXMGOAU1cAlwNnDPcWAxwb9C+Nu98A3ACwZMmStP5F2tfayeNbG2npVMtipNyddfUt7Ghs59g5k5hSXpzpkERkGMMmDXfvMbOvu/tNQDuwa6QXcfcBd/ozsyOBBcCToceLOcBjZra0X1LaAsxNejwH2DbSOFJtzZ4WntvVjGdZjahc09rVw0Mb9nHw1AoOqqsg+l0QkSwUt3vqNjM7K9UXd/en3X2au8+PFhBuAY4boBXzCHCQmS0ws2LgPMLq9Izo7XUe29LAqp1NShgp4u48v6uZFZv3q7tKJIvFnT1VCvzSzP5CGFt48VPt7u9JR2BmNgu40d2XuXu3mV0M3E6YyXWTu69Mx3WH09vr/G1zg8Yu0mRHUzsPb+zl1QdM1roOkSwUN2k8E93SKrlcibtvA5YlPV4OLE93DENxdx7bul8JI83qWzt5dEsDr5pbo64qkSwTt/bUVekOJBesrW9le2N7psOYEHY2dfDC7pZRbVErIumjokAxNXd0v7jXtoyP1XtatJZDJMsoacS0amdz1u2kl+/cnWd3KlGLZBMljRga27vY0aRuqUzY3dzBvtbOTIchIhEljRjW1bdmOoQJba3+/4tkjVhJw4IPmtk9ZvZUdOx1ZnZuesPLvM7uXrbuVysjk3Y0ddDe1ZPpMESE+C2Nq4H3E0pzzIuObSEFtaey3eaGNo1lZJi7s6mhLdNhiAhZtJ9Gttq4T3+sssHGvW1afS+SBeImjbTup5Gt6ls6VYgwS7R397C7WQPiIpkWN2ksJ737aWSlLfvVysgm+vcQyby4SeNTwCzSsJ9GNtvZpHIh2WRXc6e6qEQyLG4ZkUbgHDObThgIH9F+GrmorauHju6Jua93turq6aWls4fKkrgl00Qk1WJ9+sysr0WyO7phZgl3z9u/qp1KGFmps0f/LiKZFLd7qhvo6n8zsw4zW29mXzOzvKosV1qkdY/ZqLRQ/y4imRT3E/gx4B7CtqyHAW8A7gYuAz4CvAb4Rhriy5iSwgJtP5plJpUWUV6srimRTIr7CfwUYUe9/dHjF8xsBfCouy8ys6eBR8cSiJldCnwVmOruewY4vwFoAnqAbndfMpbrxXH4jCoeXL9Xi/uygJlx+IyqTIchMuHFbWlUA+X9jpUTZlIB7ADKRhuEmc0FTgc2DfPUU939mPFIGAA1ZUUcM7taGwFlgaNmVlFboZafSKbFbWn8CLjTzL5J2O51DvAJ4Jbo/BnA82OI41pCV9dvxvAeaTF7UhlFiQSPbd1PlwZhx11hwjh29iRmVJdmOhQRIX7S+BdgNXAeYb3GduA7wPej8/cC940mADN7C7DV3Z8c5hu9A3eYmQPXu/sNo7neaEyrKuGURbU8tb1RazfG0dTKEo6eVU1ZUUGmQxGRSNx1Gr3A96LbQOeHLANrZncBMwY4dQVwOaGlMpwT3X2bmU0jtHqec/c/DXK9C4ELAebNmzfQU0astKiApfMms6OxnWd3Nqu8SBqVFxWweEYVM9W6EMk6saeiRAv7lgJ1wItNAne/abjXuvtpg7znkcACoK+VMQd4zMyW9l886O7bop+7zOzWKJYBk0bUCrkBYMmSJSkdxZ5RXcq0yhK27G/jhd0ttKlkd8qUFhZw8NQK5taUkUhoHEkkG8Vd3HcO8L+ELqrDgZXAEcADwLBJYzDu/jQwLek6G4Al/WdPmVkFkHD3puj+GYRy7RmRSBjzJpczZ1IZW/e3s7a+haYOtTxGq6K4kAPrwv9PJQuR7Ba3pfEF4AJ3/4WZ7XP3Y83sAkICSQszmwXc6O7LgOnArVFrpBD4ibv/MV3XjiuRMOZOLmNOTSm7mjtZV9/CnhZVYo2rtryYhbXlTK8q0Qw1kRwRN2nMc/df9Dt2C2Gq7aWpCsbd5yfd3wYsi+6vA45O1XVSzcyYXlXC9KoSGtu7WL+3lS0N7VrfMYCEGbOqS1lYW86ksqJMhyMiIxQ3aewys+nuvhPYYGYnAHsI+2xIkurSIo6eNYnDplWxcV8rG/a20d6tcY+SwgTzp5RzwOQySgr1ayOSq+Imje8DJwG/IqypuBfoBb6WprhyXnFhgoOmVrKotoIdTR2s39vK3taJ13VVU1bEwtpyZlaVarxCJA/ETRpf7ato6+4/MrP7gAp3X5W2yPJEImHMmlTKrEmlNLR1sa6+hW2NHXm9L4SZMaOqhIW15arfJZJnhk0aZlYANJtZjbt3ALj7cOU+ZAA1ZUUcN6eGw7p6WFffysZ9rfT05k/yKEgY82rKWFhbrsKCInlq2E+2u/eY2QtALbAt/SHlv7KiAg6fUcXBUytYv7eVdfWtOV2ipKggwfzJZSysraBYpctF8lrcr4M/Bn4X1Z7aQijpAYC735OOwCaCooIEB0+tZOGUcjbsa2PNnpacSh5FBQkW1pazYEo5RQVKFiITQdyk8ZHo55X9jjuwMGXRTFCFBQkOrKtg/uQy1u1tZe2eFrqzuNuqIGEsnFLOoroKJQuRCSZu7akF6Q5EQvI4eGolB0wu44XdLWzc15ZVA+ZmxtyaUg6ZWkmpigiKTEgjqT1VBBwPzHL3n0XlPHD3lnQFN1GVFBZw5MxqFkwp55kdTexuznxl3dryYo6YWUV1qRbkiUxkcWtPHQn8FuggFBX8GXAy8F7gH9MW3QRXWVLI8QdMZntjO89sb8rIIsGSwgSHz6hi9qRR77ElInkkbof0d4HPufuhQFd07H7Cgj9Js5nVpZxyYC3zJo/vH+45k8o49cA6JQwReVHc7qnDCVVuIZo55e4tZqa/JuOkqCDB0bMmMau6lCe2Nqa11VFckODoWdXaLU9EXiFuS2MD8HfJB8xsKbAm1QHJ0KZWlnDyolqmVZak5f3rKoo55cBaJQwRGVDcpPHvwO/N7Cqg2Mw+C/wC+Le0RSaDKi5MsHReDYdMq0zp+x5YV8HxB0xWQUERGVSspOHuvwPeCEwljGUcALzN3e9IY2wyBDPj4KmV/N2cGhJj3IvCzDhm1iQOm16lfS1EZEixkoaZ1bn7Y+5+kbu/yd0/7O6PpiIAM7vSzLaa2RPRbdkgzzvTzJ43szVm9q+puHY+mDWplKXzRp84EmYsmTOJueM8yC4iuSlu99QmM1tuZu/qW5+RYte6+zHRbXn/k1HRxO8QWjuLgfPNbHEa4shJUytLWDK3ZlSthGNnT9L4hYjEFjdpzAN+RygnssPM/s/MzjKz8SpluhRY4+7r3L0T+Clw9jhdOydMryrhsBGOcRwyrZJZk5QwRCS+uGMae9z9Onc/iTD99kngGmB7iuK42MyeMrObzGzyAOdnA5uTHm+JjkmShbXl1FXE279iclkRB9Wlo9EoIvlsNNXmpke3OqAhzgvM7C4ze2aA29mEhYOLgGMISWig3QAH6ncZtCiTmV1oZivMbMXu3bvjhJgXzIwjZlTHeu6RM6s16C0iIxa3jMhi4HzgnUAp8HPgHHf/W5zXu/tpMa/zfUI3WH9bgLlJj+cwxN4e7n4DcAPAkiVLsqfi3zioKi1kelUJO5sGr1dVV1HMpDLVkBKRkYvb0ngQmAlcCMxx90viJozhmNnMpIdvBZ4Z4GmPAAeZ2QIzKwbOI9TCkgHMHmacYrjzIiKDiTuQPT0agE6Hr5jZMYTupg3AhwDMbBZwo7svc/duM7sYuB0oAG5y95VpiifnDbcvt/btFpHRirufRqeZTSfMYqojaYzB3W8aSwDu/k+DHN8GLEt6vBx4xXRceaWyogISZvQOshdHRbFWfIvI6MQd0ziHULBwNWH21ErgCOABYExJQ9KjqMDo6H5l0ihMmAbARWTU4o5pfAG4wN2PBVqinxcCKVkVLiIiuSH24j53/0W/Y7cA70lxPJIiPYPsMd4zoeaSiUiqxU0au6IxDYANZnYCYW2FOsezUG+v0z1I0nB3unp6xzkiEckXcZPG93lpl75rgXsJq8KvS0dQMjadwySFzm4lDREZnbizp76cdP9HZnYfUOHuq9IVmIzesEmjpxcVEBGR0RhVwUF335TqQCR1BuuainteRGQw41WlVgZxynUPpfw9u3p6aWzvfvHxuvpWAC5f/hwA1SWFFBWOpuzY0O676DUpf08RyS5KGnmoIGFUJi3gO2pm1SvOi4iMhpJGhunbuYjkktT3UYiISN5S0hARkdiUNEREJDYlDRERiU1JQ0REYlPSEBGR2JQ0REQkNiUNERGJzXyQLUHzhZntBjZmOo4sUAfsyXQQIllOn5PgAHefOtCJvE8aEpjZCndfkuk4RLKZPifDU/eUiIjEpqQhIiKxKWlMHDdkOgCRHKDPyTA0piEiIrGppSEiIrEpaYiISGxKGiIiEpuShoiIxKakIS8yM20eLpJiZpaIfi42sw+ZWXGmYxoLJQ15kUdT6czsjWZWl+l4RPKBu/dGd98AXAF8w8xOz2BIY6IptwKEVoa7u5mdC/wf8BPgu8Dj7t6W2ehEcpOZFbp7t5l9EFgG1ACtgANPAz909xcyGOKIKWkIZlbg7j1mNhv4NHAwcACheNv/At8H1rt7VwbDFMlJZlYGbAde6+5Pm9kC4C3ABwjFEW8FfuTuDZmLMj51Twnu3hPdvQVIuPub3f1I4P3APwL/D3i3mZWCxj5ERugg4DlgB4C7r3f3bwK/BXqBJcC7MxfeyChpCABmVg40ArdFj83dlwPnAwZ8AfiFmZW6mqciI7Ea6AEu63f8bqAeeBD4spkdP96BjYaShgDg7q3ABuC/zWxJUmL4G/BX4O+B+cAnMxGfSK6KxgQvB841sxVm9jYz+wfgJuBOd78e+AOwOJNxxqUxDXmRmZUA3wbKgSeANYQZH0vd/Tgz+yxwBPA+jW+IDCxp8HsyYXxwp7tvMLM5wEcJXVGPAKvc/YroNbuAf3T3ezMWeExKGhOUmSXcvTcan5hDaEU8QUgYHwQOA15L6K76irtvNLOfA/Xu/pHMRC2SO8zsL4TenFcBvwQud/c10bkyd28zswrgm8CB7n5KxoIdgcJMByAZ91XCVMAGoAW4xd2/ABAtQuoGis3sAuBUYEGG4hTJeklfxj5MGAs8DZgOfA9YaWbXAte4e1P0klnAnwndVzlBLY0JKGlNxkLgIeB0oBY4GjgH2A/c7O7/L3p+MXAe0ODuv81I0CJZLilhlANXAw/0fYai86cBNxPWaCxy987oeEHSDMasp6QxwfQljOj+ucAJ7n5J9HgSIXEsIySP97v7g5mKVSQXmFkhoaBCT/T4fOA6YBXw+v6LY83sVe7+SN/Yx/hHPDZKGhOUmR0H/IDQdH4/YRZHd3RuNqGP9f7ocSKpFIKIJDGzy4G73f3h6PF8Quv9fYRWxQ/c/YcZCzDFlDQmCDN7LzA3abwiAVwEvA3YSxjwfsDd1/Z73YstExF5uajr9nJ3vzJ6fDzwrLs3mtnhwNnAGYTP2PXufnvGgk0RJY0JwsxeCxS7+91mdpS7PxUdnw1cAhwPPAncB9zu7o0ZC1YkB5nZiYQB71uB3wGPEiYbvRq4AGh2949lLsLUUNKYYMysCNgItAH/5O4PRcdPICSPqcBZ7t6cuShFckP/rtuoRX8uYartH4Dl7r7GzGqBTndvyvXuXiWNCSj6Bb4K+AihW+oD7r4nOne4u6/M9V9skfGQNBMxeYJJGfAx4CzCF7S7gV/kyxcxlRGZAPo2genj7vXufjFwDFANbIrmj+PuK6OfShgiQ4imynq098ylZna7mX3Q3dvc/SvAe4Bm4F1AZ0aDTSG1NCYQM7sYmA0UALe6+1+i428jVLh9p7vflsEQRXKOmd1LKEj4DKFEyF7g032fJTOb7e5b86X1rqSR55IWHL0X+BLwR8Iq7yMIxQj/y903ZzJGkVyT9LmaD3wNeHtS99SXgY8DjxPGDdcO/k65R91TeS7pm83bgXPd/QLgGuBbwDTg51EhQu2TIRJT0ufqMsLnaEbSuc8QNjHrAhaOf3TppdpTeSxpkO4UQutiAfDnqOLmFuApwg5ibfDSHuEiMriknS4PB3YRNln6oZld1jeV3d13ASdnMs50UfdUnkpqPlcC1wMnEkqdfxF4zN33Rc+rBFr7Kt4qcYgML9pG4AFCN1Q1oTbbIYR1Tl/vm42Yj5Q08lRSK+MbwG8Iszc+DVQRpgDeCax09/bMRSmSW5K+jL0NeIe7nx8dXwC8DngzYbzwAnf/awZDTRsljTyU9It9EGF16jFJdaU+SFh81At8391/mcFQRXKOmR0A/Aew190vTfq8FQFHElr1383FYoRxaEwjDyUN0p1L6JIqIoxp4O7fN7NbgSuArZmJUCSnHUS06tvM7nD3OwCi3SwfM7Onop378rK7V7On8pSZLSIs3jsd+IaZzes75+573P2SvnUaIhKfu99FWO90HbDczG4zs+TZU31f0PIuYYC6p/KamS0G/j66dRPGMX7i7i0ZDUwkhyXvg2FmhwLXEr6cXefuH8/XFkYfJY08MtAva7T24gTgjcBiwmrwy9392QyEKJJzkqbY/j3wDkK3/jrgN32fIzN7O3Bi34Zm+UxJI08k/WJPBc4nLOa7ndCyWB9tYP8GQgK5POp/FZEhJA1yVwPPAz8DWgifpb2ESrY/jtZl9L0mp7ZvHSkljTxjZn8gjFXdCXwUaAJuAn7k7nvNrMzd2/KlDo5IOiVNXb8RqHT386LxwqcJYxrvAF4ALnP3xzMZ63jRQHge6Cv/YWavB45w9ze4+38B9cBzwOcIA3aneLRfsRKGyPCihFEFGPBf0eEbgGvc/VLC5+uBiZIwQEkjLySNY7wOuBnAzC4BSt39XMKK8CKgNSMBiuS2dkKi2GlmBwN10WMIe4DfDa/cgiBfaZ1GfvkucHB0//WEFkafa939b+MfkkjuSZ5UEo3/PRwdryDUm3p3NI19nrs/ED1vQrTelTTyiLvvAHZEDzcCHzWzHuBfCHVxBpxhJSIv6bcL3ynAGcCzwH53v83M7gc+Rag99a7oeXk9+J1MA+E5LGlmx98RFvIdCPwPsAo4GvgKUEqYGvi1ifSLLTJaSTMRvwCcRPgi1g0cDrzOwz7fZQDRpJIJ9UVMSSNH9SvPfBthc6WpwD8Ar3X3B6NaOAl374heM6F+uUVGy8ymACsJSWJ1VHqn3t0/EK3XeCqfK9kOZUIM3OSjpBbDtcBN7n4RcDWwJ0oYrwYOcPeOvtlVShgiQzOzgujuDOBvUcI4FngtoUo0wKXAsZmILxsoaeQoCyoJC4xujw7/mlCIEOAc4CJQshAZTtR6SP4ythM4OqoK/d/Ale6+38zeAhzi7ndmKNSMU/dUDolKMpe4+wtJx64lbGrfCLzZ3ZdG35a2AP/g7g9pIZ/I4MxsNvD/gG3At6KChJjZmwhfwo4ClhH2pLkJuNrdfzpRxwiVNHKImf2asNbiZ4Sm804zOwL4DnA88GHCStUPA3Xufo7GMUSGFk2jfQ2h6OBJhLGMr7v7KjN7I2Fjpb6dL+93929lLNgsoKSRQ8xsKfAhwiypB4Hl7v5AVM323wkL+GYSBsW/7e771MoQicfMZhKSx9mEPTN+RWh59E0kqeirED2RP1dKGjnIzM4BPkhYZ3M78DN33xoVK2wG2qPyB2pliAyj/+fEzA4BTiHMRCwgTDT5cYbCyzpKGjkk+dtNNJ32YuCtwG7Ct6IH3X1jBkMUyVkDJI8jCK2OtxCKEr5HX8KUNHKSmRW7e2d0fyrwr8BSYAPwL9HKcBEZgaS1TycBb3H3y8xsMmE8oyHqCp6w3VJ9NOU2RyRVsi1IShhfB1rd/dPAvwHPK2GIjFy0G1/fTKjvAdsB3H2fu/9uotWXGopaGjkqmmp7vLufMMC5Cf9tSGQ0zOws4JvuvjB6rM9SP0oaWc7MFhI2eqklzCN/EvgzoVjaB9396WhdRq/6W0XiMbOjgAuASYQZhze6+6/MrJwwXX1T8l7g8hIljSyU1Lf6HsKai17CwiKAMmA1cK+7/1C/2CLxJBX4PAu4ClhPGAecApwHPA6c5+6bMhdl9lPSyFJmVkiYFfVu4I9REjkMOJOwP/Fj7n55JmMUyUVmthW4wt1vTjp2EGH71unA2e6+PkPhZT0NhGevy4C/uPvvifY9cfdV7n4t8H/Ap8xsWSYDFMk1ZnYusM3dbzaz4r7j7r4a+ChhgexbMxVfLlDSyF7tvLShUhe8NIPK3W8B/grMi45PzkSAIjloF9BXKaEz+URU0+2PwFwzmxvtzCf9KGlkr43ACWZWFfXDFgCYWUnS+avN7B7grUklnUVkcA3Aq4Hzkj8z0QA4hMKfnwB+Cbw3qkslSTSmkaWibzn3Egbrznf33UnnZhGSxr8BKwgbwuwe8I1E5GWi9U3HA1cCf3X3xuh4DWFg/BLgKcLnqkvleF5OSSOLmdmBwI+AA4AfE3boO43wTWmNu1+cwfBEcpKZTScs4FsG/JyQKKYC84F97n5+xoLLAUoaWc7M5hCmA36AMEi3lZBAfujunRO1pr/IaCS3GszsTEL9tk7COqivEaayN+lzNTgljRwSbRazO6mMiJrNIiPUfzGs1jqNjJKGiEwoSYv8+hbR6svXCChpiMiEkNzlpJpSo6ekISJ5qa/bKdoX43RgRnTqOyoVMnpapyEieSfqcuobp/gVcBSh4Oc5gMrvjIFaGiKSt8zsEuCN7n6GmdUBzwMnEWYhngHc7u5NmYwx16ilISJ5IXmjsn4VEp6Lfl5HKP65ClhCKI3eiYxIYaYDEBFJkTozq3b3tUnHVhJKhryJsG1rXz2pzxNaGR0aFB8ZtTREJF/8E7DazL7d19Jw9zsIieO3hNI7c83sK0CVu18dvU599COgpCEiecHdvw6cApxAqGT7sej4PwNvi552C7APeC+8OA1XSWMENBAuInnHzD4AfAnYS9gW+f7oeJ2778locDlOLQ0RyTvufiMwF1gO3G1my81sjhLG2ClpiEjOi7ZHxszmm9lJZvYud29z908ChwO9wKZof3AZA3VPiUjOMrMKd2+J7i8CfgfsIez1PRP4rLt/Ozr/euABd+/IVLz5QC0NEclJ0bYBV5jZSdGh/wRuBc5w94OBDwNXmdkXowHvu6MptpapmPOB1mmISK5aSNiU7HAzOwZoAX7p7m1RGZEfm5kD7waKgTYAzZYaG3VPiUjOMrMywkZKZxGSyJPAx/sW+JlZFfAscLq7PzfoG0lsShoikpOSN0+KtkZ+P6HlsQK4n1Bf6g3Aq9399IwFmmeUNEQkZ5lZKbDI3VdGj18PfAQ4lDDl9ibgy+6+Q1u4poYGwkUkl70NeNrMfmhmZdFg99uBq4B1wENRwjAljNRQS0NEcpqZHQ98DTgWuNrdvxQdr3H3hui+tnRNESUNEckLZvZO4L+BTcBV7v4bVbBNPXVPiUjOSNoz41gz+7SZTeo75+4/IezKNwe4NDqmhJFiShoikjOSuphOAs4EvmFmb016yi8JlWzfDC+VF5HUUfeUiOSMvrEJM6sEXg2cDSwiDHrfS0gWs9z9zAyGmdeUNEQk6/WNTZhZAqiJbo1APWEG1RnAqcBq4AJ336XxjPRQ0hCRrJeUND5L6JYqBwqA2wnTa7uAIqDQ3ZuVMNJHSUNEslpSwngVYX+M1wOVQB1wIbAFuNTdmzMY5oShQSIRyWpJLYa3A79w96cgJBOgAbiBsGfGwxkJcILR7CkRyWpRcgD4G3BIVDoEwmSqPwGPEBb2yThQ0hCRrJO850VSS+MJQj2pn5nZVKDMzOYCy4haGdorI/3UPSUiWSeaVltBmBU1nVC5dhWwFPgfYDshUSSAm939cQ1+jw8lDRHJGkmD3m8glDo/AXCghFDu/GZ3PyvadOlI4EFC2RAZJ5o9JSJZIWnhXiFhvcVXgfvc/VkzOwP4OHAY8Bl3/2X/12Um6olHSUNEskJSK+N7wAx3P2eA51wL/CNwmrs/O94xigbCRSRLRAljEWFP70/0HbegKHr4BcJCvroMhCgoaYhIdnkrYbX3aWZ2ULTbnrt7VzT1tokw9XZaRqOcwNQ9JSJZxczeBXyGUFvqO8BdwD537zazOYSkcaK7r9d4xvhT0hCRrBOtt7gSeA9h8d733P0eM/sF0OzuF2iKbWYoaYhI1jKzecAXgSXASuBkQunzDiWNzNCYhohkLXff5O7vAj5E2JHvs1HCKFTCyAy1NEREJDa1NEREJDYlDRERiU1JQ0REYlPSEBGR2JQ0REQkNiUNERGJTUlDRERiU9IQEZHY/j+vGCEITrJIWAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Q-learning and double Q-learning.\n",
    "# Do not modify this cell.\n",
    "\n",
    "def target_policy(q, a):\n",
    "  max_q = np.max(q)\n",
    "  pi = np.array([1. if qi == max_q else 0. for qi in q])\n",
    "  return pi / sum(pi)\n",
    "\n",
    "def behaviour_policy(q):\n",
    "  return epsilon_greedy(q, epsilon)\n",
    "\n",
    "mean_reward_q_learning = []\n",
    "mean_reward_double_q_learning = []\n",
    "for _ in range(30):\n",
    "  grid = Grid(noisy=True)\n",
    "  q_agent = GeneralQ(\n",
    "      grid._layout.size, 4, grid.get_obs(),\n",
    "      target_policy, behaviour_policy, double=False, step_size=step_size)\n",
    "  dq_agent = GeneralQ(\n",
    "      grid._layout.size, 4, grid.get_obs(),\n",
    "      target_policy, behaviour_policy, double=True, step_size=step_size)\n",
    "  mean_reward_q_learning.append(run_experiment(grid, q_agent, int(2e5)))\n",
    "  mean_reward_double_q_learning.append(run_experiment(grid, dq_agent, int(2e5)))\n",
    "\n",
    "plt.violinplot([mean_reward_q_learning, mean_reward_double_q_learning])\n",
    "plt.xticks([1, 2], [\"Q-learning\", \"Double Q-learning\"], rotation=60, size=12)\n",
    "plt.ylabel(\"average reward during learning\", size=12)\n",
    "ax = plt.gca()\n",
    "ax.grid(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "924db620",
   "metadata": {},
   "source": [
    "## Task 3 - Analyse results \n",
    "\n",
    "Which among Q-learning and Double Q-learning has a higher average reward? Explain why, discussing at least a) the dynamics of the algorithm, b) how this affects behaviour, and c) why the behaviour yields higher rewards for one algorithm rather than the other. Be concise.\n",
    "\n",
    "> The Q-learning is a popular reinforcement learning algorithm based on the principle of the Markov Decision Process (MDP) in general. The Q-learning uses a single estimator, which means that it estimates the value of the next state. Moreover, the Q-learning algorithm shows that overestimations of the action-values can be generated because of using stochastic MDPs and maximum action-values so that the performance would be poor. On the other hand, the Double Q-learning uses a double estimator, which means that it stores a couple of Q function, such as $Q_{a}$ and $Q_{b}$. One of the Q function is updated a value for the next state from the other and this algorithm does not inefficient using data because both Q functions are all used to choose an action. Therefore, the Double Q-learning prevents the overestimations of the action values based on using the two Q functions. A paper [1] proved that even if the Double Q-learning sometimes occurs underestimations the action values, it yields higher rewards and provide better performance than the Q-learning.\n",
    "\n",
    "> [1] H. V. Hasselt, \"Double Q-learning\", *Multi-agent and Adaptive Computation Group*, Centrum Wiskunde & Informatica, 2010."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30ea124b",
   "metadata": {},
   "source": [
    "## Task 4 - Target Q-learning \n",
    "\n",
    "Consider a new algorithm which we will refer to as **target Q-learning**. The target Q-learning algorithm applies Q-learning updates in the form \n",
    "$\\ \\ q(S_t, A_t) \\leftarrow R_{t+1} + \\gamma \\max_a q'(S_{t+1}, a) \\ \\ $  \n",
    "where the values $q'$ have been pre-trained by running $\\epsilon$-greedy Expected Sarsa, and are then held fixed throughout training.\n",
    "\n",
    "We can now imagine now to train both Q-learning and target Q-learning on the deterministic Grid problem.  In both Q-learning and target Q-learning the behaviour policy will select actions based on the same $\\epsilon$-greedy policy that was used to pre-train the $q'$ values, but using the current action-value estimates (not the fixed target values $q'$).\n",
    "\n",
    "Explain concisely which of the two algorithms will perform better and why. (If you're tempted to answer 'it depends', be sure to be clear what you think it depends on, and why. Don't be vague: point could be subtracted for including irrelevant or false statements, even if the correct answer is also given.)\n",
    "\n",
    "> The Q-learning algorithm can basically use to determine optimal policy by updating Q values based on empirical data. $\\epsilon$ greedy action selection has been used as a stochastic policy to select an action that maximises the Q function with 1-$\\epsilon$ probability. Various actions are randomly explored in the early stages of learning. As learning progresses, good action is taken according to the action-value function that has become more reliable. On the other hand, the target Q-learning is a bootstrap method, and the target value is determined based on the Bellman equation. Furthermore, it might be improved that the target policy cannot be inadvertently updated immediately based on the concept of a deadly triad if the fixed target values are not used. However, for both algorithms, dynamic changes make the learning process unstable, and the Q function likely to be divergence if it applies to approximation or nonlinear function. A paper [2] proved that the target Q-learning is more efficient and less divergence than the Q-learning if bootstrapping on separate networks.\n",
    "\n",
    "> $q(S_t, A_t) \\leftarrow R_{t+1} + \\gamma \\max_a q'(S_{t+1}, a)$ : self._q[self._s, self.last_action] = reward + discount * np.dot(self._q[next_state], self._target_policy(self._q[next_state], action) \n",
    "\n",
    "> [2] H. v. Hasselt, Y. Doron, F. Strub, M. Hessel, N. Sonnerat, J. Modayil, \"Deep Reinforcement Learning and the Deadly Triad\", *Google DeepMind*, Dec, 2018"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fd4a9745",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}