PyTorch/NoteBooks/AutoML/AutoML_BYOC.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": "# Clara Train SDK Bring your Own Component to AutoML (BYOC-AutoML)\n\nBy the end of this notebook you would be able to:\n1. Write your own handlers for AutoML to display AutoML summary using [MLFlow](https://mlflow.org/)\n2. Write your own controller for a random controller\n3. Use AutoML admin tool to control running jobs\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "## Prerequisites\n- Familiar with Clara train main concepts. See [Getting Started Notebook](../GettingStarted/GettingStarted.ipynb)\n- Familiar with AutoML. See [AutoML Notebook](./AutoML.ipynb)\n- Nvidia GPU with 8Gb of memory  \n- Have multiple GPUs is preferred   \n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "## Resources\nYou could watch the free GTC 2021 talks covering Clara Train SDK\n- [Clara Train 4.0 - 101 Getting Started [SE2688]](https://gtc21.event.nvidia.com/media/Clara%20Train%204.0%20-%20101%20Getting%20Started%20%5BSE2688%5D/1_0qgfrql2)\n- [What’s New in Clara Train 4.0 [D3114]](https://gtc21.event.nvidia.com/media/What%E2%80%99s%20New%20in%20Clara%20Train%204.0%20%5BD3114%5D/1_umvjidt2)\n- [Take Medical AI from Concept to Production using Clara Imaging [S32482]](https://gtc21.event.nvidia.com/media/Take%20Medical%20AI%20from%20Concept%20to%20Production%20using%20Clara%20Imaging%20%20%5BS32482%5D/1_6bvnvyg7)\n- [Get Started Now on Medical Imaging AI with Clara Train on Google Cloud Platform [S32518]](https://gtc21.event.nvidia.com/media/Get%20Started%20Now%20on%20Medical%20Imaging%20AI%20with%20Clara%20Train%20on%20Google%20Cloud%20Platform%20%5BS32518%5D/1_2yjdekmi)\n- [Automate 3D Medical Imaging Segmentation with AutoML and Neural Architecture Search [S32083]](https://gtc21.event.nvidia.com/media/Automate%203D%20Medical%20Imaging%20Segmentation%20with%20AutoML%20and%20Neural%20Architecture%20Search%20%5BS32083%5D/1_r5swh2jn)\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "## DataSet \nThis notebook uses sample dataset (Single image of spleen dataset) provided in the package to train small networks for a couple of epochs. \nThis single file is duplicated 32 times in the training set and 9 times to mimic the full spleen data set. \n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "# Lets get started\nBefore we get started lets check that we have an NVIDIA GPU available in the docker by running the cell below",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "# following command should show all gpus available \n!nvidia-smi",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "Next cell we define root path for AutoML",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "MMAR_ROOT\u003d\"/claraDevDay/AutoML/\"\nprint (\"setting MMAR_ROOT\u003d\",MMAR_ROOT)\n%ls $MMAR_ROOT\n",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "Next cell defines functions that we will use throughout the notebook",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "!chmod 777 $MMAR_ROOT/commands/*\ndef runAutoML(configPath,HANDLER_JSON):\n    %cd $MMAR_ROOT/commands\n    ! ./automlBYOC.sh $configPath $HANDLER_JSON\ndef printFile(filePath,lnSt,lnOffset):\n    print (\"showing \",str(lnOffset),\" lines from file \",filePath, \"starting at line\",str(lnSt))\n    lnOffset\u003dlnSt+lnOffset\n    !\u003c $filePath head -n \"$lnOffset\" | tail -n +\"$lnSt\"\n",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "# 1- Writing your own Handler \n\n## [MLFLOW](https://mlflow.org/)\nMLflow is an open source platform to manage the ML lifecycle, \nincluding experimentation, reproducibility, deployment, and a central model registry. \n\nHere, we will integrate with MLFlow using BYO Handler to show the results of AutoML experiments. \nWe will walk through installing MLFlow, start up the web UI service, then write the BYO Handler.   \n\n### Install MLflow\nIn this cell below will install MLflow for more in information see MLflow Documentation",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "!pip install jinja2\u003d\u003d2.11.1\n!pip install mlflow\n",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "### Start the UI web using 5000 which maps to 3031 external  ",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "import subprocess\n%cd /claraDevDay/AutoML\n%pwd\na \u003d subprocess.Popen([\"mlflow\",\"ui\",\"-h\",\"0.0.0.0\",\"-p\",\"5000\"])",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%% \n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "\n## Changes to AutoML scripts\nAutoML is explained in details in [AutoML notebook](AutoML.ipynb). \nHere, we will use same examples as in AutoML notebook and focus on new BYO handlers\n- [automlBYOC.sh](./commands/automlBYOC.sh) is where we configure the number of workers and which gpus to use. \nIn this notebook we will use very small networks that will use \u003c2GB. \nWe are setting number of workers to 8 and they will all use GPU 0 by specifying \n`workers\u003d0:0:0:0:0:0:0:0`. \nMoreover we set the automlconf to point to a new config which includes our BYO handlers\n`--automlconf config_automl_BYO_handler.json`\n\nlets check out the automlBYOC.sh",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "configFile\u003dMMAR_ROOT+\"/commands/automlBYOC.sh\"\nprintFile(configFile,0,30)",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "Now lets see the contenet of the new BYO handler we set in the script above \n- [config_automl_BYO_handler.json](./config/config_automl_BYO_handler.json) where any custom controller would be defined \nalong with maximum number of MMARs to search and the number of MMARs to keep. \u003c/br\u003e\nNote we add our handler as \n` \"path\": \"myAutoMLController.MyHandler\"`",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "configFile\u003dMMAR_ROOT+\"/config/config_automl.json\"\nprintFile(configFile,0,30)\n",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "- [automl_train_round.sh](./commands/automl_train_round.sh) \nThis is the script which will be triggered in each job  \n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "## Write your handler to write to MLFlow \n\nTo write your handler you need to implement `Handler` class found in \n`automl.components.handlers.handler`. You can then implement the following functions:\n- `recommendations_available(self, ctx):`\n- `startup(self, ctx: Context)`\n- `shutdown(self, ctx: Context)`\n- `start_job(self, ctx: Context)`\n- `round_ended(self, ctx: Context)`\n- `end_job(self, ctx: Context)`\n\nIn this example we have used the `end_job` function to get the parameters and the final accuracy \nthen write it to mlflow. \nCell below prints out the function we wrote  ",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "configFile\u003dMMAR_ROOT+\"/BYOC/myAutoMLController.py\"\nprintFile(configFile,85,60)\n",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "Now lets use the [trn_autoML_Enum.json](./config/trn_autoML_Enum.json) to see how it works. \nRecall from basic autoML notebook that this file uses Enum option to search different network architecture arguments for SegResnet network)\n\nlets run it a see configurations generated\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "runAutoML(\"trn_autoML_Enum\",\"config_automl_BYO_handler.json\")\n",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%% \n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "Now you can navigate to \u003cyourIP\u003e:5000 and see the MLFlow results shown. \nIt should look similar to image below\n\n\u003cbr\u003e![mlflow](screenShots/MLFlow.png)  \n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "## 2- Write your own controller     \nIn this Example, we will write a random controller to use instead of the \nreinforcement learning controller used by AutoML. \n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "configFile\u003dMMAR_ROOT+\"/BYOC/myAutoMLController.py\"\nprintFile(configFile,27,20)\n",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "To write a controller you need to implement the `Controller` class found in `automl.components.controllers.controller`. \nYou will need to implement the following functions:\n- `set_search_space(self, space, ctx)`\n- `initial_recommendation(self, ctx)`\n- `refine_recommendation(self, outcome: Outcome, ctx: Context)`\n\nNow lets examine the code to do random pick for enums ",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "configFile\u003dMMAR_ROOT+\"/config/config_automl_BYO_RandomController.json\"\nprintFile(configFile,27,20)",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "Lets now run autoML with our controller",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "runAutoML(\"trn_autoML_Loss_Optimizer\",\"config_automl_BYO_RandomController.json\")",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "We can see debugging statements showing our custom code running. \nFinally, you can use tensorboard text tab to see the the parameters passed from the random controller to AutoML.  ",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "# 3- Using the AutoML admin tool \n\nOnce you start AutoML, user can now use use the autoMl admin. \nTo get started lets run a simple autoML run using cell below \n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "%cd $MMAR_ROOT/commands\n! ./automl.sh trn_autoML_Enum ",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "\nTo start the AutoML admin you can run \n```\npython -m dlmed.hci.tools.admin \\\n--host localhost \\\n--port 33330 \\\n--prompt \u0027AutoML\u003e \u0027\n``` \nor simply run [automl_admin.sh](commands/automl_admin.sh) \nThis is a shell which will allow you to control automl workers. \nTo see list of commands type `?` to see list of commands as below \n\u003cbr\u003e![](screenShots/AdminCmds.png)\n",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%% md\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "You can list workers using `list_workers` so you should see some thing similar to \n```\nAutoML\u003e list_workers\n------------------------------\n| NAME | GPUS | STATE | JOBS |\n------------------------------\n| W1   | 2    | busy  | 3    |\n| W2   | 3    | busy  | 3    |\n------------------------------\n```  \n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "\nRunning `summary` should produce something similar to\n\n```\nAutoML summary\nTop MMARs to keep: 30\n-------------------------------------------------------------------------------------------\n| START                      | END                        | ROOT    | SCORE      | WORKER |\n-------------------------------------------------------------------------------------------\n| 2020-08-13 21:42:46.352932 | 2020-08-13 21:43:13.502620 | W1_1_J1 | -1000000.0 | W1     |\n| 2020-08-13 21:42:47.354996 | 2020-08-13 21:43:15.417571 | W2_1_J2 | -1000000.0 | W2     |\n-------------------------------------------------------------------------------------------\n\n``` \n",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%% md\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "You can also manipulate worker as:\n- resume_worker\n- suspend_worker\n or shut down all AutoML using `stop` command\n\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    }
  ],
  "metadata": {
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 2
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython2",
      "version": "2.7.6"
    },
    "kernelspec": {
      "name": "pycharm-e5c8d846",
      "language": "python",
      "display_name": "PyCharm (claraTrain2.0)"
    },
    "stem_cell": {
      "cell_type": "raw",
      "source": "\u003c!--- SPDX-License-Identifier: Apache-2.0 --\u003e\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}