PyTorch/NoteBooks/Data/DownloadDecathlonDataSet.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": "# 1. Download Decathlon Dataset with Monai APIs \n\n[Medical Decathlon](http://medicaldecathlon.com/) dataset is public avalibe dataset which you can manually download. \nThis notebook using monai API to download any of the decathlon data sets to your Data folder. \nYou can then use this data to train models using the SDK. \nMedical Decathlon has multiple data sets for different organs as:\nLiver Tumours, Brain Tumours, Hippocampus, Lung Tumours, \nProstate, Cardiac, Pancreas Tumour, Colon Cancer, Hepatic Vessels, Spleen",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "DataRoot\u003d\"/claraDevDay/Data/DecathlonDataset\"\n%cd $DataRoot\n!pwd\n",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "import monai\nmonai.config.print_config()\n",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "## 1.1.1. Brain Tumor Data Set\nThis dataset has the following details:\n- Target: Gliomas segmentation necrotic/active tumour and oedema\n- Modality: Multimodal multisite MRI data (FLAIR, T1w, T1gd,T2w)\n- Size: 750 4D volumes (484 Training + 266 Testing)\n- Source: BRATS 2016 and 2017 datasets.\n- Challenge: Complex and heterogeneously-located targets",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "myTask\u003d\u0027Task01_BrainTumour\u0027\ndataset \u003d monai.apps.DecathlonDataset(root_dir\u003dDataRoot, task\u003dmyTask, section\u003d\"training\", download\u003dTrue)",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "\n## 1.1.2. Heart Data Set\nThis dataset has the following details:\n- Target: Left Atrium\n- Modality: Mono-modal MRI  \n- Size: 30 3D volumes (20 Training + 10 Testing)\n- Source: King’s College London\n- Challenge: Small training dataset with large variability",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "myTask\u003d\u0027Task02_Heart\u0027\ndataset \u003d monai.apps.DecathlonDataset(root_dir\u003dDataRoot, task\u003dmyTask, section\u003d\"training\", download\u003dTrue)",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "\n## 1.1.3. Liver Data Set\nThis dataset has the following details:\n- Target: Liver and tumour\n- Modality: Portal venous phase CT\n- Size: 201 3D volumes (131 Training + 70 Testing)\n- Source: IRCAD Hôpitaux Universitaires\n- Challenge: Label unbalance with a large (liver) and small (tumour) target",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "myTask\u003d\"Task03_Liver\" # will take along time (3+ hours )\ndataset \u003d monai.apps.DecathlonDataset(root_dir\u003dDataRoot, task\u003dmyTask, section\u003d\"training\", download\u003dTrue)",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "\n## 1.1.4. Hippocampus Data Set\nThis dataset has the following details:\n- Target: Hippocampus head and body\n- Modality: Mono-modal MRI \n- Size: 394 3D volumes (263 Training + 131 Testing)\n- Source: Vanderbilt University Medical Center\n- Challenge: Segmenting two neighbouring small structures with high precision \n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "myTask\u003d\u0027Task04_Hippocampus\u0027\ndataset \u003d monai.apps.DecathlonDataset(root_dir\u003dDataRoot, task\u003dmyTask, section\u003d\"training\", download\u003dTrue)",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "\n## 1.1.5. Prostate Data Set\nThis dataset has the following details:\n- Target: Prostate central gland and peripheral zone \n- Modality: Multimodal MR (T2, ADC)\n- Size: 48 4D volumes (32 Training + 16 Testing)\n- Source: Radboud University, Nijmegen Medical Centre\n- Challenge: Segmenting two adjoint regions with large inter-subject variations",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "myTask\u003d\u0027Task05_Prostate\u0027\ndataset \u003d monai.apps.DecathlonDataset(root_dir\u003dDataRoot, task\u003dmyTask, section\u003d\"training\", download\u003dTrue)",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "\n## 1.1.6. Lung Data Set\nThis dataset has the following details:\n- Target: Lung and tumours\n- Modality: CT\n- Size: 96 3D volumes (64 Training + 32 Testing)\n- Source: The Cancer Imaging Archive\n- Challenge: Segmentation of a small target (cancer) in a large image",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "myTask\u003d\u0027Task06_Lung\u0027\ndataset \u003d monai.apps.DecathlonDataset(root_dir\u003dDataRoot, task\u003dmyTask, section\u003d\"training\", download\u003dTrue)",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "\n## 1.1.7. Pancreas Data Set\nThis dataset has the following details:\n- Target: Liver and tumour\n- Modality: Portal venous phase CT\n- Size: 420 3D volumes (282 Training +139 Testing)\n- Source: Memorial Sloan Kettering Cancer Center\n- Challenge: Label unbalance with large (background), \nmedium (pancreas) and small (tumour) structures. ",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "myTask\u003d\u0027Task07_Pancreas\u0027\ndataset \u003d monai.apps.DecathlonDataset(root_dir\u003dDataRoot, task\u003dmyTask, section\u003d\"training\", download\u003dTrue)",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "\n## 1.1.8. Hepatic Vessel Data Set\nThis dataset has the following details:\n- Target: Hepatic vessels and tumour\n- Modality: CT\n- Size: 443 3D volumes (303 Training + 140 Testing)\n- Source: Memorial Sloan Kettering Cancer Center\n- Challenge: Tubular small structures next to heterogeneous tumour. ",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "myTask\u003d\u0027Task08_HepaticVessel\u0027\ndataset \u003d monai.apps.DecathlonDataset(root_dir\u003dDataRoot, task\u003dmyTask, section\u003d\"training\", download\u003dTrue)",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "\n## 1.1.9. Spleen Data Set\nThis dataset has the following details:\n- Target: Spleen\n- Modality: CT  \n- Size: 61 3D volumes (41 Training + 20 Testing)\n- Source: Memorial Sloan Kettering Cancer Center\n- Challenge: Large ranging foreground size",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "myTask\u003d \u0027Task09_Spleen\u0027\ndataset \u003d monai.apps.DecathlonDataset(root_dir\u003dDataRoot, task\u003dmyTask, section\u003d\"training\", download\u003dTrue)",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "## 1.1.10. Colon Data Set\nThis dataset has the following details:\n- Target: Colon Cancer Primaries\n- Modality: CT  \n- Size: 190 3D volumes (126 Training + 64 Testing)\n- Source: Memorial Sloan Kettering Cancer Center\n- Challenge: Heterogeneous appearance",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "myTask\u003d\u0027Task10_Colon\u0027\ndataset \u003d monai.apps.DecathlonDataset(root_dir\u003dDataRoot, task\u003dmyTask, section\u003d\"training\", download\u003dTrue)\n",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "## 1.2 Remove unneeded tar files ",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "! rm $DataRoot/*.tar",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "## 1.3 Check on data\nFor any of the downloaded data set above you can run cell below to check on number of data sets and the image shape",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "print(dataset.get_properties(\"numTraining\"))\nprint(dataset.get_properties(\"description\"))\n",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "print(dataset[0][\u0027image\u0027].shape)\nprint(dataset[0][\u0027label\u0027].shape)\n",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "# 2. Load images into OHIF\nIf you are using AIAA with OHIF integration, \nthen you would need to Dicom images.\nWe will convert nifti to Dicom then upload it to the pacs backend.\nFirst lets install needed packages",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "! apt-get install -y plastimatch\n! apt-get install -y dcmtk",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "\nNext we will do set some directories and do some clean up ",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "spleenDir\u003dDataRoot+\"/Task09_Spleen/imagesTs/\"\nspleenDcmDir\u003dDataRoot+\"/Task09_Spleen/imagesTsDCM/\"\ncmd \u003d \"rm \"+spleenDir+\"._spleen*\" # remove some mac files from download\n! $cmd",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "## 2.1 Convert nifti to dicoms ",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%% md\n"
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "import glob\nimport os\nfor i,patName in enumerate(os.listdir(spleenDir)): #glob.glob(spleenDir+\"/*.nii.gz\"):\n    patName\u003dpatName[:-7]\n    print(\"------converting \",patName)\n    PatDcmDir\u003dspleenDcmDir+patName\n    os.makedirs(PatDcmDir, exist_ok\u003dTrue)\n    fName\u003dspleenDir+patName+\".nii.gz\"\n    cmd \u003d \"plastimatch convert --input \" + fName + \" --output-dicom \" + PatDcmDir +\" --patient-name \"+ patName \n    cmd +\u003d\" --series-description \"+ patName \n#     cmd +\u003d\" --series-number 1\"\n#     cmd +\u003d\" --series-uid \u003carg\u003e        series UID for image metadata: string \n#     cmd +\u003d\" --patient-pos ffp\" #hfs hfp ffs ffp\n#     cmd +\u003d\" --direction-cosines rotated-3\" #{identity,rotated-{1,2,3},sheared}\n    !$cmd\n    if i\u003e1:\n        break",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "## 2.2 Push dicom to orthanc",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "cmd\u003d\u0027storescu -v +sd +r -xb -v -aet \"fromtest\" -aec \"ORTHANC\" orthanc 4242 \u0027+spleenDcmDir\n!$cmd",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.8"
    },
    "stem_cell": {
      "cell_type": "raw",
      "source": "\u003c!--- SPDX-License-Identifier: Apache-2.0 --\u003e\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}