cspb3202-hw5-histopathologic-cancer-testmhenderson.ipynb

{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.10","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# HW5 Histopathologic Cancer Test\n\n\n1. brief description of the problem, data (e.g. size and dimension, structure etc)\n\nIn this competition, we create an algorithm to identify metastatic cancer in small image patches taken from larger digital pathology scans. You may view and download the official Pcam dataset from GitHub https://github.com/basveeling/pcam. The data is provided under the CC0 License, following the license of Camelyon16.\n\n\n\n2. Exploratory data analysis showing a few visualization, histogram, etc, and a plan of analysis. Any data cleaning procedure.\n\n\n3. You model architecture and reasoning why you believe certain architecture would be suitable for this problem\n\n\n4. results (tables, figures etc) and analysis (reasoning of why or why not something worked well, also troubleshooting and hyperparameter optimization procedure summary)\n5. your conclusion.","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os\nimport cv2\nimport matplotlib.pyplot as plt\nimport matplotlib.patches as patches\nimport random\nfrom sklearn.utils import shuffle\nfrom tqdm import tqdm_notebook\n\ndata = pd.read_csv('/kaggle/input//histopathologic-cancer-detection/train_labels.csv')\ntrain_path = '/kaggle/input/histopathologic-cancer-detection/train/'\ntest_path = '/kaggle/input/histopathologic-cancer-detection/test/'\n# quick look at the label stats\ndata['label'].value_counts()\n","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:10:28.074201Z","iopub.execute_input":"2021-11-10T14:10:28.075076Z","iopub.status.idle":"2021-11-10T14:10:28.647636Z","shell.execute_reply.started":"2021-11-10T14:10:28.075030Z","shell.execute_reply":"2021-11-10T14:10:28.646856Z"},"trusted":true},"execution_count":8,"outputs":[]},{"cell_type":"code","source":"cancer_data = pd.read_csv('/kaggle/input/histopathologic-cancer-detection/train_labels.csv')\ntrain_path = '/kaggle/input/histopathologic-cancer-detection/train/'\ntest_path = '/kaggle/input/histopathologic-cancer-detection/test/'\n# quick look at the label stats\ncancer_data['label'].value_counts()\n","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2021-11-10T14:10:28.649744Z","iopub.execute_input":"2021-11-10T14:10:28.651169Z","iopub.status.idle":"2021-11-10T14:10:28.864692Z","shell.execute_reply.started":"2021-11-10T14:10:28.651122Z","shell.execute_reply":"2021-11-10T14:10:28.863740Z"},"trusted":true},"execution_count":9,"outputs":[]},{"cell_type":"markdown","source":"We will be looking at what kind of labels there are in the dataset.\n\nIn our case, the training data contains 130908 negative samples, and 89117 positive samples.  This is roughly 60:40 split in our training set. It means that we might be more biased to negative samples as we are training. We probably want to balance it out during our splits.","metadata":{}},{"cell_type":"code","source":"# Pie chart, where the slices will be ordered and plotted counter-clockwise:\nlabels = 'Negative', 'Positive'\nsizes = [130908, 89117]\nexplode = (0, 0)  # only \"explode\" the 2nd slice (i.e. 'Hogs')\n\nfig1, ax1 = plt.subplots()\nax1.pie(sizes, explode=explode, labels=labels, autopct='%1.1f%%',\n        shadow=True, startangle=90)\nax1.axis('equal')  # Equal aspect ratio ensures that pie is drawn as a circle.\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:10:28.866352Z","iopub.execute_input":"2021-11-10T14:10:28.866615Z","iopub.status.idle":"2021-11-10T14:10:29.002581Z","shell.execute_reply.started":"2021-11-10T14:10:28.866580Z","shell.execute_reply":"2021-11-10T14:10:29.001915Z"},"trusted":true},"execution_count":10,"outputs":[]},{"cell_type":"code","source":"plt.imshow(cv2.imread(train_path + \"00001b2b5609af42ab0ab276dd4cd41c3e7745b5.tif\"))","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:10:29.004062Z","iopub.execute_input":"2021-11-10T14:10:29.004448Z","iopub.status.idle":"2021-11-10T14:10:29.271108Z","shell.execute_reply.started":"2021-11-10T14:10:29.004403Z","shell.execute_reply":"2021-11-10T14:10:29.270346Z"},"trusted":true},"execution_count":11,"outputs":[]},{"cell_type":"code","source":"def readImage(path):\n    # OpenCV reads the image in bgr format by default\n    bgr_img = cv2.imread(path)\n    # We flip it to rgb for visualization purposes\n    b,g,r = cv2.split(bgr_img)\n    rgb_img = cv2.merge([r,g,b])\n    return rgb_img","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:10:29.273246Z","iopub.execute_input":"2021-11-10T14:10:29.275160Z","iopub.status.idle":"2021-11-10T14:10:29.280258Z","shell.execute_reply.started":"2021-11-10T14:10:29.275116Z","shell.execute_reply":"2021-11-10T14:10:29.279569Z"},"trusted":true},"execution_count":12,"outputs":[]},{"cell_type":"markdown","source":"Here we show the raw image in which we will convert the colors by flipping the RGB.","metadata":{}},{"cell_type":"code","source":"plt.imshow(readImage(train_path + \"00001b2b5609af42ab0ab276dd4cd41c3e7745b5.tif\"))","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:10:29.281048Z","iopub.execute_input":"2021-11-10T14:10:29.281326Z","iopub.status.idle":"2021-11-10T14:10:29.521489Z","shell.execute_reply.started":"2021-11-10T14:10:29.281289Z","shell.execute_reply":"2021-11-10T14:10:29.520759Z"},"trusted":true},"execution_count":13,"outputs":[]},{"cell_type":"markdown","source":"# Looking at the raw data\nNow that we have looked at the label distribution for training, let's take a look at the raw data. This image is in the train/directory and contains images in tif format. We will use open cv's library cv2 to read an image and show it.\n\nWe added a function called readImage becasue cv2 reads the image in BGR order or 'blue' 'green' 'red' order and we want to plot the image using matplotlib imshow which takes images in RGB order. readImage does this reordering","metadata":{}},{"cell_type":"code","source":"sd = shuffle(data)","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:10:29.522876Z","iopub.execute_input":"2021-11-10T14:10:29.523628Z","iopub.status.idle":"2021-11-10T14:10:29.557691Z","shell.execute_reply.started":"2021-11-10T14:10:29.523588Z","shell.execute_reply":"2021-11-10T14:10:29.556981Z"},"trusted":true},"execution_count":14,"outputs":[]},{"cell_type":"code","source":"sd[sd['label'] == 0].head(4)","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:10:29.558877Z","iopub.execute_input":"2021-11-10T14:10:29.559229Z","iopub.status.idle":"2021-11-10T14:10:29.579980Z","shell.execute_reply.started":"2021-11-10T14:10:29.559193Z","shell.execute_reply":"2021-11-10T14:10:29.579179Z"},"trusted":true},"execution_count":15,"outputs":[]},{"cell_type":"code","source":"fig,ax = plt.subplots(1,3)\nfor i in range(3):\n    id_ = sd[sd['label'] == 0].iloc[i]['id']\n    image_path_ = train_path + id_ + \".tif\"\n    ax[i].imshow(readImage(image_path_))\nfig.suptitle('Negative samples')","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:10:29.581258Z","iopub.execute_input":"2021-11-10T14:10:29.581568Z","iopub.status.idle":"2021-11-10T14:10:30.123453Z","shell.execute_reply.started":"2021-11-10T14:10:29.581532Z","shell.execute_reply":"2021-11-10T14:10:30.122787Z"},"trusted":true},"execution_count":16,"outputs":[]},{"cell_type":"code","source":"fig,ax = plt.subplots(1,3)\nfor i in range(3):\n    id_ = sd[sd['label'] == 1].iloc[i]['id']\n    image_path_ = train_path + id_ + \".tif\"\n    ax[i].imshow(readImage(image_path_))\nfig.suptitle('Positive samples')","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:10:30.125024Z","iopub.execute_input":"2021-11-10T14:10:30.125449Z","iopub.status.idle":"2021-11-10T14:10:30.553474Z","shell.execute_reply.started":"2021-11-10T14:10:30.125409Z","shell.execute_reply":"2021-11-10T14:10:30.552790Z"},"trusted":true},"execution_count":17,"outputs":[]},{"cell_type":"code","source":"plt.imshow(readImage(train_path + \"00001b2b5609af42ab0ab276dd4cd41c3e7745b5.tif\"))","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:10:30.554935Z","iopub.execute_input":"2021-11-10T14:10:30.555221Z","iopub.status.idle":"2021-11-10T14:10:30.847330Z","shell.execute_reply.started":"2021-11-10T14:10:30.555184Z","shell.execute_reply":"2021-11-10T14:10:30.846545Z"},"trusted":true},"execution_count":18,"outputs":[]},{"cell_type":"markdown","source":"# Data Augmentation\n\nWe can improve our results of our machine learning pipeline by having more data.  Omne techniqure for adding more data is to use the existing set of data and apply transformations to it.\n\nIn this case we will be applying the following transformations:\n* rotation\n* cropping\n* lighting\n* flip\n\n","metadata":{}},{"cell_type":"markdown","source":"## Image rotation","metadata":{}},{"cell_type":"code","source":"sample_ = '../input/histopathologic-cancer-detection/test/00006537328c33e284c973d7b39d340809f7271b.tif'\nimg_ = cv2.imread(sample_)\nprint(img_.shape)\n\n\n#apply rotation\nwidth_ = img_.shape[0]\ncenter_ = (width_//2, width_//2)\ndims_ = (width_, width_)\nrotation_ = 10\nM = cv2.getRotationMatrix2D(center_,rotation_,1)   # the center point is the rotation anchor\nimg_rot_ = cv2.warpAffine(img_,M,dims_,flags=cv2.INTER_CUBIC)\n# cv2.INTER_AREA for shrinking and cv2.INTER_CUBIC (alow) & cv2.INTER_LINEAR for zooming\n#plt.imshow(img_rot_)\n\n#showing results\nfig,ax = plt.subplots(1,2)\n\nax[0].imshow(img_)\nax[0].set_title('original')\nax[1].imshow(img_rot_)\nax[1].set_title('rotated')\n","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:10:30.851509Z","iopub.execute_input":"2021-11-10T14:10:30.853554Z","iopub.status.idle":"2021-11-10T14:10:31.391586Z","shell.execute_reply.started":"2021-11-10T14:10:30.853513Z","shell.execute_reply":"2021-11-10T14:10:31.390685Z"},"trusted":true},"execution_count":19,"outputs":[]},{"cell_type":"markdown","source":"## Cell Images\n\n","metadata":{}},{"cell_type":"markdown","source":"# Preparing train and test split\n\nWe will be using `train_test_split` from `sklearn.model_selection` to split the dataset into a training set and a validation set. \n\nThe idea behind the train_test_split below is to use the 'id' column which is the column of filenames, as the thing we want to split. We are using the `data.index` as the `y` value which is typically the label for `x`. This is just a little trick to get the row number that corresponds to the split.\n\nThe `train_test_split` function will then break up the `x` and `y` into two sets, a training set and a validation set. What we are interested in here are the indices of the rows that `sklearn` uses for the split.\n\nThe reason for using this function is the stratify part which allows one to make sure there is an equal distribution of labels in both the train and validation parts. Here we use a 0.1 split, i.e. a 9:1 split of training and validation.","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\n\ntrain_df = cancer_data.set_index('id')\ntrain_names = train_df.index.values\ntrain_labels = np.asarray(train_df['label'].values)\n\n#Should be x_train, x_val, y_train, y_val\ntr_n, val_n, tr_idx, val_idx = train_test_split(\n    train_names, range(len(train_names)),\n    test_size=0.1, stratify=train_labels,\n    random_state=123)\n\ntr_n1,val_n1, tr_idx1, val_idx1 = train_test_split(\n    cancer_data['id'].values, cancer_data.index,\n    test_size=0.1, stratify=cancer_data['label'],\n    random_state=123)\n\nprint(tr_n.shape, val_n.shape,22003/198022)\nlen(val_n), len(val_idx)\nprint(tr_n[0], tr_idx[0], tr_n1[0], tr_idx1[0])\nprint(cancer_data[cancer_data['id'] == tr_n[0]].index, tr_idx[0])\n#print(all(tr_idx == tr_idx1), all(tr_n == tr_n1))\n\n\n#all(val_n1 == val_n)\n#all(tr_n1 == tr_n)\n#data[data['id'] == tr_n[0]], val\n#data['id'].values\n#data['label'].values\n#train_n, traind_idx, validation_n, validation_idx = train_test_split(data['id'].values, )","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:10:31.395977Z","iopub.execute_input":"2021-11-10T14:10:31.396817Z","iopub.status.idle":"2021-11-10T14:10:32.144652Z","shell.execute_reply.started":"2021-11-10T14:10:31.396776Z","shell.execute_reply":"2021-11-10T14:10:32.143929Z"},"trusted":true},"execution_count":20,"outputs":[]},{"cell_type":"markdown","source":"`stratify` allos us to have the same proportion of positive and negative samples in the train set and the validation sets. We can check that is the case (see cell below). We find that dtratify keeps a 0.68 ratio of positive to negative samples in both the training and in the validation sets. And this ratio also matches that of the original distribution in our labeled data.","metadata":{}},{"cell_type":"code","source":"tr_tmp = cancer_data.iloc[tr_idx, :]\nvl_tmp = cancer_data.iloc[val_idx,:]\nprint(\n    tr_tmp[tr_tmp['label'] == 0].count(),\n    tr_tmp[tr_tmp['label'] == 1].count(),\n    vl_tmp[vl_tmp['label'] == 0].count(),\n    vl_tmp[vl_tmp['label'] == 1].count(),\n)\nprint('Ratios:')\nprint('Training:', 80205/117817)\nprint('Validation:', 8912/13091)\nprint('Original:', 40.5//59.5 )\n#40.5","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:10:32.151719Z","iopub.execute_input":"2021-11-10T14:10:32.153838Z","iopub.status.idle":"2021-11-10T14:10:32.384882Z","shell.execute_reply.started":"2021-11-10T14:10:32.153796Z","shell.execute_reply":"2021-11-10T14:10:32.384161Z"},"trusted":true},"execution_count":21,"outputs":[]},{"cell_type":"markdown","source":"## Libraries to use for classification\nWe will be using `fastai.vision` and `torchvision` models.","metadata":{}},{"cell_type":"code","source":"from fastai import *\nfrom fastai.vision import *\nfrom torchvision.models import *\n","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:10:32.389156Z","iopub.execute_input":"2021-11-10T14:10:32.389644Z","iopub.status.idle":"2021-11-10T14:10:32.399820Z","shell.execute_reply.started":"2021-11-10T14:10:32.389605Z","shell.execute_reply":"2021-11-10T14:10:32.398943Z"},"trusted":true},"execution_count":22,"outputs":[]},{"cell_type":"code","source":"import fastai\nfastai.__version__","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:10:32.401256Z","iopub.execute_input":"2021-11-10T14:10:32.401658Z","iopub.status.idle":"2021-11-10T14:10:32.414251Z","shell.execute_reply.started":"2021-11-10T14:10:32.401617Z","shell.execute_reply":"2021-11-10T14:10:32.413437Z"},"trusted":true},"execution_count":23,"outputs":[]},{"cell_type":"code","source":"arch = densenet169\nbatch_size = 128\nsz = 90 # imagesize is 96\nmodel_path = str(arch).split()[1]\n","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:10:32.415883Z","iopub.execute_input":"2021-11-10T14:10:32.416185Z","iopub.status.idle":"2021-11-10T14:10:32.421958Z","shell.execute_reply.started":"2021-11-10T14:10:32.416147Z","shell.execute_reply":"2021-11-10T14:10:32.421165Z"},"trusted":true},"execution_count":24,"outputs":[]},{"cell_type":"code","source":"df = cancer_data.copy()\ndf.columns = ['name', 'label']\ndf['name'] = train_path + df['name'] + '.tif'\n\ndf_test = pd.DataFrame([test_path + f for f in os.listdir(test_path)],\n             columns=['name'])\n","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:10:32.423819Z","iopub.execute_input":"2021-11-10T14:10:32.424122Z","iopub.status.idle":"2021-11-10T14:10:33.545371Z","shell.execute_reply.started":"2021-11-10T14:10:32.424087Z","shell.execute_reply":"2021-11-10T14:10:33.544543Z"},"trusted":true},"execution_count":25,"outputs":[]},{"cell_type":"markdown","source":"## Convert filenames to images\n\nReferences:\nhttps://docs.fast.ai/tutorial.datablock.html\nhttps://docs.fast.ai/vision.data.html\nhttps://docs.fast.ai/tutorial.vision\nhttps://docs.fast.ai/data.core.html#DataLoaders\n\nWe will be using `fast.ai` datablocks to specify the image filenames. `DataBlocl` is not specific to images, but by specifying the `ImageBlock` and `CategoryBlock` as the conversion between filenames to labels, filenames in our dataframe `df` can be converted later to actual image data, and the labels in `df` will be used as categories.\n\n`DataBlock` is really a specification, as it doesn't take in our dataframe `df` right away. This will happen when we call `datasets` on the `DataBlock`. At this point it will generate `DataSets` given the input dataframe and the specification in `DataBlock`.\n\nSince the input to `datasets` is a generic dataframe, `DataBlock` must also be given hints as to what part of the dataframe to use for the data and what part to use for the label. This is accomplished by two auxiliary functions `get_items` and `get_y`. In our case we pass two functions, one to gt the `name` and the other to get the `label` for a particular ro in the dataframe, i.e. these functions expect a row of data and we can extract different parts of the row.","metadata":{}},{"cell_type":"code","source":"from fastai.data.all import *\nfrom fastai.vision.all import *","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:41:38.273528Z","iopub.execute_input":"2021-11-10T14:41:38.273857Z","iopub.status.idle":"2021-11-10T14:41:38.279917Z","shell.execute_reply.started":"2021-11-10T14:41:38.273821Z","shell.execute_reply":"2021-11-10T14:41:38.278942Z"},"trusted":true},"execution_count":62,"outputs":[]},{"cell_type":"code","source":"fastai.vision.all.get_image_files\n    ","metadata":{"trusted":true},"execution_count":63,"outputs":[]},{"cell_type":"code","source":"dblock = DataBlock(blocks    = (ImageBlock, CategoryBlock),\n                   get_x = lambda r: r['name'],\n                   get_y     = lambda r: r['label'])","metadata":{"execution":{"iopub.status.busy":"2021-11-10T15:02:18.358545Z","iopub.execute_input":"2021-11-10T15:02:18.359040Z","iopub.status.idle":"2021-11-10T15:02:18.364496Z","shell.execute_reply.started":"2021-11-10T15:02:18.358993Z","shell.execute_reply":"2021-11-10T15:02:18.363838Z"},"trusted":true},"execution_count":74,"outputs":[]},{"cell_type":"code","source":"#dblock = DataBlock()\ndsets = dblock.datasets(df)\nlen(dsets.train),len(dsets.valid)","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:10:33.579079Z","iopub.execute_input":"2021-11-10T14:10:33.579332Z","iopub.status.idle":"2021-11-10T14:10:34.303315Z","shell.execute_reply.started":"2021-11-10T14:10:33.579299Z","shell.execute_reply":"2021-11-10T14:10:34.302651Z"},"trusted":true},"execution_count":29,"outputs":[]},{"cell_type":"code","source":"dsets.train[0]","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:10:34.304799Z","iopub.execute_input":"2021-11-10T14:10:34.305084Z","iopub.status.idle":"2021-11-10T14:10:34.497742Z","shell.execute_reply.started":"2021-11-10T14:10:34.305047Z","shell.execute_reply":"2021-11-10T14:10:34.496880Z"},"trusted":true},"execution_count":30,"outputs":[]},{"cell_type":"code","source":"dblock = DataBlock(\n    blocks=(ImageBlock, CategoryBlock),\n    get_x = lambda r: r['name'], \n    get_y = lambda r: r['label'])\ndsets = dblock.datasets(df)\ndsets.train[0]","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:10:34.498923Z","iopub.execute_input":"2021-11-10T14:10:34.499137Z","iopub.status.idle":"2021-11-10T14:11:10.098503Z","shell.execute_reply.started":"2021-11-10T14:10:34.499112Z","shell.execute_reply":"2021-11-10T14:11:10.097838Z"},"trusted":true},"execution_count":31,"outputs":[]},{"cell_type":"markdown","source":"# Batches for training\nImages are not directly used until they are needed. We can use the `dataloaders` function which works in a similar way as `datasets` above in which it can take a dataframe as an input and uses the specification given in the `DataBlock`.\n\nThis allows the images to be read in on the fly. For example, we show a batch of data below, the title on top of each image corresponds to whether it is positive (1) or negative (0.","metadata":{}},{"cell_type":"code","source":"dls = dblock.dataloaders(df)\ndls.show_batch()","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:11:10.099831Z","iopub.execute_input":"2021-11-10T14:11:10.100113Z","iopub.status.idle":"2021-11-10T14:11:15.537464Z","shell.execute_reply.started":"2021-11-10T14:11:10.100078Z","shell.execute_reply":"2021-11-10T14:11:15.536871Z"},"trusted":true},"execution_count":32,"outputs":[]},{"cell_type":"markdown","source":"# Specifying the splitter\nNote we have already shown previously how to use `train_test_split` from `sklearn`. We are going to use it here on conjunction with `DataBlock`. We use this functionality becasue `DataBlock` doesn't have a native option to stratify the data as we have done previously.\n\nTo add a specification of train and validation sets we incorporate the indices obtained from `train_test_split` as outpit to the `splitter` function below which will be passed to the specification in `DataBlock` through the `splitter` option.","metadata":{}},{"cell_type":"code","source":"def splitter(df):\n    train = df.index[~df['is_valid']].tolist()\n    valid = df.index[df['is_valid']].tolist()\n    return train,valid","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:11:15.538821Z","iopub.execute_input":"2021-11-10T14:11:15.539350Z","iopub.status.idle":"2021-11-10T14:11:15.544076Z","shell.execute_reply.started":"2021-11-10T14:11:15.539306Z","shell.execute_reply":"2021-11-10T14:11:15.543484Z"},"trusted":true},"execution_count":33,"outputs":[]},{"cell_type":"code","source":"def splitter(df):\n    tr_n, tr_idx, val_n, val_idx = train_test_split(\n        df['name'].values, df.index,\n        test_size=0.1, stratify=df['label'],\n        random_state=123)\n    return val_n.tolist(), val_idx.tolist()","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:11:15.545048Z","iopub.execute_input":"2021-11-10T14:11:15.545703Z","iopub.status.idle":"2021-11-10T14:11:15.554789Z","shell.execute_reply.started":"2021-11-10T14:11:15.545666Z","shell.execute_reply":"2021-11-10T14:11:15.553980Z"},"trusted":true},"execution_count":34,"outputs":[]},{"cell_type":"code","source":"#splitter(df)","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:11:15.556213Z","iopub.execute_input":"2021-11-10T14:11:15.557057Z","iopub.status.idle":"2021-11-10T14:11:15.565260Z","shell.execute_reply.started":"2021-11-10T14:11:15.556973Z","shell.execute_reply":"2021-11-10T14:11:15.564566Z"},"trusted":true},"execution_count":35,"outputs":[]},{"cell_type":"code","source":"dblock = DataBlock(blocks=(ImageBlock, CategoryBlock),\n                   splitter=splitter,\n                   get_x = lambda r: r['name'], \n                   get_y = lambda r: r['label'])\ndls = dblock.dataloaders(df)","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:11:15.568455Z","iopub.execute_input":"2021-11-10T14:11:15.568689Z","iopub.status.idle":"2021-11-10T14:11:55.515629Z","shell.execute_reply.started":"2021-11-10T14:11:15.568655Z","shell.execute_reply":"2021-11-10T14:11:55.514828Z"},"trusted":true},"execution_count":36,"outputs":[]},{"cell_type":"code","source":"dls.show_batch()","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:11:55.516818Z","iopub.execute_input":"2021-11-10T14:11:55.517106Z","iopub.status.idle":"2021-11-10T14:11:57.015096Z","shell.execute_reply.started":"2021-11-10T14:11:55.517070Z","shell.execute_reply":"2021-11-10T14:11:57.014378Z"},"trusted":true},"execution_count":37,"outputs":[]},{"cell_type":"markdown","source":"# Training the data with convolutional neural nets\n\nYou model architecture and reasoning why you believe certain architecture would be suitable for this problem\n\nresnet and densenet. \n\n","metadata":{}},{"cell_type":"code","source":"learn = cnn_learner(dls, resnet18)","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:11:57.016367Z","iopub.execute_input":"2021-11-10T14:11:57.016727Z","iopub.status.idle":"2021-11-10T14:12:00.820247Z","shell.execute_reply.started":"2021-11-10T14:11:57.016692Z","shell.execute_reply":"2021-11-10T14:12:00.819430Z"},"trusted":true},"execution_count":38,"outputs":[]},{"cell_type":"code","source":"x,y = to_cpu(dls.train.one_batch())\nactivs = learn.model(x)\nactivs.shape","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:12:00.821514Z","iopub.execute_input":"2021-11-10T14:12:00.822257Z","iopub.status.idle":"2021-11-10T14:12:02.446720Z","shell.execute_reply.started":"2021-11-10T14:12:00.822211Z","shell.execute_reply":"2021-11-10T14:12:02.445894Z"},"trusted":true},"execution_count":39,"outputs":[]},{"cell_type":"code","source":"activs[0]","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:12:02.448338Z","iopub.execute_input":"2021-11-10T14:12:02.448616Z","iopub.status.idle":"2021-11-10T14:12:02.473009Z","shell.execute_reply.started":"2021-11-10T14:12:02.448575Z","shell.execute_reply":"2021-11-10T14:12:02.472216Z"},"trusted":true},"execution_count":40,"outputs":[]},{"cell_type":"code","source":"arch","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:12:02.474529Z","iopub.execute_input":"2021-11-10T14:12:02.474982Z","iopub.status.idle":"2021-11-10T14:12:02.631688Z","shell.execute_reply.started":"2021-11-10T14:12:02.474939Z","shell.execute_reply":"2021-11-10T14:12:02.630884Z"},"trusted":true},"execution_count":41,"outputs":[]},{"cell_type":"code","source":"# Next, we create a convnet learner object\n# ps = dropout percentage (0-1) in the final layer\n#def getLearner():\n#    return create_cnn(imgDataBunch, arch, pretrained=True, path='.', metrics=accuracy, ps=0.5, callback_fns=ShowGraph)\n\n#ps = dropout percentage (0-1) in the final layer\nlearner = cnn_learner(dls, \n                     arch, \n                     pretrained=True, \n                     path='.', \n                     metrics=accuracy, \n                     ps=0.5) \n                     #callback_fns=ShowGraph)","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:12:02.633132Z","iopub.execute_input":"2021-11-10T14:12:02.633960Z","iopub.status.idle":"2021-11-10T14:12:06.909060Z","shell.execute_reply.started":"2021-11-10T14:12:02.633914Z","shell.execute_reply":"2021-11-10T14:12:06.908283Z"},"trusted":true},"execution_count":42,"outputs":[]},{"cell_type":"markdown","source":"# Hyperparameters\n\nhyperparameter optimization procedure summary\n\nWe use `Learning rate` for our hyperparameter\n\ntroubleshooting ","metadata":{}},{"cell_type":"code","source":"learner.lr_find(start_lr=1e-03, end_lr=3e-01, num_it=100)","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:12:06.910541Z","iopub.execute_input":"2021-11-10T14:12:06.911025Z","iopub.status.idle":"2021-11-10T14:12:51.284636Z","shell.execute_reply.started":"2021-11-10T14:12:06.910957Z","shell.execute_reply":"2021-11-10T14:12:51.283958Z"},"trusted":true},"execution_count":43,"outputs":[]},{"cell_type":"markdown","source":"Here we are using just one round to save on time. But if we were to do more rounds we can get the accuracy even higher.  Just doing one epoch of training we already get a very high score of 92%.","metadata":{}},{"cell_type":"code","source":"lr_max = 0.01302280556410551\nwd = 1e-2\nlearner.fit_one_cycle(1, lr_max=lr_max)\nlearner.save('_stage')","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:12:51.286008Z","iopub.execute_input":"2021-11-10T14:12:51.286437Z","iopub.status.idle":"2021-11-10T14:32:05.363746Z","shell.execute_reply.started":"2021-11-10T14:12:51.286395Z","shell.execute_reply":"2021-11-10T14:32:05.362904Z"},"trusted":true},"execution_count":44,"outputs":[]},{"cell_type":"code","source":"learner.recorder.plot_sched()","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:32:05.365294Z","iopub.execute_input":"2021-11-10T14:32:05.365981Z","iopub.status.idle":"2021-11-10T14:32:05.702513Z","shell.execute_reply.started":"2021-11-10T14:32:05.365934Z","shell.execute_reply":"2021-11-10T14:32:05.701815Z"},"trusted":true},"execution_count":45,"outputs":[]},{"cell_type":"markdown","source":"Next, we plot the losses\n\nhttps://docs.fast.ai/learner.html#Recorder.plot.loss","metadata":{}},{"cell_type":"code","source":"# before we continue, lets save the model at this stage\nlearner.save('_stage1')","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:32:05.704044Z","iopub.execute_input":"2021-11-10T14:32:05.704534Z","iopub.status.idle":"2021-11-10T14:32:05.944140Z","shell.execute_reply.started":"2021-11-10T14:32:05.704494Z","shell.execute_reply":"2021-11-10T14:32:05.942513Z"},"trusted":true},"execution_count":46,"outputs":[]},{"cell_type":"code","source":"!ls","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:32:05.945562Z","iopub.execute_input":"2021-11-10T14:32:05.945832Z","iopub.status.idle":"2021-11-10T14:32:06.684913Z","shell.execute_reply.started":"2021-11-10T14:32:05.945796Z","shell.execute_reply":"2021-11-10T14:32:06.684066Z"},"trusted":true},"execution_count":47,"outputs":[]},{"cell_type":"markdown","source":"# Results (tables, figures etc) and analysis (reasoning of why or why not something worked well","metadata":{}},{"cell_type":"code","source":"learner.recorder.plot_loss()","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:32:06.690640Z","iopub.execute_input":"2021-11-10T14:32:06.690875Z","iopub.status.idle":"2021-11-10T14:32:06.990505Z","shell.execute_reply.started":"2021-11-10T14:32:06.690848Z","shell.execute_reply":"2021-11-10T14:32:06.989826Z"},"trusted":true},"execution_count":48,"outputs":[]},{"cell_type":"markdown","source":"We would also like to plot the confusion matrix. Fast.ai provides an interpretation class:\n\n- https://docs.fast.ai/interpret.html","metadata":{}},{"cell_type":"code","source":"# predict the validation set with our model\ninterp = ClassificationInterpretation.from_learner(learner)\ninterp.plot_confusion_matrix(title='Confusion matrix')","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:32:06.991877Z","iopub.execute_input":"2021-11-10T14:32:06.992146Z","iopub.status.idle":"2021-11-10T14:33:01.202079Z","shell.execute_reply.started":"2021-11-10T14:32:06.992110Z","shell.execute_reply":"2021-11-10T14:33:01.201032Z"},"trusted":true},"execution_count":49,"outputs":[]},{"cell_type":"markdown","source":"To get a number for the accuracy of our predictions, we can use the accuracy function, which takes the predictions, and the labeled data to give us a percentage. \n\n$\n\\rm{accuracy} = \\frac{number of correct predictions}{Total number of predictions}\n$","metadata":{}},{"cell_type":"code","source":"preds,y, loss = learner.get_preds(with_loss=True)\n# get accuracy\nacc = accuracy(preds, y)\nprint('The accuracy is {0} %.'.format(acc))","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:33:01.206945Z","iopub.execute_input":"2021-11-10T14:33:01.207342Z","iopub.status.idle":"2021-11-10T14:33:50.970580Z","shell.execute_reply.started":"2021-11-10T14:33:01.207307Z","shell.execute_reply":"2021-11-10T14:33:50.969662Z"},"trusted":true},"execution_count":50,"outputs":[]},{"cell_type":"code","source":"from sklearn.metrics import roc_curve, auc\n# probs from log preds\nprobs = np.exp(preds[:,1])\n# Compute ROC curve\nfpr, tpr, thresholds = roc_curve(y, probs, pos_label=1)\n\n# Compute ROC area\nroc_auc = auc(fpr, tpr)\nprint('ROC area is {0}'.format(roc_auc))","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:33:50.972345Z","iopub.execute_input":"2021-11-10T14:33:50.972862Z","iopub.status.idle":"2021-11-10T14:33:50.988353Z","shell.execute_reply.started":"2021-11-10T14:33:50.972819Z","shell.execute_reply":"2021-11-10T14:33:50.987517Z"},"trusted":true},"execution_count":51,"outputs":[]},{"cell_type":"markdown","source":"# Test set\n\nIn this section we will run our model prediction against the test set.  There is a separate directory in the competition labeled `test`.\n\nThis directory just contains a set of images. We need to load this set of images into the `learner`. Fortunately there is a function in the learner that creates a data loader for our purposes. `leader.dls.test_dl` accepts a list of files and returns a DataLoaders object. And then we can use the model's `get_preds` to directly get prediction just on the files from the test set by passing the corresponding `DataLoaders` object to it.\n","metadata":{}},{"cell_type":"code","source":"\ntest_files = fastai.vision.all.get_image_files(test_path)","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:43:05.062207Z","iopub.execute_input":"2021-11-10T14:43:05.062976Z","iopub.status.idle":"2021-11-10T14:43:45.511430Z","shell.execute_reply.started":"2021-11-10T14:43:05.062934Z","shell.execute_reply":"2021-11-10T14:43:45.510670Z"},"trusted":true},"execution_count":64,"outputs":[]},{"cell_type":"code","source":"test_dl = learner.dls.test_dl(test_files)\ntest_preds = learner.get_preds(dl=test_dl)","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:49:09.607677Z","iopub.execute_input":"2021-11-10T14:49:09.607940Z","iopub.status.idle":"2021-11-10T14:51:00.723504Z","shell.execute_reply.started":"2021-11-10T14:49:09.607910Z","shell.execute_reply":"2021-11-10T14:51:00.722719Z"},"trusted":true},"execution_count":68,"outputs":[]},{"cell_type":"markdown","source":"Now that we have predictions, `test_preds`, we need to prepare them for submission. The competition gives a `sample_submission.csv` from which we need to use the correct column headings. In this case the columns should be called `id` and `label`. We also note that in the `id` column all the id's are not filenames but just the 'stem' of the file. We use `pathlib`'s stem function to return just the stem of the path.\n\n`get_preds` returns a tuple, and we just want to operate on the first element of the tuple which is a $n$ by 2 tensor, where $n$ is the number of elements in the tedt set.  We need to convert the predicted probability into a classification into one of the labels. We can do that by taking the max probability from these two predictions and assign the class to the `argmax` of these two predictions. Note in the code below `axis-1` corresponds to the second dimension in the tensor, which is 2 in our case, that is, for every element in the test set we will take the argmax for the prediction possibilities.\n\nThen we create a pandas DataFrame with columns labeled as 'id' and 'label', with the stem of the filenames in the first column and the classes from the argmax in the second column. And we save the DataFrame to a file called 'submission.csv' for Kaggle.","metadata":{}},{"cell_type":"code","source":"test_stem = [x.stem for x in test_files]","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:51:26.134930Z","iopub.execute_input":"2021-11-10T14:51:26.135748Z","iopub.status.idle":"2021-11-10T14:51:26.223322Z","shell.execute_reply.started":"2021-11-10T14:51:26.135706Z","shell.execute_reply":"2021-11-10T14:51:26.222490Z"},"trusted":true},"execution_count":69,"outputs":[]},{"cell_type":"code","source":"test_classes = np.argmax(test_preds[0], axis=1)","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:51:27.572354Z","iopub.execute_input":"2021-11-10T14:51:27.572607Z","iopub.status.idle":"2021-11-10T14:51:27.590830Z","shell.execute_reply.started":"2021-11-10T14:51:27.572578Z","shell.execute_reply":"2021-11-10T14:51:27.589858Z"},"trusted":true},"execution_count":70,"outputs":[]},{"cell_type":"code","source":"my_submission = pd.DataFrame({'id': test_stem, 'label' : test_classes})\nmy_submission.to_csv('submission.csv', index=False)","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:54:55.914053Z","iopub.execute_input":"2021-11-10T14:54:55.914754Z","iopub.status.idle":"2021-11-10T14:54:56.094421Z","shell.execute_reply.started":"2021-11-10T14:54:55.914717Z","shell.execute_reply":"2021-11-10T14:54:56.093628Z"},"trusted":true},"execution_count":71,"outputs":[]},{"cell_type":"markdown","source":"# Conclusion\n\nWe can see that using our densenet model we can quickly scan many images to look for cancerous cells appearing.  The speed of the model to scan several thousands of images is particularly impressive.  Having this model work together with a doctor would likely result in a very comprehnive review of all images to determine if there are cancerous cells or not.  Using densenet and not even tweaking it a while lot we were able to get very good scores.","metadata":{}},{"cell_type":"code","source":"!ls","metadata":{"execution":{"iopub.status.busy":"2021-11-10T14:55:21.540656Z","iopub.execute_input":"2021-11-10T14:55:21.541146Z","iopub.status.idle":"2021-11-10T14:55:22.302368Z","shell.execute_reply.started":"2021-11-10T14:55:21.541107Z","shell.execute_reply":"2021-11-10T14:55:22.301473Z"},"trusted":true},"execution_count":73,"outputs":[]},{"cell_type":"markdown","source":"## ","metadata":{}}]}