![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "# Fine-tuning Video Vision Transformers for Video Classification"
+ ],
+ "metadata": {
+ "id": "d_o0pftn5k-e"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Introduction"
+ ],
+ "metadata": {
+ "id": "4r9-V7wUAucR"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "Video classification is a task in machine learning where algorithms are trained to recognize and categorize the content of videos into predefined classes or categories by analyzing the spatial and temporal information present in the video frames. The field of video classification has been dominated by deep learning models that leverage convolutional neural networks (CNNs) for a long now. However, the emergence of transformer based models has sparked a revolution across various domains, including computer vision.\n",
+ "\n",
+ "Video classification has various applications including action recognition, surveillance, content filtering, and recommendation systems. One of the transformers based algorithms developed for video classification is ViViT, which is also one of the initial successful pure transformer architecuter model for video processing and understanding.\n",
+ "\n",
+ "The Video Vision Transformer (ViViT) is like an upgraded version of Vision Transformer (ViT), but for videos instead of just images. The reason we need a separate algorithm for videos is because ViT, the one for images, only looks at each frame individually without considering how they relate to each other over time. But understanding how frames connect in a video is crucial for truly understanding its content, so we need a special algorithm, like ViViT, that can handle that temporal aspect. It has set new benchmarks for video classification accuracy. The model outperforms previous methodologies based on deep 3D convolutional networks across a variety of video classification benchmarks."
+ ],
+ "metadata": {
+ "id": "fh5OfYw9AxKE"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Installation"
+ ],
+ "metadata": {
+ "id": "JDQFEgcHdGEW"
+ }
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {
+ "id": "REjcTBIYrpOn",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "984213f6-49cd-4089-ab98-747622723bc5"
+ },
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "\u001b[?25l \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/132.7 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m132.7/132.7 kB\u001b[0m \u001b[31m3.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[?25h Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m50.2/50.2 kB\u001b[0m \u001b[31m1.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[?25h Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m42.2/42.2 kB\u001b[0m \u001b[31m1.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[?25h Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m84.1/84.1 kB\u001b[0m \u001b[31m3.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m547.8/547.8 kB\u001b[0m \u001b[31m23.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m116.3/116.3 kB\u001b[0m \u001b[31m6.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m316.1/316.1 kB\u001b[0m \u001b[31m17.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m33.5/33.5 MB\u001b[0m \u001b[31m17.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m5.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m194.1/194.1 kB\u001b[0m \u001b[31m10.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m39.9/39.9 MB\u001b[0m \u001b[31m10.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[?25h Building wheel for pytorchvideo (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
+ " Building wheel for fvcore (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
+ " Building wheel for iopath (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
+ "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
+ "cudf-cu12 24.4.1 requires pyarrow<15.0.0a0,>=14.0.1, but you have pyarrow 17.0.0 which is incompatible.\n",
+ "gcsfs 2024.6.1 requires fsspec==2024.6.1, but you have fsspec 2024.5.0 which is incompatible.\n",
+ "ibis-framework 8.0.0 requires pyarrow<16,>=2, but you have pyarrow 17.0.0 which is incompatible.\u001b[0m\u001b[31m\n",
+ "\u001b[0m"
+ ]
+ }
+ ],
+ "source": [
+ "# Install required libraries\n",
+ "!pip install pytorchvideo evaluate accelerate transformers -q"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "We will now uninstall the latest version of torchvision and depricate it to version 0.14.1 since we are going to use packages and functions in pytorchvideo that are no longer supported in new versions."
+ ],
+ "metadata": {
+ "id": "0hFJR3RjryTY"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "!pip uninstall -y torchvision -q\n",
+ "!pip install torchvision==0.14.1 -q"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "qjHUi8uLYkzM",
+ "outputId": "b2317e9a-ff24-471c-f625-07b9a68c6e9a"
+ },
+ "execution_count": 2,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m24.2/24.2 MB\u001b[0m \u001b[31m49.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m887.5/887.5 MB\u001b[0m \u001b[31m2.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m317.1/317.1 MB\u001b[0m \u001b[31m5.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m21.0/21.0 MB\u001b[0m \u001b[31m84.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m849.3/849.3 kB\u001b[0m \u001b[31m43.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m557.1/557.1 MB\u001b[0m \u001b[31m3.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
+ "\u001b[?25h\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
+ "torchaudio 2.3.1+cu121 requires torch==2.3.1, but you have torch 1.13.1 which is incompatible.\n",
+ "torchtext 0.18.0 requires torch>=2.3.0, but you have torch 1.13.1 which is incompatible.\u001b[0m\u001b[31m\n",
+ "\u001b[0m"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "## Authentication"
+ ],
+ "metadata": {
+ "id": "Ahz4UqE5vj_R"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "from huggingface_hub import notebook_login\n",
+ "\n",
+ "notebook_login()"
+ ],
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 145,
+ "referenced_widgets": [
+ "4fe1a498f90e4b37b1b3b4ad83cf91b2",
+ "d190cb16248641bd92b8fbe0820c98b4",
+ "20d93cfe8dcf43cfa0d2e7f7a9ea2afc",
+ "6f2a4f367bcd42c6a500ab61a79e705c",
+ "7e9f00ae7acf4e28932669e044627c49",
+ "9b1c4e5b3d1145c59e4f80e21d5e1815",
+ "306b38aa31044fe2a9e37f0b8997a833",
+ "74532c0fb8f642da870af9adccdb1e53",
+ "299e031dbc564e269e9be5ea3bc1be18",
+ "e5082f35ff3a4ea984d856abb2516c8e",
+ "bcb81a9e63f04c7fb5b8f7ca53e6b2d1",
+ "4e4204bc814a4185b95aed66c8c07a3c",
+ "5625e6e3bab04879860b6421012051d6",
+ "f3f9da4cff924e85aab8886c8712952f",
+ "bbe7798a1cc6491688e2260bcf1600eb",
+ "11a99a42ad2c4602995715c7944b4fb9",
+ "c357a680046c49529bb8c2c35758d135",
+ "eb51a4be61ee4e939929f8627a416cd2",
+ "f006d7d59678475ca190801e67fa05e3",
+ "d9ab970258764074adff455c6262f527",
+ "a885db853b01424bae352ef9003d12e2",
+ "cefb31705ae34a6d9053d0dcd909fdfb",
+ "c811c761a0cb4f37882233eacdae6a13",
+ "d9380b3e2802443ca7c15c54448de1af",
+ "7f2937adfa3644fdaa7b712bb13d8ae6",
+ "73ec36b0fc2e47c0bbdbf67bb26de2b2",
+ "14e56564858d448485594597305e10b5",
+ "ce823b9584d6413493e10580e56d0c6c",
+ "a059a901f5c54557bdd7e27f5ef08156",
+ "431f4b2941ff414a90bc45c8ba485ae0",
+ "43ad112b85fc4a658d588106bbc376ba",
+ "2f6e013db6074780afacfe3f3bd0dbab"
+ ]
+ },
+ "id": "TGi9tQbNvssf",
+ "outputId": "1c100b33-3aab-40d6-e930-f41b18961fda"
+ },
+ "execution_count": 3,
+ "outputs": [
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ "VBox(children=(HTML(value='| Epoch | \n", + "Training Loss | \n", + "Validation Loss | \n", + "Accuracy | \n", + "
|---|---|---|---|
| 0 | \n", + "0.006500 | \n", + "0.005599 | \n", + "1.000000 | \n", + "
| 1 | \n", + "0.014400 | \n", + "0.005713 | \n", + "1.000000 | \n", + "
"
+ ]
+ },
+ "metadata": {}
+ }
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "trainer.save_model()\n",
+ "test_results = trainer.evaluate(test_dataset)\n",
+ "trainer.log_metrics(\"test\", test_results)\n",
+ "trainer.save_metrics(\"test\", test_results)\n",
+ "trainer.save_state()"
+ ],
+ "metadata": {
+ "id": "m4np55a3x-VT",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "04429e1e-073d-4ebd-abc5-836a20a42f02"
+ },
+ "execution_count": 44,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "***** test metrics *****\n",
+ " epoch = 1.2\n",
+ " eval_accuracy = 1.0\n",
+ " eval_loss = 0.0122\n",
+ " eval_runtime = 0:01:00.68\n",
+ " eval_samples_per_second = 1.434\n",
+ " eval_steps_per_second = 0.363\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ "We can now upload our training result to the hub."
+ ],
+ "metadata": {
+ "id": "vx5y4mKZDpWi"
+ }
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "trainer.push_to_hub()"
+ ],
+ "metadata": {
+ "id": "OaYtbggjyAx8",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 244,
+ "referenced_widgets": [
+ "ba73a0b4b78342e195c8d56188870af0",
+ "c062c3b3f62f4ac8b2c50741ef14ccbe",
+ "d24fe51b6fc4405eb38d689f4547e62f",
+ "52cc999f539f43c8876b54c4644c1cec",
+ "e577cfe812e440a8a288f7b1a1c1520c",
+ "f39e1157966946608e08a799c8ba1776",
+ "e259a5c2421646a390baf8c5ea9a1b26",
+ "fe0d33c17e8b4016bd364c64d9e2255a",
+ "f1e8b3d5970241a58e6fb31f3268725a",
+ "f855f923f187460b84d512869a8683c7",
+ "92f7179d482b472cb1ab16ee83b0295a",
+ "c97dd9d0dcf44d68b7cc6def43b9516b",
+ "6e5e6fd19af34048a4ca80f4b6a2fe5b",
+ "8bb9c14a17044ffe87ce6b65a9295bc8",
+ "c39c477555be472d979ae246eedb9c5f",
+ "c6a45a8269c44af48f6ce851b5c5c9e0",
+ "1f1a524105664da3ac4d672dc81d0567",
+ "790beefea6a4491a9337b8295f8eb712",
+ "a1f91db399f147ffa151d894ef4d1000",
+ "aa1bedb37b5f4be7b41e16b389220a43",
+ "875b0b523f794c7f8f225fae23a21b6b",
+ "9df124317af3485181f1330e89ffbc31",
+ "9cd2ebbdd3274ed89fb438433c345f11",
+ "d244bc7e15914904ad7781c3eea0185e",
+ "4f61773e6d38431b8d1e74c36b885ac8",
+ "2d3dfcf4d2ac47ba875577ed03f2aee2",
+ "c964cf0d5fae4839957134ffb3d36946",
+ "144fcc757bfc463f860298bae09bc306",
+ "3799cbb9a1a94de59bec43c3a3a036de",
+ "5b99ce1a407b4e68b4af17172737db53",
+ "8f6bb4229fa7440c9c70bd5491d6a3bd",
+ "77ca9aac841d45bda3ac1f4ff3c0a48b",
+ "b306333e0e3c4b15b0ced069b0452605",
+ "6f72ea2f36374427b02af6bba1ae73e6",
+ "cb9e13d174624cfead5d52250300f27a",
+ "97e9468aeb514dfd907453f653ebfe05",
+ "07bc06213bfe4863bece098190dc9dbe",
+ "e4c3761e09a6480280f088b00a80cbd6",
+ "0e91b058199944c9a81fe86b79cbdd91",
+ "4d76a5e5cb684d56a112365539295f70",
+ "5c5d414f943a44ad82b0ef157d0053a1",
+ "3c8b442873ed4af1b4531cc80bfc37ab",
+ "6946e469524542c7ab6c86efcd31cbdb",
+ "faab4e2a2bb64a5cb868620e32b9f806",
+ "0bc2ee4841d845b28d18dbafadc99ca6",
+ "875c797ef0fa497192224eff4e0eb232",
+ "dc24ed8ff7b343adbd501e01ec2704bf",
+ "d94720c230a44b50b3fbcb2fe43edaac",
+ "6273631d95c64f92ab11233e58f29bf7",
+ "8ab7328f473b4c2ba5ebb19723c48aba",
+ "5149e59100dd4786b7d5c13d0c9ea664",
+ "63fac523b1e4447ebf2767269f280fc1",
+ "bc5b8c56b3ab4b8bb24def8c8168b6f4",
+ "c5b1a48f0fb44fa58f88dea4c4dbd5ac",
+ "512d64b1bf2d4751a60ed8e25cb84a1a",
+ "d23b159a719642f9b11d18fa0c576085",
+ "7874fc3181b349aaa7485d8ddf11f915",
+ "1ba39f95cd7443c5a25b54c3f587ec63",
+ "2ea426d50bd848f1808a954293f98537",
+ "78e5a033fbbf409486bac54250227b67",
+ "0be3f39168084bcb98d23716a0eb11ee",
+ "111fe69f57c546a39ef0e79e4c1a750c",
+ "8c032de4c9974dedab4f3ff49ede4757",
+ "60be9ae8b1b2442fb010577fdea0ac57",
+ "efecf6471ebb431c87687702cbc6e49b",
+ "37545942ed5546c59ad8d01cfa1ffb84"
+ ]
+ },
+ "outputId": "5e9e3798-428e-4104-d512-25ded5f0692d"
+ },
+ "execution_count": 45,
+ "outputs": [
+ {
+ "output_type": "display_data",
+ "data": {
+ "text/plain": [
+ "events.out.tfevents.1723289519.c82016455c61.1590.3: 0%| | 0.00/24.9k [00:00"
+ ]
+ },
+ "metadata": {},
+ "execution_count": 50
+ }
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [
+ "predicted_class_idx = logits.argmax(-1).item()\n",
+ "print(\"Predicted class:\", model.config.id2label[predicted_class_idx])"
+ ],
+ "metadata": {
+ "id": "_5gBIfqtySkA",
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "outputId": "ab91b1d8-9132-47ab-ac98-cef2224507a9"
+ },
+ "execution_count": 51,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "name": "stdout",
+ "text": [
+ "Predicted class: BenchPress\n"
+ ]
+ }
+ ]
+ },
+ {
+ "cell_type": "code",
+ "source": [],
+ "metadata": {
+ "id": "vIop5V4GqZMQ"
+ },
+ "execution_count": null,
+ "outputs": []
+ }
+ ]
+}
\ No newline at end of file
From 5160213709f20b6f9a6542ac68d32e4ea6b00735 Mon Sep 17 00:00:00 2001
From: DiwakarBasnet