Course_on_Large_Language_Models.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<a href=\"https://colab.research.google.com/github/CogitoNTNU/course-on-large-language-models/blob/main/Course_on_Large_Language_Models.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9sGjEWSy1FrY"
      },
      "source": [
        "# Course on Large Language Models\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nGJkS1ctKNvq"
      },
      "source": [
        "**NOTE:** You're only meant to change code marked with \"# TODO:\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fEKMx87cR105"
      },
      "source": [
        "## Table of Contents\n",
        "\n",
        "1. **Setting Up**\n",
        "    - API Key Configuration\n",
        "    - Connecting to OpenAI API\n",
        "2. **Exploring the API**\n",
        "    - Creating Chat Completions\n",
        "    - Understanding Completion Parameters\n",
        "3. **Prompt Engineering**\n",
        "    - Crafting Effective Prompts\n",
        "    - Strategies and Best Practices\n",
        "4. **Advanced Techniques**\n",
        "    - Utilizing Embeddings\n",
        "    - Function Calling in LLMs\n",
        "5. **Extras**\n",
        "    - Creating an API key\n",
        "    - Local Development with LLMs\n",
        "    - Context Windows\n",
        "    - Fine-Tuning LLMs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pSiOx5S_DYlf"
      },
      "source": [
        "## Part 0: Setup\n",
        "To be able to use OpenAI one needs to configure an API key to the be allowed responses to requests. Remember not to commit this key to any repository or upload it as OpenAI will disable the key if it is found, and others can use it to make requests that you or your organisation (Cogito) will pay for."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LlRpazpLEtP2"
      },
      "outputs": [],
      "source": [
        "%pip install openai\n",
        "%pip install langchain\n",
        "%pip install numpy\n",
        "%pip install python-dotenv"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Laq8ql09DtyE"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "from dotenv import load_dotenv\n",
        "\n",
        "load_dotenv()\n",
        "\n",
        "# Once you add your API key below, make sure to not share it with anyone! The API key should remain private.\n",
        "OPENAI_API_KEY: str = os.getenv(\"OPENAI_API_KEY\")\n",
        "\n",
        "# There are many different models to try out \"gpt-4\", \"gpt-4-turbo-preview\", \"gpt-3.5-turbo\"\n",
        "MODEL_NAME: str = \"gpt-4o-mini\"\n",
        "\n",
        "if not OPENAI_API_KEY:\n",
        "  print(\"[ERROR] The key is not configured correctly\")\n",
        "else:\n",
        "  print(\"[SUCCESS] API Key is configured correctly.\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AJ7gr0Q81D-a"
      },
      "outputs": [],
      "source": [
        "from openai import OpenAI\n",
        "\n",
        "client = OpenAI(\n",
        "  api_key=OPENAI_API_KEY,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PrJPFSSqDWgw"
      },
      "source": [
        "## Part 1: API Connections (10 min)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Mj506eumJEMT"
      },
      "outputs": [],
      "source": [
        "completion = client.chat.completions.create(\n",
        "  model=MODEL_NAME,\n",
        "  messages=[\n",
        "    {\"role\": \"system\", \"content\": \"You are a poetic assistant, skilled in explaining complex AI concepts with creative flair.\"},\n",
        "    {\"role\": \"user\", \"content\": \"Create a limerick about Large Language Models\"}\n",
        "  ]\n",
        ")\n",
        "print(\"The Answer for the language model \")\n",
        "print(completion)\n",
        "print(\"\\nThe answer of the model: \")\n",
        "print(completion.choices[0].message.content)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nnVvFY5KTBzL"
      },
      "source": [
        "## Part 2: Understanding Completion Parameters (15 min)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TXb8iu43y2Bu"
      },
      "source": [
        "### Key Parameters:\n",
        "* **Model Name:** Specifies the particular model version you want to use (e.g., text-davinci-003). Different models have varying capabilities, sizes, and costs.\n",
        "\n",
        "* **Messages:** The list of input text that you provide to the model. This is where the art of prompt engineering comes into play, guiding the model to generate the desired output.\n",
        "\n",
        "* **Temperature:** Controls the randomness of the output. A higher temperature leads to more varied responses, while a lower temperature results in more deterministic outputs. It's typically set between 0 and 2.\n",
        "\n",
        "* **Max Tokens:** Determines the maximum length of the model's response, measured in tokens (words or pieces of words). This helps control output verbosity.\n",
        "\n",
        "* **Top P:** Influences sample diversity by only considering the top P percent of probability mass when generating responses. Adjusting this can affect the creativity and relevance of the output.\n",
        "\n",
        "* **Frequency Penalty:** Discourages repetition by penalizing words based on their frequency in the text so far. This can help generate more diverse and interesting responses.\n",
        "\n",
        "* **Presence Penalty:** Similar to frequency penalty but penalizes based on the presence of words, encouraging the model to introduce new concepts and terms.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wb3V46jsfYZW"
      },
      "source": [
        "\n",
        "### **Task 2.1** Experimenting with Parameters\n",
        "Now that you're familiar with the parameters that can influence the behavior of LLMs, let's put this knowledge to the test. Your task is to experiment with these parameters to see firsthand how they affect the model's outputs.\n",
        "\n",
        "**Choose a Prompt:** *Start with a simple prompt, such as asking the model to write a short story about a space adventure.*\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3_U5DyXgkrUr"
      },
      "outputs": [],
      "source": [
        "# TODO: Fill in your own prompt\n",
        "prompt: str = \"Write a paragraph about a space adventure\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F1DQV5OFkq7h"
      },
      "source": [
        "### **Task 2.2**\n",
        "*Vary the Temperature: Generate three completions using temperatures of 0.0, 1.0, and 2.0. Observe how the creativity and variability of the responses change.*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "I1EY_E3ofc7U"
      },
      "outputs": [],
      "source": [
        "# TODO: Change the temperature\n",
        "TEMPERATURE: float = 1.0\n",
        "\n",
        "completion = client.chat.completions.create(\n",
        "  model=MODEL_NAME,\n",
        "  temperature=TEMPERATURE,\n",
        "  messages=[\n",
        "    {\"role\": \"user\", \"content\": prompt}\n",
        "  ]\n",
        ")\n",
        "output = completion.choices[0].message.content\n",
        "print(f\"The Model responded with: '{output}'\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TfXf7CeYi27x"
      },
      "source": [
        "\n",
        "### **Task 2.3**\n",
        "*Adjust Max Tokens: Try generating responses with different limits on length, such as 50, 100, and 2000 tokens, to see how it impacts the detail and depth of the story.*\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lQPLKOVDjecx"
      },
      "outputs": [],
      "source": [
        "# TODO: Change the MAX_TOKENS\n",
        "MAX_TOKENS: int = 50\n",
        "\n",
        "completion = client.chat.completions.create(\n",
        "  model=MODEL_NAME,\n",
        "  max_tokens=MAX_TOKENS,\n",
        "  messages=[\n",
        "    {\"role\": \"user\", \"content\": prompt}\n",
        "  ]\n",
        ")\n",
        "output = completion.choices[0].message.content\n",
        "print(f\"The Model responded with: '{output}'\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hM0XCIVhje2V"
      },
      "source": [
        "\n",
        "### **Task 2.4**\n",
        "*Experiment with Top P, Frequency Penalty, and Presence Penalty: Adjust these parameters to explore their effects on repetition, novelty, and thematic diversity.*\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yMhkJ1u2jg5k"
      },
      "outputs": [],
      "source": [
        "# TODO: Change the different parameters and check effect on output\n",
        "# TOP_P can be any float number between 0 and 1\n",
        "TOP_P: float = 0.1\n",
        "# FREQUENCY_PENALTY can be any float Number between -2.0 and 2.0.\n",
        "FREQUENCY_PENALTY: float = 0\n",
        "# PRESENCE_PENALTY  can be any float Number between -2.0 and 2.0.\n",
        "PRESENCE_PENALTY: float = 0\n",
        "\n",
        "completion = client.chat.completions.create(\n",
        "  model=MODEL_NAME,\n",
        "  top_p=TOP_P,\n",
        "  frequency_penalty=FREQUENCY_PENALTY,\n",
        "  presence_penalty=PRESENCE_PENALTY,\n",
        "\n",
        "  messages=[\n",
        "    {\"role\": \"user\", \"content\": prompt}\n",
        "  ]\n",
        ")\n",
        "output = completion.choices[0].message.content\n",
        "print(f\"The Model responded with: '{output}'\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3fKfM7-gjgbk"
      },
      "source": [
        "\n",
        "Reflect on how each parameter influenced the model's output. This exercise will enhance your understanding of how to control and guide the AI to achieve results that best fit your objectives."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WwGSpodOb4LR"
      },
      "source": [
        "## Part 3: Prompt engineering (15 min)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lUeHYM4lS4jh"
      },
      "source": [
        "\n",
        "Prompt engineering is an art and science of designing inputs that guide Large Language Models (LLMs), such as Generative Pre-trained Transformer (GPT), to produce specific, high-quality responses or outputs. This process is foundational in the field of artificial intelligence because the precision with which we articulate our prompts significantly affects the AI's performance. A well-crafted prompt can lead to outputs that are not only accurate but also creative and contextually relevant, showcasing the model's capabilities to their fullest extent.\n",
        "\n",
        "### Engaging with Prompt Engineering\n",
        "Before we dive into specific tactics for effective prompt engineering, it's important to understand that the goal is to communicate with the model in its language. This means being clear, direct, and detailed in your requests.\n",
        "\n",
        "#### Tactics:\n",
        "<ul> \n",
        "    <li> <b>Include details in your query</b> to get more relevant answers.  \n",
        "        <details>\n",
        "            <summary>Example</summary>\n",
        "            Often people ask questions that are too broad or vague. Remember, the AI can't read your mind ;) \n",
        "            <br>\n",
        "            Your goal is to extract specific information from the AI.\n",
        "            <p> <i>Bad:</i> \"Tell me about dogs.\"</p>\n",
        "            <p> <i>Good:</i> \"Provide a detailed comparison between the adaptability, exercise needs, and temperament of Labrador Retrievers and Border Collies for potential dog owners.\"</p>\n",
        "        </details>\n",
        "    </li>\n",
        "    <li> <b>Ask the model to adopt a persona</b> for more tailored responses. \n",
        "        <details>\n",
        "            <summary>Example</summary>\n",
        "            Your goal is to make the interaction more engaging or specific.\n",
        "            <p> <i>Bad:</i> \"Explain quantum physics.\"</p>\n",
        "            <p> <i>Good:</i> \"Pretend you're a renowned physicist explaining the concepts of quantum physics to a high school student in a way that's easy to understand.\"</p>\n",
        "        </details>\n",
        "    </li>\n",
        "    <li> <b>Use delimiters</b> to clearly indicate distinct parts of the input.\n",
        "        <details>\n",
        "            <summary>Example</summary>\n",
        "            Your goal is to organize a multi-part question.\n",
        "            <p> <i>Bad:</i> \"What is the capital of France and tell me about its history.\"</p>\n",
        "            <p> <i>Good:</i> \"Question 1: What is the capital of France? | Question 2: Provide a brief history of the capital.\"</p>\n",
        "        </details>\n",
        "    </li>\n",
        "    <li> <b>Specify the steps</b> required to complete a task.\n",
        "        <details>\n",
        "            <summary>Example</summary>\n",
        "            Your goal is to get a walkthrough.\n",
        "            <p> <i>Bad:</i> \"How to bake a cake.\"</p>\n",
        "            <p> <i>Good:</i> \"List all the steps necessary to bake a chocolate cake, then create a list of needed ingredients with quantities, and baking time. Before estimating total time needed.\"</p>\n",
        "        </details>\n",
        "    </li>\n",
        "    <li> <b>Provide examples</b> to illustrate the type of response you're seeking. \n",
        "        <details>\n",
        "            <summary>Example</summary>\n",
        "            Your goal is to clarify your expectations.\n",
        "            <p> <i>Bad:</i> \"Generate a catchy slogan for my product.\"</p>\n",
        "            <p> <i>Good:</i> \"Generate a catchy slogan for my eco-friendly water bottle product. For example, something like 'Hydrate Sustainably' or 'Drink Green, Live Clean'.\"</p>\n",
        "        </details>\n",
        "    </li>\n",
        "    <li> <b>Specify the desired length</b> of the output to control verbosity. \n",
        "        <details>\n",
        "            <summary>Example</summary>\n",
        "            Your goal is to manage the depth of the response.\n",
        "            <p> <i>Bad:</i> \"Write an article on climate change.\"</p>\n",
        "            <p> <i>Good:</i> \"Write a concise 300-word article on the impacts of climate change on global weather patterns.\"</p>\n",
        "        </details>\n",
        "    </li>\n",
        "</ul>\n",
        "\n",
        "\n",
        "### Applying What We've Learned\n",
        "Now that we've outlined the key tactics for effective prompt engineering, let's put this knowledge into practice."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P8-IT4a3YB6j"
      },
      "source": [
        "### **Task 3.1**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pSS-0U6sgp7v"
      },
      "source": [
        "Imagine you're working on the Cogito Project **TutorAI**, a cutting-edge AI tool designed to support students in their study efforts by creating concise, informative flashcards from dense academic texts. Your challenge is to engineer a prompt that instructs the LLM to distill complex material into easy-to-review flashcards, focusing on key concepts, definitions, and examples relevant to an upcoming exam.\n",
        "\n",
        "* **Extract Key Concepts and Definitions:** The AI must identify and summarize the main ideas and definitions found in a given academic text. This involves discerning the most important points that are crucial for understanding the subject matter.\n",
        "\n",
        "* **Format the Information for Flashcards:** The output should be structured in a way that is suitable for flashcard creation. Each flashcard will have a term or concept on one side and its definition or explanation on the other side, along with an example if appropriate.\n",
        "\n",
        "* **Control the Length:** Each flashcard content (term/definition/example) should be concise, aiming for no more than 50 words per side to facilitate quick review and memorization.\n",
        "\n",
        "This task will test your ability to use detailed queries, specify a structure, and control the output length—all crucial aspects of prompt engineering. Remember, the effectiveness of your prompt will directly influence the quality and relevance of the AI's response. Good luck!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "931QTomvYCpj"
      },
      "outputs": [],
      "source": [
        "book_paragraphs: str = \"\"\"\n",
        "Chapter 1 - Epic Introduction\n",
        "Since the dawn of time, humans have tried to define how we think, and this struggle has led us to create artificial intelligence. Historically, four approaches to artificial intelligence have been followed, each described below.\n",
        "\n",
        "Acting Humanly\n",
        "If we can't distinguish between a computer and a human, the computer is said to act humanly. The computer's capability to act humanly can be tested by performing a turing test. A computer passes the turing test if a human interrogator cannot tell whether he is communicating with a computer or a person. To pass a turing test, the computer would need to possess the following capabilities:\n",
        "\n",
        "Natural language processing to enable it to communicate successfully.\n",
        "Knowledge representation to store what it knows or hears.\n",
        "Automated reasoning to use the stored information to draw conclusions.\n",
        "Machine learning to adapt to new circumstances and to detect patterns.\n",
        "\n",
        "Thinking humanly\n",
        "To make a computer think like a human, we must know how humans think. The computers ability to think humanly can be determined by comparing the computer's input-output mechanism by the corresponding human behaviour.\n",
        "\n",
        "Acting Rationally\n",
        "An agent is something that acts. A rational agent is an agent that does the right thing based on what it knows, its functions, and the surrounding environment; it acts so that it achieves the best expected outcome.\n",
        "\n",
        "Thinking rationally\n",
        "Using sound logic rules to reach the right conclusion.\n",
        "\n",
        "A relevant quote, demonstrating the logical rule of modus ponens: \"Socrates is a man; all men are mortal; therefore, Socrates is mortal.\n",
        "\"\"\"\n",
        "\n",
        "def generate_flashcards_from_paragraphs(paragraph: str) -> str:\n",
        "  completion = client.chat.completions.create(\n",
        "    model=MODEL_NAME,\n",
        "    messages=[\n",
        "      # TODO: Create a prompt or combination of \"system\" and \"user\" prompts to achieve tasks objectives\n",
        "      {\"role\": \"system\", \"content\": \"\"},\n",
        "      {\"role\": \"user\", \"content\": book_paragraphs},\n",
        "    ]\n",
        "  )\n",
        "  return completion.choices[0].message.content\n",
        "\n",
        "flashcards = generate_flashcards_from_paragraphs(book_paragraphs)\n",
        "print(f\"The Model responded with the following flashcards: \\n'{flashcards}'\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mbtcDwricare"
      },
      "source": [
        "## Part 4: Embeddings (15 min)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UdYuCgxLcdZW"
      },
      "outputs": [],
      "source": [
        "def create_embedding(prompt: str, model=\"text-embedding-ada-002\") -> list[float]:\n",
        "    return client.embeddings.create(model=model, input=prompt).data[0].embedding\n",
        "\n",
        "print(create_embedding(\"This is an embedding!\"))\n",
        "\n",
        "database: list[list[list[float]], str] = []\n",
        "\n",
        "# Create embedding-text key-value pairs and add them to the database\n",
        "corresponding_text_1 = \"This is an embedding!\"\n",
        "embedding_1 = create_embedding(corresponding_text_1)\n",
        "database.append([embedding_1, corresponding_text_1])\n",
        "\n",
        "corresponding_text_2 = \"Sverre is CTO of Cogito NTNU\"\n",
        "embedding_2 = create_embedding(corresponding_text_2)\n",
        "database.append([embedding_2, corresponding_text_2])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dfi_Eg6FUB7D"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "\n",
        "def cosine_similarity(a: list[float], b: list[float]) -> float:\n",
        "    \"\"\" \n",
        "    Takes 2 vectors a, b and finds how similar they are using the cosine similarity \n",
        "\n",
        "    Args:\n",
        "    a (list[float]): A list of floats\n",
        "    b (list[float]): A list of floats\n",
        "\n",
        "    Returns:\n",
        "        The similarity of the two vectors a and b described as a float between 0 and 1\n",
        "    \n",
        "    \"\"\"\n",
        "    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))\n",
        "\n",
        "def search_docs(query: str, database: list[list[list[float]], str], top_k: int=1):\n",
        "    \"\"\"\n",
        "    Searches the database for the most similar documents to the query\n",
        "\n",
        "    Args:\n",
        "        query (str): The query to search for\n",
        "        database (list[list[list[float]], str]): The database to search in\n",
        "        top_k (int): The number of documents to return\n",
        "    Returns:\n",
        "        A list of the top_k most similar documents to the query\n",
        "    \"\"\"\n",
        "    query_embedding = create_embedding(query)\n",
        "    results = []\n",
        "    for (doc_embedding, doc) in database:\n",
        "        similarity = cosine_similarity(query_embedding, doc_embedding)\n",
        "        results.append((similarity, doc))\n",
        "    return sorted(results, reverse=True)[:top_k]\n",
        "\n",
        "search_docs(\"Who is the CTO of Cogito?\", database)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V4yWzB9s9caE"
      },
      "source": [
        "### **Task 4.1**\n",
        "*Create a new embedding with some text of your choice, and add it to the database. See if you can make the model find it.*\n",
        "\n",
        "<details>\n",
        "    <summary><strong>Hint:</strong></summary>\n",
        "      - Look at the previous two cells\n",
        "</details>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "g-1zWV3u--Ih"
      },
      "outputs": [],
      "source": [
        "# TODO: Create an embedding for some text and append it to the database\n",
        "\n",
        "while True:\n",
        "  user_input: str = input(\"What would you like to ask the model: \")\n",
        "\n",
        "  if user_input == \"q\":\n",
        "      print(\"[SUCCESS] Shut down\")\n",
        "      break\n",
        "\n",
        "  answer = search_docs(user_input, database, top_k=1)\n",
        "\n",
        "  print(f\"The AI gave the answer: {answer}\\n\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bw5KIpeihb6B"
      },
      "source": [
        "## Part 5: Function Calling (35 min)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4vsalQXOSh1N"
      },
      "source": [
        "Example of Yr application using Langchain"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JiJQuCniheLG"
      },
      "outputs": [],
      "source": [
        "from langchain.llms.openai import OpenAI\n",
        "from langchain.tools import StructuredTool\n",
        "from langchain.agents import AgentType\n",
        "from langchain.memory import ConversationBufferMemory\n",
        "from langchain.agents import initialize_agent\n",
        "import json\n",
        "import matplotlib.pyplot as plt\n",
        "import requests\n",
        "\n",
        "# Give the agent a list of tools to use\n",
        "def get_weather(latitude: float, longitude: float) -> str:\n",
        "    \"\"\"\n",
        "    Narrate the story based on the given prompt.\n",
        "    \"\"\"\n",
        "\n",
        "    url = f\"https://api.met.no/weatherapi/locationforecast/2.0/compact?lat={latitude}&lon={longitude}\"\n",
        "    headers = {\n",
        "        'User-Agent': 'Mozilla/5.0 (compatible; LangChain/1.0; +https://langchain.ai/)'\n",
        "    }\n",
        "    response = requests.get(url, headers=headers)\n",
        "\n",
        "    plot_weather_data(response.text)\n",
        "\n",
        "    # Cap the response at 2000 characters\n",
        "    response = response.text[:2000]\n",
        "\n",
        "    return response\n",
        "\n",
        "\n",
        "def plot_weather_data(weather_data):\n",
        "        # Parse the JSON data\n",
        "        data = json.loads(weather_data)\n",
        "\n",
        "        # Extract the temperature data from nested JSON\n",
        "        temperatures = [data['properties']['timeseries'][i]['data']['instant']['details']['air_temperature'] for i in range(len(data['properties']['timeseries']))]\n",
        "\n",
        "        # Get the temperature at every round hour\n",
        "        temperatures = temperatures[::2]\n",
        "\n",
        "        if len(temperatures) > 24:\n",
        "            temperatures = temperatures[:24]\n",
        "\n",
        "        # Plot the data\n",
        "        plt.plot(temperatures, 'r-')\n",
        "        plt.xlabel('Time')\n",
        "        plt.ylabel('Temperature (C)')\n",
        "        plt.legend()\n",
        "        plt.show()\n",
        "\n",
        "\n",
        "def average_temperature(temperatures: list[int]) -> float:\n",
        "    sum = 0\n",
        "    for temp in temperatures:\n",
        "      sum += temp\n",
        "    return sum // len(temperatures)\n",
        "\n",
        "\n",
        "tools: list[StructuredTool] = [\n",
        "    StructuredTool.from_function(\n",
        "        name= \"Weather at location\",\n",
        "        func=get_weather,\n",
        "        description=\"Get the weather at a location given a latitude and longitude.\",\n",
        "    ),\n",
        "    StructuredTool.from_function(\n",
        "        name= \"Find average temperature\",\n",
        "        func=average_temperature,\n",
        "        description=\"Get average temperature from a list of temperatures\",\n",
        "    ),\n",
        "]\n",
        "\n",
        "# Make a memory for the agent to use\n",
        "memory = ConversationBufferMemory(memory_key=\"chat_history\")\n",
        "\n",
        "llm = OpenAI(temperature=0, openai_api_key=OPENAI_API_KEY)\n",
        "agent_chain = initialize_agent(\n",
        "    tools,\n",
        "    llm,\n",
        "    agent=AgentType.STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION,\n",
        "    verbose=True,\n",
        "    memory=memory,\n",
        "    max_iterations=10,\n",
        ")\n",
        "\n",
        "def run_agent(prompt: str) -> str:\n",
        "    \"\"\"Run the agent chain.\"\"\"\n",
        "    if not isinstance(prompt, str):\n",
        "        raise TypeError(\"Prompt must be a string.\")\n",
        "\n",
        "    if (len(prompt) < 1) or (len(prompt) > 1000):\n",
        "        raise ValueError(\"Prompt must be at least 1 character or less than 1000 characters.\")\n",
        "\n",
        "    result = agent_chain.run(prompt)\n",
        "    return result\n",
        "\n",
        "run_agent(\"Give me the weather at lat 63.41710242319078, long -10.4066603487495 and get the average temperature\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LOq8xApISqe1"
      },
      "source": [
        "#### Task 5.1\n",
        "*Create your own function for dividing two numbers and add it to the AI model tools. \n",
        "Try to ask it to do several additions or a combination of dividings and additions.*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "background_save": true
        },
        "id": "itPrDSoSSm5k"
      },
      "outputs": [],
      "source": [
        "from langchain.llms.openai import OpenAI\n",
        "from langchain.tools import StructuredTool\n",
        "from langchain.agents import AgentType\n",
        "from langchain.memory import ConversationBufferMemory\n",
        "from langchain.agents import initialize_agent\n",
        "\n",
        "\n",
        "def add(a: int, b: int) -> int:\n",
        "    \"\"\"Add two numbers\"\"\"\n",
        "    return a + b\n",
        "\n",
        "# TODO: Create the divide function:\n",
        "\n",
        "# TODO: Give the agent the new StructuredTool to use\n",
        "tools: list[StructuredTool] = [\n",
        "    StructuredTool.from_function(\n",
        "        name= \"Add two numbers\",\n",
        "        func=add,\n",
        "        description=\"Adds two numbers together.\",\n",
        "    ),\n",
        "]\n",
        "\n",
        "# Make a memorybuffer for the agent to use\n",
        "memory = ConversationBufferMemory(memory_key=\"chat_history\")\n",
        "\n",
        "llm = OpenAI(temperature=0, openai_api_key=OPENAI_API_KEY)\n",
        "agent_chain = initialize_agent(\n",
        "    tools,\n",
        "    llm,\n",
        "    agent=AgentType.STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION,\n",
        "    verbose=True, # Change to False if you do not want to see the chain process and only output\n",
        "    memory=memory,\n",
        "    max_iterations=3, # Number of times the model has to think about its answer\n",
        "    )\n",
        "\n",
        "def run_agent(prompt: str) -> str:\n",
        "    \"\"\"Run the agent.\"\"\"\n",
        "    if not isinstance(prompt, str):\n",
        "        raise TypeError(\"Prompt must be a string.\")\n",
        "\n",
        "    if (len(prompt) < 1) or (len(prompt) > 1000):\n",
        "        raise ValueError(\"Prompt must be at least 1 character or less than 1000 characters.\")\n",
        "\n",
        "    result = agent_chain.run(prompt)\n",
        "    return result\n",
        "\n",
        "\n",
        "while True:\n",
        "  user_input: str = input(\"What would you like to ask the model: \")\n",
        "\n",
        "  if user_input == \"q\":\n",
        "      print(\"[SUCCESS] Shutting down...\")\n",
        "      break\n",
        "\n",
        "  answer = run_agent(user_input)\n",
        "  print(f\"The AI gave the answer: {answer}\\n\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ExQMSdXYSyVe"
      },
      "source": [
        "#### Task 5.2\n",
        "*Work together with others and create something cool, try to utilize the different lesseons you have learned examples are:*\n",
        "* Create external API access some live data\n",
        "* Create more complex math operations to do calculus\n",
        "* Create bash scripts to create folders or organize a folder\n",
        "* Access a database for getting info\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6ClLpQnGTlcX"
      },
      "outputs": [],
      "source": [
        "# TODO: Copy relevant code from this notebook and create something"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eq8bhGmey8h9"
      },
      "source": [
        "## Extras (out of this workshops scope):\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-6shmlVr_Yd6"
      },
      "source": [
        "### Creating an API key\n",
        "If you want to start using these models in your own applications you will need to create a user at OpenAI, create an API key and add credits.\n",
        "[Create API key here](https://platform.openai.com/api-keys)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Coyz5uZE1vYd"
      },
      "source": [
        "\n",
        "### Running local LLMs\n",
        "For those interested in experimenting with Large Language Models (LLMs) without incurring the costs associated with API calls to services like OpenAI's, or dealing with sensitive or proprietary data, running pre-trained models on your own hardware presents a viable alternative. The open-source community, particularly [Hugging Face's](https://huggingface.co/models) Transformers library, offers access to a wide range of models, including some developed by leading tech companies.\n",
        "\n",
        "One of the standout models available is Google's FLAN-T5-XL, part of the T5 (Text-to-Text Transfer Transformer) family, which has been fine-tuned for a broad set of tasks. This model combines the flexibility of T5's architecture with training on a mixture of supervised and unsupervised tasks, making it particularly adept at understanding and generating human-like text.\n",
        "\n",
        "To get started with using FLAN-T5-XL or any other model from the Transformers library, you will need to install the necessary packages and understand how to load and interact with the model. Below is a basic Python script that demonstrates how to set up and use FLAN-T5-XL for generating text based on input prompts:\n",
        "```python\n",
        "import sys\n",
        "from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, GenerationConfig\n",
        "\n",
        "line = 'What is the value of being accepted into Cogito NTNU, Norway's largest technical AI student organisation, in the middle of an AI revolution?'\n",
        "\n",
        "model_name = 'google/flan-t5-xl'\n",
        "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
        "model = AutoModelForSeq2SeqLM.from_pretrained(model_name)\n",
        "\n",
        "config = GenerationConfig(max_new_tokens=200)\n",
        "for line in sys.stdin:\n",
        "    tokens = tokenizer(line, return_tensors=\"pt\")\n",
        "    outputs = model.generate(**tokens, generation_config=config)\n",
        "    print(tokenizer.batch_decode(outputs, skip_special_tokens=True))\n",
        "```\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l3nKpepS9Xmw"
      },
      "source": [
        "### Context windows\n",
        "\n",
        "The context window refers to the maximum amount of text (measured in tokens) the model can consider at one time when generating responses or performing tasks. This limit is intrinsic to the model's architecture and significantly influences how we design prompts and interpret model outputs.\n",
        "\n",
        "#### Significance of the Context Window\n",
        "The size of the context window determines how much information the model can \"see\" and use at any given moment. For example, GPT-3 has a context window of 2048 tokens. This means it can consider up to 2048 tokens of preceding text to generate its responses. The implications are twofold:\n",
        "\n",
        "* **Prompt Design:** When crafting prompts for an LLM, it's vital to ensure that the most relevant information is within the model's context window. Information beyond this limit won't influence the model's output, emphasizing the need for concise and focused prompt design.\n",
        "\n",
        "* **Sequential Tasks:** For tasks requiring more information than the context window allows, you may need to design a series of prompts that build on each other, ensuring each segment of the task remains within the model's view.\n",
        "\n",
        "While advancements have led to models supporting context windows surpassing 100,000 tokens (gpt-4 and other open source ones), challenges persist. Specifically, such models tend to focus on the beginning and end of the provided text, potentially underutilizing the middle portion. This is know as [lost in the middle](https://arxiv.org/pdf/2307.03172.pdf).\n",
        "\n",
        "#### New insights by a Operative System inspired model\n",
        "[MemGPT](https://memgpt.readme.io/docs/index) introduces a strategic approach to memory management, organized around two core concepts relevant to understanding context windows in LLMs:\n",
        "\n",
        "* **Memory Hierarchy:** It segments memory into two types: a \"main context\" analogous to RAM, which is smaller and faster, and an \"external context\" similar to disk storage, which is larger but slower. This structure necessitates the deliberate transfer of information between these contexts, using virtual memory.\n",
        "\n",
        "* **Process Management:** Similar to an operating system's role in managing tasks, MemGPT regulates the flow of information between the memory segments, the LLM, and users, ensuring efficient handling of processes."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Oc8BDSKT1y-C"
      },
      "source": [
        "### Fine-tuning Large Language Models\n",
        "Fine-tuning is a process that adjusts a pre-trained model to a specific task or dataset, enhancing its ability to perform on tasks it wasn't specifically trained for initially. This method leverages the general understanding that the model has developed during its initial training phase, applying it to a more focused domain or problem set. Fine-tuning can significantly improve the performance of LLMs on specialized tasks, making it a powerful tool for developers and researchers.\n",
        "\n",
        "#### Why Fine-tune?\n",
        "Customization: Tailors the model to understand and generate responses based on specific jargon, styles, or formats unique to your dataset.\n",
        "Improved Performance: Enhances the model's accuracy and efficiency on tasks that may differ from the data it was originally trained on.\n",
        "Cost-Effectiveness: Utilizes the foundational knowledge the model has gained, reducing the need for training from scratch on vast datasets.\n",
        "\n",
        "1. **How to Fine-tune an LLM:**\n",
        "Select a Pre-trained Model: Choose a model that closely aligns with your task in terms of language and domain. Models available on platforms like Hugging Face offer a good starting point.\n",
        "\n",
        "2. **Prepare Your Dataset:** Your dataset should be representative of the task at hand and formatted in a way that the model can understand. It typically involves splitting the data into training, validation, and test sets.\n",
        "\n",
        "3. **Customize Training Parameters:** Adjust parameters such as learning rate, batch size, and the number of epochs to balance between retaining learned knowledge and adapting to the new dataset.\n",
        "\n",
        "4. **Train the Model:** Use a suitable environment and framework, like PyTorch or TensorFlow, along with Hugging Face's Transformers library, to fine-tune the model on your dataset.\n",
        "\n",
        "5. **Evaluate and Iterate:** Test the model's performance on a separate validation set, and iteratively adjust your approach based on the results.\n",
        "\n",
        "An example of this using OpenAI can be found in the Cogito Project [MarketingAI](https://github.com/CogitoNTNU/MarketingAI/blob/main/src/fine_tuning/fine_tuning_job.py)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ht-CtKNT7w8v"
      },
      "source": [
        "### Leveraging OpenAI Across Diverse Programming Environments\n",
        "While this course primarily focuses on Python to interact with OpenAI's models. Thera are other supported languages. Supported languages include, but are not limited to, TypeScript/JavaScript, Java, C#, Go, C++, and PHP, alongside others like Clojure, Kotlin, Ruby, Rust, and Scala. This wide-ranging support extends the potential of OpenAI's AI models to virtually any software development domain, from web development and mobile applications to enterprise solutions and beyond.\n",
        "\n",
        "[Read more at OpenAI Docs](https://platform.openai.com/docs/libraries/community-libraries)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.12.3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}