Praw-Reddit.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "94b76cf6-24c5-418b-81de-be6d3ea56558",
   "metadata": {},
   "source": [
    "### Jupyter Notebook for Extracting Subreddit Data with PRAW and Saving Outputs\n",
    "\n",
    "**Introduction**\n",
    "This notebook extracts data from a subreddit using PRAW (Python Reddit API Wrapper), \n",
    "processes it into a pandas DataFrame, and saves it to JSON, CSV, and SQLite formats.\n",
    "\n",
    "# Importing Required Libraries\n",
    "- `praw`: Python Reddit API Wrapper to interact with Reddit.\n",
    "- `pandas`: Powerful data manipulation and analysis library, ideal for tabular data.\n",
    "- `sqlite3`: Python's built-in SQLite database library to store data in relational format.\n",
    "- `tqdm`: Progress bar library to track iterations in loops."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76aea837-6c46-4d10-bd6e-a53632e29539",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install praw tqdm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b72988f-ebe6-4d5b-ae5a-366d49e825fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from getpass import getpass\n",
    "import praw\n",
    "import pandas as pd\n",
    "import sqlite3\n",
    "from tqdm import tqdm\n",
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ed4da6b-48a1-41a9-be11-7912759c79fb",
   "metadata": {},
   "source": [
    "### Prompting User for Reddit Credentials\n",
    "In Jupyter notebooks, we avoid hardcoding sensitive credentials. Instead, we use the following approach:\n",
    "- First, check if the credentials are defined as global variables.\n",
    "- If not, check environment variables.\n",
    "- Finally, prompt the user to input them securely using `getpass`.\n",
    "\n",
    "If you need to change or reset the credentials below after running the cell, then please restart your kernel from the menu above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85ded713-c88f-42da-9449-2f71a9d16dc4",
   "metadata": {},
   "outputs": [],
   "source": [
    "client_id = globals().get(\"client_id\") or os.getenv(\"REDDIT_CLIENT_ID\") or getpass(\"Enter your Reddit client ID: \")\n",
    "client_secret = globals().get(\"client_secret\") or os.getenv(\"REDDIT_CLIENT_SECRET\") or getpass(\"Enter your Reddit client secret: \")\n",
    "username = globals().get(\"username\") or os.getenv(\"REDDIT_USERNAME\") or getpass(\"Enter your Reddit username: \")\n",
    "\n",
    "\n",
    "# Initialize PRAW\n",
    "reddit = praw.Reddit(\n",
    "    client_id=client_id,\n",
    "    client_secret=client_secret,\n",
    "    user_agent=f\"script:praw:{praw.__version__} (by u/{username})\"\n",
    ")\n",
    "\n",
    "# This cell will prompt for Reddit API credentials only once.\n",
    "# If you need to change or reset the credentials, then please restart your kernel from the menu above."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b740951-6650-400a-8380-54a123b0a878",
   "metadata": {},
   "source": [
    "## Extracting Data from a Subreddit\n",
    "The function `extract_subreddit_data` fetches the latest posts from a subreddit using the Reddit API.\n",
    "Using `tqdm`, we track the progress of fetching submissions to enhance user experience."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15991792-39ca-4cc8-982b-8cf7618bbb70",
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_subreddit_data(subreddit_name, limit=100):\n",
    "    subreddit = reddit.subreddit(subreddit_name)\n",
    "    data = []\n",
    "    try:\n",
    "        for submission in tqdm(subreddit.new(limit=limit), desc=f\"Fetching posts from r/{subreddit_name}\"):\n",
    "            submission.comments.replace_more(limit=0)\n",
    "            comments = [comment.body for comment in submission.comments.list()]\n",
    "            data.append({\n",
    "                'post_id': submission.id,\n",
    "                'title': submission.title,\n",
    "                'selftext': submission.selftext,\n",
    "                'author': str(submission.author),\n",
    "                'created_utc': submission.created_utc,\n",
    "                'comments': comments\n",
    "            })\n",
    "    except Exception as e:\n",
    "        print(f\"An error occurred while extracting data from r/{subreddit_name}: {e}\")\n",
    "    return pd.DataFrame(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6422f42-6c75-4583-aa22-66e790f1af00",
   "metadata": {},
   "source": [
    "### Saving the Data\n",
    "Using pandas, we export the data to multiple formats:\n",
    "- JSON: A standard, human-readable format for data sharing and APIs.\n",
    "- JSONL: JSON Lines, a machine-readable format where each record is a line, suitable for processing with tools like `grep` or loading into databases like BigQuery.\n",
    "- CSV: A widely-used format for spreadsheets and data exchange.\n",
    "- SQLite: A relational database format, useful for structured queries and relationships (e.g., posts and comments).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67059f32-9469-4626-99d7-daa84d6c6fef",
   "metadata": {},
   "outputs": [],
   "source": [
    "def save_data(df, base_filename):\n",
    "    try:\n",
    "        # Save to JSON\n",
    "        json_filename = f\"{base_filename}.json\"\n",
    "        df.to_json(json_filename, orient='records', indent=4)\n",
    "        print(f\"Data saved to {json_filename}\")\n",
    "\n",
    "        # Save to JSONL\n",
    "        jsonl_filename = f\"{base_filename}.jsonl\"\n",
    "        df.to_json(jsonl_filename, orient='records', lines=True)\n",
    "        print(f\"Data saved to {jsonl_filename}\")\n",
    "\n",
    "        # Save to CSV\n",
    "        csv_filename = f\"{base_filename}.csv\"\n",
    "        df.to_csv(csv_filename, index=False)\n",
    "        print(f\"Data saved to {csv_filename}\")\n",
    "\n",
    "        # Save to SQLite\n",
    "        db_filename = f\"{base_filename}.db\"\n",
    "        conn = sqlite3.connect(db_filename)\n",
    "        df[['post_id', 'title', 'selftext', 'author', 'created_utc']].to_sql('posts', conn, if_exists='replace', index=False)\n",
    "        comments_data = df.explode('comments')[['post_id', 'comments']].rename(columns={'comments': 'comment'})\n",
    "        comments_data.to_sql('comments', conn, if_exists='replace', index=False)\n",
    "        conn.close()\n",
    "        print(f\"Data saved to {db_filename}\")\n",
    "    except Exception as e:\n",
    "        print(f\"An error occurred while saving data: {e}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d06e9e3-3d84-43ea-8a74-c96813d5276d",
   "metadata": {},
   "source": [
    "### Specifying Subreddit and Running Extraction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "76b8959f-fa2f-4746-b12b-8eb751ebce4f",
   "metadata": {},
   "outputs": [],
   "source": [
    "subreddit_name = input(\"Enter the subreddit name: \")\n",
    "base_filename = f\"subreddit_{subreddit_name.lower()}_{datetime.now().strftime('%Y-%m-%d')}\"\n",
    "\n",
    "# Fetch data and save outputs\n",
    "data = extract_subreddit_data(subreddit_name, limit=100)\n",
    "save_data(data, base_filename)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d76ea49-c964-40b4-84e3-6124cdd8188f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}