카페크롤링_네이버.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: selenium in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (4.9.1)\n",
      "Requirement already satisfied: urllib3[socks]<3,>=1.26 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from selenium) (2.0.2)\n",
      "Requirement already satisfied: trio-websocket~=0.9 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from selenium) (0.10.2)\n",
      "Requirement already satisfied: trio~=0.17 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from selenium) (0.22.0)\n",
      "Requirement already satisfied: certifi>=2021.10.8 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from selenium) (2023.5.7)\n",
      "Requirement already satisfied: outcome in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from trio~=0.17->selenium) (1.2.0)\n",
      "Requirement already satisfied: async-generator>=1.9 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from trio~=0.17->selenium) (1.10)\n",
      "Requirement already satisfied: exceptiongroup>=1.0.0rc9 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from trio~=0.17->selenium) (1.1.1)\n",
      "Requirement already satisfied: sniffio in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from trio~=0.17->selenium) (1.3.0)\n",
      "Requirement already satisfied: attrs>=19.2.0 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from trio~=0.17->selenium) (23.1.0)\n",
      "Requirement already satisfied: sortedcontainers in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from trio~=0.17->selenium) (2.4.0)\n",
      "Requirement already satisfied: idna in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from trio~=0.17->selenium) (3.4)\n",
      "Requirement already satisfied: wsproto>=0.14 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from trio-websocket~=0.9->selenium) (1.2.0)\n",
      "Requirement already satisfied: pysocks!=1.5.7,<2.0,>=1.5.6 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from urllib3[socks]<3,>=1.26->selenium) (1.7.1)\n",
      "Requirement already satisfied: h11<1,>=0.9.0 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from wsproto>=0.14->trio-websocket~=0.9->selenium) (0.14.0)\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip available: \u001b[0m\u001b[31;49m22.2.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.2.1\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip3 install --upgrade pip\u001b[0m\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "pip install selenium"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: requests in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (2.31.0)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from requests) (2.0.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from requests) (3.4)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from requests) (3.2.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (from requests) (2023.5.7)\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip available: \u001b[0m\u001b[31;49m22.2.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m23.2.1\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip3 install --upgrade pip\u001b[0m\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "pip install requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# selenium의 webdriver를 사용하기 위한 import\n",
    "from selenium import webdriver\n",
    "\n",
    "# selenium으로 키를 조작하기 위한 import\n",
    "from selenium.webdriver.common.keys import Keys\n",
    "from selenium.webdriver.common.by import By\n",
    "\n",
    "# 페이지 로딩을 기다리는데에 사용할 time 모듈 import\n",
    "import time\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "from bs4 import BeautifulSoup\n",
    "import requests\n",
    "\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 크롤링\n",
    "\n",
    "네이버 지도의 경우, 크롤링 자체를 막아뒀음. 출력이 아예 안 됨.\n",
    "그러나 아래 전체로 보는 페이지(카톡으로 보낸 링크)에서는 리뷰 크롤링이 가능함.\n",
    "\n",
    "따라서 우회(?)하는 방식을 사용함.\n",
    "\n",
    "링크 중간에 '1098819515' 이렇게 숫자가 나오는데, 이게 카페/음식점의 고유번호임\n",
    "이 고유번호만 바꿔주면 크롤링 가능한 페이지로 들어갈 수 있음.\n",
    "\n",
    "문제는 강남역 주변의 카페의 고유번호를 크롤링해야 하는데, 네이버지도에서는 못함.\n",
    "그래서 네이버에 '강남역 카페'를 검색해서 나오는 몇 개의 카페는 고유번호를 가져올 수 있음.\n",
    "\n",
    "이걸 가져와서 하나씩 링크에 넣고 리뷰를 크롤링하는 방법을 사용함."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 네이버 지도 말고 네이버에서"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/y3/_25c1lg15r71b9q06fr4j0dr0000gn/T/ipykernel_42116/2079816369.py:6: DeprecationWarning: executable_path has been deprecated, please pass in a Service object\n",
      "  driver = webdriver.Chrome('/Users/imsoyeong/Downloads/chromedriver_mac64/chromedriver.exe')\n"
     ]
    }
   ],
   "source": [
    "#크롬 드라이버에 url 주소 넣고 실행\n",
    "url = 'https://search.naver.com/search.naver?where=nexearch&sm=top_hty&fbm=0&ie=utf8&query=%EA%B0%95%EB%82%A8%EC%97%AD+%EC%B9%B4%ED%8E%98'\n",
    "# 강남역 카페\n",
    "\n",
    "# 크롬 브라우저 시작\n",
    "driver = webdriver.Chrome('/Users/imsoyeong/Downloads/chromedriver_mac64/chromedriver.exe') \n",
    "driver.get(url)\n",
    "\n",
    "# 페이지가 완전히 로딩되도록 1초동안 기다림\n",
    "time.sleep(1)\n",
    "\n",
    "webpage = requests.get(url)\n",
    "soup = BeautifulSoup(webpage.content, \"html.parser\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 총 5페이지 하나씩 넘기기\n",
    "cafe_num = []\n",
    "cnt = 1\n",
    "while True:\n",
    "    # 정규표현식 패턴을 사용하여 'place/'와 '?' 사이의 값을 추출\n",
    "    pattern = r'place/([^?]+)'\n",
    "    \n",
    "    # 스크롤 로딩을 위함\n",
    "    while True:\n",
    "        bh = driver.execute_script('return document.body.scrollHeight')\n",
    "        ah = driver.execute_script('return document.body.scrollHeight')\n",
    "        if ah == bh:\n",
    "            break\n",
    "        bh = ah\n",
    "        \n",
    "    time.sleep(1)\n",
    "    # 추출\n",
    "    infos = soup.select('.CHC5F > .tzwk0')\n",
    "    for info in infos:\n",
    "        info = str(info.select)\n",
    "        try:\n",
    "            match = re.search(pattern, info)\n",
    "            extracted_value = match.group(1)\n",
    "            cafe_num.append(int(extracted_value))\n",
    "            \n",
    "        except:\n",
    "            pass\n",
    "    \n",
    "    driver.find_element('xpath', '//*[@id=\"place-main-section-root\"]/section/div/div[5]/a[2]').click()\n",
    "    cnt +=1\n",
    "    \n",
    "    if cnt == 5:\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 리스트에서 중복된 값 제거\n",
    "list(set(cafe_num))\n",
    "cafe_num.extend([1483295656, 1186606414, 1754767885, 1081873068, 1478381424, 21082407, 1490841787])\n",
    "cafe_num = list(set(cafe_num))\n",
    "cafe_num"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 고유번호 하나씩 넣어서 리뷰 크롤링\n",
    "url = https://pcmap.place.naver.com/restaurant/[34015652]/review/visitor?shouldKeywordUnfold=true"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "metadata": {},
   "outputs": [],
   "source": [
    "cafe_name = []\n",
    "comments = []\n",
    "for num in cafe_num:\n",
    "    url = 'https://pcmap.place.naver.com/restaurant/{}/review/visitor?shouldKeywordUnfold=true'.format(num)\n",
    "    #크롬 드라이버에 url 주소 넣고 실행\n",
    "    driver.get(url)\n",
    "    time.sleep(2)\n",
    "    \n",
    "    # 카페이름\n",
    "    name = driver.find_element(By.CSS_SELECTOR, '.Fc1rA').text\n",
    "    \n",
    "    # 스크롤 로딩을 위함\n",
    "    while True:\n",
    "        bh = driver.execute_script('return document.body.scrollHeight')\n",
    "        time.sleep(2)\n",
    "        ah = driver.execute_script('return document.body.scrollHeight')\n",
    "        if ah == bh:\n",
    "            break\n",
    "        bh = ah\n",
    "    \n",
    "    while True:\n",
    "        try:\n",
    "            # 리뷰 크롤링\n",
    "            reviews = driver.find_elements(By.CSS_SELECTOR, '.zPfVt')\n",
    "            for re in reviews:\n",
    "                cafe_name.append(name)\n",
    "                comments.append(re.text)\n",
    "                \n",
    "            # 더보기 버튼 누르기\n",
    "            driver.find_element('xpath', '//*[@id=\"app-root\"]/div/div/div/div[7]/div[3]/div[3]/div[2]/a').click()\n",
    "            time.sleep(1.5)\n",
    "        except:\n",
    "            break\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cafe_name</th>\n",
       "      <th>comments</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>더달달</td>\n",
       "      <td>초코디저트에 진심인 강남역카페 ‘더달달’ 🍫더달달 초코 디저트 먹으면 다른 매장은 ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>더달달</td>\n",
       "      <td>강남역카페 더달달 달달핑크빙수, 쇼콜라 등 초콜릿 디저트 퀄리티가 아주 좋았어요. ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>더달달</td>\n",
       "      <td>강남에서 조용하고 넓은 카페를 찾는다면 추천할 곳이에요! 1층 뿐만 아니라 2층도 ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>더달달</td>\n",
       "      <td>딸기 우유 빙수 쫀맛탱이에요 딸기가 밑에까지 가득 들어있어서 먹어도 먹어도 딸기가 ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>더달달</td>\n",
       "      <td>더달달 진짜 최고,,❤️ 엔틱한 분위기에 내부 인테리어도 너무 예쁘고 진짜 찐초코란...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3249</th>\n",
       "      <td>더달달</td>\n",
       "      <td>빙수랑 와플 너무 맛있어요♡</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3250</th>\n",
       "      <td>더달달</td>\n",
       "      <td>여자분들이 많은 곳은 찐 디저트맛집이라던데 여기네요..자리에 사람들 다 여자분이시거...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3251</th>\n",
       "      <td>더달달</td>\n",
       "      <td>종류도 다양하고 저렴해서 좋아요😊</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3252</th>\n",
       "      <td>더달달</td>\n",
       "      <td>양이 많아요</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3253</th>\n",
       "      <td>더달달</td>\n",
       "      <td>에그타르트 맛집을 찾다가 발견한 곳이에요 ^^</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3254 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     cafe_name                                           comments\n",
       "0          더달달  초코디저트에 진심인 강남역카페 ‘더달달’ 🍫더달달 초코 디저트 먹으면 다른 매장은 ...\n",
       "1          더달달  강남역카페 더달달 달달핑크빙수, 쇼콜라 등 초콜릿 디저트 퀄리티가 아주 좋았어요. ...\n",
       "2          더달달  강남에서 조용하고 넓은 카페를 찾는다면 추천할 곳이에요! 1층 뿐만 아니라 2층도 ...\n",
       "3          더달달  딸기 우유 빙수 쫀맛탱이에요 딸기가 밑에까지 가득 들어있어서 먹어도 먹어도 딸기가 ...\n",
       "4          더달달  더달달 진짜 최고,,❤️ 엔틱한 분위기에 내부 인테리어도 너무 예쁘고 진짜 찐초코란...\n",
       "...        ...                                                ...\n",
       "3249       더달달                                    빙수랑 와플 너무 맛있어요♡\n",
       "3250       더달달  여자분들이 많은 곳은 찐 디저트맛집이라던데 여기네요..자리에 사람들 다 여자분이시거...\n",
       "3251       더달달                                 종류도 다양하고 저렴해서 좋아요😊\n",
       "3252       더달달                                             양이 많아요\n",
       "3253       더달달                          에그타르트 맛집을 찾다가 발견한 곳이에요 ^^\n",
       "\n",
       "[3254 rows x 2 columns]"
      ]
     },
     "execution_count": 222,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame({'cafe_name':cafe_name,\n",
    "                   'comments':comments})\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}