This is a web crawler implementation for the first homework for a course at FRI, UL.
The following packages are required:
- os
- threading
- urllib.request
- selenium
- bs4
- multiprocessing
- urllib.parse
- time
- requests
- hashlib
- psycopg2
- datetime
- re
- posixpath
- urllib.robotparser
You need to install the chromedriver for selenium.
Run the Crawler.py in directory crawler.