To run the script, just $ python crawler.py To run the tests: $ python test.py Dependencies: argparse==1.2.1 hurry.filesize==0.9 lxml==3.6.0 readline==6.2.4.1 requests==2.10.0 wsgiref==0.1.2