-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Assorted improvements #6
base: master
Are you sure you want to change the base?
Conversation
…een requests to prevent being blocked 1. adds the -R flag 2. Should fix scoliono#1 and adds the -nt flag
Because we no longer overwrite files without the user asking explicitly for it (-R flag) The prompt also got in the way of running the script non-interactively
It's more informative to log the actual filename, which includes the page number. I also feel that gauging the progress is easy enough with (N/N) to make a percentage indicator unnecessary. Changed from: 12% (1/8) done 25% (2/8) done 37% (3/8) done 50% (4/8) done 62% (5/8) done 75% (6/8) done 87% (7/8) done 100% (8/8) done To: Got ./OL370939M/100.jpg (1/8) Got ./OL370939M/101.jpg (2/8) Got ./OL370939M/102.jpg (3/8) Got ./OL370939M/103.jpg (4/8) Got ./OL370939M/104.jpg (5/8) Got ./OL370939M/105.jpg (6/8) Got ./OL370939M/106.jpg (7/8) Got ./OL370939M/107.jpg (8/8) (The command used to generate these logs was: `python3 ripper.py OL370939M -s 100 -e 107 -S 10`)
@scoliono merge? |
This does not appear to totally circumvent Archive.org's rate limiting, from my testing. Around 100 pages or so, you start downloading 5 KB HTML documents instead of images.
|
If you're too persistent with the requests, it looks like you can also get this traceback:
|
@scoliono here are some changes for review. 8569763 should fix #1.
Also your email visible in
git log
is very nice.