NOTE: You must be in venv if you installed using install_requirements.bat! You can enter venv by entering "venv/Scripts/Activate" including the quotations in the CMD while in the project directory. You are in venv if and only if the CMD prompt displays (venv).
EXTREMELY IMPORTANT: YOU MUST VISIT KEMONO'S WEBSITE AND SOLVE THE CAPTCHA PUZZLE BEFORE RUNNING THE PROGRAM. THERE IS NO OTHER WAY TO GET AROUND DDOS PROTECTION OTHERWISE!!!
Command line arguments:
DOWNLOAD CONFIG - How files are downloaded
-f --bulkfile <textfile.txt> : Bulk download from text file containing links -d --downloadpath <path> : REQUIRED if and only if updater switch is not used - Set download path for single instance, must use '\' or '/' -c --chunksz <#> : Adjust download chunk size in bytes (Default is 64M) -t --threadct <#> : Change download thread count (default is 1, max is 5) -w --wait <#> : Delay between downloads in seconds (default is 0.25s) -b --track : Track artists which can updated later, not supported for discord.
- How to set up bulk file for -f switch
- Download file path is bugged, if -d is not the last switch, cannot end with '\'
- Files are downloaded in chunks at a time, higher chunk size speeds up downloads but uses more memory
- Thread count boosts download speed significantly at the risk of rate limiting.
- -w switch may help with rate limiting
- -b database is updated if and only if the program terminates successfully.
EXCLUSION - Exclusion of specific downloads
-x --excludefile "txt, zip, ..., png" : Exclude files with listed extensions, NO '.'s -p --excludepost "keyword1, keyword2,..." : Keyword in excluded posts, not case sensitive -l --excludelink "keyword1, keyword2,..." : Keyword in excluded link, not case sensitive. Is for link plaintext, not its target -o --omitcomment : Do not download any post comments -m --omitcontent : Do not download any textual post contents -n --minsize : Minimum file size in bytes
- There might be some links in post content which causes the program to halt, use --omitcontent if the program freezes
- --minsize is helpful for skipping the cropped header that some posts have.
DOWNLOAD FILE STRUCTURE - How to organize downloads
-s --partialunpack : If a artist post is text only, do not create a dedicated directory for it, partially unpacks files -u --unpacked : Enable unpacked file organization, all works will not have their own folder, overrides partial unpack -e --hashname : Download server name instead of program defined naming scheme, may lead to issues if Kemono does not store links correctly. Not supported for Discord -v --unzip : Enables unzipping of files automatically, requires 7z and setup to be done correctly
- Default download structure is that each work will be placed in their own dedicated directory, even if there is nothing within the work.
- -e switch isn't recommended due to invalid file names
- For unzipping, read 3 and 4
UTILITIES - Things that can be done besides downloading
--UPDATE : Update all tracked artist works
- --UPDATE is an Experimental feature, needs more testing
TROUBLESHOOTING - Solutions to possible issues
-z --httpcode "500, 502,..." : HTTP codes to retry downloads on, default is 429 and 403 -r --maxretries <#> : Maximum number of HTTP code retries, default is infinite -h --help : Help --EXPERIMENTAL : Enable experimental mode --BENCHMARK : Benchmark experiemental mode's scraping speed, does not download anything
- --httpcode is very helpful as Kemono uses a variety of http codes to rate limit its users. View KMPDownloader's logs when files failed to download to check if a http code is being used to rate limit.
- --maxretries is the number of retries used to retry downloads for http codes described by --httpcode, if a code is caught which isn't registered, the download will automatically be registered as a failure and written to log.
- --EXPERIMENTAL mode downloads by fetching all urls first then downloading while the default downloading mode processes artist works one by one. Experimental mode is recommended for smaller download sizes as it has a helpful progress bar as opposed to the default mode. However, default mode may be faster (and more unstable) as there are up to 2x active number of threads at once compared to experimental mode.
Bulk File Download
An example bulk file ("examples.txt") has been included.
A bulk file is a text file where each line is either a url in one of the supported url formats or a new line. The contents of the example file is included below:
Supported URL formats:
Downloads all artist works:
Downloads all artist works on that specific page:
Downloads specific artist work
Logging
TODO
Updater
KMPDownloader has the ability to track artists and update the tracked artists' archives.
Tracking and Viewing
To track artists, you will need to use the -b or --track switch when downloading normally. For example:
python KMPDownloader.py –d "." –f -b "./example.txt"The tracked artist entry will be generated and placed in KMP.db within directory you are calling KMPDownloader from. Usually, this is the root directory of KMPDownloader.
In order to view and edit this database, download DB Browser or any equivalent database viewer. When you have download the browser, you will need to select an app to open KMP.db with:
Click "Choose an app on your PC". This will open up a file explorer which will point to your default download directory for programs. If you have not changed the download path mentioned during the DB browser installation, this directory will contain the DB browser files. Navigate to the "DB Browser for SQLite" folder and click on "DB Browser for SQLite.exe". Afterwards, click "always" in the "select an app" window so you won't have to repeat these steps.
Afterwards, when you double click on KMP.db, this window should open:
In order to browse data within KMP.db, you can click on the "Browse Data" tab on the top of the window:
Column definitions:
url: Artist url the downloaded work is associated with.
artist: Artist name on Kemono.
type: Website downloaded from.
latest: Newest work downloaded from the artist.
destination: Where the artist work is located.
config: Download configuration used to download the artist work.
Editing Data
When editing data, the most important columns are "latest" and "destination". Modifications are made by clicking on the table entry you would like to change then editing the database cell content on the right of the window then clicking "apply" below the edit cell. Upon making modifications to latest or destination, click "Write Changes" on top of the window:
NOTE: changes made to the database are IRREVERSIBLE once "Write Changes" has been clicked. Any invalid changes made can break the updater! Make sure to backup "KMP.db" before making any changes.
Updating
When updating, use the --UPDATE switch instead of the -d or --downloadpath switch. Also and the -f or --bulkfile switch if it exists and --track or -b if it exists. You can use any other switches if you like to change download behavior. For example:
python KMPDownloader.py --threadct 3 --unzip --unpacked --EXPERIMENTAL -z "429, 403, 502" --UPDATE
When updating, all new artist works are downloaded to the "destination" field of the artist's url in KMP.db. The newest, most recent downloaded url is updated in KMP.db.
Example Usage
Minimum switches:
python KMPDownloader.py -d "."
Downloads to the current directory. User is told to input a valid Kemono.party url.
Highly Customized Run:
python KMPDownloader.py --threadct 3 --track --unzip --unpacked --EXPERIMENTAL -z "429, 403, 502" -d "E:\User Files\gallery"
Use 3 threads, add downloaded archives to updater database, unzip files automatically, download files in unpacked file structure, download using fetch-all then download approach, retry on http codes 429, 403, 502, download to "E:\User Files\gallery".
Bulk download
python KMPDownloader.py –d "." –f "./example.txt"
Bulk download url from example.txt to current directory
--RENAME switch
Rename switch has several purposes, it will rename files according to the switches used (it cannot be used to switch between unpacked and packed file modes) AND remove duplicate file. Here is an example of the switch in use:
We start with this directory, notice that there is quite a large number of duplicate files denoted by (0) towards the end of the file name. The files also has the id and date of the work. However, what if I wanted to keep the file naming scheme but remove the dupes? I will use the same switches I used when downloading the original directory but include the --RENAME switch.
Now, what happens if I want to change the naming scheme? In this case, I want to remove only the date. This is accomplished by using the --date switch in tandem with the --RENAME switch.
We start with this directory, notice that there is quite a large number of duplicate files denoted by (0) towards the end of the file name. The files also has the id and date of the work. However, what if I wanted to keep the file naming scheme but remove the dupes? I will use the same switches I used when downloading the original directory but include the --RENAME switch.
Now we have a directory with the naming scheme we want but free of any duplicates.
As you can see, the files only have ids now and no longer has the date.Limitations and notes
This switch is compatible with only the --id, --date, and --REUPDATE switches for customization.
- --REUPDATE is useful for mass renaming files but will be need the -x "zip, rar, 7z" switch as the program cannot detect if a folder belongs to a zip file.
- Extracted zip files cannot be renamed, moved, or deleted with the --RENAME switch. It will also be re downloaded if zip file itself (not the extracted files) does not exist.
- When using packed file mode (the mode that puts each work in its own folder) and a switch that changes naming scheme, folders will be generated first that fit the naming scheme. Each file will be moved from the folders with the old scheme to the folder with the new scheme. Afterwards, if the old folder is empty (check above bullet), the old folder will be deleted.
- If --RENAME switch is not used, if the program founds a duplicate file on the system, it will skip the download. It will never rename or delete files if the switch is not used.
- Some files cannot be renamed because a newer import may change a work name for a work that was already downloaded. The program does not use hashes but relies solely on file names and file sizes to determine if a file is a duplicate or not.
- Note that new filenames with date info will have "Published" before the date when the older filenames did not. This is a change to Kemono's front end. The program will apply --RENAME switch functionality to transform any existing files without the "Published" keyword to files with the "Published" keyword.
