# thingy_grabber
-Script for archiving thingiverse things. Due to this being a glorified webscraper, it's going to be very fragile.
+Script for archiving thingiverse things.
## Usage:
````
-usage: thingy_grabber.py [-h] [-l {debug,info,warning}] [-d DIRECTORY] [-f LOG_FILE] [-q] {collection,thing,user,batch,version} ...
+usage: thingy_grabber.py [-h] [-l {debug,info,warning}] [-d DIRECTORY] [-f LOG_FILE] [-q] [-c] [-a API_KEY]
+ {collection,thing,user,batch,version} ...
positional arguments:
{collection,thing,user,batch,version}
-f LOG_FILE, --log-file LOG_FILE
Place to log debug information to
-q, --quick Assume date ordering on posts
+ -c, --compress Compress files
+ -a API_KEY, --api-key API_KEY
+ API key for thingiverse
````
+## Warning
+Currently downloading individual things by ID is broken. This should be fixed soon.
+
+## API KEYs
+Thingy_grabber v0.10.0 accesses thingiverse in a _substantially_ different way to before. The plus side is it should be more reliable, possibly faster and no longer needs selenium or a firefox instance (and so drastically reduces memory overhead). The downside is you are _going_ to have to do something to continue using the app - basically get yourself an API KEY.
+
+To do this, go to https://www.thingiverse.com/apps/create and create your own selecting Desktop app.
+Once you have your key, either specify it on the command line or put it in a text file called `api.key` whereever you are running the script from - the script will auto load it.
+
+### Why can't I use yours?
+Because API keys can (are?) rate limited.
+
+## Downloads
+The latest version can be downloaded from here: https://github.com/cwoac/thingy_grabber/releases/. Under the 'assets' triangle there is precompiled binaries for windows (no python needed!).
+
+## Getting started
+First download the code. Either grab the source, or get the windows binary from above and extract it somewhere. If you are running from source, see `requirements.yaml` for the packages you need. You will also need an API key (as above) and to make a directory to store your downloads in.
+
+oh, and you need to know what you want to download, ofc. It can be either things, collections or just the designs of a user.
+once you have done all this you need to open a command prompt and run it.
+
+Let's say you are running windows and using the precompiled binary and extracted the release to the `thingy_grabber` directory on your desktop and you made a `things` directory in your `Documents` directory.
+When you open the command window, it will start in your home directory (say `c:\Users\cwoac`)
+`cd Desktop\thingy_grabber` to get to `c:\Users\cwoac\Desktop\thingy_grabber` and check that you are right by trying to run `thingy_grabber` - you should get a long list of possible command line options that looks a lot like the list further up.
+Supposing you want to download all of my stuff (for some crazy reason), then the command will look like this
+
+`thingy_grabber -a YOURAPIKEY -d "c:\Users\cwoac\Documents\things" -c user cwoac`
+
+The `-c` will cause the script to compress the download to a 7z file to save space. If you prefer to leave it uncompressed, just omit the `-c`
+That's the basics. Well, acutally, there isn't much more than that to be honest. There is a batch mode so if you create a text file with a list of lines like
+```
+user cwoac
+user solutionlesn
+collection cwoac at2018
+```
+then you can use the `batch` target to run each of these in turn. If you run it a second time with the same options it will only download things which have changed or been added.
+
+## Modes
### Things
`thingy_grabber.py thing thingid1 thingid2 ...`
This will create a directory named after the title of the thing(s) with the given ID(s) and download the files into it.
`
## Requirements
-python3, beautifulsoup4, requests, lxml
+python3, requests, py7xr (>=0.8.2)
## Current features:
- can download an entire collection, creating seperate subdirs for each thing in the collection
- If there is an updated file, the old directory will be moved to `name_timestamp` where `timestamp` is the last upload time of the old files. The code will then copy unchanged files across and download any new ones.
## Changelog
+* v0.10.2
+ - Fixed regression in rest API
+* v0.10.1
+ - A couple of minor bug fixes on exception handling.
+* v0.10.0
+ - API access! new -a option to provide an API key for more stable access.
+* v0.9.0
+ - Compression! New -c option will use 7z to create an archival copy of the file once downloaded.
+ Note that although it will use the presence of 7z files to determine if a file has been updated, it currently _won't_ read old files from inside the 7z for handling updates, resulting in marginally larger bandwidth usage when dealing with partially updated things. This will be fixed later.
+ - Internal tidying of how old directories are handled - I've tested this fairly heavily, but do let me know if there are issues.
+* v0.8.7
+ - Always, Always generate a valid time stamp.
+* v0.8.6
+ - Handle thingiverse returning no files for a thing gracefully.
+* v0.8.5
+ - Strip '.'s from the end of filenames
+ - If you fail a download for an already failed download it no longer throws an exception
+ - Truncates paths that are too long for windows
* v0.8.4
- Just use unicode filenames - puts the unicode characters back in!
- Force selenium to shutdown firefox on assert and normal exit
HCoop / clinton / thingy_grabber.git / blobdiff