163 lines
7.4 KiB
Markdown
163 lines
7.4 KiB
Markdown
# got
|
||
|
||
A small, Rob-Pike-style command-line download tool in Go: one program that
|
||
downloads HTTP(S) URLs over several connections and BitTorrent
|
||
magnets/torrents, with a focused set of options.
|
||
|
||
The design leans on Go's runtime instead of a hand-rolled reactor. There is no
|
||
single-threaded event-poll loop: one goroutine per download blocks on real I/O,
|
||
a buffered channel of slots bounds concurrency, and `context` carries shutdown.
|
||
What remains is the data — and, per Pike, the data is the design.
|
||
|
||
## Build
|
||
|
||
```sh
|
||
go build -o got .
|
||
```
|
||
|
||
Requires Go 1.25+. The BitTorrent engine is
|
||
[`anacrolix/torrent`](https://github.com/anacrolix/torrent); everything else is
|
||
the standard library plus `golang.org/x/{term,time}`.
|
||
|
||
## Use
|
||
|
||
```sh
|
||
# segmented HTTP download, 16 connections, into ./downloads
|
||
got -x16 -s16 -d downloads https://example.com/big.iso
|
||
|
||
# resume an interrupted download
|
||
got -c -x16 https://example.com/big.iso
|
||
|
||
# the same file from several mirrors at once (one file, connections spread
|
||
# across servers, a dead mirror falls over); -Z downloads them separately instead
|
||
got -x4 https://a.example/big.iso https://b.example/big.iso
|
||
|
||
# a torrent or magnet; seed for 30 minutes after finishing
|
||
got --seed-time=30 ubuntu.torrent
|
||
got 'magnet:?xt=urn:btih:...'
|
||
|
||
# list the files in a torrent without downloading
|
||
got -S -T some.torrent
|
||
|
||
# several downloads at once, two running in parallel
|
||
got -j2 -i urls.txt
|
||
```
|
||
|
||
Run `got --help` (or `--help=http`, `--help=bittorrent`, ...) for the full
|
||
option list.
|
||
|
||
### Options
|
||
|
||
A focused subset of options, with familiar short flags and sensible defaults.
|
||
|
||
| flag | meaning | default |
|
||
|------|---------|---------|
|
||
| `-d, --dir` | output directory | `.` |
|
||
| `-o, --out` | output filename (single download) | from URL |
|
||
| `-i, --input-file` | read downloads line by line (TAB-separated URLs = mirrors; `-` = stdin) | |
|
||
| `-c, --continue` | resume a partial download | false |
|
||
| `-x, --max-connection-per-server` | connections to one server (1–16) | 1 |
|
||
| `-s, --split` | split a download into N connections | 5 |
|
||
| `-Z, --force-sequential` | download each command-line URL separately, not as mirrors | false |
|
||
| `-k, --min-split-size` | do not split below this size | 20M |
|
||
| `-j, --max-concurrent-downloads` | parallel downloads | 5 |
|
||
| `--max-overall-download-limit` | global speed cap | 0 (off) |
|
||
| `--max-download-limit` | per-download speed cap | 0 (off) |
|
||
| `--retry-wait` | seconds to wait between retries | 0 |
|
||
| `--load-cookies` / `--save-cookies` | read/write a Netscape `cookies.txt` | |
|
||
| `--conditional-get` | skip the download if the local file is up to date | false |
|
||
| `--remote-time` | set the file's mtime from the server | false |
|
||
| `--checksum` | verify the finished file: `TYPE=DIGEST` (sha-256, sha-1, md5, …) | |
|
||
| `--dry-run` | check the file is available but do not download it | false |
|
||
| `--ca-certificate` | verify HTTPS against the CA certificates in FILE (PEM) | |
|
||
| `--stop` | stop the program after N seconds | 0 (off) |
|
||
| `--disable-ipv6` | force IPv4-only connections | false |
|
||
| `-T, --torrent-file` | a `.torrent` file | |
|
||
| `--seed-time` | minutes to seed after finishing | seed by ratio |
|
||
| `--seed-ratio` | stop seeding at this ratio | 1.0 |
|
||
| `--listen-port` | port (range) for incoming peers | 6881-6999 |
|
||
| `--enable-dht` | use the BitTorrent DHT | true |
|
||
| `--select-file` | fetch only these file indexes (`1,3-5`) | all |
|
||
| `-S, --show-files` | list torrent files and exit | |
|
||
| `--bt-stop-timeout` | give up if no download progress for N s | 0 (off) |
|
||
| `--bt-metadata-timeout` | give up if a magnet can't fetch metadata in N s | 60 |
|
||
| `--save-session` | on exit, write unfinished downloads to FILE | |
|
||
| `-q, --quiet` | no progress readout | false |
|
||
|
||
### Resuming across runs
|
||
|
||
There is no daemon — resume is a plain file. Point
|
||
`--save-session` and `-i` at the same file and add `-c`: unfinished downloads
|
||
are loaded at start and the still-unfinished ones are written back on exit.
|
||
|
||
```sh
|
||
got -c --save-session=got.session -i got.session <new urls/magnets...>
|
||
```
|
||
|
||
On the first run the session file need not exist. HTTP downloads resume from
|
||
their `.got` control file; torrents resume from the data already on disk
|
||
(re-checked by the engine). A magnet with no reachable peers no longer hangs:
|
||
`--bt-metadata-timeout` bounds the metadata fetch and defaults to 60s (set it to
|
||
0 to wait forever).
|
||
|
||
## Design
|
||
|
||
```
|
||
main classify URIs, wire options, run, report exit code
|
||
└─ download the Download interface + the scheduler (-j semaphore, ctx)
|
||
├─ httpdl []Segment + worker pool + WriteAt + JSON resume sidecar
|
||
└─ bt anacrolix/torrent behind the Download interface
|
||
├─ cli one flat option table -> hand parser -> layered config
|
||
└─ progress one ticker goroutine, pull-snapshot, \r line redraw
|
||
```
|
||
|
||
The contract is one small interface:
|
||
|
||
```go
|
||
type Download interface {
|
||
Name() string
|
||
Run(ctx context.Context) error // blocks in its own goroutine
|
||
Stat() Stat // a value snapshot, lock-free to read
|
||
}
|
||
```
|
||
|
||
- **HTTP** splits the file into byte-range `Segment`s handed to a pool of
|
||
workers. Each worker streams its range straight to one shared file with
|
||
`os.File.WriteAt` — safe for concurrent non-overlapping writes, so there is no
|
||
shared seek offset and no mutex. Resume is a small JSON sidecar
|
||
(`<file>.got`) holding per-segment progress plus validators (length +
|
||
ETag/Last-Modified) so a stale file is never trusted.
|
||
- **BitTorrent** is `anacrolix/torrent` configured from the CLI options and
|
||
driven to completion, then seeded for `--seed-time` minutes or up to
|
||
`--seed-ratio`.
|
||
- **Options** are one flat `[]Opt` table — the single source of truth for
|
||
parsing, validation, defaults and `--help`. Layers apply in order:
|
||
built-in defaults, config file, proxy environment, command line.
|
||
- **Progress** is a single goroutine that ticks once a second, pulls a snapshot
|
||
of the running downloads, derives speeds from the change since the last tick,
|
||
and redraws one line with `\r` + erase-to-end-of-line.
|
||
|
||
## Status
|
||
|
||
Implemented: segmented HTTP(S) with resume — including a foreign/browser-started
|
||
partial file (`-c`) — multi-mirror downloads (several URLs, or TAB-grouped `-i`
|
||
lines, fetch one file with connections spread across servers and fallover; `-Z`
|
||
to download them separately), retries, rate limits, auto-renaming,
|
||
`Content-Disposition` naming, conditional GET, cookies
|
||
(`--load-cookies`/`--save-cookies`), HTTP basic auth, `--remote-time`, whole-file
|
||
checksum verification (`--checksum`), and a `--dry-run` availability check;
|
||
BitTorrent download + seeding (magnet and `.torrent`,
|
||
DHT, trackers, file selection, show-files, metadata-fetch timeout), seeding that
|
||
stops on whichever of `--seed-time`/`--seed-ratio` comes first, and a listen port
|
||
chosen from the whole range so a busy port no longer disables BitTorrent;
|
||
a command-line interface with documented exit codes, config file, live progress,
|
||
and session save/reload for resume across runs.
|
||
|
||
Deferred (not implemented yet): FTP/SFTP, Metalink, and the
|
||
JSON-RPC server. The architecture leaves room for
|
||
each — a new protocol is just another `Download`. One known limitation: the
|
||
BitTorrent engine (`anacrolix/torrent`) applies download/upload rate limits
|
||
client-wide, so `--max-upload-limit`/`--max-download-limit` act per-run rather
|
||
than per-torrent when several torrents run at once (`--max-overall-*` are exact);
|
||
DHT/PEX likewise cannot yet be disabled per private torrent.
|