Hojun-Cho/got
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

httpdl: add segment work-stealing and --auto-split

Browse Source 
an idle worker now steals the back half of the busiest in-flight segment
instead of exiting, so one slow segment no longer drains alone over a single
connection. control file persists stolen tails; resume re-tiles.

add --auto-split (opt-in): derive the connection count from file size (one
per min-split-size, up to 16), overriding -x/-s. default stays at 1
connection for a single server.
This commit is contained in:
Hojun-Cho
2026-06-19 13:56:31 +09:00
parent eb8d9b9460
commit 554beb6b48
7 changed files with 407 additions and 171 deletions
Download Patch File Download Diff File Expand all files Collapse all files

154
README.md
View File

@@ -1,13 +1,8 @@
# got
A small, Rob-Pike-style command-line download tool in Go: one program that
downloads HTTP(S) URLs over several connections and BitTorrent
magnets/torrents, with a focused set of options.
The design leans on Go's runtime instead of a hand-rolled reactor. There is no
single-threaded event-poll loop: one goroutine per download blocks on real I/O,
a buffered channel of slots bounds concurrency, and `context` carries shutdown.
What remains is the data — and, per Pike, the data is the design.
A command-line download tool in Go. It fetches HTTP(S) files over multiple
connections and downloads BitTorrent magnets and `.torrent` files — one binary,
familiar flags.
## Build
@@ -15,148 +10,67 @@ What remains is the data — and, per Pike, the data is the design.
go build -o got .
```
Requires Go 1.25+. The BitTorrent engine is
[`anacrolix/torrent`](https://github.com/anacrolix/torrent); everything else is
the standard library plus `golang.org/x/{term,time}`.
Needs Go 1.25+.
## Use
## Usage
```sh
# segmented HTTP download, 16 connections, into ./downloads
got -x16 -s16 -d downloads https://example.com/big.iso
# HTTP download with 16 connections
got -x16 -s16 https://example.com/big.iso
# let got pick the connection count from the file size
got --auto-split https://example.com/big.iso
# resume an interrupted download
got -c -x16 https://example.com/big.iso
got -c https://example.com/big.iso
# the same file from several mirrors at once (one file, connections spread
# across servers, a dead mirror falls over); -Z downloads them separately instead
# one file from several mirrors (connections spread across servers,
# a dead mirror falls over); -Z downloads them as separate files instead
got -x4 https://a.example/big.iso https://b.example/big.iso
# a torrent or magnet; seed for 30 minutes after finishing
got --seed-time=30 ubuntu.torrent
got 'magnet:?xt=urn:btih:...'
# list the files in a torrent without downloading
got -S -T some.torrent
# several downloads at once, two running in parallel
# many downloads, two at a time
got -j2 -i urls.txt
```
Run `got --help` (or `--help=http`, `--help=bittorrent`, ...) for the full
option list.
Run `got --help` (or `--help=all`) for every option.
### Options
A focused subset of options, with familiar short flags and sensible defaults.
## Common options
| flag | meaning | default |
|------|---------|---------|
| `-d, --dir` | output directory | `.` |
| `-o, --out` | output filename (single download) | from URL |
| `-i, --input-file` | read downloads line by line (TAB-separated URLs = mirrors; `-` = stdin) | |
| `-o, --out` | output filename | from URL |
| `-c, --continue` | resume a partial download | false |
| `-x, --max-connection-per-server` | connections to one server (1–16) | 1 |
| `-s, --split` | split a download into N connections | 5 |
| `-Z, --force-sequential` | download each command-line URL separately, not as mirrors | false |
| `-k, --min-split-size` | do not split below this size | 20M |
| `-j, --max-concurrent-downloads` | parallel downloads | 5 |
| `--auto-split` | choose connections from file size (overrides `-x`/`-s`) | false |
| `-j, --max-concurrent-downloads` | downloads at once | 5 |
| `--max-overall-download-limit` | global speed cap | 0 (off) |
| `--max-download-limit` | per-download speed cap | 0 (off) |
| `--retry-wait` | seconds to wait between retries | 0 |
| `--load-cookies` / `--save-cookies` | read/write a Netscape `cookies.txt` | |
| `--conditional-get` | skip the download if the local file is up to date | false |
| `--remote-time` | set the file's mtime from the server | false |
| `--checksum` | verify the finished file: `TYPE=DIGEST` (sha-256, sha-1, md5, …) | |
| `--dry-run` | check the file is available but do not download it | false |
| `--ca-certificate` | verify HTTPS against the CA certificates in FILE (PEM) | |
| `--stop` | stop the program after N seconds | 0 (off) |
| `--disable-ipv6` | force IPv4-only connections | false |
| `-T, --torrent-file` | a `.torrent` file | |
| `--seed-time` | minutes to seed after finishing | seed by ratio |
| `--checksum` | verify the finished file: `TYPE=DIGEST` (sha-256, sha-1, …) | |
| `--seed-time` | minutes to seed after a torrent finishes | by ratio |
| `--seed-ratio` | stop seeding at this ratio | 1.0 |
| `--listen-port` | port (range) for incoming peers | 6881-6999 |
| `--enable-dht` | use the BitTorrent DHT | true |
| `--select-file` | fetch only these file indexes (`1,3-5`) | all |
| `-S, --show-files` | list torrent files and exit | |
| `--bt-stop-timeout` | give up if no download progress for N s | 0 (off) |
| `--bt-metadata-timeout` | give up if a magnet can't fetch metadata in N s | 60 |
| `--save-session` | on exit, write unfinished downloads to FILE | |
| `-q, --quiet` | no progress readout | false |
| `-q, --quiet` | no progress output | false |
### Resuming across runs
## Resume
There is no daemon — resume is a plain file. Point
`--save-session` and `-i` at the same file and add `-c`: unfinished downloads
are loaded at start and the still-unfinished ones are written back on exit.
HTTP downloads resume from a small `<file>.got` sidecar — just re-run with `-c`.
Torrents resume from the data already on disk. To resume across runs, point
`--save-session` and `-i` at the same file:
```sh
got -c --save-session=got.session -i got.session <new urls/magnets...>
got -c --save-session=got.session -i got.session <urls/magnets...>
```
On the first run the session file need not exist. HTTP downloads resume from
their `.got` control file; torrents resume from the data already on disk
(re-checked by the engine). A magnet with no reachable peers no longer hangs:
`--bt-metadata-timeout` bounds the metadata fetch and defaults to 60s (set it to
0 to wait forever).
## Notes
## Design
```
main classify URIs, wire options, run, report exit code
└─ download the Download interface + the scheduler (-j semaphore, ctx)
├─ httpdl []Segment + worker pool + WriteAt + JSON resume sidecar
└─ bt anacrolix/torrent behind the Download interface
├─ cli one flat option table -> hand parser -> layered config
└─ progress one ticker goroutine, pull-snapshot, \r line redraw
```
The contract is one small interface:
```go
type Download interface {
Name() string
Run(ctx context.Context) error // blocks in its own goroutine
Stat() Stat // a value snapshot, lock-free to read
}
```
- **HTTP** splits the file into byte-range `Segment`s handed to a pool of
workers. Each worker streams its range straight to one shared file with
`os.File.WriteAt` — safe for concurrent non-overlapping writes, so there is no
shared seek offset and no mutex. Resume is a small JSON sidecar
(`<file>.got`) holding per-segment progress plus validators (length +
ETag/Last-Modified) so a stale file is never trusted.
- **BitTorrent** is `anacrolix/torrent` configured from the CLI options and
driven to completion, then seeded for `--seed-time` minutes or up to
`--seed-ratio`.
- **Options** are one flat `[]Opt` table — the single source of truth for
parsing, validation, defaults and `--help`. Layers apply in order:
built-in defaults, config file, proxy environment, command line.
- **Progress** is a single goroutine that ticks once a second, pulls a snapshot
of the running downloads, derives speeds from the change since the last tick,
and redraws one line with `\r` + erase-to-end-of-line.
## Status
Implemented: segmented HTTP(S) with resume — including a foreign/browser-started
partial file (`-c`) — multi-mirror downloads (several URLs, or TAB-grouped `-i`
lines, fetch one file with connections spread across servers and fallover; `-Z`
to download them separately), retries, rate limits, auto-renaming,
`Content-Disposition` naming, conditional GET, cookies
(`--load-cookies`/`--save-cookies`), HTTP basic auth, `--remote-time`, whole-file
checksum verification (`--checksum`), and a `--dry-run` availability check;
BitTorrent download + seeding (magnet and `.torrent`,
DHT, trackers, file selection, show-files, metadata-fetch timeout), seeding that
stops on whichever of `--seed-time`/`--seed-ratio` comes first, and a listen port
chosen from the whole range so a busy port no longer disables BitTorrent;
a command-line interface with documented exit codes, config file, live progress,
and session save/reload for resume across runs.
Deferred (not implemented yet): FTP/SFTP, Metalink, and the
JSON-RPC server. The architecture leaves room for
each — a new protocol is just another `Download`. One known limitation: the
BitTorrent engine (`anacrolix/torrent`) applies download/upload rate limits
client-wide, so `--max-upload-limit`/`--max-download-limit` act per-run rather
than per-torrent when several torrents run at once (`--max-overall-*` are exact);
DHT/PEX likewise cannot yet be disabled per private torrent.
- BitTorrent uses [`anacrolix/torrent`](https://github.com/anacrolix/torrent);
HTTP uses the standard library.
- Not implemented: FTP/SFTP, Metalink, and the JSON-RPC server.
- When several torrents run at once, `--max-download-limit` /
`--max-upload-limit` apply per run rather than per torrent (`--max-overall-*`
are exact).