tx: Fast file transfer for the cloud

tx copies directories and files at line rate by fully saturating modern cloud hardware - SSD, NIC, and CPU - simultaneously. On large servers it achieves up to 10x the throughput of a simple rsync --fsync + happycache while providing above line-rate encryption, compression, and page cache transfer.

Four capabilities make this possible:

• Adaptive concurrency. A probe measures link speed and server capacity, then opens a pool of pre-authenticated TCP connections sized to keep every core and disk busy. Work is packed into fixed-size batches - many small files are bundled into one request, and large files are split into parallel chunks - so every connection does roughly equal work with no head-of-line blocking.

• Adaptive compression. The server monitors read vs. write latency per 4 MiB frame and walks a none → lz4 → zstd ladder. Compressible files effectively exceed line rate; incompressible files (media, archives) stream with zero CPU overhead.

• Background durability. Per-file fdatasync is moved off the download critical path into a background batcher that deduplicates by inode. It starts with a 512 MiB flush target, grows the batch size under backlog pressure. A final syncfs ensures the entire filesystem is durable before the transfer reports success.

• Lightweight verification. Metadata checks (size, mtime, permissions) run by default after every copy. Sampled data verification (--verify 5%data) reads about 5% of bytes and catches corruption without a full re-read or time budget verification (--verify 30s) verifies all metadata and as much sampled data as it can in the time budget - practical on slow network attached drives such as EBS.

See docs/README.md for the documentation index and docs/arch/OVERVIEW.md for a deeper dive. If you are curious about benchmarking on your own setup check out bench/README.md which has a data generator that can both generate incompressible (random) and compressible (Silesia) test datasets as well as run tx or alternative rsync commands.

# Serve the current directory on the default port (127.0.0.1:3453)
tx send tree

# Serve a specific directory on all interfaces
tx send tree --listen 0.0.0.0:3453 /srv/data

# Copy /srv/data from a remote server to a local directory
tx recv copy tx://10.0.0.1:3453/srv/data /var/lib/data

Rerun the same command to sync deltas — copy detects an existing destination and only transfers new or changed files. Metadata verification runs by default; add --verify 5%data for sampled data checks or --verify 30s to verify as much as you can in a 30s budget.

# Server — require encryption and proof of the client's private key
tx send tree --require-auth --listen 0.0.0.0:3453 /srv/data

# Client — generates an in-memory key
tx recv copy --encrypt auto tx://10.0.0.1:3453/srv/data /var/lib/data

This accepts any client with a key pair. For a long-running server with a known age public key, the client can also authenticate the server with --require-server-recipient age1.... Obtain that key through a trusted channel. Each connection runs one Noise handshake, then encrypts every command and response. Pooled connections reuse the session.

Allowlist entries imply --require-auth and restrict which clients can connect.

# The token is secret: don't log it! Share it with the client out of band.
export TX_AUTH_TOKEN=$(openssl rand -hex 32)

# Server
tx send tree --require-auth-token "$TX_AUTH_TOKEN" --listen 0.0.0.0:3453 /srv/data

# Client — reads TX_AUTH_TOKEN automatically; no key files needed
tx recv copy tx://10.0.0.1:3453/srv/data /var/lib/data

# Generate a secret token; copy the file to the client securely
umask 077
openssl rand -hex 32 > client.token

# Server
tx send tree --require-auth-token-file client.token --listen 0.0.0.0:3453 /srv/data

# Client
tx recv copy --auth-token-file client.token tx://10.0.0.1:3453/srv/data /var/lib/data

Tokens never cross the wire: each side proves it holds the token inside the handshake. Use one token per client so each can be revoked alone.

# Client — generate a private key and share only its public recipient
mkdir -p client-keys
age-keygen -o client-keys/key
age-keygen -y client-keys/key

# Server — allow the recipient printed above
tx send tree --require-client-recipient age1... --listen 0.0.0.0:3453 /srv/data

# Client — proves it holds the corresponding private key
tx recv copy --keys ./client-keys tx://10.0.0.1:3453/srv/data /var/lib/data

See docs/pub/CLI.md for the complete flag reference including --mode gentle, --verify, --deadline, progress reporting, and the get / status subcommands. FTCP wire-format details live in docs/ftcp/OVERVIEW.md.

go build ./cmd/tx
go test ./...