Situatie
wget is a command-line tool to download files from the web using HTTP, HTTPS, and FTP. It works non-interactively (can run in background), supports recursion, resuming, and rate-limiting.
Solutie
Basic usage
- Download a single file:
wget https://example.com/file.zip - Save with a different filename:
wget -O myfile.zip https://example.com/file.zip
- Download in background (log to file):
wget -b https://example.com/largefile.iso
- Resume an interrupted download:
wget -c https://example.com/largefile.iso
Downloading multiple files
- From a list in a file (one URL per line):
wget -i urls.txt
Recursive download (mirror a site or directory)
- Basic mirror (downloads site, follows links, converts links for local viewing):
wget --mirror -p --convert-links -P localdir https://example.com/
- –mirror = -r -N -l inf –no-remove-listing
- -p downloads all files necessary to display pages (images, CSS)
- –convert-links adjusts links for local browsing
- -P localdir saves into localdir
Authentication and headers
- HTTP basic auth:
wget --user=username --password=secret https://example.com/protected/file
- Send a custom header (e.g., API key):
wget --header="Authorization: Bearer TOKEN" https://api.example.com/data.json
Limit speed, retries, and timeout
- Limit download rate:
wget --limit-rate=200k https://example.com/file.iso
- Set number of retries (default 20) and timeout:
wget --tries=10 --timeout=30 https://example.com/file
FTP, proxies, and passive mode
- Use FTP URL:
wget ftp://ftp.example.com/pub/file.tar.gz
- Force passive FTP:
wget --ftp-user=user --ftp-password=pass --passive-ftp ftp://ftp.example.com/file
Common useful flags (summary)
- -O : write output to file
- -c : continue/resume
- -b : background
- -q : quiet
- -nv : non-verbose
- -r : recursive
- -l : recursion depth
- -np : no parent (don’t ascend to parent directories)
- –limit-rate=RATE : throttle speed
- –header=”Header: value” : custom header
Examples:
- Download a webpage and its resources for offline viewing:
wget –mirror -p –convert-links -P site_copy https://example.com
- Download 10 files listed in urls.txt concurrently (using xargs for parallelism):
cat urls.txt | xargs -n1 -P10 wget -q
Tips:
- Check wget version:
wget --version - Use -q for scripting (quiet), and check exit status ($?) for success
- wget is simpler for straightforward downloads and mirroring.
Leave A Comment?