drumsergio/web-mirror repository overview

⁠web-mirror

A Playwright script that saves the pages of one website as local HTML files, videos included, plus an nginx config to serve the copy offline.

The site and the page list are set in the code, not by a setting: src/main2.py ships with the placeholder domain www.place.holder and a one-page list (line 34), so you edit it before you build. It runs in Docker and writes the copy to /data.

⁠Features

• Renders each page in headless Chromium through Playwright, so pages built by JavaScript are captured.
• Saves each page as index.html in a folder named after its URL path (a page with a video as <path>.html).
• Downloads the <video> source of a page next to its HTML.
• Rewrites links to the mirrored site into local paths.
• Can read the page list from the site's sitemap.xml (swap line 34 for return links).
• Ships an nginx config (nginx/default.conf) that serves the copy with directory listings.

⁠Quick start

git clone https://github.com/GeiserX/web-mirror.git && cd web-mirror
# edit src/main2.py: replace www.place.holder with your site (lines 27, 34, 71, 72) and set the pages on line 34
docker build -t web-mirror . && docker run --rm -v "$PWD/data:/data" web-mirror

On an Apple Silicon Mac, add --platform linux/amd64 to docker build and docker run, and let Docker emulate amd64 with Rosetta (Colima --vz-rosetta, or Docker Desktop's Rosetta setting): the base image exists for amd64 only, and Chromium crashes under QEMU.

The copy lands in ./data. To serve it, run docker run --rm -p 8080:80 -v "$PWD/data:/usr/share/nginx/html:ro" -v "$PWD/nginx/default.conf:/etc/nginx/conf.d/default.conf:ro" nginx:1.29-alpine and open http://localhost:8080⁠. The published image (drumsergio/web-mirror:1.0.2, amd64) carries the placeholder, so it only helps as a base for your own build.

⁠Documentation

The full documentation is at geiserx.github.io/web-mirror⁠.

• Getting started⁠: what you need, the four lines to edit, build, run, and what a finished run looks like
• Configuration⁠: every constant in src/main2.py, and how to save the whole sitemap
• Usage⁠: where each page lands, serving the copy with nginx, running again, the published image without a build
• How it works⁠: what happens to each page, which links are rewritten, what is not saved
• Troubleshooting⁠: the arm64 build error, the QEMU crash, a full Docker disk, the placeholder's ConnectionError, video pages
• Development⁠: tests, CI, releases and the docs build
• Related projects⁠: the other web-archiving tools

⁠Related projects

⁠License

GPL-3.0-or-later⁠