mysql-binlog-archiver

A minimal production-oriented runtime for continuous MySQL binlog archiving using mysqlbinlog --read-from-remote-server --raw --stop-never.

Why this exists

Binlog archiving closes the RPO gap between full/incremental backups. This runtime continuously streams remote binlogs to local storage, tracks stream state, applies safe retention rules, and optionally pushes operational metrics.

Architecture

• Stream loop: one long-running supervisor per source instance.
• Maintain loop: periodic retention, health probes, state/report update, optional metrics push.
• State store: atomic JSON files per source plus global maintain state.
• Metrics push: Prometheus text exposition over HTTP POST via Python stdlib.

flowchart LR
  A["MySQL source"] -->|"mysqlbinlog remote raw stream"| B["Stream service"]
  B --> C["binlog directory"]
  B --> D["per-source state JSON"]
  E["Maintain service"] --> C
  E --> D
  E --> F["maintain report JSON"]
  E -->|"optional push"| G["Metrics endpoint"]

Loading

Installation modes

Use Python 3.10 or newer.

Mode A: venv install (recommended)

Use this when you want isolated dependencies and predictable upgrades.

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -U pip
python -m pip install -e .

For development:

python -m pip install -e '.[dev]'
python -m pip install pylint

Mode B: system-wide install (no venv)

Use this when you manage runtime dependencies at host level.

python3 -m pip install -U pip
python3 -m pip install .

After install, verify the binary path:

command -v mysql-binlog-archiver

venv is not required for systemd operation as long as mysql-binlog-archiver is installed and available at the path used in your unit files.

Configuration model

Instance JSON schema

Each source uses one JSON file with fields:

• source_fqdn
• source_id
• connect_host
• port (default: 3306)
• server_id (default: 4242)
• defaults_file
• binlog_dir
• bootstrap_policy (latest, oldest, explicit, default: latest)
• bootstrap_file (required for explicit)

See examples/instances/source.example.json.

MySQL defaults-extra-file

The file referenced by defaults_file must contain credentials and target endpoint for mysql and mysqlbinlog:

• host
• port
• user
• password

See examples/credentials/source.example.my.cnf.

Runtime directories

• Instance definitions: operator-defined directory with *.json
• State directory: JSON files managed by runtime
• Reports directory: maintain-latest.json
• Log directory: one stream log file per source

Runtime commands

Stream mode

mysql-binlog-archiver stream \
  --instance /etc/mysqlbinlog_archiver/instances/mysql-01.example.com.json \
  --state-dir /var/lib/mysqlbinlog_archiver/state \
  --log-dir /var/log/mysqlbinlog_archiver \
  --backoff-initial-seconds 5 \
  --backoff-max-seconds 300

Maintain mode

mysql-binlog-archiver maintain \
  --instances-dir /etc/mysqlbinlog_archiver/instances \
  --state-dir /var/lib/mysqlbinlog_archiver/state \
  --reports-dir /var/lib/mysqlbinlog_archiver/reports \
  --retention-days 14 \
  --close-after-seconds 900 \
  --retention-interval-seconds 900 \
  --metrics-interval-seconds 300 \
  --metrics-url http://127.0.0.1:8429/api/v1/import/prometheus \
  --metrics-timeout-seconds 5 \
  --metrics-retries 2 \
  --precheck-timeout-seconds 3

Systemd deployment (quickstart)

This project ships unit templates in examples/systemd/. You install and manage them.

1. Install runtime (Mode A or Mode B).
2. Create runtime directories and service user.
3. Place instance JSON and credentials files.
4. Copy and adjust unit files.
5. Reload systemd, enable services, and validate status.

Example bootstrap commands:

sudo useradd --system --home /var/lib/mysqlbinlog_archiver --shell /usr/sbin/nologin archiver
sudo install -d -o archiver -g archiver -m 0750 /var/lib/mysqlbinlog_archiver/state
sudo install -d -o archiver -g archiver -m 0750 /var/lib/mysqlbinlog_archiver/reports
sudo install -d -o archiver -g archiver -m 0750 /var/log/mysqlbinlog_archiver
sudo install -d -o root -g root -m 0755 /etc/mysqlbinlog_archiver/instances
sudo install -d -o root -g root -m 0755 /etc/mysqlbinlog_archiver/credentials

Install units and start services:

sudo cp examples/systemd/[email protected] /etc/systemd/system/
sudo cp examples/systemd/mysqlbinlog-archiver-maintain.service /etc/systemd/system/
sudo cp examples/systemd/mysqlbinlog-archiver-maintain.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now [email protected]
sudo systemctl enable --now mysqlbinlog-archiver-maintain.timer

Validate:

sudo systemctl status [email protected]
sudo systemctl status mysqlbinlog-archiver-maintain.timer
sudo journalctl -u [email protected] -n 100 --no-pager

See the runbook for production checks and troubleshooting.

Failure handling behavior

• connectivity: stream process lost connectivity.
• connectivity_bootstrap: source unreachable during bootstrap (SHOW BINARY LOGS).
• auth: access denied or authentication problem.
• source_gap: requested start file is no longer available on source.
• disk_full: ENOSPC/EDQUOT/read-only filesystem errors.
• permission: write/access permission errors.
• bootstrap: start-file resolution failed for non-connectivity reason.
• process: mysqlbinlog exited with generic non-zero process error.
• unexpected_exit: mysqlbinlog exited with code 0 unexpectedly.

Retry is continuous with bounded exponential backoff.

Security notes

• Keep credentials in --defaults-extra-file, not CLI flags.
• Store credential files with mode 0600.
• Treat state and logs as operational data; avoid writing secrets there.

Limitations and non-goals

• This runtime does not manage MySQL user grants.
• This runtime does not deploy systemd units automatically.
• This runtime does not compress, checksum, or tier archived binlogs.

Operations docs

• Runbook
• Configuration Reference
• Metrics Reference