Remove --autotune

CLAUDE.md

@@ -232,30 +232,33 @@ walkers.**
 - Cold-walk cost is fundamental btrfs metadata I/O (`statx→btrfs_iget→btree`);
   SQLite is <2%, so parallelizing the consumer wouldn't help.
 
-## io-threads auto-tuning (--autotune)
+## io-threads default (storage heuristic)
 
-`--io-threads` sizes three pools (walkers, csum/read, dedupe). Two mechanisms
-refine its default, both **only when the user didn't pass `--io-threads`**
-(`options.io_threads_set`) and both resolved on the main thread after the roots
-are known (`auto_tune_io_threads()` in `oans.c`).
+`--io-threads` sizes three pools (walkers, csum/read, dedupe). One mechanism
+refines its default, **only when the user didn't pass `--io-threads`** (the
+sentinel is `options.io_threads == 0`), resolved on the main thread after the
+roots are known (`apply_storage_defaults()` in `oans.c`).
 
 - **`src/storage.{c,h}` — heuristic from device type.** `storage_detect()`
   reports rotational-ness + device count (btrfs pools enumerated via
   `BTRFS_IOC_FS_INFO`/`DEV_INFO` → `/sys/.../queue/rotational`).
   `storage_recommend_io_threads()` is pure and unit-tested: SSD/unknown keep
-  `min(nproc,8)` (the validated path, unchanged); single HDD ≤4; HDD pool
-  ~2/device capped at 8. **The HDD constants are unmeasured guesses** — treat
-  `--autotune` as authoritative.
-- **`src/autotune.{c,h}` — empirical (`--autotune`).** Re-execs oans on a bounded
-  sample (`-` file list, no hashfile → pure in-memory read+hash) at each thread
-  count, interleaved, keeping each candidate's fastest run. Drops the page cache
-  between trials (`drop_caches`, needs root). Bounds:
-  `DUPEREMOVE_AUTOTUNE_{MAX_FILES,MAX_BYTES,ROUNDS}`. With `--hashfile` it stores
-  the winner (config key `autotune_io_threads`); a later plain run reads it back
-  (an explicit `--io-threads` still overrides). Pinned by `test_autotune.py`.
-- **Measuring autotune on the dev box is misleading:** on an unsupported fs
-  (ext) trials read 0 bytes and the numbers are startup noise; needs a btrfs/xfs
-  target.
+  `min(nproc,8)` (the validated path); single HDD ≤4; HDD pool ~2/device capped
+  at 8.
+- **The HDD/pool constants are unmeasured guesses** and there is now no in-tree
+  way to validate them — we have no spinning-disk target. The SSD/unknown path
+  is the measured one (see the walker plateau in *Scan parallelism*). If you get
+  access to real rotational storage, measure with `scripts/bench.py
+  --walk-threads 4,8,16,32` and fix the constants; don't guess again.
+- **`--autotune` was removed** (#153) — it measured warm-cache throughput
+  whenever it couldn't drop caches (i.e. without root), recommended and
+  *persisted* a thread count above the measured btrfs plateau, and the stored
+  value then won over the heuristic on every later run. Don't reintroduce a
+  self-measuring mode that writes to the hashfile. `scripts/bench.py` answers
+  the same question properly, as a dev tool.
+- The same `storage_detect()` call also feeds the scan-ETA rotational weight
+  (`pscan_set_storage_rotational()`), which must be set **even when io-threads
+  is fixed** by an explicit flag — hence the ordering in that function.
 
 ## Self-describing hashfile, history & scheduling (fork features)
 
@@ -276,9 +279,10 @@ are known (`auto_tune_io_threads()` in `oans.c`).
   reuses the progress counters, so a later read is 0. (Not `files_examined`,
   which counts every file the walk *visited*, up-to-date ones included, not just
   those hashed.) Pinned by `test_history_metrics.py`.
-- **Report modes** `-L`/`-R`/`--stats`/`--history`/`--json`/`--autotune` are
-  mutually exclusive (one `report_count` check in `parse_options`): `--stats` =
-  hashfile report, `-L` lists files, `-R` removes paths.
+- **Report modes** `-L`/`-R`/`--stats`/`--history`/`--json` are mutually
+  exclusive (one `report_count` check in `parse_options`): `--stats` = hashfile
+  report, `-L` lists files, `-R` removes paths. All but `-R` open the hashfile
+  read-only.
 - **Scheduling.** `systemd/oans@.{service,timer}` (via `make install-systemd`,
   kept out of `make install`) run `oans --hashfile=/var/cache/oans/%i.hash` on a
   timer, replaying the stored config. Guide: `docs/nas-quickstart.md`.
@@ -480,8 +484,9 @@ check unlinks and recreates (it's only a cache).
   file (a full re-scan). Do **not** bump for purely *additive* changes (a new
   `CREATE TABLE IF NOT EXISTS`, or an optional `config` key): `create_tables()`
   runs every open, so additive tables appear on old files and old binaries ignore
-  the extras. Self-describing, run-history and autotune were all added this way,
-  left at `5.0`.
+  the extras. Self-describing and run-history were both added this way, left at
+  `5.0`. Removing a key is equally safe: `--autotune`'s orphaned
+  `autotune_io_threads` row is simply never read again (no bump, no migration).
 - A from-scratch build sets `hashfile_rebuilt` → `dbfile_maybe_vacuum()` forces a
   one-off `VACUUM` (a fresh build is at insert density, ~15-20% larger
   un-vacuumed). Incremental runs only VACUUM at ≥25% free.

README.md

@@ -101,8 +101,7 @@ sudo systemctl enable --now [email protected]
 
 📖 **Setting this up on a NAS or home server?** Follow the
 **[NAS quick-start guide](docs/nas-quickstart.md)** — a complete walkthrough
-from first scan to scheduled, monitored dedupe (including `--autotune` to pick
-the fastest I/O settings for your disks).
+from first scan to scheduled, monitored dedupe.
 
 ## Watching it work
 
@@ -266,8 +265,8 @@ commands and comparison binary are documented in the
   for zero-config scheduling.
 - **systemd `oans@` service/timer templates** (`make install-systemd`) for
   weekly, idle-priority dedupe, with a **[NAS quick-start guide](docs/nas-quickstart.md)**.
-- **`--autotune`** empirically picks the fastest `--io-threads` for the backing
-  storage (and a device-type heuristic sizes the default for HDD/RAID pools).
+- **Storage-aware thread defaults:** `--io-threads` is sized from the detected
+  backing storage (SSD vs single spinning disk vs multi-device pool).
 - **`--min-filesize`**, **`--cpu-threads`**, **`--no-color`** and **`-q`** added;
   the legacy `--fdupes` mode and other dead/testing options removed.
 - **Automatic housekeeping:** prune deleted files from the hashfile after a scan
@@ -326,8 +325,8 @@ Differences to know about:
   SQLite `application_id`; oans and duperemove will each rebuild rather than
   read the other's. Hashfiles are only caches, so nothing is lost.
 - **CLI**: a few additions (`--stats`, `--history`, `--json`, `--progress=json`,
-  `--autotune`, `--min-filesize`, `--cpu-threads`, `--no-color`, `-q`), a few
-  legacy/testing options removed.
+  `--min-filesize`, `--cpu-threads`, `--no-color`, `-q`), a few legacy/testing
+  options removed.
 - Scripts that expect the `duperemove` binary keep working via the installed
   compatibility symlink, including the stable
   `net change in shared extents` output line.

completion/zsh/_oans

@@ -32,7 +32,6 @@ _duperemove() {
 		'--stats[print a report about the hashfile and exit]'
 		'--history[print the run history recorded in the hashfile and exit]'
 		'--json[print machine-readable hashfile metrics as JSON and exit]'
-		'--autotune[measure the fastest --io-threads for this machine and exit]'
 		'--skip-zeroes[read data blocks and skip any zeroed blocks]'
 		'-b[use the specified block size]:block size: '
 		'--io-threads=[use N threads for IO]:number of thread: '

docs/benchmarks.md

@@ -279,8 +279,8 @@ Under the hood, for each round and interleaving the binaries, it:
   `compsize` **Disk Usage** before/after; `Referenced` staying constant proves
   nothing was lost.
 - **Walker parallelism plateaus.** On btrfs, scaling `--io-threads` past ~8 gives
-  no wall-clock gain — it is metadata b-tree lock contention, not I/O. Use
-  `--autotune` to pick the fastest count for your disks.
+  no wall-clock gain — it is metadata b-tree lock contention, not I/O. oans sizes
+  the default from the detected storage; pass `--io-threads=N` to override.
 - **One machine, one filesystem.** A fast NVMe here — on a slow HDD (a typical
   NAS) the cold dedupe re-read is far more expensive, so the larger-than-RAM gap
   would be *larger*, not smaller. The cgroup cap models RAM pressure; a real

docs/man/oans.8

@@ -27,10 +27,6 @@ kernel\[cq]s deduplication ioctl.
 .PD
 \f[B]oans\f[R] \f[B]\-R\f[R] \f[B]\-\-hashfile\f[R]=\f[I]FILE\f[R]
 \f[I]file\f[R]\&...
-.PD 0
-.P
-.PD
-\f[B]oans\f[R] \f[B]\-\-autotune\f[R] \f[I]file\f[R]\&...
 .SH DESCRIPTION
 \f[CR]oans\f[R] finds regions of identical data across a set of files
 and, on request, asks the kernel to make them share storage \[em]
@@ -84,10 +80,10 @@ A summary of the space reclaimed is printed at the end (see
 \f[I]OUTPUT\f[R]).
 .PP
 \f[B]Report and maintenance modes.\f[R] \f[B]\-\-stats\f[R],
-\f[B]\-\-history\f[R], \f[B]\-\-json\f[R], \f[B]\-L\f[R], \f[B]\-R\f[R],
-and \f[B]\-\-autotune\f[R] each inspect or maintain a hashfile and then
-exit without scanning for dupes.
-The first four open the hashfile \f[B]read\-only\f[R]; see
+\f[B]\-\-history\f[R], \f[B]\-\-json\f[R], \f[B]\-L\f[R], and
+\f[B]\-R\f[R] each inspect or maintain a hashfile and then exit without
+scanning for dupes.
+All but \f[B]\-R\f[R] open the hashfile \f[B]read\-only\f[R]; see
 \f[I]NOTES\f[R] for running them alongside an active \f[CR]oans\f[R].
 .SH OPTIONS
 The non\-option arguments are the \f[I]files\f[R] and
@@ -214,8 +210,6 @@ storage and chooses automatically: the host CPU count capped at
 for a single spinning disk (seek\-bound); and roughly two per device for
 a multi\-device btrfs pool, still capped at 8.
 Run with \f[B]\-v\f[R] to see the detected storage and the value chosen.
-A stored \f[B]\-\-autotune\f[R] result, if present, is used in
-preference to the heuristic.
 Passing \f[I]N\f[R] explicitly disables all of this and uses \f[I]N\f[R]
 verbatim.
 .TP
@@ -265,24 +259,6 @@ reads the list from standard input.
 Requires \f[B]\-\-hashfile\f[R].
 (Deleted files are also pruned automatically on the next scan;
 \f[B]\-R\f[R] is for removing paths that still exist.)
-.TP
-\f[B]\-\-autotune\f[R]
-Measure the fastest \f[B]\-\-io\-threads\f[R] for this machine and exit.
-\f[CR]oans\f[R] reads and hashes a bounded sample of the given tree at
-several thread counts, dropping the page cache between trials (run as
-\f[B]root\f[R] for meaningful cold\-read numbers on spinning disks), and
-prints a throughput table with the winner.
-Directories are always sampled recursively \[em] the sample is about the
-storage, not the \f[B]\-r\f[R] flag.
-With \f[B]\-\-hashfile\f[R] the winning value \f[I]and\f[R] the scan
-configuration are stored, so \f[CR]\-\-autotune\f[R] doubles as
-one\-time setup: a later bare \f[CR]oans \-\-hashfile=FILE\f[R] replays
-it, using the tuned thread count unless you pass an explicit
-\f[B]\-\-io\-threads\f[R].
-This is the hardware\-measured counterpart to the storage heuristic and
-the reliable way to tune a NAS or disk array.
-Sample bounds are controlled by the \f[CR]DUPEREMOVE_AUTOTUNE_*\f[R]
-variables (see \f[I]ENVIRONMENT\f[R]).
 .SS Output and information
 .TP
 \f[B]\-q\f[R], \f[B]\-\-quiet\f[R]
@@ -437,13 +413,6 @@ oans \-\-stats   \-\-hashfile=foo.hash
 oans \-\-history \-\-hashfile=foo.hash
 oans \-\-json    \-\-hashfile=foo.hash | jq .reclaimed_total_bytes
 .EE
-.PP
-Measure the best thread count for a NAS array (as root), storing it for
-later runs:
-.IP
-.EX
-sudo oans \-\-autotune \-\-hashfile=media.hash /srv/media
-.EE
 .SS Scheduled deduplication
 Because the hashfile is self\-describing, a scheduled run only needs to
 name it.
@@ -475,10 +444,6 @@ deduped\f[R] line, or run with \f[B]\-v\f[R], to detect them.
 \f[B]NO_COLOR\f[R]
 If set (to any value), disables colored output, as
 \f[B]\-\-no\-color\f[R] does.
-.TP
-\f[B]DUPEREMOVE_AUTOTUNE_MAX_FILES\f[R], \f[B]DUPEREMOVE_AUTOTUNE_MAX_BYTES\f[R], \f[B]DUPEREMOVE_AUTOTUNE_ROUNDS\f[R]
-Bound the sample \f[B]\-\-autotune\f[R] reads (file count, total bytes,
-and number of interleaved measurement rounds).
 .SH FILES
 .TP
 \f[I]hashfile\f[R]

docs/man/oans.md

@@ -18,7 +18,6 @@ deduplication ioctl.
 | **oans** **\--hashfile**=*FILE*
 | **oans** {**\--stats** | **\--history** | **\--json** | **-L**} **\--hashfile**=*FILE*
 | **oans** **-R** **\--hashfile**=*FILE* *file*...
-| **oans** **\--autotune** *file*...
 
 # DESCRIPTION
 
@@ -64,10 +63,10 @@ skipped, so repeated runs over a mostly-stable tree are cheap. A summary of the
 space reclaimed is printed at the end (see *OUTPUT*).
 
 **Report and maintenance modes.**
-**\--stats**, **\--history**, **\--json**, **-L**, **-R**, and **\--autotune**
-each inspect or maintain a hashfile and then exit without scanning for dupes.
-The first four open the hashfile **read-only**; see *NOTES* for running them
-alongside an active `oans`.
+**\--stats**, **\--history**, **\--json**, **-L**, and **-R** each inspect or
+maintain a hashfile and then exit without scanning for dupes. All but **-R**
+open the hashfile **read-only**; see *NOTES* for running them alongside an
+active `oans`.
 
 # OPTIONS
 
@@ -174,9 +173,8 @@ directory scans the regular files directly inside it; add **-r** to recurse.
     non-rotational disks (SSD/NVMe) or unknown media; fewer for a single
     spinning disk (seek-bound); and roughly two per device for a multi-device
     btrfs pool, still capped at 8. Run with **-v** to see the detected storage
-    and the value chosen. A stored **\--autotune** result, if present, is used
-    in preference to the heuristic. Passing *N* explicitly disables all of this
-    and uses *N* verbatim.
+    and the value chosen. Passing *N* explicitly disables all of this and uses
+    *N* verbatim.
 
 **\--cpu-threads**=*N*
   ~ Number of threads for the CPU-bound duplicate-extent-finding stage. Default
@@ -218,20 +216,6 @@ directory scans the regular files directly inside it; add **-r** to recurse.
     pruned automatically on the next scan; **-R** is for removing paths that
     still exist.)
 
-**\--autotune**
-  ~ Measure the fastest **\--io-threads** for this machine and exit. `oans`
-    reads and hashes a bounded sample of the given tree at several thread
-    counts, dropping the page cache between trials (run as **root** for
-    meaningful cold-read numbers on spinning disks), and prints a throughput
-    table with the winner. Directories are always sampled recursively — the
-    sample is about the storage, not the **-r** flag. With **\--hashfile** the
-    winning value *and* the scan configuration are stored, so `--autotune`
-    doubles as one-time setup: a later bare `oans --hashfile=FILE` replays it,
-    using the tuned thread count unless you pass an explicit **\--io-threads**.
-    This is the hardware-measured counterpart to the storage heuristic and the
-    reliable way to tune a NAS or disk array. Sample bounds are controlled by
-    the `DUPEREMOVE_AUTOTUNE_*` variables (see *ENVIRONMENT*).
-
 ## Output and information
 
 **-q**, **\--quiet**
@@ -369,13 +353,6 @@ oans --history --hashfile=foo.hash
 oans --json    --hashfile=foo.hash | jq .reclaimed_total_bytes
 ```
 
-Measure the best thread count for a NAS array (as root), storing it for later
-runs:
-
-```
-sudo oans --autotune --hashfile=media.hash /srv/media
-```
-
 ## Scheduled deduplication
 
 Because the hashfile is self-describing, a scheduled run only needs to name it.
@@ -404,10 +381,6 @@ check the *Not deduped* line, or run with **-v**, to detect them.
 **NO_COLOR**
   ~ If set (to any value), disables colored output, as **\--no-color** does.
 
-**DUPEREMOVE_AUTOTUNE_MAX_FILES**, **DUPEREMOVE_AUTOTUNE_MAX_BYTES**, **DUPEREMOVE_AUTOTUNE_ROUNDS**
-  ~ Bound the sample **\--autotune** reads (file count, total bytes, and number
-    of interleaved measurement rounds).
-
 # FILES
 
 *hashfile*

docs/nas-quickstart.md

@@ -1,9 +1,9 @@
 # oans on a NAS — quick start
 
 A practical, copy-pasteable path to running oans as a scheduled deduplication
-job on a NAS or home server. It uses two features that make this painless: the
-**self-describing hashfile** (a run remembers its own options and paths) and
-**`--autotune`** (measures the fastest thread count for your disks).
+job on a NAS or home server. It leans on the **self-describing hashfile**: a
+run remembers its own options and paths, so everything after the first run
+needs no arguments.
 
 Throughout, replace `/srv/media` with your data directory and `media` with a
 short name for the job.
@@ -48,56 +48,39 @@ sudo make install            # installs the oans binary (+ duperemove symlink)
 sudo make install-systemd    # installs the oans@ timer/service templates
 ```
 
-## Step 2 — Autotune the thread count for your disks
+## Step 2 — The first run (the slow one)
 
-Give the job a name and tune into the hashfile the timer will later use
-(`/var/cache/oans/<name>.hash`). Pass the same `-dr` and path you'll dedupe
-with — autotune records them too, so this one command both measures the
-threads **and** sets up the job:
+Give the job a name — the hashfile the timer will later use is
+`/var/cache/oans/<name>.hash`:
 
 ```sh
 sudo install -d -m 0755 /var/cache/oans
-sudo oans --autotune -dr --hashfile=/var/cache/oans/media.hash /srv/media
-```
-
-Run it **as root** — autotune drops the page cache between trials to get honest
-cold-read numbers, which matters a lot on spinning disks and RAID. It reads only
-a bounded sample (quick relative to a full scan, though on a big cold tree the
-trials still take a few minutes; it prints each one as it goes), then prints a
-throughput-vs-threads table and stores the winning `--io-threads` value plus the
-scan configuration in the hashfile. This is the reliable way to size threads for
-a NAS; without it, oans falls back to a storage-type heuristic whose HDD/RAID
-numbers are only educated guesses.
-
-## Step 3 — The first run (the slow one)
-
-```sh
 sudo oans -dr --hashfile=/var/cache/oans/media.hash /srv/media
 ```
 
 This is the expensive pass: it hashes everything and deduplicates. It
 
-- reuses the thread count autotune stored in Step 2,
-- re-confirms the stored options and paths (already recorded by Step 2), and
+- records the options and paths, so later runs need no arguments,
+- sizes its I/O threads from the detected backing storage (run with **-v** to
+  see what it picked; pass `--io-threads=N` to override), and
 - is safe to interrupt — the kernel does each dedupe atomically and
   byte-verified, so Ctrl+C can only waste work, never corrupt data.
 
-Add `-v` once if you want to see the detected storage and the chosen thread
-count. Check the result:
+Check the result:
 
 ```sh
 oans --stats --hashfile=/var/cache/oans/media.hash
 ```
 
-## Step 4 — Schedule it
+## Step 3 — Schedule it
 
 ```sh
 sudo systemctl enable --now [email protected]
 ```
 
 The name after `@` is the **basename of your hashfile** in `/var/cache/oans/`:
 `oans@media` runs `--hashfile=/var/cache/oans/media.hash`. So match it to the
-hashfile you created in Steps 2–3 — if yours is
+hashfile you created in Step 2 — if yours is
 `/var/cache/oans/data.hash`, enable `[email protected]` instead. (The unit skips
 cleanly until that hashfile exists, so set it up first.)
 
@@ -109,7 +92,7 @@ To change the frequency, override `OnCalendar=`:
 sudo systemctl edit [email protected]     # e.g. OnCalendar=daily
 ```
 
-## Step 5 — Monitor
+## Step 4 — Monitor
 
 ```sh
 systemctl list-timers 'oans@*'                          # when it next runs
@@ -136,7 +119,7 @@ stream to parse.
 ## Notes for NAS users
 
 - **Run as root** so oans can read and re-extent every file in the tree.
-- **Multiple datasets:** repeat Steps 2–4 with different names (`oans@photos`,
+- **Multiple datasets:** repeat Steps 2–3 with different names (`oans@photos`,
   `oans@backups`, …). Each is an independent timer you can schedule separately.
 - **Report-only mode:** set the job up with `-r` instead of `-dr` and the
   scheduled runs will only refresh hashes and report, never change data.