Deduplication of in-use executables (Error 26: Text file busy while opening Skipping dedupe)

[tim-seoss]

Hi,

Firstly thanks for writing duperemove - it's a great tool!

I use it along with "incus" container manager to deduplicate the filesystems of several ("OS-level") Linux containers, most of which are running the same Debian OS distributions.

I run duperemove weekly. Typically automated OS security updates replace binaries and then duperemove later de-duplicates the newly installed files.

As expected, executable files which are in-use (i.e. copies of the binaries are running) fail to de-dupe:

Error 26: Text file busy while opening "/srv/incus/containers/deb12test3/rootfs/usr/sbin/mariadbd" (write=1)
/srv/incus/containers/deb12test3/rootfs/usr/sbin/mariadbd: Skipping dedupe.

A couple of possible workarounds:

1. duperemove could produce a list of busy files for later processing. This list could then be deduplicated again rapidly at a time when the binaries are not in use e.g. at boot time before the containers have been started.
2. Make a reflink copy instead, and then use file renaming to atomically replace (e.g. renameat2 with RENAME_EXCHANGE) the binary before unlinking it (if the file contents are still the same). This has a few potential races (e.g. would be unreliable if any writeable file descriptors are still open on the original binary so might not be acceptable without additional OS support).

[Zygo]

As expected, executable files which are in-use (i.e. copies of the binaries are running) fail to de-dupe:

That is not expected. Dedupe of running executables is an explicitly supported use case for the dedupe ioctl.

The correct fix is:

3. Open the dedupe target file with O_RDONLY, then dedupe normally.

and duperemove has that change since v0.13 (78e2727). The same commit removes the error message above from the code.