duperemove 0.15.2 hangs

[kevincox]

duperemove 0.15.2 (installed from nixpkgs) was running for a while but then hung for a week at 100% CPU usage.

duperemove -rd --cpu-threads=1 --io-threads=1 --dedupe-options=partial --hashfile=/root/duperemove.db /mnt /var/lib/mydir

The last log line was:

Dedupe 1 extents (id: bc6fd1db) with target: (0, 6413089905), "/var/lib/mydir/somelargefile"

I tried attaching an strace and it just printed half of a FUTEX_WAIT and never anything more.

I killed it and pulled the stack traces from the core dump:

Thread 3 (Thread 0x710b27fff6c0 (LWP 607050)):
#0  0x0000710b2852320d in syscall () from /nix/store/57iz36553175g3178pvxjij8z5rcsd4n-glibc-2.42-61/lib/libc.so.6
#1  0x0000710b28ae2457 in g_cond_wait () from /nix/store/jlyahda14aya375lv7k9fsin2zk90nxz-glib-2.88.1/lib/libglib-2.0.so.0
#2  0x0000710b28a720ac in g_async_queue_pop_intern_unlocked () from /nix/store/jlyahda14aya375lv7k9fsin2zk90nxz-glib-2.88.1/lib/libglib-2.0.so.0
#3  0x0000710b28ae2ee4 in g_thread_pool_spawn_thread () from /nix/store/jlyahda14aya375lv7k9fsin2zk90nxz-glib-2.88.1/lib/libglib-2.0.so.0
#4  0x0000710b28ae2c2e in g_thread_proxy () from /nix/store/jlyahda14aya375lv7k9fsin2zk90nxz-glib-2.88.1/lib/libglib-2.0.so.0
#5  0x0000710b2849dd53 in start_thread () from /nix/store/57iz36553175g3178pvxjij8z5rcsd4n-glibc-2.42-61/lib/libc.so.6
#6  0x0000710b2852563c in __clone3 () from /nix/store/57iz36553175g3178pvxjij8z5rcsd4n-glibc-2.42-61/lib/libc.so.6

Thread 2 (Thread 0x710b262fc6c0 (LWP 2475624)):
#0  0x0000710b28520fbd in ioctl () from /nix/store/57iz36553175g3178pvxjij8z5rcsd4n-glibc-2.42-61/lib/libc.so.6
#1  0x00006229574f31cc in dedupe_extents ()
#2  0x00006229574fb9df in dedupe_worker ()
#3  0x0000710b28ae33fa in g_thread_pool_thread_proxy () from /nix/store/jlyahda14aya375lv7k9fsin2zk90nxz-glib-2.88.1/lib/libglib-2.0.so.0
#4  0x0000710b28ae2c2e in g_thread_proxy () from /nix/store/jlyahda14aya375lv7k9fsin2zk90nxz-glib-2.88.1/lib/libglib-2.0.so.0
#5  0x0000710b2849dd53 in start_thread () from /nix/store/57iz36553175g3178pvxjij8z5rcsd4n-glibc-2.42-61/lib/libc.so.6
#6  0x0000710b2852563c in __clone3 () from /nix/store/57iz36553175g3178pvxjij8z5rcsd4n-glibc-2.42-61/lib/libc.so.6

Thread 1 (Thread 0x710b283deb00 (LWP 607042)):
#0  0x0000710b2852320d in syscall () from /nix/store/57iz36553175g3178pvxjij8z5rcsd4n-glibc-2.42-61/lib/libc.so.6
#1  0x0000710b28ae2457 in g_cond_wait () from /nix/store/jlyahda14aya375lv7k9fsin2zk90nxz-glib-2.88.1/lib/libglib-2.0.so.0
#2  0x0000710b28ae3d6e in g_thread_pool_free () from /nix/store/jlyahda14aya375lv7k9fsin2zk90nxz-glib-2.88.1/lib/libglib-2.0.so.0
#3  0x00006229574fc203 in dedupe_results ()
#4  0x00006229574eb3e9 in main ()

[Saroumane]

Hello @JackSlateur @trofi , I assume you had the courage to take over the maintenace of duperemove from @markfasheh , thanks for that !!
It seems there are 13 PR waiting to be reviewed / merged (most from @martinus , some of them maybe fixing hangs / loops). Would you mind taking care of them if you have the time ?
Again, thanks a lot.