Implement multi-processing support

[ItsIgnacioPortal]

Is your feature request related to a problem? Please describe.
Hacker-scoper already has multi-threading support, but there is still no multi-processing support. Multi-processing support would mean that multiple instances of hacker-scoper would be able to run on the same host without over-writing the local database multiple times. Ideally, other instances of hacker-scoper would detect when there is already an instance trying to update the database, and so it would wait until the update is finished before moving on to processing.

Describe the solution you'd like
AI overview of two possible solutions:

• What platform advisory locks are

  • Kernel-managed locks on a file descriptor (e.g., flock/fcntl on Unix, LockFileEx on Windows).
  • They are process-bound: the kernel releases them automatically if the process exits or closes the handle.
  • They are “advisory”: cooperating processes must attempt the same locking protocol for them to be effective.

• How they work (Unix + Windows, conceptually)

  • Unix: call flock(fd, LOCK_EX) (exclusive) or LOCK_SH (shared). LOCK_NB makes it non-blocking. Locks are per-file-descriptor and released on close() or process exit.
  • Windows: use LockFileEx on a file handle to lock a byte range; locks are released when the handle is closed or process exits.
  • In Go you hold an *os.File open while locked; pass its Fd() to the syscalls. Use non-blocking to implement waiting/backoff behavior.

• Short Go sketches (illustrative, not applied here)

  • Unix (using x/sys/unix):

    f, _ := os.OpenFile(lockPath, os.O_CREATE|os.O_RDWR, 0600)
    defer f.Close()
    err := unix.Flock(int(f.Fd()), unix.LOCK_EX) // blocks until acquired
    // on exit:
    unix.Flock(int(f.Fd()), unix.LOCK_UN)
    

  • Windows (concept): open file handle and call LockFileEx, release with UnlockFileEx or closing handle.
  • Cross-platform: use a small library (see below) to avoid platform syscalls.

• Key differences vs atomic lockfile (create-with-O_EXCL + PID/timestamp)

  • Automatic cleanup: advisory locks are kernel-managed and are automatically released if a process crashes. Atomic lockfiles require TTL/stale detection + manual cleanup (read PID and check liveness or timestamp), because a process crash can leave the lock file behind.
  • Cooperation vs enforcement: both are cooperative — other processes must follow the same protocol. But advisory locks are less error-prone because the kernel enforces release on exit.
  • Portability & complexity: atomic lockfiles are simpler to implement in pure Go with no OS-specific syscalls. Advisory locks require platform-specific code (or a tested library) and attention to NFS/mount semantics.
  • Visibility: an atomic lockfile is a real file visible to anyone; advisory locks are not visible except via try-to-lock behavior. That makes atomic files useful if you want humans/scripts to inspect lock state (PID/timestamp), but also means more potential for stale state.
  • Edge cases: advisory locks can have surprising behavior on some network filesystems (NFS) and between different lock APIs (flock vs fcntl), so test on your deployment environment.

• Pros / Cons (brief)

  • Advisory locks: + no stale-locks on crashes, simpler failure semantics; − requires platform handling, NFS caveats.
  • Atomic lockfile: + extremely portable and easy to implement; − must implement stale detection and cleanup, possible races if deletion/creation logic is brittle.

• Practical recommendation for this project

  • If your instances run on the same host(s) and you can add a small OS-specific file for locking, use advisory locks (or a cross-platform library) because they avoid stale-lock pitfalls.
  • If you want the simplest cross-platform solution without syscalls, use atomic lockfile + PID+timestamp + careful stale-lock detection and short TTL; ensure download writes to a temp file and os.Rename() to the final path atomically.

• Libraries & implementation shortcuts

  • github.com/gofrs/flock — cross-platform file lock abstraction (recommended for simple adoption).
  • github.com/nightlyone/lockfile — PID-file style helper.
  • Or implement with golang.org/x/sys/unix (flock) for Unix and syscall/windows calls on Windows if you prefer no third-party deps.

• Integration notes (what to change in updateFireBountyJSON)

  • Acquire the lock before starting the download. Prefer exercising a blocking acquire with a reasonable timeout or non-blocking + backoff.
  • Download to a temp file, fsync if you want, then os.Rename() to firebountyJSONPath to make the swap atomic.
  • Release the lock (or close the handle) and return. Add signal handling so the file handle is closed on termination.

Implementations where multiple people are using the same database on a NFS is not something that is currently within the scope of the project. But I would be willing to implement it as a sponsored enterprise endeavour. See: https://github.com/sponsors/itsignacioportal

Describe alternatives you've considered

Alternatively to lockfiles, it would also be really handy to have --update and --no-update arguments. If specified, the --update argument would simply update the db and then exit. If the --no-update argument is specified, then the current instance of hacker-scoper would not check for db updates. If both are specified, then the program should quit with exit code 2 (command line syntax error).

Additional context
Currently, multi-processing isn't necessarily "broken". But if multiple instances are launched at the same time and the database is outdated, all instances will attempt to update the database at the same time, likely triggering crashes on all but one instance. For example, this could happen on a system where someone is testing multiple companies at the same time.