Stats vacuum command

[amadeovezz]

TODO

• Encapsulate ledger stats and vacuum so there is a consistent code path for api and metis_commands
• Add ledger stats and vaccum to commands
• Add blacklisted column to data block (previously deleted)
• Use deleted column in vacuuming logic
• Add event_meta to ledger to track interesting events (currently tracking upload times)
• Add RESTORE_DATABLOCK event to track datablocks that have been deleted, but then later re-
• Add lifecycle of datablocks/ledgers to README so we have a source of truth document
• Add logic to deal with when a datablock is removed (see below)
• Lots of specs

OTHER

• Add web-down, metis-clear to Makefile-mac

DATABLOCK REMOVE LOGIC

Problem:

When a data block is vacuumed, it's marked as removed: true. If someone later uploads a file with the same content, the compute_hash! method would find the removed block and try to reuse it, causing files to point to a "removed" block.

Solution

If we find a removed datablock during hash creation, set it back to true.

MANUAL CHECKS

Backfill

Check data_block_ledger, data_blocks as you move through:

• Single project single file delete

  • LEDGER OFF. Add 3 files in one project (all different). Run checksum. Delete one file. LEDGER ON. Run add link command. Run add unlink command. Make sure file that was deleted can be reported as vacuumed.

• Single project multi file delete block

  • LEDGER OFF. Add 3 files in one project (2 same content, 1 different). Run checksum. Delete one of the file with the same content. LEDGER ON. Run add link command. Run add unlink command. Make sure file that could not be deleted.

• Multi project delete

  • LEDGER OFF. Add 1 files to Metis in one project and another file in the different project. Run checksum. Delete 1 file in both projects. LEDGER ON. Run add link command. Run add unlink command. Run Vacuum. That file should be able to be deleted.

• Multi project block

  • LEDGER OFF. Add 3 files in one project. Add the same 3 in another. Run checksum. Delete one file. LEDGER OFF. Run add link command. Run add unlink command. Deleted file should be allowed to be vacuumed

• No temp files delete

  • LEDGER OFF. Add 2 files to Metis in one project. Delete 2 files. LEDGER ON. Run add link command. Run add unlink command. Run Vacuum. No files should be deleted

[amadeovezz]

@graft

here is all our vacuum eligibility in english - would be good to have ya review and make sure we havent missed anything

[graft]

can we move this stuff to a decision document? seems more appropriate there than in this README...