Refactor with matcher registry and expanded file type detection#10
Merged
Conversation
…to stderr rather than stdout.
… ovpn, eml, python script, bash script, powershell script.
…ased on categories of files.
…ection; based on tests.
DER PKCS#7 Signed Data Detects DER sequence containing signedData OID (1.2.840.113549.1.7.2) Description: DER Encoded PKCS#7 Signed Data CRDA wireless regulatory database Detects RGDB header Description: CRDA wireless regulatory database file COFF object files Detects i386/x86-64 COFF object header shape Descriptions: Intel i386 COFF object file x86-64 COFF object file
OpenType font (OTF) Signature: OTTO Description: OpenType font data MIME: font/otf Embedded OpenType (EOT) Signature: EOT magic LP at offset 34 + valid version field Description: Embedded OpenType font MIME: vnd.ms-fontobject G-IR Typelib
Roslyn portable PDB v1 Signature: BSJB + version text containing PDB v1.0 Description: Microsoft Roslyn C# debugging symbols version 1.0 magic.mgc Signature: libmagic compiled header (1C 04 1E F1 and endian variant) Description: magic binary file for file(1) cmd ISO-8859/non-UTF text fallback
MIME refactor: - Add mime field to fileMatcher struct; detection pipeline returns (desc, mime) so MIME is co-located with each matcher rather than looked up by fragile string matching in a 350-line mimeForDescription switch - Replace mimeForDescription with a lean dynamicMIME() covering only the matchers whose describe() outputs vary (zip sub-types, ar, tar, 3gpp, pem, text, xml); all other matchers carry a static mime field - Correct several previously-wrong MIMEs: HTML now text/html, JSON application/json, FLAC audio/flac, MP3 audio/mpeg, WASM application/wasm, RTF application/rtf, WOFF/WOFF2 font/woff*, etc. Remove activeFile global: - Thread *os.File through fileMatcher.match signature so readTail and tail-checking helpers (DMG, Parquet, Feather, TGA) receive the file directly; eliminates package-level state and makes detection goroutine-safe New binary matchers: - AIFF/AIFC audio (audio/aiff) - AAC/ADTS audio — 0xFF 0xF1/0xF9 sync word (audio/aac) - PostScript and Encapsulated PostScript — %!PS prefix with %%BoundingBox detection for EPS (application/postscript) New text subtypes: - Go source (package + func/import heuristic) - Rust source (fn/use std::/let mut/impl/trait/derive heuristic) - Java source (public class/interface + import java. heuristic) - Dockerfile (FROM + directive count heuristic) - Makefile (target: + tab-indented recipe heuristic) - TOML configuration ([[array-of-tables]] and typed-value heuristic) - Shell shebang gaps: env bash, zsh, fish - Node.js shebang: #!/usr/bin/env node CLI: accept multiple FILE arguments / glob patterns (fil *.go *.md) go.mod: bump go directive from 1.20 to 1.24 to match CI Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Add several genuinely PS-specific signals that catch cmdlet-only scripts
(no function/param block, variables mid-line rather than at line start):
Strong signals (+2 each):
- [CmdletBinding()] and [Parameter(] — PS-specific attributes
- $PSScriptRoot / $PSCommandPath / $PSVersionTable — PS automatic vars
- $env: — colon-in-variable-name is unique to PS environment var syntax
- -ErrorAction / -WhatIf — PS common parameters absent in other languages
Moderate signals (+1 each):
- begin{}/process{}/end{} advanced-function block keywords
- $_ / $true / $false / $null — pipeline variable and PS literals
- -join / -split (space-bounded) — PS binary operators
- @{} hashtable and @() array subexpression — PS-specific literals
- Expanded Verb-Noun coverage: remove-, stop-, start-, invoke-, test-,
add-, where-object, foreach-object, select-object, sort-object,
import-, out-, write-verbose, write-error
Fix test data to use generic PS patterns rather than content copied
from a specific real-world script.
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Binary matchers: - CPIO archive — SVR4 no-CRC (070701), SVR4 CRC (070702), portable ASCII (070707), old binary (0xC771/0x71C7) (application/x-cpio) - LHA/LZH archive — -lhX- header magic (application/x-lzh-compressed) - Squashfs filesystem — sqsh/hsqs/sqlz/qshs magic (application/x-squashfs) - zlib compressed data — 0x78 CMF byte + (CMF*256+FLG)%31==0 checksum (application/zlib); covers 0x789C, 0x78DA, 0x7801, 0x785E - LUKS encrypted volume — LUKS\xBA\xBE magic (application/x-luks) - Device Tree Blob — 0xD00DFEED magic (application/x-dtb) - Android boot image — ANDROID! magic (application/vnd.android.boot-image) - PGP — ASCII-armored (all -----BEGIN PGP X----- variants with specific descriptions) and binary (0x99 old-format public-key packet); dynamic MIME: application/pgp-keys, application/pgp-encrypted, application/pgp-signature Text subtypes: - SQL script — DML (SELECT/INSERT/UPDATE/DELETE) and DDL (CREATE TABLE etc.) multi-word patterns to avoid English-prose false positives - HCL/Terraform configuration — resource/terraform/provider/variable blocks - Lua script — ~= not-equal, .. concat, ipairs/pairs/pcall, local keyword - R script — <- arrow assignment + library/data.frame/ggplot/function - Protocol Buffers source — syntax="proto" (immediate) or message/service/rpc - CMake script — cmake_minimum_required (immediate) or add_executable etc. - Assembly source — GAS (.section/.global/.byte) and NASM (section .text/db) directives, x86 register names, tab-indented instruction mnemonics Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Text subtype quality: YAML — fix list-heavy false negatives: - Track sequence items (- ) separately as listLines alongside keyValLines - Pure-list documents with --- marker (e.g. GitHub Actions matrices) now detected via: startMarker && listLines >= 3 && structuredRatio >= 0.5 - Mixed mapping+sequence files detected via combined structured ratio - Recognise ... document-end marker alongside --- - Extract isYAMLKeyValueLine() helper to deduplicate logic TypeScript — broaden signal set: - Add readonly, declare, import type, namespace as +1 signals - Add generic type parameters <T>, <T, , <T extends as +1 signal These catch .d.ts ambient declarations and generic utility types that previously slipped through as JavaScript Properties / env detection (new subtypes): - environment variable file: ALL_CAPS_KEY=value lines, no sections; strict — every non-comment line must be a valid uppercase env var - Java properties file: dot.separated.key=value style; requires dot keys in at least half the lines, no section headers Binary matchers: ext2/3/4 filesystem images: - Magic \x53\xEF at superblock offset 1080 (within our 36KB read window) - Distinguishes ext2 / ext3 (HAS_JOURNAL compat flag) / ext4 (EXTENTS or 64BIT incompat flag) by reading feature flags at offsets 1116/1120 GUID Partition Table disk image: - EFI PART magic at byte offset 512 (LBA 1) - Registered before matcherDosMbrBootSector so GPT disks with a protective MBR are not misclassified as plain MBR images U-Boot legacy image: - 0x27051956 big-endian magic at offset 0 ARJ archive: - 0x60EA magic with basic-header-size sanity check (bytes 2-3, 26-2600) to reduce false positives from random binary data Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
…sion) Matches the richer output the real file command provides: Images: - PNG: width x height, bit depth, color type (RGB/RGBA/grayscale/etc.), interlace from IHDR chunk (bytes 16–28) - GIF: version (87a/89a) always; width x height when non-zero (bytes 6–9 LE) - JPEG: width x height by scanning to first SOF0/SOF2 marker; flags progressive - BMP: width x height, bits-per-pixel from DIB header (bytes 18–29) - WebP: distinguishes VP8 (lossy + frame dimensions), VP8L (lossless), VP8X (extended/animated + canvas dimensions from VP8X chunk) Audio: - WAV: sample rate, mono/stereo/N-channel, bits/sample, format name (PCM/IEEE float/a-law/mu-law/extensible) from fmt chunk (bytes 20–35) - FLAC: sample rate, channels, bits/sample from bit-packed STREAMINFO block; uses integer arithmetic to avoid importing math - AIFF/AIFC: sample rate, channels, bits/sample from COMM chunk; decodes 80-bit IEEE 754 extended-precision float for sample rate without math import (parse80BitExtended added to matchers_helpers.go) Documents / data: - PDF: version string from %PDF-X.Y header literal - SQLite: page size from bytes 16-17 big-endian (value 1 → 65536) ELF: - Dynamically linked ELF now reports interpreter path from PT_INTERP segment, e.g. "(interpreter /lib64/ld-linux-x86-64.so.2)"; falls back gracefully when segment data is beyond the read buffer ZIP: - Generic Zip fallthrough now reports entry count from central directory: "Zip archive data, 12 files"; dynamicMIME updated to use HasPrefix Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
Member
|
Good work @presack! Thanks for the PR and test data! I'll release a new version now. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refactored code to use a matcher registry. Added dozens of new file types covering most major media and document types, as well as plain text detection types for many programming and scripting languages. Added additional detail in output for certain file types to more closely match the file command's. Code development was heavily AI assisted.