core-timeline: a core dump with the last minute still attached

A tiny native flight recorder whose event ring is mapped into the process address space, survives into the core image, and lets a reader reconstruct the path to SIGSEGV without rr-scale replay.

The Vision

A core dump tells you where a process died. It usually does not tell you how it got there. The last allocator decision, syscall, user marker, config reload, or request id is in a log stream somewhere else, if it was logged at all. If the heap is corrupted or the process dies before flushing, the trail is gone exactly when you need it.

core-timeline makes the last minute part of the crash artifact. A small shared-memory ring is mapped into the process, updated by async-signal-safe writes, periodically synced to a sidecar, and deliberately included in the core dump. The reader opens the core, finds the timeline mapping, and stitches events to the crashed address space: event pointers resolve against the same mappings, thread ids match the dead process, and markers point to code addresses in the dumped binary. The claim is not replay. The claim is address-space-correlated post-mortem context.

The narrow falsifiable claim: given two crash paths that die at the same instruction, core-timeline distinguishes them from the core artifact alone, even when the heap is corrupted and normal logging cannot flush.

The Novel Idea

The core idea is not a ring buffer, not tracing, and not post-mortem debugging. The defensible delta is:

The event ring is inside the core image. It is a mapped region with a magic header that the kernel includes in the dump. A reader does not join a separate log stream by timestamp; it reads the crashed process's own memory image.
Crash-time flushing is best-effort but not load-bearing. The ring is mmap-backed and periodically synced, so a corrupt heap or unsafe signal context does not have to execute complex code at SIGSEGV.
Events carry address-space keys. Return addresses, mmap generation, thread id, and optional pointer snapshots let the reader resolve "this marker came from this function in this exact core", not "a log line near the same time".

This sits between printf logging and rr. It gives enough history to debug the common "how did we reach this impossible state?" failure without recording every instruction.

Prior Art & How We Differ

Core dumps / ELF notes / coredump_filter. Core dumps already preserve mappings and registers. We add a deliberately structured mapping that is useful as a timeline, plus a reader that treats it as part of the core.
rr. rr records enough nondeterminism to replay. core-timeline does not replay anything. It is lower overhead and much less complete.
LTTng / perf / eBPF ring buffers. Excellent external tracing. They are separate streams. core-timeline's differentiator is address-space correlation: the ring is in the crashed image.
Java Flight Recorder / Windows Error Reporting. Mature runtime crash telemetry. They are runtime/platform systems. core-timeline targets native C/C++/Rust services and exposes a tiny C ABI.
Linux pstore/ramoops. Kernel crash logs survive reboot. Different layer: system-level logs, not per-process address-space events.
Wasmer journal. Wasmer records a linear history of WASM process events that can be replayed to a deterministic state (wasmer/docs/journal.md:7-14), captures memory/thread state at snapshot triggers (wasmer/docs/journal.md:100-112), and represents those events as typed JournalEntry variants such as memory updates and thread snapshots (wasmer/lib/journal/src/entry.rs:78-128). Its effector writes events into an active journal (wasmer/lib/wasix/src/journal/effector/save_event.rs:3-24) and flushes snapshots (wasmer/lib/wasix/src/journal/effector/memory_and_snapshot.rs:184-193). core-timeline borrows only the "state changes as an inspectable artifact" philosophy. We do not own a WASM runtime, pause threads, or replay execution.

Artifact Contract

The artifact is valid only if the reader can answer these from the core alone:

Which ring mapping belonged to the crashed process?
Which executable/shared-library mappings were active when each return address was recorded?
Which thread wrote each event?
Whether the final crash slot was present or only the pre-crash history survived.
Whether any slot was torn, overwritten, or skipped.

If the reader needs timestamps from a separate log stream to join evidence back to the core, we lost the novelty claim. The sidecar can improve recovery, but it is never the demo's primary evidence.

Architecture

`libtimeline`

A tiny preloadable/static library:

void tl_mark(uint32_t id, uintptr_t arg);
void tl_mark_ptr(uint32_t id, const void *ptr, uint32_t len_cap);
void tl_thread_name(const char *name);

The hot path is a single atomic fetch-add on write_idx, then fixed-size stores into a ring slot. No malloc, no locks, no formatting.

Mapping Layout

The library creates a memfd or file-backed mmap:

/tmp/core-timeline.<pid>.ring

Header:

magic CTLN
version
slot size
ring length
process start time
build id of executable if available
write index
last synced index
dropped counter
mmap generation counter
process pid namespace id if readable at init
sidecar inode/device for degraded-mode validation
CRC over the immutable header fields

Slot:

sequence number
monotonic timestamp from vDSO/clock cache when safe, or zero in signal path
tid
event kind
event id
return address
arg0/arg1
optional inline bytes for small pointer snapshots
begin/end commit words for torn-slot detection

The mapping is marked dumpable with madvise(MADV_DODUMP). If the process uses coredump_filter, install docs tell the user to include file-backed private/shared mappings. The sidecar path is stored in the header, but the core mapping is the primary artifact.

Slot write protocol:

slot.begin_seq = seq | WRITING
store event fields
release fence
slot.end_seq = seq
slot.begin_seq = seq

The reader accepts a slot only when begin and end match and the sequence is within the retained window. A crash in the middle of tl_mark produces a skipped torn slot, not a fake event.

Async-Signal-Safe Crash Path

The SIGSEGV/SIGABRT handler does only async-signal-safe work:

sigaltstack is installed at init.
Handler writes one final CRASH slot with raw stores and only atomics verified lock-free at init; if not lock-free, crash-slot writing is disabled and the pre-crash ring remains the artifact.
Handler copies siginfo_t fields and ucontext instruction pointer into the slot.
Handler calls write(crash_fd, &small_record, sizeof small_record) to a pre-opened O_APPEND sidecar fd.
The handler is installed with SA_ONSTACK | SA_RESETHAND. After the minimal write it returns to the faulting instruction; the reset disposition produces the real core on the second fault.

No malloc, no printf, no backtrace, no mutex, no symbolization, no file open.

The signal path deliberately does not call clock_gettime, dladdr, backtrace, pthread_mutex_lock, fprintf, open, malloc, or msync. Those are common "helpful crash handler" mistakes. Symbolization is offline only.

mmap + Periodic Sync Fallback

Crash-time flush is not trusted. A background helper thread wakes every 100 ms or every N slots and calls msync(MS_ASYNC) on the ring mapping. For services that cannot tolerate a helper thread, tl_mark can trigger sync every N writes. The sidecar is a recovery fallback if core patterns exclude the mapping, not the main path.

If the heap is corrupted and the handler cannot run safely, the core still contains the already-written ring pages because the mapping is part of the address space. msync matters for sidecar freshness, not for the core-only claim.

There are two durability layers:

Core inclusion layer: MADV_DODUMP plus coredump_filter configuration makes the mapping part of the core image. This is the primary path and does not require msync.
Sidecar freshness layer: periodic msync and the pre-opened fd make recent events recoverable if the core policy excludes the mapping. The reader labels this degraded-sidecar and the demo avoids relying on it.

The M0 demo fails if core-timeline-read --core-only cannot distinguish the paths.

Event Sources

MVP includes:

Manual tl_mark.
Optional LD_PRELOAD malloc/free wrappers storing pointer and size.
Optional syscall markers via wrappers for open, read, write, connect, close.

eBPF syscall ingestion is future work; it risks permissions and is not required for the core claim.

`core-timeline-read`

Reader script/tool:

Opens core ELF.
Scans PT_LOAD segments for CTLN magic.
Reconstructs the ring by sequence number.
Resolves return addresses using the executable and shared-library build ids in the same core.
Prints timeline with thread, event id, source symbol, and pointer snapshots.

If the sidecar exists, the reader can merge it, but the demo must work from the core file alone.

Reader validation steps:

Verify header CRC and supported version.
Verify slot size/ring length are sane before trusting offsets.
Reject mappings whose build id or process start time conflicts with the core's executable mapping.
Build a mapping table from the core's PT_LOAD segments, not from the live host.
Resolve return addresses by (mapping start, file offset, build id) so ASLR does not matter.
Mark torn slots, overwritten sequence ranges, and dropped count explicitly.
Print pointer snapshots as bytes plus the mapped object they pointed into, never as trusted strings by default.

Example reader output:

core-timeline: core-only, ring seq 8112..8175, dropped=0
tid 7124  REQ_PARSE        id=41    at parse_request+0x2a
tid 7124  FREE             ptr=0x55555581a2a0 size=256
tid 7124  BEFORE_DIE       arg=1001  at path_a+0x61
tid 7124  CRASH            sig=SIGSEGV pc=die+0x4 addr=0x0

The Hard Problem & Our Approach

The hard problem is reliable post-mortem evidence from a corrupted process.

Do not do complex work in the signal handler. Backtracing, formatting, opening files, locking, and allocation are all forbidden. The handler writes one fixed record to a pre-opened fd and re-raises.

Do not depend on the handler. The ring is already mapped and periodically synced. The crash path adds the final event if possible; it is not required for the preceding history.

Make the core self-describing. A magic header, version, slot size, and build id make the reader independent of external process state. If the sidecar is missing, the core still contains the ring.

Corrupted heap. Ring storage is separate from malloc-managed heap. tl_mark never allocates. The crash fixture deliberately corrupts heap metadata before dying to prove this.

Thread races. Slots use sequence numbers. A partially written slot is detected by mismatched begin/end sequence and skipped. Dropped/overwritten events are counted, not hidden.

Core policy. Linux can exclude mappings through coredump_filter, systemd-coredump policy, ulimit -c, PR_SET_DUMPABLE, or container settings. M0 includes a preflight: deliberately crash after writing one marker and assert core-timeline-read --core-only sees it. If not, the demo environment is invalid.

Address-space correlation. External tracers can say "thread 7 logged X around 12:01." core-timeline says "the return address in event X belongs to this PT_LOAD mapping in this core, with this build id." That is the difference. The reader never resolves against whatever binary is currently installed unless its build id matches the core.

Allocator corruption. LD_PRELOAD malloc/free wrappers are optional and can be disabled if they destabilize the fixture. Manual tl_mark is enough to prove the core claim. The heap-corruption step exists to prove the ring is outside malloc, not to make malloc wrapping load-bearing.

Control / Data Flow

Process starts with LD_PRELOAD=libtimeline.so or links libtimeline.a.
libtimeline creates the ring mapping, marks it dumpable, opens sidecar fd, installs altstack and signal handler.
Application calls tl_mark(REQUEST_START, id) or wrappers record malloc/free/syscall events.
Each event reserves a slot with atomic fetch-add and writes fixed fields.
Periodic sync flushes dirty pages to backing file.
Process crashes. Handler writes final crash slot if safe and re-raises.
Kernel writes core dump including the ring mapping.
core-timeline-read ./core ./binary finds the ring and prints the final events.

ABI and Build Integration

The C ABI stays intentionally primitive:

#define TL_EVENT_REQUEST_START  0x1001
#define TL_EVENT_CONFIG_RELOAD  0x1002
#define TL_EVENT_BEFORE_DIE     0x1003

void tl_init(const struct tl_options *opts);
void tl_mark(uint32_t id, uintptr_t arg);
void tl_mark2(uint32_t id, uintptr_t arg0, uintptr_t arg1);
void tl_mark_ptr(uint32_t id, const void *ptr, uint32_t len_cap);
void tl_flush_async(void);

tl_init is optional for the preload path; a constructor installs the default ring. Static-link users call it explicitly so services can choose ring size, sidecar path, signal list, and whether to install handlers. Event ids are numbers because string formatting in hot paths is exactly the thing this project avoids. A companion .tlmap file can map ids to names for reader output:

0x1001 REQUEST_START
0x1002 CONFIG_RELOAD
0x1003 BEFORE_DIE

The preload mode is useful for demos, but the honest production path is static or direct dynamic linking. LD_PRELOAD wrappers are too easy to perturb allocator/syscall behavior; manual markers prove the core mechanism without making interception the claim.

Overhead Target

The hot path budget is small enough to measure live:

disabled marker: compiled branch + return, target below 2 ns in a tight loop;
enabled marker uncontended: one atomic fetch-add plus fixed stores, target below 50 ns on a laptop-class CPU;
enabled marker under 8 threads: no locks, dropped/overwritten count allowed, target below 200 ns p95;
helper sync: amortized by interval; never on every marker in default mode.

The benchmark is not the main pitch, but it protects the design from turning into logging with extra ceremony. If tl_mark formats strings, resolves symbols, or allocates, the benchmark fails immediately.

Measured reality (the M5 microbench, honest): disabled ~0.2 ns and uncontended enabled ~2.6 ns crush their targets; the 8-thread p95 does NOT — it measures ~560 ns on the demo container, ~2.8× over the 200 ns target. That is real, not measurement noise (the bench reports an effective_tick_ns ~41 ns and only flags absolute_ns_unreliable when the tick is coarse relative to the number — here it is not). The cause is inherent: every thread does an atomic fetch-add on the single shared write_index, so the cache line bounces between cores under contention. Closing the gap means sharding the counter (per-thread/per-core sub-rings the reader merges by sequence), a real change deferred as out-of-scope. The lock-free, allocation-free, format-free property the budget exists to protect holds at every thread count; the absolute 8-thread p95 target is documented as not-yet-met rather than hidden (see docs/prior-art.md, M5 limitations).

Key Design Decisions & Tradeoffs

Mapped ring over external logs. Slight setup cost, but address-space correlation is the whole point.
Fixed-size events. No arbitrary strings in the hot path. User strings become ids; symbolization happens offline.
Signal handler is minimal. Less pretty final crash metadata, much more trustworthy.
Periodic sync by default. A helper thread is inelegant but removes the need for heroic crash flushing.
Manual markers first. LD_PRELOAD wrappers are useful, but the core demo needs only explicit markers and malloc/free.
Overwrite oldest. Bounded memory and predictable overhead matter more than retaining all history.

MVP / Scope

Delivers:

C library with tl_mark, fixed ring mapping, altstack crash handler, periodic sync.
MADV_DODUMP/core inclusion check.
Reader that scans core PT_LOAD segments for the ring and resolves symbols.
malloc/free wrapper mode.
Two-path crash fixture with corrupted heap.
Overhead benchmark for disabled and enabled markers.

Punts:

eBPF event source.
Rich string formatting.
Cross-platform crash integration.
Perfect behavior if the kernel excludes mappings from core dumps.
Replay.

Demo Environment Preflight

Before the presentation machine is trusted:

ulimit -c unlimited
cat /proc/self/coredump_filter
./scripts/core_timeline_preflight.sh   # builds a one-marker fixture, crashes it, verifies + controls

The preflight (scripts/core_timeline_preflight.sh, real and runnable) writes one marker, crashes, and verifies the marker is recovered from the core via core_timeline_read --core-only. It detects a piped core_pattern (a leading |, e.g. systemd-coredump): in that case the core does not land in cwd, so the script fails loudly with the coredumpctl dump <pid-or-exe> --output core.preflight recovery command rather than silently "passing" — the silent-vanish case is exactly what the preflight exists to catch. If containers suppress dumps, we run the demo in a VM. No debugging session should start by discovering the OS discarded the artifact.

The preflight also runs the negative controls (it exits 0 only if all behave as specified):

mapping included -> reader finds CTLN and recovers the marker (exit 0);
mapping excluded (MADV_DONTDUMP build) -> reader says timeline mapping absent (exit 2);
no library linked -> reader says no CTLN mapping found (exit 1).

(The copied-sidecar-mismatch control lives in the full suite as tests/test_m4_foreign_sidecar.py (exit 6); the preflight focuses on the core-inclusion controls that gate trusting the machine.)

Build Sequence

M0: crash fixture produces a core whose embedded ring distinguishes two same-PC crash paths with the heap corrupted. If this fails, cut the idea.
M1: fixed-size ring and tl_mark.
M2: MADV_DODUMP mapping and core reader.
M3: async-signal-safe handler and altstack.
M4: periodic sync and sidecar fallback.
M5: malloc/free wrappers and overhead benchmark.

Acceptance Gates

Gate A: core-timeline-read --core-only core.a and core.b produce different timelines even though gdb shows the same crashing PC.
Gate B: disabling MADV_DODUMP or excluding the mapping makes the reader fail loudly with timeline mapping absent.
Gate C: corrupting heap metadata before crash does not prevent manual marker recovery.
Gate D: removing the signal handler still preserves pre-crash markers in the core; only the final CRASH slot disappears.
Gate E: a deliberately torn slot is skipped and reported, not rendered as a valid event.

Differential / Adversarial Test

Fixture:

void die(void) { *(volatile int *)0 = 1; }

void path_a(void) {
    tl_mark(REQ_PARSE, 41);
    free(global);
    corrupt_heap_metadata();
    tl_mark(BEFORE_DIE, 1001);
    die();
}

void path_b(void) {
    tl_mark(CONFIG_RELOAD, 7);
    malloc(128);
    corrupt_heap_metadata();
    tl_mark(BEFORE_DIE, 2002);
    die();
}

Both paths crash at the same instruction inside die. Normal log flushing is disabled and heap metadata is corrupted. The reader must distinguish A from B from the core artifact alone:

A shows REQ_PARSE -> free -> BEFORE_DIE(1001) -> CRASH.
B shows CONFIG_RELOAD -> malloc -> BEFORE_DIE(2002) -> CRASH.

Negative control: remove MADV_DODUMP or exclude the mapping through coredump_filter; the reader must fail loudly with timeline mapping absent, not silently fall back to stale sidecar data.

Second negative control: build the same fixture without linking/preloading libtimeline. The reader must return no CTLN mapping found; it must not mistake arbitrary heap bytes for a ring.

Third negative control: copy the sidecar from path A next to core B. --core-only ignores it; normal mode refuses the merge (exit 6) unless the sidecar's identity matches a fact the core itself attests — the crashed pid from the core's NT_PRPSINFO and the executable build-id — not a timestamp. (Implementation note: the spec originally named inode here, but the reader treats inode/dev as advisory only, never a refusal trigger: the demo's FUSE bind mount drifts a genuine file's inode across opens, so refusing on it would false-positive a process's own sidecar. Two runs of the fixture produce two pids, so the copied-sidecar attack is caught on pid; build-id catches a different binary. See docs/prior-art.md M4.) This prevents the exact "separate log stream joined by vibes" dismissal.

The Demo

Compile crash_demo with two flags: --path=a and --path=b.
Run both; both produce SIGSEGV at the same PC.
Open each core with gdb briefly: same crashing instruction.
Run core-timeline-read core.a: it prints the A path.
Run core-timeline-read core.b: it prints the B path.
Show strings core | grep CTLN or reader output proving the ring came from the core mapping.
Corrupt heap before crash and repeat; evidence still exists.

The line for the panel: "This is not a log near the crash. It is memory from the crashed process, resolved against the same address space as the core."

Honest Limitations

If core dumps are disabled or coredump_filter excludes the ring mapping, the primary artifact is absent. The tool detects this and can use the sidecar only as a degraded mode.
A piped core_pattern (systemd-coredump and friends, where core_pattern starts with |) hands the core to a handler instead of writing it to cwd; in a container/namespace it can land somewhere the workflow does not see. core-timeline is not in that pipe, so it cannot itself warn at crash time — but the preflight (scripts/core_timeline_preflight.sh) detects a piped pattern and fails loudly with the coredumpctl dump recovery command, so the gap is caught before the demo rather than mid-incident. Once the core is recovered, the embedded ring is intact (it is mapped memory); if policy excluded the mapping, Gate B still fires (timeline mapping absent).
The signal handler is best-effort. SIGKILL, immediate kernel death, or stack/altstack corruption can skip the final crash slot.
Ring overwrite loses old events by design.
Pointer snapshots are capped and can fault if the pointer is invalid; MVP copies only when the caller explicitly uses tl_mark_ptr outside signal context.
Symbolization depends on debug info/build ids.
When the core carries no build-id (e.g. a binary built --build-id=none), the reader cannot verify that a supplied binary is the crashed image, so by default it refuses to symbolize (raw addresses + a loud note) rather than risk resolving against the wrong file. --unverified-symbols opts into symbolizing anyway, but the result is tagged unverified-no-core-buildid and never presented as verified fact. The raw core-sourced timeline (event ids, return addresses, thread) is always recovered regardless.
The ring's write_index (and dropped) counter is updated by the hot path via atomic fetch-add, so it lives outside the header CRC region (the CRC covers only the immutable prefix). A torn/corrupted write_index therefore passes the CRC. The reader does not trust it blindly: it independently finds the highest validly-committed sequence in the physical slots and cross-checks; a counter that disagrees is flagged loudly (counter_inconsistent), the timeline is rendered from the slots actually committed (so a deflated counter never silently drops evidence), and the scan is clamped to what the dumped segment can hold (so an inflated counter cannot drive an out-of-memory render). This is integrity-by-cross-check, not by CRC — the CRC structurally cannot cover a field the hot path mutates. The boundary value write_index == 0 is handled the same way: it is taken as authoritative "empty" only when no real committed slot exists, where a never-written slot is distinguished from a genuine seq 0 by its all-zero head (a real tl_mark always stamps a non-zero tid, kind, and return address) — so a counter torn to 0 on a populated ring (even a single-event one) is flagged and recovered, never silently dropped, while a genuinely empty ring still renders no phantom event.
This is not rr. It cannot prove what would have happened next.

Why Linus Respects It

It draws a hard line around the claim. No replay, no magic crash AI, no pretending signal handlers are safe places to do work. The artifact is just a mapped ring with fixed records, included in the core dump, read back as part of the dead process's address space. The make-or-break test is nasty: two paths, same crashing instruction, corrupted heap, no normal logs, distinguishable from the core alone. That is a small mechanism with a real debugging payoff.

FilesExpand file tree

spec.md

Latest commit

History