Expose build steps and per-step output to API consumers#276
Conversation
Add an optional OnStep callback to Builder that makes builds
observable and controllable programmatically, e.g. for tools that
embed xcaddy and want to show live, per-stage build progress. The
zero value changes nothing: with OnStep nil, output goes to the
standard log package and os.Stdout/os.Stderr exactly as before, and
the CLI is unaffected.
The build is modeled as a sequence of named steps (create_environment,
initialize_module, pin_versions, windows_resources, tidy_module,
compile, cleanup). For each step, the callback receives a StepEvent
whose Output is an io.Reader over everything the step produces:
xcaddy's own log lines as well as the stdout and stderr of the
underlying go commands. The callback runs in its own goroutine,
concurrently with its step, so a consumer needs no goroutine of its
own -- it simply reads Output (e.g. with a bufio.Scanner) until EOF,
which arrives when the step ends:
OnStep: func(e *xcaddy.StepEvent) error {
scanner := bufio.NewScanner(e.Output)
for scanner.Scan() {
showProgress(e.Step, scanner.Text())
}
return scanner.Err()
}
Output is backed by a builder-owned buffered pipe whose write side
never blocks, so a slow or absent consumer can never stall or
deadlock the build; output is readable the moment it is produced,
the builder closes the write side at the step boundary, and unread
output is discarded. Callbacks never overlap: the next step does not
begin until the previous step's callback has returned, and Build
does not return until the final callback has.
Returning a non-nil error from the callback aborts the build at the
end of the step, and the error is returned from Build wrapped with
the step name; aborting never leaks the temporary build folder.
Errors from the deferred cleanup path (environment.Close) are now
joined into Build's returned error instead of being discarded.
Two additions for consumers that publish build logs (world-readable live progress, CI services, multi-tenant builders): Builder.Env, if non-nil, is used as the entire environment of the underlying go commands instead of inheriting the process's. This enables hermetic, credential-free builds -- the strongest guarantee for publishable logs: if the build never had access to a credential (GOPROXY userinfo, git config, .netrc, cloud keys), no error message or verbose output can leak one. Builder.Secrets and Builder.RedactCredentials add a masking backstop between the build's output and StepEvent.Output. Secrets performs exact-value redaction; RedactCredentials masks common credential shapes (userinfo in URLs, GitHub/GitLab/Slack tokens, AWS access key IDs, and PEM private key blocks, including multi-line bodies). The redactor is line-buffered so a secret can never slip through by straddling a write boundary, and it flushes any unterminated final line, masked, when the step ends.
|
Thanks, AI seemed to do a fairly thorough job here. Do you think we should also redact variables found in ENV? Not sure if really necessary but if we'd rather be conservative since the logs would become public-facing... (Another option is to simply do a callback when certain milestones are reached, like go.mod constructed, modules downloaded/updated, and build started/finished, rather than full logs/output.) |
|
my thought is that redaction or even full exclusion of logs could be done on the portal or buildworker level, while xcaddy itself should probably show everything. the callback already occurs when certain milestones are reached, you could create more by just parsing log lines within it, which again i think would be better done at the consumer level |
|
Ok. So do we still want to merge this or should we simplify it here and let redactions happen higher up? |
|
I'm happy to say we should merge this if you are on board with the idea that redactions should be done at a higher level (portal, probably) @mholt |
|
Alright thanks! |
|
After merging, there's a test failing, not sure if it's a red herring: |
2 participants
Motivation
Programs that embed xcaddy as a library currently have no way to observe a build: all progress goes to the global
logpackage andos.Stdout/os.Stderr, hardwired inenvironment.newCommand. This PR makes builds observable and controllable — e.g. to show live, per-stage progress in a UI — without changing any existing behavior: everything is additive, the zero value ofBuilderbehaves exactly as before, and the CLI is untouched.API
One new callback on
Builder, plus a step vocabulary:create_environment→initialize_module→pin_versions→ (windows_resources) →tidy_module→compile→cleanup.StepEvent.Outputis anio.Readerover everything the step produces: xcaddy's own log lines and the stdout/stderr of the underlyinggocommands. It's backed by a builder-owned buffered pipe whose write side never blocks, so a slow or absent consumer can never stall or deadlock the build. The builder closes the write side at the step boundary, so reading until EOF is the only lifecycle a consumer needs.Buildjoins the last one before returning), so consumers need no goroutines, locks, orWaitGroups of their own.errors.Is/errors.As); aborting never leaks the temp folder. As a related fix, errors from the deferredenvironment.Closeare now joined intoBuild's returned error instead of being discarded.Safe-to-publish logs (second commit)
For consumers that publish build logs (live progress pages, CI services, multi-tenant builders):
Builder.Env []string— hermetic builds: when non-nil, thegocommands receive exactly this environment instead of inheriting the process's. If the build never had access to a credential (GOPROXY userinfo, git config,.netrc, cloud keys), no error message can leak one.Builder.Secrets []string— exact-value redaction: occurrences are replaced with[REDACTED]before reachingStepEvent.Output. The redactor is line-buffered, so a secret can't slip through by straddling a write boundary.Builder.RedactCredentials bool— best-effort masking of common credential shapes: userinfo in URLs, GitHub/GitLab/Slack token formats, AWS access key IDs, and PEM private key blocks (whole block, stateful across lines).Testing
New tests cover: per-step output containment, live line-scanning against a real build, abort semantics with custom errors, callback sequencing guarantees (no overlap, joins at boundaries), unread-output discard, exec's concurrent stdout/stderr drains under
-race, hermetic-env builds, and redaction (including boundary-straddling secrets, end to end). Network-dependent tests are skipped under-short(how CI runs); several integration tests run real partial builds that abort beforepin_versions, so they exerciseBuildend to end with only localgocommands.go vet,gofmt, andgo test -short -race ./...are clean; the full networked suite passes as well.