feat: add output_realtime to stream task output as produced (#83)#131
Open
esetnik wants to merge 4 commits into
Open
feat: add output_realtime to stream task output as produced (#83)#131esetnik wants to merge 4 commits into
esetnik wants to merge 4 commits into
Conversation
) Adds an opt-in 'output_realtime' config flag (default false). When enabled, each task's output is streamed to the console as it is produced (stdout for standard output, stderr for error output) instead of being buffered and emitted only after the task finishes. This helps long-running tasks (e.g. queue workers) whose logs would otherwise be invisible until completion, which is common when output is collected from the process stdout (Docker/CloudWatch). In realtime mode the successful-output end-of-job emission (per-event log, global log_output, and the console display fallback) is replaced by the live stream to avoid duplicating output. Output is forwarded verbatim (OUTPUT_RAW). The in-memory buffer is still retained, so email_output and the entire error path are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Deep review found the realtime handleOutput early-return suppressed ALL end-of-job sinks, not just the console-duplicating one. For a per-event sendOutputTo() file or a log_output->file target, the realtime stream goes to the console (a different destination), so the file silently received nothing — and start() had already truncated it. The error path also double-printed on failures (live stream + handleError console blob). Make realtime streaming additive: keep every logger-based sink (per-event log files, log_output) and email firing to their own destinations. Suppress only the two end-of-job writes that go directly to the runner's console and would duplicate the live stream: handleOutput's display() fallback and handleError's <error> console blob. If a log destination is itself php://stdout/php://stderr it collides with the live stream and double-prints; that is now a documented, operator-controllable config choice (disable that log) rather than silent data loss. Adds tests: log_output and per-event log files still written under realtime; email still sent; stderr routed to the error stream (new SpyConsoleOutput double); failed-task output not duplicated. Docs/spec updated to match. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…crunzphp#83) The original issue asks for output to reach the LOG FILE as it is produced, not just the console. Resolve the realtime destination with the same precedence the success path uses (per-event sendOutputTo file, else global output_log_file, else console) and stream the raw chunks there live: a file/stream path is opened with fopen('ab') and appended per chunk; the console keeps the stdout/stderr split. The handle is closed once the event is handled. In realtime mode the end-of-job framed record for that destination is replaced by the live raw stream (handleOutput skips the framed emission), so the file receives the output incrementally instead of one buffered record at the end. email_output and the error log are unchanged. Adds unit tests asserting the per-event and global log files receive the raw streamed output (no crunz.INFO framing), and an E2E test proving a sendOutputTo() file is written incrementally (first marker present before the second is produced). Docs/spec/CHANGELOG updated. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The mb_str_functions rule (enforced by .php-cs-fixer.dist.php) maps substr_count to mb_substr_count; mb_substr_count exists since PHP 8.0, so the rule fires on the CI static-analysis job (PHP 8.2). Use mb_substr_count in the EventRunner realtime tests to satisfy it. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #83.
What
Adds an opt-in
output_realtimeconfig flag (defaultfalse). When enabled, each task's output is streamed to its destination as it is produced, instead of being buffered and emitted only after the task process exits.This addresses the long-standing request (originally lavary/crunz#152) to stream output for long-running tasks — e.g. a worker that polls a queue in a loop, whose logs are otherwise invisible until it ends, which is painful when logs are collected from the process' stdout (Docker/CloudWatch).
Behavior
The output is streamed to the same destination it would normally be written to at the end of the job, resolved in the usual precedence:
sendOutputTo()/appendOutputTo();output_log_file, iflog_outputis enabled;stdoutfor standard output,stderrfor errors).So a task logging to a file gets that file written line-by-line as it runs, and a task whose output goes to
stdoutgets it streamed live. The live stream is the raw task output (not the framed[date] crunz.INFO: ...record), and it replaces the end-of-job framed record for that destination (so output is not written twice).email_outputand the error sinks (log_errors→errors_log_file,email_errors) are unaffected.Default is
false, so existing setups are unchanged.Notes / limitations
email_outputcan send the complete output at the end).before()/then()callbacks still goes viaemail_output.output_log_file: php://stdoutplus a separate stdout consumer), output can appear twice on that stream — point the log at a real file or disable it when relying on the live stream.Tests
EventRunnerTest,EventTest): raw streaming + no duplication; per-event and global log files receive the raw streamed output;email_outputstill sent; stderr routed to the error stream; failed-task output not duplicated.RealtimeOutputTest): proves true incremental streaming to both the console and a file (first marker present before the second is produced), plus a buffered-mode control.🤖 Generated with Claude Code