Use ATEX file upload appending for line-by-line log streaming#633
Open
matusmarhefka wants to merge 1 commit into
Open
Use ATEX file upload appending for line-by-line log streaming#633matusmarhefka wants to merge 1 commit into
matusmarhefka wants to merge 1 commit into
Conversation
Add atex_upload_log_data() to lib/results.py, which streams raw data
to a named ATEX log file using partial result appending (same filename
in multiple partial=True calls causes ATEX to append to the file).
This allows ansible-playbook, oscap and virt-install output to be
uploaded to ATEX incrementally as it is produced, providing crash-safety
(if the test dies mid-execution, ATEX already has all output up to that
point) and keeping output.txt smaller.
The ATEX protocol is consolidated into shared helpers: _atex_write_all()
retries partial writes and EINTR, and _atex_send() sends the JSON header
then either inline data (per-line streaming) or log files read in 64 KiB
chunks. The control fd is cached at module level to avoid repeated env
var lookups.
Changes to library modules:
- lib/results.py: add _atex_write_all(), _atex_send(), _get_atex_fd()
cache, and atex_upload_log_data(); modify add_log() and report_atex()
to skip re-uploading logs already streamed via _streamed_atex_logs
- lib/ansible.py: report_from_output() now streams to ATEX when to_file
is set, using the file's basename as the ATEX log name
- lib/oscap.py: add to_file parameter to report_from_verbose(), writing
all oscap output to a file (and streaming to ATEX) instead of stdout;
keep rules_from_verbose() as a pure parsing generator
- lib/virt.py: stream virt-install output to ATEX as virt-install.log
Changes to tests:
- All oscap.report_from_verbose() callers now pass to_file='oscap.log'
and call results.add_log('oscap.log') to submit the file for TMT
Resolves: RHSecurityCompliance#521
comps
reviewed
Jun 15, 2026
comps
left a comment
comps
left a comment
Contributor
There was a problem hiding this comment.
I glanced a bit through the code, provided a few comments inline.
Comment on lines
+54
to
+55
| when ATEX is available, the output is streamed to ATEX under the file's | ||
| basename. |
Contributor
There was a problem hiding this comment.
Why basename? ATEX supports full sub-paths in file names, so if somebody uses to_file='foo/bar.log' it should AFAIK work just fine.
Comment on lines
+242
to
+247
| for line in lines: | ||
| if atex_name: | ||
| results.atex_upload_log_data(atex_name, f'{line}\n') | ||
| out_file.write(f'{line}\n') | ||
| out_file.flush() | ||
| yield line |
Contributor
There was a problem hiding this comment.
Why do we write to console here via tee(), but not for ansible (above) where the writing-to-stdout was removed completely?
Comment on lines
+102
to
+109
| _atex_fd = None | ||
|
|
||
|
|
||
| def _get_atex_fd(): | ||
| global _atex_fd | ||
| if _atex_fd is None: | ||
| _atex_fd = int(os.environ['ATEX_TEST_CONTROL']) | ||
| return _atex_fd |
Contributor
There was a problem hiding this comment.
Why the caching? Is getting a variable from os.environ and converting it to an integer an expensive operation?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add
atex_upload_log_data()tolib/results.py, which streams raw data to a named ATEX log file using partial result appending (same filename in multiplepartial=Truecalls causes ATEX to append to the file).This allows
ansible-playbook,oscapandvirt-installoutput to be uploaded to ATEX incrementally as it is produced, providing crash-safety (if the test dies mid-execution, ATEX already has all output up to that point) and keepingoutput.txtsmaller.The ATEX protocol is consolidated into shared helpers:
_atex_write_all()retries partial writes andEINTR, and_atex_send()sends the JSON header then either inline data (per-line streaming) or log files read in 64 KiB chunks. The control fd is cached at module level to avoid repeated env var lookups.Changes to library modules:
lib/results.py: add_atex_write_all(),_atex_send(),_get_atex_fd()cache, andatex_upload_log_data(); modifyadd_log()andreport_atex()to skip re-uploading logs already streamed via_streamed_atex_logslib/ansible.py:report_from_output()now streams to ATEX whento_fileis set, using the file's basename as the ATEX log namelib/oscap.py: addto_fileparameter toreport_from_verbose(), writing alloscapoutput to a file (and streaming to ATEX) instead of stdout; keeprules_from_verbose()as a pure parsing generatorlib/virt.py: streamvirt-installoutput to ATEX asvirt-install.logChanges to tests:
oscap.report_from_verbose()callers now passto_file='oscap.log'and callresults.add_log('oscap.log')to submit the file for TMTResolves: #521