Vector logging#1497
Draft
BenGalewsky wants to merge 4 commits into
Draft
Conversation
Add an optional Vector sidecar to the ServiceX app pod that receives application logs over a localhost TCP socket and writes them to a Postgres log_messages table. App side (servicex_app/__init__.py): a VectorFormatter emits JSON with stable, column-matching keys, and the socket send runs on a QueueListener background thread (app.logger -> QueueHandler -> listener -> TCPLogstashHandler) so logging never blocks the app. Enabled via VECTOR_HOST/VECTOR_PORT; the existing logstash and stdout handlers are untouched. Schema: new LogMessage model plus Alembic migration v1_8_4 creating log_messages. The id is a UUID string minted by Vector (not a serial) so the sink's json_populate_recordset INSERT never has to populate an auto-generated primary key. Helm: logging.vector.* values (default disabled), a vector-configmap.yaml (socket source -> remap add_id -> postgres sink), and a sidecar container in the app deployment. Vector's DB credentials are sourced identically to the app's SQLALCHEMY_DATABASE_URI (same secret/inline password). Requires postgres.enabled. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Pull request_id and dataset_id out of the Vector log `extra` JSON blob into dedicated nullable columns on the log_messages table so they can be filtered and joined directly. VectorFormatter now hoists both keys to the top level of the event (removing them from `extra`), and migration 1.8.4 plus the LogMessage model gain the matching columns. Assisted-by: ClaudeCode:claude-opus-4.8
…st page Show log_messages rows for a transformation request at the bottom of its detail page, joined on the request's request_id UUID. Supports filtering by log level and paginates (50/page, newest first) so requests with hundreds of messages stay usable. Assisted-by: ClaudeCode:claude-opus-4.8
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #1497 +/- ##
===========================================
- Coverage 89.20% 88.89% -0.31%
===========================================
Files 103 103
Lines 3732 3773 +41
Branches 460 463 +3
===========================================
+ Hits 3329 3354 +25
- Misses 326 341 +15
- Partials 77 78 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
for more information, see https://pre-commit.ci
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces a new application logging pipeline using a Vector sidecar container, which ships structured application logs directly to a Postgres table. It includes all the necessary Helm chart, application, and database changes, and adds a paginated log viewer to the transformation request page. The most important changes are summarized below.
Vector Sidecar Logging Pipeline
logging.vector.enabled) that receives logs over a localhost TCP socket and writes them to a new Postgreslog_messagestable. Configurable via Helm values for image, port, batching, and resource limits. (helm/servicex/templates/app/deployment.yaml,helm/servicex/values.yaml,docs/deployment/reference.md, [1] [2] [3] [4] [5] [6]vector-configmap.yamlfor the Vector sidecar configuration, including source, transform (adds UUID), and Postgres sink details. (helm/servicex/templates/app/vector-configmap.yaml, helm/servicex/templates/app/vector-configmap.yamlR1-R36)log_messagestable, and a corresponding SQLAlchemy model for schema management. (servicex_app/migrations/versions/1.8.4.py,servicex_app/servicex_app/models.py, [1] [2]Application Logging Integration
VectorFormatterto serialize log records as JSON matching thelog_messagesschema. Application logs are shipped to the Vector sidecar via a non-blockingQueueHandlerandQueueListenerthread, ensuring logging does not block the app. (servicex_app/servicex_app/__init__.py, [1] [2] [3]User Interface Enhancements
log_messagestable, with level-based filtering and Bootstrap styling. (servicex_app/servicex_app/templates/transformation_request.html,servicex_app/servicex_app/web/transformation_request.py, [1] [2] [3] [4]Testing
servicex_app/servicex_app_test/web/test_transformation_request.py, [1] [2]