Separate iDRAC telemetry into distinct Kafka topics: logs and metrics

[abhishek-sa1]

Problem Statement

Omnia currently publishes all iDRAC telemetry to a single Kafka topic (idrac). This mixes structured metrics and unstructured logs in the same stream, which complicates downstream processing:

• Consumers must parse and filter mixed payloads to extract only metrics or only logs
• Metrics pipelines ingest log messages (noise) and log pipelines ingest metric messages (noise)
• No clean separation for retention, partitioning, or access control at the data type level

Current Behavior

• One Kafka topic: idrac
• iDRAC telemetry producer pushes both metrics and logs to the same topic
• Consumers receive a mixed stream and must filter by message type

Desired Behavior

• Two Kafka topics for iDRAC:
  • idrac-metrics for structured metric data
  • idrac-logs for log/trace/event data
• Producer routes messages by type to the appropriate topic
• Consumers can subscribe to the relevant topic only