Skip to content

feat(Messaging): Add info about Kafka #844

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
srp3rd wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat(Messaging): Add info about Kafka #844

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 27 additions & 3 deletions docs/operator/monitoring.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -182,7 +182,29 @@ Different options are shown in figure and described below.
password =
```

2. Direct Delivery
2. Kafka Messages

Hermes can deliver events to a Kafka topic, which can be consumed by a Kafka client.

Config for this is described below.
```cfg
[hermes]
# List of services Hermes should send messages to.
services_list = kafka

[messaging-hermes-kafka]
# Kafka options
# Non-SSL port (used is use_ssl=False
nonssl_port = 9092
# Use SSL for Kafka connection
use_ssl = False
# Broker host name or DNS alias
brokers = kafka
# Destination topic
topic = hermestopic
```

3. Direct Delivery

These options send events directly to storage or alerting systems, bypassing queues.
Hermes can write events straight to Elasticsearch, OpenSearch, or InfluxDB. In addition can also deliver events via email which supports custom SMTP servers, credentials, and SSL/TLS.
Expand All @@ -197,8 +219,8 @@ Different options are shown in figure and described below.

[hermes]
# List of services Hermes should send messages to.
# Supported values: influx, elastic, email, activemq
services_list = elastic, influx, email, activemq
# Supported values: influx, elastic, email, activemq, kafka
services_list = elastic, influx, email, activemq, kafka

# Toggle query behavior:
# True -> fetch bulk messages for each service individually
Expand Down Expand Up @@ -276,6 +298,8 @@ The final format of the message is determined by the destination service, as Her

- ActiveMQ (STOMP Message): The body is a streamlined JSON object containing only `event_type`, `payload`, and `created_at`. The message uses STOMP headers to set the event_type and flag the message as persistent.

- Kakfa: The body is a JSON object containing only `event_type`, `payload`, and `created_at`.

- Elasticsearch / OpenSearch (Bulk API): Hermes sends the raw database JSON message (including `id` and `services`) as a document using Bulk API format (via a POST request).

- InfluxDB (Line Protocol): Hermes performs on-the-fly aggregation of transfers and deletions, counting successes/failures and bytes. It does not send the raw event JSON. The final format is the InfluxDB Line Protocol, which consists of a single text line combining the measurement, tags (e.g., RSE, activity), fields (e.g., `nb_done=10`), and a timestamp.
Expand Down