Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -22,26 +22,26 @@ nodes:
remediationHint: "Check that the suse-observability-agent k8s-resource-collector pod is running, has Kubernetes API/RBAC access, and can export OTLP telemetry to SUSE Observability."

- _type: "Monitor"
name: "K8s resource collector - oversized CR payloads dropped"
name: "K8s resource collector - payloads dropped by budget"
tags:
- k8s-resource-collector
- otel-k8s-crd
function: urn:stackpack:common:monitor-function:threshold
identifier: "urn:stackpack:suse-observability-2:shared:monitor:k8s-resource-collector-oversized-cr-payloads"
identifier: "urn:stackpack:suse-observability-2:shared:monitor:k8s-resource-collector-payloads-dropped"
status: "ENABLED"
description: "Monitors CR log records dropped because their serialized payload exceeded max_cr_data_size. "
description: "Monitors CR or Kubernetes object payloads dropped because they did not fit the configured total payload budget."
arguments:
metric:
query: sum by (k8s_cluster_name, k8s_namespace_name, k8s_pod_name, api_group, kind) (increase(receiver_k8sresource_oversized_cr_payloads_total[5m]))
query: sum by (k8s_cluster_name, k8s_namespace_name, k8s_pod_name, source, api_group, kind) (increase(receiver_k8sresource_payloads_dropped_total[5m]))
Comment thread
deontaljaard marked this conversation as resolved.
unit: "short"
aliasTemplate: "Oversized CR payloads dropped in 5m"
aliasTemplate: "Payloads dropped in 5m"
comparator: GT
threshold: 0
failureState: "DEVIATING"
urnTemplate: "urn:kubernetes:/${k8s_cluster_name}:${k8s_namespace_name}:pod/${k8s_pod_name}"
titleTemplate: "K8s resource collector - oversized CR payloads dropped"
titleTemplate: "K8s resource collector - payloads dropped by budget"
intervalSeconds: 300
remediationHint: "Review which Custom Resources are being collected. Use the [Metrics Explorer](/#/metrics?metrics=%5B%5B%5B%22histogram_quantile%28%5Cn%20%200.95%2C%5Cn%20%20sum%20by%20%28le%2C%20api_group%2C%20kind%29%20%28%5Cn%20%20%20%20rate%28receiver_k8sresource_oversized_cr_payload_size_bytes_bucket%5B5m%5D%29%5Cn%20%20%29%5Cn%29%22%5D%5D%2C%5B0%2C%5B0%2C%22bytes%22%2C6%2C%5Btrue%2C%22%22%5D%5D%5D%5D) to see the 95th percentile size of dropped CR payloads by API group and kind. Increase `otel.k8sResourceCollector.crDiscovery.maxCrDataSize` only when larger payloads are expected and safe to forward."
remediationHint: "Review which CRs and Kubernetes objects are being collected. Use the [Metrics Explorer](/#/metrics?metrics=%5B%5B%5B%22histogram_quantile%28%5Cn%20%200.95%2C%5Cn%20%20sum%20by%20%28le%2C%20source%2C%20api_group%2C%20kind%2C%20outcome%29%20%28%5Cn%20%20%20%20rate%28receiver_k8sresource_payload_size_bytes_bucket%7Boutcome%3D%5C%22dropped%5C%22%7D%5B5m%5D%29%5Cn%20%20%29%5Cn%29%22%5D%5D%2C%5B0%2C%5B0%2C%22bytes%22%2C6%2C%5Btrue%2C%22%22%5D%5D%5D%5D) to see the 95th percentile size of dropped payloads by source, API group, and kind. Increase `otel.k8sResourceCollector.dataLimits.maxCrTotalDataSizeBytes` or `otel.k8sResourceCollector.dataLimits.maxObjectTotalDataSizeBytes` only when larger payloads are expected and safe to forward."

- _type: "Monitor"
name: "K8s resource collector - informer reconcile failures"
Expand Down
2 changes: 1 addition & 1 deletion stackpacks/suse-observability-2/stackpack.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: suse-observability-2
version: "0.0.5"
version: "0.0.6"
schemaVersion: "2.0"
displayName: "SUSE Observability 2.0"
categories: [ "SUSE Observability" ]
Expand Down
Loading