From 41df519878b12ef3a2a9f6c1303f89f1c273c1ad Mon Sep 17 00:00:00 2001 From: Deon Taljaard Date: Fri, 10 Jul 2026 15:03:07 +0200 Subject: [PATCH 1/3] STAC-24730: k8s-resource-collector docs --- docs/latest/modules/en/nav.adoc | 1 + .../modules/en/pages/setup/otel/agent.adoc | 17 ++- .../setup/otel/k8s-resource-collector.adoc | 140 ++++++++++++++++++ .../pages/setup/otel/telemetry-gateway.adoc | 2 +- 4 files changed, 153 insertions(+), 7 deletions(-) create mode 100644 docs/latest/modules/en/pages/setup/otel/k8s-resource-collector.adoc diff --git a/docs/latest/modules/en/nav.adoc b/docs/latest/modules/en/nav.adoc index 4a123c780..4258c67e9 100644 --- a/docs/latest/modules/en/nav.adoc +++ b/docs/latest/modules/en/nav.adoc @@ -138,6 +138,7 @@ ifdef::ss-ff-stackpacks2_enabled[] *** xref:setup/otel/agent.adoc[OpenTelemetry pipeline] *** xref:setup/otel/telemetry-gateway.adoc[Telemetry gateway] +*** xref:setup/otel/k8s-resource-collector.adoc[K8s resource collector] endif::ss-ff-stackpacks2_enabled[] *** xref:k8s-suse-rancher-prime-agent-air-gapped.adoc[Air-gapped installation using Helm charts] *** xref:setup/k8s-network-configuration-saas.adoc[Network configuration for SaaS] diff --git a/docs/latest/modules/en/pages/setup/otel/agent.adoc b/docs/latest/modules/en/pages/setup/otel/agent.adoc index 34b358c17..9bac20a87 100644 --- a/docs/latest/modules/en/pages/setup/otel/agent.adoc +++ b/docs/latest/modules/en/pages/setup/otel/agent.adoc @@ -7,12 +7,13 @@ ifdef::ss-ff-stackpacks2_enabled[] == Overview -The SUSE Observability Agent ships with an OpenTelemetry Collector that supplements the existing agent components. It is the recommended path for ingesting telemetry from any workload that already speaks OTLP or exposes a Prometheus / OpenMetrics endpoint, and lets you consolidate metrics and traces into the SUSE Observability pipeline without running a separate collector. OTLP logs are accepted by the gateway pipeline for future compatibility, but platform log ingestion is not supported yet. +The SUSE Observability Agent ships with OpenTelemetry Collector based components that supplement the existing agent components. They cover push-based application telemetry, Prometheus/OpenMetrics scraping, and Kubernetes Custom Resource topology collection without requiring a separately managed collector. -The OTel pipeline covers two tracks, each implemented by its own sub-component and independently toggleable: +The OTel support in the agent is split into independently toggleable components: -* *Telemetry gateway* (`otel.telemetryGateway`) — receives *metrics and traces* pushed over OTLP from application-embedded OpenTelemetry SDKs and forwards them to SUSE Observability. The gateway also has a logs pipeline, but OTLP log ingestion is not supported by the platform yet; keep SDK log export disabled. Point your SDKs at the `suse-observability-agent-otel-telemetry-gateway` service in the agent's namespace on port `4317` (gRPC) or `4318` (HTTP). +* *Telemetry gateway* (`otel.telemetryGateway`) — receives *metrics and traces* pushed over OTLP from application-embedded OpenTelemetry SDKs and forwards them to SUSE Observability. Point your SDKs at the `suse-observability-agent-otel-telemetry-gateway` service in the agent's namespace on port `4317` (gRPC) or `4318` (HTTP). * *Prometheus / OpenMetrics scraping* (`otel.prometheusScraping`) — discovers `ServiceMonitor` and `PodMonitor` Custom Resources, scrapes the targets they describe, and ships the resulting *metrics* over OTLP. This track is metrics-only; use it for workloads that expose a Prometheus / OpenMetrics endpoint but are not instrumented with an OTel SDK. See xref:/use/metrics/k8s-otel-prometheus-scraping.adoc[Scraping OpenMetrics with ServiceMonitor and PodMonitor] for the full how-to. +* *K8s resource collector* (`otel.k8sResourceCollector`) — watches Custom Resource Definitions (CRDs), selected Custom Resource instances, and optional additional Kubernetes resources, then forwards topology logs to SUSE Observability. See xref:/setup/otel/k8s-resource-collector.adoc[K8s resource collector] for the full configuration guide. == Default state @@ -56,22 +57,26 @@ otel: enabled: false # stop accepting OTLP push from SDKs prometheusScraping: enabled: false # stop scraping ServiceMonitor / PodMonitor targets + k8sResourceCollector: + enabled: false # stop collecting CRD/CR topology ---- == Resource impact -When enabled, the OTel pipeline can add up to three workloads, depending on which sub-pipelines are turned on: +When enabled, the OTel support can add up to four workloads, depending on which components are turned on: -* A *telemetry gateway* OpenTelemetry Collector (when `otel.telemetryGateway.enabled=true`) that terminates the OTLP push receiver and forwards metrics and traces to SUSE Observability. The gateway logs pipeline is present for future compatibility, but platform log ingestion is not supported yet. +* A *telemetry gateway* OpenTelemetry Collector (when `otel.telemetryGateway.enabled=true`) that terminates the OTLP push receiver and forwards metrics and traces to SUSE Observability. * A *scraper* OpenTelemetry Collector (when `otel.prometheusScraping.enabled=true`) that scrapes the targets selected by the Target Allocator. * A *Target Allocator* (when `otel.prometheusScraping.enabled=true`) that watches `ServiceMonitor` / `PodMonitor` resources and distributes scrape targets across the scraper collectors. +* A *k8s resource collector* OpenTelemetry Collector (when `otel.k8sResourceCollector.enabled=true`) that discovers CRDs and selected Custom Resources. -All three have conservative CPU and memory requests / limits that can be tuned in the agent Helm values under `otel.telemetryGateway`, `otel.prometheusScraping.collector`, and `otel.prometheusScraping.targetAllocator`. Refer to the agent Helm chart `values.yaml` for the full set of knobs. +All workloads have conservative CPU and memory requests / limits that can be tuned in the agent Helm values under `otel.telemetryGateway`, `otel.prometheusScraping.collector`, `otel.prometheusScraping.targetAllocator`, and `otel.k8sResourceCollector`. Refer to the agent Helm chart `values.yaml` for the full set of knobs. == See also * xref:/k8s-quick-start-guide.adoc[Agent quick start guide] * xref:/setup/otel/telemetry-gateway.adoc[Telemetry gateway (OTLP push)] +* xref:/setup/otel/k8s-resource-collector.adoc[K8s resource collector] * xref:/use/metrics/k8s-otel-prometheus-scraping.adoc[Scraping OpenMetrics with ServiceMonitor and PodMonitor] * xref:/setup/otel/concepts.adoc[OpenTelemetry concepts] * xref:/setup/otel/troubleshooting.adoc[OpenTelemetry troubleshooting] diff --git a/docs/latest/modules/en/pages/setup/otel/k8s-resource-collector.adoc b/docs/latest/modules/en/pages/setup/otel/k8s-resource-collector.adoc new file mode 100644 index 000000000..ea870b02e --- /dev/null +++ b/docs/latest/modules/en/pages/setup/otel/k8s-resource-collector.adoc @@ -0,0 +1,140 @@ +ifdef::ss-ff-stackpacks2_enabled[] += K8s resource collector +:page-languages: [en, de, es, fr, ja, pt, zh] +:revdate: 2026-07-09 +:page-revdate: {revdate} +:description: SUSE Observability + +== Overview + +The *k8s resource collector* is an OpenTelemetry Collector component in the SUSE Observability Agent. It watches Kubernetes Custom Resource Definitions (CRDs), selected Custom Resource (CR) instances, and optional additional Kubernetes resources, then forwards topology logs to SUSE Observability. + +CRDs are always collected. CR instances are filtered by API group so you can control ingest volume and avoid forwarding large or sensitive resource payloads. + +== Enable + +The collector is enabled by default when StackPacks 2.0 support is enabled: + +[,yaml] +---- +global: + features: + experimentalStackpacks: true +---- + +If you enable OTel without StackPacks 2.0, keep the collector enabled explicitly: + +[,yaml] +---- +otel: + enabled: true + k8sResourceCollector: + enabled: true +---- + +== Include and exclude Custom Resource API groups + +Configure CR collection with `otel.k8sResourceCollector.crDiscovery.apiGroups`: + +[,yaml] +---- +otel: + k8sResourceCollector: + crDiscovery: + discoveryMode: api_groups + apiGroups: + include: + "policies.kubewarden.io": true + "kubevirt.io": true + exclude: + "internal.example.com": true +---- + +By default, the chart does not collect every CR API group. Instead, the enabled integration presets add common SUSE-related API groups, such as Kubewarden, SUSE Runtime Enforcer, and SUSE Virtualization. Add more API groups explicitly when you want their CR instances collected. + +To disable an integration-provided API group in an override file, set it to `false`: + +[,yaml] +---- +otel: + k8sResourceCollector: + crDiscovery: + apiGroups: + include: + "kubevirt.io": false +---- + +Set `discoveryMode: all` to collect CR instances for every CRD API group. In this mode, `apiGroups` filters are ignored. + +[NOTE] +==== +The Kubernetes Custom Resources StackPack shows all collected CRDs. It marks whether CR instances for that CRD API group are collected. If CR instances are not collected, add the API group to `crDiscovery.apiGroups.include` and upgrade the agent. +==== + +== Restricted RBAC + +By default the collector uses wildcard read permissions for custom resources. For restricted RBAC, set `rbac.useWildcard: false`. The chart uses the truthy `crDiscovery.apiGroups.include` entries to render API-group RBAC rules: + +[,yaml] +---- +otel: + k8sResourceCollector: + crDiscovery: + discoveryMode: api_groups + apiGroups: + include: + "policies.kubewarden.io": true + "kubevirt.io": true + rbac: + useWildcard: false +---- + +Kubernetes RBAC only supports exact API groups or `"*"`. Wildcard filter patterns such as `"*.example.com"` require `rbac.useWildcard: true`. + +== Per-resource payload limit + +The collector drops oversized CR log records before forwarding them. The default limit is 16 KiB: + +[,yaml] +---- +otel: + k8sResourceCollector: + crDiscovery: + maxCrDataSize: 16384 +---- + +Increasing this value can increase ingest volume and may forward larger or sensitive CR payloads. Dropped records are counted by `receiver_k8sresource_oversized_cr_payloads_total`; dropped payload sizes are recorded in `receiver_k8sresource_oversized_cr_payload_size_bytes`. + +== Additional Kubernetes resources + +Use `objects` to watch non-CRD resources alongside CRDs and CRs: + +[,yaml] +---- +otel: + k8sResourceCollector: + objects: + pods: + group: "" + namespaces: ["kube-system"] + deployments: + group: apps + labelSelector: "app=foo" +---- + +Entries that overlap a CRD covered by `crDiscovery.apiGroups` are rejected at collector startup. With `rbac.useWildcard: false`, the chart derives resource-scoped RBAC for each `objects` entry automatically. + +== Operational monitoring + +The collector exposes OpenTelemetry Collector self-metrics on `:8888` and forwards those self-metrics to SUSE Observability. The Kubernetes Custom Resources StackPack includes monitors for: + +* No CRD/CR records emitted. +* Oversized CR payloads dropped. +* Informer reconcile failures. + +== See also + +* xref:/setup/otel/agent.adoc[OpenTelemetry pipeline overview] +* xref:/setup/otel/telemetry-gateway.adoc[Telemetry gateway] +* xref:/use/metrics/k8s-otel-prometheus-scraping.adoc[Scraping OpenMetrics with ServiceMonitor and PodMonitor] +endif::ss-ff-stackpacks2_enabled[] diff --git a/docs/latest/modules/en/pages/setup/otel/telemetry-gateway.adoc b/docs/latest/modules/en/pages/setup/otel/telemetry-gateway.adoc index e1eec32a9..fc9e5f4a4 100644 --- a/docs/latest/modules/en/pages/setup/otel/telemetry-gateway.adoc +++ b/docs/latest/modules/en/pages/setup/otel/telemetry-gateway.adoc @@ -7,7 +7,7 @@ ifdef::ss-ff-stackpacks2_enabled[] == Overview -The *telemetry gateway* is an OpenTelemetry Collector component in the SUSE Observability Agent that accepts metrics and traces pushed over OTLP from application-embedded OpenTelemetry SDKs. It enriches the data with Kubernetes metadata and forwards it to the SUSE Observability platform. The gateway includes a logs pipeline for future compatibility, but OTLP log ingestion is not supported by the platform yet. +The *telemetry gateway* is an OpenTelemetry Collector component in the SUSE Observability Agent that accepts metrics and traces pushed over OTLP from application-embedded OpenTelemetry SDKs. It enriches the data with Kubernetes metadata and forwards it to the SUSE Observability platform. Use the telemetry gateway when your workloads are instrumented with an OTel SDK (Java, Go, Python, Node.js, etc.) and export telemetry using OTLP. It is the recommended integration path for natively instrumented services. From 537f4e7a3feab408247ee5fb0cc22cd434c04d60 Mon Sep 17 00:00:00 2001 From: Deon Taljaard Date: Mon, 13 Jul 2026 13:05:15 +0200 Subject: [PATCH 2/3] STAC-24730: notes on the data budget model for crs and objects --- .../en/pages/setup/otel/k8s-resource-collector.adoc | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/latest/modules/en/pages/setup/otel/k8s-resource-collector.adoc b/docs/latest/modules/en/pages/setup/otel/k8s-resource-collector.adoc index ea870b02e..cb21351d0 100644 --- a/docs/latest/modules/en/pages/setup/otel/k8s-resource-collector.adoc +++ b/docs/latest/modules/en/pages/setup/otel/k8s-resource-collector.adoc @@ -91,19 +91,20 @@ otel: Kubernetes RBAC only supports exact API groups or `"*"`. Wildcard filter patterns such as `"*.example.com"` require `rbac.useWildcard: true`. -== Per-resource payload limit +== Payload budgets -The collector drops oversized CR log records before forwarding them. The default limit is 16 KiB: +The collector uses total payload budgets to limit how much CR and object data is forwarded per collection cycle. CRDs themselves are always forwarded and do not count against these budgets. CRs and configured Kubernetes objects are considered smallest-first, then by stable identity. Objects that do not fit are dropped for that cycle. [,yaml] ---- otel: k8sResourceCollector: - crDiscovery: - maxCrDataSize: 16384 + dataLimits: + maxCrTotalDataSizeBytes: 1048576 + maxObjectTotalDataSizeBytes: 1048576 ---- -Increasing this value can increase ingest volume and may forward larger or sensitive CR payloads. Dropped records are counted by `receiver_k8sresource_oversized_cr_payloads_total`; dropped payload sizes are recorded in `receiver_k8sresource_oversized_cr_payload_size_bytes`. +Increasing these values can increase ingest volume and may forward larger or sensitive payloads. Dropped records are counted by `receiver_k8sresource_payloads_dropped_total`; payload sizes are recorded in `receiver_k8sresource_payload_size_bytes`, labelled by `source` (`cr` or `object`), API group, kind, and outcome. == Additional Kubernetes resources From f9ca1f34ee43988a918512fe380b39f0aee17a40 Mon Sep 17 00:00:00 2001 From: Deon Taljaard Date: Mon, 13 Jul 2026 16:51:49 +0200 Subject: [PATCH 3/3] STAC-24730: bump default cr/objects size budget to 10 MiB --- .../modules/en/pages/setup/otel/k8s-resource-collector.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/latest/modules/en/pages/setup/otel/k8s-resource-collector.adoc b/docs/latest/modules/en/pages/setup/otel/k8s-resource-collector.adoc index cb21351d0..068aaab70 100644 --- a/docs/latest/modules/en/pages/setup/otel/k8s-resource-collector.adoc +++ b/docs/latest/modules/en/pages/setup/otel/k8s-resource-collector.adoc @@ -100,8 +100,8 @@ The collector uses total payload budgets to limit how much CR and object data is otel: k8sResourceCollector: dataLimits: - maxCrTotalDataSizeBytes: 1048576 - maxObjectTotalDataSizeBytes: 1048576 + maxCrTotalDataSizeBytes: 10485760 # 10 MiB + maxObjectTotalDataSizeBytes: 10485760 # 10 MiB ---- Increasing these values can increase ingest volume and may forward larger or sensitive payloads. Dropped records are counted by `receiver_k8sresource_payloads_dropped_total`; payload sizes are recorded in `receiver_k8sresource_payload_size_bytes`, labelled by `source` (`cr` or `object`), API group, kind, and outcome.