diff --git a/content/en/ninja-workshops/17-appd-ingest/1-overview/_index.md b/content/en/ninja-workshops/17-appd-ingest/1-overview/_index.md index 105ca5ebfc..97f3a54bb5 100644 --- a/content/en/ninja-workshops/17-appd-ingest/1-overview/_index.md +++ b/content/en/ninja-workshops/17-appd-ingest/1-overview/_index.md @@ -11,7 +11,7 @@ description: Use case, architecture, prerequisites, and the differences between Your organization runs AppDynamics for APM today. As part of a data visibility and governance initiative, leadership wants application performance data flowing into **Splunk Observability Cloud** as well giving teams a unified view alongside infrastructure metrics, logs, and other signals already in Splunk. -Rather than re-instrumenting every service with a separate OpenTelemetry SDK, the AppDynamics Java Agent supports **dual signal mode**: a single agent produces both AppDynamics APM data and OpenTelemetry traces simultaneously. This lets you maintain full AppDynamics functionality while streaming the same telemetry to Splunk Observability Cloud through an OpenTelemetry Collector. +Rather than re-instrumenting every service with a separate OpenTelemetry SDK, the **AppDynamics Java Agent** supports **dual signal mode**: a single agent produces both AppDynamics APM data and OpenTelemetry traces simultaneously. This lets you maintain full AppDynamics functionality while streaming the same telemetry to Splunk Observability Cloud through an OpenTelemetry Collector. This is especially helpful for your current L1 and L2 teams who currently know and rely on AppDynamics. Dual ingest helps maintain context as the applications and services they are responsible become more connected to new services hosted in SaaS platforms in the cloud. @@ -19,15 +19,17 @@ This is especially helpful for your current L1 and L2 teams who currently know a By the end of this workshop, you will: -- Build and run a simple Java service with the AppDynamics Java Agent -- Understand the difference between **hybrid mode** and **dual signal mode** -- Enable dual signal mode to send APM data to both AppDynamics and Splunk Observability Cloud -- Verify traces and metrics in both platforms -- Create a **global data link** in Splunk Observability Cloud for one-click navigation to AppDynamics +1. Build and run a simple Java service with the AppDynamics Java Agent +2. Add infrastructure dual signal with the **AppDynamics Combined Machine Agent**, the lowest-effort step to get hosts into both AppDynamics and Splunk Observability Cloud + - Understand the difference between **hybrid mode** and **dual signal mode** on the Java agent +3. Customize the bundled OTel collector inside the AppDynamics machine agent with the workshop config and see how much additional configuration is required +4. Replace the bundled collector with the **standalone Splunk OpenTelemetry Collector** (the configuration we recommend for production observability) and watch that friction disappear + - Verify traces and metrics in both platforms +5. Create a **global data link** in Splunk Observability Cloud for one-click navigation to AppDynamics ## Architecture -In this workshop you will run a Spring Boot Java application directly on your EC2 instance. The AppDynamics Java Agent attaches to the JVM process. +In this workshop you will run a Spring Boot Java application directly on your EC2 instance. The AppDynamics Java Agent attaches to the JVM process. Each phase changes one part of the picture. **Phase 1: Normal AppD instrumentation:** @@ -35,19 +37,47 @@ In this workshop you will run a Spring Boot Java application directly on your EC Java App + AppD Agent ──▶ AppD Controller ``` -**Phase 2: Dual signal mode enabled:** +**Phase 2: Combined Machine Agent with the bundled config (infrastructure dual signal):** ```text -Java App + AppD Agent ──▶ AppD Controller (AppD protocol, unchanged) - ──▶ OTel Collector (OTLP on localhost:4317) - │ - ▼ - Splunk Observability Cloud +Java App + AppD Agent ──▶ AppD Controller (APM, unchanged) + +Machine Agent (combined) ──▶ AppD Controller (Server Visibility) + └─▶ Bundled OTel Collector ──▶ Splunk Observability Cloud (hostmetrics) + (config: agent_config.yaml, AppD default, metrics-only) +``` + +The AppDynamics Combined Machine Agent ships with a bundled OTel collector built in (this is **not** a fully-configured Splunk OTel Collector, just the Splunk Distribution binary AppD packages and updates on its own cadence). The default config is metrics-only. APM still flows only to AppDynamics. + +**Phase 3: Bundled collector running the workshop config (APM dual signal, the hard way):** + +```text +Java App + AppD Agent (dual) ──▶ AppD Controller (APM, unchanged) + ──▶ localhost:4318 (OTLP) + │ +Machine Agent (combined) │ + └─▶ Bundled OTel Collector ◀───────────┘ + (config: workshop collector-config.yaml in place of agent_config.yaml) + (env vars: SPLUNK_INGEST_URL, SPLUNK_API_URL, SPLUNK_HEC_*, SPLUNK_BUNDLE_DIR, ... exported by hand) + ──▶ Splunk Observability Cloud (APM + Hosts + Logs, mostly) +``` + +Same Splunk Distribution binary as Phase 4, just running inside the AppDynamics install. You manually export every `SPLUNK_*` env var the workshop config needs, and the Smart Agent monitor bundle is missing because AppD does not ship it. The path works for APM and host metrics, the friction is the lesson. + +**Phase 4: Standalone Splunk OTel Collector (Migration Phase):** + +```text +Java App + AppD Agent (dual) + ──▶ AppD Controller (APM, unchanged) + ──▶ localhost:4318 (OTLP) + | + └─▶ Splunk OTel Collector (systemd-managed) + ──▶ Splunk Observability Cloud (APM + Hosts + Logs + Profiling + Process List) ``` -The OpenTelemetry Collector runs on the same EC2 instance, receives OTLP from the agent, and exports traces and metrics to Splunk Observability Cloud. +Same workshop config as Phase 3, run by a `systemd`-managed standalone install. Every `SPLUNK_*` env var lives in `/etc/otel/collector/splunk-otel-collector.conf, and the collector lifecycle is independent of the AppD machine agent. This is the configuration we recommend for production observability and the natural step toward an OTel-native footprint. -**Note:** It is possible to send data directly from the agent to our [OTLP ingest endpoint](https://dev.splunk.com/observability/docs/datamodel/ingest/#Send-data-points) without a collector but you may lose some attributes and association taking part in your OTel config. +**Note:** It is possible to send data directly from the Java agent to our [OTLP ingest endpoint](https://dev.splunk.com/observability/docs/datamodel/ingest/#Send-data-points) without a collector, but you lose the resource processors, high-cardinality trim, and other pipeline configuration the workshop config provides. ## Hybrid Mode vs Dual Signal Mode @@ -83,59 +113,6 @@ Dual signal mode adds **correlation attributes** to root spans that hybrid mode These attributes enable **global data links** clickable links on Splunk Observability Cloud traces that navigate directly to the corresponding AppDynamics view. Additionally, AppDynamics snapshots captured in dual mode include the OTel `TraceId` in the Data Collectors tab, enabling navigation in both directions. -## OBI vs Traditional Zero-Code Instrumentation - -OBI and existing traditional language-specific zero-code instrumentation (Java, JS, .NET, Python, Go, PHP) serve complementary roles in an observability strategy. Understanding the differences helps determine when to use each approach. - -### 1. Instrumentation Model - -| Aspect | OBI | Traditional Zero-Code Instrumentations | -|---|---|---| -| Execution model | Out-of-process | In-process | -| Instrumentation layer | Linux kernel / network | Application runtime | -| Requires code changes | No | No or minimal | -| Requires application restart | No | Yes | -| Security profile | Isolated | Same permission profile as the application | - -### 2. Level of Visibility - -| Capability | OBI | Traditional Zero-Code Instrumentations | -|---|---|---| -| Distributed tracing | Protocol-level | Full-fidelity | -| RED metrics | Yes | Yes | -| Application logs collection | No | Yes | -| Application log-trace correlation | Yes | Yes | -| Application internals (frameworks, functions) | No (partial, mostly in Go) | Yes | -| Custom spans / business attributes | No | Yes | -| Runtime metrics (JVM, memory, threads) | No, for now | Yes | - -### 3. Coverage & Compatibility - -| Scenario | OBI | Traditional Zero-Code Instrumentations | -|---|---|---| -| Multi-language environments | Strong (protocol-based) | Language-specific | -| Third-party applications | Supported | Limited, contrib repos | -| Legacy systems | Supported | Limited | -| Compiled languages (C/C++/Rust) | Supported (with some limitations in async) | Limited | -| Async / complex frameworks | Limited in some cases | Strong | - -### 4. Operational Characteristics - -| Aspect | OBI | Traditional Zero-Code Instrumentations | -|---|---|---| -| Deployment effort | Low (drop-in) | Medium (agent attachment) | -| Time to first visibility | Minutes | "More" minutes | -| Changes to application lifecycle | No | Yes | -| Performance overhead | Minimal and isolated | Varies by language/runtime | - -### 5. Splunk Distro Features - -| Feature | OBI | Traditional Zero-Code Instrumentations | -|---|---|---| -| Always-on Profiling | No (may be bundled with eBPF profiler in future) | CPU for most, Memory for some | -| Call graphs | No | CPU for most, Memory for some | -| File-based configuration | Incoming | Java, Node.js, .NET, Python (incoming) | -| No-code instrumentation | N/A | Yes | ## Prerequisites diff --git a/content/en/ninja-workshops/17-appd-ingest/2-run-with-appd/3-run-and-verify.md b/content/en/ninja-workshops/17-appd-ingest/2-run-with-appd/3-run-and-verify.md index dfeb270ba6..d4b99660cf 100644 --- a/content/en/ninja-workshops/17-appd-ingest/2-run-with-appd/3-run-and-verify.md +++ b/content/en/ninja-workshops/17-appd-ingest/2-run-with-appd/3-run-and-verify.md @@ -12,8 +12,12 @@ Replace `` with the AppDynamics token from the previous step: {{< tabs >}} {{% tab title="Command" %}} Export your environment variables +Use your AppD key you obtained previously ```bash export APPD_ACCESS_KEY= +``` +Then export the appd app name with your unique instance identifier +```bash export APPD_APP_NAME=Dual-Ingest-${INSTANCE} ``` diff --git a/content/en/ninja-workshops/17-appd-ingest/3-enable-dual-mode/2-enable-dual-mode.md b/content/en/ninja-workshops/17-appd-ingest/3-enable-dual-mode/2-enable-dual-mode.md deleted file mode 100644 index e78e106f76..0000000000 --- a/content/en/ninja-workshops/17-appd-ingest/3-enable-dual-mode/2-enable-dual-mode.md +++ /dev/null @@ -1,96 +0,0 @@ ---- -title: 2. Enable Dual Mode -weight: 2 ---- - -Now restart the application with dual signal mode flags added to the JVM command line. - -## Stop the Running Application - -Stop the app and load generator from Phase 1: - -```bash -kill %2 2>/dev/null # stop load generator -kill %1 2>/dev/null # stop the java app -``` - -{{% notice title="Tip" style="primary" icon="lightbulb" %}} -If `kill %1` doesn't work, find the PID with `ps aux | grep ingest-workshop` and kill it directly. -{{% /notice %}} - -## Restart with Dual Mode - -Run the app again with the same AppD flags, plus the dual mode and OTel exporter flags. We will leverage the variables we set for `${APPD_ACCESS_KEY}` and `${APPD_APP_NAME}` with the same values you used in Phase 1: - -We are adding 4 lines just before we invoke the application `-jar app/target/ingest-workshop-1.0.0.jar &` - -```bash -cd ~/workshop/appd - -java -javaagent:agent/javaagent.jar \ - -Dappdynamics.controller.hostName=se-lab.saas.appdynamics.com \ - -Dappdynamics.controller.port=443 \ - -Dappdynamics.controller.ssl.enabled=true \ - -Dappdynamics.agent.applicationName=${APPD_APP_NAME} \ - -Dappdynamics.agent.tierName=OrderService \ - -Dappdynamics.agent.nodeName=OrderService-Node \ - -Dappdynamics.agent.accountName=se-lab \ - -Dappdynamics.agent.accountAccessKey=${APPD_ACCESS_KEY} \ - -Dagent.deployment.mode=dual \ - -Dotel.traces.exporter=otlp \ - -Dotel.exporter.otlp.endpoint=http://localhost:4318 \ - -Dotel.resource.attributes=service.name=OrderService,service.namespace=Dual-Ingest-${INSTANCE},deployment.environment=${INSTANCE}-appd-dual \ - -jar app/target/ingest-workshop-1.0.0.jar & -``` - -Wait for the Spring Boot startup banner to appear. -Press return to go back to your prompt. - -### What the new flags do - -| Flag | Purpose | -|---|---| -| `-Dagent.deployment.mode=dual` | Enables dual signal mode the full OTel Java auto-instrumentation runs alongside the AppD agent | -| `-Dotel.traces.exporter=otlp` | Tells the OTel instrumentation to export spans via OTLP | -| `-Dotel.exporter.otlp.endpoint` | Points to the local OTel Collector on port 4318 (HTTP/protobuf) | -| `-Dotel.resource.attributes` | Sets OTel resource attributes: `service.name` maps to the AppD tier, `service.namespace` maps to the AppD application, `deployment.environment` tags data for your workshop instance | - -## Restart the Load Generator - -```bash -while true; do - curl -s localhost:8080/order > /dev/null - curl -s localhost:8080/inventory > /dev/null - sleep 2 -done & -``` - -## Verify Dual Mode is Active - -Check the application logs for confirmation that dual mode started: - -{{< tabs >}} -{{% tab title="Command" %}} - -```bash -ps aux | grep "deployment.mode=dual" | grep -v grep -``` - -{{% /tab %}} -{{% tab title="Example Output" %}} - -```text -splunk@ip-172-31-77-108 ~/workshop/appd $ ps aux | grep "deployment.mode=dual" | grep -v grep -splunk 181598 172 2.1 14402900 717736 pts/0 SNl 21:31 1:02 java -javaagent:agent/javaagent.jar -Dappdynamics.controller.hostName=se-lab.saas.appdynamics.com -Dappdynamics.controller.port=443 -Dappdynamics.controller.ssl.enabled=true -Dappdynamics.agent.applicationName=Dual-Ingest-shw-1123 -Dappdynamics.agent.tierName=OrderService -Dappdynamics.agent.nodeName=OrderService-Node -Dappdynamics.agent.accountName=se-lab -Dappdynamics.agent.accountAccessKey=hj9999999999 -Dagent.deployment.mode=dual -Dotel.traces.exporter=otlp -Dotel.exporter.otlp.endpoint=http://localhost:4318 -Dotel.resource.attributes=service.name=OrderService,service.namespace=Dual-Ingest-shw-a79e,deployment.environment=shw-a79e-appd-dual -jar app/target/ingest-workshop-1.0.0.jar -``` - -{{% /tab %}} -{{< /tabs >}} - -You should see the java process with the `deployment.mode=dual` flag. - -The AppDynamics agent is now sending: - -- **AppD APM data** to the AppDynamics Controller (unchanged) -- **OTLP traces** to the local OTel Collector on `localhost:4318`, which forwards to Splunk Observability Cloud - - use `env` in your instance to see the `{INSTANCE}` value used for your environment `deployment.environment=${INSTANCE}-appd-dual` diff --git a/content/en/ninja-workshops/17-appd-ingest/3-enable-dual-mode/3-verify-in-both.md b/content/en/ninja-workshops/17-appd-ingest/3-enable-dual-mode/3-verify-in-both.md deleted file mode 100644 index 1448955c39..0000000000 --- a/content/en/ninja-workshops/17-appd-ingest/3-enable-dual-mode/3-verify-in-both.md +++ /dev/null @@ -1,56 +0,0 @@ ---- -title: 3. Verify in Both Platforms -weight: 3 ---- - -With dual mode enabled and load flowing, traces should be arriving in Splunk Observability Cloud within a few minutes. Let's verify both destinations. - -## Verify AppDynamics (Unchanged) - -Go back to the [AppDynamics Controller](https://se-lab.saas.appdynamics.com/controller/), open your application, and confirm: - -- The **OrderService** tier is still visible in the flow map -- Business transactions for `/order` and `/inventory` are still recording -- No errors or degradation from adding dual mode - -Dual mode should not impact AppDynamics data collection both streams operate independently. - -## Verify Splunk Observability Cloud - -1. Navigate to [Splunk Observability Cloud](https://app.signalfx.com) and log in with the credentials provided by your instructor. -2. Click **APM** in the left navigation panel. -3. In the **Environment** dropdown, select `-appd-dual` (this matches the `deployment.environment` value you set in the resource attributes). -![AppDynamics Application](../../_images/o11y-service.png?width=30vw) - -{{% notice title="Give it a few minutes" style="info" icon="info-circle" %}} -It can take 2-5 minutes for traces to appear after enabling dual mode. If you don't see your service yet, wait a moment and refresh the page. -{{% /notice %}} - -4. You should see **OrderService** in the service list. - -## Explore a Trace - -1. Click on the **OrderService** service. -2. Click **Traces** to view individual traces. -3. Select a trace for `GET /order` to open the trace detail waterfall. - -In the trace waterfall, you'll see spans generated by the OTel Java auto-instrumentation. These are the same requests that AppDynamics is also monitoring. -![APM waterall](../../_images/waterfall.png) -## Examine the AppDynamics Correlation Attributes - -Click on a **root span** and look at the span attributes. You should see the AppDynamics correlation attributes: - -| Attribute | Example Value | -|---|---| -| `appd.app.name` | `Dual-Ingest-YOURINITIALS` | -| `appd.tier.name` | `OrderService` | -| `appd.bt.name` | `/order` or `/inventory` | -| `appd.request.guid` | *(the AppDynamics request GUID)* | - -These attributes are automatically added by the AppDynamics agent in dual mode. They create a direct link between this OTel trace and the corresponding data in the AppDynamics Controller. - -{{% notice title="Key Insight" style="primary" icon="lightbulb" %}} -The `appd.tier.name` attribute also appears on spans in the middle of a trace whenever the tier changes. In a multi-tier application, each span carries the correct AppDynamics tier identity. -{{% /notice %}} - -You now have the same application sending APM data to **two platforms simultaneously** from a single agent. In the next section, you'll connect the two by creating a global data link. diff --git a/content/en/ninja-workshops/17-appd-ingest/3-enable-dual-mode/_index.md b/content/en/ninja-workshops/17-appd-ingest/3-enable-dual-mode/_index.md deleted file mode 100644 index 6a2cd9deb4..0000000000 --- a/content/en/ninja-workshops/17-appd-ingest/3-enable-dual-mode/_index.md +++ /dev/null @@ -1,12 +0,0 @@ ---- -title: "Phase 2: Enable Dual Signal Mode" -linkTitle: 3. Enable Dual Mode -weight: 3 -archetype: chapter -time: 15 minutes -description: Install the OpenTelemetry Collector, enable dual signal mode on the AppDynamics agent, and verify traces appear in both AppDynamics and Splunk Observability Cloud. ---- - -In this phase you will deploy an OpenTelemetry Collector to forward data to Splunk Observability Cloud, then restart the application with dual signal mode enabled. - -The same agent that was sending data only to AppDynamics will now send to both destinations simultaneously. diff --git a/content/en/ninja-workshops/17-appd-ingest/3-machine-agent-dual/1-download-machine-agent.md b/content/en/ninja-workshops/17-appd-ingest/3-machine-agent-dual/1-download-machine-agent.md new file mode 100644 index 0000000000..6f42135384 --- /dev/null +++ b/content/en/ninja-workshops/17-appd-ingest/3-machine-agent-dual/1-download-machine-agent.md @@ -0,0 +1,44 @@ +--- +title: 1. Download the Machine Agent +weight: 1 +--- + +The same `download-appd-agent.sh` script you used to fetch the Java agent in Phase 1 takes an optional argument. Pass `machine` to fetch the latest 64-bit Linux machine agent bundle (with a bundled JRE and a bundled OTel collector binary). + +## Run the Download Script + +{{< tabs >}} +{{% tab title="Command" %}} + +```bash +cd ~/workshop/appd +./download-appd-agent.sh machine +``` + +{{% /tab %}} +{{% tab title="Example Output" %}} + +```text +Downloading agent from: https://download-files.appdynamics.com/download-file/machine-bundle/26.x.y.zzzz/machineagent-bundle-64bit-linux-26.x.y.zzzz.zip +Archive: ./MachineAgent.zip + inflating: machine-agent/machineagent.jar + inflating: machine-agent/conf/controller-info.xml + ... +``` + +{{% /tab %}} +{{< /tabs >}} + +The bundle unpacks into `~/workshop/appd/machine-agent/`. The pieces you will touch in the next page: + +| Path | What it is | +|---|---| +| `bin/machine-agent` | Launcher script. Knows how to start both the Java machine agent and the bundled OTel collector. | +| `conf/controller-info.xml` | AppDynamics Controller settings (we will pass these via `-D` flags instead, so we do not need to edit it). | +| `conf/otel/agent_config.yaml` | Default OTel Collector config. Hostmetrics + signalfx exporter, which is exactly what we need for this phase. We use it as-is. | +| `otel-collector/bin/otelcol_linux_amd64` | The bundled OTel Collector binary. | +| `jre/bin/java` | Bundled JRE so you do not need a system JDK. | + +{{% notice title="Why a bundled JRE" style="info" icon="info-circle" %}} +The machine agent runs as a Java process. Using `jre/bin/java` from the bundle keeps the workshop independent of whatever JDK happens to be on the instance and matches the version AppDynamics qualifies the agent against. +{{% /notice %}} diff --git a/content/en/ninja-workshops/17-appd-ingest/3-machine-agent-dual/2-configure-and-run.md b/content/en/ninja-workshops/17-appd-ingest/3-machine-agent-dual/2-configure-and-run.md new file mode 100644 index 0000000000..4cb24b2ffa --- /dev/null +++ b/content/en/ninja-workshops/17-appd-ingest/3-machine-agent-dual/2-configure-and-run.md @@ -0,0 +1,98 @@ +--- +title: 2. Configure and Run +weight: 2 +--- + +You will: + +1. Set the environment variables that flip the launcher into combined mode. +2. Start the combined machine agent. + +The Java app from Phase 1 keeps running unchanged. + +## Set Combined Mode Environment Variables + +Per the [Combined Agent docs](https://help.splunk.com/en/appdynamics-on-premises/infrastructure-visibility/26.4.0/machine-agent/combined-agent-for-infrastructure-visibility), environment variables have the highest precedence. Map your existing `${ACCESS_TOKEN}` and `${REALM}` to the names the bundled collector expects, and tag the host so it is easy to find in Splunk: + +```bash +export SPLUNK_OTEL_ENABLED=true +export SPLUNK_ACCESS_TOKEN=${ACCESS_TOKEN} +export SPLUNK_REALM=${REALM} +export DEPLOYMENT_ENVIRONMENT=${INSTANCE}-appd-machine +``` + +`SPLUNK_OTEL_ENABLED=true` is what flips the launcher into combined mode. With it set, the launcher starts the bundled OTel collector at `otel-collector/bin/otelcol_linux_amd64` alongside the Java machine agent process and configures it from the bundled `conf/otel/agent_config.yaml`. + +## Start the Combined Machine Agent + +The launcher accepts the same `-D` system properties you used for the Java agent in Phase 1, plus `-d` for daemon mode. The launcher requires a pidfile when running detached, so we write one alongside the install: + +{{< tabs >}} +{{% tab title="Command" %}} + +```bash +cd ~/workshop/appd/machine-agent + +./bin/machine-agent -p ./machine-agent.pid -d \ + -Dappdynamics.controller.hostName=se-lab.saas.appdynamics.com \ + -Dappdynamics.controller.port=443 \ + -Dappdynamics.controller.ssl.enabled=true \ + -Dappdynamics.agent.applicationName=${APPD_APP_NAME} \ + -Dappdynamics.agent.tierName=OrderService \ + -Dappdynamics.agent.nodeName=OrderService-Machine-Node \ + -Dappdynamics.agent.accountName=se-lab \ + -Dappdynamics.agent.accountAccessKey=${APPD_ACCESS_KEY} \ + -Dappdynamics.agent.uniqueHostId=${INSTANCE} \ + -Dappdynamics.sim.enabled=true +``` + +{{% /tab %}} +{{% tab title="Example Output" %}} + +```text +Using java executable at /home/splunk/workshop/appd/machine-agent/jre/bin/java +Starting OTel Collector: /home/splunk/workshop/appd/machine-agent/otel-collector/bin/otelcol_linux_amd64 +OTel config: /home/splunk/workshop/appd/machine-agent/conf/otel/agent_config.yaml +OTel Collector started (PID=151396) +``` + +{{% /tab %}} +{{< /tabs >}} + +The flags worth calling out: + +| Flag | Purpose | +|---|---| +| `-p ./machine-agent.pid` | **Required by the launcher whenever `-d` is set.** Write the launcher PID to this file. | +| `-d` | Run the launcher detached as a background daemon. | +| `-Dappdynamics.agent.uniqueHostId=${INSTANCE}` | Pin the host identity so it matches `host.name` in Splunk and the AppDynamics machine identity. | +| `-Dappdynamics.sim.enabled=true` | Enable Server Visibility (SIM) so AppDynamics surfaces host metrics, not just JVM-side data. | + +The launcher writes its own startup log to `machine-agent/logs/startup.out` and the bundled collector's log to `machine-agent/logs/otel/otel-collector.log`. + +## Verify the Bundled Collector is Listening + +{{< tabs >}} +{{% tab title="Command" %}} + +```bash +curl -s http://localhost:13133/ | jq +``` + +{{% /tab %}} +{{% tab title="Example Output" %}} + +```json +{ + "status": "Server available", + "upSince": "2026-05-04T19:46:26.711526053Z", + "uptime": "11.697620361s" +} +``` + +{{% /tab %}} +{{< /tabs >}} + +This means the bundled collector is now scraping host metrics every 10 seconds and exporting them to Splunk Observability Cloud through the SignalFx exporter! The machine agent process also reports the same hardware data to the AppDynamics Controller over the AppD protocol. + +The Java app from Phase 1 is still running and still sending APM data to AppDynamics only. Move on to verify both halves of the dual signal. diff --git a/content/en/ninja-workshops/17-appd-ingest/3-machine-agent-dual/3-verify.md b/content/en/ninja-workshops/17-appd-ingest/3-machine-agent-dual/3-verify.md new file mode 100644 index 0000000000..10413e995c --- /dev/null +++ b/content/en/ninja-workshops/17-appd-ingest/3-machine-agent-dual/3-verify.md @@ -0,0 +1,52 @@ +--- +title: 3. Verify in Both Platforms +weight: 3 +--- + +You should now see infrastructure metrics for the same host in both AppDynamics Server Visibility and Splunk Observability Cloud Infrastructure. + +## Verify Infrastructure in AppDynamics + +1. Open the [AppDynamics Controller](https://se-lab.saas.appdynamics.com/controller/). +2. Open your application (`Dual-Ingest-${INSTANCE}`). +3. Navigate to **Servers** in the left navigation panel. +4. You should see your host (matching the `${INSTANCE}` you set as `appdynamics.agent.uniqueHostId`). + +Server Visibility (SIM) shows CPU, memory, disk, network, and process metrics. The Java APM data from Phase 1 still flows under the **OrderService** tier in the same application. + +{{% notice title="Patience" style="info" icon="info-circle" %}} +It can take 2 to 5 minutes for the host to appear in **Servers** after the machine agent first starts. +{{% /notice %}} + +![AppD Server Visibility row showing the workshop host with CPU, memory, disk, and network metrics](../../_images/appd-machine-agent-metrics.png) + +## Verify Infrastructure in Splunk Observability Cloud + +The fastest way to confirm the bundled collector is shipping host metrics tagged with your workshop identity is to chart a single metric and filter by `host.name`: + +1. Open Splunk Observability Cloud (E.G. https://app.us1.signalfx.com). +2. Click **Metric Finder** in the left navigation panel. +3. Search for [`cpu.utilization`](https://app.us1.signalfx.com/#/chart/v2/new?template=default&filters=sf_metric:cpu.utilization) and click the metric to open the chart. +4. Add a filter: `host.name=${INSTANCE}` (use your actual `${INSTANCE}` value, for example `shw-40fb`). + +You should see a live time series for that single host. The data path is the bundled `hostmetrics` receiver inside the machine agent's collector, exported through the `signalfx` exporter to Splunk Observability Cloud. `host.name` matches `${INSTANCE}` because the workshop instance's OS hostname is set to that value, and the bundled config's `resourcedetection` processor reads it from the system. + +`deployment.environment` is also stamped on every metric, set from the `DEPLOYMENT_ENVIRONMENT=${INSTANCE}-appd-machine` env var you exported on the previous page. You can add that as a second filter to narrow further or use it to find the host in **Infrastructure > Hosts**. + +![Splunk Observability Cloud Infrastructure Monitoring](../../_images/o11y-infra-metrics-appd-machine-agent.png) + +## What You Don't See Yet + +Open **APM** in Splunk Observability Cloud. The **OrderService** service is **not** there. APM data from the Java agent still flows only to AppDynamics because we have not enabled dual signal mode on the Java agent yet. + +You also will not see this host correctly light up the **Infrastructure -> Overview -> Hosts** Navigators due to missing dimensions stamped by the workshop's `resource/workshop` processor, which is not part of the bundled config. + +**Phase 3** closes both gaps without leaving the bundled collector. You drop the workshop's full `collector-config.yaml` in place of the AppD-shipped `agent_config.yaml`, export the env vars the workshop config needs, and turn on dual signal mode on the Java agent. **Phase 4** then takes the same workshop config and runs it under the standalone Splunk OTel Collector install for the lifecycle, env var, and Smart Agent bundle wins. + +## Where to Look if Something is Off + +| Symptom | First place to look | +|---|---| +| No host in Splunk Infrastructure | `~/workshop/appd/machine-agent/logs/otel/otel-collector.log` | +| No host in AppD Server Visibility | `~/workshop/appd/machine-agent/logs/machine-agent.log`, confirm `-Dappdynamics.sim.enabled=true` | +| Bundled collector did not start | Confirm `SPLUNK_OTEL_ENABLED=true` is exported in the same shell that ran `bin/machine-agent` | diff --git a/content/en/ninja-workshops/17-appd-ingest/3-machine-agent-dual/_index.md b/content/en/ninja-workshops/17-appd-ingest/3-machine-agent-dual/_index.md new file mode 100644 index 0000000000..f49e9d4ff8 --- /dev/null +++ b/content/en/ninja-workshops/17-appd-ingest/3-machine-agent-dual/_index.md @@ -0,0 +1,28 @@ +--- +title: "Phase 2: Combined Machine Agent" +linkTitle: 3. Machine Agent Dual +weight: 3 +archetype: chapter +time: 10 minutes +description: Add infrastructure dual signal to your AppDynamics-only setup using the AppDynamics Combined Machine Agent, which ships with a bundled OTel collector and reports host metrics to both AppDynamics and Splunk Observability Cloud. +--- + +In Phase 1 your Java application reports APM data to the AppDynamics Controller and nothing else. The simplest first step toward Splunk Observability Cloud is **infrastructure dual signal**: report host metrics (CPU, memory, disk, network) to both platforms from a single agent. + +The **AppDynamics Machine Agent** ships with a minimal OpenTelemetry Collector binary bundled inside the install. +- One environment variable (`SPLUNK_OTEL_ENABLED=true`) starts both processes and the bundled collector forwards host metrics to Splunk while the machine agent process continues reporting to the AppDynamics Controller. + +What you **do not** get from the AppD bundle is a fully-configured Splunk OTel Collector deployment with all of the correlation and log goodness. +The bundled config is intentionally narrow: a `hostmetrics` receiver and the `signalfx` exporter, no traces, logs, or profiling pipelines. That is fine for infrastructure dual signal, but it is not a substitute for the workshop config we apply in Phase 3 or the standalone Splunk OTel Collector install in Phase 4. + +In this phase you will: + +1. Download the machine agent bundle using the same `download-appd-agent.sh` script you used for the Java agent in Phase 1, with a `machine` argument. +2. Start the combined machine agent. +3. Verify host metrics in **AppDynamics Server Visibility** and in **Splunk O11y Infrastructure**. + +APM data still flows only to AppDynamics at the end of this phase. **Phase 3** broadens that by dropping the workshop's full `collector-config.yaml` into the same bundled collector and turning on dual signal mode on the Java agent. **Phase 4** then takes the same workshop config and runs it under the standalone Splunk OTel Collector install, the configuration we recommend for production observability. + +{{% notice title="Why infra first" style="info" icon="info-circle" %}} +The combined machine agent is the lowest-effort dual signal step for an AppDynamics shop: it is one extra environment variable on an agent you already deploy. It is a great way to see your hosts side-by-side in both platforms before you take on the broader APM migration. +{{% /notice %}} diff --git a/content/en/ninja-workshops/17-appd-ingest/4-customize-bundled-collector/1-apply-workshop-config.md b/content/en/ninja-workshops/17-appd-ingest/4-customize-bundled-collector/1-apply-workshop-config.md new file mode 100644 index 0000000000..5a94143a0a --- /dev/null +++ b/content/en/ninja-workshops/17-appd-ingest/4-customize-bundled-collector/1-apply-workshop-config.md @@ -0,0 +1,159 @@ +--- +title: 1. Apply the Workshop Config to the Bundled Collector +weight: 1 +--- + +You will: + +1. Stop the bundled collector that came up in Phase 2 (which means stopping the machine agent, since the launcher owns the OTel process). +2. Back up the bundled `agent_config.yaml`. +3. Drop the workshop `collector-config.yaml` into the AppD install in place of it. +4. Export the additional environment variables the workshop config needs that the AppD launcher does not set. +5. Start the machine agent again so the bundled collector comes back up running the workshop config. + +## Stop the Combined Machine Agent + +Use the pidfiles the launcher wrote in Phase 2: + +```bash +cd ~/workshop/appd/machine-agent + +kill "$(cat machine-agent.pid)" 2>/dev/null || true +kill "$(cat otel-collector.pid)" 2>/dev/null || true +rm -f machine-agent.pid otel-collector.pid +``` + +Confirm the OTLP and health-check ports are clear before continuing: + +```bash +sudo ss -lntp | grep -E ':4317|:4318|:13133' || echo "ports clear" +``` + +You should see `ports clear`. + +## Back Up the Bundled Config + +The default `conf/otel/agent_config.yaml` is what the AppD machine agent ships. Save it before overwriting so you can revert by restoring it: + +```bash +cd ~/workshop/appd/machine-agent + +cp conf/otel/agent_config.yaml conf/otel/agent_config.yaml.appd-default +``` + +## See How Much You're About to Add + +Before you swap the file, compare what AppD ships against what you are about to drop in. The bundled config is intentionally narrow per the [Combined Agent docs](https://help.splunk.com/en/appdynamics-on-premises/infrastructure-visibility/26.4.0/machine-agent/combined-agent-for-infrastructure-visibility): a `hostmetrics` receiver feeding the `signalfx` exporter, plus the minimal extensions and processors AppD wires up to make that work. The workshop config adds additional receivers for OTLP data, custom resource, correlation, and high-cardinality protection processors. But most importantly they include the `splunk_hec` and `otlp_http` exporters along with the trace, log, and profiling pipelines those exporters need. + +Count the line delta first: + +```bash +wc -l conf/otel/agent_config.yaml.appd-default ~/workshop/appd/collector-config.yaml +``` + +Then look at the diff. The full output is several hundred lines, so pipe through `less`: + +```bash +diff -u conf/otel/agent_config.yaml.appd-default ~/workshop/appd/collector-config.yaml | less +``` + + +Everything that diff shows is something the AppD bundle does not configure for you. That is the work the standalone install takes care of all of this. + +## Drop the Workshop Config Into the AppD Install + +```bash +cp ~/workshop/appd/collector-config.yaml conf/otel/agent_config.yaml +``` + +That's it. The same config you would point the standalone collector at in Phase 4 now sits inside the AppD machine agent install. + +## Export the Environment Variables the Workshop Config Needs + +The workshop `collector-config.yaml` references variables the AppD launcher does not set. Export them in the same shell you will start the machine agent from: + +```bash +export SPLUNK_OTEL_ENABLED=true +export SPLUNK_ACCESS_TOKEN=${ACCESS_TOKEN} +export SPLUNK_REALM=${REALM} + +export SPLUNK_INGEST_URL=https://ingest.${REALM}.signalfx.com +export SPLUNK_API_URL=https://api.${REALM}.signalfx.com +export SPLUNK_HEC_TOKEN=${HEC_TOKEN} +export SPLUNK_HEC_URL=${HEC_URL} +export SPLUNK_LISTEN_INTERFACE=0.0.0.0 +export SPLUNK_MEMORY_LIMIT_MIB=512 +export SPLUNK_BUNDLE_DIR=/usr/lib/splunk-otel-collector/agent-bundle +export SPLUNK_COLLECTD_DIR=/usr/lib/splunk-otel-collector/agent-bundle/run/collectd +``` + +Notice the count. The first three lines are what Phase 2 needed. The next eight are required to be manually set for the machine agent. The Splunk OpenTelemetry Collector Distro by default handles most of these environment variables. + +The workshop config's `resource/workshop` processor stamps `deployment.environment=${INSTANCE}-appd-dual` on every signal regardless of what the Java agent sends, so we do not need `DEPLOYMENT_ENVIRONMENT` exported here. + +| Variable | Why the workshop config needs it | Set by AppD launcher? | +|---|---|---| +| `SPLUNK_INGEST_URL` | `signalfx.ingest_url`, `splunk_hec/profiling.endpoint`, `otlp_http.traces_endpoint` | No | +| `SPLUNK_API_URL` | `signalfx.api_url`, `http_forwarder` egress | No | +| `SPLUNK_HEC_TOKEN` | `splunk_hec.token` (logs export) | No | +| `SPLUNK_HEC_URL` | `splunk_hec.endpoint` (logs export) | No | +| `SPLUNK_LISTEN_INTERFACE` | All receivers and extensions bind to `${SPLUNK_LISTEN_INTERFACE}:` | No | +| `SPLUNK_MEMORY_LIMIT_MIB` | `memory_limiter.limit_mib` | No | +| `SPLUNK_BUNDLE_DIR` | `smartagent` extension `bundleDir` (the bundle is not present on disk) | No | +| `SPLUNK_COLLECTD_DIR` | `smartagent` extension `collectd.configDir` (also not present) | No | + +The bundle directory `/usr/lib/splunk-otel-collector/agent-bundle` does not exist yet because the AppD machine agent install does not ship it. The smartagent extension will start with the value, the `smartagent/processlist` receiver will fail every scrape because there is no bundle to load monitors from. That is one of the gaps Phase 4 closes. + +## Restart the Machine Agent + +Use the same launcher invocation as Phase 2. The launcher detects `SPLUNK_OTEL_ENABLED=true` and starts the bundled collector with the new config: + +```bash +cd ~/workshop/appd/machine-agent + +./bin/machine-agent -p ./machine-agent.pid -d \ + -Dappdynamics.controller.hostName=se-lab.saas.appdynamics.com \ + -Dappdynamics.controller.port=443 \ + -Dappdynamics.controller.ssl.enabled=true \ + -Dappdynamics.agent.applicationName=${APPD_APP_NAME} \ + -Dappdynamics.agent.tierName=OrderService \ + -Dappdynamics.agent.nodeName=OrderService-Machine-Node \ + -Dappdynamics.agent.accountName=se-lab \ + -Dappdynamics.agent.accountAccessKey=${APPD_ACCESS_KEY} \ + -Dappdynamics.agent.uniqueHostId=${INSTANCE} \ + -Dappdynamics.sim.enabled=true +``` + +You should see the launcher pick up the workshop config + +## Verify the Bundled Collector Started With the Workshop Config + +The workshop config opens the same OTLP, Jaeger, Zipkin, and health-check ports the standalone install does. Confirm a few are listening now that they were not before: + +{{< tabs >}} +{{% tab title="Command" %}} + +```bash +curl -s http://localhost:13133/ | jq +sudo ss -lntp | grep -E ':4317|:4318|:9411|:13133' +``` + +{{% /tab %}} +{{% tab title="Example Output" %}} + +```text +{ + "status": "Server available", + "upSince": "2026-05-04T20:30:11.412Z", + "uptime": "8.221s" +} +LISTEN 0 4096 *:4317 *:* users:(("otelcol_linux_a",pid=...,fd=...)) +LISTEN 0 4096 *:4318 *:* users:(("otelcol_linux_a",pid=...,fd=...)) +LISTEN 0 4096 *:9411 *:* users:(("otelcol_linux_a",pid=...,fd=...)) +LISTEN 0 4096 *:13133 *:* users:(("otelcol_linux_a",pid=...,fd=...)) +``` + +{{% /tab %}} +{{< /tabs >}} + +The bundled collector is now ready to accept OTLP from the Java agent. Move on to enable dual signal mode and point the Java app at it. diff --git a/content/en/ninja-workshops/17-appd-ingest/4-customize-bundled-collector/2-enable-dual-mode.md b/content/en/ninja-workshops/17-appd-ingest/4-customize-bundled-collector/2-enable-dual-mode.md new file mode 100644 index 0000000000..6bc5f6e7e2 --- /dev/null +++ b/content/en/ninja-workshops/17-appd-ingest/4-customize-bundled-collector/2-enable-dual-mode.md @@ -0,0 +1,87 @@ +--- +title: 2. Enable Dual Mode on the Java Agent +weight: 2 +--- + +The bundled collector is now running the workshop config and listening for OTLP on `localhost:4318`. Restart the Java app from Phase 1 in dual signal mode so its traces flow into that collector and onward to Splunk APM. + +## Stop the Running Application + +```bash +kill %2 2>/dev/null # stop load generator +kill %1 2>/dev/null # stop the java app +``` + +{{% notice title="Tip" style="primary" icon="lightbulb" %}} +If `kill %1` does not work, find the PID with `ps aux | grep ingest-workshop` and kill it directly. +{{% /notice %}} + +## Restart with Dual Mode + +Same AppD flags as Phase 1, with four additional lines that turn on dual signal mode and point the OTel side at the bundled collector. We reuse the `${APPD_ACCESS_KEY}` and `${APPD_APP_NAME}` variables from Phase 1. + +```bash +cd ~/workshop/appd + +java -javaagent:agent/javaagent.jar \ + -Dappdynamics.controller.hostName=se-lab.saas.appdynamics.com \ + -Dappdynamics.controller.port=443 \ + -Dappdynamics.controller.ssl.enabled=true \ + -Dappdynamics.agent.applicationName=${APPD_APP_NAME} \ + -Dappdynamics.agent.tierName=OrderService \ + -Dappdynamics.agent.nodeName=OrderService-Node \ + -Dappdynamics.agent.accountName=se-lab \ + -Dappdynamics.agent.accountAccessKey=${APPD_ACCESS_KEY} \ + -Dagent.deployment.mode=dual \ + -Dotel.traces.exporter=otlp \ + -Dotel.exporter.otlp.endpoint=http://localhost:4318 \ + -Dotel.resource.attributes=service.name=OrderService,service.namespace=Dual-Ingest-${INSTANCE},deployment.environment=${INSTANCE}-appd-dual \ + -jar app/target/ingest-workshop-1.0.0.jar & +``` + +Wait for the Spring Boot startup banner. Press return to get back to your prompt. + +### What the new flags do + +| Flag | Purpose | +|---|---| +| `-Dagent.deployment.mode=dual` | Enables dual signal mode. The full OTel Java auto-instrumentation runs alongside the AppD agent. | +| `-Dotel.traces.exporter=otlp` | Exports spans via OTLP. | +| `-Dotel.exporter.otlp.endpoint` | Points to the bundled collector on `localhost:4318` (OTLP HTTP). | +| `-Dotel.resource.attributes` | Stamps `service.name`, `service.namespace`, and `deployment.environment` on every span. `deployment.environment` here matches the value the workshop config's `resource/workshop` processor stamps, so spans, host metrics, and traces all land in the same Splunk APM environment. | + +## Restart the Load Generator + +```bash +while true; do + curl -s localhost:8080/order > /dev/null + curl -s localhost:8080/inventory > /dev/null + sleep 2 +done & +``` + +## Confirm Dual Mode is Active + +{{< tabs >}} +{{% tab title="Command" %}} + +```bash +ps aux | grep "deployment.mode=dual" | grep -v grep +``` + +{{% /tab %}} +{{% tab title="Example Output" %}} + +```text +splunk 181598 172 2.1 14402900 717736 pts/0 SNl 21:31 1:02 java -javaagent:agent/javaagent.jar ... -Dagent.deployment.mode=dual -Dotel.traces.exporter=otlp -Dotel.exporter.otlp.endpoint=http://localhost:4318 -Dotel.resource.attributes=service.name=OrderService,service.namespace=Dual-Ingest-shw-a79e,deployment.environment=shw-a79e-appd-dual -jar app/target/ingest-workshop-1.0.0.jar +``` + +{{% /tab %}} +{{< /tabs >}} + +The Java process now sends: + +- **AppD APM data** to the AppDynamics Controller, unchanged. +- **OTLP traces** to the bundled collector on `localhost:4318`, which the workshop config forwards to Splunk APM via the `otlp_http` and `signalfx` exporters. + +Move on to verify both destinations and then take a look at what the bundled-collector path actually cost. diff --git a/content/en/ninja-workshops/17-appd-ingest/4-customize-bundled-collector/3-verify-and-friction.md b/content/en/ninja-workshops/17-appd-ingest/4-customize-bundled-collector/3-verify-and-friction.md new file mode 100644 index 0000000000..a815e37561 --- /dev/null +++ b/content/en/ninja-workshops/17-appd-ingest/4-customize-bundled-collector/3-verify-and-friction.md @@ -0,0 +1,74 @@ +--- +title: 3. Verify +weight: 3 +--- + +You should now have APM and infrastructure in both AppDynamics and Splunk Observability Cloud, all served by a single OTel collector process the AppD launcher manages. Confirm that, then look at exactly what running in this configuration required. + +## Verify Splunk APM + +1. Open **Splunk Observability Cloud** +2. Click **APM -> Service Map** in the left navigation panel. +3. In the **Environment** dropdown, choose `-appd-dual`. The workshop config's `resource/workshop` processor stamps that environment name on every signal regardless of what the Java agent sends. +4. Click the **OrderService** service bubble. + +![Splunk Observability Cloud APM](../../_images/o11y-apm.png?width=50vw) + +5. Open **Traces** from the right menu +6. Click a `GET /order` trace. +7. Click the root span and confirm the AppDynamics correlation attributes are present: + +![Splunk Observability Cloud APM with AppD attributes](../../_images/o11y-appd-attributes.png?width=50vw) + +| Attribute | Example value | +|---|---| +| `appd.app.name` | `Dual-Ingest-YOURINITIALS` | +| `appd.tier.name` | `OrderService` | +| `appd.bt.name` | `/order` or `/inventory` | +| `appd.request.guid` | The AppDynamics request GUID | + +These attributes are the proof that the trace went through a Java agent in dual mode and that the OTel pipeline preserved them all the way to Splunk. + +{{% notice title="Give it a few minutes" style="info" icon="info-circle" %}} +Allow 2 to 5 minutes for the first traces to appear after restarting the Java app. +{{% /notice %}} + +## Verify Splunk Infrastructure + +The fastest way to confirm the bundled collector is shipping host metrics tagged with the same dimensions needed to correlate with your traces is to check the tiles for Infrastructure (and in some cases Logs) at the bottom of the screen. + +![Splunk Observability Cloud Infrastructure Correlation Tile](../../_images/o11y-Infra-Correlation.png?width=50vw) + + +## Verify AppDynamics is Unaffected + +The AppD side of both signals continues to flow unchanged: + +- **AppDynamics Controller > Applications > `Dual-Ingest-${INSTANCE}` > OrderService** still shows business transactions for `/order` and `/inventory`. +- **AppDynamics Controller > Servers** still shows the host with CPU, memory, disk, and network from the machine agent's SIM data. + +Dual mode and the workshop OTel config do not touch the AppDynamics protocols, they only add an OTel pipeline alongside them. + +## The Hidden Friction + +Everything works! The problem is what it took to get here, and what is silently still broken. + +### The lifecycle is glued to AppD + +Stopping the machine agent stops the OTel collector with it. There is no `systemctl status otel-collector` and no separate journal (though they could be manually added). Restart the machine agent to pick up a config change and you also bounce all your OTel pipelines. + +More importantly an AppDynamics machine agent upgrade can also overwrite `conf/otel/agent_config.yaml` because it is part of the AppD install tree. You would need to manage that specific otel configuration file on the instance in some fashion. + +### The collector binary is on AppD's release cadence + +The OTel binary inside the AppD machine agent ships when AppD ships. + +OTel features and security fixes you might want from a more recent Splunk Distribution release have to wait for the next AppD machine agent release. + +## What This Means + +The bundled collector path is not wrong. The bundled collector is primarily intended as a stop gap or migration method. + +For most teams running a Splunk Observability stack alongside AppDynamics, the standalone Splunk OTel Collector is a much better deal. The install script writes every `SPLUNK_*` environment variable into `/etc/otel/collector/splunk-otel-collector.conf`, ships the Smart Agent monitors like `smartagent/processlist` process data correctly, runs the collector as its own `systemd` service with independent restart and upgrade behavior, and follows the Splunk Distribution release cadence. + +That is what Phase 4 sets up, using the same workshop `collector-config.yaml` you just used here. The difference between this phase and the next is almost entirely operational. diff --git a/content/en/ninja-workshops/17-appd-ingest/4-customize-bundled-collector/_index.md b/content/en/ninja-workshops/17-appd-ingest/4-customize-bundled-collector/_index.md new file mode 100644 index 0000000000..a3260b1fbd --- /dev/null +++ b/content/en/ninja-workshops/17-appd-ingest/4-customize-bundled-collector/_index.md @@ -0,0 +1,30 @@ +--- +title: "Phase 3: Customize the Bundled Collector" +linkTitle: 4. Customize Bundled Collector +weight: 4 +archetype: chapter +time: 15 minutes +description: Reuse the OTel binary already inside the AppDynamics machine agent by overwriting its config with the full workshop collector config. Get APM, traces, and host metrics in Splunk Observability Cloud through the same process the AppD launcher manages, and see exactly what that costs in environment variables and missing pieces. +--- + +Phase 2 turned on the bundled OTel collector with the default `agent_config.yaml` AppDynamics ships. That gave you host metrics in Splunk Observability Cloud through one environment variable, but you stopped short of APM, profiling, and the rest of the pipeline set you would normally run in production. + +You do not need to install anything else to get those. The OTel binary AppD ships at `otel-collector/bin/otelcol_linux_amd64` is the **Splunk Distribution of the OpenTelemetry Collector** the same binary you would install standalone in Phase 4. It will run any config you point it at. So in this phase you point it at the workshop's full `collector-config.yaml`. +- **NOTE:** The packaged version of the collector in the AppD Machine Agent does not follow the Splunk release cadence for new Collector binaries + +So why don't we just run the Machine Agent all the time? The AppDynamics launcher only knows about the env vars the bundled config needs (`SPLUNK_OTEL_ENABLED`, `SPLUNK_ACCESS_TOKEN`, `SPLUNK_REALM`) and only contains a configuration for infrastructure metrics. +To get the rest of our data into Splunk our config needs to reference a great deal more OpenTelemetry configuration. We will wire all of that up and you will see exactly how much "by hand" work is required. + +In this phase you will: + +1. Stop the bundled collector, drop the workshop `collector-config.yaml` into the AppD install in place of `agent_config.yaml`, and export the additional environment variables the workshop config needs. +2. Restart the machine agent so the bundled collector comes back up running the workshop config. +3. Enable dual signal mode on the Java agent so APM traces flow into the bundled collector on `localhost:4318`. +4. Verify that APM, host metrics, and the AppDynamics correlation attributes all reach Splunk Observability Cloud. +5. Tally the friction the env vars you had to remember, the smart agent components that silently do nothing, and the lifecycle coupling between the AppD machine agent and the OTel collector. + +Phase 4 then takes the same workshop config, runs it through the official Splunk OTel Collector install script, and you watch the friction disappear. That is the natural next step on the path toward an OTel-native instrumentation footprint. + +{{% notice title="Why bother showing this path at all" style="info" icon="info-circle" %}} +Some teams genuinely run this way. They do not want a second OTel process on the host, or they want to keep all telemetry plumbing inside something the AppDynamics admin already operates. The path works for metrics and traces. Walking it once makes it obvious why we recommend the standalone install for everything else. +{{% /notice %}} diff --git a/content/en/ninja-workshops/17-appd-ingest/5-standalone-otel-collector/1-tear-down-bundled-collector.md b/content/en/ninja-workshops/17-appd-ingest/5-standalone-otel-collector/1-tear-down-bundled-collector.md new file mode 100644 index 0000000000..46f5b5705f --- /dev/null +++ b/content/en/ninja-workshops/17-appd-ingest/5-standalone-otel-collector/1-tear-down-bundled-collector.md @@ -0,0 +1,65 @@ +--- +title: 1. Free Up the OTel Listener for the Standalone Install +weight: 1 +--- + +The bundled OTel collector inside the AppDynamics machine agent and the standalone Splunk OTel Collector both bind to OTLP ports `4317` and `4318` on the same host. Stop the bundled side first, leaving the AppDynamics machine agent itself running so AppDynamics Server Visibility keeps reporting. The Java app stays running in dual mode the whole time the OTLP outage is short and the AppD APM channel is unaffected. + +## Stop the Machine Agent and Its Bundled Collector + +Use the pidfiles the launcher wrote in Phase 3: + +```bash +cd ~/workshop/appd/machine-agent + +kill "$(cat machine-agent.pid)" 2>/dev/null || true +kill "$(cat otel-collector.pid)" 2>/dev/null || true +rm -f machine-agent.pid otel-collector.pid +``` + +If either pidfile is missing or stale, fall back to `pkill -f machineagent.jar` and `pkill -f otelcol_linux_amd64`. + +**NOTE: IGNORE AN OTEL ERRORS THAT MAY HAPPEN IN THE BACKROUND WHILE CHANGING OUT AGENTS** + +## Confirm the OTLP Ports Are Free + +```bash +sudo ss -lntp | grep -E ':4317|:4318|:13133' || echo "ports clear" +``` + +You should see `ports clear`. If anything is still listening, kill it before continuing. + +## Restart the Machine Agent in AppD-only Mode + +Re-run the launcher **without** `SPLUNK_OTEL_ENABLED=true` so the bundled OTel collector stays off and the machine agent process keeps feeding Server Visibility: + +```bash +unset SPLUNK_OTEL_ENABLED + +cd ~/workshop/appd/machine-agent + +./bin/machine-agent -p ./machine-agent.pid -d \ + -Dappdynamics.controller.hostName=se-lab.saas.appdynamics.com \ + -Dappdynamics.controller.port=443 \ + -Dappdynamics.controller.ssl.enabled=true \ + -Dappdynamics.agent.applicationName=${APPD_APP_NAME} \ + -Dappdynamics.agent.tierName=OrderService \ + -Dappdynamics.agent.nodeName=OrderService-Machine-Node \ + -Dappdynamics.agent.accountName=se-lab \ + -Dappdynamics.agent.accountAccessKey=${APPD_ACCESS_KEY} \ + -Dappdynamics.agent.uniqueHostId=${INSTANCE} \ + -Dappdynamics.sim.enabled=true +``` + +You should now have: + +- **AppDynamics**: APM from the Java agent (still in dual mode) **plus** Server Visibility from the machine agent. +- **Splunk Observability Cloud**: a brief gap while the OTLP listener is down. The Java app continues to send to `localhost:4318` and will retry once the standalone collector comes up next. + +{{% notice title="Why not stop the Java app too" style="info" icon="info-circle" %}} +The Java app from Phase 3 is already in dual signal mode and configured to export OTLP to `localhost:4318`. As soon as the standalone collector starts listening on those ports in the next step, the Java app's traces resume without restart. Killing it here would just create more work. +{{% /notice %}} + +You should be seeing errors coming from your Java app running in the background trying to send opentelemetry data. We address that next. + +Move on to install the standalone Splunk OTel Collector. diff --git a/content/en/ninja-workshops/17-appd-ingest/3-enable-dual-mode/1-install-otel-collector.md b/content/en/ninja-workshops/17-appd-ingest/5-standalone-otel-collector/2-install-otel-collector.md similarity index 82% rename from content/en/ninja-workshops/17-appd-ingest/3-enable-dual-mode/1-install-otel-collector.md rename to content/en/ninja-workshops/17-appd-ingest/5-standalone-otel-collector/2-install-otel-collector.md index e77d036a48..6d2c172a70 100644 --- a/content/en/ninja-workshops/17-appd-ingest/3-enable-dual-mode/1-install-otel-collector.md +++ b/content/en/ninja-workshops/17-appd-ingest/5-standalone-otel-collector/2-install-otel-collector.md @@ -1,25 +1,13 @@ --- -title: 1. Install the OTel Collector -weight: 1 +title: 2. Install the Splunk OTel Collector +weight: 2 --- -The AppDynamics agent in dual mode emits OpenTelemetry data over OTLP. You need a collector on the same host to receive that data and forward it to Splunk Observability Cloud. - -## Verify Environment Variables - -Your instance should have these variables pre-set. Confirm they are available with `env` or: - -```bash -echo "REALM=$REALM" -echo "ACCESS_TOKEN=$ACCESS_TOKEN" -echo "INSTANCE=$INSTANCE" -``` - -All three should have values. If any are empty, check with your instructor. +The Java app from Phase 3 is still emitting OpenTelemetry traces over OTLP to `localhost:4318`. This step puts a real collector behind that endpoint, on its own `systemd` service, and applies the same workshop `collector-config.yaml` you used in Phase 3. ## Install the Splunk OpenTelemetry Collector -Run the Splunk OTel Collector install script. This installs the collector as a `systemd` service: +Run the Splunk OTel Collector install script. This installs the collector as a `systemd` service and ships the Smart Agent monitor bundle that was missing in Phase 3: {{< tabs >}} {{% tab title="Command" %}} @@ -41,13 +29,16 @@ Splunk OpenTelemetry Collector has been successfully installed. ## Apply the Workshop Collector Configuration -The default collector config is general-purpose. We will replace it with the workshop-specific config that receives OTLP from the AppDynamics agent and exports to Splunk Observability Cloud. But first lets take a look at what we're adding: +The default collector config the install script writes is general-purpose. Replace it with the same `~/workshop/appd/collector-config.yaml` you used in Phase 3. The processors below are why the workshop ships its own config in the first place: + +We will copy the custom config we used previously in the workshop over the `agent_config.yaml` to verify we are comparing apples to apples with our previous agent: ```bash -vim ~/workshop/appd/collector-config.yaml +sudo cp ~/workshop/appd/collector-config.yaml /etc/otel/collector/agent_config.yaml ``` -look for this section under `processors:`: +Lets check out an important element in the custom config we used in both phases of our workshop. The processors below help us maintain telemetry cleanliness when ingesting into the backend: + {{< tabs >}} {{% tab title="collector-config.yaml" %}} @@ -106,15 +97,9 @@ Finally we do a check for available space afterwards to add a dimension for `car Each of these processors is included at the end of the `pipeline:` for metrics in our configuration. -We will then copy that custom config over the `agent_config.yaml`: - -```bash -sudo cp ~/workshop/appd/collector-config.yaml /etc/otel/collector/agent_config.yaml -``` - ## Set the Collector Environment Variables -The collector service reads environment variables from a config file. The Splunk OTel Collector binary requires `SPLUNK_REALM` and `SPLUNK_ACCESS_TOKEN` (with the `SPLUNK_` prefix). Your instance has `REALM` and `ACCESS_TOKEN`, so we map them: +In Phase 3 you exported these variables in your shell every time you started the machine agent. The standalone install reads them from `/etc/otel/collector/splunk-otel-collector.conf` instead. Write them once and `systemd` carries them across reboots and restarts: ```bash sudo bash -c "cat > /etc/otel/collector/splunk-otel-collector.conf << EOF diff --git a/content/en/ninja-workshops/17-appd-ingest/5-standalone-otel-collector/3-verify-in-both.md b/content/en/ninja-workshops/17-appd-ingest/5-standalone-otel-collector/3-verify-in-both.md new file mode 100644 index 0000000000..fc36b15bde --- /dev/null +++ b/content/en/ninja-workshops/17-appd-ingest/5-standalone-otel-collector/3-verify-in-both.md @@ -0,0 +1,100 @@ +--- +title: 3. Verify Under the New Substrate +weight: 3 +--- + +The Java app from Phase 3 has been emitting OTLP at `localhost:4318` continuously through this whole transition. Once the standalone collector came up in the previous step, those traces resumed flowing through the new pipeline. Verify the data is intact and look at what was painful in Phase 3 that is now solved. + +## Verify AppDynamics is Unaffected + +Open the [AppDynamics Controller](https://se-lab.saas.appdynamics.com/controller/), open your application, and confirm: + +- The **OrderService** tier is still visible in the flow map. +- Business transactions for `/order` and `/inventory` are still recording. +- **Servers** still shows the host with CPU, memory, disk, and network from the machine agent's SIM data. + +The AppD side of both APM and infrastructure flows the same way it did in Phase 3. Swapping the OTel runtime did not touch AppDynamics. + +## Verify Splunk APM + +1. Open [Splunk Observability Cloud](https://app.signalfx.com). +2. Click **APM** in the left navigation panel. +3. In the **Environment** dropdown, select `-appd-dual`. The `resource/workshop` processor in the workshop config stamps that environment regardless of which collector substrate is running it. + +![AppDynamics Application](../../_images/o11y-service.png?width=30vw) + +{{% notice title="Give it a few minutes" style="info" icon="info-circle" %}} +Allow 2 to 5 minutes after the standalone collector starts for any in-flight queue to drain and the new pipeline to fully populate. +{{% /notice %}} + +You should still see **OrderService** in the service list, with traces continuous from Phase 3. + +## Explore a Trace + +1. Click on the **OrderService** service. +2. Click **Traces** to view individual traces. +3. Select a trace for `GET /order` to open the trace detail waterfall. + +![APM waterall](../../_images/waterfall.png) + +## Confirm the AppDynamics Correlation Attributes + +Click on a **root span** and look at the span attributes. The AppDynamics correlation attributes are still there because they are added by the AppD agent itself, independent of which collector forwards the span: + +| Attribute | Example value | +|---|---| +| `appd.app.name` | `Dual-Ingest-YOURINITIALS` | +| `appd.tier.name` | `OrderService` | +| `appd.bt.name` | `/order` or `/inventory` | +| `appd.request.guid` | The AppDynamics request GUID | + +{{% notice title="Key insight" style="primary" icon="lightbulb" %}} +The `appd.tier.name` attribute also appears on spans in the middle of a trace whenever the tier changes. In a multi-tier application, each span carries the correct AppDynamics tier identity. +{{% /notice %}} + +## Verify Splunk Infrastructure + +1. Click **Metric Finder** in the left navigation panel. +2. Search for `cpu.utilization` and open the chart. +3. Filter on `host.name=${INSTANCE}`. + +You should see a continuous time series. The data path is now the standalone collector's `hostmetrics` receiver, exporting via `signalfx`. The `resource/workshop` processor still stamps `deployment.environment=${INSTANCE}-appd-dual`, same as Phase 3. + +## What Got Better Versus Phase 3 + +Now look at the things that hurt in Phase 3 and confirm the standalone install handled them. + +### Environment variables are persistent + +```bash +sudo cat /etc/otel/collector/splunk-otel-collector.conf +``` + +Every `SPLUNK_*` variable you exported by hand in Phase 3 is now in this file. `systemd` reads it on every restart and reboot. No more shell-rc plumbing. + +### The Smart Agent process list works + +The install script shipped the bundle at `/usr/lib/splunk-otel-collector/agent-bundle`. The workshop config's `smartagent/processlist` receiver now finds it, scrapes process data, and exports it through `signalfx`. In Splunk Observability Cloud, open **Infrastructure > Hosts**, click your host, and the **Top processes** view should populate within a couple of minutes. In Phase 3 that view stayed empty. + +### Lifecycle is independent + +```bash +sudo systemctl status splunk-otel-collector --no-pager +sudo journalctl -u splunk-otel-collector -n 20 --no-pager +``` + +The collector is its own service. Restart it, reload its config, or upgrade it without touching the AppDynamics machine agent and vice versa. AppD agent upgrades cannot overwrite `/etc/otel/collector/agent_config.yaml` because it is not in their install tree. + +### Update cadence is Splunk's + +`apt list --installed 2>/dev/null | grep splunk-otel-collector` shows the version. Future upgrades come through the same package channel on the Splunk OTel Collector release cadence rather than waiting for the next AppDynamics machine agent release. + +## The Path Forward + +You walked the workshop from AppDynamics-only, through infrastructure dual signal, through "yes you can run the workshop config inside AppD's bundled collector if you accept the friction", and finally to the standalone install we recommend for production. + +The next step on this path is OTel-native: drop the AppDynamics agent on a service, instrument it with, eBPF (OBI), or the OpenTelemetry Java auto-instrumentation directly, and let the standalone Splunk OTel Collector you just installed pick the data up unchanged. The `appd.*` correlation attributes go away on those services because there is no AppD agent to add them, but everything else in the pipeline (resource processors, high-cardinality trim, host metrics, profiling, log forwarding) keeps working with no changes. + +For services where AppDynamics still earns its place (SIM, the L1/L2 mental model, business transaction analytics), dual mode and the standalone collector are how you keep both teams happy while the migration runs. + +Move on to wire up global data links so any Splunk trace gives you a one-click jump back into the matching AppDynamics view. diff --git a/content/en/ninja-workshops/17-appd-ingest/5-standalone-otel-collector/_index.md b/content/en/ninja-workshops/17-appd-ingest/5-standalone-otel-collector/_index.md new file mode 100644 index 0000000000..33bdf6e0af --- /dev/null +++ b/content/en/ninja-workshops/17-appd-ingest/5-standalone-otel-collector/_index.md @@ -0,0 +1,42 @@ +--- +title: "Phase 4: Standalone Splunk OpenTelemetry Collector" +linkTitle: 5. Standalone OTel Collector +weight: 5 +archetype: chapter +time: 15 minutes +description: Take the same workshop collector config you used in Phase 3 and run it under the standalone Splunk OpenTelemetry Collector install. Get every environment variable, the Smart Agent monitor bundle, and `systemd`-managed lifecycle for free, and move one step closer to an OTel-native footprint. +--- + +In the previous phase you proved you can run the workshop collector config inside the AppDynamics machine agent. You also tallied the cost: eight environment variables in your shell, a Smart Agent receiver that does nothing because the bundle is not on disk, and an OTel collector lifecycle glued to the AppD machine agent's process. + +This phase swaps the runtime substrate underneath. The same workshop `collector-config.yaml` runs under the **standalone Splunk Distribution of the OpenTelemetry Collector**, installed via the official install script. The config is unchanged. The Java app stays in dual mode. The transitions are all operational. + +Here is what changes: + +| Aspect | Phase 3 (bundled collector path) | Phase 4 (standalone install) | +|---|---|---| +| Where the OTel binary lives | `~/workshop/appd/machine-agent/otel-collector/bin/otelcol_linux_amd64` | `/usr/lib/splunk-otel-collector/bin/otelcol` | +| Who starts it | The AppDynamics machine agent launcher | `systemd` | +| Config location | `~/workshop/appd/machine-agent/conf/otel/agent_config.yaml` (inside the AppD install tree, can be overwritten on AppD upgrade) | `/etc/otel/collector/agent_config.yaml` (independent of AppD) | +| Env vars | Manually exported in your shell | Written once into `/etc/otel/collector/splunk-otel-collector.conf` by the install script | +| Smart Agent monitor bundle | Not present, `smartagent/processlist` silently does nothing | Shipped at `/usr/lib/splunk-otel-collector/agent-bundle`, `smartagent/processlist` works | +| Lifecycle | Stops when the machine agent stops | `systemctl restart splunk-otel-collector` independent of AppD | +| Update cadence | Tracks AppD machine agent releases | Tracks the Splunk OTel Collector release cadence | +| Diagnosis | `tail -f machine-agent/logs/otel/otel-collector.log` | `journalctl -u splunk-otel-collector -f` | +| AlwaysOn Profiling and call graphs | Bundled config has no logs pipeline; would need additional manual setup | Workshop config already includes the `splunk_hec/profiling` exporter | + +In this phase you will: + +1. Disable the bundled OTel side of the AppDynamics machine agent (set `SPLUNK_OTEL_ENABLED=false`) so the bundled collector stops, while the machine agent keeps reporting infrastructure to AppDynamics Server Visibility. +2. Install the standalone Splunk OTel Collector with the official script and apply the same workshop `collector-config.yaml` you used in Phase 3. +3. Verify that APM traces, host metrics, and the Smart Agent process list all flow under the new substrate, and check what was painful in Phase 3 is now handled by the install. + +{{% notice title="Why would we keep the AppDynamics machine agent running?" style="info" icon="info-circle" %}} +When you still require **AppDynamics Server Visibility** to receive host metrics for the AppD side of dual signal. The machine agent process is what feeds Server Visibility. We just turn off its bundled OTel side so the standalone collector owns the OTel pipeline cleanly. But in the future the agent could be fully removed from the system. +{{% /notice %}} + +{{% notice title="Skipped Phase 3" style="info" icon="info-circle" %}} +If you came here directly from Phase 2 without doing Phase 3 (not using the AppD Machine Agent), the Java app is still in single signal mode and only sending to AppDynamics. After you finish the install steps below, restart the Java app with the dual mode flags shown in [Phase 3 Step 2](../4-customize-bundled-collector/2-enable-dual-mode/) so it begins emitting OTLP into the new collector. +{{% /notice %}} + +This is the configuration we recommend for production observability and the natural next step on the path toward dropping the AppDynamics agent entirely and running an OTel-native footprint when you are ready. diff --git a/content/en/ninja-workshops/17-appd-ingest/5-wrap-up/_index.md b/content/en/ninja-workshops/17-appd-ingest/5-wrap-up/_index.md deleted file mode 100644 index 78a5bc4423..0000000000 --- a/content/en/ninja-workshops/17-appd-ingest/5-wrap-up/_index.md +++ /dev/null @@ -1,52 +0,0 @@ ---- -title: Wrap Up -linkTitle: 5. Wrap Up -weight: 5 -archetype: chapter -time: 5 minutes -description: Summary, cleanup, and next steps. ---- - -## What You Accomplished - -In this workshop you: - -1. **Built and ran a Java service with normal AppDynamics instrumentation**: a single agent sending APM data to the AppDynamics Controller only. - -2. **Learned the difference between hybrid and dual signal mode**: hybrid reuses AppD's own instrumentation to generate OTel spans (lower overhead, narrower coverage), while dual runs the full OTel Java auto-instrumentation alongside AppD (broader coverage, adds correlation attributes). - -3. **Enabled dual signal mode**: by adding four JVM flags to the same process. No code changes, no second agent, no recompilation. The same AppDynamics agent now sends data to both AppDynamics and Splunk Observability Cloud simultaneously. - -4. **Created a global data link**: in Splunk Observability Cloud that uses the `appd.*` span attributes to navigate directly to the corresponding AppDynamics tier view. - -## Cleanup - -Stop the application and load generator: - -```bash -kill %2 2>/dev/null # load generator -kill %1 2>/dev/null # java app -``` - -Optionally stop the collector: - -```bash -sudo systemctl stop splunk-otel-collector -``` - -## Key Takeaways - -- **Dual mode is a configuration change, not a code change.** You enabled it by adding JVM flags to an already-instrumented application. This makes it practical to roll out across an organization without touching application code. - -- **The `appd.*` correlation attributes are what make the integration valuable.** Without them (hybrid mode), you get OTel traces in Splunk O11y but no way to link back to the specific AppDynamics business transaction, tier, or application. Dual mode provides that linkage. - -- **Global data links turn correlation into workflow.** Instead of manually cross-referencing two tools, engineers can click from a Splunk O11y trace directly into the AppDynamics view - -- **This pattern supports gradual migration.** Organizations can run dual mode for a period to validate that Splunk Observability Cloud captures the same signal quality, then decide per-service whether to continue dual, switch to Splunk-only instrumentation, or stay with AppDynamics. - -## Further Reading - -- [Enable Dual Signal Mode](https://help.splunk.com/en/appdynamics-on-premises/virtual-appliance-self-hosted/25.7.0/splunk-appdynamics-for-opentelemetry/instrument-applications-with-splunk-appdynamics-for-opentelemetry/enable-opentelemetry-in-the-java-agent/enable-dual-signal-mode) (AppDynamics docs) -- [Enable Hybrid Mode](https://help.splunk.com/en/appdynamics-on-premises/virtual-appliance-self-hosted/25.7.0/splunk-appdynamics-for-opentelemetry/instrument-applications-with-splunk-appdynamics-for-opentelemetry/enable-opentelemetry-in-the-java-agent/enable-hybrid-mode) (AppDynamics docs) -- [Java Agent Frameworks for OpenTelemetry](https://help.splunk.com/en/appdynamics-on-premises/virtual-appliance-self-hosted/25.7.0/splunk-appdynamics-for-opentelemetry/support-for-appdynamics-for-opentelemetry/java-agent-frameworks-for-opentelemetry) (Supported framework list) -- [Global Data Links](https://docs.splunk.com/observability/en/data-visualization/navigate-with-data-links.html) (Splunk Observability docs) diff --git a/content/en/ninja-workshops/17-appd-ingest/4-global-data-links/_index.md b/content/en/ninja-workshops/17-appd-ingest/6-global-data-links/_index.md similarity index 98% rename from content/en/ninja-workshops/17-appd-ingest/4-global-data-links/_index.md rename to content/en/ninja-workshops/17-appd-ingest/6-global-data-links/_index.md index 65fe22e5b2..e66b813301 100644 --- a/content/en/ninja-workshops/17-appd-ingest/4-global-data-links/_index.md +++ b/content/en/ninja-workshops/17-appd-ingest/6-global-data-links/_index.md @@ -1,7 +1,7 @@ --- -title: "Phase 3: Global Data Links" -linkTitle: 4. Global Data Links -weight: 4 +title: "Phase 5: Global Data Links" +linkTitle: 6. Global Data Links +weight: 6 archetype: chapter time: 10 minutes description: Create a global data link in Splunk Observability Cloud that navigates directly to the corresponding AppDynamics tier view using the appd.* span attributes. diff --git a/content/en/ninja-workshops/17-appd-ingest/7-wrap-up/_index.md b/content/en/ninja-workshops/17-appd-ingest/7-wrap-up/_index.md new file mode 100644 index 0000000000..b69c4ce900 --- /dev/null +++ b/content/en/ninja-workshops/17-appd-ingest/7-wrap-up/_index.md @@ -0,0 +1,70 @@ +--- +title: Wrap Up +linkTitle: 7. Wrap Up +weight: 7 +archetype: chapter +time: 5 minutes +description: Summary, cleanup, and next steps. +--- + +## What You Accomplished + +You walked an AppDynamics shop through a four-step progression that gets progressively more capable and progressively closer to an OTel-native footprint: + +1. **Phase 1: Built and ran a Java service with normal AppDynamics instrumentation.** A single Java agent sending APM data to the AppDynamics Controller. The starting point most AppDynamics customers run today. + +2. **Phase 2: Added infrastructure dual signal with the AppDynamics Combined Machine Agent.** One environment variable (`SPLUNK_OTEL_ENABLED=true`) flipped the launcher into combined mode and the bundled OTel collector forwarded host metrics to Splunk Observability Cloud while the machine agent kept reporting to AppDynamics Server Visibility. The lowest-effort way to put hosts in both platforms. + +3. **Phase 3: Customized the bundled collector with the workshop config and enabled dual signal mode on the Java agent.** You proved you can run a real production-ish OTel config inside the AppD machine agent's bundled collector and saw exactly what it costs: eight extra `SPLUNK_*` environment variables, a Smart Agent monitor bundle that is not on disk, and an OTel collector lifecycle glued to the AppDynamics machine agent. + +4. **Phase 4: Replaced the bundled collector with the standalone Splunk OpenTelemetry Collector.** Same workshop config, but the install script wires every environment variable into `/etc/otel/collector/splunk-otel-collector.conf`, ships the Smart Agent bundle, and runs the collector as its own `systemd` service. Independent lifecycle, independent upgrade cadence, the configuration we recommend for production observability. + +5. **Learned the difference between hybrid and dual signal mode on the Java agent.** Hybrid reuses AppD's own instrumentation to emit OTel spans (lower overhead, narrower coverage). Dual runs the full OTel Java auto-instrumentation alongside AppD (broader coverage, adds the `appd.*` correlation attributes that make the next step possible). + +6. **Created a global data link** in Splunk Observability Cloud that uses the `appd.*` span attributes to navigate directly to the corresponding AppDynamics tier view. + +## Cleanup + +Stop the load generator, the Java app, the machine agent, and the standalone collector: + +```bash +kill %2 2>/dev/null # load generator +kill %1 2>/dev/null # java app + +cd ~/workshop/appd/machine-agent +kill "$(cat machine-agent.pid)" 2>/dev/null || true +rm -f machine-agent.pid + +sudo systemctl stop splunk-otel-collector +sudo systemctl disable splunk-otel-collector +``` + +If you want to leave the bundled config swap in place for later experiments, restore the original AppD-shipped config: + +```bash +cd ~/workshop/appd/machine-agent +cp conf/otel/agent_config.yaml.appd-default conf/otel/agent_config.yaml +``` + +## Key Takeaways + +- **Dual mode is a configuration change, not a code change.** You enabled it by adding JVM flags to an already-instrumented application. This makes it practical to roll out across an organization without touching application code. + +- **The AppD bundle is not the Splunk OTel Collector, but the binaries are interchangeable.** The OTel binary inside `~/workshop/appd/machine-agent/otel-collector/bin/otelcol_linux_amd64` is the Splunk Distribution. AppD packages and updates it on its own cadence and ships it with a metrics-only default config. You can absolutely point it at a richer config (Phase 3), but everything that the standalone install gives you for free becomes manual work. + +- **The standalone Splunk OTel Collector is the configuration we ship to production.** Phase 4 demonstrated why: persistent env vars, the Smart Agent bundle on disk, `systemd`-managed lifecycle, and a release cadence Splunk controls. None of that is impossible to recreate inside the AppD bundle, but the cost piles up quickly. + +- **The `appd.*` correlation attributes are what make the integration valuable.** Without them (hybrid mode), you get OTel traces in Splunk O11y but no way to link back to the specific AppDynamics business transaction, tier, or application. Dual mode provides that linkage. + +- **Global data links turn correlation into workflow.** Engineers click from a Splunk O11y trace directly into the AppDynamics view instead of cross-referencing two tools by hand. + +- **This pattern supports gradual migration toward OTel-native.** Run Phase 4 to validate that Splunk Observability Cloud captures the same signal quality. Then for new services, skip the AppDynamics agent entirely, use the OpenTelemetry Java SDK or auto-instrumentation, and let the standalone collector pick the data up unchanged. AppDynamics stays where it earns its place; everything else moves. + +## Further Reading + +- [Enable Dual Signal Mode](https://help.splunk.com/en/appdynamics-on-premises/virtual-appliance-self-hosted/25.7.0/splunk-appdynamics-for-opentelemetry/instrument-applications-with-splunk-appdynamics-for-opentelemetry/enable-opentelemetry-in-the-java-agent/enable-dual-signal-mode) (AppDynamics docs) +- [Enable Hybrid Mode](https://help.splunk.com/en/appdynamics-on-premises/virtual-appliance-self-hosted/25.7.0/splunk-appdynamics-for-opentelemetry/instrument-applications-with-splunk-appdynamics-for-opentelemetry/enable-opentelemetry-in-the-java-agent/enable-hybrid-mode) (AppDynamics docs) +- [Java Agent Frameworks for OpenTelemetry](https://help.splunk.com/en/appdynamics-on-premises/virtual-appliance-self-hosted/25.7.0/splunk-appdynamics-for-opentelemetry/support-for-appdynamics-for-opentelemetry/java-agent-frameworks-for-opentelemetry) (Supported framework list) +- [Combined Agent for Infrastructure Visibility](https://help.splunk.com/en/appdynamics-on-premises/infrastructure-visibility/26.4.0/machine-agent/combined-agent-for-infrastructure-visibility) (AppDynamics docs for the bundled OTel collector) +- [Splunk Distribution of the OpenTelemetry Collector](https://docs.splunk.com/observability/en/gdi/opentelemetry/opentelemetry.html) (Splunk docs) +- [Global Data Links](https://docs.splunk.com/observability/en/data-visualization/navigate-with-data-links.html) (Splunk Observability docs) diff --git a/content/en/ninja-workshops/17-appd-ingest/_images/O11y-OrderService-Overview.png b/content/en/ninja-workshops/17-appd-ingest/_images/O11y-OrderService-Overview.png new file mode 100644 index 0000000000..30e5291c39 Binary files /dev/null and b/content/en/ninja-workshops/17-appd-ingest/_images/O11y-OrderService-Overview.png differ diff --git a/content/en/ninja-workshops/17-appd-ingest/_images/appd-machine-agent-metrics.png b/content/en/ninja-workshops/17-appd-ingest/_images/appd-machine-agent-metrics.png new file mode 100644 index 0000000000..152c15013b Binary files /dev/null and b/content/en/ninja-workshops/17-appd-ingest/_images/appd-machine-agent-metrics.png differ diff --git a/content/en/ninja-workshops/17-appd-ingest/_images/o11y-Infra-Correlation.png b/content/en/ninja-workshops/17-appd-ingest/_images/o11y-Infra-Correlation.png new file mode 100644 index 0000000000..5f32a3d26a Binary files /dev/null and b/content/en/ninja-workshops/17-appd-ingest/_images/o11y-Infra-Correlation.png differ diff --git a/content/en/ninja-workshops/17-appd-ingest/_images/o11y-apm.png b/content/en/ninja-workshops/17-appd-ingest/_images/o11y-apm.png new file mode 100644 index 0000000000..ca7eb84ac7 Binary files /dev/null and b/content/en/ninja-workshops/17-appd-ingest/_images/o11y-apm.png differ diff --git a/content/en/ninja-workshops/17-appd-ingest/_images/o11y-appd-attributes.png b/content/en/ninja-workshops/17-appd-ingest/_images/o11y-appd-attributes.png new file mode 100644 index 0000000000..c8ea8b2ade Binary files /dev/null and b/content/en/ninja-workshops/17-appd-ingest/_images/o11y-appd-attributes.png differ diff --git a/content/en/ninja-workshops/17-appd-ingest/_images/o11y-infra-metrics-appd-machine-agent.png b/content/en/ninja-workshops/17-appd-ingest/_images/o11y-infra-metrics-appd-machine-agent.png new file mode 100644 index 0000000000..138363bd1e Binary files /dev/null and b/content/en/ninja-workshops/17-appd-ingest/_images/o11y-infra-metrics-appd-machine-agent.png differ diff --git a/content/en/ninja-workshops/17-appd-ingest/_index.md b/content/en/ninja-workshops/17-appd-ingest/_index.md index 93dd2ee482..f03da7239b 100644 --- a/content/en/ninja-workshops/17-appd-ingest/_index.md +++ b/content/en/ninja-workshops/17-appd-ingest/_index.md @@ -5,9 +5,9 @@ title: AppDynamics Dual Ingest to Splunk Observability linkTitle: AppD Dual Ingest weight: 17 archetype: chapter -time: 45 minutes +time: 75 minutes authors: ["Jeremy Hicks"] -description: A hands-on workshop demonstrating how to use the AppDynamics Java Agent's dual signal mode to send APM data to both AppDynamics and Splunk Observability Cloud simultaneously, then connect the two with global data links. +description: A hands-on workshop covering the AppDynamics Java Agent's dual signal mode for APM and the AppDynamics Combined Machine Agent for infrastructure, sending both signals to AppDynamics and Splunk Observability Cloud from a single bundle, then connecting the two platforms with global data links. --- Learn about Dual Ingest with the AppDynamics Java agent and how to send data to both AppDynamics and Splunk Observability Cloud \ No newline at end of file diff --git a/workshop/appd/download-appd-agent.sh b/workshop/appd/download-appd-agent.sh index 51cb749d98..294d45eada 100755 --- a/workshop/appd/download-appd-agent.sh +++ b/workshop/appd/download-appd-agent.sh @@ -1,16 +1,15 @@ - #!/bin/bash ####################################################################### -# This script will fetch the latest AppDynamics Sun/Oracle Java agent # -# and place it in the current directory under /appdynamics. # -# This script requires: # -# curl # -# unzip # -# jq # +# Fetch the latest AppDynamics agent (Java or Machine) and extract # +# it under ./agent or ./machine-agent. Defaults to the Java agent. # +# Usage: # +# ./download-appd-agent.sh # Java agent (default) # +# ./download-appd-agent.sh java # Java agent # +# ./download-appd-agent.sh machine # Machine agent (Linux x64) # +# Requires: curl, unzip, jq # ####################################################################### -# Function to check if a command is available and install it if not check_command() { if ! command -v "$1" > /dev/null 2>&1; then echo "$1 is not installed. Installing..." @@ -21,42 +20,50 @@ check_command() { fi } -# Check and install required commands check_command curl check_command unzip check_command jq -# Create the directory for the agent -INSTALL_DIR="./agent" -mkdir -p "$INSTALL_DIR" +AGENT_TYPE="${1:-java}" -# Download Root URL -DOWNLOAD_PATH="https://download-files.appdynamics.com/" +case "$AGENT_TYPE" in + java) + INSTALL_DIR="./agent" + PATTERN="java-jdk8" + ZIP_NAME="JavaAgent.zip" + ;; + machine) + INSTALL_DIR="./machine-agent" + # The trailing [0-9] excludes the aarch64 build (whose name continues with letters). + PATTERN="machineagent-bundle-64bit-linux-[0-9]" + ZIP_NAME="MachineAgent.zip" + ;; + *) + echo "Usage: $0 [java|machine]" >&2 + exit 1 + ;; +esac -# Fetch latest Sun Java Agent download path from AppD -FILE_PATH=$(curl -s https://download.appdynamics.com/download/downloadfilelatest/ | jq -r '.[].s3_path' | grep java-jdk8) +mkdir -p "$INSTALL_DIR" -# Construct the full URL -DOWNLOAD_PATH=$DOWNLOAD_PATH$FILE_PATH +DOWNLOAD_ROOT="https://download-files.appdynamics.com/" -# Print URL to stdout -echo "Downloading agent from: $DOWNLOAD_PATH" +FILE_PATH=$(curl -s https://download.appdynamics.com/download/downloadfilelatest/ \ + | jq -r '.[].s3_path' \ + | grep -E "$PATTERN" \ + | head -1) -# Fetch agent and write into working directory -curl -L $DOWNLOAD_PATH -o ./JavaAgent.zip +if [ -z "$FILE_PATH" ]; then + echo "Could not find a matching $AGENT_TYPE agent download." >&2 + exit 1 +fi -# Unzip the agent into the installation directory -unzip ./JavaAgent.zip -d "$INSTALL_DIR" +DOWNLOAD_PATH="${DOWNLOAD_ROOT}${FILE_PATH}" -# Pass in the custom interceptors file -# cp ./custom-interceptors.xml "$INSTALL_DIR"/ver*/conf +echo "Downloading agent from: $DOWNLOAD_PATH" -# Pass in the custom correlation file -# cp ./custom-activity-correlation.xml "$INSTALL_DIR"/ver*/conf +curl -L "$DOWNLOAD_PATH" -o "./$ZIP_NAME" -# Pass in custom log4j, log4j2 config file -# cp ./log4j.xml "$INSTALL_DIR"/ver*/conf/logging -# cp ./log4j2.xml "$INSTALL_DIR"/ver*/conf/logging +unzip -o "./$ZIP_NAME" -d "$INSTALL_DIR" -# Remove the zip -rm -f ./JavaAgent.zip \ No newline at end of file +rm -f "./$ZIP_NAME"