Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
113 changes: 45 additions & 68 deletions content/en/ninja-workshops/17-appd-ingest/1-overview/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,43 +11,73 @@ description: Use case, architecture, prerequisites, and the differences between

Your organization runs AppDynamics for APM today. As part of a data visibility and governance initiative, leadership wants application performance data flowing into **Splunk Observability Cloud** as well giving teams a unified view alongside infrastructure metrics, logs, and other signals already in Splunk.

Rather than re-instrumenting every service with a separate OpenTelemetry SDK, the AppDynamics Java Agent supports **dual signal mode**: a single agent produces both AppDynamics APM data and OpenTelemetry traces simultaneously. This lets you maintain full AppDynamics functionality while streaming the same telemetry to Splunk Observability Cloud through an OpenTelemetry Collector.
Rather than re-instrumenting every service with a separate OpenTelemetry SDK, the **AppDynamics Java Agent** supports **dual signal mode**: a single agent produces both AppDynamics APM data and OpenTelemetry traces simultaneously. This lets you maintain full AppDynamics functionality while streaming the same telemetry to Splunk Observability Cloud through an OpenTelemetry Collector.

This is especially helpful for your current L1 and L2 teams who currently know and rely on AppDynamics. Dual ingest helps maintain context as the applications and services they are responsible become more connected to new services hosted in SaaS platforms in the cloud.

## What You'll Learn

By the end of this workshop, you will:

- Build and run a simple Java service with the AppDynamics Java Agent
- Understand the difference between **hybrid mode** and **dual signal mode**
- Enable dual signal mode to send APM data to both AppDynamics and Splunk Observability Cloud
- Verify traces and metrics in both platforms
- Create a **global data link** in Splunk Observability Cloud for one-click navigation to AppDynamics
1. Build and run a simple Java service with the AppDynamics Java Agent
2. Add infrastructure dual signal with the **AppDynamics Combined Machine Agent**, the lowest-effort step to get hosts into both AppDynamics and Splunk Observability Cloud
- Understand the difference between **hybrid mode** and **dual signal mode** on the Java agent
3. Customize the bundled OTel collector inside the AppDynamics machine agent with the workshop config and see how much additional configuration is required
4. Replace the bundled collector with the **standalone Splunk OpenTelemetry Collector** (the configuration we recommend for production observability) and watch that friction disappear
- Verify traces and metrics in both platforms
5. Create a **global data link** in Splunk Observability Cloud for one-click navigation to AppDynamics

## Architecture

In this workshop you will run a Spring Boot Java application directly on your EC2 instance. The AppDynamics Java Agent attaches to the JVM process.
In this workshop you will run a Spring Boot Java application directly on your EC2 instance. The AppDynamics Java Agent attaches to the JVM process. Each phase changes one part of the picture.

**Phase 1: Normal AppD instrumentation:**

```text
Java App + AppD Agent ──▶ AppD Controller
```

**Phase 2: Dual signal mode enabled:**
**Phase 2: Combined Machine Agent with the bundled config (infrastructure dual signal):**

```text
Java App + AppD Agent ──▶ AppD Controller (AppD protocol, unchanged)
──▶ OTel Collector (OTLP on localhost:4317)
Splunk Observability Cloud
Java App + AppD Agent ──▶ AppD Controller (APM, unchanged)

Machine Agent (combined) ──▶ AppD Controller (Server Visibility)
└─▶ Bundled OTel Collector ──▶ Splunk Observability Cloud (hostmetrics)
(config: agent_config.yaml, AppD default, metrics-only)
```

The AppDynamics Combined Machine Agent ships with a bundled OTel collector built in (this is **not** a fully-configured Splunk OTel Collector, just the Splunk Distribution binary AppD packages and updates on its own cadence). The default config is metrics-only. APM still flows only to AppDynamics.

**Phase 3: Bundled collector running the workshop config (APM dual signal, the hard way):**

```text
Java App + AppD Agent (dual) ──▶ AppD Controller (APM, unchanged)
──▶ localhost:4318 (OTLP)
Machine Agent (combined) │
└─▶ Bundled OTel Collector ◀───────────┘
(config: workshop collector-config.yaml in place of agent_config.yaml)
(env vars: SPLUNK_INGEST_URL, SPLUNK_API_URL, SPLUNK_HEC_*, SPLUNK_BUNDLE_DIR, ... exported by hand)
──▶ Splunk Observability Cloud (APM + Hosts + Logs, mostly)
```

Same Splunk Distribution binary as Phase 4, just running inside the AppDynamics install. You manually export every `SPLUNK_*` env var the workshop config needs, and the Smart Agent monitor bundle is missing because AppD does not ship it. The path works for APM and host metrics, the friction is the lesson.

**Phase 4: Standalone Splunk OTel Collector (Migration Phase):**

```text
Java App + AppD Agent (dual)
──▶ AppD Controller (APM, unchanged)
──▶ localhost:4318 (OTLP)
|
└─▶ Splunk OTel Collector (systemd-managed)
──▶ Splunk Observability Cloud (APM + Hosts + Logs + Profiling + Process List)
```

The OpenTelemetry Collector runs on the same EC2 instance, receives OTLP from the agent, and exports traces and metrics to Splunk Observability Cloud.
Same workshop config as Phase 3, run by a `systemd`-managed standalone install. Every `SPLUNK_*` env var lives in `/etc/otel/collector/splunk-otel-collector.conf, and the collector lifecycle is independent of the AppD machine agent. This is the configuration we recommend for production observability and the natural step toward an OTel-native footprint.

**Note:** It is possible to send data directly from the agent to our [OTLP ingest endpoint](https://dev.splunk.com/observability/docs/datamodel/ingest/#Send-data-points) without a collector but you may lose some attributes and association taking part in your OTel config.
**Note:** It is possible to send data directly from the Java agent to our [OTLP ingest endpoint](https://dev.splunk.com/observability/docs/datamodel/ingest/#Send-data-points) without a collector, but you lose the resource processors, high-cardinality trim, and other pipeline configuration the workshop config provides.

## Hybrid Mode vs Dual Signal Mode

Expand Down Expand Up @@ -83,59 +113,6 @@ Dual signal mode adds **correlation attributes** to root spans that hybrid mode

These attributes enable **global data links** clickable links on Splunk Observability Cloud traces that navigate directly to the corresponding AppDynamics view. Additionally, AppDynamics snapshots captured in dual mode include the OTel `TraceId` in the Data Collectors tab, enabling navigation in both directions.

## OBI vs Traditional Zero-Code Instrumentation

OBI and existing traditional language-specific zero-code instrumentation (Java, JS, .NET, Python, Go, PHP) serve complementary roles in an observability strategy. Understanding the differences helps determine when to use each approach.

### 1. Instrumentation Model

| Aspect | OBI | Traditional Zero-Code Instrumentations |
|---|---|---|
| Execution model | Out-of-process | In-process |
| Instrumentation layer | Linux kernel / network | Application runtime |
| Requires code changes | No | No or minimal |
| Requires application restart | No | Yes |
| Security profile | Isolated | Same permission profile as the application |

### 2. Level of Visibility

| Capability | OBI | Traditional Zero-Code Instrumentations |
|---|---|---|
| Distributed tracing | Protocol-level | Full-fidelity |
| RED metrics | Yes | Yes |
| Application logs collection | No | Yes |
| Application log-trace correlation | Yes | Yes |
| Application internals (frameworks, functions) | No (partial, mostly in Go) | Yes |
| Custom spans / business attributes | No | Yes |
| Runtime metrics (JVM, memory, threads) | No, for now | Yes |

### 3. Coverage & Compatibility

| Scenario | OBI | Traditional Zero-Code Instrumentations |
|---|---|---|
| Multi-language environments | Strong (protocol-based) | Language-specific |
| Third-party applications | Supported | Limited, contrib repos |
| Legacy systems | Supported | Limited |
| Compiled languages (C/C++/Rust) | Supported (with some limitations in async) | Limited |
| Async / complex frameworks | Limited in some cases | Strong |

### 4. Operational Characteristics

| Aspect | OBI | Traditional Zero-Code Instrumentations |
|---|---|---|
| Deployment effort | Low (drop-in) | Medium (agent attachment) |
| Time to first visibility | Minutes | "More" minutes |
| Changes to application lifecycle | No | Yes |
| Performance overhead | Minimal and isolated | Varies by language/runtime |

### 5. Splunk Distro Features

| Feature | OBI | Traditional Zero-Code Instrumentations |
|---|---|---|
| Always-on Profiling | No (may be bundled with eBPF profiler in future) | CPU for most, Memory for some |
| Call graphs | No | CPU for most, Memory for some |
| File-based configuration | Incoming | Java, Node.js, .NET, Python (incoming) |
| No-code instrumentation | N/A | Yes |

## Prerequisites

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,12 @@ Replace `<YOUR-ACCESS-KEY>` with the AppDynamics token from the previous step:
{{< tabs >}}
{{% tab title="Command" %}}
Export your environment variables
Use your AppD key you obtained previously
```bash
export APPD_ACCESS_KEY=<Your-AppDynamics-access-key>
```
Then export the appd app name with your unique instance identifier
```bash
export APPD_APP_NAME=Dual-Ingest-${INSTANCE}
```

Expand Down

This file was deleted.

This file was deleted.

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
---
title: 1. Download the Machine Agent
weight: 1
---

The same `download-appd-agent.sh` script you used to fetch the Java agent in Phase 1 takes an optional argument. Pass `machine` to fetch the latest 64-bit Linux machine agent bundle (with a bundled JRE and a bundled OTel collector binary).

## Run the Download Script

{{< tabs >}}
{{% tab title="Command" %}}

```bash
cd ~/workshop/appd
./download-appd-agent.sh machine
```

{{% /tab %}}
{{% tab title="Example Output" %}}

```text
Downloading agent from: https://download-files.appdynamics.com/download-file/machine-bundle/26.x.y.zzzz/machineagent-bundle-64bit-linux-26.x.y.zzzz.zip
Archive: ./MachineAgent.zip
inflating: machine-agent/machineagent.jar
inflating: machine-agent/conf/controller-info.xml
...
```

{{% /tab %}}
{{< /tabs >}}

The bundle unpacks into `~/workshop/appd/machine-agent/`. The pieces you will touch in the next page:

| Path | What it is |
|---|---|
| `bin/machine-agent` | Launcher script. Knows how to start both the Java machine agent and the bundled OTel collector. |
| `conf/controller-info.xml` | AppDynamics Controller settings (we will pass these via `-D` flags instead, so we do not need to edit it). |
| `conf/otel/agent_config.yaml` | Default OTel Collector config. Hostmetrics + signalfx exporter, which is exactly what we need for this phase. We use it as-is. |
| `otel-collector/bin/otelcol_linux_amd64` | The bundled OTel Collector binary. |
| `jre/bin/java` | Bundled JRE so you do not need a system JDK. |

{{% notice title="Why a bundled JRE" style="info" icon="info-circle" %}}
The machine agent runs as a Java process. Using `jre/bin/java` from the bundle keeps the workshop independent of whatever JDK happens to be on the instance and matches the version AppDynamics qualifies the agent against.
{{% /notice %}}
Loading